Gradient boosted trees (GBT) is an machine learning technique for classification and regression problems which produces a prediction model in the form of an ensemble of weak prediction models, typically decision trees. It builds the model in a stage-wise fashion like other boosting methods do, and it generalizes them by allowing optimization of an arbitrary differentiable loss function. GBT typically demonstrates strong predictive performance and it is used widely in many commercial applications.

The core idea of GBT is to combine weak learners into a single strong learner. It differs from a traditional decision tree algorithm in two key ways:

It builds the trees in a sequential, stage-wise fashion where each successive tree aims to improve upon the previous.

It fit the trees not only on the target but also on the negative gradient of the loss function with respect to the prediction of the previous trees in the ensemble. This is done to directly minimize the loss function.

The algorithm starts with an initial prediction, usually the mean value of the target attribute in the training data (for regression) or the most probable class (for classification). It then builds the trees sequentially as follows:

In the first iteration, it builds a tree that best predicts the negative gradient of the loss function with respect to the initial prediction on the training data. It does so by splitting the training data into regions based on the values of the predictor attributes. Then within each region it fits a simple model (e.g. mean value for regression) and produces a new set of predictions.

In the next iteration, a tree is added to improve upon the existing ensemble by considering the negative gradient of the loss function with respect to the current ensemble’s prediction from the previous iteration. This process continues for a fixed number of iterations or until no further improvement in predictive performance is observed on a validation set.

The process can be summarized as follows:

Fit tree h1(x) to residuals r-1=y-yn=0 where yn=0 is the initial prediction (e.g. mean of y)

Update model: f1(x)=yn=0+h1(x)

Compute residuals: r1=y-f1(x)

Fit tree h2(x) to residuals r1

Update model: f2(x)=f1(x)+h2(x)

Compute residuals: r2=y-f2(x)

Repeat until terminal condition is met.

The predictions of the final additive model are the predictions of the grown trees combined. Importantly, the trees are not pure decision trees but are fit to approximations of the negative gradients – this turns the boosting process into an optimization algorithm that directly minimizes the loss function.

Some key aspects in which GBT can be optimized include:

Number of total trees (or boosting iterations): More trees generally lead to better performance but too many may lead to overfitting. A value between 50-150 is common.

Learning rate: Shrinks the contribution of each tree. Lower values like 0.1 prevent overfitting but require more trees for convergence. It is tuned by validation.

Tree depth: Deeper trees have more flexibility but risk overfitting. A maximum depth of 5 is common but it also needs tuning.

Minimum number of instances required in leaf nodes: Prevents overfitting by not deeply splitting on small subsets of data.

Subsample ratio of training data: Takes a subset for training each tree to reduce overfitting and adds randomness. 0.5-1 is typical.

Column or feature sampling: Samples a subset of features to consider for splits in trees.

Loss function: Cross entropy for classification, MSE for regression. Other options exist but these are most widely used.

Extensive parameter tuning is usually needed due to complex interactions between hyperparmeters. Grid search, random search or Bayesian optimization are commonly applied techniques. The trained model can consist of anywhere between a few tens to a few thousands of trees depending on the complexity of the problem.

Gradient boosted trees rely on the stage-wise expansion of weak learners into an ensemble that directly optimizes a differentiable loss function. Careful hyperparameter tuning is needed to balance accuracy versus complexity for best generalization performance on new data. When implemented well, GBT can deliver state-of-the-art results on a broad range of tasks.