Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Staged Prediction

All fit NGBoost objects support staged prediction.

For instance, to get the predictions on the first 5 examples after fitting 415 base learners, use:

array([0, 0, 1, 1, 1])
{'p0': array([0.99265637, 0.99271317, 0.00378088, 0.00378084, 0.00378084]), 'p1': array([0.00690245, 0.00690285, 0.99598636, 0.99597765, 0.99597765]), 'p2': array([0.00044118, 0.00038399, 0.00023277, 0.0002415 , 0.0002415 ])}

This is useful in conjunction with tracking errors on a validation set, which you can do by passing the X_val and Y_val arguments and then inspecting the .best_val_loss_itr instance attribute.

[iter 0] loss=5.7614 val_loss=5.7625 scale=1.0000 norm=65.4912
[iter 100] loss=5.4182 val_loss=5.5458 scale=1.0000 norm=46.3533
[iter 200] loss=5.1799 val_loss=5.5265 scale=1.0000 norm=36.9395
[iter 300] loss=5.0221 val_loss=5.6125 scale=1.0000 norm=32.8915
[iter 400] loss=4.9100 val_loss=5.7951 scale=1.0000 norm=30.5070
147

Early Stopping

NGBoost also has early stopping. If an integer early_stopping_rounds and a validation set (X_val,Y_val) are passed to fit(), the algorithm will stop running after the validation loss has increased for early_stopping_rounds of consecutive iterations.

[iter 0] loss=5.7614 val_loss=5.7625 scale=1.0000 norm=65.4912
[iter 100] loss=5.4182 val_loss=5.5458 scale=1.0000 norm=46.3533
== Early stopping achieved.
== Best iteration / VAL142 (val_loss=5.5141)

Validation set sample weights can be passed using the val_sample_weight argument to fit.

Overflow Warnings and Diverging Fits

In each iteration, NGBoost does a line search. It tries steps of different sizes along the fitted gradient and keeps a step that makes the training loss smaller. Some trial steps are very large, and the loss for them can overflow. NGBoost rejects these steps, so they do not change the model and do not show warnings.

Two settings can cause real problems:

  • Plain gradients on a target with large units. With natural_gradient=False, the size of the gradient for each parameter depends on the units of Y, and it is different for each parameter. All parameters use the same step size, so the step must be small enough for the largest gradient, and the other parameters change very little. For example, with Normal, CRPScore and a target in the thousands, the predicted mean stays almost at its starting value. Standardize Y before you fit, or keep the default natural_gradient=True. The natural gradient does not depend on the units of Y.

  • Deep trees with very small leaves. A base learner such as DecisionTreeRegressor(max_depth=None, min_samples_leaf=1) can fit each training point exactly. The predicted scale then goes toward 0 on the training data, and the predicted distributions are much too narrow for new data. With a high learning_rate, a predicted parameter can overflow or become 0, and fit stops with an error that says the fit diverged. Use shallower trees or larger leaves, a lower learning_rate, or early stopping with a validation set as shown above.

Using sklearn Model Selection

sklearn methods are compatible with NGBoost.

/opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/sklearn/tree/_classes.py:1347: FutureWarning: Value `"friedman_mse"` for `criterion` is deprecated and will be removed in 1.11. It maps to `"squared_error"` as both were always equivalent. Use `criterion="squared_error"` to remove this warning.
  warnings.warn(
{'Base': DecisionTreeRegressor(max_depth=2), 'minibatch_frac': 0.5}