from ngboost import NGBClassifier, NGBRegressor
from ngboost.distns import k_categorical, Normal
from ngboost.scores import LogScore
from sklearn.datasets import load_breast_cancer, load_diabetes
from sklearn.model_selection import train_test_split
X, Y = load_diabetes(return_X_y=True)
X_reg_train, X_reg_test, Y_reg_train, Y_reg_test = train_test_split(X, Y, test_size=0.2)
X, y = load_breast_cancer(return_X_y=True)
y[0:15] = 2 # artificially make this a 3-class problem instead of a 2-class problem
X_cls_train, X_cls_test, Y_cls_train, Y_cls_test = train_test_split(X, y, test_size=0.2)Staged Prediction¶
All fit NGBoost objects support staged prediction.
ngb_cls = NGBClassifier(Dist=k_categorical(3), Score=LogScore, n_estimators=500, verbose=False).fit(X_cls_train, Y_cls_train)For instance, to get the predictions on the first 5 examples after fitting 415 base learners, use:
preds = ngb_cls.staged_predict(X_cls_test)
preds[415][0:5]array([0, 0, 1, 1, 1])pred_dists = ngb_cls.staged_pred_dist(X_cls_test)
pred_dists[415][0:5].params{'p0': array([0.99265637, 0.99271317, 0.00378088, 0.00378084, 0.00378084]),
'p1': array([0.00690245, 0.00690285, 0.99598636, 0.99597765, 0.99597765]),
'p2': array([0.00044118, 0.00038399, 0.00023277, 0.0002415 , 0.0002415 ])}This is useful in conjunction with tracking errors on a validation set, which you can do by passing the X_val and Y_val arguments and then inspecting the .best_val_loss_itr instance attribute.
ngb = NGBRegressor()
ngb.fit(X_reg_train, Y_reg_train, X_val=X_reg_test, Y_val=Y_reg_test) # use a validation set instead of test set here in your own work
print(ngb.best_val_loss_itr)
best_preds = ngb.predict(X_reg_test, max_iter=ngb.best_val_loss_itr)[iter 0] loss=5.7614 val_loss=5.7625 scale=1.0000 norm=65.4912
[iter 100] loss=5.4182 val_loss=5.5458 scale=1.0000 norm=46.3533
[iter 200] loss=5.1799 val_loss=5.5265 scale=1.0000 norm=36.9395
[iter 300] loss=5.0221 val_loss=5.6125 scale=1.0000 norm=32.8915
[iter 400] loss=4.9100 val_loss=5.7951 scale=1.0000 norm=30.5070
147
Early Stopping¶
NGBoost also has early stopping. If an integer early_stopping_rounds and a validation set (X_val,Y_val) are passed to fit(), the algorithm will stop running after the validation loss has increased for early_stopping_rounds of consecutive iterations.
_ = NGBRegressor().fit(X_reg_train, Y_reg_train, X_val=X_reg_test, Y_val=Y_reg_test, early_stopping_rounds=2)[iter 0] loss=5.7614 val_loss=5.7625 scale=1.0000 norm=65.4912
[iter 100] loss=5.4182 val_loss=5.5458 scale=1.0000 norm=46.3533
== Early stopping achieved.
== Best iteration / VAL142 (val_loss=5.5141)
Validation set sample weights can be passed using the val_sample_weight argument to fit.
Overflow Warnings and Diverging Fits¶
In each iteration, NGBoost does a line search. It tries steps of different sizes along the fitted gradient and keeps a step that makes the training loss smaller. Some trial steps are very large, and the loss for them can overflow. NGBoost rejects these steps, so they do not change the model and do not show warnings.
Two settings can cause real problems:
Plain gradients on a target with large units. With
natural_gradient=False, the size of the gradient for each parameter depends on the units ofY, and it is different for each parameter. All parameters use the same step size, so the step must be small enough for the largest gradient, and the other parameters change very little. For example, withNormal,CRPScoreand a target in the thousands, the predicted mean stays almost at its starting value. StandardizeYbefore you fit, or keep the defaultnatural_gradient=True. The natural gradient does not depend on the units ofY.Deep trees with very small leaves. A base learner such as
DecisionTreeRegressor(max_depth=None, min_samples_leaf=1)can fit each training point exactly. The predicted scale then goes toward 0 on the training data, and the predicted distributions are much too narrow for new data. With a highlearning_rate, a predicted parameter can overflow or become 0, andfitstops with an error that says the fit diverged. Use shallower trees or larger leaves, a lowerlearning_rate, or early stopping with a validation set as shown above.
Using sklearn Model Selection¶
sklearn methods are compatible with NGBoost.
from sklearn.model_selection import GridSearchCV
from sklearn.tree import DecisionTreeRegressor
b1 = DecisionTreeRegressor(criterion='friedman_mse', max_depth=2)
b2 = DecisionTreeRegressor(criterion='friedman_mse', max_depth=4)
param_grid = {
'minibatch_frac': [1.0, 0.5],
'Base': [b1, b2]
}
ngb = NGBRegressor(Dist=Normal, verbose=False)
grid_search = GridSearchCV(ngb, param_grid=param_grid, cv=5)
grid_search.fit(X_reg_train, Y_reg_train)
print(grid_search.best_params_)/opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/sklearn/tree/_classes.py:1347: FutureWarning: Value `"friedman_mse"` for `criterion` is deprecated and will be removed in 1.11. It maps to `"squared_error"` as both were always equivalent. Use `criterion="squared_error"` to remove this warning.
warnings.warn(
{'Base': DecisionTreeRegressor(max_depth=2), 'minibatch_frac': 0.5}