We’ll start with a probabilistic regression example on the diabetes dataset from sklearn:
from ngboost import NGBRegressor
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
X, Y = load_diabetes(return_X_y=True)
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2)
ngb = NGBRegressor().fit(X_train, Y_train)
Y_preds = ngb.predict(X_test)
Y_dists = ngb.pred_dist(X_test)
# test Mean Squared Error
test_MSE = mean_squared_error(Y_preds, Y_test)
print('Test MSE', test_MSE)
# test Negative Log Likelihood
test_NLL = -Y_dists.logpdf(Y_test).mean()
print('Test NLL', test_NLL)[iter 0] loss=5.7728 val_loss=0.0000 scale=1.0000 norm=66.7238
[iter 100] loss=5.4292 val_loss=0.0000 scale=1.0000 norm=46.7438
[iter 200] loss=5.1771 val_loss=0.0000 scale=1.0000 norm=37.6246
[iter 300] loss=5.0371 val_loss=0.0000 scale=2.0000 norm=67.9881
[iter 400] loss=4.9411 val_loss=0.0000 scale=1.0000 norm=31.7381
Test MSE 3240.8108757548052
Test NLL 5.803523385088415
Getting the estimated distributional parameters at a set of points is easy. This returns the predicted mean and standard deviation of the first five observations in the test set:
Y_dists[0:5].params{'loc': array([139.5271247 , 72.22036499, 108.55159442, 179.97048689,
80.50540421]),
'scale': array([42.50819001, 21.90917279, 27.98238063, 43.8521467 , 28.49242368])}Distributions¶
NGBoost can be used with a variety of distributions, broken down into those for regression (support on an infinite set) and those for classification (support on a finite set).
Regression Distributions¶
| Distribution | Parameters | Implemented Scores | Reference |
|---|---|---|---|
Normal | loc, scale | LogScore, CRPScore | scipy.stats normal |
LogNormal | s, scale | LogScore, CRPScore | scipy.stats lognormal |
Exponential | scale | LogScore, CRPScore | scipy.stats exponential |
The parameters in this table are the keys of params on a predicted distribution, and they have the same meaning as in scipy.stats. For LogNormal, has a normal distribution with mean and standard deviation . So params['s'] is and params['scale'] is . The attributes mu and sigma of a LogNormal give and .
Regression distributions can be used through the NGBRegressor() constructor by passing the appropriate class as the Dist argument. Normal is the default.
from ngboost.distns import Exponential, Normal
X, Y = load_diabetes(return_X_y=True)
X_reg_train, X_reg_test, Y_reg_train, Y_reg_test = train_test_split(X, Y, test_size=0.2)
ngb_norm = NGBRegressor(Dist=Normal, verbose=False).fit(X_reg_train, Y_reg_train)
ngb_exp = NGBRegressor(Dist=Exponential, verbose=False).fit(X_reg_train, Y_reg_train)There are two prediction methods for NGBRegressor objects: predict(), which returns point predictions as one would expect from a standard regressor, and pred_dist(), which returns a distribution object representing the conditional distribution of at the points in the test set.
ngb_norm.predict(X_reg_test)[0:5]array([104.62095952, 98.98535385, 100.0188204 , 125.6078506 ,
191.75325253])ngb_exp.predict(X_reg_test)[0:5]array([115.60509973, 90.76704316, 97.6749406 , 112.33359466,
178.70143437])ngb_exp.pred_dist(X_reg_test)[0:5].params{'scale': array([115.60509973, 90.76704316, 97.6749406 , 112.33359466,
178.70143437])}Survival Regression¶
NGBoost supports analyses of right-censored data. Any distribution that can be used for regression in NGBoost can also be used for survival analysis in theory, but this requires the implementation of the right-censored version of the appropriate score. At the moment, LogNormal and Exponential have these scores implemented. To do survival analysis, use NGBSurvival and pass both the time-to-event (or censoring) and event indicator vectors to fit():
import numpy as np
from ngboost import NGBSurvival
from ngboost.distns import LogNormal
X, Y = load_diabetes(return_X_y=True)
X_surv_train, X_surv_test, Y_surv_train, Y_surv_test = train_test_split(X, Y, test_size=0.2)
# introduce administrative censoring to simulate survival data
T_surv_train = np.minimum(Y_surv_train, 250) # time of an event or censoring
E_surv_train = Y_surv_train <= 250 # 1 if T[i] is the time of an event, 0 if it's a time of censoring
ngb = NGBSurvival(Dist=LogNormal).fit(X_surv_train, T_surv_train, E_surv_train)[iter 0] loss=5.0379 val_loss=0.0000 scale=2.0000 norm=1.4887
[iter 100] loss=4.6837 val_loss=0.0000 scale=1.0000 norm=0.5561
[iter 200] loss=4.5090 val_loss=0.0000 scale=1.0000 norm=0.5028
[iter 300] loss=4.3792 val_loss=0.0000 scale=1.0000 norm=0.4688
[iter 400] loss=4.2807 val_loss=0.0000 scale=1.0000 norm=0.4510
The scores currently implemented assume that the censoring is independent of survival, conditional on the observed predictors.
Classification Distributions¶
| Distribution | Parameters | Implemented Scores | Reference |
|---|---|---|---|
k_categorical(K) | p0, p1... p{K-1} | LogScore | Categorical distribution on Wikipedia |
Bernoulli | p | LogScore | Bernoulli distribution on Wikipedia |
Classification distributions can be used through the NGBClassifier() constructor by passing the appropriate class as the Dist argument. Bernoulli is the default and is equivalent to k_categorical(2).
from ngboost import NGBClassifier
from ngboost.distns import k_categorical, Bernoulli
from sklearn.datasets import load_breast_cancer
X, y = load_breast_cancer(return_X_y=True)
y[0:15] = 2 # artificially make this a 3-class problem instead of a 2-class problem
X_cls_train, X_cls_test, Y_cls_train, Y_cls_test = train_test_split(X, y, test_size=0.2)
ngb_cat = NGBClassifier(Dist=k_categorical(3), verbose=False) # tell ngboost that there are 3 possible outcomes
_ = ngb_cat.fit(X_cls_train, Y_cls_train) # Y should have only 3 values: {0,1,2}When using NGBoost for classification, the outcome vector Y must consist only of integers from 0 to K-1, where K is the total number of classes. This is consistent with the classification standards in sklearn.
NGBClassifier objects have three prediction methods: predict() returns the most likely class, predict_proba() returns the class probabilities, and pred_dist() returns the distribution object.
ngb_cat.predict(X_cls_test)[0:5]array([1, 1, 1, 1, 1])ngb_cat.predict_proba(X_cls_test)[0:5]array([[7.87717623e-03, 9.91473938e-01, 6.48885655e-04],
[3.67414604e-01, 6.32530259e-01, 5.51374851e-05],
[7.87717623e-03, 9.91473938e-01, 6.48885655e-04],
[3.21991574e-02, 9.67441710e-01, 3.59132539e-04],
[7.87717623e-03, 9.91473938e-01, 6.48885655e-04]])ngb_cat.pred_dist(X_cls_test)[0:5].params{'p0': array([0.00787718, 0.3674146 , 0.00787718, 0.03219916, 0.00787718]),
'p1': array([0.99147394, 0.63253026, 0.99147394, 0.96744171, 0.99147394]),
'p2': array([6.48885655e-04, 5.51374851e-05, 6.48885655e-04, 3.59132539e-04,
6.48885655e-04])}Scores¶
NGBoost supports the log score (LogScore, also known as negative log-likelihood) and CRPS (CRPScore), although each score may not be implemented for each distribution. The score is specified by the Score argument in the constructor.
from ngboost.scores import LogScore, CRPScore
NGBRegressor(Dist=Exponential, Score=CRPScore, verbose=False).fit(X_reg_train, Y_reg_train)
NGBClassifier(Dist=k_categorical(3), Score=LogScore, verbose=False).fit(X_cls_train, Y_cls_train)Base Learners¶
NGBoost can be used with any sklearn regressor as the base learner, specified with the Base argument. The default is a depth-3 regression tree.
from sklearn.tree import DecisionTreeRegressor
learner = DecisionTreeRegressor(criterion='friedman_mse', max_depth=5)
NGBSurvival(Dist=Exponential, Score=CRPScore, Base=learner, verbose=False).fit(X_surv_train, T_surv_train, E_surv_train)/opt/hostedtoolcache/Python/3.12.16/x64/lib/python3.12/site-packages/sklearn/tree/_classes.py:1347: FutureWarning: Value `"friedman_mse"` for `criterion` is deprecated and will be removed in 1.11. It maps to `"squared_error"` as both were always equivalent. Use `criterion="squared_error"` to remove this warning.
warnings.warn(
Other Arguments¶
The learning rate, number of estimators, minibatch fraction, and column subsampling are also easily adjusted:
ngb = NGBRegressor(n_estimators=100, learning_rate=0.01,
minibatch_frac=0.5, col_sample=0.5)
ngb.fit(X_reg_train, Y_reg_train)[iter 0] loss=5.7959 val_loss=0.0000 scale=1.0000 norm=67.8270
Sample weights (for training) are set using the sample_weight argument to fit.
ngb = NGBRegressor(n_estimators=100, learning_rate=0.01,
minibatch_frac=0.5, col_sample=0.5)
weights = np.random.random(Y_reg_train.shape)
ngb.fit(X_reg_train, Y_reg_train, sample_weight=weights)[iter 0] loss=5.8137 val_loss=0.0000 scale=1.0000 norm=71.3426