U ^{jCz@sdZddlmZmZddlZddlmZmZddl m Z m Z ddl m Z ddlmZGd d d ed ZGd d d eed ZGdddeZGdddeZGdddeZGdddeZGdddeed ZGdddeZGdddeZGdddeZeeeededZdS)zZLosses and corresponding default initial estimators for gradient boosting decision trees. )ABCMetaabstractmethodN)expit logsumexp)DummyClassifierDummyRegressor) TREE_LEAF)_weighted_percentilec@sdeZdZdZdZddZeddZeddd Zed d Z dddZ eddZ eddZ dS) LossFunctionaMAbstract base class for various loss functions. Parameters ---------- n_classes : int Number of classes. Attributes ---------- K : int The number of regression trees to be induced; 1 for regression and binary classification; ``n_classes`` for multi-class classification. FcCs ||_dSN)Kself n_classesrZ/var/www/html/bwave/python/venv/lib/python3.8/site-packages/sklearn/ensemble/_gb_losses.py__init__!szLossFunction.__init__cCsdS)z-Default ``init`` estimator for loss function.Nrrrrrinit_estimator$szLossFunction.init_estimatorNcCsdS)adCompute the loss. Parameters ---------- y : ndarray of shape (n_samples,) True labels. raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves). sample_weight : ndarray of shape (n_samples,), default=None Sample weights. Nrryraw_predictions sample_weightrrr__call__(szLossFunction.__call__cKsdS)NCompute the negative gradient. Parameters ---------- y : ndarray of shape (n_samples,) The target labels. raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble at iteration ``i - 1``. Nrrrrkargsrrrnegative_gradient8szLossFunction.negative_gradient皙?rc Cs||} | } d| |<t|jtkdD](} ||| | ||||dd| f|q0|dd| f||jddddfj| dd7<dS)azUpdate the terminal regions (=leaves) of the given tree and updates the current predictions of the model. Traverses tree and invokes template method `_update_terminal_region`. Parameters ---------- tree : tree.Tree The tree object. X : ndarray of shape (n_samples, n_features) The data array. y : ndarray of shape (n_samples,) The target labels. residual : ndarray of shape (n_samples,) The residuals (usually the negative gradient). raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble at iteration ``i - 1``. sample_weight : ndarray of shape (n_samples,) The weight of each sample. sample_mask : ndarray of shape (n_samples,) The sample mask to be used. learning_rate : float, default=0.1 Learning rate shrinks the contribution of each tree by ``learning_rate``. k : int, default=0 The index of the estimator being updated. rNZaxis) applycopynpwhereZ children_leftr _update_terminal_regionvaluetake) rtreeXrresidualrr sample_mask learning_ratekterminal_regionsZmasked_terminal_regionsleafrrrupdate_terminal_regionsFs$)   &z$LossFunction.update_terminal_regionsc CsdS)z=Template method for updating terminal regions (i.e., leaves).Nr rr)r/r0r*rr+rrrrrr&s z$LossFunction._update_terminal_regioncCsdS)aLReturn the initial raw predictions. Parameters ---------- X : ndarray of shape (n_samples, n_features) The data array. estimator : object The estimator to use to compute the predictions. Returns ------- raw_predictions : ndarray of shape (n_samples, K) The initial raw predictions. K is equal to 1 for binary classification and regression, and equal to the number of classes for multiclass classification. ``raw_predictions`` is casted into float64. Nr)rr* estimatorrrrget_init_raw_predictionssz%LossFunction.get_init_raw_predictions)N)rr) __name__ __module__ __qualname____doc__is_multi_classrrrrrr1r&r4rrrrr s     A r ) metaclasscs0eZdZdZfddZddZddZZS)RegressionLossFunctionz)Base class for regression loss functions.cstjdddS)Nr)superrr __class__rrrszRegressionLossFunction.__init__cCs t|drt|dstddS)zMake sure estimator has the required fit and predict methods. Parameters ---------- estimator : object The init estimator to check. fitpredictzNThe init parameter must be a valid estimator and support both fit and predict.Nhasattr ValueErrorrr3rrrcheck_init_estimatorsz+RegressionLossFunction.check_init_estimatorcCs||}|ddtjS)Nr r<)rBreshapeastyper$float64)rr*r3Z predictionsrrrr4s z/RegressionLossFunction.get_init_raw_predictions)r5r6r7r8rrGr4 __classcell__rrr?rr;s r;c@s<eZdZdZddZdddZddZdd d Zd dZdS)LeastSquaresErrorzLoss function for least squares (LS) estimation. Terminal regions do not need to be updated for least squares. Parameters ---------- n_classes : int Number of classes. cCs tddS)Nmeanstrategyrrrrrrsz LeastSquaresError.init_estimatorNcCsH|dkrt||dSd|t|||dSdS)arCompute the least squares loss. Parameters ---------- y : ndarray of shape (n_samples,) True labels. raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves). sample_weight : ndarray of shape (n_samples,), default=None Sample weights. Nrr<)r$rMravelsumrrrrrszLeastSquaresError.__call__cKs ||S)aTCompute half of the negative gradient. Parameters ---------- y : ndarray of shape (n_samples,) The target labels. raw_predictions : ndarray of shape (n_samples,) The raw predictions (i.e. values from the tree leaves) of the tree ensemble at iteration ``i - 1``. rQrrrrrs z#LeastSquaresError.negative_gradientrrc Cs*|dd| f|||7<dS)aLeast squares does not need to update terminal regions. But it has to update the predictions. Parameters ---------- tree : tree.Tree The tree object. X : ndarray of shape (n_samples, n_features) The data array. y : ndarray of shape (n_samples,) The target labels. residual : ndarray of shape (n_samples,) The residuals (usually the negative gradient). raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble at iteration ``i - 1``. sample_weight : ndarray of shape (n,) The weight of each sample. sample_mask : ndarray of shape (n,) The sample mask to be used. learning_rate : float, default=0.1 Learning rate shrinks the contribution of each tree by ``learning_rate``. k : int, default=0 The index of the estimator being updated. N)rBrQ) rr)r*rr+rrr,r-r.rrrr1s(z)LeastSquaresError.update_terminal_regionsc CsdSr rr2rrrr& s z)LeastSquaresError._update_terminal_region)N)rr) r5r6r7r8rrrr1r&rrrrrLs   *rLc@s2eZdZdZddZd ddZddZd d ZdS) LeastAbsoluteErrorzLoss function for least absolute deviation (LAD) regression. Parameters ---------- n_classes : int Number of classes cCs tdddSNquantile?rOrVrPrrrrr7sz!LeastAbsoluteError.init_estimatorNc CsJ|dkrt||Sd|t|t||SdS)atCompute the least absolute error. Parameters ---------- y : ndarray of shape (n_samples,) True labels. raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves). sample_weight : ndarray of shape (n_samples,), default=None Sample weights. Nr<)r$absrQrMrRrrrrr:szLeastAbsoluteError.__call__cKs|}d||dkdS)aCompute the negative gradient. 1.0 if y - raw_predictions > 0.0 else -1.0 Parameters ---------- y : ndarray of shape (n_samples,) The target labels. raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble at iteration ``i - 1``. rrr<rSrrrrrQsz$LeastAbsoluteError.negative_gradientc CsZt||kd} |j| dd}|j| dd|j| dd} t| |dd|j|ddf<dS)z1LAD updates terminal regions to median estimates.rr!2 percentileN)r$r%r(r r') rr)r/r0r*rr+rrterminal_regiondiffrrrr&bs z*LeastAbsoluteError._update_terminal_region)N)r5r6r7r8rrrr&rrrrrT.s  rTcsFeZdZdZdfdd ZddZddd Zdd d Zd d ZZ S)HuberLossFunctionahHuber loss function for robust regression. M-Regression proposed in Friedman 2001. Parameters ---------- alpha : float, default=0.9 Percentile at which to extract score. References ---------- J. Friedman, Greedy Function Approximation: A Gradient Boosting Machine, The Annals of Statistics, Vol. 29, No. 5, 2001. ?cst||_d|_dSr )r>ralphagammarrar?rrrs zHuberLossFunction.__init__cCs tdddSrUrPrrrrrsz HuberLossFunction.init_estimatorNc Cs|}||}|j}|dkrX|dkr@tt||jd}ntt|||jd}t||k}|dkrtd||d}t|t|||d}|||jd} nZtd||||d}t|||t|||d}|||} | S)aCompute the Huber loss. Parameters ---------- y : ndarray of shape (n_samples,) True labels. raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble. sample_weight : ndarray of shape (n_samples,), default=None Sample weights. NdrWrr) rQrbr$r\rYrar rRshape) rrrrr^rb gamma_maskZsq_lossZlin_losslossrrrrs4"zHuberLossFunction.__call__c Ks|}||}|dkr2tt||jd}ntt|||jd}t||k}tj|jdftjd}||||<|t ||||<||_ |S)aCompute the negative gradient. Parameters ---------- y : ndarray of shape (n_samples,) The target labels. raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble at iteration ``i - 1``. sample_weight : ndarray of shape (n_samples,), default=None Sample weights. NrdrZdtype) rQr$r\rYrar zerosrerJsignrb) rrrrrr^rbrfr+rrrrs z#HuberLossFunction.negative_gradientc  Cst||kd} |j| dd}|j} |j| dd|j| dd} t| |dd} | | } | tt| tt| | |j |df<dS)Nrr!rZr[) r$r%r(rbr rMrjminimumrYr')rr)r/r0r*rr+rrr]rbr^ZmedianZdiff_minus_medianrrrr&s z)HuberLossFunction._update_terminal_region)r`)N)N r5r6r7r8rrrrr&rKrrr?rr_xs  ) r_csDeZdZdZdfdd ZddZddd Zd d Zd d ZZ S)QuantileLossFunctionzLoss function for quantile regression. Quantile regression allows to estimate the percentiles of the conditional distribution of the target. Parameters ---------- alpha : float, default=0.9 The percentile. r`cst||_|d|_dS)Nrd)r>rrar\rcr?rrrs zQuantileLossFunction.__init__cCstd|jdS)NrVrX)rrarrrrrsz#QuantileLossFunction.init_estimatorNcCs|}||}|j}||k}|dkrX|||d||||jd}nD|t||||d|t|||||}|S)aCompute the Quantile loss. Parameters ---------- y : ndarray of shape (n_samples,) True labels. raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble. sample_weight : ndarray of shape (n_samples,), default=None Sample weights. Nr<r)rQrarRrer$)rrrrr^ramaskrgrrrrs$ zQuantileLossFunction.__call__cKs,|j}|}||k}||d||S)rr<)rarQ)rrrrrarnrrrr s z&QuantileLossFunction.negative_gradientc Cs\t||kd} |j| dd|j| dd} |j| dd}t| ||j} | |j|df<dS)Nrr!)r$r%r(r r\r') rr)r/r0r*rr+rrr]r^valrrrr&1s z,QuantileLossFunction._update_terminal_region)r`)Nrlrrr?rrms   rmc@s0eZdZdZeddZeddZddZdS) ClassificationLossFunctionz-Base class for classification loss functions.cCsdS)aTemplate method to convert raw predictions into probabilities. Parameters ---------- raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble. Returns ------- probas : ndarray of shape (n_samples, K) The predicted probabilities. Nrrrrrr_raw_prediction_to_probaIsz3ClassificationLossFunction._raw_prediction_to_probacCsdS)aTemplate method to convert raw predictions to decisions. Parameters ---------- raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble. Returns ------- encoded_predictions : ndarray of shape (n_samples, K) The predicted encoded labels. Nrrqrrr_raw_prediction_to_decisionYsz6ClassificationLossFunction._raw_prediction_to_decisioncCs t|drt|dstddS)zMake sure estimator has fit and predict_proba methods. Parameters ---------- estimator : object The init estimator to check. rA predict_probazTThe init parameter must be a valid estimator and support both fit and predict_proba.NrCrFrrrrGisz/ClassificationLossFunction.check_init_estimatorN)r5r6r7r8rrrrsrGrrrrrpFs   rpcsZeZdZdZfddZddZdddZd d Zd d Zd dZ ddZ ddZ Z S)BinomialDevianceaBinomial deviance loss function for binary classification. Binary classification is a special case; here, we only need to fit one tree instead of ``n_classes`` trees. Parameters ---------- n_classes : int Number of classes. cs.|dkrtd|jj|tjdddSNrz-{0:s} requires 2 classes; got {1:d} class(es)r<r=rEformatr@r5r>rrr?rrrszBinomialDeviance.__init__cCs tddSNZpriorrNrrrrrrszBinomialDeviance.init_estimatorNc Cs\|}|dkr.dt||td|Sd|t|||td|SdS)aCompute the deviance (= 2 * negative log-likelihood). Parameters ---------- y : ndarray of shape (n_samples,) True labels. raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble. sample_weight : ndarray of shape (n_samples,), default=None Sample weights. Nr)rQr$rMZ logaddexprRrrrrrszBinomialDeviance.__call__cKs|t|S)aPCompute half of the negative gradient. Parameters ---------- y : ndarray of shape (n_samples,) True labels. raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble at iteration ``i - 1``. )rrQrrrrrs z"BinomialDeviance.negative_gradientc Cst||kd} |j| dd}|j| dd}|j| dd}t||} t|||d||} t| dkrd|j|ddf<n| | |j|ddf<dS)zMake a single Newton-Raphson step. our node estimate is given by: sum(w * (y - prob)) / sum(w * prob * (1 - prob)) we take advantage that: y - prob = residual rr!r<u?j/ N)r$r%r(rRrYr' rr)r/r0r*rr+rrr] numerator denominatorrrrr&s z(BinomialDeviance._update_terminal_regioncCsZtj|jddftjd}t||dddf<|dddf|dddf8<|S)Nrrrhr<r$ZonesrerJrrQrrZprobarrrrrs$z)BinomialDeviance._raw_prediction_to_probacCs||}tj|ddSNr<r!rrr$Zargmaxrrrrrss z,BinomialDeviance._raw_prediction_to_decisioncCs`||}|dddf}ttjj}t||d|}t|d|}|ddtj S)Nr<r rtr$finfofloat32epscliplogrHrIrJrr*r3probasZproba_pos_classrrrrrr4s  z)BinomialDeviance.get_init_raw_predictions)N r5r6r7r8rrrrr&rrrsr4rKrrr?rruxs  !rucs`eZdZdZdZfddZddZddd Zdd d Zd dZ ddZ ddZ ddZ Z S)MultinomialDeviancezMultinomial deviance loss function for multi-class classification. For multi-class classification we need to fit ``n_classes`` trees at each stage. Parameters ---------- n_classes : int Number of classes. Tcs*|dkrtd|jjt|dS)Nz#{0:s} requires more than 2 classes.rwrr?rrrs  zMultinomialDeviance.__init__cCs tddSryrzrrrrr sz"MultinomialDeviance.init_estimatorNcCsjtj|jd|jftjd}t|jD]}||k|dd|f<q&tjd||jddt|dd|dS)aCompute the Multinomial deviance. Parameters ---------- y : ndarray of shape (n_samples,) True labels. raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble. sample_weight : ndarray of shape (n_samples,), default=None Sample weights. rrhNr r<r!)weights) r$rirer rJrangeZaveragerRr)rrrrYr.rrrr szMultinomialDeviance.__call__rc Ks,|tt|dd|ft|ddS)aCompute negative gradient for the ``k``-th class. Parameters ---------- y : ndarray of shape (n_samples,) The target labels. raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble at iteration ``i - 1``. k : int, default=0 The index of the class. Nr<r!)r$ nan_to_numexpr)rrrr.kwargsrrrr&s z%MultinomialDeviance.negative_gradientc Cst||kd} |j| dd}|j| dd}|j| dd}t||} | |jd|j9} t|||d||} t| dkrd|j|ddf<n| | |j|ddf<dS)z"Make a single Newton-Raphson step.rr!r<r|r}N)r$r%r(rRr rYr'r~rrrr&9s  z+MultinomialDeviance._update_terminal_regionc Cs*tt|t|ddddtjfSr)r$rrrZnewaxisrqrrrrrUs z,MultinomialDeviance._raw_prediction_to_probacCs||}tj|ddSrrrrrrrs\s z/MultinomialDeviance._raw_prediction_to_decisioncCs@||}ttjj}t||d|}t|tj}|S)Nr<) rtr$rrrrrrIrJ)rr*r3rrrrrrr4`s  z,MultinomialDeviance.get_init_raw_predictions)N)r)r5r6r7r8r9rrrrr&rrrsr4rKrrr?rrs    rcsZeZdZdZfddZddZdddZd d Zd d Zd dZ ddZ ddZ Z S)ExponentialLossaExponential loss function for binary classification. Same loss as AdaBoost. Parameters ---------- n_classes : int Number of classes. References ---------- Greg Ridgeway, Generalized Boosted Models: A guide to the gbm package, 2007 cs.|dkrtd|jj|tjdddSrvrwrr?rrrwszExponentialLoss.__init__cCs tddSryrzrrrrrszExponentialLoss.init_estimatorNcCs`|}|dkr.ttd|d |Sd|t|td|d |SdS)aCompute the exponential loss Parameters ---------- y : ndarray of shape (n_samples,) True labels. raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble. sample_weight : ndarray of shape (n_samples,), default=None Sample weights. N@?rr<)rQr$rMrrRrrrrrs zExponentialLoss.__call__cKs$d|d}|t| |S)aUCompute the residual (= negative gradient). Parameters ---------- y : ndarray of shape (n_samples,) True labels. raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble at iteration ``i - 1``. rr)r$rrQ)rrrry_rrrrs z!ExponentialLoss.negative_gradientc Cst||kd} |j| dd}|j| dd}|j| dd}d|d} t| |t| |} t|t| |} t| dkrd|j|ddf<n| | |j|ddf<dS)Nrr!rrr|r})r$r%r(rRrrYr') rr)r/r0r*rr+rrr]rrrrrrr&s   z'ExponentialLoss._update_terminal_regioncCs^tj|jddftjd}td||dddf<|dddf|dddf8<|S)Nrrrhrr<rrrrrrrs$z(ExponentialLoss._raw_prediction_to_probacCs|dktS)Nr)rQrIintrqrrrrssz+ExponentialLoss._raw_prediction_to_decisioncCsd||}|dddf}ttjj}t||d|}dt|d|}|ddtj S)Nr<rWr rrrrrr4s  z(ExponentialLoss.get_init_raw_predictions)Nrrrr?rrhs  r)Z squared_errorZabsolute_errorZhuberrVZlog_lossZ exponential)r8abcrrnumpyr$Z scipy.specialrrdummyrrZ tree._treer Z utils.statsr r r;rLrTr_rmrprurrZLOSS_FUNCTIONSrrrrs0  jJuY2}su