U _{ju@s ddlZddlZddlmZddlmZmZmZddlZ ddl Z ddl m Z ddl mZddlmZddlmZmZddlmZdd lmZdd lmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.dd l/m0Z0dd l1m2Z2dd l3m4Z4m5Z5ddl6m7Z7ddl8m9Z9ddl:m;Z;mZ>m?Z?m@Z@ddlAmBZBddlCmDZDdddZEddZFddZGe jHIdddde jJgddZKdd ZLd!d"ZMe@d#d$ZNe@d%d&ZOe@d'd(ZPd)d*ZQe jHId+dddd,ge Rd-d.d/gd0d1d1gd/d2d/gd.d1d3ggfdddddddddddg dd/d/d0d3d4d4d5d5ddg fgd6d7ZSe jHId+d,d,dddge Rd.d1d3gd.d1d3gd0d3d1gd0d3d1gd2d3d1ggfdddgd3d3d4gfgd8d9ZTd:d;ZUdd?ZWd@dAZXdBdCZYdDdEZZe jHIdFdGdHdIdJgdKdLZ[dMdNZ\e jHIdOe Rddddddge RddddddgdPdQfe Rddddddge RddddddgdPdRfe Rddddddge RddddddgdPdSfe Rddddddge RddddddgdPdTfe Rddddddge RddddddgdPdUfgdVdWZ]e jHIdXe Rdddddge Rddddd,gdPdYfgdZd[Z^d\d]Z_d^d_Z`d`daZae jHIddde jJge jHIdbdgdgfggfge jHIdce"ee#ddde,e-gdedfZbe jHIdbdgdgfggfge jHIdce"ee#ddde,e-gdgdhZcdidjZddkdlZedmdnZfdodpZge jHIdqdrdsgdtduZhdvdwZie jHIdxdydzd{d|dgd}d~ZjddZkddZlddZme jHjIdgdfddgdfgddgdddZne jHjIddddgddd,gfdddgdddZoddZpe jHIddddgddZqddZrddZsddZtddZuddZvddZwddZxddZyddZze@ddZ{ddZ|ddZ}ddZ~ddZddZddZddZe jHIdddgddÄZe@ddńZe@ddDŽZe@e jHIdddde jJe jJfgdd̄Ze jHIddge jHIdxd{dzd|dyge jHIddde jJgddτZe jHIdxd{dzd|dygddфZe jHIddde jJgddӄZddՄZddׄZe jHIddde jJgddلZe jHIdddde jJgddۄZe jHIdddde jJgdd݄Ze jHIdddde jJgdd߄ZddZddZddZddZddZddZddZddZddZddZddZddZddZddZddZddZe jHIdddddgddddgfddddgddddgfddddgddddgfgddZe jHIdce&e"ee#d3dde+e,e-ege jHIdddd d gd d Ze jHId e(e"e jJde(e#d,e jJde(e,e jJde(e-e jJdgddZdS(N)partial)chain permutationsproduct)linalg)hamming) bernoulli)datasetssvm)make_multilabel_classification)UndefinedMetricWarning)accuracy_scoreaverage_precision_scorebalanced_accuracy_scorebrier_score_lossclass_likelihood_ratiosclassification_reportcohen_kappa_scoreconfusion_matrixf1_score fbeta_score hamming_loss hinge_loss jaccard_scorelog_loss make_scorermatthews_corrcoefmultilabel_confusion_matrixprecision_recall_fscore_supportprecision_score recall_score zero_one_loss)_check_targets)cross_val_score)LabelBinarizerlabel_binarize)DecisionTreeClassifier) MockDataFrame)assert_allcloseassert_almost_equalassert_array_almost_equalassert_array_equalassert_no_warningsignore_warnings) _nanaverage)check_random_stateFc Cs|dkrt}|j}|j}|r:||dk||dk}}|j\}}t|}td}||||||}}t |d}tj d}tj || |d|f}tjdddd} | |d||d|||d} |r| ddd f} | ||d} ||d} | | | fS) zMake some classification predictions on a toy dataset using a SVC If binary is True restrict to a binary classification problem instead of a multiclass classification problem N%rlinearT)ZkernelZ probability random_state)r load_irisdatatargetshapenparanger/shuffleintrandom RandomStateZc_Zrandnr ZSVCfitZ predict_probaZpredict) datasetbinaryXy n_samples n_featuresprngZhalfZclfZ probas_predy_predy_truerKh/var/www/html/bwave/python/venv/lib/python3.8/site-packages/sklearn/metrics/tests/test_classification.pymake_prediction:s*     * rMc Cstt}t|dd\}}}ddddddd d d dd d ddddddddddddddd}t||tt|j|jdd}||kst |D]x}|dkrt ||t st ||||kst q||||kst ||D]}t ||||||qqt |ddt s(t t |ddt s@t t |ddt sXt t |ddt spt dS) NFrArBg7Mo?gUUUUUU?ghQ?) precisionrecallf1-scoresupportUUUUUU?gc1Ƹ?g433333?g)k??$I$I?gCFQ?gc?gƢ?KrRrPrQrSg?gD~WG?g]3p?)setosaZ versicolorZ virginica macro avgaccuracy weighted avgT)labels target_names output_dictr]r[rPr\rS)r r6rMrr:r;lenr`keysAssertionError isinstancefloatr)r=)irisrJrI_expected_reportreportkeymetricrKrKrL,test_classification_report_dictionary_outputjs`"  rmcCstggdd}dtjtjtjddtjtjtjddd}t|tsHt||ks\t|D]x}|dkrt||ts~t||||kstq`||||kst||D]}t||||||qq`dS)NT)rJrIrarrZ)r]r\r^r]) rr:nanredictrdrcrfr))rjrirkrlrKrKrL2test_classification_report_output_dict_empty_inputs, rq zero_divisionwarnr5c Csdddgdddg}}tjddX}t|||dd|dkrnt|d ksLt|D]}d }|t|jksPtqPn|rvtW5QRXdS) NabcdTrecord)rrrarsr5z7Use `zero_division` parameter to control this behavior.)warningscatch_warningsrrbrdstrmessage)rrrJrIryitemmsgrKrKrL0test_classification_report_zero_division_warningsrcCstdddgdddgg}tdddgdddgg}t||dksFtt||dksXtt||dksjtt|t|dkstt|t|dkstt|t|jdkstt|t|jdkstdSNrr5?)r:arrayr rd logical_notzerosr9y1y2rKrKrL.test_multilabel_accuracy_score_subset_accuracysrc Cstdd\}}}t||dd\}}}}t|ddgdt|ddgdt|d d gdt|d d gitfd d itffD]\}}|t||f|} t| dd|t||f|} t| dd|t||f|} t| d dt|t ||fddi|d| | d| | dqxdS)NTrBaverageg\(\?g333333?r0g)\(?g(\?皙?gRQ?rrBbeta) rMrr*r+r,rr rr)r) rJrIrhrGrfskwargsZ my_assertpsrsfsrKrKrL%test_precision_recall_f1_score_binarys(    rcCsdtddgddgkstdtddgddgks4tdtddgddgksNtdtddgddgddksltdtddgddgkstdtddgddgkstdtddgddgkstdtddgddgtddksttddgddgtddttddgddgddkstdS) N?r5rrrninfgj@)rrdr rrrfpytestapproxrKrKrKrL+test_precision_recall_f_binary_single_class s"rc Csddddg}ddddg}t|tdd}t|tdd}||f||fg}t|D]\}\}}t||dddddgdd}td d d d d g|t||dddddgd d}ttd d d d d g|d D]B}|dkr|dkrqtt||dddddg|dt||d|dqqTdD]`}t t t||td|dW5QRXt t t||tdd|dW5QRXqt dddgdddgg}t dddgdddgg}t ||dddgd\}} } } tt || | gt dddgdS)Nr5r0rclassesrrr_rrnrrmacro)microweightedsamplesr)Nrrrrrr_?竪?) r%r:r; enumerater r*meanr)rraises ValueErrorrr) rJrI y_true_bin y_pred_binr7iactualrrGrrrhrKrKrL$test_precision_recall_f_extra_labelssH      rc Csddddg}ddddg}t|tdd}t|tdd}||f||fg}t|D]\}\}}tt||ddgd}tt||dd}tddg|dd td |d d td |d d td |dd dD]}||d ||d kstqqTdS)Nr5r0rrrr_rrrrrUUUUUU?rr)rrr) r%r:r;rrr r*r)rd) rJrIrrr7rZ recall_13Z recall_allrrKrKrL&test_precision_recall_f_ignored_labelsJs  rc Cstdddgdddgdddgdddgdddgdddgg}tdddgdddgdd dgddd gdddgdddgg}d }tjt|d t||dd W5QRXdS)z:Test multiclass-multiouptut for `average_precision_score`.r0r5rffffff?皙?皙?皙?333333?rrz.multiclass-multioutput format is not supportedmatch pos_labelN)r:rrrrr)rJy_scoreerr_msgrKrKrL-test_average_precision_score_non_binary_classas*  rzy_true, y_scorer0rrrrrrr333333?rVcCst||dkstdS)a( Duplicate values with precision-recall require a different processing than when computing the AUC of a ROC, because the precision-recall curve is a decreasing curve The following situation corresponds to a perfect test statistic, the average_precision_score should be 1. r5NrrdrJrrKrKrL-test_average_precision_score_duplicate_values|srcCst||dkstdS)NrrrrKrKrL(test_average_precision_score_tied_valuessrc Cs>d}tjt|d"tdddgdddgdddW5QRXdS)NzNote that pos_label \(set to 2\) is ignored when average != 'binary' \(got 'macro'\). You may use labels=\[pos_label\] to specify a single positive class.rr5r0rrr)rwarns UserWarningrrrKrKrL(test_precision_recall_f_unused_pos_labelsrcCsDtdd\}}}dd}||||dd|Ddd|DdS)NTrc Sst||}t|ddgddgg|\}}}}||||}t||||||||}|dkrrdn||} t||} t| | ddt| ddddS) Nrrr0decimal= ףp=?)rr+flattenr:sqrtrr*) rJrIcmtpfpfntnnumZdenZtrue_mccZmccrKrKrLtests & z*test_confusion_matrix_binary..testcSsg|] }t|qSrKr|.0rDrKrKrL sz0test_confusion_matrix_binary..rMrJrIrhrrKrKrLtest_confusion_matrix_binarys rcCsDtdd\}}}dd}||||dd|Ddd|DdS)NTrcSs4t||}t|ddgddggddgddgggdS)Nrrrrrr+)rJrIrrKrKrLrs z5test_multilabel_confusion_matrix_binary..testcSsg|] }t|qSrKrrrKrKrLrsz;test_multilabel_confusion_matrix_binary..rrrKrKrL'test_multilabel_confusion_matrix_binarys rcCsJtdd\}}}d dd}||||dd|Ddd|DdddS) NFrcSst||}t|ddgddggddgddggd d gd d ggg|rLd ddgndd dg}t|||d}t|ddgddggd d gd d ggddgddggg|rd dddgn dd ddg}t|||d}t|ddgddggd d gd d ggddgddggddgddgggdS)N/rr&rrrr0021rr5r3rYr)rJrI string_typerr_rKrKrLrs* ,,z9test_multilabel_confusion_matrix_multiclass..testcSsg|] }t|qSrKrrrKrKrLr sz?test_multilabel_confusion_matrix_multiclass..T)r)FrrrKrKrL+test_multilabel_confusion_matrix_multiclasss  rcCsddlm}m}tdddgdddgdddgg}tdddgdddgdddgg}||}||}||}||}tdddg}ddgddggddgddggddgddggg} |||g} |||g} | D]"} | D]} t| | }t|| qqt||dd}t|ddgddggddgddggddgddgggt||ddgd}t|ddgddggddgddgggt||ddgdd }t|ddgddggddgddggddgddgggt|||dd }t|ddgddggddgddggddgd dgggdS) Nr) csc_matrix csr_matrixr5r0rT samplewiser)r_r) sample_weightrr)Z scipy.sparserrr:rrr+)rrrJrIZ y_true_csrZ y_pred_csrZ y_true_cscZ y_pred_cscrZreal_cmZtruespredsZ y_true_tmpZ y_pred_tmprrKrKrL+test_multilabel_confusion_matrix_multilabels8"".   4&4rc Cstdddgdddgdddgg}tdddgdddgdddgg}tjtddt||ddgdW5QRXtjtdd,t||dddgddd gdd d ggdW5QRXd }tjt|dt||d gd W5QRXd }tjt|dt||dgd W5QRXtjtdd tdddgdddgddW5QRXd}tjt|d0tdddgdddggdddgdddggW5QRXdS)Nr5rzinconsistent numbers of samplesrr0rzshould be a 1d arrayrrrz%All labels must be in \[0, n labels\)rrzSamplewise metricsTrz'multiclass-multioutput is not supported)r:rrrrr)rJrIrrKrKrL'test_multilabel_confusion_matrix_errors8s*""$rz%normalize, cm_dtype, expected_results)truerTUU?)predrr)allrgeq?)Nrr0cCsPdddgd}tttdddg}t|||d}t|||jj|ksLtdS)Nrr5r0r normalize)listrrrr(dtypekindrd)rZcm_dtypeZexpected_resultsy_testrIrrKrKrLtest_confusion_matrix_normalizeVs  rc Csddddddddg}ddddddddg}t||dd}|tdksLtt tdtt||dd}W5QRX|tdkstt tdtt||ddW5QRXdS) Nrr5rr@errorrr) rsumrrrdrzr{ simplefilterRuntimeWarning)rrIZcm_trueZcm_predrKrKrL,test_confusion_matrix_normalize_single_classgs    r zparams, warn_msgrJrIz2samples of only one class were seen during testingz:positive_likelihood_ratio ill-defined and being set to nanz+no samples predicted for the positive classz:negative_likelihood_ratio ill-defined and being set to nanz@no samples of the positive class were present in the testing setc Cs(tjt|dtf|W5QRXdSNr)rrrr)paramsZwarn_msgrKrKrLtest_likelihood_ratios_warningszs1rzparams, err_msgzeclass_likelihood_ratios only supports binary classification problems, got targets of type: multiclassc Cs(tjt|dtf|W5QRXdSr)rrrr)rrrKrKrLtest_likelihood_ratios_errorssrcCstdgddgd}tdgddgddgd}t||\}}t|dt|d t||\}}t|tjdt|tdd d td gd dgd}t|||d\}}t|dt|ddS)Nr5rrrr0 rg?g_B{ %?g-q=)ZrtolrrnrrgUUUUUU@gqq?)r:rrr(r+ror)rJrIposnegrrKrKrLtest_likelihood_ratioss$   rcCstdgddgd}tdgddgddgddgd}t||}t|dd d |t||ksrtt|d gd }t|d gd }t||ddgd |ksttt||dtdgddgdd gd}tdgddgdd gd}tt||dd d tdgddgdd gd}tdgddgdd gd}tt||dd d tt||dddd d tt||dddd d dS)Nr(r5<rXr2gʡE?rrr0rrr.,4 g??g+?r3weightsg_vO?Z quadraticg#?)r:rrr)rdappend)rrkapparKrKrLtest_cohen_kappas*. $$$$ r#cCs4tdgdgdksttddgddgdks0tdS)Nrr5rn)rrdrKrKrKrLtest_matthews_corrcoef_nansr$zy_true, y_predrlrc CsVttd||||d}W5QRXt|rFt|sRtn ||ksRtdS)zmCheck the behaviour of `zero_division` when setting to 0, 1 or np.nan. No warnings should be raised. rrrN)rzr{r r:isnanrd)rlrJrIrrresultrKrKrL!test_zero_division_nan_no_warnings    r(c Cs4tt|||dd}W5QRX|dks0tdS)ztCheck the behaviour of `zero_division` when setting to "warn". A `UndefinedMetricWarning` should be raised. rsr%rnN)rrr rd)rlrJrIr'rKrKrLtest_zero_division_nan_warnings r)cCsNtjd}|jdddd}|jdddd}tt||t||dddS)Nrr0rXsizerr5r)r:r>r?randintr)rZcorrcoef)rHrJrIrKrKrL-test_matthews_corrcoef_against_numpy_corrcoef.s r.c stjd}|jdddd}|jdddd}|d}t|||dttfddtD}tfddtD}tfd dtD}|t ||}t |||d}t ||d dS) Nrr0rXr*rc sXg|]P}tD]B}tD]4}||f||f||f||fqqqSrKrange)rkmlCNrKrLrDs   z9test_matthews_corrcoef_against_jurman..cs@g|]8ddftfddtDqS)Ncs.g|]&}tD]}|kr||fqqSrKr/rrgr5r6r1rKrLrNs  Dtest_matthews_corrcoef_against_jurman...r r:r0rr4r1rLrLscs@g|]8ddftfddtDqS)Ncs.g|]&}tD]}|kr||fqqSrKr/r7r9rKrLrUs  r:r;r<r4r=rLrSsr) r:r>r?r-Zrandrrbr r0rrr)) rHrJrIrZcov_ytypZcov_ytytZcov_ypypZ mcc_jurmanZmcc_oursrKr4rL%test_matthews_corrcoef_against_jurman8s0     r>cCsftjd}dd|jddddD}tt||ddd|D}tt||d t|d d gd }t|d d }tt||d ttddddgddddgd tt|d gt|d ddddddddddddddddddddg}ddddddddddddddddddddg}tt||d dgddgd}t t tt|||dd W5QRXdS)NrcSsg|]}|dkrdndqS)rrtrurKrrrKrKrLrasz*test_matthews_corrcoef..r0rXr*rcSsg|]}|dkrdndqS)rtrurKr?rKrKrLrgsrrtrurrnr5rr) r:r>r?r-r)rr%whererbrrrd)rHrJZ y_true_invZ y_true_inv2y_1y_2maskrKrKrLtest_matthews_corrcoef_s   ,, rDc stjd}tdd}fdd|jd|ddD}tt||dddd d d d g}d d ddd d g}tt||d ddd d d d g}d d ddddg}tt||d td dd d g}dddg}tt||ddddg}dd d g}tt||ddd d dd d dd d g }d d d d d d dddg }tt||dddd d d g}d d ddd g}d d d d dg}tt|||ddddd d g}ddd d g}d d ddg}tt|||dddS)Nrrtrcsg|]}t|qSrK)chrr?Zord_arKrLrsz5test_matthews_corrcoef_multiclass..rXr*rr5r0giirrnrr)r:r>r?ordr-r)rr) rH n_classesrJZ y_pred_badZ y_pred_minrIrArBrrKrFrL!test_matthews_corrcoef_multiclasssD          rIn_pointsdi'cstjddd}fdd}tddg|}tt||dtdddg|}tt||d||\}}tt||dtt|||||dS) Ni3c Ssxt||}|d}|d}|d}t|}|||}|||}||||} ||d|d|} | t| S)Nr5r5)r5rr,r5)rrbr:r) rJrIZ conf_matrixZtrue_posZ false_posZ false_negrJZpos_rateZactivityZ mcc_numeratorZmcc_denominatorrKrKrLmcc_safes   z1test_matthews_corrcoef_overflow..mcc_safecs8|}|d|d}|dk}|dk}||fS)Nrr)Z random_sample)rJZx_trueZx_predrJrIrHrKrL random_yss  z2test_matthews_corrcoef_overflow..random_ysrnrr)r:r>r?repeatr)r)rJrMrOZarrrJrIrKrNrLtest_matthews_corrcoef_overflows   rQc CsRtdd\}}}t||dd\}}}}t|dddgdt|dd d gdt|d d d gdt|dddgt||ddd}t|ddt||dd}t|ddt||dd} t| ddt||dd}t|ddt||dd}t|ddt||dd} t| ddt||dd}t|ddt||dd}t|ddt||dd} t| ddtt t||ddW5QRXtt t||ddW5QRXtt t||ddW5QRXtt t ||dddW5QRXt||dddgdd\}}}}t|dddgdt|dd d gdt|d d d gdt|dddgdS)!NFrr(\?gQ?gzG?r0gHzG?g ףp= ?rVgQ?g333333?rrOrUrXr5rrg(\?rrgRQ?rgGz?rrrrrrg= ףp=?r) rMrr*r+rr rrrrr) rJrIrhrGrrrrrrrKrKrL)test_precision_recall_f1_score_multiclasssT             rTrrrrrcCstddddgg}tddddgg}t||ddddgg|d\}}}}t|dt|dt|d|dkrt|ddddgdS)Nr5rrr0)r_warn_forr)r:rrr+)rrJrIrGrrrrKrKrL;test_precision_refcall_f1_score_multilabel_unordered_labelss    rVc Cstdddddddddddddddg}tdddddddddddddddg}t||dd\}}}}t||dd\}}}}|t|kst|t|kst|t|kstt||dd\}}}}t|} |tj|| dkst|tj|| dkst|tj|| dkstdS)Nrr5rrrr)r:rrrrdZbincountr) rJrIrrrrhrGrrrSrKrKrL.test_precision_recall_f1_score_binary_averaged%s(( rWc Cstjdd}zrtddddddg}tddddddg}tt||ddddtt||ddddtt||ddddW5tjf|XdS) Nraise)rrr5r0rrrn)r:Zseterrrr)rr r)Zold_error_settingsrJrIrKrKrLtest_zero_precision_recall6s rYcCstdd\}}}t||ddgd}t|ddgddggt||d dgd}t|d d gd dggt|d}t||d |gd}t|d dgddggdS) NFrrr5rrrrr0rrO)rMrr+r:max)rJrIrhrZ extra_labelrKrKrL.test_confusion_matrix_multiclass_subset_labelsGsr[zlabels, err_msgz,'labels' should contains at least one label.rrz.At least one label specified must be in y_truez empty listzunknown labels)Zidsc Cs<tdd\}}}tjt|dt|||dW5QRXdS)NFrrr)rMrrrr)r_rrJrIrhrKrKrLtest_confusion_matrix_errorZs r\r_NonerB multiclasscCs>|r t|nd}tj||ftd}tgg|d}t||dS)Nrrr)rbr:rr=rr+)r_Zexpected_n_classesexpectedrrKrKrL*test_confusion_matrix_on_zero_length_inputhsracCs>dddg}tt|}t||}|jtjks2ttjtjtj fD],}t|||j |ddd}|jtjksBtqBtj tj dt fD],}t|||j |ddd}|jtj kstqtjt|dtjd}t|||d}|ddkst|d d ksttjt|d tjd}t|||d}|dd ks(t|d d ks:tdS) Nrr5F)copyrlr_rrrLll)r:Zonesrbrrint64rdZbool_Zint32Zuint64ZastypeZfloat32float64objectfullZuint32)rDweightrrrKrKrLtest_confusion_matrix_dtypers$  rjrZInt64ZFloat64booleanc Csvtd}tdddddddddg }|j||d}|jdddddddddg dd}t||}t||}t||dS)zkChecks that confusion_matrix works with pandas nullable dtypes. Non-regression test for gh-25635. pandasr5rr_reN)rZ importorskipr:rSeriesrr+)rpdZ y_ndarrayrJZ y_predictedoutputZexpected_outputrKrKrL%test_confusion_matrix_pandas_nullables    rpcCsLt}t|dd\}}}d}t||tt|j|jd}||ksHtdS)NFrNa| precision recall f1-score support setosa 0.83 0.79 0.81 24 versicolor 0.33 0.10 0.15 31 virginica 0.42 0.90 0.57 20 accuracy 0.53 75 macro avg 0.53 0.60 0.51 75 weighted avg 0.51 0.53 0.47 75 r_r` r r6rMrr:r;rbr`rdrgrJrIrhrirjrKrKrL%test_classification_report_multiclasss rtc CsLdddddddddg dddddddddg }}d}t||}||ksHtdS)Nrr5r0a| precision recall f1-score support 0 0.33 0.33 0.33 3 1 0.33 0.33 0.33 3 2 0.33 0.33 0.33 3 accuracy 0.33 9 macro avg 0.33 0.33 0.33 9 weighted avg 0.33 0.33 0.33 9 )rrd)rJrIrirjrKrKrL.test_classification_report_multiclass_balanceds. rucCs8t}t|dd\}}}d}t||}||ks4tdS)NFrNa| precision recall f1-score support 0 0.83 0.79 0.81 24 1 0.33 0.10 0.15 31 2 0.42 0.90 0.57 20 accuracy 0.53 75 macro avg 0.53 0.60 0.51 75 weighted avg 0.51 0.53 0.47 75 )r r6rMrrdrsrKrKrL:test_classification_report_multiclass_with_label_detections  rvcCsNt}t|dd\}}}d}t||tt|j|jdd}||ksJtdS)NFrNa| precision recall f1-score support setosa 0.82609 0.79167 0.80851 24 versicolor 0.33333 0.09677 0.15000 31 virginica 0.41860 0.90000 0.57143 20 accuracy 0.53333 75 macro avg 0.52601 0.59615 0.50998 75 weighted avg 0.51375 0.53333 0.47310 75 r)r_r`digitsrrrsrKrKrL1test_classification_report_multiclass_with_digitss rxcCsztdd\}}}tdddg|}tdddg|}d}t||}||ksRtd}t||dd d gd }||ksvtdS) NFrbluegreenreda| precision recall f1-score support blue 0.83 0.79 0.81 24 green 0.33 0.10 0.15 31 red 0.42 0.90 0.57 20 accuracy 0.53 75 macro avg 0.53 0.60 0.51 75 weighted avg 0.51 0.53 0.47 75 a| precision recall f1-score support a 0.83 0.79 0.81 24 b 0.33 0.10 0.15 31 c 0.42 0.90 0.57 20 accuracy 0.53 75 macro avg 0.53 0.60 0.51 75 weighted avg 0.51 0.53 0.47 75 rtrurvr`rMr:rrrd)rJrIrhrirjrKrKrL7test_classification_report_multiclass_with_string_labels   r~cCsNtdd\}}}tdddg}||}||}d}t||}||ksJtdS)NFrublue¢ugreen¢ured¢u precision recall f1-score support blue¢ 0.83 0.79 0.81 24 green¢ 0.33 0.10 0.15 31 red¢ 0.42 0.90 0.57 20 accuracy 0.53 75 macro avg 0.53 0.60 0.51 75 weighted avg 0.51 0.53 0.47 75 r}rJrIrhr_rirjrKrKrL8test_classification_report_multiclass_with_unicode_labels rcCsNtdd\}}}tdddg}||}||}d}t||}||ksJtdS)NFrryZgreengreengreengreengreenr{a precision recall f1-score support blue 0.83 0.79 0.81 24 greengreengreengreengreen 0.33 0.10 0.15 31 red 0.42 0.90 0.57 20 accuracy 0.53 75 macro avg 0.53 0.60 0.51 75 weighted avg 0.51 0.53 0.47 75 r}rrKrKrLpos_label=2 is not a valid label. It should be one of \[0, 1\]rrBr0rrTarget is multilabel-indicator but average='binary'. Please choose another average setting, one of \[None, 'micro', 'macro', 'weighted', 'samples'\].rTarget is multiclass but average='binary'. Please choose another average setting, one of \[None, 'micro', 'macro', 'weighted'\].rzJSamplewise metrics are not available outside of multilabel classification.rzNote that pos_label \(set to 3\) is ignored when average != 'binary' \(got 'micro'\). You may use labels=\[pos_label\] to specify a single positive class.rr)r:rrrrrrr)rJrIrZmsg1msg2msg3rrKrKrLtest_jaccard_score_validations0rc Cs$tdddgdddgg}tdddgdddgg}t||dddksJtt||dddks`tt||dddksvtt|t|dddkstt|t|dddkstt|t|jdddkstt|t|jdddksttdddgdddgg}tdddgdddgg}tt||dddtt||ddd tt||ddd tt||ddd gd d tt||ddd gd d tt||ddtd dd gtdddgdddgg}tdddgdddgg}tt||dddtt||dddd}t j t |dt||dgddW5QRXd}t j t |dt||dgddW5QRXd}t j t |d6ttddggtddggddd kstW5QRXd}t j t |dBttddgddggtddgddggddd kstW5QRXt|r tdS)Nrr5rrrrrrrg?r0rrrrrg?z Got 4 > 2rrrz Got -1 < 0rzXJaccard is ill-defined and being set to 0.0 in labels with no true or predicted samples.zXJaccard is ill-defined and being set to 0.0 in samples with no true or predicted labels.)r:rrrdrrr9r)r+rrrrr r)recwarnrrrJrIrrrrKrKrLtest_multilabel_jaccard_scoresn $ rc Csbddddddddg}ddddddddg}dddg}t}||||}||}tt||}tt||}ddgddgddgdgdgdgdg} ddgddgddgdgdgdgdg} dD]2} t| | D]"\} } t|| | d|| | dqqtddgddgddgg}tddgddgddgg}t t||d d dksFt W5QRXt |r^t dS) Nantcatbirdrr5r0)rrrNrrr) r$r@Z transformrrzipr)r:rr-rdr)rrJrIr_ZlbrrZmulti_jaccard_scoreZbin_jaccard_scoreZmulti_labels_listZbin_labels_listrZm_labelZb_labelrKrKrLtest_multiclass_jaccard_scores:       $  "rc Cstdgdgdddkstd}tjt|d$tddgddgdddksLtW5QRXtdgdgdddd ksrttdddddg}tdddddg}tt||ddd tt||ddd d t|rtdS) Nr5rrBrrnzOJaccard is ill-defined and being set to 0.0 due to no true or predicted samplesrrrrrr) rrdrrr r:rr)r)rrrJrIrKrKrL!test_average_binary_jaccard_score=s(rc Csxtdddgdddgg}tdddgdddgg}d}tjt|d(t||ddd}|tdksjtW5QRXdS) Nr5rzJaccard is ill-defined and being set to 0.0 in samples with no true or predicted labels. Use `zero_division` parameter to control this behavior.rrrsrrrrn)r:rrrr rrrd)rJrIrscorerKrKrL(test_jaccard_score_zero_division_warningTsrzzero_division, expected_scorerc)r5rc Csztdddgdddgg}tdddgdddgg}t"tdtt||d|d}W5QRX|t|ksvt dS)Nr5rrrr) r:rrzr{r r rrrrd)rrZexpected_scorerJrIrrKrKrL*test_jaccard_score_zero_division_set_valuecs  rcCsftddddgddddgddddgg}tddddgddddgddddgg}t||dd\}}}}t|ddddgdt|ddddgdt|ddddgdt|ddddgdt||ddd }|}t|dd ddgdt||d d\}}}}t|d t|dt|d |dks ttt||dd d t|t||dd\}}}}t|dt|dt|d|dks~ttt||ddd d||d||t||dd\}}}}t|d t|dt|d |dksttt||ddd tj||dt||dd\}}}}t|dt|dt|d|dksLttt||ddd ddS)Nr5rrrnrrr0rrrrRrg?g?rrrrrr r:rrr*rr)rdrrrJrIrGrrrf2rSrKrKrL+test_precision_recall_f1_score_multilabel_1psV((             rcCshtddddgddddgddddgg}tddddgddddgddddgg}t||dd\}}}}t|ddddgdt|ddddgdt|ddddgdt|ddddgdt||ddd }|}t|dd ddgdt||d d\}}}}t|d t|d t|d |dks ttt||dd d d ||d||t||dd\}}}}t|d t|dt|d|dksttt||ddd t|t||dd\}}}}t|dt|d t|d|dksttt||ddd tj||dt||dd\}}}}t|dt|dt|d|dksLttt||ddd dddS)Nr5rrrnrr0rgQ?r皙?r?rrrg?rrrTrrg&S?rrrKrKrL+test_precision_recall_f1_score_multilabel_2s^((             rz%zero_division, zero_division_expected)rsrrLcCstddddgddddgddddgg}tddddgddddgddddgg}t||d|d\}}}}t||dddgdt|ddd|gdt|sdntj}t||dd|gdt|ddddgdt||dd|d } |} t| |d d|gdt||d |d\}}}}t|rdn|} d t| } t|d| | t|d | | dt|shdnd}t|||dksttt||dd |d t | ddt||d|d\}}}}t|dt|dt|d|dksttt||dd|d d||d||t||d|d\}}}}t||dkr@dndt|dt|s`dnd } t|d| |dksttt||dd|d t | | dt||dd\}}}}t|dt|dt|d|dkstddtjdi} tt||dd|d | |dddS)Nrr5rrrnr0rrrrrrrrr?g?rrrrWrrrg@rrrTgZd;O?gZd;O?) r:rrr*r&rorr)rdr.get)rrZzero_division_expectedrJrIrGrrrZ expected_frrSZ value_to_sumZvalues_to_averageZexpected_resultrKrKrL7test_precision_recall_f1_score_with_an_empty_predictions((            rrc Cstd}t|}tt|||||d\}}}}tt|||||d} |dksPtt|r~|||| fD]} t| sftqfdSt|}t ||t ||t ||t | t|dS)NrXrrrrrr) r:rrr,rrrdr&rfr)) rrrrrJrIrGrrrfbetarlrKrKrL"test_precision_recall_f1_no_labelsWs8       rc Cstd}t|}t}tt||||dd\}}}}W5QRXt|dt|dt|d|dksptttt |||dd}W5QRXt|ddS)NrrrSr) r:rrrrrr r)rdr) rrJrIfuncrGrrrrrKrKrL1test_precision_recall_f1_no_labels_check_warnings~s   "     rcCstd}t|}tt||dd|d\}}}}tt||dd|d}t|}t||||gdt||||gdt||||gdt|dddgdt||||gddS)Nrrrrr0r)r:rrr,rrrfr*)rrrJrIrGrrrrrKrKrL/test_precision_recall_f1_no_labels_average_nones0   rc Cstd}t|}ttt||ddd\}}}}W5QRXt|dddgdt|dddgdt|dddgdt|dddgdttt||ddd}W5QRXt|dddgddS)Nrr5rSrr0r) r:rrrrr rr*r)rJrIrGrrrrrKrKrL4test_precision_recall_f1_no_labels_average_none_warns    rc Cstt}}dD]t}d}tj||d |dddgdddg|dW5QRXd}tj||d |dddgdddg|dW5QRXqd }tj||d8|tddgddggtddgddggd dW5QRXd }tj||d8|tddgddggtddgddggd dW5QRXd }tj||d8|tddgddggtddgddggd dW5QRXd}tj||d8|tddgddggtddgddggd dW5QRXd }tj||d|ddgddgddW5QRXd}tj||d|ddgddgddW5QRXtjdd^}tdtddgddgddd}t | j |ksvt d }t | j |kst W5QRXdS)NNrrzPrecision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.rrr5r0rzRecall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.zPrecision and F-score are ill-defined and being set to 0.0 in samples with no predicted labels. Use `zero_division` parameter to control this behavior.rzRecall and F-score are ill-defined and being set to 0.0 in samples with no true labels. Use `zero_division` parameter to control this behavior.zPrecision and F-score are ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.rzRecall and F-score are ill-defined and being set to 0.0 due to no true samples. Use `zero_division` parameter to control this behavior.rrBTrxalways) rr rrr:rrzr{r r|popr}rd)rrrrryrKrKrLtest_prf_warningssV $&<<<<   rc Cst}dD]@}t|dddgdddg||dt|dddgdddg||dqt|tddgddggtddgddggd|dt|tddgddggtddgddggd|dt|tddgddggtddgddggd|dt|tddgddggtddgddggd|dt|ddgddgd |dt|ddgddgd |dtjd d :}td tddgddgd |dt|dkstW5QRXdS) Nrrr5r0rrrrrBTrxr) rr,r:rrzr{r rbrd)rrrrryrKrKrL)test_prf_no_warnings_if_zero_division_set1s   rc Cstttddgddggtddgddggd|dtjdd}tdttddgddggtddgddggd|d|dkrt|j d kst nt |dkst tddgddg|dkrt|j d kst W5QRXdS) Nr5rrrTrxrrszzRecall is ill-defined and being set to 0.0 due to no true samples. Use `zero_division` parameter to control this behavior.) r,r r:rrzr{r r|rr}rdrbrrryrKrKrLtest_recall_warningsqs6   rc Cstjdd}tdttddgddggtddgddggd|d|dkrlt|jd ks|t nt |dks|t tddgddg|dkrt|jd kst W5QRXt ttddgddggtddgddggd|ddS) NTrxrr5rrrrszPrecision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.) rzr{r rr:rr|rr}rdrbr,rrKrKrLtest_precision_warningss6   rc Cs.tjdd}tdtttddfD]}|tddgddggtddgddggd|d t|dksrt |tddgddggtddgddggd|d t|dkst |tddgddggtddgddggd|d |d krt | j d kst q*t|dks*t q*W5QRXdS) NTrxrr0rr5rrrrszF-score is ill-defined and being set to 0.0 due to no true nor predicted samples. Use `zero_division` parameter to control this behavior.) rzr{r rrrr:rrbrdr|rr})rrryrrKrKrLtest_fscore_warningss:    rc Csddddg}ddddg}d}tdddgdddgdddgg}tdddgdddgdddgg}d}|||f|||ffD]H\}}}tttttddfD](} tjt |d| ||W5QRXqqxdS) Nr5r0rrrrrr) r:rrr rrrrrr) Z y_true_mcZ y_pred_mcZmsg_mcZ y_true_indZ y_pred_indZmsg_indrJrIrrlrKrKrL'test_prf_average_binary_data_non_binarys$  "" rc*Cs~d}d}d}d}d}d}|tdddgdddgdddggf|tddgddgddggf|d d dgf|dddgf|d d d gf|td gd gdggf|tdgdgdggf|td gd gd ggf|tdd gdd gd d ggf|tddgddgddggfg }||f|||f|||f|||fd||fd||f|||fd||fd||fd||fd||fd||fd||fd||fd||fd||fd||fd||fd||fd||fd||fdi}t|d dD]\\}} \} } z||| f} Wn"tk r|| |f} YnX| dkrttt| | W5QRX|| krjd|| } tjt| dt| | W5QRXn>||||fkr>d|} tjt| dt| | W5QRXnt| | \}}}|| kst | dr|jdkst |jdkst n t |t | t |t | ttt| dd| W5QRXqddg} ddg} d}tjt|dt| | W5QRXdS)Nzmultilabel-indicatorr^rBZ continuouszmulticlass-multioutputzcontinuous-multioutputrr5r0rrnrrrrg?g@)rPz@Classification metrics can't handle a mix of {0} and {1} targetsrz{0} is not supportedZ multilabelZcsrr)r5r0)rr0r)r0)rr0zYou appear to be using a legacy multi-label data representation. Sequence of sequences are no longer supported; use a binary array or sparse matrix instead - the MultiLabelBinarizer transformer can convert to this format.) r:rrKeyErrorrrrr"formatrd startswithr+Zsqueeze)ZINDZMCZBINZCNTZMMCZMCNZEXAMPLESZEXPECTEDZtype1rZtype2rr`rZ merged_typeZy1outZy2outrrKrKrLtest__check_targets s$          rcCs*ddg}ddg}t||ddks&tdS)Nrr5rr^)r"rdr rKrKrLAtest__check_targets_multiclass_with_both_y_true_and_y_pred_binaryi srcCsptddddg}tddddg}t||dks6ttdd d dg}tddddg}t||dksltdS) Nrr5g!rrg333333ӿrrr0)r:rrrdrJ pred_decisionrKrKrLtest_hinge_loss_binaryp s rc Cs6tddddgddddgddd dgdd ddgd d d d gddd dgg}tddddddg}td|dd|ddd|dd|ddd|dd|ddd|dd|ddd|dd|ddd|dd|ddg}tj|dd|dt|}t|||ks2tdS)N ףp= ?(\ſ(\GzgHzGgGz׿Q޿333333RQؿzGHzGHzGѿQ?rr5r0rrroutr:rcliprrrd)rrJ dummy_lossesdummy_hinge_lossrKrKrLtest_hinge_loss_multiclassz s,         rc Csptddddg}tddddgdd d d gd d ddgd d ddgg}d}tjt|dt||W5QRXdS)Nrr5r0gRQ?g rh?g(\gffffffrrrrrrrrzDPlease include all labels in y_true or pass labels as third argumentr)r:rrrrr)rJr error_messagerKrKrL:test_hinge_loss_multiclass_missing_labels_with_labels_none s     rc Cstdddddddg}tdddddddg}d}tjtt|dt||dW5QRXtddgddgddgddgddgddgddgg}dddg}d}tjtt|dt|||dW5QRXdS) Nr0r5rzThe shape of pred_decision cannot be 1d arraywith a multiclass target. pred_decision shape must be (n_samples, n_classes), that is (7, 3). Got: (7,)rrzThe shape of pred_decision is not consistent with the number of classes. With a multiclass target, pred_decision shape must be (n_samples, n_classes), that is (7, 3). Got: (7, 2))rJrr_)r:rrrrreescaper)rJrrr_rKrKrLtd d dgd ddgd d d gd d d ggd dtjkst W5QRXd d dg}ddgddgddgg}ttt||W5QRXdd d dg}ddgddgddgddgg}tj t |dt||}W5QRXt|d!ddddg}ddgddgg}tddgddgg}d"}tjt|dt||W5QRXddgddgddgg}d#}t|t||ftt|ddd f }t||d dgd$}t||d ddg}dddgdddgdddgg} tj t |dt|| d dd%gd$}W5QRXt|d&dddS)'NnoyesrrrVg{Gz?gGz?rrgMbP?g+?r5rr0rrrrTrg躕ʀ?Fg.L`@rry_pred values do not sum to one)repsrrrhamspamL7?zly_true contains only one label \(2\). Please provide the true labels explicitly through the labels argument.zBFound input variables with inconsistent numbers of samples: [3, 2]rrgPv0?)r:rrrrZlogpmfr)Zasarrayrr FutureWarningrrrdrrrlog) rJrIlossZ loss_trueZuser_warning_msgrZ error_strZ true_log_lossZcalculated_log_lossZy_score2rKrKrL test_log_loss sv& *    $ * , , @ 0     rcCs:tjddg|d}|}t||dd}t|s6tdS)zCheck the behaviour of `eps="auto"` that changes depending on the input array dtype. Non-regression test for: https://github.com/scikit-learn/scikit-learn/issues/24315 rr5r_autorN)r:rrbrisfiniterd)Z global_dtyperJrIrrKrKrLtest_log_loss_eps_autor srcCs<tjddgtjd}|}t||dd}t|s8tdS)z2Check the behaviour of `eps="auto"` for np.float16rr5r_rrN)r:rZfloat16rbrrrd)rJrIrrKrKrLtest_log_loss_eps_auto_float16 src Cstddddg}tddgddgddgddgg}ttfg}z"d d lm}m}|||fWntk rtYnX|D]L\}}||||}}tj t d d t ||} W5QRXt | d ddqzdS)Nrrrrrrrrr) DataFramermrrrrr) r:rr'rlrrmr! ImportErrorrrrrr)) Zy_trZy_prtypesrrmZ TrueInputTypeZ PredInputTyperJrIrrKrKrLtest_log_loss_pandas_input s"  rc Cstddddddg}tddddddg}t||d t|}tt||d tt|||ttd|||ttd |d||tt t||ddW5QRXtt t||dW5QRXtt t||dW5QRXtddd dg}tdd d d g}d}tjt |dt||W5QRXttdgd gdttdgd gdttdgd gdttdgd gdddttdgd gddddS)Nrr5rrrVrrgffffff?r0rnrrrzMOnly binary classification is supported. The type of the target is multiclassrrg{Gz?rZfoobarr) r:rrZnormrbr)rrrr)rJrIZ true_scorerrKrKrLtest_brier_score_loss s0   rc Cs8d}tjt|dtdddgdddgW5QRXdS)Nz%y_pred contains classes not in y_truerrr5)rrrrrrKrKrL#test_balanced_accuracy_score_unseen srz y_true,y_predrtrurvc Cst||dt|d}tt||}W5QRX|t|ksDtt||dd}t|t||d}|||d|kstdS)NrrT)adjustedrr5) r r:uniquer-rrrrdZ full_like)rJrIZ macro_recallZbalancedrZchancerKrKrLtest_balanced_accuracy_score s rr)FTr,)rnr)zeroonecCsntjd}d|d}}|j||dd}|tkr@|j|d}n|}||||d}tt|rjt dS) zCheck that the metric works with different types of `pos_label`. We can expect `pos_label` to be a bool, an integer, a float, a string. No error should be raised for those types. *rrT)r+replacer*rN) r:r>r?choiceruniformrbanyr&rd)rlrrHrErrJrIr'rKrKrL*test_classification_metric_pos_label_types s rscoringr%)rrrcCs<tjdd\}}tddd||}t||||ddddS) aZCheck that we validate `np.nan` properly for classification metrics. With `n_jobs=2` in cross-validation, the `np.nan` used for the singleton will be different in the sub-process and we should not use the `is` operator but `math.isnan`. Non-regression test for: https://github.com/scikit-learn/scikit-learn/issues/27563 r)r4r) max_depthr4r0rX)rZn_jobsZ error_scoreN)r Zmake_classificationr&r@r#)rrCrD classifierrKrKrL9test_classification_metric_division_by_zero_nan_validaton sr)NF)rrz functoolsr itertoolsrrrnumpyr:rZscipyrZscipy.spatial.distancerrZ scipy.statsrZsklearnr r Zsklearn.datasetsr Zsklearn.exceptionsr Zsklearn.metricsr rrrrrrrrrrrrrrrrrrr r!Zsklearn.metrics._classificationr"Zsklearn.model_selectionr#Zsklearn.preprocessingr$r%Z sklearn.treer&Zsklearn.utils._mockingr'Zsklearn.utils._testingr(r)r*r+r,r-Zsklearn.utils.extmathr.Zsklearn.utils.validationr/rMrmrqmarkZ parametrizerorrrrrrrrrrrrrrrrrr rrrr#r$r(r)r.r>rDrIrQrTrVrWrYr[r\rarjrprtrurvrxr~rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrKrKrKrLs&      \        0A "  -        #)     -       '#8 !:     #  (M%  B @ ^$  b ? & ' 'b  Z  #&&&