U ^{jߕ @sdZddlZddlZddlZddlZddlmZmZddlZ ddl m Z ddl m Z ddlmZddlmZdd lmZdd lmZmZdd lmZmZmZdd lmZmZdd lmZm Z m!Z!ddl"m#Z#m$Z$m%Z%ddZ&ddZ'dddddde (e j)j*dddZ+ddZ,edgddgdgdgddd ddddddde (e j)j*dd! d"d#Z-Gd$d%d%e#Z.Gd&d'd'e.Z/ddddddde (e j)j*fd(d)Z0Gd*d+d+e.Z1dS),zUGraphicalLasso: sparse inverse covariance estimation with an l1-penalized estimator. N)IntegralReal)linalg) _fit_context)ConvergenceWarning)_cd_fast)lars_path_gram)check_cvcross_val_score)Interval StrOptionsvalidate_params)Paralleldelayed)_is_arraylike_not_scalarcheck_random_state check_scalar)EmpiricalCovarianceempirical_covariancelog_likelihoodcCsZ|jd}dt|||tdtj}||t|tt|7}|S)zEvaluation of the graphical-lasso objective function the objective function is made of a shifted scaled version of the normalized log-likelihood (i.e. its empirical mean over the samples) and a penalisation term to promote sparsity rr)shapernplogpiabssumdiag)Zmle precision_alphapcostr$^/var/www/html/bwave/python/venv/lib/python3.8/site-packages/sklearn/covariance/_graph_lasso.py _objective$s "*r&cCsJt||}||jd8}||t|tt|7}|S)zExpression of the dual gap convergence criterion The specific definition is given in Duchi "Projected Subgradient Methods for Learning Sparse Gaussians". r)rrrrr)emp_covr r!gapr$r$r% _dual_gap1s*r)cd-C6?dF)cov_initmodetolenet_tolmax_iterverboseepscCs|j\} } |dkrdt|} dt|| } | | tdtj7} t|| | } || | | fdfS|dkrv|}n|}|d9}|j dd| d}||j dd| d<t |} t | }d}t }|dkrt ddd }n t dd }ztj} tj|ddddfd d }t|D]L}t| D]}|dkr||d}||||k||<|dd|f||k|dd|f<n|ddddf|dd<||||kf}tjf||dkr| ||k|f| ||fd | }t||d|||||tdd \}} } } n(t|||j|| dd|ddd\} } }W5QRXd|||ft|||k|f|| ||f<| ||f || ||k|f<| ||f || |||kf<t||}|||||kf<||||k|f<q*t| stdt|| |} t|| |} |r&td|| | f|| | ft| |krJqt| s|dkrtdqtd|| ft Wn:tk r}z|j!ddf|_!|W5d}~XYnX|| ||dfS)Nrrrgffffff?rr*raiseignore)Zoverinvalid)r6C)orderiFTlars)ZXyZGramZ n_samplesZ alpha_minZ copy_Gramr3methodZ return_pathg?z1The system is too ill-conditioned for this solverz<[graphical_lasso] Iteration % 3i, cost % 3.2e, dual gap %.3ezANon SPD result: the system is too ill-conditioned for this solverzDgraphical_lasso: did not converge after %i iteration: dual gap: %.3ez3. The system is too ill-conditioned for this solver)"rrinvrrrrrcopyflatZpinvhZarangelistdictinfrangeZerrstatecd_fastZenet_coordinate_descent_gramrr sizedotisfiniteFloatingPointErrorr)r&printappendrwarningswarnrargs)r'r!r-r.r/r0r1r2r3_Z n_featuresr r#Zd_gap covariance_ZdiagonalindicesiZcostserrorsZsub_covarianceidxZdirowZcoefser$r$r%_graphical_lasso>s       &         rTcCs4t|}d|jdd|jdd<tt|S)aFind the maximum alpha for which there are some non-zeros off-diagonal. Parameters ---------- emp_cov : ndarray of shape (n_features, n_features) The sample covariance matrix. Notes ----- This results from the bound for the all the Lasso that are solved in GraphicalLasso: each time, the row of cov corresponds to Xy. As the bound for alpha is given by `max(abs(Xy))`, the result follows. rNr)rr<r=rmaxr)r'Ar$r$r% alpha_maxs rW array-likeboolean)r'r- return_costs return_n_iterZprefer_skip_nested_validation) r-r.r/r0r1r2rZr3r[c Csh|dk rtdtt||d||||| dd |} | j| jg} |rP| | j| r`| | j t | S)a L1-penalized covariance estimator. Read more in the :ref:`User Guide `. .. versionchanged:: v0.20 graph_lasso has been renamed to graphical_lasso Parameters ---------- emp_cov : array-like of shape (n_features, n_features) Empirical covariance from which to compute the covariance estimate. alpha : float The regularization parameter: the higher alpha, the more regularization, the sparser the inverse covariance. Range is (0, inf]. cov_init : array of shape (n_features, n_features), default=None The initial guess for the covariance. If None, then the empirical covariance is used. .. deprecated:: 1.3 `cov_init` is deprecated in 1.3 and will be removed in 1.5. It currently has no effect. mode : {'cd', 'lars'}, default='cd' The Lasso solver to use: coordinate descent or LARS. Use LARS for very sparse underlying graphs, where p > n. Elsewhere prefer cd which is more numerically stable. tol : float, default=1e-4 The tolerance to declare convergence: if the dual gap goes below this value, iterations are stopped. Range is (0, inf]. enet_tol : float, default=1e-4 The tolerance for the elastic net solver used to calculate the descent direction. This parameter controls the accuracy of the search direction for a given column update, not of the overall parameter estimate. Only used for mode='cd'. Range is (0, inf]. max_iter : int, default=100 The maximum number of iterations. verbose : bool, default=False If verbose is True, the objective function and dual gap are printed at each iteration. return_costs : bool, default=False If return_costs is True, the objective function and dual gap at each iteration are returned. eps : float, default=eps The machine-precision regularization in the computation of the Cholesky diagonal factors. Increase this for very ill-conditioned systems. Default is `np.finfo(np.float64).eps`. return_n_iter : bool, default=False Whether or not to return the number of iterations. Returns ------- covariance : ndarray of shape (n_features, n_features) The estimated covariance matrix. precision : ndarray of shape (n_features, n_features) The estimated (sparse) precision matrix. costs : list of (objective, dual_gap) pairs The list of values of the objective function and the dual gap at each iteration. Returned only if return_costs is True. n_iter : int Number of iterations. Returned only if `return_n_iter` is set to True. See Also -------- GraphicalLasso : Sparse inverse covariance estimation with an l1-penalized estimator. GraphicalLassoCV : Sparse inverse covariance with cross-validated choice of the l1 penalty. Notes ----- The algorithm employed to solve this problem is the GLasso algorithm, from the Friedman 2008 Biostatistics paper. It is the same algorithm as in the R `glasso` package. One possible difference with the `glasso` R package is that the diagonal coefficients are not penalized. NzdThe cov_init parameter is deprecated in 1.3 and will be removed in 1.5. It does not have any effect. precomputedT) r!r. covariancer/r0r1r2r3assume_centered) rIrJ FutureWarningGraphicalLassofitrMr rHcosts_n_iter_tuple) r'r!r-r.r/r0r1r2rZr3r[modeloutputr$r$r%graphical_lassos0r    rhc seZdZUejeeddddgeeddddgeeddddgeddhgdgeeddd dgd Ze e d <e d d d ddde e jjdffdd ZZS)BaseGraphicalLassorNrightclosedleftr*r9r2both)r/r0r1r.r2r3_parameter_constraintsZstore_precisionr+r,Fcs6tj|d||_||_||_||_||_||_dS)Nr_)super__init__r/r0r1r.r2r3)selfr/r0r1r.r2r3r_ __class__r$r%rrqs zBaseGraphicalLasso.__init__)__name__ __module__ __qualname__rror rrr r?__annotations__poprfinfofloat64r3rr __classcell__r$r$rtr%ries"    ric seZdZUdZejeeddddgedhdgdZe e d<dd dd d d d e e j jd dfdd ZedddddZZS)raaSparse inverse covariance estimation with an l1-penalized estimator. Read more in the :ref:`User Guide `. .. versionchanged:: v0.20 GraphLasso has been renamed to GraphicalLasso Parameters ---------- alpha : float, default=0.01 The regularization parameter: the higher alpha, the more regularization, the sparser the inverse covariance. Range is (0, inf]. mode : {'cd', 'lars'}, default='cd' The Lasso solver to use: coordinate descent or LARS. Use LARS for very sparse underlying graphs, where p > n. Elsewhere prefer cd which is more numerically stable. covariance : "precomputed", default=None If covariance is "precomputed", the input data in `fit` is assumed to be the covariance matrix. If `None`, the empirical covariance is estimated from the data `X`. .. versionadded:: 1.3 tol : float, default=1e-4 The tolerance to declare convergence: if the dual gap goes below this value, iterations are stopped. Range is (0, inf]. enet_tol : float, default=1e-4 The tolerance for the elastic net solver used to calculate the descent direction. This parameter controls the accuracy of the search direction for a given column update, not of the overall parameter estimate. Only used for mode='cd'. Range is (0, inf]. max_iter : int, default=100 The maximum number of iterations. verbose : bool, default=False If verbose is True, the objective function and dual gap are plotted at each iteration. eps : float, default=eps The machine-precision regularization in the computation of the Cholesky diagonal factors. Increase this for very ill-conditioned systems. Default is `np.finfo(np.float64).eps`. .. versionadded:: 1.3 assume_centered : bool, default=False If True, data are not centered before computation. Useful when working with data whose mean is almost, but not exactly zero. If False, data are centered before computation. Attributes ---------- location_ : ndarray of shape (n_features,) Estimated location, i.e. the estimated mean. covariance_ : ndarray of shape (n_features, n_features) Estimated covariance matrix precision_ : ndarray of shape (n_features, n_features) Estimated pseudo inverse matrix. n_iter_ : int Number of iterations run. costs_ : list of (objective, dual_gap) pairs The list of values of the objective function and the dual gap at each iteration. Returned only if return_costs is True. .. versionadded:: 1.3 n_features_in_ : int Number of features seen during :term:`fit`. .. versionadded:: 0.24 feature_names_in_ : ndarray of shape (`n_features_in_`,) Names of features seen during :term:`fit`. Defined only when `X` has feature names that are all strings. .. versionadded:: 1.0 See Also -------- graphical_lasso : L1-penalized covariance estimator. GraphicalLassoCV : Sparse inverse covariance with cross-validated choice of the l1 penalty. Examples -------- >>> import numpy as np >>> from sklearn.covariance import GraphicalLasso >>> true_cov = np.array([[0.8, 0.0, 0.2, 0.0], ... [0.0, 0.4, 0.0, 0.0], ... [0.2, 0.0, 0.3, 0.1], ... [0.0, 0.0, 0.1, 0.7]]) >>> np.random.seed(0) >>> X = np.random.multivariate_normal(mean=[0, 0, 0, 0], ... cov=true_cov, ... size=200) >>> cov = GraphicalLasso().fit(X) >>> np.around(cov.covariance_, decimals=3) array([[0.816, 0.049, 0.218, 0.019], [0.049, 0.364, 0.017, 0.034], [0.218, 0.017, 0.322, 0.093], [0.019, 0.034, 0.093, 0.69 ]]) >>> np.around(cov.location_, decimals=3) array([0.073, 0.04 , 0.038, 0.143]) rNrnrkr])r!r^ro{Gz?r*r+r,F)r.r^r/r0r1r2r3r_c s*tj||||||| d||_||_dSN)r/r0r1r.r2r3r_)rqrrr!r^) rsr!r.r^r/r0r1r2r3r_rtr$r%rrs  zGraphicalLasso.__init__Tr\c Cs|j|ddd}|jdkr6|}t|jd|_n4t||jd}|jr^t|jd|_n | d|_t ||j d|j |j |j|j|j|jd \|_|_|_|_|S) aFit the GraphicalLasso model to X. Parameters ---------- X : array-like of shape (n_samples, n_features) Data from which to compute the covariance estimate. y : Ignored Not used, present for API consistency by convention. Returns ------- self : object Returns the instance itself. r)ensure_min_featuresZensure_min_samplesr]rrprNr!r-r.r/r0r1r2r3)_validate_datar^r<rzerosr location_rr_meanrTr!r.r/r0r1r2r3rMr rcrd)rsXyr'r$r$r%rbs(   zGraphicalLasso.fit)r~)N)rvrwrx__doc__riror rr r?ryrr{r|r3rrrrbr}r$r$rtr%ras$ t  rac  Csbtd|d} t|} |dkr(| } n|} t} t}t}|dk rNt|}|D]}zLt| || ||||| | d \} }}}| | |||dk rt||}Wn4tk rtj }| tj |tj YnX|dk rt |stj }|||dkrt j dqR|dkrR|dk r8td||fqRtd|qR|dk rZ| ||fS| |fS)a l1-penalized covariance estimator along a path of decreasing alphas Read more in the :ref:`User Guide `. Parameters ---------- X : ndarray of shape (n_samples, n_features) Data from which to compute the covariance estimate. alphas : array-like of shape (n_alphas,) The list of regularization parameters, decreasing order. cov_init : array of shape (n_features, n_features), default=None The initial guess for the covariance. X_test : array of shape (n_test_samples, n_features), default=None Optional test matrix to measure generalisation error. mode : {'cd', 'lars'}, default='cd' The Lasso solver to use: coordinate descent or LARS. Use LARS for very sparse underlying graphs, where p > n. Elsewhere prefer cd which is more numerically stable. tol : float, default=1e-4 The tolerance to declare convergence: if the dual gap goes below this value, iterations are stopped. The tolerance must be a positive number. enet_tol : float, default=1e-4 The tolerance for the elastic net solver used to calculate the descent direction. This parameter controls the accuracy of the search direction for a given column update, not of the overall parameter estimate. Only used for mode='cd'. The tolerance must be a positive number. max_iter : int, default=100 The maximum number of iterations. This parameter should be a strictly positive integer. verbose : int or bool, default=False The higher the verbosity flag, the more information is printed during the fitting. eps : float, default=eps The machine-precision regularization in the computation of the Cholesky diagonal factors. Increase this for very ill-conditioned systems. Default is `np.finfo(np.float64).eps`. .. versionadded:: 1.3 Returns ------- covariances_ : list of shape (n_alphas,) of ndarray of shape (n_features, n_features) The estimated covariance matrices. precisions_ : list of shape (n_alphas,) of ndarray of shape (n_features, n_features) The estimated (sparse) precision matrices. scores_ : list of shape (n_alphas,), dtype=float The generalisation error (log-likelihood) on the test data. Returned only if test data is passed. rrNr.z/[graphical_lasso_path] alpha: %.2e, score: %.2ez"[graphical_lasso_path] alpha: %.2e)rUrr<r>rTrHrrFrr@nanrEsysstderrwriterG)ralphasr-X_testr.r/r0r1r2r3 inner_verboser'rMZ covariances_Z precisions_Zscores_Z test_emp_covr!r rL this_scorer$r$r%graphical_lasso_pathDs`K          rc seZdZUdZejeedddddgeeddddgdgedgd Zee d <d d dd d d ddde e j j dd fdd ZedddddZZS)GraphicalLassoCVa?Sparse inverse covariance w/ cross-validated choice of the l1 penalty. See glossary entry for :term:`cross-validation estimator`. Read more in the :ref:`User Guide `. .. versionchanged:: v0.20 GraphLassoCV has been renamed to GraphicalLassoCV Parameters ---------- alphas : int or array-like of shape (n_alphas,), dtype=float, default=4 If an integer is given, it fixes the number of points on the grids of alpha to be used. If a list is given, it gives the grid to be used. See the notes in the class docstring for more details. Range is [1, inf) for an integer. Range is (0, inf] for an array-like of floats. n_refinements : int, default=4 The number of times the grid is refined. Not used if explicit values of alphas are passed. Range is [1, inf). cv : int, cross-validation generator or iterable, default=None Determines the cross-validation splitting strategy. Possible inputs for cv are: - None, to use the default 5-fold cross-validation, - integer, to specify the number of folds. - :term:`CV splitter`, - An iterable yielding (train, test) splits as arrays of indices. For integer/None inputs :class:`~sklearn.model_selection.KFold` is used. Refer :ref:`User Guide ` for the various cross-validation strategies that can be used here. .. versionchanged:: 0.20 ``cv`` default value if None changed from 3-fold to 5-fold. tol : float, default=1e-4 The tolerance to declare convergence: if the dual gap goes below this value, iterations are stopped. Range is (0, inf]. enet_tol : float, default=1e-4 The tolerance for the elastic net solver used to calculate the descent direction. This parameter controls the accuracy of the search direction for a given column update, not of the overall parameter estimate. Only used for mode='cd'. Range is (0, inf]. max_iter : int, default=100 Maximum number of iterations. mode : {'cd', 'lars'}, default='cd' The Lasso solver to use: coordinate descent or LARS. Use LARS for very sparse underlying graphs, where number of features is greater than number of samples. Elsewhere prefer cd which is more numerically stable. n_jobs : int, default=None Number of jobs to run in parallel. ``None`` means 1 unless in a :obj:`joblib.parallel_backend` context. ``-1`` means using all processors. See :term:`Glossary ` for more details. .. versionchanged:: v0.20 `n_jobs` default changed from 1 to None verbose : bool, default=False If verbose is True, the objective function and duality gap are printed at each iteration. eps : float, default=eps The machine-precision regularization in the computation of the Cholesky diagonal factors. Increase this for very ill-conditioned systems. Default is `np.finfo(np.float64).eps`. .. versionadded:: 1.3 assume_centered : bool, default=False If True, data are not centered before computation. Useful when working with data whose mean is almost, but not exactly zero. If False, data are centered before computation. Attributes ---------- location_ : ndarray of shape (n_features,) Estimated location, i.e. the estimated mean. covariance_ : ndarray of shape (n_features, n_features) Estimated covariance matrix. precision_ : ndarray of shape (n_features, n_features) Estimated precision matrix (inverse covariance). costs_ : list of (objective, dual_gap) pairs The list of values of the objective function and the dual gap at each iteration. Returned only if return_costs is True. .. versionadded:: 1.3 alpha_ : float Penalization parameter selected. cv_results_ : dict of ndarrays A dict with keys: alphas : ndarray of shape (n_alphas,) All penalization parameters explored. split(k)_test_score : ndarray of shape (n_alphas,) Log-likelihood score on left-out data across (k)th fold. .. versionadded:: 1.0 mean_test_score : ndarray of shape (n_alphas,) Mean of scores over the folds. .. versionadded:: 1.0 std_test_score : ndarray of shape (n_alphas,) Standard deviation of scores over the folds. .. versionadded:: 1.0 n_iter_ : int Number of iterations run for the optimal alpha. n_features_in_ : int Number of features seen during :term:`fit`. .. versionadded:: 0.24 feature_names_in_ : ndarray of shape (`n_features_in_`,) Names of features seen during :term:`fit`. Defined only when `X` has feature names that are all strings. .. versionadded:: 1.0 See Also -------- graphical_lasso : L1-penalized covariance estimator. GraphicalLasso : Sparse inverse covariance estimation with an l1-penalized estimator. Notes ----- The search for the optimal penalization parameter (`alpha`) is done on an iteratively refined grid: first the cross-validated scores on a grid are computed, then a new refined grid is centered around the maximum, and so on. One of the challenges which is faced here is that the solvers can fail to converge to a well-conditioned estimate. The corresponding values of `alpha` then come out as missing values, but the optimum may be close to these missing values. In `fit`, once the best parameter `alpha` is found through cross-validation, the model is fit again using the entire training set. Examples -------- >>> import numpy as np >>> from sklearn.covariance import GraphicalLassoCV >>> true_cov = np.array([[0.8, 0.0, 0.2, 0.0], ... [0.0, 0.4, 0.0, 0.0], ... [0.2, 0.0, 0.3, 0.1], ... [0.0, 0.0, 0.1, 0.7]]) >>> np.random.seed(0) >>> X = np.random.multivariate_normal(mean=[0, 0, 0, 0], ... cov=true_cov, ... size=200) >>> cov = GraphicalLassoCV().fit(X) >>> np.around(cov.covariance_, decimals=3) array([[0.816, 0.051, 0.22 , 0.017], [0.051, 0.364, 0.018, 0.036], [0.22 , 0.018, 0.322, 0.094], [0.017, 0.036, 0.094, 0.69 ]]) >>> np.around(cov.location_, decimals=3) array([0.073, 0.04 , 0.038, 0.143]) rNrmrkrXrZ cv_object)r n_refinementscvn_jobsror+r,r*F) rrrr/r0r1r.rr2r3r_c s6tj||||| | | d||_||_||_||_dSr)rqrrrrrr) rsrrrr/r0r1r.rr2r3r_rtr$r%rrs zGraphicalLassoCV.__init__Tr\c s jddjr(tjd_n d_tjd}tj |dd}t }j }t dj dt|rj D]}t|dtdtjd d q|j d}n:j}t|} d | } tt| t| |d d d t} t|D]} tBtdttjj dfdd||D} W5QRXt| \}}}t|}t|}| t||t!|t"#ddd}tj }d}t$|D]Z\}\}}}t|}|dt%tj&j'krtj(}t)|r|}||kr|}|}q|dkr|dd} |dd} n||krV|t*|dksV||d} ||dd} nP|t*|dkr||d} d ||d} n ||dd} ||dd} t|stt| t| |ddd j r|dkrt+d| d|t| fqt t|}t |d}t |d,d|,t-t.|jdt/|}dt/i_0t|jdD]$} |d d | fj0d| d<qztj|ddj0d<tj1|ddj0d<|}|_2t3||j4j5j6j7j'd\_8_9_:_;S)aFit the GraphicalLasso covariance model to X. Parameters ---------- X : array-like of shape (n_samples, n_features) Data from which to compute the covariance estimate. y : Ignored Not used, present for API consistency by convention. Returns ------- self : object Returns the instance itself. r)rrrrpF) classifierr!rj)Zmin_valZmax_valZinclude_boundariesr~Nr5)rr2c 3sJ|]B\}}tt||jjjtdjjd VqdS)皙?)rrr.r/r0r1r2r3N)rrr.r/r0intr1r3).0traintestrrrrsr$r% s  z'GraphicalLassoCV.fit..T)keyreverserz8[GraphicalLassoCV] Done refinement % 2i out of %i: % 3is)rrr2rsplitZ _test_score)ZaxisZmean_test_scoreZstd_test_score)r!r.r/r0r1r2r3)rrUr2rrrr@rrWZlogspacelog10timerArIcatch_warnings simplefilterrrrrzipextendsortedoperator itemgetter enumerater{r|r3rrElenrGrHr rarrayZ cv_results_ZstdZalpha_rTr.r/r0r1rMr rcrd)rsrrr'rpathZn_alphasr!rZalpha_1Zalpha_0t0rOZ this_pathZcovsrLZscoresZ best_scoreZlast_finite_idxindexrZ best_indexZ grid_scoresZ best_alphar$rr%rbs  $                 " zGraphicalLassoCV.fit)N)rvrwrxrriror rr?ryrr{r|r3rrrrbr}r$r$rtr%rs, 8  r)2rrrrrInumbersrrnumpyrZscipyrbaser exceptionsrZ linear_modelrrBr Zmodel_selectionr r Zutils._param_validationr r rZutils.parallelrrZutils.validationrrrrrrr&r)r{r|r3rTrWrhrirarrr$r$r$r%sz         D