U ʹj<@sdZddlZddlZddlZddlmZddlmZddlm Z ddl m Z m Z m Z mZmZmZmZddlmZmZdd lmZdd lmZd d lmZeeZed ZdgZeddGdddZ eddGdddZ!eddGdddZ"eddGdddZ#eddGdddZ$d+eee%e fe$dddZ&e e"ddd Z'e(e%d!d"d#Z)d$Z*e+e%d%d&d'Z,e e%dd(d)d*Z-dS),z4Contains utilities to manage the HF cache directory.N) defaultdict) dataclass)Path)Dict FrozenSetListLiteralOptionalSetUnion) CacheNotFoundCorruptedCacheException)tabulate) HF_HUB_CACHE)logging)modeldatasetspacez .DS_StoreT)frozenc@sxeZdZUdZeed<eed<eed<eed<eed<eed<e edd d Z e edd d Z e edd dZ dS)CachedFileInfoaFrozen data structure holding information about a single cached file. Args: file_name (`str`): Name of the file. Example: `config.json`. file_path (`Path`): Path of the file in the `snapshots` directory. The file path is a symlink referring to a blob in the `blobs` folder. blob_path (`Path`): Path of the blob file. This is equivalent to `file_path.resolve()`. size_on_disk (`int`): Size of the blob file in bytes. blob_last_accessed (`float`): Timestamp of the last time the blob file has been accessed (from any revision). blob_last_modified (`float`): Timestamp of the last time the blob file has been modified/created. > [!WARNING] > `blob_last_accessed` and `blob_last_modified` reliability can depend on the OS you > are using. See [python documentation](https://docs.python.org/3/library/os.html#os.stat_result) > for more details. file_name file_path blob_path size_on_diskblob_last_accessedblob_last_modifiedreturncCs t|jS)z (property) Timestamp of the last time the blob file has been accessed (from any revision), returned as a human-readable string. Example: "2 weeks ago". )_format_timesincerselfr#c/var/www/html/bwave/python/venv/lib/python3.8/site-packages/huggingface_hub/utils/_cache_manager.pyblob_last_accessed_strJsz%CachedFileInfo.blob_last_accessed_strcCs t|jS)z (property) Timestamp of the last time the blob file has been modified, returned as a human-readable string. Example: "2 weeks ago". )r rr!r#r#r$blob_last_modified_strTsz%CachedFileInfo.blob_last_modified_strcCs t|jS)zi (property) Size of the blob file as a human-readable string. Example: "42.2K".  _format_sizerr!r#r#r$size_on_disk_str^szCachedFileInfo.size_on_disk_strN) __name__ __module__ __qualname____doc__str__annotations__rintfloatpropertyr%r&r)r#r#r#r$r(s   rc@seZdZUdZeed<eed<eed<ee ed<eeed<e ed<e edd d Z e edd d Z e edd dZdS)CachedRevisionInfoa.Frozen data structure holding information about a revision. A revision correspond to a folder in the `snapshots` folder and is populated with the exact tree structure as the repo on the Hub but contains only symlinks. A revision can be either referenced by 1 or more `refs` or be "detached" (no refs). Args: commit_hash (`str`): Hash of the revision (unique). Example: `"9338f7b671827df886678df2bdd7cc7b4f36dffd"`. snapshot_path (`Path`): Path to the revision directory in the `snapshots` folder. It contains the exact tree structure as the repo on the Hub. files: (`FrozenSet[CachedFileInfo]`): Set of [`~CachedFileInfo`] describing all files contained in the snapshot. refs (`FrozenSet[str]`): Set of `refs` pointing to this revision. If the revision has no `refs`, it is considered detached. Example: `{"main", "2.4.0"}` or `{"refs/pr/1"}`. size_on_disk (`int`): Sum of the blob file sizes that are symlink-ed by the revision. last_modified (`float`): Timestamp of the last time the revision has been created/modified. > [!WARNING] > `last_accessed` cannot be determined correctly on a single revision as blob files > are shared across revisions. > [!WARNING] > `size_on_disk` is not necessarily the sum of all file sizes because of possible > duplicated files. Besides, only blobs are taken into account, not the (negligible) > size of folders and symlinks. commit_hash snapshot_pathrfilesrefs last_modifiedrcCs t|jS)z (property) Timestamp of the last time the revision has been modified, returned as a human-readable string. Example: "2 weeks ago". r r8r!r#r#r$last_modified_strsz$CachedRevisionInfo.last_modified_strcCs t|jSzn (property) Sum of the blob file sizes as a human-readable string. Example: "42.2K". r'r!r#r#r$r)sz#CachedRevisionInfo.size_on_disk_strcCs t|jS)zC (property) Total number of files in the revision. )lenr6r!r#r#r$nb_filesszCachedRevisionInfo.nb_filesN)r*r+r,r-r.r/rr0rrr1r2r:r)r=r#r#r#r$r3hs "   r3c@seZdZUdZeed<eed<eed<eed<eed<e e ed<e ed<e ed <e ed d d Z e ed d dZe ed ddZe eee fd ddZdS)CachedRepoInfoaFFrozen data structure holding information about a cached repository. Args: repo_id (`str`): Repo id of the repo on the Hub. Example: `"google/fleurs"`. repo_type (`Literal["dataset", "model", "space"]`): Type of the cached repo. repo_path (`Path`): Local path to the cached repo. size_on_disk (`int`): Sum of the blob file sizes in the cached repo. nb_files (`int`): Total number of blob files in the cached repo. revisions (`FrozenSet[CachedRevisionInfo]`): Set of [`~CachedRevisionInfo`] describing all revisions cached in the repo. last_accessed (`float`): Timestamp of the last time a blob file of the repo has been accessed. last_modified (`float`): Timestamp of the last time a blob file of the repo has been modified/created. > [!WARNING] > `size_on_disk` is not necessarily the sum of all revisions sizes because of > duplicated files. Besides, only blobs are taken into account, not the (negligible) > size of folders and symlinks. > [!WARNING] > `last_accessed` and `last_modified` reliability can depend on the OS you are using. > See [python documentation](https://docs.python.org/3/library/os.html#os.stat_result) > for more details. repo_id repo_type repo_pathrr= revisions last_accessedr8rcCs t|jS)z (property) Last time a blob file of the repo has been accessed, returned as a human-readable string. Example: "2 weeks ago". )r rCr!r#r#r$last_accessed_strsz CachedRepoInfo.last_accessed_strcCs t|jS)z (property) Last time a blob file of the repo has been modified, returned as a human-readable string. Example: "2 weeks ago". r9r!r#r#r$r:sz CachedRepoInfo.last_modified_strcCs t|jSr;r'r!r#r#r$r)szCachedRepoInfo.size_on_disk_strcCsdd|jDS)zQ (property) Mapping between `refs` and revision data structures. cSsi|]}|jD] }||qqSr#)r7).0revisionrefr#r#r$ s z'CachedRepoInfo.refs..)rBr!r#r#r$r7szCachedRepoInfo.refsN)r*r+r,r-r.r/ REPO_TYPE_Trr0rr3r1r2rDr:r)rr7r#r#r#r$r>s"    r>c@sjeZdZUdZeed<eeed<eeed<eeed<eeed<ee ddd Z d dd d Z d S) DeleteCacheStrategyaFrozen data structure holding the strategy to delete cached revisions. This object is not meant to be instantiated programmatically but to be returned by [`~utils.HFCacheInfo.delete_revisions`]. See documentation for usage example. Args: expected_freed_size (`float`): Expected freed size once strategy is executed. blobs (`FrozenSet[Path]`): Set of blob file paths to be deleted. refs (`FrozenSet[Path]`): Set of reference file paths to be deleted. repos (`FrozenSet[Path]`): Set of entire repo paths to be deleted. snapshots (`FrozenSet[Path]`): Set of snapshots to be deleted (directory of symlinks). expected_freed_sizeblobsr7repos snapshotsrcCs t|jS)zt (property) Expected size that will be freed as a human-readable string. Example: "42.2K". )r(rKr!r#r#r$expected_freed_size_strsz+DeleteCacheStrategy.expected_freed_size_strNcCsx|jD]}t|ddq|jD]}t|ddq|jD]}t|ddq6|jD]}t|ddqNtd|jddS) aExecute the defined strategy. > [!WARNING] > If this method is interrupted, the cache might get corrupted. Deletion order is > implemented so that references and symlinks are deleted before the actual blob > files. > [!WARNING] > This method is irreversible. If executed, cached files are erased and must be > downloaded again. repo) path_typeZsnapshotrGZblobzCache deletion done. Saved .N)rM_try_delete_pathrNr7rLloggerinforO)r"pathr#r#r$execute"s    zDeleteCacheStrategy.execute) r*r+r,r-r0r/rrr2r.rOrWr#r#r#r$rJs     rJc@sjeZdZUdZeed<eeed<ee ed<e e dddZ e e dd d Zd d ee d ddZdS) HFCacheInfoasFrozen data structure holding information about the entire cache-system. This data structure is returned by [`scan_cache_dir`] and is immutable. Args: size_on_disk (`int`): Sum of all valid repo sizes in the cache-system. repos (`FrozenSet[CachedRepoInfo]`): Set of [`~CachedRepoInfo`] describing all valid cached repos found on the cache-system while scanning. warnings (`List[CorruptedCacheException]`): List of [`~CorruptedCacheException`] that occurred while scanning the cache. Those exceptions are captured so that the scan can continue. Corrupted repos are skipped from the scan. > [!WARNING] > Here `size_on_disk` is equal to the sum of all repo sizes (only blobs). However if > some cached repos are corrupted, their sizes are not taken into account. rrMwarningsrcCs t|jS)z (property) Sum of all valid repo sizes in the cache-system as a human-readable string. Example: "42.2K". r'r!r#r#r$r)_szHFCacheInfo.size_on_disk_str)rBrcGst|}tt}|jD]4}|jD](}|j|kr |||||jq qt|dkrnt dd |t}t}t}t} d} | D]\} } | j| } t| dkr|| j | | j 7} q| D]}| |j|jD]}|| j d|q|jD]p}|j|krd}| D]8}|jD]}|j|jkr$d}qDq$|sqTq|r||j| |j 7} qqqtt|t|t|t| | dS)aPrepare the strategy to delete one or more revisions cached locally. Input revisions can be any revision hash. If a revision hash is not found in the local cache, a warning is thrown but no error is raised. Revisions can be from different cached repos since hashes are unique across repos, Examples: ```py >>> from huggingface_hub import scan_cache_dir >>> cache_info = scan_cache_dir() >>> delete_strategy = cache_info.delete_revisions( ... "81fd1d6e7847c99f5862c9fb81387956d99ec7aa" ... ) >>> print(f"Will free {delete_strategy.expected_freed_size_str}.") Will free 7.9K. >>> delete_strategy.execute() Cache deletion done. Saved 7.9K. ``` ```py >>> from huggingface_hub import scan_cache_dir >>> scan_cache_dir().delete_revisions( ... "81fd1d6e7847c99f5862c9fb81387956d99ec7aa", ... "e2983b237dccf3ab4937c97fa717319a9ca1a96d", ... "6c0e6080953db56375760c0471a8c5f2929baf11", ... ).execute() Cache deletion done. Saved 8.6G. ``` > [!WARNING] > `delete_revisions` returns a [`~utils.DeleteCacheStrategy`] object that needs to > be executed. The [`~utils.DeleteCacheStrategy`] is not meant to be modified but > allows having a dry run before actually executing the deletion. rz,Revision(s) not found - cannot delete them: , r7TF)rLr7rMrNrK)setrrMrBr4addremover<rTwarningjoinitemsrArr5r7r6rrJ frozenset)r"rBZhashes_to_deleteZrepos_with_revisionsrPrFZdelete_strategy_blobsZdelete_strategy_refsZdelete_strategy_reposZdelete_strategy_snapshotsZ#delete_strategy_expected_freed_sizeZ affected_repoZrevisions_to_deleteZother_revisionsZrevision_to_deleterGfileZ is_file_aloneZrev_filer#r#r$delete_revisionsisX#                zHFCacheInfo.delete_revisionsr) verbosity)rdrc Cst|dkr>> from huggingface_hub.utils import scan_cache_dir >>> hf_cache_info = scan_cache_dir() HFCacheInfo(...) >>> print(hf_cache_info.export_as_table()) REPO ID REPO TYPE SIZE ON DISK NB FILES LAST_ACCESSED LAST_MODIFIED REFS LOCAL PATH --------------------------------------------------- --------- ------------ -------- ------------- ------------- ---- -------------------------------------------------------------------------------------------------- roberta-base model 2.7M 5 1 day ago 1 week ago main ~/.cache/huggingface/hub/models--roberta-base suno/bark model 8.8K 1 1 week ago 1 week ago main ~/.cache/huggingface/hub/models--suno--bark t5-base model 893.8M 4 4 days ago 7 months ago main ~/.cache/huggingface/hub/models--t5-base t5-large model 3.0G 4 5 weeks ago 5 months ago main ~/.cache/huggingface/hub/models--t5-large >>> print(hf_cache_info.export_as_table(verbosity=1)) REPO ID REPO TYPE REVISION SIZE ON DISK NB FILES LAST_MODIFIED REFS LOCAL PATH --------------------------------------------------- --------- ---------------------------------------- ------------ -------- ------------- ---- ----------------------------------------------------------------------------------------------------------------------------------------------------- roberta-base model e2da8e2f811d1448a5b465c236feacd80ffbac7b 2.7M 5 1 week ago main ~/.cache/huggingface/hub/models--roberta-base/snapshots/e2da8e2f811d1448a5b465c236feacd80ffbac7b suno/bark model 70a8a7d34168586dc5d028fa9666aceade177992 8.8K 1 1 week ago main ~/.cache/huggingface/hub/models--suno--bark/snapshots/70a8a7d34168586dc5d028fa9666aceade177992 t5-base model a9723ea7f1b39c1eae772870f3b547bf6ef7e6c1 893.8M 4 7 months ago main ~/.cache/huggingface/hub/models--t5-base/snapshots/a9723ea7f1b39c1eae772870f3b547bf6ef7e6c1 t5-large model 150ebc2c4b72291e770f58e6057481c8d2ed331a 3.0G 4 5 months ago main ~/.cache/huggingface/hub/models--t5-large/snapshots/150ebc2c4b72291e770f58e6057481c8d2ed331a ``` Args: verbosity (`int`, *optional*): The verbosity level. Defaults to 0. Returns: `str`: The table as a string. rc SsDg|]<}|j|jd|j|j|j|jdt|j t |j gqS){:>12}rZ) r?r@formatr)r=rDr:r_sortedr7r.rArErPr#r#r$ s  z/HFCacheInfo.export_as_table..cSs|jSNrArPr#r#r$z-HFCacheInfo.export_as_table..keyzREPO IDz REPO TYPEz SIZE ON DISKzNB FILESZ LAST_ACCESSEDZ LAST_MODIFIEDZREFSz LOCAL PATH)rowsheadersc Ss\g|]T}t|jdddD]<}|j|j|jd|j|j|jd t|j t |j gqqS)cSs|jSrj)r4)rFr#r#r$rmrnz8HFCacheInfo.export_as_table...rorerZ) rgrBr?r@r4rfr)r=r:r_r7r.r5)rErPrFr#r#r$ris  cSs|jSrjrkrlr#r#r$rmrnZREVISIONN)rrgrM)r"rdr#r#r$export_as_tables:(  zHFCacheInfo.export_as_tableN)r*r+r,r-r0r/rr>rr r2r.r)rJrcrsr#r#r#r$rXEs    drX) cache_dirrc Cs|dkr t}t|}|s8td|d|d|rPtd|dt}g}| D]P}|j dkrrqbz| t |Wqbt k r}z||W5d}~XYqbXqbtt|tdd|D|d S) afScan the entire HF cache-system and return a [`~HFCacheInfo`] structure. Use `scan_cache_dir` in order to programmatically scan your cache-system. The cache will be scanned repo by repo. If a repo is corrupted, a [`~CorruptedCacheException`] will be thrown internally but captured and returned in the [`~HFCacheInfo`] structure. Only valid repos get a proper report. ```py >>> from huggingface_hub import scan_cache_dir >>> hf_cache_info = scan_cache_dir() HFCacheInfo( size_on_disk=3398085269, repos=frozenset({ CachedRepoInfo( repo_id='t5-small', repo_type='model', repo_path=PosixPath(...), size_on_disk=970726914, nb_files=11, revisions=frozenset({ CachedRevisionInfo( commit_hash='d78aea13fa7ecd06c29e3e46195d6341255065d5', size_on_disk=970726339, snapshot_path=PosixPath(...), files=frozenset({ CachedFileInfo( file_name='config.json', size_on_disk=1197 file_path=PosixPath(...), blob_path=PosixPath(...), ), CachedFileInfo(...), ... }), ), CachedRevisionInfo(...), ... }), ), CachedRepoInfo(...), ... }), warnings=[ CorruptedCacheException("Snapshots dir doesn't exist in cached repo: ..."), CorruptedCacheException(...), ... ], ) ``` You can also print a detailed report directly from the `hf` command line using: ```text > hf cache scan REPO ID REPO TYPE SIZE ON DISK NB FILES REFS LOCAL PATH --------------------------- --------- ------------ -------- ------------------- ------------------------------------------------------------------------- glue dataset 116.3K 15 1.17.0, main, 2.4.0 /Users/lucain/.cache/huggingface/hub/datasets--glue google/fleurs dataset 64.9M 6 main, refs/pr/1 /Users/lucain/.cache/huggingface/hub/datasets--google--fleurs Jean-Baptiste/camembert-ner model 441.0M 7 main /Users/lucain/.cache/huggingface/hub/models--Jean-Baptiste--camembert-ner bert-base-cased model 1.9G 13 main /Users/lucain/.cache/huggingface/hub/models--bert-base-cased t5-base model 10.1K 3 main /Users/lucain/.cache/huggingface/hub/models--t5-base t5-small model 970.7M 11 refs/pr/1, main /Users/lucain/.cache/huggingface/hub/models--t5-small Done in 0.0s. Scanned 6 repo(s) for a total of 3.4G. Got 1 warning(s) while scanning. Use -vvv to print details. ``` Args: cache_dir (`str` or `Path`, `optional`): Cache directory to cache. Defaults to the default HF cache directory. > [!WARNING] > Raises: > > `CacheNotFound` > If the cache directory does not exist. > > [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) > If the cache directory is a file, instead of a directory. Returns: a [`~HFCacheInfo`] object. NzCache directory not found: zM. Please use `cache_dir` argument or set `HF_HUB_CACHE` environment variable.)rtz1Scan cache expects a directory but found a file: z.lockscss|] }|jVqdSrj)rrhr#r#r$ sz!scan_cache_dir..)rMrrY)rr expanduserresolveexistsr is_file ValueErrorr[iterdirnamer\_scan_cached_repor appendrXrasum)rtrMrYrAer#r#r$scan_cache_dir,s4S    r)rArc s&|std|d|jkr.td||jjddd\}}|dd}|dd}|d krxtd |d |d i|d }|d}|r|std|tt}|r.|rtd|| dD]P}|s|jt krqt | |}| }|} W5QRX|| |qt} |D]<} | jt krRq<| rjtd| t} | dD]|} | rqzt| }|std||kr||<| t| j| |j||j|jdqzt| dkrtfdd| D}n | j}| t| jt| t|| jttfddtdd| DD| |dqrrrz8Repo type must be `dataset`, `model` or `space`, found `z` (z).rNr7z,Snapshots dir doesn't exist in cached repo: z!Refs directory cannot be a file: z**/*z*Snapshots folder corrupted. Found a file: zBlob missing (broken symlink): )rrrrrrrc3s|]}|jjVqdSrj)rst_mtimerErbZ blob_statsr#r$rusz$_scan_cached_repo..c3s|]}|jVqdSrjst_size)rErrr#r$ruscss|] }|jVqdSrj)rrr#r#r$rus)r4r6r7rr5r8z-Reference(s) refer to missing commit hashes: z (css|] }|jVqdSrj)st_atimerEstatr#r#r$ruscss|] }|jVqdSrj)rrr#r#r$ruscss|] }|jVqdSrjrrr#r#r$ru s)r=r?rAr@rBrrCr8)!is_dirr r|splitreplacerxrr[ryglobFILES_TO_IGNOREr. relative_toopenreadr\r{rrwrrrrrr<maxr3rapoprdictvaluesr>)rAr@r?Zsnapshots_pathZ refs_pathZ refs_by_hashZref_pathref_namefr4Zcached_revisionsZ revision_path cached_filesrrZrevision_last_modifiedZrepo_last_accessedZrepo_last_modifiedZ repo_statsr#rr$r}s               r})numrcCsDt|}dD]*}t|dkr.|d|S|d}q |ddS)zkFormat size in bytes into a human-readable string. Taken from https://stackoverflow.com/a/1094933 )KMGTPEZg@@z3.1fz.1fY)r1abs)rZnum_funitr#r#r$r(&s   r())secondr<)minuterr)houri)dayiQ)weeki: r)monthi' )yeari3N)tsrcCsht|}|dkrdStD]*\}}}t||}|dk r||krqHq|d||dkr^dnddS) zFormat timestamp in seconds into a human-readable string, relative to now. Vaguely inspired by Django's `timesince` formatter. za few seconds agoN rsrz ago)time_TIMESINCE_CHUNKSround)rdeltalabeldividerZ max_valuevaluer#r#r$r ?s  r )rVrQrc Cstd|d|z"|r,t|n t|Wn^tk rhtjd|d|dddYn0t k rtjd|d|dddYnXd S) aETry to delete a local file or folder. If the path does not exists, error is logged as a warning and then ignored. Args: path (`Path`) Path to delete. Can be a file or a folder. path_type (`str`) What path are we deleting ? Only for logging purposes. Example: "snapshot". zDelete z: zCouldn't delete z: file not found ()T)exc_infoz: permission denied (N) rTrUryosr]shutilrmtreeFileNotFoundErrorr^PermissionError)rVrQr#r#r$rSNs   rS)N).r-rrr collectionsr dataclassesrpathlibrtypingrrrrr r r Zhuggingface_hub.errorsr r Zcommands._cli_utilsr constantsrrrZ get_loggerr*rTrIrrr3r>rJrXr.rr}r0r(rr1r rSr#r#r#r$s@   $    ?FOEgs