U ʹj"@sNddlZddlmZddlmZddlmZmZmZm Z m Z m Z ddl m Z mZe eZeGdddZeGdd d Ze e eeefe eeed d d ZGd ddeZGdddeZGdddeZeeeefe eeefdddZddZeeeeeeefdddZe eee eedddZdS)N) defaultdict) dataclass)AnyDictListOptionalTupleUnion)logging yaml_dumpc@s:eZdZUdZeed<eed<eed<eed<eed<dZeeed<dZ eeed <dZ eeed <dZ eeed <dZ ee eefed <dZeeed <dZeeed<dZee eefed<dZeeed<dZeeed<dZeeed<dZeeed<eedddZdedddZddddZdS) EvalResultu Flattened representation of individual evaluation results found in model-index of Model Cards. For more information on the model-index spec, see https://github.com/huggingface/hub-docs/blob/main/modelcard.md?plain=1. Args: task_type (`str`): The task identifier. Example: "image-classification". dataset_type (`str`): The dataset identifier. Example: "common_voice". Use dataset id from https://hf.co/datasets. dataset_name (`str`): A pretty name for the dataset. Example: "Common Voice (French)". metric_type (`str`): The metric identifier. Example: "wer". Use metric id from https://hf.co/metrics. metric_value (`Any`): The metric value. Example: 0.9 or "20.0 ± 1.2". task_name (`str`, *optional*): A pretty name for the task. Example: "Speech Recognition". dataset_config (`str`, *optional*): The name of the dataset configuration used in `load_dataset()`. Example: fr in `load_dataset("common_voice", "fr")`. See the `datasets` docs for more info: https://hf.co/docs/datasets/package_reference/loading_methods#datasets.load_dataset.name dataset_split (`str`, *optional*): The split used in `load_dataset()`. Example: "test". dataset_revision (`str`, *optional*): The revision (AKA Git Sha) of the dataset used in `load_dataset()`. Example: 5503434ddd753f426f4b38109466949a1217c2bb dataset_args (`Dict[str, Any]`, *optional*): The arguments passed during `Metric.compute()`. Example for `bleu`: `{"max_order": 4}` metric_name (`str`, *optional*): A pretty name for the metric. Example: "Test WER". metric_config (`str`, *optional*): The name of the metric configuration used in `load_metric()`. Example: bleurt-large-512 in `load_metric("bleurt", "bleurt-large-512")`. See the `datasets` docs for more info: https://huggingface.co/docs/datasets/v2.1.0/en/loading#load-configurations metric_args (`Dict[str, Any]`, *optional*): The arguments passed during `Metric.compute()`. Example for `bleu`: max_order: 4 verified (`bool`, *optional*): Indicates whether the metrics originate from Hugging Face's [evaluation service](https://huggingface.co/spaces/autoevaluate/model-evaluator) or not. Automatically computed by Hugging Face, do not set. verify_token (`str`, *optional*): A JSON Web Token that is used to verify whether the metrics originate from Hugging Face's [evaluation service](https://huggingface.co/spaces/autoevaluate/model-evaluator) or not. source_name (`str`, *optional*): The name of the source of the evaluation result. Example: "Open LLM Leaderboard". source_url (`str`, *optional*): The URL of the source of the evaluation result. Example: "https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard". task_type dataset_type dataset_name metric_type metric_valueN task_namedataset_config dataset_splitdataset_revision dataset_args metric_name metric_config metric_argsverified verify_token source_name source_urlreturncCs|j|j|j|j|jfS)z9Returns a tuple that uniquely identifies this evaluation.)r rrrrselfr"\/var/www/html/bwave/python/venv/lib/python3.8/site-packages/huggingface_hub/repocard_data.pyunique_identifiers zEvalResult.unique_identifier)otherrcCsD|jD]4\}}|dkrq |dkr t||t||kr dSq dS)zx Return True if `self` and `other` describe exactly the same metric but with a different value. rrFT)__dict__itemsgetattr)r!r%key_r"r"r#is_equal_except_values z EvalResult.is_equal_except_valuecCs |jdk r|jdkrtddS)NzAIf `source_name` is provided, `source_url` must also be provided.)rr ValueErrorr r"r"r# __post_init__szEvalResult.__post_init__)__name__ __module__ __qualname____doc__str__annotations__rrrrrrrrrrrrboolrrrpropertytupler$r+r-r"r"r"r#r s, 3 r c@seZdZdZd"edddZddZdd Zd#ee e e d d d Z ddZ ddZ d$e eedddZd%e eedddZe edddZe ed dddZe edddZedd d!Zd S)&CardDataaStructure containing metadata from a RepoCard. [`CardData`] is the parent class of [`ModelCardData`] and [`DatasetCardData`]. Metadata can be exported as a dictionary or YAML. Export can be customized to alter the representation of the data (example: flatten evaluation results). `CardData` behaves as a dictionary (can get, pop, set values) but do not inherit from `dict` to allow this export step. F)ignore_metadata_errorscKs|j|dSN)r&update)r!r8kwargsr"r"r#__init__szCardData.__init__cCs(t|j}||dd|DS)zConverts CardData to a dict. Returns: `dict`: CardData represented as a dictionary ready to be dumped to a YAML block for inclusion in a README.md file. cSsi|]\}}|dk r||qSr9r").0r)valuer"r"r# sz$CardData.to_dict..)copydeepcopyr&_to_dictr'r!Z data_dictr"r"r#to_dicts  zCardData.to_dictcCsdS)zUse this method in child classes to alter the dict representation of the data. Alter the dict in-place. Args: data_dict (`dict`): The raw dict representation of the card data. Nr"rCr"r"r#rBszCardData._to_dictN)original_orderrcsH|r2fdd|ttjt|D_td|dS)a Dumps CardData to a YAML block for inclusion in a README.md file. Args: line_break (str, *optional*): The line break to use when dumping to yaml. Returns: `str`: CardData represented as a YAML block. cs"i|]}|jkr|j|qSr"r&)r=kr r"r#r?s z$CardData.to_yaml..F) sort_keys line_break)listsetr&keysr rDstrip)r!rIrEr"r r#to_yamls  zCardData.to_yamlcCs t|jSr9)reprr&r r"r"r#__repr__szCardData.__repr__cCs|Sr9)rNr r"r"r#__str__szCardData.__str__)r)defaultrcCs|j|}|dkr|S|S)#Get value for a given metadata key.N)r&get)r!r)rRr>r"r"r#rTs z CardData.getcCs|j||S)z#Pop value for a given metadata key.)r&pop)r!r)rRr"r"r#rUsz CardData.pop)r)rcCs |j|S)rSrFr!r)r"r"r# __getitem__szCardData.__getitem__)r)r>rcCs||j|<dS)z#Set value for a given metadata key.NrF)r!r)r>r"r"r# __setitem__szCardData.__setitem__cCs ||jkS)z%Check if a given metadata key is set.rFrVr"r"r# __contains__szCardData.__contains__rcCs t|jS)z'Return the number of metadata keys set.)lenr&r r"r"r#__len__szCardData.__len__)F)NN)N)N)r.r/r0r1r4r<rDrBrrr2rNrPrQrrTrUrWrXrYintr[r"r"r"r#r7s  r7) eval_results model_namercCs`|dkr gSt|tr|g}t|tr8tdd|DsLtdt|d|dkr\td|S)Ncss|]}t|tVqdSr9) isinstancer )r=rr"r"r# sz)_validate_eval_results..zM`eval_results` should be of type `EvalResult` or a list of `EvalResult`, got .z7Passing `eval_results` requires `model_name` to be set.)r_r rJallr,type)r]r^r"r"r#_validate_eval_resultss recseZdZdZdddddddddddddd eeeeefeeeeefeeeeeeeefeeeeeeeeeeeeeeeeeee d fddZ ddZ Z S) ModelCardDataaQModel Card Metadata that is used by Hugging Face Hub when included at the top of your README.md Args: base_model (`str` or `List[str]`, *optional*): The identifier of the base model from which the model derives. This is applicable for example if your model is a fine-tune or adapter of an existing model. The value must be the ID of a model on the Hub (or a list of IDs if your model derives from multiple models). Defaults to None. datasets (`Union[str, List[str]]`, *optional*): Dataset or list of datasets that were used to train this model. Should be a dataset ID found on https://hf.co/datasets. Defaults to None. eval_results (`Union[List[EvalResult], EvalResult]`, *optional*): List of `huggingface_hub.EvalResult` that define evaluation results of the model. If provided, `model_name` is used to as a name on PapersWithCode's leaderboards. Defaults to `None`. language (`Union[str, List[str]]`, *optional*): Language of model's training data or metadata. It must be an ISO 639-1, 639-2 or 639-3 code (two/three letters), or a special value like "code", "multilingual". Defaults to `None`. library_name (`str`, *optional*): Name of library used by this model. Example: keras or any library from https://github.com/huggingface/huggingface.js/blob/main/packages/tasks/src/model-libraries.ts. Defaults to None. license (`str`, *optional*): License of this model. Example: apache-2.0 or any license from https://huggingface.co/docs/hub/repositories-licenses. Defaults to None. license_name (`str`, *optional*): Name of the license of this model. Defaults to None. To be used in conjunction with `license_link`. Common licenses (Apache-2.0, MIT, CC-BY-SA-4.0) do not need a name. In that case, use `license` instead. license_link (`str`, *optional*): Link to the license of this model. Defaults to None. To be used in conjunction with `license_name`. Common licenses (Apache-2.0, MIT, CC-BY-SA-4.0) do not need a link. In that case, use `license` instead. metrics (`List[str]`, *optional*): List of metrics used to evaluate this model. Should be a metric name that can be found at https://hf.co/metrics. Example: 'accuracy'. Defaults to None. model_name (`str`, *optional*): A name for this model. It is used along with `eval_results` to construct the `model-index` within the card's metadata. The name you supply here is what will be used on PapersWithCode's leaderboards. If None is provided then the repo name is used as a default. Defaults to None. pipeline_tag (`str`, *optional*): The pipeline tag associated with the model. Example: "text-classification". tags (`List[str]`, *optional*): List of tags to add to your model that can be used when filtering on the Hugging Face Hub. Defaults to None. ignore_metadata_errors (`str`): If True, errors while parsing the metadata section will be ignored. Some information might be lost during the process. Use it at your own risk. kwargs (`dict`, *optional*): Additional metadata that will be added to the model card. Defaults to None. Example: ```python >>> from huggingface_hub import ModelCardData >>> card_data = ModelCardData( ... language="en", ... license="mit", ... library_name="timm", ... tags=['image-classification', 'resnet'], ... ) >>> card_data.to_dict() {'language': 'en', 'license': 'mit', 'library_name': 'timm', 'tags': ['image-classification', 'resnet']} ``` NF) base_modeldatasetsr]language library_namelicense license_name license_linkmetricsr^ pipeline_tagtagsr8c  sF||_||_||_||_||_||_||_||_| |_| |_ | |_ t | |_ | dd}|rzt|\} }| |_ ||_WnNttfk r}z,| rtdntd|jd|dW5d}~XYnXtjf||jrBzt|j|j |_WnNtk r@}z.| r td|dntd||W5d}~XYnXdS)N model-indexz1T', and 'other'. source_datasets (`List[str]]`, *optional*): Indicates whether the dataset is an original dataset or extended from another existing dataset. Options are: 'original' and 'extended'. task_categories (`Union[str, List[str]]`, *optional*): What categories of task does the dataset support? task_ids (`Union[str, List[str]]`, *optional*): What specific tasks does the dataset support? paperswithcode_id (`str`, *optional*): ID of the dataset on PapersWithCode. pretty_name (`str`, *optional*): A more human-readable name for the dataset. (ex. "Cats vs. Dogs") train_eval_index (`Dict`, *optional*): A dictionary that describes the necessary spec for doing evaluation on the Hub. If not provided, it will be gathered from the 'train-eval-index' key of the kwargs. config_names (`Union[str, List[str]]`, *optional*): A list of the available dataset configs for the dataset. NF)rirkannotations_creatorslanguage_creatorsmultilingualitysize_categoriessource_datasetstask_categoriestask_idspaperswithcode_id pretty_nametrain_eval_index config_namesr8c sl||_||_||_||_||_||_||_||_| |_| |_ | |_ | |_ | pV| dd|_ tjf|dS)Ntrain-eval-index)rrrirkrrrrrrrrrUrrzr<)r!rirkrrrrrrrrrrrr8r;rr"r#r<szDatasetCardData.__init__cCs|d|d<dS)Nrr)rUrCr"r"r#rBszDatasetCardData._to_dict) r.r/r0r1rr r2rrr4r<rBrr"r"rr#rs@* $rcseZdZdZddddddddddddd eeeeeeeeeeeeeeeeeeeeeeeeeed fddZ Z S) SpaceCardDataa Space Card Metadata that is used by Hugging Face Hub when included at the top of your README.md To get an exhaustive reference of Spaces configuration, please visit https://huggingface.co/docs/hub/spaces-config-reference#spaces-configuration-reference. Args: title (`str`, *optional*) Title of the Space. sdk (`str`, *optional*) SDK of the Space (one of `gradio`, `streamlit`, `docker`, or `static`). sdk_version (`str`, *optional*) Version of the used SDK (if Gradio/Streamlit sdk). python_version (`str`, *optional*) Python version used in the Space (if Gradio/Streamlit sdk). app_file (`str`, *optional*) Path to your main application file (which contains either gradio or streamlit Python code, or static html code). Path is relative to the root of the repository. app_port (`str`, *optional*) Port on which your application is running. Used only if sdk is `docker`. license (`str`, *optional*) License of this model. Example: apache-2.0 or any license from https://huggingface.co/docs/hub/repositories-licenses. duplicated_from (`str`, *optional*) ID of the original Space if this is a duplicated Space. models (List[`str`], *optional*) List of models related to this Space. Should be a dataset ID found on https://hf.co/models. datasets (`List[str]`, *optional*) List of datasets related to this Space. Should be a dataset ID found on https://hf.co/datasets. tags (`List[str]`, *optional*) List of tags to add to your Space that can be used when filtering on the Hub. ignore_metadata_errors (`str`): If True, errors while parsing the metadata section will be ignored. Some information might be lost during the process. Use it at your own risk. kwargs (`dict`, *optional*): Additional metadata that will be added to the space card. Example: ```python >>> from huggingface_hub import SpaceCardData >>> card_data = SpaceCardData( ... title="Dreambooth Training", ... license="mit", ... sdk="gradio", ... duplicated_from="multimodalart/dreambooth-training" ... ) >>> card_data.to_dict() {'title': 'Dreambooth Training', 'sdk': 'gradio', 'license': 'mit', 'duplicated_from': 'multimodalart/dreambooth-training'} ``` NF) titlesdk sdk_versionpython_versionapp_fileapp_portrkduplicated_frommodelsrhrpr8c  sX||_||_||_||_||_||_||_||_| |_| |_ t | |_ t j f| dSr9)rrrrrrrkrrrhrsrprzr<)r!rrrrrrrkrrrhrpr8r;rr"r#r< s zSpaceCardData.__init__) r.r/r0r1rr2r\rr4r<rr"r"rr#rs64   r)r|rcCsFg}|D]2}|d}|d}|D]}|dd}|dd}|dd}|dd} |dd} |dd} |dd} |dd } |d id}|d id }|d D]z}|d}|d }|d}|d }|d}|d}|d}t||| |||| | | | |||||||d}||qq"q||fS)aTakes in a model index and returns the model name and a list of `huggingface_hub.EvalResult` objects. A detailed spec of the model index can be found here: https://github.com/huggingface/hub-docs/blob/main/modelcard.md?plain=1 Args: model_index (`List[Dict[str, Any]]`): A model index data structure, likely coming from a README.md file on the Hugging Face Hub. Returns: model_name (`str`): The name of the model as found in the model index. This is used as the identifier for the model on leaderboards like PapersWithCode. eval_results (`List[EvalResult]`): A list of `huggingface_hub.EvalResult` objects containing the metrics reported in the provided model_index. Example: ```python >>> from huggingface_hub.repocard_data import model_index_to_eval_results >>> # Define a minimal model index >>> model_index = [ ... { ... "name": "my-cool-model", ... "results": [ ... { ... "task": { ... "type": "image-classification" ... }, ... "dataset": { ... "type": "beans", ... "name": "Beans" ... }, ... "metrics": [ ... { ... "type": "accuracy", ... "value": 0.9 ... } ... ] ... } ... ] ... } ... ] >>> model_name, eval_results = model_index_to_eval_results(model_index) >>> model_name 'my-cool-model' >>> eval_results[0].task_type 'image-classification' >>> eval_results[0].metric_type 'accuracy' ``` nameresultstaskrddatasetconfigsplitrevisionargssourceurlrnr>r verifyToken)r rrrrrrrrrrrrrrrr)rTr append)r|r]elemrrresultr rrrrrrrrrZmetricrrrrrrr eval_resultr"r"r#rt+sX8           rtcCsRt|tttfr&t|dd|DSt|trJt|dd|DS|SdS)zk Recursively remove `None` values from a dict. Borrowed from: https://stackoverflow.com/a/20558778 css|]}|dk rt|VqdSr9 _remove_none)r=xr"r"r#rasz_remove_none..css2|]*\}}|dk r|dk rt|t|fVqdSr9r)r=rGvr"r"r#rasN)r_rJr6rKrddictr')objr"r"r#rs  r)r^r]rc Cstt}|D]}||j|q g}|D]}|d}|j|jd|j|j|j |j |j |j ddd|Dd}|j dk rd|j i}|jdk r|j|d <||d <||q.||d g} t| S) aTakes in given model name and list of `huggingface_hub.EvalResult` and returns a valid model-index that will be compatible with the format expected by the Hugging Face Hub. Args: model_name (`str`): Name of the model (ex. "my-cool-model"). This is used as the identifier for the model on leaderboards like PapersWithCode. eval_results (`List[EvalResult]`): List of `huggingface_hub.EvalResult` objects containing the metrics to be reported in the model-index. Returns: model_index (`List[Dict[str, Any]]`): The eval_results converted to a model-index. Example: ```python >>> from huggingface_hub.repocard_data import eval_results_to_model_index, EvalResult >>> # Define minimal eval_results >>> eval_results = [ ... EvalResult( ... task_type="image-classification", # Required ... dataset_type="beans", # Required ... dataset_name="Beans", # Required ... metric_type="accuracy", # Required ... metric_value=0.9, # Required ... ) ... ] >>> eval_results_to_model_index("my-cool-model", eval_results) [{'name': 'my-cool-model', 'results': [{'task': {'type': 'image-classification'}, 'dataset': {'name': 'Beans', 'type': 'beans'}, 'metrics': [{'type': 'accuracy', 'value': 0.9}]}]}] ``` r)rdr)rrdrrrrc Ss.g|]&}|j|j|j|j|j|j|jdqS))rdr>rrrrr)rrrrrrr)r=rr"r"r# s z/eval_results_to_model_index..)rrrnNrrr)rr)rrJr$rvaluesr rrrrrrrrrr) r^r]Ztask_and_ds_types_maprZmodel_index_datarZ sample_resultdatarr|r"r"r#rsB%      r)rprcCs0|dkr |Sg}|D]}||kr||q|Sr9)r)rpZ unique_tagstagr"r"r#rss rs)r@ collectionsr dataclassesrtypingrrrrrr Zhuggingface_hub.utilsr r Z get_loggerr.rwr r7r2rerfrrrtrrrsr"r"r"r#s,    T PQ(h "\