U ʹjvH@sFdZddlmZddlmZmZddlZddlmZddlm Z m Z ddl m Z ddl mZmZmZd d lmZd d lmZd d lmZmZmZd gZGdd d eZdaddZdbddZeedddedddedddedddddedddd edd!dd edd"dd edd#dddd$edd%dedd&dddedd'dedd(dedd)dddedd*dedd+dddedd,dedd-dd.edd/dddedd0dd.edd1dd.edd2dddedd3dd.edd4dddedd5dd.d6Zedced7d8d9Zedded7d:d;Zedeed7dd?Z!edged7d@dAZ"edhed7dBdCZ#edied7dDdEZ$edjed7dFdGZ%edked7dHdIZ&edled7dJdKZ'edmed7dLdMZ(edned7dNdOZ)edoed7dPdQZ*edped7dRdSZ+edqed7dTdUZ,edred7dVdWZ-ee.dXdYdZd[d\d]d^d_d`dS)sa[ DeiT - Data-efficient Image Transformers DeiT model defs and weights from https://github.com/facebookresearch/deit, original copyright below paper: `DeiT: Data-efficient Image Transformers` - https://arxiv.org/abs/2012.12877 paper: `DeiT III: Revenge of the ViT` - https://arxiv.org/abs/2204.07118 Modifications copyright 2021, Ross Wightman )partial)SequenceUnionN)nnIMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD)resample_abs_pos_embed)VisionTransformer trunc_normal_checkpoint_filter_fn)build_model_with_cfg)checkpoint_seq)generate_default_cfgsregister_modelregister_model_deprecationsVisionTransformerDistilledcseZdZdZfddZdfdd Zejjddd Z ejjd d Z dd dZ ejjdddZ ddZ deejdddZZS)rz Vision Transformer w/ Distillation Token and Head Distillation token & head support for `DeiT: Data-efficient Image Transformers` - https://arxiv.org/abs/2012.12877 cs|dd}tj||ddi|jdks0td|_tt dd|j |_ tt d|j j |j|j |_|jdkrt|j |jnt|_d|_||dS) N weight_initskip)tokenr rF)popsuper__init__ global_poolAssertionErrornum_prefix_tokensr ParametertorchZzeros embed_dim dist_tokenZ patch_embedZ num_patches pos_embed num_classesLinearIdentity head_distdistilled_training init_weights)selfargskwargsr __class__O/var/www/html/bwave/python/venv/lib/python3.8/site-packages/timm/models/deit.pyr$s $z#VisionTransformerDistilled.__init__rcs t|jddtj|ddS)Ng{Gz?)std)mode)r r"rr))r*r2r-r/r0r)2sz'VisionTransformerDistilled.init_weightsFcCstdddgdS)Nz+^cls_token|pos_embed|patch_embed|dist_token)z^blocks\.(\d+)N)z^norm)i)stemblocks)dict)r*Zcoarser/r/r0 group_matcher6s z(VisionTransformerDistilled.group_matchercCs |j|jfSNheadr')r*r/r/r0get_classifier?sz)VisionTransformerDistilled.get_classifierNcCsL||_|dkrt|j|nt|_|dkr>t|j|jnt|_dS)Nr)r$rr%r!r&r9r')r*r$rr/r/r0reset_classifierCs z+VisionTransformerDistilled.reset_classifierTcCs ||_dSr7)r()r*enabler/r/r0set_distilled_trainingHsz1VisionTransformerDistilled.set_distilled_trainingcCs|jrD|j\}}}}t|j||f|jr*dn|jd}||d|}n|j}|jr||}tj|j |jddd|j |jddd|fdd}n@tj|j |jddd|j |jddd|fdd}||}| |S)Nr)rr )dim) Zdynamic_img_sizeshaper r#no_embed_classrviewr catZ cls_tokenexpandr"Zpos_drop)r*xBHWCr#r/r/r0 _pos_embedLs6z%VisionTransformerDistilled._pos_embed) pre_logitsreturncCst|dddf|dddf}}|r2||dS||}||}|jrd|jrdtjsd||fS||dSdS)Nrr r)r9r'r(Ztrainingr jitZ is_scripting)r*rErKZx_distr/r/r0 forward_headks"   z'VisionTransformerDistilled.forward_head)r)F)N)T)F)__name__ __module__ __qualname____doc__rr)r rMignorer6r:r;r=rJboolZTensorrN __classcell__r/r/r-r0rs     FcKsD|ddrtd|rtnt}t|||fdttddi|}|S)NZ features_onlyzhttps://dl.fbaipublicfiles.com/deit/deit_3_huge_224_21k_v1.pth)zdeit_tiny_patch16_224.fb_in1kzdeit_small_patch16_224.fb_in1kzdeit_base_patch16_224.fb_in1kzdeit_base_patch16_384.fb_in1kz'deit_tiny_distilled_patch16_224.fb_in1kz(deit_small_distilled_patch16_224.fb_in1kz'deit_base_distilled_patch16_224.fb_in1kz'deit_base_distilled_patch16_384.fb_in1kzdeit3_small_patch16_224.fb_in1kzdeit3_small_patch16_384.fb_in1kz deit3_medium_patch16_224.fb_in1kzdeit3_base_patch16_224.fb_in1kzdeit3_base_patch16_384.fb_in1kzdeit3_large_patch16_224.fb_in1kzdeit3_large_patch16_384.fb_in1kzdeit3_huge_patch14_224.fb_in1k(deit3_small_patch16_224.fb_in22k_ft_in1k(deit3_small_patch16_384.fb_in22k_ft_in1k)deit3_medium_patch16_224.fb_in22k_ft_in1k'deit3_base_patch16_224.fb_in22k_ft_in1k'deit3_base_patch16_384.fb_in22k_ft_in1k(deit3_large_patch16_224.fb_in22k_ft_in1k(deit3_large_patch16_384.fb_in22k_ft_in1k'deit3_huge_patch14_224.fb_in22k_ft_in1k)rLcKs.tddddd}tdd|it|f|}|S) z DeiT-tiny model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit.  r] patch_sizer!depth num_headsdeit_tiny_patch16_224rY)rvr5r\rYr,Z model_argsr[r/r/r0rvsrvcKs.tddddd}tdd|it|f|}|S) z DeiT-small model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. rorfrqrrdeit_small_patch16_224rY)rzrwrxr/r/r0rzsrzcKs.tddddd}tdd|it|f|}|S)z DeiT base model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. rorqrrdeit_base_patch16_224rY)r|rwrxr/r/r0r|sr|cKs.tddddd}tdd|it|f|}|S)z DeiT base model @ 384x384 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. ror{rqrrdeit_base_patch16_384rY)r}rwrxr/r/r0r}sr}cKs0tddddd}td |ddt|f|}|S) z DeiT-tiny distilled model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. rorprqr]rrdeit_tiny_distilled_patch16_224TrYrZ)r~rwrxr/r/r0r~s r~cKs0tddddd}td |ddt|f|}|S) z DeiT-small distilled model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. rorfrqryrr deit_small_distilled_patch16_224Tr)rrwrxr/r/r0r&s rcKs0tddddd}td|ddt|f|}|S) z DeiT-base distilled model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. ror{rqrrdeit_base_distilled_patch16_224Tr)rrwrxr/r/r0r1s rcKs0tddddd}td|ddt|f|}|S) z DeiT-base distilled model @ 384x384 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. ror{rqrrdeit_base_distilled_patch16_384Tr)rrwrxr/r/r0r<s rcKs2tddddddd}td d |it|f|}|S) z DeiT-3 small model @ 224x224 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. rorfrqryTư>rsr!rtrurAZ init_valuesdeit3_small_patch16_224rY)rrwrxr/r/r0rGsrcKs2tddddddd}td d |it|f|}|S) z DeiT-3 small model @ 384x384 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. rorfrqryTrrdeit3_small_patch16_384rY)rrwrxr/r/r0rQsrcKs2tddddddd}td d |it|f|}|S) z DeiT-3 medium model @ 224x224 (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. roirqTrrdeit3_medium_patch16_224rY)rrwrxr/r/r0r[srcKs2tddddddd}td d|it|f|}|S) z DeiT-3 base model @ 224x224 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. ror{rqTrrdeit3_base_patch16_224rY)rrwrxr/r/r0resrcKs2tddddddd}td d|it|f|}|S) DeiT-3 base model @ 384x384 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. ror{rqTrrdeit3_base_patch16_384rY)rrwrxr/r/r0rosrcKs2tddddddd}td d|it|f|}|S) z DeiT-3 large model @ 224x224 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. roTrrdeit3_large_patch16_224rY)rrwrxr/r/r0rysrcKs2tddddddd}td d|it|f|}|S) z DeiT-3 large model @ 384x384 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. rorrTrrdeit3_large_patch16_384rY)rrwrxr/r/r0rsrcKs2tddddddd}td d |it|f|}|S) ri roTrrdeit3_huge_patch14_224rY)rrwrxr/r/r0rsrrgrhrirjrkrlrmrn)Z deit3_small_patch16_224_in21ft1kZ deit3_small_patch16_384_in21ft1kZ!deit3_medium_patch16_224_in21ft1kZdeit3_base_patch16_224_in21ft1kZdeit3_base_patch16_384_in21ft1kZ deit3_large_patch16_224_in21ft1kZ deit3_large_patch16_384_in21ft1kZdeit3_huge_patch14_224_in21ft1k)FF)r)F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)F)/rR functoolsrtypingrrr rZ timm.datarrZ timm.layersr Ztimm.models.vision_transformerr r r Z_builderrZ _manipulater _registryrrr__all__rr\rdZ default_cfgsrvrzr|r}r~rrrrrrrrrrrrOr/r/r/r0sf     \  a