U ʹj@sxdZddlZddlZddlmZmZmZmZmZddl Z ddl m Z ddl m Z mZddlmZmZmZmZmZmZmZmZmZmZmZmZddlmZddlmZdd l m!Z!m"Z"dd l#m$Z$m%Z%m&Z&dd l'm(Z(d gZ)e*e+Z,ee-ee-e-ffZ.e j/ee-e-fe j/d ddZ0eee-e-fe-e-dddZ1e-e-dddZ2Gddde j3Z4Gddde j3Z5Gddde j3Z6Gddde j3Z7Gdd d e j3Z8dd Z9d\d"d#Z:d]d%d&Z;e$e;d'd(d)e;d'd*d)e;d'd+d,d-d.d/e;d'd0d)e;d'd1d,d-d.d/e;d'd2d)e;d'd3d)e;d'd4d)e;d'd5d,d-d.d/e;d'd6d)e;d'd7d8d9e;d'd:d8d9e;d'd;d8d9e;d'dd8d9e;d'd?d,d-d.d8d=e;d'd@d)e;d'dAd)e;d'dBd)dCZe%d`e8dDdIdJZ?e%dae8dDdKdLZ@e%dbe8dDdMdNZAe%dce8dDdOdPZBe%dde8dDdQdRZCe%dee8dDdSdTZDe%dfe8dDdUdVZEe&e+dWdXdYdZd[dS)ga Swin Transformer A PyTorch impl of : `Swin Transformer: Hierarchical Vision Transformer using Shifted Windows` - https://arxiv.org/pdf/2103.14030 Code/weights from https://github.com/microsoft/Swin-Transformer, original copyright/license info below S3 (AutoFormerV2, https://arxiv.org/abs/2111.14725) Swin weights from - https://github.com/microsoft/Cream/tree/main/AutoFormerV2 Modifications and additions for timm hacked together by / Copyright 2021, Ross Wightman N)CallableListOptionalTupleUnionIMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD) PatchEmbedMlpDropPathClassifierHead to_2tuple to_ntuple trunc_normal__assertuse_fused_attnresize_rel_pos_bias_tableresample_patch_embedndgrid)build_model_with_cfg)register_notrace_function)checkpoint_seq named_apply)generate_default_cfgsregister_modelregister_model_deprecations)get_init_weights_vitSwinTransformer)x window_sizereturncCsj|j\}}}}||||d|d||d|d|}|ddddddd|d|d|}|S)aU Partition into non-overlapping windows with padding if needed. Args: x (tensor): input tokens with [B, H, W, C]. window_size (int): window size. Returns: windows: windows after partition with [B * num_windows, window_size, window_size, C]. (Hp, Wp): padded height and width before partition rrshapeviewpermute contiguous)r r!BHWCwindowsr2[/var/www/html/bwave/python/venv/lib/python3.8/site-packages/timm/models/swin_transformer.pywindow_partition)s,,r4)r!r.r/cCs^|jd}|d||d||d|d|d|}|ddddddd|||}|S)z Args: windows: (num_windows*B, window_size, window_size, C) window_size (int): Window size H (int): Height of image W (int): Width of image Returns: x: (B, H, W, C) r'rrr#r$r%r&r()r1r!r.r/r0r r2r2r3window_reverse=s ,$r5)win_hwin_wcCsttt|t|}t|d}|dddddf|dddddf}|ddd}|dddddf|d7<|dddddf|d7<|dddddfd|d9<|dS)Nrr$rr')torchstackrZarangeflattenr+r,sum)r6r7ZcoordsZcoords_flattenZrelative_coordsr2r2r3get_relative_position_indexOs ,""&r<c speZdZUdZejjeed<de e e e e ee e dfdd Z ejd d d Zde ejd ddZZS)WindowAttentionz Window based multi-head self attention (W-MSA) module with relative position bias. It supports shifted and non-shifted windows. fused_attnNT)dim num_headshead_dimr!qkv_bias attn_drop proj_dropc st||_t||_|j\}} || |_||_|p>||}||} |d|_tdd|_ t t d|dd| d||_|jdt|| ddt j|| d |d |_t ||_t | ||_t ||_t|jd d t jd d|_dS)a Args: dim: Number of input channels. num_heads: Number of attention heads. head_dim: Number of channels per head (dim // num_heads if not set) window_size: The height and width of the window. qkv_bias: If True, add a learnable bias to query, key, value. attn_drop: Dropout ratio of attention weight. proj_drop: Dropout ratio of output. gT)Z experimentalr$rrelative_position_indexF persistentr#Zbiasg{Gz?)stdr')rAN)super__init__rArr! window_arearBscalerr>nn Parameterr8zerosrelative_position_bias_tableregister_bufferr<LinearqkvZDropoutrEprojrFrZSoftmaxsoftmax) selfrArBrCr!rDrErFr6r7Zattn_dim __class__r2r3rMas$       (  zWindowAttention.__init__r"cCs<|j|jd|j|jd}|ddd}|dS)Nr'r$rr)rSrGr*rNr+r, unsqueeze)rYZrelative_position_biasr2r2r3_get_rel_pos_biass z!WindowAttention._get_rel_pos_biasmaskc Cs|j\}}}||||d|jdddddd}|d\}}} |jr|} |dk r|jd} |d| d|| || d|jdd}| |d|j||} t j j j ||| | |jr|jjndd }n||j}||d d} | |} |dk r8|jd} | d| |j|||dd} | d|j||} || } || } | | }|dd||d}||}||}|S) z Args: x: input features with shape of (num_windows*B, N, C) mask: (0/-inf) mask with shape of (num_windows, Wh*Ww, Wh*Ww) or None r#r'r$rrr%Nr@) attn_maskZ dropout_p)r)rVreshaperBr+Zunbindr>r^r*expandr8rP functionalZscaled_dot_product_attentionZtrainingrEprOZ transposer]rXrWrF) rYr r`ZB_Nr0rVqkvraZnum_winattnr2r2r3forwards< ( &    $    zWindowAttention.forward)Nr?Tr@r@)N)__name__ __module__ __qualname____doc__r8jitFinalbool__annotations__intr_int_or_tuple_2_tfloatrMTensorr^rl __classcell__r2r2rZr3r=[s$ -r=cseZdZdZdddddddddejejf eeee eeee e e e e e e d fd d Z eeeefeeeffd d dZddZddZZS)SwinTransformerBlockz Swin Transformer Block. r%Nr?r@Tr@) rAinput_resolutionrBrCr! shift_size mlp_ratiorDrFrE drop_path act_layer norm_layerc sJt||_||_|||\}}||_||_|jd|jd|_||_| ||_ t |||t |j|| | d|_ | dkrt | nt|_| ||_t|t||| | d|_| dkrt | nt|_t|jr2|j\}}t||jd|jd}t||jd|jd}td||df}d}td|jd t|jd |jd t|jd dfD]j}td|jd t|jd |jd t|jd dfD]&}||dd||ddf<|d7}qqrt||j}|d|j}|d|d}||dkt d |dkt d}nd}|j!d |d d dS) a Args: dim: Number of input channels. input_resolution: Input resolution. window_size: Window size. num_heads: Number of attention heads. head_dim: Enforce the number of channels per head shift_size: Shift size for SW-MSA. mlp_ratio: Ratio of mlp hidden dim to embedding dim. qkv_bias: If True, add a learnable bias to query, key, value. proj_drop: Dropout rate. attn_drop: Attention dropout rate. drop_path: Stochastic depth rate. act_layer: Activation layer. norm_layer: Normalization layer. rr)rBrCr!rDrErFr@)Z in_featuresZhidden_featuresrZdropNr'r$gYraFrH)"rLrMrAr|_calc_window_shiftr!r}rNr~norm1r=rrkr rPIdentity drop_path1norm2r rumlp drop_path2anymathceilr8rRslicer4r*r]Z masked_fillrwrT)rYrAr|rBrCr!r}r~rDrFrErrrwsssr.r/Zimg_maskZcnthwZ mask_windowsrarZr2r3rMsb        &zSwinTransformerBlock.__init__r\cCsNt|}t|}ddt|j|D}ddt|j||D}t|t|fS)NcSs g|]\}}||kr|n|qSr2r2).0rrr2r2r3 sz;SwinTransformerBlock._calc_window_shift..cSs"g|]\}}}||krdn|qSrr2)rrrsr2r2r3rs)rzipr|tuple)rYZtarget_window_sizeZtarget_shift_sizer!r}r2r2r3rs z'SwinTransformerBlock._calc_window_shiftc CsV|j\}}}}t|j}|rBtj||jd |jd fdd}n|}|jd||jd|jd}|jd||jd|jd} tjj|ddd| d|f}|||| } } t ||j} | d|j |} |j | |j d} | d|jd|jd|} t| |j| | }|ddd|d|ddf}|rNtj||jdd}n|}|S)Nrr)rr$)ZshiftsZdimsr'r_)r)rr}r8Zrollr!rPrepadr4r*rNrkrar5r,)rYr r-r.r/r0Z has_shiftZ shifted_xZpad_hZpad_wZHpZWpZ x_windowsZ attn_windowsr2r2r3_attns& &"" $zSwinTransformerBlock._attncCsd|j\}}}}|||||}||d|}|||||}|||||}|S)Nr')r)rrrrcrrrrYr r-r.r/r0r2r2r3rlAs zSwinTransformerBlock.forward)rmrnrorprPZGELU LayerNormrurvrrwrsrrMrrrrlryr2r2rZr3rzs<X&#rzcs>eZdZdZdejfeeeedfdd Z ddZ Z S) PatchMergingz Patch Merging Layer. NrAout_dimrcsHt||_|pd||_|d||_tjd||jdd|_dS)z Args: dim: Number of input channels. out_dim: Number of output channels (or 2 * dim if None) norm_layer: Normalization layer. r$r%FrJN)rLrMrArnormrPrU reduction)rYrArrrZr2r3rMNs zPatchMerging.__init__cCs|j\}}}}t|ddkd|dt|ddkd|d|||dd|dd|dddddd d}||}||}|S) Nr$rz x height (z) is not even.z x width (rr#r%r&)r)rrcr+r:rrrr2r2r3rl`s2  zPatchMerging.forward) rmrnrorprPrrurrrMrlryr2r2rZr3rJsrcsxeZdZdZdddddddddejf eeeeefeeee ee e ee e e e e e fedfd d Zd d ZZS) SwinTransformerStagez3 A basic Swin Transformer layer for one stage. Tr%Nr?r{r@rArr|depth downsamplerBrCr!r~rDrFrErrc st| _| _|r,tdd|Dn| _| _d _t  tdd D |rnt |d _ n|kszt t _ t j f ddt|D _dS) a Args: dim: Number of input channels. out_dim: Number of output channels. input_resolution: Input resolution. depth: Number of blocks. downsample: Downsample layer at the end of the layer. num_heads: Number of attention heads. head_dim: Channels per head (dim // num_heads if not set) window_size: Local window size. mlp_ratio: Ratio of mlp hidden dim to embedding dim. qkv_bias: If True, add a learnable bias to query, key, value. proj_drop: Projection dropout rate. attn_drop: Attention dropout rate. drop_path: Stochastic depth rate. norm_layer: Normalization layer. css|]}|dVqdS)r$Nr2rir2r2r3 sz0SwinTransformerStage.__init__..FcSsg|] }|dqSr$r2)rrr2r2r3rsz1SwinTransformerStage.__init__..rcsPg|]H}t j |ddkr&dn ttrB|nd qS)r$r) rAr|rBrCr!r}r~rDrFrErr)rzoutput_resolution isinstancelistr rErrCr~rrBrrFrDrYr}r!r2r3rsN)rLrMrAr|rrrgrad_checkpointingrrrAssertionErrorrPr Sequentialrangeblocks)rYrArr|rrrBrCr!r~rDrFrErrrZrr3rMns&"    $zSwinTransformerStage.__init__cCs6||}|jr(tjs(t|j|}n ||}|SN)rrr8rqZ is_scriptingrrrYr r2r2r3rls   zSwinTransformerStage.forward)rmrnrorprPrrurrsrrvrwrrrrMrlryr2r2rZr3rjs8  HrcseZdZdZdddddddd d d d d ddddeejdfeeeee ee edfe edfe eee e e e e e eee efe dfdd Zejjd)ddZejjddZejjd*ddZejjd+ddZejjddZd,d d!Zd"d#Zd-e d$d%d&Zd'd(ZZS).rz Swin Transformer A PyTorch impl of : `Swin Transformer: Hierarchical Vision Transformer using Shifted Windows` - https://arxiv.org/pdf/2103.14030 r%r#avg`r$r$r$r#r Nr?r{Tr@g?.)img_size patch_sizein_chans num_classes global_pool embed_dimdepthsrBrCr!r~rD drop_rateproj_drop_rateattn_drop_ratedrop_path_rate embed_layerr weight_initc s0t|dkst||_||_d|_t||_|_t d|jd|_ g|_ t t tfs|fddt|jD||||d|dd|_|jj|_t|j| } t | tt fst|j| } nt| dkr| f|j} t| |jkstt|j| } d dtd|t||D}g}d}d}t|jD]}|}|t|||jd||jd|f|||dk||| || || || |||||d g7}|}|dkr|d9}|j t|d |d |d g7_ q@tj||_||j |_t|j ||| |jd|_|dkr,| |dS)a~ Args: img_size: Input image size. patch_size: Patch size. in_chans: Number of input image channels. num_classes: Number of classes for classification head. embed_dim: Patch embedding dimension. depths: Depth of each Swin Transformer layer. num_heads: Number of attention heads in different layers. head_dim: Dimension of self-attention heads. window_size: Window size. mlp_ratio: Ratio of mlp hidden dim to embedding dim. qkv_bias: If True, add a learnable bias to query, key, value. drop_rate: Dropout rate. attn_drop_rate (float): Attention dropout rate. drop_path_rate (float): Stochastic depth rate. embed_layer: Patch embedding layer. norm_layer (nn.Module): Normalization layer. )rrZNHWCr$rcsg|]}td|qSr)rurrr2r3rsz,SwinTransformer.__init__..r)rrrrr output_fmtcSsg|] }|qSr2)tolist)rr r2r2r3rsrr%layers.)Znum_chsrmodule) pool_typerZ input_fmtskipN)!rLrMrrrrlenZ num_layersrruZ num_featuresZ feature_inforrrr patch_embedZ grid_sizeZ patch_gridrr8Zlinspacer;splitrdictrPrlayersrr head init_weights)rYrrrrrrrrBrCr!r~rDrrrrrrrkwargsZdprrZin_dimrOrrrZrr3rMs*      "    (   zSwinTransformer.__init__cCs<|dks td|kr"t|j nd}tt||d|dS)N)ZjaxZjax_nlhbZmocorZnlhbr@) head_bias)rrlogrrr)rYmoderr2r2r3r<s zSwinTransformer.init_weightscCs.t}|D]\}}d|kr||q|S)NrS)setZnamed_parametersadd)rYZnwdn_r2r2r3no_weight_decayBs  zSwinTransformer.no_weight_decayFcCstd|r dndddgdS)Nz ^patch_embedz^layers\.(\d+))z^layers\.(\d+).downsampler)z^layers\.(\d+)\.\w+\.(\d+)N)z^norm)i)stemr)r)rYZcoarser2r2r3 group_matcherJszSwinTransformer.group_matchercCs|jD] }||_qdSr)rr)rYenablelr2r2r3set_grad_checkpointingUs z&SwinTransformer.set_grad_checkpointingcCs|jjSr)rfc)rYr2r2r3get_classifierZszSwinTransformer.get_classifiercCs||_|jj||ddS)N)r)rrreset)rYrrr2r2r3reset_classifier^sz SwinTransformer.reset_classifiercCs"||}||}||}|Sr)rrrrr2r2r3forward_featuresbs   z SwinTransformer.forward_features pre_logitscCs|r|j|ddS||S)NTr)r)rYr rr2r2r3 forward_headhszSwinTransformer.forward_headcCs||}||}|Sr)rrrr2r2r3rlks  zSwinTransformer.forward)r)F)T)N)F)rmrnrorpr rPrrvrustrrrrwrsrrrMr8rqignorerrrrrrrrrlryr2r2rZr3rsn   u      c s.d}d|krd}ddl}i}|d|}|d|}|D]\}tfdd d Dr\q.)rGrazpatch_embed.proj.weightrbr'bicubic) interpolationZ antialiasverboserSir)Znew_window_sizeZnew_bias_shapezlayers.(\d+).downsamplecSsdt|dddS)Nrrz .downsample)rugroup)r r2r2r3z&checkpoint_filter_fn..zhead.zhead.fc.)regetitemsrrrWweightr)rendswithZ get_submodulerSr!rsubreplace) rrZ old_weightsrZout_dictrjrr.r/mr2rr3checkpoint_filter_fnqsB   "  rFcKsPtddt|ddD}|d|}tt||fttd|dd|}|S) Ncss|]\}}|VqdSrr2)rrrr2r2r3rsz+_create_swin_transformer..r)rrr#r out_indicesT)Zflatten_sequentialr)Zpretrained_filter_fnZ feature_cfg)r enumeraterpoprrrr)variant pretrainedrZdefault_out_indicesrrr2r2r3_create_swin_transformers  rrc Ks"|ddddddttddd d |S) Nr)r#rr)r?r?g?rTzpatch_embed.projzhead.fcZmit) urlr input_size pool_sizecrop_pctrZfixed_input_sizeZmeanrKZ first_conv classifierlicenser)r rr2r2r3_cfgsrztimm/zvhttps://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_small_patch4_window7_224_22kto1k_finetune.pth) hf_hub_idr zlhttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224_22kto1k.pthzmhttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window12_384_22kto1k.pth)r#r)rrg?)rr r r r zmhttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window7_224_22kto1k.pthznhttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window12_384_22kto1k.pthzdhttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_tiny_patch4_window7_224.pthzehttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_small_patch4_window7_224.pthzdhttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224.pthzehttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window12_384.pthzuhttps://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_tiny_patch4_window7_224_22kto1k_finetune.pthzhhttps://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_tiny_patch4_window7_224_22k.pthiQU)rr rzihttps://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_small_patch4_window7_224_22k.pthzhhttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224_22k.pthzihttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window12_384_22k.pth)rr r r r rzihttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window7_224_22k.pthzjhttps://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window12_384_22k.pthzbhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/s3_t-1d53f6a8.pthzbhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/s3_s-3bb4c69d.pthzbhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/s3_b-a1e95db4.pth)z.swin_small_patch4_window7_224.ms_in22k_ft_in1kz-swin_base_patch4_window7_224.ms_in22k_ft_in1kz.swin_base_patch4_window12_384.ms_in22k_ft_in1kz.swin_large_patch4_window7_224.ms_in22k_ft_in1kz/swin_large_patch4_window12_384.ms_in22k_ft_in1kz$swin_tiny_patch4_window7_224.ms_in1kz%swin_small_patch4_window7_224.ms_in1kz$swin_base_patch4_window7_224.ms_in1kz%swin_base_patch4_window12_384.ms_in1kz-swin_tiny_patch4_window7_224.ms_in22k_ft_in1kz%swin_tiny_patch4_window7_224.ms_in22kz&swin_small_patch4_window7_224.ms_in22k%swin_base_patch4_window7_224.ms_in22k&swin_base_patch4_window12_384.ms_in22k&swin_large_patch4_window7_224.ms_in22k'swin_large_patch4_window12_384.ms_in22kzswin_s3_tiny_224.ms_in1kzswin_s3_small_224.ms_in1kzswin_s3_base_224.ms_in1kr\cKs,tdddddd}td d|it|f|S) z+ Swin-T @ 224x224, trained ImageNet-1k r%r?rrrrr!rrrBswin_tiny_patch4_window7_224r)rrrrrZ model_argsr2r2r3rs rcKs,tdddddd}td d|it|f|S) z Swin-S @ 224x224 r%r?rr$r$r$rrswin_small_patch4_window7_224r)rrrr2r2r3rs rcKs,tdddddd}td d|it|f|S) z Swin-B @ 224x224 r%r?rr% rswin_base_patch4_window7_224r)r"rrr2r2r3r"s r"cKs,tdddddd}td d|it|f|S) z Swin-B @ 384x384 r%rrrrrswin_base_patch4_window12_384r)r#rrr2r2r3r#s r#cKs,tdddddd}td d|it|f|S) z Swin-L @ 224x224 r%r?rrrr0rswin_large_patch4_window7_224r)r'rrr2r2r3r'!s r'cKs,tdddddd}td d|it|f|S) z Swin-L @ 384x384 r%rr$rr%rswin_large_patch4_window12_384r)r(rrr2r2r3r(*s r(cKs,tdddddd}td d|it|f|S) z; Swin-S3-T @ 224x224, https://arxiv.org/abs/2111.14725 r%r?r?r?rrrrswin_s3_tiny_224r)r+rrr2r2r3r+3sr+cKs,tdddddd}td d|it|f|S) z; Swin-S3-S @ 224x224, https://arxiv.org/abs/2111.14725 r%)r*r*r*r?rrrrswin_s3_small_224r)r,rrr2r2r3r,<sr,cKs,tdddddd}td d|it|f|S) z; Swin-S3-B @ 224x224, https://arxiv.org/abs/2111.14725 r%r)r)r$r$r$rrswin_s3_base_224r)r.rrr2r2r3r.Esr.rrrr)Z"swin_base_patch4_window7_224_in22kZ#swin_base_patch4_window12_384_in22kZ#swin_large_patch4_window7_224_in22kZ$swin_large_patch4_window12_384_in22k)F)r)F)F)F)F)F)F)F)F)F)Frploggingrtypingrrrrrr8Ztorch.nnrPZ timm.datarr Z timm.layersr r r r rrrrrrrrZ_builderrZ _features_fxrZ _manipulaterr _registryrrrZvision_transformerr__all__ getLoggerrmZ_loggerrurvrxr4r5r<Moduler=rzrrrrrrZ default_cfgsrrr"r#r'r(r+r,r.r2r2r2r3s6 8       ` V2) K