U ʹjĆ@sLdZddlZddlmZmZmZmZmZmZddl Z ddl m Z ddl m m ZddlmmZddlmZmZddlmZmZmZmZmZmZmZmZmZddlm Z ddl!m"Z"dd l#m$Z$m%Z%m&Z&d gZ'ee(ee(e(ffZ)e j*ee(e(fe j*d d d Z+e"e j*ee(e(fee(e(fe j*dddZ,Gddde j-Z.Gddde j-Z/Gddde j-Z0Gddde j-Z1Gdd d e j-Z2ddZ3dYddZ4dZd d!Z5e$e5d"d#d$e5d"d%d&d'd(d)e5d"d*d$e5d"d+d&d'd(d)e5d"d,d$e5d"d-d$e5d"d.d$e5d"d/d$e5d"d0d$e5d"d1d$e5d"d2d3d4d5d6e5d"d7d3d4d5d6d8 Z6e%d[e2d9d:d;Z7e%d\e2d9dd?Z9e%d^e2d9d@dAZ:e%d_e2d9dBdCZ;e%d`e2d9dDdEZe%dce2d9dJdKZ?e%dde2d9dLdMZ@e%dee2d9dNdOZAe%dfe2d9dPdQZBe&eCdRdSdTdUdVdWdXdS)gaK Swin Transformer V2 A PyTorch impl of : `Swin Transformer V2: Scaling Up Capacity and Resolution` - https://arxiv.org/abs/2111.09883 Code/weights from https://github.com/microsoft/Swin-Transformer, original copyright/license info below Modifications and additions for timm hacked together by / Copyright 2022, Ross Wightman N)CallableOptionalTupleUnionSetDictIMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD) PatchEmbedMlpDropPath to_2tuple trunc_normal__assertClassifierHeadresample_patch_embedndgrid)build_model_with_cfg)register_notrace_function)generate_default_cfgsregister_modelregister_model_deprecationsSwinTransformerV2)x window_sizereturncCsj|j\}}}}||||d|d||d|d|}|ddddddd|d|d|}|S)z Args: x: (B, H, W, C) window_size (int): window size Returns: windows: (num_windows*B, window_size, window_size, C) rrshapeviewpermute contiguous)rrBHWCwindowsr-^/var/www/html/bwave/python/venv/lib/python3.8/site-packages/timm/models/swin_transformer_v2.pywindow_partition#s ,,r/)r,rimg_sizercCsf|\}}|jd}|d||d||d|d|d|}|ddddddd|||}|S)z Args: windows: (num_windows * B, window_size[0], window_size[1], C) window_size (Tuple[int, int]): Window size img_size (Tuple[int, int]): Image size Returns: x: (B, H, W, C) r"rrrrr r!r#)r,rr0r)r*r+rr-r-r.window_reverse2s  ,$r1c sdeZdZdZd eeeefeeeeeeefddfdd Zd e j e e j e j d d d Z Z S)WindowAttentiona Window based multi-head self attention (W-MSA) module with relative position bias. It supports both of shifted and non-shifted window. Args: dim (int): Number of input channels. window_size (tuple[int]): The height and width of the window. num_heads (int): Number of attention heads. qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True attn_drop (float, optional): Dropout ratio of attention weight. Default: 0.0 proj_drop (float, optional): Dropout ratio of output. Default: 0.0 pretrained_window_size (tuple[int]): The height and width of the window in pre-training. TrrN)dimr num_headsqkv_bias attn_drop proj_droppretrained_window_sizerc st||_||_||_||_tt dt |ddf|_ t tj ddddtjddtj d|dd|_t|jd d |jd tj}t|jdd |jdtj} tt|| } | ddd d } |d d krP| ddddddd f|d d<| dddddddf|dd<n\| ddddddd f|jd d<| dddddddf|jdd<| d 9} t| tt| d td } |jd | dd t|jd } t|jd} tt| | } t| d}|dddddf|dddddf}|ddd }|ddddd f|jd d7<|dddddf|jdd7<|ddddd fd|jdd9<|d}|jd|dd tj ||ddd|_|rTtt ||_!|jdt |dd tt ||_"nd|_!d|_#d|_"t$||_%t |||_&t$||_'tj(dd|_)dS)N rriTbias)ZinplaceFr?relative_coords_table persistentr"relative_position_indexrk_biasr5)*super__init__r5rr:r6nn ParametertorchlogZones logit_scale SequentialLinearZReLUcpb_mlpZarangetoZfloat32stackrr&r' unsqueezesignlog2absmathregister_bufferflattensumqkvzerosq_biasv_biasrDZDropoutr8projr9ZSoftmaxsoftmax)selfr5rr6r7r8r9r:Zrelative_coords_hZrelative_coords_wr@Zcoords_hZcoords_wZcoordsZcoords_flattenZrelative_coordsrC __class__r-r.rGRsd " &&,...   ,((,   zWindowAttention.__init__)rmaskrcCs|j\}}}d}|jdk r0t|j|j|jf}tj||jj |d}| ||d|j d ddddd}| d\}} } tj|dd tj| dd d d} tj|jtd d } | | } ||jd|j } | |jd|jd|jd|jd|jdd}| ddd}d t|}| |d} |dk r|jd}| d||j |||dd} | d|j ||} || } n || } || } | | dd |||}||}| |}|S)z Args: x: input features with shape of (num_windows*B, N, C) mask: (0/-inf) mask with shape of (num_windows, Wh*Ww, Wh*Ww) or None N)inputweightr=rr"rrrr rEgY@)max)!r$r\rJcatrDr]FZlinearrZrereshaper6r&Zunbind normalizeZ transposeclamprLrVrKexprOr@r%rCrr'ZsigmoidrRr_r8r^r9)r`rrcZB_Nr+r7rZqkvattnrLZrelative_position_bias_tableZrelative_position_biasZnum_winr-r-r.forwards<  "$  $     zWindowAttention.forward)Tr3r3r4)N)__name__ __module__ __qualname____doc__intrboolfloatrGrJTensorrrt __classcell__r-r-rar.r2Ds   Er2cseZdZdZdddddddejejdf eeeeee e e e e ej ej eddfd d Z eee e eefe eeffd d d ZejejdddZejejdddZZS)SwinTransformerV2Blockz Swin Transformer Block. r@Tr3N)r5input_resolutionr6r shift_size mlp_ratior7r9r8 drop_path act_layer norm_layerr:rc st||_t||_||_|||\}}||_||_|jd|jd|_ ||_ t |t|j||| |t| d|_ | ||_ | dkrt| nt|_t|t||| |d|_| ||_| dkrt| nt|_t|jr|j\}}td||df}d}td|jd t|jd |jd t|jd dfD]j}td|jd t|jd |jd t|jd dfD]&}||dd||ddf<|d7}qqDt||j}|d|j }|d|d}||dktd |dktd}nd}|j d |d d dS) a Args: dim: Number of input channels. input_resolution: Input resolution. num_heads: Number of attention heads. window_size: Window size. shift_size: Shift size for SW-MSA. mlp_ratio: Ratio of mlp hidden dim to embedding dim. qkv_bias: If True, add a learnable bias to query, key, value. proj_drop: Dropout rate. attn_drop: Attention dropout rate. drop_path: Stochastic depth rate. act_layer: Activation layer. norm_layer: Normalization layer. pretrained_window_size: Window size in pretraining. rr)rr6r7r8r9r:r3)Z in_featuresZhidden_featuresrZdropNr"rgY attn_maskFrA)!rFrGr5rrr6_calc_window_shiftrr window_arearr2rsnorm1r rHIdentity drop_path1r rymlpnorm2 drop_path2anyrJr[slicer/r%rRZ masked_fillr{rW)r`r5rr6rrrr7r9r8rrrr:wsssr)r*Zimg_maskZcnthwZ mask_windowsrrar-r.rGs`        &zSwinTransformerV2Block.__init__)target_window_sizetarget_shift_sizercCsNt|}t|}ddt|j|D}ddt|j||D}t|t|fS)NcSs g|]\}}||kr|n|qSr-r-).0rrr-r-r. !sz=SwinTransformerV2Block._calc_window_shift..cSs"g|]\}}}||krdn|qSrr-)rrrsr-r-r.r"s)rziprtuple)r`rrrrr-r-r.rs z)SwinTransformerV2Block._calc_window_shiftrrc Cs|j\}}}}t|j}|rBtj||jd |jd fdd}n|}t||j}|d|j|}|j ||j d} | d|jd|jd|} t | |j|j }|rtj||jdd}n|}|S)Nrr)rr)ZshiftsZdimsr")rc) r$rrrJZrollr/rr%rrsrr1r) r`rr(r)r*r+Z has_shiftZ shifted_xZ x_windowsZ attn_windowsr-r-r._attn%s & zSwinTransformerV2Block._attncCsd|j\}}}}|||||}||d|}|||||}|||||}|S)Nr")r$rrrrkrrrr`rr(r)r*r+r-r-r.rtAs zSwinTransformerV2Block.forward)rurvrwrxrHZGELU LayerNormry_int_or_tuple_2_tr{rzModulerGrrrJr|rrtr}r-r-rar.r~sBX r~csNeZdZdZdejfeeeejddfdd Z e j e j dddZ Z S) PatchMergingz Patch Merging Layer. N)r5out_dimrrcsFt||_|pd||_tjd||jdd|_||j|_dS)z Args: dim (int): Number of input channels. out_dim (int): Number of output channels (or 2 * dim if None) norm_layer (nn.Module, optional): Normalization layer. Default: nn.LayerNorm rr Fr<N)rFrGr5rrHrN reductionnorm)r`r5rrrar-r.rGNs  zPatchMerging.__init__rcCs|j\}}}}t|ddkd|dt|ddkd|d|||dd|dd|dddddd d}||}||}|S) Nrrz x height (z) is not even.z x width (rrr r!)r$rrkr&rXrrrr-r-r.rt[s2  zPatchMerging.forward)rurvrwrxrHrryrrrGrJr|rtr}r-r-rar.rJs& rcs~eZdZdZddddddejddf eeeeeeee ee e e ej eeddfd d Z e j e j d d d ZddddZZS)SwinTransformerV2Stagez" A Swin Transformer V2 Stage. FrTr3rN)r5rrdepthr6r downsamplerr7r9r8rrr: output_nchwrcst| _| _|r,tdd|Dn| _| _| _d _t  tdd D |rtt |d _ n|kst t  _ t  f ddt|D _dS) a; Args: dim: Number of input channels. out_dim: Number of output channels. input_resolution: Input resolution. depth: Number of blocks. num_heads: Number of attention heads. window_size: Local window size. downsample: Use downsample layer at start of the block. mlp_ratio: Ratio of mlp hidden dim to embedding dim. qkv_bias: If True, add a learnable bias to query, key, value. proj_drop: Projection dropout rate attn_drop: Attention dropout rate. drop_path: Stochastic depth rate. norm_layer: Normalization layer. pretrained_window_size: Local window size in pretraining. output_nchw: Output tensors on NCHW format instead of NHWC. css|]}|dVqdS)rNr-rir-r-r. sz2SwinTransformerV2Stage.__init__..FcSsg|] }|dqSrr-)rrr-r-r.rsz3SwinTransformerV2Stage.__init__..)r5rrcsPg|]H}t j |ddkr$dn ttr@|nd qS)rr) r5rr6rrrr7r9r8rrr:)r~output_resolution isinstancelistr r8rrrr6rr:r9r7r`rrr-r.rsN)rFrGr5rrrrrgrad_checkpointingrrrAssertionErrorrHrZ ModuleListrangeblocks)r`r5rrrr6rrrr7r9r8rrr:rrarr.rGis $   $zSwinTransformerV2Stage.__init__rcCs@||}|jD]*}|jr2tjs2t||}q||}q|SN)rrrrJjitZ is_scripting checkpoint)r`rblkr-r-r.rts    zSwinTransformerV2Stage.forwardrcCsX|jD]L}tj|jjdtj|jjdtj|jjdtj|jjdqdS)Nr)rrHinit constant_rr=rer)r`rr-r-r._init_respostnorms  z(SwinTransformerV2Stage._init_respostnorm)rurvrwrxrHrryrrzr{rrGrJr|rtrr}r-r-rar.res< G rcseZdZdZdddddddd d d d d d d dejdfeeeeeee edfe edfee e e e e e e e edfdfdd Z ddZejjddZejjd)ddZejjd*ddZejjddZd+d d!Zd"d#Zd,e d$d%d&Zd'd(ZZS)-rz Swin Transformer V2 A PyTorch impl of : `Swin Transformer V2: Scaling Up Capacity and Resolution` - https://arxiv.org/abs/2111.09883 r ravg`rrrrr rrTr3g?)rrrr.)r0 patch_sizein_chans num_classes global_pool embed_dimdepthsr6rrr7 drop_rateproj_drop_rateattn_drop_ratedrop_path_raterpretrained_window_sizesc st||_|dkst||_d|_t||_|_t d|jd|_ g|_ t t tfs|fddt|jDt|||d|dd|_d dtd|t||D}g}d}d}t|jD]}|}|t|||jjd||jjd|f|||dk||| | | | ||||||d g7}|}|dkrN|d9}|j t|d |d |d g7_ qtj||_||j |_t|j ||| |jd|_||j |jD]}|!qdS)a/ Args: img_size: Input image size. patch_size: Patch size. in_chans: Number of input image channels. num_classes: Number of classes for classification head. embed_dim: Patch embedding dimension. depths: Depth of each Swin Transformer stage (layer). num_heads: Number of attention heads in different layers. window_size: Window size. mlp_ratio: Ratio of mlp hidden dim to embedding dim. qkv_bias: If True, add a learnable bias to query, key, value. drop_rate: Head dropout rate. proj_drop_rate: Projection dropout rate. attn_drop_rate: Attention dropout rate. drop_path_rate: Stochastic depth rate. norm_layer: Normalization layer. patch_norm: If True, add normalization after patch embedding. pretrained_window_sizes: Pretrained window sizes of each layer. output_fmt: Output tensor format if not None, otherwise output 'NHWC' by default. )rZNHWCrrcsg|]}td|qSr)ryrrr-r.rsz.SwinTransformerV2.__init__..r)r0rrrr output_fmtcSsg|] }|qSr-)tolist)rrr-r-r.r s)r5rrrrr6rrr7r9r8rrr:r layers.)Znum_chsrmodule)Z pool_typerZ input_fmtN)"rFrGrrrrlenZ num_layersrryZ num_featuresZ feature_inforrrrr patch_embedrJZlinspacerYsplitrZ grid_sizedictrHrMlayersrrheadapply _init_weightsr)r`r0rrrrrrr6rrr7rrrrrrkwargsZdprrZin_dimscalerrZblyrarr.rGsv*    "  &    zSwinTransformerV2.__init__cCsDt|tjr@t|jddt|tjr@|jdk r@tj|jddS)Ng{Gz?)stdr)rrHrNrrer=rr)r`mr-r-r.r6s zSwinTransformerV2._init_weightscs<t}|D](\}tfdddDr|q|S)Ncsg|] }|kqSr-r-)rkwnr-r.r@sz5SwinTransformerV2.no_weight_decay..)rOrL)setZ named_modulesradd)r`Znodrr-rr.no_weight_decay<s  z!SwinTransformerV2.no_weight_decayFcCstd|r dndddgdS)Nz^absolute_pos_embed|patch_embedz^layers\.(\d+))z^layers\.(\d+).downsampler)z^layers\.(\d+)\.\w+\.(\d+)N)z^norm)i)stemr)r)r`Zcoarser-r-r. group_matcherDszSwinTransformerV2.group_matchercCs|jD] }||_qdSr)rr)r`enablelr-r-r.set_grad_checkpointingOs z(SwinTransformerV2.set_grad_checkpointingcCs|jjSr)rfc)r`r-r-r.get_classifierTsz SwinTransformerV2.get_classifierNcCs||_|j||dSr)rrreset)r`rrr-r-r.reset_classifierXsz"SwinTransformerV2.reset_classifiercCs"||}||}||}|Sr)rrrr`rr-r-r.forward_features\s   z"SwinTransformerV2.forward_features pre_logitscCs|r|j|ddS||S)NTr)r)r`rrr-r-r. forward_headbszSwinTransformerV2.forward_headcCs||}||}|Sr)rrrr-r-r.rtes  zSwinTransformerV2.forward)F)T)N)F)rurvrwrxrHrrrystrrr{rzrrGrrJrignorerrrrrrrrtr}r-r-rar.rsd   m     c s|d|}|d|}d|k}i}ddl}|D]\}tfdddDrTq4dkr|jjjj\}}}}|jd |ks|jd |krt|||fd d d d }|s| ddd dd||<q4|S)Nmodel state_dictzhead.fc.weightrcsg|] }|kqSr-r-)rrrqr-r.rrsz(checkpoint_filter_fn..)rCr@rzpatch_embed.proj.weightrfr"bicubicT) interpolationZ antialiasverbosezlayers.(\d+).downsamplecSsdt|dddS)Nrrz .downsample)rygroup)rr-r-r.z&checkpoint_filter_fn..zhead.zhead.fc.) getreitemsrrr^rer$rsubreplace) rrZnative_checkpointZout_dictrrr_r)r*r-rr.checkpoint_filter_fnks.    r FcKsPtddt|ddD}|d|}tt||fttd|dd|}|S) Ncss|]\}}|VqdSrr-)rrr r-r-r.rsz._create_swin_transformer_v2..r)rrrr out_indicesT)Zflatten_sequentialr )Zpretrained_filter_fnZ feature_cfg)r enumeraterpoprrr r)variant pretrainedrZdefault_out_indicesr rr-r-r._create_swin_transformer_v2s  rrc Ks"|ddddddttddd d |S) Nr)rr)r>r>g?rTzpatch_embed.projzhead.fcZmit) urlr input_size pool_sizecrop_pctrZfixed_input_sizeZmeanrZ first_conv classifierlicenser)rrr-r-r._cfgsrztimm/z{https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window12to16_192to256_22kto1k_ft.pth) hf_hub_idrz{https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window12to24_192to384_22kto1k_ft.pth)rr)rrr?)rrrrrz|https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_large_patch4_window12to16_192to256_22kto1k_ft.pthz|https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_large_patch4_window12to24_192to384_22kto1k_ft.pthzfhttps://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_tiny_patch4_window8_256.pthzghttps://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_tiny_patch4_window16_256.pthzghttps://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_small_patch4_window8_256.pthzhhttps://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_small_patch4_window16_256.pthzfhttps://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window8_256.pthzghttps://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window16_256.pthzkhttps://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window12_192_22k.pthiQU)rr)rr)rrrrrzlhttps://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_large_patch4_window12_192_22k.pth) 2swinv2_base_window12to16_192to256.ms_in22k_ft_in1k2swinv2_base_window12to24_192to384.ms_in22k_ft_in1k3swinv2_large_window12to16_192to256.ms_in22k_ft_in1k3swinv2_large_window12to24_192to384.ms_in22k_ft_in1kzswinv2_tiny_window8_256.ms_in1kz swinv2_tiny_window16_256.ms_in1kz swinv2_small_window8_256.ms_in1kz!swinv2_small_window16_256.ms_in1kzswinv2_base_window8_256.ms_in1kz swinv2_base_window16_256.ms_in1k!swinv2_base_window12_192.ms_in22k"swinv2_large_window12_192.ms_in22krcKs*tddddd}tdd|it|f|S)  rhrrrrrrr6swinv2_tiny_window16_256r)r&rrrrZ model_argsr-r-r.r&s r&cKs*tddddd}tdd|it|f|S) r$r>rrrr%swinv2_tiny_window8_256r)r)r'r(r-r-r.r)s r)cKs*tddddd}tdd|it|f|S) r$rhrrrrrr%swinv2_small_window16_256r)r,r'r(r-r-r.r,s r,cKs*tddddd}tdd|it|f|S) r$r>rr*rr%swinv2_small_window8_256r)r-r'r(r-r-r.r-s r-cKs*tddddd}tdd|it|f|S) r$rhr*r r>rh r%swinv2_base_window16_256r)r1r'r(r-r-r.r1s r1cKs*tddddd}tdd|it|f|S) r$r>r.r*r/r%swinv2_base_window8_256r)r2r'r(r-r-r.r2s r2cKs*tddddd}tdd|it|f|S) r$rr.r*r/r%swinv2_base_window12_192r)r3r'r(r-r-r.r3s r3cKs,tdddddd}td d|it|f|S) r$rhr.r*r/rrrrrrrr6r!swinv2_base_window12to16_192to256r)r6r'r(r-r-r.r6s r6cKs,tdddddd}td d|it|f|S) r$rr.r*r/r4r5!swinv2_base_window12to24_192to384r)r7r'r(r-r-r.r7$s r7cKs*tddddd}tdd|it|f|S) r$rrr*rrr0r%swinv2_large_window12_192r)r:r'r(r-r-r.r:/s r:cKs,tdddddd}td d|it|f|S) r$rhrr*r8r4r5"swinv2_large_window12to16_192to256r)r;r'r(r-r-r.r;8s r;cKs,tdddddd}td d|it|f|S) r$rrr*r8r4r5"swinv2_large_window12to24_192to384r)r<r'r(r-r-r.r<Cs r<r"rrr#r r!)Zswinv2_base_window12_192_22kZ)swinv2_base_window12to16_192to256_22kft1kZ)swinv2_base_window12to24_192to384_22kft1kZswinv2_large_window12_192_22kZ*swinv2_large_window12to16_192to256_22kft1kZ*swinv2_large_window12to24_192to384_22kft1k)F)r)F)F)F)F)F)F)F)F)F)F)F)F)DrxrVtypingrrrrrrrJZtorch.nnrHZtorch.nn.functionalZ functionalrjZtorch.utils.checkpointutilsrZ timm.datar r Z timm.layersr r r rrrrrrZ_builderrZ _features_fxr _registryrrr__all__ryrr|r/r1rr2r~rrrr rrZ default_cfgsr&r)r,r-r1r2r3r6r7r:r;r<rur-r-r-r.s  ,  *} ]* :