U ʹj@sddlZddlmZddlmZmZmZmZddlZddl m m Z ddlm Z m Z ddlmZmZddlmZddlmZmZdd lmZd d lmZmZmZd d lmZd d lmZm Z dddddddddddddg Z!ej ej dddZ"ej#$dej ej ee%ej ddd Z&ej#$d Gd!d"d"e j'Z(Gd#d$d$e j'Z)dGe e e e ee%e%ee%e*e*ee ee eej e+e d'd(d)Z,ej#$d)Gd*d+d+e j'Z-Gd,d-d-e-Z.Gd.d/d/e j'Z/Gd0d1d1e/Z0Gd2dde j'Z1ee%e%ee%ee%ee%e*eee+ee1d3 d4d5Z2d6eiZ3Gd7ddeZ4Gd8ddeZ5Gd9ddeZ6Gd:ddeZ7Gd;ddeZ8Gddd&d?ee4e+ee1d@dAdZ;ee d=e5j:fd>dd&d?ee5e+ee1d@dBdZdd&d?ee6e+ee1d@dCdZ=ee d=e7j:fd>dd&d?ee7e+ee1d@dDdZ>ee d=e8j:fd>dd&d?ee8e+ee1d@dEdZ?ee d=e9j:fd>dd&d?ee9e+ee1d@dFdZ@dS)HN)partial)AnyCallableListOptional)nnTensor)MLPPermute)StochasticDepth)ImageClassificationInterpolationMode)_log_api_usage_once)register_modelWeights WeightsEnum)_IMAGENET_CATEGORIES)_ovewrite_named_paramhandle_legacy_interfaceSwinTransformerSwin_T_WeightsSwin_S_WeightsSwin_B_WeightsSwin_V2_T_WeightsSwin_V2_S_WeightsSwin_V2_B_Weightsswin_tswin_sswin_b swin_v2_t swin_v2_s swin_v2_bxreturnc Cs|jdd\}}}t|ddd|dd|df}|dddddddddf}|dddddddddf}|dddddddddf}|dddddddddf}t||||gd}|S)Nrr .r)shapeFpadtorchcat)r%HW_Zx0x1Zx2Zx3r2b/var/www/html/bwave/python/venv/lib/python3.8/site-packages/torchvision/models/swin_transformer.py_patch_merging_pad#s     r4)relative_position_bias_tablerelative_position_index window_sizer&cCsB|d|d}||}|||d}|dddd}|S)Nrrr(r )viewpermute contiguous unsqueeze)r5r6r7Nrelative_position_biasr2r2r3_get_relative_position_bias1s r>csFeZdZdZejfeedejfdfdd Z e dddZ Z S) PatchMergingzPatch Merging Layer. Args: dim (int): Number of input channels. norm_layer (nn.Module): Normalization layer. Default: nn.LayerNorm. .dim norm_layercsDtt|||_tjd|d|dd|_|d||_dSNr Fbiassuper__init__rrArLinear reductionnormselfrArB __class__r2r3rIEs  zPatchMerging.__init__r%cCs t|}||}||}|Sz Args: x (Tensor): input tensor with expected layout of [..., H, W, C] Returns: Tensor with layout of [..., H/2, W/2, 2*C] )r4rLrKrNr%r2r2r3forwardLs  zPatchMerging.forward __name__ __module__ __qualname____doc__r LayerNormintrModulerIrrT __classcell__r2r2rOr3r?>s$r?csFeZdZdZejfeedejfdfdd Z e dddZ Z S) PatchMergingV2zPatch Merging Layer for Swin Transformer V2. Args: dim (int): Number of input channels. norm_layer (nn.Module): Normalization layer. Default: nn.LayerNorm. .r@csDtt|||_tjd|d|dd|_|d||_dSrCrGrMrOr2r3rI`s  zPatchMergingV2.__init__rQcCs t|}||}||}|SrR)r4rKrLrSr2r2r3rTgs  zPatchMergingV2.forwardrUr2r2rOr3r^Ys$r^T)input qkv_weight proj_weightr=r7 num_heads shift_sizeattention_dropoutdropoutqkv_bias proj_bias logit_scaletrainingr&c $ Cs|j\} }}}|d||d|d}|d||d|d}t|ddd|d|f}|j\}}}}|}|d|krd|d<|d|krd|d<t|dkrtj||d |d fdd}||d||d}|| ||d|d||d|d|}|dddddd | ||d|d|}| d k rv| d k rv| } | d}| |d| t ||| }| |d|dd|||ddddd}|d|d|d}}}| d k r tj|d d tj|d d d d }tj| td d} || }n"|||d}||d d }||}t|dkr|||f}d|d f|d |d f|d d ff}d|d f|d |d f|d d ff} d}!|D]>}"| D]2}#|!||"d|"d|#d|#df<|!d7}!qҐq|||d|d||d|d}|dddd ||d|d}|d|d}||dktd|dktd}||d||||d|d}||dd}|d ||d|d}tj|d d }tj||| d}||dd |d|d|}t ||| }tj||| d}|| ||d||d|d|d|}|dddddd | |||}t|dkrtj||d|dfdd}|d d d |d |d d f}|S)a Window based multi-head self attention (W-MSA) module with relative position bias. It supports both of shifted and non-shifted window. Args: input (Tensor[N, H, W, C]): The input tensor or 4-dimensions. qkv_weight (Tensor[in_dim, out_dim]): The weight tensor of query, key, value. proj_weight (Tensor[out_dim, out_dim]): The weight tensor of projection. relative_position_bias (Tensor): The learned relative position bias added to attention. window_size (List[int]): Window size. num_heads (int): Number of attention heads. shift_size (List[int]): Shift size for shifted window attention. attention_dropout (float): Dropout ratio of attention weight. Default: 0.0. dropout (float): Dropout ratio of output. Default: 0.0. qkv_bias (Tensor[out_dim], optional): The bias tensor of query, key, value. Default: None. proj_bias (Tensor[out_dim], optional): The bias tensor of projection. Default: None. logit_scale (Tensor[out_dim], optional): Logit scale of cosine attention for Swin Transformer V2. Default: None. training (bool, optional): Training flag used by the dropout parameters. Default: True. Returns: Tensor[N, H, W, C]: The output tensor after shifted window attention. rr)rr )ZshiftsZdimsr rDNr()rAgY@)maxggYr_)prj)r)r*r+copysumr,Zrollr8r9Zreshapeclonenumelzero_Zlinearsize normalizeZ transposeclampmathlogexpmatmulZ new_zerosr;Z masked_fillfloatZsoftmaxrfr:)$r`rarbr=r7rcrdrerfrgrhrirjBr.r/CZpad_rZpad_br%r0Zpad_HZpad_WZ num_windowslengthqkvqkvattnZ attn_maskZh_slicesZw_slicescounthwr2r2r3shifted_window_attentiontsp#    ,. 0 $ ..$($$((, $rc sneZdZdZdeeeeeeeeeedfdd ZddZ d d Z e j d d d Z e e dddZZS)ShiftedWindowAttentionz/ See :func:`shifted_window_attention`. Tr_rAr7rdrcrgrhrerfc stt|dks"t|dkr*td||_||_||_||_||_t j ||d|d|_ t j |||d|_ | |dS)Nr z.window_size and shift_size must be of length 2rkrE)rHrIlen ValueErrorr7rdrcrerfrrJrproj#define_relative_position_bias_tabledefine_relative_position_index) rNrAr7rdrcrgrhrerfrOr2r3rIs zShiftedWindowAttention.__init__cCsLttd|jddd|jdd|j|_tjj|jdddS)Nr rr{Gz?Zstd) r Parameterr,Zzerosr7rcr5init trunc_normal_rNr2r2r3r s,z:ShiftedWindowAttention.define_relative_position_bias_tablecCst|jd}t|jd}ttj||dd}t|d}|dddddf|dddddf}|ddd}|dddddf|jdd7<|dddddf|jdd7<|dddddfd|jdd9<|d}| d|dS)NrrijZindexingr r(r6) r,aranger7stackmeshgridflattenr9r:rqregister_buffer)rNZcoords_hZcoords_wZcoordsZcoords_flattenZrelative_coordsr6r2r2r3rs ,((,z5ShiftedWindowAttention.define_relative_position_indexr&cCst|j|j|jSN)r>r5r6r7rr2r2r3get_relative_position_biass z1ShiftedWindowAttention.get_relative_position_biasr$cCsD|}t||jj|jj||j|j|j|j|j |jj |jj |j d S) Args: x (Tensor): Tensor with layout of [B, H, W, C] Returns: Tensor with same layout as input, i.e. [B, H, W, C] )rdrerfrgrhrj) rrrweightrr7rcrdrerfrFrjrNr%r=r2r2r3rT#szShiftedWindowAttention.forward)TTr_r_)rVrWrXrYr[rboolr|rIrrr,rrrTr]r2r2rOr3rs& rc sdeZdZdZdeeeeeeeeeedfdd ZddZ e j d d d Z e d d dZ ZS)ShiftedWindowAttentionV2z2 See :func:`shifted_window_attention_v2`. Tr_rc stj||||||||dttdt|ddf|_ttj ddddtj ddtj d|d d|_ |r|j j d } |j j | d| jdS) N)rgrhrerf rr iTrE)inplaceFrk)rHrIrrr,ryZonesri SequentialrJZReLUcpb_mlprrFrsdatart) rNrAr7rdrcrgrhrerfrrOr2r3rI@s&  " z!ShiftedWindowAttentionV2.__init__cCstj|jdd |jdtjd}tj|jdd |jdtjd}ttj||gdd}|dddd}|dddddddf|jdd<|dddddddf|jdd<|d9}t |t t |dd }| d |dS) Nrr)Zdtyperrr g?g@relative_coords_table) r,rr7Zfloat32rrr9r:r;signlog2absr)rNZrelative_coords_hZrelative_coords_wrr2r2r3r_s$$.. zrrr8rcr6r7r,Zsigmoid)rNr=r2r2r3rosz3ShiftedWindowAttentionV2.get_relative_position_biasrQcCsH|}t||jj|jj||j|j|j|j|j |jj |jj |j |j d S)r)rdrerfrgrhrirj) rrrrrr7rcrdrerfrFrirjrr2r2r3rTxs z ShiftedWindowAttentionV2.forward)TTr_r_)rVrWrXrYr[rrr|rIrr,rrrTr]r2r2rOr3r;s$  rc sreZdZdZddddejefeeeeeee e e e e dej fe dej fd fdd Z e dd d ZZS) SwinTransformerBlocka Swin Transformer Block. Args: dim (int): Number of input channels. num_heads (int): Number of attention heads. window_size (List[int]): Window size. shift_size (List[int]): Shift size for shifted window attention. mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. Default: 4.0. dropout (float): Dropout rate. Default: 0.0. attention_dropout (float): Attention dropout rate. Default: 0.0. stochastic_depth_prob: (float): Stochastic depth rate. Default: 0.0. norm_layer (nn.Module): Normalization layer. Default: nn.LayerNorm. attn_layer (nn.Module): Attention layer. Default: ShiftedWindowAttention @r_. rArcr7rd mlp_ratiorfrestochastic_depth_probrB attn_layerc stt|| ||_| ||||||d|_t|d|_| ||_t|t |||gt j d|d|_ |j D]:} t| t jrtt j| j| jdk rtt jj| jddqtdS)N)rerfrow)Zactivation_layerrrfgư>r)rHrIrnorm1rr stochastic_depthnorm2r r[rZGELUmlpmodules isinstancerJrZxavier_uniform_rrFZnormal_) rNrArcr7rdrrfrerrBrmrOr2r3rIs&    "  zSwinTransformerBlock.__init__rQcCs8|||||}|||||}|Sr)rrrrrrSr2r2r3rTszSwinTransformerBlock.forward)rVrWrXrYrrZrr[rr|rr\rIrrTr]r2r2rOr3rs(  #rc sreZdZdZddddejefeeeeeee e e e e dej fe dej fd fdd Z e dd d ZZS) SwinTransformerBlockV2a Swin Transformer V2 Block. Args: dim (int): Number of input channels. num_heads (int): Number of attention heads. window_size (List[int]): Window size. shift_size (List[int]): Shift size for shifted window attention. mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. Default: 4.0. dropout (float): Dropout rate. Default: 0.0. attention_dropout (float): Attention dropout rate. Default: 0.0. stochastic_depth_prob: (float): Stochastic depth rate. Default: 0.0. norm_layer (nn.Module): Normalization layer. Default: nn.LayerNorm. attn_layer (nn.Module): Attention layer. Default: ShiftedWindowAttentionV2. rr_.rc s$tj||||||||| | d dS)N)rrfrerrBr)rHrI) rNrArcr7rdrrfrerrBrrOr2r3rIs zSwinTransformerBlockV2.__init__rQcCs8|||||}|||||}|Sr)rrrrrrSr2r2r3rTszSwinTransformerBlockV2.forward)rVrWrXrYrrZrr[rr|rr\rIrrTr]r2r2rOr3rs(  rcseZdZdZdddddddefeeeeeeeeeeeeeeee de j fee de j fe de j fd fd d Z d d Z ZS) ra; Implements Swin Transformer from the `"Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" `_ paper. Args: patch_size (List[int]): Patch size. embed_dim (int): Patch embedding dimension. depths (List(int)): Depth of each Swin Transformer layer. num_heads (List(int)): Number of attention heads in different layers. window_size (List[int]): Window size. mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. Default: 4.0. dropout (float): Dropout rate. Default: 0.0. attention_dropout (float): Attention dropout rate. Default: 0.0. stochastic_depth_prob (float): Stochastic depth rate. Default: 0.1. num_classes (int): Number of classes for classification head. Default: 1000. block (nn.Module, optional): SwinTransformer Block. Default: None. norm_layer (nn.Module, optional): Normalization layer. Default: None. downsample_layer (nn.Module): Downsample layer (patch merging). Default: PatchMerging. rr_g?iN.) patch_size embed_dimdepthsrcr7rrfrer num_classesrBblockdownsample_layercstt|| |_| dkr$t} | dkr:ttjdd} g}|t tj d||d|df|d|dfdt ddddg| |t |}d}t t|D]}g}|d|}t ||D]R| t||d}|| ||||fdd |D||||| d |d7}q|tj ||t|dkr|| || qtj ||_|dt|d}| ||_t ddddg|_td|_td|_t|| |_|D]@}t|tjrtjj|jd d |jdk rtj|jqdS) Ngh㈵>)epsrkrr)Z kernel_sizeZstrider cs$g|]}ddkrdn|dqS)r rr2).0rZi_layerr2r3 Csz,SwinTransformer.__init__..)r7rdrrfrerrBrr) rHrIrrrrrrZappendrZConv2dr rqrangerr|featuresrLr9ZAdaptiveAvgPool2davgpoolZFlattenrrJheadrrrrrrFZzeros_)rNrrrrcr7rrfrerrrBrrZlayersZtotal_stage_blocksZstage_block_idZi_stageZstagerAZsd_probZ num_featuresrrOrr3rIsp          zSwinTransformer.__init__cCs@||}||}||}||}||}||}|Sr)rrLr9rrrrSr2r2r3rT_s      zSwinTransformer.forward)rVrWrXrYr?rr[r|rrrr\rIrTr]r2r2rOr3rs2 O) rrrrcr7rweightsprogresskwargsr&c KsZ|dk rt|dt|jdtf||||||d|} |dk rV| |j|dd| S)Nr categories)rrrrcr7rT)rZ check_hash)rrmetarZload_state_dictZget_state_dict) rrrrcr7rrrrmodelr2r2r3_swin_transformeris  rrc @sJeZdZedeeddejdeddddd d d id d dddZ e Z dS)rz7https://download.pytorch.org/models/swin_t-704ceda3.pthZ crop_sizeZ resize_size interpolationibrrUhttps://github.com/pytorch/vision/tree/main/references/classification#swintransformer ImageNet-1KguV^T@glW@zacc@1zacc@5gX9@g\( [@YThese weights reproduce closely the results of the paper using a similar training recipe.Z num_paramsZmin_sizeZrecipeZ_metricsZ_ops _file_sizeZ_docsurlZ transformsrN rVrWrXrrr rZBICUBIC _COMMON_META IMAGENET1K_V1DEFAULTr2r2r2r3rs.c @sJeZdZedeeddejdeddddd d d id d dddZ e Z dS)rz7https://download.pytorch.org/models/swin_s-5e29d889.pthrrirrrrgCT@gףp= X@rgZd{!@gx&g@rrrNrr2r2r2r3rs.c @sJeZdZedeeddejdeddddd d d id d dddZ e Z dS)rz7https://download.pytorch.org/models/swin_b-68c6b09e.pthrri<;rrrgh|?T@g)\(X@rg&1.@gt@rrrNrr2r2r2r3rs.c @sJeZdZedeeddejdeddddd d d id d dddZ e Z dS)rz:https://download.pytorch.org/models/swin_v2_t-b137f0e2.pthriRrrXhttps://github.com/pytorch/vision/tree/main/references/classification#swintransformer-v2rgS㥛T@g rX@rg(\@gMb([@rrrNrr2r2r2r3rs.c @sJeZdZedeeddejdeddddd d d id d dddZ e Z dS)rz:https://download.pytorch.org/models/swin_v2_s-637d8ceb.pthrrrirrrg!rhT@gNbX94X@rgd;O'@gg@rrrNrr2r2r2r3rs.c @sJeZdZedeeddejdeddddd d d id d dddZ e Z dS)rz:https://download.pytorch.org/models/swin_v2_b-781e5279.pthriri=rrrgI +U@gK7X@rg33333S4@gˡEu@rrrNrr2r2r2r3r s.Z pretrained)r)rr)rrrr&c KsBt|}tfddgdddddgddddgddgd ||d |S) a Constructs a swin_tiny architecture from `Swin Transformer: Hierarchical Vision Transformer using Shifted Windows `_. Args: weights (:class:`~torchvision.models.Swin_T_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.Swin_T_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.swin_transformer.SwinTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.Swin_T_Weights :members: rD`r rk 皙?rrrrcr7rrr)rverifyrrrrr2r2r3r"s    c KsBt|}tfddgdddddgddddgd d gd ||d |S) a Constructs a swin_small architecture from `Swin Transformer: Hierarchical Vision Transformer using Shifted Windows `_. Args: weights (:class:`~torchvision.models.Swin_S_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.Swin_S_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.swin_transformer.SwinTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.Swin_S_Weights :members: rDrr rkrrrr333333?r)rrrrr2r2r3rHs    c KsBt|}tfddgdddddgddddgddgd ||d |S) a Constructs a swin_base architecture from `Swin Transformer: Hierarchical Vision Transformer using Shifted Windows `_. Args: weights (:class:`~torchvision.models.Swin_B_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.Swin_B_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.swin_transformer.SwinTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.Swin_B_Weights :members: rDr rrr r?r)rrrrr2r2r3r ns    c KsFt|}tfddgdddddgddddgddgd ||ttd |S) a Constructs a swin_v2_tiny architecture from `Swin Transformer V2: Scaling Up Capacity and Resolution `_. Args: weights (:class:`~torchvision.models.Swin_V2_T_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.Swin_V2_T_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.swin_transformer.SwinTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.Swin_V2_T_Weights :members: rDrr rrkrrrr rrrrcr7rrrrr)rrrrr^rr2r2r3r!s    c KsFt|}tfddgdddddgddddgd d gd ||ttd |S) a Constructs a swin_v2_small architecture from `Swin Transformer V2: Scaling Up Capacity and Resolution `_. Args: weights (:class:`~torchvision.models.Swin_V2_S_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.Swin_V2_S_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.swin_transformer.SwinTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.Swin_V2_S_Weights :members: rDrr rrkrrrrrr)rrrrr^rr2r2r3r"s    c KsFt|}tfddgdddddgddddgddgd||ttd |S) a Constructs a swin_v2_base architecture from `Swin Transformer V2: Scaling Up Capacity and Resolution `_. Args: weights (:class:`~torchvision.models.Swin_V2_B_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.Swin_V2_B_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.swin_transformer.SwinTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.Swin_V2_B_Weights :members: rDrr rrrrrr)rrrrr^rr2r2r3r#s    )r_r_NNNT)Arx functoolsrtypingrrrrr,Ztorch.nn.functionalrZ functionalr*rZops.miscr r Zops.stochastic_depthr Ztransforms._presetsr rutilsrZ_apirrr_metarZ_utilsrr__all__r4Zfxwrapr[r>r\r?r^r|rrrrrrrrrrrrrrrrrrr r!r"r#r2r2r2r3s      # s QV92n $$$$$$$&$&