U ʹj| @s^ddlZddlmZddlmZddlmZmZmZm Z m Z m Z ddl Z ddlZddlmmZddlmZmZddlmZmZmZddlmZddlmZmZdd lmZmZdd l m!Z!dd l"m#Z#m$Z$dd l%m&Z&d ddgZ'e e(e(fe(e(e(e e(e(fdddZ)e e(e(fe(ee e(e(fdddZ*e(e(ejdddZ+Gdddej,Z-Gdddej,Z.Gdddej,Z/Gdd d ej,Z0Gd!d"d"ej,Z1Gd#d$d$ej,Z2Gd%d&d&ej,Z3Gd'd(d(ej,Z4Gd)d d ej,Z5d5e(ee(ee(e6e(e(e ee7ee5d+ d,d-Z8Gd.ddeZ9eed/e9j:fd0dd1d2e e9e7ee5d3d4dZ;dS)6N) OrderedDict)partial)AnyCallableListOptionalSequenceTuple)nnTensor)register_modelWeights WeightsEnum)_IMAGENET_CATEGORIES)_ovewrite_named_paramhandle_legacy_interface)Conv2dNormActivationSqueezeExcitation)StochasticDepth)ImageClassificationInterpolationMode)_log_api_usage_onceMaxVitMaxVit_T_Weightsmaxvit_t) input_size kernel_sizestridepaddingreturncCs8|d|d||d|d|d||dfS)Nr)rrrrr"r"X/var/www/html/bwave/python/venv/lib/python3.8/site-packages/torchvision/models/maxvit.py_get_conv_output_shapesr$)rn_blocksrcCs<g}t|ddd}t|D]}t|ddd}||q|S)zQUtil function to check that the input size is correct for a MaxVit configuration.r r!)r$rangeappend)rr%ZshapesZblock_input_shape_r"r"r#_make_block_input_shapes s   r*)heightwidthrcCsttt|t|g}t|d}|dddddf|dddddf}|ddd}|dddddf|d7<|dddddf|d7<|dddddfd|d9<|dS)Nr!r r)torchstackZmeshgridZarangeflattenpermute contiguoussum)r+r,ZcoordsZ coords_flatZrelative_coordsr"r"r#_get_relative_position_index*s  ,""&r4c s\eZdZdZd eeeeeedejfedejfedd fdd Z e e dd d Z Z S) MBConva=MBConv: Mobile Inverted Residual Bottleneck. Args: in_channels (int): Number of input channels. out_channels (int): Number of output channels. expansion_ratio (float): Expansion ratio in the bottleneck. squeeze_ratio (float): Squeeze ratio in the SE Layer. stride (int): Stride of the depthwise convolution. activation_layer (Callable[..., nn.Module]): Activation function. norm_layer (Callable[..., nn.Module]): Normalization function. p_stochastic_dropout (float): Probability of stochastic depth. .N) in_channels out_channelsexpansion_ratio squeeze_ratioractivation_layer norm_layerp_stochastic_dropoutrc  s*t||dkp||k} | rdtj||ddddg} |dkrVtjd|ddg| } tj| |_n t|_t||} t||} |rt |dd|_ n t|_ t } ||| d <t || ddd ||dd | d <t | | d|d||| dd | d<t | | tjd| d<tj| |ddd| d<t| |_dS)Nr!T)rrbiasr r&rrrrowmodeZpre_normr)rrrr;r<inplaceZconv_a)rrrr;r<groupsrCZconv_b)Z activationZsqueeze_excitation)r7r8rr>Zconv_c)super__init__r Conv2dZ AvgPool2d SequentialprojIdentityintrstochastic_depthrrrZSiLUlayers)selfr7r8r9r:rr;r<r=Z should_projrIZ mid_channelsZ sqz_channelsZ_layers __class__r"r#rFCsP        zMBConv.__init__xrcCs"||}|||}||S)z Args: x (Tensor): Input tensor with expected layout of [B, C, H, W]. Returns: Tensor: Output tensor with expected layout of [B, C, H / stride, W / stride]. )rIrLrMrNrRresr"r"r#forwards zMBConv.forward)r6) __name__ __module__ __qualname____doc__rKfloatrr ModulerFr rU __classcell__r"r"rOr#r55s  =r5csLeZdZdZeeeddfdd ZejdddZeed d d Z Z S) $RelativePositionalMultiHeadAttentionzRelative Positional Multi-Head Attention. Args: feat_dim (int): Number of input features. head_dim (int): Number of features per head. max_seq_len (int): Maximum sequence length. N)feat_dimhead_dim max_seq_lenrcst||dkr*td|d||||_||_tt||_||_ t ||j|jd|_ |d|_ t |j|j||_t jtjd|jdd|jd|jftjd|_|d t|j|jtj jj|jd d dS) Nrz feat_dim: z must be divisible by head_dim: r&gr r!)Zdtyperelative_position_index{Gz?Zstd)rErF ValueErrorn_headsr_rKmathsqrtsizer`r Linearto_qkv scale_factormergeZ parameter Parameterr.emptyZfloat32relative_position_bias_tableZregister_bufferr4initZ trunc_normal_)rNr^r_r`rOr"r#rFs    ,z-RelativePositionalMultiHeadAttention.__init__rcCs@|jd}|j||j|jd}|ddd}|dS)Nr-r rr!)raviewror`r1r2Z unsqueeze)rNZ bias_indexZ relative_biasr"r"r#get_relative_positional_biass zARelativePositionalMultiHeadAttention.get_relative_positional_biasrQcCs|j\}}}}|j|j}}||}tj|ddd\} } } | |||||ddddd} | |||||ddddd} | |||||ddddd} | |j} t d| | } | } t j | | dd} t d | | }|ddddd||||}| |}|S) z Args: x (Tensor): Input tensor with expected layout of [B, G, P, D]. Returns: Tensor: Output tensor with expected layout of [B, G, P, D]. r&r-)dimrr!r z!B G H I D, B G H J D -> B G H I Jz!B G H I J, B G H J D -> B G H I D)shaperer_rjr.chunkreshaper1rkZeinsumrsFZsoftmaxrl)rNrRBGPDHZDHZqkvqkvZdot_prodZpos_biasoutr"r"r#rUs      z,RelativePositionalMultiHeadAttention.forward) rVrWrXrYrKrFr.r rsrUr\r"r"rOr#r]s r]cs>eZdZdZeeddfdd ZejejdddZZ S) SwapAxeszPermute the axes of a tensor.N)abrcst||_||_dSN)rErFrr)rNrrrOr"r#rFs zSwapAxes.__init__rQcCst||j|j}|Sr)r.ZswapaxesrrrSr"r"r#rUszSwapAxes.forward) rVrWrXrYrKrFr.r rUr\r"r"rOr#rsrcs8eZdZdZddfdd ZeeedddZZS) WindowPartitionzB Partition the input tensor into non-overlapping windows. NrqcstdSrrErFrNrOr"r#rFszWindowPartition.__init__)rRprcCsf|j\}}}}|}|||||||||}|dddddd}|||||||||}|S)z Args: x (Tensor): Input tensor with expected layout of [B, C, H, W]. p (int): Number of partitions. Returns: Tensor: Output tensor with expected layout of [B, H/P, W/P, P*P, C]. rr rur&r!rvrxr1)rNrRrrzCr~Wr|r"r"r#rUs  zWindowPartition.forward rVrWrXrYrFr rKrUr\r"r"rOr#rsrcs<eZdZdZddfdd ZeeeeedddZZS) WindowDepartitionzo Departition the input tensor of non-overlapping windows into a feature volume of layout [B, C, H, W]. NrqcstdSrrrrOr"r#rFszWindowDepartition.__init__)rRr h_partitions w_partitionsrc Cs`|j\}}}}|} ||} } ||| | | | |}|dddddd}|||| | | | }|S)ar Args: x (Tensor): Input tensor with expected layout of [B, (H/P * W/P), P*P, C]. p (int): Number of partitions. h_partitions (int): Number of vertical partitions. w_partitions (int): Number of horizontal partitions. Returns: Tensor: Output tensor with expected layout of [B, C, H, W]. rrr!r&r rur) rNrRrrrrzr{ZPPrr|ZHPZWPr"r"r#rUs  zWindowDepartition.forwardrr"r"rOr#rsrc sheZdZdZeeeeeeefeedej fedej fe e e dd fdd Z e e ddd Z ZS) PartitionAttentionLayera Layer for partitioning the input tensor into non-overlapping windows and applying attention to each window. Args: in_channels (int): Number of input channels. head_dim (int): Dimension of each attention head. partition_size (int): Size of the partitions. partition_type (str): Type of partitioning to use. Can be either "grid" or "window". grid_size (Tuple[int, int]): Size of the grid to partition the input tensor into. mlp_ratio (int): Ratio of the feature size expansion in the MLP layer. activation_layer (Callable[..., nn.Module]): Activation function to use. norm_layer (Callable[..., nn.Module]): Normalization function to use. attention_dropout (float): Dropout probability for the attention layer. mlp_dropout (float): Dropout probability for the MLP layer. p_stochastic_dropout (float): Probability of dropping out a partition. .N) r7r_partition_sizepartition_type grid_size mlp_ratior;r<attention_dropout mlp_dropoutr=rc s(t|||_||_|d||_||_||_|dkrDtd|dkr^||j|_|_ n|j||_|_ t |_ t |_ |dkrtddnt|_|dkrtddnt|_t||t|||dt| |_tt|t||||t|||t| |_t| d d |_dS) Nr)gridwindowz0partition_type must be either 'grid' or 'window'rrr r@rA)rErFrer_Z n_partitionsrrrdrgr partition_oprdepartition_oprr rJpartition_swapdepartition_swaprHr]ZDropout attn_layer LayerNormri mlp_layerrstochastic_dropout) rNr7r_rrrrr;r<rrr=rOr"r#rF,s8    z PartitionAttentionLayer.__init__rQcCs|jd|j|jd|j}}t|jd|jdkoL|jd|jdkd|j|j|||j}||}||||}||| |}| |}| ||j||}|S)z Args: x (Tensor): Input tensor with expected layout of [B, C, H, W]. Returns: Tensor: Output tensor with expected layout of [B, C, H, W]. rr!z[Grid size must be divisible by partition size. Got grid size of {} and partition size of {}) rrr.Z_assertformatrrrrrrr)rNrRghZgwr"r"r#rUfs "&  zPartitionAttentionLayer.forward)rVrWrXrYrKstrr rr r[rZrFr rUr\r"r"rOr#rs   :rcsneZdZdZeeeeeedejfedejfeeeeeee eefddfdd Z e e ddd Z Z S) MaxVitLayera MaxVit layer consisting of a MBConv layer followed by a PartitionAttentionLayer with `window` and a PartitionAttentionLayer with `grid`. Args: in_channels (int): Number of input channels. out_channels (int): Number of output channels. expansion_ratio (float): Expansion ratio in the bottleneck. squeeze_ratio (float): Squeeze ratio in the SE Layer. stride (int): Stride of the depthwise convolution. activation_layer (Callable[..., nn.Module]): Activation function. norm_layer (Callable[..., nn.Module]): Normalization function. head_dim (int): Dimension of the attention heads. mlp_ratio (int): Ratio of the MLP layer. mlp_dropout (float): Dropout probability for the MLP layer. attention_dropout (float): Dropout probability for the attention layer. p_stochastic_dropout (float): Probability of stochastic depth. partition_size (int): Size of the partitions. grid_size (Tuple[int, int]): Size of the input feature grid. .N)r7r8r:r9rr<r;r_rrrr=rrrc stt}t|||||||| d|d<t||| d|| |tj| | | d |d<t||| d|| |tj| | | d |d<t||_dS)N)r7r8r9r:rr;r<r=ZMBconvr) r7r_rrrrr;r<rrr=Zwindow_attentionrZgrid_attention) rErFrr5rr rrHrM)rNr7r8r:r9rr<r;r_rrrr=rrrMrOr"r#rFsN    zMaxVitLayer.__init__rQcCs||}|Sz Args: x (Tensor): Input tensor of shape (B, C, H, W). Returns: Tensor: Output tensor of shape (B, C, H, W). rM)rNrRr"r"r#rUs zMaxVitLayer.forward)rVrWrXrYrKrZrr r[r rFr rUr\r"r"rOr#rs$   ArcsreZdZdZeeeeedejfedejfeeeeee eefee eddfdd Z e e ddd Z ZS) MaxVitBlocka( A MaxVit block consisting of `n_layers` MaxVit layers. Args: in_channels (int): Number of input channels. out_channels (int): Number of output channels. expansion_ratio (float): Expansion ratio in the bottleneck. squeeze_ratio (float): Squeeze ratio in the SE Layer. activation_layer (Callable[..., nn.Module]): Activation function. norm_layer (Callable[..., nn.Module]): Normalization function. head_dim (int): Dimension of the attention heads. mlp_ratio (int): Ratio of the MLP layer. mlp_dropout (float): Dropout probability for the MLP layer. attention_dropout (float): Dropout probability for the attention layer. p_stochastic_dropout (float): Probability of stochastic depth. partition_size (int): Size of the partitions. input_grid_size (Tuple[int, int]): Size of the input feature grid. n_layers (int): Number of layers in the block. p_stochastic (List[float]): List of probabilities for stochastic depth for each layer. .N)r7r8r:r9r<r;r_rrrrinput_grid_sizen_layers p_stochasticrcstt|| ks,td| d|dt|_t| dddd|_t |D]V\}}|dkrddnd}|jt |dkr||n|||||||||| | | |j|d g7_qPdS) Nz'p_stochastic must have length n_layers=z, got p_stochastic=.r&r r!r?r)r7r8r:r9rr<r;r_rrrrrr=) rErFlenrdr ModuleListrMr$r enumerater)rNr7r8r:r9r<r;r_rrrrrrridxrrrOr"r#rFs2   zMaxVitBlock.__init__rQcCs|jD] }||}q|Srr)rNrRlayerr"r"r#rU,s  zMaxVitBlock.forward)rVrWrXrYrKrZrr r[r rrFr rUr\r"r"rOr#rs$   3rcseZdZdZdejddddddfeeefeeeeeeee e e dej fe dej fe e ee e eddfd d Z eed d d ZddZZS)ray Implements MaxVit Transformer from the `MaxViT: Multi-Axis Vision Transformer `_ paper. Args: input_size (Tuple[int, int]): Size of the input image. stem_channels (int): Number of channels in the stem. partition_size (int): Size of the partitions. block_channels (List[int]): Number of channels in each block. block_layers (List[int]): Number of layers in each block. stochastic_depth_prob (float): Probability of stochastic depth. Expands to a list of probabilities for each layer that scales linearly to the specified value. squeeze_ratio (float): Squeeze ratio in the SE Layer. Default: 0.25. expansion_ratio (float): Expansion ratio in the MBConv bottleneck. Default: 4. norm_layer (Callable[..., nn.Module]): Normalization function. Default: None (setting to None will produce a `BatchNorm2d(eps=1e-3, momentum=0.99)`). activation_layer (Callable[..., nn.Module]): Activation function Default: nn.GELU. head_dim (int): Dimension of the attention heads. mlp_ratio (int): Expansion ratio of the MLP layer. Default: 4. mlp_dropout (float): Dropout probability for the MLP layer. Default: 0.0. attention_dropout (float): Dropout probability for the attention layer. Default: 0.0. num_classes (int): Number of classes. Default: 1000. Ng?rur6i.)r stem_channelsrblock_channels block_layersr_stochastic_depth_probr<r;r:r9rrr num_classesrcstt|d}|dkr.ttjddd}t|t|}t|D]J\}}|d|dksl|d|dkrDt d|d|d |d |d qDt t ||dd || d ddt ||dddddd|_ t |dd dd}||_t|_|g|dd}|}td|t|}d}t|||D]X\}}}|jt||| | || || | ||||||||d|jdj}||7}q t tdtt|dt|d|dttj|d|d d|_|dS)Nr&gMbP?gGz?)epsZmomentumrr!z Input size z of block z$ is not divisible by partition size zx. Consider changing the partition size or the input size. Current configuration yields the following block input sizes: rr F)rr<r;r>rCT)rr<r;r>r?r-)r7r8r:r9r<r;r_rrrrrrr)r>) rErFrrr BatchNorm2dr*rrrdrHrstemr$rrblocksnpZlinspacer3tolistzipr(rrZAdaptiveAvgPool2dZFlattenrriZTanh classifier _init_weights)rNrrrrrr_rr<r;r:r9rrrrZinput_channelsZblock_input_sizesrZblock_input_sizer7r8rZp_idxZ in_channelZ out_channelZ num_layersrOr"r#rFMs        zMaxVit.__init__rQcCs,||}|jD] }||}q||}|Sr)rrr)rNrRblockr"r"r#rUs     zMaxVit.forwardcCs|D]}t|tjrDtjj|jdd|jdk rtj|jqt|tj rrtj |jdtj |jdqt|tj rtjj|jdd|jdk rtj|jqdS)Nrbrcr!r) modules isinstancer rGrpZnormal_weightr>Zzeros_rZ constant_ri)rNmr"r"r#rs      zMaxVit._init_weights)rVrWrXrYr ZGELUr rKrrZrrr[rFr rUrr\r"r"rOr#r8s:%  vF) rrrrrr_weightsprogresskwargsrc Ks|dk rPt|dt|jd|jdd|jddks>tt|d|jd|dd} tf||||||| d|} |dk r| |j|d d | S) Nr categoriesmin_sizerr!rr)rrrrr_rrT)rZ check_hash)rrmetaAssertionErrorpoprZload_state_dictZget_state_dict) rrrrrr_rrrrmodelr"r"r#_maxvits&   rc @sHeZdZedeeddejdedddddd d id d d ddZ e Z dS)rz9https://download.pytorch.org/models/maxvit_t-bc5ab103.pthr)Z crop_sizeZ resize_size interpolationirzLhttps://github.com/pytorch/vision/tree/main/references/classification#maxvitz ImageNet-1KgT@g|?5.X@)zacc@1zacc@5gZd;@gK7]@zYThese weights reproduce closely the results of the paper using a similar training recipe.)rZ num_paramsrZrecipeZ_metricsZ_ops _file_sizeZ_docs)urlZ transformsrN) rVrWrXr rrrZBICUBICr IMAGENET1K_V1DEFAULTr"r"r"r#rs.Z pretrained)rT)rr)rrrrc Ks:t|}tfdddddgddddgddd ||d |S) a Constructs a maxvit_t architecture from `MaxViT: Multi-Axis Vision Transformer `_. Args: weights (:class:`~torchvision.models.MaxVit_T_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.MaxVit_T_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.maxvit.MaxVit`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.MaxVit_T_Weights :members: @ir r g?)rrrr_rrrr)rverifyr)rrrr"r"r#rs    )NF)sb      &&  WI haU. *