U ʹjZ@sldZddlmZddlmZddlZddlmZddlmm Z ddlm Z ddl m Z mZddlmZmZmZmZmZmZmZddlmZmZd d lmZd d lmZd d lmZd d lm Z m!Z!dgZ"Gdddej#Z$Gdddej#Z%Gdddej#Z&Gdddej#Z'Gdddej#Z(e ee)e)fdddZ*ee ee)e)fe)e)dddZ+Gdd d ej#Z,Gd!d"d"ej#Z-Gd#d$d$ej#Z.Gd%ddej#Z/d&d'Z0d>d)d*Z1d?d,d-Z2e e2d.d/e2d.d/e2d.d/e2e2e2d0Z3e!d@e/d1d2d3Z4e!dAe/d1d4d5Z5e!dBe/d1d6d7Z6e!dCe/d1d8d9Z7e!dDe/d1d:d;Z8e!dEe/d1dr; norm_layerr(r*r+r!3s z Stem.__init__r,c Cs|j\}}}}t|d|jd||jd|jdf}t|ddd|jd||jd|jdf}||}||}|S)Nrr)shapeFpadr;r@rAr'r-BCHWr*r*r+r/Is .2  z Stem.forward) r0r1r2__doc__r r!rr/r5r*r*r(r+r6.sr6cs.eZdZeffdd ZedddZZS) Downsamplecs:t||_||_|||_tj||dddd|_dS)Nrrr:)r r!r=r>rAr"r#r@)r'r=r>rBr(r*r+r!Ss  zDownsample.__init__r,c Csb|j\}}}}||}t|dd|ddf}t|dddd|ddf}||}|S)Nrr)rCrArDrEr@rFr*r*r+r/fs    zDownsample.forward)r0r1r2r r!rr/r5r*r*r(r+rLRsrLcs,eZdZdfdd ZedddZZS) ChannelAttentionFcsJt||_||}|d|_tj||d|d|_t|||_dS)Nrbias)r r! num_headsscaler"Linearqkvr$)r'rrRqkv_biashead_dimr(r*r+r!qs   zChannelAttention.__init__r,c Cs|j\}}}||||d|j||jddddd}|d\}}}||j}|dd|} | jdd} | |dddd}|dd|||}| |}|S) Nrrrrr8r) rCrUreshaperRpermuteunbindrS transposesoftmaxr$) r'r-rGNrHrUqrvZ attentionr*r*r+r/zs .   zChannelAttention.forward)rNF)r0r1r2r!rr/r5r*r*r(r+rMos rMcs>eZdZdddejejddffdd Zeddd ZZ S) ChannelBlock@FTc stt|d| d|_||_|||_t|||d|_|dkrJt|nt |_ t|d| d|_ |jr|||_ t|t|||d|_|dkrt|nt |_nd|_ d|_d|_dSNrr)rRrVre)Z in_featuresZhidden_features act_layer)r r!rcpe1ffnnorm1rMattnrr"r& drop_path1cpe2norm2r r3mlp drop_path2) r'rrR mlp_ratiorV drop_pathrgrBricpe_actr(r*r+r!s$    zChannelBlock.__init__r,cCs|j\}}}}||ddd}||}||}|||}||dd||||}|j dk r|ddd}|| | | |}|dd||||}|S)Nrr) rCrhflattenr^rjrkrlrmviewrorprn)r'r-rGrHrIrJcurr*r*r+r/s   zChannelBlock.forward) r0r1r2r"r% LayerNormr!rr/r5r*r*r(r+rcs"rc)r- window_sizecCsj|j\}}}}||||d|d||d|d|}|ddddddd|d|d|}|S)z Args: x: (B, H, W, C) window_size (int): window size Returns: windows: (num_windows*B, window_size, window_size, C) rrrrr8rXrCrur\ contiguous)r-rxrGrIrJrHwindowsr*r*r+window_partitions,,r})r|rxrIrJcCs^|jd}|d||d||d|d|d|}|ddddddd|||}|S)z Args: windows: (num_windows*B, window_size, window_size, C) window_size (int): Window size H (int): Height of image W (int): Width of image Returns: x: (B, H, W, C) rXrrrrr8ryrz)r|rxrIrJrHr-r*r*r+window_reverses ,$r~csBeZdZUdZejjeed<d fdd Z e dddZ Z S) WindowAttentiona Window based multi-head self attention (W-MSA) module with relative position bias. It supports both of shifted and non-shifted window. Args: dim (int): Number of input channels. window_size (tuple[int]): The height and width of the window. num_heads (int): Number of attention heads. qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True fused_attnTcslt||_||_||_||}|d|_t|_tj ||d|d|_ t |||_ tj dd|_ dS)NrOrrPrXrZ)r r!rrxrRrSrrr"rTrUr$ZSoftmaxr_)r'rrxrRrVrWr(r*r+r!s  zWindowAttention.__init__r,c Cs|j\}}}||||d|j||jddddd}|d\}}}|jr`t|||}n,||j }|| dd} | | } | |}| dd|||}| |}|S)Nrrrrr8rYrX) rCrUr[rRr\r]rrDZscaled_dot_product_attentionrSr^r_r$) r'r-ZB_r`rHrUrarrbrkr*r*r+r/s .   zWindowAttention.forward)T) r0r1r2rKtorchjitFinalr4__annotations__r!rr/r5r*r*r(r+rs rcsDeZdZdZddddejejddffdd Zed d d Z Z S) SpatialBlocka< Windows Block. Args: dim (int): Number of input channels. num_heads (int): Number of attention heads. window_size (int): Window size. mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True drop_path (float, optional): Stochastic depth rate. Default: 0.0 act_layer (nn.Module, optional): Activation layer. Default: nn.GELU norm_layer (nn.Module, optional): Normalization layer. Default: nn.LayerNorm r9rdTreFc st||_| |_||_t||_||_t|d| d|_ |||_ t ||j||d|_ |dkrjt |nt|_t|d| d|_|jr|||_t||} t|| |d|_|dkrt |nt|_nd|_d|_d|_dSrf)r r!rrirRr rxrqrrhrjrrkrr"r&rlrmrnr3r rorp) r'rrRrxrqrVrrrgrBrirsZmlp_hidden_dimr(r*r+r!s8     zSpatialBlock.__init__r,c Cs|j\}}}}||ddd}||}|||||}d}}|jd||jd|jd} |jd||jd|jd} t|dd|| || f}|j\} } } } t ||j}|d|jd|jd|}| |}|d|jd|jd|}t ||j| | }|ddd|d|ddf }|||||}|| |}||dd||||}|jdk r|ddd}|||||}|dd||||}|S)NrrrrX)rCrhrtr^rjrurxrDrEr}rkr~r{rlrmrorprn)r'r-rGrHrIrJZshortcutZpad_lZpad_tZpad_rZpad_b_ZHpZWpZ x_windowsZ attn_windowsr*r*r+r/Es. ""  $ zSpatialBlock.forward) r0r1r2rKr"r%rwr!rr/r5r*r*r(r+r s-rc sXeZdZddddddddeejddf fd d Zejj dd d Z e d ddZ Z S) DaVitStagerTspatialchannelrr9r8)rrFcstd|_|r&t||| d|_n t|_g}t|D]}g}t|D]`\}}|dkr| t |||| | || | ||d qL|dkrL| t |||| | || | |dqL| tj |qdepthr attn_typesrRrxrqrVdrop_path_ratesrB norm_layer_clrirsZ stage_blocksZ block_idxZdual_attention_blockZattn_idxZ attn_typer(r*r+r!nsD     zDaVitStage.__init__cCs ||_dSr.)r)r'enabler*r*r+set_grad_checkpointingsz!DaVitStage.set_grad_checkpointingr,cCs6||}|jr(tjs(t|j|}n ||}|Sr.)rrrr is_scriptingrrr'r-r*r*r+r/s   zDaVitStage.forward)T)r0r1r2r r"rwr!rrignorerrr/r5r*r*r(r+rms @ rcseZdZdZd%fdd ZddZejjd&ddZ ejjd'ddZ ejjddZ d(ddZ ddZ d)ed d!d"Zd#d$ZZS)*ra DaViT A PyTorch implementation of `DaViT: Dual Attention Vision Transformers` - https://arxiv.org/abs/2204.03645 Supports arbitrary input sizes and pyramid feature extraction Args: in_chans (int): Number of input image channels. Default: 3 num_classes (int): Number of classes for classification head. Default: 1000 depths (tuple(int)): Number of blocks in each stage. Default: (1, 1, 3, 1) embed_dims (tuple(int)): Patch embedding dimension. Default: (96, 192, 384, 768) num_heads (tuple(int)): Number of attention heads in different layers. Default: (3, 6, 12, 24) window_size (int): Window size. Default: 7 mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. Default: 4 qkv_bias (bool): If True, add a learnable bias to query, key, value. Default: True drop_path_rate (float): Stochastic depth rate. Default: 0.1 norm_layer (nn.Module): Normalization layer. Default: nn.LayerNorm. rrrrrr7r r9r8T layernorm2d layernormh㈵>rFreavgcstt|}|t|kr.t|ks4nttt|| d}tt| | d} ||_|d|_||_d|_ g|_ t ||d|d|_ |d}ddt d|t||D}g}t|D]n}||}t|||||dk| ||||||||| | | d}|}|||j t|d d |d g7_ qtj||_|rf||j|_t|j|||jd |_n"t|_t|j|||j|d |_||jdS)N)epsrXFrrcSsg|] }|qSr*)tolist).0r-r*r*r+ sz"DaVit.__init__..) rrrrRrxrqrVrrBrrirsrzstages.)Znum_chsZ reductionmodule) pool_type drop_rate)rrrB)r r!lenr?rr num_classesZ num_featuresrrZ feature_infor6stemrZlinspacesumsplitrrrdictr"rstagesnorm_prerheadr&rapply _init_weights)r'Zin_chansdepths embed_dimsrRrxrqrVrBrZnorm_epsrrirsrZdrop_path_rater global_poolZhead_norm_firstZ num_stagesr=ZdprrZ stage_idxr>stager(r*r+r!sj " "  "    zDaVit.__init__cCsDt|tjr@t|jddt|tjr@|jdk r@tj|jddS)Ng{Gz?)stdr) isinstancer"rTr weightrQinitZ constant_)r'mr*r*r+r s zDaVit._init_weightscCstd|r dndddgdS)Nz^stemz^stages\.(\d+))z^stages\.(\d+).downsample)r)z^stages\.(\d+)\.blocks\.(\d+)N)z ^norm_pre)i)rr)r)r'Zcoarser*r*r+ group_matcher&szDaVit.group_matchercCs"||_|jD]}|j|dq dS)N)r)rrr)r'rrr*r*r+r1s zDaVit.set_grad_checkpointingcCs|jjSr.)rfc)r'r*r*r+get_classifier7szDaVit.get_classifierNcCs|j||dSr.)rreset)r'rrr*r*r+reset_classifier;szDaVit.reset_classifiercCs@||}|jr(tjs(t|j|}n ||}||}|Sr.)rrrrrrrrrr*r*r+forward_features>s    zDaVit.forward_features pre_logitscCs|r|j|ddS||S)NTr)r)r'r-rr*r*r+ forward_headGszDaVit.forward_headcCs||}||}|Sr.)rrrr*r*r+r/Js  z DaVit.forward)rrrrr9r8TrrrrTFrererrF)F)T)N)F)r0r1r2rKr!rrrrrrrrrr4rr/r5r*r*r(r+rs>S     cCsd|kr |Sd|kr|d}ddl}i}|D]t\}}|dd|}|dd|}|d d }|d d }|d d}|dd}|dd}|dd}|||<q0|S)z Remap MSFT checkpoints -> timm zhead.fc.weight state_dictrNzpatch_embeds.([0-9]+)zstages.\1.downsamplezmain_blocks.([0-9]+)zstages.\1.blockszdownsample.projzdownsample.convzstages.0.downsamplerzhead.zhead.fc.znorms.z head.norm.zcpe.0rhzcpe.1rm)reitemssubreplace)rmodelrZout_dictrrbr*r*r+checkpoint_filter_fnPs"       rFcKsPtddt|ddD}|d|}tt||fttd|dd|}|S) Ncss|]\}}|VqdSr.r*)rirr*r*r+ hsz _create_davit..rr out_indicesT)Zflatten_sequentialr)Zpretrained_filter_fnZ feature_cfg)tuplergetpoprrrr)variant pretrainedkwargsZdefault_out_indicesrrr*r*r+ _create_davitgs  rc Ks|dddddttddd |S) Nr)rr)r9r9gffffff?Zbicubicz stem.convzhead.fc) urlrZ input_sizeZ pool_sizeZcrop_pct interpolationZmeanrZ first_conv classifierr)rrr*r*r+_cfgvsrztimm/)Z hf_hub_id)zdavit_tiny.msft_in1kzdavit_small.msft_in1kzdavit_base.msft_in1k davit_large davit_huge davit_giant)returncKs(tdddd}tdd|it|f|S)NrrrrrrR davit_tinyr)rrrrrZ model_argsr*r*r+rsrcKs(tdddd}tdd|it|f|S)Nrr rrrr davit_smallr)rrrr*r*r+rsrcKs(tdddd}tdd|it|f|S)Nr))r8rN r davit_baser)rrrr*r*r+rsrcKs(tdddd}tdd|it|f|S)Nr)rrr)rrr0rrr)rrrr*r*r+rsrcKs(tdddd}tdd|it|f|S)Nr)rrri)rNrr@rrr)rrrr*r*r+rsrcKs(tdddd}tdd|it|f|S)N)rrrr)rrri )rrrr7rrr)rrrr*r*r+rsr)F)r)F)F)F)F)F)F):rK functoolsrtypingrrZtorch.nnr"Ztorch.nn.functionalZ functionalrDrZ timm.datarrZ timm.layersrr r r r r rrrZ_builderrZ _features_fxrZ _manipulater _registryrr__all__Modulerr6rLrMrcr3r}r~rrrrrrrZ default_cfgsrrrrrrr*r*r*r+sp    $    $7,bN