U ʹjT @sVdZddlZddlZddlZddlmZddlZddlm m Z ddlm Z ddl m Z mZddlmZmZmZmZmZmZddlmZmZmZmZmZdd lmZdd lmZdd lm Z m!Z!dd l"m#Z#m$Z$m%Z%d gZ&e'e(Z)Gddde j*Z+Gddde j*Z,Gddde j*Z-e.dddZ/ee.dddZ0Gddde j*Z1Gdd d e j*Z2d;e j*e3e4ddd Z5d!d"Z6d#d$Z7de2d-d.d/Z;e#d?e2d-d0d1Ze#dBe2d-d6d7Z?e#dCe2d-d8d9Z@e%e(d5d7d9d:dS)Da Nested Transformer (NesT) in PyTorch A PyTorch implement of Aggregating Nested Transformers as described in: 'Aggregating Nested Transformers' - https://arxiv.org/abs/2105.12723 The official Jax code is released and available at https://github.com/google-research/nested-transformer. The weights have been converted with convert/convert_nest_flax.py Acknowledgments: * The paper authors for sharing their research, code, and model weights * Ross Wightman's existing code off which I based this Copyright 2021 Alexander Soare N)partial)nnIMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD) PatchEmbedMlpDropPathcreate_classifier trunc_normal__assert) create_conv2d create_pool2d to_ntupleuse_fused_attn LayerNorm)build_model_with_cfg)register_notrace_function)checkpoint_seq named_apply)register_modelgenerate_default_cfgsregister_model_deprecationsNestcs<eZdZUdZejjeed<d fdd Z dd Z Z S) Attentionz This is much like `.vision_transformer.Attention` but uses *localised* self attention by accepting an input with an extra "image block" dim fused_attnFcsjt||_||}|d|_t|_tj|d||d|_t ||_ t|||_ t ||_ dS)Ng)bias) super__init__ num_headsscalerrrLinearqkvDropout attn_dropproj proj_drop)selfdimr#qkv_biasr(r*Zhead_dim __class__O/var/www/html/bwave/python/venv/lib/python3.8/site-packages/timm/models/nest.pyr"/s   zAttention.__init__c Cs|j\}}}}|||||d|j||jdddddd}|d\}}} |jrxtj||| |j rn|j j ndd}n8||j }|| d d } | jd d } | | } | | }|ddddd||||}||}||}|S) zm x is shape: B (batch_size), T (image blocks), N (seq length per image block), C (embed dim) rrrr)Z dropout_p)r,)shaper&reshaper#permuteZunbindrFZscaled_dot_product_attentionZtrainingr(pr$ transposeZsoftmaxr)r*) r+xBTNCr&qkvattnr0r0r1forward;s2"     zAttention.forward)rFrr) __name__ __module__ __qualname____doc__torchjitFinalbool__annotations__r"rF __classcell__r0r0r.r1r(s  rcs<eZdZdZdddddejejffdd ZddZZ S) TransformerLayerz This is much like `.vision_transformer.Block` but: - Called TransformerLayer here to allow for "block" as defined in the paper ("non-overlapping image blocks") - Uses modified Attention layer that handles the "block" dimension @Frc snt| ||_t|||||d|_|dkr8t|nt|_| ||_ t ||} t || ||d|_ dS)N)r#r-r(r*r)Z in_featuresZhidden_features act_layerZdrop) r!r"norm1rrEr rIdentity drop_pathnorm2intrmlp) r+r,r# mlp_ratior-r*r(rVrS norm_layerZmlp_hidden_dimr.r0r1r"Zs$    zTransformerLayer.__init__cCs<||}||||}|||||}|SN)rTrVrErYrW)r+r=yr0r0r1rFys zTransformerLayer.forward) rGrHrIrJrGELUrr"rFrPr0r0r.r1rQTs rQcs&eZdZdfdd ZddZZS)ConvPoolcs>tt||d|dd|_|||_tddd|d|_dS)NrT) kernel_sizepaddingr maxr3)raZstriderb)r!r"r convnormrpool)r+Z in_channelsZ out_channelsr[pad_typer.r0r1r"s  zConvPool.__init__cCsjt|jdddkdt|jdddkd||}||dddddddd}||}|S)z: x is expected to have shape (B, C, H, W) r5r3rz1BlockAggregation requires even input spatial dimsr6rr)r r7rdrer9rfr+r=r0r0r1rFs  " zConvPool.forward)r`)rGrHrIr"rFrPr0r0r.r1r_sr_) block_sizecCsv|j\}}}}t||dkdt||dkd||}||}|||||||}|dd|||d|}|S)zimage to blocks Args: x (Tensor): with shape (B, H, W, C) block_size (int): edge length of a single square block in units of H, W rz,`block_size` must divide input height evenlyz+`block_size` must divide input width evenlyr3rr6)r7r r8r<)r=rir>HWrAZ grid_heightZ grid_widthr0r0r1blockifysrlc CsX|j\}}}}tt|}||}}|||||||}|dd||||}|S)zblocks to image Args: x (Tensor): with shape (B, T, N, C) where T is number of blocks and N is sequence size per block block_size (int): edge length of a single square block in units of desired H, W r3r)r7rXmathsqrtr8r<) r=rir>r?_rAZ grid_sizeheightwidthr0r0r1 deblockifys  rrc s<eZdZdZdddddgdddf fdd Zd d ZZS) NestLevelz7 Single hierarchical level of a Nested Transformer NrRTrr`c st||_d|_ttd|||_|dk rJt ||d|_ n t |_ t rpt |kspt dtjf ddt|D|_dS)NFr)r[rgzDMust provide as many drop path rates as there are transformer layersc s*g|]"}t|d qS)) r,r#rZr-r*r(rVr[rS)rQ.0i rSr(rV embed_dimrZr[r#r*r-r0r1 s z&NestLevel.__init__..)r!r"rigrad_checkpointingr ParameterrKZzeros pos_embedr_rfrUlenAssertionError Sequentialrangetransformer_encoder)r+ num_blocksri seq_lengthr#depthrxZprev_embed_dimrZr-r*r(rVr[rSrgr.rwr1r"s   zNestLevel.__init__cCst||}|dddd}t||j}||j}|jrNtjsNt |j |}n | |}t ||j}|ddddS)z+ expects x as (B, C, H, W) rr3rr) rfr9rlrir|rzrKrLZ is_scriptingrrrrrhr0r0r1rFs     zNestLevel.forward)rGrHrIrJr"rFrPr0r0r.r1rss .rscseZdZdZd&fdd Zejjd'ddZejjddZ ejjd(ddZ ejjd)ddZ ejjddZ d*ddZ dd Zd+ed!d"d#Zd$d%ZZS),rz Nested Transformer (NesT) A PyTorch impl of : `Aggregating Nested Transformers` - https://arxiv.org/abs/2105.12723 rr2ir2rr3r3rRTr?Nr`avgcstdD]8}t|}t|tjjrt||kstd|dqt ||}t ||}t ||}||_ |d|_ g|_ |pt }|ptj}| |_||_t|tjjr|d|dkstd|d}||dkstd||_d t|d|_||t|jddks(td t||t|jd|_t||||dd d |_|jj|_|j|jd|_g}d dt d|t!|"|D}d}d }t#t|jD]|}||}|$t%|j||j|j||||||| | | | |||||d|j t&||d|dg7_ |}|d9}qtj'||_(||d|_)t*|j |j |d\}}||_+t,| |_-||_.|/|dS)a Args: img_size (int, tuple): input image size in_chans (int): number of input channels patch_size (int): patch size num_levels (int): number of block hierarchies (T_d in the paper) embed_dims (int, tuple): embedding dimensions of each level num_heads (int, tuple): number of attention heads for each level depths (int, tuple): number of transformer layers for each level num_classes (int): number of classes for classification head mlp_ratio (int): ratio of mlp hidden dim to embedding dim for MLP of transformer layers qkv_bias (bool): enable bias for qkv if True drop_rate (float): dropout rate for MLP of transformer layers, MSA final projection layer, and classifier attn_drop_rate (float): attention dropout rate drop_path_rate (float): stochastic depth rate norm_layer: (nn.Module): normalization layer for transformer layers act_layer: (nn.Module): activation layer in MLP of transformer layers pad_type: str: Type of padding to use '' for PyTorch symmetric, 'same' for TF SAME weight_init: (str): weight init scheme global_pool: (str): type of pooling operation to apply to final feature map Notes: - Default values follow NesT-B from the original Jax code. - `embed_dims`, `num_heads`, `depths` should be ints or tuples with length `num_levels`. - For those following the paper, Table A1 may have errors! - https://github.com/google-research/nested-transformer/issues/2  embed_dimsr#depthsz Require `len(z) == num_levels`r6rrz Model only handles square inputsz*`patch_size` must divide `img_size` evenlyr2zUFirst level blocks don't fit evenly. Check `img_size`, `patch_size`, and `num_levels`F)img_size patch_sizein_chansrxflattencSsg|] }|qSr0)tolist)rur=r0r0r1ry[sz!Nest.__init__..N)rZr-r*r(rVr[rSrgzlevels.)Znum_chsZ reductionmoduler3Z pool_type)0r!r"locals isinstance collectionsabcSequencer}r~r num_classes num_featuresZ feature_inforrr^ drop_rate num_levelsrrKZarangefliprrrmrnrXrir patch_embedZ num_patchesrZlinspacesumsplitrappendrsdictrlevelsrer global_poolr' head_drophead init_weights)r+rrrrrr#rrrZr-rZproj_drop_rateZattn_drop_rateZdrop_path_rater[rSrgZ weight_initr param_nameZ param_valuerZdp_ratesZprev_dimZ curr_stridervr,rr.r0r1r"s1         "    z Nest.__init__cCsZ|dks td|kr"t|j nd}|jD]}t|jddddq,ttt |d|dS) N)nlhbr`rr{Gz?r5r3stdab) head_bias) r~rmlogrrr r|rr_init_nest_weights)r+moderlevelr0r0r1rs   zNest.init_weightscCsddtt|jDS)NcSsh|]}d|dqS)zlevel.z .pos_embedr0rtr0r0r1 sz'Nest.no_weight_decay..)rr}rr+r0r0r1no_weight_decayszNest.no_weight_decayFcCs"td|r dnddfddgd}|S)Nz ^patch_embedz^levels\.(\d+)z*^levels\.(\d+)\.transformer_encoder\.(\d+))z"^levels\.(\d+)\.(?:pool|pos_embed))r)z^norm)i)stemblocks)r)r+ZcoarseZmatcherr0r0r1 group_matcherszNest.group_matchercCs|jD] }||_qdSr\)rrz)r+enablelr0r0r1set_grad_checkpointings zNest.set_grad_checkpointingcCs|jSr\)rrr0r0r1get_classifierszNest.get_classifiercCs$||_t|j|j|d\|_|_dS)Nr)rr rrr)r+rrr0r0r1reset_classifiers zNest.reset_classifiercCs:||}||}||dddddddd}|S)Nrr3rr)rrrer9rhr0r0r1forward_featuress  "zNest.forward_features) pre_logitscCs&||}||}|r|S||Sr\)rrr)r+r=rr0r0r1 forward_heads  zNest.forward_headcCs||}||}|Sr\)rrrhr0r0r1rFs  z Nest.forward)rrr2rrrrrrRTrrrrNNr`r`r)r`)F)T)r)F)rGrHrIrJr"rKrLignorerrrrrrrrNrrFrPr0r0r.r1rsH      r`rrnamercCst|tjrf|dr:t|jddddtj|j|qt|jdddd|jdk rtj |jn6t|tj rt|jdddd|jdk rtj |jdS)zn NesT weight initialization Can replicate Jax implementation. Otherwise follows vision_transformer.py rrr5r3rN) rrr% startswithr weightinitZ constant_r Zzeros_ZConv2drr0r0r1rs     rcCstd|j|j|jd}|jdd\}}tt||}t|tt|dddd}tj |||gddd}t |ddddtt|}|S) z Rescale the grid of position embeddings when loading from state_dict Expected shape of position embeddings is (1, T, N, C), and considers only square images z$Resized position embedding: %s to %sr3rrrbicubicF)sizerZ align_corners) _loggerinfor7rXrmrnrrr9r:Z interpolaterl)ZposembZ posemb_newZseq_length_oldZnum_blocks_newZseq_length_newZsize_newr0r0r1resize_pos_embeds   rcCsNdd|D}|D]2}||jt||jkrt||t||||<q|S)z4 resize positional embeddings of pretrained weights cSsg|]}|dr|qS)Z pos_embed_)r)rurCr0r0r1rys z(checkpoint_filter_fn..)keysr7getattrr)Z state_dictmodelZpos_embed_keysrCr0r0r1checkpoint_filter_fns rFcKs&tt||ftdddtd|}|S)N)rrr3T)Z out_indicesZflatten_sequential)Z feature_cfgZpretrained_filter_fn)rrrr)variant pretrainedkwargsrr0r0r1 _create_nests  rc Ks$|ddddgdddttddd |S) Nr)rrrg?rTzpatch_embed.projr) urlrZ input_sizeZ pool_sizeZcrop_pct interpolationZfixed_input_sizeZmeanrZ first_conv classifierr)rrr0r0r1_cfgsrztimm/)Z hf_hub_id)znest_base.untrainedznest_small.untrainedznest_tiny.untrainedznest_base_jx.goog_in1kznest_small_jx.goog_in1kznest_tiny_jx.goog_in1k)returncKs,tfdddd|}tdd|i|}|S) Nest-B @ 224x224 rrrr nest_baser)rrrrrZ model_kwargsrr0r0r1rsrcKs,tfdddd|}tdd|i|}|S) Nest-S @ 224x224 `ir rr nest_smallr)rrrr0r0r1rsrcKs,tfdddd|}tdd|i|}|S) Nest-T @ 224x224 rrr3r3rr nest_tinyr)rrrr0r0r1rsrcKs8|ddtfdddd|}td d|i|}|S) rrgsamerrrr nest_base_jxr)r setdefaultrrrr0r0r1r$s rcKs8|ddtfdddd|}td d|i|}|S) rrgrrrrr nest_small_jxr)rrrr0r0r1r/s rcKs8|ddtfdddd|}td d|i|}|S) rrgrrrrr nest_tiny_jxr)rrrr0r0r1r9s r)Z jx_nest_baseZ jx_nest_smallZ jx_nest_tiny)r`r)F)r`)F)F)F)F)F)F)ArJcollections.abcrloggingrm functoolsrrKZtorch.nn.functionalrZ functionalr:Z timm.datarrZ timm.layersrrr r r r r rrrrZ_builderrZ _features_fxrZ _manipulaterr _registryrrr__all__ getLoggerrGrModulerrQr_rXrlrrrsrstrfloatrrrrrZ default_cfgsrrrrrrr0r0r0r1sn      ,,BE