U ʹj@sdZddlZddlmZddlmmZddlmZm Z m Z Gdddej Z Gdddej Z Gd d d ej ZGd d d ej ZdS) a Convolution with Weight Standardization (StdConv and ScaledStdConv) StdConv: @article{weightstandardization, author = {Siyuan Qiao and Huiyu Wang and Chenxi Liu and Wei Shen and Alan Yuille}, title = {Weight Standardization}, journal = {arXiv preprint arXiv:1903.10520}, year = {2019}, } Code: https://github.com/joe-siyuan-qiao/WeightStandardization ScaledStdConv: Paper: `Characterizing signal propagation to close the performance gap in unnormalized ResNets` - https://arxiv.org/abs/2101.08692 Official Deepmind JAX code: https://github.com/deepmind/deepmind-research/tree/master/nfnets Hacked together by / copyright Ross Wightman, 2021. N) get_paddingget_padding_valuepad_samecs*eZdZdZd fdd Zdd ZZS) StdConv2dzConv2d with Weight Standardization. Used for BiT ResNet-V2 models. Paper: `Micro-Batch Training with Batch-Channel Normalization and Weight Standardization` - https://arxiv.org/abs/1903.10520v2 rNFư>c s:|dkrt|||}tj||||||||d| |_dS)Nstridepaddingdilationgroupsbias)rsuper__init__eps) self in_channel out_channels kernel_sizer r r r r r __class__S/var/www/html/bwave/python/venv/lib/python3.8/site-packages/timm/layers/std_conv.pyr s zStdConv2d.__init__c CsRtj|jd|jddddd|jd|j}t|||j|j |j |j |j }|SNrT)trainingmomentumr) F batch_normweightreshaperr reshape_asconv2dr r r r r rxr rrrforward*s zStdConv2d.forward)rNrrFr__name__ __module__ __qualname____doc__rr& __classcell__rrrrrs rcs*eZdZdZd fdd Zdd ZZS) StdConv2dSamezConv2d with Weight Standardization. TF compatible SAME padding. Used for ViT Hybrid model. Paper: `Micro-Batch Training with Batch-Channel Normalization and Weight Standardization` - https://arxiv.org/abs/1903.10520v2 rSAMEFrc s@t||||d\}} tj||||||||d| |_| |_dS)Nr r r)rrrsame_padr) rrrrr r r r r r is_dynamicrrrr8szStdConv2dSame.__init__c Csl|jrt||j|j|j}tj|jd|j ddddd|j d |j}t |||j |j|j|j|j}|Sr)r0rrr r rrr r!rrr"r#r r r r$rrrr&Bs zStdConv2dSame.forward)rr.rrFrr'rrrrr-2s r-cs*eZdZdZd fdd Zd d ZZS) ScaledStdConv2da4Conv2d layer with Scaled Weight Standardization. Paper: `Characterizing signal propagation to close the performance gap in unnormalized ResNets` - https://arxiv.org/abs/2101.08692 NOTE: the operations used in this impl differ slightly from the DeepMind Haiku impl. The impact is minor. rNT?rc sp|dkrt|||}tj||||||||dtt|jdddf| |_| |j d d|_ | |_ dS)Nrrr) rrrnn Parametertorchfullrgainr numelscaler) r in_channelsrrr r r r r gammar gain_initrrrrUs zScaledStdConv2d.__init__c Cs^tj|jd|jddd|j|jddd|jd |j}t |||j |j |j |j|jSNrrTr)r rrr)rrr r!rr9r;viewrr"r#r r r r r r$rrrr&aszScaledStdConv2d.forward)rNrrTr3rr3r'rrrrr2Ls  r2cs*eZdZdZd fdd Zd d ZZS) ScaledStdConv2dSamea\Conv2d layer with Scaled Weight Standardization and Tensorflow-like SAME padding support Paper: `Characterizing signal propagation to close the performance gap in unnormalized ResNets` - https://arxiv.org/abs/2101.08692 NOTE: the operations used in this impl differ slightly from the DeepMind Haiku impl. The impact is minor. rr.Tr3rc svt||||d\}} tj||||||||dtt|jdddf| |_| |j d d|_ | |_ | |_ dS)Nr/rrrr4)rrrr5r6r7r8rr9r r:r;r0r) rr<rrr r r r r r=rr>r1rrrrrszScaledStdConv2dSame.__init__c Csx|jrt||j|j|j}tj|jd|j ddd|j |j  ddd|j d|j}t|||j|j|j|j|jSr?)r0rrr r rrr r!rr9r;r@rr"r#r r r r$rrrr&~szScaledStdConv2dSame.forward)rr.rrTr3rr3r'rrrrrAis  rA)r+r7Ztorch.nnr5Ztorch.nn.functionalZ functionalrr rrrZConv2drr-r2rArrrrs