U ʹj@sdZddlZddlmZddlZddlmZddlmmZ ddl m Z m Z m Z mZGdddejZGdd d ejZGd d d ejZGd d d ejZejedddZejjejejejedddZejejejedddZGdddejZGdddejZdS)z Normalization layers and wrappers Norm layer definitions that support fast norm and consistent channel arg order (always first arg). Hacked together by / Copyright 2022 Ross Wightman N)Tuple) is_fast_normfast_group_normfast_layer_norm fast_rms_normcs&eZdZdfdd ZddZZS) GroupNorm h㈵>Tcs tj||||dt|_dS)N)epsaffinesuper__init__r fast_norm)self num_channels num_groupsr r  __class__O/var/www/html/bwave/python/venv/lib/python3.8/site-packages/timm/layers/norm.pyrszGroupNorm.__init__cCs<|jrt||j|j|j|jSt||j|j|j|jSdSNrrrweightbiasr FZ group_normrxrrrforwardszGroupNorm.forward)r r T)__name__ __module__ __qualname__rr __classcell__rrrrrsrcs4eZdZdZfddZejejdddZZS) GroupNorm1zL Group Normalization with 1 group. Input: tensor in shape [B, C, *] c stjd|f|t|_dS)Nrr )rrkwargsrrrr#szGroupNorm1.__init__rreturncCs<|jrt||j|j|j|jSt||j|j|j|jSdSrrrrrrr'szGroupNorm1.forward r r!r"__doc__rtorchTensorrr#rrrrr$s r$cs6eZdZdZd fdd ZejejdddZZS) LayerNormz# LayerNorm w/ fast norm option ư>Tcstj|||dt|_dSN)r elementwise_affinerrr _fast_normrrr r rrrr1szLayerNorm.__init__r&cCs>|jr t||j|j|j|j}nt||j|j|j|j}|Sr)r1rnormalized_shaperrr r layer_normrrrrr5szLayerNorm.forward)r-Tr(rrrrr,.sr,cs6eZdZdZd fdd ZejejdddZZS) LayerNorm2dz5 LayerNorm for channels of '2D' spatial NCHW tensors r-Tcstj|||dt|_dSr.r0r2rrrr?szLayerNorm2d.__init__r&cCs^|dddd}|jr0t||j|j|j|j}nt||j|j|j|j}|dddd}|SNrr) permuter1rr3rrr rr4rrrrrCs zLayerNorm2d.forward)r-Tr(rrrrr5=sr5)tensorr'cCs$tjr|S|jtjdSdS)N)Z memory_format)r*jitZ is_scriptingZ is_contiguousZcontiguous_format)r:rrr_is_contiguousMs r<)rrrr cCsXtj|dddd\}}||t||}||ddddf|ddddf}|S)NrFT)dimZunbiasedkeepdim)r*Zvar_meanrsqrt)rrrr surrr_layer_norm_cfUs(rBcCsl|jddd}||jddd||d}||t||}||dddd|dddd}|S)NrT)r=r>r)Zmeanclampr*r?view)rrrr rAr@rrr_layer_norm_cf_sqm]s  $rFcs2eZdZdZdfdd ZejdddZZS) LayerNormExp2da_ LayerNorm for channels_first tensors with 2d spatial dimensions (ie N, C, H, W). Experimental implementation w/ manual norm for tensors non-contiguous tensors. This improves throughput in some scenarios (tested on Ampere GPU), esp w/ channels_last layout. However, benefits are not always clear and can perform worse on other GPUs. r-cstj||ddS)N)r )rr)rrr rrrrnszLayerNormExp2d.__init__r'cCsTt|rs