U ʹj=@sdZddlZddlmZddlmmZddlmZde e e e e e d d d Z dej e e e e e d d dZGdddejZde e e dddZGdddejZdS)a> DropBlock, DropPath PyTorch implementations of DropBlock and DropPath (Stochastic Depth) regularization layers. Papers: DropBlock: A regularization method for convolutional networks (https://arxiv.org/abs/1810.12890) Deep Networks with Stochastic Depth (https://arxiv.org/abs/1603.09382) Code: DropBlock impl inspired by two Tensorflow impl that I liked: - https://github.com/tensorflow/tpu/blob/master/models/official/resnet/resnet_model.py#L74 - https://github.com/clovaai/assembled-cnn/blob/master/nets/blocks.py Hacked together by / Copyright 2020 Ross Wightman N)ndgrid皙??F) drop_prob block_size gamma_scale with_noiseinplace batchwisecCs|j\}}} } | | } t|t| | } ||| | d| |d| |d} ttj| |jdtj| |jd\}}|| dk|| | ddk@|| dk|| | ddk@@}t|dd| | fj|jd}|rtj d|| | f|j|jd}n t |}d| ||dkj|jd}t j | | d| dd }|r|rbtj d|| | f|j|jdnt|}|r|||d|n|||d|}nJ||jtjdd|j}|r|||n |||}|S)a  DropBlock. See https://arxiv.org/pdf/1810.12890.pdf DropBlock with an experimental gaussian noise option. This layer has been tested on a few training runs with success, but needs further validation and possibly optimization for lower runtime impact. r)devicedtype)rrZ kernel_sizeZstridepaddinggHz>)shapeminrtorchZarangerZreshapetorZrandZ rand_likeF max_pool2dZrandnZ randn_likemul_add_numelfloat32sumadd)xrrr r r r BCHW total_sizeclipped_block_sizegammaZw_iZh_iZ valid_blockZ uniform_noise block_mask normal_noisenormalize_scaler*O/var/www/html/bwave/python/venv/lib/python3.8/site-packages/timm/layers/drop.py drop_block_2ds>& ,( r,)rrrr r r cCs|j\}}}} | |} t|t| |} ||| | d| |d||d} t|| } tj| |j| d| dd} |rt| }|r| d|  || n|d| || }nTd| } | | jtj ddj|jd}|r| | |n || |}|S)z DropBlock. See https://arxiv.org/pdf/1810.12890.pdf DropBlock with an experimental gaussian noise option. Simplied from above without concern for valid block mask at edges. r rrrrgư>)rrrZ empty_like bernoulli_rrrrZnormal_rrrrrr)rrrr r r r r!r"r#r$r%r&r'r(r)r*r*r+drop_block_fast_2dNs0  * r.c s<eZdZdZd eeeeeeedfdd Zd d ZZ S) DropBlock2dz9 DropBlock. See https://arxiv.org/pdf/1810.12890.pdf rrrFT)rrr r r r fastcs<tt|||_||_||_||_||_||_||_ dSN) superr/__init__rr rr r r r0)selfrrr r r r r0 __class__r*r+r3ys zDropBlock2d.__init__cCsV|jr |js|S|jr2t||j|j|j|j|jSt||j|j|j|j|j|j SdSr1) trainingrr0r.rr r r r,r r4rr*r*r+forwards( zDropBlock2d.forward)rrrFFFT) __name__ __module__ __qualname____doc__floatintboolr3r9 __classcell__r*r*r5r+r/us$r/T)rr7 scale_by_keepcCs`|dks |s|Sd|}|jdfd|jd}|||}|dkrX|rX||||S)a(Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use 'survival rate' as the argument. rBrr)r)rndimZ new_emptyr-Zdiv_)rrr7rCZ keep_probrZ random_tensorr*r*r+ drop_paths   rEcs:eZdZdZd eedfdd ZddZd d ZZ S) DropPathz^Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). rBT)rrCcstt|||_||_dSr1)r2rFr3rrC)r4rrCr5r*r+r3szDropPath.__init__cCst||j|j|jSr1)rErr7rCr8r*r*r+r9szDropPath.forwardcCsdt|jddS)Nz drop_prob=z0.3f)roundr)r4r*r*r+ extra_reprszDropPath.extra_repr)rBT) r:r;r<r=r>r@r3r9rIrAr*r*r5r+rFsrF)rrrFFF)rrrFF)rBFT)r=rZtorch.nnnnZtorch.nn.functionalZ functionalrgridrr>r?r@r,ZTensorr.Moduler/rErFr*r*r*r+sD   8 '!