U ʹj@sdZddlZddlmZddlmZd2edddZGd d d ejZ d3edd d Z Gd ddejZ d4edddZ GdddejZ d5edddZGdddejZd6edddZGdddejZd7edddZGdddejZd8eddd ZGd!d"d"ejZGd#d$d$ejZd9ejeejd%d&d'ZGd(d)d)ejZd:ejeejd%d*d+ZGd,d-d-ejZd;ejeejd%d.d/ZGd0d1d1ejZdS)|r |d|djdddSd||djdddSdS)z Hard Mish Experimental, based on notes by Mish author Diganta Misra at https://github.com/digantamisra98/H-Mish/blob/0da20d4bc58e696b6803f2523c58d3c8a82782d0/README.md g?r)minmaxN)rclampr r r r hard_mishhsr4cs,eZdZdedfdd ZddZZS)HardMishFrcstt|||_dSr)rr5rrrrr r rtszHardMish.__init__cCs t||jSr)r4rrr r r rxszHardMish.forward)Frr r rr r5ssr5csBeZdZdZd eeeddfdd Zej ej d d d Z Z S) PReLUz)Applies PReLU (w/ dummy inplace arg) ?FN)num_parametersinitrreturncstt|j||ddS)N)r9r:)rr6r)rr9r:rrr r rszPReLU.__init__inputr;cCst||jSr)rZpreluweightrr=r r r rsz PReLU.forward)r7r8F) rrrr"intfloatrrtorchTensorrrr r rr r6|sr6)r rr;cCs t|Srrgelur r r r rEsrEcs<eZdZdZd edfdd ZejejdddZZ S) GELULApplies the Gaussian Error Linear Units function (w/ dummy inplace arg) Frcstt|dSr)rrFrrrr r rsz GELU.__init__r<cCs t|SrrDr?r r r rsz GELU.forward)F rrrr"rrrBrCrrr r rr rFsrFcCstj|ddSNr)Z approximaterDr r r r gelu_tanhsrJcs<eZdZdZd edfdd ZejejdddZZ S) GELUTanhrGFrcstt|dSr)rrKrrrr r rszGELUTanh.__init__r<cCstj|ddSrIrDr?r r r rszGELUTanh.forward)FrHr r rr rKsrKcCs|td|S)NgZd;?)rBrr r r r quick_gelusrLcs<eZdZdZd edfdd ZejejdddZZ S) QuickGELUrGFrcstt|dSr)rrMrrrr r rszQuickGELU.__init__r<cCst|Sr)rLr?r r r rszQuickGELU.forward)FrHr r rr rMsrM)F)F)F)F)F)F)F)F)F)F)r"rBrZtorch.nnrrrr Modulerr r!rr$rr&r,r-r.r/r4r5r6rCrErFrJrKrLrMr r r r s0