U ʹj_O @sddlZddlZddlmZddlZddlmZddZddZdAdd Z d d Z d d Z dBddZ dCee e edddZdDee e edddZdEee e e e eejedddZee edddZeed d!d"Zeed d#d$Zd%d&ZdFd(d)Zd*d+ZdGee ed,d-d.ZdHee ed,d/d0Zd1d2ZdIee eed5d6d7ZdJee eed5d8d9ZdKd:d;ZdLd=d>Zd?d@Z e eZ!e eZ"e eZ#e eZ$e eZ%e eZ&e eZ'e eZ(e eZ)e eZ*e eZ+dS)MN)Tensor)Optionalc Cs,t|||W5QRSQRXdSN)torchno_graduniform_tensorabr L/var/www/html/bwave/python/venv/lib/python3.8/site-packages/torch/nn/init.py_no_grad_uniform_ s rc Cs,t|||W5QRSQRXdSr)rrnormal_r meanstdr r r _no_grad_normal_s rc Csdd}||d|ks(||d|kr6tjdddt||||}||||}|jd|dd|d|d|||td| ||j ||d |W5QRSQRXdS) NcSsdt|tddS)N?@)matherfsqrt)xr r r norm_cdfsz(_no_grad_trunc_normal_..norm_cdfzjmean is more than 2 std from [a, b] in nn.init.trunc_normal_. The distribution of values may be incorrect. stacklevel generatorr)minmax) warningswarnrrrZerfinv_mul_rrZadd_Zclamp_) r rrr r r rlur r r _no_grad_trunc_normal_s    r(c Cs*t||W5QRSQRXdSr)rrZfill_r valr r r _no_grad_fill_9s r+c Cs(t|W5QRSQRXdSr)rrzero_r r r r _no_grad_zero_>s r.cCsdddddddg}||ks"|dkr&d S|d kr2d S|d krDtd S|dkr|dkrZd}n4t|tsnt|tsxt|tr~|}ntd|dtd d |dS|dkrdStd|dS)aReturn the recommended gain value for the given nonlinearity function. The values are as follows: ================= ==================================================== nonlinearity gain ================= ==================================================== Linear / Identity :math:`1` Conv{1,2,3}D :math:`1` Sigmoid :math:`1` Tanh :math:`\frac{5}{3}` ReLU :math:`\sqrt{2}` Leaky Relu :math:`\sqrt{\frac{2}{1 + \text{negative\_slope}^2}}` SELU :math:`\frac{3}{4}` ================= ==================================================== .. warning:: In order to implement `Self-Normalizing Neural Networks`_ , you should use ``nonlinearity='linear'`` instead of ``nonlinearity='selu'``. This gives the initial weights a variance of ``1 / N``, which is necessary to induce a stable fixed point in the forward pass. In contrast, the default gain for ``SELU`` sacrifices the normalization effect for more stable gradient flow in rectangular layers. Args: nonlinearity: the non-linear function (`nn.functional` name) param: optional parameter for the non-linear function Examples: >>> gain = nn.init.calculate_gain('leaky_relu', 0.2) # leaky_relu with negative_slope=0.2 .. _Self-Normalizing Neural Networks: https://papers.nips.cc/paper/2017/hash/5d44ee6f2c3f71b73125876103c8f6c4-Abstract.html ZlinearZconv1dZconv2dZconv3dZconv_transpose1dZconv_transpose2dZconv_transpose3dZsigmoidrtanhg?Zrelur leaky_reluN{Gz?znegative_slope z not a valid numberrZselug?zUnsupported nonlinearity )rr isinstanceboolintfloat ValueError) nonlinearityparamZ linear_fnsZnegative_sloper r r calculate_gainCs"! r9r)r r r returncCs0tj|r$tjjt|f|||dSt|||S)adFills the input Tensor with values drawn from the uniform distribution :math:`\mathcal{U}(a, b)`. Args: tensor: an n-dimensional `torch.Tensor` a: the lower bound of the uniform distribution b: the upper bound of the uniform distribution Examples: >>> w = torch.empty(3, 5) >>> nn.init.uniform_(w) r)r overrideshas_torch_function_variadichandle_torch_functionrrrr r r rzs r)r rrr;cCs0tj|r$tjjt|f|||dSt|||S)azFills the input Tensor with values drawn from the normal distribution :math:`\mathcal{N}(\text{mean}, \text{std}^2)`. Args: tensor: an n-dimensional `torch.Tensor` mean: the mean of the normal distribution std: the standard deviation of the normal distribution Examples: >>> w = torch.empty(3, 5) >>> nn.init.normal_(w) r)rr<r=r>rrrr r r rs rr)r rrr r r r;cCst||||||dS)aFills the input Tensor with values drawn from a truncated normal distribution. The values are effectively drawn from the normal distribution :math:`\mathcal{N}(\text{mean}, \text{std}^2)` with values outside :math:`[a, b]` redrawn until they are within the bounds. The method used for generating the random values works best when :math:`a \leq \text{mean} \leq b`. Args: tensor: an n-dimensional `torch.Tensor` mean: the mean of the normal distribution std: the standard deviation of the normal distribution a: the minimum cutoff value b: the maximum cutoff value Examples: >>> w = torch.empty(3, 5) >>> nn.init.trunc_normal_(w) r)r()r rrr r r r r r trunc_normal_sr@)r r*r;cCs,tj|r"tjjt|f||dSt||S)zFills the input Tensor with the value :math:`\text{val}`. Args: tensor: an n-dimensional `torch.Tensor` val: the value to fill the tensor with Examples: >>> w = torch.empty(3, 5) >>> nn.init.constant_(w, 0.3) r))rr<r=r> constant_r+r)r r r rAs rA)r r;cCs t|dS)zFills the input Tensor with the scalar value `1`. Args: tensor: an n-dimensional `torch.Tensor` Examples: >>> w = torch.empty(3, 5) >>> nn.init.ones_(w) r)r+r-r r r ones_s rBcCst|S)zFills the input Tensor with the scalar value `0`. Args: tensor: an n-dimensional `torch.Tensor` Examples: >>> w = torch.empty(3, 5) >>> nn.init.zeros_(w) )r.r-r r r zeros_s rCc CsB|dkrtdttj|j||jdW5QRX|S)a=Fills the 2-dimensional input `Tensor` with the identity matrix. Preserves the identity of the inputs in `Linear` layers, where as many inputs are preserved as possible. Args: tensor: a 2-dimensional `torch.Tensor` Examples: >>> w = torch.empty(3, 5) >>> nn.init.eye_(w) r,Only tensors with 2 dimensions are supported)out requires_grad) ndimensionr6rreyeshaperFr-r r r eye_s   rJrc Cs&|}|dkrtd|}|d|dkr8td|d|}t||d}t|t|D]}t|D]}|dkrd||||||ddf<qx|dkrd||||||dd|ddf<qxd||||||dd|dd|ddf<qxqlW5QRX|S) aFFills the {3, 4, 5}-dimensional input `Tensor` with the Dirac delta function. Preserves the identity of the inputs in `Convolutional` layers, where as many input channels are preserved as possible. In case of groups>1, each group of channels preserves identity Args: tensor: a {3, 4, 5}-dimensional `torch.Tensor` groups (int, optional): number of groups in the conv layer (default: 1) Examples: >>> w = torch.empty(3, 16, 5, 5) >>> nn.init.dirac_(w) >>> w = torch.empty(3, 24, 5, 5) >>> nn.init.dirac_(w, 3) )z5Only tensors with 3, 4, or 5 dimensions are supportedrz!dim 0 must be divisible by groupsrrKrrL)rGr6sizer!rrr,range)r groups dimensionssizesZout_chans_per_grpZmin_dimgdr r r dirac_s2    "  rUcCsp|}|dkrtd|d}|d}d}|dkrX|jddD] }||9}qJ||}||}||fS)NrzNFan in and fan out can not be computed for tensor with fewer than 2 dimensionsrr)dimr6rNrI)r rQZnum_input_fmapsZnum_output_fmapsZreceptive_field_sizesfan_infan_outr r r _calculate_fan_in_and_fan_out#s    rZ)r gainr;cCsBt|\}}|tdt||}td|}t|| |S)aFills the input `Tensor` with values according to the method described in `Understanding the difficulty of training deep feedforward neural networks` - Glorot, X. & Bengio, Y. (2010), using a uniform distribution. The resulting tensor will have values sampled from :math:`\mathcal{U}(-a, a)` where .. math:: a = \text{gain} \times \sqrt{\frac{6}{\text{fan\_in} + \text{fan\_out}}} Also known as Glorot initialization. Args: tensor: an n-dimensional `torch.Tensor` gain: an optional scaling factor Examples: >>> w = torch.empty(3, 5) >>> nn.init.xavier_uniform_(w, gain=nn.init.calculate_gain('relu')) r@)rZrrr5r)r r[rXrYrr r r r xavier_uniform_6s r]cCs2t|\}}|tdt||}t|d|S)aFills the input `Tensor` with values according to the method described in `Understanding the difficulty of training deep feedforward neural networks` - Glorot, X. & Bengio, Y. (2010), using a normal distribution. The resulting tensor will have values sampled from :math:`\mathcal{N}(0, \text{std}^2)` where .. math:: \text{std} = \text{gain} \times \sqrt{\frac{2}{\text{fan\_in} + \text{fan\_out}}} Also known as Glorot initialization. Args: tensor: an n-dimensional `torch.Tensor` gain: an optional scaling factor Examples: >>> w = torch.empty(3, 5) >>> nn.init.xavier_normal_(w) rr:)rZrrr5r)r r[rXrYrr r r xavier_normal_Qs r^cCsH|}ddg}||kr,td|d|t|\}}|dkrD|S|S)NrXrYzMode z" not supported, please use one of )lowerr6rZ)r modeZ valid_modesrXrYr r r _calculate_correct_fanks  rarXr0r r r`r7c Cstj|r&tjjt|f||||dSd|jkr>td|St||}t ||}|t |}t d|}t | | |W5QRSQRXdS)aFills the input `Tensor` with values according to the method described in `Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification` - He, K. et al. (2015), using a uniform distribution. The resulting tensor will have values sampled from :math:`\mathcal{U}(-\text{bound}, \text{bound})` where .. math:: \text{bound} = \text{gain} \times \sqrt{\frac{3}{\text{fan\_mode}}} Also known as He initialization. Args: tensor: an n-dimensional `torch.Tensor` a: the negative slope of the rectifier used after this layer (only used with ``'leaky_relu'``) mode: either ``'fan_in'`` (default) or ``'fan_out'``. Choosing ``'fan_in'`` preserves the magnitude of the variance of the weights in the forward pass. Choosing ``'fan_out'`` preserves the magnitudes in the backwards pass. nonlinearity: the non-linear function (`nn.functional` name), recommended to use only with ``'relu'`` or ``'leaky_relu'`` (default). Examples: >>> w = torch.empty(3, 5) >>> nn.init.kaiming_uniform_(w, mode='fan_in', nonlinearity='relu') rbr,Initializing zero-element tensors is a no-opr\N)rr<r=r>kaiming_uniform_rIr#r$rar9rrrr)r r r`r7fanr[rboundr r r rdus$      rdc Csfd|jkrtd|St||}t||}|t|}t| d|W5QRSQRXdS)aFills the input `Tensor` with values according to the method described in `Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification` - He, K. et al. (2015), using a normal distribution. The resulting tensor will have values sampled from :math:`\mathcal{N}(0, \text{std}^2)` where .. math:: \text{std} = \frac{\text{gain}}{\sqrt{\text{fan\_mode}}} Also known as He initialization. Args: tensor: an n-dimensional `torch.Tensor` a: the negative slope of the rectifier used after this layer (only used with ``'leaky_relu'``) mode: either ``'fan_in'`` (default) or ``'fan_out'``. Choosing ``'fan_in'`` preserves the magnitude of the variance of the weights in the forward pass. Choosing ``'fan_out'`` preserves the magnitudes in the backwards pass. nonlinearity: the non-linear function (`nn.functional` name), recommended to use only with ``'relu'`` or ``'leaky_relu'`` (default). Examples: >>> w = torch.empty(3, 5) >>> nn.init.kaiming_normal_(w, mode='fan_out', nonlinearity='relu') rrcN) rIr#r$rar9rrrrr)r r r`r7rer[rr r r kaiming_normal_s     rgc Cs|dkrtd|dkr$|S|d}||}|||dd}||kr^|tj |\}}t |d}| }||9}||kr|t | ||||W5QRX|S)a]Fills the input `Tensor` with a (semi) orthogonal matrix, as described in `Exact solutions to the nonlinear dynamics of learning in deep linear neural networks` - Saxe, A. et al. (2013). The input tensor must have at least 2 dimensions, and for tensors with more than 2 dimensions the trailing dimensions are flattened. Args: tensor: an n-dimensional `torch.Tensor`, where :math:`n \geq 2` gain: optional scaling factor Examples: >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_LAPACK) >>> w = torch.empty(3, 5) >>> nn.init.orthogonal_(w) rz4Only tensors with 2 or more dimensions are supportedrr)rGr6ZnumelrNnewrZt_rZlinalgZqrZdiagsignrZview_asZcopy_r%) r r[rowscolsZ flattenedqrrTphr r r orthogonal_s&      ror1c Cs|dkrtd|j\}}tt||}tB|d|t |D]&}t |}|d|}d|||f<qNW5QRX|S)aNFills the 2D input `Tensor` as a sparse matrix, where the non-zero elements will be drawn from the normal distribution :math:`\mathcal{N}(0, 0.01)`, as described in `Deep learning via Hessian-free optimization` - Martens, J. (2010). Args: tensor: an n-dimensional `torch.Tensor` sparsity: The fraction of elements in each column to be set to zero std: the standard deviation of the normal distribution used to generate the non-zero values Examples: >>> w = torch.empty(3, 5) >>> nn.init.sparse_(w, sparsity=0.1) rrDrN) rGr6rIr4rceilrrrrOZrandperm) r ZsparsityrrjrkZ num_zerosZcol_idxZ row_indicesZ zero_indicesr r r sparse_s       rqcsFjddfdd}dddd|_|_|S)Ncs&tjddddd||S)Nznn.init.z' is now deprecated in favor of nn.init..rr)r#r$)argskwargsmethnew_nameZold_namer r deprecated_initsz(_make_deprecate..deprecated_initz z_(...) .. warning:: This method is now deprecated in favor of :func:`torch.nn.init.z"`. See :func:`~torch.nn.init.z` for details.)__name____doc__)rwryr rvr _make_deprecates  r|)N)N)r:r)r:r)r:rr?rN)r)r)r)rrXr0)rrXr0)r)r1),rr#rrtypingrZ _Optionalrrr(r+r.r9r5rr Generatorr@rArBrCrJrUrZr]r^rastrrdrgrorqr|uniformnormalZconstantrHZdiracZxavier_uniformZ xavier_normalZkaiming_uniformZkaiming_normalZ orthogonalsparser r r r s   # 7    +  2 ' -