U ʹjG@sddlZddlmZmZmZmZmZmZmZddl Z ddl m Z m Z ddl m Z mZee j ee j fZdddgZdeeeeeee j d d dZdeeeeeee j d d dZdeeeedd ddZdS)N)UnionIterableListDictTupleOptionalcast)Tensorinf)"_group_tensors_by_device_and_dtype_has_foreach_supportclip_grad_norm_clip_grad_normclip_grad_value_@F) parametersmax_norm norm_typeerror_if_nonfiniteforeachreturncst|tjr|g}dd|D}t|}tt|dkrFtdS|djtdd|Dg}tkrfdd|D}t|dkr|dnt t |}ng}| D]p\\} } \\}} |dks|rt || d r| t|q|rtd | jd q| fd d|Dqtjt fd d|D}|rpt||rptdd||d} tj| dd} | D]\\} } \\}} |dks|rt || d rt|| | n>|rtd | jd n$| | } |D]}|| qq|S)aDClips gradient norm of an iterable of parameters. The norm is computed over all gradients together, as if they were concatenated into a single vector. Gradients are modified in-place. Args: parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a single Tensor that will have gradients normalized max_norm (float): max norm of the gradients norm_type (float): type of the used p-norm. Can be ``'inf'`` for infinity norm. error_if_nonfinite (bool): if True, an error is thrown if the total norm of the gradients from :attr:`parameters` is ``nan``, ``inf``, or ``-inf``. Default: False (will switch to True in the future) foreach (bool): use the faster foreach-based implementation. If ``None``, use the foreach implementation for CUDA and CPU native tensors and silently fall back to the slow implementation for other device types. Default: ``None`` Returns: Total norm of the parameter gradients (viewed as a single vector). cSsg|]}|jdk r|jqSNgrad.0prW/var/www/html/bwave/python/venv/lib/python3.8/site-packages/torch/nn/utils/clip_grad.py 's z#clip_grad_norm_..rgcSsg|] }|qSr)detachrgrrrr.scs$g|]}tj|tqSr)torchlinalg vector_normr r tor! first_devicerrr1sNdevice:foreach=True was passed, but can't use the foreach API on  tensorscsg|]}tj|qSr)r#r$r%r!)rrrr;scsg|]}|qSr)r&)rZnormr'rrr=szThe total norm of order z for gradients from `parameters` is non-finite, so it cannot be clipped. To disable this error and scale the gradients by the non-finite norm anyway, set `error_if_nonfinite=False`gư>g?)max) isinstancer#r floatlenZtensorr+r r r.stackitemsr extendZ _foreach_norm RuntimeErrortyper$r% logical_orisnanisinfclampZ _foreach_mul_r&r Zmul_)rrrrrgrads grouped_gradsZnormsZ total_normr+_Z clip_coefZclip_coef_clampedZclip_coef_clamped_devicer"r)r(rrr sH    &"   cCstjdddt|||||S)zClips gradient norm of an iterable of parameters. .. warning:: This method is now deprecated in favor of :func:`torch.nn.utils.clip_grad_norm_`. z[torch.nn.utils.clip_grad_norm is now deprecated in favor of torch.nn.utils.clip_grad_norm_.) stacklevel)warningswarnr )rrrrrrrrrWs )r clip_valuerrc Cst|tjr|g}t|}dd|D}t|g}|D]\\}}\\}}|dksX|rtttt||drt ttt|| t ttt||q:|rt d|j dq:t &|D]}tt|j| |dqW5QRXq:dS)aClips gradient of an iterable of parameters at specified value. Gradients are modified in-place. Args: parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a single Tensor that will have gradients normalized clip_value (float): maximum allowed value of the gradients. The gradients are clipped in the range :math:`\left[\text{-clip\_value}, \text{clip\_value}\right]` foreach (bool): use the faster foreach-based implementation If ``None``, use the foreach implementation for CUDA and CPU native tensors and silently fall back to the slow implementation for other device types. Default: ``None`` cSsg|]}|jdk r|jqSrrrrrrrys z$clip_grad_value_..Nr*r,r-)minr.)r/r#r r0r r3r rrZ_foreach_clamp_min_Z_foreach_clamp_max_r5r6Zno_gradZclamp_)rrBrr;r<r+r=rrrrres  " )rFN)rFN)N)r@typingrrrrrrrr#r r Ztorch.utils._foreach_utilsr r Z_tensor_or_tensors__all__r0boolr rrrrrrs:$  L