U ʹjn@sddlZddlZddlmZmZddlmZddlmZm Z m Z m Z m Z m Z ddlZddlmZddgZGd d d ZGd ddeZd d ZGdddZdS)N)abc defaultdict)Enum)AnycastDictListOptionalTuple)amp_definitely_not_availableOptState GradScalerc@s2eZdZdZejddddZejdddZdS) _MultiDeviceReplicatorz_ Lazily serves copies of a tensor to requested devices. Copies are cached per-device. N) master_tensorreturncCs&|js|jjdkst||_i|_dS)Nxla)is_cudadevicetypeAssertionErrormaster_per_device_tensors)selfrrY/var/www/html/bwave/python/venv/lib/python3.8/site-packages/torch/cuda/amp/grad_scaler.py__init__sz_MultiDeviceReplicator.__init__rcCs6|j|d}|dkr2|jj|ddd}||j|<|S)NT)r non_blockingcopy)rgetrto)rrretvalrrrr s  z_MultiDeviceReplicator.get)__name__ __module__ __qualname____doc__torchTensorrr rrrrrsrc@seZdZdZdZdZdS)r rr N)r#r$r%READYUNSCALEDSTEPPEDrrrrr %scCs tjidS)N)stagefound_inf_per_device)r r*rrrr_refresh_per_optimizer_state+sr/c@s eZdZUeejed<eejed<eeee e ffed<d=d d Z e ejejfd d d Z ddZddZddZddZddZddZd>ddZddZdd Zd!d"Zd#d$Zd%d&Zd'd(Zd)d*Zd+d,Zd-d.Zd/d0Zd1d2Zd3d4Z d5d6Z!d7d8Z"d9d:Z#d;d<Z$dS)?r_scaleZ_grows_tracker_per_optimizer_states@@?TcCs|rtrtdd|_n||_|jr||dks8td|dksHtd||_d|_||_||_||_ d|_ d|_ t t |_dS)NzLtorch.cuda.amp.GradScaler is enabled, but CUDA is not available. Disabling.F?z The growth factor must be > 1.0.z!The backoff factor must be < 1.0.r)r warningswarn_enabledr _init_scaler0_growth_factor_backoff_factor_growth_interval_init_growth_tracker_growth_trackerrr/r1)rZ init_scale growth_factorbackoff_factorgrowth_intervalZenabledrrrrts" zGradScaler.__init__rcCsLd}|jdk s"td|d||jdk s@td|d||j|jfS)NzaThis may indicate your script did not use scaler.scale(loss or outputs) earlier in the iteration.z Attempted z but _scale is None. z but _growth_tracker is None. )r0rr?)rfuncnameZfixrrr_check_scale_growth_trackers  z&GradScaler._check_scale_growth_trackercCsF|jdkstdtjd|jtj|d|_tjd|jtj|d|_dS)Nz)_growth_tracker initialized before _scalerdtyper) r?rr'fullr:float32r0r>Zint32)rdevrrr_lazy_init_scale_growth_trackersz*GradScaler._lazy_init_scale_growth_trackercsjs |St|tjrf|js,|jjdks,tjdkrB |jjdk sPt|jj |jddSgfdd|S)a2 Multiplies ('scales') a tensor or list of tensors by the scale factor. Returns scaled outputs. If this instance of :class:`GradScaler` is not enabled, outputs are returned unmodified. Args: outputs (Tensor or iterable of Tensors): Outputs to scale. rNTrrcst|tjrv|js"|jjdks"ttdkrbjdkrD |jjdk sRt t j|d |jSt|t jrt|}t|ttfrt||S|SntddS)Nrrz2outputs must be a Tensor or an iterable of Tensors) isinstancer'r(rrrrlenr0rJappendrr rIterablemaplisttuple ValueError)valiterable apply_scalerstashrrrWs       z%GradScaler.scale..apply_scale) r9rLr'r(rrrrr0rJr!)routputsrrVrscales    zGradScaler.scalec Cst|}t|}tdd}t|jD]}|dD]t} | jdkrHq8|sb| jjtjkrbtd| jj r| jjtjkr| j | _| j } n| j} || j | j | q8q,|D]0\} } | D]} t| || || qqW5QRX|jS)NcSsttSN)rrQrrrrz,GradScaler._unscale_grads_..paramsz%Attempting to unscale FP16 gradients.)rrr'Zno_gradZ param_groupsZgradrFZfloat16rSZ is_sparseZcoalesceZ_valuesrrNitemsvaluesZ*_amp_foreach_non_finite_check_and_unscale_r r)r optimizer inv_scale found_infZ allow_fp16Zper_device_inv_scaleZper_device_found_infZper_device_and_dtype_gradsgroupparamZ to_unscalerZper_dtype_gradsZgradsrrr_unscale_grads_s8        zGradScaler._unscale_grads_cCs|js dS|d|jt|}|dtjkr:tdn|dtjkrPtd|jdk s^t |j }t jddt j|jjd}||||d |d <tj|d<dS) as Divides ("unscales") the optimizer's gradient tensors by the scale factor. :meth:`unscale_` is optional, serving cases where you need to :ref:`modify or inspect gradients` between the backward pass(es) and :meth:`step`. If :meth:`unscale_` is not called explicitly, gradients will be unscaled automatically during :meth:`step`. Simple example, using :meth:`unscale_` to enable clipping of unscaled gradients:: ... scaler.scale(loss).backward() scaler.unscale_(optimizer) torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm) scaler.step(optimizer) scaler.update() Args: optimizer (torch.optim.Optimizer): Optimizer that owns the gradients to be unscaled. .. note:: :meth:`unscale_` does not incur a CPU-GPU sync. .. warning:: :meth:`unscale_` should only be called once per optimizer per :meth:`step` call, and only after all gradients for that optimizer's assigned parameters have been accumulated. Calling :meth:`unscale_` twice for a given optimizer between each :meth:`step` triggers a RuntimeError. .. warning:: :meth:`unscale_` may unscale sparse gradients out of place, replacing the ``.grad`` attribute. Nunscale_r-zMunscale_() has already been called on this optimizer since the last update().z(unscale_() is being called after step().rrEFr.)r9rDr1idr r+ RuntimeErrorr,r0rdoubleZ reciprocalfloatr'rGrHrrf)rraoptimizer_staterbrcrrrrgs(  zGradScaler.unscale_cOs.d}tdd|dDs*|j||}|S)Ncss|]}|VqdSr[)item).0vrrr :sz-GradScaler._maybe_opt_step..r.)sumr`step)rrarmargskwargsr"rrr_maybe_opt_step8s zGradScaler._maybe_opt_stepc s|js|j||Sd|kr"td|d|jt|}|dtjkrPtdd}t|dr*|j r*|}dt |jj k}|rt d t|d|ind|dtjkr|||ttjtfd d |d D}|dtjkrdn|_||_|j||}tj|d<|s&|`|`|S|dtjkrD||t|d d ks^td|j||f||}tj|d<|S)a :meth:`step` carries out the following two operations: 1. Internally invokes ``unscale_(optimizer)`` (unless :meth:`unscale_` was explicitly called for ``optimizer`` earlier in the iteration). As part of the :meth:`unscale_`, gradients are checked for infs/NaNs. 2. If no inf/NaN gradients are found, invokes ``optimizer.step()`` using the unscaled gradients. Otherwise, ``optimizer.step()`` is skipped to avoid corrupting the params. ``*args`` and ``**kwargs`` are forwarded to ``optimizer.step()``. Returns the return value of ``optimizer.step(*args, **kwargs)``. Args: optimizer (torch.optim.Optimizer): Optimizer that applies the gradients. args: Any arguments. kwargs: Any keyword arguments. .. warning:: Closure use is not currently supported. closurez@Closure use is not currently supported if GradScaler is enabled.rsr-z7step() has already been called since the last update().N_step_supports_amp_scalingZ grad_scalerzGradScaler is going to stop passing itself as a keyword argument to the passed optimizer. In the near future GradScaler registers `grad_scale: Tensor` and `found_inf: Tensor` to the passed optimizer and let the optimizer use them directly.csg|]}|jjddqS)T)r)r!r)rotZscalerrr sz#GradScaler.step..r.rz/No inf checks were recorded for this optimizer.) r9rsrjrDr1rir r,hasattrrxinspect signature parametersr7r8 FutureWarningupdater*_check_inf_per_device_get_scale_asyncrr'r(rrr`r+Z grad_scalercrgrMrrv) rrartrurmr"Zkwargs_Zhas_grad_scaler_kwargrcrrzrrs>sn          zGradScaler.stepNcs|js dS|d\}|dk rt|tr8|j|nLd}t|tjjsRt || dksft ||j dksxt ||j |nzfdd|j D}t|dkst d |d}t|dkrtdt|D]}|||7}qt|||j|j|jtt|_ dS) aS Updates the scale factor. If any optimizer steps were skipped the scale is multiplied by ``backoff_factor`` to reduce it. If ``growth_interval`` unskipped iterations occurred consecutively, the scale is multiplied by ``growth_factor`` to increase it. Passing ``new_scale`` sets the new scale value manually. (``new_scale`` is not used directly, it's used to fill GradScaler's internal scale tensor. So if ``new_scale`` was a tensor, later in-place changes to that tensor will not further affect the scale GradScaler uses internally.) Args: new_scale (float or :class:`torch.cuda.FloatTensor`, optional, default=None): New scale factor. .. warning:: :meth:`update` should only be called at the end of the iteration, after ``scaler.step(optimizer)`` has been invoked for all optimizers used this iteration. .. warning:: For performance reasons, we do not check the scale factor value to avoid synchronizations, so the scale factor is not guaranteed to be above 1. If the scale falls below 1 and/or you are seeing NaNs in your gradients or loss, something is likely wrong. For example, bf16-pretrained models are often incompatible with AMP/fp16 due to differing dynamic ranges. Nrz[new_scale should be a float or a 1-element torch.cuda.FloatTensor with requires_grad=False.r Fcs.g|]&}|dD]}|jjddqqS)r.TrK)r`r!r)rostatercr0rrr{sz%GradScaler.update..rz,No inf checks were recorded prior to update.)r9rDrLrlr0fill_r'cudaZ FloatTensorrZnumelZ requires_gradZcopy_r1r`rMrangeZ_amp_update_scale_r;r<r=rr/)rZ new_scaler?reasonZ found_infsZfound_inf_combinedirrrrs8    zGradScaler.updatecCs|jSr[rrrrrrszGradScaler._get_scale_asynccCs*|jr"|jdkr|jS|SdSdS)z Returns a Python float containing the current scale, or 1.0 if scaling is disabled. .. warning:: :meth:`get_scale` incurs a CPU-GPU sync. Nr6)r9r0r:rrnrrrr get_scales  zGradScaler.get_scalecCs|jS)zL Returns a Python float containing the scale growth factor. r;rrrrget_growth_factorszGradScaler.get_growth_factorcCs ||_dS)zd Args: new_scale (float): Value to use as the new scale growth factor. NrrZ new_factorrrrset_growth_factorszGradScaler.set_growth_factorcCs|jS)zM Returns a Python float containing the scale backoff factor. r<rrrrget_backoff_factor szGradScaler.get_backoff_factorcCs ||_dS)ze Args: new_scale (float): Value to use as the new scale backoff factor. Nrrrrrset_backoff_factorszGradScaler.set_backoff_factorcCs|jS)zF Returns a Python int containing the growth interval. r=rrrrget_growth_intervalszGradScaler.get_growth_intervalcCs ||_dS)za Args: new_interval (int): Value to use as the new growth interval. Nr)rZ new_intervalrrrset_growth_intervalszGradScaler.set_growth_intervalcCs(|jr |jdkr|jS|jSdSdS)Nr)r9r?r>rnrrrr_get_growth_tracker%s zGradScaler._get_growth_trackercCs|jS)zM Returns a bool indicating whether this instance is enabled. )r9rrrr is_enabled/szGradScaler.is_enabledcCs(|jr$||j|j|j|dSiS)a Returns the state of the scaler as a :class:`dict`. It contains five entries: * ``"scale"`` - a Python float containing the current scale * ``"growth_factor"`` - a Python float containing the current growth factor * ``"backoff_factor"`` - a Python float containing the current backoff factor * ``"growth_interval"`` - a Python int containing the current growth interval * ``"_growth_tracker"`` - a Python int containing the number of recent consecutive unskipped steps. If this instance is not enabled, returns an empty dict. .. note:: If you wish to checkpoint the scaler's state after a particular iteration, :meth:`state_dict` should be called after :meth:`update`. )rZr@rArBr?)r9rr;r<r=rrrrr state_dict5szGradScaler.state_dictcCs|js dSt|dkrtd|d|_|jdk rB|j|d|d|_|d|_|d|_|d|_ |j dk r|j |ddS) z Loads the scaler state. If this instance is disabled, :meth:`load_state_dict` is a no-op. Args: state_dict(dict): scaler state. Should be an object returned from a call to :meth:`state_dict`. NrzeThe source state dict is empty, possibly because it was saved from a disabled instance of GradScaler.rZr@rArBr?) r9rMrjr:r0rr;r<r=r>r?)rrrrrload_state_dictQs        zGradScaler.load_state_dictcCsR|j}|jrNt|jdks&td||d<||d<d|d<d|d<|S)NrzpA GradScaler instance may only be pickled at the beginning of an iteration, or at the end after scaler.update().r:r>r0r?)__dict__rr9rMr1rrrrrrrr __getstate__ks   zGradScaler.__getstate__cCs|j|dSr[)rrrrrr __setstate__{szGradScaler.__setstate__cCsj|d\}}tjddtj|jd}tjddtj|jd}||||d|jt|d<|jt|dS)Nrrr6rErhTr.)rDr'rGrHrrfr1ri)rrar0_Zdummy_inv_scalercrrrr~s z GradScaler._check_inf_per_devicecCs|jt|dS)Nr.)r1ri)rrarrr_found_inf_per_devicesz GradScaler._found_inf_per_device)r2r3r4r5T)N)%r#r$r%r r'r(__annotations__rintstrrrr rDrJrZrfrgrvrsrrrrrrrrrrrrrrrrrrrrrr/sD D ! .,8h E  )r}r7 collectionsrrenumrtypingrrrrr r r'commonr __all__rr r/rrrrrs