U ʹj/ @sddlZddlZddlmZmZmZmZmZmZm Z m Z m Z m Z ddl Z ddlmZddl mZddlmZddlmZddlmZgZeee e edfe edffdd d Zee jeee eefd d d Ze edfe edfe e edfeeeffdddZe deee Z!e de j"eZ#e e!e j$e%ee!dddZ&e e#e j$e%e e#dddZ&ddZ&d8eee%ddddZ'e j"e j(e%dddZ)e j"e%dd d!Z*e d"Z+e d#eee e,e eeZ-e ee j"ge+fe j"e+d$d%d&Z.e ee j"gefe-e-d$d'd&Z.d(d&Z.e edfeeeefe j$e%e e edfe eeefdffd)d*d+Z/d9ej0ee j"eej1d,d-d.Z2d:ej3ej0e4e4eee%dd/d0d1Z5ej0ee j"e4e4dd2d3d4Z6eeefeedd5d6d7Z7dS);N) AnyCallable ContainerDictListOptional OrderedDictTupleTypeVaroverload)nn) _get_stream)_is_namedtuple)PackedSequence.)argskwargsreturncOsBg}t|}|D]\}}||||qt|t|fS)a Turn argument list into separate key list and value list (unpack_kwargs does the opposite) Inspiration: https://github.com/facebookresearch/fairscale/blob/eeb6684/fairscale/internal/containers.py#L70 Usage:: kwarg_keys, flat_args = pack_kwargs(1, 2, a=3, b=4) assert kwarg_keys == ("a", "b") assert flat_args == (1, 2, 3, 4) args, kwargs = unpack_kwargs(kwarg_keys, flat_args) assert args == (1, 2) assert kwargs == {"a": 3, "b": 4} Returns: Tuple[Tuple[Any, ...], Tuple[str, ...]]: The first tuple element gives gives both positional args and kwarg values, where the positional args proceed kwarg values and kwarg values are ordered consistently with the kwarg keys. The second tuple element gives the kwarg keys. The second tuple element's length is at most the first tuple element's length. )listitemsappendtuple)rr kwarg_keys flat_argskvrV/var/www/html/bwave/python/venv/lib/python3.8/site-packages/torch/distributed/utils.py _pack_kwargss   r)dtyperrrcs<dkr||fStjtjdfdd }t||t||fS)z Casts floating point tensors in ``args`` and ``kwargs`` to ``input_dtype``. This respects the existing ``requires_grad`` on the tensors. N)xrcs"t|r|jkr|S|SN)torchZis_floating_pointrto)rrrrcast_fn6sz%_cast_forward_inputs..cast_fn)r!Tensor_apply_to_tensors)rrrr$rr#r_cast_forward_inputs*s r')rrrcCsvt|t|ks,tdt|dt|t|dkr@|ifS|dt| }tt||t| d}||fS)zSee _pack_kwargs.ztoo many keys z vs. rN)lenAssertionErrordictzip)rrrrrrr_unpack_kwargs=s r,ST)inputs target_device!use_side_stream_for_tensor_copiesrcCsdSr rr/r0r1rrr _recursive_toMsr3cCsdSr rr2rrrr3Rscs(fddz |}W5dX|S)z7 Recursively moves input to the target_device. c sttjtfrttr"jjnj}|kr6fSsFfStt|jd}|jdksf|dkrrfSt }| |}W5QRX|j J| }| |ttr|j|nt|tjst||W5QRX|fStr"fddttDSttrNtdkrNtttSttrtdkrddttDSttrtdkrfddttDSgS)Ncpucsg|]}t|qSrtype).0robjrr {sz1_recursive_to..to_map..rcSsg|] }t|qSr)rr7irrrr:scsg|]}t|qSrr5r;r8rrr:s) isinstancer!r%rdatadevicer"getattrr6r streamindexcurrent_streamZ wait_streamZ record_streamr)rr+maprr(rr*r)r9r?Z device_modrAoutputrCr0to_mapr1r8rrG\s:       z_recursive_to..to_mapNr)r/r0r1resrrFrr3Ws ) T)condsraise_assertion_errorrcCs$|s t|t|r t|dS)zThis is used as an alternate to ``assert`` when in the backward context to print the error message ``s`` since otherwise, it is swallowed.N)print traceback print_stackr))rIrJrKrrr _p_asserts rO)tensorsizerc Cspt^||k}|sT|}t|dkd|||| W5QRSQRXdS)z Allocate storage for ``tensor`` with the given size. Returns: bool: ``True`` if this method allocated storage and ``False`` if the storage was already allocated. rz8Tensor storage should have been resized to be 0 but got N)r!no_grad_typed_storage_sizeZnumelrO_resize_)rPrQZalready_allocatedZtensor_storage_sizerrr_alloc_storages  rV)rPrc Cszth|dk}|s^t|dkd|d|d|j|d| W5QRSQRXdS)z Frees the underlying storage of ``tensor``. Returns: bool: ``True`` if the method freed the storage and ``False`` if the storage was already freed. rzVFreeing a tensor's storage is unsafe when it is not the sole occupant storage offset: z storage size: z tensor shape: N)r!rRrSrTrOZstorage_offsetshaperU)rPZ already_freedrrr _free_storages  "rXQR)fn containerrcCsdSr rr[r\rrrr&sr&cCsdSr rr]rrrr&scsfdd|S)zFRecursively apply to all tensor in different kinds of container types.cst|tjr|St|drXt|}t|D] }|j}t||t ||q2|St|t r| }| D]\}}|||<qr|St|t r|j|St|trćfdd| DSt|rfdd|D}t||St|tttfrt|fdd|DS|SdS)N__dataclass_fields__csi|]\}}||qSrr)r7keyvalueapplyrr sz4_apply_to_tensors..apply..c3s|]}|VqdSr rr7elrarr sz3_apply_to_tensors..apply..c3s|]}|VqdSr rrdrarrrfs)r=r!r%hasattr dataclassesreplacefieldsnamesetattrr@r __class__rrr>r*rr6rrset)rZdcfrkZodr_r`rHrbr[rrrbs0        z _apply_to_tensors..applyrr]rrprr&s)r/rr0r1rcCs|rt|||ng}|r$t|||ng}t|t|kr^|ddtt|t|Dn4t|t|kr|ddtt|t|Dt|t|fS)NcSsg|]}dqS)rrr7_rrrr:sz_to_kwargs..cSsg|]}iqSrrrqrrrr:s)r3r(extendranger)r/rr0r1Z moved_inputsZ moved_kwargsrrr _to_kwargss&$ru process_groupZtensorsloggercCst|||Sr )distZ_verify_params_across_processesrvrrr$_verify_param_shape_across_processessrz)modulerwbroadcast_bucket_sizesrcparams_and_buffers_to_ignorebroadcast_buffersrc Csjg}|D]\}}||kr ||q |rX|D]\}} ||kr8|| q8t||||dS)a2 Syncs ``module``'s parameters and buffers state so that all ranks contain the same module state across all ranks. Note that this API assumes that all parameter shapes are consistent before running the synchronization. This can be checked with ``_verify_param_shape_across_processes``. N)Znamed_parametersrdetachZ named_buffers_sync_params_and_buffers) r{rwr|r}r~r module_statesrkparambufferrrr_sync_module_states sr)rwrr|r}rcCs t|dkrt||||dS)zu Synchronizes ``module_states`` (list of tensors) across all processes by broadcasting them from rank 0. rN)r(ryZ_broadcast_coalesced)rwrr|r}rrrr's r) state_dict old_prefix new_prefixrcCsX||krtdt|D]6}||s,q||t|d}||||<||=qdS)a Replace all keys that match a given old_prefix with a new_prefix (in-place). Usage:: state_dict = {"layer.xyz": torch.tensor(1)} replace_by_prefix_(state_dict, "layer.", "module.layer.") assert state_dict == {"module.layer.xyz": torch.tensor(1)} z*old_prefix and new_prefix must be distinctN) ValueErrorrkeys startswithr()rrrr_Znew_keyrrr_replace_by_prefix7s  r)T)N)T)8rhrMtypingrrrrrrrr r r r!Ztorch.distributed distributedryr Ztorch.nn.parallel._functionsr Z torch.nn.parallel.scatter_gatherrZtorch.nn.utils.rnnr__all__strrrr'r,r*rrr-r%r.r?boolr3rOSizerVrXrYrnrZr&ruZ ProcessGroupLoggerrzModuleintrrrrrrrs0     * : 5 " " "