U ʹj1T@sddlZddlZddlmZmZddlmZe ej dshedej j d<edej j d<edej j d<dd l m Z mZmZd d Zd d ZGdddej jZGdddZdddZdS)N) tree_flattentree_unflatten) _dummy_typeZ_CudaStreamBase _CUDAGraph_graph_pool_handle_cuda_isCurrentStreamCapturing)rrrcCstS)z Returns True if CUDA graph capture is underway on the current CUDA stream, False otherwise. If a CUDA context does not exist on the current device, returns False without initializing the context. )rr r P/var/www/html/bwave/python/venv/lib/python3.8/site-packages/torch/cuda/graphs.pyis_current_stream_capturingsr cCstS)z Returns an opaque token representing the id of a graph memory pool. See :ref:`Graph memory management`. .. warning:: This API is in beta and may change in future releases. )rr r r r graph_pool_handle$sr csveZdZdZfddZdfdd Zfdd Zfd d Zfd d ZfddZ fddZ fddZ Z S) CUDAGraphzw Wrapper around a CUDA graph. .. warning:: This API is in beta and may change in future releases. cs t|SN)super__new__)cls __class__r r r8szCUDAGraph.__new__Nglobalcstj||ddS)a Begins capturing CUDA work on the current stream. Typically, you shouldn't call ``capture_begin`` yourself. Use :class:`~torch.cuda.graph` or :func:`~torch.cuda.make_graphed_callables`, which call ``capture_begin`` internally. Arguments: pool (optional): Token (returned by :func:`~torch.cuda.graph_pool_handle` or :meth:`other_Graph_instance.pool()`) that hints this graph may share memory with the indicated pool. See :ref:`Graph memory management`. capture_error_mode (str, optional): specifies the cudaStreamCaptureMode for the graph capture stream. Can be "global", "thread_local" or "relaxed". During cuda graph capture, some actions, such as cudaMalloc, may be unsafe. "global" will error on actions in other threads, "thread_local" will only error for actions in the current thread, and "relaxed" will not error on these actions. Do NOT change this setting unless you're familiar with `cudaStreamCaptureMode `_ )poolcapture_error_modeN)r capture_begin)selfrrrr r r;szCUDAGraph.capture_begincstdS)aP Ends CUDA graph capture on the current stream. After ``capture_end``, ``replay`` may be called on this instance. Typically, you shouldn't call ``capture_end`` yourself. Use :class:`~torch.cuda.graph` or :func:`~torch.cuda.make_graphed_callables`, which call ``capture_end`` internally. N)r capture_endrrr r rOs zCUDAGraph.capture_endcstdS)z? Replays the CUDA work captured by this graph. N)rreplayrrr r rZszCUDAGraph.replaycstdS)zD Deletes the graph currently held by this instance. N)rresetrrr r r`szCUDAGraph.resetcs tS)z Returns an opaque token representing the id of this graph's memory pool. This id can optionally be passed to another graph's ``capture_begin``, which hints the other graph may share the same memory pool. )rrrrr r rfszCUDAGraph.poolcs tS)zB Enables debugging mode for CUDAGraph.debug_dump. )renable_debug_moderrr r rnszCUDAGraph.enable_debug_modecs t|S)z Arguments: debug_path (required): Path to dump the graph to. Calls a debugging function to dump the graph if the debugging is enabled via CUDAGraph.enable_debug_mode() )r debug_dump)rZ debug_pathrr r rtszCUDAGraph.debug_dump)Nr) __name__ __module__ __qualname____doc__rrrrrrrr __classcell__r r rr r 0s     r c@s4eZdZdZdZd edddZddZd d ZdS) grapha Context-manager that captures CUDA work into a :class:`torch.cuda.CUDAGraph` object for later replay. See :ref:`CUDA Graphs ` for a general introduction, detailed use, and constraints. Arguments: cuda_graph (torch.cuda.CUDAGraph): Graph object used for capture. pool (optional): Opaque token (returned by a call to :func:`~torch.cuda.graph_pool_handle()` or :meth:`other_Graph_instance.pool()`) hinting this graph's capture may share memory from the specified pool. See :ref:`Graph memory management`. stream (torch.cuda.Stream, optional): If supplied, will be set as the current stream in the context. If not supplied, ``graph`` sets its own internal side stream as the current stream in the context. capture_error_mode (str, optional): specifies the cudaStreamCaptureMode for the graph capture stream. Can be "global", "thread_local" or "relaxed". During cuda graph capture, some actions, such as cudaMalloc, may be unsafe. "global" will error on actions in other threads, "thread_local" will only error for actions in the current thread, and "relaxed" will not error on actions. Do NOT change this setting unless you're familiar with `cudaStreamCaptureMode `_ .. note:: For effective memory sharing, if you pass a ``pool`` used by a previous capture and the previous capture used an explicit ``stream`` argument, you should pass the same ``stream`` argument to this capture. .. warning:: This API is in beta and may change in future releases. .. _cudaStreamCaptureMode: https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__STREAM.html#group__CUDART__STREAM_1g9d0535d93a214cbf126835257b16ba85 Nr)rcCsr|jjdkrtj|j_|dkr&dn|f|_|dk r:|n|jj|_|jdk sRttj|j|_ ||_ ||_ dS)Nr ) rdefault_capture_streamtorchcudaStreamrZcapture_streamAssertionErrorstream stream_ctx cuda_graphr)rr,rr*rr r r __init__s zgraph.__init__cCs@tjttj|j|jj |j d|j idS)Nr) r&r' synchronizegcZcollectZ empty_cacher+ __enter__r,rrrrr r r r0s   zgraph.__enter__cCs|j|j|||dSr)r,rr+__exit__)rexc_type exc_value tracebackr r r r1s zgraph.__exit__)NNr) rr r!r"r%strr-r0r1r r r r r$s r$Fc% s.trtrtdd}t|ts6d}|f}|f}gt||D]\}}t|tjjrt |j dkrt |j dkrt |j dkst dtdd|Dst dt|\}}t|td d|DsDt d qDd d D} d d |Dfdd tt |D} dd tt |D} dd tt |D} t} tjtjtjt||| D]p\}}}t|D]V}t||\}}tjjtdd|Dtdd|Dtdd|Dd|d}q~~qtW5QRXtjg}g}t||| D]T\}}}tjj|| d||}W5QRXt|\}}|t|||qg}g}tt| t|t| tD]\}}}}tdd|D}tjj|| dFtjjtdd|Dtdd|Dtdd|Dd|d}W5QRXg}d}|D]0}|jr4||||d7}n |dqt|}||||qtt|}tt|}dd} g}!t|D]\}"}| | |"| |"|"| |"||"| |"||"||"||" }#t|tjjr dd}$|$||j|#|j |_ |!|n |!|#q|r&|!dSt|!S) a Accepts callables (functions or :class:`nn.Module`\ s) and returns graphed versions. Each graphed callable's forward pass runs its source callable's forward CUDA work as a CUDA graph inside a single autograd node. The graphed callable's forward pass also appends a backward node to the autograd graph. During backward, this node runs the callable's backward work as a CUDA graph. Therefore, each graphed callable should be a drop-in replacement for its source callable in an autograd-enabled training loop. See :ref:`Partial-network capture` for detailed use and constraints. If you pass a tuple of several callables, their captures will use the same memory pool. See :ref:`Graph memory management` for when this is appropriate. Arguments: callables (torch.nn.Module or Python function, or tuple of these): Callable or callables to graph. See :ref:`Graph memory management` for when passing a tuple of callables is appropriate. If you pass a tuple of callables, their order in the tuple must be the same order they'll run in the live workload. sample_args (tuple of Tensors, or tuple of tuples of Tensors): Samples args for each callable. If a single callable was passed, ``sample_args`` must be a single tuple of argument Tensors. If a tuple of callables was passed, ``sample_args`` must be tuple of tuples of argument Tensors. num_warmup_iters (int): The number of warmup iterations. Currently, ``DataDistributedParallel`` needs 11 iterations for warm up. Default: ``3``. allow_unused_input (bool): If False, specifying inputs that were not used when computing outputs (and therefore their grad is always zero) is an error. Defaults to False. .. note:: The ``requires_grad`` state of each Tensor in ``sample_args`` must match the state that's expected for the corresponding real input in the training loop. .. warning:: This API is in beta and may change in future releases. .. warning:: ``sample_args`` for each callable must contain only Tensors. Other types are not allowed. .. warning:: Returned callables do not support higher order differentiation (e.g., double backward). .. warning:: In any :class:`~torch.nn.Module` passed to :func:`~make_graphed_callables`, only parameters may be trainable. Buffers must have ``requires_grad=False``. .. warning:: After you pass a :class:`torch.nn.Module` through :func:`~make_graphed_callables`, you may not add or remove any of that Module's parameters or buffers. .. warning:: :class:`torch.nn.Module`\s passed to :func:`~torch.cuda.make_graphed_callables` must not have module hooks registered on them at the time they are passed. However, registering hooks on modules *after* passing them through :func:`~torch.cuda.make_graphed_callables` is allowed. .. warning:: When running a graphed callable, you must pass its arguments in the same order and format they appeared in that callable's ``sample_args``. .. warning:: The automatic mixed precision is supported in :func:`~torch.cuda.make_graphed_callables` only with disabled caching. The context manager `torch.cuda.amp.autocast()` must have `cache_enabled=False`. z_make_graphed_callables does not support the autocast caching. Please set `cache_enabled=False`.FTrzModules must not have hooks registered at the time they are passed. However, registering hooks on modules after passing them through make_graphed_callables is allowed.css|]}|jdkVqdS)FN requires_grad.0br r r 'sz)make_graphed_callables..zIn any :class:`~torch.nn.Module` passed to :func:`~make_graphed_callables`, only parameters may be trainable. All buffers must have ``requires_grad=False``.css|]}t|tjVqdSr) isinstancer&ZTensor)r:argr r r r<.szfIn the beta API, sample_args for each callable must contain only Tensors. Other types are not allowed.cSsg|] }t|qSr )len)r:argsr r r 5sz*make_graphed_callables..cSs*g|]"}t|tjjr"t|ndqS)r )r=r&nnModuletuple parameters)r:cr r r rA6scsg|]}||qSr r r:iZflatten_sample_argsZper_callable_module_paramsr r rA:scSsg|]}tjqSr r&r'r r:_r r r rA?scSsg|]}tjqSr rJrKr r r rA@scss|]}|jr|VqdSrr7r:or r r r<Oscss|]}|jr|VqdSrr7rGr r r r<Pscss|]}|jrt|VqdSrr8r&Z empty_likerMr r r r<Qs)outputsinputsZ grad_outputsZ only_inputsZ allow_unused)rcss"|]}|jrt|ndVqdSrrOrMr r r r<tscss|]}|jr|VqdSrr7rMr r r r<zscss|]}|jr|VqdSrr7rGr r r r<{scss|]}|dk r|VqdSrr rMr r r r<|srNc s8G fdddtjjfdd} | S)Ncs@eZdZefddZeejjjfddZ dS)zOmake_graphed_callables..make_graphed_autograd_function..Graphedcs`tD].}|||kr|||qttsNttddDS)Ncss|]}|VqdSrdetachrMr r r r<szjmake_graphed_callables..make_graphed_autograd_function..Graphed.forward..)rangedata_ptrcopy_rr=rDr))ctxrQrH) fwd_graph len_user_argsstatic_input_surfacestatic_outputsr r forwards  zWmake_graphed_callables..make_graphed_autograd_function..Graphed.forwardcsrt|tkstt|D]*\}}|dk r||kr||qtts`ttddDS)Ncss"|]}|dk r|n|VqdSrrRr9r r r r<szkmake_graphed_callables..make_graphed_autograd_function..Graphed.backward..)r?r)ziprUrVrr=rD)rWZgradsggrad) bwd_graphstatic_grad_inputsstatic_grad_outputsr r backwards zXmake_graphed_callables..make_graphed_autograd_function..Graphed.backwardN) rr r! staticmethodr\r&autogradfunctionZonce_differentiablercr )r`rXrYrarbrZr[r r Grapheds  rgcs(t|\}}jt|}t|Sr) _tree_flattenapplyrD_tree_unflatten) user_argsZflatten_user_argsrLout)rg module_paramsoutput_unflatten_specr r functionalizeds zVmake_graphed_callables..make_graphed_autograd_function..functionalized)r&reFunction) rXr`rmrYrnrZr[rbraror ) rgr`rXrYrmrnrarbrZr[r make_graphed_autograd_functions $z>make_graphed_callables..make_graphed_autograd_functioncsfdd}|S)Ncsjkr|S|SdSr)training)rkfuncgraph_training_stategraphedorig_fwdr r new_fwds zEmake_graphed_callables..make_graphed_forward..new_fwdr )rtrurvrwrxr rsr make_graphed_forwardsz4make_graphed_callables..make_graphed_forward)!r&Zis_autocast_enabledZis_autocast_cache_enabled RuntimeErrorr=rDr]rBrCr?Z_backward_hooksZ_forward_hooksZ_forward_pre_hooksr)allbuffersrhappendrTr r'r.r*r(rer_r$reversedr8list enumeraterrr\)%Z callablesZ sample_argsZnum_warmup_itersZallow_unused_inputZjust_one_callablerFr@Z flatten_argrLZper_callable_len_user_argsZ"per_callable_static_input_surfacesZ fwd_graphsZ bwd_graphsZmempoolrtrZrPZ grad_inputsZper_callable_static_outputsZ"per_callable_output_unflatten_specrXZflatten_outputsspecZ per_callable_static_grad_outputsZper_callable_static_grad_inputsr[r`rmrbraZgrad_idxr>rqretrHrvryr rIr make_graphed_callablessE                 3   r)r6F)r/r&Ztorch.utils._pytreerrhrrjZ_utilsrhasattrZ_C__dict__Ztorch._Crrrr r r r$rr r r r s"     OL