U ʹj@sUdZddlmZddlmZmZmZmZmZddl Z ddl m Z ddl m Z m Z mZmZmZmZdd d gZeeed <ee ZGd ddZGd d d e jjZGdd d e jjZdS)zkAutograd functions for stream-aware CUDA copy. It is used to overlap copy and computation on the same GPU. )deque)DequeListOptionalTupleSequenceN)Tensor)AbstractStreamcurrent_stream get_device record_stream use_stream wait_streamContextCopyWait__all__c@seZdZUeed<eed<dS)r prev_stream next_streamN)__name__ __module__ __qualname__r __annotations__rrc/var/www/html/bwave/python/venv/lib/python3.8/site-packages/torch/distributed/pipeline/sync/copy.pyrs c@sJeZdZdZeeeeedddZeee e e e dfdddZ d S) rz#Copies tensors on specific streams.ctxrrreturnc Gs||_||_g}tt|}t|jt|V|D]J}t|rt|jt|dd}||t ||t ||q4||q4W5QRXW5QRXt |S)NTZ non_blocking) rrr r rtorch is_tensortoappendr tuple)rrrinputoutputZ output_streamxyrrrforward!s      z Copy.forward.)r grad_outputrc Gs|j}|j}tt|d}tt|}t|Xt|Dt|D]4}|jt|dd}| |t ||t ||qBW5QRXW5QRXd}|t |S)N)maxlenTrNN) rrrlenr r rreversedr" appendleftr r$) rr*rr grad_inputZ input_streamr'r( grad_streamsrrrbackward:s     z Copy.backwardN rrr__doc__ staticmethodrr Tensorsr)rrrr2rrrrrs c@sJeZdZdZeeeeedddZeee e e e dfdddZ d S) rzSynchronizes a stream to another stream. Place it just before you want to start an operation on the next stream, provided that all operations on the previous stream are done. rcGs(||_||_t||tdd|DS)Ncss$|]}t|r|n|VqdS)N)r r!detach).0r'rrr aszWait.forward..)rrrr$)rrrr%rrrr)Ys z Wait.forward.)rr0rcGs"|j}|j}t||d}||S)Nr,)rrr)rr0rrr1rrrr2cs  z Wait.backwardNr3rrrrrQs )r4 collectionsrtypingrrrrrr rstreamr r r r rrrstrrr6rZautogradFunctionrrrrrrs   3