U ʹj)@sddlZddlZddlmZddlmZmZmZmZm Z m Z ddl m Z dddddZ dd d Zdd d Zdd dZdddddZdddddZdS)N)nccl) _take_tensors_flatten_dense_tensors_unflatten_dense_tensors_reorder_tensors_as_get_device_index_handle_complex)List)outcCsbt|}|dk|dkAs,td|d||dk rPdd|D}tj||Stj||SdS)aBroadcasts a tensor to specified GPU devices. Args: tensor (Tensor): tensor to broadcast. Can be on CPU or GPU. devices (Iterable[torch.device, str or int], optional): an iterable of GPU devices, among which to broadcast. out (Sequence[Tensor], optional, keyword-only): the GPU tensors to store output results. .. note:: Exactly one of :attr:`devices` and :attr:`out` must be specified. Returns: - If :attr:`devices` is specified, a tuple containing copies of :attr:`tensor`, placed on :attr:`devices`. - If :attr:`out` is specified, a tuple containing :attr:`out` tensors, each containing a copy of :attr:`tensor`. NzFExactly one of 'devices' and 'out' must be specified, but got devices=z and out=cSsg|] }t|qSr.0dr r U/var/www/html/bwave/python/venv/lib/python3.8/site-packages/torch/nn/parallel/comm.py "szbroadcast..)r RuntimeErrortorch_CZ _broadcastZ_broadcast_out)tensordevicesr r r r broadcastsrcCs,dd|D}dd|D}tj|||S)a/Broadcasts a sequence tensors to the specified GPUs. Small tensors are first coalesced into a buffer to reduce the number of synchronizations. Args: tensors (sequence): tensors to broadcast. Must be on the same device, either CPU or GPU. devices (Iterable[torch.device, str or int]): an iterable of GPU devices, among which to broadcast. buffer_size (int): maximum size of the buffer used for coalescing Returns: A tuple containing copies of :attr:`tensor`, placed on :attr:`devices`. cSsg|] }t|qSr r r r r rr7sz'broadcast_coalesced..cSsg|] }t|qSr rrtr r rr8s)rrZ_broadcast_coalesced)tensorsr buffer_sizer r rbroadcast_coalesced(src sbt|dd}|d}dt|D]~\}}|jjdks@td||krP|||kr$ddd |D}dd d |D}td |d |d |q$dkrt dt |dkr|dSt |rt |}t j||dnlt |jj|}fddt|D} || dj|dd}| ddD]} || j|ddqB|S)aSums tensors from multiple GPUs. All inputs should have matching shapes, dtype, and layout. The output tensor will be of the same shape, dtype, and layout. Args: inputs (Iterable[Tensor]): an iterable of tensors to add. destination (int, optional): a device on which the output will be placed (default: current device). Returns: A tensor containing an elementwise sum of all inputs, placed on the :attr:`destination` device. T)optionalrNcpuz+reduce_add expects all inputs to be on GPUsxcss|]}t|VqdSNstrrr!r r r Sszreduce_add..css|]}t|VqdSr"r#r%r r rr&Tszinput z has invalid size: got z, but expected zLreduce_add expects destination to be on the same GPU with one of the tensors)outputrootcsg|]\}}|kr|qSr r )rirZ root_indexr rraszreduce_add..)deviceZ non_blocking)rsize enumerater,typeAssertionErrorZ get_devicejoin ValueErrorrlenrZ is_availablerZ empty_likereducetoZadd_) inputs destinationZ input_sizer*ZinpgotexpectedresultZdestination_deviceZnonroototherr r+r reduce_add<s0      r<csdd|D}g}g}t|D]x}tdd|DrXt||}||||dqt||D] \}} || jr|| n| qb||ddqfdd|D} t| D]<} dd| D} t| |} t| | dD]} || jqqtt ||S) a]Sums tensors from multiple GPUs. Small tensors are first coalesced into a buffer to reduce the number of synchronizations. Args: inputs (Iterable[Iterable[Tensor]]): iterable of iterables that contain tensors from a single device. destination (int, optional): a device on which the output will be placed (default: current device). buffer_size (int): maximum size of the buffer used for coalescing Returns: A tuple of tensors containing an elementwise sum of each group of inputs, placed on the ``destination`` device. cSsg|]}gqSr r )r_r r rr|sz(reduce_add_coalesced..css|] }|jVqdSr") is_sparserr r rr&sz'reduce_add_coalesced..rcsg|]}t|qSr )r)rrrr rrscSsg|] }t|qSr )r)rchunkr r rrs) zipallr<appendr>Zto_denserdatatupler)r6r7rZ dense_tensorsr(Z ref_orderZtensor_at_gpusr:ZcollrZitrschunksZ flat_tensorsZ flat_resultr r@rreduce_add_coalescedis$     rHcCs|t|}|dkr6dd|D}ttj|||||S|dk rLtd||dk rbtd|ttj||||SdS)a<Scatters tensor across multiple GPUs. Args: tensor (Tensor): tensor to scatter. Can be on CPU or GPU. devices (Iterable[torch.device, str or int], optional): an iterable of GPU devices, among which to scatter. chunk_sizes (Iterable[int], optional): sizes of chunks to be placed on each device. It should match :attr:`devices` in length and sums to ``tensor.size(dim)``. If not specified, :attr:`tensor` will be divided into equal chunks. dim (int, optional): A dimension along which to chunk :attr:`tensor`. Default: ``0``. streams (Iterable[torch.cuda.Stream], optional): an iterable of Streams, among which to execute the scatter. If not specified, the default stream will be utilized. out (Sequence[Tensor], optional, keyword-only): the GPU tensors to store output results. Sizes of these tensors must match that of :attr:`tensor`, except for :attr:`dim`, where the total size must sum to ``tensor.size(dim)``. .. note:: Exactly one of :attr:`devices` and :attr:`out` must be specified. When :attr:`out` is specified, :attr:`chunk_sizes` must not be specified and will be inferred from sizes of :attr:`out`. Returns: - If :attr:`devices` is specified, a tuple containing chunks of :attr:`tensor`, placed on :attr:`devices`. - If :attr:`out` is specified, a tuple containing :attr:`out` tensors, each containing a chunk of :attr:`tensor`. NcSsg|] }t|qSr r r r r rrszscatter..zI'devices' must not be specified when 'out' is specified, but got devices=zQ'chunk_sizes' must not be specified when 'out' is specified, but got chunk_sizes=)rrFrrZ_scatterrZ _scatter_out)rrZ chunk_sizesdimstreamsr r r rscatters"rKcCspdd|D}|dkrF|dkr(tdt|ddd}tj|||S|dk r\td|tj|||SdS) aGathers tensors from multiple GPU devices. Args: tensors (Iterable[Tensor]): an iterable of tensors to gather. Tensor sizes in all dimensions other than :attr:`dim` have to match. dim (int, optional): a dimension along which the tensors will be concatenated. Default: ``0``. destination (torch.device, str, or int, optional): the output device. Can be CPU or CUDA. Default: the current CUDA device. out (Tensor, optional, keyword-only): the tensor to store gather result. Its sizes must match those of :attr:`tensors`, except for :attr:`dim`, where the size must equal ``sum(tensor.size(dim) for tensor in tensors)``. Can be on CPU or CUDA. .. note:: :attr:`destination` must not be specified when :attr:`out` is specified. Returns: - If :attr:`destination` is specified, a tensor located on :attr:`destination` device, that is a result of concatenating :attr:`tensors` along :attr:`dim`. - If :attr:`out` is specified, the :attr:`out` tensor, now containing results of concatenating :attr:`tensors` along :attr:`dim`. cSsg|] }t|qSr rrr r rrszgather..Nr?zjUsing -1 to represent CPU tensor is deprecated. Please use a device object or string instead, e.g., "cpu".T)Z allow_cpurzQ'destination' must not be specified when 'out' is specified, but got destination=)warningswarnrrrZ_gatherrZ _gather_out)rrIr7r r r rgathersrN)N)r)N)Nr)NNrN)rN)rLrZ torch.cudarZ torch._utilsrrrrrrtypingr rrr<rHrKrNr r r rs     - -0