U ʹj9@sUdZddlZddlmZmZmZmZmZmZddlZddlm Z ddl Zddddd gZ ee e d <ee Zee efZeegeeee ffZGd ddZGd ddZdd ddZeeedddZeeddd ZdS)zManipulation of micro-batches.N)AnyCallableListUnioncastSequence)TensorNoChunkBatchcheckscattergather__all__c@s*eZdZdZedddZeddZdS)r a Wrapper for a Tensor in :meth:`Pipe.forward` indicating that the tensor should not be chunked on the batch dimension and instead be replicated as-is across all micro-batches. This is useful for tensors which might not have any 'batch' semantics for the model. )inpcCs"t|std|||_dS)Nz+NoChunk only supported for tensors, found: )torch is_tensor TypeError_tensor)selfrri/var/www/html/bwave/python/venv/lib/python3.8/site-packages/torch/distributed/pipeline/sync/microbatch.py__init__s zNoChunk.__init__cCs|jSN)rrrrrtensor#szNoChunk.tensorN)__name__ __module__ __qualname____doc__rrpropertyrrrrrr sc@seZdZdZeeeefddddZe edddZ e d d Z d d Z d dZ eddddZedddZddZedddZedddZejeeddddZejeeddddZeeefddd dZeddd!d"Zeddd#d$ZdS)%r zC An abstraction representing a microbatch in the pipeline. N)valuesreturncCs@||_t||_|js3sz!Batch.__init__..zNo tensors found in batch: )_valuesrratomicanyr)rr rrrr-s  zBatch.__init__r!cCs|jstdtt|jS)z Retrieves the underlying tensor.znot atomic batch)r'AttributeErrorrrr&rrrrr6sz Batch.tensorcCs|jS)z-Retrieves the underlying values for the batch)r&rrrrr =sz Batch.valuescCs<|jr dSt|jD]\}}t|r|SqtddS)z< Retrieves the index of first tensor found. rzNo tensor found!N)r' enumerater&rrr)rir$rrrfind_tensor_idxBs   zBatch.find_tensor_idxcCs2|jr|jjS|jD]}t|r|jSqdS)z; Retrieves the device for this microbatch. N)r'r&devicerr)rr$rrr get_deviceNs   zBatch.get_device)functionr!cCs&|jrt||jSt||jSdS)zbCalls a function on the microbatch. It also wraps the output with :class:`Batch`. N)r'r r&)rr0rrrcallYsz Batch.callcCsd|jd|jdS)Nz Batch[atomic=z]()r'r&rrrr__repr__bszBatch.__repr__ccs |jr|jVn |jEdHdSrr3rrrr__iter__es zBatch.__iter__cCs|jr dSt|jS)N)r'lenr&rrrr__len__ksz Batch.__len__)indexcCs&|js|j|S|dkr td|jS)Nr atomic batch allows index 0 onlyr'r& IndexError)rr9rrr __getitem__ns  zBatch.__getitem__)r9r$r!cCsdSrrrr9r$rrr __setitem__xszBatch.__setitem__cCsdSrrr>rrrr?|s)r9r!cCs(t|tr|||n |||dSr) isinstanceint_setitem_by_index_setitem_by_slicer>rrrr?s cCsP|js6|}|jd||f|j|dd|_dS|dkrFtd||_dS)Nr6rr:r;)rr9r$r,rrrrBs(zBatch._setitem_by_indexcCs`|j|jkr$|jkr$dks.ntd|js>||_dSt|dkrRtd|d|_dS)Nzonly slice [:] supportedr6z5atomic batch cannot be replaced with multiple tensorsr)startstopstepNotImplementedErrorr'r&r7r<r>rrrrCs& zBatch._setitem_by_slice)rrrrrrrrrrrr r-r/Functionr1strr4r5rAr8r=typingoverloadr?sliceTensorsrBrCrrrrr (s(       r)csBtdd|Ds td|tfdd|Dr>tddS)z Checks whether the input contains at least one tensor and each tensor is on the same device as the first partition. Raises: ValueError: input does not contain at least one tensor css|]}t|VqdSrr"r#inputrrrr%szcheck..z inputs do not have any tensors: c3s"|]}t|o|jkVqdSr)rrr.rN first_devicerrr%sz>All inputs should be on the same device as the first partitionN)r(r ValueError)rQinputsrrPrr s )chunksr!cGst|dkr2t|dtr2dd|d|DSddt|D}d}|D]}t|r||}|dkr|t|krtd|dt|t|}t|D]\}}|| |qqLt|D].}t|t r|| |j q|| |qqL|d |}d d|DS) z7Splits an input mini-batch into multiple micro-batches.r6rcSsg|] }t|qSrr r#xrrr szscatter..cSsg|]}gqSrr)r#_rrrrXsz6Found different number of chunks produced for inputs:  and NcSsg|] }t|qSrrUrVrrrrXs) r7r@rchunkrangerr RuntimeErrorr+appendr r)rTrSZbatchesZ num_chunksrOtensorsr,rrrrr s$     )outputsc Cs|djr(tdd|D}t|}ng}tt|dD]}t|d|}g}|D]>}|t||krtd|dt|||||qXt |d|r|t|q<||q.z2Types for microbatch outputs do not match, found: r[) r'tuplercatr]r7typerr_r)rar`outputZ output_bufr, output_typeZcurrent_outputsbatchrrrr s    )rrJrrrrrrrrZtorch.cuda.commrrI__annotations__rMZTensorOrTensorsrHr r r rAr r rrrrs   w#