U ʹj~[@sddlZddlZddlZddlZddlZddlZddlZddlZddlZddl m Z ddl m Z ddl mZmZmZmZddlZddlmZddlZddlmZddlmZddlmZddlmZmZd d lm Z d d lm!Z!m"Z"e#e$Z%ed Z&e d Z'e'(Z)e)rddl*Z+gZ,dZ-e)rXddddgZ,e+j.j/j012ddZ3d4dde,DZ-dddgZ5GdddZ6ddZ7dZ8Gd d!d!Z9e:dd"d#Z;d$d%d&d'ZGd,d-d-e?Z@d.d/ZAdJd0d1ZBdKd$d$d2d3d4ZCd5d6ZDd7d8ZEd9d:ZFdLd$d$d2d;d<ZGeeeHeeHeeHd=d>d?ZIe&eee&ge&fd@dAdBZJeJejKZLeJeMdCZNeJdZOeJd$ZPeJd$ZQGdDdEdEZRGdFdGdGZSGdHdIdIZTdS)MN)Counter) import_module)CallableOptionalSequenceTypeVar) rand_strided)is_float_dtype)StorageWeakRef)ContentStoreReaderContentStoreWriter)config) clone_inputs get_debug_dirTztorch._inductor.configz1//caffe2/torch/fb/sparsenn:sparsenn_operators_gpuz-//caffe2/torch/fb/sparsenn:sparsenn_operatorsz///deeplearning/fbgemm/fbgemm_gpu:sparse_ops_cpuz+//deeplearning/fbgemm/fbgemm_gpu:sparse_opszfbcode:// cCsg|]}d|dqS)ztorch.ops.load_library("z").0xrrX/var/www/html/bwave/python/venv/lib/python3.8/site-packages/torch/_dynamo/debug_utils.py 0srZbuck2runz@mode/dev-nosanc@s&eZdZddZddZd ddZdS) BuckTargetWritercCstjtj|\|_|_|jdd|_|jddd|j|_|j|jdd|_|jdd|_|j}||dddd}d|d |j|_ dS) Nz.pyr/.zfbcode.zfbcode/r:) ospathsplitabspathsubdirpy_filereplacetargetfind cmd_line_path)selffilenametmprrr__init__7szBuckTargetWriter.__init__c CsDdddtD}td|jd|jdtd|d|jd S) NrcSsg|]}d|dqS)z "z",rrrrrrFsz*BuckTargetWriter.build..za load("@fbcode_macros//build_defs:python_binary.bzl", "python_binary") python_binary( name="z", srcs = ["z"], compile = False, deps = [ "//caffe2:torch", "//caffe2/functorch:functorch", "//triton:triton", "z", ], cpp_deps = [ z ], main_module = "z", ) )join extra_depstextwrapdedentr(r& cur_targetr")r+Zextra_cpp_depsrrrbuildEs zBuckTargetWriter.buildTc CsZtj|jd}t|d}||W5QRXt|jg}|rVt dd||S)NZTARGETSwzFFound an example that reproduces the error. Run this cmd to repro - %s ) r!r"r/r%openwriter4BUCK_CMD_PREFIXr*logwarning)r+Z print_msgZ target_filefdZ cmd_splitrrrr8]s  zBuckTargetWriter.writeN)T)__name__ __module__ __qualname__r.r4r8rrrrr6srcCsLtjtd}|dkr.tdt}tj|sHtj |dd|S)NZminifierz /minifier_T)exist_ok) r!r"r/rtempfile gettempdirgetpassgetuserexistsmakedirs)r"rrr minifier_dirks  rGc@seZdZejjejjejjejjejj ejj ejj ejj ejj ejjejjejjejjejjejjejjejjejjejjejjejjgZeddZeddZdS)NNModuleToStringcCsLt}|D] \}}t|tjkr||qt|dkrHtd|dS)Nrz-We have not tested reprs of some modules - %sT) setnamed_childrentyperI safe_reprsaddlenr:r;)gmZ cant_convert_modulerrrcan_convert_to_strings   z&NNModuleToString.can_convert_to_stringcCsddlm}d}td}|D]V\}}|}t|d}|dk rZ|jrZ|d}||dd|d|d 7}q"|j D]\}} | dkrq| t krdd l m} | jt kstt| } n@t| rd t| jd | jd } ndt| jd | jd } | jr| d} ||dd|d| d7}q|j D]d\} } | dkrXqBd}| jrhd}dt| jd | j|d} ||dd| d| d 7}qB|||jdd 7}|S)Nr) _addindent z from torch.nn import * class Repro(torch.nn.Module): def __init__(self): super().__init__() z.cuda()zself.z = r) PRINT_OPTSz torch.randn(z, dtype=)ztorch.randint(1, size=zself.register_buffer('z', z) rz, device="cuda"ztorch.nn.Parameter(torch.randn(z))rH)Ztorch.nn.modules.modulerTr1r2rK__repr__next parametersZis_cuda_buffersitemsZnumelMAX_CONSTANT_NUMEL_INLINEZtorch._tensor_strrW thresholdAssertionErrorreprtorchis_floating_pointlistshapedtype _parameterscode)rPrTtab model_str module_namerRZ module_strZ example_paramZ buffer_namebufferrWZ tensor_str param_nameparam maybe_devicerrrconvertsF            " zNNModuleToString.convertN)r=r>r?rbnnZLinearZConv1dZConv2dZConv3dZ BatchNorm1dZ BatchNorm2dZ BatchNorm3dZ LayerNormZDropoutZSoftmaxZReLUZGELUZIdentityZ MaxPool2dZ EmbeddingZTanhZConvTranspose1dZGLUZLSTMZFlattenZAdaptiveAvgPool2drM staticmethodrSrprrrrrIws4 rIcCstjsdSd}zJtjddgtjd}|jd}d dd |D}||d7}Wnt k rx|d 7}YnXt d d t tj D}|d 7}|D]\}}|d|d|d7}q|d7}|S)Nz:# torch.cuda.is_available()==False, no GPU info collected z# CUDA Info: Znvccz --version)stdoutrrcSs g|]}|dkrd|dqS))r#  r)rsrrrrsz-_cuda_system_info_comment..z# nvcc not found css|]}tj|VqdSN)rbcudaZget_device_name)rirrr sz,_cuda_system_info_comment..z# GPU Hardware Info: rtz : ru)rbrxZ is_available subprocessrPIPErsdecoder#r/FileNotFoundErrorrrangeZ device_countr])rjZcuda_version_outZcuda_version_linescommentZ gpu_namesnamecountrrr_cuda_system_info_comments$  rF) stable_outputcCsHddl}ddl}|rdSd|jjd|jjd|jjdS)Nrz*# config omitted due to stable_output=TruezYimport torch._dynamo.config import torch._inductor.config import torch._functorch.config r)Ztorch._functorch.configZtorch._inductor.configZ_dynamorZcodegen_configZ _inductorZ _functorch)rrbrrrgenerate_config_strings   rcCstjtdS)Nzminifier_launcher.py)r!r"r/rGrrrrget_minifier_repro_pathsrc Cst}td|tr"t|z$t|d}||W5QRXWn6tk r|}zt|t d|W5d}~XYnXdS)NzWriting minified repro to: %sr5z(Could not write to {minified_repro_path}) rr:r;use_buckrr8r7OSError exceptionNotImplementedError)contentsZminified_repro_pathr<errrhelper_for_dump_minify s    rc@s eZdZdS) AccuracyErrorN)r=r>r?rrrrrsrcCsBt|}tt|D](}t||tjr||||jq|S)z This clone inputs is different from utils clone_input. In case of minifier, all the tensors are leaf tensors while creating a new graph. So, we set the requires_grad field w/o checking the leafness of the tensor. )rrrO isinstancerbTensorZrequires_grad_ requires_grad)example_inputsZ cloned_inputsidxrrrclone_inputs_retaining_gradnesss rc Csddlm}ddlm}m}m}t|}|s6t|}t |drJ| dt |dd}t |d d} t |d s|dk s|| dk r||}|dk r||_ | dk r| |_ ||} |r| S|| r|| } | ||| d|S) z Runs a forward and possibly backward iteration for a given mod and args. When disable_clone is True, we will use args as-is without cloning. This is higher fidelity but we may destroy the args in the process. r)make_boxed_funcr )collect_resultsreduce_to_scalar_lossrequires_bwd_pass zero_gradTnamed_parametersN named_buffersZ _boxed_call)Ztorch._functorch.aot_autogradrtestingrrrcopydeepcopyrhasattrrgetattrrrZbackward) rPargsonly_fwdZ disable_clonerrrrZorig_named_parametersZorig_named_buffersoutZlossrrrrun_fwd_maybe_bwd)s4       r require_fp64 ignore_non_fpc Cs.ddlm}ddlm}m}ddlm} t||rF|||_|||_ t||rd|||_|||_ t |||} d} t j rz(t t|t|\} } t | | |} Wn*tk r|rtdtdYnXzt |||}Wn4tk r}ztdWYdSd}~XYnX| | || t jd|d }|S) aa Check two models have same accuracy. require_fp64: if True, raise an error if we unable to calculate the fp64 reference ignore_non_fp: if True, do not compare outputs which are not floating point. This is mostly useful for the minifier (which wants to avoid quantizing floating point error into integer/boolean error) r )OptimizedModule)"named_buffers_for_optimized_module%named_parameters_for_optimized_module)sameNzCould not generate fp64 outputszWhile minifying the program in accuracy minification mode, ran into a runtime exception which is likely an unrelated issue. Skipping this graph.T)ZtolZ equal_nanr)Z eval_framerrrrutilsrrrrrrZsame_two_models_use_fp64 cast_to_fp64rrr Exception RuntimeErrorr:r;rZrepro_tolerance)rPZopt_gmrrrrrrrrrefZfp64_refZ fp64_modelZ fp64_examplesresrZpassingrrrsame_two_modelsPsL         rcCs|jjD]b}|jdkr|jtjjjjkrt |j dks:t t |j dr|j dtj kr|j dtj f|_ q|j||S)NZ call_functionrVr r)graphZnodesopr(rbZopsZprimsZconvert_element_typedefaultrOrr`r float64ZlintZ recompile)modelnoderrr!cast_convert_element_type_to_fp64s  rcsBddlm}|}tjkr(t|}|fdd|}||fS)Nr)tree_mapcs"t|tjr|r|S|Srw)rrbrrctorrfrrs   zcast_to..)Ztorch.utils._pytreerrrbrr)rfrinputsrrrrcast_tos    rcCsttj||Srw)rrbr)rrrrrrsrc Csdz,|t|t|}t||||||d WStk r^}ztdWYdSd}~XYnXdS)NrzWhile minifying the program in accuracy minification mode, ran into a runtime exception which is likely an unrelated issue. Skipping this graphF)rrrrrr:r)rPrZ compiler_fnrrrZ compiled_gmrrrrbackend_accuracy_failss$  r)striderereturncCs|dk r |St|Srw)rZmake_contiguous_strides_for)rrerrr_stride_or_defaultsr)drcs fddS)Ncs|dk r |SSrwrrrrrrz_mk_defaulter..rrrrr _mk_defaultersrcpuc@s4eZdZddZdddddZddZd d ZdS) NopInputReadercCs d|_dS)Nrtotal)r+rrrr.szNopInputReader.__init__Ndevice dtype_hintcCs|jd7_dS)Nr r)r+ storage_hashnbytesrrrrrstorageszNopInputReader.storagecOsdSrwrr+rkwargsrrrtensorszNopInputReader.tensorcOsdSrwrrrrrsymintszNopInputReader.symintr=r>r?r.rrrrrrrrsrc@sJeZdZd ddddZdddddZddddddd d Zd d ZdS) InputReaderN)pbarcCs8|dkrtd|dk r"t|nd|_g|_||_dS)Nz0no save_dir specified, will generate random data)r:r;r storerr)r+save_dirrrrrr.s  zInputReader.__init__rcCs|jdk r|jdt|}t|}|jdk r||dk r|z|j|}Wntk r\Yn X||jkrxt d||j|St d|||j f}t d|d}t |||| S)Nr zdevice mismatch: %s != %sz1could not load %s, generating random data insteadre)rupdate_device_or_default_dtype_or_defaultrZ read_storager~rr:r;itemsizerruntyped_storage)r+rrrrrrerrrrrs       zInputReader.storage)storage_offsetrfris_leafc Kst||d}t|}t|}t|}t|}tjg||j|d} t| ||||W5QRX|st | j tj d} W5QRXt| ||||W5QRXtj j| |ksttj| ||j| | S)Nr)rfrr)Z memory_format)r_storage_offset_or_defaultr_is_leaf_or_default_requires_grad_or_defaultrbrrZno_gradset_Z enable_gradcloneZpreserve_format _subclasses meta_utils safe_is_leafr`_utilsZset_tensor_metadatarappend) r+rrerrrfrrmetadatatrrrr%s,     zInputReader.tensorcCs|j||Srw)rr)r+valrrrrFs zInputReader.symint)N)Nrrrrrrs  !rc@sTeZdZddddZddZddded d d Zdd d d Zdd ddZdS) InputWriterF stable_hashcCs:g|_t|_||_|dk r*t||dnd|_i|_dS)Nr)_lines itertoolsrstorage_counterrr r seen_storages)r+rrrrrr.Xs zInputWriter.__init__cCs*dg}|dd|jD|d|S)Nzdef load_args(reader):css|]}d|VqdS)rUNr)rlrrrrzhsz$InputWriter.lines..zload_args._version = 0)extendrr)r+rrrrlinesds  zInputWriter.linesNr device_hint)rc Cst|}|j|}|dk r |Sdt|j}d}tdt|krNd|}d}|j}|jdkrr|dk snt|}t d|krd|}| } d} |j dk r|jjdkr|j |} |j |d| d| ||d||j|<|S) Nbufrz , dtype_hint=metaz , device=z = reader.storage(, rX)r rgetrZrrrrLr`rrrZ write_storagerr) r+rrrwsvZmaybe_dtype_hintrorrrrrrrss0        zInputWriter.storagec Cs4|j||j|jd}g}td|jd|krH|tt |t d|jkrh|d|jt d| kr|d| t j|}|r|dd|Dtd|jkr|d|jt jj|}td|kr|d||jd d |tt |jf|d |dS) Nrrzdtype=zstorage_offset=css |]\}}|d|VqdS)=Nr)rkrrrrrzsz%InputWriter.tensor..zrequires_grad=zis_leaf=zreader.tensor(r) # )rrrfrrrerrstrtuplerrrrbrZget_tensor_metadatarr]rrrrrrrr/)r+rrrrZtensor_metadatarrrrrs6  zInputWriter.tensorcCs0t|tjr|jj}|jd|d|dS)Nzreader.symint(r)rrbZSymIntrhintrr)r+rrrrrrs zInputWriter.symint) r=r>r?r.rrrrrrrrrrWs  r)FF)F)F)Ur functoolsrCrloggingr!r{rAr1 collectionsr importlibrtypingrrrrrbZtorch._prims_commonZ _prims_commonrZtorch._subclasses.meta_utilsZtorch._dynamo.testingrr Z torch.multiprocessing.reductionsr Ztorch.utils._content_storer r rrrr getLoggerr=r:rZinductor_configZ is_fbcoderZlibfb.py.build_infoZlibfbr0Z extra_importspyZ build_infoZ BuildInfoZget_build_ruler'r3r/r9rrGr^rI lru_cacherrrrrrrrrrrrrintrrZfloat32rrrrrrrrrrrrrs         5 d  + E *   T