U ʹj(@sdZddlZddlmZddlZddlmZddlmZm Z m Z m Z m Z ddl mZdd Zefd d Zd d ZddZddZGdddZddddgfddZgddfddZgdfddZgdfddZd$ejjejjd!d"d#ZdS)%zM Model / state_dict utils Hacked together by / Copyright 2020 Ross Wightman Ndeepcopy)FrozenBatchNorm2d)BatchNormAct2dSyncBatchNormActFrozenBatchNormAct2dfreeze_batch_norm_2dunfreeze_batch_norm_2d)ModelEmacCs,t|trt|jSt|dr$|jS|SdS)Nmodule) isinstancer unwrap_modelZemahasattrr )modelrO/var/www/html/bwave/python/venv/lib/python3.8/site-packages/timm/utils/model.pyrs  rcCs ||SN)Z state_dict)rZ unwrap_fnrrrget_state_dictsrcCs t|jdddgddS)zA calculate average channel square mean of output activations rZaxis)torchmeanitemrinputoutputrrravg_sq_ch_meansrcCst|jdddgdSz> calculate average channel variance of output activations rrrrrrvarrrrrr avg_ch_var!sr"cCst|jdddgdSrr rrrravg_ch_var_residual'sr#c@s(eZdZdZddZddZddZdS) ActivationStatsHookaIterates through each of `model`'s modules and matches modules using unix pattern matching based on `hook_fn_locs` and registers `hook_fn` to the module if there is a match. Arguments: model (nn.Module): model from which we will extract the activation stats hook_fn_locs (List[str]): List of `hook_fn` locations based on Unix type string matching with the name of model's modules. hook_fns (List[Callable]): List of hook functions to be registered at every module in `layer_names`. Inspiration from https://docs.fast.ai/callback.hook.html. Refer to https://gist.github.com/amaarora/6e56942fcb46e67ba203f3009b30d950 for an example on how to plot Signal Propogation Plots using `ActivationStatsHook`. cCsb||_||_||_t|t|kr*tdtdd|D|_t||D]\}}|||qHdS)Nz_Please provide `hook_fns` for each `hook_fn_locs`, their lengths are different.css|]}|jgfVqdSr)__name__).0hook_fnrrr Fsz/ActivationStatsHook.__init__..) r hook_fn_locshook_fnslen ValueErrordictstatszip register_hook)selfrr)r* hook_fn_locr'rrr__init__?szActivationStatsHook.__init__csfdd}|S)Ncs"|||}jj|dSr)r.r%append)r rroutr'r1rrappend_activation_statsKs zAActivationStatsHook._create_hook..append_activation_statsr)r1r'r7rr6r _create_hookJsz ActivationStatsHook._create_hookcCs6|jD]&\}}t||s q |||q dSr)r named_modulesfnmatchZregister_forward_hookr8)r1r2r'namer rrrr0Qs z!ActivationStatsHook.register_hookN)r% __module__ __qualname____doc__r3r8r0rrrrr$-s r$rcCs*tdd|}t|||d}||}|jS)a Extract average square channel mean and variance of activations during forward pass to plot Signal Propogation Plots (SPP). Paper: https://arxiv.org/abs/2101.08692 Example Usage: https://gist.github.com/amaarora/6e56942fcb46e67ba203f3009b30d950 gg?)r)r*)rnormalr$r.)rr)r*Z input_shapexhook_rrrextract_spp_statsXs rETfreezec s"|dkstdttjjjjtjjjjtt fr:tdt|t rJ|g}|}fdd|D}t |s|t t \}}t ||D]\}}|D]}|dkrdnd|_q|rd d }|dkrt|} t|tjjjjtjjjjtt fr||| qt|} t|ttfr||| qd S) a4 Freeze or unfreeze parameters of the specified modules and those of all their hierarchical descendants. This is done in place. Args: root_module (nn.Module, optional): Root module relative to which the `submodules` are referenced. submodules (list[str]): List of modules for which the parameters will be (un)frozen. They are to be provided as named modules relative to the root module (accessible via `root_module.named_modules()`). An empty list means that the whole root module will be (un)frozen. Defaults to [] include_bn_running_stats (bool): Whether to also (un)freeze the running statistics of batch norm 2d layers. Defaults to `True`. mode (bool): Whether to freeze ("freeze") or unfreeze ("unfreeze"). Defaults to `"freeze"`. )rFunfreezez,`mode` must be one of "freeze" or "unfreeze"zYou have provided a batch norm layer as the `root module`. Please use `timm.utils.model.freeze_batch_norm_2d` or `timm.utils.model.unfreeze_batch_norm_2d` instead.csg|]}|qSr) get_submodule)r&m root_modulerr sz$_freeze_unfreeze..rFFTcSsD|dd}t|dkr4||d|d|n |||dS)N.r r)rsplitr+rHZ add_module)r r; submodulesplitrrr_add_submodules  z(_freeze_unfreeze.._add_submoduleN)AssertionErrorr rnnmodulesZ batchnormZ BatchNorm2dZ SyncBatchNormrrstrr+listr/named_children parametersZ requires_gradrr rr) rK submodulesinclude_bn_running_statsmoder9nrIprQresrrJr_freeze_unfreezejsB       r_cCst|||dddS)a Freeze parameters of the specified modules and those of all their hierarchical descendants. This is done in place. Args: root_module (nn.Module): Root module relative to which `submodules` are referenced. submodules (list[str]): List of modules for which the parameters will be frozen. They are to be provided as named modules relative to the root module (accessible via `root_module.named_modules()`). An empty list means that the whole root module will be frozen. Defaults to `[]`. include_bn_running_stats (bool): Whether to also freeze the running statistics of `BatchNorm2d` and `SyncBatchNorm` layers. These will be converted to `FrozenBatchNorm2d` in place. Hint: During fine tuning, it's good practice to freeze batch norm stats. And note that these are different to the affine parameters which are just normal PyTorch parameters. Defaults to `True`. Hint: If you want to freeze batch norm ONLY, use `timm.utils.model.freeze_batch_norm_2d`. Examples:: >>> model = timm.create_model('resnet18') >>> # Freeze up to and including layer2 >>> submodules = [n for n, _ in model.named_children()] >>> print(submodules) ['conv1', 'bn1', 'act1', 'maxpool', 'layer1', 'layer2', 'layer3', 'layer4', 'global_pool', 'fc'] >>> freeze(model, submodules[:submodules.index('layer2') + 1]) >>> # Check for yourself that it works as expected >>> print(model.layer2[0].conv1.weight.requires_grad) False >>> print(model.layer3[0].conv1.weight.requires_grad) True >>> # Unfreeze >>> unfreeze(model) rFrZr[Nr_rKrYrZrrrrFscCst|||dddS)a Unfreeze parameters of the specified modules and those of all their hierarchical descendants. This is done in place. Args: root_module (nn.Module): Root module relative to which `submodules` are referenced. submodules (list[str]): List of submodules for which the parameters will be (un)frozen. They are to be provided as named modules relative to the root module (accessible via `root_module.named_modules()`). An empty list means that the whole root module will be unfrozen. Defaults to `[]`. include_bn_running_stats (bool): Whether to also unfreeze the running statistics of `FrozenBatchNorm2d` layers. These will be converted to `BatchNorm2d` in place. Defaults to `True`. See example in docstring for `freeze`. rGr`NrarbrrrrGs rGF)rreturncs$|s t|}fdd||S)Ncs`|D]R\}}t|dr,t|||n&t|dr@|nt|drR||qdS)Nfusereparameterizeswitch_to_deploy)rWrsetattrrdrerf)rIZ child_namechild_fuserrrjs    z#reparameterize_model.._fuser)rZinplacerrirreparameterize_models  rk)F)r>r:copyrrZtorchvision.ops.miscrZ timm.layersrrrrr Z model_emar rrrr"r#r$rEr_rFrGrSModulerkrrrrs$    / E"