U ʹjs0@s6ddlZddlmZddlmZmZmZmZmZmZm Z m Z ddl m Z m Z ddgZGdddeZd d e d e d ed e_de ee ee ee ee ee eeeeeeeed ddZe ee ee ee ee eeeeeeeed ddZe ee ee ee ee eeeeeeeed ddZdS)N)Tensor) Optimizer_use_grad_for_differentiable _get_value_stack_if_compiling_default_to_fused_or_foreach_differentiable_doc _maximize_doc _foreach_doc)ListOptionalAdamaxadamaxcsVeZdZddddeeeedfd d Zfd d Zd dZedddZ Z S)rMb`?g?g+?:0yE>rNF)maximizedifferentiable)foreachrrc sd|kstd|d|ks,td|d|dkrDdksXntd|dd|dkrpdksntd|dd|kstd |t|||||||d } t|| dS) NzInvalid learning rate: zInvalid epsilon value: rg?z#Invalid beta parameter at index 0: rz#Invalid beta parameter at index 1: zInvalid weight_decay value: )lrbetaseps weight_decayrrr) ValueErrordictsuper__init__) selfparamsrrrrrrrdefaults __class__Q/var/www/html/bwave/python/venv/lib/python3.8/site-packages/torch/optim/adamax.pyr s(  zAdamax.__init__cst||jD](}|dd|dd|ddqt|j}t|dkoft |dd}|s|D]}t t |d|d<qpdS)NrrFrrstep) r __setstate__ param_groups setdefaultliststatevalueslentorchZ is_tensortensorfloat)rr+groupZ state_valuesZstep_is_tensorsr"r$r%r'.s     zAdamax.__setstate__c Cs|dD]}|jdkrq|||jjr2td||j|j|}t|dkrtd|d<tj|tj d|d<tj|tj d|d<||d||d||dqdS) Nr z(Adamax does not support sparse gradientsrrr&)Z memory_formatexp_avgexp_inf) gradappendZ is_sparse RuntimeErrorr+r-r.r/Z zeros_likeZpreserve_format) rr1params_with_gradgradsexp_avgsexp_infs state_stepspr+r$r$r% _init_group<s*        zAdamax._init_groupcCsd}|dk r&t |}W5QRX|jD]}g}g}g}g}g}|d\} } |d} |d} |d} |d}|d}|d}|||||||t|||||| | | | | |||d q,|S) zPerforms a single optimization step. Args: closure (Callable, optional): A closure that reevaluates the model and returns the loss. Nrrrrrrr)rbeta1beta2rrrrr)r.Z enable_gradr(r>r)rclosureZlossr1r8r9r:r;r<r?r@rrrrrrr$r$r%r&UsD   z Adamax.step)rrrrN)N) __name__ __module__ __qualname__r boolrr'r>rr& __classcell__r$r$r"r%r s" " aImplements Adamax algorithm (a variant of Adam based on infinity norm). .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \gamma \text{ (lr)}, \beta_1, \beta_2 \text{ (betas)},\theta_0 \text{ (params)},f(\theta) \text{ (objective)}, \: \lambda \text{ (weight decay)}, \\ &\hspace{13mm} \epsilon \text{ (epsilon)} \\ &\textbf{initialize} : m_0 \leftarrow 0 \text{ ( first moment)}, u_0 \leftarrow 0 \text{ ( infinity norm)} \\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}if \: \lambda \neq 0 \\ &\hspace{10mm} g_t \leftarrow g_t + \lambda \theta_{t-1} \\ &\hspace{5mm}m_t \leftarrow \beta_1 m_{t-1} + (1 - \beta_1) g_t \\ &\hspace{5mm}u_t \leftarrow \mathrm{max}(\beta_2 u_{t-1}, |g_{t}|+\epsilon) \\ &\hspace{5mm}\theta_t \leftarrow \theta_{t-1} - \frac{\gamma m_t}{(1-\beta^t_1) u_t} \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to `Adam: A Method for Stochastic Optimization`_. a  Args: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, optional): learning rate (default: 2e-3) betas (Tuple[float, float], optional): coefficients used for computing running averages of gradient and its square eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-8) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) z zd .. _Adam\: A Method for Stochastic Optimization: https://arxiv.org/abs/1412.6980 F) r r9r:r;r<rrrrr?r@rrcCstdd|Dstd|dkr4t||dd\} }|rJtjrJtd|r^tjs^t}nt}|||||||| | | | ||d dS) zrFunctional API that performs adamax algorithm computation. See :class:`~torch.optim.Adamax` for details. css|]}t|tjVqdS)N) isinstancer.r).0tr$r$r% szadamax..zPAPI has changed, `state_steps` argument must contain a list of singleton tensorsNF)Z use_fusedz6torch.jit.script not supported with foreach optimizers)rr?r@rrrr)allr7rr.ZjitZ is_scripting_multi_tensor_adamax_single_tensor_adamax)r r9r:r;r<rrrrr?r@rr_funcr$r$r%rs2) r r9r:r;r<rr?r@rrrrcCs*t|D]\} } || }| s"|n| }|| }|| }|| }|d7}| dkr^|j| | d}t| rt| } t|}t|}t|}||d|t||d| | dgd}| stj |dd|dn| tj |dddd|t|}||}| j||| dqdS)NrralphaFkeepdimout)rS)value) enumerateaddr. is_complex view_as_realZlerp_catZmul_ unsqueezeabsadd_ unsqueeze_ZamaxZcopy_rZaddcdiv_)r r9r:r;r<rr?r@rrrriparamr5r3r4Zstep_tnorm_bufbias_correctionclrr$r$r%rMs2     "rM) r r9r:r;r<r?r@rrrrrc s| r tdt|dkrdSt|||||g} | D]@\\} }}}}}| rZt|}dd| D} dd|D}dd|D}dd|D}t|d|dkr| rtj|| |d ntj|| |d }t ||dt ||t ||D]L\}}t | d|| dgd}tj|dd ||fd qfd d|D}tfd d|D}t| |||q8dS)Nz#_foreach ops don't support autogradrcSs$g|]}t|rt|n|qSr$r.rXrYrHxr$r$r% 8sz(_multi_tensor_adamax..cSs$g|]}t|rt|n|qSr$rdrer$r$r%rg9scSs$g|]}t|rt|n|qSr$rdrer$r$r%rg:scSs$g|]}t|rt|n|qSr$rdrer$r$r%rg;srrPFrRcsg|]}dt|qS)r)r)rHr&)r?r$r%rgSscsg|]}d|qS)r$)rHrb)rr$r%rgTs)AssertionErrorr-rZ"_group_tensors_by_device_and_dtyper,r.Z _foreach_negZ _foreach_add_Z _foreach_addZ_foreach_lerp_Z _foreach_mul_ziprZr[r\r]r^maxnewlongrZ_foreach_addcdiv_)r r9r:r;r<r?r@rrrrrZgrouped_tensorsZgrouped_paramsZ grouped_gradsZgrouped_exp_avgsZgrouped_exp_infsZgrouped_state_stepsrNr4r5raZbias_correctionsrcr$)r?rr%rLs6      rL)NFF)r.rZ optimizerrrrrrr r r typingr r __all__r__doc__rEr0rrMrLr$r$r$r%st ({   5 8 5