U ʹjF@sxdZddlmZmZddlZddlmZddlmZddl m Z m Z ddl m Z Gdd d ejZGd d d ejZdS) a: Attention Pool 2D Implementations of 2D spatial feature pooling using multi-head attention instead of average pool. Based on idea in CLIP by OpenAI, licensed Apache 2.0 https://github.com/openai/CLIP/blob/3b473b0e682c091a9e53623eebc1ca1657385717/clip/model.py Hacked together by / Copyright 2021 Ross Wightman )UnionTupleN) to_2tuple)apply_rot_embedRotaryEmbedding) trunc_normal_cs8eZdZdZd eeeeedfdd Zdd ZZS) RotAttentionPool2daB Attention based 2D feature pooling w/ rotary (relative) pos embedding. This is a multi-head attention based replacement for (spatial) average pooling in NN architectures. Adapted from the AttentionPool2d in CLIP w/ rotary embedding instead of learned embed. https://github.com/openai/CLIP/blob/3b473b0e682c091a9e53623eebc1ca1657385717/clip/model.py NOTE: While this impl does not require a fixed feature size, performance at differeing resolutions from train varies widely and falls off dramatically. I'm not sure if there is a way around this... -RW NT) in_features out_features embed_dim num_headsqkv_biascst|p|}|p|}tj||d|d|_t|||_||_||dksTt|||_|jd|_ t |j|_ t |jj |ddtj|jjdS)Nbiasr࿩Zstd)super__init__nnLinearqkvprojrAssertionErrorhead_dimscaler pos_embedrweightinitzeros_r)selfr r r rr __class__[/var/www/html/bwave/python/venv/lib/python3.8/site-packages/timm/layers/attention_pool2d.pyrs    zRotAttentionPool2d.__init__cCs|j\}}}}||}||d|ddd}tj|jddd|gdd}||||dd|j|jddddd }|d|d|d}}} |ddddddf|ddddddf} }|j ||f\} } t || | }tj| |gdd}|ddddddf|ddddddf} }t || | }tj| |gdd}|| d d|j }|jdd}||  dd||dd}||}|dddfS) NrrTZkeepdimdimrr )shapereshapepermutetorchcatmeanrrrrZ get_embedr transposersoftmaxr)r"xB_HWNqkvZqcZsin_embZcos_embkcattnr%r%r&forward4s$.6 6   zRotAttentionPool2d.forward)NNr T) __name__ __module__ __qualname____doc__intboolrr@ __classcell__r%r%r#r&r s r csJeZdZdZd eeeeeeffeeeedfdd Zdd Z Z S) AttentionPool2da Attention based 2D feature pooling w/ learned (absolute) pos embedding. This is a multi-head attention based replacement for (spatial) average pooling in NN architectures. It was based on impl in CLIP by OpenAI https://github.com/openai/CLIP/blob/3b473b0e682c091a9e53623eebc1ca1657385717/clip/model.py NOTE: This requires feature size upon construction and well prevent adaptive sizing of the network. Nr T)r feat_sizer r rrcst|p|}|p|}||dks*tt||_tj||d|d|_t|||_||_ |||_ |j d|_ |jd|jd}t t |d||_t|j|ddt|jj|ddtj|jjdS)Nrrrrrr)rrrrrIrrrrrrr Parameterr0Zzerosrrrr r!r)r"r rIr r rrZ spatial_dimr#r%r&rXs    zAttentionPool2d.__init__c Cs*|j\}}}}||}|jd|ks(t|jd|ks:t||d|ddd}tj|jddd|gdd}||j d |j }| |||dd|j |jddddd }|d|d|d}}} ||d d|j} | jdd} | | dd||dd}||}|dddfS) Nrrr'r(Tr)r*rr r,)r-rIrr.r/r0r1r2rZ unsqueezetoZdtyperrrr3rr4r) r"r5r6r7r8r9r:r;r<r=r?r%r%r&r@ss.  zAttentionPool2d.forward)NNr T) rArBrCrDrErrrFrr@rGr%r%r#r&rHOs rH)rDtypingrrr0Ztorch.nnrZhelpersrZpos_embed_sincosrrZ weight_initrModuler rHr%r%r%r&s    ;