U ʹj8@sdZddlZddlmZmZmZmZddlZddlmZddl m Z ddl m Z d/e eeeejd d d Zd0e ee eejejdddZdd ddejdfee e eeeejeejejdddZdddd ddd dejdf ee eeje e eeeeeee ejeejeejd ddZGdddejZddZejdd d!Zeejdd"d#Zejdd$d%Zd&d'Zdddd dd dejdf ee eeje e eeeeee ejeejd( d)d*ZGd+d,d,ejZ Gd-d.d.ejZ!dS)1zv Sin-cos, fourier, rotary position embedding modules and functions Hacked together by / Copyright 2022 Ross Wightman N)ListTupleOptionalUnion)nn)ndgrid)_assertl@T) num_bandsmax_freq linear_bandsdevicecCsP|r tjd|d|tj|d}n&dtjdt|dd|tj|d}|tjS)N?dtyperrr)torchlinspacefloat32mathlogpi)r r r rbandsr[/var/www/html/bwave/python/venv/lib/python3.8/site-packages/timm/layers/pos_embed_sincos.pypixel_freq_bandss&r@r)r temperaturesteprreturncCs2tjd||tj|dtj|}d||}|S)NrrrrZarangeZint64tor)r rrrexprrrr freq_bandss" r$@F) feat_shapedimr reverse_coordinterleave_sin_cosrrr c s|ddkstd|d}t||dd}|r>|ddd}ttfdd |Dddd} | d|d} |rd nd} tjt| t | g| d d} | j |d S) a Args: feat_shape: dim: temperature: reverse_coord: stack grid order W, H instead of H, W interleave_sin_cos: sin, cos, sin, cos stack instead of sin, sin, cos, cos dtype: device: Returns: rzHEmbed dimension must be divisible by 4 for sin-cos 2D position embeddingrrrrNcs&g|]}tj|tjdtjqS)rrr!.0srrr Dsz,build_sincos2d_pos_embed..rr'r) AssertionErrorr$rstackrflatten transpose unsqueezesincosr") r&r'rr(r)rrZpos_dimrgridpos2Z stack_dimZpos_embrr1rbuild_sincos2d_pos_embed's"  $r>) r&rr max_resrr include_grid in_pixelsref_feat_shaperrr c s|dkr4|r"t|t||d}qPt||dd}ndkrB|j| dkrP|j} |rhfdd|D} nfdd|D} |dk rddt| ||D} tjt| d d } | d } | |} | j | d | | }}|r| ||gn||g}|S) a[ Args: feat_shape: Feature shape for embedding. bands: Pre-calculated frequency bands. num_bands: Number of frequency bands (determines output dim). max_res: Maximum resolution for pixel based freq. temperature: Temperature for non-pixel freq. linear_bands: Linear band spacing for pixel based freq. include_grid: Include the spatial grid in output. in_pixels: Output in pixel freq. ref_feat_shape: Reference feature shape for resize / fine-tune. dtype: Output dtype. device: Output device. Returns: N)r rrr+c s"g|]}tjdd|tjdqS)gr)Zstepsrr)rrrr.r1rrr2sz+build_fourier_pos_embed..cs&g|]}tj|tjdtjqSr-r!r.r1rrr2scSsg|]\}}}|||qSrr)r/xfrrrrr2sr,r3r4) rfloatr$rrziprr6rr9r:r"r;)r&rr r@rr rArBrCrrtr<posZpos_sinZpos_cosoutrr1rbuild_fourier_pos_embedPs:  rLcs.eZdZd eedfdd Zdd ZZS) FourierEmbedr?r%TF)r@r cs<t||_||_||_||_|jdt||dddS)NrF persistent)super__init__r@r concat_grid keep_spatialregister_bufferr)selfr@r rRrS __class__rrrQs zFourierEmbed.__init__cCs|jdd\}}|jdd}t||j|j|j|jd}tj|dd}|dd t |}|fd|j d}|j rtj|| d|dd ddgdd}n.) isinstancerTensorrlrrprapply_rot_embed_lists rscCsZ|dd\}}|jdkrF||d|t||d|S||t||S)Nrr,rYr)Z tensor_splitr]r9rkrj)rDrbrmrnrrrapply_rot_embed_cats ,rtc Cs@|d|jddd}|d|ddd|jd}|S)Nrr,r)r9r^rZZgather)rD pos_embedZ keep_indicesrrrapply_keep_indices_nlcs"rv) r&rr'r@rr rBrCrrc  Csjt|||d|||||| |d \} } d} |D] } | | 9} q,| | ddd} | | ddd} | | fS)a Args: feat_shape: Spatial shape of the target tensor for embedding. bands: Optional pre-generated frequency bands dim: Output dimension of embedding tensor. max_res: Maximum resolution for pixel mode. temperature: Temperature (inv freq) for non-pixel mode linear_bands: Linearly (instead of log) spaced bands for pixel mode in_pixels: Pixel vs language (inv freq) mode. dtype: Output dtype. device: Output device. Returns: r*) rr r@rr rBrCrrrr,r)rLr_Zrepeat_interleave)r&rr'r@rr rBrCrrrmrnZnum_spatial_dimrDrrrbuild_rotary_pos_embeds$  rwcs\eZdZdZdeeeeeeedfdd Zdeeed d d Z d dZ Z S)RotaryEmbeddinga Rotary position embedding NOTE: This is my initial attempt at impl rotary embedding for spatial use, it has not been well tested, and will likely change. It will be moved to its own file. The following impl/resources were referenced for this impl: * https://github.com/lucidrains/vit-pytorch/blob/6f3a5fcf0bca1c5ec33a35ef48d97213709df4ba/vit_pytorch/rvt.py * https://blog.eleuther.ai/rotary-embeddings/ r?'TFNr r&rCc st||_||_||_||_||_||_|dkr|rRt|dt ||d}nt |d|dd}t ||j d|ddd|_ d|_n@t||||||jd\} } d|_|j d | dd|j d | dddS) Nr*r rrrrFrNr&r'r@r rBrC pos_embed_sin pos_embed_cos)rPrQr'r@rrBr&rCrrGr$printrTr~rrwr) rUr'r@rrBr r&rCrZemb_sinZemb_cosrVrrrQs\  zRotaryEmbedding.__init__rZcCs8|jdk r(|dk stt||j|jdS|j|jfSdS)N)rB)rr5rwrBr~r)rUrZrrr get_embedYs  zRotaryEmbedding.get_embedcCs$||jdd\}}t|||SNr)rrZro)rUrDrmrnrrrrceszRotaryEmbedding.forward)r?ryTFNN)N rdrerf__doc__boolrrrgrQrrcrhrrrVrrxs   > rxcs\eZdZdZdeeeeeeedfdd Zdeeed d d Z d dZ Z S)RotaryEmbeddingCata Rotary position embedding w/ concatenatd sin & cos The following impl/resources were referenced for this impl: * https://github.com/lucidrains/vit-pytorch/blob/6f3a5fcf0bca1c5ec33a35ef48d97213709df4ba/vit_pytorch/rvt.py * https://blog.eleuther.ai/rotary-embeddings/ r?ryTFNrzc st||_||_||_||_||_||_|dkr||rRt|dt ||d}nt |d|dd}|j d|ddd|_ n4t ||||||jd} d|_|j d t| d dddS) Nr*r{rr|rFrNr}rur,)rPrQr'r@rrBr&rCrrGr$rTrurwrrr[) rUr'r@rrBr r&rCrembedsrVrrrQssN  zRotaryEmbeddingCat.__init__rcCsT|jdk r4|dk r4t||j|j|jd}t|dS|jdk rD|jSdsPtddS)N)rBrCr,FzPget_embed() requires pre-computed pos_embed or valid shape w/ pre-computed bands)rrwrBrCrr[rur5)rUrZrrrrrs  zRotaryEmbeddingCat.get_embedcCs||jdd}t||Sr)rrZrt)rUrDrurrrrcszRotaryEmbeddingCat.forward)r?ryTFNN)NrrrrVrrks   7r)r TN)rrN)"rrtypingrrrrrrr<rZ trace_utilsr rgrGrrrrrr$rrr>rLModulerMrjrorsrtrvrwrxrrrrrs      +  F,  1[