bi$dZddlZddlmZddlmZGddejZGddejZGd d ejZ dS) a Generic interface to various configurations of the Perceiver Resampler, that simply takes in a series of (potentially time-indexed) contextual embeddings, and "resamples" (compresses) them down to a pre-specified number of latents! Note that the Perceiver in general resamples based solely off the *long-range* context; there's a nice opportunity here to prime the Perceiver Resampler with say a single layer's worth of language embeddings (the target domain), and use that to softly "retrieve & compress" what we need --> this would be a novel contribution we should explore. References: - DeepMind's Flamingo: https://www.deepmind.com/blog/tackling-multiple-tasks-with-a-single-visual-language-model - Code borrowed w/ love from: https://github.com/lucidrains/flamingo-pytorch N) IdeficsConfigc`eZdZdededededededdffd Zd ejdejfd ZxZ S) IdeficsPerceiverResamplerconfig embed_dimdepthn_headshead_dim n_latentsreturnNc$t||||f\____jj_tj tj jjd_ tjds jdznjjdz_tjfdt%|D_tjj_dS)ao Instantiates a Perceiver Resampler that operates over a sequence of embeddings (say from a ResNet or ViT or MAE) of a given dimension, performs `depth` blocks of cross-attention with a fixed `n_latents` inputs, then returns a Tensor of shape [bsz, n_latents, embed_dim]. :param embed_dim: Dimensionality of embeddings being fed to the Perceiver Resampler (also dimensionality of latent embeddings *returned* by the Perceiver Resampler. Could be e.g., VIT embed_dim, ResNet pool dim, and so on. Args: config (`IdeficsConfig`): config object embed_dim (`int`): The size of each embedding vector depth (`int`): Depth of the Perceiver Resampler (Transformer w/ cross attention). Should be shallow (< 3). n_heads (`int`): Number of heads in each Transformer block (for multi-headed self-attention). head_dim (`int`): Dimensionality of each head projection in the Transformer block. n_latents (`int`): Number of latent embeddings to resample ("compress") the input sequence to (usually < 128). T) requires_gradrc g|]O}tjtjjjjtjgPS) nn ModuleListIdeficsPerceiverAttentionrr r qk_layer_norms IdeficsMLPintermediate_dim).0_rselfs `/root/projects/butler/venv/lib/python3.11/site-packages/transformers/models/idefics/perceiver.py z6IdeficsPerceiverResampler.__init__..Qsi     1$.$,PTP]_c_rss"4#8&AA   N)super__init__rr r r perceiver_configqk_layer_norms_perceiverrr Parametertorchrandnlatentshasattr vision_configrrrangeblocks LayerNorm layer_norm)rrrr r r r __class__s`` rr z"IdeficsPerceiverResampler.__init__/s( FOQXZbdmFmC dmT^$5N|EK$O$O_cddd 6/== 4DNQ  %/!3  m     u       ,t~66rcontextc|j|jddd}|jD]"\}}||||z}|||z}#||S)zWResample arbitrary length context & *compress* down to self.n_latents latent embeddingsrr)r&repeatshaper*r,)rr.r&attnffs rforwardz!IdeficsPerceiverResampler.forward]sw,%%gmA&61==  , ,HD"d7G,,w6GbkkG+GGw'''r) __name__ __module__ __qualname__rintr r$Tensorr4 __classcell__r-s@rrr.s,7#,703,7 let long-form inputs be `context`, resampled embeddings be `latents`gFbiasN)rr rr r rrr+context_layer_normlatents_layer_norm q_layer_norm k_layer_normqk_scaleLinearq_projk_projv_proj output_proj)rrr r rr-s rr z"IdeficsPerceiverAttention.__init__ks7 6?(3 dm,"$,t~">">"$,t~">">   < " T] ; ;D  " T] ; ;D  t+ i t}0LSXYYY i t}0LSXYYY i t}0LSXYYY 9T\DM%A9SXYYYrr.r&cH |}|}|jdd\ }}|}t j||gd}t j||gd} fd|||fD\}}}jr* |} |}t j d|j z|}|| dd z } | d} t j d | |} | d d dS) aF Runs Perceiver Self-Attention, with special (context, latents) appended along the `seq` dimension! Args: context (`torch.Tensor`): Tensor of shape `[bsz, seq, embed_dim]` representing long-form context to resample. latents (`torch.Tensor`): Tensor of shape `[bsz, n_latents, embed_dim]` representing fixed length latents to compress to. Returns: `torch.Tensor`: Tensor of shape `[bsz, n_latents, embed_dim]` representing attention over latents w/ cross from context. N)dimcg|]C}||jdjjddDS)r)reshaper1r r transpose)rx batch_sizers rrz5IdeficsPerceiverAttention.forward..sHuuufg199ZT\4=QQ[[\]_`aauuurz... i d, ... j d -> ... i jT)rMkeepdimz... i j, ... j d -> ... i drrO)r@rAr1rFrGr$catrHrrBrCeinsumrDamaxdetachsoftmaxrIrQflatten) rr.r& seq_lengthrqkvscoresstabilized_scoresr2 resampledrSs ` @rr4z!IdeficsPerceiverAttention.forwards))'22))'22,3M"1",=) J  KK  KK 7G"4"=== > > KK 7G"4"=== > > vuuuulmoprsktuuu1a   %!!!$$A!!!$$A;Q=NPQRR"fkkb$k&G&G&N&N&P&PQ ((R(00L!>aHH  3 3Aq 9 9 A A" E EFFFr) r5r6r7r8boolr r$r9r4r:r;s@rrrjsZ#ZZsZTXZ]aZZZZZZ*(Gu|(Gel(Gu|(G(G(G(G(G(G(G(GrrcZeZdZdeffd ZdeejdzdejfdZxZ S)rrcZt|jj|_t j|j|_t j|j|d|_t j |_ t j||jd|_ dS)z:Simple MLP block with intermediate_size and embedding sizeFr>N) rr r(rrr+lnrEfcReLUactc_proj)rintermediate_sizerr-s rr zIdeficsMLP.__init__s -7,t~..)DN,=EJJJ799i 14>NNN r hidden_statesNr c||}||}||}||}|S)N)rfrgrirj)rrls rr4zIdeficsMLP.forwardsL ..  .. //  M22 r) r5r6r7rr tupler$ FloatTensorr4r:r;s@rrrs}O-OOOOOOU5+<%=%DIZrr) __doc__r$torch.nnrconfiguration_ideficsrModulerrrrrrrts4   0000009(9(9(9(9( 9(9(9(x>G>G>G>G>G >G>G>GBr