bi NddlZddlmZmZddlmZejeZeZ dej dej j dej fdZ ddej j dej d ej d ej d ej dzd ededzdedzdedzdedzdeej dffdZdS)N)_flash_attention_forward!flash_attn_supports_top_left_mask)loggingquerymodulereturnc,|jtjkr~tjdrtjdSt |jdr |jjStd|Dj jSdS)ziIf the query is in float32, return a target dtype compatible with flash attention. Return None otherwise.cuda _is_quantizedc3XK|]%}t|tjj!|V&dS)N) isinstancetorchnnLinear).0layers d/root/projects/butler/venv/lib/python3.11/site-packages/transformers/integrations/flash_attention.py z#get_target_dtype..s8bb%z%QVQYQ`?a?abbbbbbbN) dtyperfloat32is_autocast_enabledget_autocast_dtypehasattrconfignextmodulesweight)rrs rget_target_dtyper s {em##  $V , , p+F33 3 V]O 4 4 p=& &bb6>>+;+;bbbbbio o 4rkeyvalueattention_maskdropoutscalingsliding_windowsoftcap is_causalc  | ddrtd|jd} t d|jDrt d|dd}|dd}|dd}t||} | | n|j} t||||f| | ||||t| |j j t|dr|jndd | } | dfS) Noutput_attentionsFzFlash Attention does not support `output_attentions=True`. Please set your attention to `eager` if you want any of these features.rc3"K|] }|dkV dS)rN)rdims rrz*flash_attention_forward../s& + +3!8 + + + + + +rzTensor query has shape with a zero dimension. FlashAttention does not support inputs with dim=0. Please check your input shapes or use SDPA instead. layer_idx) query_lengthr)r% softmax_scaler'r(use_top_left_mask target_dtypeattn_implementationr0)getlogger warning_onceshapeany ValueError transposer r)r_use_top_left_maskr_attn_implementationrr0)rrr"r#r$r%r&r'r(r)kwargsseq_lenr4 attn_outputs rflash_attention_forwardrBsVzz%u--  W   k!nG + +u{ + + +++  B   OOAq ! !E --1  C OOAq ! !E$E622L'2 8HI*     %,!"M>&-fk&B&BL&"" K$  r)r!NNNN)rmodeling_flash_attention_utilsrrutilsr get_logger__name__r7r=TensorrModulerr floatintbooltuplerBr-rrrMsW hhhhhhhh  H % %6688 EL %(/ ek    & !% !== HO= <= = < = L4' =  =T\=$J=T\=d{= 5< ======r