biAAUddlZddlZddlZddlmZddlmZddlm Z ddl m Z m Z ddlmZdd lmZe jeZ dd lmZmZmZmZmZdd lmZdd lmZ dd lmZn #e $rdZYnwxYw ddlm!Z"dZ#n #e $rdZ#YnwxYwej$%dd&Z'dZ(e'e vZ)de*fdZde*fdZ!deddideddiej+eddiej+eddiej+edd iej+ed!d iej+eddid"ded#d$iej,ed%d&ej+ed%d&iej+ed'd&iej+ed%d&id(dej+ej-zed)d*d+iidej+ej-zed)d,d+iidej+ej-zed)d-d+iidej+ej-zed)d.d+iidej+ej-zed)d/d+iidej+ej-zed)d0d+iid1 Z.e/e*e/ee*zee/eefzffe0d2<e,ej+ed3d45iej+ed3d45id6e.d7<d8Z1dUd9Z2n+#e $r#dZ(dZ)d:Zd;Z!Gd<d=Zd>Zd?Zd@Z2YnwxYwdAddBdCddBdCddBdDZ3e/e*e/e*e*ffe0dE<iZ4e/e*edzfe0dF<dGe*dzdHe5fdIZ6 dUdGe*dJedzdHedzfdKZ7e4fdLe*dMe/e*edzffdNZ8dVdLe*dOe*dzdPe9e*zdzdHefdQZdRe:eezfdSZ;gdTZz-use_kernel_forward_from_hub..Ess)_kernels_enabled$_kernels_use_kernel_forward_from_hublogger warning_once_TRANSFORMERS_USE_HUB_KERNELS)rs rrr>sF  #7 CC C   xYvxx   #? "r! func_namectrtrt|Ststdn"tdt dS)Nzwuse_kernel_func_from_hub is not available in the installed kernels version. Please upgrade kernels to use this feature.rc|Srrfuncs rr z*use_kernel_func_from_hub..Tsr!)r"_has_use_kernel_func_from_hub!_kernels_use_kernel_func_from_hubr$r%r&)r's rrrGs|  % = %4Y?? ?0 ##B ##|]z||%$ $r!cudaz!kernels-community/deformable-detrMultiScaleDeformableAttention)repo_idrzkernels-community/moe Llama4TextMoezkernels-community/liger_kernels LigerRMSNormzkernels-community/rmsnormRMSNormzkernels-community/mlx_rmsnorm)r.rocmxpumpsnpuzmedmekk/triton-llama-mlpTritonLlamaMLPzkernels-community/megablocksMegaBlocksMoeMLPzahadnagy/megablocks)r.r4r5zkernels-community/activationFastGELU)r0rr QuickGELUNewGELUSiluGeluGeluTanh) r/r1r3MLPr9r:r;r<SiLUGeLUr?_KERNEL_MAPPINGzkernels-community/rotaryapply_rotary_transformers)r0r')r5r.rotary_pos_embcf|vp,tfd|DS)Nc3bK|])}t|tot|V*dSr) isinstancedicthas_key).0vkeys r zhas_key..s:[[1z!T22Fwq#[[[[[[r!)anyvalues)drMs `rrJrJs<ax[3[[[[PQPXPXPZPZ[[[[[[r!c|t}t|drtdstdt |dS)Nr5z0.10.2) MIN_VERSIONzfkernels uses an incompatible version. Please install the latest version with `pip install -U kernels`.)rCrJr ImportErrorr)mappings r$register_kernel_mapping_transformersrVs\ ?%G 7E " " +?H+U+U+U x  (((((r!c d}|S)Nc|Srrrs r decoratorz.use_kernel_forward_from_hub..decoratorsJr!rargskwargsrYs rrrs   r!c d}|S)Nc|Srrr*s rrYz+use_kernel_func_from_hub..decoratorsKr!rrZs rrrs   r!ceZdZdZdS)r c td)NzNLayerRepository requires `kernels` to be installed. Run `pip install kernels`. RuntimeError)selfr[r\s r__init__zLayerRepository.__init__sopp pr!N)__name__ __module__ __qualname__rdrr!rr r s( q q q q qr!r c td)Nz^replace_kernel_forward_from_hub requires `kernels` to be installed. Run `pip install kernels`.rar[r\s rrrs l   r!c td)NzVregister_kernel_mapping requires `kernels` to be installed. Run `pip install kernels`.raris rrr sstttr!c td)Nzcregister_kernel_mapping_transformers requires `kernels` to be installed. Run `pip install kernels`.raris rrVrVs q   r!zkernels-community/causal-conv1d)r0rzkernels-community/mamba-ssm)z causal-conv1dz mamba-ssmzfalcon_mamba-ssm_HUB_KERNEL_MAPPING_KERNEL_MODULE_MAPPINGattn_implementationreturnc8|duotjd|duS)zJCheck whether `attn_implementation` matches a kernel pattern from the hub.Nz'^[^/:]+/[^/:]+(?:@[^/:]+)?(?::[^/:]+)?$)research)rns r is_kernelrss. 4' c I@BU V V^b br!attention_wrappercddlm}ddlm}d|vr|ddn|}t |sdSt stdd|vr-|d\}}|}n|}d}|}| d \}}}|}|r|nd} t|| } n(#t$r} td |d | d d} ~ wwxYwt| dr |t}|} n|t| |} ||| |||d| S)a< Load and register the kernel associated to `attn_implementation`. Args: attn_implementation: A string, usually a kernel repo like "kernels-community/flash-mla". attn_wrapper: a callable for the wrapper around the attention implementation. In `transformers` we have a wrapper around the `flash_attn_var_len` call, and the same goes for `sdpa` and `eager`. They just prepare the arguments properly. This is mostly used for continious batching, where we want the `paged` wrapper, which calls the paged cache. r)ALL_MASK_ATTENTION_FUNCTIONS)ALL_ATTENTION_FUNCTIONS|r Nz`kernels` is either not installed or uses an incompatible version. Please install the latest version with `pip install -U kernels`.:@)revisionz-An error occurred while trying to load from 'z': .flash_attn_varlen_funcflash_attention_2) masking_utilsrvmodeling_utilsrwsplitrs_kernels_availablerTstrip partitionr Exception ValueErrorhasattrr getattrregister) rnrtrvrwactual_attn_namer0 kernel_name_revkernelekernel_functions rload_and_register_attn_kernelr%s=<<<<<888888||vr#t||tr||S|tvr%td|dd||<dSt r t|d}t|dd}t|dd}t|||}|||<n#t$rd||<Ynt$rd||<YnwxYwddl }| dd }d |d } |j d t} t| |d} n#t$rd} YnwxYwt!| r8| r. |j |} | ||<| S#t$rd||<Yn wxYwd||<||S) NzKernel z! not found in _HUB_KERNEL_MAPPINGr0r{rr{rr-ris_ _availablez..utils.import_utils)rHrrlr$r%rgetrFileNotFoundErrorAssertionError importlibreplace import_module __package__rrcallable) rrUr0r{rrrnew_kernel_namer' utils_modis_kernel_availablemodules rlazy_load_kernelr_s=g*W[-A:"N"N{##---TkTTTUUU# t#( ()+6yAG*;7;;JMMH)+6::9dKKG(GLLLF#)GK   ( ( (#'GK  ( ( (#'GK  ( %--c3775/555  '/ /0F TTI"))Y"E"E   ' ' '"&    ' ' ( ( (-@-@-B-B ( ,00O1EFF'- $  , , ,'+ $$$ ,$(GK ; s=A,C C-C-,C-'D99 EE%E==FFr{rc"ddlm}d||d}trntjd}t j|t jdkrt||||St|||Std ) Nr) __version__ transformers) frameworkrr0kernelsz0.10.4)r{r user_agentrzFkernels is not installed, please install it with `pip install kernels`) rrrmetadatar pkg_versionparseget_kernel_hubrT)rr{rrrkernels_versions rrrs-+R]^^Jd#,44Y??  _ - -1B81L1L L L!+'^hiii i!+'RRR Rbcccr! module_namescBttrgfd}|S)a This decorator attaches the target function as an attribute of the module. The function must already be decorated with @use_kernel_func_from_hub this decorator then wraps it as an nn.Module internally. When kernelize is later applied to the full model, the function can be accessed as a regular module attribute and kernelized just like any other layer. The kernelization is performed in place, modifying the module directly. c2|jfd}||_|S)NcL|g|Ri|D]}t|d|dS)N rotary_fn)setattr)rcr[r\fnr orig_inits rnew_initz8use_kernelized_func..decorator..new_initsQ Id ,T , , ,V , , ," / /k2.... / /r!)rd)rrrrs @rrYz&use_kernelized_func..decorators7L  / / / / / /    r!)rHr)rrYs` ruse_kernelized_funcrs?,))&$~       r!) r rrrrVrrrrr)NN)=importlib.metadatarosrqcollections.abcrtypesr packagingrrutilsrrutils.import_utilsr flash_attentionr get_loggerrer$rr r rrrrrrr#rrTrr-r,environrupperr&rr"str INFERENCETRAINING TORCH_COMPILErCrI__annotations__rJrVrlrmboolrsrrintlistr__all__rr!rrsk $$$$$$,,,,,,11111111555555444444  H % %t  ******* .YYYYYY(,%% ...(-%%%.%'JNN3De$L$L$R$R$T$T!48LL##### %C % % % %" OO;:*  OO/* =-!!!=-!!!7(!!!;(!!!=-!!!5   D OO2+  :1   :1!!! 11!!!:1!!!#  2 !33__:)666  !33__:*666  !33__:(666  !33__:vWX666  !33__:vWX666  !33__:z[\666 k|e|eOT#tFSL/DI^D_2_$_``a|||~!6B]!!! 6B]!!! - - ()\\\)))))! ! !   qqqqqqqq   uuu     =! J"CqQQ:qII$AaPP22T#tCH~-. 8:S*t"3349993:$DH77719D7$7777tPf, , #, S*t:K5K0L, , , , ^ d dC d3: dsSySWGW dcm d d d dd8nx&?4   s[J2 A'&J2'A1.J20A11J25A>=J2>BJ2BH)J22%KK