j53ddlZddlZddlmZddlZddlmZmZddlmZddl m Z ddl m Z e j eZedGd d ejZed Gd d ejZedGddejZedGddejZedGddejZedGddejZGddejZGddejZGddejZGd d!ejZGd"d#ejZGd$d%ejZGd&d'eZGd(d)ejZid*ed+ed,d-d.fd/ed0ed1ed2d3ifd4ed5ed6d3ifd7ed8ed9ejd:ed;eded?ej!d@ej"eej#ej$ej%edAZ&ee&Z'dBZ(e(d1Z)e(d0Z*e(d*Z+e(d/Z,e(d<Z-e(dCZ.e(d;Z/e(d:Z0dS)DN) OrderedDict)Tensornn)use_kernel_forward_from_hub)logging)is_torchdynamo_compilingGeluTanhcJeZdZdZd deffd ZdedefdZdedefdZxZ S) GELUTanha& A fast C implementation of the tanh approximation of the GeLU activation function. See https://huggingface.co/papers/1606.08415. This implementation is equivalent to NewGELU and FastGELU but much faster. However, it is not an exact numerical match due to rounding errors. Fuse_gelu_tanh_pythonct|r|j|_dSt jt jjd|_dS)Ntanh) approximate) super__init___gelu_tanh_pythonact functoolspartialr functionalgelu)selfr __class__s {/home/lighthouse/badge-speaker-benchmark/.benchmark-venv-community/lib/python3.11/site-packages/transformers/activations.pyrzGELUTanh.__init__(sP   Q-DHHH ();PPPDHHHinputreturnc |dzdtjtjdtjz |dtj|dzzzzzSN??@Hm?g@torchrmathsqrtpipowrrs rrzGELUTanh._gelu_tanh_python/sPs{cEJtytw/G/G5S[^c^ghmor^s^sSsKs/t$u$uuvvrc,||SNrr+s rforwardzGELUTanh.forward2xxrF) __name__ __module__ __qualname____doc__boolrrrr/ __classcell__rs@rr r sQQTQQQQQQwvw&wwwwVrr NewGELUc"eZdZdZdedefdZdS)NewGELUActivationz Implementation of the GELU activation function currently in Google BERT repo (identical to OpenAI GPT). Also see the Gaussian Error Linear Units paper: https://huggingface.co/papers/1606.08415 rrc d|zdtjtjdtjz |dtj|dzzzzzSr r%r+s rr/zNewGELUActivation.forward=sPU{cEJtytw/G/G5S[^c^ghmor^s^sSsKs/t$u$uuvvrNr2r3r4r5rr/rrr;r;6sH wVwwwwwwwrr;GeLUcJeZdZdZd deffd ZdedefdZdedefdZxZ S) GELUActivationa Original Implementation of the GELU activation function in Google BERT repo when initially created. For information: OpenAI GPT's GELU is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) This is now written in C in nn.functional Also see the Gaussian Error Linear Units paper: https://huggingface.co/papers/1606.08415 Fuse_gelu_pythonct|r|j|_dStjj|_dSr-)rr _gelu_pythonrrrr)rrBrs rrzGELUActivation.__init__Js?   *(DHHH})DHHHrrrcf|dzdtj|tjdz zzS)Nr!r"r#)r&erfr'r(r+s rrDzGELUActivation._gelu_pythonQs-s{cEIedinn.D$E$EEFFrc,||Sr-r.r+s rr/zGELUActivation.forwardTr0rr1) r2r3r4r5r6rrrDr/r7r8s@rrArAAs********G&GVGGGGVrrASiLUc"eZdZdZdedefdZdS)SiLUActivationa See Gaussian Error Linear Units (Hendrycks et al., https://arxiv.org/abs/1606.08415) where the SiLU (Sigmoid Linear Unit) was originally introduced and coined, and see Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning (Elfwing et al., https://arxiv.org/abs/1702.03118) and Swish: a Self-Gated Activation Function (Ramachandran et al., https://arxiv.org/abs/1710.05941v1) where the SiLU was experimented with later. rrc@tj|Sr-)rrsilur+s rr/zSiLUActivation.forwardbs}!!%(((rNr=r>rrrJrJXs@)V)))))))rrJFastGELUc"eZdZdZdedefdZdS)FastGELUActivationz} Applies GELU approximation that is slower than QuickGELU but more accurate. See: https://github.com/hendrycks/GELUs rrcZd|zdtj|dzdd|z|zzzzzS)Nr!r"g3E?r$)r&rr+s rr/zFastGELUActivation.forwardls;U{cEJu|/CsXX]M]`eMeGe/f$g$gghhrNr=r>rrrOrOfsHiViiiiiiirrO QuickGELUc"eZdZdZdedefdZdS)QuickGELUActivationzr Applies GELU approximation that is fast but somewhat inaccurate. See: https://github.com/hendrycks/GELUs rrc6|tjd|zzS)NgZd;?)r&sigmoidr+s rr/zQuickGELUActivation.forwardvsu}UU]3333rNr=r>rrrSrSps@4V4444444rrSc<eZdZdZdedeffd ZdedefdZxZS)ClippedGELUActivationa Clip the range of possible GeLU outputs between [min, max]. This is especially useful for quantization purpose, as it allows mapping negatives values in the GeLU spectrum. For more information on this trick, please refer to https://huggingface.co/papers/2004.09602. Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))). See https://huggingface.co/papers/1606.08415 minmaxc||krtd|d|dt||_||_dS)Nzmin should be < max (got min: z, max: )) ValueErrorrrrXrY)rrXrYrs rrzClippedGELUActivation.__init__sV 99PcPP#PPPQQ Q rxrc\tjt||j|jSr-)r&cliprrXrY)rr]s rr/zClippedGELUActivation.forwards z$q''48TX666r) r2r3r4r5floatrrr/r7r8s@rrWrWzsw  E77F77777777rrWc2eZdZdZfdZdedefdZxZS)AccurateGELUActivationz Applies GELU approximation that is faster than default and more accurate than QuickGELU. See: https://github.com/hendrycks/GELUs Implemented along with MEGA (Moving Average Equipped Gated Attention) cttjdtjz |_dS)N)rrr'r(r)precomputed_constantrrs rrzAccurateGELUActivation.__init__s7 $(Ia$'k$:$:!!!rrrc ~d|zdtj|j|dtj|dzzzzzS)Nr!rr$)r&rrer*r+s rr/zAccurateGELUActivation.forwardsCU{a%*T-F%RZ]b]fglno]p]pRpJp-q"r"rrssr)r2r3r4r5rrr/r7r8s@rrbrbsn;;;;;tVtttttttttrrbcBeZdZdZfdZdedefdZdedefdZxZS)MishActivationz See Mish: A Self-Regularized Non-Monotonic Activation Function (Misra., https://huggingface.co/papers/1908.08681). Also visit the official repository for the paper: https://github.com/digantamisra98/Mish ctttjj|_dSr-)rrrrmishrrfs rrzMishActivation.__init__s) =%rrrcj|tjtj|zSr-)r&rrrsoftplusr+s r _mish_pythonzMishActivation._mish_pythons'uz"-"8"8"?"?@@@@rc,||Sr-r.r+s rr/zMishActivation.forwardr0r) r2r3r4r5rrror/r7r8s@rrjrjs &&&&&A&AVAAAAVrrjc"eZdZdZdedefdZdS)LinearActivationz[ Applies the linear activation function, i.e. forwarding input directly to output. rrc|Sr-r>r+s rr/zLinearActivation.forwards rNr=r>rrrrrrs@VrrrceZdZdZddZdS)LaplaceActivationz Applies elementwise activation based on Laplace function, introduced in MEGA as an attention activation. See https://huggingface.co/papers/2209.10655 Inspired by squared relu, but with bounded range and gradient for better stability 绹۞? ^/ ?c||z |tjdz}ddtj|zzS)Nr#r!r")divr'r(r&rF)rrmusigmas rr/zLaplaceActivation.forwards@  3!788cEIe,,,--rN)rvrwr2r3r4r5r/r>rrrurus2......rruceZdZdZdZdS)ReLUSquaredActivationz` Applies the relu^2 activation introduced in https://huggingface.co/papers/2109.08668v2 cltj|}tj|}|Sr-)rrrelur&square)rr relu_appliedsquareds rr/zReLUSquaredActivation.forwards+}))%00 ,|,,rNr|r>rrr~r~s-rr~ceZdZfdZxZS)ClassInstantierct|}t|tr|n|if\}}|di|S)Nr>)r __getitem__ isinstancetuple)rkeycontentclskwargsrs rrzClassInstantier.__getitem__sL''%%c**!+GU!;!;Ngg'2 Vs}}V}}r)r2r3r4rr7r8s@rrrs8rrcjeZdZdZddddejdffd Zdedefd Zdedefd Z d edefd Z xZ S) XIELUActivationz Applies the xIELU activation function introduced in https://arxiv.org/abs/2411.13010 If the user has installed the nickjbrowning/XIELU wheel, we import xIELU CUDA Otherwise, we emit a single warning and use xIELU Python g?r!gưFc ttjt jt jt j||d|_ tjt jt jt j||z |d|_ | dt j||| dt j||||_ t|j |_t|j |_d|_ ddl}tjj|_d} ddlm} | |j|_|dz }n,#t<$r} |d| d z }|j|_Yd} ~ nd} ~ wwxYwt> |dS#t<$r3} t> d tC| Yd} ~ dSd} ~ wwxYw) N)dtyperbetaepszUsing experimental xIELU CUDA.)allow_in_graphz& Enabled torch._dynamo for xIELU CUDA.z+ Could not enable torch._dynamo for xIELU (z*) - this may result in slower performance.uCUDA-fused xIELU not available (%s) – falling back to a Python version. For CUDA xIELU (experimental), `pip install git+https://github.com/nickjbrowning/XIELU`)"rrr Parameterr&logexpm1tensor unsqueezealpha_palpha_nregister_bufferwith_vector_loadsr`rdetachcpuitem _beta_scalarr _eps_scalar_xielu_cuda_obj xielu.opsclassesxieluXIELU torch._dynamor _xielu_cuda_xielu_cuda_fn Exceptionlogger warning_oncestr) r alpha_p_init alpha_n_initrrrrrmsgrerrrs rrzXIELUActivation.__init__s |EIek%,|[`:a:a:a.b.b$c$c$m$mno$p$pqq | Iek%,|d/B%"P"P"PQQ R R \ \]^ _ _    VU\$e%D%D%DEEE UELE$B$B$BCCC!2!$)"2"2"4"4"8"8":":"@"@"B"B"G"G"I"IJJ !2!2!6!6!8!8!>!>!@!@!E!E!G!GHH#     #(=#6#<#<#>#>D 2C 7888888&4nT5E&F&F#?? 7 7 7tStttt&*&6###### 7    $ $ $ $ $      jC          sB.J* I"!J*" J ,JJ*J  J** K'4(K""K'r]rc ntj|j}|jtj|jz}t j|dk||z|z|j|zzt jt j ||j |z |z|j|zzS)Nr) rrrnrrrr&whererrXr)rr]rrs r _xielu_pythonzXIELUActivation._xielu_pythons-((66)bm44T\BBB{ E aK!Odi!m + [1dh// 0 01 4 ?$)a- O   rcp|j}|dkr-|d}|dk-|dkr*|dd|d}||jkr!t d||j|j||j |j |j |j |j |j|j}||S)zDFirewall function to prevent torch.compile from seeing .item() callsrhrrz_Warning: xIELU input tensor expects 3 dimensions but got (shape: %s). Reshaping to (shape: %s).)shapedimrviewsizerrrr/rtorrrrr)rr]original_shaperesults rrzXIELUActivation._xielu_cudaseeggkk AAeeggkk 5577Q;;r1affRjj))A QW $ $   q    %-- LOOAG $ $ LOOAG $ $      "  {{>***rrc|jD|jr=ts||Std||S)Nz:torch._dynamo is compiling, using Python version of xIELU.)ris_cudar rrrrr+s rr/zXIELUActivation.forward1s^   + ++-- b**5111##$`aaa!!%(((r) r2r3r4r5r&bfloat16rrrrr/r7r8s@rrrs  n))))))V v &    +V+++++2)V)))))))))rrrgelu_10i )rXrY gelu_fastgelu_new gelu_pythonrBTgelu_pytorch_tanhgelu_python_tanhr gelu_accuratelaplace leaky_relulinearrl quick_gelurrelu2relu6rU)rLswishrprelurc |tvr t|Std|dtt)Nz function z not found in ACT2FN mapping )ACT2FNKeyErrorlistkeys)activation_strings rget_activationrUsOF""'((h#4hhSWX^XcXcXeXeSfSfhhiiirrL)1rr' collectionsrr&rrintegrations.hub_kernelsrutilsrutils.import_utilsr get_loggerr2rModuler r;rArJrOrSrWrbrjrrrur~rr LeakyReLUReLUReLU6SigmoidrHTanhPReLUACT2CLSrrrrrrrrLrl linear_actr>rrrsy ###### AAAAAA888888  H % %Z((ry)(.Y''wwwww ww('wV$$RY%$,V$$ ) ) ) ) )RY ) )%$ )Z((iiiiiii)(i[))44444")44*)477777BI7772 t t t t tRY t t t RY"ry . . . . . . . .BIk[)[)[)[)[)bi[)[)[)|  N  %s2'>'>? # !   N%6$=>     $:D#AB +   ",    N %  BG  "   RX! "rz# $  W G X -   0  ! !jjjn]++ >* % %~f N; ' ' ^L ) ) ~f~f ^H % % r