Mܶiu'ldZddlZddlmZddlmZmZddlZeje Z ddgZ edd e d e fd ZGd deZejd i d,dejdejdejdejdejde de de d eejejejffdZej d,dejdejdejdejdejde de de d eejejejffdZ d-dejdejdejdejdejde de de dedzd ejeejejfzfdZdedeedfded dfd Zejd!id"ejdejdejdejd#ejd$ejdejdejde de de d%ejd eejejejffd&Zejd"ejdejdejdejd#ejd$ejdejdejde de de d%ejd eejejejffd'Zded"ejd(ejd)ejd eejdzdff d*Zeee+dS).z Variable-length attention implementation using Flash Attention. This module provides a high-level Python interface for variable-length attention that calls into the optimized Flash Attention kernels. N) lru_cache)Any NamedTuple varlen_attn AuxRequest)maxsize device_indexreturncdS)z;Cache device capability check to avoid repeated CUDA calls.F)r s ^/root/projects/openclaw-proxy/venv/lib64/python3.11/site-packages/torch/nn/attention/varlen.py_should_use_cudnnrs  5c"eZdZUdZdZeed<dS)rz Request which auxiliary outputs to compute from varlen_attn. Each field is a boolean indicating whether that auxiliary output should be computed. FlseN)__name__ __module__ __qualname____doc__rbool__annotations__r rrrrs. Crztorch_attn::_varlen_attn) mutates_argsFquerykeyvaluecu_seq_qcu_seq_kmax_qmax_k is_causalc|jot|jj}|rbtdt jj |||d||||dd|d } | d| d| d} } } nNtd t jj |||||||d|d \} } } } } t j d t j |j }| | |fS) z Private custom op for variable-length attention. This is the internal implementation. Users should use the public varlen_attn function instead. #Using cuDNN backend for varlen_attnNTFr-Using Flash Attention backend for varlen_attn)return_debug_maskdtypedevice) is_cudarr-indexloginfotorchopsaten_cudnn_attention_forward_flash_attention_forwardzerosuint64)rrrrrrr r! use_cudnnresultoutput softmax_lse rng_state_ rng_state_s r _varlen_attnr@$s " G"3EL4F"G"GI  677788               *0F1IvayY  @AAA/4y~/V/V         #0W 0 0 , Y1 ELJ ; **rc$tj|}|d} |d} tj| | ftj|j} tjdtj|j} || | fS)z Fake implementation for meta tensor computation and tracing. Based on the 3D varlen path from meta__flash_attention_forward: - query shape: (total, num_heads, head_dim) - logsumexp shape: (num_heads, total_q) rr%r+r))r2 empty_likesizeemptyfloatr-r8) rrrrrrr r!r;total_q num_heads logsumexpr=s r_varlen_attn_fakerI^s& e $ $FjjmmG 1 I GEK I D U\JJJI 9i ''r return_auxc ~tjj||||||||\} } } | |jr| | fS| S)a9 Compute variable-length attention using Flash Attention. This function is similar to scaled_dot_product_attention but optimized for variable-length sequences using cumulative sequence position tensors. Args: - query (Tensor): Query tensor; shape :math:`(T_q, H, D)` - key (Tensor): Key tensor; shape :math:`(T_k, H, D)` - value (Tensor): Value tensor; shape :math:`(T_k, H, D)` - cu_seq_q (Tensor): Cumulative sequence positions for queries; shape :math:`(N+1,)` - cu_seq_k (Tensor): Cumulative sequence positions for keys/values; shape :math:`(N+1,)` - max_q (int): Maximum query sequence length in the batch. - max_k (int): Maximum key/value sequence length in the batch. - is_causal (bool, optional): If set to True, applies causal masking (default: False). - return_aux (Optional[AuxRequest]): If not None and ``return_aux.lse`` is True, also returns the logsumexp tensor. Shape legend: - :math:`N`: Batch size - :math:`T_q`: Total number of query tokens in the batch (sum of all query sequence lengths) - :math:`T_k`: Total number of key/value tokens in the batch (sum of all key/value sequence lengths) - :math:`H`: Number of attention heads - :math:`D`: Head dimension Returns: - Tensor: Output tensor from attention computation - If ``return_aux`` is not None and ``return_aux.lse`` is True, returns a tuple of Tensors: (output, lse), where lse is the logsumexp Example:: >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_CUDA) >>> batch_size, max_seq_len, embed_dim, num_heads = 2, 512, 1024, 16 >>> head_dim = embed_dim // num_heads >>> seq_lengths = [] >>> for _ in range(batch_size): ... length = torch.randint(1, max_seq_len // 64 + 1, (1,)).item() * 64 ... seq_lengths.append(min(length, max_seq_len)) >>> seq_lengths = torch.tensor(seq_lengths, device="cuda") >>> total_tokens = seq_lengths.sum().item() >>> >>> # Create packed query, key, value tensors >>> query = torch.randn( ... total_tokens, num_heads, head_dim, dtype=torch.float16, device="cuda" ... ) >>> key = torch.randn( ... total_tokens, num_heads, head_dim, dtype=torch.float16, device="cuda" ... ) >>> value = torch.randn( ... total_tokens, num_heads, head_dim, dtype=torch.float16, device="cuda" ... ) >>> >>> # Build cumulative sequence tensor >>> cu_seq = torch.zeros(batch_size + 1, device="cuda", dtype=torch.int32) >>> cu_seq[1:] = seq_lengths.cumsum(0) >>> max_len = seq_lengths.max().item() >>> >>> # Call varlen_attn >>> output = varlen_attn( ... query, key, value, cu_seq, cu_seq, max_len, max_len, is_causal=False ... ) )r2r3 torch_attnr@r) rrrrrrr r!rJoutrr>s rrrsSN)&33 sE8XueYKCa*.Cx Jrctxinputs.r;c |\}}}}}}} } |\} } } ||||||| | | ||_| |_| |_dSN)save_for_backwardrr r!)rNrOr;rrrrrrr r!rMrr=s r_setup_contextrSs_EKBE3x5% Ci%eXxc9UUUCICICMMMrz!torch_attn::_varlen_attn_backwardgrad_outrMrr=c tjd|j} |jot |jj} | rPt dtjj |||||||||| d| | | \}}}nOt dtjj |||||||||| d| | | \}}}|||fS)Nr)r-r#r$r') r2rDr-r.rr/r0r1r3r4_cudnn_attention_backward_flash_attention_backward)rTrrrrMrrrrr r!r=unusedr9dqdkdvs r_varlen_attn_backwardr\s[5< 0 0 0F G"3EL4F"G"GI#  6777Y^==                B" @AAAY^==                B r2:rc tj|} tj|} tj|}| | |fS)zF Fake implementation for meta tensor computation and tracing. )r2rB)rTrrrrMrrrrr r!r= grad_querygrad_key grad_values r_varlen_attn_backward_fakerasA&!%((J$$H!%((J x ++rgrad_lsegrad_rngc|j\}}}}}} } } |j} |j} |j}tjj||||| | ||| | || \}}}|||ddddddf SrQ) saved_tensorsrr r!r2r3rLr\)rNrTrbrcrrrrrrMrr=rr r!rYrZr[s r _backwardrf,sBEAR>E3x3Y IE IE I%;;         JBB r2tT4tT 99r) setup_context)F)FN)rlogging functoolsrtypingrrr2 getLoggerrr0__all__intrrrlibrary custom_opTensortupler@ register_fakerIrrSr\rarfregister_autogradr rrrts)"""""""" g!! , ' 1CD 3"EE6+6+ <6+ 6+ <6+l 6+ l 6+  6+ 6+6+ 5<u| 346+6+6+FE6+r(( <( ( <(l ( l (  ( (( 5<u| 34((((P$(LL <L L <Ll L l L  L LLT!L \E%, 455LLLL^U38_cd<2NN5l5 <5 5 < 5  5  5l5l5 5 55|5 5<u| 34555ON5p$,l, <, , < ,  ,  ,l,l, , ,,|, 5<u| 34,,,%$,2: : :05 :HM : 5<$  #$::::4 yGGGGGr