bi:1NdZddlmZddlZddlmZddlmZddlm Z ddl m Z dd l m Z mZe r dd lmZd d lmZiZeed GddZGddZeddZd,dZd-d"Zd.d/d'ZejZd0d(Zd.d)d*d+ZdS)1z Contains the logic for automatic additional output capture with our forward decorators. This mostly describe the hooks used and the logic to make capture thread/context safe. ) annotationsN) ContextVar) dataclasswraps) TYPE_CHECKING)is_torchdynamo_compilingrequires)nnPreTrainedModel)torch)backendscHeZdZUdZded<dZded<dZded <dZded <dS) OutputRecordera Configuration for recording outputs from a model via hooks. Attributes: target_class (Type): The class (e.g., nn.Module) to which the hook will be attached. index (Optional[int]): If the output is a tuple/list, optionally record only at a specific index. layer_name (Optional[str]): Name of the submodule to target (if needed), e.g., "transformer.layer.3.attn". class_name (Optional[str]): Name of the class to which the hook will be attached. Could be the suffix of class name in some cases. ztype[nn.Module] target_classrintindexN str | None layer_name class_name)__name__ __module__ __qualname____doc____annotations__rrr^/root/projects/butler/venv/lib/python3.11/site-packages/transformers/utils/output_capturing.pyrr'sY"!!!ENNNN!J!!!!!J!!!!!!r rc*eZdZdZdZdZdZdZdS)CompileableContextVara Convenience wrapper around a ContextVar for usage with `torch.compile`. This behaves exactly as a `ContextVar`, except when compilation is triggered in which case it behaves as a simple global variable. This is useful as `torch.compile` cannot trace the `get` method of `ContextVar`. This however means that the access to the underlying variable is not thread-safe when compilation is triggered. cNt|||_||_d|_dS)NdefaultF)r context_var global_var compiling)selfnamer&s r!__init__zCompileableContextVar.__init__Bs)%dG<<<!r c|jr|jStrd|_|jS|jSNT)r)r(r is_compilingr'get)r*s r!r0zCompileableContextVar.getGsI > .? "()) .$(!&'++---r crtr||_d|_dS|j|Sr.)r r(r)r'set)r*values r!r2zCompileableContextVar.setSs: # % % /#DO!DN4#''.. .r ch|jrd|_d|_dS|j|dS)NF)r)r(r'reset)r*tokens r!r5zCompileableContextVar.reset[s; > *"DO"DNNN   " "5 ) ) ) ) )r N)rrrrr,r0r2r5rr r!r#r#:sZ . . .///*****r r#output_collectorr%module nn.ModulekeystrrrreturnNonec@fd}||dS)zaInstall the forward hook needed to capture the output described by `key` and `index` in `module`.ct}||vrdSdkr:t|dkr!||dt |t s||dS|#||dSdS)N hidden_statesr)_active_collectorr0keyslenappend isinstancetuple)r8argsoutputcollected_outputsrr:s r!output_capturing_hookz;install_output_capuring_hook..output_capturing_hookjs-1133  $3D3I3I3K3K(K(K F / ! !c*;C*@&A&AQ&F&F c " ) )$q' 2 2 2&%(( 9 c " ) )& 1 1 1 1 1 E] & c " ) )&- 8 8 8 8 8' &r N)register_forward_hook)r8r:rrJs `` r!install_output_capuring_hookrLgs? 9 9 9 9 9 9   !677777r parent_module module_name capture_tasks list[tuple[str, OutputRecorder]]cddlm}|D]B\}}t||st ||d||,t ||d|C|D]i\}}|jt||js!|jA||jr'|j |j |vrSt|||j jdS)a Recursively install all output capturing hooks on all submodules of `parent_module`. Note that we need to use this recursive approach instead of simply iterating over all modules, because we want to respect the `capture_tasks` of all individual submodels (`PreTrainedModel` instances) in the graph. That is, once we reach a submodel in the graph, its children should use this submodel's `capture_tasks`, but other parts of the graph should not. r r.)prefixN) modeling_utilsrnamed_childrenrErecursively_install_hooks"install_all_output_capturing_hooksrrendswithrrLr)rMrNrOrr+r8r:specss r!rVrV{s 100000&4466WW f&/22 W %f.E.Et.E.E} U U U U /v>U>Ut>U>U V V V V V$JJ U   *z-I[/\/\ *   ([-A-A%BR-S-S (+0@ 0S0S (U[ I I IJJr modelrrSrctt|jpi}g}|D]\}}t |t s|g}|D]z}t |tsLd|vrdnd}t |tsdn|}t |ts|nd} t| ||}|||f{||nd}t|||d|_ dS)z Install the output recording hooks on all the modules in `model`. Tis will take care of correctly dispatching the `_can_record_outputs` property of each individual submodels in case of composite models. r@rr N)rrrT) _CAN_RECORD_REGISTRYr0r; __class__itemsrElistrrDrV!_output_capturing_hooks_installed) rZrS capture_flagsrOr: layer_specsrYrrrs r!rWrWs. ),,S-A-ABBHbMM)//11 / /[+t,, (&-K  / /Ee^44 f,33)3E3)?)?JTTU ,6uc,B,BLuu &LZdeee  #u . . . .  /)VVrFeV];;;.2E+++r ct|ddrdSt5t|ddr ddddSt|ddddS#1swxYwYdS)z Check if the model already has output capturing hooks installed, and install them if it is not already the case. Note that this is thread-safe, in case 2 (or more) threads want to install them concurrently. raFN)getattr_hook_installation_lockrW)rZs r!maybe_install_capturing_hooksrgsu95AA 22 5=u E E   22222222 +5111 222222222222222222sAAAAT)tie_last_hidden_statesc,fd}| ||S|S)a Decorator to intercept specific layer outputs through hooks. The hooks are installed only once and lazily, the first time output capture is requested with the `output_xxx` kwargs/config. The implementation is fully context/thread safe, except when using `torch.compile`, as dynamo is unable to trace through `ContextVar` methods. Args: tie_last_hidden_states (`bool`, *optional*, defaults to `True`): Whether to overwrite `out.hidden_states[-1]` with the `out.last_hidden_state`. This is true for all language models and should be toggled off only if `out.hidden_states[-1]` has to be the hidden state before last layer norm, which is needed for some vision models (e.g. CLIP, SigLIP) c@tfd}|S)Ncdtjdd}tt jpi}fd|D}d|vr-dtjdd|d<d|vr-dtjdd|d <d |D}t|d krtt |}  g|Ri}t |n#t |wxYw|D]_} | d kr snt|d r4|| dd|| <|| |jnCt|dr3|| dd|| <|| |jt#|| || <| dkrt%|| t&r`t|| dkrGt#|| d dd|| <t#|| ddd|d| z<-t#|| || <Gt#|| || <a|dur|}|S)N return_dictTc xi|]6}d|d|tjd|d7S)output_F)r0reconfig).0kkwargsr*s r! zHcapture_outputs..wrapped_fn..wrapper..s[! vzz-A--mXYmm]b9c9cddr cross_attentionsoutput_attentionsFoutput_cross_attentionsmask_decoder_attentionsoutput_mask_decoder_attentionscFi|]\}}||ddgS)rnr\)replace)rprqvs r!rszHcapture_outputs..wrapped_fn..wrapper..s2 g g g$!Qef g9b!9!92 g g gr rr@vision_hidden_stateslast_hidden_state attentionsr r cross_)popreror]r0r;r^r_rCrgrAr2r5hasattrrDr|r~rFrEr`to_tuple) r*rGrrrlcapturable_flagsrecordable_keysrI output_tokenoutputsr:funcrhs ` ` r!wrapperz4capture_outputs..wrapped_fn..wrappers]!**]GDKX\4]4]^^K 477DN8K8KLLRPR )O "%555=CZZ'>QSX)Y)Y>> 9:),<<QSX)Y)YEE @A!h g/J_J_JaJa g g g $%%))-d333,001BCCL 6$t5d555f55"'' 5555!'' 5555) A A/))1Q *@AAQ1B31G1L)#.)#.55g6RSSSS *=>>Q1B31G1L)#.)#.55g6OPPP#():3)?#@#@GCLLL((!"23"7>>E3GWX[G\C]C]abCbCb',->s-CADqD-I'J'J 278I#8NqtRSt8T2U2U3//',->s-C'D'D #():3)?#@#@GCLLe##!**,,Ns ( EE+r)rrrhs` r! wrapped_fnz#capture_outputs..wrapped_fns: t= = = = =  = ~r r)rrhrs ` r!capture_outputsrs@AAAAAF z$ r )r8r9r:r;rrr<r=)rMr9rNr;rOrPr<r=)N)rZrrSrr<r=)rZrr<r=)r __future__r threading contextvarsr dataclassesr functoolsrtypingr import_utilsr r rr rTrr]rr#rArLrVrWLockrfrgrrr r!rs #"""""""""""!!!!!! <<<<<<<<1000000  :""""""" ""&*&*&*&*&*&*&*&*T*)*