bi0(`ddlmZmZerddlZdZdZdZGddeZdS) )PushToHubMixinis_torch_availableNcd} t|}n #t$rtd|wxYw|jj}|dkrt jjdS|S)z Infers the device type from the model parameters. Args: model: The model instance. Returns: The device type. z { "RMSNorm": { "cuda": "kernels-community/layer_norm:LlamaRMSNorm", ... }, ... } zPCannot determine model device, please provide a device to the mapping. Example: cudaNrocm) next parameters StopIteration ValueErrordevicetypetorchversionhip)modelEXAMPLE_MAPPINGparamdev_types [/root/projects/butler/venv/lib/python3.11/site-packages/transformers/utils/kernel_config.py infer_devicers O U%%''((     p_n p p    | H6 =  (6 Os !&Acddlm}|dvrtd||dd}|dd}|||||ii||<dS)NrLayerRepositoryrrxpunpu5Only cuda, rocm, xpu and npu devices supported, got: :)repo_id layer_name)kernelsrr split)r"r repo_namemodecompatible_mappingrrepo_layer_namer!s radd_to_mappingr)9s'''''' 333YQWYYZZZooc**1-Oooc""1%G //* &z"""c(ddlm}ddlm}|dvrt d||dd}|dd}|dd } |||||| | ii||<dS) Nr)Path)LocalLayerRepositoryrrrr /) repo_path package_namer")pathlibr,r#r-r r$) r"r r%r&r'r,r-r(r0repo_package_names radd_to_mapping_localr4Js,,,,,, 333YQWYYZZZooc**1-O$$Q'I!,,R0 &&$y//.* &z"""r*c:eZdZdZidfdZd dZdZdZd dZdS) KernelConfigze Kernel configuration class. This class is used to configure the kernel mapping for a model. Fc0||_i|_||_dSN)kernel_mappingregistered_layer_namesuse_local_kernel)selfr9r;s r__init__zKernelConfig.__init__ds ,&(# 0r*NcFddlm}||||||ii|j|<dS)Nrr)r!r"revision)r#rr9)r<r!registered_namer"r r&r?rs r update_kernelzKernelConfig.update_kernelisR++++++ oo#)%0 O,,,r*cx|D]$\}}t|dr|j|j|<%dS)Nkernel_layer_name) named_moduleshasattrrCr:)r<rnamemodules rstore_registered_layer_namesz)KernelConfig.store_registered_layer_namesvsT!//11 M MLD&v233 M4:4L+D1 M Mr*cd}||t|jts't d|dt |j|jD]\}}||jvrt d|dt|trd|vsd|vrt d|d |ft|trc|D]M\}}|d vrt d |t|trd|vsd|vrt d|d |Nt d |d|d S)aJ Validates the kernel_mapping to ensure that: 1. Each layer_name in the mapping is registered in the model (i.e., the model contains a module with a matching kernel_layer_name). 2. Each kernel value is either a string of the form 'org/repo:layer_name' or a dict mapping device types ("cuda", "rocm", "xpu", "npu") to such strings. 3. Each device key in a dict is one of "cuda", "rocm", "xpu", or "npu". 4. Each repo_name is a valid repository and layer name in the format 'org/repo:layer_name' (i.e., a string containing both a slash and a colon). 5. If a local path is detected, it should be in the format '/abs/path:layer_name'. The absolute path must include the `package_name`, like "/home/user/layer_norm". Args: model: The model instance whose modules are checked for registered kernel_layer_name attributes. Raises: ValueError: If a layer_name is not registered in the model, if a device is not supported, or if a repo_name is not a valid 'org/repo:layer_name' string. a For single device form remote { "RMSNorm": "kernels-community/layer_norm:LlamaRMSNorm", ... }, For multiple devices form remote { "RMSNorm": { "cuda": "kernels-community/layer_norm:LlamaRMSNorm", "rocm": "kernels-community/layer_norm:LlamaRMSNorm", ... }, ... } For single device form local { "RMSNorm": "/abs/path:LlamaRMSNorm", ... }, For multiple devices form local { "RMSNorm": { "cuda": "/abs/path:LlamaRMSNorm", "rocm": "/abs/path:LlamaRMSNorm", ... }, ... } z7Kernel mapping must be a dict of the following format: z, got: zLayer z` is not registered in the model, please register it first using register_kernel_forward_from_hubr.rzKernel mapping for 'zl' must be a valid repo name with a layer name (e.g., 'org/repo:layer_name' or '/abs/path:layer_name'), got: rrz'Kernel mapping must follow the format: N) rH isinstancer9dictr ritemsr:valuesstr)r<rMAPPING_FORMATr"kernelr r%s rsanitize_kernel_mappingz$KernelConfig.sanitize_kernel_mapping{s, # H ))%000$-t44 |.||aefjfyazaz|| #'"5";";"="= l l J!!j!jbh!j!jkkk- l lr*c ddlm}i}t|}|jD]\}}|jr|jn|j}|r ||jz}t|tr1|} |j st||| ||Xt||| ||lt|trO|D]:\} } | |kr |j st|| | ||'t|| | ||;||_dS)aW Transforms a simple kernel_mapping of the form: { "RMSNorm": "kernels-community/layer_norm:LlamaRMSNorm", ... }, or for local path: { "RMSNorm": "/home/user/liger_kernels:LigerRMSNorm", ... }, into a nested mapping: { "RMSNorm": { "cuda": { Mode.INFERENCE: LayerRepository( repo_id="kernels-community/layer_norm", layer_name="LlamaRMSNorm", ) } } } or for local path: { "RMSNorm": { "cuda": { Mode.INFERENCE: LocalLayerRepository( repo_path=Path("/home/user/liger_kernels"), package_name="liger_kernels", layer_name="LigerRMSNorm", ) } } } that's compatible with the kernels library. The device is inferred from the model's parameters if not provided. The Mode is inferred from the model's training state. r)ModeN)r#rSrr9rLtrainingTRAINING INFERENCE TORCH_COMPILErJrNr;r)r4rK) r<rcompilerSr'current_devicer"rPr&r%r s rcreate_compatible_mappingz&KernelConfig.create_compatible_mappings_b !     %e,,"&"5";";"="= f f J$)NF4==D 1d00&#&& f" ,j":~y$Pbcccc(^YPTVhiiiiFD)) f)/ff%FI// 0f&z69dL^____,ZDRdeeee0r*r8)F) __name__ __module__ __qualname____doc__r=rArHrQrZr*rr6r6_s')51111     MMM QlQlQlfJ1J1J1J1J1J1r*r6)utilsrrrrr)r4r6r_r*rras76666666LLL   F"*y1y1y1y1y1>y1y1y1y1y1r*