bi %ddlmZddlmZddlmZmZddlmZddl m Z ddl m Z erdd l Z erdd lmZejeZGd d e Zd S) ) annotations) TYPE_CHECKING)is_torch_availablelogging) SinqConfig) HfQuantizer)get_module_from_nameN)PreTrainedModelceZdZUdZdZded<d fd Zd!d Zed!d Z d Z d"dZ d#dZ d$dZ d%dZdZdZ d&d'dZd(dZxZS))SinqHfQuantizera HF v5 quantizer for SINQ. Modes: - method="sinq" (default): * weight-only SINQ * param-level ConversionOps (`SinqQuantize`) during load for pure language models (each Linear.weight is turned into a SINQLinear module) * module-level quantization after load for multimodal models - method="asinq": * A-SINQ (activation-aware) SINQ quantization Tbool requires_parameters_quantizationquantization_configrc Xtj|fi|d|_d|_dS)NF)super__init___normalized_device_str_do_param_level_sinq)selfrkwargs __class__s a/root/projects/butler/venv/lib/python3.11/site-packages/transformers/quantizers/quantizer_sinq.pyrzSinqHfQuantizer.__init__1s9,7777726#*/!!!returncdSNTrs ris_serializablezSinqHfQuantizer.is_serializable7strcdSrrr s r is_trainablezSinqHfQuantizer.is_trainable:strc|atjr!dtji}nddi}td|d|S)Ncpuz:The device_map was not initialized. Setting device_map to zJ. If you want to use the model for inference, please set device_map='auto')torchcuda is_availablecurrent_deviceloggerinfo)r device_maps rupdate_device_mapz!SinqHfQuantizer.update_device_map>sz  z&&(( ) %*";";"="=>  %[ KK[)3[[[    rdtype torch.dtypec0| tj}||_|SN)r'bfloat16r/)rr/s r update_dtypezSinqHfQuantizer.update_dtypeKs =NE  rNonecddlm}|stdtjst d|d}t|trTt| }t|dkr tdt|d|jjd kr|jst'd dSdS) Nr)is_sinq_availablezMThe 'sinq' package is not installed. Please install it with: pip install sinqzNo CUDA device is available. Quantization and inference will run on the CPU. Please note that this will significantly slow down inference speed and increase quantization time.r-r zkSinqHfQuantizer: multi-GPU device_map detected, but SINQ currently supports only a single CUDA device. Got z. Please use device_map=None.asinqzYou are using `method='asinq'` in the quantization config. Right now the calibrated version of SINQ is not supported in Hugging Face, please refer and use the official SINQ repository `to quantize a model with this method. )utilsr7 ImportErrorr'r(r)r+warningget isinstancedictsetvalueslen RuntimeErrorsortedrmethod pre_quantized ValueError)rargsrr7r-device_map_valuess rvalidate_environmentz$SinqHfQuantizer.validate_environmentQs0------  "" omnn nz&&((  NNB   ZZ -- j$ ' '  #J$5$5$7$7 8 8 $%%))"\#)*;#<#<\\\  # *g 5 5d>P 5:  6 5 5 5rcfgr>c ddlm}|j}|t|j|jt|jndddddt |j|S)zI Build the dict that SINQLinear expects as quant_config. r)sinq_base_quant_configNFr )nbits group_size quant_zero quant_scale view_as_floataxis tiling_moderD)sinq.sinqlinear_hfrLrDintrMrNstrrS)rrJsinq_base_quant_config_fnrDs r_build_sinq_quant_dictz&SinqHfQuantizer._build_sinq_quant_dictmsx [ZZZZZ((ci...1n.Hs3>***dCO,,    rmodelr param_namerVc ddlm}|jrdS|jjdkrdS|jsdSt ||\}}|dkrdSt||}t|dd}|o| } | S)a- Called per-parameter to decide whether to run `SinqQuantize` on it. - If `self.pre_quantized`, we do *not* quantize again (handled by SinqDeserialize instead). - For method="asinq": return False (ASINQ is not supported in Hugging Face). - For method="sinq": True only for SINQLinear.weight not in modules_to_not_convert. Note: After _process_model_before_weight_loading(), the modules are already SINQLinear, not nn.Linear. We check for SINQLinear modules that are not yet quantized (ready=False). r) SINQLinearFr8weightreadyT) rTr\rErrDrr r=getattr) rrYrZrr\module tensor_nameis_sinqis_readyresults rparam_needs_quantizationz(SinqHfQuantizer.param_needs_quantizations 211111   5  # *g 5 55( 525*EE ( " "5VZ0067D11)\ rc$ddlm}||S)z Return the ConversionOps used for param-level quantization (Sinq). The actual SINQLinear construction is in integrations/sinq.py. r) SinqQuantize)integrations.sinqrg)rrgs rget_quantize_opsz SinqHfQuantizer.get_quantize_opss& 544444|D!!!rcdddlm}|jr"ddlm}|gddg||ggSgS)a4 If `pre_quantized=True`, interpret a checkpoint produced by SINQLinear.state_dict: .W_q .bias .meta via a WeightConverter + SinqDeserialize so that we reconstruct a SINQLinear module instead of a plain nn.Linear. r)WeightConverter)SinqDeserialize)z.W_qz.metaz.biasz.weight)source_patternstarget_patterns operations)core_model_loadingrkrErhrl)rrkrls rget_weight_conversionsz&SinqHfQuantizer.get_weight_conversionss 988888    ; ; ; ; ; ; %%% &/K / 5 56   rNkeep_in_fp32_moduleslist[str] | Nonec Dddlm}|||jjpg||_|jjdko|j |_|jrdn||j}t|trZtt| d}t|trd|}n2t|}n"t jrdnd}|||j||j||j }dS) a Called on meta-initialized model, before loading any weights. For SINQ, we replace nn.Linear modules with empty SINQLinear modules here. The actual quantization happens later in SinqQuantize.convert() when weights are loaded. r)replace_with_sinq_linearsinqNrzcuda:zcuda:0r&)modules_to_not_convert quant_config compute_dtypedevicerE)rhruget_modules_to_not_convertrrwrDrErrXr=r>nextiterr@rUrVr'r(r)r/) rrYr-rrrrusinq_quant_dict first_device device_strs r$_process_model_before_weight_loadingz4SinqHfQuantizer._process_model_before_weight_loadings? A@@@@@&*&E&E D,CIrL`' ' # %)$<$Cv$M$hVZVhRh!"&"4o$$$:U:UVZVn:o:o j$ ' ' JZ%6%6%8%8 9 91==L,,, /3\33  .. %*Z%<%<%>%>IEJ(( #'#>(*,    rc &ddlm}||S)aq Called after *all* weights have been loaded. For SINQ: 1. Move non-SINQLinear modules to GPU (embeddings, norms, lm_head, etc.) - SINQLinear modules already have GemLite buffers on GPU - We skip moving SINQLinear's W_q/meta to avoid memory duplication 2. Patch HF save/load methods for SINQ serialization r)patch_hf_pretrained_io) sinq.hf_ior)rrYrrs r#_process_model_after_weight_loadingz3SinqHfQuantizer._process_model_after_weight_loadings, 655555     r)rr)rr)r/r0rr0)rr5)rJrrr>)rYr rZrVrrr2)rYr rrrs)rYr )__name__ __module__ __qualname____doc__r__annotations__rr!propertyr#r.r4rIrXrerirqrr __classcell__)rs@rrr!sC  .2$1111000000 X    8    $    D"""B26 ) ) ) ) ) Vrr) __future__rtypingrr9rrutils.quantization_configrbaser quantizers_utilsr r'modeling_utilsr get_loggerrr+rrrrrs#""""" ////////222222222222LLL1000000  H % %dddddkdddddr