bi*Z"UdZddlmZddlZddlZddlmZddlm Z ddl m Z m Z ddl Z ddlmZddl mZd d lmZd d lmZd d lmZd d lmZmZd dlmZmZmZmZmZm Z m!Z!m"Z"m#Z#e r d dl$m%Z%d dlm&Z&ej'e(Z)e GddZ*d>dZ+ee*ge,e e-e.e fffZ/d?dZ0 d@dAd)Z1dBd+Z2dBd,Z3dBd-Z4dBd.Z5dBd/Z6dBd0Z7dBd1Z8dBd2Z9dBd3Z:dBd4Z;dBd5ZdBd8Z?dBd9Z@dBd:ZAdBd;ZBejCejDejEejFejGejHejIejJejKejLejMejNejOejPejQgZRejSejTejUejVejWejXgZYejZej[gZ\ej]ej^gZ_ej`ejaejbgZcejde2ejee3ejfe3ejge4ejhe5ejie6ejje8ejke9ejle:ejme;ejne?ejoeBejpe>ejqe>ie-reRe7e-reYe<e-re\e=e-re_e@e-receAZsdz-_setup_low_rank_optimizer..s(XXX41aqDW?W?WQ?W?W?Wr*c"g|] \}}|v | Sr)r)rPs r+rUz-_setup_low_rank_optimizer..s(```tq!1L_C_C_C_C_C_r*paramsr z Layerwise z( does not support gradient accumulation!c|j6||dSdS)N)gradstep zero_grad)paramoptimizer_dicts r+optimizer_hookz1_setup_low_rank_optimizer..optimizer_hooksGz%u%**,,,u%//11111&%r*r])lowerendswith parallel_moder DISTRIBUTEDNotImplementedErroroptim_target_modules ValueErrorr=listrE TypeErrorr4 named_modulesr r Linearloggerwarningappendlennamed_parametersgradient_accumulation_steps parameters requires_grad"register_post_accumulate_grad_hookr update)rrrDrFrGr rH is_layerwise optimizer_cls all_linear module_namemoduletarget_module_existsis_regex target_paramsnon_target_params param_groupsr\r^r]rTs @@r+_setup_low_rank_optimizerr~es"''))22;??Ld*l.FFFKaF!"b"b"b"bccc%n5M  (h^hhhiii d/$ = =  0- 0 0    }^>^^^___ 4,c22rt7P7X7XY\^a7b7bfr7r$2244 < < V)C  %{D* * * &h&"),, # tH t+rr^rrrsss # J  "";#:;;;; 1$$ggg$JcggghhhXXXX5#9#9#;#;XXXM````u'='='?'?``` $% =1L1L D  +q 0 0b.bbbcc c& ] ]E$1MHug3F2G$\$\K[$\$\N5 ! !" m mE$1MHug3V3V2W$l$l[k$l$lN5 ! ! 2 2 2 2 2 %%'' I IE" I88HHH/ !1> BCCCX|4555 * **r*ctxcX|jdddt|jfS)zGet Adafactor optimizer.Fscale_parameter relative_step)r rsr rs r+_get_adafactorrs/EE R RSSS c* **r*cddlm}|j|j|jjtjkr|jddi||jfS)z/Get PyTorch AdamW optimizer (regular or fused).rAdamWfusedT) torch.optimrr rsr!rrArADAMW_TORCH_FUSEDrrs r+_get_adamw_torchrsh!!!!!!000 x~999 ##WdO444 #& &&r*c ddlm}|j|j||jfS#t $rt dwxYw)z'Get Torch XLA syncfree AdamW optimizer.rrz7Trainer failed to import syncfree AdamW from torch_xla.)torch_xla.amp.syncfreerr rsr! ImportErrorrers r+_get_adamw_torch_xlarsoT000000 ##CO444c*** TTTRSSST -0A c ddlm}|j|j||jfS#t $rt dwxYw)zGet NPU Fused AdamW optimizer.r) NpuFusedAdamWz3Trainer failed to import FusedAdamW from torch_npu.)torch_npu.optimrr rsr!rre)rrs r+_get_adamw_torch_npu_fusedrsoP111111 ##CO444c222 PPPNOOOPrc ddlm}|j|j||jfS#t $rt dwxYw)zGet Apex Fused Adam optimizer.r) FusedAdamzFTrainer tried to instantiate apex FusedAdam but apex is not installed!)apex.optimizersrr rsr!rre)rrs r+_get_adamw_apex_fusedrsoc------ ##CO444#... cccabbbcrctstdddlm}m}m}|jj}d|v}d|vrdnd}d}|j}d |vr|}nod |vr|}d |jj |jj fi}nMd |vr |}|j }n>d |vr9ddlm } | }t|j d|jj t|j d|jj t|j ddft|j ddt|j d|jjd}d|j vrt!|j d|d<d|j vrt!|j d|d<d|i} d |vr|| d<|j||j| ||jfS)z;Get bitsandbytes optimizer (AdamW, Lion, RMSprop variants).ziYou need to install `bitsandbytes` in order to use bitsandbytes optimizers: `pip install -U bitsandbytes`r)rLionRMSproppaged8bit Nadamlionbetasrmspropademamix)AdEMAMixbeta1beta2beta3gH.?alphag@eps)rrrt_alphat_beta3 optim_bitsis_paged)rrbitsandbytes.optimrrrrrAr! adam_beta1 adam_beta2r#rfloatget adam_epsilonintr rs) rrrr optim_namerrruadditional_optim_kwargsr bnb_kwargss r+_get_bitsandbytes_optimizerrsV $ & &  w   8777777777J*$H **JM!o  :   #*SX-@#(BU,V"W j  "%. z ! !////// cn((#(2EFFGGcn((#(2EFFGGcn((&99:: 3>--gs;;<<++E383HIIJJ# #   & &14S^I5N1O1O #I .  & &14S^I5N1O1O #I . +J ""!) : 7888 +++ #. ..r*c , ddlm}|j|j|jt |jddtt|jddtt|jddtt|jdd d ||jfS#t$rtd wxYw) z!Get AnyPrecision AdamW optimizer.r)AnyPrecisionAdamWuse_kahan_summationFalsemomentum_dtypefloat32variance_dtypecompensation_buffer_dtypebfloat16)rrrrz4Please install https://github.com/pytorch/torchdistx) torchdistx.optimizersrr rsr!rr#rgetattrr@rre)rrs r+_get_adamw_anyprecisionr&sQ;;;;;; ##CO444 ##'01C1CDY[b1c1c'd'd")%1C1CDTV_1`1`"a"a")%1C1CDTV_1`1`"a"a-43>--.I:VV..    !#"666 QQQOPPPQs C6C99Dc2tjj|jfS)zGet SGD optimizer.)r@rASGDr rs r+_get_sgdr;s ;?C0 00r*c2tjj|jfS)zGet Adagrad optimizer.)r@rAAdagradr rs r+ _get_adagradr@ ;  4 44r*c2tjj|jfS)zGet RMSprop optimizer.)r@rArr rs r+ _get_rmsproprErr*c tstdddlm}m}m}t j|t j|t j |t j |t j |t j |i}t|jddt|jddt!|jdd |jd d d }t#|j|j|jj|||j\}}|jjt j kr|d d d||fS)zGet GaLore optimizer.zYou need to install `galore_torch` in order to use GaLore optimizers. Install it with `pip install git+https://github.com/jiaweizzhao/GaLore`r)GaLoreAdafactor GaLoreAdamWGaLoreAdamW8bitrankupdate_proj_gapscaleg? proj_typestd)rrrrFr)rr galore_torchrrrr GALORE_ADAMWGALORE_ADAMW_8BITGALORE_ADAFACTORGALORE_ADAMW_LAYERWISEGALORE_ADAMW_8BIT_LAYERWISEGALORE_ADAFACTOR_LAYERWISErr#poprr~rrrAr rs)rrrrrFgalore_optim_kwargsrur s r+_get_galore_optimizerrJsn $ & &  V   KJJJJJJJJJ #[(/'-{2O1? CN&&vs3344s~112CSIIJJs~))'48899^'' U;; '@ #)SX^->@SUXUi''#M# x~888EE R RSSS * **r*c tstdddlm}tj|tj|i}t|j dd|j dd|j dd t|j d d t|j d d |j ddd}| |j t|j|j|jj|||jS)zGet Apollo optimizer.zYou need to install `apollo_torch` in order to use APOLLO optimizers. Install it with `pip install git+https://github.com/zhuhanqing/APOLLO`r) APOLLOAdamWrrprojrandom scale_typechannelrrr?rr)rrrrrr)rr apollo_torchrr APOLLO_ADAMWAPOLLO_ADAMW_LAYERWISErr#rrrsr!r~rrrAr )rrrFapollo_optim_kwargss r+_get_apollo_optimizerrksA $ & &  U   )((((( #[-{ CN&&vs3344""6844n((yAAs~112CSIIJJs~))'37788^'' U;; s/// $ #)SX^->@SUXUi  r*ctstd|jtdddlm}m}d|jjvr|n|}|j d|ji||j fS)zGet LOMO optimizer.zjYou need to install `lomo_optim` in order to use LOMO optimizers. Install it with `pip install lomo-optim`NzMYou need to pass a `model` in order to correctly initialize a LOMO optimizer.r)AdaLomoLomoadar) rrrre lomo_optimrrrrAr rs)rrrrus r+_get_lomo_optimizerrs     7    yhiii(((((((($66GGDM#) 4555 #. ..r*c tstdddlm}|jt |jddt |jddt |jdd t |jd d t |jd d d ||jfS)zGet GrokAdamW optimizer.z5Please install grokadamw with `pip install grokadamw`r) GrokAdamW alpha_initg\(\?lamb@gammag?grokking_signal_decay_rategradient_clippingr)rrrrr) rre grokadamwrr rsrr#r)rrs r+_get_grokadamwrs ! # #RPQQQ###### 2 2< F FGG#.,,VS99::3>--gs;;<<*/0B0BC_ad0e0e*f*f!&s~'9'9:Ms'S'S!T!T    c* **r*c ltrGtjtjdtjdkrt dtjtjdtjdkrt dtjtjdtjdkr dd lm}m}ndd l m}m}|j j tj kr|}n|}|j|jd d t%|jd d d|j|j||jfS)z%Get TorchAO 4-bit or 8-bit optimizer.torchaoz0.4.0zYou need to have `torchao>=0.4.0` in order to use torch 4-bit optimizers. Install it with `pip install torchao` or follow the instructions here: https://github.com/pytorch/aor@z2.4zYou need to have `torch>2.4` in order to use torch 4-bit optimizers. Install it with `pip install --upgrade torch` it is available on pipy. Otherwise, you need to install torch nightly.z0.11.0r) AdamW4bit AdamW8bit block_sizebf16_stochastic_roundr)rr)rrparse importlibmetadatar torchao.optimrrtorchao.prototype.low_bit_optimrrArADAMW_TORCH_4BITr rsr#rrr!)rrrrus r+_get_torchao_optimizerrs  ! ! W]93E3M3Mi3X3X%Y%Y\c\ijq\r\r%r%r ,   }Y'//8899W]5=Q=QQQ <   }Y'// ::;;w}X?V?VVV666666666HHHHHHHH x~888! ! .,,\3??%.s~/A/ABY[b/c/c%d%d   000 #. ..r*c tstdddlm}m}i}d}|jjtjkr0tdstdddlm }|}|j }d}nP|jjtj kr |}|j }n,|jjtj kr|}ntd |jj|d <|r|jj|d <|t#|jd d t#|jddd|j|||jfS)zGet ScheduleFree optimizer.zwYou need to install `schedulefree` in order to use schedulefree optimizers. Install it with `pip install schedulefree.`r)AdamWScheduleFreeSGDScheduleFreeTz1.4.0zYou need to install `schedulefree>=1.4.0` in order to use RAdamScheduleFree optimizer. Install it with `pip install schedulefree.`)RAdamScheduleFreeFzInvalid schedulefree optimizer weight_decay warmup_stepsweight_lr_powerrrg)rr)rr schedulefreerrrrArSCHEDULE_FREE_RADAMrr!SCHEDULE_FREE_ADAMWSCHEDULE_FREE_SGDrerrrsrr#rr )rrrrrequire_warmuprrus r+_get_schedule_free_optimizerrs $ & &  :   @??????? N x~;;;(11 >  322222) "%/ >= = =) "%/ >; ; ;' 9:::.1h.CN+H25(2G/""$S^%7%78I3%O%OPPs~))#s3344    7888 #. ..r*ctstdddlm}|jdd}|t |}|jdd}|t|}|jj |j d<t|jdd ||d }|j |j |j |||j fS) z,Get StableAdamW optimizer from torch-optimi.zwYou need to install `torch-optimi` in order to use stable_adamw optimizers. Install it with `pip install torch-optimi`.r) StableAdamWmax_lrN kahan_sumr decouple_lrF)r"r r!) rroptimirr#rrr;rrr!r rs)rrr r!stable_adamw_kwargss r+_get_stable_adamwr%s  $ & &  :   #""""" ^  $ / /F v"";55IOO &)h&;CON#CN..}eDDEE 000 3444 , ,,r*zdict[str, OptimizerHandler]_OPTIMIZER_HANDLERS)r,r-r.r")r:rr.r;)T)rrrrrDrErFrrGrr rrHr;r.rI)rrr.rI)ur' __future__rimportlib.metadatar loggingcollections.abcr dataclassesrtypingrrr@ packagingrr optimizationr trainer_pt_utilsr trainer_utilsr training_argsrrutilsrrrrrrrrrmodeling_utilsrr getLoggerr$rjrr9tupledictrEOptimizerHandlerrCr~rrrrrrrrrrrrrrrrr% ADAMW_BNB ADAMW_8BIT PAGED_ADAMWPAGED_ADAMW_8BITADEMAMIX ADEMAMIX_8BITPAGED_ADEMAMIXPAGED_ADEMAMIX_8BITLION LION_8BIT PAGED_LIONPAGED_LION_8BIT RMSPROP_BNB RMSPROP_8BIT RMSPROP_32BIT_BITSANDBYTES_OPTIMIZERSrrrrrr_GALORE_OPTIMIZERSrr_APOLLO_OPTIMIZERSrADAMW_TORCH_8BIT_TORCHAO_OPTIMIZERSrrr_SCHEDULE_FREE_OPTIMIZERS ADAFACTOR ADAMW_TORCHrADAMW_TORCH_XLAADAMW_TORCH_NPU_FUSEDADAMW_APEX_FUSEDADAMW_ANYPRECISIONrADAGRADRMSPROP GROKADAMW STABLE_ADAMWLOMOADALOMOfromkeysr&r(r)r*r+rZs#"""""$$$$$$!!!!!!%%%%%%%% ######55555555555577777777                      1//////000000  8 $ $  -.c4S>6I0JJK8$(V+V+V+V+V+|++++ ''''TTTTPPPPcccc////////dQQQQ*1111 5555 5555 ++++B:////$++++& / / / /F)/)/)/)/X----D# !&" &$#).- ) ## &&$n 0$&6"$8(*D#%:%'>LLn!2,/4 mm,.IJJ4  mm&(=>>!4" mm&(=>>#4$ mm')?@@%4& mm-/KLL'4r*