bi&|ddlZddlZddlZddlmZddlmZmZmZm Z e j e Z defdZ dZd dZdS) N) DataLoader) WEIGHTS_NAMEPushToHubMixinis_torch_xla_availablelogging dataloaderctreddlmcm}t ||js Jdddlmcm}|| d}||j d<|S|S)NrzPThe dataloader must be a `torch_xla.distributed.parallel_loader.MpDeviceLoader`.)fsdpNinput_sharding) r%torch_xla.distributed.parallel_loader distributedparallel_loader isinstanceMpDeviceLoadertorch_xla.distributed.spmdspmd ShardingSpecget_global_mesh_parallel_loader_kwargs)r plxs sharding_specs X/root/projects/butler/venv/lib/python3.11/site-packages/transformers/integrations/tpu.pytpu_spmd_dataloaderrs:::::::::*b&788  ^   0////////(:(:(<(= 2.0._no_split_modulestransformer_layer_cls_to_wrapmin_num_params)r&z@Could not find the transformer layer class to wrap in the model.)transformer_layer_clsxla_fsdp_grad_ckptzX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.Fc:sn}||g|Ri|SN)margskwargs target_clsFSDPFSDPv2r is_fsdp_xla_v2_enableds rauto_wrapper_callablez2wrap_model_xla_fsdp..auto_wrapper_callablets=%;GJ://22DTDDDVDD Drc ddlm}d}t|tjr|}n5t|t r |d}nt||r|j}|td||ddS)Nr)CausalLMOutputWithPastrzASomething went wrong, the output of the model shouldn't be `None`)r NN) modeling_outputsr5rtorchTensortuplelogits ValueError mark_sharding)outputmeshr5 real_outputrs r shard_outputz)wrap_model_xla_fsdp..shard_output{s A A A A A AK&%,// ,$ FE** ,$Qi F$:;; ,$m " !deee   [$0D E E E E Er)r@auto_wrap_policyr3)rAr3cN|jdi|}|r|S)Nr+)step mark_step) optimizerbarrieroptimizer_argslossxms rpatched_optimizer_stepz3wrap_model_xla_fsdp..patched_optimizer_steps4y~////   LLNNN r)torch_xla.core.xla_modelcore xla_modelrrrtrainer_pt_utilsrtorch_xla.distributed.fsdprr torch_xla.distributed.fsdp.wrapr!r"7torch_xla.experimental.spmd_fully_sharded_data_parallelr# ImportErrorgetattr fsdp_configget functoolspartialset Exceptionaddxla_fsdp_configconfig use_cachelogger warning_onceoptimizer_step)modelr-r2rr!r"rAr3%default_transformer_cls_names_to_wrap"fsdp_transformer_layer_cls_to_wraptransformer_cls_to_wrap layer_classtransformer_cls fsdp_kwargsr@rJr0r1r rIrs ` @@@@@rwrap_model_xla_fsdprh.s,*))))))))+++++++++====== hRRRRRR@@@@@@        "        hhhfgggh ,3E;NPT,U,U))-)9)=)=')N**& ()A--$, '8HIY8Z    , 7"%%%= = =K88 LLO& bccc'++O<<<<$, ("9    &K ,- E < ! +   j   &+EL " E E E E E E E E   F F F F F %-"7      -"7     38 /B Ls =Ac @ddlmcm}||n|j}td|||drItj |dtj |tj |dtf}|d |rq||d }tj |d |jd |jd t(} | || d|d|jrddlm} | tj |ddt(d\} } |jj}||} t5| |r| || ntd| | tj |t(nFt5||st5|||rV||||j||ntd||}| |tj |t(nB|||j||||jr||dSdSdS)a Saves a model checkpoint on TPU/XLA devices. Handles FSDP v1 sharded checkpoints (with consolidation on master), as well as standard XLA model saving via `save_pretrained` or `xm.save`. Args: model (`torch.nn.Module`): The model to save. args (`TrainingArguments`): The training arguments. accelerator (`Accelerator`): The accelerator instance. processing_class: The processing class (tokenizer/processor) to save alongside the model. is_fsdp_xla_v1_enabled (`bool`): Whether FSDP XLA v1 is enabled. output_dir (`str`, *optional*): The directory to save to. Defaults to `args.output_dir`. rNzSaving model checkpoint to F)localT)exist_okztraining_args.binsaving_checkpoint)rashard_metadatarankz-of--) master_onlysave_full_checkpoints)%consolidate_sharded_model_checkpointsz rank*-of-*-) ckpt_prefix ckpt_suffix save_model) state_dictzETrainer.model is not a `PreTrainedModel`, only saving its state dict.)is_main_processrw)rKrLrM output_dirr^inforDis_master_ordinalosmakedirsr7savepathjoinr rendezvousrwget_shard_metadata process_index world_sizer should_saverOrrmodule unwrap_modelrsave_pretrained_maybe_convert_to_cpu)rar- acceleratorprocessing_classis_fsdp_xla_v1_enabledryrIsupported_classesckpt ckpt_pathrrfull_state_dict_unwrapped_modelrws rsave_tpu_checkpointrs*)))))))))54?J KK:j::;;;LLNNN %((H J.... 4j2EFFGGG()MM%&&&* %%''#6688  GLL-lD4F-l-lDO-l-l^j-l-lmm  iU333 -...   Q X X X X X X!F!FGLLR888,88 """ OQ L'E)66u==O/+<== Q// /WWWW cdddj,)O)OPPP 0 1 1 k..u557H I I H  $ $U + + ; ; $ 033E4D4D4F4FGG <     KK_ ` ` `11%2B2B2D2DEEJ GGJ Z F F G G G G   ,//0@0@0B0BCC    #(8#((44444$###rr*)rVr|r7torch.utils.datarutilsrrrr get_logger__name__r^rrhrr+rrrs ''''''QQQQQQQQQQQQ  H % %J&tttnJ5J5J5J5J5J5r