bidZddlZddlZddlZddlmZddlmZddlm Z ddl m Z ddl m ZddlmZddlmZmZdd lmZdd lmZdd lmZdd lmZmZdd lmZm Z m!Z!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0ddl1m2Z2m3Z3m4Z4e4j5e6Z7dZ8dZ9dZ:dZ;dZe3e*Gdde-Z?e?Z@dS)z Tokenization classes for fast tokenizers (provided by HuggingFace's tokenizers library). For slow (python) tokenizers see tokenization_utils.py N) defaultdict)Iterable)copyfile)Any)is_offline_mode) AddedToken processors)Encoding) Tokenizer)Decoder)BPEUnigram) BpeTrainerUnigramTrainerWordLevelTrainerWordPieceTrainer cached_file)convert_gguf_tokenizer)load_gguf_checkpoint)INIT_TOKENIZER_DOCSTRING BatchEncodingPreTokenizedInputPreTrainedTokenizerBase TextInputTruncationStrategygenerate_merges)PaddingStrategyadd_end_docstringsloggingztokenizer.jsonzspecial_tokens_map.jsonztokenizer_config.jsonztokenizer.modelzadded_tokens.jsonu tokenizer_object ([`tokenizers.Tokenizer`]): A [`tokenizers.Tokenizer`] object from 🤗 tokenizers to instantiate from. See [Using tokenizers from 🤗 tokenizers](../fast_tokenizers) for more information. tokenizer_file ([`str`]): A path to a local JSON file representing a previously serialized [`tokenizers.Tokenizer`] object from 🤗 tokenizers. )r r WordLevel WordPiece)tokenizer_file vocab_filec)eZdZdZeZdZdZedLdZ fdZ e de fdZ e de fdZdMd ed edzdeefd Zd Ze d Ze dZejdZejdZdZe defdZdeeeffdZe deeeffdZe deeeffdZe deeeffdZeZeZ deeeffdZ!de fdZ"defdZ#e de$fdZ%e de&fdZ' dNde(de dzde dzd e d!e d"e d#e d$e deeee)fe*e(ffd%Z+d&edefd'Z,d(ededzfd)Z-dLd*e*eezdefd+Z.dLd,e defd-Z/dLd.ee*ezd/e dee*ezfd0Z0dOd1ed,edzd2e de*efd3Z1d4e2d5e3d6ed7ed8edzd9edzf d:Z4dde2j5e3j6dd;ddddddddddddfd1e7e8ze*e7ze*e8zde dzde dzde dzd e d!e d"e d#e d$e d?e dzde9f(d@Z:dAe*edefdBZ; dPdCee*ezd/e dDe dzdefdEZ< dQd ee=j>zdFeedGfdHe dzd edzdeedGff dIZ? dRdJZ@e dSdKZAxZBS)TTokenizersBackendaQ Base class for all fast tokenizers (wrapping HuggingFace tokenizers library). Inherits from [`~tokenization_utils_base.PreTrainedTokenizerBase`]. Handles all the shared methods for tokenization and special tokens, as well as methods for downloading/caching/loading pretrained tokenizers, as well as adding tokens to the vocabulary. This class also contains the added tokens in a unified way on top of all tokenizers so we don't have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...). NFc t|}|dd}|Ltj|r-|t us d|jvs|rtj||d<|S|tj|rtj|j }t|d5}tj |}dddn #1swxYwY| di dd}|j8t|t r"t!t#t$|}n|jjd kr1|r.t|d t t$fr d |D}nx|jjd krd t)|D}nN|jjdks|jjdkr.t|t rdt)|D}||d<t+|dd} d| divr,| r*| jdkr|dd} d| D} | |d<|||d<|S| d} | d} | d}| d} t| t,rd| drOtj| r0ddlm} | | | \|d<|d<|St| t,rqtj| rQ| dr; ddlm}|| j|jfi|} ddlm}| |j}|t=|dr |jd-i|}n=#t@$r0}tB"d|jd|d Yd}~nd}~wwxYwt=|d!r |j#d-i|}nz#t@$rm}tB"d"| d#|d$dd%lm$}|| | d&'| \|d<|d<Yd}~nd}~wwxYw|S|At| t,r,tj| r | |d<|d}| At| t,r,tj| r | |d<|d} | |j|jjdkrt|trd(tJtLd)t t,ffd* gd+}tO}|D]+}||vr%|(||g,tS||,} | |d<|S).zs Build a `tokenizers.Tokenizer` backend from the available serialization files (tokenizer.json, sentencepiece models, tekken.json, vocab/merges). r$N__init__tokenizer_objectutf-8encodingmodelvocabrrc,g|]}t|S)tuple).0items e/root/projects/butler/venv/lib/python3.11/site-packages/transformers/tokenization_utils_tokenizers.py z>TokenizersBackend.convert_to_native_format..s;;;TU4[[;;;r"ci|]\}}|| Sr1r1r3itokens r5 z>TokenizersBackend.convert_to_native_format..sCCChaCCCr7r r#cTi|]%\}}t|tr|dn||&Sr) isinstancelistr9s r5r<z>TokenizersBackend.convert_to_native_format..s8pppS[STV[E4)@)@KU1XXeQpppr7mergescg|]H}t|tr"t|dnt|IS) )r?strr2split)r3merges r5r6z>TokenizersBackend.convert_to_native_format..sKrrrbgZs5K5K]% C 0 0111QVW\Q]Q]rrrr7post_processorr% merges_filez tekken.jsonr)MistralConverter)r%.model)SentencePieceExtractor)SLOW_TO_FAST_CONVERTERSconvert_from_spmz,Could not reorder vocab using converter for z due to z/. Falling back to raw SentencePiece extraction.convert_from_spm_modelz+Could not extract SentencePiece model from z$ using sentencepiece library due to z%. Falling back to TikToken extractor.)TikTokenConverterextra_special_tokens)r%rPvaluesreturncg}|D]b}|t|ttfr||@|t |c|SN)r?r@r2extendappendrD)rQ collectedval_iter_special_tokenss r5rYzHTokenizersBackend.convert_to_native_format.._iter_special_tokenss{') !33C{ !#e}553!(()=)=c)B)BCCCC!((S2222  r7) pad_token unk_token bos_token eos_token sep_token cls_token mask_tokenadditional_special_tokensrP) skip_tokensr1)*dictpopospathisfiler'__dict__ TokenizerFast from_filerGopenjsonloadgetr.r?r@mapr2__name__ enumerategetattrrDendswithconvert_slow_tokenizerrIextract_vocab_merges_from_modelrKextractrLhasattrrM ExceptionloggerwarningrNrOrrsetupdater)clstrust_remote_codekwargs local_kwargsfast_tokenizer_file processortokenizer_handletokenizer_jsonr/ model_typerAr%rHrIrKrLconverter_classerOspecial_tokens_keysrbkeyrYs @r5convert_to_native_formatz*TokenizersBackend.convert_to_native_formatds[F|| *../?FF +233 ,)))Zs|-K-KO`-K/--gr:::: :zObfkOkOk'0:rrkqrrr)/ X&$1: -. !%%l33 "&&}55   ))!!(++ j# & & :+>+>}+M+M RTRYR`R`akRlRl @ @ @ @ @ @>*+E+E *J]J]^fJgJg  >JJJJJJI55jAAI#)ddWcdd  OOOOOO&=&A&A#,&O&OO&2wPb7c7c2'G'G'W'W,'W'W  NNPs|PP]^PPP 3 899N#=3#=#M#M #M#ML > > >:*::rs:::FEEEEE@Q@Q) @P@PQg@h@hAAA11*==> W%|H'='='='='='= >  =Z C88=RW^^J=W=W=$.L ! )E >jc::>rw~~k?Z?Z>%0L "!(+F >ci3 8Je8S8SXbchjnXoXo8S !Xc] !tCy ! ! ! ! ! ! # # # %(EEK* R R,&&&&';';\#=N. s%CkCkCkYQPVQRTUAqCkCkCkr7T)r/rAfuse_unkdropoutunk_id)r/ra9Couldn't instantiate the backend tokenizer from one of: (1) a `tokenizers` library serialization file, (2) a slow tokenizer instance to convert or (3) an equivalent slow tokenizer class to instantiate and convert. You need to have sentencepiece or tiktoken installed to convert a slow tokenizer to a fast one.r\zr]zz3The backend tokenizer is not correctly initialized. max_lengthtruncation_side directionstridetruncation_strategystrategyrZpad_token_type_id pad_type_id padding_sidelengthpad_to_multiple_ofbackend tokenizers add_bos_token add_eos_tokenrGcFh|]}tt|Sr1hashreprr3r;s r5 z-TokenizersBackend.__init__..Ks&$^$^$^5T$u++%6%6$^$^$^r7cVg|]%\}}tt|v#|&Sr1r)r3indexr;added_tokens_decoder_hashs r5r6z.TokenizersBackend.__init__..LsA   uDKK  (AAA AAAr7c|dSNrr1)xs r5z,TokenizersBackend.__init__..Ns STUVSWr7rc,g|]}t|Sr1rDrs r5r6z.TokenizersBackend.__init__..Qs;b;b;b5CJJ;b;b;br7c0g|]}|t|Sr1r)r3ts r5r6z.TokenizersBackend.__init__..bs$WWW1UVWAWWWr7)speciali pre_tokenizerfix_mistral_regex) init_kwargsrr1):rdrncopydeepcopyrerfrgrirjrrrr|len _tokenizerr?rcrqr r@r2r ValueError setdefault truncationenable_truncation no_truncationpaddingenable_padding_add_bos_token_add_eos_tokenrG_should_update_post_processorsuperr)r%rsplit_special_tokensencode_special_tokensrsorteditemsadded_tokens_encoderkeys_special_tokens_maprQrDrV_extra_special_tokensrr add_tokensget_vocab_sizeNotImplementedErrorrr_patch_mistral_regexrupdate_post_processor) selfargsrr*rrrrr%r/rAfast_tokenizer gguf_path gguf_param architecturetokenizer_dictradditional_kwargs vocab_dict _truncation_paddingexplicit_bos_eos_in_kwargsrG tokens_to_addencoderspecial_token_valuer;tokensall_named_tokens vocab_sizer __class__s @r5r)zTokenizersBackend.__init__sv!::&8$??JJ{D11 $jj)94@@%zz*@"EE!::&8%@@ZZ --  7##H%%  '!]+;< > >   3[5L M M M M O ) ) + + +?*   *DO * 6 6X 6 6 6   k8K+@ A A A   18M3J K K K   nh{.C D D D   lHX,> ? ? ?   2H=Q4R S S S F " " ,F9 %4%>%[/U[B["$jj%@@$jj%@@#ZZ(8$?? ?> <-;DO *-G-q4?KimqKq*""6"""  !(DO 0040I-$^$^DD]$^$^$^!    &';'A'A'C'C X X X   t0557788;b;bTa;b;b;bb$(#;#B#B#D#D : : "*&''w66;NVc;c;c$$%8999/ , ,E5zz((U--G-G$$U+++ }   ! !FWW0H0O0O0Q0QWWW & % %eS))-&ud;;;EEz22- =-SZZ;K-K-K(,  e$$$$ (''' 7799JJ"   JJJ    74?OT#R#R#^ JJ{D ) ) )7d7 $$^T::!,"(**-@"A"A   DO  . X$/2PTX2X *  - )  & & ( ( ( ( ( ) )s[.. [=<[=rRcdS)NTr1rs r5is_fastzTokenizersBackend.is_faststr7cd|jvr]|jddr=t|dr+|jr$tj|jSdSdS)z `bool`: Whether or not the slow tokenizer can be saved. For a sentencepiece based slow tokenizer, this can only be `True` if the original `"sentencepiece.model"` was not deleted. r%rJFT)vocab_files_namesrsrwr%rerfrgrs r5can_save_slow_tokenizerz)TokenizersBackend.can_save_slow_tokenizersi 41 1 1d6L\6Z6c6cdl6m6m 1t\** 7t 7w~~do66654r7save_directoryfilename_prefixctj|s td|ddStj||r|dzndt dz}tj|jtj|krt|j||fS)NzVocabulary path (z) should be a directory-rr%) rerfisdirryerrorjoinVOCAB_FILES_NAMESabspathr%r)rrrout_vocab_files r5save_vocabularyz!TokenizersBackend.save_vocabularysw}}^,,  LLT^TTT U U U F oM_s222QbcoQpp   7??4? + +rw~/N/N N N T_n 5 5 5  r7c|j}|j}||jrtd|j}|j}||jr d|_dS|jr|dzndd|jrd|zdznd}||jrd|zd zndd |jrd|zd znd}g}|jr|||f|jr|||ftj ||| |j _ dS) ze Updates the underlying post processor with the current `bos_token` and `eos_token`. Nz)add_bos_token = True but bos_token = NoneFz:0 rz$A:0rCz:0z:1z $B:1)singlepairspecial_tokens) r\ bos_token_idrrr] eos_token_idrrVr TemplateProcessingrrG)rbosreosrrrrs r5rz'TokenizersBackend.update_post_processorskn( ;4-;HII In(  ;4-;!&D  F%)%7?S5[[Rww[_[mEucCiRVFVFVsuwwD0BJ39t++DDgkgyRBRUX[R[^bRbRb@BDD   7  ! !3 "5 6 6 6   7  ! !3 "5 6 6 6)3)F^* * * &&&r7c$t|ddS)NrFrrrs r5rzTokenizersBackend.add_eos_tokent-u555r7c$t|ddS)NrFr rs r5rzTokenizersBackend.add_bos_tokenr r7cft|d||dS)Nrobject __setattr__rrvalues r5rzTokenizersBackend.add_eos_token34!15999 ""$$$$$r7cft|d||dS)Nrr rs r5rzTokenizersBackend.add_bos_tokenrr7cg}|jD]j}|t|tr||0t|t r%|t|ddk|jD]g}t|tr||-t|t r%|t|ddh|r||dt|dds |j j | dSdS)a[ Post-initialization hook that runs after the tokenizer is fully set up. This is called by from_pretrained() after loading the tokenizer, which allows us to add any special tokens that may have been passed as AddedToken objects. Child classes should call super()._post_init() if they override this method. NTF)r normalized)rr) rrQr?rrVrDrrrrrrGr)rr token_valuer;s r5 _post_initzTokenizersBackend._post_initsl 3::<< ^ ^K"+z22 ^$$[1111K-- ^$$Z TV[%\%\%\]]]/ X XE%,, X$$U++++E3'' X$$ZtPU%V%V%VWWW  @ OOM$O ? ? ? 48$ ? ? )4?CaCi  & & ( ( ( ( (DjCir7c8|jdS)zP `int`: Size of the base vocabulary (without the added tokens). Fwith_added_tokensrrrs r5rzTokenizersBackend.vocab_sizes ---FFFr7c8|jdS)NTr)r get_vocabrs r5rzTokenizersBackend.get_vocabs((4(@@@r7c*|SrT)rrs r5r/zTokenizersBackend.vocabs~~r7chdt|jdDS)z Returns the sorted mapping from string to index. The added tokens encoder is cached for performance optimisation in `self._added_tokens_encoder` for the slow tokenizers. c$i|] \}}|j|Sr1contentr3vks r5r<z:TokenizersBackend.added_tokens_encoder.. mmmA 1mmmr7c|dSrr1r4s r5rz8TokenizersBackend.added_tokens_encoder.. dhijdkr7rrrrrs r5rz&TokenizersBackend.added_tokens_encoders9 nm0I0O0O0Q0QWkWk)l)l)lmmmmr7c4|jS)z Returns the added tokens in the vocabulary as a dictionary of index to AddedToken. Returns: `dict[str, int]`: The added tokens. )rget_added_tokens_decoderrs r5rz&TokenizersBackend.added_tokens_decoder s77999r7chdt|jdDS)z Returns the added tokens in the vocabulary as a dictionary of token to index. Returns: `dict[str, int]`: The added tokens. c$i|] \}}|j|Sr1r!r#s r5r<z5TokenizersBackend.get_added_vocab.. r&r7c|dSrr1r(s r5rz3TokenizersBackend.get_added_vocab.. r)r7rr*rs r5get_added_vocabz!TokenizersBackend.get_added_vocabs9nm0I0O0O0Q0QWkWk)l)l)lmmmmr7cdS)zN Returns True, to avoid expensive `assert tokenizer` gotchas. Tr1rs r5__bool__zTokenizersBackend.__bool__"s tr7c8|jdS)zD Size of the full vocabulary with the added tokens. Trrrs r5__len__zTokenizersBackend.__len__(s---EEEr7c|jS)zc `tokenizers.implementations.BaseTokenizer`: The Rust tokenizer used as a backend. )rrs r5backend_tokenizerz#TokenizersBackend.backend_tokenizer.s r7c|jjS)zU `tokenizers.decoders.Decoder`: The Rust decoder for this tokenizer. )rdecoderrs r5r8zTokenizersBackend.decoder5s &&r7Tr-return_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverbosec F| d|jv}| d|jv}|r|j |g|jz} n|g} tt} | D]} | d| j|r | d| j|r | d| j|r | d| j|r | d| j |r-| dt| j| | fS)a Convert the encoding representation (from low-level HuggingFace tokenizer output) to a python Dict and a list of encodings, take care of building a batch from overflowing tokens. Overflowing tokens are converted to additional examples (like batches) so the output values of the dict are lists (overflows) of lists (tokens). Output shape: (overflows, sequence length) Ntoken_type_idsattention_mask input_idsspecial_tokens_maskoffset_mappingr) model_input_names overflowingrr@rVidstype_idsrBrDoffsetsr) rr-r9r:r;r<r=r>r? encodings encoding_dictrs r5_convert_encodingz#TokenizersBackend._convert_encoding<sV( ! ($48N$N ! ($48N$N ! $ #)=)I! X%99II! I#D))  ; ;A + & - -ae 4 4 4$ C./66qzBBB$ I./66q7GHHH) S34;;A This encodes a dummy input and checks the number of added tokens, and is therefore not efficient. Do not put this inside your training loop. Args: pair (`bool`, *optional*, defaults to `False`): Whether the number of added tokens should be computed in the case of a sequence pair or a single sequence. Returns: `int`: Number of special tokens added to sequences. )rnum_special_tokens_to_add)rrs r5r[z+TokenizersBackend.num_special_tokens_to_addzs&88>>>r7rHskip_special_tokensc<t|tr|j|Sg}|rt |jn t }|D]C}t|}||vr||j|D|S)a Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and added tokens. Args: ids (`int` or `list[int]`): The token id (or token ids) to convert to tokens. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Returns: `str` or `list[str]`: The decoded token(s). )r?rTrrSr{all_special_idsrV)rrHr\r ids_to_skiprs r5convert_ids_to_tokensz'TokenizersBackend.convert_ids_to_tokenss c3   4?..s33 33FQc$.///CEE  > >EJJE ## MM$/55e<< = = = = r7textrXc H|jd|||d|S)N)ra text_pairrXr1) _encode_plusr)rrarrXrs r5tokenizezTokenizersBackend.tokenizes2 t lddOallekllssuuur7padding_strategyrrrrrc |jj |jj}|tjkr |jn<|||j|jd} d} n fd|D} | |kr|jjdi||tj kr||j dSdS|tj kr|nd} | ||n|j |j|j|j|d}||kr|jjdi|dSdS)a Define the truncation and the padding strategies for fast tokenizers (provided by HuggingFace tokenizers library) and restore the tokenizer settings afterwards. The provided tokenizer has no padding / truncation strategy before the managed section. If your tokenizer set a padding / truncation strategy before, then it will be reset to no padding / truncation when exiting the managed section. Args: padding_strategy ([`~utils.PaddingStrategy`]): The kind of padding that will be applied to the input truncation_strategy ([`~tokenization_utils_base.TruncationStrategy`]): The kind of truncation that will be applied to the input max_length (`int`): The maximum size of a sequence. stride (`int`): The stride to use when handling overflow. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. N)rrrrc>i|]}||dSrTrn)r3r%rs r5r<z@TokenizersBackend.set_truncation_and_padding..s)GGG11kooa66GGGr7)rrpad_idrZrrr1)rrrrDO_NOT_TRUNCATErrrrr DO_NOT_PAD no_padding MAX_LENGTHr pad_token_idrZrr) rrfrrrrrrtargetcurrentrrs @r5set_truncation_and_paddingz,TokenizersBackend.set_truncation_and_paddingsiBo0 ?* "4"D D D&--///) /5!1 F"GGGGGGG&  11;;F;;; 9 9 9#**,,,,,$#$47Q#Q#QZZW[F -9-E\\4K\+!^#5&8 F6!!..8888888"!r7rrcis_split_into_wordsreturn_tensorsrc  #d}||std|||std|rAt|ttfo#|o!t|dttf}nt|ttf}|rt|trt d|Pt |t |kr0tdt |dt |d|tt||n|}n |r||fgn|g}t|ttfs t dt|d  ||||| | |j }j j |kr |j _ j ||| }  fd |D}i}|ddD]##fd |D}||#<d|D}r;g}t|D]$\}\}} ||gt |dzz }%||d<|dD]}!|!|t!||| }"|s5| 3s1t!d|"D|"j}"|"S)Ncrt|trdSt|ttfrt |dkrdSt|dtrdSt|dttfrt |ddks!t|ddtrdSt|ddttfrFt |dddkp&t|dddtSdSdSdS)NTrF)r?rDr@r2r)rs r5_is_valid_text_inputz._is_valid_text_inputs!S!! tAe}-- q66Q;;4!c** !4!tUm44!1Q4yyA~~AaDGS)A)A~#t#AaDGdE];;%"1Q47||q0OJqtAwqz34O4OO$u 5ur7ztext input must be of type `str` (single example), `list[str]` (batch or single pretokenized example) or `list[list[str]]` (batch of pretokenized examples) or `list[tuple[list[str], list[str]]]` (batch of pretokenized sequence pairs).rzdwhen tokenizing batches of text, `text_pair` must be a list or tuple with the same length as `text`.zbatch length of `text`: z- does not match batch length of `text_pair`: .z:batch_text_or_text_pairs has to be a list or a tuple (got ))rfrrrrr)rXis_pretokenizedc Jg|]}|  S))r-r9r:r;r<r=r>r?)rM) r3r-r:r>r=r;r<r9rr?s r5r6z2TokenizersBackend._encode_plus..dsX   " "!&;&;*C+E'=+ #   r7c0g|]\}}|D]}|Sr1r1)r3r4rrrs r5r6z2TokenizersBackend._encode_plus..us.NNN74DINNqQNNNNr7c"g|] \}}|D]}| Sr1r1)r3rr4rs r5r6z2TokenizersBackend._encode_plus..ws)SSSWQdSSqSSSSr7rCoverflow_to_sample_mapping) tensor_typeci|]>\}}|t|dkr#t|dtr|dn|?Sr>)rr?r@)r3rrs r5r<z2TokenizersBackend._encode_plus..sW"Uc%jj1nnE!Hd9S9Sn%((Y^r7)rr?r@r2rD TypeErrorrziptyperrrrr encode_batchrq&_eventual_warn_about_too_long_sequencerrrK)$rrarcrXrfrrrrsrrrtr9r:r;r<r=r>r?rrrw is_batchedbatch_text_or_text_pairsrKtokens_and_encodingssanitized_tokensstacksanitized_encodingsr~r:toksrrCbatched_outputrs$` ``````` @r5rdzTokenizersBackend._encode_pluss0   ($#D)) W   )=)=i)H)H W   9#D4-88hThjQUVWQX[_afZgFhFhJJ#D4-88J  T)S)) $Tc)nn)D)D *s4yy**I***FOEZtCi,@,@'A'A'A`d $ $?H'Sy(9':':dV $2UDMBB nTRjMkMknnn  ''- 3!1% (    '#'#< ? 04H H H4HDO 1O00 $1/1   & '*1- * *CNNNN&:NNNE$) S ! !SS0DSSS % X)+ & )*> ? ? K K 9D!*qcC[8I4J4J.JJ**=W 9 :)+6 X XI  7 7 :w W W W W&'79LZhiii n4=V4*&4&:&:&<&<( Nr7rc|jj|jj|nd|S)NrC)r6r8decoder)rrs r5convert_tokens_to_stringz*TokenizersBackend.convert_tokens_to_strings@%-9  " * 1 1& 9 9 9&!! r7 token_idsclean_up_tokenization_spacesc |ddt|tr|g}t|tr|d}|j||}||n|j}|rt|dr*t|j r| |}n| dd dd d d  d d  d d dd dd dd dd dd}|S)Nuse_source_tokenizerrC)r\clean_up_tokenizationz .rxz ??z !!z ,,z ' 'z n'tzn'tz 'mz'mz 'sz'sz 'vez'vez 'rez're) rdr?rTrcrrrrwcallablerreplace)rrr\rrras r5_decodezTokenizersBackend._decodesn  )4000 i % % $" I i & & /!+.I%%iEX%YY,7 ) (2 % ( t455 (4C]:^:^ 11$77LLs++WT3''WT3''WT3''WUC((WVU++WUD))WUD))WVU++WVU++ r7 file_names. legacy_formatct|}tj||r|dzndtz}|j|||fz}|S)Nrr)rDrerfrTOKENIZER_FILEr6save)rrrrrr$s r5_save_pretrainedz"TokenizersBackend._save_pretrainedsl^,, oM_s222Q__   ##N333>"33 r7c  tj|j}|d}|d} d} |dddkri|dd<g|dd<n|ddd kr^|dd O|dd } |dd| d }  | vr| } d |dd <| d gg|dd<n;|ddd vr i|dd<nt d|ddd7d|dvr-|ddvr|dd|dd<t jtj|g} |D]} | dd}| dd}|ddd kr|sC| dvr| d| d<| td*i| || ||dddkr#d|vr|dd|dd|d<|dddkr#d|vr|dd|dd|d<|ddd kr| | |d<|dt|dddksA|dddkrPd|dvrFtd|ddDr!tj|d<t |dd}|d*|| d|}|||| ,tj}d| vr| dD]}| d|d}fd |D}|| d|d<|D](}|}|t d!)fd"|D| d|d#<d$D]L}|| vrF| |\}} |vr|}|}|t d!||g| |<M| |d<t jtj||j}t*jD]}t/||t/||} |vr|}|j|d}t5|tr-t||j|j|j|jd%&||<|||<|jr|jng}|| |tA|d kr||d'<|d(< |j!d*i|S#tD$rH}d)tG|vr1|d(d|j!d*i|}|_|cYd}~Sd}~wwxYw)+uf Trains a tokenizer on a new corpus with the same defaults (in terms of special tokens or tokenization pipeline) as the current one. Args: text_iterator (generator of `list[str]`): The training corpus. Should be a generator of batches of texts, for instance a list of lists of texts if you have everything in memory. vocab_size (`int`): The size of the vocabulary you want for your tokenizer. length (`int`, *optional*): The total number of sequences in the iterator. This is used to provide meaningful progress tracking new_special_tokens (list of `str` or `AddedToken`, *optional*): A list of new special tokens to add to the tokenizer you are training. special_tokens_map (`dict[str, str]`, *optional*): If you want to rename some of the special tokens this tokenizer uses, pass along a mapping old special token name to new special token name in this argument. kwargs (`dict[str, Any]`, *optional*): Additional keyword arguments passed along to the trainer from the 🤗 Tokenizers library. Returns: [`PreTrainedTokenizerFast`]: A new tokenizer of the same type as the original one, trained on `text_iterator`. added_tokensrGNr.rr r/rArrrg)r"r#z;This method does not support this type of tokenizer (found z-) only BPE, Unigram, WordLevel and WordPiece.r[ridr"continuing_subword_prefixend_of_word_suffixr ByteLevelSequence pretokenizersc3.K|]}|ddkVdS)rrNr1)r3 pretokenizers r5 z.;s@$!(K7r7initial_alphabet)rr)rtrainerrrc<g|]}||Sr1ri)r3r;special_tokens_maps r5r6z=TokenizersBackend.train_new_from_iterator..Ms*![![![5"4"8"8"F"F![![![r7zQAttempted to set a token in the post processor that does not exist in the mappingc:g|]}|Sr1)rO)r3r;rs r5r6z=TokenizersBackend.train_new_from_iterator..Vs)CuCuCuejIDYDYZ_D`D`CuCuCur7rH)r}sepT) single_wordlstriprstriprrrPr*z7multiple values for keyword argument 'tokenizer_object'r1)$rlloadsrto_strrdrrifrom_strdumpsrVrrUanypre_tokenizers_fastralphabetMODEL_TO_TRAINER_MAPPINGtrain_from_iteratorrOrrrSPECIAL_TOKENS_ATTRIBUTESrrrrnr?rrrrrPrrrrD)r text_iteratorrrnew_special_tokensrrrrrGr[rr added_tokenrr trainer_classrtrained_tokenizer_jsonrrr;token_id special_tokenspecial_token_fullrPr new_tokenizerrs ` @r5train_new_from_iteratorz)TokenizersBackend.train_new_from_iteratorsDDO$:$:$<$<==%)).99 '++,<== ' "6 *e 3 3/1N7 #G ,02N7 #H - - G $V , 9 9g&x0<'0:*73G4D3Ew'0 G $V ,0J J J/1N7 #G , ,>n]dNeflNm>>>   *~g666w' 48JJJ3EnU\F]^iFj3kN7 #K 0!*4:n+E+EFF ' = =K!ooi66Gd++Ag&v.);;G;!-+i2HL^2^2^);K 11)*:; v vC+,<=cB8LF)5![![![![TZ![![![FLN#34S9(C!'#,#8#8#?#?#+", s##, DvCuCuCuntCuCuCuN#34S9%@@!/ F F  N22-m++F++ +   HCPQFFRR -t444 . 8 8 8 8 +4 ($$$$$$$ s$! V.. X8> Model type one of the mistral models (on older versions) 2) Remote models on the hub from official mistral models >> Tags including `base_model:.*mistralai` rN) model_info)versionrmodel_idrRc|}|j0dd|jrdSdS)Nzbase_model:.*mistralairTF)tagssearchr)rr.rres r5is_base_mistralz?TokenizersBackend._patch_mistral_regex..is_base_mistralsIJx((Ez%995rwwuz7J7JKK 45r7Tz config.jsonF) cache_dirr;local_files_only%_raise_exceptions_for_missing_entries'_raise_exceptions_for_connection_errors _commit_hashr+r,transformers_versionrz4.57.2)mistralmistral3voxtral ministralpixtralz4.57.3rz$The tokenizer you are loading from 'a' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.z[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+|[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]+[\p{Ll}\p{Lm}\p{Lo}\p{M}]*|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n/]*|\s*[\r\n]+|\s+(?!\S)|\s+isolated)patternbehavior)r use_regex)rhuggingface_hubr packagingrtransformers.utils.hubrrDboolrrkrlrmrnparsesetattrrrryrzrpre_tokenizersSplitRegexr6rr?r Metaspacer)r}rpretrained_model_name_or_pathr;rrris_localrrrrrr _config_filemistral_config_detectedf_configrtransformers_model_typersplit_pretokenizercurrent_pretokenizerrrs @@r5rz&TokenizersBackend._patch_mistral_regexs*  ......%%%%%%666666 c d           H ( 4  5% 5*9/:W*X*X 5';-#!16;8=)   L', #',999+Q"illG+++++++++++++++'.{{3I'J'J$*1++l*C*C' (%GMM:N,O,OSZS`S`aiSjSj,j,j  )3?3  )()%gmmAAAA tCH~   X nd38nnnnXn:d3 ?&;:::X:10nc3hnnnn$ FFFFF =X ''''X'.2-1*/+0',#-(-(-( $d{-( $d{ -( $( -( %) -(!%-(-(-( tCH~tL11 2-(-(-(-(^ 7#7#*777766d3+;&<6WZ6666 ??d?s????*tCyt`cfjknfo`o4vvSvd vtvjnorjsvvvvI9)I90I9 I9  I9  $J I9DjI9I9I9I9\gk#',;,F2D2T!%$))-#'&*-1-1*/+0',#,0)XX++d9o=EV@WWX004 ?BTJ[E\\_ccX! X * X 0 X$JXX"X $JXDjXt X $d{X $d{X$(X %)!X"!%#X$%X&'X(#Tk)X, -XXXXt tCy S    %*48 ((c?("('+Tk ( ((((\&*&* bk)#s(Od{  t  sCx * CCCCJ }}}[}}}}}r7r')Arrrlre collectionsrcollections.abcrshutilrtypingrtokenizers.pre_tokenizersrrrrrrr r rr ritokenizers.decodersr rtokenizers.modelsr rtokenizers.trainersrrrrrrintegrations.ggmlrmodeling_gguf_pytorch_utilsrtokenization_utils_baserrrrrrrutilsrr r! get_loggerrpryrSPECIAL_TOKENS_MAP_FILETOKENIZER_CONFIG_FILETIKTOKEN_VOCAB_FILEADDED_TOKENS_FILErrr'PreTrainedTokenizerFastr1r7r5rs  ######$$$$$$777777++++++--------//////111111666666********^^^^^^^^^^^^......555555======@?????????  H % %"3/'( !! (6EXYY,--CCCCC/CC.-CN&,r7