jgS)rddlZddlZddlZddlZddlZddlmZmZmZm Z m Z m Z ddl m Z mZmZmZmZmZdZ d9dedee ed e ee ed ed ed eef d Z d:dedee edee ed ed ed eef dZ d;dedee ed ed ed ee f dZ d:dedee ed ed ed eef dZ dN) AsyncIterableCallableIterableListOptionalUnion)GenerationResultGenerationStepResult Generator ScoringResultTranslationResult Translatorcttdtttdtttdttt dt tt dttt dttt dtdS)z4Registers additional attributes to compiled modules.translate_iterablescore_iterablegenerate_tokensgenerate_iterableasync_generate_tokensN) setattrrtranslator_translate_iterabletranslator_score_iterabletranslator_generate_tokensr generator_generate_iterablegenerator_score_iterablegenerator_generate_tokensgenerator_async_generate_tokens{/home/lighthouse/badge-speaker-benchmark/.benchmark-venv-community/lib64/python3.11/site-packages/ctranslate2/extensions.pyregister_extensionsr s J,.KLLL J(*CDDD J)+EFFF I*,GHHH I')ABBB I(*CDDD I.0OPPPPPr examples translatorsource target_prefixmax_batch_size batch_typereturnc+tK|g}|||t|j|||fi|Ed{VdS)a6Translates an iterable of tokenized examples. This method is built on top of :meth:`ctranslate2.Translator.translate_batch` to efficiently translate an arbitrarily large stream of data. It enables the following optimizations: * stream processing (the iterable is not fully materialized in memory) * parallel translations (if the translator has multiple workers) * asynchronous batch prefetching * local sorting by length Arguments: source: An iterable of tokenized source examples. target_prefix: An optional iterable of tokenized target prefixes. max_batch_size: The maximum batch size. batch_type: Whether :obj:`max_batch_size` is the number of "examples" or "tokens". **kwargs: Any translation options accepted by :meth:`ctranslate2.Translator.translate_batch`. Returns: A generator iterator over :class:`ctranslate2.TranslationResult` instances. Example: This method can be used to efficiently translate text files: .. code-block:: python # Replace by your own tokenization and detokenization functions. tokenize_fn = lambda line: line.strip().split() detokenize_fn = lambda tokens: " ".join(tokens) with open("input.txt") as input_file: source = map(tokenize_fn, input_file) results = translator.translate_iterable(source, max_batch_size=64) for result in results: tokens = result.hypotheses[0] target = detokenize_fn(tokens) print(target) N)append_process_iterabletranslate_batch)r#r$r%r&r'kwargs iterabless rrrs`I ''' "   r@targetc+DKt|j||g||fi|Ed{VdS)a}Scores an iterable of tokenized examples. This method is built on top of :meth:`ctranslate2.Translator.score_batch` to efficiently score an arbitrarily large stream of data. It enables the following optimizations: * stream processing (the iterable is not fully materialized in memory) * parallel scoring (if the translator has multiple workers) * asynchronous batch prefetching * local sorting by length Arguments: source: An iterable of tokenized source examples. target: An iterable of tokenized target examples. max_batch_size: The maximum batch size. batch_type: Whether :obj:`max_batch_size` is the number of "examples" or "tokens". **kwargs: Any scoring options accepted by :meth:`ctranslate2.Translator.score_batch`. Returns: A generator iterator over :class:`ctranslate2.ScoringResult` instances. Nr+ score_batch)r#r$r0r&r'r-s rrr[se<!    r generator start_tokensc+BKt|j|g||fi|Ed{VdS)acGenerates from an iterable of tokenized prompts. This method is built on top of :meth:`ctranslate2.Generator.generate_batch` to efficiently run generation on an arbitrarily large stream of data. It enables the following optimizations: * stream processing (the iterable is not fully materialized in memory) * parallel generations (if the generator has multiple workers) * asynchronous batch prefetching * local sorting by length Arguments: start_tokens: An iterable of tokenized prompts. max_batch_size: The maximum batch size. batch_type: Whether :obj:`max_batch_size` is the number of "examples" or "tokens". **kwargs: Any generation options accepted by :meth:`ctranslate2.Generator.generate_batch`. Returns: A generator iterator over :class:`ctranslate2.GenerationResult` instances. N)r+generate_batch)r4r5r&r'r-s rrrsb8!    rtokensc+BKt|j|g||fi|Ed{VdS)a9Scores an iterable of tokenized examples. This method is built on top of :meth:`ctranslate2.Generator.score_batch` to efficiently score an arbitrarily large stream of data. It enables the following optimizations: * stream processing (the iterable is not fully materialized in memory) * parallel scoring (if the generator has multiple workers) * asynchronous batch prefetching * local sorting by length Arguments: tokens: An iterable of tokenized examples. max_batch_size: The maximum batch size. batch_type: Whether :obj:`max_batch_size` is the number of "examples" or "tokens". **kwargs: Any score options accepted by :meth:`ctranslate2.Generator.score_batch`. Returns: A generator iterator over :class:`ctranslate2.ScoringResult` instances. Nr2)r4r8r&r'r-s rrrsb8!    rFi) max_decoding_lengthmin_decoding_length sampling_topk sampling_toppsampling_temperaturereturn_log_probrepetition_penaltyno_repeat_ngram_size disable_unksuppress_sequences end_tokenmax_input_lengthuse_vmapr<r=r>r?r@rArBrCrDrErFrGrHc #jKt|j|g||gnd| | | | | ||||||||Ed{VdS)a{Yields tokens as they are generated by the model. Arguments: source: Source tokens. target_prefix: Optional target prefix tokens. max_decoding_length: Maximum prediction length. min_decoding_length: Minimum prediction length. sampling_topk: Randomly sample predictions from the top K candidates. sampling_topp: Keep the most probable tokens whose cumulative probability exceeds this value. sampling_temperature: Sampling temperature to generate more random samples. return_log_prob: Include the token log probability in the result. repetition_penalty: Penalty applied to the score of previously generated tokens (set > 1 to penalize). no_repeat_ngram_size: Prevent repetitions of ngrams with this size (set 0 to disable). disable_unk: Disable the generation of the unknown token. suppress_sequences: Disable the generation of some sequences of tokens. end_token: Stop the decoding on one of these tokens (defaults to the model EOS token). max_input_length: Truncate inputs after this many tokens (set 0 to disable). use_vmap: Use the vocabulary mapping file saved in this model Returns: A generator iterator over :class:`ctranslate2.GenerationStepResult` instances. Note: This generation method is not compatible with beam search which requires a complete decoding. N) rBrCrDrErFr<r=r>r?r@ return_scoresrGrH)_generate_tokensr,)r#r$r%r<r=r>r?r@rArBrCrDrErFrGrHs rrrs\ " (4$-1-//##1%)!riT) max_length min_lengthr>r?r@rArBrCrDrErF static_promptcache_static_promptcallbackpromptrLrMrNrOrPc#Kt|dkrt|dtr|g}t|j|fid|d|d| d| d| d| d|d |d |d |d |d |d| d|d|ddd|Ed{VdS)aYields tokens as they are generated by the model. Arguments: prompt: Batch of start tokens. If the decoder starts from a special start token like , this token should be added to this input. max_batch_size: The maximum batch size. batch_type: Whether :obj:`max_batch_size` is the number of "examples" or "tokens". max_length: Maximum generation length. min_length: Minimum generation length. sampling_topk: Randomly sample predictions from the top K candidates. sampling_topp: Keep the most probable tokens whose cumulative probability exceeds this value. sampling_temperature: Sampling temperature to generate more random samples. return_log_prob: Include the token log probability in the result. repetition_penalty: Penalty applied to the score of previously generated tokens (set > 1 to penalize). no_repeat_ngram_size: Prevent repetitions of ngrams with this size (set 0 to disable). disable_unk: Disable the generation of the unknown token. suppress_sequences: Disable the generation of some sequences of tokens. end_token: Stop the decoding on one these tokens (defaults to the model EOS token). static_prompt: If the model expects a static prompt (a.k.a. system prompt) it can be set here to simplify the inputs and optionally cache the model state for this prompt to accelerate future generations. cache_static_prompt: Cache the model state after the static prompt and reuse it for future generations using the same static prompt. callback: Optional function that is called for each generated token when obj:`beam_size` is 1. If the callback function returns ``True``, the decoding will stop for this batch index. Returns: A generator iterator over :class:`ctranslate2.GenerationStepResult` instances. Note: This generation method is not compatible with beam search which requires a complete decoding. rr&r'rBrCrDrErFrLrMr>r?r@rJrNrOinclude_prompt_in_resultFrPN)len isinstancestrrKr7)r4rQr&r'rLrMr>r?r@rArBrCrDrErFrNrOrPs rrrsJp 6{{Q:fQi55 &~:  .-  21  K.-)::$m$m21&o $m!"0/#$"'%&'rcKt|dkrt|dtr|g}t|j|fid|d|d| d| d| d| d|d |d |d |d |d |d| d|d|ddd|23d{V}|WV 6dS)aYields tokens asynchronously as they are generated by the model. Arguments: prompt: Batch of start tokens. If the decoder starts from a special start token like , this token should be added to this input. max_batch_size: The maximum batch size. batch_type: Whether :obj:`max_batch_size` is the number of "examples" or "tokens". max_length: Maximum generation length. min_length: Minimum generation length. sampling_topk: Randomly sample predictions from the top K candidates. sampling_topp: Keep the most probable tokens whose cumulative probability exceeds this value. sampling_temperature: Sampling temperature to generate more random samples. return_log_prob: Include the token log probability in the result. repetition_penalty: Penalty applied to the score of previously generated tokens (set > 1 to penalize). no_repeat_ngram_size: Prevent repetitions of ngrams with this size (set 0 to disable). disable_unk: Disable the generation of the unknown token. suppress_sequences: Disable the generation of some sequences of tokens. end_token: Stop the decoding on one of these tokens (defaults to the model EOS token). static_prompt: If the model expects a static prompt (a.k.a. system prompt) it can be set here to simplify the inputs and optionally cache the model state for this prompt to accelerate future generations. cache_static_prompt: Cache the model state after the static prompt and reuse it for future generations using the same static prompt. callback: Optional function that is called for each generated token when obj:`beam_size` is 1. If the callback function returns ``True``, the decoding will stop for this batch index. Returns: An async generator iterator over :class:`ctranslate2.GenerationStepResult` instances. Note: This generation method is not compatible with beam search which requires a complete decoding. rr&r'rBrCrDrErFrLrMr>r?r@rJrNrOrSFrPN)rTrUrVAsyncGeneratorr7)r4rQr&r'rLrMr>r?r@rArBrCrDrErFrNrOrP step_results rrr_sep 6{{Q:fQi55+ &~:  .-  21  K.-)::$m$m21&o $m!"0/#$"'%&'k*+s8Bc&eZdZdZdZdZdZdS)rXctj|_tj|_d|_||_||_||_ dSN) asyncioQueuequeue threadingEventshutdown_event iterator_task process_funcargsr-)selfrdrer-s r__init__zAsyncGenerator.__init__sA]__ 'o//!(  rc8Kt|jg|jRi|jD]W}|j|d{Vt jdd{V|j rnX|jdd{VdS)Ng-C6?) rKrdrer-r_putr]sleeprbis_set)rfrYs rproducerzAsyncGenerator.producers+   $    -1[    K*..-- - - - - - - --'' ' ' ' ' ' ' '"))++  jnnT"""""""""""rc\tj||_|Sr\)r] create_taskrlrc)rfs r __aiter__zAsyncGenerator.__aiter__s"$0AA rc4K|jrt |jd{V}| |jt|S#t j$r!|jtwxYwr\)rbrkStopAsyncIterationr_getsetr]CancelledError)rfitems r __anext__zAsyncGenerator.__anext__s   % % ' ' %$ $ %))))))))D|#'')))((K% % % %   # # % % %$ $ %s AA''0BN)__name__ __module__ __qualname__rgrlrorvrrrrXrXsP # # # % % % % %rrXc/  Ktj tj|dd d  fd}|dd|d||i| fd}tj|d}| }|nBt|tr| |Vn$#t$r YnwxYwY| dS) NrPcdS)NFr)rYs rz"_generate_tokens..sErcp|}|p|Sr\)rirk)rYuser_callback_resultgenerator_closed step_results user_callbacks r _callbackz#_generate_tokens.._callbacks?,}[99%%%&&((@,@@rTr;) asynchronous beam_sizerPc D]}|n,#t$r}|Yd}~nd}~wwxYwddSr\)result Exceptionri)re async_resultsrs r_catch_exceptionz*_generate_tokens.._catch_exceptions '         Q         s AAA)r0daemon) r_r^r`rarrupdateThreadstartrUr GeneratorExitrsjoin) rdrer-rrthreadrYrrrrs @@@@rrKrKs;==L ((JJz400M11 AAAAAAA  MM !  !L$1&11M %5d C C CF LLNNN "&&((    k9 - -            " " " E    KKMMMMMsCC65C6c+VK|dkrtdt|dkr |d}ntj|}|||dd|dkr|dzn|}t j}t|||D]s}|j||i||rY|d r?|j V|r|d ?t|r'|j V|%dSdS)Nr;zmax_batch_size must be >= 1rT)r&r'r) ValueErrorrT itertools zip_longestr collectionsdeque_batch_iteratorextenddonepopleftr) rdr.r&r'r-iterableread_batch_sizer_streamss rr+r+sx6777 9~~Q<()4 MM,$   .\}}|"t|dkrt d||?'||VdSdS)Nrr"r8r;zInvalid batch type %sc3K|]}gVdSr\r).0_s r z"_batch_iterator..Fs"001B000000rz+Input iterables do not have the same length)rUtuplerTmaxrzipr*)r batch_sizer'rrLexamplebatchelements rrr-sGJ""'5)) !jG  # # 3wqz??j88  8 # #ZWQZ99J -C OOa/:= JJ  __ 4zABB B ?0000000G!'733 " "NE73w<rs KKKKKKKKKKKKKKKKQQQ48 ::: T#Y :HT#Y/0: :  : ::::B $$$ T#Y $ T#Y $ $  $m$$$$T """49%"" "  """"P """ T#Y "" " m """"P*.? # "#! ! !48<@ #???? I?DI&?  ?  ??? ?????!d3i1?c49d3i789? !?"#?$"#%????J N "#! ! !48<@)- $7;'NNNN $s)T$s)_, -NN N  NNNN NNNNN!d3i1N c49d3i789!N"DI&#N$%N&,-t34'N("#)NNNNh O "#! ! !48<@)- $7;'OOOO $s)T$s)_, -OO O  OOOO OOOOO!d3i1O c49d3i789!O"DI&#O$%O&,-t34'O('()OOOOd%%%%%%%%%%%%%%%%P333l'''<     r