j#dZddlmZddlmZ ddlZn #e$rdZYnwxYwGddZdZ e dkr e dSdS) a A module for language identification using the TextCat algorithm. An implementation of the text categorization algorithm presented in Cavnar, W. B. and J. M. Trenkle, "N-Gram-Based Text Categorization". The algorithm takes advantage of Zipf's law and uses n-gram frequencies to profile languages and text-yet to be identified-then compares using a distance measure. Language n-grams are provided by the "An Crubadan" project. A corpus reader was created separately to read those files. For details regarding the algorithm, see: https://www.let.rug.nl/~vannoord/TextCat/textcat.pdf For details about An Crubadan, see: https://borel.slu.edu/crubadan/index.html )maxsize)trigramsNcHeZdZdZiZdZdZiZdZdZ dZ dZ dZ d d Z dS) TextCatN<>ctstdddlm}||_|jD]}|j|dS)Nzclassify.textcat requires the regex module that supports unicode. Try '$ pip install regex' and see https://pypi.python.org/pypi/regex for further details.r)crubadan)reOSError nltk.corpusr _corpuslangs lang_freq)selfr langs x/home/lighthouse/badge-speaker-benchmark/.benchmark-venv-community/lib/python3.11/site-packages/nltk/classify/textcat.py__init__zTextCat.__init__7s| #  )((((( L&&(( ) )D L " "4 ( ( ( ( ) )c.tjdd|S)z)Get rid of punctuation except apostrophesz [^\P{P}\']+)r sub)rtexts rremove_punctuationzTextCat.remove_punctuationGsvnb$///rcddlm}m}||}||}|}|D]L}t |j|z|jz}d|D} | D]} | |vr|| xxdz cc<d|| <M|S)z'Create FreqDist of trigrams within textr)FreqDist word_tokenizec8g|]}d|S)r)join).0tris r z#TextCat.profile..Us"KKKsbggcllKKKr)nltkrrrr _START_CHAR _END_CHAR) rrrr clean_texttokens fingerprintttoken_trigram_tuplestoken_trigrams cur_trigrams rprofilezTextCat.profileKs00000000,,T22 z**hjj  1 1A#+D,,Q#R#R KK6JKKKN- 1 1 +-- ,,,1,,,,/0K ,,  1 rcJ|j|}d}||vr{t||}t||}t ||z }nt }|S)zgCalculate the "out-of-place" measure between the text and language profile for a single trigramr)rrlistkeysindexabsr)rrtrigram text_profilelang_fddistidx_lang_profileidx_texts r calc_distzTextCat.calc_dist_s,((.. g  #GLLNN3399'BB L--//0066w??H'(233DD D rci}||}dt|D}|jjD]o}dt|j|D}d}|D]2}||vr"|t ||||z z }(|tz }3|||<p|S)zOCalculate the "out-of-place" measure between the text and all languagesci|]\}}|| Sr=)r ir4s r z&TextCat.lang_dists..sFFFZQgqFFFrci|]\}}|| Sr=r=)r r>r*s rr?z&TextCat.lang_dists..sSSS41a!QSSSrr)r. enumerater_all_lang_freqr1rr3r) rr distancesr. text_ranksr lang_ranks lang_distr4s r lang_distszTextCat.lang_diststs ,,t$$GF9W3E3EFFF L/4466 ( (DSS9T\5K5KD5Q5Q+R+RSSSJI" ) )j((Z%8:g;N%N!O!OOII(II'IdOOrFc|||_|js|rgSdSt|jfd|jD}t |j}t|t|kr|rgSdS|r|St|dkr|dSdS)a Determines the most likely language(s) for the given text. Parameters ---------- text : str The text whose language is to be identified. return_all : bool, optional If False (default), returns a single ISO 639-3 language code as a str, or None if the language is ambiguous or cannot be determined. If True, returns a list of all language codes sharing the minimal distance. The list will have one element if there is a unique best match, multiple elements for ties, or be empty if no language is found. Returns ------- str or None, or list of str If return_all is False: - str: language code if unique minimum found - None: if ambiguous or not classifiable If return_all is True: - list: possible language code(s), or empty list if not classifiable Examples -------- >>> from nltk.classify.textcat import TextCat >>> cat = TextCat() >>> print(cat.guess_language('The quick brown fox jumps over the lazy dog.')) eng A case with no information, returns None or an empty list: >>> print(cat.guess_language('', return_all=True)) [] >>> print(cat.guess_language('')) None A case where a single short input ties between Catalan and French: >>> print(sorted(cat.guess_language('ent', return_all=True))) ['cat', 'fra'] By default (`return_all=False`), in a tie, guess_language returns None: >>> print(cat.guess_language('ent')) None Note: For short or generic inputs, or for closely related languages, the classifier may return an unexpected language. For example, the following is a perfectly grammatical English sentence, but may be classified as Scots ('sco') due to profile similarity: >>> print(cat.guess_language('This is a short English sentence.')) sco This behavior is not a bug, but an artifact of the underlying n-gram profiles. The classifier should be used with sufficiently distinctive and longer text fragments for best accuracy. Nc&g|] \}}|k |Sr=r=)r rr7min_dists rr"z*TextCat.guess_language..s.   T4$(BRBRDBRBRBRrr#r)rGlast_distancesminvaluesitemsr0r1len)rr return_all candidates all_languagesrJs @rguess_languagezTextCat.guess_languagesx#ood33"   4t*113344    #'#6#<#<#>#>   T0557788  z??c-00 0 0  4    z??a  a= tr)F)__name__ __module__ __qualname__r fingerprintsr%r&rKrrr.r:rGrSr=rrrr/sGLKIN))) 000(*2QQQQQQrrc N ddlm}gd}dddddd d d d d }t}|D]}|| t dz }t t t }d}td|D]@ dd fdtd| Dz}||z }Atd|ddzdz| |} td| d|| dtddS)Nr)udhr) z Kurdish-UTF8z Abkhaz-UTF8zFarsi_Persian-UTF8z Hindi-UTF8z Hawaiian-UTF8z Russian-UTF8zVietnamese-UTF8zSerbian_Srpski-UTF8zEsperanto-UTF8zNorthern Kurdish AbkhazianzIranian PersianHindiHawaiianRussian VietnameseSerbian Esperanto) kmrabkpeshinhawrusviesrpepor#r c,g|]}|Sr=r=)r jr> raw_sentencess rr"zdemo..s#&V&V&Vq}Q'7':&V&V&VrzLanguage snippet: z...zLanguage detection: z ()z############################################################################################################################################) r rYrsentsrOr0maprangerprintrS) rYrfriendlytccur_langrowscolssamplecur_sentguessr>rms @@rdemor|s         E"   H B 8,, =!!A%C]++,,q$  ASXX&V&V&V&V&VE!TRSWDUDU&V&V&VWWWH h FF "VAcE]2U:;;;!!&)) @U@@huo@@@AAA i#r__main__) __doc__sysr nltk.utilrregexr ImportErrorrr|rTr=rrrs*  BBBooooooood...b zDFFFFFs