j@ dZddlZddlmZddlZddlmZddlmZdZ dZ dZ d Z e e d d d d dde df Z e de ge ddRZejdZeje ejejzejzZejdZejdZd!dZd"dZGddeZdZdZ d#d ZdS)$a Twitter-aware tokenizer, designed to be flexible and easy to adapt to new domains and tasks. The basic logic is this: 1. The tuple REGEXPS defines a list of regular expression strings. 2. The REGEXPS strings are put, in order, into a compiled regular expression object called WORD_RE, under the TweetTokenizer class. 3. The tokenization is done by WORD_RE.findall(s), where s is the user-supplied string, inside the tokenize() method of the class TweetTokenizer. 4. When instantiating Tokenizer objects, there are several options: * preserve_case. By default, it is set to True. If it is set to False, then the tokenizer will downcase everything except for emoticons. * reduce_len. By default, it is set to False. It specifies whether to replace repeated character sequences of length 3 or greater with sequences of length 3. * strip_handles. By default, it is set to False. It specifies whether to remove Twitter handles of text used in the `tokenize` method. * match_phone_numbers. By default, it is set to True. It indicates whether the `tokenize` method should look for phone numbers. N)List)redos) TokenizerIac (?: [<>]? [:;=8] # eyes [\-o\*\']? # optional nose [\)\]\(\[dDpP/\:\}\{@\|\\] # mouth | [\)\]\(\[dDpP/\:\}\{@\|\\] # mouth [\-o\*\']? # optional nose [:;=8] # eyes [<>]? | {}\[\]]+ # Run of non-space, non-()<>{}[] | # or \([^\s()]{0,255}?\([^\s()]{1,255}\)[^\s()]{0,255}?\) # balanced parens, one level deep: (...(...)...) | \([^\s]{1,255}?\) # balanced parens, non-recursive: (...) )+ (?: # End with: \([^\s()]{0,255}?\([^\s()]{1,255}\)[^\s()]{0,255}?\) # balanced parens, one level deep: (...(...)...) | \([^\s]{1,255}?\) # balanced parens, non-recursive: (...) | # or [^\s`!()\[\]{};:'".,<>?«»“”‘’] # not a space or one of these punct chars ) | # OR, the following to match naked domains: (?: (?\s]+>z [\-]+>|<[\-]+z (?:@[\w_]+)z(?:\#+[\w_]+[\w\'_\-]*[\w_]+)z3[\w.+-]{1,64}@[\w-]{1,63}\.(?:[\w-]\.?){1,251}[\w-]uR.(?: [🏻-🏿]?(?:‍.[🏻-🏿]?)+ | [🏻-🏿] )a (?:[^\W\d_](?:[^\W\d_]|['\-_])+[^\W\d_]) # Words with apostrophes or dashes. | (?:[+\-]?\d+[,/.:-]\d+[+\-]?) # Numbers, including fractions, decimals. | (?:[\w_]+) # Words without apostrophes or dashes. | (?:\.(?:\s*\.){1,}) # Ellipsis dots. | (?:\S) # Everything else that isn't whitespace. z([^\p{L}\p{N}])\1{3,}z&(#?(x?))([^&;\s]+);zZ(?>> from nltk.tokenize.casual import _replace_html_entities >>> _replace_html_entities(b'Price: £100') 'Price: \xa3100' >>> print(_replace_html_entities(b'Price: £100')) Price: £100 >>> cP|d}|dr} |drt|d}nt|d}d|cxkrdkr&nn#t|fdSnO#t$rd}YnAwxYw|vr|d St jj|}|' t|S#ttf$rYnwxYwrd n|d S) Nr cp1252r) groupintr r ValueErrorhtmlentitiesname2codepointgetchr OverflowError)match entity_bodynumberkeepremove_illegals r_convert_entityz/_replace_html_entities.._convert_entitysOkk!nn ;;q>> C ;;q>>2 b11FF b11F 6))))T))))) &++228<<<    d""{{1~~%]155kBBF   6{{" .    $7rrQ7s$A(B B&%B&)C88D  D )ENT_REsubr)r r*r+rr,s `` r_replace_html_entitiesr/sB88888888 ::otX'F'F G GGrcveZdZdZdZdZ d dZdedeefdZ e d d Z e d d Z dS)TweetTokenizera Tokenizer for tweets. >>> from nltk.tokenize import TweetTokenizer >>> tknzr = TweetTokenizer() >>> s0 = "This is a cooool #dummysmiley: :-) :-P <3 and some arrows < > -> <--" >>> tknzr.tokenize(s0) # doctest: +NORMALIZE_WHITESPACE ['This', 'is', 'a', 'cooool', '#dummysmiley', ':', ':-)', ':-P', '<3', 'and', 'some', 'arrows', '<', '>', '->', '<--'] Examples using `strip_handles` and `reduce_len parameters`: >>> tknzr = TweetTokenizer(strip_handles=True, reduce_len=True) >>> s1 = '@remy: This is waaaaayyyy too much for you!!!!!!' >>> tknzr.tokenize(s1) [':', 'This', 'is', 'waaayyy', 'too', 'much', 'for', 'you', '!', '!', '!'] NTFc>||_||_||_||_dS)ae Create a `TweetTokenizer` instance with settings for use in the `tokenize` method. :param preserve_case: Flag indicating whether to preserve the casing (capitalisation) of text used in the `tokenize` method. Defaults to True. :type preserve_case: bool :param reduce_len: Flag indicating whether to replace repeated character sequences of length 3 or greater with sequences of length 3. Defaults to False. :type reduce_len: bool :param strip_handles: Flag indicating whether to remove Twitter handles of text used in the `tokenize` method. Defaults to False. :type strip_handles: bool :param match_phone_numbers: Flag indicating whether the `tokenize` method should look for phone numbers. Defaults to True. :type match_phone_numbers: bool N preserve_case reduce_len strip_handlesmatch_phone_numbers)selfr4r5r6r7s r__init__zTweetTokenizer.__init__Ms).+$*#6   rr returnct|}|jrt|}|jrt |}t d|} |jr'|j |tj }n&|j |tj }n,#t$rtdtj ddwxYw|jstt!d|}|S)zTokenize the input text. :param text: str :rtype: list(str) :return: a tokenized list of strings; joining this list returns the original string if `preserve_case=False`. \1\1\1)timeoutzTweetTokenizer exceeded its zGs time limit; the input may be adversarial (e.g. a long run of digits).Ncbt|r|n|S)N) EMOTICON_REsearchlower)xs rz)TweetTokenizer.tokenize..s%K$6$6q$9$9Hqqqwwyyr)r/r6remove_handlesr5reduce_lengtheningHANG_REr.r7 PHONE_WORD_REfindallrDEFAULT_TIMEOUTWORD_RE TimeoutErrorr4listmap)r8r safe_textwordss rtokenizezTweetTokenizer.tokenizeis.&d++   (!$''D ? ,%d++DKK 400 ' W*22u'<3 ,,Y@U,VV   Lu/DLLL   ! HH5QQE s AB--)C regex.Patternc t|jsgtjddt dtjtjztjzt|_t|jS)zCore TweetTokenizer regex(|)) type_WORD_REregexcompilejoinREGEXPSVERBOSEIUNICODEr8s rrJzTweetTokenizer.WORD_REsmDzz" "'-(CHHW%%((( '%-7##DJJ Dzz""rc t|jsgtjddt dtjtjztjzt|_t|jS)z#Secondary core TweetTokenizer regexrSrTrU) rV_PHONE_WORD_RErXrYrZ REGEXPS_PHONEr\r]r^r_s rrGzTweetTokenizer.PHONE_WORD_REsmDzz( (- .CHH]++... '%-7))DJJ %Dzz((rTFFT)r:rQ) __name__ __module__ __qualname____doc__rWrar9strrLrPpropertyrJrGrrrr1r13s(HN  77778'S'T#Y''''R###X#)))X)))rr1cVtjd}|d|S)ze Replace repeated character sequences of length 3 or greater with sequences of length 3. z (.)\1{2,}r<)rXrYr.)r patterns rrErEs' mL))G ;;y$ ' ''rc8td|S)z4 Remove Twitter username handles from text.  ) HANDLES_REr.)r s rrDrDs >>#t $ $$rFcNt|||||S)z: Convenience function for wrapping the tokenizer. r3)r1rP)r r4r5r6r7s rcasual_tokenizerps4 ##/     htnn r)Nr)rTr rc)rgr!typingrrXnltkrnltk.tokenize.apir EMOTICONSURLSFLAGS PHONE_REGEXr[rbrYrFr\r]r^r?r-rnrr/r1rErDrprrrrxs@  (((((($ $)f   $  (>   /" J[77122;77  %-0 1 1emIu}uw'>'NOO  . / /U]H 8H8H8H8H|s)s)s)s)s)Zs)s)s)v(((%%% r