jU+dZddlZdddddddd iZd Zdd Zdd ZGd dZdZeddZddZ dS)ut HuggingFace datasets integration for NLTK. Provides a PathPointer subclass that reads directly from the HuggingFace datasets cache, and a ``download()`` function that populates that cache. Usage:: import nltk nltk.download('stopwords', hf=True) # download to HF cache nltk.corpus.stopwords.words('portuguese') # HF fallback if not in ~/nltk_data nltk.corpus.stopwords.words('portuguese', hf=True) # HF cache directly Registry schema --------------- Each entry in ``REGISTRY`` must declare: ``repo`` (str) HuggingFace dataset repo id, e.g. ``"nltk-data-hub/stopwords"``. ``split`` (str) HF split name to load, e.g. ``"stopwords"``, ``"train"``. ``structure`` (str) How the corpus is organised on HF: ``"multi_config"`` One HF config per NLTK fileid. No assumption about what that dimension represents (language, category, author, etc.). ``fileid`` → config name. ``"flat"`` Single config, flat table. A ``fileid_column`` value is used to select rows for a given fileid. ``"single"`` Single config, no sub-selection. The whole split is the corpus. ``content_type`` (str) How rows are serialised to the byte/text stream NLTK readers expect: ``"word_list"`` Each row is one entry; ``text_column`` holds the string. Serialised as one entry per line. ``"raw_text"`` Rows have a ``text_column`` with full document text. When a fileid is given, rows are filtered by ``fileid_column``. Serialised as the raw text string. (Add new types here as more corpora are onboarded.) ``cache_probe`` (str) A single parquet path inside the repo used to detect whether the corpus has already been downloaded locally. No network request is made; ``huggingface_hub.try_to_load_from_cache`` inspects the local filesystem only. Optional keys (required by certain content types): ``text_column`` column that holds the main text / word value. ``fileid_column`` column that identifies which NLTK fileid a row belongs to. ``label_column`` column for classification labels (future use). N stopwordsznltk-data-hub/stopwords multi_config word_listwordzdata/english/stopwords.parquet)reposplit structure content_type text_column cache_probect|}|dS ddlm}||d|dd}|duo|d kS#t$rYdSwxYw) zHReturn True if the corpus parquet exists in the local HF datasets cache.NFr)try_to_load_from_cacherr dataset)repo_idfilename repo_type no_connection)REGISTRYgethuggingface_hubr Exception) corpus_idinforresults {/home/lighthouse/badge-speaker-benchmark/.benchmark-venv-community/lib/python3.11/site-packages/nltk/huggingface/dataset.py _is_cachedr_s << " "D |u ::::::''L-(    T!?f&?? uus)A AAc6|dd}|dkr!d||dS|dkrD|d}||}t|dkrd|n |r|dndStd |d ) uz Convert an HF dataset ``ds`` to the byte/text content that an NLTK corpus reader would find in a plain file. :param ds: ``datasets.Dataset`` already filtered/selected for this corpus. :param info: REGISTRY entry for the corpus. :param fileid: the NLTK fileid being requested (used by some types). :returns: str — file content as NLTK expects it. r raw_textr r rz content_type=zL is not implemented. Add a handler in nltk.huggingface.dataset._serialise().)rjoinlenNotImplementedError)dsrfileidr coltextss r _serialiser)vs88NJ77L{""yyD/0111z!!=!3#&u::>>tyyE8QarR  B  B B B  cZddlm}|dd}|dkr+td||d|d  S|d kr>||d|d  }!|d |fd }|S||d|d  S)z Load the appropriate HF dataset slice for *fileid*, respecting structure. :param info: REGISTRY entry. :param fileid: NLTK fileid (may be None for single-structure corpora). :returns: ``datasets.Dataset``. r load_datasetr singlerNz`fileid is required for multi_config corpora. Pass the config name (e.g. a language or category).rrrflat fileid_columnc|kSN)rowr'r&s rz"_load_hf_dataset..ss3x6'9r*)datasetsr-r ValueErrorfilter)rr&r-r r%r's ` @r_load_hf_datasetr:s&%%%%%h//IN"" >F |DL&W FFFFF \$v,d7m < < <  'C99999::B  <V DM : : ::r*cVeZdZdZd dZd dZdZdZdZe dZ d Z d Z dS) HFDatasetPathPointeraK A ``PathPointer``-compatible object backed by a HuggingFace dataset stored in the local HF datasets cache (~/.cache/huggingface/datasets/). Satisfies the NLTK PathPointer interface (``open`` / ``file_size`` / ``join``) so that existing corpus readers work unchanged after ``nltk.download(..., hf=True)``. Nc"||_||_dSr3rr&)selfrr&s r__init__zHFDatasetPathPointer.__init__s" r*ct|j}t||j}t |||j}|rt j|St j|dS)z>Return a stream of file content as NLTK corpus readers expect.r&zutf-8) rrr:r&r)ioStringIOBytesIOencode)r?encodingrr%contents ropenzHFDatasetPathPointer.opensl' d4; 7 7 7Rdk:::  (;w'' 'z'..11222r*cdS)Nrr4r?s r file_sizezHFDatasetPathPointer.file_sizesqr*c,t|j|Sr3)r<r)r?r&s rr"zHFDatasetPathPointer.joins#DNF;;;r*ct|j}|t|jsgS|dd} |dkr$ddlm}t ||dS|dkrGdd lm}||d|d  }t ||d S|d gS#t$rgcYSwxYw) z8Return sorted list of fileids available for this corpus.Nr r.rrget_dataset_config_namesrr0r,rr/r1) rrrrr7rPsortedr-uniquer)r?rr rPr-r%s rfileidszHFDatasetPathPointer.fileidss||DN++ >???M? "   III s)C9A CC CCc~t|jid|j}d|S)Nrzhf://)rrr)r?rs rpathzHFDatasetPathPointer.paths5||DNB//33FDNKKt~~r*c@|jr|jd|jn|jS)N/)r&rUrKs r__str__zHFDatasetPathPointer.__str__s(/3{I$)++dk+++ Ir*c(d|jd|jdS)NzHFDatasetPathPointer(z, )r>rKs r__repr__zHFDatasetPathPointer.__repr__sKt~KK4;KKKKr*r3) __name__ __module__ __qualname____doc__r@rIrLr"rSpropertyrUrXr[r4r*rr<r<s 3333<<< ,XJJJLLLLLr*r<cj ddlm}|tdS#t$rYdSwxYw)Nr PathPointer) nltk.datarcregisterr<rrbs r_register_path_pointerrfsV ))))))122222      s $ 22Fc ~ t| 'td|dttddlm |rd|ini dd}|d krdd lm}| d } fd |D}|s[td |D}td|d d dt|d|dd |S d fd di }|s-td|d d dt|dd|S)af Download an NLTK corpus from HuggingFace into the HF datasets cache (``~/.cache/huggingface/datasets/``). :param corpus_id: NLTK corpus id, e.g. ``'stopwords'``. :param token: optional HuggingFace API token for private repos. :param quiet: suppress progress output. :raises ValueError: if *corpus_id* is not in the HF registry. NCorpus z- is not available on HuggingFace. Available: rr,tokenr r.rrOrc Bi|]}|d|fddiS)rrr4).0cfgrkwargsr-s r zdownload..#sL    d6lCOOtG}OOO   r*c34K|]}t|VdSr3)r#)rkds r zdownload..(s(881A888888r*z [nltk_hf] 'z' downloaded from z (z configs, ,z rows)r) rrr8rQr7r-rPsumvaluesprintr#) rriquietr rPconfigsrtotalr%rrmr-s @@@rdownloadry s << " "D | -i - - ** - -   &%%%%%!& .gu  BFh//IN""555555**4<88          88 88888E r}s??B )##7  ".4;;;;H@L@L@L@L@L@L@L@LH   ----`//////r*