biC ddlmZddlZddlZddlZddlmZddlmZddl Z ddl Z ddl m Z ddl mZddlmZdd lmZejeZerdd lmZd d d dddedddf d8d&Zd dddefd9d)Zd:d+Zddd,efd;d/Z d) annotationsN)Callable) TYPE_CHECKING)Tensor)tqdm)cos_sim)normalize_embeddings)SentenceTransformerF iii dmodelr sentences list[str]show_progress_barbool batch_sizeintquery_chunk_sizecorpus_chunk_size max_pairstop_kscore_function"Callable[[Tensor, Tensor], Tensor] truncate_dim int | None prompt_name str | Nonepromptreturnlist[list[float | int]]c d||||d| | | } t| |||||S)a@ Given a list of sentences / texts, this function performs paraphrase mining. It compares all sentences against all other sentences and returns a list with the pairs that have the highest cosine similarity score. Args: model (SentenceTransformer): SentenceTransformer model for embedding computation sentences (List[str]): A list of strings (texts or sentences) show_progress_bar (bool, optional): Plotting of a progress bar. Defaults to False. batch_size (int, optional): Number of texts that are encoded simultaneously by the model. Defaults to 32. query_chunk_size (int, optional): Search for most similar pairs for #query_chunk_size at the same time. Decrease, to lower memory footprint (increases run-time). Defaults to 5000. corpus_chunk_size (int, optional): Compare a sentence simultaneously against #corpus_chunk_size other sentences. Decrease, to lower memory footprint (increases run-time). Defaults to 100000. max_pairs (int, optional): Maximal number of text pairs returned. Defaults to 500000. top_k (int, optional): For each sentence, we retrieve up to top_k other sentences. Defaults to 100. score_function (Callable[[Tensor, Tensor], Tensor], optional): Function for computing scores. By default, cosine similarity. Defaults to cos_sim. truncate_dim (int, optional): The dimension to truncate sentence embeddings to. If None, uses the model's ones. Defaults to None. prompt_name (Optional[str], optional): The name of a predefined prompt to use when encoding the sentence. It must match a key in the model `prompts` dictionary, which can be set during model initialization or loaded from the model configuration. Ignored if `prompt` is provided. Defaults to None. prompt (Optional[str], optional): A raw prompt string to prepend directly to the input sentence during encoding. For instance, `prompt="query: "` transforms the sentence "What is the capital of France?" into: "query: What is the capital of France?". Use this to override the prompt logic entirely and supply your own prefix. This takes precedence over `prompt_name`. Defaults to None. Returns: List[List[Union[float, int]]]: Returns a list of triplets with the format [score, id1, id2] T)rrconvert_to_tensorrrr)rrrrr)encodeparaphrase_mining_embeddings) rrrrrrrrrrrr embeddingss _/root/projects/butler/venv/lib/python3.11/site-packages/sentence_transformers/util/retrieval.pyparaphrase_miningr(s]\+!J ()+%    r&rc |dz }tj}d}d}tdt||D]x} tdt||D]U} ||| | |z|| | |z} t j| t |t| dddd\} } | } | } tt| D]}t| |D]r\}}| |z}| |z}||kr]| |||krK| | ||||f|dz }||kr| }|d}sWzt}g}| s{| \}}}t||g\}}||kr5||f|vr/|||f||||g| {t|dd}|S) a Given a list of sentences / texts, this function performs paraphrase mining. It compares all sentences against all other sentences and returns a list with the pairs that have the highest cosine similarity score. Args: embeddings (Tensor): A tensor with the embeddings query_chunk_size (int): Search for most similar pairs for #query_chunk_size at the same time. Decrease, to lower memory footprint (increases run-time). corpus_chunk_size (int): Compare a sentence simultaneously against #corpus_chunk_size other sentences. Decrease, to lower memory footprint (increases run-time). max_pairs (int): Maximal number of text pairs returned. top_k (int): For each sentence, we retrieve up to top_k other sentences score_function (Callable[[Tensor, Tensor], Tensor]): Function for computing scores. By default, cosine similarity. Returns: List[List[Union[float, int]]]: Returns a list of triplets with the format [score, id1, id2] rrTFdimlargestsortedc|dS)Nrxs r'z.paraphrase_mining_embeddings..s !A$r)keyreverse)queue PriorityQueuerangelentorchtopkmincputolist enumerateputgetsetemptyr/addappend)r&rrrrrpairs min_score num_addedcorpus_start_idxquery_start_idxscoresscores_top_k_valuesscores_top_k_idx query_itr top_k_idx corpus_itrijentry added_pairs pairs_listscoresorted_isorted_js r'r%r%Ys0 QJE   ! !EII!!S__6GHH11$QJ9IJJ 1 1O#^?_?O-OOP+.>AR.RRSF 5:JE3vay>>224PU555 1 !1#6"9"9";";"B"B"D"D /3355<<>> "3v;;// 1 1 -67G 7R-S-S 1 1)Iz')3A(:5AAvv"5i"@"Ki"W"W #6y#A)#LaQR"STTT!Q $ 11$)IIKKE(-aI 1 1 14%%KJkkmm;iikk q!#QF^^( x  Xx$8 $K$K OOXx0 1 1 1   uh9 : : : kkmm; EEEJ r)"list[list[dict[str, int | float]]]ct|i|S)z8This function is deprecated. Use semantic_search instead)semantic_search)argskwargss r'information_retrievalr`s D +F + ++r) query_embeddingscorpus_embeddingsc t|tjtjfrt j|}n)t|t rt j|}t|j dkr| d}t|tjtjfrt j|}n)t|t rt j|}|j |j kr| |j }dtt|D}tdt||D]"}t||zt|}|jr3t j|||j } |d| } n |||} tdt||D]} t| |zt|} |jr3t j| | |j } |d| } n || | } || | }t j|t|t|dddd\}}|}|}tt|D]}t+||||D]c\}}| |z}||z}t|||krt-j||||fFt-j||||fd$tt|D]b}tt||D]!}|||\}}||d|||<"t3||d d ||<c|S) a3 This function performs by default a cosine similarity search between a list of query embeddings and a list of corpus embeddings. It can be used for Information Retrieval / Semantic Search for corpora up to about 1 Million entries. Args: query_embeddings (:class:`~torch.Tensor`): A 2 dimensional tensor with the query embeddings. Can be a sparse tensor. corpus_embeddings (:class:`~torch.Tensor`): A 2 dimensional tensor with the corpus embeddings. Can be a sparse tensor. query_chunk_size (int, optional): Process 100 queries simultaneously. Increasing that value increases the speed, but requires more memory. Defaults to 100. corpus_chunk_size (int, optional): Scans the corpus 100k entries at a time. Increasing that value increases the speed, but requires more memory. Defaults to 500000. top_k (int, optional): Retrieve top k matching entries. Defaults to 10. score_function (Callable[[:class:`~torch.Tensor`, :class:`~torch.Tensor`], :class:`~torch.Tensor`], optional): Function for computing scores. By default, cosine similarity. Returns: List[List[Dict[str, Union[int, float]]]]: A list with one entry for each query. Each entry is a list of dictionaries with the keys 'corpus_id' and 'score', sorted by decreasing cosine similarity scores. rrcg|]}gSr1r1).0_s r' z#semantic_search..sDDD!2DDDr)deviceTFr,) corpus_idrXc|dS)NrXr1r2s r'r4z!semantic_search..s \]^e\fr)r5) isinstancenpndarraygenericr< from_numpyliststackr;shape unsqueezerjtor:r> is_sparsearange index_selectr=r?r@zipheapqheappush heappushpopr/)rbrcrrrrqueries_result_listrL query_end_idxindices query_chunkrKcorpus_end_idx corpus_chunk cos_scorescos_scores_top_k_valuescos_scores_top_k_idxrP sub_corpus_idrXrkquery_iddoc_itrs r'r]r]se0"RZ$<==9 +,<== $d + +9 ;'788  !""a''+55a88#bj"*%=>>;!,->?? %t , ,;!K(9::#3#:::+../@/GHHDDuS1A-B-B'C'CDDD C(8$9$9;KLL$]$]O.>>DT@U@UVV  % Jl?MJZJabbbG*777CCKK*?=+HIK!&a->)?)?AR S S ] ]  !14E!EsK\G]G]^^N * R,'7PaPhiii0==aII 01A.1PQ ( \BBJ=BJCs:a='9'9::4X]=== 9 #%9'>&A&A&C&C&J&J&L&L ##7#;#;#=#=#D#D#F#F "3z??33 ] ] ,/0DY0OQhirQs,t,t]](M5 0= @I.:H.x899EAA/9E9;M)*=h*G%QZI[\\\\] ]% ]<#12233vvS!4X!>??@@ ^ ^G28.Ss Ar)r5c t|Srrr2s r'r4z%community_detection..cs #a&&r))rmr<rtensorrjr r>r;maxrr:Ttypesumanyr=rzrGr@r/rDrAupdate)r&rrrrextracted_communities sort_max_size start_idxrthreshold_maskrow_wise_countlarge_enough_maskrrg top_k_indicescountr top_k_valuesrS top_val_large top_idx_largeunique_communities extracted_ids cluster_id communitynon_overlapped_communityidxs r'community_detectionrs0 j%, / /.\*--  Yz/@AAAI%j11J/ZAAA 22B77ZIIM aZ*--4FTePe*e*e  I ,B BCjlR    !_ 4 4'94N+//22N!/2D D $((** +,=>N#$56J""$$A)DAA A}#&nm"D"D G Gw%,,WVeV_-C-C-E-EFFFF G)oo0BDoQQOL!3|,,-- e e?2&)333=a=3E3E _c3E3d3d0M=(+i77MCPZOO<[<[(+A ,=s:(O(O 7A!}7I7IMcg7I7h7h4 }(+i77MCPZOO<[<[*00}PY?Y1Z1a1a1c1cddd e##8>N>NX\]]]EEM!*+@!A!A;; I#%  5 5C-''(//444 ' ( (,> > >  % %&> ? ? ?  !9 : : : 28H8HRVWWW r))rr rrrrrrrrrrrrrrrrrrrrrrr r!)r&rrrrrrrrrrrr r!)r r[)rbrrcrrrrrrrrrr r[)rrarF) r&rrrrrrrrrr r) __future__rr{loggingr8collections.abcrtypingrnumpyrnr<rtqdm.autonotebookr similarityr rr getLogger__name__logger)sentence_transformers.SentenceTransformerr r(r%r`r]rr1r)r'rs""""""  $$$$$$  """"""((((((  8 $ $NMMMMMM $ #9@#"?????H!#9@ FFFFFR,,,, #9@ XXXXXz # cccccccr)