biddlmZddlmZddlmZddlZddlmZmZddl m Z ddl m Z Gdd ej ZdS) ) annotations)Iterable)AnyN)Tensornn)util)SentenceTransformerc`eZdZdejfdfd Zdd ZddZddZe ddZ xZ S) CoSENTLossg4@modelr scalefloatreturnNonecrt||_||_||_dS)a This class implements CoSENT (Consistent SENTence embedding) loss. It expects that each of the InputExamples consists of a pair of texts and a float valued label, representing the expected similarity score between the pair. It computes the following loss function: ``loss = logsum(1+exp(s(k,l)-s(i,j))+exp...)``, where ``(i,j)`` and ``(k,l)`` are any of the input pairs in the batch such that the expected similarity of ``(i,j)`` is greater than ``(k,l)``. The summation is over all possible pairs of input pairs in the batch that match this condition. Anecdotal experiments show that this loss function produces a more powerful training signal than :class:`CosineSimilarityLoss`, resulting in faster convergence and a final model with superior performance. Consequently, CoSENTLoss may be used as a drop-in replacement for :class:`CosineSimilarityLoss` in any training script. Args: model: SentenceTransformerModel similarity_fct: Function to compute the PAIRWISE similarity between embeddings. Default is ``util.pairwise_cos_sim``. scale: Output of similarity function is multiplied by scale value. Represents the inverse temperature. References: - For further details, see: https://penghao-bdsc.github.io/papers/CoSENT_TASLP2024.pdf Requirements: - Sentence pairs with corresponding similarity scores in range of the similarity function. Default is [-1,1]. Inputs: +--------------------------------+------------------------+ | Texts | Labels | +================================+========================+ | (sentence_A, sentence_B) pairs | float similarity score | +--------------------------------+------------------------+ Relations: - :class:`AnglELoss` is CoSENTLoss with ``pairwise_angle_sim`` as the metric, rather than ``pairwise_cos_sim``. - :class:`CosineSimilarityLoss` seems to produce a weaker training signal than CoSENTLoss. In our experiments, CoSENTLoss is recommended. Example: :: from sentence_transformers import SentenceTransformer, SentenceTransformerTrainer, losses from datasets import Dataset model = SentenceTransformer("microsoft/mpnet-base") train_dataset = Dataset.from_dict({ "sentence1": ["It's nice weather outside today.", "He drove to work."], "sentence2": ["It's so sunny.", "She walked to the store."], "score": [1.0, 0.3], }) loss = losses.CoSENTLoss(model) trainer = SentenceTransformerTrainer( model=model, train_dataset=train_dataset, loss=loss, ) trainer.train() N)super__init__r similarity_fctr )selfr r r __class__s b/root/projects/butler/venv/lib/python3.11/site-packages/sentence_transformers/losses/CoSENTLoss.pyrzCoSENTLoss.__init__s6|  , sentence_featuresIterable[dict[str, Tensor]]labelsrcLfd|D}||S)NcFg|]}|dS)sentence_embedding)r ).0sentence_featurers r z&CoSENTLoss.forward..Rs-sssM]djj!1223GHsssr)compute_loss_from_embeddings)rrr embeddingss` rforwardzCoSENTLoss.forwardQs3ssssarsss 00VDDDrr# list[Tensor]c||d|d}||jz}|dddf|dddfz }|dddf|dddfk}|}|d|z dzz }tjtjd|j|dfd}tj |d}|S)z Compute the CoSENT loss from embeddings. Args: embeddings: List of embeddings labels: Labels indicating the similarity scores of the pairs Returns: Loss value rNgmB)dim) rr rtorchcatzerostodeviceview logsumexp)rr#rscoreslosss rr"z'CoSENTLoss.compute_loss_from_embeddingsVs$$Z]JqMBB$*$46$'?246$'?21v:--EKNN--fm<rGs""""""$$$$$$ &&&&&&IIIIIIssssssssssr