bi dddlmZddlZddlmZddlmZeje Z d d d Z dd Z dS)) annotationsN)loggingFtensor torch.Tensor with_gradboolreturnctjrtjr|r%tjj}n[tj}fdt|D}tj|tj }||<tj |dSd}t |S)ap Gathers a tensor from each distributed rank into a list. Always retains gradients for the local rank's tensor, and optionally retains gradients for the gathered tensors if `with_grad` is True. Args: tensor (torch.Tensor): The tensor to gather from each rank. with_grad (bool, optional): If True, the local rank's tensor retains its gradients. Defaults to False. Returns: torch.Tensor: A tensor containing the gathered tensors from all ranks, concatenated along the first dimension. If torch.distributed is not available or not initialized, returns the original tensor. c8g|]}tjS)torch zeros_like).0_rs a/root/projects/butler/venv/lib/python3.11/site-packages/sentence_transformers/util/distributed.py zall_gather..s$TTTQ 0 8 8TTTr)dimzTrying to gather while torch.distributed is not available or has not been initialized, returning the original (local) tensor. This is expected if you are only using one GPU; consider not using gathering to remove this warning.) dist is_availableis_initializedr distributednn all_gatherget_world_sizerangeget_rankcatlogger warning_once)rrgathered_tensors world_size local_rankwarnings` rrr s  2t244 2  2$03>>vFF  ,..JTTTT% BSBSTTT  O,f 5 5 5J+1 Z (y)q1111 S     Mrc$t|dS)a Gathers a tensor from each distributed rank into a list, retaining gradients for the local rank's tensor. Args: tensor (torch.Tensor): The tensor to gather from each rank. Returns: torch.Tensor: A tensor containing the gathered tensors from all ranks, concatenated along the first dimension. If torch.distributed is not available or not initialized, returns the original tensor. T)r)r)rs rall_gather_with_gradr&2s f - - --r)F)rrrrr r)rrr r) __future__rr torch.distributedrrtransformers.utilsr get_logger__name__rrr&r rrr,s"""""" &&&&&&  H % %$$$$$N . . . . . .r