Zi&N dZddlZddlZddlZddlZddlmZddlmZmZm Z ddl Z ddl m Z ddl mZmZmZmZmZmZmZddlmZmZmZmZdd lmZmZmZmZmZm Z m!Z!m"Z"e j#e$Z%e"d Z&d e'd e(e'e'ffdZ)dde e*e+fde,d e'fdZ-ddeede,d e'fdZ. dde/e eefde,de,d e/e'fdZ0 dde1de/e'de'de*de*de,d dfdZ2e&3dgd  dd!ee'e j4d" fd#ee,e j5d$d% fd&ee,e j5d'd( fded)ed df d*Z6d e'd e,fd+Z7e&3d,gd- ddddej8ddfd eee'e j4d. fdee,e j5d/d0d1 fd2ee,e j5d3d4 fd5ee,e j5d6d7d8 fd9eded)ed dfd:Z9d;ee'de,d2e,d5e,d9ede,d)ee'd dfd<Z:d e'de,d2e,d5e,d9ede,d)ee'd dfd=Z;e&3d>d?d@g  dd!ee'e j4dA fded)ed dfdBZe&3dYgdZ  dd[ee'e j4d\ fd]ee'e j4d^ fd)ed dfd_Z?e&3d`gda  ddbeee'e j4dc fddeee'e j4de fdCee,e j5df fdgee,e j5dhdi fdjee,e j5dkdl fdmeee'e j5dn fdoeee'e j5dp fdQee,e j5dRdq fdTeee/e'e j5dr fdVeee/e'e j5ds fdteee'e j5du fdvee,e j5dwdx fdyee,e j5dzd{ fd|ee,e j5d}d~d fdee,e j5ddd fd)ed df"dZ@e&3dgd  ddee'e j4d fdeee'e j4d fded)ed df dZAdS)z7Contains commands to interact with buckets via the CLI.N)datetime) AnnotatedOptionalUnion)logging) BUCKET_PREFIX BucketFile BucketFolder FilterMatcher_is_bucket_path_parse_bucket_path_split_bucket_id_and_prefix) StatusLineare_progress_bars_disableddisable_progress_barsenable_progress_bars) FormatOpt OutputFormatQuietOptTokenOptapi_object_to_dict get_hf_apiprint_list_output typer_factoryz"Commands to interact with buckets.)helpargumentreturnc|trt|S t|S#t$rt d|dtdwxYw)zParse a bucket argument accepting both 'namespace/name(/prefix)' and 'hf://buckets/namespace/name(/prefix)'. Returns: tuple: (bucket_id, prefix) where bucket_id is "namespace/bucket_name" and prefix may be empty string. zInvalid bucket argument: -. Must be in format namespace/bucket_name or znamespace/bucket_name) startswithrr r ValueError)rs V/root/projects/butler/venv/lib/python3.11/site-packages/huggingface_hub/cli/buckets.py_parse_bucket_argumentr$8s =)),!(+++ *8444     8 8 8  8 8 8    s :&A Fsizehuman_readablec|st|SdD]&}|dkr|dkr |d|cS|dd|cS|dz}'|ddS)zFormat a size in bytes.)BKBMBGBTBir( z.1fz PB)str)r%r&units r# _format_sizer0Is 4yy- $;;s{{'''''''''''' ' ' '      mtimecb|dS|r|dS|dS)z0Format mtime datetime to a readable date string.Nz %b %d %H:%Mz%Y-%m-%d %H:%M:%S)strftime)r2r&s r# _format_mtimer6Ws9 }r-~~m,,, >>- . ..r1itemsquietcNi}|D]t}|jd}|}|ddD]}||vrdii||<||d}|d}t|tr ||vrdii||<md|i||<ud} d} d} |s|D]{}t|trdt |j|} t| t| } t|j |} t| t| } || dkr| dz| z} g}t||d| | | |S) aBuild a tree representation of files and directories. Produces ASCII tree with size and date columns before the tree connector. When quiet=True, only the tree structure is shown (no size/date). Args: items: List of BucketFile/BucketFolder items human_readable: Whether to show human-readable sizes and short dates quiet: If True, show only the tree structure without sizes/dates Returns: List of formatted tree lines /N __children____item__rr4 prefix_widthmax_size_widthr&) pathsplit isinstancer r r0r%maxlenr6r2 _render_tree)r7r&r8treeitempartscurrentpart final_partr@rAmax_date_widthsize_strdate_strliness r# _build_treerR`s&D 5 5 $$#2#J 4 4D7""!/ 4 dmN3GG2Y dL ) ) 5(('5r&: ##-t"4GJ  LNN ? D DD$ ++ D' >BB!$^S]]!C!C(^DD!$^S]]!C!C A  )A->LE   !%%  Lr1noderQindentr@rAc tt|}t|D]\}\}} |t|dz k} | rdnd} d| v} | di} |dkr| rd|z}nR| d}|6t |j|}t|j|}|d |d |}nd|z}| |d || || rd nd n!| || || rd nd | r|| rd ndz}t| |||||dS)z:Recursively render a tree structure with size+date prefix.ru └── u ├── r<rr-r=N> r:r4z u│ r?) sortedr7 enumeraterFgetr0r%r6r2appendrG)rSrQrTr@rAr&r7inamevalueis_last connectoris_dirchildrenprefixrIrOrP child_indents r#rGrGs 4::<< E%e,,=D%s5zzA~%$+=LL 5(99^R00 !   0|+yy,,#+DI~FFH,TZHHH (H>HHHHhHHFF </F LLFVVfViVVf?TssRTVV W W W W LLFLILtLF5JSSLL M M M  !w%DVVHEL )--     1r1create)zhf buckets create my-bucketz hf buckets create user/my-bucketz-hf buckets create hf://buckets/user/my-bucketz*hf buckets create user/my-bucket --privatez+hf buckets create user/my-bucket --exist-ok)r]examples bucket_idzTBucket ID: bucket_name, namespace/bucket_name, or hf://buckets/namespace/bucket_nameprivatez --privatezCreate a private bucket.exist_okz --exist-okz3Do not raise an error if the bucket already exists.tokenct|}|trk t|\}}n3#t$r&}t jt|d}~wwxYw|r t jd|dtd|}|||r|nd|} |rt| j dStd| j d| j d dS) zCreate a new bucket.rjNz-Cannot specify a prefix for bucket creation: . Use namespace/bucket_name or namespace/bucket_name.)rhrizBucket created: z (handle: )) rr!rr$r"typer BadParameterr. create_bucketprinthandleurl) rgrhrir8rjapi parsed_idrce bucket_urls r#reresBF 5 ! ! !CM**  - 6y A A Ivv - - -$SVV,, , -  $W WW1>WWW  """,#J  Q j      OOO:;LOOOPPPPPs? A/ !A**A/c||tr|ttd}n|}d|vS)zFCheck if argument is a bucket ID (namespace/name) vs just a namespace.Nr:)r!rrF)rrBs r# _is_bucket_idr{sA=))M**,,- $;r1z list | ls) zhf buckets listzhf buckets list huggingfacezhf buckets list user/my-bucketz!hf buckets list user/my-bucket -Rz!hf buckets list user/my-bucket -hz%hf buckets list user/my-bucket --treez(hf buckets list user/my-bucket --tree -hz+hf buckets list hf://buckets/user/my-bucketz%hf buckets list user/my-bucket/sub -RzyNamespace (user or org) to list buckets, or bucket ID (namespace/bucket_name(/prefix) or hf://buckets/...) to list files.z--human-readablez-hz$Show sizes in human readable format.as_treez--treez3List files in tree format (only for listing files). recursivez --recursivez-Rz0List files recursively (only for listing files).formatc |duot|}|rt|||||||dSt|||||||dS)zList buckets or files in a bucket. When called with no argument or a namespace, lists buckets. When called with a bucket ID (namespace/bucket_name), lists files in the bucket. N)rr&r|r}r~r8rj) namespacer&r|r}r~r8rj)r{ _list_files _list_buckets)rr&r|r}r~r8rj is_file_modes r#list_cmdrsr4'CM(,C,CL )       )      r1rcb|rtjd|rtjd|K|tr1|t td}|d}t |}d||D}|sE|sC|tj kr3||n| d} td | d dSgd } d td ttffd } ddd} t|||| | | dS)zList buckets in a namespace.z'Cannot use --tree when listing buckets.z,Cannot use --recursive when listing buckets.Nr:rlc,g|]}t|Sr).0buckets r# z!_list_buckets..xs!^^^f!&))^^^r1)rr]z"No buckets found under namespace ''.)idrhr% total_files created_atrIrc Dddlm}||d||dt|dd||d||d gS) Nr) _format_cellrrhr%rr&rr) _cli_utilsrrZr0)rIrr&s r#row_fnz_list_buckets..row_fns,,,,,, L$ ( ( L),, - - &!,,^ L L L L-00 1 1 L,// 0 0   r1right)r%r)r~r8headersr alignments)rprqr!rrFrstripr list_bucketsrjsonwhoamirsdictlistr.r) rr&r|r}r~r8rjrvresultsresolved_namespacerrrs ` r#rras}L !JKKKQ !OPPP!5!5m!D!Dc-00223 $$S)) 5 ! ! !C^^8H8HS\8H8]8]^^^G  <#444.7.CV\I]  M7IMMM N N N FDDDG  T  d3i       "'::JgfE7SYfpqqqqqqr1c |r$|tjkrtjdt |} t |\}} n3#t $r&} tjt| d} ~ wwxYwt| || pd|} | stddStd| D} |tjkr1d| D} ttj | d n |r't| || }|D]}t|n|rG| D]C}t|trt|jd /t|jDn| D]}t|tr5t#|j|}td d d|dd|jd Lt'|j|}t#|j|}t|d d|dd|j|s%| r%t-ddSdSdS)zList files in a bucket.z%Cannot use --tree with --format json.rlNrcr}z(empty)c3@K|]}t|tVdSN)rDr rrIs r# z_list_files..s,KKT*T<88KKKKKKr1c,g|]}t|Srrrs r#rz_list_files..s!>>>%d++>>>r1r>rT)r&r8r:r4z>12rWz>19z!Use -R to list files recursively.)rrrprqrr$r"r.rlist_bucket_treersanydumpsrRrDr rBr6 uploaded_atr0r%r2rdone)rr&r|r}r~r8rjrvrgrcrxr7has_directoriesr tree_lineslinerI mtime_strrOs r#rrsJ6\... !HIII 5 ! ! !C)28<< 66 ))) Q((()   >T     E  iKKUKKKKKO """>>>>> dj+++,,,, G ~USSS   D $KKKK  G ! !D$ -- !ooo&&&&di     ! G GD$ -- G)$*:NKK @@@9@@@DI@@@AAAA' >BB)$*nEE EEEEEE$)EEFFFF ?? =>>>>>????sA A;!A66A;infozhf buckets info user/my-bucketz+hf buckets info hf://buckets/user/my-bucketzFBucket ID: namespace/bucket_name or hf://buckets/namespace/bucket_namecnt|} t|\}}n3#t$r&}tjt |d}~wwxYw||}|rt|jdSttj t|ddS)zGet info about a bucket.rlNr>r) rr$r"rprqr. bucket_inforsrrrr)rgr8rjrvrw_rxrs r#rrs$ 5 ! ! !C)-i88 11 ))) Q((()__Y ' 'F @ fi dj+F33A>>>?????s% A!AAdelete)z hf buckets delete user/my-bucketz-hf buckets delete hf://buckets/user/my-bucketz&hf buckets delete user/my-bucket --yesz-hf buckets delete user/my-bucket --missing-okyesz--yesz-yzSkip confirmation prompt. missing_okz --missing-okz3Do not raise an error if the bucket does not exist.cn|trl t|\}}n3#t$r&}t jt |d}~wwxYw|r t jd|dtd|}n$d|vr t jd|dtd|sWWW  I    a) a a;H a a a    - X9 X X XYY *   +--  5 ! ! !CiJ777 . i ,,,-----s/ A!AAz remove | rm)z)hf buckets remove user/my-bucket/file.txtz2hf buckets rm hf://buckets/user/my-bucket/file.txtz.hf buckets rm user/my-bucket/logs/ --recursivez:hf buckets rm user/my-bucket --recursive --include "*.tmp"z8hf buckets rm user/my-bucket/data/ --recursive --dry-runzBucket path: namespace/bucket_name/path or hf://buckets/namespace/bucket_name/path. With --recursive, namespace/bucket_name is also accepted to target all files.z0Remove files recursively under the given prefix.dry_runz --dry-runz8Preview what would be deleted without actually deleting.includezQInclude only files matching pattern (can specify multiple). Requires --recursive.excludezLExclude files matching pattern (can specify multiple). Requires --recursive.c t|\}} n3#t$r&} tjt | d} ~ wwxYw| dkr|stjd|d|d|s|r|stjdt |} |rOt | } | d g} | || d pdd D]R}t|tr;| || d t| dS| d t| d|s|r t||fd| D}n| }d|D}t!d|D}t#|d }|s|st%ddSt|d|}|sZ|sX|s|D]}t%d|tjd|d|d}|s"t%dtj|r6|D]}t%dt*|d |t%d|ddS| |||r|D]}t%|dS|D]}t%dt*|d |t%d|d|d dS| d }|stjd!|r-t%dt*|d |t%d"dS|s?tjd#|d$|d}|s"t%dtj| ||g|rt%|dSt%dt*|d |dS)%zbRemove files from a bucket. To delete an entire bucket, use `hf buckets delete` instead. Nr4z?No file path specified. To remove files, provide a path (e.g. 'zd/FILE') or use --recursive to remove all files. To delete the entire bucket, use `hf buckets delete z`.z,--include and --exclude require --recursive.rl)enabledzListing files from remoter:TrzListing files from remote (z files))include_patternsexclude_patternscHg|]}|j|Sr)matchesrB)rfmatchers r#rzremove..s,MMM1W__QV5L5LMQMMMr1cg|] }|j Sr)rBrrs r#rzremove..s444af444r1c3$K|] }|jV dSr)r%rs r#rzremove..s$77A777777r1rzNo files to remove.z file(s) totaling rWzRemove z from 'rrzdelete: z (dry run) z would be removed.)rzRemoved rzFile path cannot be empty.z"(dry run) 1 file would be removed.zRemove 'z' from ')r$r"rprqr.rrupdaterrrDr r[rFrr sumr0rsrrrbatch_bucket_files)rr}rrrrr8rjrgrcrxrvstatus all_filesrI matched_files file_paths total_sizerO count_labelrBr file_pathrs @r#remover7s+|)28<< 66 ))) Q((()||I|  R  R RDM R R R   Q7QIQ !OPPP 5 ! ! !CLEI... 1222&( (( ==%%-)   U UD $ ++ U  &&& SC NNSSSTTT I#i..IIIJJJ  &g &#WwWWWGMMMM MMMMM%M44m444 7777777  4@@@  -+,,, FZFFHFF  $7 $ '&''D+t++&&&&m$Ok$O$O)$O$O$OPPG $j!!!kmm#  " D DBB BBDBBCCCC >{>>> ? ? ? F y<<<  @"  d   # D DBB BBDBBCCCC >[>>>>> ? ? ? ? ?MM#&&  C$%ABB B   C]CICC CC D D D 6 7 7 7 F $m$Oy$O$O)$O$O$OPPG $j!!!kmm# y)===  E )      C]CICC CC D D D D Ds A!AAmove)z/hf buckets move user/old-bucket user/new-bucketz/hf buckets move user/my-bucket my-org/my-bucketzIhf buckets move hf://buckets/user/old-bucket hf://buckets/user/new-bucketfrom_idzMSource bucket ID: namespace/bucket_name or hf://buckets/namespace/bucket_nameto_idzRDestination bucket ID: namespace/bucket_name or hf://buckets/namespace/bucket_namecNt|\}}|r tjd|dtdt|\}}|r tjd|dtdt |}|||t d|d|dS) z2Move (rename) a bucket to a new name or namespace.z)Cannot specify a prefix for bucket move: rmrnrl)rrzBucket moved:  -> N)r$rprqrr move_bucketrs)rrrjparsed_from_id from_prefix parsed_to_id to_prefixrvs r#rrs2#9"A"ANK   S S S-: S S S   5U;;L)   S S S-: S S S   5 ! ! !COON,O??? => = =| = =>>>>>r1sync)z2hf buckets sync ./data hf://buckets/user/my-bucketz2hf buckets sync hf://buckets/user/my-bucket ./dataz;hf buckets sync ./data hf://buckets/user/my-bucket --deletez^hf buckets sync hf://buckets/user/my-bucket ./data --include "*.safetensors" --exclude "*.tmp"zIhf buckets sync ./data hf://buckets/user/my-bucket --plan sync-plan.jsonlz'hf buckets sync --apply sync-plan.jsonlzhf buckets cp my-config.json hf://buckets/user/my-bucket/logs/zKhf buckets cp my-config.json hf://buckets/user/my-bucket/remote-config.jsonz7hf buckets cp - hf://buckets/user/my-bucket/config.jsonsrcz9Source: local file, hf://buckets/... path, or - for stdindstz?Destination: local path, hf://buckets/... path, or - for stdoutc  t|}t|}|duot|}|dk}|dk}|r|rtjd|s.|s,|s*|tjdtjd|r|stjd|rG|rE|Jt |\} } | dks| d rtjd |r|stjd |s5|s3t j|rtjd |rt |\} } | dks| d rtjd | d dd} |r t} | st tj 5}t j|| }|| | |fgt!|d5}t"jj|dddn #1swxYwYdddn #1swxYwY| st-dSdS#| st-wwxYw|| }nvt j|s4|t js|d r!t j|| }n|}t j|}|rt j|d|rt || | |fg|rt-n#|rt-wwxYw|st5d|d|dSdS|rt |\} }t"jj}|rt || ||fg|rt-n#|rt-wwxYw|st5d|dSdSt j|stjd|t |\} } | dkr t j|}n:| d r#| t j|z}n| }|rt || ||fg|rt-n#|rt-wwxYw|s!t5d|dt>| d |dSdS)z'Copy a single file to or from a bucket.rlN-z$Remote-to-remote copy not supported.z2Missing destination. Provide a bucket path as DST.z;One of SRC or DST must be a bucket path (hf://buckets/...).z+Stdin upload requires a bucket destination.r4r:zAStdin upload requires a full destination path including filename.z"Cannot pipe to stdout for uploads.zNSource must be a file, not a directory. Use `hf buckets sync` for directories.zJSource path must include a file name, not just a bucket or directory path.rr;rbT)riz Downloaded: r)addzUploaded: stdin -> zSource file not found: z Uploaded: ) rr rprqr endswithosrBisdirrsplitrrtempfileTemporaryDirectoryjoindownload_bucket_filesopensysstdoutbufferwritereadrsepdirnamemakedirsrsstdinrisfilebasenamer)rrr8rjrv src_is_bucket dst_is_bucket src_is_stdin dst_is_stdoutrrcrgfilenamepbar_was_disabledtmp_dirtmp_pathr local_path parent_dir remote_pathdatas r#rrs, 5 ! ! !C#C((MtO<(<(>H--i68:L9MNNNh--: )//999:::::::::::::::::::::::::::::: )+(*****++(+(****+{% s## !s||BF';'; !s||C?P?P !W\\#x88  44J 7 J6666 (%''' +)))vz6J5KLLL+(***+(****+ <:S::j::;;;;; < < 'R!3C!8!8 ;y$$&&  $ ! # # # '  " "9D+3F2G " H H H '$&&& '$&&&& ' / --- . . . . . / / w~~c"" F$%Ds%D%DEE E.s33 6 R<<'**3//KK __S ! ! ! 27#3#3C#8#88KK K  $ ! # # # '  " "9C3E2F " G G G '$&&& '$&&&& ' R PsPP PyPP;PP Q Q Q Q Q R Rsy9J A I07I I0I I0 I !I0$ J0I44J7I48JJ"(NN% P66Q UU)F)FF)rrF)FFFN)FN)FFFNNFNr)NNFFFNNFNNNFFFFN)NFN)B__doc__rrrrrtypingrrrrphuggingface_hubrhuggingface_hub._bucketsrr r r r r rhuggingface_hub.utilsrrrrrrrrrrrrr get_logger__name__logger buckets_clir.tupler$intfloatboolr0r6rrRrrGcommandArgumentOptionrer{tablerrrrrrrrrrr1r#r&s>= ---------- ######vuuuuuuuuuuu                      H % %m!EFFF  S U38_    "  uS%Z( $ 3    //(+/T/c////!<< j,./ 0<< < #Y <<<<F )) ) 9) ) )  )  ) ))))X    .  -/Q/Q g    /Q  +    /Q  F    /Q* +/Q, -/Q. //Q/Q/Q  /QdCD       .    $*G@ @  W    @    7    @ $  F    %@ 2   C    3@ B C@ D E@ F G@ H I@ @ @   @ F-r}-r-r-r -r  -r  -r C=-r -r-r-r-r`>?>?>?>? >?  >?  >? C=>? >?>?>?>?B  (5@@ Y    @ @ @ @@@@2  .  /9.9. Y    9.    ,     9.  F    9., -9.. /9.0 19.9.9.9.x    6    ]QEQE a    QE   C    QE$    ,     %QE4  K    5QEBc d    CQENc _    OQEZ [QE\ ]QE^ _QEQEQE  QEh  *"?"?  `    "?  e     "? "? "?"?"?"?T       &          Ixx  ^     x  c     x   B     x&  N    'x4  O    5xB   P     CxN   6     OxZ  H    [xhc I    ixtc I    ux@  ;    AxL  T    MxZ  V    [xh   8    ixx    "     yxH IxJ Kxxx  x@       "  RR 3,ghhhh iR  ~u~+lmmmm R  R  R RRR  RRRr1