jdZ ddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddlmZddlmZddlmZmZdZdZdadadZd Zd3d Zd Z d4dZeddZejdZ ejdZ!dZ"dZ#dZ$d5dZ%Gdde j&j'Z(dZ)dZ*Gddej+j,Z-Gdd ej+j.Z/Gd!d"e j&j0Z1Gd#d$e j&j2Z3d%Z4d&Z5d'Z6d(Z7d)Z8d*Z9d+Z:e:Z;d6d-dd.d/Z<Gd0d1e j=Z=gd2Z>dS)7z+Centralized I/O security sentinel for NLTK.N) lru_cache)Path)unquoteurlparseTFc0 tj|}n#t$rYdSwxYwtj|jsdStjdkrdS|jtjdfvrdS|jtjtj zzrdSdS)aReturn True if *path* is a directory safe to trust as a data root: another *unprivileged local user* cannot plant files in it. This is the test that distinguishes a shared, world-writable temp directory (Linux ``/tmp``, mode ``1777``) -- which a local attacker could use to plant trusted-looking data (CWE-377/CWE-378) -- from a *private* per-user temp directory (macOS ``$TMPDIR`` ``/var/folders/...`` mode ``0700``, Windows ``%TEMP%`` under the ACL-protected user profile), which is safe. Cross-platform: * POSIX: the directory must be owned by the current user (or root) and be neither group- nor world-writable (unless the writable group/other bit is paired with the sticky bit *and* the dir is user-owned -- but we keep it strict and simply reject any group/other-writable dir). * Windows: ``st_uid``/mode are not meaningful; the per-user temp/profile is ACL-protected, so a plain "exists and is a directory" check is used. FposixTr) osstatOSErrorS_ISDIRst_modenamest_uidgetuidS_IWGRPS_IWOTH)pathsts o/home/lighthouse/badge-speaker-benchmark/.benchmark-venv-community/lib/python3.11/site-packages/nltk/pathsec.pyis_private_dirr*s$ WT]] uu < # #u w't ya(((u zT\DL01u 4s  %%ctg}dtjvr.tttjddg}tjdd}||f}tt|krtSt}|| tj zD]}|r t|dr|j n|}|tt!|r#t&t(t*f$rYwxYwddg}|D]|} t|}|r||`#t&t(t*f$rYywxYw tt/j}t3|r||n#t&t(t*f$rYnwxYw|a|a|S)zDDynamically determines allowed directories based on NLTK data paths.z nltk.datar NLTK_DATANz ~/nltk_dataz/usr/share/nltk_data)sysmoduleslistgetattrr environget_ALLOWED_ROOTS_CACHE_LAST_DATA_PATHSsetsplitpathsephasattrraddrstr expanduserresolver ValueError RuntimeErrorexiststempfile gettempdirr) current_paths env_paths current_staterootspraw_pcandidate_locsloctmps r_get_allowed_rootsr8Os1Mck!!WS[%=vrJJKK  {B//I"I.M',< ,M,M## EEE Y__RZ88 8   ")!V"4"4;! $s5zz**5577??AABBBBZ6      $%;z validate_path..s:WWv~<!6!6t!.s'==tsd{======rHzS]: CWD access restricted in ENFORCE mode. Authorize via: nltk.data.path.append('.')zSecurity Warning [z allowed via CWD. stacklevelz]: Unauthorized path ) isinstanceintr'stripr%rrschemerrr)r r*lowerfindrCr8anyr getcwdENFORCEPermissionErrorwarningswarnRuntimeWarning Exception) path_inputcontext required_rootrawparsed lower_rawzip_idxroot_raw scoped_root allowed_rootsmsgrKrFs @@r validate_pathrhs*c""*C OOK+EB(K+,A(J?JK+J,)K++J,,=K++#LLctjjr|ddn|}tj|ds|drdStjj|dvS)aTrue if a ZIP member is written somewhere other than where it is validated. ``zipfile.ZipFile.extract`` sanitises a member name by *dropping* the drive and every empty / ``.`` / ``..`` component while keeping the rest, whereas ``Path.resolve`` collapses a ``..`` against its *preceding* component. For a member such as ``a/../b/x`` the two disagree: it is validated as ``/b/x`` but written to ``/a/b/x``, which can escape through a pre-existing symlink at ``/a/b`` that the collapsed validation path never visits. The mismatch only ever arises from absolute / drive-qualified / ``..`` members -- exactly the shapes a legitimate archive never uses -- so rather than keep two different normalizations in sync we reject them outright. This both closes the validate/extract gap and is the proactive block the hardened extractor promises (CWE-22 / CWE-59). \/rT)r raltsepreplace splitdrive startswithpardirr#)name_str normalizeds r_zip_member_is_unsaferss{"13L!!$,,,HJ w(##A&**?*?*D*Dt 7>Z--c22 22rHZipAuditc t|fd}t|tjr ||dStj|d5}||ddddS#1swxYwYdS#t t f$rttjf$rtrt dYdSwxYw)zEEnhanced Zip-Slip protection using Pathlib for cross-platform safety.cgn|}|D]}t|dr|jnt|}d|vrt d||z }t |s|ksP|s;dd|d}trt|tj |tddS) NfilenamezNull byte in ZIP member: r@z]: Traversal member 'z ' detected.rLrM) namelistr%rwr'r*r)rsrCrWrXrYrZr[) zfmembersrrq member_pathrgr^specific_memberrFs r_auditz$validate_zip_archive.._audits%4%@!!bkkmm   I I,3D*,E,ET4==3t998##$%K%K%KLLL &099;; (22I6))[-G-G-O-O)dddxdddCI-c222 c>aHHHH# I IrHrNzZip validation failed) rr)rOzipfileZipFilerXr*r BadZipFilerW)zip_obj_or_path target_rootr}r^r~rzrFs `` @rvalidate_zip_archivers\#;k""**,, I I I I I I I. ow 7 7  F? # # # # ##66 "r                    Z ( W' (;;;  ;!"9:: : ; ; ;;s<A BB) B5 BBB B B=CC)maxsizecx tj|dtjS#ttf$rgcYSwxYw)a?Cached hostname resolution for the early SSRF pre-check. Note: the cache alone does NOT prevent DNS rebinding, because the connection layer re-resolves the hostname independently. The actual rebinding protection is the connect-time IP pinning in ``_SafeHTTPConnection`` / ``_SafeHTTPSConnection``. Nproto)socket getaddrinfo IPPROTO_TCPr r*)hostnames r_resolve_hostnamersK!(D8JKKKK Z  s #99z 64:ff9b::/96z::/96ct|tjsdS|j}||S|tvs |t vr!tj|jddSdS)a<The embedded IPv4 for IPv6 forms that *are* an IPv4 address, else None. Covers IPv4-mapped (``::ffff:0:0/96``), IPv4-compatible (``::/96``) and the NAT64 well-known prefix (``64:ff9b::/96``). For these the IPv6 wrapper has no independent routable meaning, so the embedded IPv4 is what gets reached. N)rO ipaddress IPv6Address ipv4_mapped_NAT64_WELL_KNOWN_IPV4_COMPATIBLE IPv4Addresspacked)ipmappeds r_embedded_ipv4r1sg b)/ 0 0t ^F   "(8"8"8$RYrss^444 4rHc#Kt|tjsdS|j}||V|j}| |Ed{VdSdS)zBIPv4 addresses tunneled by routable IPv6 wrappers (6to4 / Teredo).N)rOrr sixtofourteredo)rrrs r_tunneled_ipv4srBsi b)/ 0 0 I YF rHct|}||}t|D]}|js|jsdS|jp|j S)amReturn True if the SSRF filter must refuse to connect to ``ip``. Policy (defense in depth): only *globally routable* addresses are allowed; anything that is not global -- loopback, link-local, private, carrier-grade NAT (100.64.0.0/10), reserved, unspecified (``0.0.0.0`` / ``::``), documentation ranges, etc. -- is forbidden. This generalises the previous explicit ``loopback / link-local / multicast / private`` list and is a strict superset of it. Multicast is still rejected explicitly because some CPython versions classify multicast addresses as ``is_global``. IPv6 addresses that embed an IPv4 address are evaluated by that embedded IPv4, not by the wrapper: the stdlib classifies the wrappers (IPv4-mapped, IPv4-compatible, NAT64 ``64:ff9b::/96``, 6to4 ``2002::/16``, Teredo ``2001:0::/32``) as globally routable, so a forbidden internal IPv4 (loopback, the link-local cloud-metadata address, ...) could otherwise be smuggled past the check and then routed to that IPv4 by a NAT64/6to4/Teredo gateway (CWE-918). This extends the previous IPv4-mapped-only unwrap. NT)rr is_multicast is_global)rembeddedtunneleds r_ip_is_forbiddenrNsf&b!!H #B''   (: 44  ? .",..rH NetworkIOc|r!t|sdS tt|}|jdkrt |j}|j}|dvrtd|d|tj dkr5t|dkr"|dd kr|d d kr |d d}t||d dS|jdvrBd|d|jd}trt|tj|t ddSt#|jpdD]k}t'j|dd}t+|r:d|d|}trt|tj|t dldS#tt,f$rt.$r trYdSwxYw)z-Hardened URL validation with SSRF protection.Nr>)r localhostr@z9.file_scheme]: Non-local file URI authority not allowed: ntrLrrk:z .file_schemer^)r;r<z]: Unsupported scheme 'z'.rMrr?z!]: SSRF attempt to restricted IP )r'rQrrRrrnetlocr r rlenrhrWrXrYrZr[rrr ip_addressrr*r\) url_inputr^ra file_pathrrgresultrs rvalidate_network_urlrjs[ C NN00222#i..)) =F " " ,,I]F...L7LLAGLL4 NNa''aLC''aLC''%abbM )-E-E-E F F F F F = 1 1 1XwXXv}XXX  A%c*** c>a@@@@ F'(=2>> E EF%fQil33B## E[W[[WY[[E)#...M#~!DDDD E E Z (       s B4F,A F,(BF,,#GGc"eZdZdZfdZxZS)_ValidatingRedirectHandlerzIEnsures that every step of a redirect chain is re-validated against SSRF.crt|dt||||||S)NNetworkRedirectr)rsuperredirect_request)selfreqfpcodergheadersnewurl __class__s rrz+_ValidatingRedirectHandler.redirect_requests9V->????ww''RsGVLLLrH)__name__ __module__ __qualname____doc__r __classcell__rs@rrrsGSSMMMMMMMMMrHrcx tj||tj}n#ttf$rgcYSwxYw|D]z} t j|dd}n#t$rY0wxYwt|r7d|}trt|tj |td{|S)a Resolve ``host`` once and SSRF-validate *every* address it resolves to. Returns the resolved ``getaddrinfo`` records so the caller can connect to a **pinned** numeric address. Because validation and the subsequent connection observe the same resolution (the connection is made to the numeric IP, which triggers no further name lookup), this closes the DNS-rebinding TOCTOU where a hostname resolves to a public IP during validation and to an internal / loopback IP during the actual connect. rr?rzDSecurity Violation [pathsec.urlopen]: SSRF attempt to restricted IP rrM) rrrr r*rrrrWrXrYrZr[)hostportaddrinforesrrgs r_resolve_and_validate_hostrs%dD8JKKK Z   = = %c!fQi00BB    H  B   =]Y[]]C +%c*** M#~! < < < < Os!$:: A## A0/A0ct||}|stjd|dd}|D]C}|dd} tj||f||cS#t$r }|}Yd}~>6 ; ; A A C C I I K KP rHctrdSd|d}trt|tj|t ddS)aFail closed on a proxied fetch NLTK cannot SSRF-validate (CWE-918). A configured proxy performs the egress, so NLTK can neither pin the validated IP nor observe the address the proxy ultimately reaches. Rather than reintroduce the DNS-rebinding / internal-routing SSRF that the pinned direct path closes, refuse the request unless the operator has asserted the proxy is trusted (``ALLOW_PROXIED_FETCH`` / ``NLTK_ALLOW_PROXIED_URLOPEN``). NzBSecurity Violation [pathsec.urlopen]: refusing a proxied fetch of a. A configured proxy performs the egress, so NLTK cannot pin the validated IP and SSRF protection cannot be enforced (CWE-918). If and only if the proxy is trusted to be SSRF-safe, opt in via NLTK_ALLOW_PROXIED_URLOPEN=1 or nltk.pathsec.ALLOW_PROXIED_FETCH=True.rLrM)rrWrXrYrZr[)url_strrgs r _reject_unpinnable_proxied_fetchrsl Q  Q Q Q#c""" M#~!444444rHct|dr|jnt|}t|dt g}d}d}t jj=t jjjD]%}t|t jj rVd}|j rt|j ni}|rd}| t j |xt|t jjr8| t j|jt|t jjr7| t j|j'|s"|s t jrd}|sw|s2| t j i| t%| t'nt)|t jj|} | j|g|Ri|S)z Secure wrapper for urllib.request.urlopen with redirect validation. Inherits NLTK proxy settings, but intentionally ignores other custom global handlers to strictly enforce the security sandbox. full_urlzpathsec.urlopenrFNT)r%rr'rrurllibrequest_openerhandlersrO ProxyHandlerproxiesdictappendProxyBasicAuthHandlerpasswdProxyDigestAuthHandler getproxiesrrr build_openeropen) urlargsrrrproxiedhas_proxy_handlerhandlerisolated_proxiesopeners rurlopenr4s9 &c:66DcllCHHG*;<<<<+,,-HG ~)~-6 W WG'6>#>?? W$(!d|vrtjtjz}n d|vrtjtjz}nd}d|vr tj}n |hdzr tj}n tj}||zS)aTranslate a :func:`builtins.open` *mode* string into ``os.open()`` flags. Mirrors the read/write/create/append/truncate intent of *mode* so the hardened opener can add ``O_NOFOLLOW`` on top of exactly what the caller asked for. POSIX only. xwar+>r$r#r") r"r O_CREATO_EXCLO_TRUNCO_APPENDO_RDWRO_WRONLYO_RDONLY)rcharscreataccesss r_os_open_flagsr0s IIE e|| RY&  RZ'  R[( e||   E>rHc ddl}t|ttddzttddz} tj||d}n<#t $r/}|j|j|jfvrtd|d|d}~wwxYw tj |} | j dkrtd |d | j d t|} t| | ntj ||| n##t$rtj|wxYw tj||fi|S#t$rtj|wxYw) aOpen ``raw_path`` for read *or* write, closing the symlink-swap TOCTOU and the hardlink escape that a path-only ``validate_path`` cannot. 1. Open ``raw_path`` with ``O_NOFOLLOW`` so its final component is never followed as a symlink. Unlike opening a re-resolved path, this is atomic: the kernel either opens the real inode at that path or fails, so a symlink swapped in after the caller's ``validate_path`` cannot redirect the open (TOCTOU). On the write side this stops an attacker who plants/swaps ``/pkg`` -> ``/etc/...`` between the check and a ``retrieve()`` / model write from redirecting the write outside the root -- the symmetric counterpart of the read-side content leak (CWE-59, GHSA-f794 class). Corpora/data files are never symlinks, so refusing a final-component symlink outright is fail-closed. 2. ``fstat`` the descriptor and refuse ``st_nlink > 1``: a hardlink is another name for the same inode with no symlink to resolve, so an in-root hardlink to an outside inode would otherwise be read or written through. 3. Re-validate the *opened descriptor's* real path (from the kernel, not by re-resolving the string) so a swapped intermediate directory symlink that redirected the open to an outside inode is still caught. A newly created file is given ``0600`` permissions so a write into a shared temp dir is not left group-/world-readable (CWE-377/378). POSIX only; callers fall back to :func:`builtins.open` elsewhere. rN O_NOFOLLOW O_CLOEXECizQSecurity Violation [pathsec.open]: refusing to follow a symlink at open time for z (TOCTOU guard, CWE-59)rzASecurity Violation [pathsec.open]: refusing multiply-linked file z (st_nlink=z9); a hardlink can point at an outside-root inode (CWE-59)r^r_)errnor0rr rr ELOOPEMLINKrXfstatst_nlinkr rhrrealpath BaseExceptionclosefdopen) raw_pathrr^r_rr5flagsrrractuals r_hardened_openrAs2LLL t "lA & & ' "k1 % % &  WXue , ,  7u{EL1 1 1!H$,HHH    Xb\\ ;??!...*,+... b!!(FFbg.>.>x.H.H'        yT,,V,,,    s1A B$*BBArr^r_rr>s rrrs$8dG=IIII}T777777  Dzz#u%%*fd##c5\++%yR9T?? RRRFGGTQR H~~c5\)) oUYZbUcUcUl o o o    H~~;x(((G=IIIIP27g%% hg}OOOOO = 7 7 7 7 77s B!!B<c:eZdZdZfdZdfd Zdfd ZxZS)rz#Secure wrapper for zipfile.ZipFile.ct|tttjfr tj|}n#t $rt ddwxYwt|ttfvr%t dt|jdt|turtj |}t|d|}n|}tj |g|Ri|dS)NrCrDrEzpathsec.ZipFiler) rOr'rHr PathLikerIrJrGrrrhr__init__)rr>r rr> file_to_openrs rrNzZipFile.__init__#s dS%5 6 6  9T??   B  H~~c5\11w]abj]k]k]twwwH~~&&;x00 (,= > > > >#LLL777777777s ?ANct||ptj|t|||S)N)r})rr rVrextract)rmemberrpwdrs rrQzZipFile.extract=s=T4#629;;OOOOwwvtS111rHct||ptjt|||dSrB)rr rVr extractall)rrr{rSrs rrUzZipFile.extractallAs@T4#629;;777 4#.....rH)NN)NNN)rrrrrNrQrUrrs@rrr sz--888884222222//////////rHr)rhrrrrrrW)r9N)Nrt)r)r)?rrF http.clientr;rr rr rr-urllib.requestrrYr functoolsrpathlibr urllib.parserrrWrr r!rr8rhrsrr ip_networkrrrrrrrHTTPRedirectHandlerrrrclientHTTPConnectionrHTTPSConnectionr HTTPHandlerr HTTPSHandlerrrrrrr r0rA_hardened_read_openrr__all__rJrHrrds211 ******** """J333lTTTTn3330AK';';';';T 3   )I(88'9'00"   ///86666rMMMMM!CMMM8!!!H     $+4    O O O O O4;6 O O O66666v~1666 AAAAA3AAA   555.:-:-:-z333 *2DDDR%-8N$-8-8-8-8-8`#/#/#/#/#/go#/#/#/L   rH