U /e3@sJUddlmZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlZddlZddlZddlZddlZddlZddlmZddlmZddlmZmZmZmZmZddlmZm Z ddl!m"Z"m#Z#ddlm$Z$dd l%m&Z&dd lm'Z'dd l(m)Z)dd l*m+Z+dd l,m-Z-ddl.m/Z/ddl0m1Z1ddl2m3Z3ddl2m4Z5ddl2m6Z6m7Z7m8Z8m9Z9ddl:Z:ddl;Z;ddlZ>Wne?k rdZ>YnXddl@ZAddlBmCZCddlDmEZEddlFZFddlFmGZGddlHmIZJddlHmKZKddlHmLZMddlNmOZOddlPmQZQddlRm.Z.zddlSmTZTWne?k rleUZTYnXeQr~ddlVZ nddlWZ eXeYZZZ[e3rddl\m]Z]e]dZ^e8dZ_d Z`d!aad"d#Zbd$d%Zcd&d'ZdeAjed(d)Zfdd,d-Zgdd/d0Zhd1d2Zidd4d5Zjdd6d7Z4Gd8d9d9ZkGd:d;d;Zlddd?d@ZnGdAdBdBZoe"dCdDZpe"dEdFZqdGdHZrdIdJdKdLdMZse9dNdNdOdPdQZte9d!dRdSdTdUdVdWdXdQZtdd!dRdSdYdQZtGdZdVdVZudd\d]ZveAjed^d_Zwe=jxyd`daZzddcddZ{dedfZ|dgdhZ}didjZ~ddkdlZdmdnZIdodpdqdrdsZddJdUdudvdwZdJdxdydzd{Zd|d}Zeeffd~dZejdfddZGdddejZddZedddZedddZddddddddddddddddddgZeAjeefddZddZe"ddJddddZddZdddddZeZGdddeZGdddeZddddddddZddZe dRddZeeejdJddddZddZGddÄdÃZeZdddJddŜddDŽZddddddddddќ ZddӄZe$dԃZded<d!ad3d!d֜dddTdd؜ddڄZdTdۜdd݄ZGdd߄d߃ZdS)) annotationsN) TimeoutError)deque)Callable Collection ContainerKeysView ValuesView)CancelledErrorThreadPoolExecutor)contextmanagersuppress) ContextVar) timedelta)wraps)md5)cache_from_source) PickleBuffer)sleep) ModuleType) TYPE_CHECKING)Any)ClassVarIteratorTypeVaroverload)gen)IOLoop)istask) ensure_bytes) key_split)parse_timedelta) get_template)WINDOWS)time) thread_state) ParamSpecPTZ__no_default__Fc Cstjd}t|}|dkrtsdg}ddlm}m}||D]6\}}zt |Wnt k rjYq@X| |q@| |da|S)aCreate a multiprocessing context The context type is controlled by the ``distributed.worker.multiprocessing-method`` configuration key. Returns ------- multiprocessing.BaseContext The multiprocessing context Notes ----- Repeated calls with the same method will return the same object (since multiprocessing.get_context returns singleton instances). z)distributed.worker.multiprocessing-methodZ forkserver distributedr)optional_packagesrequired_packagesT)daskconfiggetmultiprocessingZ get_context_forkserver_preload_setZdistributed.versionsr*r+ importlib import_module ImportErrorappendZset_forkserver_preload)methodctxZpreloadr*r+pkg_r95/tmp/pip-unpacked-wheel-g426oqom/distributed/utils.pyget_mp_contextWs     r;cCspz,t|}|t|jt|jBkr*WdSWntk rDYqlYnXz |j}Wqtk rhYqlYqXqdS)zE Whether the function takes an argument with the given name. TF)inspectgetfullargspecsetargs kwonlyargs TypeError __wrapped__AttributeError)funcargnameZargspecr9r9r:has_arg}s     rFcCs tdk rttjdSdSdS)z; Get the maximum number of open files per process. Nri)resourceZ getrlimitZ RLIMIT_NOFILEr9r9r9r:get_fileno_limitsrHc Cst|tj}zz$|||f|d}|WWnStk r}zJtd||ftt t ||tjtj d}|ddWYWSd}~XYnXW5|XdS)NrzQCouldn't detect a suitable IP address for reaching %r, defaulting to hostname: %s) socket SOCK_DGRAMcloseconnect getsocknameOSErrorwarningswarnRuntimeWarning getaddrinfo gethostname IPPROTO_UDP)hostportfamilysockipeZ addr_infor9r9r:_get_ips.  *r\8.8.8.8PcCst||tjdS)z Get the local IP address through which the *host* is reachable. *host* defaults to a well-known Internet host (one of Google's public DNS servers). rX)r\rJAF_INETrVrWr9r9r:get_ipsrb2001:4860:4860::8888cCst||tjdS)z- The same as get_ip(), but for IPv6. r_)r\rJAF_INET6rar9r9r:get_ipv6srecCsdt}||kr,t|}td||||D]}|jtjkr4|j Sq4td|ddS)z Get the local IPv4 address of a network interface. KeyError is raised if the interface doesn't exist. ValueError is raised if the interface does no have an IPv4 address associated with it. zG{!r} is not a valid network interface. Valid network interfaces are: {}z interface z doesn't have an IPv4 addressN) psutil net_if_addrslistkeys ValueErrorformatrXrJr`address)ZifnamergZallowed_ifnamesinfor9r9r:get_ip_interfaces    rnr9cs|tjttj|dd|D}sxzIdH}Wn0tk rjtjfdd}|YnX||j <q |S)a!Wait on many tasks at the same time Err once any of the tasks err. See https://github.com/tornadoweb/tornado/issues/1546 Parameters ---------- args: futures to wait for quiet_exceptions: tuple, Exception Exception types to avoid logging if they fail cSsg|]}dqSNr9.0r8r9r9r: szAll..Nc 3s4tjD]$}z |VWq k r,Yq Xq dSzWatch unfinished tasks Otherwise if they err they get logged in a way that is hard to control. They need some other task to watch them so that they are not orphaned NrhZ _unfinishedZtaskquiet_exceptionsZtasksr9r:quiets  zAll..quiet rZ WaitIteratormapasyncio ensure_futuredonenext Exception coroutineZ current_indexr?rwresultsresultrxr9rvr:Alls   rcs~tjttj|dd|D}szzIdH}Wn0tk rjtjfdd}|YnX||j <qzq |S)aWait on many tasks at the same time and return when any is finished Err once any of the tasks err. Parameters ---------- args: futures to wait for quiet_exceptions: tuple, Exception Exception types to avoid logging if they fail cSsg|]}dqSror9rpr9r9r:rrszAny..Nc 3s4tjD]$}z |VWq k r,Yq Xq dSrsrtrurvr9r:rxs  zAny..quietryrr9rvr:rs   rc@seZdZdZddZdS) NoOpAwaitablezAn awaitable object that always returns None. Useful to return from a method that can be called in both asynchronous and synchronous contextscCsdd}|S)NcsdSror9r9r9r9r:f2sz"NoOpAwaitable.__await__..f) __await__)selfrr9r9r:r1szNoOpAwaitable.__await__N)__name__ __module__ __qualname____doc__rr9r9r9r:r+src@s,eZdZdZeddZdddddZdS)SyncMethodMixina$ A mixin for adding an `asynchronous` attribute and `sync` method to a class. Subclasses must define a `loop` attribute for an associated `tornado.IOLoop`, and may also add a `_asynchronous` attribute indicating whether the class should default to asynchronous behavior. cCst|jt|dddS)z!Are we running in the event loop? _asynchronousF)default) in_async_callloopgetattrrr9r9r: asynchronousAszSyncMethodMixin.asynchronousN)rcallback_timeoutcOs\t|}|dkr|j}|r<|||}|dk r8t||}|St|j|f|d|i|SdS)z`Call `func` with `args` synchronously or asynchronously depending on the calling contextNr)_parse_timedeltarr{wait_forsyncr)rrDrrr?kwargsfuturer9r9r:rFs$  zSyncMethodMixin.sync)rrrrpropertyrrr9r9r9r:r8s rcCs>z|jtkWStk r8|js2|YSYdSXdS)z3Whether this call is currently within an async callFN) asyncio_loopr{get_running_loop RuntimeError is_running)rrr9r9r:rWs  r)rc stdjrtdtt d tj f dd}fddfdd } |dk r|st d d n s|d qrΈ\}}} | | n SdS) z; Run coroutine in loop running in separate thread. szIOLoop is closedNc3szpzNtkrtdtjVdk r>ttVWnt k rlt YnXW5XdS)Nz)sync() called from thread of running loop) r> threading get_identrrZmomentr{rr|rsysexc_infor9) r?rr[errorrDrrmain_tidrr9r:ros     zsync..fcsdk rdSro)cancelr9)rr9r:rszsync..cancelcs2z |WStk r,YnXdSro)waitKeyboardInterrupt add_callbacktimeout)rr[rr9r:rs   zsync..waitztimed out after z s. ) rrZ is_closedrrEventrrrrris_setwith_traceback) rrDrr?rrrtypexctbr9) r?rrr[rrDrrrrrr:rcs(        rc@szeZdZUdZeZded<e Z dddZ dd Z d d Z dd dZddZddZddZddZeddZdS) LoopRunnerag A helper to start and stop an IO loop in a controlled way. Several loop runners can associate safely to the same IO loop. Parameters ---------- loop: IOLoop (optional) If given, this loop will be re-used, otherwise an appropriate one will be looked up or created. asynchronous: boolean (optional, default False) If false (the default), the loop is meant to run in a separate thread and will be started if necessary. If true, the loop is meant to run in the thread this object is instantiated from, and will not be started automatically. zJClassVar[weakref.WeakKeyDictionary[IOLoop, tuple[int, LoopRunner | None]]] _all_loopsNFc Cs|dkrT|rJz tWn$tk r<tjdtddYnXt|_qtt|_n |j sntjdtdd||_||_ d|_ d|_ |j|j|jdW5QRXdS)NzQConstructing a LoopRunner(asynchronous=True) without a running loop is deprecated stacklevelzGConstructing LoopRunner(loop=loop) without a running loop is deprecatedF)rN)r{rrrPrQDeprecationWarningrcurrent_looprrr _loop_thread_started_lockr setdefault)rrrr9r9r:__init__s0     zLoopRunner.__init__c Cs|j|W5QRXdS)z Start the IO loop if required. The loop is run in a dedicated thread. If the loop is already running, this method does nothing. N)r_start_unlockedrr9r9r:startszLoopRunner.startcs|jr t|j|j\}}|js0|dk s0|dkrN|d|f|j|j<d|_dS|jdks\t|dkshtttdgdgfdd|jffdd }tj|dd }d|_ | j d d d|_d}||k rP d ddk r:t dt s:t dts2td dd|ddf|j|j<n0ddksft||_|d|f|j|j<dS)NrTcstd<dSNr)rcurrent_threadr>r9) in_threadloop_evtr9r:loop_cbs z+LoopRunner._start_unlocked..loop_cbc s`|zFz|js |Wn*tk rL}z |d<W5d}~XYnXW5XdSr)rr>rrrr)rr[)done_evtr start_excr9r:run_loops   z,LoopRunner._start_unlocked..run_loopzIO loop)targetnamerrznot an exception: )rAssertionErrorrrrrrrThreaddaemonrr isinstancerrrA)rcount real_runnerrthreadZ actual_threadr9)rrrrrr:rsF       zLoopRunner._start_unlockedrc Cs |j||W5QRXdS)zv Stop and close the loop if it was created by us. Otherwise, just mark this object "stopped". N)r_stop_unlockedrrr9r9r:stopszLoopRunner.stopcCsj|js dSd|_|j|j\}}|dkr>|d|f|j|j<n(|dksJt|j|j=|dk rf||dS)NFr)rrrr _real_stop)rrrrr9r9r:rs  zLoopRunner._stop_unlockedc Csf|jdk st|jdk rbz@|j|jj|jj|dtt|jW5QRXW5d|_XdS)Nr) rrrrrjoinr KeyErrorrLrr9r9r:r)s  zLoopRunner._real_stopcCs|jS)zP Return True between start() and stop() calls, False otherwise. )rrr9r9r: is_started4szLoopRunner.is_startedcOsJ|jrt|j|f||S|zt|j|f||WS|XdS)z Convenience helper: start the loop if needed, run sync(func, *args, **kwargs), then stop the loop again. N)rrrrr)rrDr?rr9r9r:run_sync:s zLoopRunner.run_synccCs&|j}|js tjdtdd|jS)NzGAccessing the loop property while the loop is not running is deprecatedrr)rrrrPrQr)rrr9r9r:rHs zLoopRunner.loop)NF)r)rrrrweakrefWeakKeyDictionaryr__annotations__rLockrrrrrrrrrrrr9r9r9r:rs    9  rcksi}|D],}ztt|||<Wqtk r2YqXq|D]\}}tt||q>z dVW5|D]<}z ||}Wntk rtt|YqdXtt||qdXdSro)rr%rCitemssetattrrdelattr)roldkvr9r9r:set_thread_stateTs   rc csZtjt|}t|d}||W5QRXz |VW5tj|rTt|XdS)Nw) ospathrtempfile gettempdiropenwriteexistsremove)filenametextfnrr9r9r:tmp_textjs   rcCs,dtjkrdSddlm}t|dddk S)zWDetermine if we're running within an IPython kernel >>> is_kernel() False IPythonFr get_ipythonkernelN)rmodulesrrrrr9r9r: is_kernelws  robjectstr)xreturncCst|tr|dSt|tr|ddkr>|ddddSt|dkrZtd|rZdS|dd kr|d dd d St|Snt|t rt | Sd SdS)avA more fine-grained version of key_split. >>> key_split_group(('x-2', 1)) 'x-2' >>> key_split_group("('x-2', 1)") 'x-2' >>> key_split_group('ae05086432ca935f6eba409a8ecd4896') 'data' >>> key_split_group('>> key_split_group('x') 'x' >>> key_split_group('x-1') 'x' r(,rz()"' z [a-f0-9]{32}data.ZOtherN) rtuplersplitstriplenrematchr byteskey_split_groupdecode)rr9r9r:rs       rCallable[P, T]rDrcCsdSror9)rDr9r9r: log_errorssrrpdb unroll_stackboolint _LogErrors)rrrcCsdSror9rr9r9r:rscCst||d}|r||S|S)aLog any errors and then reraise them. This can be used: - As a context manager:: with log_errors(...): ... - As a bare function decorator:: @log_errors def func(...): ... - As a function decorator with parameters:: @log_errors(...) def func(...): ... Parameters ---------- pdb: bool, optional Set to True to break into the debugger in case of exception unroll_stack: int, optional Number of levels of stack to unroll when determining the module's name for the purpose of logging. Normally you should omit this. Set to 2 if you are writing a helper function, context manager, or decorator. r)r)rDrrler9r9r:rs c@sReZdZUdZded<ded<dddddZddd d d Zd d ZddZdS)rrrrrrcCs||_||_dSror)rrrr9r9r:rsz_LogErrors.__init__rrcsBjd7_tr(fdd}nfdd}t|S)Nrc s,||IdHW5QRSQRXdSror9r?rrDrr9r:wrappersz$_LogErrors.__call__..wrapperc s&||W5QRSQRXdSror9rrr9r:rs)rr<iscoroutinefunctionr)rrDrr9rr:__call__s  z_LogErrors.__call__cCs|Sror9rr9r9r: __enter__sz_LogErrors.__enter__c Csddlm}|r t||tjfr$dSt}||j}t|d}|j }zt |} | |Wnt k rvYnX|jrddl} | dS)Nr)CommClosedError)Zdistributed.commr  issubclassrZReturnr<stackr getmodulerlogging getLogger exceptionrr set_trace) rexc_type exc_value tracebackr r"framemodmodnameloggerrr9r9r:__exit__s   z_LogErrors.__exit__N) rrr __slots__rrrrr/r9r9r9r:rs r)cCsRt|trtt|}d}t|}|jD] }t|tjr,|j}| |q,|S)zK Change all StreamHandlers for the given logger to the given level N) rrrr$upperr%handlers StreamHandlerlevelsetLevel)r4rootrr.handlerr9r9r:silence_logging s     r8c Csz|sd}tjtjg}|D]X}zt|d|tj}Wn(tjk rZ}z|}W5d}~XYqX|dddSq|dS)a%Ensure that address is an IP address Examples -------- >>> ensure_ip('localhost') '127.0.0.1' >>> ensure_ip('') # Maps as localhost for binding e.g. 'tcp://:8811' '127.0.0.1' >>> ensure_ip('123.123.123.123') # pass through IP addresses '123.123.123.123' localhostiNrrI)rJr`rdrS SOCK_STREAMgaierror)hostnameZfamiliesZfamrr[rr9r9r: ensure_ips  r=csht\}}tjddtjddtjddtjddg}rdtfdd |DrdjqBS) Nr)ZworkerZ schedulertornadozgen.pyZ concurrentZfuturesc3s|]}|jjjkVqdSro)tb_framef_code co_filename)rqb exc_tracebackr9r: Dsz get_traceback..)rrrrranytb_next)r(r)badr9rCr: get_traceback<s    rI'c Cshtt||kr`zt|dt|d|WStk r\tdt|t|d|YSXn|SdS)z/Truncate exception to be about a certain lengthzLong error messageN)r rtyper)r[nr9r9r:truncate_exceptionKs $rMcCs2t|}|tk r.|tk r.td|d|ddS)z'Validate a key as received on a stream.zUnexpected key type z (value: )N)rKrr rA)rrr9r9r: validate_keyVsrOcCs@t|p>t|tkr"ttt|p>t|tko>ttt|S)zPossibly contains a nested task)rrKrhrFrz_maybe_complexdictvaluesrur9r9r:rP]s   rPcCs|dkrdSd}||}|s&dS||}z2||}||t||t|WdStk rtYnX|t| d}qdS)aSeek current file to next byte after a delimiter bytestring This seeks the file to the next byte following the delimiter. It does not return anything. Use ``file.tell()`` to see location afterwards. Parameters ---------- file: a file delimiter: bytes a delimiter like ``b' '`` or message sentinel blocksize: int Number of bytes to read from the file at once. rN)tellreadindexseekr rj)file delimiter blocksizelastrfullir9r9r:seek_delimiterhs   "r^cCst|r\||t||d|}|||8}|||t||d|}|}||}||||}|S)a$Read a block of bytes from a file Parameters ---------- f: file File-like object supporting seek, read, tell, etc.. offset: int Byte offset to start read length: int Number of bytes to read delimiter: bytes (optional) Ensure reading starts and stops at delimiter bytestring If using the ``delimiter=`` keyword argument we ensure that the read starts and stops at delimiter boundaries that follow the locations ``offset`` and ``offset + length``. If ``offset`` is zero then we start at zero. The bytestring returned WILL include the terminating delimiter string. Examples -------- >>> from io import BytesIO # doctest: +SKIP >>> f = BytesIO(b'Alice, 100\nBob, 200\nCharlie, 300') # doctest: +SKIP >>> read_block(f, 0, 13) # doctest: +SKIP b'Alice, 100\nBo' >>> read_block(f, 0, 13, delimiter=b'\n') # doctest: +SKIP b'Alice, 100\nBob, 200\n' >>> read_block(f, 10, 10, delimiter=b'\n') # doctest: +SKIP b'Bob, 200\nCharlie, 300' i)rWr^rTrU)roffsetlengthrYrendr r9r9r: read_blocks"      rbcCstjdtddt|S)aAttempt to turn `s` into bytes. Parameters ---------- s : Any The object to be converted. Will correctly handled * str * bytes * objects implementing the buffer protocol (memoryview, ndarray, etc.) Returns ------- b : bytes Raises ------ TypeError When `s` cannot be converted Examples -------- >>> ensure_bytes('123') b'123' >>> ensure_bytes(b'123') b'123' z`distributed.utils.ensure_bytes` is deprecated. Please switch to `dask.utils.ensure_bytes`. This will be removed in `2022.6.0`.rr)rPrQr _ensure_bytes)rr9r9r:rs rz-bytes | bytearray | memoryview | PickleBuffer memoryview)objrcCs\t|tst|}|js"ttS|js4tt|S|jdksH|jdkrTt|S|SdS)z5Ensure `obj` is a 1-D contiguous `uint8` `memoryview`rBN) rrdnbytes bytearray contiguousndimrkrraw)rer9r9r:ensure_memoryviews    rl)rVrc CsPttjtj6}||df|d|d}|W5QRSQRXdS)zReturn a probably-open port There is a chance that this port will be taken by the operating system soon after returning from this function. rrN)rJr`r:bindlistenrN)rVrrWr9r9r: open_ports   rpzlist[ModuleType])rrc Cs:tj|\}}tj|\}}g}d}|dkrH|tjkr>|}|||dkrvt|}ttt |W5QRX|dkr|tjkrtj d|ddt |gD}| |g} |std|njt|dk rtj d|z2|D](}td ||| tt|qW5|dk r4tj |X| S) z*Loads modules for a file (.py, .zip, .egg)N).pyrq)z.eggz.zipz.pyzrcss|] }|jVqdSro)r)rqZmod_infor9r9r:rEszimport_file..zFound nothing to import from %szReload module %s from %s file)rrrsplitextrr4rr rOrinsertpkgutil iter_modulesextendr.warningr1invalidate_cachesrmreloadr2) r directoryrrextZnames_to_importZtmp_python_pathZ cache_filenamesZloadedr9r9r: import_files<      r}csdd|D}tdd|D}tddtt||D}ddt||t|}dd d d|D}d fd d|D}d |||||gS) zFormats an ascii table for given columns and rows. Parameters ---------- columns : list The column names rows : list of tuples The rows in the table. Each tuple must be the same length as ``columns``. cSsg|]}tdd|DqS)css|]}t|VqdSrorrqr]r9r9r:rE7sz(asciitable...)rrqrr9r9r:rr7szasciitable..css|]}t|VqdSror~rr9r9r:rE8szasciitable..css*|]"\}}tttt|t|VqdSro)maxrzr )rqrcr9r9r:rE9s|z %%-%ds |z+%s++css|]}d|dVqdS)-rNr9)rqrr9r9r:rE<s c3s|]}|VqdSror9rZ row_templater9r:rE=s)rzipr r)columnsZrowsZwidthsheaderbarrr9rr: asciitable,s  rcCs<t||rt|Sz|jWStk r6t|YSXdS)z(Number of bytes of a frame or memoryviewN)rr rgrC)r+Z _bytes_liker9r9r:rgAs  rgc Cst|}|jr|jrptj|rfz*t|}||}W5QRX|rJ|WSWntt fk rdYnXt dq t d|ddS)z@Reads a JSON file from disk that may be being written as we readg?zCould not load file after zs.N) Deadlineafterexpires remainingrrrrrjrrr)rloadrdeadlinercfgr9r9r:json_load_robustLs      rcsTeZdZUdZeZded<ddfdd Zdd Z d d Z e d d Z Z S) DequeHandlerz3A logging.Handler that records records into a dequez'ClassVar[weakref.WeakSet[DequeHandler]] _instancesrJ)rLcs*t|d|_tj|||j|dS)N)maxlen)rsuperrradd)rrLr?r __class__r9r:rbs zDequeHandler.__init__cCs|j|dSro)rr4)rrecordr9r9r:emitgszDequeHandler.emitcCs|jdS)z) Clear internal storage. N)rclearrr9r9r:rjszDequeHandler.clearcCst|jD] }|q dS)zG Clear the internal storage of all live DequeHandlers. N)rhrr)clsinstr9r9r:clear_all_instancespsz DequeHandler.clear_all_instances)rrrrrWeakSetrrrrr classmethodr __classcell__r9r9rr:r]s rcCs2tjjjD]}t|jD] }|qqdS)znPython 2's logger's locks don't survive a fork event https://github.com/dask/distributed/issues/1491 N)r$Loggermanager loggerDictrir%r2 createLock)rr7r9r9r:reset_logger_locksysricCs|t|jkSro)r< signature parameters)rDkeywordr9r9r: has_keywordsrcCs|dk rt|trLzddlm}||}Wn tk rJtd|YnXt|jtjjrb|j}t|tjjrdd|j D}||kSdS)Nr)r2z&Module for command %s is not availablecSs h|]}t|tjjr|jqSr9)rclickcoreOptionZhuman_readable_namerqpr9r9r: sz&command_has_keyword..F) rrr1r2r3mainrrCommandparams)cmdrr2Z cmd_paramsr9r9r:command_has_keywords   rz#440154z#471669z#472A79z#433C84z#3C4D8Az#355D8Cz#2E6C8Ez#287A8Ez#23898Dz#1E978Az#20A585z#2EB27Cz#45BF6Fz#64CB5Dz#88D547z#AFDC2Ez#D7E219z#FDE724cCs6tt|}t|ddd}||t|S)N)rrencoder hexdigestr )rpalettehrLr9r9r:color_ofsrcCst|pt|Sro)r<rris_coroutine_function)rr9r9r:rsrzstr | float | timedeltazIterator[None])durationmsgrccs@t}dVt}||}|t|kr>> parse_ports(8787) [8787] or a string: >>> parse_ports("8787") [8787] A sequential range of ports can be specified by a string which indicates the first and last ports which should be included in the sequence of ports: >>> parse_ports("8787:8790") [8787, 8788, 8789, 8790] An input of ``None`` is also valid and can be used to indicate that no port has been specified: >>> parse_ports(None) [None] :zzWhen specifying a range of ports like port_start:port_stop, port_stop must be greater than port_start, but got port_start=z and port_stop=rNcss|]}t|tVqdSro)rrrr9r9r:rEszparse_ports..) rrrzrrrjrhrangerallrA)rWZ port_startZ port_stopr9r9r: parse_portss (   rc@seZdZdZddZdS)Logz7A container for newline-delimited string of log entriescCstdj|dS)Nz log.html.j2)logr"renderrr9r9r: _repr_html_$szLog._repr_html_Nrrrrrr9r9r9r:r!src@seZdZdZddZdS)Logsz>A container for a dict mapping names to strings of log entriescCstdj|dS)Nz logs.html.j2)Zlogsrrr9r9r:r+szLogs._repr_html_Nrr9r9r9r:r(srzdict[str, AnyType]zCallable | Nonezstr | ModuleType | Nonez list[str])drrrcsddlm}|s|r|D]r}t||st||s|rT|rTtd|||||fq|rrtd||d|qtd||d|qddtfd d |DgS) aConvert a kwargs dictionary into a list of CLI keywords Parameters ---------- d : dict The keywords to convert cls : callable The callable that consumes these terms to check them for validity cmd : string or object A string with the name of a module, or the module containing a click-generated command with a "main" function, or the function itself. It may be used to parse a module's custom arguments (that is, arguments that are not part of Worker class), such as nworkers from dask worker CLI or enable_nvlink from dask-cuda-worker CLI. Examples -------- >>> cli_keywords({"x": 123, "save_file": "foo.txt"}) ['--x', '123', '--save-file', 'foo.txt'] >>> from dask.distributed import Worker >>> cli_keywords({"x": 123}, Worker) Traceback (most recent call last): ... ValueError: Class distributed.worker.Worker does not support keyword x r)typenamez0Neither class %s or module %s support keyword %szClass z does not support keyword zModule cSs0t|}d|kr,d|kr,d|kr,d|d}|S)N '"r~)routr9r9r: convert_valueas z#cli_keywords..convert_valuec3s*|]"\}}d|dd|gVqdS)z--r8rN)replacerqrrrr9r:rEhszcli_keywords..) dask.utilsrrrrjsumr)rrrrrr9rr: cli_keywords/s, rcCstjj|dk Sro)xmletree ElementTree fromstring)rr9r9r: is_valid_xmllsrz Dask-Offload) max_workersZthread_name_prefixAnyType)rrcCsHz|dd\}}Wntk r2t|YSXt|}t||S)zReturn the fully qualified term Examples -------- >>> import_term("math.sin") # doctest: +SKIP rr)rsplitrjr1r2r)r module_name attr_namemoduler9r9r: import_termts  rcs0t}t|tfddIdHS)NcsjfSro)runr9r?contextrrr9r:rSzoffload..)r{r contextvarsZ copy_contextZrun_in_executor_offload_executor)rr?rrr9rr:offloads rc@s,eZdZddZddZddZddZd S) EmptyContextcCsdSror9rr9r9r:rszEmptyContext.__enter__cCsdSror9rr(r)r*r9r9r:r/szEmptyContext.__exit__csdSror9rr9r9r: __aenter__szEmptyContext.__aenter__csdSror9rr9r9r: __aexit__szEmptyContext.__aexit__N)rrrrr/rrr9r9r9r:rsrz list[dict])addrsdefault_listen_iprc Cs|dkr d}t|tr |d}t|ts0|g}g}|D]}z t|}Wnttfk r`YnXt|trv|d}t|ttfrt|dkr|dt|d}}qt|dkr|d\}}qt|nt|tr|}|}| ||dq8|S) a Examples -------- >>> clean_dashboard_address(8787) [{'address': '', 'port': 8787}] >>> clean_dashboard_address(":8787") [{'address': '', 'port': 8787}] >>> clean_dashboard_address("8787") [{'address': '', 'port': 8787}] >>> clean_dashboard_address("8787") [{'address': '', 'port': 8787}] >>> clean_dashboard_address("foo:8787") [{'address': 'foo', 'port': 8787}] >>> clean_dashboard_address([8787, 8887]) [{'address': '', 'port': 8787}, {'address': '', 'port': 8887}] >>> clean_dashboard_address(":8787,:8887") [{'address': '', 'port': 8787}, {'address': '', 'port': 8887}] z0.0.0.0rmrrrrr)rlrW) rrrrhrrArjrr r4)rr addressesaddrrVrWr9r9r:clean_dashboard_addresss2          rzdask.config.deserializezdask.config.serializezdask.utils.format_byteszdask.utils.format_timezdask.utils.funcnamezdask.utils.parse_byteszdask.utils.parse_timedeltazdask.utils.typenamezdask.utils.tmpfile) Zdeserialize_for_cliZserialize_for_cliZ format_bytesZ format_timefuncnameZ parse_bytesr!rZtmpfilecCsL|tkr4t|}tj|d|dtddt|Stdtd|dS)NzC is deprecated and will be removed in a future release. Please use z instead.r)categoryrzmodule z has no attribute ) _deprecationsrPrQ FutureWarningrrCr)rZ use_insteadr9r9r: __getattr__sr_recursive_to_dict_seenzContextVar[set[int]])excludememberszContainer[str])rerrrc st|ttttfs|dkr|St|ttfr4t|S|rPfddt |D}z t }Wnt k rvt }YnX|}t |}z>t||krt|W&St|drtrt|W S|t|daz|jdWWSdaX|t|t|dr|jdWSt|ttt ttttfrLfd d |DWSt|tri}|D]R\}}t|d}t|d}z |||<Wn"t k r||t|<YnXqd|WSt|WS|j|XdS) aRecursively convert arbitrary Python objects to a JSON-serializable representation. This is intended for debugging purposes only. The following objects are supported: list, tuple, set, frozenset, deque, dict, dict_keys, dict_values Descended into these objects recursively. Python-specific collections are converted to JSON-friendly variants. Classes that define ``_to_dict(self, *, exclude: Container[str] = ())``: Call the method and dump its output Classes that define ``_to_dict_no_nest(self, *, exclude: Container[str] = ())``: Like above, but prevents nested calls (see below) Other Python objects Dump the output of ``repr()`` Objects already encountered before, regardless of type Dump the output of ``repr()``. This breaks circular references and shortens the output. Parameters ---------- exclude: A list of attribute names to be excluded from the dump. This will be forwarded to the objects ``_to_dict`` methods and these methods are required to accept this parameter. members: If True, convert the top-level Python object to a dict of its public members **``_to_dict_no_nest`` vs. ``_to_dict``** The presence of the ``_to_dict_no_nest`` method signals ``recursive_to_dict`` to have a mutually exclusive full dict representation with other objects that also have the ``_to_dict_no_nest``, regardless of their class. Only the outermost object in a nested structure has the method invoked; all others are dumped as their string repr instead, even if they were not encountered before. Example: .. code-block:: python >>> class Person: ... def __init__(self, name): ... self.name = name ... self.children = [] ... self.pets = [] ... ... def _to_dict_no_nest(self, exclude=()): ... return recursive_to_dict(self.__dict__, exclude=exclude) ... ... def __repr__(self): ... return self.name >>> class Pet: ... def __init__(self, name): ... self.name = name ... self.owners = [] ... ... def _to_dict_no_nest(self, exclude=()): ... return recursive_to_dict(self.__dict__, exclude=exclude) ... ... def __repr__(self): ... return self.name >>> alice = Person("Alice") >>> bob = Person("Bob") >>> charlie = Pet("Charlie") >>> alice.children.append(bob) >>> alice.pets.append(charlie) >>> bob.pets.append(charlie) >>> charlie.owners[:] = [alice, bob] >>> recursive_to_dict({"people": [alice, bob], "pets": [charlie]}) { "people": [ {"name": "Alice", "children": ["Bob"], "pets": ["Charlie"]}, {"name": "Bob", "children": [], "pets": ["Charlie"]}, ], "pets": [ {"name": "Charlie", "owners": ["Alice", "Bob"]}, ], } If we changed the methods to ``_to_dict``, the output would instead be: .. code-block:: python { "people": [ { "name": "Alice", "children": [ { "name": "Bob", "children": [], "pets": [{"name": "Charlie", "owners": ["Alice", "Bob"]}], }, ], pets: ["Charlie"], ], "Bob", ], "pets": ["Charlie"], } Also notice that, if in the future someone will swap the creation of the ``children`` and ``pets`` attributes inside ``Person.__init__``, the output with ``_to_dict`` will change completely whereas the one with ``_to_dict_no_nest`` won't! Ncs0i|](\}}|ds|krt|s||qS)r8) startswithcallablerrr9r: js  z%recursive_to_dict.._to_dict_no_nestTFr _to_dictcsg|]}t|dqS)r )recursive_to_dict)rqelr r9r:rrsz%recursive_to_dict..)!rrfloatrrrKr reprr< getmembersrr. LookupErrorr>copyvarresetidhasattr_to_dict_no_nest_flagrr rrhr frozensetrrr rQrrrA)rerrseentokresrrr9r r:rsTm            rrcCsddlm}|S)zIs the interpreter shutting down now? This is a variant of ``sys.is_finalizing`` which can return True inside the ``__del__`` method of classes defined inside the distributed package. r_python_shutting_down)r)r!r r9r9r:is_python_shutting_downs r"c@seZdZUdZded<ded<ddddd Zedddd d d Zedd ddZ edd ddZ edd ddZ edd ddZ edd ddZ dS)rzrZtornado.iolooprr,rrrrcr r!rZ dask.widgetsr"Zdistributed.compatibilityr#Zdistributed.metricsZ dask.contextr%localZ!multiprocessing.popen_spawn_win32Z!multiprocessing.popen_spawn_posixr%rr.Z_loggerZtyping_extensionsr&r'r(Z no_defaultr0r;rFrHmemoizer\rbrernrrrrrrrrrrrrr8r=Zpickling_supportinstallrIrMrOrPr^rbrlrpr}rr rhrgrrHandlerrr lru_cacherrrrrrrrrrrrQrrrrfinalizeshutdownrrrZ empty_contextrrrrrrrr"rr9r9r9r:sb                          &    ( (  7;  !#5     ! 4&&     @ =  6 '