U /eV@sjddlmZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl m Z ddlmZmZmZddlmZmZerddlmZmZeeZGdddZGd d d ZGd d d Zd ddddZGdddeejZ Gddde Z!Gddde Z"GdddeZ#GdddeZ$GdddeZ%GdddZ&Gd d!d!eZ'dS)") annotationsN) Awaitable) TYPE_CHECKINGAnyClassVar)funcnametmpfile) SchedulerTaskStateStatec@seZdZdZdddddZdddd Zddd d Zdd d dddddZdddddZdddddddddZ ddddddZ ddddddZ ddddd d!Z ddddd"d#Z dddd$d%d&Zd'S)(SchedulerPluginaInterface to extend the Scheduler The scheduler operates by triggering and responding to events like ``task_finished``, ``update_graph``, ``task_erred``, etc.. A plugin enables custom code to run at each of those same events. The scheduler will run the analogous methods on this class when each event is triggered. This runs user code within the scheduler thread that can perform arbitrary operations in synchrony with the scheduler itself. Plugins are often used for diagnostics and measurement, but have full access to the scheduler and could in principle affect core scheduling. To implement a plugin: 1. subclass this class 2. override some of its methods 3. add the plugin to the scheduler with ``Scheduler.add_plugin(myplugin)``. Examples -------- >>> class Counter(SchedulerPlugin): ... def __init__(self): ... self.counter = 0 ... ... def transition(self, key, start, finish, *args, **kwargs): ... if start == 'processing' and finish == 'memory': ... self.counter += 1 ... ... def restart(self, scheduler): ... self.counter = 0 >>> plugin = Counter() >>> scheduler.add_plugin(plugin) # doctest: +SKIP r None) schedulerreturncsdS)zhRun when the scheduler starts up This runs at the end of the Scheduler startup process Nselfr rrB/tmp/pip-unpacked-wheel-g426oqom/distributed/diagnostics/plugin.pystart=szSchedulerPlugin.startrcsdS)z*Runs prior to any Scheduler shutdown logicNrrrrr before_closeCszSchedulerPlugin.before_closecsdS)zRun when the scheduler closes down This runs at the beginning of the Scheduler shutdown process, but after workers have been asked to shut down gracefully NrrrrrcloseFszSchedulerPlugin.closezset[str]zdict[str, float]r)r keys restrictionskwargsrcKsdS)z0Run when a new graph / tasks enter the schedulerNr)rr rrrrrr update_graphMszSchedulerPlugin.update_graphcCsdS)z&Run when the scheduler restarts itselfNrrrrrrestartVszSchedulerPlugin.restartstrr )keyrfinishargsrrcOsdS)aRun whenever a task changes state Parameters ---------- key : string start : string Start state of the transition. One of released, waiting, processing, memory, error. finish : string Final state of the transition. *args, **kwargs : More options passed when transitioning This may include worker ID, compute time, etc. Nr)rrrrr rrrr transitionYszSchedulerPlugin.transitionzNone | Awaitable[None])r workerrcCsdS)z(Run when a new worker enters the clusterNrrr r"rrr add_workerpszSchedulerPlugin.add_workercCsdS)z$Run when a worker leaves the clusterNrr#rrr remove_workersszSchedulerPlugin.remove_worker)r clientrcCsdS)zRun when a new client connectsNrrr r&rrr add_clientxszSchedulerPlugin.add_clientcCsdS)zRun when a client disconnectsNrr'rrr remove_client{szSchedulerPlugin.remove_client)topicmsgrcCsdS)zRun when an event is loggedNr)rr*r+rrr log_event~szSchedulerPlugin.log_eventN)__name__ __module__ __qualname____doc__rrrrrr!r$r%r(r)r,rrrrr s$ r c@s(eZdZdZddZddZddZdS) WorkerPluginaInterface to extend the Worker A worker plugin enables custom code to run at different stages of the Workers' lifecycle: at setup, during task state transitions, when a task or dependency is released, and at teardown. A plugin enables custom code to run at each of step of a Workers's life. Whenever such an event happens, the corresponding method on this class will be called. Note that the user code always runs within the Worker's main thread. To implement a plugin implement some of the methods of this class and register the plugin to your client in order to have it attached to every existing and future workers with ``Client.register_worker_plugin``. Examples -------- >>> class ErrorLogger(WorkerPlugin): ... def __init__(self, logger): ... self.logger = logger ... ... def setup(self, worker): ... self.worker = worker ... ... def transition(self, key, start, finish, *args, **kwargs): ... if finish == 'error': ... ts = self.worker.tasks[key] ... exc_info = (type(ts.exception), ts.exception, ts.traceback) ... self.logger.error( ... "Error during computation of '%s'.", key, ... exc_info=exc_info ... ) >>> import logging >>> plugin = ErrorLogger(logging) >>> client.register_worker_plugin(plugin) # doctest: +SKIP cCsdS)z Run when the plugin is attached to a worker. This happens when the plugin is registered and attached to existing workers, or when a worker is created after the plugin has been registered. Nrrr"rrrsetupszWorkerPlugin.setupcCsdS)z@Run when the worker to which the plugin is attached to is closedNrr2rrrteardownszWorkerPlugin.teardowncKsdS)a Throughout the lifecycle of a task (see :doc:`Worker `), Workers are instructed by the scheduler to compute certain tasks, resulting in transitions in the state of each task. The Worker owning the task is then notified of this state transition. Whenever a task changes its state, this method will be called. Parameters ---------- key : string start : string Start state of the transition. One of waiting, ready, executing, long-running, memory, error. finish : string Final state of the transition. kwargs : More options passed when transitioning Nr)rrrrrrrrr!szWorkerPlugin.transitionN)r-r.r/r0r3r4r!rrrrr1s%r1c@s$eZdZdZdZddZddZdS) NannyPluginaInterface to extend the Nanny A worker plugin enables custom code to run at different stages of the Workers' lifecycle. A nanny plugin does the same thing, but benefits from being able to run code before the worker is started, or to restart the worker if necessary. To implement a plugin implement some of the methods of this class and register the plugin to your client in order to have it attached to every existing and future nanny by passing ``nanny=True`` to :meth:`Client.register_worker_plugin`. The ``restart`` attribute is used to control whether or not a running ``Worker`` needs to be restarted when registering the plugin. See Also -------- WorkerPlugin SchedulerPlugin FcCsdS)z Run when the plugin is attached to a nanny. This happens when the plugin is registered and attached to existing nannies, or when a nanny is created after the plugin has been registered. Nrrnannyrrrr3szNannyPlugin.setupcCsdS)z?Run when the nanny to which the plugin is attached to is closedNrr6rrrr4szNannyPlugin.teardownN)r-r.r/r0rr3r4rrrrr5sr5z,SchedulerPlugin | WorkerPlugin | NannyPluginr)pluginrcCs0t|dr|jStt|dttSdS)zUReturn plugin name. If plugin has no name attribute a random name is used. name-N)hasattrr9rtyperuuiduuid4)r8rrr_get_plugin_names r?c@seZdZUdZded<ded<ded<ded <ddd d d Zd dZejddddZ ddZ ddZ ddZ ddZ ddZddZdS) PackageInstalla1Abstract parent class for a worker plugin to install a set of packages This accepts a set of packages to install on all workers. You can also optionally ask for the worker to restart itself after performing this installation. .. note:: This will increase the time it takes to start up each worker. If possible, we recommend including the libraries in the worker environment or image. This is primarily intended for experimentation and debugging. Parameters ---------- packages A list of packages (with optional versions) to install restart Whether or not to restart the worker after installing the packages Only functions if the worker has an attached nanny process See Also -------- CondaInstall PipInstall z ClassVar[str] INSTALLERrr9 list[str]packagesboolr)rCrcCs&||_||_|jdt|_dS)Nz -install-)rCrrAr=r>r9)rrCrrrr__init__szPackageInstall.__init__c sddlm}|dtddIdH4IdH||IdHsjtd|j|j| |IdH| ntd|j|j r|j r| |IdHstd||IdH|jj|jd|jd d W5QIdHRXdS) Nr) SemaphoreT)Z max_leasesr9registerz(%s installing the following packages: %sz6The following packages have already been installed: %sz)Restarting worker to refresh interpreter.z-setup)rreason)Zdistributed.semaphorerFsocket gethostname _is_installedloggerinforArC_set_installedinstallrr7 _is_restarted_set_restartedZloopZ add_callbackZclose_gracefullyr9)rr"rFrrrr3!s.     zPackageInstall.setupr rcCsdS)zInstall the requested packagesNrrrrrrP<szPackageInstall.installcs|jj|ddIdHSNF)default)r& get_metadata_compose_installed_keyr2rrrrL@szPackageInstall._is_installedcs|j|dIdHdSNT)r& set_metadatarVr2rrrrOEszPackageInstall._set_installedcCs|jdtgS)N installed)r9rJrKrrrrrVKsz%PackageInstall._compose_installed_keycs|jj||ddIdHSrS)r&rU_compose_restarted_keyr2rrrrQRszPackageInstall._is_restartedcs|j||dIdHdSrW)r&rXrZr2rrrrRXszPackageInstall._set_restartedcCs|jd|jgS)NZ restarted)r9r7r2rrrrZ^sz%PackageInstall._compose_restarted_keyN)r-r.r/r0__annotations__rEr3abcabstractmethodrPrLrOrVrQrRrZrrrrr@s  r@csHeZdZUdZdZded<ddddd fd d Zd d ddZZS) CondaInstallamA Worker Plugin to conda install a set of packages This accepts a set of packages to install on all workers as well as options to use when installing. You can also optionally ask for the worker to restart itself after performing this installation. .. note:: This will increase the time it takes to start up each worker. If possible, we recommend including the libraries in the worker environment or image. This is primarily intended for experimentation and debugging. Parameters ---------- packages A list of packages (with optional versions) to install using conda conda_options Additional options to pass to conda restart Whether or not to restart the worker after installing the packages Only functions if the worker has an attached nanny process Examples -------- >>> from dask.distributed import CondaInstall >>> plugin = CondaInstall(packages=["scikit-learn"], conda_options=["--update-deps"]) >>> client.register_worker_plugin(plugin) See Also -------- PackageInstall PipInstall ZcondarB conda_optionsNFlist[str] | NonerD)rCr_rcstj||d|pg|_dSN)r)superrEr_)rrCr_r __class__rrrEszCondaInstall.__init__r rc Cszddlm}m}Wn:tk rN}zd}t|t||W5d}~XYnXz||j|j|j \}}}Wn:t k r}zd}t|t||W5d}~XYnX|dkrd| d}t|t|dS)Nr)Commands run_commandz`conda install failed because conda could not be found. Please make sure that conda is installed.zconda install failedzconda install failed with '') Zconda.cli.python_apirerfModuleNotFoundErrorrMerror RuntimeErrorINSTALLr_rC Exceptiondecodestrip)rrerfer+_stderr returncoderrrrPs(    zCondaInstall.install)NF r-r.r/r0rAr[rErP __classcell__rrrcrr^bs % r^csHeZdZUdZdZded<ddddd fd d Zd d ddZZS) PipInstalla]A Worker Plugin to pip install a set of packages This accepts a set of packages to install on all workers as well as options to use when installing. You can also optionally ask for the worker to restart itself after performing this installation. .. note:: This will increase the time it takes to start up each worker. If possible, we recommend including the libraries in the worker environment or image. This is primarily intended for experimentation and debugging. Parameters ---------- packages A list of packages (with optional versions) to install using pip pip_options Additional options to pass to pip restart Whether or not to restart the worker after installing the packages Only functions if the worker has an attached nanny process Examples -------- >>> from dask.distributed import PipInstall >>> plugin = PipInstall(packages=["scikit-learn"], pip_options=["--upgrade"]) >>> client.register_worker_plugin(plugin) See Also -------- PackageInstall CondaInstall piprB pip_optionsNFr`rD)rCrwrcstj||d|pg|_dSra)rbrErw)rrCrwrrcrrrEszPipInstall.__init__r rcCsptjtjdddg|j|jtjtjd}|\}}|}|dkrld| d}t |t |dS)Nz-mrvrP)stdoutrqrzpip install failed with 'rg) subprocessPopensys executablerwrCPIPE communicatewaitrmrnrMrirj)rprocrprqrrr+rrrrPs  zPipInstall.install)NFrsrrrcrrus % ruc@s$eZdZdZdZddZddZdS) UploadFileaQA WorkerPlugin to upload a local file to workers. Parameters ---------- filepath: str A path to the file (.py, egg, or zip) to upload Examples -------- >>> from distributed.diagnostics.plugin import UploadFile >>> client.register_worker_plugin(UploadFile("/path/to/file.py")) # doctest: +SKIP upload_filec Cs2tj||_t|d}||_W5QRXdS)S Initialize the plugin by reading in the data from the given file. rbN)ospathbasenamefilenameopenreaddata)rfilepathfrrrrEs zUploadFile.__init__cs4|j|j|jddIdH}t|j|dks0tdS)NT)rrloadnbytes)rrrlenAssertionError)rr"responserrrr3 s  zUploadFile.setupN)r-r.r/r0r9rEr3rrrrrsrc@s(eZdZdZd ddddZddZdS) EnvironTNz dict | None)environcCs |pi}dd|D|_dS)NcSsi|]\}}|t|qSr)r).0kvrrr sz$Environ.__init__..)itemsr)rrrrrrEszEnviron.__init__cs|j|jdSN)envupdaterr6rrrr3sz Environ.setup)N)r-r.r/rrEr3rrrrrsrc@s0eZdZdZdddddffddZdd Zd S) UploadDirectoryaUA NannyPlugin to upload a local file to workers. Parameters ---------- path: str A path to the directory to upload Examples -------- >>> from distributed.diagnostics.plugin import UploadDirectory >>> client.register_worker_plugin(UploadDirectory("/path/to/directory"), nanny=True) # doctest: +SKIP F)z.gitz.githubz .pytest_cachetestsZdocscCstj|ddkS)NrGz.pyc)rrsplitext)fnrrr/zUploadDirectory.c s.tj|}tj|d|_||_||_dtj|d|_tdd}t |dtj }t |D]\}} | D]|} tj || t fdd|Drqvtjt fdd|Drqvtjtj || tj |d } || qvqhW5QRXt|d } | |_W5QRXW5QRXd S) rzupload-directory-zip) extensionwc3s|]}|VqdSrr)r predicate)rrr @sz+UploadDirectory.__init__..c3s|]}|kVqdSrr)rword)dirsrrrCsz..rN)rr expandusersplitr update_pathr9rzipfileZipFile ZIP_DEFLATEDwalkjoinanyseprelpathwriterrr) rrrrZ skip_wordsskiprzrootfilesfile archive_namerr)rrrrE)s,     zUploadDirectory.__init__c stj|jdtd}t|d}||jW5QRXddl }| |}|j |jdW5QRX|j rtj|j|j}|t jkrt jd|t|dS)Nztmp-z.zipwbr)r)rrrZlocal_directoryr=r>rrrrr extractallrr{insertremove)rr7rrrrrrrrr3Ns   zUploadDirectory.setupN)r-r.r/r0rEr3rrrrrs %rc@sTeZdZddZddZddZddZd d Zd d Zd dZ ddZ ddZ dS)forward_streamcCsd||_i|_tt||_|dkr(d|_n |dkr8d|_ntd|d|dkrTdnd|_g|_dS)NrxrGrqz1Expected stream to be 'stdout' or 'stderr'; got 'rg)_worker_original_methodsgetattrr{_stream_file ValueError_buffer)rstreamr"rrrrEas  zforward_stream.__init__cCs||||dSr)_forward)rZwrite_fnrrrr_writeqs zforward_stream._writecCs(|j|d|ksd|kr$|dS)N  )rappend_send)rrrrrrus zforward_stream._forwardcCs*|j|jddd}|jd|g|_dS)N)r rrendprint)rrrr,)rr+rrrr{szforward_stream._sendcCs||dSrr)rZflush_fnrrr_flushszforward_stream._flushcCs||dSrr)rZclose_fnrrr_closeszforward_stream._closecCs0t|j|}||j|<t|j|t||dSr)rrrsetattr functoolspartial)r method_nameZ interceptorZoriginal_methodrrr _intercepts   zforward_stream._interceptcCs0|d|j|d|j|d|j|jS)Nrflushr)rrrrrrrrr __enter__szforward_stream.__enter__cCs6|j|jD]\}}t|j||qi|_dSr)rrrrr)rexc_type exc_value tracebackattroriginalrrr__exit__s zforward_stream.__exit__N) r-r.r/rErrrrrrrrrrrrr`src@s eZdZdZddZddZdS) ForwardOutputa#A Worker Plugin that forwards ``stdout`` and ``stderr`` from workers to clients This plugin forwards all output sent to ``stdout`` and ``stderr` on all workers to all clients where it is written to the respective streams. Analogous to the terminal, this plugin uses line buffering. To ensure that an output is written without a newline, make sure to flush the stream. .. warning:: Using this plugin will forward **all** output in ``stdout`` and ``stderr`` from every worker to every client. If the output is very chatty, this will add significant strain on the scheduler. Proceed with caution! Examples -------- >>> from dask.distributed import ForwardOutput >>> plugin = ForwardOutput() >>> client.register_worker_plugin(plugin) cCs6t|_|jtd|d|jtd|ddS)Nrx)r"rq) contextlib ExitStack _exit_stack enter_contextrr2rrrr3s zForwardOutput.setupcCs|jdSr)rrr2rrrr4szForwardOutput.teardownN)r-r.r/r0r3r4rrrrrsr)( __future__rr\rrloggingrrJryr{r=rcollections.abcrtypingrrrZ dask.utilsrrZdistributed.schedulerr r getLoggerr-rMr r1r5r?ABCr@r^rurrrrrrrrrs8   jE# lLB  E<