U /e(@sddlmZddlZddlZddlZddlZddlmZddlm Z ddl m Z ddl m Z ddlmZddlmZdd lmZdd lmZdd lmZeeZGd d d e ZdS)) annotationsN) CPU_COUNT) get_template) SpecCluster)nprocesses_nthreads)Nanny) Scheduler)Security)Worker)parse_memory_limitcsjeZdZdZdddddddddejddddddddddddddffdd Zd d Zd fd d ZZ S) LocalClusterugCreate local Scheduler and Workers This creates a "cluster" of a scheduler and workers running on the local machine. Parameters ---------- n_workers: int Number of workers to start memory_limit: str, float, int, or None, default "auto" Sets the memory limit *per worker*. Notes regarding argument data type: * If None or 0, no limit is applied. * If "auto", the total system memory is split evenly between the workers. * If a float, that fraction of the system memory is used *per worker*. * If a string giving a number of bytes (like ``"1GiB"``), that amount is used *per worker*. * If an int, that number of bytes is used *per worker*. Note that the limit will only be enforced when ``processes=True``, and the limit is only enforced on a best-effort basis — it's still possible for workers to exceed this limit. processes: bool Whether to use processes (True) or threads (False). Defaults to True, unless worker_class=Worker, in which case it defaults to False. threads_per_worker: int Number of threads per each worker scheduler_port: int Port of the scheduler. 8786 by default, use 0 to choose a random port silence_logs: logging level Level of logs to print out to stdout. ``logging.WARN`` by default. Use a falsey value like False or None for no change. host: string Host address on which the scheduler will listen, defaults to only localhost ip: string Deprecated. See ``host`` above. dashboard_address: str Address on which to listen for the Bokeh diagnostics server like 'localhost:8787' or '0.0.0.0:8787'. Defaults to ':8787'. Set to ``None`` to disable the dashboard. Use ':0' for a random port. worker_dashboard_address: str Address on which to listen for the Bokeh worker diagnostics server like 'localhost:8787' or '0.0.0.0:8787'. Defaults to None which disables the dashboard. Use ':0' for a random port. diagnostics_port: int Deprecated. See dashboard_address. asynchronous: bool (False by default) Set to True if using this cluster within async/await functions or within Tornado gen.coroutines. This should remain False for normal use. blocked_handlers: List[str] A list of strings specifying a blocklist of handlers to disallow on the Scheduler, like ``['feed', 'run_function']`` service_kwargs: Dict[str, Dict] Extra keywords to hand to the running services security : Security or bool, optional Configures communication security in this cluster. Can be a security object, or True. If True, temporary self-signed credentials will be created automatically. protocol: str (optional) Protocol to use like ``tcp://``, ``tls://``, ``inproc://`` This defaults to sensible choice given other keyword arguments like ``processes`` and ``security`` interface: str (optional) Network interface to use. Defaults to lo/localhost worker_class: Worker Worker class used to instantiate workers from. Defaults to Worker if processes=False and Nanny if processes=True or omitted. **worker_kwargs: Extra worker arguments. Any additional keyword arguments will be passed to the ``Worker`` class constructor. Examples -------- >>> cluster = LocalCluster() # Create a local cluster # doctest: +SKIP >>> cluster # doctest: +SKIP LocalCluster("127.0.0.1:8786", workers=8, threads=8) >>> c = Client(cluster) # connect to local cluster # doctest: +SKIP Scale the cluster to three workers >>> cluster.scale(3) # doctest: +SKIP Pass extra keyword arguments to Bokeh >>> LocalCluster(service_kwargs={'dashboard': {'prefix': '/foo'}}) # doctest: +SKIP Nrz:8787Fc s|dk r |}| dk r"td| } |dkr8tdd}d|krJtd|dkrd|dkpbt|t}|dkrx|rttnt}d|_||_|dkrt}n$|dkrt}nt |tst d|dkr|rd|kr| dd}n$|r|j rd }n|js| sd }nd }| ds|d}|dkr6|d s6|s6d }|p>i}|pHi}|dkrx|dkrx|rpt\}}nd}t}|dkr|dk r|rtdt|nd}|r|dkrtdttt|}|rd|krtdd|td|d<||||| | dk |||| d ttt||||| ||| dk | |d |p>id}||dfddt|D}tj|||||| ||d dS)NzMdiagnostics_port has been deprecated. Please use `dashboard_address=` insteadrz_Setting `threads_per_worker` to 0 has been deprecated. Please set to None or to a specific int. dashboardzSetting `dashboard` is discouraged. Please set `dashboard_address` to affect the scheduler (more common) and `worker_dashboard_address` for the worker (less common).Tz"security must be a Security objectz://ztls://z inproc://ztcp://Zinprocz 127.0.0.1r Z memory_limitauto)logger) hostZnthreadsservicesdashboard_addressr interfaceprotocolsecurity silence_logs) rrservice_kwargsrportrrrrblocked_handlers)clsoptionscsi|] }|qSr).0iworkerrsz)LocalCluster.__init__..) name schedulerworkersr!loop asynchronousrrscheduler_sync_interval)warningswarn issubclassrr status processesr temporary isinstance TypeErrorsplitZrequire_encryptionendswith startswithrrmaxintmathceilr rupdatertoolzmergedictrangesuper__init__)selfr$Z n_workersZthreads_per_workerr.r'startripZscheduler_portrrZworker_dashboard_addressZdiagnostics_portrZworker_servicesrr(rrrrZ worker_classZscheduler_kwargsr)Z worker_kwargsr%r& __class__r r"r?qs              zLocalCluster.__init__cOs tddS)NzdThe `cluster.start_worker` function has been removed. Please see the `cluster.scale` method instead.)NotImplementedError)r@argskwargsrrr" start_worker szLocalCluster.start_workercs(tdj|jj|j|d}tj|dS)Nzlocal_cluster.html.j2)r-r.cluster_status)rI)rrenderr-r$r.r> _repr_html_)r@rIrCrr"rKs zLocalCluster._repr_html_)N) __name__ __module__ __qualname____doc__loggingWARNr?rHrK __classcell__rrrCr"r s:[r ) __future__rrPr7r*r:Z dask.systemrZ dask.widgetsrZdistributed.deploy.specrZdistributed.deploy.utilsrZdistributed.nannyrZdistributed.schedulerrZdistributed.securityr Zdistributed.workerr Zdistributed.worker_memoryr getLoggerrLrr rrrr"s