U /e)@s>ddlmZddlZddlmZmZddlmZddlm Z ddl m Z ddl Z z ddlZWnek rtdZYnXGdddeZGd d d e ZGd d d e Zedkrejnejadad ZddZddZddZddZddZddZddZddZddZ d d!Z!d"d#Z"d$d%Z#d&d'Z$d(d)Z%d*d+Z&dS),) annotationsN)IntEnumauto)uname) NamedTuple)parsec@s0eZdZeZeZeZeZeZeZ dS) NVMLStateN) __name__ __module__ __qualname__r UNINITIALIZED INITIALIZEDDISABLED_PYNVML_NOT_AVAILABLEDISABLED_CONFIGDISABLED_LIBRARY_NOT_FOUND DISABLED_WSL_INSUFFICIENT_DRIVERrr@/tmp/pip-unpacked-wheel-g426oqom/distributed/diagnostics/nvml.pyrsrc@s2eZdZUdZded<dZded<dZded<dS)CudaDeviceInfoNz bytes | Noneuuidz int | None device_index mig_index)r r r r__annotations__rrrrrrr!s   rc@s"eZdZUded<dZded<dS) CudaContextbool has_contextNzCudaDeviceInfo | None device_info)r r r rrrrrrr's rz512.15cCsttjkottkS)z&Is pynvml initialized on this process?) NVML_STATErr NVML_OWNER_PIDosgetpidrrrris_initialized;sr!cCs dtjkS)zCheck if we are in Windows Subsystem for Linux; some PyNVML queries are not supported there. Taken from https://www.scivision.dev/python-detect-wsl/ zmicrosoft-standard)rreleaserrrr_in_wsl@sr#c CsttjtjtjtjhkrdSttjkr6tt kr6dSttj krVt j dsVtjadSttjkrltt ksxttj krz tWn(tjtjtjfk rtjaYdSXtrttttkrtjadSddlm}tjat a|ntdtdtddS)zIdempotent (per-process) initialization of PyNVML Notes ----- Modifies global variables NVML_STATE and NVML_OWNER_PIDNzdistributed.diagnostics.nvmlradd_gpu_metricsz+Unhandled initialisation state (NVML_STATE=z, NVML_OWNER_PID=))rrrrrrr rrr r daskconfiggetpynvmlZnvmlInitZNVMLError_LibraryNotFoundZNVMLError_DriverNotLoadedZNVMLError_Unknownr# parse_versionZnvmlSystemGetDriverVersiondecodeMINIMUM_WSL_VERSIONZdistributed.workerr% RuntimeErrorr$rrr init_onceGsV      r/cCsttsdStSdS)Nr)r/r!r*ZnvmlDeviceGetCountrrrrdevice_get_count}sr0cCst}ttjkrtdnttjkr.tdnttjkrJtdtdnlttjkr^tdnX|dkrptdnFz"t t t t j dd d }Wntk rd}YnXt|SdS) Nz8NVML monitoring requires PyNVML and NVML to be installedz$PyNVML is installed, but NVML is notz3Outdated NVIDIA drivers for WSL, please upgrade to z or newerzKPyNVML monitoring disabled by 'distributed.diagnostics.nvml' config settingrzNo GPUs availableZCUDA_VISIBLE_DEVICES,)r0rrrr.rrr-rnextmapintrenvironr)split ValueErrorr*nvmlDeviceGetHandleByIndex)countZgpu_idxrrr_pynvml_handless,         r;cCs8tttdrt|}n t|}tdd|DS)a)Check whether the current process is same as that of handle Parameters ---------- handle : pyvnml.nvml.LP_struct_c_nvmlDevice_t NVML handle to CUDA device Returns ------- out : bool Whether the device handle has a CUDA context on the running process. 'nvmlDeviceGetComputeRunningProcesses_v2css|]}t|jkVqdSN)rr pid).0procrrr sz+_running_process_matches..)r/hasattrr*r<Z$nvmlDeviceGetComputeRunningProcessesany)handleZrunning_processesrrr_running_process_matchess    rEc CsttrttD]}t|}zt|\}}Wntjk rRtj}YnX|tj krtt |D]`}zt ||}Wntj k rYqlYnXt |rlt|}tdt|||ddSqlqt |rt|}tdt||ddSqtddS)aKCheck whether the current process already has a CUDA context created. Returns ------- out : CudaContext Object containing information as to whether the current process has a CUDA context created, and in the positive case containing also information about the device the context belongs to. T)rrr)rrrrF)r)r/r!ranger0r*r9nvmlDeviceGetMigModeNVMLError_NotSupportedZNVML_DEVICE_MIG_DISABLEZNVML_DEVICE_MIG_ENABLEZnvmlDeviceGetMaxMigDeviceCountZ#nvmlDeviceGetMigDeviceHandleByIndexZNVMLError_NotFoundrEnvmlDeviceGetUUIDrr)indexrDZmig_current_modeZmig_pending_moderZ mig_handlerrrrhas_cuda_contextsD        rLcCs~tz t|}t|}t|}WnJtk rpt|trD|nt|d}t|}t |}t|}YnXt ||dS)a`Get both device index and UUID from device index or UUID Parameters ---------- device : int, bytes or str An ``int`` with the index of a GPU, or ``bytes`` or ``str`` with the UUID of a CUDA (either GPU or MIG) device. Returns ------- out : CudaDeviceInfo Object containing information about the device. Examples -------- >>> get_device_index_and_uuid(0) # doctest: +SKIP {'device-index': 0, 'uuid': b'GPU-e1006a74-5836-264f-5c26-53d19d212dfe'} >>> get_device_index_and_uuid('GPU-e1006a74-5836-264f-5c26-53d19d212dfe') # doctest: +SKIP {'device-index': 0, 'uuid': b'GPU-e1006a74-5836-264f-5c26-53d19d212dfe'} >>> get_device_index_and_uuid('MIG-7feb6df5-eccf-5faa-ab00-9a441867e237') # doctest: +SKIP {'device-index': 0, 'uuid': b'MIG-7feb6df5-eccf-5faa-ab00-9a441867e237'} utf-8rF) r/r5r*r9rJr8 isinstancebytesnvmlDeviceGetHandleByUUIDZnvmlDeviceGetIndexr)devicerZ device_handlerZ uuid_handlerrrget_device_index_and_uuids   rRcCstzt|}t|}Wn6tk rRt|tr:|nt|d}t|}YnXz t|WStj k r~ddgYSXdS)aXGet MIG mode for a device index or UUID Parameters ---------- device: int, bytes or str An ``int`` with the index of a GPU, or ``bytes`` or ``str`` with the UUID of a CUDA (either GPU or MIG) device. Returns ------- out : list A ``list`` with two integers ``[current_mode, pending_mode]``. rMrN) r/r5r*r9r8rNrOrPrHrI)rQrrDrrrrget_device_mig_mode s rScCs,zt|jWStjk r&YdSXdSr=)r*ZnvmlDeviceGetUtilizationRatesZgpurIhrrr_get_utilization(srVcCs,zt|jWStjk r&YdSXdSr=)r*nvmlDeviceGetMemoryInfousedrIrTrrr_get_memory_used/srYcCs,zt|jWStjk r&YdSXdSr=)r*rWtotalrIrTrrr_get_memory_total6sr[cCs.zt|WStjk r(YdSXdSr=)r*ZnvmlDeviceGetNamer,rIrTrrr _get_name=sr\cCst}t|t|dS)N)Z utilizationz memory-used)r;rVrYrTrrr real_timeDsr]cCst}t|t|dS)N)z memory-totalname)r;r[r\rTrrrone_timeLsr_)' __future__rrenumrrplatformrtypingrZpackaging.versionrr+r'r* ImportErrorrrrrr rrr-r!r#r/r0r;rErLrRrSrVrYr[r\r]r_rrrrsD       6-)