U /e/ @sdZddlZddlmZmZmZddlmZddlm Z ddl Z ddl m Z ddlmZdd Ze jZe jZe jZe jZee jZee jZee jZee jZe jZe jZe jZe jZe jZe jZe jZe j Z e j!Z!e"e# e j$Z$W5QRXe j%Z%e"e# e j&Z&W5QRXe j'Z'e"e# e j(Z(W5QRXd)d d Z)d*d dZ*ddZ+ddZ,ddZ-ddZ.ddZ/d+ddZ0d,ddZ1d-ddZ2d.d!d"Z3d#d$Z4d%d&Z5d'd(Z6dS)/z- A set of NumPy functions to apply per chunk N) ContainerIterableSequencewraps)Integral)concat)flattencstdfdd }|S)zU A wrapper for functions that don't provide keepdims to ensure that they do. Nc s|f|d|i|}|s |S|}|dkr6t|j}t|tttfsL|g}t}t|jD]$}||krr|d7}q\|tdf7}q\||}|S)NaxisN)rangendim isinstancerrrtupleslice) xr keepdimsargskwargsraxesZr_sliceZ each_axis a_callable4/tmp/pip-unpacked-wheel-dbjnr7gq/dask/array/chunk.pykeepdims_wrapped_callables  z3keepdims_wrapper..keepdims_wrapped_callable)NNr)rrrrrkeepdims_wrapper srFc stjD]}|kr d|<q |rHtfddtjD}|ttfddtjD}||fdttdjddi|S)aCoarsen array by applying reduction to fixed size neighborhoods Parameters ---------- reduction: function Function like np.sum, np.mean, etc... x: np.ndarray Array to be coarsened axes: dict Mapping of axis to coarsening factor Examples -------- >>> x = np.array([1, 2, 3, 4, 5, 6]) >>> coarsen(np.sum, x, {0: 2}) array([ 3, 7, 11]) >>> coarsen(np.max, x, {0: 3}) array([3, 6]) Provide dictionary of scale per dimension >>> x = np.arange(24).reshape((4, 6)) >>> x array([[ 0, 1, 2, 3, 4, 5], [ 6, 7, 8, 9, 10, 11], [12, 13, 14, 15, 16, 17], [18, 19, 20, 21, 22, 23]]) >>> coarsen(np.min, x, {0: 2, 1: 3}) array([[ 0, 3], [12, 15]]) You must avoid excess elements explicitly >>> x = np.array([1, 2, 3, 4, 5, 6, 7, 8]) >>> coarsen(np.min, x, {0: 3}, trim_excess=True) array([1, 4]) c3s<|]4\}}||r*td|| ntddVqdS)rNr).0idrrr szcoarsen..cs&g|]}j|||fqSr)shaperr rrrr szcoarsen..r )r r r enumerater$rZreshape)Z reductionrrZ trim_excessrr indZnewshaperr&rcoarsenRs(  "r+csNttrg|jttr8fddt|jD|tddDS)aCTrim boundaries off of array >>> x = np.arange(24).reshape((4, 6)) >>> trim(x, axes={0: 0, 1: 1}) array([[ 1, 2, 3, 4], [ 7, 8, 9, 10], [13, 14, 15, 16], [19, 20, 21, 22]]) >>> trim(x, axes={0: 1, 1: 1}) array([[ 7, 8, 9, 10], [13, 14, 15, 16]]) csg|]}|dqS)r)getr%r"rrr'sztrim..css"|]}t||r| ndVqdSr r)rZaxrrrr#sztrim..)rrr dictr r)rrrr"rtrims    r.cs||dks tdt||jkr*|Stj|| d}|dkrPt| dnt| |tfddt|jDS)aChunk and combine function of topk Extract the k largest elements from a on the given axis. If k is negative, extract the -k smallest elements instead. Note that, unlike in the parent function, the returned elements are not sorted internally. Trr Nc3s"|]}|krntdVqdSr rr%r Zk_slicerrr#sztopk..) AssertionErrorabsr$np partitionrrr r akr rrr0rtopks r8cs\|dks tt|||}dtj|d}|dkr<|S|tfddt|jDS)zmFinal aggregation function of topk Invoke topk one final time and then sort the results internally. Trr/c3s*|]"}|krtdddntdVqdSNrr%r/rrr#sz!topk_aggregate..)r1r8r3sortrr r r5rr/rtopk_aggregates  r<cCs||fS)z^Preparatory step for argtopk Put data together with its original indices in a tuple. r)r6idxrrrargtopk_preprocesssr>cs|dks tdt|trXtt|}tdd|D}tdd|D}n|\}}t||jkrv|Stj|| d}|dkrt | dnt | |t fdd t |j D}t ||t ||fS) a)Chunk and combine function of argtopk Extract the indices of the k largest elements from a on the given axis. If k is negative, extract the indices of the -k smallest elements instead. Note that, unlike in the parent function, the returned elements are not sorted internally. TrcSsg|] \}}|qSrr)rai_rrrr'szargtopk..cSsg|]\}}t||jqSr)r3Z broadcast_tor$)rr?Zidxirrrr'sr/Nc3s"|]}|krntdVqdSr rr%r0rrr#szargtopk..)r1rlistr r3Z concatenater2r$Z argpartitionrrr r take_along_axisZ a_plus_idxr7r rr6r=Zidx2rr0rargtopks     "rDcs|dks tt|dkr|n|d}t|||\}}dtj|d}t||}|dkrf|S|tfddt|jDS)zFinal aggregation function of argtopk Invoke argtopk one final time, sort the results internally, drop the data and return the index only. Trrr/c3s*|]"}|krtdddntdVqdSr9rr%r/rrr#sz$argtopk_aggregate..) r1lenrDr3ZargsortrBrr r rCrr/rargtopk_aggregates  rFcCs:ddlm}||||||d}t||kr6|ddS|S)Nr) arange_safeliker:)dask.array.utilsrGrE)startstopsteplengthdtyperIrGresrrraranges rQTcCsDddlm}t||r|}t||r0|}tj|||||dS)Nr)Array)endpointrO)Zdask.array.corerRrZcomputer3linspace)rKrLnumrSrOrRrrrrTs    rTcKs|j|f|Sr )astype)rZ astype_dtyperrrrrVsrVCcCs|dkrDztj||d}Wntk r8t|}YnX||Sztj||d}Wntk rtt|}YnX|j|jSdS)NrWrH)r3Zascontiguousarray TypeErrorviewZasfortranarrayT)rrOorderrrrrYs rYcsddlm}m}|||dtjtdk||dk|jk@}||tfddt |j DS)aChunk function of `slice_with_int_dask_array_on_axis`. Slice one chunk of x by one chunk of idx. Parameters ---------- x: ndarray, any dtype, any shape i-th chunk of x idx: ndarray, ndim=1, dtype=any integer j-th chunk of idx (cartesian product with the chunks of x) offset: ndarray, shape=(1, ), dtype=int64 Index of the first element along axis of the current chunk of x x_size: int Total size of the x da.Array along axis axis: int normalized axis to take elements from (0 <= axis < x.ndim) Returns ------- x sliced along axis, using only the elements of idx that fall inside the current chunk. r) asarray_safemeta_from_arrayrHc3s"|]}|krntdVqdSr rr%r r=rrr#Qsz,slice_with_int_dask_array..) rJr\r]rVr3int64wherer$rr r )rr=offsetZx_sizer r\r] idx_filterrr^rslice_with_int_dask_array&s rcc s|tj}t|dk|t||}d}d}t||D]Z}||k|||k@}t|}t||d|d7||7}|jdkr<||d7}q<|tfddt |j DS)aSFinal aggregation function of `slice_with_int_dask_array_on_axis`. Aggregate all chunks of x by one chunk of idx, reordering the output of `slice_with_int_dask_array`. Note that there is no combine function, as a recursive aggregation (e.g. with split_every) would not give any benefit. Parameters ---------- idx: ndarray, ndim=1, dtype=any integer j-th chunk of idx chunk_outputs: ndarray concatenation along axis of the outputs of `slice_with_int_dask_array` for all chunks of x and the j-th chunk of idx x_chunks: tuple dask chunks of the x da.Array along axis, e.g. ``(3, 3, 2)`` axis: int normalized axis to take elements from (0 <= axis < x.ndim) Returns ------- Selection from all chunks of x for the j-th chunk of idx, in the correct order rrr:c3s"|]}|krntdVqdSr rr%r Z idx_finalrrr#sz6slice_with_int_dask_array_aggregate..) rVr3r_r`sumZ zeros_likeZcumsumsizerr r ) r=Z chunk_outputsZx_chunksr Zx_chunk_offsetZchunk_output_offsetZx_chunkrbZidx_cumrrdr#slice_with_int_dask_array_aggregateTs"    rgc Csxz ||}Wn,tk r8}ztd|W5d}~XYnXz$|jjs\|jd|jkr\|}Wntk rrYnX|S)afGetitem function This function creates a copy of the desired selection for array-like inputs when the selection is smaller than half of the original array. This avoids excess memory usage when extracting a small portion from a large array. For more information, see https://numpy.org/doc/stable/reference/arrays.indexing.html#basic-slicing-and-indexing. Parameters ---------- obj: ndarray, string, tuple, list Object to get item from. index: int, list[int], slice() Desired selection to extract from obj. Returns ------- Selection obj[index] zTArray chunk size or shape is unknown. Possible solution with x.compute_chunk_sizes()Nr() IndexError ValueErrorflagsZowndatarfcopyAttributeError)objindexresulterrrgetitems  rq)F)N)N)TN)N)rW)7__doc__ contextlibcollections.abcrrr functoolsrZnumbersrZnumpyr3ZtlzrZ dask.corer rreprodminmaxZargminZ nanargminZargmaxZ nanargmaxanyallZnansumZnanprodZ nancumprodZ nancumsumZnanminZnanmaxZmeansuppressrlZnanmeanvarZnanvarZstdZnanstdr+r.r8r<r>rDrFrQrTrVrYrcrgrqrrrrs^    #        9    .8