U /e)@sddlZddlZddlmZddlmZddlmZddlm Z ddl Z ddl m Z ddlmZmZddlmZdd lmZdd lmZdd lmZdd lmZmZd ZddZddZddZdddZ dS)N)Counter)reduce)product)mul)config)Arraynormalize_chunks)meta_from_array)tokenize)flatten)HighLevelGraph)M parse_bytesa Dask's reshape only supports operations that merge or split existing dimensions evenly. For example: >>> x = da.ones((6, 5, 4), chunks=(3, 2, 2)) >>> x.reshape((3, 2, 5, 4)) # supported, splits 6 into 3 & 2 >>> x.reshape((30, 4)) # supported, merges 6 & 5 into 30 >>> x.reshape((4, 5, 6)) # unsupported, existing dimensions split unevenly To work around this you may call reshape in multiple passes, or (if your data is small enough) call ``compute`` first and handle reshaping in ``numpy`` directly. c sTtddDsttd}t|d}ddttD}ddtt|D}|dksl|dkrD|||kr|||<|||<|d8}|d8}qZ|}||}|dkrd||<|d8}qZ|dkrd||<|d8}qZ||krZ|d} | dkr.tt| |d|kr.| d8} qtt| |d|krRtttfd dt|Drt|dD]} | || <q||t t t| |||<nt| d|dD]} | f|| <qttt t| d|d} t | | || <tt| d|dt fd d|| D||<|d8}| d}qZ||krZ|d} | dkrtt|| |d|kr| d8} qjtt|| |d|krtttt|| d|dt |||<t| d|dD]} || f|| <qt fd d||D|| <| d}|d8}qZt |t |fS) Ncss|]}t|tVqdSN) isinstancetuple.0cr6/tmp/pip-unpacked-wheel-dbjnr7gq/dask/array/reshape.py "sz"reshape_rechunk..cSsg|]}dqSrrrirrr %sz#reshape_rechunk..cSsg|]}dqSrrrrrrr&srrc3s"|]}t||kVqdSr)lenr)inchunksinshaperrrBsc3s|]}|VqdSrrr)prodrrrPsc3s|]}|VqdSrrr)csrrrds)allAssertionErrorrrangerrNotImplementedError_not_implemented_messagemathr map expand_tuplercontract_tuple) rZoutshaperiiZoiZresult_inchunksZresult_outchunksdindoutZileftrZchunk_reductionZoleftr)r!rrr rreshape_rechunk!sv              &  r.cCs~|dkr |Sg}|D]L}|}t||d}|d|krR|t||t|8}q*|r||qt|t|ksvtt|S)z >>> expand_tuple((2, 4), 2) (1, 1, 2, 2) >>> expand_tuple((2, 4), 3) (1, 1, 1, 1, 2) >>> expand_tuple((3, 4), 2) (1, 2, 2, 2) >>> expand_tuple((7, 4), 3) (2, 2, 3, 1, 1, 2) r)maxappendintsumr#r)chunksfactoroutrxpartrrrr)ls  r)cCs\t||dkstg}d}|D]2}||7}||}||}||}|r ||q t|S)zReturn simple chunks tuple such that factor divides all elements Examples -------- >>> contract_tuple((2, 2, 8, 4), 4) (4, 8, 4) r)r3r#r1r)r4r5r6ZresidualchunkdivZgoodrrrr*s  r*Tcs>ddlm}ddlm}tt||}dd|D}t|t|krt|t|dkr`tdt|dkrz|jdkrz|S||j t t |dtfdd |D}t t|jrtd t|jt t |d|j krtd |j|kr|St|t|}d t||}|jdkrztt|} |fd t|tj| |fi} tdd |D} tj|| |gd} t| || |dSt|j} t|}|s| |kr|ddt| |D}t|j||j\}}t t dd|D|j j!}|dkrt"t#$d}t#$dd}n t"|}d}||kr|dkrDd}t%j&||ddnn|rt'|t'|@}g}|D]:}||dkr|(|||d8<n |(dqbt)||||j |d}||}t*t+|j,gfdd|D}t*t+|gfdd|D}t*t+|}ddt-|||D} tj|| |gd} t| |||dS) abReshape array to new shape Parameters ---------- shape : int or tuple of ints The new shape should be compatible with the original shape. If an integer, then the result will be a 1-D array of that length. One shape dimension can be -1. In this case, the value is inferred from the length of the array and remaining dimensions. merge_chunks : bool, default True Whether to merge chunks using the logic in :meth:`dask.array.rechunk` when communication is necessary given the input array chunking and the output shape. With ``merge_chunks==False``, the input array will be rechunked to a chunksize of 1, which can create very many tasks. limit: int (optional) The maximum block size to target in bytes. If no limit is provided, it defaults to using the ``array.chunk-size`` Dask config value. Notes ----- This is a parallelized version of the ``np.reshape`` function with the following limitations: 1. It assumes that the array is stored in `row-major order`_ 2. It only allows for reshapings that collapse or merge dimensions like ``(1, 2, 3, 4) -> (1, 6, 4)`` or ``(64,) -> (4, 4, 4)`` .. _`row-major order`: https://en.wikipedia.org/wiki/Row-_and_column-major_order When communication is necessary this algorithm depends on the logic within rechunk. It endeavors to keep chunk sizes roughly the same when possible. See :ref:`array-chunks.reshaping` for a discussion the tradeoffs of ``merge_chunks``. See Also -------- dask.array.rechunk numpy.reshape r)PerformanceWarning)sanitize_indexcSsg|]}|dkr|qS)rrsrrrrszreshape..rz&can only specify one unknown dimensionc3s|]}|dkrn|VqdS)r=Nrr>Z missing_sizerrrszreshape..z_Array chunk size or shape is unknown. shape: %s Possible solution with x.compute_chunk_sizes()z)total size of new array must be unchangedzreshape-)rcss|] }|fVqdSrr)rdrrrrs)Z dependencies)metacSsi|] }|dqSrrrrrr szreshape..cSsg|] }t|qSr)r0rrrrrsNzarray.chunk-sizez array.slicing.split-large-chunksTaReshaping is producing a large chunk. To accept the large chunk and silence this warning, set the option >>> with dask.config.set(**{'array.slicing.split_large_chunks': False}): ... array.reshape(shape) To avoid creating the large chunks, set the option >>> with dask.config.set(**{'array.slicing.split_large_chunks': True}): ... array.reshape(shape)Explictly passing ``limit`` to ``reshape`` will also silence this warning >>> array.reshape(shape, limit='128 MiB')) stacklevelauto)shapelimitdtypeZprevious_chunkscSsg|]}tt|qSrr$rrrrrr&scSsg|]}tt|qSrrJrrrrr'scSs i|]\}}}|tj||fqSr)r reshape)rabrGrrrrC)s).dask.array.corer;Zdask.array.slicingr<rr(r ValueErrorndimsizerrnpisnanr3rGstrr r Z npartitionsnextr Z __dask_keys__r rKr Zfrom_collectionsrZrechunkr$r.r4rIitemsizerrgetwarningswarnrr1rlistrnamezip)r7rGZ merge_chunksrHr;r<Z known_sizesrBr[keyZdskr4graphr,r-rZ outchunksZmax_chunksize_in_bytessplitmsgZmatching_chunksZ chunk_planr6Zx2Zin_keysZout_keysZshapesrr@rrKs*             rK)TN)!r'rX collectionsr functoolsr itertoolsroperatorrZnumpyrRZdaskrrNrrZdask.array.utilsr Z dask.baser Z dask.corer Zdask.highlevelgraphr Z dask.utilsr rr&r.r)r*rKrrrrs$         K