U /e@sddlmZddlZddlZddlmZddlZddlmZej ej dZ dddDZd ed <d ed <e fd d ddddZddZddZddZdddddZejdZdd ddd ZdS)!) annotationsN)Sequence)nbyteszdistributed.comm.shardcCsi|]}d|dqS)z max_%s_leni).0xrr>/tmp/pip-unpacked-wheel-g426oqom/distributed/protocol/utils.py sr )strbinarraymapextFstrict_map_keyrawzbytes | memoryviewintzlist[memoryview])framenreturncsR|pt}tj|kr gSjj}|jfddtd|DS)z Split a frame into a list of frames of maximum size This helps us to avoid passing around very large bytestrings. Examples -------- >>> frame_split_size([b'12345', b'678'], n=3) # doctest: +SKIP [b'123', b'45', b'678'] csg|]}||qSrr)rirZitems_per_shardrr +sz$frame_split_size..r)BIG_BYTES_SHARD_SIZE memoryviewritemsizerange)rrZnitemsrrrframe_split_sizes    rcCs,t|}tt|}tjd|d|f|S)NQ)lenr rstructpack)framesZnframesZ nbytes_framesrrrpack_frames_prelude.s r"cCsdt|f|S)zPack frames into a byte-like object This prepends length information to the front of the bytes-like object See Also -------- unpack_frames )joinr")r!rrr pack_frames4s r%c Csvt|}d}t|}t||\}t||||}g}|d|}|D]"}||}|||||}qN|S)zUnpack bytes into a sequence of frames This assumes that length information is at the front of the bytestring, as performed by pack_frames See Also -------- pack_frames r)rrcalcsize unpack_fromappend) bfmtZfmt_sizeZn_frameslengthsr!startlengthendrrr unpack_frames@s   r0zSequence[memoryview]r)mvsrc Cs|sttSt|dkr"|dS|d}t|tsFtdt||j}|j}d}d}t|D]\}}t|tst|dt||j dkrqb|j|k rt |d|jd||j st |d|j dkrt |d|j d |j|krt |d |jd|t |}|dkr.|}n4||} || krbt d |d | d d|| d||j 7}qb|dkrt|dkst|S|dkstdt|d} t | } || } | | | ||S)a Zero-copy "concatenate" a sequence of contiguous memoryviews. Returns a new memoryview which slices into the underlying buffer to extract out the portion equivalent to all of ``mvs`` being concatenated. All the memoryviews must: * Share the same underlying buffer (``.obj``) * When merged, cover a continuous portion of that buffer with no gaps * Have the same strides * Be 1-dimensional * Have the same format * Be contiguous Raises ValueError if these conditions are not met. r&rzExpected memoryview; got z: expected memoryview; got z#: memoryview has different buffer: z vs z: memoryview non-contiguousz: memoryview has z dimensions, not 1z: inconsistent format: z memoryview z2 does not start where the previous ends. Expected rz , starts z byte(s) away.z#Underlying buffer is null pointer?!B)r bytearrayr isinstance TypeErrortypeobjformat enumerater ValueError contiguousndimaddress_of_memoryviewAssertionErrorcast) r1firstr7r8Zfirst_start_addrrrmvZ start_addrZ expected_addrZbase_mvZbase_start_addr start_indexrrrmerge_memoryviews\sV            rCr&)rArcCstzt|}Wntk r"Yn Xt|Sz ddl}Wn$tk r^td|dYnX||j ddS)z Get the pointer to the first byte of a memoryview's data. If the memoryview is read-only, NumPy must be installed. rNz+Cannot get address of read-only memoryview z since NumPy is not installed.data) one_byte_carr from_bufferr5ctypes addressofZnumpy ImportErrorr:ZasarrayZ__array_interface__)rAZcarrnprrrr=s    r=) __future__rrGrcollections.abcrZdaskZdistributed.utilsrutilsZ parse_bytesconfiggetrZ msgpack_optsrr"r%r0rCc_byterEr=rrrrs&    J