U /e@snddlZddlZddlmZmZddlmZmZddlm Z ddl m Z ddl m Z mZdd d Zd d ZdS)N)OpenFileget_fs_token_paths)infer_compression read_block)tokenize)delayed) is_integer parse_bytesF128 MiB10 kiBc s"t|ttttjfstdt|d|d\}} t| dkrHt d||dk rzt|trbt |}t |srtdt |}|dkrdggt| } dggt| } n8g} g} | D](dkrt } n} | dk rtd d } | dkrtd q| dkr| g| gq| |r@| |kr@| | |}n|}d}dg}g}| ||d d kr||7}|t |||d|dqR|| |d|rd |d<|dd 8<| || |qttg}t| | | D]b\}}t||fdd|D}fddt|||D}||q|r |dkrnd}t|trt |}t| ddp}dkr||}nT||}||}|sʐq|kr||d d}q||}q|}W5QRX|r||| fS||fS)aU Given a path or paths, return delayed objects that read from those paths. The path may be a filename like ``'2015-01-01.csv'`` or a globstring like ``'2015-*-*.csv'``. The path may be preceded by a protocol, like ``s3://`` or ``hdfs://`` if those libraries are installed. This cleanly breaks data by a delimiter if given, so that block boundaries start directly after a delimiter and end on the delimiter. Parameters ---------- urlpath : string or list Absolute or relative filepath(s). Prefix with a protocol like ``s3://`` to read from alternative filesystems. To read from multiple files you can pass a globstring or a list of paths, with the caveat that they must all have the same protocol. delimiter : bytes An optional delimiter, like ``b'\n'`` on which to split blocks of bytes. not_zero : bool Force seek of start-of-file delimiter, discarding header. blocksize : int, str Chunk size in bytes, defaults to "128 MiB" compression : string or None String like 'gzip' or 'xz'. Must support efficient random access. sample : int, string, or boolean Whether or not to return a header sample. Values can be ``False`` for "no sample requested" Or an integer or string value like ``2**20`` or ``"1 MiB"`` include_path : bool Whether or not to include the path with the bytes representing a particular file. Default is False. **kwargs : dict Extra options that make sense to a particular storage connection, e.g. host, port, username, password, etc. Examples -------- >>> sample, blocks = read_bytes('2015-*-*.csv', delimiter=b'\n') # doctest: +SKIP >>> sample, blocks = read_bytes('s3://bucket/2015-*-*.csv', delimiter=b'\n') # doctest: +SKIP >>> sample, paths, blocks = read_bytes('2015-*-*.csv', include_path=True) # doctest: +SKIP Returns ------- sample : bytes The sample header blocks : list of lists of ``dask.Delayed`` Each list corresponds to a file, and each delayed object computes to a block of bytes from that file. paths : list of strings, only included if include_path is True List of same length as blocks, where each item is the path to the file represented in the corresponding block. z3Path should be a string, os.PathLike, list or tuplerb)modeZstorage_optionsrz%s resolved to no filesNzblocksize must be an integerZinferzHCannot do chunked reads on compressed files. To read, set blocksize=NonesizezfBacking filesystem couldn't determine file size, cannot do chunked reads. To read, set blocksize=None.csg|]}d|dqS)z read-block--).0o)tokenr3/tmp/pip-unpacked-wheel-dbjnr7gq/dask/bytes/core.py szread_bytes..c s.g|]&\}}}td|||dqS) compression)Z dask_key_name)r)rrkeyl)r delayed_read delimiterfspathrrrs Tr r) isinstancestrlisttupleosPathLike TypeErrorrlenOSErrorr rintr ValueErrorinfoappendrread_block_from_fileziprZukeyrreadsplit)ZurlpathrZnot_zero blocksizesamplerZ include_pathkwargsZfs_tokenpathsoffsetslengthscomprZ blocksize1ZplaceofflengthoutoffsetkeysvaluesfZ sample_buffnewr)rrrr r!rr read_bytes sB                     rBc CsTt|@}|dkr0|dkr0|W5QRSt||||W5QRSQRXdS)Nr)copyr1r)Z lazy_filer:bsrr@rrrr/s r/)NFr r NF)rCr&Z fsspec.corerrZ fsspec.utilsrrZ dask.baserZ dask.delayedrZ dask.utilsrr rBr/rrrrs   1