U f/emA @sfdZddlmZddlZddlZddlmZmZddlm Z ddl m Z m Z ddl mZddlmZdd lmZdd lmZmZmZdd lmZdd lmZdd lmZmZmZmZm Z dddddZ!d2dddddddddZ"GdddZ#Gddde#Z$Gd d!d!e#Z%eej&d"d#d3d&d'dd(d)dd*d+d,d-d.Z'eej&d"d#d4dddd/d0d1Z(dS)5z parquet compat ) annotationsN)AnyAnyStr)catch_warnings)FilePathOrBufferStorageOptions)import_optional_dependencyAbstractMethodError)doc) DataFrame MultiIndex get_option)generic)Version) IOHandles get_handle is_fsspec_urlis_urlstringify_pathstrBaseImpl)enginereturnc Cs|dkrtd}|dkr|ttg}d}|D]D}z |WStk rj}z|dt|7}W5d}~XYq(Xq(td||dkrtS|dkrtStd dS) zreturn our implementationautozio.parquet.enginez - NzUnable to find a usable engine; tried using: 'pyarrow', 'fastparquet'. A suitable version of pyarrow or fastparquet is required for parquet support. Trying to import the above resulted in these errors:pyarrow fastparquetz.engine must be one of 'pyarrow', 'fastparquet')r PyArrowImplFastParquetImpl ImportErrorr ValueError)rZengine_classesZ error_msgsZ engine_classerrr#5/tmp/pip-unpacked-wheel-tiezk1ph/pandas/io/parquet.py get_engine%s$ $ r%rbFrrrboolz.tuple[FilePathOrBuffer, IOHandles | None, Any])pathfsstorage_optionsmodeis_dirrcCst|}t|r:|dkr:td}|jj|f|p0i\}}n|rVt|rN|dkrVtdd}|s|st|trt j |st ||d|d}d}|j }|||fS)zFile handling for PyArrow.Nfsspecr&z8storage_options passed with buffer, or non-supported URLFZis_textr*)rrrcoreZ url_to_fsrr! isinstancerosr(isdirrhandle)r(r)r*r+r,path_or_handler-handlesr#r#r$_get_path_or_handleGs8   r6c@s6eZdZeddddZddddZd dd ZdS) rr )dfcCsxt|tstdt|jtr>tdd|jjDsRtdn|jjdkrRtdtdd|jj D}|sttddS) Nz+to_parquet only supports IO with DataFramescss|]}|jdkVqdS)>stringemptyN) inferred_type).0xr#r#r$ vsz.BaseImpl.validate_dataframe..z parquet must have string column names for all values in each level of the MultiIndex >r8r9z%parquet must have string column namescss |]}|dk rt|tVqdSN)r0r)r;namer#r#r$r=sz!Index level names must be strings) r0r r!columnsr alllevelsr:indexnames)r7Z valid_namesr#r#r$validate_dataframens    zBaseImpl.validate_dataframecKs t|dSr>r )selfr7r( compressionkwargsr#r#r$writeszBaseImpl.writeNcKs t|dSr>r )rFr(r@rHr#r#r$readsz BaseImpl.read)N)__name__ __module__ __qualname__ staticmethodrErIrJr#r#r#r$rmsc@s>eZdZddZdddddd d d d d Zdd dddZdS)rcCs&tdddddl}ddl}||_dS)Nrz(pyarrow is required for parquet support.extrar)rZpyarrow.parquetZpandas.core.arrays._arrow_utilsapi)rFrpandasr#r#r$__init__szPyArrowImpl.__init__snappyNr zFilePathOrBuffer[AnyStr] str | None bool | Nonerlist[str] | None)r7r(rGrCr*partition_colsc Ks||d|ddi}|dk r*||d<|jjj|f|} t||dd|d|dk d\} } |d<zH|dk r|jjj| | f||d|n|jjj | | fd|i|W5| dk r| XdS)NZschemaZpreserve_index filesystemwb)r*r+r,)rGrXrG) rEpoprQZTableZ from_pandasr6closeparquetZwrite_to_datasetZ write_table) rFr7r(rGrCr*rXrHZfrom_pandas_kwargstabler4r5r#r#r$rIsB    zPyArrowImpl.writeFr*c KsHd|d<i}|rddl}|j||j||j||j| |j | |j | |j||j||j||j|i }|j|d<td} | dkrd|d<t||dd|d d \} } |d<zB|jjj| fd |i|jf|} | dkr(| jdd d } | WS| dk rB| XdS)NTZuse_pandas_metadatarZ types_mapperzmode.data_managerarrayZ split_blocksrYr&)r*r+r@F)copy)rRrQZint8Z Int8DtypeZint16Z Int16DtypeZint32Z Int32DtypeZint64Z Int64DtypeZuint8Z UInt8DtypeZuint16Z UInt16DtypeZuint32Z UInt32DtypeZuint64Z UInt64DtypeZbool_Z BooleanDtyper8Z StringDtypegetrr6r[r\r]Z read_table to_pandasZ _as_manager) rFr(r@use_nullable_dtypesr*rHZto_pandas_kwargspdmappingmanagerr4r5resultr#r#r$rJsd    zPyArrowImpl.read)rTNNN)NFNrKrLrMrSrIrJr#r#r#r$rs/rc@s6eZdZddZd ddddd Zddd d d ZdS)rcCstddd}||_dS)Nrz,fastparquet is required for parquet support.rO)rrQ)rFrr#r#r$rSs zFastParquetImpl.__init__rTNr r)r7r*c s||d|kr$|dk r$tdnd|kr6|d}|dk rFd|d<t|}t|rrtdfdd|d<n r~td td d $|jj||f|||d |W5QRXdS) N partition_onzYCannot use both partition_on and partition_cols. Use partition_cols for partitioning dataZhiveZ file_schemer-csj|dfpiS)NrZopenr(_r-r*r#r$#s z'FastParquetImpl.write.. open_withz?storage_options passed with file object or non-fsspec file pathT)record)rGZ write_indexrj) rEr!r[rrrrrQrI)rFr7r(rGrCrXr*rHr#ror$rIs8   zFastParquetImpl.writer_c  s|dd}|rtdt|}i}d}t|r|tdt|jjtdkrhj|dfp\ij |d<qfdd |d <n,t |t rt j |st|ddd }|j}|jj|f|}|jfd |i|} |dk r|| S) NrdFzNThe 'use_nullable_dtypes' argument is not supported for the fastparquet enginer-z0.6.1r&r)csj|dfpiS)Nr&rkrmror#r$rpIs z&FastParquetImpl.read..rqr.r@)r[r!rrrrrQ __version__rlr)r0rr1r(r2rr3Z ParquetFilercr\) rFr(r@r*rHrdZparquet_kwargsr5Z parquet_filerhr#ror$rJ5s>  zFastParquetImpl.read)rTNNN)NNrir#r#r#r$rs 3rr*r_rrTr zFilePathOrBuffer | NonerUrVrWz bytes | None)r7r(rrGrCr*rXrc Ksrt|tr|g}t|}|dkr(tn|} |j|| f||||d||dkrjt| tjsbt| SdSdS)a Write a DataFrame to the parquet format. Parameters ---------- df : DataFrame path : str or file-like object, default None If a string, it will be used as Root Directory path when writing a partitioned dataset. By file-like object, we refer to objects with a write() method, such as a file handle (e.g. via builtin open function) or io.BytesIO. The engine fastparquet does not accept file-like objects. If path is None, a bytes object is returned. .. versionchanged:: 1.2.0 engine : {{'auto', 'pyarrow', 'fastparquet'}}, default 'auto' Parquet library to use. If 'auto', then the option ``io.parquet.engine`` is used. The default ``io.parquet.engine`` behavior is to try 'pyarrow', falling back to 'fastparquet' if 'pyarrow' is unavailable. compression : {{'snappy', 'gzip', 'brotli', None}}, default 'snappy' Name of the compression to use. Use ``None`` for no compression. index : bool, default None If ``True``, include the dataframe's index(es) in the file output. If ``False``, they will not be written to the file. If ``None``, similar to ``True`` the dataframe's index(es) will be saved. However, instead of being saved as values, the RangeIndex will be stored as a range in the metadata so it doesn't require much space and is faster. Other indexes will be included as columns in the file output. partition_cols : str or list, optional, default None Column names by which to partition the dataset. Columns are partitioned in the order they are given. Must be None if path is not a string. {storage_options} .. versionadded:: 1.2.0 kwargs Additional keyword arguments passed to the engine Returns ------- bytes if no path argument is provided else None N)rGrCrXr*)r0rr%ioBytesIOrIAssertionErrorgetvalue) r7r(rrGrCr*rXrHimplZ path_or_bufr#r#r$ to_parquet]s&9  ry)rr*rdcKs"t|}|j|f|||d|S)a Load a parquet object from the file path, returning a DataFrame. Parameters ---------- path : str, path object or file-like object Any valid string path is acceptable. The string could be a URL. Valid URL schemes include http, ftp, s3, gs, and file. For file URLs, a host is expected. A local file could be: ``file://localhost/path/to/table.parquet``. A file URL can also be a path to a directory that contains multiple partitioned parquet files. Both pyarrow and fastparquet support paths to directories as well as file URLs. A directory path could be: ``file://localhost/path/to/tables`` or ``s3://bucket/partition_dir`` If you want to pass in a path object, pandas accepts any ``os.PathLike``. By file-like object, we refer to objects with a ``read()`` method, such as a file handle (e.g. via builtin ``open`` function) or ``StringIO``. engine : {{'auto', 'pyarrow', 'fastparquet'}}, default 'auto' Parquet library to use. If 'auto', then the option ``io.parquet.engine`` is used. The default ``io.parquet.engine`` behavior is to try 'pyarrow', falling back to 'fastparquet' if 'pyarrow' is unavailable. columns : list, default=None If not None, only these columns will be read from the file. {storage_options} .. versionadded:: 1.3.0 use_nullable_dtypes : bool, default False If True, use dtypes that use ``pd.NA`` as missing value indicator for the resulting DataFrame (only applicable for ``engine="pyarrow"``). As new dtypes are added that support ``pd.NA`` in the future, the output with this option will change to use those dtypes. Note: this is an experimental option, and behaviour (e.g. additional support dtypes) may change without notice. .. versionadded:: 1.2.0 **kwargs Any additional kwargs are passed to the engine. Returns ------- DataFrame )r@r*rd)r%rJ)r(rr@r*rdrHrxr#r#r$ read_parquets;rz)Nr&F)NrrTNNN)rNNF))__doc__ __future__rrtr1typingrrwarningsrZpandas._typingrrZpandas.compat._optionalrZ pandas.errorsr Zpandas.util._decoratorsr rRr r rZ pandas.corerZpandas.util.versionrZpandas.io.commonrrrrrr%r6rrrZ _shared_docsryrzr#r#r#r$sF        %&$ic O