U
e{j5 @ s d Z ddlZddlZddlZddlmZ ddlmZ ddl m
Z
dZ g Z dd Z
dd
d Zdd d
Zdd Zdd Zdd ZdS )z
This package contains functions for reading and writing Parquet
tables that are not meant to be used directly, but instead are
available as readers/writers in `astropy.table`. See
:ref:`astropy:table_io` for more details.
N)
minversion)AstropyUserWarning)NOT_OVERWRITING_MSGs PAR1c O sb |dk rHz | }W n tk
r* Y dS X |d}|| |tkS |dk rZ|dS dS dS )a Checks if input is in the Parquet format.
Parameters
----------
origin : Any
filepath : str or None
fileobj : `~pyarrow.NativeFile` or None
*args, **kwargs
Returns
-------
is_parquet : bool
True if 'fileobj' is not None and is a pyarrow file, or if
'filepath' is a string ending with '.parquet' or '.parq'.
False otherwise.
NF )z.parquetz.parq)tellAttributeErrorreadseekPARQUET_SIGNATUREendswith)originfilepathfileobjargskwargspos signature r V/var/www/html/bwave/python/venv/lib/python3.8/site-packages/astropy/io/misc/parquet.pyparquet_identify s
r Fc s t \}}}t| ttjfs.t| ds.td|| }|jdk rXdd |j D } ni } ddl
m }
m } m
} i }
d| kr| dd }| |}d
|kr|d
}
nd}d d |jD }d }d
|
krd}|
d
}|D ] }t|| D ] || < qqt| |dk r| |dk r,| ttfdd| D g }D ]& fdd| D }|| qT|std|rt|
d
D ] }|kr|
d
| q|s|j | ||d}|j }nd}g }|D ] | j |! |" fkr&|#| j $ qd }|| krHt%| | }nP|rhd}t&'d| dt( n0t)dd | D }t&'d| d| dt( |#| j |! krd | nd | qt*j+|tt,||d }|s|D ] | - | dd< q|
||
d } |dk rzd d |d! D } | j. D ]8} d"D ],} | | | j/ krHt0| | | | j/ | qHq@| 1| } | S )#a
Read a Table object from a Parquet file.
This requires `pyarrow `_
to be installed.
The ``filters`` parameter consists of predicates that are expressed
in disjunctive normal form (DNF), like ``[[('x', '=', 0), ...], ...]``.
DNF allows arbitrary boolean logical combinations of single column
predicates. The innermost tuples each describe a single column predicate.
The list of inner predicates is interpreted as a conjunction (AND),
forming a more selective and multiple column predicate. Finally, the most
outer list combines these filters as a disjunction (OR).
Predicates may also be passed as List[Tuple]. This form is interpreted
as a single conjunction. To express OR in predicates, one must
use the (preferred) List[List[Tuple]] notation.
Each tuple has format: (``key``, ``op``, ``value``) and compares the
``key`` with the ``value``.
The supported ``op`` are: ``=`` or ``==``, ``!=``, ``<``, ``>``, ``<=``,
``>=``, ``in`` and ``not in``. If the ``op`` is ``in`` or ``not in``, the
``value`` must be a collection such as a ``list``, a ``set`` or a
``tuple``.
Examples:
.. code-block:: python
('x', '=', 0)
('y', 'in', ['a', 'b', 'c'])
('z', 'not in', {'a','b'})
Parameters
----------
input : str or path-like or file-like object
If a string or path-like object, the filename to read the table from.
If a file-like object, the stream to read data.
include_names : list [str], optional
List of names to include in output. If not supplied, then
include all columns.
exclude_names : list [str], optional
List of names to exclude from output (applied after ``include_names``).
If not supplied then no columns are excluded.
schema_only : bool, optional
Only read the schema/metadata with table information.
filters : list [tuple] or list [list [tuple] ] or None, optional
Rows which do not match the filter predicate will be removed from
scanned data. See `pyarrow.parquet.read_table()` for details.
Returns
-------
table : `~astropy.table.Table`
Table will have zero rows and only metadata information
if schema_only is True.
r z5pyarrow can only open path-like or file-like objects.Nc S s" i | ]\}}| d | d qS zUTF-8)decode.0kvr r r
s z&read_table_parquet..r )Tablemeta serializetable_meta_yaml
r c S s i | ]
}||qS r r r namer r r r s FZ__serialized_columns__Tc s g | ]}| kr|qS r r r x) use_namesr r
s z&read_table_parquet..c s g | ]\}} |kr|qS r r )r ncolr# r r r' s z3No include_names specified were found in the table.)columnsfilters table::len::
zNo z1 found in metadata. Guessing {strlen} for schema.c s s | ]}t | V qd S )N)lenZas_py)r rowr r r s z%read_table_parquet..z* found in metadata. Using longest string (z
characters).Uz|S)dtype)datar c S s i | ]}|d |qS r* r r$ r r r r s datatype) descriptionformatunitr )2 get_pyarrow
isinstancestrosPathLikehasattr TypeErrorZ read_schemametadataitems
astropy.tabler r r popsplitZget_header_from_yamlnames
_get_namessetvaluesintersection_updatedifference_updatelistdictfromkeysextend
ValueErrorkeysZ
read_tablenum_rowsfieldtypestringbinaryappendZto_pandas_dtypeintwarningswarnr maxnpZzeroszipZto_numpyr+ r# setattrZ _construct_mixins_from_columns) inputZ
include_namesZ
exclude_namesZ schema_onlyr, paparquet_schemamdr r r Z meta_dict meta_yamlZmeta_hdrZfull_table_columnsZhas_serialized_columnsZserialized_columnsZscolZ
names_to_readrE pa_tablerQ r3 Zmd_nameZstrlenr4 tableZ header_colsr) attrr )r# r& r read_table_parquet; s ;
(
rh c s ddl m}m} ddlm} t \}}t|ttj fsFt
d| |d | | W 5 Q R X | }d
|} i }
j D ]^\} } | jjtjkrt| jjd |
d| < n$| jjtjkrt| jj|
d| < | |
d < qd
d |
D }
fd d
jjD }j||
d}tj|rL|r>t| ntt ||j |||d6}fdd
D }j j!||d}|"| W 5 Q R X dS )a
Write a Table object to a Parquet file
This requires `pyarrow `_
to be installed.
Parameters
----------
table : `~astropy.table.Table`
Data table that is to be written to file.
output : str or path-like
The filename to write the table to.
overwrite : bool, optional
Whether to overwrite any existing file without warning. Default `False`.
r )r r )serialize_context_asz.`output` should be a string or path-like, not r` r! r r- r c S s" i | ]\}}| d | d qS r )encoder r r r r - s z'write_table_parquet..c s" g | ]}| j| jfqS r )Zfrom_numpy_dtyper3 rS r" Z encode_tabler_ r r r' 3 s z'write_table_parquet..)r@ )versionc s g | ]} |qS r )array)r r) )r_ r r r' C s )rb N)#rB r r Zastropy.utils.data_infori r9 r: r; r< r= r? Zrepresent_mixins_as_columnsZget_yaml_from_tablejoinr+ rA r3 rS r[ Zstr_itemsizebytes_rE rb pathexistsremoveOSErrorr r7 Z
ParquetWriterZitercolsr Z from_arraysZ write_table)rf output overwriter r ri r` writer_versionrd Z
meta_yaml_strr@ r# r) Zmetadata_encodeZ type_listrb writerZarraysre r rk r write_table_parquet s>
ry c C sF g }| D ]4\}}t|tr.|t| q |dkr || q |S )a" Recursively find the names in a serialized column dictionary.
Parameters
----------
_dict : `dict`
Dictionary from astropy __serialized_columns__
Returns
-------
all_names : `list` [`str`]
All the column names mentioned in _dict and sub-dicts.
r# )rA r: rL rN rF rV )_dictZ all_namesr r r r r rF J s
rF c C sF ddl m} ddlm} | d|t | d|t | d|t dS )z,
Register Parquet with Unified I/O.
r )registry)r r` N)
Z
astropy.ior{ rB r Zregister_readerrh Zregister_writerry Zregister_identifierr )Z io_registryr r r r register_parquet` s
r| c C sT zdd l } ddl m} W n tk
r4 tdY nX t| drFd}nd}| ||fS )Nr )r` z3pyarrow is required to read and write parquet filesz6.0.0z2.4z2.0)Zpyarrowr` ImportError Exceptionr )r_ r` rw r r r r9 l s
r9 )NNFN)F)__doc__r< rX numpyr[ Z
astropy.utilsr Zastropy.utils.exceptionsr Zastropy.utils.miscr r
__all__r rh ry rF r| r9 r r r r s& "
D
L