j0 d dl mZ d dlZd dlZd dlmZmZ d dlmZ d dl m
Z
d dl m Z er,d dl
m
Z
d dlmZ d d lmZ d d
lmZ d d lmZmZ d d lmZ d d
lmZ dddddddddd(d$Zdddddddd%d)d'ZdS )* ) annotationsN)
TYPE_CHECKINGAny)wrap_ldf)NoPickleOption) DeltaDataset)datetime)Path)Literal
DeltaTable) DataFrame LazyFrame)StorageOptionsDict)CredentialProviderFunctionautoF)versioncolumnsrechunkstorage_optionscredential_providerdelta_table_options use_pyarrowpyarrow_optionssourcestr | Path | DeltaTabler int | str | datetime | Noner list[str] | Noner bool | Noner StorageOptionsDict | Noner 3CredentialProviderFunction | Literal['auto'] | Noner dict[str, Any] | Noner boolr returnr c
t | ||||||| } || | } | S )a7
Reads into a DataFrame from a Delta lake table.
Parameters
----------
source
DeltaTable or a Path or URI to the root of the Delta lake table.
Note: For Local filesystem, absolute and relative paths are supported but
for the supported object storages - GCS, Azure and S3 full URI must be provided.
version
Numerical version or timestamp version of the Delta lake table.
Note: If `version` is not provided, the latest version of delta lake
table is read.
columns
Columns to select. Accepts a list of column names.
rechunk
Make sure that all columns are contiguous in memory by
aggregating the chunks into a single array.
storage_options
Extra options for the storage backends supported by `deltalake`.
For cloud storages, this may include configurations for authentication etc.
More info is available `here
`__.
credential_provider
Provide a function that can be called to provide cloud storage
credentials. The function is expected to return a dictionary of
credential keys along with an optional credential expiry time.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
delta_table_options
Additional keyword arguments while reading a Delta lake Table.
use_pyarrow
Flag to enable pyarrow dataset reads.
pyarrow_options
Keyword arguments while converting a Delta lake Table to pyarrow table.
Returns
-------
DataFrame
Examples
--------
Reads a Delta table from local filesystem.
Note: Since version is not provided, the latest version of the delta table is read.
>>> table_path = "/path/to/delta-table/"
>>> pl.read_delta(table_path) # doctest: +SKIP
Reads a specific version of the Delta table from local filesystem.
Note: This will fail if the provided version of the delta table does not exist.
>>> pl.read_delta(table_path, version=1) # doctest: +SKIP
Time travel a delta table from local filesystem using a timestamp version.
>>> pl.read_delta(
... table_path, version=datetime(2020, 1, 1, tzinfo=timezone.utc)
... ) # doctest: +SKIP
Reads a Delta table from AWS S3.
See a list of supported storage options for S3 `here
`__.
>>> table_path = "s3://bucket/path/to/delta-table/"
>>> storage_options = {
... "AWS_ACCESS_KEY_ID": "THE_AWS_ACCESS_KEY_ID",
... "AWS_SECRET_ACCESS_KEY": "THE_AWS_SECRET_ACCESS_KEY",
... }
>>> pl.read_delta(table_path, storage_options=storage_options) # doctest: +SKIP
Reads a Delta table from Google Cloud storage (GCS).
See a list of supported storage options for GCS `here
`__.
>>> table_path = "gs://bucket/path/to/delta-table/"
>>> storage_options = {"SERVICE_ACCOUNT": "SERVICE_ACCOUNT_JSON_ABSOLUTE_PATH"}
>>> pl.read_delta(table_path, storage_options=storage_options) # doctest: +SKIP
Reads a Delta table from Azure.
Following type of table paths are supported,
* az:///
* adl:///
* abfs:///
See a list of supported storage options for Azure `here
`__.
>>> table_path = "az://container/path/to/delta-table/"
>>> storage_options = {
... "AZURE_STORAGE_ACCOUNT_NAME": "AZURE_STORAGE_ACCOUNT_NAME",
... "AZURE_STORAGE_ACCOUNT_KEY": "AZURE_STORAGE_ACCOUNT_KEY",
... }
>>> pl.read_delta(table_path, storage_options=storage_options) # doctest: +SKIP
Reads a Delta table with additional delta specific options. In the below example,
`without_files` option is used which loads the table without file tracking
information.
>>> table_path = "/path/to/delta-table/"
>>> delta_table_options = {"without_files": True}
>>> pl.read_delta(
... table_path, delta_table_options=delta_table_options
... ) # doctest: +SKIP
)r r r r r r r r )
scan_deltaselectcollect)
r r r r r r r r r dfs
^/home/longshao/multi-rider-rag/.venv/lib/python3.11/site-packages/polars/io/delta/functions.py
read_deltar+ sX v
' / / '
B
YYw
::<< )r r r r r r r r c ddl m} ddlm} d}
t j d ddlm} t | | r| }
|
| || |d } n||dk rd }
t |
d} ~|
|
j |i |
j pi |pi }t t |
|
t | nd||| ||||pd
}t | | S ) aY
Lazily read from a Delta lake table.
Parameters
----------
source
DeltaTable or a Path or URI to the root of the Delta lake table.
Note: For Local filesystem, absolute and relative paths are supported but
for the supported object storages - GCS, Azure and S3 full URI must be provided.
version
Numerical version or timestamp version of the Delta lake table.
Note: If `version` is not provided, the latest version of delta lake
table is read.
storage_options
Extra options for the storage backends supported by `deltalake`.
For cloud storages, this may include configurations for authentication etc.
More info is available `here
`__.
credential_provider
Provide a function that can be called to provide cloud storage
credentials. The function is expected to return a dictionary of
credential keys along with an optional credential expiry time.
.. warning::
This functionality is considered **unstable**. It may be changed
at any point without it being considered a breaking change.
delta_table_options
Additional keyword arguments while reading a Delta lake Table.
use_pyarrow
Flag to enable pyarrow dataset reads.
pyarrow_options
Keyword arguments while converting a Delta lake Table to pyarrow table.
Use this parameter when filtering on partitioned columns or to read
from a 'fsspec' supported filesystem.
rechunk
Make sure that all columns are contiguous in memory by
aggregating the chunks into a single array.
Returns
-------
LazyFrame
Examples
--------
Creates a scan for a Delta table from local filesystem.
Note: Since version is not provided, the latest version of the delta table is read.
>>> table_path = "/path/to/delta-table/"
>>> pl.scan_delta(table_path).collect() # doctest: +SKIP
Creates a scan for a specific version of the Delta table from local filesystem.
Note: This will fail if the provided version of the delta table does not exist.
>>> pl.scan_delta(table_path, version=1).collect() # doctest: +SKIP
Time travel a delta table from local filesystem using a timestamp version.
>>> pl.scan_delta(
... table_path, version=datetime(2020, 1, 1, tzinfo=timezone.utc)
... ).collect() # doctest: +SKIP
Creates a scan for a Delta table from AWS S3.
See a list of supported storage options for S3 `here
`__.
>>> table_path = "s3://bucket/path/to/delta-table/"
>>> storage_options = {
... "AWS_REGION": "eu-central-1",
... "AWS_ACCESS_KEY_ID": "THE_AWS_ACCESS_KEY_ID",
... "AWS_SECRET_ACCESS_KEY": "THE_AWS_SECRET_ACCESS_KEY",
... }
>>> pl.scan_delta(
... table_path, storage_options=storage_options
... ).collect() # doctest: +SKIP
Creates a scan for a Delta table from Google Cloud storage (GCS).
See a list of supported storage options for GCS `here
`__.
>>> table_path = "gs://bucket/path/to/delta-table/"
>>> storage_options = {"SERVICE_ACCOUNT": "SERVICE_ACCOUNT_JSON_ABSOLUTE_PATH"}
>>> pl.scan_delta(
... table_path, storage_options=storage_options
... ).collect() # doctest: +SKIP
Creates a scan for a Delta table from Azure.
Supported options for Azure are available `here
`__.
Following type of table paths are supported,
* az:///
* adl:///
* abfs[s]:///
>>> table_path = "az://container/path/to/delta-table/"
>>> storage_options = {
... "AZURE_STORAGE_ACCOUNT_NAME": "AZURE_STORAGE_ACCOUNT_NAME",
... "AZURE_STORAGE_ACCOUNT_KEY": "AZURE_STORAGE_ACCOUNT_KEY",
... }
>>> pl.scan_delta(
... table_path, storage_options=storage_options
... ).collect() # doctest: +SKIP
Creates a scan for a Delta table with additional delta specific options.
In the below example, `without_files` option is used which loads the table without
file tracking information.
>>> table_path = "/path/to/delta-table/"
>>> delta_table_options = {"without_files": True}
>>> pl.scan_delta(
... table_path, delta_table_options=delta_table_options
... ).collect() # doctest: +SKIP
r ) PyLazyFrame)!_init_credential_provider_builderN deltalaker r& r z?cannot use credential_provider when passing a DeltaTable objectF) table_
table_uri_r r credential_provider_builderr r r r ) polars._plrr. ,polars.io.cloud.credential_provider._builderr/ importlibutil find_specr0 r
isinstance
ValueError_storage_optionsr r strr new_from_dataset_object)r r r r r r r r r. r/ tabler
r3 msgdatasets r* r& r& sn @ (''''' $E~ ,,8(((((( fj ) ) E }&G&G ,'
'
##
(-@F-J-JOoo &*#
*o.I
%+
$"
e$$"'-3v;;;T'$? / ' 5
G K77@@ A AAr, )r r r r r r r r r r r r! r r" r r# r r" r$ r )r r r r r r r r! r r" r r# r r" r r r$ r )
__future__r r6 importlib.utiltypingr r polars._utils.wrapr polars.io.cloud._utilsr polars.io.delta._datasetr r pathlibr
r r0 r
polarsr r polars._typingr polars.io.cloudr r+ r& r, r* rL s " " " " " " % % % % % % % % ' ' ' ' ' ' 1 1 1 1 1 1 1 1 1 1 1 1 ;!!!!!! $$$$$$++++++++111111:::::: ,0 $ 15OU15 -1H H H H H H\ ,015OU15 -1 mB mB mB mB mB mB mB mBr,