jddlmZddlmZddlmZmZddlZddl m Z erddl m Z ddlmZmZddd ddZ ddddddZdS)) annotations)partial) TYPE_CHECKINGAnyNpyarrow)Iterator) DataFrame LazyFrameT)allow_pyarrow_filter batch_sizedspa.dataset.Datasetr boolr int | Nonereturnr ctt|||}tj|j||S)a' Pickle the partially applied function `_scan_pyarrow_dataset_impl`. The bytes are then sent to the polars logical plan. It can be deserialized once executed and ran. Parameters ---------- ds pyarrow dataset allow_pyarrow_filter Allow predicates to be pushed down to pyarrow. This can lead to different results if comparisons are done with null values as pyarrow handles this different than polars does. batch_size The maximum row count for scanned pyarrow record batches. r user_batch_sizer)r_scan_pyarrow_dataset_implplr _scan_python_functionschema)rr r funcs m/home/longshao/multi-rider-rag/.venv/lib/python3.11/site-packages/polars/io/pyarrow_dataset/anonymous_scan.py_scan_pyarrow_datasetrsO2 " 1"    D < - - 4!5 .  r with_columnslist[str] | None predicatestr | bytes | Nonen_rowsr tuple[Iterator[DataFrame], bool]c d}|r:|8ddlm}m} m} m} ddlm} m} m}t|t| | ||| | | d}|}||d||n|}||d<d fd }|du}||fS) an Take the projected columns and materialize an arrow table. Parameters ---------- ds pyarrow dataset. with_columns Columns that are projected. predicate pyarrow expression string (when `allow_pyarrow_filter=True`) or serialized Polars predicate bytes (when `allow_pyarrow_filter=False`). n_rows: Materialize only `n` rows from the arrow dataset. batch_size The maximum row count for scanned pyarrow record batches. allow_pyarrow_filter If True, evaluate predicate and return DataFrame directly. If False, return `(generator, False)` tuple for IOPlugin path. user_batch_size User-specified `batch_size` (takes precedence over Rust-provided `batch_size`). Warnings -------- Don't use this if you accept untrusted user inputs. Predicates will be evaluated with python 'eval'. There is sanitation in place, but it is a possible attack vector. Returns ------- tuple[Iterator[DataFrame], bool] A generator over the DataFrames and a boolean indicating if the predicates is applied. Nr) to_py_dateto_py_datetime to_py_timeto_py_timedelta)DateDatetimeDuration)par)r*r+r%r&r'r()columnsfilterr rIterator[DataFrame]c34Kdkr$tjjfiVdS}jdiD]Z}|jdkr|4|dkrdS|j|kr|d|}||jz}tj|V[dS)Nr)rr head to_batchesnum_rowsslice from_arrow) remainingbatch common_paramsrr"s rframesz*_scan_pyarrow_dataset_impl..framess Q;;,wrwv????@@ @ @ @ F "R]33]33 ' 'E~""$>>EE>I--!KK955EU^+ -&& & & & & ' 'r)rr/) polars._utils.convertr%r&r'r(polars.datatypesr)r*r+evalr,)rrr r"r r rfilter_r%r&r'r(r)r*r+vr:"applies_predicate_in_this_functionr9s` ` @rrr3s(^G 5            >=========  $$("0(#2    >G0<$P$PM$3$?ZJ&0 l#''''''''**1)<& 6887 77r)rrr rr rrr )N)rrrrr r!r"rr rr rrrrr#) __future__r functoolsrtypingrrpolarsrpolars._dependenciesrr,collections.abcr r r rrr1rrrGs""""""%%%%%%%%......,((((((++++++++ "&! !!!!!!R" f8"&"&f8f8f8f8f8f8f8f8r