Batch Reader Library Selection
Evaluated per format on four axes: pure Rust, maintained, decodes straight to Arrow, and version-coupling risk. “Pure Rust” is not one verdict — it splits per format, and two formats cannot be arrow-native by nature.
| Format | Library | Arrow-native | Version risk | Verdict |
|---|---|---|---|---|
| Parquet | parquet (arrow-rs) | yes, direct | lockstep with Arrow | adopt |
| CSV | arrow-csv | yes, direct | lockstep | adopt |
| JSON | arrow-json | yes, direct | lockstep | adopt |
| Excel | calamine | no — row→Arrow assembly | independent, low churn | adopt |
| XML | quick-xml + our mapping | no — bespoke mapping | independent, low churn | adopt |
| — | Polars | — | nightly-API breakage | reject |
Why not Polars
The predecessor system used Polars for batch reading and had to vendor patches to
compile on stable Rust, because polars-ooc/polars-stream depended on
nightly-only APIs. The selection above removes the need for Polars entirely:
parquet + arrow-csv + arrow-json cover the arrow-native formats and version
in lockstep with the workspace Arrow through the codec-core re-export;
calamine covers Excel; quick-xml plus our mapping covers XML. Every reader is
pure Rust with no C dependency. Dropping Polars is not a preference — it removes
the specific version-coupling failure the predecessor documented.
The two non-arrow-native readers
Excel (calamine) yields cell-by-cell values, not Arrow, so it needs a
row→Arrow assembly step the others don’t. Excel’s loose typing (dates as serial
floats, numbers-as-text) is a classic mapping minefield, so the cell→Arrow
mapping is governed by type-map and the date-serial gotcha is fixed once, with
a regression test.
XML splits into parsing (quick-xml — pure Rust, streaming, not in question)
and mapping XML→Arrow, which is genuinely bespoke and ours. XML→Arrow is not a
solved library problem the way Parquet→Arrow is: attributes vs elements, mixed
content, repeated-elements-as-lists, namespaces and optional-everything make the
mapping opinionated. codec-xml (streaming payloads) and the batch XML reader
share one mapping core, as Avro registry and OCF share codec-avro.
Contract schema beats inference
CSV, JSON and XML inference is lossy — inherently for XML (everything is a string
until typed), and sampling-based for CSV/JSON. Where a contract (ODCS) or XSD
supplies a schema, batch reads use it and skip inference; inference is the
fallback only when no schema is available. This makes the reads both safer and
faster, and is another place contract-core pays for itself. Stated once as a
cross-format source-batch rule (ADR-0035).