ADR-0003: Arrow version strategy across the workspace and DataFusion

Reference

Gates ADR-0023 (Iceberg/Delta crate maturity), which shares the binding constraint. Constrains every crate that exposes an Arrow type. Interacts with ADR-0047 (independent publishing), which makes arrow a public dependency.

Context

arrow-rs makes breaking API changes at most releases. Types from two Arrow versions are distinct types to the compiler even when structurally identical, so a RecordBatch cannot pass between them directly.

The workspace pins one Arrow version through a single re-export in twg-codec-core, which prevents internal skew. It cannot prevent skew with dependencies that pin Arrow themselves — DataFusion above all, and also delta-rs and iceberg-rust. DataFusion’s internal Arrow version is outside our control, so any batch entering or leaving a DataFusion plan crosses a version boundary whenever the workspace runs ahead.

Three forces pull against each other:

• Security and currency. The preference is to pin latest secure releases. Deliberately running an older Arrow means inheriting someone else’s upgrade cadence for security fixes.
• Cost per crossing. A predecessor system ran a newer workspace Arrow than its query engine and bridged via Arrow IPC, measuring roughly 5–10% CPU overhead — paid on every transform batch, and recorded as a regret.
• Public API. Because crates publish independently (ADR-0047), arrow is a public dependency of the codec crates. Its version is part of our API, an Arrow major bump is a breaking change for us, and nothing exposing an Arrow type can stabilise until this is settled.

Options considered + consequences

Dimensions: currency and security posture, cost per crossing, blast radius, maintenance burden.

Option 1: Pin the workspace to DataFusion’s Arrow version

Track whatever DataFusion depends on; no boundary exists, so nothing to bridge.

• Pros: zero crossing cost. Simplest possible dependency graph. No unsafe. Trivially compatible with delta-rs/iceberg-rust if they track similarly.
• Cons: currency is delegated. An Arrow security advisory cannot be acted on until DataFusion moves, which conflicts directly with the stated preference for latest secure releases. Newer Arrow features unavailable. Our published crates’ Arrow version — a public API commitment — is then set by a third party’s release cadence.
• Mitigation to evaluate: DataFusion’s cadence is frequent and tends to track Arrow closely, so the staleness window may be weeks rather than versions. Measure the actual historical lag before treating this as disqualifying.

Option 2: Newest Arrow workspace-wide, bridge via Arrow IPC

The predecessor’s choice.

• Pros: full currency and immediate security response. Wire format is stable across versions, so the bridge is straightforward and safe code.
• Cons: serialisation copies every buffer, so cost scales with bytes — the measured 5–10% CPU on every transform batch. Directly contradicts the no-intermediate-materialisation principle this project applies elsewhere. Recorded upstream as a regret.

Option 3: Newest Arrow workspace-wide, bridge via the Arrow C Data Interface

The C Data Interface is a stable, versioned ABI intended for exchange between Arrow implementations — including two versions of the same implementation. arrow-rs exposes to_ffi/from_ffi over #[repr(C)] structs, so the crossing becomes pointer passing plus a release callback rather than serialisation.

• Pros: full currency, and cost scales with the number of arrays rather than bytes — a different order of magnitude for wide or large batches. The machinery is already scoped: twg-ffi exists as the single audited unsafe crate for the C Data Interface (ADR-0042/0043), so this is existing code serving a second purpose rather than new surface.
• Cons: unsafe at the boundary, with release-callback lifetime discipline — contained in twg-ffi, but real. Both sides must implement compatible C Data Interface spec versions; this is a stable spec but the assumption needs verifying, not assuming. Not free: per-array export/import still costs something.
• To measure: crossing cost against Option 2 on representative batch widths. The hypothesis is a large win; it is a hypothesis.

Option 4: Minimise crossings rather than cheapen them

Unlike the predecessor, DataFusion here is the optional SQL transform stage. Passthrough, neutral-value and direct-Arrow paths bypass it entirely, so the bridge tax applies only to traffic that actually runs SQL.

• Pros: attacks the volume rather than the unit cost. Simple projections and filters could run on our own Arrow kernels, reserving DataFusion — and its boundary — for genuinely complex SQL.
• Cons: two transform paths to maintain and to keep semantically identical, which is its own correctness risk. Only helps if the traffic split is favourable.
• To measure: what fraction of realistic pipelines need full SQL versus a projection or filter that our own kernels could serve.

Option 5: Run DataFusion out of process, over Flight

Version skew is irrelevant across a process boundary.

• Pros: total decoupling; each side upgrades independently. twg-pipeline-flight already exists for sidecar transforms.
• Cons: a network hop and a lifecycle to manage for what is otherwise an in-process stage. Heavyweight as a default, but a legitimate escape hatch if the version conflict ever becomes intractable.

Option 6: Fork or patch DataFusion onto our Arrow

• Rejected. Sustained maintenance of a fork of a large, fast-moving project, for a version-alignment problem. The cost is unbounded and recurring.

Evidence

Measured against the crates.io registry on 2026-07-26.

DataFusion tracks Arrow closely

Median 28 days, range 9–46. Arrow ships a major roughly every three months, so DataFusion is typically one month behind and never a version generation behind.

The previous Arrow line keeps receiving patches

arrow 59.1.0 shipped 2026-07-07; arrow 58.4.0 shipped 2026-07-22 — after it. The N-1 major line is actively maintained, not abandoned on the next major.

This is decisive for Option 1’s central objection. DataFusion pins with a caret (arrow ^58.3.0), so any 58.x at or above that satisfies it. A fix landing in the 58 line is therefore available immediately, without waiting for DataFusion. Only a fix that lands exclusively in 59.x and is never backported would be blocked — a narrower exposure than “delegating security currency” implies.

The ecosystem has converged, which resolves ADR-0023’s binding constraint

DataFusion and iceberg-rust, released within a day of each other, pin the same Arrow major. Pinning 58 therefore removes the version boundary with both — so the question of which of them is the binding constraint dissolves: they agree.

Recommended option

Option 1 — pin the workspace to the Arrow major that DataFusion and the table crates share. Today that is Arrow 58.

The objection to Option 1 was that it delegates security currency to a third party’s cadence. The evidence narrows that materially: DataFusion’s adoption lag is a month, the N-1 Arrow line keeps receiving patches, and the caret pin means those patches are available to us immediately. The remaining exposure is a fix that lands only in the newest major and is never backported — real, but narrow, and detectable.

Choosing it buys the properties every other option spends something to approximate: no version boundary at all, so no bridge to build, no crossing cost to pay per batch, no unsafe at the boundary, and the simplest dependency graph. Options 2 and 3 both exist only to make a boundary cheaper; not having one is strictly better than having a cheap one.

Options 2, 3 and 4 are retained as a designed escape path, not discarded. If the ecosystem diverges — DataFusion and the table crates pinning different majors, or a security fix we cannot wait for — the workspace moves to the newer Arrow and bridges the DataFusion stage alone. Option 3 (the C Data Interface) is the bridge to reach for rather than Option 2 (IPC): its cost scales with array count rather than bytes, and twg-ffi already exists as the single audited unsafe crate for that interface, so the machinery is present rather than hypothetical. Option 4 (minimising crossings, available because DataFusion is the optional transform stage here) reduces whatever that bridge would cost.

Deferred measurements. The IPC-versus-C-Data-Interface crossing cost, and the traffic split between SQL-requiring and projection-only pipelines, are no longer gating: with no boundary, neither number changes the decision. They become the first work of the escape path, if it is ever taken.

Consequences

• The workspace pins Arrow 58, expressed once in twg-codec-core’s re-export.
• Moving Arrow major is a breaking change for our published crates. Because crates publish independently (ADR-0047), arrow is a public dependency and its version is part of the API. The supported version must be stated in each codec crate’s README, and a bump requires a major version bump of those crates.
• Arrow-major moves are scheduled behind DataFusion, roughly quarterly, rather than being taken as they land.
• cargo-audit in CI is the detector for the residual exposure — a Arrow advisory unfixed in the pinned line is the signal to take the escape path, and should be treated as such rather than waited out.
• ADR-0023 loses its Arrow-compatibility dimension entirely; what remains there is feature maturity, which needs hands-on evaluation rather than registry data.

Advice Received

Document version history