ADR-0001: Build a pure-Rust, sans-io streaming crate ecosystem with co-equal byte, native-value and Arrow representations

Reference

Foundational ADR for the stream-arrow workspace. Downstream ADRs (consumer-group protocol, neutral value model, Arrow union mapping, offset-correlation contract, record-metadata representation, table-catalog write semantics, connector flow-control model, transform DAG semantics, WASM sandbox contract) will reference this document.

Summary

Stub — to be written when the decision approaches Accepted.

Context

We ingest from Kafka across a multi-brand estate spanning Confluent, Redpanda, MSK, Azure Event Hubs and StreamNative, under demanding SLAs. Every current Rust option forces a compromise. rdkafka wraps librdkafka: mature and complete, but it is a C dependency with C memory-safety characteristics, a build-time burden in CI images (cmake, OpenSSL), and known cancel-safety hazards around StreamConsumer::recv() inside select!. The pure-Rust alternatives (rskafka, kafka-rust) deliberately omit consumer groups, which disqualifies them for our workloads.

Separately, no crate in the ecosystem offers a clean path from broker bytes to Arrow RecordBatch. Every available library is shaped by JVM conventions: decode message-at-a-time into an intermediate dynamic value type, then convert. That intermediate materialisation is precisely the cost Arrow exists to eliminate, and it sits directly on the hot path into our Lakehouse and Lakebase targets.

Arrow is not, however, the only representation we need. Regulated retention requires the original wire bytes be kept for audit and replay, unmodified and envelope-intact. Proxying, tee-to-archive and forward-to-third-party workloads want bytes and never want a schema resolved at all. And record-at-a-time consumers — request/response handlers, validators, small services — want a decoded value without paying for a columnar dependency they will not use. A design that treats Arrow as the sole exit forces all three through decode-and-re-encode, which is wasteful for the first two and lossy for the first.

Fidelity is a first-class requirement rather than a quality goal. Kafka’s record model carries more than key and value — ordered headers permitting duplicate keys and null values, a timestamp with a broker-authoritative type, leader epoch, and batch-level producer identity. Abstractions routinely collapse these (headers into a map, null into empty) and the loss is invisible until an audit or a replay needs them. The design commits to carrying the full inventory and to documenting the three fields that are broker-authoritative and therefore not round-trippable by equality.

Schema resolution also has two distinct sources in our estate. Message-schema registries (Confluent, Glue Schema Registry) answer “what shape is this payload”; table catalogs (Glue Data Catalog) answer “what shape is the table this feeds”. They are different APIs answering different questions, and both are needed — the second in both directions, since batches must register and evolve the tables they land in.

Pulsar is now in scope as a first-class transport rather than a later possibility, and as a fresh protocol implementation rather than a dependency on pulsar-rs. Our own evaluation found that crate’s limitations to be architectural rather than incidental: backpressure is implicit in whether the caller polls its Stream, with the permit window tunable only through batch size; there is no rate limiter and no explicit pause/resume; seek destroys and recreates the consumer; and selective partition subscription forfeits automatic discovery. Each is fixable in isolation, but fixing them inside a design shaped around implicit backpressure means working against its grain permanently.

That decision surfaces a second one. Sink and source semantics, flow control, observability, retry and dead-lettering are not transport-specific, and placing them in either client would deny them to the other. They belong in a transport-neutral connector-core that both implement — and the Kafka client is retrofitted onto it as soon as it exists, because an abstraction designed against a single transport is a guess.

A middle layer is also in scope, and it changes what the project is. Up to this point the system is a library ecosystem configured at compile time by its authors. A SQL-configured transform layer makes it additionally a deployable configured at runtime by people who did not write it — which introduces a threat model the rest of the design does not have, since user-supplied queries and user-supplied binaries then execute inside our process.

The shape required is a DAG, not a flat fan-out: raw records tap to one sink while the same source is decoded, unpacked, split into virtual tables for data quality and filtering, and further unnested into derived tables with their own sinks. Sinks attach at interior nodes. Two properties of that shape are load-bearing and silent when wrong — intermediate nodes with multiple dependents must be materialised or the plan re-executes per dependent and multiplies broker load, and nodes that change cardinality break the offset-correlation contract unless derived rows carry their originating locator.

The intent is an ecosystem, not a crate. Kafka is the first transport; Pulsar and MQ follow. XML/XSLT joins the codec layer later, reusing existing XML-to-Arrow work. That trajectory only pays off if the layering holds from the first commit — a transport-coupled codec cannot be reused by the second transport, and a codec-coupled transport cannot be reused by the second format.

Recommended option

Option 3 — pure-Rust ecosystem with sans-io codec crates exposing three co-equal representations. Options 1 and 2 both solve the immediate ingestion problem faster, but neither produces reusable assets. The decisive argument is not memory safety in isolation; it is that the codec crates are the durable value, and they are only durable if they are runtime-free, transport-free, and not mandatorily columnar.

Three representations are supported as peers rather than as a primary with fallbacks:

• Bytes — passthrough, envelope intact, no schema resolution. Available with no codec crate present at all.
• Native value — decoded into a neutral value model owned by codec-core, not the upstream libraries’ types.
• Arrow — batch-oriented decode straight into ArrayBuilders, no intermediate materialisation.

Record metadata travels with the decoded representations as reserved __kafka_-prefixed columns rather than out-of-band alongside the batch. The out-of-band alternative keeps schemas clean but does not survive an Arrow Flight boundary, which would lose every metadata field at precisely the point our Lakehouse writes need them.

Extension is provided by two mechanisms with deliberately distinct contracts rather than one general-purpose mechanism. In-process WASM modules serve pure transforms and close the real DataFusion gap — user-defined functions currently require compiling into our binary — with a fuel-metered sandbox granted no network or filesystem capability. Out-of-process Flight sidecars serve callouts, stateful work and model inference. Merging them was considered and rejected: granting network access to the embedded sandbox dissolves the isolation that justifies it and makes per-batch latency unpredictable inside a query plan, while forcing pure transforms through a network hop is needless cost. Inline model inference specifically belongs in the sidecar, since WASM-compiled inference is materially slower than native and GPU-backed inference is impossible in-sandbox.

Packaging takes the statically linked musl binary as the primary artefact, from which OCI images, systemd deployments and function packages all derive; an image-first approach yields only an image. OCI rather than LXC because Kubernetes and its runtimes consume OCI, and multi-arch rather than arm64-only because the second architecture costs a build-matrix entry while excluding most on-premise and much of Azure.

Observability is designed in rather than added later. Spans follow the record lifecycle and propagate through message headers so traces cross the broker; metric names and labels are defined in connector-core so both transports are interchangeable behind one dashboard. Retrofitting this after two clients exist means two incompatible instrumentation schemes and a migration.

Table catalogs are a separate crate (table-catalog) from message-schema registries, not an implementation behind the same trait. Conflating them would put table-format and partition-registration concerns behind an interface shaped for wire envelopes.

The neutral value model is chosen over re-exporting apache_avro::Value / prost_reflect::DynamicMessage / serde_json::Value deliberately. Re-exporting is cheaper initially but couples our public API to three upstream release cadences, makes format-agnostic consumers impossible to write, and means a format swap is a breaking change for users. Owning the model costs a mapping layer per codec and one hard design problem — a value model expressive enough for Avro unions, Protobuf oneof, JSON’s dynamism and XML’s mixed content, without becoming a lowest common denominator. We accept that cost because format-agnostic consumption is a stated requirement of the MQ and Pulsar phases, and retrofitting a neutral model after three codecs have shipped upstream types is a breaking change we would rather not schedule.

Arrow is a default-on feature of the codec crates. The common case is columnar and defaults should serve it; default-features = false yields a build with no arrow dependency for slim and embedded consumers. This has one consequence worth stating plainly: codec-core’s Arrow re-export — the version-pinning chokepoint that keeps arrow-rs churn contained — exists only when the feature is enabled, so the no-Arrow build has a materially smaller public API.

The cost is honest and large: consumer groups, the idempotent producer and transactions are months of work, and rebalance correctness is unforgiving. We accept it because that gap is exactly what keeps the Rust ecosystem dependent on librdkafka — the difficulty is the moat, not a reason to retreat. It is planned in full rather than deferred.

Two purity boundaries are accepted and documented rather than hidden. Zstd compression has no production-grade pure-Rust encoder; ruzstd decodes, so we decode natively and feature-gate zstd-encode off by default, to be developed when a producer path needs it. GSSAPI/Kerberos has no production-grade pure-Rust implementation; it is an optional feature over cross-krb5 that explicitly breaks the guarantee when enabled. Both are stated in the README rather than discovered by a user.

Options considered + consequences

Dimensions: end-to-end safety, time-to-first-ingest, reuse across transports and formats, ops burden, hot-path performance.

Option 1: Thin ergonomic layer over rdkafka

Description: Wrap librdkafka bindings, add a schema-registry client and Arrow decoding above it.

Consequences:

• Pros: Protocol correctness is solved — SASL mechanisms, all broker variants, consumer groups, transactions work today. Fastest route to production ingest, plausibly a working consumer in weeks.
• Cons: C dependency on the hot path defeats the end-to-end safety goal; CI images carry cmake and OpenSSL; cancel-safety footguns leak into every consumer we write; nothing built is reusable when Pulsar and MQ arrive. Zero-copy passthrough is constrained by the C buffer ownership model.
• Ops burden: low. Time-to-deliver: lowest. Reuse: none.

Option 2: Pure-Rust transport, Arrow-only codecs over upstream value types

Description: Build the Kafka client in pure Rust; decode via apache-avro/prost into their native types and convert to Arrow afterwards. Arrow is the only supported output.

Consequences:

• Pros: Achieves memory safety across the transport; meaningfully less codec work than Option 3; no neutral value model to design.
• Cons: Per-message intermediate materialisation on the hot path is the exact cost Arrow removes, and the conversion is where the CPU goes. Codecs stay coupled to upstream value types, so format-agnostic consumption is impossible and the reuse story for Pulsar/MQ/XML does not materialise. Passthrough and record-at-a-time consumers are unserved. Delivers the hard part of Option 3 without its main payoff.
• Ops burden: medium. Time-to-deliver: medium. Reuse: partial.

Option 3: Full pure-Rust ecosystem, sans-io codecs, three co-equal representations

Description: Layered workspace — shared transport primitives (wire-tls, wire-sasl, wire-compression), a Kafka client over kafka-protocol, a registry crate splitting wire envelope from schema resolution, and independent codec crates exposing bytes, neutral values and Arrow builders with no runtime dependency.

Consequences:

• Pros: End-to-end memory and type safety with two documented exceptions. Codec crates reusable by Pulsar, MQ, DataFusion TableProviders and Lambda without modification, and usable without Arrow at all. Passthrough is near-free given refcounted slices, serving audit-retention and proxy workloads with no codec present. Batch-oriented push/finish preserves the columnar benefit. Arrow Flight and IPC fall out of the pipeline layer.
• Cons: Consumer-group protocol, idempotent producer and transactions are substantial and correctness-critical. The neutral value model is a hard design problem with real risk of becoming a lowest common denominator, and every codec now owes three surfaces rather than one — roughly 30–40% more codec work per format, incurred up front rather than amortised. Full-fidelity round-tripping multiplies the test matrix by three representations. The catalog write path carries blast radius beyond this system, since a mis-evolved table affects every consumer of it. Long runway before the first Arrow ingest. Two protocol implementations to maintain rather than one, which is a sustained cost and the largest single commitment in the plan — mitigated but not removed by shared transport primitives and a shared connector layer. Runtime configurability introduces a threat model the library-only design avoids, and a dependence on DataFusion’s API stability confined to one crate.
• Ops burden: highest initially, lowest at steady state. Time-to-deliver: longest. Reuse: full.

Advice Received

Document version history

Note (repo import): This document predates the v2.2.0-informed revision pass. The confirmations, departures, and new decisions agreed during design review are tracked in docs/adr/README.md and will be folded in when the full ADR set is written. Treat the phase structure and crate layout as current; treat specific version pins and the single-ADR framing as pending update.