Expand description
Object-store sink. Scaffold only.
One sink, three format backends selected by config — so routing, durable-ack,
offset colocation and primary-raw eligibility are implemented ONCE rather than
per format. The format-* crates own the encoding and define what “durable”
means for each; this crate consumes that through the sink contract.
parquet — plain files to object store, no table layer; durable = flush+close iceberg — Iceberg table; durable = commit to the catalog
Primary-raw eligibility: a format backend qualifies as ReplayableSink when it
is both readable back for recovery and able to confirm durable acknowledgement
before returning success. Plain Parquet on object store qualifies (readable +
flush/close durability), so a Parquet raw lane is a valid, cheap primary-raw
target with no Delta/catalog dependency.
§Only one open table format is supported, and it is Iceberg
Delta and Iceberg are converging — Iceberg v3 closes most of the feature gap, and cross-format metadata translation exists — so carrying two table-format implementations buys duplicated maintenance rather than capability. Iceberg is the one to keep: Apache-governed, the widest engine support, and it pins the same Arrow major we do (ADR-0003).
The backend seam here is what makes that reversible. Adding a Delta backend later is additive rather than a rewrite, which is the right posture for a decision resting partly on how convergence plays out.
A Delta-writing service can still hold primary-raw — conditionally. Delta
and Iceberg are both Parquet data files plus a metadata layer, and a Delta
table can be configured to generate Iceberg metadata alongside its own without
rewriting data, so one copy of the files serves both. An Iceberg reader can
then read it, and the catalog exposes it through an Iceberg REST endpoint with
vended credentials — the same seam twg-table-catalog already provides.
This is a precondition to VALIDATE, never to assume. See
twg-sink-zerobus for the conditions and the read-after-write hazard.