How a Sink Should Work

A generic design blueprint for sinks — the destination/write side of a sync — grounded in thalweg’s current plans.

Status. This is a design document, not a description of shipped code. thalweg is pre-implementation: every sink crate is a scaffold (a README.md plus a doc-comment lib.rs — “the documented contract exists, the implementation does not”), and the only crate with real, tested code is twg-type-map. Landing order in the delivery plan: the BatchSink substrate and the first four sinks in Phase 5a, the coverage store + recovery in Phase 5b, Postgres/Zerobus + catalog + Iceberg in Phase 7, and the transform DAG in Phase 9. Where this document leans on a thalweg decision it names the crate/ADR; capabilities that are intent rather than build are marked (planned). The aim is a generic, reusable model of what a correct sink must do, using thalweg’s specification as the worked example.

1. What a sink is

A sink is the component that takes the pipeline’s record stream and writes it to a destination — a table, an object store, a message transport — durably, and reports back precisely what it committed so the source can safely advance. A sink is the last hop on the data path:

… ─▶ transform (DAG) ─▶ SINK(s)
                          │
                          └─ durable ack ─▶ source position advances
                                            per-sink coverage recorded

A sink treats every destination uniformly: a dead-letter / quarantine target is not a special downstream of the sink layer, it is just another sink with its own table and coverage. The sink itself is agnostic of what role a target plays.

A sink is responsible for four things, and this document expands each:

1. Accepting and writing a batch of Arrow RecordBatches with a well-defined lifecycle (§2). The write side is Arrow-only — upstream decode/transform normalise every representation into Arrow before a sink sees it.
2. Confirming durability and returning a precise record of what landed, so delivery is correct and replay-safe (§2, §5, §9).
3. Provisioning and evolving the destination table/schema, including comments/descriptions (§8).
4. Reporting progress, coverage, and errors back to the engine and operators without ever blocking the clean path (§10, §11).

2. The sink contract

thalweg models the write side at two layers, and a concrete sink usually implements against both.

2.1 BatchSink — the Arrow write substrate (planned; twg-stream-arrow, ADR-0001)

A sink consumes Arrow RecordBatches and nothing else — there is a single sink input type, not a family of per-representation sinks. Upstream decode/transform normalise raw bytes and the neutral value model into Arrow before the write side, so the write substrate is one abstraction over RecordBatch rather than a RawSink/ValueSink/ArrowSink/TeeSink split.

#![allow(unused)]
fn main() {
pub trait BatchSink {
    fn push(&mut self, batch: &RecordBatch) -> Result<()>;
    fn finish(&mut self) -> Result<(CommitInfo, OffsetSpan)>;
}
}

• Lifecycle: push-many → finish-once, returning (CommitInfo, OffsetSpan).
• The critical property: OffsetSpan returns identically from every sink, so commit-after-write, replay positioning, and the soak harness are written once and work across every sink.
• _twg_emit_ts is stamped on write, when a BatchSink commits the batch.

2.2 Sink — the transport-neutral delivery trait (planned; twg-connector-core, ADR-0012)

#![allow(unused)]
fn main() {
pub trait Sink {
    fn send(&mut self, batch: Payload) -> BoxFuture<Result<Vec<Receipt>>>;
    fn flush(&mut self) -> BoxFuture<Result<()>>;
}
}

• Visible lifecycle: send → flush. send returns Vec<Receipt> — the per-batch acknowledgements.
• Note there is deliberately no open/commit/close/ack method in the trait. Commit and acknowledgement are modelled as data — the returned Receipt / OffsetSpan plus the per-sink coverage record in the offset store — not as extra trait methods. This keeps the trait minimal and pushes durability semantics into the values a sink returns.

2.3 Durable acknowledgement — defined per sink type

The governing invariant of the whole system is that the source position is committed only after a durable sink acknowledgement. “Durable” is defined per sink type:

2.4 Batch, transaction & materialisation model

• Batch-granular by default: the source advances to the batch high-water offset once the whole batch is durably acked. Per-record acknowledgement is the low-latency opt-in.
• Partial-failure recovery is per-sink and optional. Two strategies: the engine replays the whole batch and dedup on _twg_record_id absorbs the records that already landed; or, where a sink auto-retries internally (transaction or transport-level retry), the sink resolves the partial failure itself and engine-level replay is not needed. Whole-batch replay + dedup is a fallback for sinks that need it, not a mandatory step for every sink.
• Transaction model: where a sink is transactional, the coverage update belongs in the same transaction as the write — that is the exactly-once path. Where it is not, the sink does write-then-record with at-least-once semantics and lets dedup absorb the replay window.
• Materialisation is a property of the transform-DAG node that owns the sink, not a sink method — see §5.

2.5 The ReplayableSink capability (planned)

A sink that can hold the primary-raw role must satisfy a stronger contract: durable = true is mandatory (enforced at config validation) and the data it wrote must be recoverable — but not necessarily readable back through the sink or the sync itself. An external read path counts: Zerobus, for example, is write-only, yet its data is retrievable via Delta / Open Sharing or a Unity Catalog SQL-warehouse query. Together these are the ReplayableSink capability. What matters is that some durable, readable copy exists for recovery — not that the sink exposes a read API. A sink that cannot confirm durability before returning success, or whose writes cannot be read back by any path, is not eligible to anchor replay.

3. Inputs a sink needs

To be defined and run, a sink needs:

• A destination identity — table/catalog identifier, object-store URI, Flight endpoint, or database DSN.
• Credentials, vended not embedded. Sink/table credentials are obtained from the table catalog via credential vending (temporary, scoped credentials that inherit the requesting principal’s privileges) rather than held statically by the sink or its format crate. Transport auth (TLS/mTLS, SASL) is a separate, lower layer.
• The Arrow schema of the incoming RecordBatches, which the sink maps to the target table — an Arrow RecordBatch → sink-table conversion (DDL + type mapping, see §8). Schema resolution (contract / registry / inference) happens upstream on the read/decode side; by the time data reaches a sink it is already Arrow, so the sink’s only schema concern is the batch-to-table mapping.
• A role and durability setting — primary-raw vs secondary/derived, and whether durable is on (mandatory for primary-raw; an explicit, documented opt-out only for non-raw secondaries).
• Config-validation preconditions, checked at startup and re-checked rather than cached where a target property can change under a running pipeline (§6.2).
• A coverage/offset regime — where this sink’s write coverage is persisted (§9).

4. What a sink must support

1. Writing Arrow RecordBatches via its BatchSink write lifecycle (push-many → finish-once).
2. Durable acknowledgement appropriate to its type, and returning a precise OffsetSpan / Receipt.
3. Transactional coverage where the target supports it (exactly-once), or safe write-then-record (at-least-once) otherwise.
4. Idempotent replay where it relies on engine-level replay — tolerating whole-batch replays without duplicating landed data, via _twg_record_id (sinks that auto-retry internally may not need this; see §2.4).
5. Destination provisioning — create + evolve the table/schema and stamp comments (§8).
6. Backpressure participation — releasing admission-control budget on ack, never blocking the clean path (§6, §10).
7. Coverage reporting and error/status surfacing (§9, §10, §11).

5. Sink roles & the transform DAG

Sinks attach to a DAG, and their role determines whether they gate the source.

5.1 Primary-raw vs secondary/derived

• The gate is the sink that owns replay authority.
  • Raw-first topology: the source acks when the primary-raw sink acks durably. Secondary sinks ack independently and asynchronously; they are recovered from raw if they lag, so they never gate the source.
  • Single-sink topology: that sole sink’s durable ack gates the source.
• Durability is role-dependent. Primary-raw MUST be durable. Non-raw secondary sinks MAY disable durable to trade safety for throughput — because raw can rebuild them — but that opt-out is explicit and documented, never a silent default.
• Any replayable sink can hold primary-raw — e.g. Parquet-on-object-store, Postgres, or Zerobus where Iceberg reads are enabled.

5.2 Materialisation

A DAG node is { name, query, optional sink, materialisation }. Materialisation becomes automatic when a node has more than one dependent (DataFusion re-executes a plan per collect(), so an unmaterialised shared intermediate is re-derived per dependent). Sinks attach at interior nodes, not only leaves. Cardinality caveat (ADR-0004): an unnest node breaks the OffsetSpan range assumption, so that node’s span is the union of contributing locators.

5.3 Recovery-from-raw is a precondition, not later hardening

A multi-sink topology without recovery is silently lossy by construction, so configuration refuses one. Recovery is required before a second sink is allowed. Concurrency notes worth stating: concurrent live + recovery writes to a single sink are safe for dedup-on-id sinks; for append-only targets they produce out-of-order arrival with correct identity — acceptable, but it must be stated.

6. Concrete sink types

All five are scaffolds today. Durability follows §2.3; primary-raw eligibility follows §2.5 and §5.

6.1 twg-sink-postgres

Writes via binary COPY; durability is transaction commit; idempotency is the engine-wide _twg_record_id dedup. Because names are spliced into CREATE TABLE and binary COPY, it depends on the identifier-safety helpers in twg-type-map (§8).

6.2 twg-sink-zerobus

Writes Arrow RecordBatch over Flight (no protobuf descriptor is involved in the write path) into Delta; durability is server-side ack. Its primary-raw role is conditional and enforced as config-validation preconditions: the table must be registered in the catalog, Iceberg reads/column mapping must be enabled, reader/writer protocol versions must meet a minimum, and deletion vectors must NOT be enabled (switching them on later silently removes recovery’s read path). The engine must refuse the primary-raw role if any precondition is unmet, and re-check rather than cache — a table property can change under a running pipeline. Two knock-ons: a read-after-write hazard (Iceberg metadata generation is async, so raw-anchored recovery may see an older table state and silently replay an incomplete window — recovery must tolerate the lag or trigger generation synchronously), and a codec knock-on (Iceberg-enabled tables use Zstandard, not Snappy, so the recovery read path needs zstd decode — decode-only, purity-safe).

6.3 twg-sink-object-store

One sink, format backend selected by config, so routing, durable-ack, offset colocation and primary-raw eligibility are implemented once rather than per format. Durability per backend: parquet = plain files to object store, no table layer, durable on flush+close; iceberg = Iceberg table, durable on commit to the catalog. A format backend qualifies as ReplayableSink when it is both readable back for recovery and able to confirm durable ack before returning success — plain Parquet qualifies, making a Parquet raw lane a valid, cheap primary-raw target with no Delta/catalog dependency. Only one open table format (Iceberg) is in scope (ADR-0023); the backend seam keeps a future Delta backend additive.

6.4 twg-sink-flight and twg-sink-arrow-ipc

Header-only scaffolds. Durability from the core invariant: Flight = server-side ack; IPC file = flush+close. Both land in Phase 5a.

7. Scalability (sink side)

A sink participates in the engine’s flow control rather than owning its own:

• Ack-driven admission control. The engine’s byte-budget semaphore (sized in bytes, from the p95 post-decode expansion ratio) releases permits when a sink’s ack arrives. A slow sink therefore throttles intake naturally, without a separate mechanism, and without blocking the clean path (the permit is acquired and released in the same select! as ack processing) — see §10.
• Batch sizing follows backpressure. The backpressure state machine scales effective batch size as memory/channel/write-failure pressure rises; a sink should honour the batch size handed to it rather than buffering unboundedly.
• Secondary sinks scale independently. Because they ack async and recover from raw, a lagging derived sink never stalls the source; it can be scaled or rebuilt on its own cadence.
• Coverage partitions horizontally. Per-sink coverage is keyed by topic+partition, so multiple pods each own a disjoint slice of a sink’s write responsibility (§9).
• Streaming transports keep a small, fixed pool of long-lived streams per destination. Never open a stream per batch or per worker: on the predecessor that tripped a broker “too many concurrent streams” limit (thousands of rejects/day) and starved throughput by churning reconnections instead of sending. On a concurrency reject, honour the server’s suggested backoff and send on an existing stream rather than reopening immediately — a self-protective path must not amplify the condition it detects. Size the pool to bandwidth, not record count: a stream planned at ~X MB/s cannot carry records/sec × record_size above X, so large records saturate one stream at a low record rate — spread them across a few streams.

8. Creating & updating destination tables, schemas & comments

A sink must be able to provision and evolve its target, not just write into a pre-made table. This is a split responsibility between the table catalog and the type-mapping vocabulary.

8.1 Type mapping & identifier safety — the built foundation (implemented: twg-type-map)

Mapping to each target dialect is a hub-and-spoke through Arrow, and the goal is to convert straight from the Arrow RecordBatch to the target types wherever possible: each column’s Arrow type is mapped directly to its target type — SQL DDL text for Postgres, and Delta / Iceberg / Zerobus-proto types for the table formats. A decoded-JSON representation remains available as an intermediary only for the paths that genuinely need it (e.g. JSONB fallback); it is never a mandatory hop on the Arrow → target conversion. Per-dialect names are emitted correctly (e.g. Iceberg long, decimal(10,2), fixed[16]; Delta nested-JSON; proto BIGINT vs UC-API LONG). Identifier safety (strict vs hyphen-relaxed validation, quoting/escaping) is explicit so names are safe to splice into CREATE TABLE and binary COPY. This is the one piece that already exists and is tested.

8.2 Table registration & the describe-vs-execute split (planned: twg-table-catalog)

The TableCatalog trait owns table schema resolution, sink registration, credential vending, and descriptive metadata. Descriptive metadata is stamped at table CREATE and at schema EVOLUTION — a newly added column gets its comment too, because evolution is a write-path event just like creation — applied via a describe-vs-execute split (compute the desired state, then apply it).

8.3 Schema evolution

Policy: additive is safe, widening is conditionally safe behind an opt-in, anything else fails loudly. The widening vocabulary is already implemented (twg-type-map::can_widen_to; e.g. lossless int32 → int64 → double → string, int → f64 refused as lossy). The missing piece a full implementation still needs is the component that diffs a live table against the desired schema and issues the ALTERs.

8.4 Comments / descriptions

• Two comment authorities. Engine-owned columns (_twg_*, dq_results, offset-store columns) get fixed built-in descriptions; payload column descriptions flow from the ODCS contract — no contract means no payload comment.
• Table-level comment plus properties/tags (owner, contract URI + version, source topic) turn the catalog into a discovery surface.
• Backend degrade-gracefully matrix: Unity Catalog → COMMENT ON + tags + properties; Iceberg → table properties + column docs in the schema; Glue → column Comment fields + table Parameters.
• Idempotent re-sync. The sink marks its comments twg:managed=true and stores a twg:comment_hash; a re-sync writes only when the contract’s description changed versus the last synced hash — no churn on no-op deploys. Human edits to managed columns are detected via the managed flag; respect-vs- restore is a policy, not a blind stomp.

8.5 Credential vending

The catalog exposes a read-only Iceberg REST endpoint with credential vending — temporary scoped credentials inheriting the requesting principal’s privileges. Recovery reads resolve the table, obtain vended credentials, and read Parquet through the Iceberg reader (no Delta implementation required). Two metadata properties the sink should surface: whether Iceberg reads are enabled, and how far generated metadata trails (§6.2).

8.6 Emit-time type reconciliation — the write boundary rejects shape mismatches

Schema resolution is upstream (§3), but two type mismatches only bite at the write boundary, where the incoming Arrow RecordBatch meets the target column type — and on the predecessor each one took out a whole silver lane for the life of a deploy, because it fails at emit, after config validation passed:

• A nested Arrow type into a STRING column. A decoded Struct/List/Map landing in a column the target declares STRING (Arrow LargeUtf8) is rejected at write (“expected LargeUtf8 but got Struct with N fields”), for every record. The sink must reconcile, not reject: either the author serialises upstream with to_json(<any>) (decode-codec-learnings.md §4), or — the clean end-state — the sink auto-projects at emit, because it already knows the declared target type from twg-type-map. Auto-projection is a hot-path change (retry-slice, byte-budget — JSON runs 3–5× larger than the packed nested value), so it is gated behind a trigger; until then to_json is the first-class escape hatch, and a STRUCT-into-STRING mismatch is a loud failure (§11), never a silent NULL.
• A timezone-aware value into a timezone-naive column. A Timestamp(µs, Some("UTC")) into a TIMESTAMP_NTZ (Timestamp(µs, None)) column is rejected the same way. The fix is upstream — tz-naive helpers (decode-codec-learnings.md §5) — but the sink must surface the mismatch loudly, naming the offending column, not fail opaquely per batch.

The principle: the sink knows the target type, so any mismatch it could reconcile or name precisely must never surface as an opaque per-batch rejection.

9. Traceability (sink side)

• _twg_emit_ts is a sink event — stamped when a BatchSink commits — and closes the provenance chain _twg_source_timestamp → _twg_ingest_ts → _twg_emit_ts, giving end-to-end latency per record.
• Per-sink write coverage is the authoritative record of what each destination holds, kept in twg-offset-store as gap-aware ranges keyed (sink, topic, partition, start) → end, prefix-scanned, and merged by union, never last-write-wins (so concurrent writers during a rebalance cannot erase each other’s progress). Note this is a different quantity from the source position: once the source is acked, coverage is the only record of what a sink is still missing. The store is two-tiered — a durable/shared tier colocated with raw (never with the sink it describes) plus a pod-local write-behind cache checkpointed on an interval.
• Governing invariant: coverage may LAG, never LEAD. A lagging sink coverage costs a dedup-absorbed replay (safe); a leading one is silent permanent loss.
• The DQ audit is itself a first-class sink (receives = "dq_results"), append-only, with the same durability/recovery guarantees as data — so which records a sink accepted/quarantined/dropped, and under which versioned rule, is queryable per record.

10. Non-blocking ack back channel

Acknowledgements drive flow control and the source-position commit, but a caller is never blocked waiting on an individual ack:

• Async delivery ack. Sink::send returns Vec<Receipt> as a future; the byte-budget permit for that batch is released when the ack arrives, processed in the same select! loop as record flow. The sink keeps moving while acks stream back asynchronously.
• Secondary sinks never stall the source. They ack independently and async and are recovered from raw if they lag (§5), so a slow destination cannot gate intake.
• Ingest is fire-and-forget; the durable wait is a background lane. For a pipelined transport (Flight/Zerobus SDK), send returns as soon as the SDK accepts the batch; the “wait for durable server ack” runs on a per-stream background ack lane that drains a bounded channel, coalesces a burst, and confirms the whole burst with one round-trip on the largest offset (offset monotonicity ⇒ one wait confirms all lesser items). The at-least-once contract (ADR-0038) is preserved by where the confirmation fires, not whether — the source-position commit and per-sink coverage update still happen only on the durable ack, now from the ack lane. Backpressure is the channel filling: when full, send awaits, which the SDK’s own bounded buffer already models.
• Advance on the delivered offset, never the submitted offset. Submitted is an observable ingest high-water only; recovery (§9, ADR-0053) reads delivered. Any durability path that reads submitted is a bug.
• The dead-letter / quarantine target is itself just another sink, wired on a non-blocking lane — its own buffer and backpressure, genuinely independent of the clean write path. When it cannot keep up, quarantine sheds — never clean (overflow → drop-with-count loud metric, safe because raw is the system of record). Quarantine never gates: it does not sit between source and ack.

Anti-pattern to design out (prior-art). Making the async ack synchronous — awaiting the durable-offset wait inline after every submit — serialises the sink to one in-flight batch. On the predecessor engine this cost a ~20,000× throughput loss (≈4 batches/sec against a 100 MB/s pipe) and manifested as everything else: server-ack-timeout false-positives (each network hiccup fully exposed to the caller deadline, “fixed” by inflating the timeout — a bandaid that trades false positives for slow real-failure detection), backpressure trips at trivial throughput, and batchers that never fill a batch because they spend their time waiting between batches. The pipelined ack lane above absorbs transient hiccups in the SDK’s buffer instead of exposing them. This refinement is recorded as ADR-0060 (pipelined fire-and-forget sink ack); see also decode-codec-learnings.md §8.

11. Monitoring & surfacing errors/messages from a sink

The status/stats channel is read-side and never on the commit path, so observing a sink can never slow it down.

• Health & stats. /health exposes liveness/readiness including backpressure state; /stats exposes throughput, per-sink ack latency, coverage gaps, and quarantine/drop counts. (planned: twg-observability.)
• OpenTelemetry (OTLP) metrics/logs/traces, with per-batch spans through the write → ack path and latency derived from the _twg_ timestamp chain.
• Errors surface on three planes, by severity:
  1. Per-record — the unified DLQ tags every diverted record with a reason code (decode / contract / DQ / retry-exhausted), and the DQ audit records the disposition; deliberately-dropped junk never appears in the DLQ, keeping it high-signal.
  2. Aggregate — /stats counters and OTLP metrics (drop counts, quarantine overflow, retry exhaustion, coverage-gap alarms, per-sink ack latency).
  3. Liveness — /health plus backpressure state.
• Loud, not silent, on the dangerous cases. A sink must emit explicit, counted signals for: overflow drops, coverage gaps, unsafe schema-evolution attempts, unmet primary-raw preconditions, read-after-write recovery hazards, write-boundary type mismatches (nested-into-STRING, tz-aware-into-NTZ; §8.6), and type-lossy decode fallbacks (over-depth collapse; decode-codec-learnings §3) — never degrade quietly. Silent data loss is the one outcome the whole design exists to prevent.
• Discriminate self-inflicted from environmental on the ack lane. Two causes drive the same downstream symptom (delayed/absent durable acks, quarantine-drain timeouts): (a) the transport is transiently slow to ack — environmental, transient, recovered by a source/stream reconnect; and (b) our own ack lane died on an unrecoverable transport error — self-inflicted, permanent until the stream is recreated. Alert on the cause signal (a distinct ack-lane poisoned counter), never on the shared symptom, or minutes of sink poisoning get misread as broker slowness. Keep the two counters semantically separable at source; the runbook branching is an operations concern.

12. Sink design invariants (summary checklist)

A sink built to this blueprint should satisfy, at all times:

• Returns a precise OffsetSpan / Receipt; source position advances only after this sink’s durable ack (definition per §2.3).
• Coverage update is in the same transaction as the write for transactional sinks (exactly-once), else safe write-then-record (at-least-once) with _twg_record_id dedup absorbing replays.
• Primary-raw role requires the full ReplayableSink capability (durable and recoverable via some read path — the sink itself, Delta/Open Sharing, or a SQL warehouse), enforced at config validation and re-checked, not cached.
• Non-raw secondary sinks may disable durable only as an explicit, documented opt-out; they ack async and recover from raw, never gating the source.
• Per-sink coverage is gap-aware and union-merged; coverage may lag, never lead.
• Creates and evolves the destination table/schema — additive by default, widening behind opt-in, else fail loud — with identifier-safe DDL from twg-type-map.
• Comments are stamped at CREATE and EVOLUTION, from the correct authority (engine _twg_* vs ODCS payload), and re-synced idempotently (twg:managed / twg:comment_hash).
• Uses vended, scoped credentials from the catalog, not embedded secrets.
• Ack is asynchronous and releases admission budget; the DLQ/quarantine lane never backpressures the clean write path — quarantine sheds, never clean.
• The async ack is never made synchronous — ingest is fire-and-forget, the durable wait runs on a per-stream background lane, and the source advances on the delivered offset, never the submitted one. (§10)
• Streaming transports use a few long-lived streams per destination, honour server backoff on a concurrency reject, and are sized to bandwidth, not record count. (§7)
• Write-boundary type mismatches (nested-into-STRING, tz-aware-into-NTZ) are reconciled (to_json / auto-project) or surfaced loudly with the column named — never an opaque per-batch rejection or a silent NULL. (§8.6)
• Errors surface on three planes (per-record DLQ + audit, aggregate stats/OTLP, liveness health), and dangerous cases are loud; ack-lane poison is discriminated from environmental slow-ack at source. (§11)

Source grounding

Derived from thalweg’s current design artefacts — chiefly docs/blueprints/thalweg-delivery-plan.md, AGENTS.md, README.md, and the doc-comment scaffolds of twg-connector-core, twg-stream-arrow, twg-offset-store, twg-table-catalog, and the sink crates (twg-sink-postgres, twg-sink-zerobus, twg-sink-object-store, twg-sink-flight, twg-sink-arrow-ipc), plus the implemented twg-type-map. Relevant ADRs include 0001 (BatchSink/OffsetSpan), 0004 (unnest/span), 0007 (DAG), 0012 (Sink), 0022 (schema evolution), 0023 (single table format), 0024/0037 (catalog), 0038 (source-ack invariant), and 0057 (recovery-from-raw). As those crates move from scaffold to implementation, retire the (planned) markers and reconcile this document against the shipped behaviour.

The write-side hazards folded into §7 (stream pool / churn / bandwidth sizing), §8.6 (emit-time type reconciliation), §10 (pipelined fire-and-forget ack + the per-batch-wait anti-pattern), and §11 (loud type-mismatch cases + ack-lane-poison discrimination) are drawn from production incidents on the predecessor engine thalweg replaces, abstracted per the repository’s no-sensitive-information rule. Their decode-side counterparts live in decode-codec-learnings.md; the pipelined-ack refinement (§10) is recorded as ADR-0060.