ADR-0058: Record-metadata columns — provenance naming and transport-neutral superset

Reference

Amends ADR-0005 (record metadata as reserved columns) and ADR-0051 (configurable metadata prefix, immutable once a target exists). Interacts with ADR-0004 (locator propagation through unnest), ADR-0044 (raw lane), ADR-0054 (content-hash dedup), and the Phase 5 connector-core retrofit and Phase 8 Pulsar work in the delivery plan.

Context

ADR-0005 established that record metadata rides with the record as reserved columns rather than out-of-band, because a sidecar (e.g. carried on OffsetSpan) is lost at the Arrow Flight boundary — precisely where metadata matters most for downstream Lakehouse writes. ADR-0051 made the column prefix configurable (default _twg_) and immutable once a target exists.

Those decisions left two things unresolved that only surfaced once the column set was written down concretely:

• The column names were Kafka-shaped and flat. The enumerated set (_twg_offset, _twg_leader_epoch, _twg_timestamp_type, …) mixes fields that only Kafka supplies with fields common to every transport, under one flat namespace. On a Pulsar stream, _twg_offset and _twg_leader_epoch are meaningless, and Pulsar’s own fields (message id, producer name, sequence id, event time, ordering key, redelivery count) have nowhere to live. The system is explicitly multi-transport, so a Kafka-only column vocabulary is incoherent.

• There was no distinction between metadata the source provides and metadata we produce, and no processing-time lineage at all. The only timestamp modelled was the source’s own (_twg_timestamp + _twg_timestamp_type). There was no record of when Thalweg read a record or when it wrote it, so a record carried no event-time-to-processing-time provenance — information operators need for latency accounting, and pipelines need for correctness when reasoning about lateness.

Options considered + consequences

Dimensions: transport neutrality, self-describing names, migration cost, collision safety.

Option 1: Keep flat Kafka-shaped names; add Pulsar fields ad hoc

Leave _twg_offset etc. as-is and bolt on Pulsar columns as needed.

• Pros: no rename; smallest diff today.
• Cons: the namespace stays Kafka-centric and a reader cannot tell a broker-supplied column from one Thalweg stamped. Pulsar fields accrete without a home. The incoherence ADR-0005 tried to avoid (metadata that doesn’t describe the actual transport) reappears per-transport.

Option 2: A transport-neutral _twg_source_* superset, with bare _twg_* reserved for Thalweg-stamped values

Rename every broker-supplied column to _twg_source_* and define one superset across transports (a column is null where a transport doesn’t supply it). Reserve bare _twg_* (no source infix) for values Thalweg itself produces, and add _twg_ingest_ts / _twg_emit_ts to complete the provenance clock.

• Pros: self-describing — the source infix is the authority marker. One schema describes Kafka and Pulsar. Event-time → processing-time lineage becomes first-class. Collision rules from ADR-0051 carry over unchanged.
• Cons: renames every metadata column — a larger schema change than the configurable-prefix case ADR-0051 governs; must land before any target exists. The Pulsar rows are net-new design until confirmed against a live broker.

Option 3: Namespace by transport (_twg_kafka_*, _twg_pulsar_*)

• Pros: unambiguous per transport.
• Cons: defeats the point of a neutral column set — a consumer would branch on transport to read topic/partition, and a stream migrated between transports would change schema. Rejected by the same reasoning that set the default prefix to transport-neutral _twg_ in ADR-0051.

Decision

Adopt Option 2.

Source-provided columns are named _twg_source_* and form one transport-neutral superset. A column is null for transports that do not supply it, so a single schema describes Kafka and Pulsar alike. This covers topic, partition, Kafka offset / Pulsar message id, leader epoch, key, source timestamp (+ Kafka timestamp type / Pulsar event time), headers (Kafka headers or Pulsar properties, as List<Struct<key, value>> since a MapArray cannot hold Kafka’s duplicate keys or null values), the Kafka batch fields (producer id / epoch / base sequence) and the Pulsar fields (producer name, sequence id, ordering key, redelivery count), and the source cluster id.

Thalweg-stamped columns keep the bare _twg_* form — the absent source infix is the signal that Thalweg, not the broker, is the authority:

• _twg_ingest_ts — stamped on read, at the Source::poll_batch boundary.
• _twg_emit_ts — stamped on write, when a BatchSink commits the batch.
• _twg_record_id — the deterministic content hash over {source cluster, topic, partition, offset, payload} that anchors dedup and recovery (ADR-0054).

_twg_source_timestamp → _twg_ingest_ts → _twg_emit_ts gives an event-time-to-processing-time provenance chain on every record.

The prefix (_twg_) remains configurable and immutable-once-set per ADR-0051; the reserved-namespace collision rule is unchanged (a payload field under the prefix is a hard decode error). MetadataPolicy continues to govern the decoded lanes only — None / Locator (default) / All; the raw lane’s Arrow form carries the full set by construction. The raw lane’s contract is an Arrow RecordBatch; the physical file schema (Parquet layout, opaque-envelope vs. exploded columns) is a sink decision, out of scope here.

Consequences

• Because this renames every source column, it must land before any target exists. Applying it to an existing target is a refused migration under ADR-0051 — a stronger version of the prefix-change case that ADR governs.
• The Pulsar rows are provisional. Pulsar’s message field model is not yet pinned in the repo (twg-pulsar-client is flow-control-only; twg-pulsar-arrow is a stub). The superset above is confirmed against a live Pulsar broker in Phase 8; column names may be adjusted then, which is cheap while the design is pre-implementation.
• RawRecord / Locator are Kafka-shaped today. Generalising them to carry this superset is part of the Phase 5 connector-core retrofit (delivery-plan risk F4).
• unnest propagation (ADR-0004) is specified for the locator only. How the full metadata superset propagates across a cardinality change is follow-up work once this naming is fixed.
• Status is Draft pending review; the delivery plan (v0.7) already reflects this model so the two stay in step.