Thalweg · API reference ← Main docs

Crate twg_dq_audit

Crate twg_dq_audit 

Source
Expand description

DQ audit-trail record shape.

The shared contract between dq-enforcer (produces verdicts) and the sinks (persist them). Scaffold only.

An audit record is per-record, per-rule (full grain): every record’s verdict on every rule it was evaluated against. To count as an audit trail it carries:

  • record_id — the deterministic record id (column named under the configured metadata prefix), the join key back to raw
  • rule_id — stable rule identifier
  • rule_version — and the contract/rule version, since rules evolve; a verdict against an unversioned rule is worthless once the rule is redefined
  • rule_source — manual | odcs | merged (which authority defined it)
  • verdict — pass | fail
  • disposition — clean | quarantined | annotated | dropped (the outcome actually applied, not just pass/fail). dropped means the record matched a drop rule — a known, intended discard, distinct from quarantined (failed a quality rule) and from a decode failure. Dropped records go nowhere: not to a clean sink, not to the DLQ (that is the point — no DLQ noise from test traffic / heartbeats).
  • provenance — source locator, ingest timestamp, producing pod/pipeline

The trail is append-only by construction and routed to a first-class sink (receives = "dq_results"), so it has the SAME durability and recovery guarantees as the data it describes — it is data, not telemetry. On by default; disableable for non-regulated deployments.

Drop audit is OFF by default: a dropped record leaves no trace, because the versioned drop rule is itself the audit artifact — “we discard all records matching this versioned rule” is provable from contract/config history without a row per record. Per-drop-rule audit = true opts a rule into per-record drop rows where individual proof is wanted; the high-volume noise case (heartbeats) stays silent.

Volume note: full grain is N_records × N_rules rows, typically an order of magnitude more than data rows, so the default target is a compact columnar sink (Delta/Iceberg/Parquet) and the record is narrow (ids + codes, no payload).