E2E Scenarios & Regression Catalogue
The authoritative list behind twg-e2e. Scenarios prove the system does
what it should; regressions prove it never again does what it once did wrong.
Scenarios
| Scenario | Proves |
|---|---|
kafka_to_postgres_happy_path | Baseline stream → transform → sink with binary COPY and offset commit. |
pulsar_to_zerobus_with_raw_primary_and_recovery | Primary-raw role, source ack on raw write, bounded windowed recovery with record-ID anti-join. |
batch_file_to_postgres_shares_streaming_pipeline | source-batch is a bounded Source feeding the identical transform/DQ/sink path — no parallel implementation. |
flight_source_to_flight_sink_roundtrip_preserves_locator | Flight as a first-class transport carries the locator through app_metadata, so recovery and primary-raw work on already-Arrow data. |
odcs_contract_violation_routes_to_dlq_with_reason_code | Contract loaded (local/HTTP/object-store), schema conformance enforced, violation lands in the unified DLQ tagged contract. |
merged_quality_rules_most_restrictive_wins | Manual + ODCS quality rules on one field resolve to the tighter threshold. |
multi_sink_fanout_dag_with_unnest_offset_correlation | DAG with interior sinks and an unnest node; derived rows carry originating locator; commit-after-write never acks unlanded descendants. |
primary_raw_ack_ordering_survives_secondary_sink_failure | A failing secondary sink with isolate does not block the source, because raw is present; offset store colocated with primary raw. |
backpressure_stuck_state_emits_scale_signal | The flow-control state machine reaches Stuck under load and emits the pressure signal rather than OOMing. |
delta_sharing_recipient_reads_cross_org_table | Delta Sharing source resolves pre-signed URLs and reads a shared table without copying. |
delta_sink_primary_raw_requires_iceberg_reads | Config validation grants primary-raw to a Delta-writing sink only when the table has Iceberg reads and column mapping enabled and deletion vectors disabled; refuses otherwise. |
deletion_vectors_enabled_later_is_detected | Enabling deletion vectors on a primary-raw table after startup is detected rather than silently removing recovery’s read path. |
recovery_does_not_read_stale_iceberg_metadata | A recovery pass either waits for or triggers Iceberg metadata generation, so it never replays an incomplete window from a trailing snapshot. |
iceberg_write_then_read_roundtrip_via_catalog | Iceberg write commits data files + catalog metadata; read resolves the current snapshot. |
iceberg_source_feeds_raw_anchored_recovery | Recovery reads the raw table directly through the Iceberg or Parquet reader, not catalog SQL. |
avro_ocf_batch_shares_streaming_decode | An .avro OCF file and a registry-framed stream decode through the same codec-avro logic. |
wasm_custom_decoder_handles_bespoke_binary | A user WASM decoder turns a proprietary binary format into Arrow, sandboxed and fuel-limited, via the batch-in/RecordBatch-out contract. |
arrow_ipc_stream_source_preserves_locator | A non-Flight Arrow IPC stream source carries the locator so it joins recovery. |
catalog_vends_credentials_across_backends | The same Delta table is read via UC-vended and S3/IAM-vended credentials through one table-catalog seam. |
degraded_secondary_sink_stays_ready_shows_in_health | A degraded secondary sink appears degraded in /health but /ready stays 200 — display-only, not work-gating. |
primary_raw_unwritable_drops_readiness_not_liveness | When the primary-raw sink cannot write, /ready returns 503 while /live stays 200 — no restart loop. |
backpressure_state_surfaces_per_stage_in_health | The four-state machine at Warning/Stuck projects to degraded/down on the affected stage in the /health tree. |
dq_audit_trail_full_grain_per_record_per_rule | Every record’s verdict on every rule persists to the dq_results sink, joinable to raw by record ID, stamped with rule id + version. |
dq_audit_survives_restart_same_durability_as_data | The audit trail recovers with the data on forced restart — it is a durable sink, not best-effort telemetry. |
dq_rule_version_stamped_survives_contract_evolution | A verdict records which rule version produced it, so it stays meaningful after the rule is redefined. |
drop_rule_discards_silently_no_dlq_no_audit_by_default | A record matching a drop rule reaches neither clean sink nor DLQ, and leaves no audit row unless the rule opts in. |
created_table_carries_metadata_and_contract_comments | A Thalweg-created table has built-in comments on _twg_* columns and contract descriptions on payload columns, plus table properties (contract URI+version, source topic). |
evolved_column_gets_its_comment_stamped | Adding a column via schema evolution stamps its comment too, not just create-time columns. |
resync_is_noop_when_nothing_changed | Re-running against an unchanged contract writes no catalog changes — no churn, no rate-limit pressure, clean audit log. |
source_advances_only_after_primary_raw_durable_ack | The source offset does not move until the primary-raw sink confirms durability; a slow secondary sink never gates it. |
crash_between_send_and_durable_replays_not_loses | A crash after send but before durable ack replays the batch; dedup on record ID absorbs already-landed records. |
non_raw_sink_may_run_non_durable_recovered_from_raw | A secondary sink with durable disabled runs faster and is rebuilt from raw on lag/failure, without weakening the source-ack invariant. |
stats_endpoint_shows_per_stage_memory_last_30s | /stats returns a per-stage throughput and per-record/per-batch memory digest over a 30s window, from stage-owned quantities. |
admission_uses_measured_p95_not_static_estimate | Once the window has data, admission budgets from the measured p95 record size; the static config is only the cold-start seed and floor. |
size_spike_does_not_blow_admission_budget | A burst of large records after small ones is absorbed because admission budgets on the window p95, not the lagging mean. |
store_managed_source_position_never_leads_durability | For a source with no broker-side position, the persisted cursor is committed only after the raw write is durable; a crash between resumes with duplicates, never a gap. |
non_replayable_source_refused_primary_raw | A source whose origin cannot be re-read is refused the primary-raw role unless its producer supports resumption, rather than presenting a topology that only looks recoverable. |
coverage_never_over_claims_an_unwritten_batch | Coverage is recorded only after the sink’s durable ack; a crash between write and record leaves the batch marked missing, never marked written. |
pod_loss_replays_at_most_one_checkpoint_interval | Losing an ingest pod discards its local cache and falls back to the last checkpoint; the overlap is absorbed by record-id dedup, and no batch is left unrecorded. |
mid_sequence_batch_failure_is_recoverable | A sink writing batches 1,2,4,5 while 3 fails leaves batch 3 identifiable as a gap — a high-water mark of 5 would lose it silently. |
quorum_member_lowers_its_throughput_budget | A worker additionally holding quorum runs with a reduced admission budget and smaller batches, so consensus fsyncs do not contend with ingest. |
scaling_down_one_worker_does_not_reconfigure_quorum | Three workers scaling to two leaves a three-member cluster with one unreachable; quorum holds and no membership change occurs. |
terminating_quorum_member_hands_off_first | A quorum member being terminated relinquishes membership before exit; the cluster treats it as departed rather than waiting out a failure detector. |
losing_two_quorum_members_at_once_is_prevented | Concurrent termination of two members is refused or serialised, since the survivor cannot reconfigure alone. |
quorum_role_assignment_is_stable_across_restarts | Role assignment settles on the same nodes rather than shuffling, so membership churn is bounded by real topology change. |
two_node_coordination_tier_refused_at_config | A coordination tier sized at two is rejected: quorum of two means both must be up, strictly worse availability than one. |
single_node_tier_serves_all_coordination_features | With one node, leases, counters and the dedup seen-set all function; only fault tolerance is absent. |
coordination_tier_is_not_autoscaled | Worker autoscaling does not change coordination membership; the tier holds its configured size under load. |
losing_the_tier_costs_continuity_not_data | Losing coordination state expires leases, empties the dedup window and loses buffered coverage — none of which is the sole record of anything. |
coordination_works_without_an_orchestrator | Leases and leader election function on bare processes with no orchestration platform present. |
graceful_shutdown_flushes_coordination_state | Stopping the last member persists state durably first, and a cold start restores it rather than beginning empty. |
dedup_costs_one_bulk_write_per_batch | A batch of N records produces one bulk lookup and one bulk write, not N of each; consensus traffic scales with batches, not records. |
within_batch_duplicates_resolved_locally | Two records in the same batch sharing a content hash are deduplicated before the bulk write, without a second round trip. |
concurrent_batches_miss_dedup_never_drop | Two pods processing the same content hash concurrently both pass the record through — a missed dedup, never a dropped legitimate record. |
dedup_window_bounded_by_configured_memory | The seen-set honours its memory bound by evicting oldest first; the window shrinks under pressure rather than the process growing without limit. |
dedup_membership_is_exact_never_probabilistic | The seen-set never reports a record as seen when it was not; eviction may cause a missed dedup, but a false positive would drop a legitimate record. |
content_dedup_refused_without_a_uniqueness_declaration | A stream whose contract declares no natural key or uniqueness guarantee cannot enable content dedup; validation refuses rather than warns. |
identical_payloads_survive_when_dedup_is_off | Two distinct events with byte-identical bodies both land when content dedup is disabled — the default. |
content_dedup_unavailable_degrades_not_stops | With the shared seen-set unreachable the pipeline continues without dedup; duplicate volume rises, nothing halts and nothing is lost. |
sinks_stay_idempotent_regardless_of_content_dedup | Record-id idempotency at the sink is unaffected by content dedup being on or off, so nothing downstream depends on a best-effort filter. |
pipeline_runs_with_a_single_source_and_sink | A minimal deployment wiring one source and one sink starts and runs with no other component present. |
coverage_store_backing_follows_raw | Coverage lands in the store kind matching where raw lands — relational, object-with-CAS, append-only log, or table — with no technology assumed present. |
object_coverage_cas_conflict_retries_and_converges | Two pods checkpointing the same coverage object contend on the precondition, re-read, re-merge by union and converge without losing either writer’s ranges. |
coverage_survives_the_sink_it_describes_being_down | With a secondary sink unavailable, the gaps it is accumulating are still recorded — the store lives with raw, not inside the failed destination. |
coverage_store_refuses_colocation_with_a_non_raw_sink | Config validation rejects placing the coverage store in a sink whose own failure it must survive. |
rebalance_owner_reads_durable_not_previous_local | A partition moving to a new owner resumes from the durable checkpoint; the previous owner’s uncheckpointed coverage is replayed, not read across. |
concurrent_coverage_writers_merge_by_union | Two pods briefly owning a partition during rebalance produce the union of their ranges; last-write-wins would discard one writer’s coverage and create an over-claim. |
coverage_scan_cost_scales_with_gaps_not_updates | Reading a destination’s coverage costs a prefix scan proportional to the number of gaps, not to how many batches have been written. |
range_compaction_never_closes_an_unverified_gap | Compacting adjacent covered ranges must not merge across a gap that was never written. |
recovery_agent_reads_durable_not_cached_coverage | A recovery agent plans windows from committed coverage, never from an ingest pod’s uncheckpointed local view. |
second_sink_refused_without_recovery | Config validation refuses a multi-sink topology where coverage and recovery are unavailable, rather than running one that is silently lossy. |
single_agent_recovery_needs_no_coordination | Recovery with one worker completes without leases, elections or any coordination substrate present. |
recovery_mode_does_not_consume_the_source | A pod in recovery mode replays from raw and never touches the source, so a backfill cannot throttle live ingest. |
two_recovery_agents_do_not_replay_the_same_window | Window leases in the offset store keep concurrent agents on disjoint work. |
expired_lease_is_reclaimed_without_corruption | An agent dying mid-window leaves work reclaimable; the replay overlap is absorbed by record-ID dedup. |
recovery_scales_to_zero_when_caught_up | With no destination behind, the backlog metric reports zero and recovery capacity costs nothing. |
staged_subscription_avoids_boot_storm | Subscribing to a many-partition topic against a backlog rises in observable steps and never spikes past the memory limit at t=0. |
shedding_is_topic_spread_not_tail_drop | Under memory pressure every topic degrades proportionally; no topic is reduced to zero coverage while others stay full. |
shed_order_differs_per_pod | Two pods under simultaneous pressure shed from different topics first, so damage is not concentrated. |
at_floor_with_unserved_partitions_emits_pressure | A pod at its shed floor that still has partitions it would serve emits the scale-out signal — shedding without it just moves pressure around the ring. |
high_memory_empty_channels_does_not_throttle | The working-set heuristic suppresses backpressure when memory is high but channels are empty: fragmentation, not live data. |
recovery_state_prevents_throughput_sawtooth | Clearing conditions enters Recovery at reduced batch size rather than returning straight to Normal; throughput does not oscillate. |
oversized_record_dlq_judged_on_grantable_not_total | During warm-up a record too large for current capacity goes to the DLQ instead of stalling everything behind it. |
budget_acquire_cannot_deadlock_against_acks | With the byte budget full, ack processing still runs and releases permits. |
quarantine_overflow_never_blocks_clean_path | A saturated quarantine lane sheds (drop-with-count) rather than backpressuring clean; clean throughput is unaffected. |
recovery_rehydrates_dropped_quarantine_rows_from_raw | The default recovery pass backfills quarantine rows that were dropped under live overflow, from raw. |
recovery_skip_dlq_mode_leaves_quarantine_incomplete_but_recovers_clean | Skip-DLQ recovery replays clean data and does not re-quarantine; the flag surfaces that quarantine is non-authoritative for that window. |
Regressions (learned the hard way)
| Regression | The failure it locks out |
|---|---|
avro_decimal_missing_scale_preserved_not_corrupted | AVRO decimals with absent precision/scale must be preserved (as string), not silently corrupted. |
avro_decimal_negative_scale_no_wraparound | Negative effective-scale must not wrap around to a huge positive. |
headers_are_bytes_not_utf8_no_lossy_normalisation | Kafka and Pulsar headers are bytes; they must not be lossily coerced to UTF-8/JSON. Ordered, dup keys and null values retained. |
kafka_vs_pulsar_offset_semantics_kept_distinct_in_cursor | Kafka offset (next) and Pulsar (last-consumed) semantics stay distinct in the typed cursor, not flattened. |
pulsar_messageid_not_flattened_to_bigint | Pulsar ledger:entry:partition message IDs must not be forced into a numeric column. |
zerobus_midstream_schema_change_rejected_by_contract_fake | Mid-stream schema change is rejected at the boundary, as the real service requires. |
arrow_version_boundary_never_mixes_without_bridge | Two Arrow versions never meet directly; the single re-export chokepoint holds. |
datafusion_sessioncontext_reused_not_recreated_per_batch | SessionContext is reused; recreation (~100ms) never appears on the per-batch path. |
deterministic_record_id_includes_source_cluster_id | Record ID hashes in source_cluster_id, so identical topic/partition/offset across clusters do not collide. |
unbounded_memory_growth_triggers_admission_control | Admission control gates before RSS runs away, rather than relying on the OOM killer. |
excel_date_serial_mapped_correctly_not_as_float | calamine’s date-serial floats map to Arrow dates via type-map, not raw floats. |
xml_repeated_elements_become_list_not_last_wins | Repeated XML elements map to an Arrow list, not silently collapsed to the last value. |
batch_csv_uses_contract_schema_when_present | A supplied contract schema overrides inference on CSV/JSON/XML batch reads. |
protobuf_imported_wrapper_types_resolve | A schema importing google.protobuf.StringValue / Timestamp decodes. Fails against any decoder whose registry walks only the root’s nested types. |
protobuf_sibling_message_reference_resolves | A field referencing another top-level message resolves, whether declared in the same file or an imported one. |
protobuf_circular_types_terminate | Struct → Value → ListValue → Value resolves and decodes without non-termination. |
protobuf_map_entries_and_deep_nesting_resolve | Synthetic map-entry types and 3+ level nesting resolve, not just top-level fields. |
protobuf_missing_imports_fails_at_load_not_decode | A descriptor set compiled without imports fails at schema load, naming both the unresolved type and its referrer — never a partial registry that fails on first matching payload. |
protobuf_repeated_message_is_list_of_struct | A repeated message field decodes to a list of structs, not a stringified subtree. |
protobuf_schema_is_data_independent | Empty and populated payloads of the same schema produce an identical Arrow schema. |
dropped_records_never_appear_in_dlq | Drop-rule matches must not reach the DLQ — test traffic and heartbeats stay out, so the DLQ remains a signal for real failures. |
configured_metadata_prefix_is_used_everywhere | With a non-default prefix set, metadata columns, DQ audit joins, catalog comments and the collision check all use it — no hardcoded _twg_ survives. |
changing_prefix_on_existing_target_is_refused | A prefix differing from the one recorded in the target’s metadata fails validation rather than silently writing a parallel set of metadata columns. |
resync_does_not_stomp_human_edited_comments | Re-sync overwrites only Thalweg-managed comments (flagged), never a human-authored edit on a managed column without policy. |
primary_raw_with_durable_disabled_rejected_at_config | Config validation refuses to assign primary-raw to a sink that cannot confirm durability, or with durable turned off. |
quarantine_never_gates_source_ack | Quarantine is never on the ack path — only primary-raw gates the source — so quarantine shedding cannot affect source-ack correctness. |
parquet_object_store_sink_is_primary_raw_eligible | A plain Parquet-files raw lane on object store confirms flush+close durability and is readable back, so it can hold the primary-raw role with no Delta/catalog dependency. |
arrow_c_data_interface_zero_copy_no_serialisation | A foreign Arrow producer hands batches in-process with no copy and no IPC serialisation; the handoff preserves the locator. |
unsafe_confined_to_ffi_crate_only | The unsafe_code lint relaxation appears in twg-ffi and nowhere else; the purity gate fails if any other crate relaxes it. |
wasm_decode_crosses_boundary_once_per_batch | The custom WASM decoder is invoked per batch, not per record — framing is native, so boundary-crossing cost is amortised like a transform UDF. |
Each row above is a comment in twg-e2e/tests/ until it is an executable,
passing test. Converting them is tracked work, not optional polish.