The fastest
line down the channel.

01How it works

A record enters from a source — a broker, a file, an Arrow stream. It is decoded once into a shared columnar form, checked against its data contract, transformed by SQL if you want it to be, and written to one or more sinks. Along the way every record carries its origin, so if a destination falls behind it can be rebuilt from the durable copy rather than re-read from the source.

Streaming and batch are the same pipeline. A file source is simply a bounded one — it ends, where a broker does not — so batch ingestion runs through the identical decode, contract, transform, sink and dead-letter path. Not a parallel implementation that drifts, and not a separate tool: one binary, with the mode as a subcommand.

Every box on the channel is a stage you can observe independently — its throughput, its memory, and whether it's healthy or building backpressure — at /health and /stats.

What travels with every record. Origin is not a side-channel — it rides with the record as reserved columns under a _twg_ prefix, so it survives every hop, including the Arrow Flight boundary where an out-of-band sidecar would be lost. Two families, and the name tells you which. _twg_source_* is what the broker handed us — topic, partition, offset (or a Pulsar message id), key, timestamp, headers and the rest — as one transport-neutral set, with a column left null where a given transport doesn't supply it. Bare _twg_* is what Thalweg itself stamps: _twg_ingest_ts when we read the record, _twg_emit_ts when we write it, and _twg_record_id, a deterministic content hash that anchors dedup and recovery.

02What's supported

Thalweg is early-stage: the architecture is decided and the workspace is scaffolded, with the first capability — user-defined functions — now built and the rest following. The table below is honest about that. It shows where each capability stands today, what is built, and what is planned but not yet started — for example RabbitMQ, NATS, and other transports are planned, not built.

Sources

Payload formats

Schema tooling

Sinks & table formats

Platform capabilities

03Design principles

These ideas shape every decision in Thalweg. They are worth reading before the architecture, because the rest follows from them.

04The layers

Crates are grouped into layers with a strictly downward dependency direction, enforced in CI. Nothing in the codec layer may depend on a transport; nothing in the codec or SASL layer may depend on an async runtime.

Reading bottom to top: raw bytes come in through transport primitives, up through sources into the shared connector contract, get decoded and validated, transformed, and written out. The codec-core crate holds the single Arrow re-export — the one place the Arrow version is named, so the whole workspace pins one version.

05Crate reference

Filter by layer, or read straight through. Every crate is currently a scaffold — the descriptions are the contract each will fulfil.

06Contracts & data quality

A data contract describes what a stream is supposed to contain. Thalweg reads ODCS — the Open Data Contract Standard — and uses it to decide both the shape of the data and what counts as acceptable.

Contracts are optional. Without one, schemas are inferred or taken from a schema registry and quality rules come from configuration. With one, the contract is authoritative on schema, and its quality expectations merge with any configured manually.

07Recovery & guarantees

The correctness backbone is one rule, and it applies to every kind of position the system holds: position may lag, but must never lead. A source position ahead of what was durably written resumes past records that never landed; a sink marked as written when it wasn't is never replayed. Both are silent loss. Behind, in either case, is a redundant replay that deduplication absorbs.

So the source is acknowledged only once the primary-raw sink confirms durability, and coverage is recorded only after the sink it describes has acknowledged. Never before, never concurrently.

Recovery is not optional past one sink

Once the source is acknowledged, the broker moves on and can no longer tell you what a downstream destination missed. From that moment the coverage store is the only record of it. A multi-sink topology without recovery isn't less resilient — it's silently lossy by construction, so configuration refuses one.

08Scaling & coordination

Workers scale freely on throughput. What they need to agree on is deliberately tiny — which recovery window each agent has claimed, which node drives recovery — so the coordination layer is sized for that rather than for the data path.

Quorum is a role, not a tier

Rather than a separate coordination deployment, a subset of workers additionally hold quorum. A node determines its roles when it starts: worker, recovery worker, and where the cluster is short of members, a quorum member too.

A node holding quorum lowers its own throughput budget. Consensus means an fsync per commit and replication to peers, which contends with ingest for the same disk and CPU — so it takes less work and leaves headroom, trading its throughput for the cluster's stability.

Two nodes is refused rather than discouraged: quorum of two means both must be up for any write, so losing either halts writes. That's strictly worse availability than a single node, which keeps serving alone — the one size worse than both its neighbours, and the one an autoscaler would step through by accident.

Membership changes far less often than pod count, which is what makes this workable: a three-member cluster tolerates one member gone, so scaling three workers to two needs no reconfiguration at all.

Catching duplicates the record ID can't

The record ID hashes source position, so broker redelivery is idempotent. It does nothing for a duplicate created upstream — a producer's own retry republishes the same record at a new offset, which hashes differently and passes through. Hashing the payload catches it.

09Delivery plan

Phases, with the constraint that shapes them: nothing ships that would be silently lossy. Where a capability can't be made safe yet, configuration refuses it rather than allowing a topology that only looks correct.

Phase 5 is the one to watch: it carries the sink layer, the connector abstraction and the recovery machinery, and recovery is a precondition for the second sink rather than hardening applied afterwards.

10Command-line tools

Most of the engine is scaffold and design, gated behind the delivery plan. One capability is built and usable today: turning a Protobuf source tree into the artefacts the rest of the platform consumes, and projecting it to a data contract.

twg proto bundle

Resolve any Protobuf source — a single .proto file, a bare directory of protos, or a Java/Maven/Gradle repository with many src/main/proto roots — into one combined descriptor set plus human-readable renderings and a machine-readable manifest.

OPTIONS

OUTPUTS

Design records: docs/adr/0063 (bundling) and docs/adr/0064 (the schema bridge). The other twg subcommands — stream, batch, config — are scaffolds pending their delivery-plan phases.

11Decision records

Every significant choice is recorded as an ADR, so the reasoning survives the people. One remains genuinely open, pending a trade study; one is drafted and accepted, and one newly drafted and in review; the rest are decided and awaiting drafting into full prose.

The full index lives in docs/adr/README.md. Each will be drafted in the house ADR style and synced to Confluence.

12Why this name

The thalweg is the line of fastest flow — the deepest continuous channel down a watercourse, the path water takes when it has a choice. For an engine whose job is the efficient path from source to destination, that is the thesis in one word.

It carries a second meaning that happens to describe a real design decision. In international law, the thalweg is the boundary drawn along a river: the line that carries the authority. That is precisely how this system treats its primary-raw lane — one sink holds the authoritative copy, the source is acknowledged against it, and everything else is reconciled back to it.

The project was drafted under the working name Millrace. That name turned out to be published on crates.io already — by another streaming event system, a direct collision in this project's own domain — so it was changed before anything was pushed or published. Thalweg is also a watercourse term, which is why the channel diagrams, the sluice-gate stages, and the weathered-copper palette on this page all survived the rename intact.

What else was considered

Other water-engineering candidates were checked for availability and brand collisions. penstock had the best literal meaning — the pressurised conduit feeding a turbine — but collides with a healthcare vendor whose product is audit software, an unfortunate neighbour given this system's own audit trail. headrace collides with an HR-tech company. adit was free but reads as a mis-typing of "audit". sluice, weir, leat, grist, winnow, spillway, braid and riffle were all already published crates.

A bare three-letter acronym with no parent word was considered and rejected: the available combinations were either colliding — trc and drc are smart-pointer crates, tde is Transparent Data Encryption, avc is H.264 video — or characterless. Deriving the prefix from a real word gives both a memorable name and a short prefix. Within that, thw was rejected in favour of twg: it is the well-known abbreviation for Technisches Hilfswerk, Germany's federal civil-protection agency, an awkward adjacency for a German-named organisation.

Crates are prefixed twg-; the command is twg. twg proto bundle is built and usable today; twg stream, twg batch and twg config are scaffolds pending their delivery-plan phases.

13Document map

Where to find things in the repository.