The fastest
line down the channel.

01How it works

A record enters from a source — a broker, a file, an Arrow stream. It is decoded once into a shared columnar form, checked against its data contract, transformed by SQL if you want it to be, and written to one or more sinks. Along the way every record carries its origin, so if a destination falls behind it can be rebuilt from the durable copy rather than re-read from the source.

Every box on the channel is a stage you can observe independently — its throughput, its memory, and whether it's healthy or building backpressure — at /health and /stats.

02What's supported

Thalweg is pre-implementation: the architecture is decided and the workspace is scaffolded, but code has not yet been written. The table below is honest about that. It shows where each capability stands today, and what is planned but not yet started — for example RabbitMQ, NATS, and other transports are planned, not built.

Sources

Payload formats

Sinks & table formats

Platform capabilities

03Design principles

Five ideas shape every decision in Thalweg. They are worth reading before the architecture, because the rest follows from them.

04The layers

Crates are grouped into layers with a strictly downward dependency direction, enforced in CI. Nothing in the codec layer may depend on a transport; nothing in the codec or SASL layer may depend on an async runtime.

Reading bottom to top: raw bytes come in through transport primitives, up through sources into the shared connector contract, get decoded and validated, transformed, and written out. The codec-core crate holds the single Arrow re-export — the one place the Arrow version is named, so the whole workspace pins one version.

05Crate reference

Forty-three crates. Filter by layer, or read straight through. Every crate is currently a scaffold — the descriptions are the contract each will fulfil.

06Recovery & guarantees

The correctness backbone is one rule: the source offset advances only after a sink has durably acknowledged the record — never on send, never on buffered. In a multi-sink setup, the sink that gates the source is the one holding the primary-raw role. Everything else recovers from raw.

07Decision records

Every significant choice is recorded as an ADR, so the reasoning survives the people. Four remain genuinely open, pending trade studies or production input; the rest are decided and awaiting drafting into full prose.

The full index of all 44 records lives in docs/adr/README.md. Each will be drafted in the house ADR style and synced to Confluence.

08Why this name

The thalweg is the line of fastest flow — the deepest continuous channel down a watercourse, the path water takes when it has a choice. For an engine whose job is the efficient path from source to destination, that is the thesis in one word.

It carries a second meaning that happens to describe a real design decision. In international law, the thalweg is the boundary drawn along a river: the line that carries the authority. That is precisely how this system treats its primary-raw lane — one sink holds the authoritative copy, the source is acknowledged against it, and everything else is reconciled back to it.

The project was drafted under the working name Millrace. That name turned out to be published on crates.io already — by another streaming event system, a direct collision in this project's own domain — so it was changed before anything was pushed or published. Thalweg is also a watercourse term, which is why the channel diagrams, the sluice-gate stages, and the weathered-copper palette on this page all survived the rename intact.

What else was considered

Other water-engineering candidates were checked for availability and brand collisions. penstock had the best literal meaning — the pressurised conduit feeding a turbine — but collides with a healthcare vendor whose product is audit software, an unfortunate neighbour given this system's own audit trail. headrace collides with an HR-tech company. adit was free but reads as a mis-typing of "audit". sluice, weir, leat, grist, winnow, spillway, braid and riffle were all already published crates.

A bare three-letter acronym with no parent word was considered and rejected: the available combinations were either colliding — trc and drc are smart-pointer crates, tde is Transparent Data Encryption, avc is H.264 video — or characterless. Deriving the prefix from a real word gives both a memorable name and a short prefix. Within that, thw was rejected in favour of twg: it is the well-known abbreviation for Technisches Hilfswerk, Germany's federal civil-protection agency, an awkward adjacency for a German-named organisation.

Crates are prefixed twg-; the command is twg, with twg stream, twg batch and twg config as subcommands.

09Document map

Where to find things in the repository.