API reference

Crate documentation

Rendered rustdoc for every crate in the Thalweg workspace — signatures and doc comments for the public surface of each crate. The prose documentation (architecture decisions, blueprints, operations, testing) is in the reference; the overview and support matrix are on the landing page.

50 crates

twg The `twg` command — Thalweg’s single entry point. twg-codec-avro Avro to neutral-value and Arrow decoding twg-codec-core Neutral value model, decode/encode traits, and the single Arrow re-export. twg-codec-custom Custom binary decode. Scaffold only. twg-codec-json JSON to neutral-value and Arrow decoding twg-codec-protobuf Protobuf to neutral-value and Arrow decoding twg-codec-xml XML to neutral-value and Arrow decoding twg-config layered config load (TOML < env < CLI), K8s file-mount + pointer resolution, secret enumeration twg-connector-core Transport-neutral Source/Sink, flow control, observability, recovery. twg-contract-core resolved validation spec (schema + quality rules) with precedence and most-restrictive merge; contract-format-agnostic twg-contract-odcs ODCS v3.x parsing and loading (local/HTTP/object-store), translated into contract-core’s spec twg-dq-audit DQ audit-trail record shape. twg-dq-enforcer data-quality enforcement modes and split twg-e2e End-to-end test harness. twg-ffi FFI boundary — both directions of foreign interop, isolated in one audited crate. Scaffold only. twg-format-iceberg Apache Iceberg table format read + write; catalog-mediated twg-format-parquet standalone Parquet file read/write to object store (no table layer) twg-kafka-arrow composition: Kafka source to Arrow pipeline twg-kafka-client pure-Rust Kafka protocol client twg-observability Unified observability surface: metrics, tracing export, and health. twg-offset-store Sink write-coverage store. Scaffold only. twg-pipeline-flight Arrow Flight sidecar transforms twg-pipeline-sql `DataFusion` transform DAG and fan-out twg-pipeline-wasm wasmtime host for in-process pure-transform UDFs twg-proto-bundle Bundle a protobuf source into the artefacts the descriptor toolkit and an operator need: one combined `FileDescriptorSet` (`.pb`), one `proto3` file per package, and a fully-qualified message-type manifest. See ADR-0063 and `docs/blueprints/proto-bundle-implementation-plan.md`. twg-proto-decode API documentation for the Rust `twg_proto_decode` crate. twg-proto-flatten Flattens a `FileDescriptorSet` into a single self-contained `DescriptorProto` compatible with a shallow, `nested_type`-only message registry (the shape built by, for example, a `MessageRegistry::from_descriptor` that only recurses `nested_type`). twg-proto-schema Flattens a protobuf message type into an Arrow schema, matching the semantic policies documented in `flatten` and `wrapper`. Generic and self-contained: no runtime wire-decode, no product-specific assumptions - just descriptor-to-schema computation. twg-pulsar-arrow composition: Pulsar source to Arrow pipeline twg-pulsar-client Pure-Rust Pulsar binary-protocol client. Scaffold only. twg-resource-loader shared local/HTTP/object-store resolver with caching and fail-fast; used by config, ODCS contracts, and transform SQL twg-schema-bridge Convert schemas between formats through a neutral hub (ADR-0064): every format converts to/from `twg_contract_core::schema::SchemaModel` (structure) so N formats cost N adapters, not N². twg-schema-registry schema registry clients and wire-envelope handling twg-sink-arrow-ipc Arrow IPC as a sink: file and stream, non-Flight twg-sink-flight Arrow Flight/IPC as a first-class sink transport twg-sink-object-store Object-store sink. Scaffold only. twg-sink-postgres `PostgreSQL` sink via binary COPY twg-sink-zerobus Databricks Zerobus sink: fire-and-forget Arrow ingest with a per-stream background ack lane (ADR-0060), a bandwidth-sized stream pool, catalog-driven table provisioning, and emit-time type reconciliation. twg-source-arrow-ipc Arrow IPC as a source: file (random-access) and stream (sequential), non-Flight, with locator support twg-source-batch Bounded file sources producing the shared record stream. Scaffold only. twg-source-delta-sharing Delta Sharing / open-sharing recipient: cross-org read via pre-signed file URLs twg-source-flight Arrow Flight/IPC as a first-class source transport, with locator and recovery support twg-stream-arrow batch sinks, IPC, Flight, offset correlation twg-table-catalog Table catalog: resolution, registration, credential vending, and descriptive metadata. Scaffold only. twg-type-map Shared type-mapping vocabulary across formats and storage targets. twg-udf `twg-udf`: user-defined functions for Thalweg’s `DataFusion` transform engine. twg-wire-compression pure-Rust compression facade shared by all transports twg-wire-sasl SASL mechanisms as sans-io state machines twg-wire-tls rustls config unification across transports xtask Build automation for the Thalweg workspace.

Generated from the workspace with cargo doc. The source view is not published; these pages carry signatures and documentation only.