Crate twg_codec_protobuf 

Protobuf to neutral-value and Arrow decoding

Scaffold only — no implementation yet. See docs/blueprints for the phase this crate lands in.

Decode strategy: selective vendoring plus our own registry and Arrow stage

Status: OPEN — licence cleared, gated on a benchmark.

Why not prost-reflect

DynamicMessage allocates an owned value tree per record, which is the intermediate materialisation this project rejects elsewhere (the Avro decision is bytes straight into builders, never through avro::Value). Published figures for a zero-copy alternative put it 4-9x faster, widening with field count — on a 100-field schema prost-reflect falls to roughly 176 MB/s against ~1.6 GB/s. Upstream’s numbers on upstream’s hardware, not independently reproduced.

Why not depend on that parser

It ships only inside a vendor ingest SDK behind a feature flag, with no standalone crate, and that SDK’s manifest pulls a multi-threaded async runtime, an HTTP client, and JSON — into a crate required to be sans-io. It would also put a vendor SDK in the decode path of deployments with no sink from that vendor, and couple decode to that SDK’s release cadence. A predecessor system took the dependency and had its decode path broken by a minor release.

Why not write all of it ourselves

Tempting, and wrong for most of it. The upstream parser is ~2,700 lines and that bulk is the protobuf edge cases: proto2 versus proto3 presence, packed repeated fields, maps as synthetic entry messages, oneof, groups, well-known types. Reimplementing reproduces that surface for no architectural gain — a varint is a varint, and “our use case is different” is not true at the wire layer. The licence (Apache-2.0, no NOTICE file) permits reuse with light obligations: retain copyright headers, state changes made, include the licence.

The split

Vendor, minimally modified and clearly marked as vendored:

• wire-format and varint decoding — textbook and stable
• the single-pass recursive parser core — where the edge cases live
• the sparse field map — small, and the O(1) lookup insight is the point

Adapt:

• value representation, which wants to be Arrow-oriented rather than built around a random-access accessor API we do not need

Ours outright:

• the registry, built from the whole FileDescriptorSet and resolving by fully-qualified name. Upstream builds from a single root descriptor by walking root.nested_type only, so imported types (google.protobuf.*) and sibling top-level messages are never registered and decode fails with an unknown-type error — while compiling and type-checking cleanly, which makes it a production-only failure. This is the actual fix, not a preference; descriptor flattening is the workaround needed only when the registry cannot be replaced.
• the Arrow output stage, which does not exist upstream.
• the error taxonomy.

An honest note on “direct to Arrow”

Fields arrive in arbitrary wire order and repeated fields can interleave, so a record’s absent fields — the ones needing nulls appended, since every column needs a value per row — are only known at end of record. Appending straight into builders mid-parse is therefore not possible; per-record staging is necessary, not incidental, and the upstream pre-sized arrays are exactly that staging. The realistic shape is parse-into-staging then flush-to-builders. Any claim of a single fused pass should be treated sceptically until measured.

What IS available to us, and worth measuring: pre-resolving field number to builder index once per schema rather than per record; pre-sizing builders from batch cardinality; and staging raw slices plus wire type rather than a tagged value enum, converting directly into the typed builder at flush.

Gate before this is settled

Benchmark with the upstream parser as the baseline to beat, on our hardware and our schemas, prost-reflect as the reference floor, including a 100+ field wide schema — that is where reflection-based decoders collapse and therefore where the question is decided. Criterion, committed baselines, CI failure on regression. If our staging-and-flush does not beat parse-then-walk by a real margin, the adaptation is not worth maintaining.

Rejected regardless

Copying a standard definition such as StringValue into a schema as a hand-written nested message makes decode succeed and is wrong: it duplicates well-known types per schema, drops their metadata, and breaks interoperability with ordinary protobuf tooling. Use standard imports and resolve them properly.

Properties to prove by test, whatever the decoder

Imported wrapper types, sibling references, circular type graphs, map entries, deep nesting, repeated messages as list-of-struct, schema data-independence, and missing-imports failing at schema load rather than on first matching payload. See docs/testing/SCENARIOS.md. Library-agnostic, and must keep passing if the decode strategy changes again.