Expand description
Live per-stage memory & throughput digest — 30-second rolling window.
Served two ways, not redundant — they serve different moments:
- OTEL gauges (continuous): raw per-stage throughput (records/sec,
bytes/sec), memory, batch sizes. Dashboards and alerting compute their
own windows (Grafana
avg_over_time([30s])). Steady-state view. - GET /stats (in-process): the digested 30s rolling breakdown as JSON — per-stage throughput and average per-record / per-batch memory — so a pod SSH’d into during an incident tells you directly where its memory went, without a round-trip to a metrics backend that may itself be degraded.
Per-record memory is an ATTRIBUTION problem, not a raw measurement: stages share an allocator, so global RSS cannot honestly be sliced per stage. The digest reports what is genuinely stage-owned — queue depth × bytes, batch count × batch size, in-flight bytes — and derives per-record/per-batch averages from those, not from carving up total RSS. A true number, not a precise-looking fiction. Pure-Rust allocation gives cleaner accounting than jemalloc-over-C, but the attribution honesty still holds.
Scaffold only.
§Flow-control metrics that make a diagnosis possible
These exist to distinguish causes that look identical from the outside:
partition_active_count / partition_total_count
coverage — how much of the topic this pod is actually serving
partition_pressure 0.0 | 1.0
1.0 means at floor WITH partitions it would otherwise serve: unmet
demand, and the signal an autoscaler should act on
backpressure_state normal | degraded | critical | recovery
source_throttled counter, per evaluation while throttled
source_throttle_transitions counter, on false->true edge only
the pair matters: duration vs episode count are different questions,
and one metric cannot answer both
byte_budget_saturation_ratio 0.0..1.0
sustained high with LOW memory means the sink is the bottleneck, not
memory — the diagnosis the operator most often gets wrong
memory_ratio 0.0..1.0, only when a cgroup limit is detected
Where no cgroup limit can be read, memory-based backpressure degrades to channel-fill only. That is an acceptable fallback but a materially weaker one, and it should be visible rather than silent.