Expand description
Pure-Rust Pulsar binary-protocol client. Scaffold only.
§Why a fresh client rather than wrapping an existing crate
Existing crates provide the protocol basics — connect, subscribe, receive, ack, reconnection — but not the flow control a memory-constrained, high-throughput consumer requires. Three gaps forced custom layers in a production predecessor and are designed in here rather than bolted on:
- No downstream coordination. The consumer takes whatever the broker prefetched; there is no way to say “I am saturated, stop”.
- All-or-nothing subscription. Subscribing to a partitioned topic creates every partition’s consumer at once, each with its own prefetch buffer.
- No memory awareness. No hook to observe memory pressure and adjust.
§The failure this exists to prevent
Memory scales with physical partition count, because each partition gets its own prefetch buffer. A partitioned topic with tens of partitions and a default prefetch of ~1000 messages holds tens of thousands of messages resident the instant it subscribes. Against a backlog on a small pod, that is an OOM before the first message is processed — a boot storm, not a steady-state problem.
Measured in production: ~54 partitions on a 2 GB pod spiked past the cgroup limit at subscription time and was killed every time. Staged subscription made the same workload stable at 60-80% of limit.
§Staged subscription
Start at zero active partitions and add in batches with a delay between them, so memory rises in observable steps instead of one spike. Growth pauses automatically when memory crosses the high-water mark, so the pod settles at whatever coverage it can actually sustain rather than dying at full coverage.
§Shedding under pressure, and the two non-obvious rules
Above a memory high-water mark, and subject to a cooldown that prevents thrashing, shed a batch of partitions; below a low-water mark, grow again. Never drop below a configured floor — a floor is what stops shed/grow oscillation from starving the pod entirely.
Shed topic-spread, never tail-drop. Dropping the highest-indexed partitions annihilates whichever topic sorts last in configuration — and if that is the high-throughput topic, it is starved to zero coverage while low-throughput topics keep full coverage. Shed one partition per topic round-robin instead, so every topic degrades proportionally and none is zeroed.
Seed the round-robin order per pod. Without it every pod walks topics in configuration order and sheds from the same topic first under simultaneous pressure, concentrating the damage. A pod-index-seeded shuffle spreads it.
§Pressure is a scale-out signal, not just a local action
Shedding relieves this pod but does not reduce total demand — with a shared subscription the load redistributes to other pods, which may themselves be under pressure. Moving pressure around the ring is not relief. So when a pod is at its floor with partitions it would otherwise serve, it emits a pressure gauge that the autoscaler consumes as unmet demand and adds capacity. Shedding without this signal is a slow-motion failure.
§Known limitation, deliberately recorded
Shedding is partition-count aware, not lag aware. If a badly-lagging partition is the one shed, this pod’s throughput does not improve; recovery comes indirectly via redistribution and scale-out. Lag-aware shedding — preferring to keep high-lag partitions and shed idle ones — needs per-partition lag from the broker and is a genuine improvement available to us, since we control the client.
§Subscription type and assignment are orthogonal
Worth stating because it is easy to conflate: the subscription type (exclusive / failover / shared / key-shared) governs how multiple consumers on the same partition divide messages. Which partitions this pod consumes is a separate, client-side decision — partitions can be subscribed individually. Both are in play at once, and a design can control assignment while still using a shared subscription for redistribution.