NotiQ — Distributed Notification Platform
Eight Rust microservices communicating over gRPC and a Postgres-native job queue. Built to explore what actually breaks when you operate a reliable delivery system at scale — gossip-based membership, consistent hashing, lock-free concurrency, and end-to-end backpressure across service boundaries.
System architecture
Services — bounded contexts
Service communication
| call | protocol | pattern | failure handling |
|---|---|---|---|
| gateway → enqueue-svc | gRPC | sync request/reply | circuit breaker trips → 503 to client |
| gateway → admin-svc | gRPC | sync request/reply | circuit breaker → graceful degradation |
| enqueue-svc → worker-svc | async queue | outbox + CDC + LISTEN/NOTIFY | worker down → jobs persist in Postgres, zero loss |
| worker-svc → delivery-* | gRPC | fan-out, bulkhead per channel | channel down → dead-letter; others unaffected |
| scheduler-svc → enqueue-svc | gRPC | saga orchestration step | enqueue fail → saga compensates in reverse |
| worker-svc ↔ worker-svc | UDP gossip | SWIM probe + epidemic broadcast | partition → suspect → dead → ring rebalance |
| all services → Postgres | TCP / sqlx pool | connection pool, schema per service | Multi-AZ failover, pool reconnect |
| all services → Redis | TCP / fred-rs | rate limit · idempotency · backpressure | Redis down → configurable allow-all fallback |
How it was built — key decisions
A Postgres-native queue (like Graphile Worker, River, Oban) keeps job state transactional with the rest of the app. No external broker, no dual-write race, no new failure domain. The honest tradeoffs: Postgres becomes the throughput ceiling at very high write rates, WAL amplifies under heavy churn, and cross-region queue replication is hard. Acceptable for this scale.
Multiple workers claim different jobs from the same table with zero serialization conflicts — no advisory locks, no SELECT FOR UPDATE blocking. Rows locked by another transaction are skipped entirely. The same mechanism behind pgqueue, Graphile Worker, and River.
Workers sleep on a Postgres channel and wake on INSERT rather than polling. Push for latency, pull for correctness — the outbox CDC task guarantees at-least-once delivery regardless of whether the notification arrived.
Job state is an append-only sequence: Enqueued → Claimed → Failed → Retried → Delivered. Current state is a projection. Audit trail is free, replay is exact. Overwriting a status column destroys history and makes debugging retried failures guesswork.
BTreeMap vnode ring over a 2³² token space. 150 virtual nodes per worker smooth distribution. On join/leave, only the adjacent token range migrates — ~1/N of total keys. Proven with a test: add a worker, measure the fraction of keys that moved.
Direct ping → suspect on timeout → indirect ping via K random peers → dead on no ack. Epidemic broadcast piggybacks membership deltas on outgoing pings — O(log N) convergence without a coordinator. A dead event triggers ring rebalance automatically.
crossbeam-channel connects the gossip, dequeue, and fan-out tasks inside each instance. No mutex on the hot path. AtomicU8 for node liveness with Acquire/Release ordering — not SeqCst, which would be correct but unnecessarily expensive.
Payloads are wrapped in bytes::Bytes. Fan-out to N delivery channels clones the Arc pointer, not the heap allocation. A criterion benchmark with allocation counting confirms zero extra heap allocations per channel.
Email, SMS, and webhook are separate ECS services with separate pipelines and failure domains. A SES outage tripping circuits has zero effect on Twilio throughput. The bulkhead makes failure boundaries explicit rather than accidental.
Each delivery service signals capacity upstream near saturation. worker-svc aggregates via an atomic queue-depth counter and propagates upstream to enqueue-svc. Unbounded queues cause silent data loss; this prevents it.
Naive exponential backoff synchronizes retries across instances and causes thundering herd after mass failure. Decorrelated jitter (sleep = min(cap, random(base, prev × 3))) keeps retry waves spread over time.
A --chaos flag randomly kills workers mid-flight, delays acks, and injects partitions. A run audits the delivery_log and proves zero message loss. That's the only real proof of reliability — review alone isn't enough.
VPC with public/private split, SGs as code, one ECS task definition per service, RDS multi-AZ + read replica, ElastiCache, Secrets Manager, KMS, Route 53, ACM. Nothing clicked in the console — infra is reproducible and reviewable.
Each ECS task assumes its own least-privilege IAM role via IRSA. enqueue-svc writes S3 and reads Secrets Manager; delivery-email calls SES. No shared credentials, no static keys in env vars or images.
worker-svc auto-scales on a custom CloudWatch metric — Postgres queue depth — not CPU. CPU is the wrong signal for a queue worker: low when idle, spiking when overwhelmed. Queue depth is what actually needs to scale against.
Every service emits traces via tracing-opentelemetry. One trace ID spans gateway → enqueue → worker → delivery across gRPC boundaries via propagated context. OTLP export to CloudWatch in prod, Jaeger locally.
gRPC gives strongly-typed protobuf contracts, generated client stubs, binary encoding (~3-10× smaller than JSON), and streaming. Breaking changes are caught at compile time, not at runtime in production.
Each service owns its tables under a named schema. No service queries another's tables directly — cross-service reads go through gRPC. This enforces bounded-context ownership at the database level.
UUID v4 is random — poor index locality, page splits under write load. UUID v7 is time-ordered: new rows land at the end of the index, giving sequential writes and far better insert throughput on jobs and job_events.
axum, tonic, sqlx, fred, and crossbeam all integrate directly with Tokio. Ecosystem coherence matters more than runtime micro-benchmarks at this scale.