14.7 Messaging, Queues & Asynchronous Processing

1. Plain English Overview

Message queues and event streams decouple fast client-facing web servers from slow, resource-heavy backend operations (such as video transcoding, sending email notifications, or processing telemetry) by holding jobs in an ordered queue until background workers process them.

---

2. Usage in Cooking Nexus

• Current State: Direct synchronous processing in Express routes ([server.js](file:///Users/avishjain/avishjain9-repos/26-cooking-nexus/server.js)).
• Target Asynchronous Pipeline: Decouple high-frequency /api/vision hazard pings and video upload transcoding into a background worker queue.

---

3. Asynchronous Queue Architecture

---

4. Messaging Technologies Comparison

Technology	Delivery Model	Delivery Semantics	Ordering Guarantees	Throughput	Fit for Cooking Nexus
Redis Streams (Chosen)	In-Memory Stream / Consumer Groups	At-least-once	Strict FIFO per stream	Ultra-High (>500k ops/sec)	Optimal for IoT frame buffering
RabbitMQ	Traditional AMQP Message Broker	At-least-once	FIFO per queue	Moderate (~50k msg/sec)	Great for complex routing keys
Apache Kafka	Distributed Commit Log	At-least-once	Strict per partition	Extreme (>1M msg/sec)	Overkill for early stage
AWS SQS	Managed Cloud Queue	At-least-once	Standard (Best effort) / FIFO	High	Vendor locked, easy setup
BullMQ (Redis)	Node.js Redis Job Queue	At-least-once	Configurable priority	High	Excellent for video transcoding

---

5. Delivery Semantics & Failure Handling

1. At-Least-Once Delivery: Workers acknowledge messages (XACK) only after the batch insert into MongoDB succeeds. If a worker crashes mid-insert, another worker claims the pending message after a 5-second visibility timeout.
2. Idempotent Consumer: Every safety telemetry payload includes a unique eventUuid generated at the camera. The worker performs an atomic $setOnInsert in MongoDB using the eventUuid, ensuring duplicate message retries never create duplicate safety incident records.
3. Dead Letter Queue (DLQ): If a corrupted payload fails validation 3 consecutive times, it is routed to a dead letter queue for developer inspection.

---

6. What Breaks First at Scale?

• At 10x: Synchronous database writes in /api/vision will block the Express event loop under concurrent load.
• At 100x: Queue memory overflow in Redis if worker processes halt and messages pile up unchecked.
• At 1,000x: Partition consumer lag in Kafka if video transcoding workers cannot keep pace with uploaded reels.

---

7. Senior Engineer Improvements

Implement Queue Backpressure & Shedding: If the telemetry queue depth exceeds 50,000 pending items, downsample low-severity normal-temperature pings while giving strict priority to high-severity smoke or flame hazard messages.

---

8. Spoken Interview Answer (30 Seconds)

"To scale kitchen safety telemetry to 10,000 concurrent kitchens, we decouple ingestion from database persistence using Redis Streams. When an optical camera posts a frame to /api/vision, Express immediately pushes the event to the stream and returns an HTTP 202 Accepted in under two milliseconds. Background worker consumers pull events in batches of 500, write them to MongoDB, and trigger push notifications, while an idempotent event UUID ensures that network retries never produce duplicate database entries."