Appearance
14.14 Observability, Logging & Site Reliability Operations
1. Plain English Overview
Observability provides deep, real-time visibility into an application's internal state through the "Three Pillars of Observability"—Logs, Metrics, and Distributed Traces—allowing engineers to detect anomalies, debug outages, and ensure high system reliability.
2. Usage in Cooking Nexus
- Current Logging: Express console logging for magic link generation and server startup ([server.js:L6](file:///Users/avishjain/avishjain9-repos/26-cooking-nexus/server.js#L6), [authController.js:L12](file:///Users/avishjain/avishjain9-repos/26-cooking-nexus/controllers/authController.js#L12)).
- Safety Incident Audits:
SafetyLogcollection preserving historical hazard records. - Client Error Boundary: Toast notification interceptors catching SSE streaming failures in [ai-stream.ts:L95](file:///Users/avishjain/avishjain9-repos/26-cooking-nexus/27-kitchen-companion-pro/src/lib/ai-stream.ts#L95).
3. Observability Architecture (The 3 Pillars)
4. Observability Tools Comparison
| Tool / Category | Primary Pillar | Overhead | Scalability | Cost Profile | Fit for Cooking Nexus |
|---|---|---|---|---|---|
| Pino (Recommended) | Structured Logs | Ultra-Low (Fastest JSON) | High | Open Source | Standard for Node backend |
| Winston | Logs | Moderate (Transport tax) | Moderate | Open Source | Popular, but slower than Pino |
| Prometheus + Grafana | Metrics (Time-series) | Low (Pull model) | High (Mimir / Cortex) | Open Source / Low cost | Industry standard |
| OpenTelemetry (OTel) | Distributed Tracing | Low | Vendor-neutral | Open Source | Essential for microservices |
| Datadog / Sentry | Full-Stack APM & Errors | Ultra-Low (Agent based) | SaaS Scale | High ($$$ per host) | Great for automated crash alerts |
5. Golden Signals & SLI / SLO Definitions
- Service Level Indicator (SLI):
- Availability: Percentage of successful HTTP 200/202 responses on
/api/vision. - Latency: 99th percentile (p99) response time for hazard ingestion.
- Availability: Percentage of successful HTTP 200/202 responses on
- Service Level Objective (SLO):
- 99.9% of
/api/visionrequests complete successfully over any rolling 30-day window. - 95% of requests complete in $< 100\text{ms}$.
- 99.9% of
- Service Level Agreement (SLA): 99.5% uptime commitment to smart kitchen appliance manufacturers.
6. What Breaks First at Scale?
- At 10x: Server disk space fills up with un-rotated
console.logstdout text files. - At 100x: Alert fatigue: poorly tuned alerts wake on-call engineers for benign 1-second transient network blips.
- At 1,000x: High-cardinality metric explosion in Prometheus if tracking metrics with dynamic user IDs as labels.
7. Senior Engineer Improvements
- Correlation IDs: Inject an
x-correlation-idheader in Express middleware. Pass this ID into every database query, log message, and upstream AI request so an engineer can trace an entire request across all services with a single search. - Crash Reporting: Integrate Sentry in both the React frontend and Express backend to capture unhandled exceptions with full stack traces and client breadcrumbs.
8. Spoken Interview Answer (30 Seconds)
"In Cooking Nexus, our observability strategy focuses on the four Golden Signals: Latency, Traffic, Errors, and Saturation. We use Pino for high-performance structured JSON logging with correlation IDs, Prometheus for tracking request duration percentiles, and Sentry for real-time frontend crash alerts. By setting our SLO to 99.9% availability for kitchen safety ingestion, our alerting rules page engineers only when burn rates threaten customer-facing reliability."