Logging and Observability Basics

Once a backend is running in production, you can't attach a debugger to it and step through a live request. Observability is the practice of instrumenting a backend so its actual behavior — errors, latency, unusual patterns — is visible after the fact, without having to reproduce the problem locally. It rests on three main pillars: logs, metrics, and traces.

Logs: what happened

Logs are timestamped records of discrete events. The key upgrade from beginner to production-grade logging is moving from unstructured text to structured logs — consistent, machine-parseable fields instead of a free-form sentence.

// Unstructured — hard to search or aggregate "User 42 failed to checkout because payment declined" // Structured — queryable, filterable, aggregatable { "timestamp": "2026-09-15T10:22:31Z", "level": "error", "event": "checkout_failed", "user_id": 42, "reason": "payment_declined", "order_id": 1024 }

Structured logs let you ask questions like "how many checkouts failed with payment_declined in the last hour" directly against a log aggregation tool, instead of grepping through text.

What to log, and what never to

Log enough to reconstruct what happened: request IDs, user IDs, key business events, errors with stack traces. Never log secrets, passwords, full credit card numbers, or other sensitive personal data — a logging pipeline is itself a place data can leak, and it's easy to forget that logs are often retained (and searchable by many engineers) far longer than the data would otherwise live.

Metrics: how much, how often

Metrics are numeric measurements aggregated over time — request count, error rate, response latency (often tracked as percentiles: p50, p95, p99), CPU and memory usage. Unlike logs, which describe individual events, metrics describe trends and are what dashboards and alerts are usually built on.

http_requests_total{route="/orders", status="500"} 42 http_request_duration_seconds{route="/orders", quantile="0.95"} 0.340

Percentiles matter more than averages for latency — an average can look fine while a meaningful slice of real users (the p95 or p99) has a genuinely bad experience that the average quietly hides.

Traces: following one request across services

A trace follows a single request as it moves across multiple services — an API gateway, an auth service, a database, a payment provider — showing exactly where time was spent. In a system made of several services, a slow endpoint can be hard to diagnose from logs and metrics alone; a trace shows precisely which downstream call in the chain was the actual bottleneck.

Trace abc-123 (total: 420ms) ├─ API gateway 12ms ├─ Auth service 8ms ├─ Order service 45ms │ └─ Database query 38ms └─ Payment service 340ms ← the actual bottleneck

Why this matters beyond debugging

Good observability turns "the app feels slow sometimes" — an unactionable complaint — into "p95 latency on /checkout spiked to 2.3s starting at 14:32, correlated with a payment provider timeout" — a specific, actionable fact. That shift is the entire point: production issues get resolved by data, not guesswork.

With individual requests observable, the next lesson zooms out to how an entire backend system is structured — starting with the monolith-vs-microservices decision.