Observability: logs, metrics, traces, alerts and SLOs

Logs and traces are not the same thing: a log lives in one service, while a trace stitches the services together.

• A correlation ID (the transaction id every downstream call carries) is the foundation of tracing.
• Distributed tracing adds a trace ID for the whole request and a span for each hop (service A, then B, then the DB) with start time, duration and parent. They travel in headers (the W3C traceparent standard).
• The waterfall view shows it at a glance: "checkout took 2.1 s, and 1.8 s of it was the inventory service".
• Tools: OpenTelemetry (the standard), Jaeger, Zipkin, Datadog.
• Generate the ID at your edge / gateway instead of trusting the client.
• Log the trace ID in every log line so logs and traces connect.
• Sample traces (for example 1% plus every slow or failed request), because tracing everything is expensive.

• CPU at 80% is a cause, and users may be perfectly happy at 80%. Alert on what users feel.
• The golden signals: latency, traffic, errors, saturation. For example "error rate above 1%" or "p99 checkout latency above 2 s".
• Autoscaling on resource metrics such as CPU or queue depth is a good use of them. It is not the same as paging a human.
• Only page a human for something actionable. Too many noisy alerts cause alert fatigue, and people start ignoring them.

• Set the SLO stricter than the SLA, so you are warned well before you break the contract.
• Error budget: 99.9% allows 0.1% failure, about 43 minutes of downtime a month. A healthy budget means ship faster. A burned budget means freeze features and fix reliability.
• Burn-rate alerts: alert when you are using up the budget too fast. A fast burn (the month budget gone in 2 days) pages someone. A slow burn opens a ticket.
• Do not alert on the SLA directly.

• Normal way: the app emits metrics directly (counters, gauges, histograms) with Prometheus, StatsD or OpenTelemetry. Tiny numbers, cheap to store.
• Log-based metrics: a tool counts log lines that match a pattern ("ERROR lines per minute") in Splunk, Datadog or CloudWatch. It is a fallback for legacy apps you cannot change. It is fragile (a reworded message breaks the regex) and costly (logs are huge).
• Splunk started as a log search and analytics platform: it ingests and indexes machine data, you query it with SPL, and it has dashboards and alerts. It is heavy in ops and in security (SIEM), and now also offers metrics and tracing through Splunk Observability Cloud.
• Many companies use one platform for all three signals (Datadog, Grafana stack, New Relic, Elastic, Splunk). The storage underneath is separate, and the signals are linked by the trace ID.
• Open-source stack: ELK or Loki for logs, Prometheus + Grafana for metrics, Jaeger / Tempo for traces.