Observability: Logs, Metrics, and Tracing with OpenTelemetry

Observability: Correlating signals to explain actual behavior

Central principle

Isolated signs show symptoms; correlation by context allows you to reconstruct cause, impact and dependencies.

In-depth edition - study material and professional reference

radio_button_unchecked Chapter presentation

Previous chapters have shown that an API transaction can traverse DNS, balancers, gateways, service meshes, services, banks and messaging systems. When something fails, no single component sees the entire journey. Observability is the discipline that transforms signals emitted by these components into the ability to understand the internal state of the system, reconstruct causality and evaluate impact for consumers and business processes.

Logs, metrics, and distributed tracing are often called the pillars of observability. The expression is useful, but it can lead to a fragmented interpretation. The value isn't in storing three types of data in different tools; is to correlate them by service identity, environment, version, request, trace and business context. A metric indicates that latency has increased, a trace locates the responsible step, and a log explains the specific error.

OpenTelemetry provides APIs, SDKs, semantic conventions , protocols, and a vendor-agnostic Collector to produce, process, and export telemetry. It does not replace the observability backend. Its role is to standardize instrumentation and transport, reducing dependence on proprietary agents and allowing consistent signals to be sent to different destinations.

This chapter delves into observability theory, the design of logs and metrics, the anatomy of traces and spans, context propagation, OTLP , Collector, semantic conventions , sampling, exemplars, SLOs, security, cardinality control and troubleshooting. The focus remains on enterprise APIs, gateways and distributed architectures.

How to study this chapter

Choose a single business journey – for example, create payment – and track how it appears across all three signals. Ask which identity describes the service, how context traverses each hop, which attributes are stable, and how the diagnosis would change if one of the signals were missing.

Learning Objectives

• Differentiate between monitoring, telemetry, observability, diagnosis and auditing.
• Design structured logs, useful metrics, and distributed traces with consistent context.
• Explain trace , span , span context, resource , event, link, status and baggage .
• Understand W3C Trace Context , traceparent, tracestate, and interprotocol propagation.
• Describe APIs, SDKs, auto-instrumentation, OTLP and OpenTelemetry Collector.
• Apply semantic conventions and control high cardinality attributes.
• Compare head sampling , parent-based sampling and tail sampling .
• Relate exemplars, logs and traces to metrics and SLOs.
• Design resilient, safe and economically sustainable pipelines.
• Diagnose instrumentation failures and correlation gaps in API Gateways and microservices.

Chapter structure

• 32.1 Observability, monitoring and telemetry
• 32.2 Signals and correlation
• 32.3 Structured logs
• 32.4 Metrics and cardinality
• 32.5 Distributed tracing and spans
• 32.6 Context propagation and baggage
• 32.7 OpenTelemetry: API, SDK and instrumentation
• 32.8 OTLP and OpenTelemetry Collector
• 32.9 Semantic conventions and Resources
• 32.10 Sampling and exemplars
• 32.11 SLI , SLO , alerts and dashboards
• 32.12 Observability in gateways, Kubernetes and messaging
• 32.13 Security, privacy, costs and troubleshooting
• Summary, checklist, laboratories, exercises, glossary and references

radio_button_unchecked 32.1 Observability, monitoring and telemetry

Telemetry is the set of data emitted by the system: event records, measurements, traces, profiles and other signals. Monitoring uses part of this data to monitor known conditions, compare values with limits and trigger alerts. Observability is a broader property: the ability to infer internal state and answer novel questions from available external signals.

A system can be intensely monitored and still be poorly observable. Dozens of CPU, memory, and availability dashboards don't explain why only consumers in one region receive 502 when querying a specific backend. Observability requires context, correlation, appropriate granularity, and a data architecture that allows you to navigate from aggregate symptom to individual evidence.

Auditing has a different objective. An audit log records actions relevant to security and compliance, preserving authorship, integrity, and retention. It can participate in investigations, but should not be confused with diagnostic log. Mixing the two uses leads to too much sensitive data in operational tools or incomplete audit trails.

radio_button_unchecked 32.2 Logs, metrics and traces as correlated signals

Logs record discrete events with detailed context. Metrics represent aggregate measurements and are powerful for trends, alerts, and capacity. Traces represent the causal path of an operation through processes and services. Each signal has different costs, query models and granularities; none of them are sufficient for all diagnoses.

Correlation depends on common attributes. Resource identifies the entity that produced the telemetry, such as service, instance, pod, cluster, and environment. Trace ID and Span ID connect logs to a specific run. Semantic attributes standardize concepts such as HTTP method, route, messaging system, database and error code. Without consistency, dashboards and queries became sets of exceptions per team.

OpenTelemetry also treats baggage and profiles as adjacent signals or mechanisms. Baggage propagates key-value pairs across the distributed context; profiles record resource usage at the code level and are evolving in the ecosystem. In this chapter, the focus remains on the three signals most present in APIs, without ignoring that modern architecture can correlate them with other data.

radio_button_unchecked 32.3 Structured logs and operational events

A structured log represents the event as typed fields or key-value pairs, typically serialized in JSON or the agent's native format. Instead of just writing "payment failed", the record includes timestamp, severity, service, environment, version, operation, error code, trace_id, span_id, and allowed business identifiers. This framework improves automatic filtering, aggregation, and correlation.

Severity must reflect the capacity for action. DEBUG and TRACE help in controlled environments; INFO records relevant operational events; WARN indicates degradation or unexpected condition that did not interrupt operation; ERROR records failure of an action; FATAL or equivalent indicates inability to continue. Logging every exception as ERROR creates noise and destroys the value of the alert.

Logs need to avoid unnecessary personal data, tokens, cookies, secrets, full payloads and financial numbers. Further masking is not a sufficient defense, because the data may have already been transported or persisted. The correct design starts at the source, with allowlist of fields, data classification, proportional retention and restricted access.

Example of a structured and correlatable event

{
  "timestamp": "2026-07-16T11:42:31.052Z",
  "severity": "ERROR",
  "service.name": "payments-api",
  "deployment.environment.name": "production",
  "http.route": "/payments/{id}/confirmation",
  "error.type": "BackendTimeout",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7"
}

radio_button_unchecked 32.4 Metrics, instruments, aggregations and cardinality

Metrics represent numerical observations aggregated over time. Counters accumulate events, such as requests and errors. Histograms distribute values, such as duration and payload size, into ranges or representations suitable for the backend. Gauges record a current value, such as open connections or queue depth. UpDownCounters represent quantities that increase and decrease.

The choice of instrument defines how the measurement can be aggregated. A request duration should not be recorded as just an average, because the average hides tails. Histograms allow you to consult percentiles and distribution. Still, incorrectly aggregated percentiles can produce erroneous conclusions; It is necessary to understand the backend model and the time window.

Cardinality is the number of distinct combinations of attributes. Adding user_id, order_id, full URL, or trace_id as a metric label creates almost unlimited series, increases memory and cost, and can crash the pipeline. Metrics should use limited, stable dimensions such as normalized route, method, status class, region, and version. Individual evidence remains in logs or traces.

Cardinality rule

Metric attributes must answer aggregate questions. Practically unique identifiers belong to logs and traces. Before adding a dimension, estimate how many different values it can take per environment, service, and retention window.

radio_button_unchecked 32.5 Distributed tracing, traces and spans

A trace represents a distributed operation, such as an API call that crosses gateway, service, messaging and bank. Each step is represented by a span , with name, time interval, context, attributes, events, status and relationships with other spans. The framework allows you to observe total latency, critical path, and triggered dependencies.

SpanKind describes the role of the span as SERVER, CLIENT, PRODUCER, CONSUMER, or INTERNAL. Classification helps you understand boundaries and calculate metrics from traces. Events record specific occurrences within the span , as an exception. Links relate spans that do not have a single parent-child hierarchy, a common situation in asynchronous and fan-out processing.

The span name must have low cardinality and reflect the operation, not the concrete identifier. In HTTP, a normalized route is preferable to the full URL. In messaging, the name of the destination and the operation help to reconstruct production and consumption. The trace must not be transformed into indiscriminate storage of payloads; attributes and events obey the same privacy rules as logs.

radio_button_unchecked 32.6 Context propagation, W3C Trace Context and Baggage

For spans from different processes to belong to the same trace , the context needs to cross the network boundary. The W3C Trace Context standardizes the traceparent and tracestate headers for HTTP. traceparent carries version, trace ID, parent ID and flags. tracestate transports vendor-specific information in an interoperable manner. Instrumentations must extract the incoming context and inject it into outgoing calls.

In gRPC, messaging, and proprietary protocols, the same principle is applied by metadata or message properties. Propagation must respect trust boundaries: accepting external IDs without validation or copying all baggage to internal systems can introduce abuse, leakage and increased payload.

Baggage carries application context in key-value pairs. He must not transport secrets, personal data or high cardinality information without justification. As it can cross many services, a small field is multiplied by the entire journey. Baggage also does not replace authorization; the application should not trust a propagated claim just because it arrived in the context.

Distributed Context Conceptual Example

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
tracestate: vendorname=opaque-value
baggage: tenant.tier=premium,region.origin=br-south

radio_button_unchecked 32.7 OpenTelemetry: API, SDK and instrumentation

OpenTelemetry separates API and SDK. The API is used by libraries and applications to create instruments without depending on a specific implementation. The SDK implements sampling, processing, aggregation and export. This separation allows a library to be instrumented without requiring the consumer to send data to a specific supplier.

Manual instrumentation is suitable for business operations and sections that generic instrumentation does not understand. Self-instrumentation uses agents, bytecode, monkey patching, eBPF or equivalent mechanisms to capture frameworks and libraries with little code change. The combination is usually superior: automatic for infrastructure coverage and manual for business semantics.

Instrumenting does not mean generating as much data as possible. Each span , attribute , log and series has a cost. The team must define a telemetry strategy: which journeys are critical, which attributes are required, which conventions are stable, which signals will be derived, and how instrumentation will be tested alongside the code.

radio_button_unchecked 32.8 OTLP and OpenTelemetry Collector

OTLP is OpenTelemetry's native protocol for transporting telemetry. It can operate over gRPC or HTTP and has templates for traces, metrics and logs. The protocol reduces the need for different formats per vendor, but does not eliminate network decisions, authentication, compression, queuing, retry, and loss protection.

The OpenTelemetry Collector receives, processes, and exports telemetry. Receivers accept OTLP and other formats. Processors apply batching, filtering, transformation, enrichment, sampling and memory protection. Exporters send data to backends. Extensions provide auxiliary capabilities such as health checks and authentication. Pipelines are declared by sign.

The Collector can be deployed as an agent close to the application, a central gateway per cluster or region, or a combination in layers. The agent pattern reduces hops and collects local data; The gateway pattern centralizes backend processing and credentials. The design must consider availability, persistent queues, isolation per tenant, scalability and the risk of making the Collector a single point of failure.

Collector pipeline summary example

receivers:
  otlp:
    protocols:
      grpc: {}
      http: {}
processors:
  memory_limiter:
    limit_mib: 1024
  batch: {}
exporters:
  otlp/backend:
    endpoint: observability.internal:4317
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/backend]

radio_button_unchecked 32.9 Semantic conventions , Resource and attribute governance

Semantic conventions define common names, types, and meanings for attributes, span names, metrics, and units. Without these conventions, each team could register an HTTP method as method, httpMethod or verb, making corporate dashboards unfeasible. Standardization allows you to consult services written in different languages with the same logic.

Resource describes the entity that produced the telemetry. Attributes like service.name, service.version, deployment.environment.name, host, container, and Kubernetes help you locate the source. Resource should not be confused with operation attributes: service.name describes the issuer; http.route describes the request.

Not all semantic conventions have the same level of stability. Governance must record the adopted version, track experimental changes, and prevent silent renames. Tel Weaver and schema tools can help organizations validate telemetry contracts, but the discipline starts with a clear catalog of allowed attributes.

radio_button_unchecked 32.10 Sampling, controlled loss and preservation of useful traces

Sampling reduces trace volume. Head sampling decides at the beginning, using probability, parent context or local rules. It's cheap and fast, but you don't know the end result. A rare error can be ruled out before it happens. Parent-based sampling helps maintain consistency between child spans and the trace decision.

Tail sampling decides after gathering enough spans in the Collector or backend. It can preserve traces with errors, high latency, or specific attributes. On the other hand, it requires memory, waiting, consistent routing of spans of the same trace and capacity planning. A distributed deployment without trace affinity may make incomplete decisions.

The policy must reflect objectives: maintaining 100% errors, traces above SLO , representative samples per route and a small percentage of healthy traffic. Sampling does not correct poorly designed telemetry. Even discarded traces can contribute to derived metrics, depending on the point at which aggregation occurs.

radio_button_unchecked 32.11 Exemplars and correlation between metrics, logs and traces

Exemplary associates a metric observation with a representative trace or context. By observing a high latency bucket, the operator can open a real trace that contributed to that value. This navigation reduces the jump between aggregated view and individual evidence, especially in latency tail incidents.

Logs can also carry trace_id and span_id. The ideal correlation allows you to navigate from an alert to the series, from the series to an exemplar , from the exemplar to the trace , and from the span to logs of the same execution. This experience relies on consistent instrumentation, time synchronization, compliant retention, and cross-backend integration.

Correlation should not be used to copy all data for all signals. Metrics remain aggregated; traces remain selective; logs continue events. The objective is to maintain common navigation keys and semantics, preserving the economic and operational advantages of each model.

radio_button_unchecked 32.12 SLI , SLO , error budget and alerts

SLI is a measurement of observed behavior, such as the proportion of valid requests that complete successfully below a latency threshold. SLO sets the target to a window, for example 99.9% in 30 days. The error budget represents the number of failures tolerated before exceeding the target. This approach connects telemetry to user expectation.

Infrastructure-only alerts produce many false positives and false negatives. High CPU may be normal; Low CPU can coexist with complete unavailability due to DNS error. Burn rate alerts observe how quickly the error budget is consumed and can combine short and long windows to balance speed and stability.

Dashboards must follow operational questions. For APIs, the golden signals - latency, traffic, errors and saturation - are a good starting point. However, route, consumer, region, version, and backend need to be controlled dimensions. Panels without owner, SLO and associated action become decoration.

radio_button_unchecked 32.13 Observability in API Gateways, Kubernetes and messaging

API Gateway must register the edge view and the upstream view separately. Total duration, time to backend, status produced by the gateway, upstream status, policy that failed, route, consumer and trace context are different evidence. A 502 needs to indicate whether there was a DNS failure, connect timeout, TLS, reset, or invalid response from the backend.

In Kubernetes, Resources and infrastructure attributes connect the logical service to cluster, namespace, workload, pod, container, and node. Collector can enrich telemetry with environmental metadata. The design needs to tolerate ephemeral pods, rollouts, and IP changes without treating the instance as a permanent identity.

In messaging, causality may not form a simple tree. Production, storage, consumption, retries and DLQ occur at different times. Links and semantic attributes help relate messages. Lag metrics, message age, redelivery rate and queue depth complement traces, but need to be interpreted within the broker's semantics.

radio_button_unchecked 32.14 Security, privacy and integrity of telemetry

Telemetry is a sensitive asset. It can reveal internal topology, service names, versions, failures, user identifiers, and security decisions. The pipeline must use authentication, encryption in transit, tenant segregation, access controls, retention, and administration trail. Collectors should not accept data from any source without validation.

Trace context propagation crosses external borders and needs policy. Received IDs can be accepted, regenerated or related by links depending on the risk and need for correlation. Baggage must be filtered. Authentication logs need to avoid tokens and credentials; SQL or HTTP attributes should not log sensitive values by default.

Integrity also matters. An attacker may attempt to hide activity by reducing telemetry, flooding the pipeline, or injecting misleading fields. Limits, mutual authentication, schema validation, monitoring the Collector itself, and immutable storage for auditing reduce this risk.

Telemetry is not an LGPD-free area

Observational data remains subject to purpose, minimization, retention, access and security. A trace ID may not identify a person on its own, but logs and baggage often carry context that makes correlation possible.

radio_button_unchecked 32.15 Pipeline costs, retention, cardinality and capacity

Observability costs CPU, memory, network, storage, indexing and querying. The cost arises in the instrumentation and is multiplied by volume, cardinality and retention. An application that adds five high cardinality attributes can increase series and indexes much more than the traffic growth suggests.

The strategy must define retention tiers, sampling, compression, aggregation and signal routing. Debug logs may remain for a few days; aggregated metrics can have long retention; full traces can be sampled; Auditing follows its own policy. Collector allows you to filter, transform, and forward data to different destinations before paying the full cost on the backend.

The pipeline also needs SLO . Internal queues, dropped spans, export failures, memory denials, processing time and Collector CPU usage must be monitored. An observability platform that silently loses data can lead teams to incorrect conclusions during incidents.

radio_button_unchecked 32.16 Observability troubleshooting

When a trace is broken, check the propagation first. Did the client inject traceparent? Did the gateway preserve or recreate the context? Did the service library extract the header? Did asynchronous calls load the correct context? A failure at any hop creates separate traces, even if all components are instrumented.

When metrics disappear, investigate the instrument, views, temporality, export interval, filters, and cardinality. In logs, check parsing, timestamp, multiline, encoding and severity mapping. In the Collector, examine health, queues, memory limiter, batch, retries and exporter errors. The pipeline itself needs to emit telemetry to be diagnosed.

Time differences produce spans with impossible order and logs outside the window. Clock synchronization is a basic requirement. It is also common for empty or inconsistent service names to group different applications together on the backend. A checklist of Resources and semantic conventions must be part of the deployment.

radio_button_unchecked 32.17 Case studies and labs

Case Study 1: Consumers intermittently receive 504 on a banking API. The latency metric only shows increase on one route. An exemplar opens a trace whose gateway span consumes little time, but the backend span remains close to timeout. The logs from the same trace indicate an exhausted connection pool. Correlation avoids changing gateway policies unnecessarily.

Case study 2: After a rollout, the cost of metrics increases tenfold. Investigation shows that a new version added customer_id as a histogram attribute . The fix removes the identifier from the metric and preserves it only in sampled spans and authorized logs. The incident demonstrates why telemetry needs contract review.

Case study 3: Kafka messages appear as separate traces from the original request. The producer created spans, but did not inject context into the message headers. After correcting the propagation and using links in the consumer when appropriate, the platform begins to reconstruct the asynchronous journey without forcing an incorrect hierarchy.

Suggested labs

1) Instrument an API with self-instrumentation and a manual business span . 2) Propagate traceparent through the gateway and confirm trace continuity. 3) Configure a Collector with OTLP receiver, memory_limiter, batch and two exporters. 4) Create a metric with low cardinality and associate exemplars. 5) Simulate error, timeout and asynchronous message and compare the three signals.

radio_button_unchecked Chapter summary

Observability is the ability to explain the internal behavior of systems based on external signals. Logs, metrics and traces have different models and must be correlated by Resource , distributed context and semantic conventions . The quality of the correlation is more important than the quantity of data.

OpenTelemetry separates APIs, SDKs, instrumentations, OTLP protocol and Collector. This architecture enables vendor-agnostic instrumentation, centralized processing, and export to multiple backends. The Collector, however, needs to be designed as a critical component, with its own capacity, queues, security and telemetry.

Sampling, exemplars, SLOs and alerts transform signals into decisions. Cardinality, privacy and cost need to be addressed by design. In gateways, Kubernetes, and messaging, each layer produces specific evidence, but the full journey only appears when context and semantics cross all boundaries.

Next step of the course

The next chapter delves deeper into Kubernetes for APIs, connecting workloads, Services, Ingress/Gateway API, probes, autoscaling, security and operations to the observability practices presented here.

radio_button_unchecked Observability checklist

• Each service defines service.name, version, environment, and ownership consistently.
• Logs are structured, correlatable, and do not contain secrets or unnecessary personal data.
• Metrics use correct instruments and low cardinality attributes.
• Spans describe stable operations, dependencies, errors, and timing without copying entire payloads.
• Trace context is propagated across HTTP, gRPC and messaging, respecting trust boundaries.
• Semantic conventions and their stability are governed as a contract.
• Sampling preserves errors, high latency and healthy traffic representation.
• Collector has memory limiter, batch, queues, retries, health checks and internal telemetry.
• Dashboards and alerts are linked to SLIs, SLOs, owners and known actions.
• Costs, retention, cardinality, and privacy are reviewed before production.

radio_button_unchecked Exercises

• Differentiate between telemetry, monitoring, observability and auditing.
• Explain why trace_id is inappropriate as a metric label.
• Describe Resource , span , event, link, status and baggage .
• Explain traceparent and tracestate in a distributed HTTP call.
• Compare manual instrumentation and self-instrumentation.
• Design a Collector pipeline with receivers, processors and exporters.
• Compare head sampling and tail sampling .
• Explain how exemplars connect metrics and traces.
• Propose an availability SLI and a latency SLI for an API.
• List controls to prevent exposure of tokens and PII in telemetry.
• Describe how to diagnose a broken trace between gateway and backend.
• Propose a laboratory to correlate HTTP request, message and asynchronous processing.

radio_button_unchecked Glossary

radio_button_unchecked Technical references

• OpenTelemetry. What is OpenTelemetry? and Observability Primer.
• OpenTelemetry. Concepts: Signals, Traces, Metrics, Logs, Baggage and Context Propagation.
• OpenTelemetry Specification. Overview, Tracing, Metrics and Logs.
• OpenTelemetry Semantic Conventions .
• OpenTelemetry Collector: Architecture, Configuration, Deployment Patterns and Scaling.
• W3C. Trace Context Level 2.
• W3C. Baggage .
• Google SRE. Service Level Objectives and Error Budgets.
• CNCF. OpenTelemetry project documentation.

Update note

OpenTelemetry evolves sign by sign, and semantic conventions can be stable or experimental. Before standardizing attributes, exporters or declarative configuration, validate the status of the adopted version and test migrations in a controlled environment.