Observability and Monitoring for Containers

In modern DevOps practices, observability and monitoring are indispensable for maintaining the health, performance, and reliability of applications, especially when they are containerized. Containers create a dynamic and ephemeral environment, making traditional monitoring approaches less effective. Fortunately, tools like Prometheus, Grafana, and Jaeger make it easier to achieve effective observability and monitoring for containerized applications.

In this article, we will provide a guide on how to implement observability and monitoring solutions for containerized applications using Prometheus, Grafana, and Jaeger.

1. Why Observability and Monitoring Matter

Observability and monitoring allow you to:

• Identify Issues Early: Detect infrastructure and application-level issues before they affect users.

• Debug and Troubleshoot: Provide insights and data necessary to pinpoint the root cause of problems.

• Measure Performance: Understand application performance and make informed decisions to optimize it.

• Ensure Reliability: Maintain the overall health and availability of your services.

In containerized environments, applications are broken down into microservices, each running in its own container, often orchestrated by Kubernetes. This architecture demands advanced observability solutions to effectively manage and monitor these distributed services.

2. Introduction to Prometheus, Grafana, and Jaeger

Prometheus

Prometheus is an open-source systems monitoring and alerting toolkit. Key features include:

• A multi-dimensional data model with time series data identified by metric name and key/value pairs.

• A flexible query language (PromQL) to use this dimensionality.

• Pull-based metrics collection over HTTP.

• Many integrations with various data sources and exporters.

Grafana

Grafana is an open-source platform for monitoring and observability. It allows you to:

• Query, visualize, and alert on metrics from different data sources, including Prometheus.

• Create, explore, and share dashboards in real-time.

• Set up alerts to notify of performance issues.

Jaeger

Jaeger is an open-source, end-to-end distributed tracing tool. It helps with:

• Monitoring and troubleshooting microservices-based distributed systems.

• Finding latency and performance bottlenecks.

• Visualizing request flows in complex microservice environments.

3. Setting Up Prometheus for Container Monitoring

Let’s start by setting up Prometheus to monitor container metrics.

Step 1: Install Prometheus

Assuming you are using Docker, create a prometheus.yml configuration file:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'docker'
    static_configs:
      - targets: ['172.17.0.1:9323']
  - job_name: 'cadvisor'
    static_configs:
      - targets: ['172.17.0.1:8080']

Here, we are adding Prometheus itself, the Docker daemon, and cAdvisor as scrape targets. The Docker daemon only reports metrics about itself (engine health, build counts). Per-container CPU, memory, and network metrics come from cAdvisor, which Step 3 sets up.

Step 2: Run Prometheus in a Docker Container

docker run -d --name=prometheus -p 9090:9090 \
  -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus

This command runs Prometheus in a Docker container and binds it to port 9090. The configuration file is mapped into the container.

Step 3: Exposing Docker Metrics

We need the Docker daemon to expose its own metrics. If you’re using Docker Desktop, the steps differ (edit the Docker Engine settings instead). On Linux, add this to /etc/docker/daemon.json and restart Docker:

{
  "metrics-addr": "172.17.0.1:9323"
}

Binding to the docker0 bridge address lets the Prometheus container reach it without exposing the port to your whole network. Binding to 0.0.0.0 works too but opens port 9323 to every interface, so firewall it. See the Docker daemon Prometheus docs.

Then run cAdvisor for the per-container metrics that the PromQL query in the Grafana section depends on:

docker run -d --name=cadvisor -p 8080:8080 \
  -v /:/rootfs:ro -v /var/run:/var/run:ro -v /sys:/sys:ro \
  -v /var/lib/docker/:/var/lib/docker:ro \
  ghcr.io/google/cadvisor:v0.60.6

cAdvisor images moved from gcr.io/cadvisor/cadvisor to ghcr.io/google/cadvisor with v0.53.0, and the old registry stops at v0.55.1. The cAdvisor and Prometheus post has a fuller Compose setup.

Step 4: Verify Prometheus Setup

Visit http://localhost:9090 in your browser to access the Prometheus interface. You should be able to see metrics being scraped.

4. Visualizing Metrics with Grafana

Grafana allows you to create dashboards that visualize metrics collected by Prometheus.

Step 1: Install Grafana

Run the following command to start Grafana in a Docker container:

docker run -d --name=grafana -p 3000:3000 grafana/grafana

Step 2: Configure Grafana to Use Prometheus

• Open Grafana by navigating to http://localhost:3000.

• Log in with the default username admin and password admin. Grafana prompts you to change it on first login, so do.

• Go to Connections -> Data sources and add Prometheus as a data source.

• Configure the HTTP URL to http://prometheus:9090. Container names only resolve on a user-defined Docker network (or inside Compose, shown below). With plain docker run on the default bridge, use http://172.17.0.1:9090 instead.

Step 3: Create a Dashboard

• Go to Dashboards -> New -> New dashboard.

• Add a new panel and select Prometheus as the data source.

• Enter a PromQL query, for example, sum(rate(container_cpu_usage_seconds_total[1m])) by (container).

Grafana will visualize the data in a graph, allowing you to monitor CPU usage across containers in real-time.

5. Distributed Tracing with Jaeger

Jaeger enables you to trace requests through various microservices, giving visibility into performance and latency issues.

Step 1: Install Jaeger

Run the following command to start Jaeger in a Docker container:

docker run -d --name=jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  cr.jaegertracing.io/jaegertracing/jaeger:2.21.0

Jaeger 2.x speaks OTLP natively (the 4317/4318 ports above are the gRPC and HTTP OTLP receivers), so you don’t need the pile of legacy agent/Zipkin ports the old guides made you copy-paste. The old jaegertracing/all-in-one image is the v1 line and stopped at 1.76, so use the jaegertracing/jaeger image instead. Grab the current tag from the getting started page.

Step 2: Instrument Your Application

Heads up: the old jaeger_client and flask_opentracing libraries were archived back in 2022, Jaeger told everyone to migrate to OpenTelemetry and then sunset its own clients. So don’t reach for those anymore. Use the OpenTelemetry SDK and point it at Jaeger’s OTLP endpoint instead. For example, in a Python Flask app:

from flask import Flask
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor

provider = TracerProvider(resource=Resource.create({"service.name": "my-service"}))
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://jaeger:4317", insecure=True))
)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)

app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)

@app.route('/')
def home():
    with tracer.start_as_current_span('home-span') as span:
        span.add_event('home-page')
        return 'Hello, World!'

if __name__ == '__main__':
    app.run(host='0.0.0.0')

Step 3: View Traces

Open Jaeger’s UI by navigating to http://localhost:16686. You should be able to see traces and inspect the details of each request, including any performance bottlenecks.

6. Putting it All Together: A Practical Example

We will now combine Prometheus, Grafana, and Jaeger to monitor a simple containerized microservices application.

Step 1: Docker Compose Setup

Create a docker-compose.yml file:

services:
  prometheus:
    image: prom/prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

  cadvisor:
    image: ghcr.io/google/cadvisor:v0.60.6
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    ports:
      - "8080:8080"

  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"

  jaeger:
    image: cr.jaegertracing.io/jaegertracing/jaeger:2.21.0
    ports:
      - "16686:16686"
      - "4317:4317"
      - "4318:4318"

Add any application services and configure them to expose Prometheus metrics and Jaeger traces.

Step 2: Run Docker Compose

In your terminal, navigate to the directory containing docker-compose.yml and run:

docker compose up -d

This command will start Prometheus, cAdvisor, Grafana, Jaeger, and your application services. In this Compose setup, change the cAdvisor scrape target in prometheus.yml to cadvisor:8080 and the Jaeger endpoint in your app to http://jaeger:4317.

Step 3: Set Up Dashboards and Alerts in Grafana

• Add Prometheus as a data source as described earlier.

• Create dashboards to visualize key metrics.

• Configure alerts to notify you of performance or availability issues.

Step 4: Collect Traces

Ensure your application services are instrumented with Jaeger. Use Jaeger’s UI to view traces across services.

7. Conclusion

Observability and monitoring keep containerized applications healthy and performing well. By using Prometheus, Grafana, and Jaeger, you can gain deep insights into your systems, detect issues early, and ensure smooth operation.

With Prometheus, you can collect and query metrics efficiently. Grafana allows you to visualize these metrics and set up alerts. Jaeger provides powerful distributed tracing capabilities to troubleshoot request flows and performance issues.

By combining these tools, you can achieve full observability for your containerized applications, helping you maintain high reliability and performance standards.

Related Reading

• Uptime Monitoring with Uptime Kuma
• Access Docker socket via TCP
• Docker Compose: Orchestrating Multi-Container Applications
• Copying Files Between Docker Containers and Host Machines
• WordPress, Docker, NGINX, and MySQL via Ansible