Centralizing Docker Container Logs With Grafana Loki

What You’ll Need

To follow this guide step by step, you will need the following infrastructure and tools:

• A virtual private server running Ubuntu 22.04 or 24.04. You can spin up an instance on Hetzner VPS or Contabo VPS. If you prefer another cloud provider, DigitalOcean is a great alternative.
• A domain or subdomain pointed to your instance using Namecheap if you plan to expose Grafana to your network with TLS.
• Docker Engine and Docker Compose V2 installed on your server.
• An optional n8n Cloud instance or self-hosted n8n pipeline if you want to automate notifications based on log alerts.

Table of Contents

• Understanding the Grafana Loki Architecture
• Deploying Grafana, Loki, and Promtail with Docker Compose
• Configuring Promtail for Docker Log Collection
• Querying and Parsing Structured JSON Logs with LogQL
• Configuring Loki Alert Rules and Grafana Dashboards
• Getting Started

Understanding the Grafana Loki Architecture

When managing microservices, SSHing into individual nodes to run docker logs -f container_name quickly becomes unsustainable. Elasticsearch, Logstash, and Kibana (the ELK stack) used to be the default solution for log aggregation. However, indexing every word in every log line requires massive amounts of RAM and storage.

Grafana Loki takes a fundamentally different approach. Inspired by Prometheus, Loki does not index the full text of your log messages. Instead, it indexes only the metadata labels assigned to your log streams, such as the environment, service name, or container ID. The actual raw log chunks are compressed and stored in object storage or local disk space. This design reduces memory usage by up to 80 percent compared to full text indexing engines.

The central logging stack consists of three distinct components:

1. Promtail: The log collection agent that runs on your Docker hosts. It discovers running containers, tails their stdout and stderr streams, attaches label metadata, and pushes the log streams to Loki via HTTP.
2. Loki: The central indexing and ingestion engine. It accepts log payloads from Promtail, organizes them into compressed chunks by stream labels, and handles LogQL query requests.
3. Grafana: The visualization frontend that connects to Loki as a data source, allowing you to query, analyze, and build alerting dashboards from your container logs.

By combining these three tools, you get a lightweight observability stack capable of ingesting thousands of log lines per second without breaking your server’s budget.

Deploying Grafana, Loki, and Promtail with Docker Compose

Let’s build a central logging node on a Hetzner VPS or a Contabo VPS instance. Create a new directory named loki-stack on your server and navigate into it:

mkdir -p ~/loki-stack
cd ~/loki-stack

Inside this directory, we will create three files: docker-compose.yml, loki-config.yaml, and promtail-config.yaml.

First, let’s create the loki-config.yaml file. This file controls Loki’s internal ports, storage directories, retention periods, and chunk configuration:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

Next, save the following complete docker-compose.yml configuration to bring up Loki, Promtail, Grafana, and an Nginx container to test log streaming:

version: "3.8"

networks:
  logging:
    driver: bridge

services:
  loki:
    image: grafana/loki:2.9.2
    container_name: loki
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/loki-config.yaml
    volumes:
      - ./loki-config.yaml:/etc/loki/loki-config.yaml
      - loki-data:/tmp/loki
    networks:
      - logging
    restart: unless-stopped

  promtail:
    image: grafana/promtail:2.9.2
    container_name: promtail
    volumes:
      - ./promtail-config.yaml:/etc/promtail/promtail-config.yaml
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock
    command: -config.file=/etc/promtail/promtail-config.yaml
    networks:
      - logging
    restart: unless-stopped

  grafana:
    image: grafana/grafana:10.2.0
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=adminpassword123
    volumes:
      - grafana-data:/var/lib/grafana
    networks:
      - logging
    restart: unless-stopped

  nginx-app:
    image: nginx:latest
    container_name: nginx-app
    ports:
      - "8080:80"
    networks:
      - logging
    restart: unless-stopped

volumes:
  loki-data:
  grafana-data:

💡 Fast-Track Your Project: Don’t want to configure this yourself? I build custom n8n pipelines and bots. Message me with code SYS3-HUGO.

Configuring Promtail for Docker Log Collection

Promtail relies on Docker service discovery to automatically extract container metadata directly from /var/run/docker.sock. By pointing Promtail to the Docker socket, it automatically detects container startups and shutdowns without requiring any changes to your application code.

Create the promtail-config.yaml file in your working directory with the exact structure below:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'
      - source_labels: ['__meta_docker_container_log_stream']
        target_label: 'stream'
    pipeline_stages:
      - json:
          expressions:
            output: log
            stream: stream
            timestamp: time
      - timestamp:
          source: timestamp
          format: RFC3339Nano
      - output:
          source: output

Notice how docker_sd_configs queries the daemon socket, extracts the container name from __meta_docker_container_name, strip the leading slash, and applies it as a label named container.

If you decide to deploy Loki on a standalone centralized server on DigitalOcean while shipping logs from edge web servers across public networks, ensure you secure your log ingestion endpoints. You can protect your Loki ingress using an Nginx reverse proxy configured with mTLS. For a full walkthrough on client certificates, see our guide on How to Implement Mutual TLS Authentication with Nginx.

Start your stack by executing:

docker compose up -d

Verify that all four containers are running:

docker compose ps

You should see loki, promtail, grafana, and nginx-app marked as healthy or running.

Querying and Parsing Structured JSON Logs with LogQL

LogQL is Loki’s query language. It consists of two components: a stream selector that uses metadata labels, and optional log pipeline expressions to parse, transform, and filter log contents.

Access Grafana in your web browser at http://your-server-ip:3000. Log in using admin and adminpassword123. Navigate to Connections > Data Sources > Add Data Source, select Loki, and set the HTTP URL to http://loki:3100. Click Save & Test.

Now go to the Explore tab in Grafana.

To select all logs emitted by our test Nginx container, enter this basic stream selector:

{container="nginx-app"}

If your containers produce structured JSON logs, Loki allows you to unpack key-value pairs on the fly. When building microservices, such as workflows designed for Building Reliable Structured Output Pipelines with OpenAI, logging structured payloads is crucial for tracing model decisions. Similarly, if your application is busy Parsing Inbound Support Tickets with OpenAI, raw output logs often include JSON fields like ticket_id, status, and latency.

Suppose an application logs the following JSON payload:

{"level":"error","service":"ticket-parser","ticket_id":"8942","message":"Failed to parse incoming payload","execution_time_ms":420}

You can parse this log stream and extract individual fields into dynamic query labels using the | json parser:

{container="app-service"} | json | level="error"

To calculate the per-second rate of error logs over 5-minute rolling windows across all containers, use the following aggregation query:

sum(rate({container=~".+"} |= "error" [5m])) by (container)

To compute the 99th percentile execution time from your extracted fields:

quantile_over_time(0.99, {container="app-service"} | json | unwrap execution_time_ms [10m]) by (service)

Configuring Loki Alert Rules and Grafana Dashboards

Loki features a built-in Ruler component that evaluates LogQL queries periodically and forwards active alerts directly to Alertmanager or external webhooks.

To configure alert rules, add an alerts.yaml file to your server directory:

groups:
  - name: docker_alerts
    rules:
      - alert: HighContainerErrorRate
        expr: |
          sum(rate({container=~".+"} |= "error" [5m])) by (container) > 5          
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "High error log rate on container {{ $labels.container }}"
          description: "Container {{ $labels.container }} logged more than 5 errors per second over 5 minutes."

To enable rule evaluation, update your loki-config.yaml file to point the ruler to this file:

ruler:
  storage:
    type: local
    local:
      directory: /tmp/loki/rules
  rule_path: /tmp/loki/rules-temp
  alertmanager_url: http://localhost:9093
  ring:
    kvstore:
      store: inmemory

Mount this alerts file in your docker-compose.yml under the Loki service volume section:

    volumes:
      - ./loki-config.yaml:/etc/loki/loki-config.yaml
      - ./alerts.yaml:/tmp/loki/rules/fake/alerts.yaml
      - loki-data:/tmp/loki

Restart Loki to apply the new alert definitions:

docker compose restart loki

Inside Grafana, you can create dashboard panels based on these LogQL queries. Add a time-series panel that maps log volume by stream, or create a log volume heatmap panel to visualize log spikes across your entire infrastructure.

Getting Started

Centralizing logs using Grafana Loki and Promtail keeps your server resources dedicated to your core applications while providing instant visibility into microservice health. You can scale this architecture across multiple VPS instances by running Promtail on every worker machine and shipping logs to a master Loki database secured behind mTLS.

To get started today, set up your compute node on Hetzner VPS or DigitalOcean, register your domain through Namecheap, and connect your alert triggers to an instance of n8n Cloud.

Outsource Your Automation

Don’t have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-HUGO for priority. Or DM at chasebot.online.