Configuring Grafana Loki For Centralized Server Logging

What You’ll Need

• Hetzner VPS or Contabo VPS for hosting
• Namecheap if domain needed
• DigitalOcean as alternative

Table of Contents

• Architecture Overview: Loki, Promtail, and Grafana
• Step 1: Deploying Loki and Grafana via Docker Compose
• Step 2: Configuring Promtail for System and Application Logs
• Step 3: Querying Logs in Grafana and Securing the Stack
• Getting Started

Architecture Overview: Loki, Promtail, and Grafana

Managing log collection across multiple distributed nodes can quickly become overwhelming if you rely on SSHing into individual instances to inspect raw text files. Traditional centralized logging tools like Elasticsearch, Logstash, and Kibana (ELK) provide deep search capabilities, but they require massive amounts of RAM and storage because they index the full text content of every log event.

Grafana Loki takes a different approach designed specifically for efficient, low-overhead operations. Inspired by Prometheus, Loki does not index the message content of the logs. Instead, it indexes only the labels and metadata associated with each log stream. This design choice dramatically reduces memory footprint and storage costs, allowing you to run a production-grade centralized logging stack on affordable cloud instances.

The logging stack consists of three main components:

1. Promtail: The agent deployed on target servers. Promtail discovers log files on local disks, attaches structured key-value labels, and streams the log entries to the Loki endpoint.
2. Loki: The central datastore and indexing engine. Loki accepts log streams from Promtail agents, aggregates them into chunk files, stores them in local disk or object storage, and processes search queries.
3. Grafana: The visualization layer. Grafana connects to Loki as a data source, providing an intuitive web interface for querying logs using LogQL, building monitoring dashboards, and dispatching alerts.

When I set up cloud infrastructure for monitoring workflows, I prefer pairing lightweight telemetry tools directly with target server configurations. For a deeper look into host orchestration strategies, check out my guide on Deploying Open Source Workflow Systems On Hetzner.

Step 1: Deploying Loki and Grafana via Docker Compose

To get started, we will deploy Loki and Grafana using Docker Compose on a central server. If you need a stable host, launching a Hetzner VPS or DigitalOcean instance with 2 vCPUs and 4 GB RAM will give you plenty of headroom for high log ingest rates.

First, create a dedicated project directory structure on your server:

mkdir -p /opt/loki-stack/config
mkdir -p /opt/loki-stack/loki-data
mkdir -p /opt/loki-stack/grafana-data
cd /opt/loki-stack

Next, create the primary configuration file for Loki at /opt/loki-stack/config/loki-config.yaml. This configuration specifies internal server ports, storage mechanisms, retention periods, and chunk parameters:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-05-15
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: true
  retention_period: 336h

Now create the docker-compose.yml file inside /opt/loki-stack/docker-compose.yml to launch both Loki and Grafana:

version: "3.8"

services:
  loki:
    image: grafana/loki:2.9.4
    container_name: loki
    ports:
      - "3100:3100"
    volumes:
      - ./config/loki-config.yaml:/etc/loki/loki-config.yaml
      - ./loki-data:/tmp/loki
    command: -config.file=/etc/loki/loki-config.yaml
    restart: unless-stopped
    networks:
      - logging

  grafana:
    image: grafana/grafana:10.2.3
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=ChangeThisSecurePassword123!
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - ./grafana-data:/var/lib/grafana
    restart: unless-stopped
    networks:
      - logging

networks:
  logging:
    driver: bridge

Start the logging server infrastructure with Docker Compose:

docker compose up -d

Verify that both containers are running properly:

docker compose ps

💡 Fast-Track Your Project: Don’t want to configure this yourself? I build custom n8n pipelines and bots. Message me with code SYS3-HUGO.

Step 2: Configuring Promtail for System and Application Logs

With Loki listening for incoming log chunks on port 3100, we now need to configure Promtail to aggregate log streams from our target machines. Promtail can run directly on the host OS as a native systemd service, which provides direct access to system log files located in /var/log and the systemd journal.

First, download and extract the Promtail binary on your client server:

cd /tmp
wget https://github.com/grafana/loki/releases/download/v2.9.4/promtail-linux-amd64.zip
unzip promtail-linux-amd64.zip
mv promtail-linux-amd64 /usr/local/bin/promtail
chmod 755 /usr/local/bin/promtail

Next, create dedicated system users and configuration directories for Promtail:

useradd --system --no-create-home promtail
usermod -a -G adm promtail
mkdir -p /etc/promtail
mkdir -p /var/lib/promtail
chown promtail:promtail /var/lib/promtail

Create the full Promtail configuration file at /etc/promtail/promtail-config.yaml. Make sure to update the url field under clients to point to the IP address or hostname of your central Loki deployment server:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /var/lib/promtail/positions.yaml

clients:
  - url: http://192.168.1.100:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          host: node-01
          __path__: /var/log/*.log

  - job_name: syslog
    static_configs:
      - targets:
          - localhost
        labels:
          job: syslog
          host: node-01
          __path__: /var/log/syslog

  - job_name: authlog
    static_configs:
      - targets:
          - localhost
        labels:
          job: auth
          host: node-01
          __path__: /var/log/auth.log

Now, create a systemd unit service file so that Promtail runs automatically in the background and restarts across system reboots. Create /etc/systemd/system/promtail.service:

[Unit]
Description=Promtail Log Collector Service
After=network.target

[Service]
Type=simple
User=promtail
Group=promtail
ExecStart=/usr/local/bin/promtail -config.file=/etc/promtail/promtail-config.yaml
Restart=on-failure
RestartSec=5s
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Set the proper ownership permissions, reload systemd daemon configurations, and start the Promtail service:

chown promtail:promtail /etc/promtail/promtail-config.yaml
systemctl daemon-reload
systemctl enable promtail
systemctl start promtail

To confirm that Promtail is successfully scraping files and forwarding records, inspect its service logs using journalctl:

journalctl -u promtail -f -n 50

Step 3: Querying Logs in Grafana and Securing the Stack

Now that Promtail is streaming host logs to Loki, open your web browser and navigate to http://<your-server-ip>:3000. Log into Grafana using the administrator credentials set in your Docker Compose configuration file.

Connecting Grafana to Loki

1. Open the left sidebar menu in Grafana, navigate to Connections, and click Data sources.
2. Click the Add data source button and select Loki from the list of available source types.
3. In the HTTP settings section, enter http://loki:3100 into the URL text input field.
4. Scroll to the bottom of the page and click Save & test. You should see a success banner confirming that Grafana successfully connected to the Loki server API.

Querying Logs with LogQL

Navigate to the Explore panel using the main navigation bar. Select Loki as your active data source in the top left selector. You can now execute targeted log search queries using LogQL (Loki Query Language).

To fetch all logs categorized under the syslog job stream:

{job="syslog"}

To search for specific error events across system authorization logs:

{job="auth"} |= "Failed password"

If you are monitoring specialized JSON application output, such as API interactions generated when Building Reliable Structured Output Pipelines with OpenAI, Loki can automatically parse the structured JSON payload without manual string splitting:

{job="varlogs"} | json | status >= 500

Securing the Infrastructure

By default, Loki exposes port 3100 without built-in authentication layers. Leaving port 3100 open to the public internet creates a security vulnerability where unauthenticated parties could read internal log entries or pollute your datastore with spoofed log streams.

To protect your installation, configure firewall access rules to block public incoming connections to port 3100, permitting access only from trusted host IPs or internal VPN subnets. For step-by-step firewall setup procedures, check out my full post on Configuring UFW Firewall Rules On Linux Servers.

Execute the following commands on your central logging node to restrict access to port 3100:

ufw default deny incoming
ufw default allow outgoing
ufw allow 22/tcp
ufw allow 80/tcp
ufw allow 443/tcp
ufw allow 3000/tcp
ufw allow from 192.168.1.0/24 to any port 3100 proto tcp
ufw enable

With firewall rules active, only trusted hosts on the 192.168.1.0/24 local network segment can send log records to Loki on port 3100, while your Grafana web board remains accessible on port 3000.

Getting Started

Building a centralized logging architecture with Grafana Loki and Promtail provides comprehensive visibility into server infrastructure while keeping memory and storage overhead to a minimum. By indexing metadata labels instead of full plain text, Loki enables fast query responses across multiple nodes using lightweight cloud infrastructure such as a Hetzner VPS or DigitalOcean droplet.

To get started, launch the Loki and Grafana stack using the provided Docker Compose configuration, install Promtail on your client nodes to capture syslog and system logs, and secure your network endpoints with UFW rules.

Outsource Your Automation

Don’t have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-HUGO for priority. Or DM at chasebot.online.