Aggregating Linux Systemd Logs With Vector

Aggregating Linux Systemd Logs With Vector

What You’ll Need

  • A Linux server running Ubuntu 22.04 LTS or Debian 12 hosted on a Hetzner VPS or Contabo VPS
  • Root or sudo privileges on your target server
  • An alternative server instance on DigitalOcean if running a multi-cloud configuration
  • Basic command line experience with systemd, systemctl, and journalctl

Table of Contents

Understanding Journald and Vector for Log Aggregation

Managing system logs across multiple Linux servers can quickly become a bottleneck if you rely solely on standard SSH sessions and journalctl commands. By default, systemd-journald captures stdout, stderr, kernel messages, and syslog events from every service on your machine. It stores these logs in a binary format inside the /var/log/journal/ directory. While binary indexing makes local querying with journalctl fast, extracting those logs for centralized storage, visualization, and alerting requires an efficient log shipper.

Many DevOps teams initially choose traditional tools like Filebeat or Fluentbit to tail text files in /var/log/. However, tailing text files bypasses key structured metadata that systemd-journald attaches to every entry. Metadata such as the systemd unit name, cgroup path, effective execution user, process ID, and precise microsecond-level kernel timestamps are lost unless you read directly from the native binary journal socket.

Vector, an open-source observability data pipeline built in Rust by Datadog, solves this efficiently. Vector uses high-level memory safety and multi-threaded concurrency to consume minimal CPU and RAM while parsing thousands of journal entries per second. It connects directly to the native systemd journal API, avoiding the CPU overhead of wrapping journalctl calls or converting logs to plaintext disk files before shipping.

When setting up high-performance log shipping on infrastructure, ensuring high I/O throughput for persistent storage is vital. If you are operating virtual machines in public cloud environments, consider reading our guide on optimizing container volume performance on Hetzner VPS to prevent disk write bottlenecks from slowing down Vector or systemd-journald.

Installing Vector on a Linux Server

To get started, log into your server terminal. We will set up the official Datadog Vector APT repository to pull the latest stable binary build for Debian and Ubuntu distributions.

Run the following commands to install dependencies, add the official repository key, and register the Vector package source:

sudo apt-get update -y
sudo apt-get install -y curl apt-transport-https gnupg debian-archive-keyring
curl -1sLf 'https://repositories.timber.io/public/vector/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/vector-archive-keyring.gpg
curl -1sLf 'https://repositories.timber.io/public/vector/config.deb.txt?distro=ubuntu&codename=jammy' | sudo tee /etc/apt/sources.list.d/vector.list
sudo apt-get update -y
sudo apt-get install -y vector

Once the installation finishes, verify that Vector is properly installed by checking its version string:

vector --version

Verify that the systemd service file was installed alongside the binary:

systemctl status vector

If you are using a cloud instance on a Hetzner VPS or DigitalOcean, the installation will automatically create a dedicated system service account named vector. For Vector to read native binary logs directly from /var/log/journal/, its service user must belong to the systemd-journal system group.

Add the vector service user to the systemd-journal group using this command:

sudo usermod -aG systemd-journal vector

💡 Fast-Track Your Project: Don’t want to configure this yourself? I build custom n8n pipelines and bots. Message me with code SYS3-HUGO.

Configuring Vector to Ingest Systemd Journal Logs

Vector uses a single main configuration file located at /etc/vector/vector.yaml. In Vector’s architecture, data flows through three main components:

  1. Sources: The inputs where data is collected (in our case, systemd_journal).
  2. Transforms: Processing steps where VRL (Vector Remap Language) parses, mutates, enriches, or drops incoming log structures.
  3. Sinks: Destinations where structured logs are delivered (such as Grafana Loki, ClickHouse, AWS S3, or remote HTTP endpoints).

We will create a complete /etc/vector/vector.yaml file. This configuration configures Vector to ingest systemd logs, drop high-volume unneeded internal journal fields, standardize log levels, and extract clean JSON structures.

Open /etc/vector/vector.yaml in your editor and add the complete configuration below:

data_dir: /var/lib/vector

sources:
  journal_source:
    type: journald
    current_boot_only: true
    exclude_units:
      - vector.service

transforms:
  parse_and_clean_journal:
    type: remap
    inputs:
      - journal_source
    source: |
      .timestamp = parse_timestamp!(._SOURCE_REALTIME_TIMESTAMP, "%s%6f")
      .host = to_string!(.host)
      .unit = to_string!(._SYSTEMD_UNIT)
      .message = to_string!(.message)
      .syslog_identifier = to_string!(.SYSLOG_IDENTIFIER)
      
      del(._SYSTEMD_CGROUP)
      del(._SYSTEMD_SLICE)
      del(._SYSTEMD_USER_SLICE)
      del(._SOURCE_REALTIME_TIMESTAMP)
      del(._CAP_EFFECTIVE)
      del(._TRANSPORT)
      del(._CMDLINE)
      del(._COMM)
      del(._EXE)
      del(._GIUD)
      del(._UID)
      del(._PID)
      
      if .PRIORITY == "0" {
        .level = "emergency"
      } else if .PRIORITY == "1" {
        .level = "alert"
      } else if .PRIORITY == "2" {
        .level = "critical"
      } else if .PRIORITY == "3" {
        .level = "error"
      } else if .PRIORITY == "4" {
        .level = "warning"
      } else if .PRIORITY == "5" {
        .level = "notice"
      } else if .PRIORITY == "6" {
        .level = "info"
      } else if .PRIORITY == "7" {
        .level = "debug"
      } else {
        .level = "unknown"
      }
      del(.PRIORITY)      

sinks:
  console_output:
    type: console
    inputs:
      - parse_and_clean_journal
    target: stdout
    encoding:
      codec: json

Test your new configuration file for syntax errors using Vector’s built-in validation tool:

vector validate /etc/vector/vector.yaml

If the validation returns clean without errors, restart Vector to activate the journal reader:

sudo systemctl restart vector
sudo systemctl status vector

To see Vector processing live journal entries directly in your terminal, check Vector’s output using systemctl log tailing:

sudo journalctl -u vector.service -f -n 50

Transforming and Routing Logs to Remote Sinks

Now that Vector reads binary systemd logs, parses syslog severities, and cleans up internal fields, we can route those logs to production storage systems. Common storage backends include Grafana Loki, Elasticsearch, or ClickHouse.

Vector allows you to split pipelines based on conditions. For instance, if you run automated background orchestration pipelines on your server like how to self host Kestra on cloud VPS, or containerized applications like deploying Telegram bots using Docker Compose, you can isolate their systemd logs into separate destinations or label indexes.

Below is a complete enterprise Vector configuration. It ingests systemd logs, routes core security events (like sshd.service) to a dedicated file backup sink, routes application container logs to a remote Grafana Loki server, and outputs system-wide errors to a local ClickHouse instance.

Replace the contents of /etc/vector/vector.yaml with the complete specification below:

data_dir: /var/lib/vector

sources:
  systemd_raw:
    type: journald
    current_boot_only: true
    exclude_units:
      - vector.service

transforms:
  normalize_logs:
    type: remap
    inputs:
      - systemd_raw
    source: |
      .timestamp = parse_timestamp!(._SOURCE_REALTIME_TIMESTAMP, "%s%6f")
      .host = to_string!(.host)
      .unit = to_string!(._SYSTEMD_UNIT)
      .message = to_string!(.message)
      .syslog_identifier = to_string!(.SYSLOG_IDENTIFIER)
      
      del(._SYSTEMD_CGROUP)
      del(._SYSTEMD_SLICE)
      del(._SYSTEMD_USER_SLICE)
      del(._SOURCE_REALTIME_TIMESTAMP)
      del(._CAP_EFFECTIVE)
      del(._TRANSPORT)
      del(._CMDLINE)
      
      if .PRIORITY == "0" {
        .level = "emergency"
      } else if .PRIORITY == "1" {
        .level = "alert"
      } else if .PRIORITY == "2" {
        .level = "critical"
      } else if .PRIORITY == "3" {
        .level = "error"
      } else if .PRIORITY == "4" {
        .level = "warning"
      } else if .PRIORITY == "5" {
        .level = "notice"
      } else if .PRIORITY == "6" {
        .level = "info"
      } else if .PRIORITY == "7" {
        .level = "debug"
      } else {
        .level = "unknown"
      }
      del(.PRIORITY)      

  route_by_service:
    type: route
    inputs:
      - normalize_logs
    route:
      ssh_events: '.unit == "ssh.service" || .unit == "sshd.service"'
      docker_events: '.unit == "docker.service"'
      critical_errors: '.level == "error" || .level == "critical" || .level == "alert" || .level == "emergency"'

sinks:
  loki_sink:
    type: loki
    inputs:
      - normalize_logs
    endpoint: http://loki.internal.network:3100
    encoding:
      codec: json
    labels:
      host: "{{ host }}"
      unit: "{{ unit }}"
      level: "{{ level }}"
    buffer:
      type: disk
      max_size: 1073741824
      when_full: block

  ssh_audit_file:
    type: file
    inputs:
      - route_by_service.ssh_events
    path: /var/log/vector/ssh_audit.log
    encoding:
      codec: json
    buffer:
      type: memory
      max_events: 500

  clickhouse_errors:
    type: clickhouse
    inputs:
      - route_by_service.critical_errors
    endpoint: http://clickhouse.internal.network:8123
    database: system_logs
    table: systemd_errors
    skip_unknown_fields: true
    auth:
      strategy: basic
      user: vector_ingest
      password: ProductionPassword123

Validate and test this full configuration:

vector validate /etc/vector/vector.yaml
sudo systemctl restart vector

Production Hardening and Buffer Management

Running log forwarding agents in production requires safeguards against memory starvation, network outages, and log spikes. If your downstream log receiver (such as Grafana Loki or ClickHouse) experiences downtime, an unbuffered agent could lock system resources or drop critical incident records.

1. Enabling Disk Buffers

In the configuration above, notice the buffer block added to loki_sink. By setting type: disk and defining a max_size of 1073741824 bytes (1 GB), Vector queues log batches onto disk inside /var/lib/vector if the remote receiver drops connection. Once the Loki endpoint recovers, Vector drains the disk buffer sequentially without overwhelming server RAM.

2. Hardening systemd Execution Limits

To ensure that Vector never consumes excessive system resources during log spikes, configure explicit systemd resource constraints. Edit the default Vector systemd unit file directly using systemctl edit:

sudo systemctl edit vector

Insert the following override settings:

[Service]
MemoryHigh=512M
MemoryMax=768M
CPUWeight=80
IOWeight=80
LimitNOFILE=65536
Restart=always
RestartSec=5s

Save and close the file. Apply the systemd unit changes and restart Vector:

sudo systemctl daemon-reload
sudo systemctl restart vector

Check the active execution parameters to confirm that your memory limits are enforced by kernel cgroups:

systemctl show vector --property=MemoryMax,MemoryCurrent,CPUWeight

By decoupling local log aggregation from remote network availability using Rust memory guarantees, native journal sockets, disk-backed buffers, and cgroup limits, Vector provides an reliable logging setup for Linux infrastructure.

Getting Started

To build your own automated systemd log aggregation infrastructure, select host hardware tailored for high network output and persistent disk I/O performance.

Outsource Your Automation

Don’t have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-HUGO for priority. Or DM at chasebot.online.

Want to automate this yourself?

Start with n8n Cloud (free tier available) or self-host on a Hetzner VPS for full control.

Want this engine running on your own VPS?

This blog publishes itself — daily, unattended, on free API tiers. The full engine, Hugo theme, and setup guide are available as System 3.

Get System 3
system online