Aggregating Linux Systemd Logs With Vector
What You’ll Need
- A Linux server running Ubuntu 22.04 LTS or Debian 12 hosted on a Hetzner VPS or Contabo VPS
- Root or sudo privileges on your target server
- An alternative server instance on DigitalOcean if running a multi-cloud configuration
- Basic command line experience with systemd, systemctl, and journalctl
Table of Contents
- Understanding Journald and Vector for Log Aggregation
- Installing Vector on a Linux Server
- Configuring Vector to Ingest Systemd Journal Logs
- Transforming and Routing Logs to Remote Sinks
- Production Hardening and Buffer Management
- Getting Started
Understanding Journald and Vector for Log Aggregation
Managing system logs across multiple Linux servers can quickly become a bottleneck if you rely solely on standard SSH sessions and journalctl commands. By default, systemd-journald captures stdout, stderr, kernel messages, and syslog events from every service on your machine. It stores these logs in a binary format inside the /var/log/journal/ directory. While binary indexing makes local querying with journalctl fast, extracting those logs for centralized storage, visualization, and alerting requires an efficient log shipper.
Many DevOps teams initially choose traditional tools like Filebeat or Fluentbit to tail text files in /var/log/. However, tailing text files bypasses key structured metadata that systemd-journald attaches to every entry. Metadata such as the systemd unit name, cgroup path, effective execution user, process ID, and precise microsecond-level kernel timestamps are lost unless you read directly from the native binary journal socket.
Vector, an open-source observability data pipeline built in Rust by Datadog, solves this efficiently. Vector uses high-level memory safety and multi-threaded concurrency to consume minimal CPU and RAM while parsing thousands of journal entries per second. It connects directly to the native systemd journal API, avoiding the CPU overhead of wrapping journalctl calls or converting logs to plaintext disk files before shipping.
When setting up high-performance log shipping on infrastructure, ensuring high I/O throughput for persistent storage is vital. If you are operating virtual machines in public cloud environments, consider reading our guide on optimizing container volume performance on Hetzner VPS to prevent disk write bottlenecks from slowing down Vector or systemd-journald.
Installing Vector on a Linux Server
To get started, log into your server terminal. We will set up the official Datadog Vector APT repository to pull the latest stable binary build for Debian and Ubuntu distributions.
Run the following commands to install dependencies, add the official repository key, and register the Vector package source:
sudo apt-get update -y
sudo apt-get install -y curl apt-transport-https gnupg debian-archive-keyring
curl -1sLf 'https://repositories.timber.io/public/vector/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/vector-archive-keyring.gpg
curl -1sLf 'https://repositories.timber.io/public/vector/config.deb.txt?distro=ubuntu&codename=jammy' | sudo tee /etc/apt/sources.list.d/vector.list
sudo apt-get update -y
sudo apt-get install -y vector
Once the installation finishes, verify that Vector is properly installed by checking its version string:
vector --version
Verify that the systemd service file was installed alongside the binary:
systemctl status vector
If you are using a cloud instance on a Hetzner VPS or DigitalOcean, the installation will automatically create a dedicated system service account named vector. For Vector to read native binary logs directly from /var/log/journal/, its service user must belong to the systemd-journal system group.
Add the vector service user to the systemd-journal group using this command:
sudo usermod -aG systemd-journal vector
💡 Fast-Track Your Project: Don’t want to configure this yourself? I build custom n8n pipelines and bots. Message me with code SYS3-HUGO.
Configuring Vector to Ingest Systemd Journal Logs
Vector uses a single main configuration file located at /etc/vector/vector.yaml. In Vector’s architecture, data flows through three main components:
- Sources: The inputs where data is collected (in our case,
systemd_journal). - Transforms: Processing steps where VRL (Vector Remap Language) parses, mutates, enriches, or drops incoming log structures.
- Sinks: Destinations where structured logs are delivered (such as Grafana Loki, ClickHouse, AWS S3, or remote HTTP endpoints).
We will create a complete /etc/vector/vector.yaml file. This configuration configures Vector to ingest systemd logs, drop high-volume unneeded internal journal fields, standardize log levels, and extract clean JSON structures.
Open /etc/vector/vector.yaml in your editor and add the complete configuration below:
data_dir: /var/lib/vector
sources:
journal_source:
type: journald
current_boot_only: true
exclude_units:
- vector.service
transforms:
parse_and_clean_journal:
type: remap
inputs:
- journal_source
source: |
.timestamp = parse_timestamp!(._SOURCE_REALTIME_TIMESTAMP, "%s%6f")
.host = to_string!(.host)
.unit = to_string!(._SYSTEMD_UNIT)
.message = to_string!(.message)
.syslog_identifier = to_string!(.SYSLOG_IDENTIFIER)
del(._SYSTEMD_CGROUP)
del(._SYSTEMD_SLICE)
del(._SYSTEMD_USER_SLICE)
del(._SOURCE_REALTIME_TIMESTAMP)
del(._CAP_EFFECTIVE)
del(._TRANSPORT)
del(._CMDLINE)
del(._COMM)
del(._EXE)
del(._GIUD)
del(._UID)
del(._PID)
if .PRIORITY == "0" {
.level = "emergency"
} else if .PRIORITY == "1" {
.level = "alert"
} else if .PRIORITY == "2" {
.level = "critical"
} else if .PRIORITY == "3" {
.level = "error"
} else if .PRIORITY == "4" {
.level = "warning"
} else if .PRIORITY == "5" {
.level = "notice"
} else if .PRIORITY == "6" {
.level = "info"
} else if .PRIORITY == "7" {
.level = "debug"
} else {
.level = "unknown"
}
del(.PRIORITY)
sinks:
console_output:
type: console
inputs:
- parse_and_clean_journal
target: stdout
encoding:
codec: json
Test your new configuration file for syntax errors using Vector’s built-in validation tool:
vector validate /etc/vector/vector.yaml
If the validation returns clean without errors, restart Vector to activate the journal reader:
sudo systemctl restart vector
sudo systemctl status vector
To see Vector processing live journal entries directly in your terminal, check Vector’s output using systemctl log tailing:
sudo journalctl -u vector.service -f -n 50
Transforming and Routing Logs to Remote Sinks
Now that Vector reads binary systemd logs, parses syslog severities, and cleans up internal fields, we can route those logs to production storage systems. Common storage backends include Grafana Loki, Elasticsearch, or ClickHouse.
Vector allows you to split pipelines based on conditions. For instance, if you run automated background orchestration pipelines on your server like how to self host Kestra on cloud VPS, or containerized applications like deploying Telegram bots using Docker Compose, you can isolate their systemd logs into separate destinations or label indexes.
Below is a complete enterprise Vector configuration. It ingests systemd logs, routes core security events (like sshd.service) to a dedicated file backup sink, routes application container logs to a remote Grafana Loki server, and outputs system-wide errors to a local ClickHouse instance.
Replace the contents of /etc/vector/vector.yaml with the complete specification below:
data_dir: /var/lib/vector
sources:
systemd_raw:
type: journald
current_boot_only: true
exclude_units:
- vector.service
transforms:
normalize_logs:
type: remap
inputs:
- systemd_raw
source: |
.timestamp = parse_timestamp!(._SOURCE_REALTIME_TIMESTAMP, "%s%6f")
.host = to_string!(.host)
.unit = to_string!(._SYSTEMD_UNIT)
.message = to_string!(.message)
.syslog_identifier = to_string!(.SYSLOG_IDENTIFIER)
del(._SYSTEMD_CGROUP)
del(._SYSTEMD_SLICE)
del(._SYSTEMD_USER_SLICE)
del(._SOURCE_REALTIME_TIMESTAMP)
del(._CAP_EFFECTIVE)
del(._TRANSPORT)
del(._CMDLINE)
if .PRIORITY == "0" {
.level = "emergency"
} else if .PRIORITY == "1" {
.level = "alert"
} else if .PRIORITY == "2" {
.level = "critical"
} else if .PRIORITY == "3" {
.level = "error"
} else if .PRIORITY == "4" {
.level = "warning"
} else if .PRIORITY == "5" {
.level = "notice"
} else if .PRIORITY == "6" {
.level = "info"
} else if .PRIORITY == "7" {
.level = "debug"
} else {
.level = "unknown"
}
del(.PRIORITY)
route_by_service:
type: route
inputs:
- normalize_logs
route:
ssh_events: '.unit == "ssh.service" || .unit == "sshd.service"'
docker_events: '.unit == "docker.service"'
critical_errors: '.level == "error" || .level == "critical" || .level == "alert" || .level == "emergency"'
sinks:
loki_sink:
type: loki
inputs:
- normalize_logs
endpoint: http://loki.internal.network:3100
encoding:
codec: json
labels:
host: "{{ host }}"
unit: "{{ unit }}"
level: "{{ level }}"
buffer:
type: disk
max_size: 1073741824
when_full: block
ssh_audit_file:
type: file
inputs:
- route_by_service.ssh_events
path: /var/log/vector/ssh_audit.log
encoding:
codec: json
buffer:
type: memory
max_events: 500
clickhouse_errors:
type: clickhouse
inputs:
- route_by_service.critical_errors
endpoint: http://clickhouse.internal.network:8123
database: system_logs
table: systemd_errors
skip_unknown_fields: true
auth:
strategy: basic
user: vector_ingest
password: ProductionPassword123
Validate and test this full configuration:
vector validate /etc/vector/vector.yaml
sudo systemctl restart vector
Production Hardening and Buffer Management
Running log forwarding agents in production requires safeguards against memory starvation, network outages, and log spikes. If your downstream log receiver (such as Grafana Loki or ClickHouse) experiences downtime, an unbuffered agent could lock system resources or drop critical incident records.
1. Enabling Disk Buffers
In the configuration above, notice the buffer block added to loki_sink. By setting type: disk and defining a max_size of 1073741824 bytes (1 GB), Vector queues log batches onto disk inside /var/lib/vector if the remote receiver drops connection. Once the Loki endpoint recovers, Vector drains the disk buffer sequentially without overwhelming server RAM.
2. Hardening systemd Execution Limits
To ensure that Vector never consumes excessive system resources during log spikes, configure explicit systemd resource constraints. Edit the default Vector systemd unit file directly using systemctl edit:
sudo systemctl edit vector
Insert the following override settings:
[Service]
MemoryHigh=512M
MemoryMax=768M
CPUWeight=80
IOWeight=80
LimitNOFILE=65536
Restart=always
RestartSec=5s
Save and close the file. Apply the systemd unit changes and restart Vector:
sudo systemctl daemon-reload
sudo systemctl restart vector
Check the active execution parameters to confirm that your memory limits are enforced by kernel cgroups:
systemctl show vector --property=MemoryMax,MemoryCurrent,CPUWeight
By decoupling local log aggregation from remote network availability using Rust memory guarantees, native journal sockets, disk-backed buffers, and cgroup limits, Vector provides an reliable logging setup for Linux infrastructure.
Getting Started
To build your own automated systemd log aggregation infrastructure, select host hardware tailored for high network output and persistent disk I/O performance.
- Provision hosting for your collectors on a Hetzner VPS or high-mem Contabo VPS
- Set up fallback cloud nodes on DigitalOcean for redundancy
Outsource Your Automation
Don’t have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-HUGO for priority. Or DM at chasebot.online.
Want to automate this yourself?
Start with n8n Cloud (free tier available) or self-host on a Hetzner VPS for full control.
Want this engine running on your own VPS?
This blog publishes itself — daily, unattended, on free API tiers. The full engine, Hugo theme, and setup guide are available as System 3.
Get System 3