Setting Up Alertmanager Webhook Alerts for Linux

Setting Up Alertmanager Webhook Alerts for Linux

What You’ll Need

  • A Linux server running Ubuntu 22.04 LTS or Debian 12 hosted on a Hetzner VPS or DigitalOcean
  • Prometheus and Alertmanager installed on your server
  • Python 3.10 or higher with pip installed
  • Root or sudo privileges on your Linux machine
  • Optional: An n8n Cloud instance if you want to forward webhook payloads into complex automation workflows

Table of Contents

Understanding Prometheus Alertmanager Webhooks

Monitoring infrastructure on Linux requires more than just collecting metrics. You need real time notification routing when things go wrong. Prometheus handles metrics collection and evaluates alerting rules, but it delegates notification delivery to Alertmanager. Alertmanager manages alert deduplication, grouping, silencing, and routing to various integrations like email, PagerDuty, Slack, or generic HTTP webhooks.

Generic HTTP webhooks are the most flexible routing target in Alertmanager. Instead of relying on built-in integrations, a webhook receiver allows you to send structured JSON payloads directly to custom scripts, API gateways, automated remediation pipelines, or chat tools.

When Alertmanager triggers a webhook, it sends an HTTP POST request containing a detailed JSON payload. This payload includes alert labels, annotations, status indicators (such as firing or resolved), timestamps, and generator URLs. By building a dedicated webhook handler on your server, you can trigger automated shell scripts, bounce failed services, send custom metrics to external databases, or push notifications to custom endpoints.

Configuring Prometheus and Alertmanager for Webhooks

To route alerts to an HTTP endpoint, you must configure Prometheus alert rules and set up receiver definitions inside alertmanager.yml.

First, configure your Linux server running on a Hetzner VPS to ensure Alertmanager is running. Open your Prometheus alert rules file at /etc/prometheus/alert.rules.yml and add high memory and high disk usage alert definitions:

groups:
  - name: system_alerts
    rules:
      - alert: HighMemoryUsage
        expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 > 85
        for: 2m
        labels:
          severity: critical
          category: system
        annotations:
          summary: "High Memory usage detected on {{ $labels.instance }}"
          description: "Memory usage is currently above 85% for more than 2 minutes."

      - alert: HostDiskSpaceFilling
        expr: (node_filesystem_avail_bytes * 100) / node_filesystem_size_bytes < 10
        for: 5m
        labels:
          severity: warning
          category: storage
        annotations:
          summary: "Disk space low on {{ $labels.instance }}"
          description: "Available disk space is below 10% on mount point {{ $labels.mountpoint }}."

Next, open your Alertmanager configuration file located at /etc/alertmanager/alertmanager.yml. You will define a route that sends all incoming alerts to a custom webhook receiver running on localhost port 5001.

global:
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'webhook-handler'

receivers:
  - name: 'webhook-handler'
    webhook_configs:
      - url: 'http://127.0.0.1:5001/alerts'
        send_resolved: true
        max_alerts: 0

Restart Prometheus and Alertmanager services using systemctl to apply the configuration:

sudo systemctl restart prometheus
sudo systemctl restart alertmanager

Verify that Alertmanager is running without errors:

sudo systemctl status alertmanager --no-pager

💡 Fast-Track Your Project: Don’t want to configure this yourself? I build custom n8n pipelines and bots. Message me with code SYS3-HUGO.

Building a Custom Python Webhook Receiver Service

Now that Alertmanager is configured to dispatch POST requests to http://127.0.0.1:5001/alerts, you need a backend service to parse incoming JSON payloads and execute appropriate actions.

We will write a robust Python service using Flask and standard libraries. Create a working directory at /opt/webhook-receiver and create the main script file.

sudo mkdir -p /opt/webhook-receiver
cd /opt/webhook-receiver

Install Flask using pip:

sudo pip3 install flask

Create /opt/webhook-receiver/app.py with the following full Python code:

import os
import sys
import json
import logging
import subprocess
from flask import Flask, request, jsonify

app = Flask(__name__)

logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s [%(levelname)s] %(message)s',
    handlers=[
        logging.FileHandler("/var/log/webhook-receiver.log"),
        logging.StreamHandler(sys.stdout)
    ]
)

def execute_remediation(alert_name, instance, status):
    logging.info(f"Processing remediation for alert: {alert_name}, Status: {status}, Instance: {instance}")
    
    if alert_name == "HighMemoryUsage" and status == "firing":
        logging.warning("High memory usage detected! Clearing OS page cache...")
        try:
            result = subprocess.run(
                ["sudo", "sync"], 
                check=True, 
                capture_output=True, 
                text=True
            )
            logging.info("Successfully synced filesystem caches.")
        except subprocess.CalledProcessError as err:
            logging.error(f"Failed to execute cache clearing: {err.stderr}")

    elif alert_name == "HostDiskSpaceFilling" and status == "firing":
        logging.warning("Low disk space detected! Cleaning temp files...")
        try:
            result = subprocess.run(
                ["sudo", "rm", "-rf", "/tmp/*.tmp"], 
                check=True, 
                capture_output=True, 
                text=True
            )
            logging.info("Successfully cleaned temporary files.")
        except subprocess.CalledProcessError as err:
            logging.error(f"Failed to clean temp directory: {err.stderr}")

@app.route('/alerts', methods=['POST'])
def receive_alert():
    if not request.is_json:
        logging.error("Received non-JSON request payload.")
        return jsonify({"status": "error", "message": "Payload must be JSON"}), 400

    payload = request.get_json()
    logging.info(f"Received webhook alert payload with keys: {list(payload.keys())}")

    status = payload.get('status', 'unknown')
    alerts = payload.get('alerts', [])

    logging.info(f"Notification status: {status}, Total alerts in payload: {len(alerts)}")

    for alert in alerts:
        labels = alert.get('labels', {})
        annotations = alert.get('annotations', {})
        alert_name = labels.get('alertname', 'UnknownAlert')
        instance = labels.get('instance', 'UnknownInstance')
        severity = labels.get('severity', 'info')
        summary = annotations.get('summary', 'No summary provided')

        logging.info(f"Alert: {alert_name} | Severity: {severity} | Instance: {instance} | Summary: {summary}")
        
        execute_remediation(alert_name, instance, alert.get('status', status))

    return jsonify({"status": "success", "processed_alerts": len(alerts)}), 200

if __name__ == '__main__':
    app.run(host='127.0.0.1', port=5001)

If you plan to relay parsed alert data to Telegram or another notification service, check out our guide on Deploying Telegram Bots Using Docker Compose to set up a dedicated messaging engine.

Deploying and Hardening the Receiver Service with Systemd

To make sure your custom Linux webhook receiver stays active across system reboots and restarts automatically on failure, you must run it as a systemd daemon.

First, create a non-privileged system user to run the service securely:

sudo useradd -r -s /bin/false webhookuser

Grant permissions for log files and directory execution:

sudo touch /var/log/webhook-receiver.log
sudo chown webhookuser:webhookuser /var/log/webhook-receiver.log
sudo chown -R webhookuser:webhookuser /opt/webhook-receiver

Configure sudo permissions so webhookuser can execute specific remediation commands without requiring a password prompt. Edit the sudoers file safely:

sudo visudo /etc/sudoers.d/webhookuser

Add these explicit rules:

webhookuser ALL=(ALL) NOPASSWD: /usr/bin/sync
webhookuser ALL=(ALL) NOPASSWD: /usr/bin/rm -rf /tmp/*.tmp

Next, create the systemd unit service file at /etc/systemd/system/webhook-receiver.service:

[Unit]
Description=Alertmanager Webhook Receiver Handler
After=network.target

[Service]
Type=simple
User=webhookuser
Group=webhookuser
WorkingDirectory=/opt/webhook-receiver
ExecStart=/usr/bin/python3 /opt/webhook-receiver/app.py
Restart=always
RestartSec=5s
Environment=PYTHONUNBUFFERED=1

# Security Sandbox Directives
ProtectSystem=full
ProtectHome=true
NoNewPrivileges=false

[Install]
WantedBy=multi-user.target

Reload the systemd manager daemon, enable the service to boot automatically on startup, and launch the service:

sudo systemctl daemon-reload
sudo systemctl enable webhook-receiver
sudo systemctl start webhook-receiver

Check the status of your webhook daemon:

sudo systemctl status webhook-receiver --no-pager

Testing Alert Endpoints and Ensuring Reliability

With Prometheus, Alertmanager, and your webhook receiver operating together, you should test the delivery of alerts to verify end to end routing.

You can simulate an Alertmanager alert POST request using curl directly against your server endpoint:

curl -X POST http://127.0.0.1:5001/alerts \
  -H "Content-Type: application/json" \
  -d '{
    "version": "4",
    "groupKey": "{}:{alertname=\"HighMemoryUsage\"}",
    "status": "firing",
    "receiver": "webhook-handler",
    "groupLabels": {
      "alertname": "HighMemoryUsage"
    },
    "commonLabels": {
      "alertname": "HighMemoryUsage",
      "severity": "critical",
      "instance": "vps-linux-node-01"
    },
    "commonAnnotations": {
      "summary": "High Memory usage detected on vps-linux-node-01"
    },
    "externalURL": "http://alertmanager.local:9093",
    "alerts": [
      {
        "status": "firing",
        "labels": {
          "alertname": "HighMemoryUsage",
          "severity": "critical",
          "instance": "vps-linux-node-01"
        },
        "annotations": {
          "summary": "High Memory usage detected on vps-linux-node-01",
          "description": "Memory usage is currently above 85% for more than 2 minutes."
        },
        "startsAt": "2026-03-30T10:00:00Z",
        "endsAt": "0001-01-01T00:00:00Z",
        "generatorURL": "http://prometheus.local:9090/graph"
      }
    ]
  }'

Verify that the payload was received and handled by viewing your log files:

sudo tail -n 20 /var/log/webhook-receiver.log

When building automated webhook processing endpoints operating across distributed networks, reliability is a major concern. Networks can drop connections, third party API rates can be exceeded, and services can crash.

To protect your system against cascading down-time, consider implementing resilience strategies. You can read our guide on Implementing Circuit Breakers for Outbound Webhook Delivery to prevent your system from overloading failing external services. Additionally, if your receiver triggers high-volume API requests, balance payload traffic by Implementing Token Bucket Rate Limiting With Redis to prevent rate limit exceptions on destination servers.

Getting Started

Building an automated Linux alert management pipeline with Prometheus and Alertmanager ensures fast notification delivery and self-healing systems.

To get your setup running:

  1. Spin up a cloud server instance on a Hetzner VPS or DigitalOcean.
  2. Configure Prometheus and Alertmanager with your standard rules and routes.
  3. Deploy the custom Python webhook handler script as a systemd service.
  4. Integrate downstream automation pipelines using self-hosted automation or n8n Cloud.

Outsource Your Automation

Don’t have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-HUGO for priority. Or DM at chasebot.online.

Want to automate this yourself?

Start with n8n Cloud (free tier available) or self-host on a Hetzner VPS for full control.

Want this engine running on your own VPS?

This blog publishes itself — daily, unattended, on free API tiers. The full engine, Hugo theme, and setup guide are available as System 3.

Get System 3
system online