How to Self Host Kestra on Cloud VPS

What You’ll Need

• Hetzner VPS or DigitalOcean (Ubuntu 22.04 LTS, minimum 4GB RAM and 2 vCPUs recommended)
• Contabo VPS as a high-memory alternative host
• Namecheap to purchase and manage your domain name and DNS records
• Basic familiarity with SSH, Docker, and command-line system administration

Table of Contents

• Server Provisioning and Initial Hardening
• Installing Docker and Configuring Resource Limits
• Configuring PostgreSQL and Kestra Backend
• Deploying Kestra with Docker Compose and Traefik
• Writing Your First Kestra Workflow
• Getting Started

Server Provisioning and Initial Hardening

Kestra is an open source event-driven orchestrator built to manage complex data pipelines, automation scripts, and scheduled jobs. While lightweight tools like Airflow require complex celery workers and queue infrastructure, Kestra consolidates its scheduler, worker, and executor logic into a streamlined Java runtime.

To host Kestra reliably, start by spinning up an Ubuntu 22.04 instance on Hetzner VPS or DigitalOcean. A server with 4GB RAM gives you plenty of headroom to run the Kestra engine alongside PostgreSQL and a Traefik reverse proxy.

Once your server instance is active, connect to your server over SSH as the root user:

ssh root@your_server_ip

First, update system packages to their latest patches:

sudo apt update && sudo apt upgrade -y

Next, create a dedicated non-root user with sudo permissions for running services:

adduser deploy
usermod -aG sudo deploy
rsync --archive --chown=deploy:deploy ~/.ssh /home/deploy

Configure Uncomplicated Firewall (UFW) to allow SSH connections, HTTP traffic, and HTTPS traffic while blocking all other incoming ports:

ufw default deny incoming
ufw default allow outgoing
ufw allow 22/tcp
ufw allow 80/tcp
ufw allow 443/tcp
ufw enable

Before running production orchestrators on open networks, ensure you follow our guide on Hardening SSH Server Security on Ubuntu VPS to disable root logins, alter key parameters, and prevent unauthorized terminal access.

💡 Fast-Track Your Project: Don’t want to configure this yourself? I build custom n8n pipelines and bots. Message me with code SYS3-HUGO.

Installing Docker and Configuring Resource Limits

Kestra runs seamlessly in containerized environments. We will install Docker Engine and the Docker Compose plugin directly from the official Docker repository to ensure access to modern features and stability patches.

Switch to your newly created non-root deploy user:

su - deploy

Execute the following terminal commands to install the required dependencies and import Docker GPG keys:

sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg lsb-release

sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

Grant your deploy user access to the Docker daemon group so you can execute docker commands without sudo:

sudo usermod -aG docker deploy
newgrp docker

When running complex automation workflows that execute Python, Node, or custom shell operations within container instances, orchestrators can quickly consume unallocated CPU cores and host RAM. Take a look at our complete guide on How to Restrict Docker Container Resource Usage to limit container resources and protect your host environment from memory leaks or CPU throttling.

Configuring PostgreSQL and Kestra Backend

Kestra supports multiple storage and queue engines. For single instance self-hosted setups, PostgreSQL acts as both the state storage layer and the queue mechanism through database pooling.

Create a working directory on your host system to store Kestra runtime configurations, dynamic logs, and database mounts:

mkdir -p ~/kestra-stack/config
mkdir -p ~/kestra-stack/postgres_data
mkdir -p ~/kestra-stack/traefik
cd ~/kestra-stack

Now, construct the explicit Kestra application configuration file named config/application.yml. This configuration defines database connectivity parameters, internal queue handlers, security configuration options, and plugin execution behaviors:

kestra:
  repository:
    type: postgres
  queue:
    type: postgres
  storage:
    type: local
    local:
      base-path: "/tmp/kestra-workdir/storage"

datasources:
  postgres:
    url: jdbc:postgresql://postgres:5432/kestra
    driver-class-name: org.postgresql.Driver
    username: kestra_user
    password: SuperSecretPassword123!
    maximum-pool-size: 20
    minimum-idle: 5

server:
  basic-auth:
    enabled: true
    username: admin@example.com
    password: StrongAdminPassword456!

logger:
  levels:
    org.kestra: INFO

If you plan on running hundreds of simultaneous workflow tasks hitting database connections concurrently, consider reading Configuring PgBouncer Connection Pooling for PostgreSQL to prevent database socket exhaustion and reduce overhead on PostgreSQL backend instances.

Deploying Kestra with Docker Compose and Traefik

We will orchestrate three main services using Docker Compose: Traefik (for automatic SSL certificates via Let’s Encrypt), PostgreSQL (state storage), and Kestra (the orchestration server).

First, create a network configuration file for Traefik to handle incoming requests securely. Create traefik/traefik.yml with the following explicit runtime settings:

api:
  dashboard: false

entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
  websecure:
    address: ":443"

certificatesResolvers:
  letsencrypt:
    acme:
      email: your-email@example.com
      storage: /letsencrypt/acme.json
      httpChallenge:
        entryPoint: web

providers:
  docker:
    exposedByDefault: false

Now, create an empty state storage file for Let’s Encrypt SSL certificates and adjust access permissions:

touch ~/kestra-stack/traefik/acme.json
chmod 600 ~/kestra-stack/traefik/acme.json

Create the full unified docker-compose.yml file in ~/kestra-stack/docker-compose.yml:

version: "3.8"

networks:
  kestra-network:
    driver: bridge

services:
  traefik:
    image: traefik:v2.10
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - kestra-network
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./traefik/traefik.yml:/traefik.yml:ro
      - ./traefik/acme.json:/letsencrypt/acme.json

  postgres:
    image: postgres:15-alpine
    container_name: kestra-postgres
    restart: unless-stopped
    networks:
      - kestra-network
    environment:
      POSTGRES_DB: kestra
      POSTGRES_USER: kestra_user
      POSTGRES_PASSWORD: SuperSecretPassword123!
    volumes:
      - ./postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U kestra_user -d kestra"]
      interval: 5s
      timeout: 5s
      retries: 5

  kestra:
    image: kestra/kestra:latest
    container_name: kestra-engine
    restart: unless-stopped
    networks:
      - kestra-network
    command: server standalone
    depends_on:
      postgres:
        condition: service_healthy
    volumes:
      - ./config/application.yml:/app/configuration.yml:ro
      - /var/run/docker.sock:/var/run/docker.sock
      - /tmp/kestra-workdir:/tmp/kestra-workdir
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.kestra.rule=Host(`kestra.yourdomain.com`)"
      - "traefik.http.routers.kestra.entrypoints=websecure"
      - "traefik.http.routers.kestra.tls.certresolver=letsencrypt"
      - "traefik.http.services.kestra.loadbalancer.server.port=8080"

Replace kestra.yourdomain.com with your real domain name configured in Namecheap pointing directly to your VPS IP address. Also replace your-email@example.com in traefik/traefik.yml with your actual email address.

Start the complete application stack using Docker Compose:

docker compose up -d

Check container status logs to ensure everything compiles and boots properly:

docker compose logs -f kestra

Once initialized, navigate to https://kestra.yourdomain.com in your browser. You will be prompted to supply the admin username (admin@example.com) and password (StrongAdminPassword456!) specified in your application.yml file.

Writing Your First Kestra Workflow

With Kestra deployed, you can build automation flows directly inside the Web UI editor or via YAML definitions uploaded over the REST API.

Here is a multi-step workflow example that runs local scripts, processes HTTP requests, and outputs execution records.

Create a workflow file named sample-pipeline.yml:

id: system_health_pipeline
namespace: dev.ops

tasks:
  - id: log_start
    type: io.kestra.plugin.core.log.Log
    message: Starting automated server metrics check...

  - id: fetch_external_ip
    type: io.kestra.plugin.core.http.Request
    uri: https://api.ipify.org?format=json

  - id: parse_ip_response
    type: io.kestra.plugin.core.log.Log
    message: "Public IP response was: {{ outputs.fetch_external_ip.body }}"

  - id: run_shell_diagnostics
    type: io.kestra.plugin.scripts.shell.Commands
    runner: PROCESS
    commands:
      - echo "Current Disk Usage:"
      - df -h
      - echo "Current Memory Usage:"
      - free -m

triggers:
  - id: daily_schedule
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "0 8 * * *"

To create this workflow via Kestra’s API directly from your terminal, execute the following curl command:

curl -X POST "https://kestra.yourdomain.com/api/v1/workflows" \
  -u "admin@example.com:StrongAdminPassword456!" \
  -H "Content-Type: application/x-yaml" \
  --data-binary @sample-pipeline.yml

You can now open the Web UI dashboard under the Flows tab, locate system_health_pipeline, and click Execute to trigger the flow manually. The platform presents detailed real-time execution logs, task topologies, and outputs for every step.

Getting Started

To launch your own production instance, acquire infrastructure resources through these providers:

• Hetzner VPS or DigitalOcean for high-performance cloud compute
• Contabo VPS if you need high-memory server specs at lower price points
• Namecheap for simple domain registration and DNS records management

Outsource Your Automation

Don’t have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-HUGO for priority. Or DM at chasebot.online.