Handling Overlapping Execution in Scheduled Python Scripts

Handling Overlapping Execution in Scheduled Python Scripts

What You’ll Need

  • n8n Cloud or self-hosted n8n for orchestrating scheduled tasks
  • Hetzner VPS or Contabo VPS for hosting Python background workers
  • DigitalOcean as an alternative cloud instance provider
  • Python 3.10 or higher installed on your server environment

Table of Contents


The Problem: Race Conditions in Scheduled Python Scripts

Every developer who manages background automation eventually runs into overlapping executions. You configure a cron job or a scheduler to trigger a script every five minutes. The script usually takes thirty seconds to complete. Everything runs smoothly until external factors intervene: an API endpoint slows down, a database query hits an unindexed lock, or network latency spikes.

When the next scheduled interval arrives, your operating system spawns a second instance of the script. Now two processes are reading from the same message queue, updating identical database records, or sending duplicate webhooks to external API consumers.

If your task involves external operations like handling rate limits when querying Anthropic models, an overlapping instance can instantly exhaust your quota or trigger cascading failure modes across your worker cluster.

Time --->
[Cron Trigger 10:00] ---> [ Script Instance A Starts ]........................[ Complete 10:07 ]
[Cron Trigger 10:05] --->                 [ Script Instance B Starts ] -------> [ OVERLAP FAILS / DUPLICATES DATA ]

To eliminate race conditions, you need determinism. The incoming execution must detect that a previous process is still running and either exit immediately or wait gracefully without corrupting state.


Strategy 1: OS-Level File Locking with fcntl

When running scheduled scripts on a single host like a Hetzner VPS, operating system locks provide the fastest and most dependable mechanism to block overlapping runs. Linux and Unix systems provide the fcntl module natively inside Python standard library.

Using fcntl.flock, we can request an exclusive non-blocking lock on a specific file descriptor. If another process holds the lock, fcntl raises an OSError (specifically EWOULDBLOCK or EAGAIN). The second script catches this error and terminates immediately.

Here is the complete implementation of a process lock manager using file locking.

import fcntl
import sys
import time
import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] PID %(process)d: %(message)s",
    handlers=[logging.StreamHandler(sys.stdout)]
)

class SingleInstanceFileLock:
    def __init__(self, lock_file_path: str):
        self.lock_file_path = lock_file_path
        self.fp = None

    def __enter__(self):
        self.fp = open(self.lock_file_path, "w")
        try:
            fcntl.flock(self.fp.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
            self.fp.write(str(sys.argv[0]))
            self.fp.flush()
            logging.info(f"Acquired lock on file: {self.lock_file_path}")
            return self
        except (OSError, IOError):
            logging.warning(f"Another instance is already running with lock file: {self.lock_file_path}")
            self.fp.close()
            sys.exit(0)

    def __exit__(self, exc_type, exc_val, exc_tb):
        if self.fp and not self.fp.closed:
            try:
                fcntl.flock(self.fp.fileno(), fcntl.LOCK_UN)
                self.fp.close()
                logging.info(f"Released lock on file: {self.lock_file_path}")
            except (OSError, IOError) as err:
                logging.error(f"Failed to release lock file: {err}")

def run_long_task():
    logging.info("Starting task execution sequence...")
    for step in range(1, 6):
        logging.info(f"Processing step {step} of 5...")
        time.sleep(2)
    logging.info("Task execution sequence finished successfully.")

if __name__ == "__main__":
    LOCK_PATH = "/tmp/my_scheduled_python_job.lock"
    with SingleInstanceFileLock(LOCK_PATH):
        run_long_task()

When you trigger this script twice in parallel, the second instance attempts to secure an exclusive lock on /tmp/my_scheduled_python_job.lock. Because the file descriptor is already locked by PID A, PID B catches the exception, logs a warning, and exits cleanly with return code 0.


💡 Fast-Track Your Project: Don’t want to configure this yourself? I build custom n8n pipelines and bots. Message me with code SYS3-HUGO.


Strategy 2: Distributed Locking with Redis

File locking works perfectly for a single host. However, modern deployment architectures split work across multiple application nodes on hosts like DigitalOcean or Contabo VPS. If your cron triggers run inside multiple Kubernetes pods or autoscaling server instances, local file locks will fail to prevent execution overlaps.

In distributed setups, we use Redis as an atomic lock manager. We acquire locks using SET key value NX EX seconds, which guarantees that the key is only created if it does not already exist, while automatically setting an expiration timeout (TTL) to avoid deadlocks if a worker node crashes unexpectedly.

To release the lock safely without deleting a lock acquired by a subsequent run, we execute an atomic Lua script that compares the stored random identifier with our instance identifier.

When handling distributed worker failures, combining this pattern with robust recovery mechanisms like implementing circuit breakers for outbound webhook delivery ensures downstream endpoints remain healthy when retries occur.

Here is the complete Python script for distributed locking using Redis:

import time
import uuid
import sys
import logging
import redis

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] PID %(process)d: %(message)s"
)

LUA_RELEASE_SCRIPT = """
if redis.call("get", KEYS[1]) == ARGV[1] then
    return redis.call("del", KEYS[1])
else
    return 0
end
"""

class RedisDistributedLock:
    def __init__(self, redis_client: redis.Redis, lock_name: str, expire_seconds: int = 60):
        self.redis_client = redis_client
        self.lock_key = f"lock:{lock_name}"
        self.expire_seconds = expire_seconds
        self.token = str(uuid.uuid4())
        self.release_lua = self.redis_client.register_script(LUA_RELEASE_SCRIPT)

    def acquire(self) -> bool:
        acquired = self.redis_client.set(
            name=self.lock_key,
            value=self.token,
            nx=True,
            ex=self.expire_seconds
        )
        return bool(acquired)

    def release(self) -> bool:
        result = self.release_lua(keys=[self.lock_key], args=[self.token])
        return bool(result)

    def __enter__(self):
        if not self.acquire():
            logging.warning(f"Could not acquire distributed lock for key: {self.lock_key}. Terminating script.")
            sys.exit(0)
        logging.info(f"Acquired distributed lock: {self.lock_key} with token: {self.token}")
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        released = self.release()
        if released:
            logging.info(f"Successfully released distributed lock: {self.lock_key}")
        else:
            logging.error(f"Failed to release lock or lock expired before task completion: {self.lock_key}")

def process_distributed_batch():
    logging.info("Executing distributed payload synchronization...")
    for i in range(1, 4):
        logging.info(f"Syncing item chunk {i}/3...")
        time.sleep(3)
    logging.info("Distributed payload synchronization completed.")

if __name__ == "__main__":
    redis_conn = redis.Redis(host="localhost", port=6379, db=0, decode_responses=True)
    LOCK_NAME = "sync_user_subscriptions"
    LOCK_TIMEOUT = 30

    with RedisDistributedLock(redis_client=redis_conn, lock_name=LOCK_NAME, expire_seconds=LOCK_TIMEOUT):
        process_distributed_batch()

Strategy 3: Atomic Lock Decorator with Graceful Exit

To keep your core application logic clean, you can abstract concurrency management into a reusable Python decorator. This enables you to wrap any standard task function with execution guarantees without cluttering the function body.

This production decorator supports configurable retry attempts, custom lock file destinations, and clean POSIX signal handlers to ensure locks release when processes encounter SIGTERM or SIGINT.

import os
import sys
import time
import fcntl
import signal
import functools
import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s]: %(message)s"
)

def prevent_concurrent_execution(lock_file_name: str):
    def decorator(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            lock_dir = "/tmp/script_locks"
            os.makedirs(lock_dir, exist_ok=True)
            lock_path = os.path.join(lock_dir, lock_file_name)
            
            lock_fd = open(lock_path, "w")
            try:
                fcntl.flock(lock_fd.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
            except (OSError, IOError):
                logging.warning(f"Skipping execution of '{func.__name__}': another instance is actively running.")
                lock_fd.close()
                return None

            def cleanup_signal_handler(signum, frame):
                logging.info(f"Received signal {signum}. Cleaning up lock file...")
                try:
                    fcntl.flock(lock_fd.fileno(), fcntl.LOCK_UN)
                    lock_fd.close()
                except Exception as e:
                    logging.error(f"Error closing lock on signal: {e}")
                sys.exit(128 + signum)

            signal.signal(signal.SIGINT, cleanup_signal_handler)
            signal.signal(signal.SIGTERM, cleanup_signal_handler)

            try:
                logging.info(f"Lock acquired for '{func.__name__}'. Executing workload.")
                result = func(*args, **kwargs)
                return result
            finally:
                try:
                    fcntl.flock(lock_fd.fileno(), fcntl.LOCK_UN)
                    lock_fd.close()
                    logging.info(f"Lock released for '{func.__name__}'.")
                except Exception as e:
                    logging.error(f"Failed releasing lock for '{func.__name__}': {e}")

        return wrapper
    return decorator

@prevent_concurrent_execution(lock_file_name="hourly_data_export.lock")
def export_hourly_analytics_data():
    logging.info("Gathering analytics records from primary database...")
    time.sleep(4)
    logging.info("Analytics data compiled and sent to storage bucket.")

if __name__ == "__main__":
    export_hourly_analytics_data()

Orchestration Strategies and Systemd Controls

While code-level locks prevent concurrent runs, moving schedule logic out of raw crontabs and into dedicated service orchestrators provides built-in concurrency controls, native failure visibility, and detailed metric logs.

If you are running stand-alone scripts on server infrastructure, Linux systemd timers offer built-in prevention against overlapping runs via Systemd service definitions.

By default, if a systemd service unit is already active when its corresponding timer triggers, systemd skips the invocation automatically without raising system errors.

Systemd Configuration Example

Create the service unit file at /etc/systemd/system/py_job.service:

[Unit]
Description=Scheduled Data Processing Job
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/opt/automation
ExecStart=/usr/bin/python3 /opt/automation/worker.py
Restart=no

Create the corresponding timer unit file at /etc/systemd/system/py_job.timer:

[Unit]
Description=Run Data Processing Job Every 5 Minutes

[Timer]
OnCalendar=*:0/5
Persistent=true

[Install]
WantedBy=timers.target

Reload systemd daemon configurations and enable the timer:

systemctl daemon-reload
systemctl enable --now py_job.timer

If your Python script takes 8 minutes to execute, the timer trigger at minute 5 will be safely ignored because py_job.service is still in an active state.

For complex workflows involving multiple dependent steps, offloading scheduling to a dedicated workflow engine is often superior to managing raw scripts. You can learn how to set up an enterprise orchestrator in our guide on how to self host Kestra on cloud VPS.

Similarly, automation platforms like n8n Cloud allow you to configure execution policies directly in the interface, choosing between queueing subsequent runs or cancelling concurrent executions entirely.


Getting Started

To implement robust execution locking in your Python infrastructure, set up your development or production nodes with these hosting providers:

  • Spin up dedicated cloud compute instances using a high-performance Hetzner VPS or Contabo VPS.
  • Deploy scalable cloud infrastructure on DigitalOcean for distributed worker setups.
  • Orchestrate script execution, manage API retries, and prevent overlapping schedules using n8n Cloud.

Outsource Your Automation

Don’t have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-HUGO for priority. Or DM at chasebot.online.

Want to automate this yourself?

Start with n8n Cloud (free tier available) or self-host on a Hetzner VPS for full control.

Want this engine running on your own VPS?

This blog publishes itself — daily, unattended, on free API tiers. The full engine, Hugo theme, and setup guide are available as System 3.

Get System 3
system online