Building Scalable Playwright Scrapers with Docker

· 9 min read ·
Building Scalable Playwright Scrapers with Docker

What You’ll Need

  • Hetzner VPS or Contabo VPS for hosting your containerized scrapers
  • n8n Cloud or self-hosted n8n for downstream workflow execution
  • Docker Engine and Docker Compose installed on your deployment host
  • Python 3.11 or newer installed on your development machine

Table of Contents

Architecting Headless Scrapers with Playwright

Building web scrapers for modern single page applications requires toolsets capable of executing JavaScript, handling client side rendering, and bypassing basic bot detection mechanisms. While legacy libraries like BeautifulSoup work well for static HTML pages, complex dynamic platforms demand headless browser automation engines. Playwright stands out in this ecosystem due to its native support for asynchronous event loops, automatic waiting mechanisms, and isolated browser contexts.

When I design extraction pipelines, my primary goal is performance per compute unit. Spinning up a fresh browser process for every URL request destroys CPU and memory throughput. Instead, I spin up a single persistent browser instance and spawn lightweight browser contexts for individual scraping sessions. Each context acts like an isolated incognito browser session with zero cookie or state leakage.

To deploy these workloads efficiently across cloud infrastructure like a Hetzner VPS, we need to write asynchronous Python scrapers. Below is a complete script demonstrating an asynchronous Playwright implementation that targets dynamic rendering targets and returns extracted data.

import asyncio
import json
import logging
from typing import Dict, List, Any
from playwright.async_api import async_playwright, Browser, Page

logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s")

async def extract_product_details(page: Page, url: str) -> Dict[str, Any]:
    logging.info(f"Navigating to URL: {url}")
    response = await page.goto(url, wait_until="domcontentloaded", timeout=30000)
    
    if not response or response.status >= 400:
        logging.error(f"Failed to load page. Status code: {response.status if response else 'No Response'}")
        return {"url": url, "status": "failed", "error": f"HTTP status {response.status if response else 'None'}"}

    try:
        await page.wait_for_selector("h1", timeout=5000)
    except Exception as e:
        logging.warning(f"Heading selector timed out on {url}: {str(e)}")

    title = await page.locator("h1").inner_text() if await page.locator("h1").count() > 0 else "N/A"
    
    meta_description = "N/A"
    meta_locator = page.locator("meta[name='description']")
    if await meta_locator.count() > 0:
        meta_description = await meta_locator.get_attribute("content") or "N/A"

    links = await page.locator("a").evaluate_all("elements => elements.map(el => el.href)")

    return {
        "url": url,
        "status": "success",
        "title": title.strip(),
        "meta_description": meta_description.strip(),
        "link_count": len(links),
    }

async def run_batch(urls: List[str]) -> List[Dict[str, Any]]:
    results = []
    async with async_playwright() as p:
        browser: Browser = await p.chromium.launch(
            headless=True,
            args=[
                "--disable-dev-shm-usage",
                "--no-sandbox",
                "--disable-setuid-sandbox",
                "--disable-gpu",
            ]
        )
        context = await browser.new_context(
            user_agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
        )
        
        for url in urls:
            page = await context.new_page()
            try:
                data = await extract_product_details(page, url)
                results.append(data)
            except Exception as exc:
                logging.error(f"Unhandled exception while scraping {url}: {str(exc)}")
                results.append({"url": url, "status": "error", "error": str(exc)})
            finally:
                await page.close()
                
        await context.close()
        await browser.close()
    return results

if __name__ == "__main__":
    target_urls = [
        "https://quotes.toscrape.com/",
        "https://books.toscrape.com/",
    ]
    extracted_data = asyncio.run(run_batch(target_urls))
    print(json.dumps(extracted_data, indent=2))

When extracting raw data from web pages, downstream applications often demand strict schema validation. If you plan to pass scraped text to large language models for structured parsing, read our guide on Handling Structured Output with OpenAI Function Calling to convert unstructured HTML strings into deterministic JSON records.

💡 Fast-Track Your Project: Don’t want to configure this yourself? I build custom n8n pipelines and bots. Message me with code SYS3-HUGO.

Containerizing Playwright with Docker

Executing Playwright scripts on bare metal servers presents dependency management issues. Playwright requires specific system level shared libraries for Chromium, Firefox, and WebKit binaries. System package updates can break browser drivers without warning. Docker solves this problem by packaging the operating system dependencies, browser engines, and application scripts into an immutable execution environment.

Microsoft maintains official Docker images for Playwright that come pre-packaged with system dependencies and compiled browser binaries. Using these base images reduces container build times and ensures runtime stability.

Below is the production grade Dockerfile for our Playwright pipeline. It sets up non-root system users, installs core Python packages, configures environment variables, and configures entry points.

FROM mcr.microsoft.com/playwright/python:v1.41.0-jammy

ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    PIP_NO_CACHE_DIR=1 \
    DEBIAN_FRONTEND=noninteractive

WORKDIR /app

RUN apt-get update && apt-get install -y --no-install-recommends \
    curl \
    ca-certificates \
    && rm -rf /var/lib/apt/lists/*

COPY requirements.txt /app/requirements.txt

RUN pip install --upgrade pip && \
    pip install -r requirements.txt

COPY . /app

RUN useradd -m -u 1000 scraperuser && \
    chown -R scraperuser:scraperuser /app

USER scraperuser

CMD ["python", "worker.py"]

To complement the Dockerfile, define a .dockerignore file in the root project directory. This prevents host binaries, cached Python artifacts, and version control files from leaking into your container builds.

__pycache__/
*.pyc
*.pyo
*.pyd
.git/
.gitignore
.dockerignore
.venv/
venv/
*.log
dist/
build/

Here is the corresponding requirements.txt file containing strict version locks for our dependencies:

playwright==1.41.0
redis==5.0.1
pydantic==2.6.1
asyncio==3.4.3

Scaling with Redis and Docker Compose

Single container scrapers are limited by single host network limits and core counts. To scale scraper throughput, we can decouple task generation from task execution using a message queue pattern. A producer process injects URLs into a Redis list, while multiple Dockerized Playwright worker containers pull jobs off the queue asynchronously.

If you want to trigger scraper queue producers on automated chron schedules rather than event driven webhooks, refer to our tutorial on How to Schedule Python Tasks using APScheduler to execute scheduled task dispatchers.

Below is the complete implementation for our consumer queue worker script (worker.py). It listens to a Redis queue, acquires scraping jobs, processes target pages, and pushes structured outputs back into a completion queue.

import asyncio
import json
import logging
import os
import signal
from typing import Dict, Any
import redis.asyncio as aioredis
from playwright.async_api import async_playwright, Browser, BrowserContext, Page

logging.basicConfig(level=logging.INFO, format="%(asctime)s - [%(levelname)s] - %(message)s")

REDIS_HOST = os.getenv("REDIS_HOST", "localhost")
REDIS_PORT = int(os.getenv("REDIS_PORT", 6379))
QUEUE_NAME = os.getenv("QUEUE_NAME", "scraper_tasks")
RESULT_QUEUE = os.getenv("RESULT_QUEUE", "scraper_results")

class ScraperWorker:
    def __init__(self):
        self.redis_client = None
        self.running = True
        self.browser: Browser = None
        self.playwright_instance = None

    async def init_browser(self):
        self.playwright_instance = await async_playwright().start()
        self.browser = await self.playwright_instance.chromium.launch(
            headless=True,
            args=["--disable-dev-shm-usage", "--no-sandbox", "--disable-setuid-sandbox"]
        )
        self.redis_client = aioredis.Redis(host=REDIS_HOST, port=REDIS_PORT, db=0, decode_responses=True)
        logging.info("Worker initialized. Connected to Redis and Playwright browser process.")

    async def process_task(self, context: BrowserContext, task_payload: Dict[str, Any]) -> Dict[str, Any]:
        url = task_payload.get("url")
        job_id = task_payload.get("job_id")
        
        page: Page = await context.new_page()
        try:
            logging.info(f"Processing Job ID: {job_id} | Target: {url}")
            response = await page.goto(url, wait_until="networkidle", timeout=40000)
            
            title = await page.title()
            content_length = len(await page.content())
            
            return {
                "job_id": job_id,
                "url": url,
                "status_code": response.status if response else 0,
                "title": title,
                "content_bytes": content_length,
                "error": None
            }
        except Exception as err:
            logging.error(f"Error executing job {job_id}: {str(err)}")
            return {
                "job_id": job_id,
                "url": url,
                "status_code": 0,
                "title": None,
                "content_bytes": 0,
                "error": str(err)
            }
        finally:
            await page.close()

    async def run(self):
        await self.init_browser()
        context = await self.browser.new_context()
        
        try:
            while self.running:
                task_data = await self.redis_client.blpop(QUEUE_NAME, timeout=5)
                if not task_data:
                    await asyncio.sleep(0.5)
                    continue
                
                queue_key, raw_payload = task_data
                try:
                    payload = json.loads(raw_payload)
                    result = await self.process_task(context, payload)
                    await self.redis_client.rpush(RESULT_QUEUE, json.dumps(result))
                    logging.info(f"Finished Job ID: {payload.get('job_id')}")
                except json.JSONDecodeError:
                    logging.error(f"Failed to parse incoming payload: {raw_payload}")
        finally:
            await context.close()
            await self.shutdown()

    async def shutdown(self):
        logging.info("Shutting down worker...")
        self.running = False
        if self.browser:
            await self.browser.close()
        if self.playwright_instance:
            await self.playwright_instance.stop()
        if self.redis_client:
            await self.redis_client.aclose()
        logging.info("Worker shutdown complete.")

if __name__ == "__main__":
    worker = ScraperWorker()
    
    def handle_signal(sig, frame):
        logging.warning("Shutdown signal received. Stopping worker loop...")
        worker.running = False

    signal.signal(signal.SIGINT, handle_signal)
    signal.signal(signal.SIGTERM, handle_signal)

    asyncio.run(worker.run())

To orchestrate the Redis instance alongside multiple scaled instances of our Playwright worker, we define a multi container configuration using docker-compose.yml.

version: "3.8"

services:
  redis:
    image: redis:7-alpine
    container_name: scraper_redis
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5
    restart: always

  scraper_worker:
    build:
      context: .
      dockerfile: Dockerfile
    environment:
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      - QUEUE_NAME=scraper_tasks
      - RESULT_QUEUE=scraper_results
    depends_on:
      redis:
        condition: service_healthy
    deploy:
      replicas: 3
      resources:
        limits:
          cpus: "1.5"
          memory: 1500M
        reservations:
          cpus: "0.5"
          memory: 512M
    restart: on-failure

volumes:
  redis_data:

With this infrastructure setup, you can scale worker capacity up or down across compute instances using Docker Compose native commands:

docker compose up -d --scale scraper_worker=5

Resilience, Proxy Rotation, and Error Handling

Scraping dynamic target sites at scale introduces blocking conditions, IP bans, network timeouts, and resource leaks. A scalable container system must mitigate these factors before they cause task dropouts.

Three core components ensure production resilience:

  1. Proxy Rotation: Route every Playwright browser context through residential or datacenter proxies.
  2. Resource Limits: Cap maximum memory and CPU usage inside containers to prevent runaway headless browser instances from crashing host nodes.
  3. Alerting and Downstream Automation: Notify ops teams when error rates exceed acceptable thresholds.

To send real time alerts to operations staff when container queues stall or proxies drop, read how to Automate WhatsApp Business Messages with n8n and 360dialog to dispatch operational telemetry directly to your emergency response channels.

Below is an enterprise ready pattern showing how to integrate proxy rotation, custom network headers, resource cleanup, and detailed exception handling into a reusable Playwright runner block.

import asyncio
import logging
from typing import Dict, Any, Optional
from playwright.async_api import async_playwright, Playwright, TimeoutError as PlaywrightTimeoutError

logging.basicConfig(level=logging.INFO)

class ResilientScraper:
    def __init__(self, proxy_list: Optional[list] = None):
        self.proxy_list = proxy_list or []
        self.current_proxy_index = 0

    def get_next_proxy(self) -> Optional[Dict[str, str]]:
        if not self.proxy_list:
            return None
        proxy_url = self.proxy_list[self.current_proxy_index]
        self.current_proxy_index = (self.current_proxy_index + 1) % len(self.proxy_list)
        return {"server": proxy_url}

    async def fetch_page_resilient(self, url: str, max_retries: int = 3) -> Dict[str, Any]:
        retries = 0
        backoff_delay = 2.0

        while retries < max_retries:
            proxy_config = self.get_next_proxy()
            logging.info(f"Attempt {retries + 1} for {url} using proxy: {proxy_config}")

            async with async_playwright() as p:
                try:
                    browser = await p.chromium.launch(
                        headless=True,
                        args=["--disable-dev-shm-usage", "--no-sandbox"]
                    )
                    
                    context_options = {
                        "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
                        "viewport": {"width": 1920, "height": 1080},
                        "ignore_https_errors": True,
                    }
                    if proxy_config:
                        context_options["proxy"] = proxy_config

                    context = await browser.new_context(**context_options)
                    
                    # Block resource-heavy assets to save memory and bandwidth
                    await context.route("**/*.{png,jpg,jpeg,gif,svg,woff,woff2,css}", lambda route: route.abort())

                    page = await context.new_page()
                    
                    response = await page.goto(url, wait_until="domcontentloaded", timeout=20000)
                    
                    if response and response.status == 200:
                        content = await page.content()
                        await browser.close()
                        return {
                            "status": "success",
                            "url": url,
                            "http_code": response.status,
                            "html_length": len(content)
                        }
                    else:
                        logging.warning(f"Bad HTTP status code {response.status if response else 'None'} on {url}")
                        
                    await browser.close()

                except PlaywrightTimeoutError:
                    logging.error(f"Timeout error fetching {url} on attempt {retries + 1}")
                except Exception as ex:
                    logging.error(f"Unexpected error fetching {url}: {str(ex)}")

            retries += 1
            await asyncio.sleep(backoff_delay)
            backoff_delay *= 2.0

        return {"status": "failed", "url": url, "error": "Max retries exceeded"}

if __name__ == "__main__":
    proxies = [
        "http://proxy1.example.com:8080",
        "http://proxy2.example.com:8080"
    ]
    scraper = ResilientScraper(proxy_list=proxies)
    result = asyncio.run(scraper.fetch_page_resilient("https://quotes.toscrape.com/"))
    print(result)

By resource blocking static assets (png, jpg, woff), your Playwright instances run faster, reduce network consumption by up to 70 percent, and decrease memory pressure inside Docker containers.

Getting Started

To spin up your scalable containerized scraper infrastructure:

  1. Provision a high network performance cloud instance on Hetzner VPS or Contabo VPS.
  2. Install Docker and the Docker Compose plugin on your remote host.
  3. Save the Dockerfile, docker-compose.yml, worker.py, and requirements.txt files into your deployment directory.
  4. Scale your scraper workers using docker compose up -d --scale scraper_worker=4.
  5. Connect your scrapers to orchestrators like n8n Cloud to manage dataset delivery to production databases.

Outsource Your Automation

Don’t have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-HUGO for priority. Or DM at chasebot.online.

Want to automate this yourself?

Start with n8n Cloud (free tier available) or self-host on a Hetzner VPS for full control.

Want this engine running on your own VPS?

This blog publishes itself — daily, unattended, on free API tiers. The full engine, Hugo theme, and setup guide are available as System 3.

Get System 3

Related Guides

system online