Building Web Scraping Data Pipelines on VPS

· 9 min read ·
Building Web Scraping Data Pipelines on VPS

What You’ll Need

Table of Contents

  1. Why VPS-based Web Scraping Matters
  2. Architecture Overview for Scraping Pipelines
  3. Setting Up Your VPS Environment
  4. Building Your First n8n Scraping Workflow
  5. Advanced Data Pipeline Patterns
  6. Storage and Data Management
  7. Getting Started

Why VPS-based Web Scraping Matters

I’ve been building data pipelines for years, and I can tell you that hosting your scraping infrastructure on a VPS fundamentally changes what’s possible. When you’re relying on SaaS platforms alone, you’re constrained by rate limits, storage quotas, and execution timeouts. A VPS gives you control.

Web scraping at scale requires persistent infrastructure. You need to handle retries gracefully, manage browser sessions, store intermediate data, and process millions of records without losing state. Cloud-only solutions charge per execution and often kill long-running jobs. On a VPS, you control everything: memory allocation, CPU scheduling, storage persistence, and parallel execution threads.

The real power emerges when you combine a VPS with n8n Cloud or self-hosted n8n. You get workflow orchestration without operational overhead, while maintaining the flexibility of dedicated infrastructure for the heavy lifting.

Architecture Overview for Scraping Pipelines

Before we write code, let’s talk structure. A production scraping pipeline on a VPS typically has these layers:

The orchestration layer runs on n8n. This handles scheduling, error handling, and conditional logic. The scraping layer uses headless browsers or HTTP libraries that run inside containers on your VPS. The data transformation layer normalizes and validates scraped content. The storage layer persists data to databases or object storage. The monitoring layer tracks pipeline health and performance.

Most teams make the mistake of building everything inside n8n. That creates bottlenecks when you need to scrape 100,000 pages or process streaming data. Instead, offload heavy operations to containerized services running on your VPS, then orchestrate them from n8n.

When you eventually need to integrate scraped data with machine learning models, you might want to connect that pipeline to vector databases using embeddings, but we’ll focus on the scraping layer first.

Setting Up Your VPS Environment

Start with a Hetzner VPS or Contabo VPS. I typically use the 4-core, 8GB RAM tier for production scraping. Spin up Ubuntu 22.04 LTS. From your terminal:

ssh root@your_vps_ip
apt-get update
apt-get upgrade -y
apt-get install -y curl wget git build-essential python3-pip

Before deploying scraping services, secure your SSH access by following the principles outlined in hardening SSH server security on Ubuntu VPS. This prevents unauthorized access to your data pipeline.

Now install Docker:

curl -fsSL https://get.docker.com -o get-docker.sh
sh get-docker.sh
usermod -aG docker root

Install Docker Compose:

curl -L "https://github.com/docker/compose/releases/download/v2.20.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
docker-compose --version

Create a project directory structure:

mkdir -p /opt/scraping-pipeline/{n8n,scrapers,data,config}
cd /opt/scraping-pipeline

Building Your First n8n Scraping Workflow

Let me walk you through deploying n8n on your VPS and creating a real scraping workflow. Create a docker-compose.yml file:

version: '3.8'

services:
  n8n:
    image: n8nio/n8n:latest
    container_name: n8n-scraper
    restart: always
    ports:
      - "5678:5678"
    environment:
      - N8N_HOST=your_domain.com
      - N8N_PROTOCOL=https
      - N8N_PORT=5678
      - N8N_SECURE_COOKIE=true
      - DB_TYPE=postgres
      - DB_POSTGRESDB_HOST=postgres
      - DB_POSTGRESDB_PORT=5432
      - DB_POSTGRESDB_DATABASE=n8n
      - DB_POSTGRESDB_USER=n8n
      - DB_POSTGRESDB_PASSWORD=secure_password_here
      - EXECUTIONS_PROCESS=queue
      - N8N_SKIP_WEBHOOK_VALIDATION=false
    volumes:
      - n8n_data:/home/node/.n8n
      - /opt/scraping-pipeline/config:/config
    networks:
      - scraping_net
    depends_on:
      - postgres

  postgres:
    image: postgres:15-alpine
    container_name: n8n-postgres
    restart: always
    environment:
      POSTGRES_DB: n8n
      POSTGRES_USER: n8n
      POSTGRES_PASSWORD: secure_password_here
    volumes:
      - postgres_data:/var/lib/postgresql/data
    networks:
      - scraping_net
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U n8n"]
      interval: 10s
      timeout: 5s
      retries: 5

  redis:
    image: redis:7-alpine
    container_name: n8n-redis
    restart: always
    networks:
      - scraping_net
    volumes:
      - redis_data:/data

  puppeteer-service:
    image: browserless/chrome:latest
    container_name: puppeteer-scraper
    restart: always
    ports:
      - "3000:3000"
    environment:
      - TOKEN=scraping_token_123
      - MAX_CONCURRENT_SESSIONS=5
    networks:
      - scraping_net

volumes:
  n8n_data:
  postgres_data:
  redis_data:

networks:
  scraping_net:
    driver: bridge

Deploy this stack:

cd /opt/scraping-pipeline
docker-compose up -d
docker-compose logs -f n8n

Wait for n8n to initialize (30-60 seconds). Access it at https://your_domain.com:5678 after setting up reverse proxy with Traefik. For production, I recommend configuring Traefik with Docker Compose for SSL termination and load balancing.

Now let’s build the actual workflow. In the n8n UI, create a new workflow. I’ll show you the JSON configuration so you can import it directly:

{
  "name": "E-commerce Product Scraper",
  "nodes": [
    {
      "parameters": {
        "rule": {
          "interval": [
            {
              "days": 0,
              "hours": 0,
              "minutes": 5,
              "seconds": 0
            }
          ]
        }
      },
      "name": "Schedule Trigger",
      "type": "n8n-nodes-base.cron",
      "typeVersion": 1,
      "position": [
        250,
        300
      ]
    },
    {
      "parameters": {
        "method": "GET",
        "url": "http://puppeteer-service:3000/session",
        "authentication": "genericCredentialType",
        "genericCredentials": {
          "password": "scraping_token_123"
        }
      },
      "name": "Initialize Browser",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.1,
      "position": [
        450,
        300
      ]
    },
    {
      "parameters": {
        "method": "POST",
        "url": "http://puppeteer-service:3000/execute",
        "authentication": "genericCredentialType",
        "genericCredentials": {
          "password": "scraping_token_123"
        },
        "sendBody": true,
        "body": {
          "code": "const page = await browser.newPage();\nawait page.goto('https://example-ecommerce.com/products', { waitUntil: 'networkidle2' });\nconst products = await page.evaluate(() => {\n  return Array.from(document.querySelectorAll('.product-item')).map(el => ({\n    title: el.querySelector('.product-title')?.innerText || '',\n    price: el.querySelector('.product-price')?.innerText || '',\n    url: el.querySelector('a')?.href || '',\n    rating: el.querySelector('.rating')?.innerText || 'N/A',\n    inStock: el.querySelector('.stock-status')?.innerText.includes('In Stock') || false\n  }));\n});\nawait page.close();\nreturn products;"
        }
      },
      "name": "Scrape Products",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.1,
      "position": [
        650,
        300
      ]
    },
    {
      "parameters": {
        "operation": "executeQuery",
        "query": "INSERT INTO products (title, price, url, rating, in_stock, scraped_at) VALUES\n{% for item in $input.all() %}\n  ('{{ item.json.title }}', '{{ item.json.price }}', '{{ item.json.url }}', '{{ item.json.rating }}', {{ item.json.inStock }}, NOW()){{ \"\" if loop.last else \",\" }}\n{% endfor %}\nON CONFLICT (url) DO UPDATE SET\n  price = EXCLUDED.price,\n  rating = EXCLUDED.rating,\n  in_stock = EXCLUDED.in_stock,\n  scraped_at = EXCLUDED.scraped_at;"
      },
      "name": "Store in Database",
      "type": "n8n-nodes-base.postgres",
      "typeVersion": 2.3,
      "position": [
        850,
        300
      ]
    },
    {
      "parameters": {
        "key": "product_count",
        "value": "{{ $input.all().length }}"
      },
      "name": "Count Products",
      "type": "n8n-nodes-base.set",
      "typeVersion": 3.3,
      "position": [
        1050,
        300
      ]
    }
  ],
  "connections": {
    "Schedule Trigger": {
      "main": [
        [
          {
            "node": "Initialize Browser",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Initialize Browser": {
      "main": [
        [
          {
            "node": "Scrape Products",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Scrape Products": {
      "main": [
        [
          {
            "node": "Store in Database",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Store in Database": {
      "main": [
        [
          {
            "node": "Count Products",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  }
}

💡 Fast-Track Your Project: Don’t want to configure this yourself? I build custom n8n pipelines and bots. Message me with code SYS3-HUGO.

Advanced Data Pipeline Patterns

Once your basic pipeline works, you’ll want to handle edge cases. Here’s a more production-ready setup with retry logic, proxy rotation, and data validation:

mkdir -p /opt/scraping-pipeline/scrapers/src
cat > /opt/scraping-pipeline/scrapers/Dockerfile << 'EOF'
FROM node:18-alpine

WORKDIR /app

RUN apk add --no-cache \
    chromium \
    ca-certificates

COPY package.json package-lock.json ./
RUN npm ci --only=production

COPY src ./src

EXPOSE 3001

CMD ["node", "src/server.js"]
EOF

Create the scraper service code:

const express = require('express');
const puppeteer = require('puppeteer');
const axios = require('axios');

const app = express();
app.use(express.json());

const PROXIES = [
  'http://proxy1:8080',
  'http://proxy2:8080',
  'http://proxy3:8080'
];

let proxyIndex = 0;

const getNextProxy = () => {
  const proxy = PROXIES[proxyIndex % PROXIES.length];
  proxyIndex++;
  return proxy;
};

const validateProduct = (product) => {
  const errors = [];
  if (!product.title || product.title.trim().length === 0) {
    errors.push('Missing product title');
  }
  if (!product.price || isNaN(parseFloat(product.price))) {
    errors.push('Invalid price format');
  }
  if (!product.url || !product.url.startsWith('http')) {
    errors.push('Invalid URL format');
  }
  return {
    isValid: errors.length === 0,
    errors
  };
};

const scrapeWithRetry = async (url, maxRetries = 3) => {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      const browser = await puppeteer.launch({
        args: [
          '--no-sandbox',
          '--disable-setuid-sandbox',
          `--proxy-server=${getNextProxy()}`
        ]
      });

      const page = await browser.newPage();
      await page.setDefaultNavigationTimeout(30000);
      await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36');

      await page.goto(url, { waitUntil: 'networkidle2' });

      const products = await page.evaluate(() => {
        return Array.from(document.querySelectorAll('[data-product-id]')).map(el => ({
          id: el.getAttribute('data-product-id'),
          title: el.querySelector('[data-field="title"]')?.innerText || '',
          price: el.querySelector('[data-field="price"]')?.innerText?.replace(/[^0-9.]/g, '') || '',
          url: el.querySelector('a[href]')?.href || '',
          rating: el.querySelector('[data-field="rating"]')?.innerText?.replace(/[^0-9.]/g, '') || '0',
          inStock: el.querySelector('[data-field="stock"]')?.innerText.toLowerCase().includes('stock') || false,
          description: el.querySelector('[data-field="description"]')?.innerText || '',
          imageUrl: el.querySelector('img[data-field="image"]')?.src || ''
        }));
      });

      await browser.close();

      const validatedProducts = products.map(p => {
        const validation = validateProduct(p);
        return {
          ...p,
          validation
        };
      });

      return {
        success: true,
        products: validatedProducts.filter(p => p.validation.isValid),
        failedProducts: validatedProducts.filter(p => !p.validation.isValid),
        scrapedAt: new Date().toISOString()
      };
    } catch (error) {
      console.error(`Attempt ${attempt} failed:`, error.message);
      if (attempt === maxRetries) {
        return {
          success: false,
          error: error.message,
          attempt
        };
      }
      await new Promise(resolve => setTimeout(resolve, 2000 * attempt));
    }
  }
};

app.post('/scrape', async (req, res) => {
  const { url } = req.body;

  if (!url) {
    return res.status(400).json({ error: 'URL is required' });
  }

  const result = await scrapeWithRetry(url);
  res.json(result);
});

app.get('/health', (req, res) => {
  res.json({ status: 'healthy', timestamp: new Date().toISOString() });
});

const PORT = process.env.PORT || 3001;
app.listen(PORT, () => {
  console.log(`Scraper service running on port ${PORT}`);
});

Create the package.json:

{
  "name": "scraper-service",
  "version": "1.0.0",
  "description": "Production web scraping service",
  "main": "src/server.js",
  "scripts": {
    "start": "node src/server.js",
    "dev": "nodemon src/server.js"
  },
  "dependencies": {
    "express": "^4.18.2",
    "puppeteer": "^21.0.0",
    "axios": "^1.5.0"
  },
  "devDependencies": {
    "nodemon": "^3.0.1"
  }
}

Storage and Data Management

Your VPS needs persistent, queryable storage. I recommend PostgreSQL in the docker-compose stack above, but let’s configure it properly for production scraping. Create a migration script:

cat > /opt/scraping-pipeline/init-db.sql << 'EOF'
CREATE TABLE IF NOT EXISTS products (
  id BIGSERIAL PRIMARY KEY,
  external_id VARCHAR(255) UNIQUE,
  title VARCHAR(500) NOT NULL,
  price NUMERIC(10, 2),
  url VARCHAR(2048) UNIQUE NOT NULL,
  rating NUMERIC(3, 1),
  in_stock BOOLEAN DEFAULT true,
  description TEXT,
  image_url VARCHAR(2048),
  source VARCHAR(100),
  scraped_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE INDEX idx_products_url ON products(url);
CREATE INDEX idx_products_scraped_at ON products(scraped_at);
CREATE INDEX idx_products_source ON products(source);

CREATE TABLE IF NOT EXISTS scrape_logs (
  id BIGSERIAL PRIMARY KEY,
  workflow_name VARCHAR(255),
  items_scraped INTEGER,
  items_failed INTEGER,
  duration_seconds INTEGER,
  status VARCHAR(50),
  error_message TEXT,
  started_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  completed_at TIMESTAMP,
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE INDEX idx_scrape_logs_workflow ON scrape_logs(workflow_name);
CREATE INDEX idx_scrape_logs_status ON scrape_logs(status);
EOF

docker exec n8n-postgres psql -U n8n -d n8n -f /docker-entrypoint-initdb.d/init-db.sql

For larger datasets, consider adding object storage for raw HTML or images:

docker run -d \
  --name minio \
  --network scraping_net \
  -p 9000:9000 \
  -p 9001:9001 \
  -e MINIO_ROOT_USER=minioadmin \
  -e MINIO_ROOT_PASSWORD=minioadmin \
  minio/minio server /data --console-address ":9001"

Now create an n8n node that stores raw scraped data in S3-compatible MinIO:

{
  "parameters": {
    "action": "upload",
    "bucketName": "scraping-raw",
    "key": "{{ $now.toFormat('yyyy/MM/dd/HHmmss') }}-{{ $node['Scrape Products'].json.url.replace(/[^a-zA-Z0-9]/g, '-') }}.json",
    "fileContent": "{{ JSON.stringify($input.all(), null, 2) }}",
    "credentials": "minio-credentials"
  },
  "name": "Store Raw Data",
  "type": "n8n-nodes-base.awsS3",
  "typeVersion": 3.0,
  "position": [
    1250,
    300
  ]
}

Getting Started

You’ve now seen the complete architecture. Start small:

  1. Spin up a Hetzner VPS or DigitalOcean instance with at least 4GB RAM.

  2. Run the docker-compose stack to launch n8n, PostgreSQL, and the Puppeteer service.

  3. Deploy your first workflow using the JSON template above.

  4. Test it with a real website, monitoring logs and database inserts.

  5. Scale up by adding proxy rotation, multi-threaded execution, and error handling as shown in the advanced patterns.

The key advantage of VPS-based scraping is control and cost. You pay one flat rate regardless of how many pages you scrape, versus cloud platforms that charge per execution. For long-running pipelines that process 100,000+ pages daily, a VPS solution costs 1/10th as much.

When you’re ready to add complexity like integrating vector embeddings or building RAG pipelines on top of your scraped data, you can extend this setup to connect OpenAI embeddings to vector databases.

Outsource Your Automation

Don’t have time? I build production n8n workflows, WhatsApp bots, and fully automated YouTube Shorts pipelines. Hire me on Fiverr, mention SYS3-HUGO for priority. Or DM at chasebot.online.

Want to automate this yourself?

Start with n8n Cloud (free tier available) or self-host on a Hetzner VPS for full control.

Want this engine running on your own VPS?

This blog publishes itself — daily, unattended, on free API tiers. The full engine, Hugo theme, and setup guide are available as System 3.

Get System 3

Related Guides

system online