Setup Guide¶

This guide provides technical details for configuring your environment.

Environment Configuration¶

Onion Peeler uses a .env file to manage secrets and operational parameters. Create one by copying the example:

cp .env.example .env

Setting Up Tor¶

The framework expects a Tor service to be running locally or via Docker. By default, it looks for the Tor SOCKS proxy on 127.0.0.1:9050.

If you are using Docker, the deploy/docker-compose.yml file provides a pre-configured Tor instance.

Docker Environment Configuration¶

For a production-ready environment, Onion Peeler uses Docker Compose with a Gluetun VPN container (using Mullvad) to route all traffic securely.

VPN Configuration (Mullvad)¶

We use Gluetun to manage the VPN connection. You must provide your Mullvad Wireguard credentials in the .env file.

Key	Description	Example / Source
`WIREGUARD_PRIVATE_KEY`	Your 32-byte private key.	Generated from Mullvad dashboard.
`WIREGUARD_ADDRESSES`	Your assigned internal IP.	e.g., `10.64.0.1/32`
`SERVER_COUNTRIES`	Target VPN exit node location.	`Netherlands`
`SERVER_HOSTNAMES`	Specific server ID.	`nl-ams-wg-101`

How to get Mullvad credentials

Log in to Mullvad.net.

Go to Account -> Wireguard Config.

Generate a new key pair or select an existing one.

Download the .conf file.

Look for PrivateKey and Address in the [Interface] section of the file.

Tor & Scraper Configuration¶

Key	Description	Default
`TOR_CONTROL_PASSWORD`	Password for the Tor control port.	`secret_tor_password`
`TOR_ROTATION_INTERVAL`	Seconds between IP rotations.	`600`
`ONION_PEELER_SITE`	The site configuration to use.	`template`

Running the Stack¶

Once your .env is configured, start the full stack:

make up

This starts three interconnected containers: 1. dw_vpn: Routes all traffic through Mullvad via Wireguard. 2. tor: Provides the Tor proxy, running behind the VPN. 3. scraper: Executes the crawl, routed through the VPN for clearweb and Tor for .onion sites.

To verify the connection:

make logs-vpn

Check the output for a "VPN is up" message and a confirmation of your assigned IP.

Makefile Reference¶

We provide a Makefile for common operations.

Command	Description
`make build`	Install all dependencies with `uv sync`.
`make test`	Run the full test suite.
`make test-tor`	Verify Tor connectivity via Scrapy shell.
`make crawl site=X`	(Recommended) Run a crawl for site `X` inside Docker.
`make scrapyd cmd=X`	Run any Scrapy command `X` inside Docker.
`make status`	Check the status of the Docker containers.
`make logs`	Follow logs from all active services.
`make up` / `make down`	Start or stop the Docker infrastructure.
`make docs-serve`	Serve this documentation locally.

Security Warning

Never commit your .env file or any credentials to a public repository. The provided .gitignore is already configured to ignore .env and config/accounts/*.json.

Expanding Onion Peeler¶

Now that your environment is set up, you can create new site configurations in config/sites/ and run custom crawls without modifying the core application. Check out the Site Configuration Guide to learn how to create your first TOML configuration.