Skip to content

CLI Usage Guide

This guide covers the practical commands for interacting with the Onion Peeler application. We recommend using the Docker-based commands to ensure your traffic is properly routed through the VPN and Tor.

Listing Available Spiders

Onion Peeler dynamically generates spiders based on your TOML configurations. To see which spiders are currently registered:

make scrapyd cmd="list"

docker compose run --rm scraper list

uv run scrapy list

Running a Full Crawl

To start a crawling process, use the crawl command followed by the spider name.

# Results saved to daunt_output.json automatically
make crawl site=daunt

docker compose run --rm scraper crawl daunt -o output.json

uv run scrapy crawl daunt -o output.json

Parsing a Single URL

The parse command is invaluable for testing selectors or debugging extraction logic without running a full crawl.

make scrapyd cmd='parse "https://daunt.link"'

docker compose run --rm scraper parse "https://daunt.link"

uv run scrapy parse "https://daunt.link"

Debugging with Scrapy Shell

To interactively test CSS or XPath selectors against a live response:

make scrapyd cmd='shell "https://daunt.link"'

docker compose run --rm scraper shell "https://daunt.link"

uv run scrapy shell "https://daunt.link"

Inside the shell, you can test your configuration selectors: 

response.css("div.mirror-list .mirror")