Development Guide¶

This guide covers the local development workflow for Onion Peeler, focusing on dependency management and testing strategies.

Dependency Management¶

We use uv as our primary package manager for its speed and robust environment isolation. However, standard venv and pip are still supported.

1. Installation & Syncing¶

First, ensure you have your chosen tool installed and the environment synchronized.

uv (Recommended)venv & pip

If you haven't installed uv yet:

curl -LsSf https://astral-sh.uv/install.sh | sh

To install the core project dependencies and create a virtual environment:

uv sync

Create the environment:

python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

Install dependencies:
```
pip install -e .
```

2. Dependency Groups¶

Onion Peeler uses optional dependency groups (e.g., dev for testing, docs for documentation).

uvvenv & pip

To install a specific group:

uv sync --group dev
uv sync --group docs

To install everything:

uv sync --all-groups

For legacy environments, you may need to install from pyproject.toml or requirements.txt manually if groups are not supported by your pip version:

pip install ".[dev,docs]"

3. Running Commands¶

Use these commands to execute scripts or CLIs within the project's environment.

uvvenv & pip

uv run scrapy list
uv run scrapy crawl daunt

Ensure your environment is activated first:

source .venv/bin/activate
scrapy list
scrapy crawl daunt

Testing¶

We use pytest for unit and integration testing.

Running All Tests¶

uvvenv & pip

uv run pytest

pytest

Testing Specific Components¶

uvvenv & pip

# Settings Loader
uv run pytest tests/test_settings.py

# Page Extraction
uv run pytest tests/test_pages.py

# Settings Loader
pytest tests/test_settings.py

# Page Extraction
pytest tests/test_pages.py

Connectivity & Quality¶

Tor Connectivity Test¶

To verify that your local or Dockerized Tor service is reachable by the Scrapy engine:

make test-tor

Linting and Formatting¶

We use Ruff for maintaining code quality.

Check for errors: make lint
Auto-fix formatting: make format