local-deep-research/docs/developing.md

# Developer Guide

## Architecture Documentation

- **[Architecture Overview](architecture/OVERVIEW.md)** - System components, research flow, and module responsibilities
- **[Database Schema](architecture/DATABASE_SCHEMA.md)** - Database models and relationships
- **[Extension Guide](developing/EXTENDING.md)** - How to add custom search engines, strategies, and LLM providers
- **[Testing and CI](CI_CD_INFRASTRUCTURE.md)** - GitHub Actions workflows, pre-commit hooks, and security scanning

## Development Setup

### Prerequisites
- **Python**: 3.12+
- **Node.js**: 24.0.0+
- **Docker**: Latest version (for production builds)
- **PDM**: Python package manager
- **SQLCipher**: Required for encrypted database support. See [SQLCIPHER_INSTALL.md](SQLCIPHER_INSTALL.md) for platform-specific instructions.

### Initial Setup

1. **Clone and Prepare Environment**
   ```bash
   git clone git@github.com:LearningCircuit/local-deep-research.git
   # Or via HTTPS: https://github.com/LearningCircuit/local-deep-research.git
   cd local-deep-research
   ```

2. **Backend Setup**
   ```bash
   # Create and activate virtual environment
   python -m venv .venv
   source .venv/bin/activate

   # Install dependencies
   pip install pdm
   pdm install --no-self
   ```

3. **Frontend Setup**
   ```bash
   # Install frontend dependencies
   npm install
   ```

4. **Enable Git Hooks**
   ```bash
   # Install pre-commit hooks
   pre-commit install
   pre-commit install-hooks
   ```
   We use the `pre-commit` framework to manage git hooks. This repository includes both standard checks (ruff, eslint, gitleaks) and custom local checks located in `.pre-commit-hooks/`.

## Running the Application

### Development Mode

1. **Start the Backend**
   ```bash
   source .venv/bin/activate

   # Option A: Using the installed entry point
   ldr-web

   # Option B: Using Python module directly
   python -m local_deep_research.web.app
   ```

2. **Start the Frontend** (in a new terminal)
   ```bash
   npm run dev
   ```
   Access the app at `http://localhost:5173`.

### Development Environment Variables

For the full list of all settings and environment variables, see [CONFIGURATION.md](CONFIGURATION.md).

For local development and testing, you may want to configure these environment variables:

| Variable | Default | Description |
|----------|---------|-------------|
| `LDR_DATA_DIR` | Platform default | Override data/database storage location |
| `LDR_BOOTSTRAP_ALLOW_UNENCRYPTED` | `false` | Allow unencrypted database (dev only!) |
| `CI` or `TESTING` | unset | Enables testing mode (bypasses some security checks) |

> **Warning**: Never set `LDR_BOOTSTRAP_ALLOW_UNENCRYPTED=true` in production. User data will be stored unencrypted.

### Docker (Production-like)

To build and run the entire stack in Docker:

```bash
docker build -t localdeepresearch/local-deep-research:dev .
docker run -p 5000:5000 -e LDR_DATA_DIR=/data -v ldr_data:/data localdeepresearch/local-deep-research:dev
```

### Testing a Release Candidate (Prerelease Image)

When a release is being cut, `.github/workflows/prerelease-docker.yml` publishes
the RC under two tags on Docker Hub:

- **`prerelease-vX.Y.Z-<sha>`** — immutable, pinned to one specific build. Use
  this when you want a reproducible test target (e.g. when filing a bug).
- **`prerelease`** — floating alias that always points at the most recent RC.
  Use this for quick "try the next release" testing without bumping the tag
  every cycle.

The simplest way to test an RC alongside your existing stable instance is to
add a second service to `docker-compose.yml` with a different port and isolated
volumes, so a broken migration in the RC can't damage your production DB:

```yaml
  local-deep-research-pre:
    image: localdeepresearch/local-deep-research:prerelease
    container_name: local-deep-research-pre
    networks:
      - ldr-network
    extra_hosts:
      - "host.docker.internal:host-gateway"
    ports:
      - "5001:5000"                # production stays on 5000
    environment:
      - LDR_WEB_HOST=0.0.0.0
      - LDR_WEB_PORT=5000
      - LDR_DATA_DIR=/data
      - LDR_LLM_OLLAMA_URL=http://ollama:11434
      - LDR_SEARCH_ENGINE_WEB_SEARXNG_DEFAULT_PARAMS_INSTANCE_URL=http://searxng:8080
    volumes:
      - ldr_data_pre:/data        # ← separate from production ldr_data
      - ldr_scripts_pre:/scripts
    restart: unless-stopped

volumes:
  ldr_data_pre:
  ldr_scripts_pre:
```

> Copy the `ulimits`, `security_opt`, `cap_drop`, and `cap_add` blocks from the
> main `local-deep-research` service for the same hardening — they are required
> for correct startup, not optional.

Then pull and start just the prerelease service:

```bash
docker compose pull local-deep-research-pre
docker compose up -d local-deep-research-pre
```

The UI will be at `http://localhost:5001`. To upgrade to the next RC after it
ships, re-run `docker compose pull local-deep-research-pre && docker compose up -d local-deep-research-pre`.

## Building

### Building a Package

To build a wheel and source distribution, simply run `pdm build`.

### Building Frontend Assets

If you're developing from source and want to use the Web UI, you need to build the frontend assets:

```bash
npm install
npm run build
```

This builds the Vite frontend into `src/local_deep_research/web/static/dist/`.

> **Note:** pip users don't need this step - pre-built assets are included in the PyPI package.

### Dependency Lockfile Management

This project uses **PDM** with `pdm.lock` to pin exact dependency versions. If you see this warning during Docker builds:

```
WARNING: Lockfile hash doesn't match pyproject.toml, packages may be outdated
```

It means `pyproject.toml` has changed but `pdm.lock` hasn't been regenerated.

**How to fix:**
```bash
pdm lock
```

Always commit `pdm.lock` alongside `pyproject.toml` changes to ensure reproducible builds.

## Testing

### Backend Tests (Pytest)

We support two modes of backend testing:

#### 1. Isolated Testing (No Server Required)
This is the default and recommended way to run unit and integration tests. It uses Flask's `test_client` to mock the server.

**How to run:**
```bash
source .venv/bin/activate
# Run all isolated tests
pytest tests/

# Run specific API or Auth tests
pytest tests/api_tests/
pytest tests/auth_tests/
```

#### 2. Live System Testing (Requires Running Server)
These tests make real HTTP requests to a running application instance.

**Prerequisites:**
1. Start the backend server in one terminal: `ldr-web` (or via Docker).
2. Run the live test scripts in another terminal.

**How to run:**
```bash
python tests/ui_tests/test_simple_research_api.py
```

### Frontend & E2E Tests (Puppeteer)
We use `Puppeteer` for UI and End-to-End testing.

**Prerequisites:**
1. Navigate to the tests directory: `cd tests`
2. Install test dependencies: `npm install`

**How to run:**
1. **Ensure the application is running** (locally on port 5000 or via Docker).
2. Run the test suite:
   ```bash
   # Run all UI tests
   node tests/ui_tests/run_all_ui_tests.js

   # Run specific test
   node tests/ui_tests/test_simple_auth.js
   ```

## Database Management

### Backup & Restore
Before testing changes, you may wish to backup your Local Deep Research data volume.

**Create a Backup**
```bash
docker run --rm \
  -v ldr_data:/from \
  -v ldr_data-backup:/to \
  debian:latest \
  bash -c "cd /from ; tar -cf - . | (cd /to ; tar -xpf -)"
```

**Restore from Backup**

*Warning: This overwrites existing data*
```bash
docker run --rm \
  -v ldr_data:/target \
  -v ldr_data-backup:/source \
  debian:latest \
  bash -c "rm -rf /target/* /target/.[!.]* ; \
           cd /source ; tar -cf - . | (cd /target ; tar -xpf -)"
```

## Troubleshooting

### SQLCipher Issues

See [SQLCIPHER_INSTALL.md](SQLCIPHER_INSTALL.md#troubleshooting) for SQLCipher-related errors.

### Permission Denied on Docker Volume

**Error:** `PermissionError: [Errno 13] Permission denied: '/app/.config/...'`

**Solution:** The volume may have been created with different ownership. Reset it:
```bash
docker volume rm ldr_data
# Re-run the container to create a fresh volume
```

### Session Lost After Server Restart

**Cause:** The application's secret key is used to sign session cookies.

**Solution:** The secret key is automatically generated on first run and persisted to a `.secret_key` file in your data directory (controlled by `LDR_DATA_DIR`). Sessions are only lost if this file is deleted. If using Docker, ensure your data volume (`ldr_data:/data`) is persistent.

### NoSettingsContextError in Background Threads

**Error:** `NoSettingsContextError: No settings context available`

**Cause:** Background threads don't have access to the Flask request context.

**Solution:** This is handled automatically by the application. If you encounter this during development, ensure your LLM settings are configured via the web UI settings page.

### PDM Lockfile Out of Sync

**Error:** `Lockfile hash doesn't match pyproject.toml`

**Solution:**
```bash
pdm lock
git add pdm.lock
```