Start

Testing Guide

This document defines the testing policy and workflows for the START project.

Core Principles

Flow

Test Structure

Unit Tests

Script & Integration Tests

Running Tests

Use the gate rather than retyping a command: uv run start-gate <arm> runs exactly what CI runs. See Gate entry point for the arm list and generated numbers.

# Default arm: tests not marked slow, network, or integration
uv run start-gate

# One arm at a time (each is also a CI step)
uv run start-gate fast
uv run start-gate full
uv run start-gate docs
uv run start-gate gnn
uv run start-gate artifacts

# Every arm with its exact argv
uv run start-gate --list

# Focused local runs (not gate arms)
uv run pytest -k "domain"
uv run pytest tests/test_domain.py

Markers

Offline-First Testing

def test_file_processing(tmp_path):
    source = tmp_path / "input.json"
    source.write_text('{"a": 1}')
    result = process_file(source)
    assert result["a"] == 1

Environment Setup for Tests

# Non-GUI matplotlib backend
export MPLBACKEND=Agg

# Optional: keys for an explicitly invoked live provider probe
export PERPLEXITY_API_KEY="..."
export OPENROUTER_API_KEY="..."

Coverage

The full arm enforces the 80% branch-aware working baseline and is the only coverage pass in CI; the measurement is recorded in _generated/gate-timings.md. The manual release workflow is fail-closed at the 90% branch-aware floor and adds the publication output gate. Linux runs cover Python 3.11–3.12 (fast on 3.11, full on 3.12); the macOS smoke job uses the locked Python 3.12 environment and runs the smoke-macos arm.

The Linux matrix explicitly passes its Python version to both uv sync and uv run; installing an interpreter alone does not override the repository’s Python 3.12 pin. Before the selected gate runs, CI checks the actual version, virtual-environment prefix and executable directory. The required test rollup still requires every existing CI job to succeed; failed, cancelled or skipped jobs cannot satisfy it.

Repository Validation

uv run start-gate docs      # strict MkDocs build + repository validation
uv run start-gate numbers   # generated gate numbers match scripts/gate.py

The repository validator parses tracked JSON/YAML/TOML files, rejects duplicate YAML keys, checks authored Markdown links, rejects references to retired language-layout paths, and blocks project-authored terminology that would imply non-real provider or test behavior. Generated curriculum/research snapshots and vendored third-party artifacts are excluded from authored-text terminology checks while remaining subject to structural config parsing when applicable.

Best Practices