Testing as the forcing function for architectural change

plsec has a three-tier test strategy: Tier 1 (pure logic, no mocking), Tier 2 (filesystem with tmp_path), and Tier 3 (subprocess mocking). Tiers 1 and 2 were clean. Tier 3 was where the trouble surfaced.

Writing subprocess-mocking tests for doctor.py required 5 simultaneous mocks and offered zero pure-logic seams. Testing scan.py meant mocking 4 bespoke run_*() functions individually. Every test was fighting the structure of the code rather than verifying its behaviour.

The test difficulty was not a testing problem. It was an architecture problem surfaced by tests. The code was hard to test because knowledge about agents, scanners, and processes was scattered across command files rather than declared in one place.

Before proposing any solution, we traced the change cost of two scenarios: adding a new AI agent and adding a new security scanner. The results were unambiguous.

The root cause was scattered metadata. Agent knowledge lived in 12+ if/elif branches across init.py, secure.py, create.py, validate.py, and doctor.py. Each branch hardcoded which file to create, which template to use, and which detector field to check. Scanner invocation was similarly ad hoc: 4 separate run_<tool>() functions in scan.py, each constructing subprocess calls with bespoke argument lists.

One specific pattern illustrated the scaling problem. The CLI used "both" as a special value for the --agent flag: AgentType = Literal["claude", "opencode", "both"]. This appeared in 3 command files. Adding a third agent would make "both" ambiguous. The concept should have been "all configured agents", not an enumerated variant.

The refactoring was designed collaboratively between a human architect and Claude. The division of labour was deliberate and consistent throughout.

The human's role: problem identification from testing experience, codebase trace, architectural direction (entity-operation separation), design principles, phase sequencing, and the decision to use existing tests as the regression safety net. Every architectural judgment was human-driven.

The agent's role: detailed design specification (dataclass definitions, registry shapes, consumer migration patterns), before/after code examples, impact analysis tables, and consistency verification across the design document. The agent did the high-volume design labour under architectural direction.

The resulting design document carries the credit line "Gylthuin (with AI Assistance from Claude)" -- reflecting this division accurately. The architecture is human-authored. The specification is collaboratively produced.

Five design principles governed the refactoring:

Declare once, consume everywhere. Metadata about agents, scanners, and processes lives in registry modules. Commands iterate the registries; they do not hardcode entity knowledge.

Entities and operations are separate concerns. Registries describe what exists (nouns). Commands describe what to do (verbs). The interface between them is iteration.

New entity, one file change. Adding a new agent requires adding one AgentSpec entry. Adding a new scanner requires one ScannerSpec entry. No if/elif changes in command files.

Testable without mocking. Registry data and check functions are pure or near-pure -- taking Path arguments, not calling global state internally. Mocking is reserved for subprocess calls, not for plumbing.

Don't over-abstract. The registries are plain dicts of dataclasses, not metaclass-driven discovery mechanisms. If it doesn't simplify testing or reduce duplication, it doesn't belong.

  Entities (Registries)              Operations (Commands)
  =====================              =====================

  AGENTS                  ------>    CREATE / INIT / SECURE
  (claude, opencode, ...)            "produce config files"

  SCANNERS                ------>    SCAN
  (trivy-secrets, bandit, ...)       "run security checks"

  PROCESSES               ------>    PROXY (start/stop/status)
  (pipelock, ...)                    "manage background services"

  All three               ------>    DOCTOR / STATUS
                                     "check environment health"

The consumer code tells the story more clearly than the registry definitions. Here is how secure.py's change calculation transformed -- from per-agent blocks to a generic loop.

# 15 lines per agent, duplicated
if "claude" in state.agents:
    content = CLAUDE_MD_STRICT \
        if is_strict else CLAUDE_MD_BALANCED
    if not info.has_claude_md:
        changes.creates.append(...)
    elif force:
        changes.modifies.append(...)
    else:
        changes.conflicts.append(...)

# Same 15 lines for opencode
if "opencode" in state.agents:
    ...

# Generic, works for any agent
for agent_id in state.agents:
    spec = AGENTS[agent_id]
    _add_agent_config_change(
        changes, spec, info,
        security_mode(state.preset),
        force,
    )

# Adding Gemini: zero changes here.
# Just add AgentSpec to registry.

The same transformation applied across every consumer. The full impact table:

The refactoring was split into phases so the test suite stayed green at every step. The existing 216 tests (Tiers 1 and 2) served as the regression safety net. If any failed during consumer rewiring, the change was incorrect.

The refactoring added 216 new tests (142 registry + 74 command), bringing the total from 216 to 432, with coverage increasing from an unmeasured baseline to 69%. The Tier 3 tests that initially exposed the problem now wrote cleanly against the registry-driven architecture.

The registry model subsequently enabled the engine architecture. The engine registry (EngineRegistry) follows the same pattern: declare engines in a registry, the orchestrator iterates them. Adding a new security engine means implementing one Engine subclass and calling register(). No control flow changes.

Testing difficulty is an architecture signal, not a testing problem. When Tier 3 tests required extensive mocking, the correct response was not "improve the mocks" but "fix the code structure so the logic is testable without mocks." The tests were functioning as a design pressure, not just a verification mechanism.

Evidence before solution. The codebase trace (35 files, 50+ lines) preceded the design. The numbers made the case for the refactoring in a way arguing about principles could not. Quantifying the change cost took an hour. It saved weeks of debate about whether the refactoring was justified.

Agents are good at elaboration, bad at judgment. The agent produced detailed dataclass definitions, migration tables, and consumer impact analyses quickly and accurately. It did not identify the architecture problem, articulate the design principles, or decide the phasing strategy. The human-agent boundary was at the judgment/elaboration line, and maintaining it was essential.

Existing tests are the best refactoring safety net. The 216 pre-existing tests caught every consumer-rewiring mistake immediately. The phased approach (registries first, consumers second, new tests last) meant we always had a green test suite to fall back to. Writing new tests before the refactoring would have been wasted effort -- they would have tested the wrong structure.

Registries reduce the blast radius of AI-assisted code changes. With scattered metadata, an agent modifying plsec's codebase had to understand 12+ if/elif branches across 7 files to make a consistent change. With registries, the same change is a single entry in a single file. The architecture now matches the granularity at which agents (and humans) can reliably work.