Testing and Quality

The test suite is treated as part of the product, not an afterthought. This page describes what it covers and the philosophy behind how it is structured.

Scope

The suite runs on every push across Python 3.10, 3.11, and 3.12, on both Linux and Windows, plus a coverage job. It covers the analyzer, the generator factory, the defensive manager and fallback ladder, the validators, the configuration and template layers, the API endpoints, and the CLI.

Execution-level testing of generated code

The most important quality decision in the project: the generated code is tested by running it, not merely by compiling it.

This distinction is not academic. An earlier version of the suite checked that generated generators parsed and compiled, and they did, while several of them raised NameError the moment they were actually executed, because a code path referenced a name only one template defined. Compilation is not execution; syntactically valid code can still be wrong. The suite now generates code for every template type, executes it, and asserts on the records it produces: correct count, correct shape, populated non-nullable columns, and identical output under a fixed seed.

Regressions are encoded, not just fixed

Several tests exist because of specific bugs found the hard way. Each is kept as a named regression test so the same mistake cannot return quietly:

  • Cycle detection in dependency sorting: a cyclic foreign-key graph must resolve to a defined order, not loop forever.

  • Fallback observability: a defect once caused a whole generation level to be silently skipped while fallbacks masked it; the suite now asserts the expected level actually runs.

  • Cross-process determinism: set-iteration order and embedded timestamps are both tested, because both silently break byte-level reproducibility.

  • Windows tempfile semantics: a NamedTemporaryFile must be closed before it can be unlinked on Windows; tests follow the safe convention.

Determinism as a tested property

Reproducibility is verified, not assumed. Tests confirm that a seeded deploy produces byte-identical output across runs and that a seeded generator instance produces identical records. See Design Principles.

Running the suite

pip install -e ".[dev]"
pytest