pytest

pytest#

Test Data Workbench generates code, and generated code is exactly the kind of artifact that is easy to get subtly wrong while looking right: it parses, it has a plausible shape, and it still fails the moment something calls it. pytest is the tool that catches that gap.

Why pytest#

The project needs a test runner that can express three things without fighting the framework: fixtures composed from other fixtures (a schema built from tables built from columns), async tests (much of the analysis and deployment path is async def), and enough test files that plain assert statements with readable failure output matter more than they would in a five-file project. pytest gives all three natively: fixtures, pytest-asyncio integration, and assertion introspection that reports what a failed assert actually compared, with no custom assertion methods to learn. The suite defines 358 test functions across 18 modules under tests/ (more once parametrized cases are expanded), runs on every push across three Python versions and two operating systems, and it is the reason the project can claim its generated generators actually work rather than merely compile.

The idea underneath#

pytest’s fixture system is dependency injection applied to test setup. A test function declares what it needs as parameters, for example def test_x(self, user_table, defensive_manager):, and pytest resolves each name against a fixture function, calls it, and hands the return value in. Compare that to the older xUnit style (Python’s own unittest.TestCase, Java’s JUnit 3/4), where setup lives in an inherited setUp/tearDown pair that runs unconditionally for every test in the class whether that test needs it or not, and where sharing setup across classes means subclassing. pytest fixtures are requested, not inherited: a test asks for exactly the fixtures it needs, fixtures can themselves request other fixtures, and a fixture can be swapped for a different implementation at any scope without touching the tests that consume it. The official documentation calls this out directly as the design’s reason for existing, better than the classic xUnit style of setup/teardown functions.

Every test in this codebase still follows the older, more universal shape underneath the fixture plumbing: arrange, act, assert. Build the input (often by requesting a fixture that already built it), call the thing under test, check the result. tests/test_pii_detection.py makes the three phases almost mechanically visible:

def test_flags_email_ssn_and_address(self, analyzer, mixed_schema_tables):
    pii = analyzer._detect_pii_columns(mixed_schema_tables)          # act
    flagged = {(p.table, p.column): p.category for p in pii}

    assert flagged[("users", "email")] == "email"                    # assert
    assert flagged[("users", "ssn")] == "ssn"
    assert flagged[("users", "street_address")] == "address"

analyzer and mixed_schema_tables are the arrange step, supplied as fixtures rather than written inline.

A second idea worth naming explicitly: the vocabulary distinction between test doubles. Martin Fowler’s “Mocks Aren’t Stubs” is the canonical reference: a stub returns canned answers, a fake is a real, working, simplified implementation, and a mock additionally records how it was called so the test can assert on those calls after the fact (behavior verification, as opposed to the state verification a stub or fake supports). tests/test_defensive_manager.py uses this distinction concretely without naming it: GoodGenerator, BadGenerator, and FallbackGenerator are hand-written classes with a real, working generate() method, that is, fakes, used to drive DefensiveGeneratorManager through its success, failure, and fallback paths deterministically. None of them record calls for later assertion, so nothing in that file is a mock in Fowler’s strict sense, even though unittest.mock.MagicMock is imported at the top of that file and of tests/test_integration.py. Neither import is actually used; the suite reaches for fakes, not mocks, everywhere it needs a test double. See Sharp edges and limits for what that implies about the pytest-mock dependency.

The last idea is specific to testing a code generator rather than a function: the unit under test does not return a value, it returns source code that, when run, produces a value. Asserting on the string is weak, because syntactically valid Python can still be behaviorally wrong. tests/test_generated_execution.py names this directly in its module docstring, and the fix it applies, actually executing the generated artifact, is the strongest form of testing a generator: closer in spirit to an acceptance test of the generator’s output than a unit test of the generator’s text.

How it fits this project#

pytest is a dev extra, not a runtime dependency: pyproject.toml lists pytest>=7.4.3, pytest-asyncio>=0.21.1, and pytest-mock>=3.12.0 under [project.optional-dependencies] dev, installed with pip install -e ".[dev]". Configuration lives in [tool.pytest.ini_options] in the same file:

[tool.pytest.ini_options]
testpaths = ["tests"]
asyncio_mode = "auto"
markers = [
    "asyncio: mark a test as an async test",
    "slow: mark a test as slow-running",
]

testpaths scopes collection to tests/. The suite is organised as 18 test_*.py modules plus one shared tests/conftest.py, with no per-subdirectory conftest.py files, no separate unit/ and integration/ trees, and one file, tests/test_postgres_integration.py, that is real-database rather than SQLite-backed. Every module groups its tests into one or more class Test...: containers (18 files, each with at least one such class); pytest does not require this, it collects bare def test_... functions equally well, but grouping by class here also groups the fixtures and helper methods those tests share.

Fixtures and conftest#

tests/conftest.py is the spine of the suite: fixtures defined there are visible to every test module without an import, because pytest auto-discovers conftest.py files by directory scope. It defines the shared schema vocabulary the rest of the suite builds on:

@pytest.fixture
def user_table():
    """Minimal users table."""
    return Table(
        name="users",
        columns=[
            Column("id", "integer", False, constraints=[ConstraintType.PRIMARY_KEY]),
            Column("username", "varchar(50)", False),
            Column("email", "varchar(100)", False),
            ...
        ],
        row_count=1000,
        entity_type=EntityType.USER,
    )

@pytest.fixture
def demo_schema(user_table, product_table, order_table,
                 sample_relationships, sample_business_rules):
    """Full demo schema with tables, relationships, and rules."""
    return SchemaInfo(
        database_name="test_ecommerce",
        tables=[user_table, product_table, order_table],
        relationships=sample_relationships,
        business_rules=sample_business_rules,
    )

demo_schema requesting user_table, product_table, and order_table as its own arguments is fixtures composing fixtures: pytest resolves the whole dependency graph before the test ever runs. Individual test files add their own local fixtures on top of the shared ones, scoped to that file only, for example customer_table and generic_table in tests/test_generated_execution.py, or the nested factory fixture redeclared inside several TestXxx classes across the suite (a fixture defined inside a class is only visible to tests in that class).

Testing generated code by executing it#

tests/test_generated_execution.py exists because tests/test_generator_factory.py only ever runs compile(code, ..., "exec") on generated source, which proves the code parses and never calls generate(). That gap let a real bug ship: a type-based fallback emitted a bare fake.xxx() expression, but only the user/customer template defines a module-level fake = Faker(); every other template raised NameError the moment generate() actually ran. The fix is a small harness that execs the generated source into an isolated namespace, pulls out the one class ending in Generator, and then calls it for real:

def _load_generator_class(code: str):
    namespace: dict = {}
    exec(compile(code, "<generated>", "exec"), namespace)
    matches = [
        value for key, value in namespace.items()
        if key.endswith("Generator") and isinstance(value, type)
    ]
    assert len(matches) == 1, ...
    return matches[0]

def test_user_generator_executes(self, factory, user_table):
    schema = SchemaInfo("test", [user_table], [], [])
    code = factory.create_generators_from_schema(schema).generators["users"]
    generator_cls = _load_generator_class(code)

    records = generator_cls().generate(5)
    _assert_valid_records(records, user_table, 5)

Every entity template (user, customer, product, order, review, generic, fallback) gets this same treatment, plus dedicated classes for sequential primary-key behavior (TestSequentialPrimaryKeys) and name-based temporal-column detection (TestNameBasedTemporalDetection), each again executing the generated class rather than reading its source text.

asyncio_mode auto and testing async code#

asyncio_mode = "auto" tells pytest-asyncio to treat every async def test_... function as an asyncio test automatically, with no per-test marker required. 22 test functions across the suite are async def, and every one of them uses async/await because the code under test is: SchemaAnalyzer.analyze_production_schema and the CLI’s analyze/ deploy entry points are coroutines, so exercising them means awaiting them.

@pytest.mark.asyncio
async def test_no_select_statements_touch_table_data(self, tmp_path):
    conn_str = _make_db_with_data(tmp_path / "meta.db")
    analyzer = SchemaAnalyzer()

    with _SqlCapture() as capture:
        schema_info = await analyzer.analyze_production_schema(conn_str, metadata_only=True)

The explicit @pytest.mark.asyncio on that test (and on the rest of the async tests in the suite) is redundant under auto mode, pytest-asyncio would collect and run it correctly without the decorator, but it is not wrong: the decorator is harmless in auto mode and documents intent at the call site. markers = ["asyncio: ...", "slow: ..."] in pyproject.toml registers both names so pytest does not emit an unknown-marker warning when it sees them, independent of whether asyncio_mode needs the marker to function.

Markers and selecting subsets#

Two custom markers are registered in pyproject.toml: asyncio (used throughout, as above) and slow. slow is declared but not currently applied to any test in the suite, there is no -m slow or -m "not slow" invocation anywhere in CI or the docs either. The mechanism to select subsets exists (pytest -m asyncio would run only the async tests, pytest -m "not slow" would skip anything tagged slow once something is), it is just unused today. The suite instead splits by file: the Postgres-only module is run as a separate, explicit invocation rather than filtered out of a combined run with a marker, see Integration tests against a real database below.

pytest.mark.skipif is used for conditional skipping instead, gating an entire module rather than individual tests:

pytestmark = pytest.mark.skipif(
    not TDW_TEST_POSTGRES_URL,
    reason="TDW_TEST_POSTGRES_URL is not set - skipping real Postgres integration test",
)

Assigning to the module-level pytestmark name applies the mark to every test collected from that module, which is how tests/test_postgres_integration.py skips itself cleanly in any environment without the environment variable set, rather than erroring.

pytest.mark.parametrize is the suite’s other marker, used for one thing: turning a long list of name/category pairs into individually reported test cases in tests/test_pii_detection.py:

@pytest.mark.parametrize("column_name,expected_category", [
    ("email", "email"),
    ("user_email", "email"),
    ("ssn", "ssn"),
    ("social_security_number", "ssn"),
    ...
])
def test_flags_expected_category(self, column_name, expected_category):
    assert classify_column_pii(column_name) == expected_category

Each tuple becomes its own test ID in the pytest report, so a failure names the exact column-name pattern that broke rather than reporting one failure for a loop over 30-odd cases.

Mocking, and mostly not needing it#

pytest-mock is a declared dev dependency, but its mocker fixture does not appear anywhere in tests/. The only mock-adjacent imports in the whole suite are from unittest.mock import MagicMock in tests/test_defensive_manager.py and tests/test_integration.py, and neither file actually constructs a MagicMock anywhere, both imports are dead. Where the suite needs a substitute implementation, it writes a small, real class instead, a fake in Fowler’s terms:

class BadGenerator:
    """Always raises."""
    def __init__(self, ref_data=None):
        pass

    def generate(self, count):
        raise RuntimeError("primary exploded")

def test_fallback_on_primary_failure(self, defensive_manager):
    defensive_manager.register_generator(
        "users", BadGenerator, fallback_class=FallbackGenerator
    )
    result = defensive_manager.generate_safely("users", 4)
    assert result.generator_used == "fallback"

This fits the project generally: most of the suite exercises real SQLite databases through tmp_path rather than mocking a database connection, and, as described above, executes real generated code rather than mocking what a generator would return. A BadGenerator that genuinely raises is a closer approximation of a broken user-contributed generator than a Mock(side_effect=RuntimeError) would be, and it exercises the real generate(count) call signature rather than an unconstrained mock that would accept any call shape.

Integration tests against a real database#

Most of the suite runs entirely against SQLite, built fresh per test with pytest’s tmp_path fixture (a pathlib.Path unique to that test, cleaned up automatically). tests/test_metadata_only.py is a strong example of testing a negative with this pattern: proving that metadata_only=True issues zero SQL against table data cannot be done by inspecting the returned SchemaInfo alone, since an absent field does not prove a query was never sent, only that its result was discarded. The test instead attaches a SQLAlchemy event listener to every engine created during the call and captures the literal SQL text:

class _SqlCapture:
    def __init__(self):
        self.statements = []

    def __call__(self, conn, cursor, statement, parameters, context, executemany):
        self.statements.append(statement)

    def __enter__(self):
        event.listen(sa.engine.Engine, "before_cursor_execute", self)
        return self

    def __exit__(self, *exc_info):
        event.remove(sa.engine.Engine, "before_cursor_execute", self)

@pytest.mark.asyncio
async def test_no_select_statements_touch_table_data(self, tmp_path):
    ...
    with _SqlCapture() as capture:
        schema_info = await analyzer.analyze_production_schema(conn_str, metadata_only=True)

    offending = _select_statements_touching(capture.statements, TABLE_NAMES)
    assert offending == [], f"metadata_only=True issued SQL against table data: {offending}"

The file also carries a deliberate contrast class, TestDefaultModeStillReadsTableData, that asserts the opposite on the default (non-metadata_only) path: SELECTs against table data ARE issued. Its docstring explains why: if that assertion ever failed, it would mean the SQL-capture mechanism itself was broken, which would make the zero-queries test above a false pass rather than a real one. Testing the capture mechanism’s ability to detect data access is what gives the negative assertion teeth.

tests/test_postgres_integration.py goes one step further and tests against a real PostgreSQL server rather than SQLite, because SQLite and Postgres do not reflect identically (self-referential foreign keys, for one, mirrored deliberately in this test’s schema). The whole module is gated behind an environment variable so the default local and CI run is unaffected:

TDW_TEST_POSTGRES_URL = os.environ.get("TDW_TEST_POSTGRES_URL")

pytestmark = pytest.mark.skipif(
    not TDW_TEST_POSTGRES_URL,
    reason="TDW_TEST_POSTGRES_URL is not set - skipping real Postgres integration test",
)

Locally that means docker compose up -d (docker-compose.yml at the repo root) plus exporting the variable; in CI, a dedicated postgres job in .github/workflows/ci.yml starts a postgres:16 service container, waits on its health check, and runs pytest tests/test_postgres_integration.py -v as its own step, separate from the main SQLite-only matrix. It requires the postgres extra (psycopg2-binary), installed only in that job.

Determinism and seeded tests#

Reproducibility is treated as a property to assert on, not an assumption. tests/test_generated_execution.py defines one small helper used across every generator-template test:

def _assert_seed_determinism(generator_cls, **extra_kwargs):
    """Two instances built with the same seed must produce identical output."""
    first = generator_cls(seed=42, **extra_kwargs).generate(5)
    second = generator_cls(seed=42, **extra_kwargs).generate(5)
    assert first == second

TestSequentialPrimaryKeys extends the same idea to auto-increment primary keys, checking that a counter continues across multiple generate() calls on one instance ([1, 2, 3] then [4, 5], not [1, 2, 3] then [1, 2]), and that a non-integer (UUID) primary key is never coerced into that sequential strategy in the first place. This is the test-level half of the byte-level reproducibility described in Design Principles; the design document explains why determinism matters for CI fixtures, this suite is what actually checks it holds.

The CI matrix#

.github/workflows/ci.yml defines three jobs. The main test job runs python -m pytest --tb=short -q across a matrix of os: [ubuntu-latest, windows-latest] times python-version: ["3.10", "3.11", "3.12"] with fail-fast: false, six combinations, none of which touch Postgres. A separate coverage job runs once, on ubuntu-latest with Python 3.12, installing pytest-cov on top of the dev extra and running pytest --cov=test_data_workbench --cov-report=term --cov-report=xml -q, uploading coverage.xml as a build artifact. The third job, postgres, is described above. Nothing in the workflow fails the build on a coverage threshold; see Sharp edges and limits.

Sharp edges and limits#

Coverage is measured, not enforced. The coverage CI job produces a percentage and an XML artifact and stops there, no --cov-fail-under, no gate on the pull request. A coverage regression is visible to a human who opens the job output, not something CI will block on by itself.

The slow marker is declared but unused. slow is registered in pyproject.toml and never applied to a test. There is currently no way to run “the fast subset” of this suite; every local pytest invocation runs everything under testpaths.

pytest-mock is a dependency the suite does not use. The mocker fixture it provides is never requested. The two unittest.mock.MagicMock imports that do exist are dead code, imported and never called. This is a defensible position, hand-written fakes are arguably a better fit for testing a fallback ladder than a generic mock object, but it does mean the dependency itself is currently unearned; either it should be used somewhere that benefits from call-assertion behavior a fake cannot give (verifying a generator was not called after a defensive fallback fires would be one candidate), or it could be dropped.

Auto mode applies to every async test, project-wide, with no per-test opt-in. asyncio_mode = "auto" treats every async def test_... function as an asyncio test automatically. That is exactly what this codebase wants today, every async test genuinely exercises an awaited code path, but it is a global setting: a future async def test_... added purely to use async with on a non-async resource, or written by mistake, would silently become an asyncio test too, rather than failing loudly as “async def used without a marker” the way strict mode would. Auto mode trades that guardrail for the convenience of never having to remember the decorator, which is a reasonable trade at this suite’s current size and consistency, but it is a trade, not a free improvement.

Rigor is not even across the suite. tests/test_generated_execution.py and tests/test_metadata_only.py are recent, tightly targeted, and each exists because of a specific bug or specific claim (“zero SQL against table data”) that needed proof. tests/test_integration.py is older and looser: assertions like assert len(generator_set.generators) >= 2 and assert result.valid or len(result.errors) == 0 tolerate a wider range of outcomes than the newer files’ exact-value assertions, and the module still carries an if __name__ == "__main__": block that shells out to subprocess.run([sys.executable, "-m", "pytest", __file__, ...]) with emoji in its printed summary, a pattern from before the project standardized on running the suite through plain pytest. It still passes and still adds coverage, but a reader leaning on it as a model for how to write a new test should look at test_generated_execution.py or test_metadata_only.py instead.

Nothing in src/ is measurably thin from the outside. Every module that tests/ names a file after (schema_analyzer, generator_factory, defensive_manager, constraint_check, validators, template_generator, config_builder, demo_coordinator, feature_flags, rapid_deployment, the API and CLI layers) has a corresponding test module. Whether every branch inside those modules is actually exercised is exactly what the uncapped coverage job would answer and does not enforce; see the first point above.

No property-based testing. Every test in the suite is example-based, a concrete input mapped to a concrete expected output, including the large @pytest.mark.parametrize tables in tests/test_pii_detection.py. There is no Hypothesis (or similar) dependency and no generated-input fuzzing anywhere in tests/. For a project whose core claim is that it adapts correctly to arbitrary schemas, the schemas actually exercised in tests are all hand-written and small; a schema shape nobody thought to write a fixture for is untested by construction.

Learn more resources#

Official documentation#

Tutorials and blogs#

  • Martin Fowler, Mocks Aren’t Stubs, the source for the mock/stub/fake vocabulary used in The idea underneath above, and the article that names the classical-versus-mockist testing styles this project’s fake-preferring approach falls on the classical side of.

  • Brian Okken, pytest fixtures: nuts and bolts, revisited, a from-the-ground-up walkthrough of fixture mechanics from the author of Python Testing with pytest, useful for the fixture-composition patterns conftest.py in this repo relies on.

Videos#