Packaging and distribution

Packaging and distribution#

Why packaging and distribution#

tdw is meant to be run by people who never clone this repository. The Documentation link in pyproject.toml and the install instructions in Installation both point at a plain pip install test-data-workbench, which means someone else’s machine, with none of this project’s source tree on it, has to end up with a working tdw command on its PATH after running one line. That does not happen by accident. Something has to turn src/test_data_workbench plus pyproject.toml into an artifact PyPI can host, pip (or pipx) can install unattended, and a single declared function, test_data_workbench.cli.main:main, into an executable command. That something is the subject of this page: the [build-system] and [project] configuration in pyproject.toml, and the GitHub Actions workflow that builds and ships it.

None of this is done by hand. .github/workflows/publish.yml builds the distribution, checks it, and uploads it to PyPI automatically whenever a GitHub Release is published (or on a manual trigger); nobody runs twine upload from a laptop.

The idea underneath#

PEP 517 and PEP 518: decoupling the build backend#

For a long time, “packaging a Python project” meant writing a setup.py that imported setuptools and called setup(...): the Python interpreter executing arbitrary code was the build system, and it was setuptools specifically, hardcoded into every tool that wanted to build a package. PEP 518 introduced pyproject.toml with one first job: let a project declare, as static TOML data rather than executable code, which packages a tool needs installed before it can even attempt a build. That is exactly what this project’s [build-system] table does: requires = ["hatchling"]. PEP 517 defined the other half, a build-backend hook protocol (build_sdist, build_wheel, and related hooks) that any conforming backend implements, so a front end such as pip or build can build any project the same way, by reading build-backend and calling those hooks, without needing to know in advance whether the backend behind it is Hatchling, setuptools, Flit, PDM, or something written after both PEPs shipped. Setuptools stopped being implicitly the build system for every Python package the moment these two PEPs existed; it became one interchangeable implementation of a public interface, and this project picked a different one.

PEP 621: metadata in one place#

PEP 621 did for project metadata what 517 and 518 did for the build backend. Before it, a project’s name, version, and dependency list were typically buried inside the same setup.py that 517 had just made optional, which meant a tool had to execute Python code just to learn a project’s name. PEP 621 standardized a [project] table that any front end reads directly out of pyproject.toml as static data, with no interpreter involved. Every field this project sets under [project], name, version, dependencies, optional-dependencies, scripts, is PEP 621 vocabulary, understood identically regardless of which build backend sits underneath it.

Source distribution vs wheel#

A PEP 517 build backend produces two different kinds of artifact, and they solve different problems. A source distribution (an sdist, a .tar.gz) is close to the repository itself: source files plus metadata, in a form pip can turn into a wheel by invoking the build backend again on the installing machine. A wheel (a .whl, a zip archive with a defined internal layout) is already built: exactly the files that need to land inside site-packages, arranged that way in advance. Installing a wheel is a copy operation, not a build; pip never re-invokes Hatchling and never re-executes anything from [build-system] on the end user’s machine. That is why wheels are the preferred, default install format: a person running pip install test-data-workbench is not expected to own a build toolchain, is not expected to trust the package enough to let it run arbitrary setup code on install, and should not have to wait on anything to compile. publish.yml’s python -m build step, run with no extra arguments, produces both from this project’s source in one call.

Console script entry points#

A Python function is not, by itself, a shell command. Something has to bridge “call this function” and “type this word at a prompt.” The console_scripts entry point group is that bridge; PEP 621 gave it a table of its own, [project.scripts], so declaring one no longer requires importing setuptools’ API directly. The left side of name = "module:function" is the command a user types; the right side is where the interpreter finds the callable. When a wheel that declares one gets installed, the installer writes a small executable launcher into the environment’s scripts directory, one that does, in effect, import sys; from module import function; sys.exit(function()). It is that generated launcher, not a copy of the project’s own source file, that ends up on PATH.

Trusted Publishing: OIDC instead of a stored token#

The traditional way a CI workflow authenticates to PyPI is a long-lived API token: generated once on pypi.org, pasted into the CI provider as a secret, valid until someone remembers to revoke it. That token is a standing liability, usable by anyone who can read it, for as long as it remains valid, which in practice is often indefinitely. PyPI’s Trusted Publishing feature replaces the stored secret with OpenID Connect (OIDC), an identity layer built on OAuth 2.0. Instead of a workflow proving who it is by presenting a password, GitHub Actions itself mints a short-lived, cryptographically signed token asserting facts about the running job (which repository, which workflow file, which environment); PyPI, having been told in advance, on pypi.org, to trust tokens carrying those specific facts, exchanges it for a PyPI-scoped publishing credential that expires shortly after. There is no long-lived secret to leak, because there is no long-lived secret: the credential is minted fresh, per run, and is worthless outside the exact GitHub Actions run that requested it. This is a meaningful supply-chain improvement precisely because it removes the thing that makes token leaks dangerous in the first place: a credential that keeps working long after the run that needed it has finished.

How it fits this project#

pyproject.toml at the repository root is the entire packaging configuration; there is no setup.py, setup.cfg, or MANIFEST.in anywhere in the tree. [build-system] names Hatchling as the backend, [project] carries the PEP 621 metadata (the distribution name test-data-workbench, one console script, two optional-dependency extras), and [tool.hatch.build.targets.wheel] tells Hatchling which directory under src/ is the actual importable package. Three GitHub Actions workflows divide the packaging-adjacent work: ci.yml installs the project as an editable install on every push and pull request to prove it still imports and passes its tests; docs.yml builds and deploys the Sphinx site and, as docs/source/conf.py shows, reads the installed package’s version through importlib.metadata rather than restating it; publish.yml is the one that actually builds the sdist and wheel and ships them to PyPI.

The build backend: Hatchling#

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

That is the entire build-system declaration: no version floor is set on Hatchling itself, so whichever release pip resolves at build time is the one used (see Sharp edges and limits for the equivalent point about this project’s own dependencies). Hatchling is the build backend shipped by Hatch, developed under the same pypa GitHub organization as setuptools and build. Beyond naming it, this project asks Hatchling for exactly one more thing it would not infer entirely on its own: where the package lives under src/, covered next.

Project metadata: classifiers, license, and requires-python#

[project] carries the PEP 621 fields:

name = "test-data-workbench"
version = "1.0.0"
description = "Schema-adaptive test data generator: point it at a database, get realistic relational test data back"
readme = "README.md"
requires-python = ">=3.10"
license = "Apache-2.0"
authors = [{name = "Volantic Systems"}]

requires-python = ">=3.10" matches ci.yml’s test matrix exactly (3.10, 3.11, 3.12), and so do the three Programming Language :: Python :: 3.1x entries in classifiers; a version this project does not test is not claimed as supported. readme is what PyPI renders as the project’s description page; keywords populates the search chips on that same page; [project.urls] (Homepage, Documentation, Repository, Issues) becomes the “Project links” sidebar there.

license = "Apache-2.0" is a bare SPDX license expression string, the format PEP 639 introduced, rather than the older license = {text = "..."} table. Consistent with that, classifiers carries no License :: OSI Approved :: Apache Software License entry: PEP 639 treats a license classifier as redundant once a license expression is present, and this project does not carry both.

The src layout and the packages option#

[tool.hatch.build.targets.wheel]
packages = ["src/test_data_workbench"]

The importable code lives under src/test_data_workbench, not in a test_data_workbench/ directory sitting next to pyproject.toml and tests/. That is the “src layout,” and its practical benefit shows up during development, before packaging ever runs: with no importable package directly at the repository root, import test_data_workbench cannot silently succeed against the working tree by accident; Python only finds it through whatever is actually installed. docs/source/conf.py shows the other side of the same fact: it has to add src/ to sys.path by hand (sys.path.insert(0, os.path.abspath("../../src"))) for Sphinx’s autodoc extension to import the package at all, precisely because nothing makes it importable by default.

The packages option tells Hatchling’s wheel builder which directory to ship, and collapses the src/ prefix away: what is packaged as src/test_data_workbench here is installed and imported as test_data_workbench, not src.test_data_workbench. Hatchling’s own default file-selection heuristic (checking, among other locations, src/<name>/__init__.py where <name> is derived from the normalized project name) would likely have found this exact layout unassisted, since test-data-workbench normalizes to test_data_workbench. Setting packages explicitly trades that inference for a mapping visible directly in the file, which stays correct even if name in [project] is ever changed without a matching directory rename.

The tdw console script#

[project.scripts]
tdw = "test_data_workbench.cli.main:main"

This is the whole reason a user who types tdw after installing the package gets a command at all. src/test_data_workbench/cli/main.py defines the target directly:

def main(argv: Optional[Sequence[str]] = None) -> int:
    parser = build_parser()
    args = parser.parse_args(argv)
    ...

if __name__ == "__main__":
    sys.exit(main())

The if __name__ == "__main__": guard is what makes python -m test_data_workbench.cli.main work as a second, manual way to reach the same main(); the console-script launcher pip generates from [project.scripts] does the equivalent of that guard automatically, for every installed copy, without anyone needing to remember the module path. main()’s own --version handling, _package_version(), prefers importlib.metadata.version("test-data-workbench"), the version the installer actually recorded, and only falls back to the package’s own hardcoded __version__ if that lookup raises PackageNotFoundError (see Version handling).

Optional dependency extras: postgres and dev#

[project.optional-dependencies]
postgres = [
    "psycopg2-binary>=2.9.9",
]
dev = [
    "pytest>=7.4.3",
    "pytest-asyncio>=0.21.1",
    "pytest-mock>=3.12.0",
    "httpx2",  # required by starlette.testclient (FastAPI TestClient) on current starlette
]

[project.optional-dependencies] is PEP 621 vocabulary too: a mapping from an extra’s name to a list of dependencies installed only when that extra is requested. postgres pulls in the PostgreSQL driver; SQLite, this project’s zero-setup default (see Installation), needs no driver at all, since sqlite3 ships in the Python standard library. dev pins what the test suite needs beyond the runtime dependencies: pytest, its asyncio_mode = "auto" companion pytest-asyncio ([tool.pytest.ini_options] turns that mode on project-wide), and pytest-mock, plus one more entry discussed in Sharp edges and limits.

ci.yml exercises both extras directly, and shows they compose: the test and coverage jobs run pip install -e ".[dev]", while the postgres job runs pip install -e ".[dev,postgres]", naming two extras in one bracketed, comma-separated list.

Version handling#

The version string 1.0.0 is written as a literal three separate times: pyproject.toml’s version = "1.0.0", src/test_data_workbench/__init__.py’s __version__ = "1.0.0", and a fallback inside docs/source/conf.py:

try:
    release = _pkg_version("test-data-workbench")
except Exception:
    release = "1.0.0"
version = release

In the common case, package actually installed, editable or not, neither hardcoded copy is what gets shown. conf.py calls importlib.metadata.version("test-data-workbench") first, which reads back whatever version the installer recorded for the currently installed distribution; cli/main.py’s _package_version() does the identical lookup for tdw --version. Both hardcoded literals, in __init__.py and in conf.py’s except branch, exist only for the case where the code is being imported straight out of a source tree with nothing installed at all, and importlib.metadata therefore has no distribution record to find. The repository carries exactly one git tag, v1.0.0, and it currently agrees with all three; how durable that agreement is is covered in Sharp edges and limits.

The release workflow: build, check, and publish#

publish.yml runs on release: types: [published] (a GitHub Release being published) or workflow_dispatch (a manual button in the Actions tab), and splits the work across two jobs.

build checks out the repository, installs Python 3.12, then:

pip install build
python -m build
pip install twine
twine check dist/*

twine check validates the built metadata before anything is uploaded, including whether the README renders as valid content for PyPI’s description page, which is exactly the kind of thing that otherwise only fails after a real upload. The resulting dist/ directory (sdist and wheel) is then handed to actions/upload-artifact under the name dist.

publish runs only if build succeeded (needs: build), downloads that same dist artifact rather than rebuilding anything, and calls pypa/gh-action-pypi-publish@release/v1 with no with: block at all: no password, no token, no repository-url. That is possible because of two other declarations on the job: permissions: id-token: write, scoped to this job only (build gets no such permission, so only the job that actually talks to PyPI can mint an OIDC token), and environment: {name: pypi, url: https://pypi.org/project/test-data-workbench/}, which names the GitHub Environment the run deploys under. The other half of the trust relationship, which repository, workflow filename, and environment name PyPI will accept a token from, is registered on pypi.org’s own Publishing settings for this project, not in this repository; nothing under .github/ shows that configuration, because it does not live here.

Sharp edges and limits#

The version is hand-maintained, and the drift this allows is not hypothetical: it has already happened once. pyproject.toml, __init__.py, and conf.py’s fallback all currently say 1.0.0, but nothing checks that they agree, and src/test_data_workbench/core/__init__.py still carries its own __version__ = "4.0.0", left over from before this project was renamed from test-data-schema-adapter (its original name and version, visible in this repository’s first commit) to test-data-workbench at 1.0.0. Nothing imports test_data_workbench.core.__version__ today, so it is currently inert, but because [tool.hatch.build.targets.wheel] ships the whole src/test_data_workbench tree, that stale "4.0.0" string is not only a git-history artifact: it is inside the wheel published to PyPI right now, for anyone who greps an installed copy.

The single git tag, v1.0.0, matching pyproject.toml today is also manual care rather than enforcement. publish.yml never reads the tag or the GitHub Release name to decide what to build; it builds whatever pyproject.toml says on the checked-out commit. The workflow_dispatch trigger makes the gap concrete: it lets a maintainer publish from the Actions tab with no GitHub Release involved at all, so a manual run ships whatever version string is sitting in pyproject.toml on the current default branch, tagged or not.

Every runtime dependency in [project] is a lower bound only: fastapi>=0.104.1, sqlalchemy>=2.0.23, pydantic>=2.5.0, and the rest through rich>=13.7.0, none capped with a < upper bound. That is a common, deliberate tradeoff rather than an oversight: an upper cap gives a false sense of safety (pinning <3.0 does not mean version 2.9 was actually tested against this code either), and no cap maximizes compatibility with whatever else is already in an installer’s environment. The cost is symmetrical: nothing here stops a future major release of, say, SQLAlchemy or Pydantic from resolving into an install and changing behavior this project depends on, and the first sign of that would be a real install breaking, not a CI failure, since ci.yml installs against whatever the resolver picks at run time rather than a pinned lockfile.

twine check verifies metadata, not behavior. No step in publish.yml, or in ci.yml, ever installs the built wheel and runs anything against it; ci.yml’s jobs install with pip install -e ".[dev]", an editable install straight from src/, which never exercises the packages mapping or Hatchling’s file-selection decisions the way an actual pip install dist/*.whl would. A packaging-only defect, a file [tool.hatch.build.targets.wheel] fails to include, for example, would pass every job in this repository and surface only after a real user installs the published wheel.

The dev extra’s fourth entry deserves a note, because the name invites a double take. Starlette’s TestClient, which tests/test_api.py uses to exercise the FastAPI app (see FastAPI), is built on httpx, and pyproject.toml pins a distribution named httpx2. That is a distinct distribution rather than a version marker or an alias, but it is the successor line: httpx2 is published from the Pydantic organisation at github.com/pydantic/httpx2 and carries its own matching httpcore2 dependency. A name one character away from a very popular package is worth verifying rather than assuming, in either direction; this one checks out. The entry is unpinned, so every ci.yml job installing the dev extra takes whatever version currently resolves, which is the more ordinary observation to make about it.

Learn more resources#

Official documentation

Tutorials and blogs

Videos