The greatest advantage in packaging Python code (when compared to other forms
of distributing programs and libraries) is that packages allow us to stand on
the shoulders of giants: you don’t need to implement everything by yourself,
you can just declare dependencies on third-party packages and
pip, PyPI and their friends will do the heavy lifting for you.
Of course, with great power comes great responsibility. Package authors must be careful when declaring the versions of the packages they depend on, so the people consuming the final work can do reliable installations, without facing dependency hell. In the opensource community, two main strategies have emerged in the last few years:
- the first one is called abstract and consists of having permissive, minimal and generic dependencies, with versions specified by ranges, so anyone can install the package without many conflicts, sharing and reusing as much as possible dependencies that are already installed or are also required by other packages
- the second, called concrete, consists of having strict dependencies, with pinned versions, so all the users will have repeatable installations
Both approaches have advantages and disadvantages, and usually are used together in different phases of a project. As a rule of thumb, libraries tend to emphasize abstract dependencies (but can still have concrete dependencies for the development environment), while applications tend to rely on concrete dependencies (but can still have abstract dependencies specially if they are intended to be distributed via PyPI, e.g. command line tools and auxiliary WSGI apps/middleware to be mounted inside other domain-centric apps). For more information about this topic check Donald Stufft post.
Since PyScaffold aims the development of Python projects that can be easily
packaged and distributed using the standard PyPI and
pip flow, we adopt the
specification of abstract dependencies using
basically means that if PyScaffold generated projects specify dependencies
setup.cfg file (using general version ranges), everything will
work as expected.
While specifying the final dependencies for packages is pretty much
straightforward (you just have to use
setup.cfg), dependencies for running the tests can be a little bit trick.
setuptools provides a
tests_require field that follows
the same convention as
install_requires, however this field is not strictly
setuptools doesn’t really do much to enforce the packages
listed will be installed before the test suite runs.
PyScaffold’s recommendation is to create a
testing field (actually you can
name it whatever you want, but let’s be explicit!) inside the
[options.extras_require] section of
setup.cfg. This way multiple test
runners can have a centralised configuration and authors can avoid double
If you use
pytest-runner adding a
--extras flag will do the trick of
making sure these dependencies are installed, and if you use
tox, the same
is accomplished with the
extras configuration field. By default PyScaffold will take care of these
configurations for you.
If you prefer to use just
tox and keep everything inside
tox.ini, please go ahead and move your test dependencies.
Every should work just fine :)
PyScaffold strongly advocates the use of test runners to guarantee
your project is correctly packaged/works in isolated environments.
A good start is to create a new project passing the
--tox option to
putup and try running
tox in your project root.
As previously mentioned, PyScaffold will get you covered when specifying the
abstract or test dependencies of your package. We provide sensible
out-of-the-box. In most of the cases this is enough, since developers in the
Python community are used to rely on tools like
virtualenv and have a
workflow that take advantage of such configurations. As an example, someone
$ pip install pyscaffold $ putup myproj --tox $ cd myproj $ python -m venv .venv $ source .venv/bin/activate # ... edit setup.cfg to add dependencies ... $ pip install -e . $ pip install tox $ tox
However, someone could argue that this process is pretty manual and laborious to maintain specially when the developer changes the abstract dependencies. Moreover, it is desirable to keep track of the version of each item in the dependency graph, so the developer can have environment reproducibility when trying to use another machine or discuss bugs with colleagues.
In the following sections, we describe how to use two popular command line tools, supported by PyScaffold, to tackle these issues.
How to integrate Pipenv¶
We can think in Pipenv as a virtual environment manager. It creates
per-project virtualenvs and generates a
Pipfile.lock file that contains a
precise description of the dependency tree and enables re-creating the exact
same environment elsewhere.
Pipenv supports two different sets of dependencies: the default one, and the dev set. The default set is meant to store runtime dependencies while the dev set is meant to store dependencies that are used only during development.
This separation can be directly mapped to PyScaffold strategy: basically the
default set should mimic the
install_requires option in
while the dev set should contain things like
ptpython or any other tool the developer uses
Test dependencies are internally managed by the test runner, so we don’t have to tell Pipenv about them
The easiest way of doing so is to add a
-e . dependency (in resemblance
with the non-automated workflow) in the default set, and all the other ones in
the dev set. After using Pipenv, you should add both
Pipfile.lock to your git repository to achieve reproducibility (maintaining
Pipfile.lock shared by all the developers in the same project can
save you some hours of sleep).
In a nutshell, PyScaffold+Pipenv workflow looks like:
$ pip install pyscaffold pipenv $ putup myproj --tox $ cd myproj # ... edit setup.cfg to add dependencies ... $ pipenv install $ pipenv install -e . # proxy setup.cfg install_requires $ pipenv install --dev tox sphinx # etc $ pipenv run tox # use `pipenv run` to access tools inside env $ pipenv lock # to generate Pipfile.lock $ git add Pipfile Pipfile.lock
After adding dependencies in
setup.cfg, you can run
pipenv update to
add them to your virtual environment.