Python requirements and virtualenvs: pip-tools, uv and pins

The goal is narrow and worth stating plainly: the same set of files should produce the same set of installed packages on your laptop, in CI and on the server, today and in four months. Python gets you most of the way there with two mechanisms - a virtual environment that isolates one application's packages from everything else, and a requirements file that says exactly which versions go in it. What breaks reproducibility is almost always the second half: a file that pins your own dependencies but not theirs, so an upgrade you never asked for arrives on the next restart.

This post covers what a virtualenv really is and how it breaks, how to write a requirements file that holds, the lock-file tools worth using, and what goes wrong when the install runs on a small server rather than your machine.

What a virtualenv actually is#

A virtual environment is a directory with a copy of the Python package machinery and its own site-packages. Creating one is built in:

$ python -m venv .venv$ .venv/bin/python -m pip install --upgrade pip$ .venv/bin/pip install -r requirements.txt

There is less magic here than people assume. .venv/bin/python is a symlink or small copy pointing at the interpreter that created it. .venv/pyvenv.cfg records which interpreter that was and whether the system's packages are visible. Activation - source .venv/bin/activate - does nothing clever either: it puts .venv/bin at the front of your PATH and sets VIRTUAL_ENV. That is the whole trick.

Which is why calling .venv/bin/python and .venv/bin/pip by their full paths is the better habit on a server. There is no shell session to activate in a start command, no risk that a script forgot to activate, and no confusion about which interpreter is running. Anything you would write as python manage.py migrate becomes .venv/bin/python manage.py migrate.

Three properties of a venv matter in production:

• It is not relocatable. Console scripts in .venv/bin carry an absolute shebang line, and pyvenv.cfg holds an absolute path. Rename the directory above it, or move the application, and the environment stops working. Recreate it rather than trying to patch the paths.
• It is tied to one interpreter version. Packages live in .venv/lib/python3.12/site-packages. If the base image moves from 3.12 to 3.13, the environment does not follow, and you get ModuleNotFoundError for things that are visibly on disk.
• It never belongs in git. Add .venv/ to .gitignore. A committed environment is hundreds of megabytes of platform-specific binaries that will not run anywhere else.

On modern Debian and Ubuntu you will also meet error: externally-managed-environment when you try to pip install into the system Python. That message is the distribution protecting its own packages, and the correct response is to make a venv. --break-system-packages is named accurately.

requirements.txt: what to pin and what breaks#

A requirements file is a list of specifiers, one per line. The specifiers you will use:

The failure is not the obvious one. Most people do pin their direct dependencies. What they do not pin is the dependencies of those dependencies, and that is where the surprise lives: you pinned the web framework, the framework depends on a template library with a loose range, that library ships a release with a behaviour change, and your next deploy picks it up because nothing said otherwise.

The usual first answer is pip freeze > requirements.txt, which does capture everything installed. It has three known problems. It records the state of your machine rather than your intent, so nobody can tell later which packages you actually chose and which came along for the ride. It picks up editable installs and local paths that mean nothing on another machine. And it happily records whatever your environment drifted into, including the package you installed once to try something.

The pattern that solves this is two files. requirements.in holds the handful of packages you chose, loosely if you like. A tool resolves it once and writes requirements.txt containing every package in the tree at an exact version. You edit the first, you commit both, and the server installs only the second.

django~=5.0gunicornpsycopg[binary]whitenoise

$ pip-compile requirements.in -o requirements.txt

asgiref==3.8.1    # via djangodjango==5.0.6    # via -r requirements.ingunicorn==22.0.0    # via -r requirements.inpackaging==24.1    # via gunicorn

The # via comments are why this format is worth the extra file: six months later you can see which of your choices dragged in a package, and deleting a line from the .in file removes it and everything it brought with it.

Lock files: pip-tools, uv, Poetry and Pipenv#

Four tools do this job. They differ in how much else they take over.

pip-tools is the smallest step from where most projects already are. It produces a plain requirements file that any pip can install, so nothing on the server needs to know the tool exists. pip-sync requirements.txt goes further than pip install by also uninstalling anything in the environment that is not in the file, which is the difference between "the packages I need are present" and "the environment matches the file".

uv does the same job an order of magnitude faster, and can replace venv and pip as well. uv venv creates an environment, uv pip compile requirements.in -o requirements.txt resolves, uv pip sync requirements.txt installs. It also has a project mode built on pyproject.toml and uv.lock, where uv sync creates the environment and installs the locked set in one command. The speed comes from a global cache and a resolver written in Rust; on a slow server the difference between a thirty-second install and a three-second one changes how willing you are to reinstall on every restart.

Poetry manages dependencies, environments and packaging together, with its own resolver and pyproject.toml as the source of truth. It is a good fit for a library and a reasonable one for an application, with one caveat for deployment: the server now needs Poetry installed before it can install anything. Exporting to a plain requirements file is possible - in recent versions through a plugin - and is the usual way to keep the server simple.

Pipenv does the same thing with Pipfile and Pipfile.lock. It is still maintained and still works; it is simply less common in new projects than it was.

If you have no opinion, use uv with an .in file and a compiled requirements.txt. You get a lock file, the server needs nothing but pip if you ever want to drop the tool, and the install is fast enough to run on every deploy.

Hashes, and when they are worth it#

A pinned version says which release to install. A hash says which bytes. With --generate-hashes, the compiled file carries a digest for every artefact, and pip refuses anything that does not match:

$ pip-compile --generate-hashes requirements.in -o requirements.txt$ pip install --require-hashes -r requirements.txt

This protects against an index serving something other than what you tested - a compromised mirror, a re-uploaded artefact, a proxy in the middle. The cost is real: hash mode requires every dependency to be pinned with a hash, so you cannot install anything ad hoc afterwards, and the file becomes long and noisy in reviews. It is worth it for anything handling money or credentials, and skippable for a hobby project. Either way, pin the versions.

While you are there, pip-audit checks an environment or a requirements file against the Python advisory database and tells you which pins have known vulnerabilities. It is a better use of five minutes than upgrading everything blindly.

Installing on the server#

The install belongs in the deploy step, and on most panels the deploy step is the start command:

$ .venv/bin/pip install --no-cache-dir -r requirements.txt \  && .venv/bin/gunicorn myproject.wsgi:application --bind 0.0.0.0:8000

pip skips anything already satisfied, so a restart with no dependency changes costs a couple of seconds rather than a full install. That makes it safe to leave in place, which is the point: the running code and the declared dependencies can never drift apart if one is installed from the other every time the process starts. The rest of that command - the worker count, the bind address, the port - is covered in deploy FastAPI or Flask.

A few environment variables make this quieter and smaller:

PIP_DISABLE_PIP_VERSION_CHECK=1PIP_NO_CACHE_DIR=1PYTHONDONTWRITEBYTECODE=1PYTHONUNBUFFERED=1

The last one is not about packaging, but it is the setting people miss most: without it Python buffers standard output when it is not attached to a terminal, and the console shows nothing until the buffer fills. Where those variables are set, and which of them are secrets, is the subject of environment variables and secrets. On RE:NODE the panel console is unfiltered live output with a command line, so it is the first place you look when an install fails - and it is empty until you set that variable.

Where a host pulls your repository from GitHub on every start, as the app plans do, the ordering is: pull, install, run. Two switches control it - pull on start, and deploy on push, which restarts a server that was already running - and each deploy gets its own record, so a deploy that installed cleanly and one that fell over during the install are distinguishable after the fact. Deploy a Node app from GitHub walks through the same two switches from the JavaScript side; the mechanism is identical.

Wheels, compilers and the packages that hurt#

Most packages install as a wheel: a pre-built archive that unpacks in seconds. Some do not, and then pip falls back to building from source on the machine doing the install. On a plan with half a core, that is the difference between a five-second install and a ten-minute one, or an install that fails with a compiler error about a header file nobody has.

The usual offenders and what to do:

• psycopg2 builds against libpq. Use psycopg2-binary, or the modern psycopg[binary], which ship wheels. The Django side of that choice is in deploy Django to production.
• cryptography needs a Rust toolchain when no wheel matches. It publishes wheels for common Linux platforms, so this only bites on unusual architectures or very old pip versions.
• Pillow, lxml and mysqlclient all want system libraries when built from source. Prefer the wheel; upgrade pip first, because wheel compatibility tags improved over time and an ancient pip will ignore a wheel that would have worked.
• numpy, pandas and anything scientific have excellent wheels and enormous ones. They install fine and they eat disk.

To find out before you find out the hard way, refuse source builds entirely:

$ pip install --only-binary=:all: -r requirements.txt

If that fails, the error names the package that has no wheel for your platform, which is a much better failure than discovering it during a deploy at midnight.

Disk, caches and small plans#

An application plan's disk is finite - 5 GB on the smallest tier, up to 50 GB at the top - and Python fills it in three ways: the environment, pip's cache, and compiled bytecode. Check with du -sh .venv and pip cache dir.

A plain web app venv is 60-150 MB. Add pandas and numpy and you are into several hundred. Add a machine-learning stack and gigabytes is normal, at which point disk becomes part of the plan you need rather than an afterthought. pip cache purge and uv cache clean free space immediately; --no-cache-dir stops it accumulating in the first place, at the cost of re-downloading on the next install.

One venv per application, always. Two apps sharing an environment means one cannot be upgraded without the other, which defeats the purpose of having environments at all. If two apps need genuinely different Python versions, they need different servers - the base image gives you one interpreter version, and installing a second one by hand is a maintenance job you will not enjoy.

What goes wrong#

`ModuleNotFoundError` for a package that is definitely installed. Two interpreters. You installed with the system pip and are running .venv/bin/python, or the reverse. .venv/bin/python -c "import sys; print(sys.executable)" settles it.

It worked locally and broke on the server. Something in the tree was not pinned. Recreate your local environment from the requirements file - delete .venv, make a new one, install - and it will usually break locally too, which is much easier to debug.

The install takes ten minutes. A source build. Run with --only-binary=:all: to identify it, then switch to a package that ships wheels.

`No space left on device` during an install. The cache plus a partially unpacked wheel. Purge the cache, then use --no-cache-dir in the start command.

A package upgraded itself on restart. A range rather than a pin, somewhere in the tree. Compile a lock file and install only from that.

`error: externally-managed-environment`. You are installing into the system Python. Make a venv.

The environment broke after a platform update. The base interpreter moved version and the venv still points at the old path. Delete .venv and recreate it; this is why the environment is never in git and the requirements file always is.

FAQ#

Do I need a virtualenv inside a container?

Strictly, no - a container already isolates the application. In practice a venv still helps: it separates your packages from the ones the image installed, it makes pip behave the same locally and remotely, and it sidesteps the externally-managed-environment error on distributions that enforce PEP 668. The cost is a directory.

Is pip freeze good enough?

For a solo project that never changes, it works. It records the state of a machine rather than your intent, loses the distinction between packages you chose and packages you inherited, and can capture local paths. Two files - one you edit, one compiled from it - answer the same question and stay readable.

Should I commit the lock file?

Yes. The lock file is the reproducible part; without it in the repository, the install on the server is a fresh resolve that can differ from the one you tested. Commit both the input file and the compiled output, and review the diff when it changes.

uv or pip-tools?

Both produce a plain requirements file that any pip can install, so the choice is reversible. uv is dramatically faster and can also create the environment and run commands; pip-tools is older, smaller in scope and completely predictable. On a slow server the speed matters more than it sounds, because it decides whether reinstalling on every start is tolerable.

How do I upgrade one package safely?

Change it in the input file, recompile, and read the diff of the generated file before committing. With pip-tools that is pip-compile --upgrade-package django; uv has the same flag. Upgrading one package deliberately and reading what moved with it is the whole discipline.