How to document requirements - pip

Question

When I program, I often experiment with third party packages.
Of course, they are installed via pip.
When installed, packages install their dependencies.
At the end I want to have a clean requirements.txt which reflect what packages are really necessary for the project (pip freeze > requirements.txt).
If I manage to find a better solution to a package, I uninstall it via pip.
The problem is that when uninstalled, a package doesn't uninstall its dependencies.
If I'm not mistaken, this it doesn't. Well, and when I make pip freeze, I can't recognize what is going on here.
Therefore I decided to document which package installed which other packages.
When uninstalled, I uninstall them manually.
But this is really troublesome and error prone:
pip freeze > requirements.txt
asgiref==3.2.3
Django==3.0.2
pkg-resources==0.0.0
pytz==2019.3
sqlparse==0.3.0
Edit requirements.txt:
asgiref==3.2.3 # Django dependency
Django==3.0.2 # Django
pkg-resources==0.0.0 # Default package
pytz==2019.3 # Django dependency
sqlparse==0.3.0 # Django dependency
When I install a new package, I make pip freeze > requirements1.txt. Then compare the files, reveal the newly installed package's dependencies, mark them, then copy-paste old comments.
Well, finally I can just make it a complete mess. I make pip freeze and understand that I just don't know which package depends on which as I just forget to make my comments or something.
Again this is my goal: when I finish my project in a year, I'd like to know exactly that:
Every installed package is absolutely necessary for the project to run.
No unnecessary packages is installed.
Could you tell me what is the best practice to do that.

Answer 1

It seems pip-tools is the exact tool you need to solve the problems you encountered in your development workflow.
pip-tools enables you to specify your high-level dependencies (e.g. Django), while taking care automatically of your low-level dependencies (e.g. pytz). It also enables you to sync your environment to a requirements.txt file, regardless of the messy state you might have.
Installing pip-tools in your project
To use pip-tools, first create a new empty Python virtual environment in your project and activate it, using the tool of your choice:
with venv (built-in): python3 -m venv .venv && source .venv/bin/activate
with pyenv: pyenv virtualenv 3.8.1 project-env && pyenv local project-env
Then, install pip-tools in the Python virtual environment of your project:
pip install pip-tools
Managing dependencies with pip-compile using a requirements.in file
pip-tools includes a pip-compile tool that takes a requirements.in file as input. This requirements.in file is similar to requirements.txt, but it contains only high-level dependencies. For example, if your project is using the latest version of Django, you could write something like this in requirements.in:
django>=3.0,<3.1
You can see that requirements.in does not contain the dependencies of Django, only Django itself.
Once you have set your dependencies in requirements.in, use pip-compile to "compile" your requirements.in into a requirements.txt file:
➜ pip-compile requirements.in
#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile requirements.in
#
asgiref==3.2.3 # via django
django==3.0.2
pytz==2019.3 # via django
sqlparse==0.3.0 # via django
The requirements.txt file generated by pip-compile indicates the source of every indirect dependency next to the package name. For example, pytz==2019.3 is a dependency of django. In addition to that, it also pins each dependency to a precise version number, to make sure that the installation of your dependencies is reproducible
Applying dependencies from generated requirements.txt with pip-sync
Now that you have a requirements.txt, you can apply it on your Python virtual environment using pip-sync:
➜ pip-sync requirements.txt
Collecting asgiref==3.2.3 (from -r /var/folders/r1/n_n031s51wz2gjwy7mb9k4rh0000gn/T/tmpvhv549si (line 1))
Using cached https://files.pythonhosted.org/packages/a5/cb/5a235b605a9753ebcb2730c75e610fb51c8cab3f01230080a8229fa36adb/asgiref-3.2.3-py2.py3-none-any.whl
Collecting django==3.0.2 (from -r /var/folders/r1/n_n031s51wz2gjwy7mb9k4rh0000gn/T/tmpvhv549si (line 2))
Using cached https://files.pythonhosted.org/packages/55/d1/8ade70e65fa157e1903fe4078305ca53b6819ab212d9fbbe5755afc8ea2e/Django-3.0.2-py3-none-any.whl
Collecting pytz==2019.3 (from -r /var/folders/r1/n_n031s51wz2gjwy7mb9k4rh0000gn/T/tmpvhv549si (line 3))
Using cached https://files.pythonhosted.org/packages/e7/f9/f0b53f88060247251bf481fa6ea62cd0d25bf1b11a87888e53ce5b7c8ad2/pytz-2019.3-py2.py3-none-any.whl
Collecting sqlparse==0.3.0 (from -r /var/folders/r1/n_n031s51wz2gjwy7mb9k4rh0000gn/T/tmpvhv549si (line 4))
Using cached https://files.pythonhosted.org/packages/ef/53/900f7d2a54557c6a37886585a91336520e5539e3ae2423ff1102daf4f3a7/sqlparse-0.3.0-py2.py3-none-any.whl
Installing collected packages: asgiref, pytz, sqlparse, django
Successfully installed asgiref-3.2.3 django-3.0.2 pytz-2019.3 sqlparse-0.3.0
pip-sync will make sure your virtual environment corresponds exactly to what is defined in the requirements.txt file by uninstalling and installing the relevant packages.
After your environment has been synced, make sure your project code works properly. If it does, commit requirements.in and requirements.txt into version control.
Reference
For more details, you can refer to the docs of pip-tools, or to the article of Hynek Schlawack: Python Application Dependency Management in 2018.

Answer 2

Dependency management is a bit rougher in python land than others, there a tools like poetry and pipenv but for just poking a round I would try the following.
create a virtual environment in your project directory
$ python3 -m venv .venv
Activate your virtual environment
$ source .venv/bin/activate
Install whatever packages you're playing with
$ echo "package1" >> requirements-dev.txt
$ echo "package2" >> requirements-dev.txt
$ pip install --upgrade -r requirements-dev.txt
When you think you have everything you need deactivate your virtual env, create a new one, make sure everything still works, then create your new requirements.txt
$ deactivate
$ rm -rf .venv
$ python3 -m venv .venv
$ pip install --upgrade -r requirements-dev.txt
# test your code still works
# create a new requirements file to be used for "prod"
$ pip freeze > requirements.txt
I wouldn't call this best practice, but its a start and should get you by until to decide which side you want to join when it comes to dep management tools :)