I have a flask app where I'm trying to automate deployment to EC2.
Not a big deal, but is there a setting in either Fabric or Distribute that reads the requirements.txt file directly for the setup.py, so I don't have to spell everything out in the setup(install_requires=[]) list, rather than writing a file reader for my requirements.txt? If not, do people have recommendations or suggestions on auto-deployment and with pip?
I'm reviewing from here and here.
Not a big deal, but is there a setting in either Fabric or Distribute
that reads the requirements.txt file directly for the setup.py, so I
don't have to spell everything out in the setup(install_requires=[])
list, rather than writing a file reader for my requirements.txt?
You might still want to checkout frb's answer to the duplicate question How can I reference requirements.txt for the install_requires kwarg in setuptools.setup?, which provides a straight forward two line solution for writing a file reader.
If you really want to avoid this, you could alternatively add the common pip install -r requirements.txtto your fabfile.py, e.g.:
# ...
# create a place where we can unzip the tarball, then enter
# that directory and unzip it
run('mkdir /tmp/yourapplication')
with cd('/tmp/yourapplication'):
run('tar xzf /tmp/yourapplication.tar.gz')
# now install the requirements with our virtual environment's
# pip installer
run('/var/www/yourapplication/env/scripts/pip install -r requirements.txt')
# now setup the package with our virtual environment's
# python interpreter
run('/var/www/yourapplication/env/bin/python setup.py install')
Related
I am trying to generate requirements.txt for someone to replicate my environment. As you may know, the standard way is
pip freeze > requirements.txt
I noticed that this will list all the packages, including the dependencies of installed packages, which makes this list unnecessary huge. I then browsed around and came across pip-chill that allows us to only list installed packages in requirements.txt.
Now, from my understanding when someone tries to replicate the environment with pip install -r requirements.txt, this will automatically install the dependencies of installed packages.
If this is true, this means it is safe to use pip-chill instead of pip to generate the requirements.txt. My question is, is there any other risk of omitting dependencies of installed packages using pip-chill that I am missing here?
I believe using pip-compile from pip-tools is a good practice when constructing your requirements.txt. This will make sure that builds are predictable and deterministic.
The pip-compile command lets you compile a requirements.txt file from your dependencies, specified in either setup.py or requirements.in
Here's my recommended steps in constructing your requirements.txt (if using requirements.in):
Create a virtual env and install pip-tools there
$ source /path/to/venv/bin/activate
(venv)$ python -m pip install pip-tools
Specify your application/project's direct dependencies your requirements.in file:
# requirements.in
requests
boto3==1.16.51
Use pip-compile to generate requirements.txt
$ pip-compile --output-file=- > requirements.txt
your requirements.txt files will have:
#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile --output-file=-
#
boto3==1.16.51
# via -r requirements.in
botocore==1.19.51
# via
# boto3
# s3transfer
certifi==2020.12.5
# via requests
chardet==4.0.0
# via requests
idna==2.10
# via requests
jmespath==0.10.0
# via
# boto3
# botocore
python-dateutil==2.8.1
# via botocore
requests==2.25.1
# via -r requirements.in
s3transfer==0.3.3
# via boto3
six==1.15.0
# via python-dateutil
urllib3==1.26.2
# via
# botocore
# requests
Your application should always work with the dependencies installed from this generated requirements.txt. If you have to update a dependency you just need to update the requirements.in file and redo pip-compile. I believe this is a much better approach than doing pip freeze > requirements.txt which I see some people do.
I guess the main advantage of using this is you can keep track of the actual direct dependencies of your project in a separate requirement.in file
I find this very similar to how node modules/dependencies are being managed in a node app project with the package.json (requirements.in) and package-lock.json (requirements.txt).
From my point of view requirements.txt files should list all dependencies, direct dependencies as well as their dependencies (indirect, transient). If for some reason, only direct dependencies are wanted there are tools that can help with that, from a cursory look, pip-chill seems inadequate since it doesn't actually look at the code to figure out what packages are directly imported. Maybe better look at projects such as pipreqs, pigar, they seem to be more accurate in figuring out what the actual direct dependencies are (based on the imports in your code).
But at the end of the day you should curate such lists by hand. When writing the code you choose carefully which packages you want to import, with the same care you should curate a list of the projects (and their versions) containing those packages. Tools can help, but the developer knows better.
When I program, I often experiment with third party packages.
Of course, they are installed via pip.
When installed, packages install their dependencies.
At the end I want to have a clean requirements.txt which reflect what packages are really necessary for the project (pip freeze > requirements.txt).
If I manage to find a better solution to a package, I uninstall it via pip.
The problem is that when uninstalled, a package doesn't uninstall its dependencies.
If I'm not mistaken, this it doesn't. Well, and when I make pip freeze, I can't recognize what is going on here.
Therefore I decided to document which package installed which other packages.
When uninstalled, I uninstall them manually.
But this is really troublesome and error prone:
pip freeze > requirements.txt
asgiref==3.2.3
Django==3.0.2
pkg-resources==0.0.0
pytz==2019.3
sqlparse==0.3.0
Edit requirements.txt:
asgiref==3.2.3 # Django dependency
Django==3.0.2 # Django
pkg-resources==0.0.0 # Default package
pytz==2019.3 # Django dependency
sqlparse==0.3.0 # Django dependency
When I install a new package, I make pip freeze > requirements1.txt. Then compare the files, reveal the newly installed package's dependencies, mark them, then copy-paste old comments.
Well, finally I can just make it a complete mess. I make pip freeze and understand that I just don't know which package depends on which as I just forget to make my comments or something.
Again this is my goal: when I finish my project in a year, I'd like to know exactly that:
Every installed package is absolutely necessary for the project to run.
No unnecessary packages is installed.
Could you tell me what is the best practice to do that.
It seems pip-tools is the exact tool you need to solve the problems you encountered in your development workflow.
pip-tools enables you to specify your high-level dependencies (e.g. Django), while taking care automatically of your low-level dependencies (e.g. pytz). It also enables you to sync your environment to a requirements.txt file, regardless of the messy state you might have.
Installing pip-tools in your project
To use pip-tools, first create a new empty Python virtual environment in your project and activate it, using the tool of your choice:
with venv (built-in): python3 -m venv .venv && source .venv/bin/activate
with pyenv: pyenv virtualenv 3.8.1 project-env && pyenv local project-env
Then, install pip-tools in the Python virtual environment of your project:
pip install pip-tools
Managing dependencies with pip-compile using a requirements.in file
pip-tools includes a pip-compile tool that takes a requirements.in file as input. This requirements.in file is similar to requirements.txt, but it contains only high-level dependencies. For example, if your project is using the latest version of Django, you could write something like this in requirements.in:
django>=3.0,<3.1
You can see that requirements.in does not contain the dependencies of Django, only Django itself.
Once you have set your dependencies in requirements.in, use pip-compile to "compile" your requirements.in into a requirements.txt file:
➜ pip-compile requirements.in
#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile requirements.in
#
asgiref==3.2.3 # via django
django==3.0.2
pytz==2019.3 # via django
sqlparse==0.3.0 # via django
The requirements.txt file generated by pip-compile indicates the source of every indirect dependency next to the package name. For example, pytz==2019.3 is a dependency of django. In addition to that, it also pins each dependency to a precise version number, to make sure that the installation of your dependencies is reproducible
Applying dependencies from generated requirements.txt with pip-sync
Now that you have a requirements.txt, you can apply it on your Python virtual environment using pip-sync:
➜ pip-sync requirements.txt
Collecting asgiref==3.2.3 (from -r /var/folders/r1/n_n031s51wz2gjwy7mb9k4rh0000gn/T/tmpvhv549si (line 1))
Using cached https://files.pythonhosted.org/packages/a5/cb/5a235b605a9753ebcb2730c75e610fb51c8cab3f01230080a8229fa36adb/asgiref-3.2.3-py2.py3-none-any.whl
Collecting django==3.0.2 (from -r /var/folders/r1/n_n031s51wz2gjwy7mb9k4rh0000gn/T/tmpvhv549si (line 2))
Using cached https://files.pythonhosted.org/packages/55/d1/8ade70e65fa157e1903fe4078305ca53b6819ab212d9fbbe5755afc8ea2e/Django-3.0.2-py3-none-any.whl
Collecting pytz==2019.3 (from -r /var/folders/r1/n_n031s51wz2gjwy7mb9k4rh0000gn/T/tmpvhv549si (line 3))
Using cached https://files.pythonhosted.org/packages/e7/f9/f0b53f88060247251bf481fa6ea62cd0d25bf1b11a87888e53ce5b7c8ad2/pytz-2019.3-py2.py3-none-any.whl
Collecting sqlparse==0.3.0 (from -r /var/folders/r1/n_n031s51wz2gjwy7mb9k4rh0000gn/T/tmpvhv549si (line 4))
Using cached https://files.pythonhosted.org/packages/ef/53/900f7d2a54557c6a37886585a91336520e5539e3ae2423ff1102daf4f3a7/sqlparse-0.3.0-py2.py3-none-any.whl
Installing collected packages: asgiref, pytz, sqlparse, django
Successfully installed asgiref-3.2.3 django-3.0.2 pytz-2019.3 sqlparse-0.3.0
pip-sync will make sure your virtual environment corresponds exactly to what is defined in the requirements.txt file by uninstalling and installing the relevant packages.
After your environment has been synced, make sure your project code works properly. If it does, commit requirements.in and requirements.txt into version control.
Reference
For more details, you can refer to the docs of pip-tools, or to the article of Hynek Schlawack: Python Application Dependency Management in 2018.
Dependency management is a bit rougher in python land than others, there a tools like poetry and pipenv but for just poking a round I would try the following.
create a virtual environment in your project directory
$ python3 -m venv .venv
Activate your virtual environment
$ source .venv/bin/activate
Install whatever packages you're playing with
$ echo "package1" >> requirements-dev.txt
$ echo "package2" >> requirements-dev.txt
$ pip install --upgrade -r requirements-dev.txt
When you think you have everything you need deactivate your virtual env, create a new one, make sure everything still works, then create your new requirements.txt
$ deactivate
$ rm -rf .venv
$ python3 -m venv .venv
$ pip install --upgrade -r requirements-dev.txt
# test your code still works
# create a new requirements file to be used for "prod"
$ pip freeze > requirements.txt
I wouldn't call this best practice, but its a start and should get you by until to decide which side you want to join when it comes to dep management tools :)
I'm having issues with pip failing to install editable packages from a local directory. I was able to install the packages manually using commands like pip install -e pkg1. I wanted to use a requirements.txt file to automate future installs, because my coworkers will be working on the same packages. My ideal development workflow is for each developer to checkout the source from version control and run pip install -r requirements.txt. The requirements file would designate all the packages as editable so we can import our code without the need for .pth files but we wouldn't have to keep updating our environments. And by using namespace packages, we can decouple the import semantics from the file structures.
But it's not working out.
I have a directory with packages like so:
index/
pkg1/
src/
pkg1/
__init__.py
pkg1.py
setup.py
pkg2/
src/
...etc.
Each setup.py file contains something like:
from setuptools import setup, find_packages
setup(
name="pkg1",
version="0.1",
packages=find_packages('src'),
package_dir={'':'src'},
)
I generated my requirements.txt file using pip freeze, which yielded something like this:
# Editable install with no version control (pkg1==0.1)
-e c:\source\pkg1
# Editable install with no version control (pkg2==0.1)
-e c:\source\pkg2
...etc...
I was surprised when pip choked on the requirements file that it created for itself:
(venv) C:\Source>pip install -r requirements.txt
c:sourcepkg1 should either be a path to a local project or a VCS url beginning with svn+, git+, hg+, or bzr+
Also, some of our packages rely on other of our packages and pip has been absolutely useless at identifying these dependencies. I have resorted to manually installing packages in dependency order.
Maybe I'm pushing pip to its limits here. The documentation and help online has not been helpful, so far. Most sources discuss editable installation, installation from requirements files, package dependencies, or namespace packages, but never all these concepts at once. Usually when the online help is scarce, it means that I'm trying to use a tool for something it wasn't intended to do or I've discovered a bug.
Is this development process viable? Do I need to make a private package index or something?
What: I've a shell script that I'd like to distribute to my LUG.
I believe that a debian package will be the easiest way to distribute it. I want to create a .deb file for the script in this repository
Where: I want it to be placed in some directory like /usr/local/bin so that it is easy to execute and maybe create some symbolic links
Problem: How to write make file for it and/or other files and folders required to do that. I researched a lot when I tried to do it couple of months ago but no luck then. Here are the files from my previous attempt Now I'm trying to pack this for a tutorial on shell script in my LUG and facing similar situation again.
I'll be really glad if someone can be patient enough to guide me through it.
Any kind of resources or details will be much appreciated.
PS: I also intend to port the script to perl soon.
As mirabilos said, you should at least have a look to the packaging-tutorial written by Lucas Nussbaum, the current Debian Project Leader. You can install it directly from a Debian repository:
# apt-get install packaging-tutorial
Then, open and skim the PDF located at /usr/share/doc/packaging-tutorial/packaging-tutorial.pdf. After skimming it you'll have the basic knowledge needed to understand the structure of a Debian package.
Now let's get our hands dirty. mv your script to a new directory. The name of the directory must follow the nomenclature upstreamname-*version*.
rul#helicon:/tmp/a$ mkdir script-0.1
rul#helicon:/tmp/a$ mv script.sh script-0.1
cd to the directory where your script is and run dh_make --createorig. Choose single binary. You'll now have a debian/ directory with lots of file in it. This files are the ones you need to make your package. In your case, most, if not all, of the *.ex files are safe to be removed. Read and modify when needed the remaining files.
Now let's write the core of our package. You want to install a script in /usr/local/bin. The good news is that there is already a program that does that for you. You just have to specify the file name and where to put it. This program is dh_install. It has a very complete man page. After reading it, you should now understand that you have to create a install file in the debian/ directory.
rul#helicon:/tmp/a/script-0.1$ echo "script.sh usr/local/bin/" > debian/install
Here you have a real example of this file usage.
That's it! You have all you need to build your package. cd to the root directory of your package and run dpkg-buildpackage. If all went well, you'll have your fresh new .deb in ../.
You really should have a look at, in this order, the inofficial packaging tutorial, the Debian New Maintainers' Guide, Debian Developer's Reference and Policy. (The order is also increasingly dry, and reversed for formalness.)
It may take two days or so, but is really worth it.
Also, look at other small packages shipping only scripts, or other mere “file installers” (like php-htmlpurifier, first example I remembered while writing this).
If your package will only have a single file (or small number of files) in it, going through the full Debian toolchain might be overkill.
For packaging single files, I recommend you use the equivs tool. Install the equivs package, then run equivs-control to create a template file.
Edit the template file (give your package a name, version number etc.).
Add the name of your script to the Files: attribute in the template, for example:
Package: my-awesome-script
Version: 4.2
Files: my-awesome-script.sh /usr/local/bin
Section: misc
Priority: optional
Standards-Version: 3.9.2
Maintainer: Me <me#gmail.com>
Description: An awesome script doing stuff
Lorem ipsum etc. pp.
Put the script file alongside the template file.
Run equivs-build which will create your Debian package.
This is much easier for these simple cases than anything else – and the package that you get is standards compliant without resorting to any hacks or jumping through hoops.
for pre install, write your script in file DEBAIN/preinst;
for post install, write your script in file DEBAIN/postinst;
Use checkinstall or fpm to build your packages in minutes not hours or days!:
sudo checkinstall --fstrans=yes --install=no -D --pkgname=script \
--maintainer='Name <name#domain.tld>' --pkgarch=all --pkgversion=0.1 \
--nodoc cp script.sh /usr/local/bin
fpm -s dir -t deb --prefix /usr/local/bin -n script -v 0.1 -a all ./script.sh
Note: checkinstall requires dpkg/dpkg-deb (only works on Debian/Ubuntu), fpm is platform independent but requires ruby.
I'm trying to install discount on my VPS which is based on Solaris and compiling works great after setting some environment variables but the install fails.
So I thought I'd do the install manually, but what does install really do? Is it simply a mv followed by a chmod? Is it magic? That error seems to show that it attempts to do a lot of searching for files all over?
Can I just copy the binary, library and header files as usual?
Googling "install" doesn't give me much relevant information so I appreciate any clarification I can get!
According to man install:
install [OPTION]... [-T] SOURCE DEST`
install [OPTION]... SOURCE... DIRECTORY
install [OPTION]... -t DIRECTORY SOURCE...
install [OPTION]... -d DIRECTORY...
In the first three forms, copy SOURCE to DEST or multiple SOURCE(s) to
the existing DIRECTORY, while setting permission modes and owner/group.
In the 4th form, create all components of the given DIRECTORY(ies).
As for the difference to using cp, according to install vs. cp; and mmap, install unlinks the existing file, creating a new one linked to the same spot.
This has the advantage that, if the file you're trying to overwrite is a currently running program, it can continue running, as the file being written is in fact in a new location, and the existing program code is still in the old one.
A cp simply tries to overwrite the existing file, which will fail if the file is locked due to being in use.
Further information
install Command