pip has a -q/--quiet flag that works ideally from the command line. I'm using an automated deployment process (Amazon Elastic Beanstalk), and the tools use pip to install from a requirements file.
Unfortunately, pip is generating non-error output that's causing EB to abort due to its logger being unable to handle non-ASCII output.
Since I can't apply the quiet flag to the pip command directly (it's run automatically), is there a per-line flag I can set in my requirements file or an environment variable that would suppress pip's output?
Pip offers the --quiet / -q option to silence output. Example:
pip install -q -r requirements.txt
After more digging, this is a pending feature request for pip in github:
https://github.com/pypa/pip/issues/271
Temporary workaround: Using a separate bash script to invoke pip per-line until this is implemented, published, and available on Elastic Beanstalk.
Related
Background
I have a multi-stage Dockerfile for a JupyterLab image. There are three stages:
server
kernel: mamba create -y -n /tmp/kernel-packages/myenv ...
runner:
FROM ... as runner
...
COPY --from=kernel /tmp/kernel-packages/ /opt/conda/envs
RUN conda config --append envs_dirs /opt/conda/envs/
...
Problem
In the resulting image, the preferred python -m pip works, but pip gives:
bash: /opt/conda/envs/myenv/bin/pip: /tmp/kernel-packages/myenv/bin/python3.9: bad interpreter: No such file or directory
The reason is that pip has #!/tmp/kernel-packages/myenv/bin/python3.9 as its shebang.
Expected
A behaviour like pip install --root /tmp/server-packages ..., which works perfectly fine in a COPY --from=server /tmp/server-packages /.
Additional information
Additionally, some other binaries, like curl or papermill also have wrong paths hardcoded by conda. I've read about Anaconda | Moving Conda Environments, but it seems like overkill to use conda-pack and conda-unpack.
Workaround
Simply create an env by name: mamba create -y -n myenv and then in the runner stage COPY --from=kernel /opt/conda/envs /opt/conda/envs.
Question
Is there a Conda (Mamba) equivalent to pip install's --root option? I'm not looking for a solution to the stated problem as I have I have already found a way that works for me. My interest is purely in the functionality of the conda binary.
The --prefix argument is the equivalent - just that some Conda packages use hardcoded paths, hence the issue you encounter.
conda-prefix-replacement
To properly move a Conda environment to a new prefix via a COPY operation one would need to run the conda-prefix-replacement tool (a.k.a., cpr) to ensure that all files with hardcoded paths get updated to the new location. Presumably, conda-pack is doing a similar thing, just under-the-hood.
For your purposes, you might consider pre-running cpr on the environment(s) in the kernel image so that they are ready to work in the deployed location. Though that would mean always COPYing to the same location.
See the cpr repository for details on use.
Most of you may know OpenAI playground, so I built an function generator app and followed all the instructions but I can't launch it via venv-python. I can actually install the requirements using pip while venv is not active and can launch the site with flask, but when I try to install requirements after activating venv, I get this error;
User#lalec ~
$ cd openai-quickstart-python
User#lalec ~/openai-quickstart-python (master)
$ . venv/Scripts/activate
(venv)
User#lalec ~/openai-quickstart-python (master)
$ pip install -r requirements.txt
Fatal error in launcher: Unable to create process using '"C:\Users\celal\openai-quickstart-python\venv\Scripts\python.exe" "C:\Users\User\openai-quickstart-python\venv\Scripts\pip.exe" install -r requirements.txt': The system cannot find the file specified.
How do I fix this? I added every possible script locations into PATH I thought it would help but no result.
I just realized two directories in the error doesn't match, and "C:\Users\celal\openai-quickstart-python\venv\Scripts\python.exe" actually does not even exist. Maybe that's the problem... How can I change this ??
Sorry if I'm asking dumb questions I'm new to all this... Also, I don't get why I need venv activated while I can just launch it by accessing the directory and type flask run in cmd, would appreciate any answers.
ss of the project's directory
ss of venv/Scripts/
ss of bash screen with errors
The error message suggests that the requirements.txt file can't be read. Check that you definitely can read it and it's in the expected location.
The perferred command for running pip is:
python3 -m pip install -r requirements.txt
This makes sure that you know which python3 binary you're running pip with - some systems have more than one python3 binary in various locations.
You could achieve similar without activating the virtual environment:
./venv/bin/python3 -m pip install -r requirements.txt
quick question - I have a big requirements file. On one system, I have a couple requirements (pytorch, torchvision) which don't install on the a particular machine. Is there a way I can still use the file to install everything BUT these? Something like
pip install -r requirements.txt --except=pytorch,torchvision
I don't see anything like this in the pip options but maybe there's another way.
There is no way. You have to somehow process the list and exclude packages before passing them to pip. Something like
pip install `grep -v 'pytorch\|torchvision' requirements.txt`
If you are working in Bash, it might be a better idea to use process substitution:
pip install -r <(grep -E -v 'pytorch|torchvision' requirements.txt)
This way, you don't have to worry about comments etc.
In pip there is an -f option which does the following:
-f, --find-links : If a url or path to an html file, then parse for links to archives. If a local path or file:// url that's a directory, then look for archives in the directory listing.
This is the preferred way of installing PyTorch, by setting the link to their overview website, e.g.:
pip3 install torch===1.3.0 -f https://download.pytorch.org/whl/torch_stable.html
For my virtual environments I use pipenv but I haven't found an option that does the same as -f. In the meantime, I can just look up the direct link to the package that is relevant for my system, but that is cumbersome.
Does pipenv provide a way to do the same thing as pip's -f?
In the new version of pipenv (I tested with version 2020.11.15), you can install packages in this way:
pipenv install https://download.pytorch.org/whl/cpu/torch-1.3.0%2Bcpu-cp36-cp36m-linux_x86_64.whl
The link can be found in the this page: https://download.pytorch.org/whl/torch_stable.html
This will be added to the Pipfile as well.
[packages]
torch = {file = "https://download.pytorch.org/whl/cpu/torch-1.3.0%2Bcpu-cp36-cp36m-linux_x86_64.whl"}
you need to manually check the link with your compute platform, os and python version.
It is possible to use environment variables recognised by pip to tweak its behaviour within pipenv's execution. E.g.:
PIP_FIND_LINKS=https://download.pytorch.org/whl/torch_stable.html pipenv install torch==1.5.1+cu101
See:
Advanced Usage of Pipenv - Configuration With Environment Variables
pip's User Guide - Environment Variables
For now, I haven't found a work-around. What you can do, of course, is enabling the pipenv shell and doing what you must with pip, e.g.
pipenv shell
python -m pip install torch===1.3.0 -f https://download.pytorch.org/whl/torch_stable.html
This will install torch in the pipenv environment but torch will not be added to the Pipfile (nor to the lock file).
I have a simple bash script that connects to a series of servers and updates a specific package, using a here-string to answer a prompt:
sudo /usr/bin/apt-get install package-name <<< Y
The prompt is:
Configuration file /etc/package-name/package-name.conf
==> File on system created by you or by a script.
==> File also in package provided by package maintainer.
What would you like to do about it? Your options are:
Y or I : install the package maintainer's version
N or O : keep your currently-installed version
D : show the differences between the versions
Z : background this process to examine the situation
The default action is to keep your current version.
*** package-name.conf (Y/I/N/O/D/Z [default=N] ?
This is the error when it doesn't work:
dpkg: error processing package-name (--configure):
EOF on stdin a conffile prompt
Errors were encountered while processing:
package-name
I'm not able to install expect or any other programs on the servers. Any help would be appreciated.
You should pass in a dpkg option to specify at the prompt the desired behavior in this scenario. Then it won't prompt.
sudo apt-get -o DPkg::Options::="--force-confnew" -y install package-name
(Untested; obtained by googling.)
If you look at the apt-get man page, you can find an option (-y) to answer yes for you. Why not try that.