I am trying to use two requirements.txt in environment.yml to create a conda environment.
One of them has --no-index --find-links=/home/myuser/mydownloadedpackage.
For some reason, it always go to download that package which I want it to install from my downloaded directory.
Is there a way that I can force it to use find-links for one requirements.txt and global configuration for the other ?
Related
I have a conda environment with packages installed via conda install. I also have two local development packages that were each installed with pip install -e .. Now, conda env export shows everything, including both local development packages. However, I don't want conda to include them when creating the same environment on other machines - I want to keep doing it via pip install -e ..
So how can I exclude both local packages when creating the environment.yml file? Do I need to manually remove them or is there a way to this from the command line?
While there are some alternative flags for conda env export that change output behavior (e.g., --from-history is most notable), there really isn't anything as specific as OP describes. Instead, manually remove the offending lines.
Note that YAMLs do support all pip install commands, so the editable installs can also be included. For example, https://stackoverflow.com/a/59864744/570918.
Consider Prioritizing the YAML Specification
In a software engineering setting, I would expect that users should not even be hitting development environments with conda install or pip install commands. Instead, the team should have a maunally-written, version-controlled YAML to begin with and all installations/changes to the environment are managed through editing the YAML file and using conda env update to propagate changes in the YAML to the environment.
That is, conda env export should not be necessary because the environment already has a well-defined means of creation.
Background
I have a multi-stage Dockerfile for a JupyterLab image. There are three stages:
server
kernel: mamba create -y -n /tmp/kernel-packages/myenv ...
runner:
FROM ... as runner
...
COPY --from=kernel /tmp/kernel-packages/ /opt/conda/envs
RUN conda config --append envs_dirs /opt/conda/envs/
...
Problem
In the resulting image, the preferred python -m pip works, but pip gives:
bash: /opt/conda/envs/myenv/bin/pip: /tmp/kernel-packages/myenv/bin/python3.9: bad interpreter: No such file or directory
The reason is that pip has #!/tmp/kernel-packages/myenv/bin/python3.9 as its shebang.
Expected
A behaviour like pip install --root /tmp/server-packages ..., which works perfectly fine in a COPY --from=server /tmp/server-packages /.
Additional information
Additionally, some other binaries, like curl or papermill also have wrong paths hardcoded by conda. I've read about Anaconda | Moving Conda Environments, but it seems like overkill to use conda-pack and conda-unpack.
Workaround
Simply create an env by name: mamba create -y -n myenv and then in the runner stage COPY --from=kernel /opt/conda/envs /opt/conda/envs.
Question
Is there a Conda (Mamba) equivalent to pip install's --root option? I'm not looking for a solution to the stated problem as I have I have already found a way that works for me. My interest is purely in the functionality of the conda binary.
The --prefix argument is the equivalent - just that some Conda packages use hardcoded paths, hence the issue you encounter.
conda-prefix-replacement
To properly move a Conda environment to a new prefix via a COPY operation one would need to run the conda-prefix-replacement tool (a.k.a., cpr) to ensure that all files with hardcoded paths get updated to the new location. Presumably, conda-pack is doing a similar thing, just under-the-hood.
For your purposes, you might consider pre-running cpr on the environment(s) in the kernel image so that they are ready to work in the deployed location. Though that would mean always COPYing to the same location.
See the cpr repository for details on use.
Working in an offline Windows 10 environment where I have available a local mirror of conda-forge and an installation of Anaconda3. I need to create an environment with packages of which some are only to be found in conda-forge mirror and some only in local Anaconda3 C:/Users/<user>/AppData/Local/Continuum/anaconda3/pkgs folder. Note that the latter is not a subset of the former (e.g. scipy).
The problem is when I create an environment from an environment.yml that includes packages from both groups, then it fails because conda create --name test --file environment.yml seems to be only searching the channel. On the other hand conda create --name test --offline --file environment.yml is only looking in the above referenced pkgs folder.
How can I point conda create to both repositories?
I create a conda environment without specifying any packages using the following command:
conda create --name test_env
I can then use all the packages in the root environment inside test_env (but they do not appear in the outputs of conda list and conda env export). This is already unexpected to me but the real problems begin when I then install something inside that environment, e.g.:
conda install pywavelets
Afterwards, pywavelets is usable but all the other packages which are no dependencies of pywavelets disappear inside the environment (e.g. pandas). I don't understand why that happens. Does anybody have an explanation for that?
More importantly, what does this mean for best practices for working with conda environments? Should I always create my environments specifying at least python (conda create --name test_env python)? However, then I have to install everything by hand in that environment which is quite cumbersome. So, my idea now is to specify anaconda for all environments I create:
conda create --name test_env anaconda
The disadvantage, however, is that the list of dependencies listed by conda list and conda env export gets unnecessarily long (e.g. even listing the Anaconda Navigator). Does anybody have a better solution for this?
The reason you can use all the packages from the root environment when you don't specify a Python version during environment creation is because you're actually using the root environment's Python executable! You can check with which python or python -c "import sys; print(sys.executable)". See also my other answer here.
When you install pywavelets, one of the dependencies is (probably) Python, so a new Python executable is installed into your environment. Therefore, when you run Python, it only picks up the packages that are installed in the test_env.
If you want all of the packages from another environment, you can create a file that lists all the packages and then use that file to create a new environment, as detailed in the Conda docs: https://conda.io/docs/user-guide/tasks/manage-environments.html#building-identical-conda-environments
To summarize
conda list --explicit > spec-file.txt
conda create --name myenv --file spec-file.txt
or to install into an existing environment
conda install --name myenv --file spec-file.txt
Since that's just a text file, you can edit and remove any packages that you don't want.
I am struggling with versionning and dependencies with conda and python packages.
When doing : conda install -c conda-forge qt==5.6.2
it installs all the dependencies or None of them (-no-dependencies).
1) How to install/update selectively the dependencies ?
(because some cause breakage for other packages).
2) I have a sandbox envs in conda where I test the install+regression test.
But, it works, I would like to reproduce the install in other environnment.
Is it a way to modify directly the config file of the environnement and add manually the new packages ?
For regression test, am also using
https://github.com/pelson/conda-execute
which allows temp envs setup with dependencies.
If it can help other people, stuck in this situation,
work around is using --force :
conda -c channel install packagename --force
it will install only the package.
If you want to selectively install packages,
conda -c channel packagename
and you can get the list of dependencies from where you can choose to install.