I'm running installing a conda env:
conda env create -f python/env/foobar.yml && . "$(conda info --base)/etc/profile.d/conda.sh" && conda activate foobar
Then running the python command:
/usr/local/envs/foobar/bin/python foobar.py
This command works locally (macos) but for some reason in the github ci (ubuntu), this gives the following error:
"Cannot run program "/usr/local/envs/foobar/bin/python": error=2, No such file or directory"
Why is this not working?
Generally, I would not recommend directly pointing at binaries in Conda environments as shown. This is because environments often have activation steps that set shell variables, and expected behavior of binaries can be dependent on those settings.
Typically, users activate the environment, then allow the software to be resolved by the shell. Something like,
conda activate foobar
python foobar.py
Alternatively, the conda run command allows for executing within an environment without manipulating the active shell. For example,
conda run -n foobar python foobar.py
Note that conda run defaults to buffering output (i.e., not interactive), but there are flags (see conda run -h) that can enable alternative modes.
Related
conda 4.10.1
airflow 2.2.2
I normally run a script in the following manner
conda activate env
python /path to script/script.py
So I put those two commands into a bash script and used the bashOperator like so:
t1 = BashOperator(
task_id='testtask',
depends_on_past=False,
bash_command='/path to bash/script.bash ',
retries=0,
)
and got the dreaded conda is not setup to activate environments.
Then I did:
conda init bash
conda activate env
python /path to script/script.py
but of course, the shell has to be restarted, which I don't know how to do in apache airflow. There has to be default args or something secret with the .bashrc etc. to activate anaconda environments in non interactive mode, but I'm a windows conda transplant and a tutorial is not handy.
There's this other solution which basically does a bunch of tricky things to start python in the environment of your choice,
How to run Airflow PythonOperator in a virtual environment
That secret hack is to just run the python in the environment:
bash_command='~/anaconda3/envs/env_of_choice/bin/python
/python_files/python_task1.py',
This guy was able to do it on anaconda 3.9!
How to change working directory and specify conda environment in Apache Airflow
But mysteriously, my environment and my base environment have the same python. When I type env for both environments the difference is in the following:
conda_shlvl=2 instead of 1
conda_prefix_1 = users/me/opt/anaconda3
path includes /users/me/opt/anaconda3/envs/env_of_choice/bin
conda_prefix=/users/me/opt/anaconda3/envs/env_of_choice
conda_default_env=sfdc
There are a few ways to go. Maybe I didn't set up the environment correctly and its using the base python instead of making a python in the virtual environment. I used a yml file. It's also really tempting just to set these environment variables in the DAG, but maybe that's not the accepted way? I couldn't find a tutorial. What's the right path? Or maybe my version, 4.10.1 is too advanced and I should downgrade to 3.9. Too many options. Advice?
The way I ended up doing this was to use the conda run command (inspired from this answer). conda run allows you to trigger a conda environment programmatically without needing to activate it - and this works within airflow.
I'm on a mac, and I'm trying to download a software called DLITE (which can be found here https://github.com/AllenCellModeling/DLITE). However, the instructions provided don't work I'm using Anaconda, and when I try and activate the environment that I've created, I get the following error message :
conda init dlite
ArgumentError: Invalid shells:
dlite
Currently available shells are:
bash
fish
powershell
tcsh
xonsh
zsh
I haven't used Anaconda or python much before, and I have no idea what to do to fix this, although I've tried a few things.
$ conda create --name dlite python=3.7
$ conda activate dlite
The command above will create the environment (as written in the dlite tutorial.
$ git clone https://github.com/AllenCellModeling/DLITE.git
$ cd DLITE
$ pip install -e .[all]
This will install the needed packages.
Hope this helped.
I have already created my own environment using conda in linux. When I check conda info --envs, it gives a list of environment like
# conda environments:
#
base * /home1/sriparna/anaconda3
copy_Env_deepgo /home1/sriparna/anaconda3/envs/copy_Env_deepgo
deepgo_2 /home1/sriparna/anaconda3/envs/deepgo_2
deepgo_3 /home1/sriparna/anaconda3/envs/deepgo_3
enzy /home1/sriparna/anaconda3/envs/enzy
parth /home1/sriparna/anaconda3/envs/parth
protein_struc /home1/sriparna/anaconda3/envs/protein_struc
py2 /home1/sriparna/anaconda3/envs/py2
But I cant activate a particular environment. e.g. when I tried conda activate parth it shows
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run
$ conda init <SHELL_NAME>
Currently supported shells are:
- bash
- fish
- tcsh
- xonsh
- zsh
- powershell
See 'conda init --help' for more information and options.
IMPORTANT: You may need to close and restart your shell after running 'conda init'.
Please help me regarding this.
This worked for me:
source {path_to_anaconda}/anaconda3/etc/profile.d/conda.sh
conda activate parth
Conda is not initialized in your shell. Run the following commands in an interactive shell,
conda init
Conda will detect the type of your shell, and write init scripts into the shell's configuration file. (Running it once is enough.)
Detail
Conda updated its environment activation after 4.6.
Quote from conda 4.6 release log
Conda 4.4 allowed “conda activate envname”. The problem was that setting up your shell to use this new feature was not always straightforward. Conda 4.6 adds extensive initialization support so that more shells than ever before can use the new “conda activate” command. For more information, read the output from “conda init –help”
In previous conda, the binaries installed by the default env "base" are exposed into the shell.
After conda init is introduced in conda 4.6, conda only expose command
conda into the PATH. And environment switch is unified by conda activate env-name and conda deactivate on all platforms. But to make these commands work, you have to do an additional initialization with conda init.
Read the conda 4.6 release log for more detail.
I had installed anaconda in Ubuntu VM environment but base was not loading, tried adding the PATH to .bashrc, didn't solve as Conda:not found was error.
Then I solved through:
Press F1>>Terminal: Select Default Profile
You might have available options :
bash
fish
tcsh
xonsh
zsh
powershell
Choose YOUR_SHELL_NAME
Run:
eval "$(/home/mishra/anaconda3/bin/conda shell.YOUR_SHELL_NAME hook)"
#example
eval "$(/home/mishra/anaconda3/bin/conda shell.bash hook)"
Immediately (base) will show
There seem to be other similar questions, but none that seem to be exactly this.
I have created a conda install on a machine. I want to create the exact same install on a 2nd machine. I could export the environment and then create an environment using that exported information. But in all the examples, I'm creating an environment that is a clone of the root. Not a new root that is a clone of the root.
So how to I create a true clone of a conda install? The idea would be
Export the environment of the root on machine 1
conda [something...] > configuration.yml
Install miniconda using Miniconda....sh on machine 2
Recreate the original environment
conda [install all the stuff including enironments] configuration.yml
I can't seem to find any explicit instructions for doing this. The closest I found suggests:
conda env export > environment.yml
conda env update -n root -f environment.yml
However the first command draws an error:
balter#server:/home/.../Applications$ conda env export > environment.yml
CondaEnvException: Conda Env Exception: Unable to determine environment
Please re-run this command with one of the following options:
* Provide an environment name via --name or -n
* Re-run this command inside an activated conda environment.
balter#server:/home/.../Applications$ conda -h
usage: conda [-h] [-V] command ...
conda is a tool for managing and deploying applications, environments and packages.
Options:
...
...
One of the posts I found on the interwebs discusses two points that I'm unclear about:
1) Whether pip installs also carry over, and
2) The difference between
conda env create
and
conda create
Conda environments exist as an easy solution to this sort of problem. However, it is possible to do what you ask.
Step 1: confirm you are in the root environment on the first machine, then export a list of the installed packages:
$ source activate root
$ conda list -e > root.yml
Step 2: on the other machine, download and install Miniconda, then install the packages from root.yml:
$ wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
$ bash Miniconda3-latest-Linux-x86_64.sh
$ conda install --file root.yml
How can I install a package under development to an Anaconda environment?
With pip:
pip install -e /path/to/mypackage
or with regular setuptools:
python /path/to/mypackage/setup.py develop
There is also conda develop available now.
http://conda.pydata.org/docs/commands/build/conda-develop.html
Update in 2019: conda develop hasn't been maintained and is not recommended. See https://github.com/conda/conda-build/issues/1992
Recommendation is to use python setup.py develop or pip install -e .
Using either of those will work with Anaconda. Make sure that you have pip or setuptools installed into the conda environment you want to install into, and that you have it activated.
This is the equivalent to pip install -e .
conda install conda-build
conda develop .
As explained in this gh issue thread, because of build isolation and dependency installation, Anaconda developers recommend using:
pip install --no-build-isolation --no-deps -e .
Build / Host Environment
To create build and host environments and a build script go to your recipe directory and use
conda debug /path/to/your/recipe-directory
as documented here. This will print an instructive message like
################################################################################
Build and/or host environments created for debugging. To enter a debugging environment:
cd /home/UserName/miniconda3/conda-bld/debug_1542385789430/work && source /home/UserName/miniconda3/conda-bld/debug_1542385789430/work/build_env_setup.sh
To run your build, you might want to start with running the conda_build.sh file.
################################################################################
(The message might tell you incorrectly, that it created a test environment.) Your source code has been copied to the .../work directory and there is also a conda_build.sh script. Note, that sourcing the build_env_setup.sh will load both build and host environments.
You can work on your code and your recipe and build with the conda_build.sh, but you won't get a proper conda package, as far as I know. When you are finished, you can remove the debug environment:
conda deactivate # maybe twice
conda build purge
Test Environment
To get the test environment, you have to build the package first and then debug that. This might be useful to fix your test files.
conda build /path/to/your/recipe-directory # creates mypackage*.tar.bz2
# find file location of mypackage*.tar.bz2 with:
conda search --info --use-local mypackage # look at the url row for the path
cd /path/to/miniconda3/conda-bld/linux-64/ # go to that path, can be different
conda debug mypackage*.tar.bz2
This will print e. g.:
################################################################################
Test environment created for debugging. To enter a debugging environment:
cd /home/UserName/miniconda3/conda-bld/debug_1542385789430/test_tmp && source /home/UserName/miniconda3/conda-bld/debug_1542385789430/work/conda_test_env_vars.sh
To run your tests, you might want to start with running the conda_test_runner.sh file.
################################################################################
Again, remove with
conda deactivate
conda build purge
Run Environment
This is actually no debugging, but the general process of building and installing a local package. With the run environment you can check, whether all dependencies are specified in the requirements/run section. Also pinning can be an issue.
(base) $ conda build /path/to/your/recipe-directory
(base) $ conda create --name package-env --use-local mypackage
(base) $ conda activate package-env
(package-env) $ python
>>> import mypackage
You can also list the dependencies of your package with (man page)
conda search --info --use-local mypackage
A last hint: If you want to know the versions of the dependencies and see, whether pinning works, try (man page)
conda render /path/to/your/recipe-directory