The conda documentation says that when you use
conda create --name myenv
The new environment
uses the same version of Python that you are currently using because
you did not specify a version.
However, that's not the case for me. I have Windows 10 and Anaconda. I am into the "base" environment created by default.
If I run
conda create --name testenv
Then when I activate the environment
conda activate testenv
There is no Python. If I write
python
to the console the Microsoft Store is opened.
To have a Python interpreter I need to manually specify it
conda create --name testenv2 python=3.8
That specific note in the Conda documentation was a hold-over from before Conda v4.4 and has since been corrected (see here and here).
Background
Previous to Conda v4.4, the base environment's bin/ directory was always on the PATH, hence why not installing a Python interpreter in a new environment it would fall back to the base Python. Conda v4.4 introduced a new strategy for managing environment isolation via defining the primary interface to Conda as a set of shell functions and allowing the base bin directory only to be included on PATH when the base environment was active. This strategy provides cleaner isolation of environments, which means that only what is in the active environment will be available.
Hence, if you want Python in the environment, it must be explicitly installed.
Related
conda 4.10.1
airflow 2.2.2
I normally run a script in the following manner
conda activate env
python /path to script/script.py
So I put those two commands into a bash script and used the bashOperator like so:
t1 = BashOperator(
task_id='testtask',
depends_on_past=False,
bash_command='/path to bash/script.bash ',
retries=0,
)
and got the dreaded conda is not setup to activate environments.
Then I did:
conda init bash
conda activate env
python /path to script/script.py
but of course, the shell has to be restarted, which I don't know how to do in apache airflow. There has to be default args or something secret with the .bashrc etc. to activate anaconda environments in non interactive mode, but I'm a windows conda transplant and a tutorial is not handy.
There's this other solution which basically does a bunch of tricky things to start python in the environment of your choice,
How to run Airflow PythonOperator in a virtual environment
That secret hack is to just run the python in the environment:
bash_command='~/anaconda3/envs/env_of_choice/bin/python
/python_files/python_task1.py',
This guy was able to do it on anaconda 3.9!
How to change working directory and specify conda environment in Apache Airflow
But mysteriously, my environment and my base environment have the same python. When I type env for both environments the difference is in the following:
conda_shlvl=2 instead of 1
conda_prefix_1 = users/me/opt/anaconda3
path includes /users/me/opt/anaconda3/envs/env_of_choice/bin
conda_prefix=/users/me/opt/anaconda3/envs/env_of_choice
conda_default_env=sfdc
There are a few ways to go. Maybe I didn't set up the environment correctly and its using the base python instead of making a python in the virtual environment. I used a yml file. It's also really tempting just to set these environment variables in the DAG, but maybe that's not the accepted way? I couldn't find a tutorial. What's the right path? Or maybe my version, 4.10.1 is too advanced and I should downgrade to 3.9. Too many options. Advice?
The way I ended up doing this was to use the conda run command (inspired from this answer). conda run allows you to trigger a conda environment programmatically without needing to activate it - and this works within airflow.
I have been using miniconda for a while and have set up conda environments for each for each of my projects. What I can't figure out after looking through the documentation, is there a way to bond/associate my conda environment to my project folder for that conda environment? So that when I activate a specific conda environment it moves directly into the associated project directory. This virtualenvwrapper etc. can do for example. Is conda able to this?
As mentioned in Activating an environment, conda automatically executes "activation scripts" when an environment is activated. These scripts are typically provided by conda packages installed in the environment.
Just add a script of your own with a cd command. See here for details:
https://stackoverflow.com/a/43415167/11451509
I want to know after creating and activating a conda environment in terminal. Say venv:
conda create -n venv
source activate venv
Then the prompt will come with the enviroment name (venv).
would the packages installed (say conda install tensorflow without --name venv) after environment activation only effect inside the environment venv? Or would it affect outside venv?
I don't want to mess up my environments.
The point of using an environment in conda is to have a separate space where you can install packages and apply your configurations.
As long as you are inside an environment, all the packages you install will only be installed there. One way you can check this is by listing the packages you installed using: conda list
You will see that the packages you installed in the active environment are not installed in the other ones.
I hope this answers your question
I create a conda environment without specifying any packages using the following command:
conda create --name test_env
I can then use all the packages in the root environment inside test_env (but they do not appear in the outputs of conda list and conda env export). This is already unexpected to me but the real problems begin when I then install something inside that environment, e.g.:
conda install pywavelets
Afterwards, pywavelets is usable but all the other packages which are no dependencies of pywavelets disappear inside the environment (e.g. pandas). I don't understand why that happens. Does anybody have an explanation for that?
More importantly, what does this mean for best practices for working with conda environments? Should I always create my environments specifying at least python (conda create --name test_env python)? However, then I have to install everything by hand in that environment which is quite cumbersome. So, my idea now is to specify anaconda for all environments I create:
conda create --name test_env anaconda
The disadvantage, however, is that the list of dependencies listed by conda list and conda env export gets unnecessarily long (e.g. even listing the Anaconda Navigator). Does anybody have a better solution for this?
The reason you can use all the packages from the root environment when you don't specify a Python version during environment creation is because you're actually using the root environment's Python executable! You can check with which python or python -c "import sys; print(sys.executable)". See also my other answer here.
When you install pywavelets, one of the dependencies is (probably) Python, so a new Python executable is installed into your environment. Therefore, when you run Python, it only picks up the packages that are installed in the test_env.
If you want all of the packages from another environment, you can create a file that lists all the packages and then use that file to create a new environment, as detailed in the Conda docs: https://conda.io/docs/user-guide/tasks/manage-environments.html#building-identical-conda-environments
To summarize
conda list --explicit > spec-file.txt
conda create --name myenv --file spec-file.txt
or to install into an existing environment
conda install --name myenv --file spec-file.txt
Since that's just a text file, you can edit and remove any packages that you don't want.
I noticed that when a conda environment is created without specifying the python version:
conda create --name snowflakes
instead of:
conda create --name snowflakes python=3.6
the environments are not separated and share the package with the default python interpreter.
Thereupon, What is the use of non-separated anaconda environments?
EDIT - 20170824:
The question has been solved. Actually non-separated environments do not exist. With the first command there is no new Python interpreter installed so that it calls the first that it finds in the PATH being the standard Python interpreter because there is no other.
I think you are misunderstanding the word "separate" in the docs. In the docs, they mean "separate" in the sense of "create a new environment, with a new name to try some new things". They do not mean that you are creating a different kind of conda environment. There is only one kind of environment in conda, what you are calling the "separated" environment. All packages in all environments are always unique. It so happens that the first command creates an empty environment with no packages. Therefore, when the new environment is activated, the PATH environment variable looks like: ~/miniconda3/envs/snowflakes/bin:~/miniconda3/bin:... Now, since there is no Python installed into ~/miniconda3/envs/snowflakes/bin (because the snowflakes environment is empty), the shell still finds Python in ~/miniconda3/bin as first on the path. The snowflakes environment does not share with the root environment. For instance, if, after creating, you type conda install -n snowflakes python it will install a new version of Python that won't find any packages! Therefore, there is only one kind of environment in conda, what you are calling the "separated" environment.