I'm trying to switch over to using Mamba for my Python installation (Windows 10). Previously, I've just been installing Python using the installer from python.org and pip.
However, I have some one-off scripts I'd like to be able to run in a default environment without having to use source activate $env every time. The newest version of Mamba uses Python 3.10, but most of these scripts were written for Python 3.9.7. I'd like to downgrade the Python installation in the default environment to 3.9.7. I've tried doing this using mamba install python=3.9.7, which initially seems to work.
However, I get the following problem. When I run Python 3.10 using python in a command prompt after installing Mamba (i.e., before running the previous command to downgrade to 3.9.7), the interpreter launches and works correctly. Any packages I add to the base environment become available.
But after I downgrade the base environment to Python 3.9.7, I get the following warning when running python:
Warning:
This Python interpreter is in a conda environment, but the environment has
not been activated. Libraries may fail to load. To activate this environment
please see https://conda.io/activation
The interpreter does launch, but packages added to the base environment indeed fail to load when I attempt to import them.
I can get around this by running mamba activate base (though for some reason I don't understand, this is only successful from cmd and not PowerShell, which is annoying, since that's the shell I use by default). After this, python launches the interpreter without any warnings, and packages added to the base environment load as expected.
But what I'd like to do is downgrade Python to 3.9.7 and make it work like the default 3.10 environment, where the base environment is available without needing to manually activate it every time, since this is what I want for my one-off scripts that I want to be able to just run by clicking on them.
Is there a way I can downgrade the version of Python in Mamba's base environment, but make it work like the prepackaged version (i.e., without needing to issue mamba activate base every time)? Is the way to do this just to use an older version of Mamba that comes with Python 3.9.7?
"Is there a way I can downgrade the version of Python in Mamba's base environment...?"
Yes, but you shouldn't. It's best to leave base alone - it should strictly be used for Conda/Mamba infrastructure, not for running user scripts. If you want a default Python 3.9.7 environment, then create one, disable auto_activate_base, and have that new environment activate when your shell starts up.
Related
conda 4.10.1
airflow 2.2.2
I normally run a script in the following manner
conda activate env
python /path to script/script.py
So I put those two commands into a bash script and used the bashOperator like so:
t1 = BashOperator(
task_id='testtask',
depends_on_past=False,
bash_command='/path to bash/script.bash ',
retries=0,
)
and got the dreaded conda is not setup to activate environments.
Then I did:
conda init bash
conda activate env
python /path to script/script.py
but of course, the shell has to be restarted, which I don't know how to do in apache airflow. There has to be default args or something secret with the .bashrc etc. to activate anaconda environments in non interactive mode, but I'm a windows conda transplant and a tutorial is not handy.
There's this other solution which basically does a bunch of tricky things to start python in the environment of your choice,
How to run Airflow PythonOperator in a virtual environment
That secret hack is to just run the python in the environment:
bash_command='~/anaconda3/envs/env_of_choice/bin/python
/python_files/python_task1.py',
This guy was able to do it on anaconda 3.9!
How to change working directory and specify conda environment in Apache Airflow
But mysteriously, my environment and my base environment have the same python. When I type env for both environments the difference is in the following:
conda_shlvl=2 instead of 1
conda_prefix_1 = users/me/opt/anaconda3
path includes /users/me/opt/anaconda3/envs/env_of_choice/bin
conda_prefix=/users/me/opt/anaconda3/envs/env_of_choice
conda_default_env=sfdc
There are a few ways to go. Maybe I didn't set up the environment correctly and its using the base python instead of making a python in the virtual environment. I used a yml file. It's also really tempting just to set these environment variables in the DAG, but maybe that's not the accepted way? I couldn't find a tutorial. What's the right path? Or maybe my version, 4.10.1 is too advanced and I should downgrade to 3.9. Too many options. Advice?
The way I ended up doing this was to use the conda run command (inspired from this answer). conda run allows you to trigger a conda environment programmatically without needing to activate it - and this works within airflow.
I use conda 4.7.11 with auto_activate_base: false in ~/.condarc. I installed htop using conda install -c conda-forge htop. It was installed at ~/miniconda3/bin/htop. When I am in base environment I am able to use htop because ~/miniconda3/bin is prepended to PATH variable. But when I am outside all environments then only ~/miniconda3/condabin is prepended to PATH. When I am in all other environments except base then ~/miniconda3/envs/CUSTOM_ENV/bin and ~/miniconda3/condabin are prepended to PATH but not ~/miniconda3/bin, that's why I can use htop only from base environment. So my question is about how to be able to use htop installed using conda from all environments, including case when all environments are deactivated.
Please, don't suggest using package managers like apt or yum in my case (CentOS), because I have no root access to use this package manager. Thank you in advance.
Conda environments aren't nested, so what is in base is not inherited by the others. Isolation of environments is the imperative requirement, so it should make sense that the content in base env is not accessible when it isn't activated.
Option 1: Environment Stacking
However, there is an option to explicitly stack environments, which at this point literally means what you're asking for, namely, keeping the previous enviroment's bin/ in the PATH variable. So, if you htop installed only in base, then you can retain access to it in other envs like so
conda activate base
conda activate --stack my_env
If you decide to go this route, I think it would be prudent to be very minimal about what you install in base. Of course, you could also create a non-base env to stack on, but then it might be a bother to have to always activate this env, whereas in default installs, base auto-activates.
Starting with Conda v4.8 there will be an auto_stack configuration option:
conda config --set auto_stack true
See the documentation on environment stacking for details.
Option 2: Install by Default
If you want to have htop in every env but not outside of Conda envs, then the naive solution is to install it in every env. Conda offers a simple solution to this called Default Packages, and is in the Conda config under the key create_default_packages. Running the following will tell Conda to always install htop when creating a new env:
conda config --add create_default_packages htop
Unfortunately that won't update any existing envs, so you'd still have to go back and do that (e.g., Install a package into all envs). There's also a --no-default-packages flag for ignoring default packages when creating new envs.
Option 3: Global Installs
A Word of Caution
The following two options are not official recommendations, so caveat emptor and, if you do ever use them, be sure to report such a non-standard manipulation of $PATH when reporting problems/troubleshooting in the future.
Linking
Another option (although more manual) is to create a folder in your user directory (e.g., ~/.local/bin) that you add to $PATH in your .bashrc and create links in there to the binaries that you wish to "export" globally. I do this with a handful of programs that I wanted to use independently of Conda (e.g., emacs) even though they are installed and managed by Conda.
Dedicated Env
If you plan to do this with a bunch of software, then it might work to dedicate an env to such global software and just add its whole ./bin dir to $PATH. Do not do this with base - Conda wants to strictly manage that itself since Conda v4.4. Furthermore, do not do this with anything Python-related: stick strictly to native (compiled) software (e.g., htop is a good example). If an additional Python of the same version ends up on your $PATH this can create a mess in library loading. I've never attempted this and prefer the manual linking because I know exactly what I'm exporting.
I downloaded and installed Anaconda3-2019.03-MacOSX-x86_64.pkg which should normally install python 3.7. But oddly this is not the case :
(base) iMac-de-Bibi:~ ac$ which python
/Users/ac/anaconda3/bin/python
(base) iMac-de-Bibi:~ ac$ /Users/ac/anaconda3/bin/python --version
Python 2.7.16 :: Anaconda, Inc
Any help would be appreciated
Note : it's my first anaconda installation on this machine
You might have mistakenly downloaded and installed the wrong Anaconda distribution.
You might have changed the Python version in the base environment after the initial installation, by doing something like this:
conda activate base
conda install python=2.7
Update: As reported in a comment, installing pyside downgrades Python.
Have you correctly activated the base environment, and not messed with PATH or PYTHONPATH? I'm not sure about it, but python might be a script that calls the actual Python interpreter. It could pick the wrong one if search paths are off.
In general, I recommend to install Miniconda and create custom environments. Then you can choose whichever Python version you want for each environment, and don't care which one is in the base environment.
I noticed that when a conda environment is created without specifying the python version:
conda create --name snowflakes
instead of:
conda create --name snowflakes python=3.6
the environments are not separated and share the package with the default python interpreter.
Thereupon, What is the use of non-separated anaconda environments?
EDIT - 20170824:
The question has been solved. Actually non-separated environments do not exist. With the first command there is no new Python interpreter installed so that it calls the first that it finds in the PATH being the standard Python interpreter because there is no other.
I think you are misunderstanding the word "separate" in the docs. In the docs, they mean "separate" in the sense of "create a new environment, with a new name to try some new things". They do not mean that you are creating a different kind of conda environment. There is only one kind of environment in conda, what you are calling the "separated" environment. All packages in all environments are always unique. It so happens that the first command creates an empty environment with no packages. Therefore, when the new environment is activated, the PATH environment variable looks like: ~/miniconda3/envs/snowflakes/bin:~/miniconda3/bin:... Now, since there is no Python installed into ~/miniconda3/envs/snowflakes/bin (because the snowflakes environment is empty), the shell still finds Python in ~/miniconda3/bin as first on the path. The snowflakes environment does not share with the root environment. For instance, if, after creating, you type conda install -n snowflakes python it will install a new version of Python that won't find any packages! Therefore, there is only one kind of environment in conda, what you are calling the "separated" environment.
I've created a conda environment so I can use runsnakerun on my Mac as follows:
conda create -n runsnake wxPython
source activate runsnake
pip install runsnakerun
However, when I now attempt to use runsnake I get:
$ runsnake
This program needs access to the screen.
Please run with a Framework build of python, and only when you are
logged in on the main display of your Mac.
How can I get runsnake to work?
The problem is that conda's python is not a "framework python" on Mac, their decision is that you have to use pythonw instead. Unfortunately, pip builds entry point scripts using python not pythonw and, to make matters worse, RunSnakeRun does a horrible subprocess dance to launch itself in 32-bit mode on 64-bit macs.
The upshot is that the only hacky workaround I can think of to launch runsnake is as follows:
VERSIONER_PYTHON_PREFER_32_BIT=yes pythonw `which runsnake32`
I'm not actually sure that environment variable is even needed anymore, but the above does what the entry point appears to be trying to do and does at least start the RunSnakeRun gui.