I had installed Anaconda on my system before I knew the difference between Anaconda and Miniconda. I would like to downsize to Miniconda since I don't want the bloat of Anaconda, but I already have a handful of environments set up.
So far the only way I can think of migrating is to completely get rid of everything right now, install Miniconda, and then recreate my environments by hand, but that seems quite arduous. Is there a smarter way of going about this?
I agree with #darthbith: Export the envs to YAML files (conda env export) then recreate them once you have Miniconda installed (conda env create).
While there are some experimental tools for packaging and moving envs (i.e., so you avoid having to redownload packages), they only work on a single env basis. So, I can't see how going this route one could avoid making multiple copies of many of the shared files. Instead, if you let Conda handle the environment (re)creation, it will leverage hardlinks to minimize disk usage, and that seems to be one of your aims.
It may be possible to avoid redownloading packages during the environment recreations by retaining the pkgs directory in the root of your Anaconda install, then copying its contents over into the pkgs of the Miniconda install. I would only copy folders/tarballs that don't conflict with the ones that come with Miniconda. After finishing environment recreation, then a conda clean -p would likely be in order, since Anaconda includes many packages that likely aren't getting reused.
Related
I am not an expert in informatics stuffs. I deleted an environment that had many packages, one of them psi4 using the command:
conda remove --name myenv --all
However, in the folder:
~/anaconda3/pkgs
there are still some folders like:
psi4-1.3.2+ecbda83-py37h06ff01c_1, psi4-rt-1.3.2-py37h6cf1279_1
And the same happened for other packages that I manually identified, therefore, I assume that the same happen for the rest of the packages that belonged to this environment. And the problem is that these files take space from my disk and I really don't know how many and what are the packages on this situation.
Is there some way to delete all these non-used folders in order to free space?
Thanks in advance.
The command you used just removes the environment or installed package not the downloaded binary files. You can clean those up using:
conda clean -a
Just wondering as what is the right way to update Anaconda and Conda installation and virtual environments. Here is my confusion step by step:
When I run command conda update anaconda, it updates/downgrades alot of packages.
Then I ran conda update conda, which again updates/downgrades some packages.
Next, I ran conda update --update-all it starts downgrading/upgrading different packages.
Lastly, just to make sure that everything's updated, I ran conda update anaconda again. I was expecting a message like Everything's up to date but to my surprise it was again showing a huge list of packages that needed to be updated/downgraded again?
What am I doing wrong here? It appears to me as if I am going in circles with these commands. Any help?
You're not doing anything wrong per se, but it just doesn't make much sense to ever run conda update anaconda and conda update --all right after each other on the same env - they represent two completely different configurations.
Update Anaconda
Anaconda is a Python distribution that bundles together a ton of packages. Presumably, a bunch of testing goes into verifying that all the package versions and builds are compatible with each other. Because this takes time to do, the Anaconda team only releases new distributions (i.e., a new anaconda version) every couple months or so. If you want a stable set of packages that have been tested for interoperability, then do conda update anaconda.
Update All
In between Anaconda releases, new versions of many packages are still released on the Anaconda channel, and if you run conda update --all you're going to inevitably get ahead of the versions specified in the anaconda bundle. If you want the newest individual package releases and don't mind potentially working with package builds that aren't thoroughly tested for integration, then run conda update --all.
It may be worth noting that people who prioritize having access to the latest versions of packages often seem to prefer Conda Forge, because it tends to have more frequent package releases. However, in my opinion, there's almost no point to installing Anaconda if you're going to switch most packages to Conda Forge anyway. Instead, just install Miniconda and only install what you want from Conda Forge at the start.
Update None
Personally, I will rarely run conda update on an env once I've harden the requirements for a project. Every time you update an env, you risk breaking code that you've already written. Instead, Conda makes it quite easy to create new envs, and if they have a lot of overlap with other envs, then the envs can be quite light due to sharing packages across envs via hardlinking.
Update Conda
The one exception to everything above is the conda package, which is the very infrastructure you're using to manage packages and envs. That, one should update just like any other package manager (e.g., a pip or a homebrew).
Found the answers in this useful post by Anaconda
Keeping Anaconda Up To Date
Below is a question that gets asked so often that I decided it would be helpful to publish an answer explaining the various ways in which Anaconda can be kept up to date. The question was originally asked on StackOverflow.
I have Anaconda installed on my computer and I’d like to update it. In
Navigator I can see that there are several individual packages that
can be updated, but also an anaconda package that sometimes has a
version number and sometimes says custom. How do I proceed?
The Answer
What 95% of People Actually Want
In most cases what you want to do when you say that you want to update Anaconda is to execute the command:
conda update --all
This will update all packages in the current environment to the latest version—with the small print being that it may use an older version of some packages in order to satisfy dependency constraints (often this won’t be necessary and when it is necessary the package plan solver will do its best to minimize the impact).
This needs to be executed from the command line, and the best way to get there is from Anaconda Navigator, then the “Environments” tab, then click on the triangle beside the root environment, selecting “Open Terminal”:
This operation will only update the one selected environment (in this case, the root environment). If you have other environments you’d like to update you can repeat the process above, but first click on the environment. When it is selected there is a triangular marker on the right (see image above, step 3). Or, from the command line, you can provide the environment name (-n envname) or path (-p /path/to/env). For example, to update your dspyr environment from the screenshot above:
conda update -n dspyr --all
Update Individual Packages
If you are only interested in updating an individual package then simply click on the blue arrow or blue version number in Navigator, e.g. for astroid or astropy in the screenshot above, and this will tag those packages for an upgrade. When you are done you need to click the “Apply” button:
Or from the command line:
conda update astroid astropy
Updating Just the Packages in the Standard Anaconda Distribution
If you don’t care about package versions and just want “the latest set of all packages in the standard Anaconda Distribution, so long as they work together,” then you should take a look at this gist.
Why Updating the Anaconda Package is Almost Always a Bad Idea
In most cases, updating the Anaconda package in the package list will have a surprising result—you may actually downgrade many packages (in fact, this is likely if it indicates the version as custom). The gist above provides details.
Leverage conda Environments
Your root environment is probably not a good place to try and manage an exact set of packages—it is going to be a dynamic working space with new packages installed and packages randomly updated. If you need an exact set of packages, create a conda environment to hold them. Thanks to the conda package cache and the way file linking is used, doing this is typically fast and consumes very little additional disk space. For example:
conda create -n myspecialenv -c bioconda -c conda-forge python=3.5 pandas beautifulsoup seaborn nltk
The conda documentation has more details and examples.
pip, PyPI, and setuptools?
None of this is going to help with updating packages that have been installed from PyPI via pip, or any packages installed using python setup.py install. conda list will give you some hints about the pip-based Python packages you have in an environment, but it won’t do anything special to update them.
Commercial Use of Anaconda or Anaconda Enterprise
It’s pretty much exactly the same story, with the exception that you may not be able to update the root environment if it was installed by someone else (say, to /opt/anaconda/latest). If you’re not able to update the environments you are using, you should be able to clone and then update:
conda create -n myenv --clone root
conda update -n myenv --all
The other way in is simply,
anaconda-navigator
The resulting GUI image is below, the only difference with respect to this question is where you see "Installed", there is a drop down menu for for "Updatable" and therein you simply click the dependencies for updating for any given environment.
General info
I'm sure everyone knows this, but for anyone who doesn't Anaconda navigator is a point and click GUI already part of the Anaconda and is simply brilliant for managing, installing, updating and deleting all the dependencies.
With respect to the question it is great for managing all the dependencies inside new envs, creating new envs, loading new channels. It works great remotely via X11 if you have Anaconda loaded on a remote cluster/server.
The bonus for me is that I've never known it fail.
conda install conda=4.8.2
works, as it installs a specific version and '''/''' will not spn for long.
I use conda 4.7.11 with auto_activate_base: false in ~/.condarc. I installed htop using conda install -c conda-forge htop. It was installed at ~/miniconda3/bin/htop. When I am in base environment I am able to use htop because ~/miniconda3/bin is prepended to PATH variable. But when I am outside all environments then only ~/miniconda3/condabin is prepended to PATH. When I am in all other environments except base then ~/miniconda3/envs/CUSTOM_ENV/bin and ~/miniconda3/condabin are prepended to PATH but not ~/miniconda3/bin, that's why I can use htop only from base environment. So my question is about how to be able to use htop installed using conda from all environments, including case when all environments are deactivated.
Please, don't suggest using package managers like apt or yum in my case (CentOS), because I have no root access to use this package manager. Thank you in advance.
Conda environments aren't nested, so what is in base is not inherited by the others. Isolation of environments is the imperative requirement, so it should make sense that the content in base env is not accessible when it isn't activated.
Option 1: Environment Stacking
However, there is an option to explicitly stack environments, which at this point literally means what you're asking for, namely, keeping the previous enviroment's bin/ in the PATH variable. So, if you htop installed only in base, then you can retain access to it in other envs like so
conda activate base
conda activate --stack my_env
If you decide to go this route, I think it would be prudent to be very minimal about what you install in base. Of course, you could also create a non-base env to stack on, but then it might be a bother to have to always activate this env, whereas in default installs, base auto-activates.
Starting with Conda v4.8 there will be an auto_stack configuration option:
conda config --set auto_stack true
See the documentation on environment stacking for details.
Option 2: Install by Default
If you want to have htop in every env but not outside of Conda envs, then the naive solution is to install it in every env. Conda offers a simple solution to this called Default Packages, and is in the Conda config under the key create_default_packages. Running the following will tell Conda to always install htop when creating a new env:
conda config --add create_default_packages htop
Unfortunately that won't update any existing envs, so you'd still have to go back and do that (e.g., Install a package into all envs). There's also a --no-default-packages flag for ignoring default packages when creating new envs.
Option 3: Global Installs
A Word of Caution
The following two options are not official recommendations, so caveat emptor and, if you do ever use them, be sure to report such a non-standard manipulation of $PATH when reporting problems/troubleshooting in the future.
Linking
Another option (although more manual) is to create a folder in your user directory (e.g., ~/.local/bin) that you add to $PATH in your .bashrc and create links in there to the binaries that you wish to "export" globally. I do this with a handful of programs that I wanted to use independently of Conda (e.g., emacs) even though they are installed and managed by Conda.
Dedicated Env
If you plan to do this with a bunch of software, then it might work to dedicate an env to such global software and just add its whole ./bin dir to $PATH. Do not do this with base - Conda wants to strictly manage that itself since Conda v4.4. Furthermore, do not do this with anything Python-related: stick strictly to native (compiled) software (e.g., htop is a good example). If an additional Python of the same version ends up on your $PATH this can create a mess in library loading. I've never attempted this and prefer the manual linking because I know exactly what I'm exporting.
I am running different conda env and I'd like to specify where the packages are downloaded to, rather than having all of them in my $home.
I have found this question which, at time of writing, has no answers. However, my question is different: I don't want to specify the pkg_dir in my .condarc because I want to have a different download dir for each project (space is not an issue).
How do I define the pkg_dir for a specific conda env?
To note, I'm creating environments as conda env create -f my_env.yml -p complete-env.
A fundamental concept of conda is that packages get downloaded and extracted to a shared cache, from which they are selectively linked into the different conda environments. You want to work against this concept, so whatever you do will be hacky and have repercussions.
You could install a separate Miniconda for each of your projects, and (try to) make sure that they don't know about eachother by removing all conda-related files and environment settings from your home directory, or even use a different HOME for each project. Before working with a project, you'd have to put the correct conda on the PATH.
Or you could install Miniconda on a dedicated drive apart from your home directory, and put the conda environments inside your home directory. That would prevent conda from hard-linking the files. It would still download the packages into the shared cache, but then copy only the relevant files into each of your projects. Of course, copying is slower than hard-linking.
Specifying the package directory per environment rather than per conda installation is not possible, as darthbith has already pointed out in a comment.
I'm trying to use a Conda installation that was installed by a user other than myself, and with the root installation directory having read only perms.
Supposedly this is doable in the latest version of Conda and the envs_dirs setting, but even with the envs_dirs setting set to a writable path, Conda still tries to create locks in the read only root install path. Is there any way around this?
For anyone looking at this in the future - https://github.com/conda/conda/issues/1329
This issue addresses a multitude of problems with running shared user environments in Conda. As of time of this posting, it's partially supported and somewhat hacky, though check the link for status.