How to activate conda environment in GitHub Actions? - installation

Question

I am setting up continuous integration using GitHub Actions. One of the prerequisites (samtools) is most easily installed by conda. The standard way to use installed packages is to activate the corresponding conda environment. I am looking for a way to activate the conda environment. The common methods to activate it failed, see details below. My current workaround is to add to the PATH a hardcoded path to samtools, installed by conda. But this is not maintainable if the number of installed packages increases. It is also not the standard way to use packages installed with conda.
DETAILS:
.github/workflows/conda.yml
name: Conda
on: [push]
jobs:
# label of the job
tests:
name: Tests
# containers must run in Linux based operating systems
runs-on: ubuntu-latest
# Docker Hub image that `postgres-job` executes in
container: node:latest
# service containers to run with `postgres-job`
steps:
- uses: conda-incubator/setup-miniconda#v2
with:
miniconda-version: "latest"
channels: bioconda, conda-forge, defaults
use-only-tar-bz2: true # IMPORTANT: This needs to be set for caching to work properly!
auto-update-conda: true
auto-activate-base: true
- name: Install samtools
run: |
set -x
echo "begin: PATH=$PATH;"
conda create -y --name samtools samtools
conda activate samtools || true
echo "after conda activate samtools: PATH=$PATH;"
which samtools || true
$CONDA/bin/activate samtools || true
echo "after CONDA/bin/activate samtools: PATH=$PATH;"
which samtools || true
export PATH="3/envs/samtools/bin:$PATH"
echo "after export PATH=3/envs/samtools/bin:PATH: PATH=$PATH;"
which samtools || true
Output:
Run set -x
begin: PATH=3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin;
+ echo begin: PATH=3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin;
+ conda create -y --name samtools samtools
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
[...]
# To activate this environment, use
#
# $ conda activate samtools
#
# To deactivate an active environment, use
#
# $ conda deactivate
+ conda activate samtools
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run
$ conda init <SHELL_NAME>
Currently supported shells are:
- bash
- fish
- tcsh
- xonsh
- zsh
- powershell
See 'conda init --help' for more information and options.
IMPORTANT: You may need to close and restart your shell after running 'conda init'.
+ true
after conda activate samtools: PATH=3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin;
+ echo after conda activate samtools: PATH=3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin;
+ which samtools
+ true
+ 3/bin/activate samtools
+ echo after CONDA/bin/activate samtools: PATH=3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin;
+ which samtools
after CONDA/bin/activate samtools: PATH=3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin;
+ true
after export PATH=3/envs/samtools/bin:PATH: PATH=3/envs/samtools/bin:3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin;
+ export PATH=3/envs/samtools/bin:3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ echo after export PATH=3/envs/samtools/bin:PATH: PATH=3/envs/samtools/bin:3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin;
+ which samtools
3/envs/samtools/bin/samtools

Answer 1

Automatic Activation
The documentation also gives examples of automatic activation. Specifically, Example #3 demonstrates this with a couple key components:
As noted in the IMPORTANT section of the documentation and in the other answer, the shell needs to use login mode, which can be set globally within a job, using
jobs:
example-3:
defaults:
run:
shell: bash -l {0}
An environment definition is provided to setup-miniconda GHA. In the example it's etc/example-environment.yml, which defines the environment anaconda-client-env and this is set to activate using the activate-environment argument.
steps:
- uses: conda-incubator/setup-miniconda#v2
with:
activate-environment: anaconda-client-env
environment-file: etc/example-environment.yml
auto-activate-base: false
- run: |
conda info
conda list
Samtools Example
I have a repository where I test environment definitions, so here's an explicit example for samtools. Note that I prefer Mamba, and also recommend capturing an explicit mamba env export to document the environment.
envs/so-samtools.yaml
name: so-samtools
channels:
- conda-forge
- bioconda
- defaults
dependencies:
- samtools
.github/workflows/so-samtools.yaml
name: so-samtools
on:
push:
paths:
- 'envs/so-samtools.yaml'
- '.github/workflows/so-samtools.yaml'
jobs:
create-env:
name: ${{ matrix.os }}
runs-on: ${{ matrix.os }}
defaults:
run:
shell: bash -l {0}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest]
steps:
- name: checkout repository
uses: actions/checkout#v2
- name: create environment with mamba
uses: conda-incubator/setup-miniconda#v2
with:
mamba-version: "*"
channels: conda-forge,bioconda,defaults
auto-activate-base: false
activate-environment: so-samtools
environment-file: envs/so-samtools.yaml
- name: check solution
run: |
mamba env export
- name: test samtools
run: |
which samtools
samtools help

Answer 2

Add this to not ignore bash profile files (fix from: https://github.com/marketplace/actions/setup-miniconda ):
defaults:
run:
shell: bash -l {0}
Complete GitHub Action yml file, .github/workflows/conda.yml:
name: Conda
on: [push]
jobs:
# label of the job
tests:
name: Tests
# containers must run in Linux based operating systems
runs-on: ubuntu-latest
# Do not ignore bash profile files. From:
# https://github.com/marketplace/actions/setup-miniconda
defaults:
run:
shell: bash -l {0}
# Docker Hub image that `postgres-job` executes in
container: node:latest
# service containers to run with `postgres-job`
steps:
- uses: conda-incubator/setup-miniconda#v2
with:
miniconda-version: "latest"
channels: bioconda, conda-forge, defaults
use-only-tar-bz2: true # IMPORTANT: This needs to be set for caching to work properly!
auto-update-conda: true
auto-activate-base: true
- name: Install samtools
run: |
echo "begin: PATH=$PATH;"
conda create -y --name samtools samtools
conda activate samtools || true
echo "after conda activate samtools: PATH=$PATH;"
which samtools || true
# Use samtools here.
Output:
begin: PATH=/__w/ngs-aggregate_results/ngs-aggregate_results/3/envs/test/bin:/__w/ngs-aggregate_results/ngs-aggregate_results/3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin;
...
after conda activate samtools: PATH=/__w/ngs-aggregate_results/ngs-aggregate_results/3/envs/samtools/bin:/__w/ngs-aggregate_results/ngs-aggregate_results/3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin;
/__w/ngs-aggregate_results/ngs-aggregate_results/3/envs/samtools/bin/samtools