Conda create from requirements.txt not finding packages - anaconda

Question

What I try
conda create --name ml --file ./requirements.txt
I created the requirements.txt file with conda list -e > requirements.txt on another computer in the past.
requirements.txt:
https://github.com/penguinsAreFunny/bugFinder-machineLearning/blob/master/requirements.txt
Error
PackagesNotFoundError: The following packages are not available from current channels:
protobuf==3.19.1=pypi_0
tensorboard-data-server==0.6.1=pypi_0
pygments==2.10.0=pypi_0
scikit-learn==1.0.1=pypi_0
tensorflow-estimator==2.4.0=pypi_0
flake8==4.0.1=pypi_0
nest-asyncio==1.5.1=pypi_0
[...]
Current channels:
https://conda.anaconda.org/conda-forge/win-64
https://conda.anaconda.org/conda-forge/noarch
https://repo.anaconda.com/pkgs/main/win-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/win-64
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/msys2/win-64
https://repo.anaconda.com/pkgs/msys2/noarch
https://conda.anaconda.org/pickle/win-64
https://conda.anaconda.org/pickle/noarch
https://conda.anaconda.org/nltk/win-64
https://conda.anaconda.org/nltk/noarch
Question
Why can´t conda find the packages in the channels?
I think the missing packages should be in conda-forge, shouldn´t they?
Used Version
conda 4.11.0

Answer 1

Issue: PyPI not compatible
The packages likely are in Conda Forge as suggested, but the build strings, "pypi_0", indicate that they had been installed from PyPI in the previous environment. The conda list -e command captures this info, but the conda create command cannot handle it.
Workarounds
Option 1: Source from Conda
The quickest fix is probably to edit the file to remove the build string specification on those packages. That is, something like:
## remove all PyPI references
sed -e 's/=pypi_0//' requirements.txt > reqs.nopip.txt
## try creating only from Conda packages
conda create -n m1 --file reqs.nopip.txt
Conda will then try to treat these PyPI package specifications as Conda packages. However, this is not always reliable, since some packages go by different names in the two repositories.
Option 2: Export YAML
Alternatively, serializing to YAML can handle both capturing and reinstalling Pip-installed packages. So, if you still have the old environment around, consider using:
conda env export > environment.yaml
which can recreate (on the same platform) with
conda env create -n m1 -f environment.yaml
Option 3: Convert requirements.txt to YAML
If the environment is no longer around, or the requirements.txt was provided by another user, then another option is to convert the file to a YAML format. Here is an AWK script for doing this:
list_export_to_yaml.awk
#!/usr/bin/env awk -f
#' Author: Mervin Fansler
#' GitHub: #mfansler
#' License: MIT
#'
#' Basic usage
#' $ conda list --export | awk -f list_export_to_yaml.awk
#'
#' Omitting builds with 'no_builds'
#' $ conda list --export | awk -v no_builds=1 -f list_export_to_yaml.awk
#'
#' Specifying channels with 'channels'
#' $ conda list --export | awk -v channels="conda-forge,defaults" -f list_export_to_yaml.awk
BEGIN {
FS="=";
if (channels) split(channels, channels_arr, ",");
else channels_arr[0]="defaults";
}
{
# skip header
if ($1 ~ /^#/) next;
if ($3 ~ /pypi/) { # pypi packages
pip=1;
pypi[i++]=" - "$1"=="$2" ";
} else { # conda packages
if ($1 ~ /pip/) pip=1;
else { # should we keep builds?
if (no_builds) conda[j++]=" - "$1"="$2" ";
else conda[j++]=" - "$1"="$2"="$3" ";
}
}
}
END {
# emit channel info
print "channels: ";
for (k in channels_arr) print " - "channels_arr[k]" ";
# emit conda pkg info
print "dependencies: ";
for (j in conda) print conda[j];
# emit PyPI pkg info
if (pip) print " - pip ";
if (length(pypi) > 0) {
print " - pip: ";
for (i in pypi) print pypi[i];
}
}
For OP's example, we get:
$ wget -O requirements.txt 'https://github.com/penguinsAreFunny/bugFinder-machineLearning/raw/master/requirements.txt'
$ awk -f list_export_to_yaml.awk requirements.txt > bugfinder-ml.yaml
which then has the contents:
channels:
- defaults
dependencies:
- brotlipy=0.7.0=py38h294d835_1003
- ca-certificates=2021.10.8=h5b45459_0
- cffi=1.15.0=py38hd8c33c5_0
- chardet=4.0.0=py38haa244fe_2
- cryptography=35.0.0=py38hb7941b4_2
- future=0.18.2=py38haa244fe_4
- h2o=3.34.0.3=py38_0
- openjdk=11.0.9.1=h57928b3_1
- openssl=1.1.1l=h8ffe710_0
- pycparser=2.20=pyh9f0ad1d_2
- pyopenssl=21.0.0=pyhd8ed1ab_0
- pysocks=1.7.1=py38haa244fe_4
- python=3.8.12=h7840368_2_cpython
- python_abi=3.8=2_cp38
- requests=2.26.0=pyhd8ed1ab_0
- setuptools=58.5.3=py38haa244fe_0
- sqlite=3.36.0=h8ffe710_2
- tabulate=0.8.9=pyhd8ed1ab_0
- ucrt=10.0.20348.0=h57928b3_0
- urllib3=1.26.7=pyhd8ed1ab_0
- vc=14.2=hb210afc_5
- vs2013_runtime=12.0.21005=1
- vs2015_runtime=14.29.30037=h902a5da_5
- wheel=0.37.0=pyhd8ed1ab_1
- win_inet_pton=1.1.0=py38haa244fe_3
- pip
- pip:
- absl-py==0.15.0
- appdirs==1.4.4
- astroid==2.7.3
- astunparse==1.6.3
- autopep8==1.6.0
- backcall==0.2.0
- backports-entry-points-selectable==1.1.0
- black==21.4b0
- cachetools==4.2.4
- certifi==2021.10.8
- cfgv==3.3.1
- charset-normalizer==2.0.7
- click==8.0.3
- cycler==0.11.0
- deap==1.3.1
- debugpy==1.5.1
- decorator==5.1.0
- dill==0.3.4
- distlib==0.3.3
- entrypoints==0.3
- filelock==3.3.2
- flake8==4.0.1
- flatbuffers==1.12
- gast==0.3.3
- google-auth==2.3.3
- google-auth-oauthlib==0.4.6
- google-pasta==0.2.0
- grpcio==1.32.0
- h5py==2.10.0
- identify==2.3.3
- idna==3.3
- importlib-resources==5.4.0
- ipykernel==6.5.0
- ipython==7.29.0
- isort==5.10.0
- jedi==0.18.0
- jinja2==3.0.2
- joblib==1.1.0
- jupyter-client==7.0.6
- jupyter-core==4.9.1
- keras-preprocessing==1.1.2
- kiwisolver==1.3.2
- markdown==3.3.4
- markupsafe==2.0.1
- matplotlib==3.4.3
- matplotlib-inline==0.1.3
- mypy==0.910
- mypy-extensions==0.4.3
- nest-asyncio==1.5.1
- nodeenv==1.6.0
- numpy==1.19.5
- oauthlib==3.1.1
- opt-einsum==3.3.0
- pandas==1.3.4
- parso==0.8.2
- pathspec==0.9.0
- pickleshare==0.7.5
- pillow==8.4.0
- platformdirs==2.4.0
- pre-commit==2.15.0
- prompt-toolkit==3.0.22
- protobuf==3.19.1
- pyasn1==0.4.8
- pyasn1-modules==0.2.8
- pycodestyle==2.8.0
- pyflakes==2.4.0
- pygments==2.10.0
- pylint==2.10.2
- pyparsing==3.0.4
- python-dateutil==2.8.2
- pytz==2021.3
- pywin32==302
- pyyaml==6.0
- pyzmq==22.3.0
- regex==2021.11.2
- requests-oauthlib==1.3.0
- rsa==4.7.2
- scikit-learn==1.0.1
- scipy==1.7.1
- six==1.15.0
- stopit==1.1.2
- sweetviz==2.1.3
- tensorboard==2.7.0
- tensorboard-data-server==0.6.1
- tensorboard-plugin-wit==1.8.0
- tensorflow==2.4.4
- tensorflow-estimator==2.4.0
- termcolor==1.1.0
- threadpoolctl==3.0.0
- tornado==6.1
- tpot==0.11.7
- tqdm==4.62.3
- traitlets==5.1.1
- typing-extensions==3.7.4.3
- update-checker==0.18.0
- virtualenv==20.10.0
- wcwidth==0.2.5
- werkzeug==2.0.2
- xgboost==1.5.0
- zipp==3.6.0
Note that since conda list --export does not capture channel information, the user must determine this on their own. By default the script inserts a defaults, but also provides an argument (channels) to specify additional channels for the YAML in a comma-separate format. E.g.
awk -f list_export_to_yaml.awk -v channels='conda-forge,defaults' requirements.txt
would output
channels:
- conda-forge
- defaults
in the YAML.
There is also a no_builds argument to suppress builds (i.e., versions only). E.g.,
awk -f list_export_to_yaml.awk -v no_builds=1 requirements.txt