I'm writing some unittests for a project, and the following works:
> conda activate env_name
(env_name) > python -m unittest
But this gives me a ModuleNotFoundError:
> conda run -n env_name python -m unittest
The module that is failing is called within my project, like this:
import snowflake.connector
def add(a,b):
return a+b
It may be worth noting that if I change the import to numpy, I don't get this issue. One possibility may be that in the site-packages for my env, numpy is just named "numpy", but snowflake.connector is named "snowlake_connector_python-2.7.4.dist-info"
(base) PS C:\Users\<username>\Desktop> conda info --envs
# conda environments:
#
base * C:\Users\<username>\Anaconda3
proj_1 C:\Users\<username>\Anaconda3\envs\proj_1
proj_2 C:\Users\<username>\Anaconda3\envs\proj_2
C:\Users\<username>\anaconda3
proj_2 C:\Users\<username>\anaconda3\envs\proj_2
This is on Windows 10. There's also an old environment present in the C:\Users\<username>\anaconda3\envs directory that I deleted using conda remove -n proj_0 --all in the past. C:\Users\<username>\.conda\environments.txt file contents:
C:\Users\<username>\anaconda3
C:\Users\<username>\anaconda3\envs\proj_1
C:\Users\<username>\Anaconda3
C:\Users\<username>\Anaconda3\envs\proj_2
What's up with all the casing and repeat environments? I don't remember changing anything manually. I've just tried removing the old environment dir (proj_0) manually, but got an error saying the folder is open in another program (I'm guessing conda).
Edit: conda config --show envs_dirs output:
envs_dirs:
- C:\Users\<username>\Anaconda3\envs
- C:\Users\<username>\.conda\envs
- C:\Users\<username>\AppData\Local\conda\conda\envs
When I try to run the experiment defined in this notebook in notebook, I encountered an error when it is creating the conda env. The error occurs when the below cell is executed:
from azureml.core import Experiment, ScriptRunConfig, Environment
from azureml.core.conda_dependencies import CondaDependencies
from azureml.widgets import RunDetails
# Create a Python environment for the experiment
sklearn_env = Environment("sklearn-env")
# Ensure the required packages are installed (we need scikit-learn, Azure ML defaults, and Azure ML dataprep)
packages = CondaDependencies.create(conda_packages=['scikit-learn','pip'],
pip_packages=['azureml-defaults','azureml-dataprep[pandas]'])
sklearn_env.python.conda_dependencies = packages
# Get the training dataset
diabetes_ds = ws.datasets.get("diabetes dataset")
# Create a script config
script_config = ScriptRunConfig(source_directory=experiment_folder,
script='diabetes_training.py',
arguments = ['--regularization', 0.1, # Regularizaton rate parameter
'--input-data', diabetes_ds.as_named_input('training_data')], # Reference to dataset
environment=sklearn_env)
# submit the experiment
experiment_name = 'mslearn-train-diabetes'
experiment = Experiment(workspace=ws, name=experiment_name)
run = experiment.submit(config=script_config)
RunDetails(run).show()
run.wait_for_completion()
Everytime I run this, I always faced the issue of creating the conda env as below:
Creating conda environment...
Running: ['conda', 'env', 'create', '-p', '/home/azureuser/.azureml/envs/azureml_000000000000', '-f', 'azureml-environment-setup/mutated_conda_dependencies.yml']
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Installing pip dependencies: ...working...
Attempting to clean up partially built conda environment: /home/azureuser/.azureml/envs/azureml_000000000000
Remove all packages in environment /home/azureuser/.azureml/envs/azureml_000000000000:
Creating conda environment failed with exit code: -15
I could not find anything useful on the internet and this is not the only script where it fail. When I am try to run other experiments I have sometimes faced this issue. One solution which worked in the above case is I moved the pandas from pip to conda and it was able to create the coonda env. Example below:
# Ensure the required packages are installed (we need scikit-learn, Azure ML defaults, and Azure ML dataprep)
packages = CondaDependencies.create(conda_packages=['scikit-learn','pip'],
pip_packages=['azureml-defaults','azureml-dataprep[pandas]'])
# Ensure the required packages are installed (we need scikit-learn, Azure ML defaults, and Azure ML dataprep)
packages = CondaDependencies.create(conda_packages=['scikit-learn','pip','pandas'],
pip_packages=['azureml-defaults','azureml-dataprep'])
The error message (or the logs from Azure) is also not much help. Would apprecite if a proper solution is available.
Edit: I have recently started learning to use Azure for Machine learning and so if I am not sure if I am missing something? I assume the example notebooks should work as is hence raised this question.
short answer
Totally been in your shoes before. This code sample seems a smidge out of date. Using this notebook as a reference, can you try the following?
packages = CondaDependencies.create(
pip_packages=['azureml-defaults','scikit-learn']
)
longer answer
Using pip with Conda is not always smooth sailing. In this instance, conda isn't reporting up the issue that pip is having. The solution is to create and test this environment locally where we can get more information, which will at least will give you a more informative error message.
Install anaconda or miniconda (or use an Azure ML Compute Instance which has conda pre-installed)
Make a file called environment.yml that looks like this
name: aml_env
dependencies:
- python=3.8
- pip=21.0.1
- pip:
- azureml-defaults
- azureml-dataprep[pandas]
- scikit-learn==0.24.1
Create this environment with the command conda env create -f environment.yml.
respond to any discovered error message
If there' no error, use this new environment.yml with Azure ML like so
sklearn_env = Environment.from_conda_specification(name = 'sklearn-env', file_path = './environment.yml')
more context
the error I'm guessing that's happening is when you reference a pip requirements file from a conda environment file. In this scenario, conda calls pip install -r requirements.txt and if that command errors out, conda can't report the error.
requirements.txt
scikit-learn==0.24.1
azureml-dataprep[pandas]
environment.yml
name: aml_env
dependencies:
- python=3.8
- pip=21.0.1
- pip:
- -rrequirements.txt
What worked for me looking at the previous notebook 05 - Train Models.ipynb:
packages = CondaDependencies.create(conda_packages=['pip', 'scikit-learn'],
pip_packages=['azureml-defaults'])
You have to:
Remove 'azureml-dataprep[pandas]' from pip_packages
Change the order of conda_packages - pip should go first
I have installed pyjnius with conda. However, when I try to import pyjnius it fails
> from jnius import autoclass
File
"C:\Users\OEM\Miniconda3\envs\example-env\lib\site-packages\jnius\__init__.py",
line 12, in <module>
from .jnius import * # noqa ImportError: DLL load failed: The specified module could not be found.
Together with pyjnius conda installs also openjdk. Next, pyjnius looks for jvm.dll in one of PATH directories. DLL could be found in
C:\Users\OEM\Miniconda3\pkgs\openjdk-11.0.1-1017\Library\bin\server
but conda does not include it in PATH. It adds another folder in PATH:
C:\Users\OEM\Miniconda3\envs\example2-env\Library\jre\bin\server
while this directory is missing: JRE has not been installed, only JDK. I can, obviously, include first directory in my PATH, however, this would bypass conda virtual environments concept. How can I solve this problem in an elegant way?
Here's environment.yml to reproduce the problem:
name: example-env
channels:
- conda-forge
dependencies:
- python=3.7
- Cython
- pyjnius
Next, I create and activate as follows:
conda env update --file environment.yml
conda activate example-env
I got Python 2.7.8 installed on my Win7 machine, which comes with pip already pre-installed. I'm successfully able to install new packages from pip and now I need to add custom repository url to the install list of pip
To do so I need to modify pip.ini which is in %APPDATA%\pip\pip.ini according to the Official Manual
However there are no pip folder anywhere (not in Roaming, not in Local, not in LocalLow)
nor there exists PyPa folder in: C:\ProgramData\PyPA\pip\pip.conf
Could you tell me where do i search for pip.ini? how to add foreign repo to the install list?
Instead of checking a list of well-known locations, you can ask pip to list the valid locations:
pip config -v list
Fun fact
On the same machine, with the same pip version, the valid locations can vary based on the actual Python version.
Environment: Win 7 x64, the HOME environment variable is set to D:\Home
Python 3.7.3:
> pip config -v list
For variant 'global', will try loading 'C:\ProgramData\pip\pip.ini'
For variant 'user', will try loading 'D:\Home\pip\pip.ini'
For variant 'user', will try loading 'C:\Users\foobar\AppData\Roaming\pip\pip.ini'
For variant 'site', will try loading 'C:\Python37\pip.ini'
Python 3.8.0:
> pip config -v list
For variant 'global', will try loading 'C:\ProgramData\pip\pip.ini'
For variant 'user', will try loading 'C:\Users\foobar\pip\pip.ini'
For variant 'user', will try loading 'C:\Users\foobar\AppData\Roaming\pip\pip.ini'
For variant 'site', will try loading 'C:\Python38\pip.ini'
Finally got it sorted.
Apparently for Windows users pip.ini config file is not created, however can be added manually!
just create new %APPDATA%\pip\pip.ini and content of custom repository:
[install]
find-links = https://<login>:<password>#your.repo.com/custom/url
Reference: https://pip.pypa.io/en/stable/user_guide/#config-file
A bit late, but for reference:
Try adding the pip.ini file in %USERPROFILE%\pip\pip.ini (usually: C:\Users\<username>\pip\pip.ini).
On windows pip.exe looks for "pip.ini" in this order:
C:\ProgramData\pip\pip.ini
C:\Users\<username>\pip\pip.ini
C:\Users\<username>\AppData\Roaming\pip\pip.ini
It's been 7 years, and I think there's now a better answer for most people -- but it does depend on version of pip. For the most recent pips I'm using:
$ pip config -v debug
lists where it's looking and you can decide which location is
most useful for what you've got in mind. It does look like a fairly recent
change: On a year-old docker image I had with pip 20.1
I got "ERROR: Need an action (edit, get, list, set, unset) to
perform." On that system, pip config -v list gave a list of files it would try, this is supposed to be 'global', 'user' or 'site' variants of pip.ini locations.
For Windows 10, for pip 21.2.4 on both 3.9.6 and 3.6.8, I get response below with pip config -v debug, while pip config -v list is silent (unless a pip.ini is found).
global:
C:\ProgramData\pip\pip.ini, exists: False
site:
c:\py\myvenv\pip.ini, exists: False
user:
C:\Users\myname\pip\pip.ini, exists: False
C:\Users\myname\AppData\Roaming\pip\pip.ini, exists: False
From a downloaded image I got from dockerhub in June 2021 with pip 21.2.2 and python 3.6.10:
pip config -v debug
env_var:
env:
global:
/etc/xdg/pip/pip.conf, exists: False
/etc/pip.conf, exists: True
global.extra-index-url: http://trynexs:8081/repository/repo_group/simple
site:
/usr/local/pip.conf, exists: False
user:
/home/tanhauser/.pip/pip.conf, exists: False
/home/tanhauser/.config/pip/pip.conf, exists: False
Pip changed the location of the config file in windows starting in pip 6.0 the pip config docs explain the location of the config files as follows.
pip --version >= 6 (as of version 18.1 hasn't changed again yet)
%APPDATA%\pip\pip.ini
pip --version < 6
%HOME%\pip\pip.ini
Inside a virtual env
%VIRTUAL_ENV%\pip.ini
Site-wide win7+ (same as of win10)
C:\ProgramData\pip\pip.ini
Site-wide winxp (note windows vista side wide not supported)
C:\Documents and Settings\All Users\Application Data\pip\pip.ini
NOTE: If multiple configuration files are found by pip then they are combined in the following order:
The site-wide file is read
The per-user file is read
The virtualenv-specific file is read
Also pip added a config command starting in pip 10.
pip config --help
I know this is a bit late, however, this post is high on the rankings when searching. Inside a virtual environment pip.ini can also be in the root of the virtual environment. From the docs and https://pip.pypa.io/en/stable/user_guide/?highlight=pip.ini#configuration
Inside a virtualenv:
On Unix and macOS the file is $VIRTUAL_ENV/pip.conf
On Windows the file is: %VIRTUAL_ENV%\pip.ini
All the answers are partially wrong and right.
It depends on how your system is configured. The only way (for me) to find out was to patch site-packages/pip/locations.py at the point where site_config_files is assigned (around line 120 for pip 9.0.1)
print('########## ' + str(site_config_files))
and then run pip search foo
On my system it printed ########## ['C:\\ProgramData\\pip\\pip.ini'], of which location I assumed I could not create/edit. But it just worked.
Btw, for my system %APPDATA% points to C:\Users\MYUSER\AppData\Roaming, which is not looked at when running pip on my system.
Rather than guessing first check if you have any default global/local config which is read by pip with the below command:
pip config list
This will give all details of the default config loaded by python.
If the above command doesn't give any output please try to find where pip tries to find for the global config file with the below command:
pip config --editor <path to editor of your choice> edit
The above command will open the config file which pip reads by default or else it will give an error saying that the file doesn't exist.
If there's an error please go ahead and create the exact directory and file structure as show in the error. Once the file has been created please make your changes e.g.
[global]
cert = /path/to/base64/ssl/certificate.pem
proxy = http://username:password#ipaddress:port
Save the file and please try to check (the above mentioned check command) if the configs are loaded by pip or not.
For more info please follow pip config documentation
Make sure you acually have a pip.ini file, not pip.ini.txt.
For me (Windows 8, pip 9.0.1, python 3.5.3), the correct path was
c:\Users\<UserName>\.pypirc <- sic!, even on windows
Windows 10:
I had to create 'pip' directory inside
C:\Users\UserName\AppData\Roaming\
then create pip.ini file inside that 'pip' directory:
C:\Users\<username>\AppData\Roaming\pip\pip.ini
No other location worked for me.
For Windows, python will load the config from path below. So, if pip.ini file is not exist in these paths you can create the new file by refer these path depend on environment scopes (global, user & site) that you need python execute.
For variant 'global', will try loading 'C:\ProgramData\pip\pip.ini'
For variant 'user', will try loading 'C:\Users\MyName\pip\pip.ini'
For variant 'user', will try loading 'C:\Users\MyName\AppData\Roaming\pip\pip.ini'
For variant 'site', will try loading 'c:\python39-32\pip.ini'
By the way, you can check the paths as above by
pip config -v list
On a Windows 10 machine with multiple users I used this:
c:\users\all users\pip\pip.ini
Using pip version 22.3.1 with python version 3.10.4..