I have a script that is making a conda virtual environment where functions need to be imported from folders outside of the repo. I have tried two approaches to add the folders to the path:
Conda develop:
os.system(conda run -n test_env conda develop ' + PACKAGE_PATH)
Results in:
Permission denied [PACKAGE_PATH]
Create conda.pth manually
path = os.path.join(env_path, 'lib')
if not os.path.exists(path):
os.mkdir(path)
path = os.path.join(path, 'site-packages')
if not os.path.exists(path):
os.mkdir(path)
env_path_file = os.path.join(path, 'conda.pth')
with open(env_path_file, 'w+') as f:
write_line(f, PACKAGE_PATH)
The file is created successfully. But, setting the conda interpreter from test_env (in Pycharm) and doing import sys; sys.path does not show [PACKAGE_PATH] and functions cannot be imported from its folders.
Any suggestions?
Thank you.
This is resolved by using conda develop from outside the environment:
conda develop [package_path]
not
conda run -n conda develop [package_path]
I'm creating an environment in an Azure DevOps pipeline from a .yml file. However, one of my modules has dependency issues, causing conda env create -n env-name --file conda.yml to get stuck. I know that I need to use --use-deprecated=legacy-resolver but since I'm creating the environment from a YAML file I don't know how to specify it in my YAML file (rather than directly running pip install).
channels:
- conda-forge
- nodefaults
dependencies:
- python=3.9.12
- pip>=19.0
- pip:
- numpy==1.22.0
- pandavro
- scikit-learn
- ipykernel
- pyspark
- mlflow
- mltable
I've tried adding [--use-deprecated=legacy-resolver] after one of my modules (e.g. pandavro [--use-deprecated=legacy-resolver]) but it seems like Conda doesn't recognize this syntax.
It seems like this feature hasn't yet been implemented, based on these posts:
https://github.com/conda/conda/issues/3763
https://github.com/conda/conda/issues/6805
The workaround that worked for me was removing all pip dependencies from my conda.yml file (i.e. all lines beginning with and including -pip:) and instead putting them in a separate requirements.txt file. I then added another step to my ADO pipeline to run pip install -r requirements.txt --use-deprecated=legacy-resolver.
I'm struggling to figure out conda virtual environments on windows. All I want is to be able to have different versions of h2o installed at the same time because of their insane decision to not allow you to be able to load files saved in even the most minor different version.
I created a virtual environment by cloning my base anaconda:
conda create -n h203_14_0_7 --clone base
I then activated the virtual environment like so:
C:\ProgramData\Anaconda3\Scripts\activate h203_14_0_7
Now that I'm in the virtual environment (I see the (h203_14_0_7) at the beginning of the prompt), i want to uninstall the version of h2o in this virtual environment so I tried:
pip uninstall h2o
But this output
which to me looks like it's going to uninstall the global h2o rather than the virtual environment h2o. So I think it's using the global pip instead of the pip it should have cloned off the base. So how to I use the virtual environment pip to uninstall h2o just for my virtual environment and how can I be sure that it's doing the right thing?
I then ran
conda intall pip
and it seems that after that I was able to use pip to uninstall h2o only from the virtual environment (I hope). I then downloaded the older h2o version from here: https://github.com/h2oai/h2o-3/releases/tag/jenkins-rel-weierstrass-7
but when I try install it I get
(h203_14_0_7) C:\ProgramData\Anaconda3\envs\h203_14_0_7>pip install C:\Users\dan25\Downloads\h2o-3-jenkins-rel-weierstrass-7.tar.gz
Processing c:\users\dan25\downloads\h2o-3-jenkins-rel-weierstrass-7.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\ProgramData\Anaconda3\envs\h203_14_0_7\lib\tokenize.py", line 452, in open
buffer = _builtin_open(filename, 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\dan25\\AppData\\Local\\Temp\\pip-sf7r_6pm-build\\setup.py'
So what now?
I had trouble (e.g. https://0xdata.atlassian.net/browse/PUBDEV-3370 ) getting that approach to ever work. It felt like some kind of global dependency was in there, somewhere.
So, I personally just uninstall, and install the desired version, as I need to move between versions. (Actually, I am more likely to use a different VirtualBox or AWS image for each.)
However I noticed searching for conda on the H2O jira that there is a lot of activity recently. They might all be pointing out the same bug you have found, but if so it sounds like it is something getting enough attention to get fixed.
Aside: finding old versions (and your edit showing install problems)
To find, e.g. 3.14.0.7, google it with "h2o". The top hit is http://h2o-release.s3.amazonaws.com/h2o/rel-weierstrass/7/index.html
The "rel-weierstrass" represents 3.14.0, and the 7 is in the URL. (I've yet to see a full list of all the rel-XXX names, but google will always find at least one in the series, even if it won't find the exact minor version.)
Download the zip file you find there. Inside you will find both an R package, and a whl package for Python. So unzip it, extract the one you want, then pip install it.
These zip files are always on S3 (AFAIK). The link you showed was a source snapshot, on github.
Install requirements:
pip install requests tabulate numpy scikit-learn
Extract the archive:
zcat h2o-3-jenkins-rel-weierstrass-7.tar.gz | tar xvf -
cd into Python directory and build:
cd h2o-py
../gradlew build
I have this working now. I think the trick is to make sure you do NOT have h2o installed on your base python. I did the following:
pip uninstall h2o
conda create --name h2o-base pip
conda activate h2o-base
conda install numpy
conda install pandas
conda install requests
conda install tabulate
conda install colorama
conda install future
conda install jupyter
python -m pip install ipykernel
conda deactivate
And now to install specific versions of h2o, you need to URL of the .whl file for that version and you can find a list of the URLs of all the old versions here: https://github.com/h2oai/h2o-3/blob/master/Changes.md
So for example to install version 3.18.0.8:
conda create --name h2o-3-18-0-8 --clone h2o-base
conda activate h2o-3-18-0-8
pip install http://h2o-release.s3.amazonaws.com/h2o/rel-wolpert/8/Python/h2o-3.18.0.8-py2.py3-none-any.whl
python -m ipykernel install --user --name h2o-3-18-0-8 --display-name "Python (h2o-3-18-0-8)"
or version 3.20.0.2 (make sure to conda deactivate first):
conda create --name h2o-3-20-0-2 --clone h2o-base
conda activate h2o-3-20-0-2
pip install http://h2o-release.s3.amazonaws.com/h2o/rel-wright/2/Python/h2o-3.20.0.2-py2.py3-none-any.whl
python -m ipykernel install --user --name h2o-3-20-0-2 --display-name "Python (h2o-3-20-0-2)"
This set-up allows me to have multiple versions of h2o installed on the same computer and if I have to use serialized models I just have to run python from the virtual environment with the correct version of h2o installed. I think this is preferable to uninstalling and reinstalling h2o each time.
Here is the environments.yml file if you want to skip all the manual installs above:
name: h2o-base
channels:
- conda-forge
- defaults
dependencies:
- asn1crypto=0.24.0=py37_1003
- backcall=0.1.0=py_0
- bleach=3.0.2=py_0
- ca-certificates=2018.10.15=ha4d7672_0
- certifi=2018.10.15=py37_1000
- cffi=1.11.5=py37hfa6e2cd_1001
- chardet=3.0.4=py37_1003
- colorama=0.4.0=py_0
- cryptography=2.3=py37h74b6da3_0
- cryptography-vectors=2.3.1=py37_1000
- decorator=4.3.0=py_0
- entrypoints=0.2.3=py37_1002
- future=0.16.0=py37_1002
- icu=58.2=vc14_0
- idna=2.7=py37_1002
- ipykernel=5.1.0=pyh24bf2e0_0
- ipython=7.0.1=py37h39e3cac_1000
- ipython_genutils=0.2.0=py_1
- ipywidgets=7.4.2=py_0
- jedi=0.13.1=py37_1000
- jinja2=2.10=py_1
- jpeg=9b=vc14_2
- jsonschema=2.6.0=py37_1002
- jupyter=1.0.0=py_1
- jupyter_client=5.2.3=py_1
- jupyter_console=6.0.0=py_0
- jupyter_core=4.4.0=py_0
- libflang=5.0.0=vc14_20180208
- libpng=1.6.34=vc14_0
- libsodium=1.0.16=vc14_0
- llvm-meta=5.0.0=0
- markupsafe=1.0=py37hfa6e2cd_1001
- mistune=0.8.4=py37hfa6e2cd_1000
- nbconvert=5.3.1=py_1
- nbformat=4.4.0=py_1
- notebook=5.7.0=py37_1000
- openblas=0.2.20=vc14_8
- openmp=5.0.0=vc14_1
- openssl=1.0.2p=hfa6e2cd_1001
- pandas=0.23.4=py37h830ac7b_1000
- pandoc=2.3.1=0
- pandocfilters=1.4.2=py_1
- parso=0.3.1=py_0
- pickleshare=0.7.5=py37_1000
- pip=18.1=py37_1000
- prometheus_client=0.4.2=py_0
- prompt_toolkit=2.0.6=py_0
- pycparser=2.19=py_0
- pygments=2.2.0=py_1
- pyopenssl=18.0.0=py37_1000
- pyqt=5.6.0=py37h764d66f_7
- pysocks=1.6.8=py37_1002
- python=3.7.0=hc182675_1005
- python-dateutil=2.7.3=py_0
- pytz=2018.5=py_0
- pywinpty=0.5.4=py37_1002
- pyzmq=17.1.2=py37hf576995_1001
- qt=5.6.2=vc14_1
- qtconsole=4.4.2=py_1
- requests=2.19.1=py37_1001
- send2trash=1.5.0=py_0
- setuptools=40.4.3=py37_0
- simplegeneric=0.8.1=py_1
- sip=4.18.1=py37h6538335_0
- six=1.11.0=py37_1001
- tabulate=0.8.2=py_0
- terminado=0.8.1=py37_1001
- testpath=0.4.2=py37_1000
- tornado=5.1.1=py37hfa6e2cd_1000
- traitlets=4.3.2=py37_1000
- urllib3=1.23=py37_1001
- vc=14=0
- vs2015_runtime=14.0.25420=0
- wcwidth=0.1.7=py_1
- webencodings=0.5.1=py_1
- wheel=0.32.1=py37_0
- widgetsnbextension=3.4.2=py37_1000
- win_inet_pton=1.0.1=py37_1002
- wincertstore=0.2=py37_1002
- winpty=0.4.3=4
- zeromq=4.2.5=vc14_2
- zlib=1.2.11=vc14_0
- blas=1.0=mkl
- icc_rt=2017.0.4=h97af966_0
- intel-openmp=2019.0=118
- m2w64-gcc-libgfortran=5.3.0=6
- m2w64-gcc-libs=5.3.0=7
- m2w64-gcc-libs-core=5.3.0=7
- m2w64-gmp=6.1.0=2
- m2w64-libwinpthread-git=5.0.0.4634.697f757=2
- mkl=2019.0=118
- mkl_fft=1.0.6=py37hdbbee80_0
- mkl_random=1.0.1=py37h77b88f5_1
- msys2-conda-epoch=20160418=1
- numpy=1.15.2=py37ha559c80_0
- numpy-base=1.15.2=py37h8128ebf_0
Windows 10, Anaconda 2 or 3
c:\leo.repo\leo-editor>c:\apps\Git\bin\git.exe --version
yields:
git version 2.8.2.windows.1
sys.path contains C:\apps\Git, C:\apps\Git\bin and C:\apps\Git\cmd, but I am getting the dreaded GitCommandNotFound exception. What am I doing wrong?
Executing:
from git import Repo
yields:
[snip]
File "c:\Anaconda3\lib\site-packages\git\cmd.py", line 602, in execute
raise GitCommandNotFound(command, err)
git.exc.GitCommandNotFound: Cmd('git') not found due to: FileNotFoundError('[WinError 2] The system cannot find the file specified')
cmdline: git version
Edward
It appears that installs are different on Anaconda.
Per this page the proper installation procedure is:
conda install -c conda-forge gitpython=2.1.1
Installation on Python 2 was straightforward.
Update: I had to add the following to sys.path for Python 3:
C:\Anaconda3\pkgs\gitdb-0.6.4-py35_1\Lib\site-packages
I have no idea why python 2 installs in site-packages as expected, while python 3 installs in the path shown above.
Update2: I also had to update the Windows PATH variable (not just sys.path) to point to the directory containing git.exe.
I got Python 2.7.8 installed on my Win7 machine, which comes with pip already pre-installed. I'm successfully able to install new packages from pip and now I need to add custom repository url to the install list of pip
To do so I need to modify pip.ini which is in %APPDATA%\pip\pip.ini according to the Official Manual
However there are no pip folder anywhere (not in Roaming, not in Local, not in LocalLow)
nor there exists PyPa folder in: C:\ProgramData\PyPA\pip\pip.conf
Could you tell me where do i search for pip.ini? how to add foreign repo to the install list?
Instead of checking a list of well-known locations, you can ask pip to list the valid locations:
pip config -v list
Fun fact
On the same machine, with the same pip version, the valid locations can vary based on the actual Python version.
Environment: Win 7 x64, the HOME environment variable is set to D:\Home
Python 3.7.3:
> pip config -v list
For variant 'global', will try loading 'C:\ProgramData\pip\pip.ini'
For variant 'user', will try loading 'D:\Home\pip\pip.ini'
For variant 'user', will try loading 'C:\Users\foobar\AppData\Roaming\pip\pip.ini'
For variant 'site', will try loading 'C:\Python37\pip.ini'
Python 3.8.0:
> pip config -v list
For variant 'global', will try loading 'C:\ProgramData\pip\pip.ini'
For variant 'user', will try loading 'C:\Users\foobar\pip\pip.ini'
For variant 'user', will try loading 'C:\Users\foobar\AppData\Roaming\pip\pip.ini'
For variant 'site', will try loading 'C:\Python38\pip.ini'
Finally got it sorted.
Apparently for Windows users pip.ini config file is not created, however can be added manually!
just create new %APPDATA%\pip\pip.ini and content of custom repository:
[install]
find-links = https://<login>:<password>#your.repo.com/custom/url
Reference: https://pip.pypa.io/en/stable/user_guide/#config-file
A bit late, but for reference:
Try adding the pip.ini file in %USERPROFILE%\pip\pip.ini (usually: C:\Users\<username>\pip\pip.ini).
On windows pip.exe looks for "pip.ini" in this order:
C:\ProgramData\pip\pip.ini
C:\Users\<username>\pip\pip.ini
C:\Users\<username>\AppData\Roaming\pip\pip.ini
It's been 7 years, and I think there's now a better answer for most people -- but it does depend on version of pip. For the most recent pips I'm using:
$ pip config -v debug
lists where it's looking and you can decide which location is
most useful for what you've got in mind. It does look like a fairly recent
change: On a year-old docker image I had with pip 20.1
I got "ERROR: Need an action (edit, get, list, set, unset) to
perform." On that system, pip config -v list gave a list of files it would try, this is supposed to be 'global', 'user' or 'site' variants of pip.ini locations.
For Windows 10, for pip 21.2.4 on both 3.9.6 and 3.6.8, I get response below with pip config -v debug, while pip config -v list is silent (unless a pip.ini is found).
global:
C:\ProgramData\pip\pip.ini, exists: False
site:
c:\py\myvenv\pip.ini, exists: False
user:
C:\Users\myname\pip\pip.ini, exists: False
C:\Users\myname\AppData\Roaming\pip\pip.ini, exists: False
From a downloaded image I got from dockerhub in June 2021 with pip 21.2.2 and python 3.6.10:
pip config -v debug
env_var:
env:
global:
/etc/xdg/pip/pip.conf, exists: False
/etc/pip.conf, exists: True
global.extra-index-url: http://trynexs:8081/repository/repo_group/simple
site:
/usr/local/pip.conf, exists: False
user:
/home/tanhauser/.pip/pip.conf, exists: False
/home/tanhauser/.config/pip/pip.conf, exists: False
Pip changed the location of the config file in windows starting in pip 6.0 the pip config docs explain the location of the config files as follows.
pip --version >= 6 (as of version 18.1 hasn't changed again yet)
%APPDATA%\pip\pip.ini
pip --version < 6
%HOME%\pip\pip.ini
Inside a virtual env
%VIRTUAL_ENV%\pip.ini
Site-wide win7+ (same as of win10)
C:\ProgramData\pip\pip.ini
Site-wide winxp (note windows vista side wide not supported)
C:\Documents and Settings\All Users\Application Data\pip\pip.ini
NOTE: If multiple configuration files are found by pip then they are combined in the following order:
The site-wide file is read
The per-user file is read
The virtualenv-specific file is read
Also pip added a config command starting in pip 10.
pip config --help
I know this is a bit late, however, this post is high on the rankings when searching. Inside a virtual environment pip.ini can also be in the root of the virtual environment. From the docs and https://pip.pypa.io/en/stable/user_guide/?highlight=pip.ini#configuration
Inside a virtualenv:
On Unix and macOS the file is $VIRTUAL_ENV/pip.conf
On Windows the file is: %VIRTUAL_ENV%\pip.ini
All the answers are partially wrong and right.
It depends on how your system is configured. The only way (for me) to find out was to patch site-packages/pip/locations.py at the point where site_config_files is assigned (around line 120 for pip 9.0.1)
print('########## ' + str(site_config_files))
and then run pip search foo
On my system it printed ########## ['C:\\ProgramData\\pip\\pip.ini'], of which location I assumed I could not create/edit. But it just worked.
Btw, for my system %APPDATA% points to C:\Users\MYUSER\AppData\Roaming, which is not looked at when running pip on my system.
Rather than guessing first check if you have any default global/local config which is read by pip with the below command:
pip config list
This will give all details of the default config loaded by python.
If the above command doesn't give any output please try to find where pip tries to find for the global config file with the below command:
pip config --editor <path to editor of your choice> edit
The above command will open the config file which pip reads by default or else it will give an error saying that the file doesn't exist.
If there's an error please go ahead and create the exact directory and file structure as show in the error. Once the file has been created please make your changes e.g.
[global]
cert = /path/to/base64/ssl/certificate.pem
proxy = http://username:password#ipaddress:port
Save the file and please try to check (the above mentioned check command) if the configs are loaded by pip or not.
For more info please follow pip config documentation
Make sure you acually have a pip.ini file, not pip.ini.txt.
For me (Windows 8, pip 9.0.1, python 3.5.3), the correct path was
c:\Users\<UserName>\.pypirc <- sic!, even on windows
Windows 10:
I had to create 'pip' directory inside
C:\Users\UserName\AppData\Roaming\
then create pip.ini file inside that 'pip' directory:
C:\Users\<username>\AppData\Roaming\pip\pip.ini
No other location worked for me.
For Windows, python will load the config from path below. So, if pip.ini file is not exist in these paths you can create the new file by refer these path depend on environment scopes (global, user & site) that you need python execute.
For variant 'global', will try loading 'C:\ProgramData\pip\pip.ini'
For variant 'user', will try loading 'C:\Users\MyName\pip\pip.ini'
For variant 'user', will try loading 'C:\Users\MyName\AppData\Roaming\pip\pip.ini'
For variant 'site', will try loading 'c:\python39-32\pip.ini'
By the way, you can check the paths as above by
pip config -v list
On a Windows 10 machine with multiple users I used this:
c:\users\all users\pip\pip.ini
Using pip version 22.3.1 with python version 3.10.4..