I try since multiple days to install solaris (https://github.com/CosmiQ/solaris) locally, on google colab or on renkulab (https://renkulab.io/). Up to now, without any luck. I tried on all platform different approaches:
Creating a conda environment (as recommended by the authors)
Directly through pip
And also cloning the repository and access the folders and functions directly
All of these approaches failed so far. Mostly there is a wheel building error for GDAL. Which i have installed first. I do not find any proper documentation or other failure descriptions which makes me question myself... Maybe here someone has experience with this library?
I highly appreciate every hint.
Thanks a lot
Colab Setup
I can get it set up in Colab with the following:
First Cell: Install Mamba/Conda
!pip install -q condacolab
import condacolab
condacolab.install()
This will trigger a runtime restart - it does this on purpose.
Second Cell: Install Solaris prerequisites
I'm assuming we want the GPU-enabled version. If not, there is another YAML in the solaris repository for a CPU-only environment.
!wget https://raw.githubusercontent.com/CosmiQ/solaris/main/environment-gpu.yml
!mamba env update -n base -f environment-gpu.yml
Manually restart the runtime after this completes!
Third Cell: Install Solaris
!pip install solaris
That should be it. Following these steps, I could import the module and use the entrypoints, e.g.,
Module import ✅
import solaris
Example entrypoint ✅
!make_masks -h
There were some future deprecation warnings from NumPy about some syntax in the TensorFlow code, but otherwise, seems functional. However, I don't personally use this tool, so I don't know if there is more to verify.
Related
first time on stack overflow. Basically pip is installing packages and Jupyter notebook can't find them to import. I've searched other similar questions and found some tips, but none of them have worked in my particular instance. I've shown the information that was helpful in other posts so you can see what I' mworking with:
My Notebook
In similar questions they've asked what "jupyter kernelspec list" returns in the terminal, so I've inlcuded that here:
KernelSpec Results
I would include my kernel.json file as well, but I tried changing with it, and upon seeing no change, tried deleting it altogether and my notebook runs fine.
So I'm thoroughly confused, and could really use some help.
Thankyou
Make sure the packages you think you installed are really installed to the running environment.
E.g. from inside notebook run:
!pip list | grep package_name
If the package in question is installed, get details using [pip show][1]
!pip show pyyaml
Name: PyYAML
Version: 5.1
Summary: YAML parser and emitter for Python
Home-page: https://github.com/yaml/pyyaml
Author: Kirill Simonov
Author-email: xi#resolvent.net
License: MIT
Location: /home/ntg/anaconda3/lib/python3.7/site-packages
Requires:
Required-by: bokeh, anaconda-client
Remember that for some libraries the import is not the name of the module,e.g. to install pyyaml I did pip install pyyaml, but to import it I will need import yaml... When in doubt, google the specific module.
Keep in mind that pip deals with modules (think libraries such as pandas).
If you cannot find some of your code, make sure it is in the directory you think it is, if not in the same directory, make sure the other directory has a file named init.py to denote the dir is a module, and possibly check
Cannot import from __init__ in a subfolder
Please, I do need a light here. I want to install numpy using a good BLAS/LAPACK lib on Windows, but absolutely no page explains the process well enough. It seems OpenBLAS is a good and fast option.
The goal is to use "theano" with "keras", and "theano" requires that the libraries be "dynamic", not static. (Not sure I understand what that means, but it causes slowness and memory issues)
Please treat me as a complete newbie. Give me a step by step tutorial on how to do it! Don't forget to tell me "where" files should go! Which folders should go in PATH! What commands exactly I should call, and what are their output, where? What do I do with their results or with compiled files? How does numpy find them? Etc. All the sites I've seen seem to think I'm a linux expert and already know everything.
What I have tried:
Downloaded the compiled version of numpy+mkl from here -- This does install numpy, it becomes usable, but theano presents the memory leak problem, besides working slowly. Is it a matter of setting the right ldflags in the .theanorc file? If so, which are the flags? - About the MKL libraries, this answer may be useful?
Tried installing Anaconda - it doesn't work either, and I had no idea about what went wrong. It gave me messages suggesting installing some extra stuff, it worked but incredibly slow. More than 10 times slower than my bugged numpy version mentioned above (so, unacceptable, impossible to work at that speed). If I have to go changing everything about Anaconda, it's better to use a regular python instead and know what is going on.
Found these already compiled BLAS/LAPACK libraries (.dll and .lib) files. But.... what am I supposed to do with them? -- Simply adding their folders to the PATH var and installing numpy gives me "numpy-atlas", not the libraries I downloaded. How do I make numpy see them?
Tried to understand this page, but yet, it seems it will lead me exactly to the previous case, what will I do with the results? Where are the libs they suggest I use? What are the suggested quickbuild scripts, where are they?
Found the Cygwin option here. I haven't tried it, but it sounds it should be something easier than reinstalling all my python and packages, all from Cygwin
Year 2021: this works with miniconda on Windows 10
conda create -n openblas python=3.8
conda activate openblas
conda install conda-forge::blas=*=openblas
conda install -c conda-forge numpy
Tested also with matplotlib-base.
Tried following the building instructions in http://scipy.github.io/devdocs/building/windows.html a number of times without success.
[MY SOLUTION]
After extensive reading of the logs and related forums, came up with a method that has worked for me in two windows machines already. Posted a batch file with the steps I used in https://github.com/jhvital/build-scipy-openblas.
The show_config method shows linkage to openblas libraries, and this was confirmed by comparing the runtimes of scripts against the scipy installed with conda install scipy, which installs the mkl package.
Still I'm not entirely sure whether the libraries were linked correctly. Feedback would be appreciated.
The build instructions for SciPy is tested a number of times and it seems to be working.
You can find the instructions here http://scipy.github.io/devdocs/building/windows.html
Note that this is still a very delicate procedure and needs to be done very carefully. It still does not guarantee success thanks to Windows being Windows. However please take the time to report any issues should you encounter any at https://github.com/scipy/scipy.org/issues (notice it is not the scipy repo but but scipy.org repo)
I am not sure what's the error you have recieved when trying to get openblas & numpy using anaconda.
For openblas on windows using anaconda3 python 3.6v this works:
conda install -c menpo openblas
ref: https://anaconda.org/menpo/openblas
For numpy:
it is :conda install numpy. if you are looking for a specific version of numpy use: conda install numpy=version_number
Jython Package installation issue, using pip
Hi, I have installed Jython2.7 configured with pydev in eclipse neon, also configured python 3.6 package
I am able to install packages for python using pip installer?
pip install "packagename"
Below are some of the packages in python/Lib/Site-packages directory
I was able to install all the packages
How do I use pip installer to install packages for jython?
I tried to install Jip package with
jython install setup.py
The binary File got installed in the Jython/Lib/Site-packages folder
However, I am not able to use it.
where and how do I get Jython package binaries like jip?
Also, Please let me know how to search jython packages?
Also, How to make pip install library packages in jython?
Any other configuration like jython home, etc that should be made?
This answer is going to be really generic but I just recently have slogged my way through the setup for jython/jip/pip and here's roughly what I had to do.
Firstly, I'm running Windows 7 64 Bit from behind a proxy (work machine.)
Had to install jython 2.7.0 instead of 2.7.1 because (I think anyway) 2.7.1 requires admin privileges which I don't have on my work PC.
Pip didn't install correctly during the Jython installation and I spent an obscene amount of time trying to get it installed and functioning as I knew it from my cpython days. NOTE: Just because you get pip installed, doesn't mean you can use any package on a python package repo. As of 2.7.0, Jython doesn't have end to end capability to interpret/compile some libraries that rely on certain python wrappers of native OS function calls. I believe 2.7.1 makes solid progress in the direction of supporting all needed native calls but don't quote me on that. For example, I tried to use wxPython to make a simple GUI to test my jython install. Trying to install it from pip kept causing really non-specific error info that took me a lot of time to figure out that the cause was jython simply couldn't compile the wxPython source so beware.
I had to set environment variables 'http_proxy' and 'https_proxy' in the form of http://proxyhosturl:port and https://proxyhosturl:port respectively to get out from behind the proxies without having to invoke pip with the proxy switch every time I called it.
To actually install pip, have a look here. These instructions are for Python and Linux/Unix but the principle is roughly the same. Just use jython -m instead of python -m and ignore the '$' at the start of each command line.
Also be sure to CD to your python_home/bin folder when invoking the ez_install exe.
If that doesn't work (didn't for me), try using get-pip.py script with these instructions https://pip.pypa.io/en/stable/installing/ (remember jython instead of python etc.). Download it, cd to the download location and follow the noted install steps. Worth noting is about half way down the install instructions where it details installing from local archives (source/binary zip or tar.gz archives of pip and setuptools as better described here: https://packaging.python.org/tutorials/installing-packages/#installing-from-local-archives).
The links to the bin archives of pip and setuptools are here:
https://pypi.python.org/pypi/setuptools
https://pypi.python.org/pypi/pip
It may also be worth making sure that your PATH environment variable has the jython/bin path in the variable value. The jython installer should do this but, again, mine did not.
If all goes well, you should be able to invoke pip with the --version switch and if it prints a line with the installed pip version info then you should be good to go
Another quirky issue I had was I could invoke a function of pip one time and any subsequent times I would get a stack trace ending with something along the lines of an object not having a certain property. I fixed that by finding my temp directory by opening a windows explorer instance and typing %TEMP% in the address bar and hitting enter, it should take you to a subdir of your AppData folder and there you may see a folder with the name of the package you were trying to install and the text "_pip" somewhere in the directory name. Delete the directory and try the pip install command again. I had to do this + invoke pip install pip -U to update my install to the latest version. Then pip began behaving correctly in my instance.
pip search numpy (or your library name) will generate a list of results with the same logic it uses to locate your desired package when you call pip install but, again, just because it returns a matching package doesn't mean it will compile when you install it (numpy doesn't work because of the missing java to C native function calls I described earlier.) The trade off is that you can import code artifacts from Java JAR files in your Jython script files and leverage their functionality with relative ease. Between the public Java APIs available and the python packages that work with the jython interpretor, you can (in my experience) come up with a way to accomplish your task. See the following info on JIP, Maven and IDEs.
IDE and jython integration (Eclipse)
- If you are stuck using Eclipse (like me) it actually has pretty decent support for python development. Install the PyDev plugin for Eclipse from Help -> Install Software. Put in this URL https://marketplace.eclipse.org/content/pydev-python-ide-eclipse, hit tab, and select the PyDev plugin and hit 'finish.'
- Setup the jython interpretor info from Windows -> Preferences -> PyDev. Provide the path to your jython.jar file.
- You should now be able to use File -> New PyDev project to create a basic python project and configure it to use your version of Jython and Java.
Brief Overview of Jip and Maven
- jip is a jython package that is invoked very similarly to pip but instead will download JAR files from the Maven Central Repository instead of python packages from pypi.com, for example. See the install instructions described here. Note the install procedure for a global jip install which differ from just pip install jip. https://pypi.python.org/pypi/jip/
- I never got jip to work exactly as I wished because there's not a ton of documentation on it outside of what I already linked. However, if you install a JAR using jip, you have to go to your project in Eclipse and actually add the JARs themselves to your PYTHONPATH in order for import statements and editing to have intellisense and so that you don't get a classnotfound exception at runtime. See following screen shot.
- There is a JIP config file that you can use similar to the pip config ini file but I have yet to find any exhaustive documentation on it's setup.
Note in the above screen shot the first entry in the External libraries entries. By default, pip places installed packages in that directory so to enable eclipse to find them, you need to also ensure that location is entered.
In Conclusion
- I have more to add to this answer and I will do so as soon as possible. In the meantime, see this example project I've loaded into github.
https://github.com/jheidlage1222/jython_java_integration_example
It shows basic config and how to interface with JARs from python code. I used the apache httpcomponents library as an example. Good luck amigo.
I am on windows 64bit, I have installed anaconda, and managed to create an environment with python 2.7
I have numpy, pylearn2, theano, and every package is built properly
I have been able to import all these modules, however I get some very esoteric messages when I try to complete the model, like
ImportError: Could not import pylearn2.models.softmax_regression but could import pylearn2.models. Original exception: No module named dnn
Then I tried to actually find the package in the installation, but inside the cuda folder, there is no module named dnn. Looking at github, I see that it should be there.
Why is theano missing modules? I installed using conda install theano, and it gave some suggestions, I have managed to pick the correct one.
I have uninstalled and installed theano many time, I can import it but I can never get the proper modules.
What is going wrong?
Ok, after a few days of search, it seems like Theano installed from anaconda is missing a lot of modules. However, installing theano by cloning the repository with
pip install --upgrade --no-deps git+git://github.com/Theano/Theano.git
seems to resolve the issue. Since windows normally does not have git, it can be easily installed (seems to take care of the environment's path variable) from here
https://git-scm.com/download/win
I got Numpy and Matplotlib running on Heroku, and I'm trying to install Scipy as well. However, Scipy requires BLAS[1] to install, which is not presented on the Heroku platform. After contacting Heroku support, they suggested me to build BLAS as a static library to deploy, and setup the necessary environment variables.
So, I compiled libblas.a on a 64bit Linux box, and set the following variables as described in [2] :
$ heroku config
BLAS => .heroku/vendor/lib/libfblas.a
LD_LIBRARY_PATH => .heroku/vendor/lib
LIBRARY_PATH => .heroku/vendor/lib
PATH => bin:/usr/local/bin:/usr/bin:/bin
PYTHONUNBUFFERED => true
After adding scipy==0.10.1 in my requirements.txt, the push still fails.
File "scipy/integrate/setup.py", line 10, in configuration
blas_opt = get_info('blas_opt',notfound_action=2)
File "/tmp/build_h5l5y31i49e8/lib/python2.7/site-packages/numpy/distutils/system_info.py", line 311, in get_info
return cl().get_info(notfound_action)
File "/tmp/build_h5l5y31i49e8/lib/python2.7/site-packages/numpy/distutils/system_info.py", line 462, in get_info
raise self.notfounderror(self.notfounderror.__doc__)
numpy.distutils.system_info.BlasNotFoundError:
Blas (http://www.netlib.org/blas/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [blas]) or by setting
the BLAS environment variable.
It seem that pip is not aware of the BLAS environment variable, so I check the environment using heroku run python:
(venv)bash-3.2$ heroku run python
Running python attached to terminal... import up, run.1
Python 2.7.2 (default, Oct 31 2011, 16:22:04)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.system('bash')
~ $ echo $BLAS
.heroku/vendor/lib/libfblas.a
~ $ ls .heroku/vendor/lib/libfblas.a
.heroku/vendor/lib/libfblas.a
~ $
And it seems fine. Now I have no idea how to solve this.
[1] http://www.netlib.org/blas/
[2] http://www.scipy.org/Installing_SciPy/Linux
I managed to get this working on the cedar stack by building numpy and scipy offline as bdists and then modifying the heroku python buildpack to unzip these onto the dyno's vendor/venv areas directly. You can also use the buildpack to set environment variables that persist.
Heroku haven't officially published buildpacks yet - search for 'heroku buildpacks' for more thirdparty/heroku ones and information.
My fork of the python build pack is here:
https://wyn#github.com/wyn/heroku-buildpack-python.git
The changes are in the bin/compile where I source two new steps, a scipy/numpy step and an openopt step. The scripts for the two steps are in bin/steps/npscipy and bin/steps/openopt. I also added some variables to bin/release. Note that I am assuming installation through a setup.py file rather than the requirements.txt approach (c.f. https://devcenter.heroku.com/articles/python-pip#traditional_distributions).
I also download the blas/lapack/atlas/gfortran binaries that were used to build numpy/scipy onto the dyno as there are c extensions that need to link through to them. The reason for building everything offline and downloading is that pip-installing numpy/scipy requires you to have a fortran compiler + associated dev environment and this made my slugs too big.
It seems to work, the slug size is now 35mb and scaling seems fast too. All but one of the numpy tests pass and all of the scipy tests pass.
This is still work in progress for me but I thought I'd share.
In case someone else comes to this like I did...
The excellent answer on this question from #coshx sadly no longer works (at least I could not get it to work). What I did, however was the following:
I forked npsicpy-binaries repository from #coshx and changed all the tar files such that they do not have venv as the root folder inside (my fork is here)
I forked the npsp-helloworld repository from #coshx and made it use a requirements.txt file instead of the setup.py (my fork is here - this means that you can use the whole pip approach).
I forked the heroku-buildpack-python repository from Heroku, took the npscipy file from #coshx and changed it to work with this latest version of the build pack (my fork is here - you can see that there is no venv set up, for example).
After doing those three things I have the npsp-helloworld application working perfectly. You just need to make sure you set the buildpack appropriately when creating the application and you are good to go:
$ heroku create --stack=cedar --buildpack=https://github.com/kmp1/heroku-buildpack-python.git
NOTE: I haven't worked out how to make my own binaries yet, so the three libraries (scipy, numpy and scikit-learn) are not the latest versions so make sure when you install it you do (if anyone can build these I would gladly accept a pull request for them):
pip install scipy==0.11.0
pip install numpy==1.7.0
pip install scikit-learn==0.13.1
By the way - I am really sorry if I have not done things in the correct way, etiquette-wise. I'm still learning git and this whole open source thing.
Another good option is the conda buildpack, which allows you to add any of the free Linux64 packages available through Anaconda/Miniconda to a Heroku app. Some of the most popular packages include numpy, scipy, scikit-learn, statsmodels, pandas, and cvxopt. While the buildpack makes it fairly simple to add packages to an app, the downsides are that the buildback takes up a lot of space and that you have to wait on Anaconda to update the libraries in the repository.
If you are starting a new Python app on Heroku, you can add the conda buildpack using the command:
$ heroku create YOUR_APP_NAME --buildpack https://github.com/kennethreitz/conda-buildpack.git
If you have already setup a Python app on Heroku, you can add the conda buildpack to the existing app using the command:
$ heroku config:add BUILDPACK_URL=https://github.com/kennethreitz/conda-buildpack.git
Or, if you need to specify the app by name:
$ heroku config:add BUILDPACK_URL=https://github.com/kennethreitz/conda-buildpack.git --app YOUR_APP_NAME
To use the buildpack, you will need to include two text files in the app directory, requirements.txt and conda-requirements.txt. Just as with the standard Python buildpack, the requirements.txt file lists packages that should be installed using pip. Packages that should be installed using conda are listed in the conda-requirements.txt file. Some of the most useful scientific packages include numpy, scipy, scikit-learn, statsmodels, pandas, and cvxopt. The full list of available conda packages can be found at repo.continuum.io.
For example:
$ cat requirements.txt
gunicorn==0.14.2
requests==0.11.1
$ cat conda-requirements.txt
scipy
numpy
cvxopt
That’s it! You can now add Anaconda packages to a Python app on Heroku.
The slug compiler is not aware of your environment variables, which is why it fails during push, and not once running.
The only real option you have is to look at the user_env_compile addon that's currently in labs beta.
http://devcenter.heroku.com/articles/labs-user-env-compile
For those who wish to use Python 3.4 in production, I've built numpy 1.8.1, scipy 0.14.0, and scikit-learn 0.15-git (0.14 doesn't work with the others for some reason) as binaries on Ubuntu 10.04 LTS 64-bit, which works on the Heroku cedar stack. Here is the git repo.
My heroku buildpack is quite similar to that of kmp. Note that the bin/steps/npscipy file links to my repository of binaries above.
I'm putting this here in case someone stumbled on this like I do. Since I am using python 3.4, thenovices buildpack doesn't work out for me.
I tried to look at conda buildpack, but it seems like a little bit of an overkill for my application which only requires scipy and numpy. What finally works for me is this excellent guide, using a multi buildpacks approach. The scipy installation was indeed a bit long though. Hope this helps! :)