what is the difference between '!pip' and '%pip'? We can use both of them in jupyter notebook and google colab. But we can not use the simple 'pip' there, can we?
%pip is a magic function and works mostly the same like pip. You can use other magic functions in jupyter which you can list with %lsmagic
(https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-lsmagic)
Using ! allows to run commands like ls or pip or what you have available on your OS.
Colab is just like jupyter so you can run native pip like pip install tensorflow.
See related answers:
What is the meaning of exclamation and question marks in Jupyter notebook?
What is %pylab?
How do I install Python packages in Google's Colab?
Related
I'm a little confused with package install in anaconda environment.
I can install my python packagse in the following ways.
open anaconda prompt and do : conda install tensorflow
launch an jupyter notebook from anaconda prompt, choose the default python kernel, and do:
!pip install tensorflow
Can someone tell me what is the difference between these two cases? Where are the python packages installed? What happens when I pip install python packages in default kernel in jupyter notebook?
I was able to install tensorflow in jupyter notebook in default kernel (python3), but trying to import tensorflow give me module note found error. Does anyone know why that happened? What is happening under the hood? Very much appreciate any help to clarification.
pip is the default package manager that ships with python. Conda is also a package manager, but it is third party. Conda was made especially for data science libraries. Libraries installed with conda usually give much better performance than pip. In pip, the packages are stored in python/scripts and conda stores them at /anaconda/pkgs/. As for the module not found error. I would need more information about it, but you can check out this video. I learnt how to install TensorFlow here and would highly recommend it.
The Jupyter notebook worked initially, but I tried importing tensorflow and that would not work, so that led to me messing up everything.
I basically messed everything up, and I feel like the only way out now is to just nuke my device and restart. I had no idea what pip and anaconda are (still don't really), tried a bunch of funky updates and installations and whatever and now everything is just dead. My jupyter notebook cannot even run the normal python kernel.
How can I hard reset everything?
As a bonus, if someone were to ELI5 the difference between conda, pip, gitbash, and PowerShell are. And what versions of stuff does Jupyter run on (since my conda and device had different versions of things I think?). I use Windows 10.
My first piece of advice is to not use Windows, though I'll probably get downvote spam for that. On Ubuntu, I could stuff Jupyter setup into one line:
# update, install python3, python3-dev, and pip3; get pip packages
sudo apt-get update && sudo apt-get install -y python3 python3-dev python3-pip && sudo -H python3 -m pip install jupyter notebook ipykernel tensorflow
Once the packages are installed, it's as easy as running jupyter notebook in the terminal.
Anaconda is a distribution of Python that includes a ton of pre-built packages, including Jupyter and scipy, numpy, pandas, etc. It's an "out of the box" solution basically, that comes with most of the tools you need. "Pip" is a package manager for Python; pip install [package] lets you use a package in your script, like import [package]. In this case, that's tensorflow.
ipykernel is a package that will open up a Python kernel for Jupyter. You could run a Jupyter notebook on a Python3.7 backend but do stuff with Python2 code by installing ipykernel with Python2's pip, usually (on Ubuntu) sudo apt-get update && sudo apt-get install -y python-pip && sudo -H python -m pip install ipykernel.
What happens when you run jupter notebook? Do you get errors? Can you get the notebook to open, but there's just no kernel to attach to a notebook?
I have just set up a new Windows 10 machine for Python, Jupyter, and Tensorflow. I did the set-up without anaconda. I did the normal set-up procedure with some special steps:
1) Python 3.8 und Jupyter as installed by "pip install" does not work. You need to add three lines of code in a module that is installed as dependency when you install Jupyter. change asyncio.py
2) Current Tensorflow does not work with Python 3.8. You need to install Python 3.7. You don't need to delete your Python 3.8 if you have one. Create a virtual environment with virtualenv as described here and give the Path to your Python 3.7 Special Python in virtualenv
3) If you want to use GPU for NVIDIA in Tensorflow, you need to deal with the fact that two things do not fit together: current Tensorflow and the current version of ‘NVIDIA GPU Computing Toolkit’ (a tools you need for GPU support). Take a look here for the fix: cudart64_XYZ.dll not found
Let's start with the basics:
As a bonus, if someone were to ELI5 the difference between conda, pip, gitbash, and powershell are
You probably know the classical cmd.exe which opens a basic terminal where you can use different commands and call programs from. It is basically a text based way to interact with your operating system.
Powershell is in my understanding just an extension of this (I don't use it myself) and has more capabilities of what you can do and also better scripting support.
gitbash is an optional tool that you probably installed when you installed git on your computer. It emulates a bash shell that many people are used to from different operating system like ubuntu where bash is often the default terminal and therefore makes it easier to use, as all the syntax and commands are then the same as these ppl are used.
Neither of these is in any way directly related to using python on your computer other than being able to type python or jupyter notebook into these terminals to start the applications.
To the more python specific questions:
conda is a package and virtual environment management tool. It can be used to install a variety of software and also create virtual environments to keep different set ups seperate from one another (e.g. different python versions on the same machine). But it is not limited to python. It is pre-installed when you download and install miniconda or anaconda which are two python distributions.
pip is a package manager only for python packages and comes pre-installed with most python distributions.
anaconda/miniconda , often times confused with conda are two python distributions, i.e. what you would consider as "I installed python on my system" that come with the conda package manager pre-installed. miniconda does thereby not ship any other packages while anaconda comes with a long list of useful packages pre-installed and is therefore a popular choice when you want an easy acces into using python for your research
For more info, you can also read understanding-conda-and-pip
How can you save your system now
I basically fucked everything up
Difficult to access the current state of your system, but I would suggest you try the following steps to get to a working condition again:
Go into Setting -> Apps and remove everything that is related to python or anaconda. Make sure that everything is deleted by also searching (using windows search feature) for python or conda folders somewhere in C:\Users. This should make sure that everything about your setup is purged
Make sure that neither python, pip or jupyter commands are working anymore in your cmd (confirming the purge)
Download and install miniconda
Now Create a virtual environment and install tf. This is a good way to go because if you should manage to f*k up the environment, you can just delete and recreate it without much trouble:
conda create -n venv pip python=3.7 #create environment
conda activate venv #activate the environment
conda install jupyter #for jupyter notebook
pip install https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-2.1.0-cp37-cp37m-win_amd64.whl
Start jupyter notebook: jupyter notebook. Since it only exists in this environment, same as tensorflow, there should be no more issues to use tensorflow normally
I wanted to use to add some python packages using pip in my Google Colab Notebook terminal and also set my environment variable(a JSON file).
Where can I find a terminal so that I can do these tasks?
Is it even possible to run Google Colab Notebook as a superuser?
Colab notebooks execute as root on ephemeral VMs. You can run pip install and configure environment variables like so:
Pip install:
!pip install -q matplotlib-venn
Environment variables:
import os
os.environ['YOUR_VARIABLE'] = 'value'
Did you encounter a particular error attempting to do something specific?
after following the instruction in http://docs.h2o.ai/h2o/latest-stable/h2o-docs/downloading.html#install-in-python, I was able to install h2o v 3.16.0.2. I was also able to use command line instructions mentioned in the above website and test that it is working.
python
import h2o
h2o.init()
h2o.demo("glm")
However, when I launch anaconda spyder, I am not able to import h2o. How do I link the h2o I have installed and bring it into Spyder python?
Update:
I have already tried {conda install -c anaconda h2o } which is mentioned on Anaconda.org, but that installs older ver 3.10 of h2o and that did not work either.
thanks for your help.
I believe the problem is due to your python environment. When you install Anaconda, you need to use anaconda pip, to ensure that the installed packages are available in conda.
The short answer is you can install the lastest stable version of H2O (3.16.0.2 as of today) using conda via h2oai channel instead of anaconda channel (h2o maintains it's own channel):
conda install -c h2oai h2o
this should solve your issue.
But more generally, the packages will appear in conda if you use anaconda pip. You can check which pip is being used by doing
which pip
and making sure that the path to the pip is within your anaconda distribution; something like /home/<userdir>/anaconda/bin/pip instead of /usr/bin/pip
Same is true also for Python. Try checking if starting Python in terminal points to anaconda Python by doing which python. If that's not the case, than you would need to add the conda installation of Python to your PATH variable. Please refer to conda docs for instructions https://conda.io/docs/user-guide/install/index.html
It would have been helpful if you had included information regarding your operating system in the question.
I installed tensorflow as described for Anaconda installation in the tensorflow.org. I could run the tensorflow in the Mac terminal just typing,
$ source activate tensorflow
$ python
import tensorflow as tf
But, when I run iPython notebook, it does not work by showing an error of no module named tensorflow. I felt like tensorflow was correctly installed but iPython notebook could not recognize it. Is there any way to fix it?
Please have a look at your path. When you do the command 'source activate tensorflow', your path may have changed. The example is following.
sherrie#sherrie-PC:~/tensorflow/tensorflow/examples/udacity$ source activate tensorflow
discarding /home/sherrie/anaconda2/bin from PATH
prepending /home/sherrie/anaconda2/envs/tensorflow/bin to PATH
Tensorflow is installed in the first path, instead of the second one.
The most important step comes. Open your directory of 'anaconda' --> find the 'envs' --> open 'tensorflow' --> copy all files in 'sitepack0ages' to 'anaconda/lib/python2.7/sitepack-ages'. Done.
My answer is based on this other answer: Trouble with TensorFlow in Jupyter Notebook. I had the exact same problem, and I solved it by doing this:
Once your environment is activated, run "which jupyter" and verify if the path returned points to the /bin folder under your environment, something like /my_environments_path/my_tensorflow_environment_name/bin/jupyter. If this is already the case, your scenario is different from the one I had;
Run "which pip" and/or "which pip3". Odds are, you are executing pip or pip3 from a different location, so Jupyter is not being installed inside your environment. In my case, Python version was 3.6, but pip3 was being called from a different location. I ran "pip install jupyter", since pip was inside the /bin folder in my environment. Once I called the correct pip, I checked again the "which jupyter" command, and this time Jupyter was under my environment and I could import Tensorflow inside my notebooks. If no pip's path points to your environment, install pip inside it by running "conda install pip".
I hope this works, along with the additional information on the link above.