Import Bio module from Bash - bash

I'm working on a remote server with Linux.
I created a new environment and got biopython installed with conda:
conda create --name new_bio python=3.5
conda activate new_bio
conda install -c conda-forge biopython
import Bio works when I am running it inside python:
(new_bio) -bash-4.2$ python
Python 3.5.5 | packaged by conda-forge | (default, Jul 23 2018, 23:45:43)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import Bio
>>> exit()
import Bio works when I am running my script like this:
(new_bio) -bash-4.2$ python my_Bio_script.py
hello
my_Bio_script.py:
from Bio import SeqIO
print('hello')
But when I try to run my script with Bash :
(new_bio) -bash-4.2$ sbash my_bash.sh
my_bash.sh bash file:
#!/bin/bash
module load anaconda ### load anaconda module
source activate new_bio ### activating environment, environment
must be configured before running the job
python my_Bio_script.py
I get this error:
Traceback (most recent call last):
File "T.py", line 1, in <module>
from Bio import SeqIO
ModuleNotFoundError: No module named 'Bio'
I have to send it as a Slurm Job in a bash file.. what can I do?

One potential way to start debugging is to print environment variables PYTHONPATH and PATH.
What I would recommend is, print $PATH and $PYTHONPATH. Analyse the python version and path, replicate the same inside the bash script.
Also, analyse the ~/.profile as well as ~/.bashrc file and see whether there is any discrepancy with PATHs for python.

Related

how to solve Module not found error? ( installed the module and error persists)

I have installed the module tradingeconomics on cmd and when i try to import the library it throws this error , how to solve it ?
'
ModuleNotFoundError Traceback (most recent call last)
<ipython-input-7-15897cf55e0a> in <module>
1 #Importing required libraries
----> 2 import tradingeconomics as te
3 import pandas as pd
4 import numpy as np
5 from datetime import datetime,date,timedelta
ModuleNotFoundError: No module named 'tradingeconomics'
It is possible that you installed the module in an environment and you are not working in the environment, or you are working in an environment but the module was installed outside of the environment. I use a conda environment, you can set up one by typing this in the terminal
conda create --new myenv
conda activate myenv
set up pip using the following
conda install -n myenv pip
then you can download modules
pip install tradingeconomics
use conda deactivate to exit the environment,
you can learn more about conda here https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#using-pip-in-an-environment

Why is python's pip install behaving strangely in a script but fine in python prompt?

I'm trying to install a module from withing a python script using pip. Here is the contents of script.py:
#/usr/bin/python2.7
# I'm the file called `script.py`
import sys, importlib, pip
print(sys.version); print(sys.path) # For debugging
try:
importlib.import_module('docopt')
except ImportError:
pip.main(['install', '-U', 'docopt'])
finally:
globals()[pack] = importlib.import_module('docopt')
Runnting this script, e.g. using python2.7 script.py gives me:
$ python2.7 script.py
2.7.9 (default, Oct 3 2016, 17:42:24)
[GCC 4.3.2 20081105 (Red Hat 4.3.2-7)]
['/usr/lib/python27.zip', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/lib/python2.7/site-packages']
No module named docopt
Downloading/unpacking docopt
Cannot fetch index base URL https://pypi.python.org/simple/
Could not find any downloads that satisfy the requirement docopt
Cleaning up...
No distributions at all found for docopt
Storing debug log for failure in /root/.pip/pip.log
Traceback (most recent call last):
[...]
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
ImportError: No module named docopt
So problems with internet connection/ssl or something transport related. However, the same commands work perfectly fine in the python interactive interpreter, when I enter them by hand (copy&paste them):
$ python2.7
Python 2.7.9 (default, Oct 3 2016, 17:42:24)
[GCC 4.3.2 20081105 (Red Hat 4.3.2-7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pip, sys
>>> print(sys.path)
['/usr/lib/python27.zip', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/lib/python2.7/site-packages']
>>> pip.main(['install', '-U', 'docopt'])
Downloading/unpacking docopt
Downloading docopt-0.6.2.tar.gz
Running setup.py (path:/tmp/pip_build_root/docopt/setup.py) egg_info for package docopt
Installing collected packages: docopt
Running setup.py install for docopt
Successfully installed docopt
Cleaning up...
0
Again, the thing that weirds me out is, that it works interactively i.e. running python2.7 in the shell and then entering the very same code by hand. However, not when I run the script file with python2.7 script.py. So In one case, on the same machine, the same interpreter has internet access in the other case it does not.
I'm running out of troubleshooting ideas. I used the same machine and user to get above results. There is no python-startup file, so that is not making the commands magically work interactively. The contents of sys.path are the same in both cases. I'm not behind a proxy. Any ideas what could be missing in the script?
Cannot fetch index base URL https://pypi.python.org/simple/
indicates that for some reason your python interpreter cannot access to internet. There can be plenty of reasons for that. Do you use a HTTP proxy ?

ipython can't load a module which python does

I have recently updated to OSX El Capitain upgrading an exhisting working osx installation. Now I have a module, installed via pip on an anaconda distribution which is now broken. Or better the module can be correctly imported from python
Python 2.7.10 |Anaconda 2.3.0 (x86_64)| (default, Sep 15 2015, 14:29:08)
[GCC 4.2.1 (Apple Inc. build 5577)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import MDSplus as mds
whereas it does not load on ipython, claiming not to finding a library.
Python 2.7.10 |Anaconda 2.3.0 (x86_64)| (default, Sep 15 2015, 14:29:08)
Type "copyright", "credits" or "license" for more information.
IPython 4.0.0 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
Using matplotlib backend: MacOSX
In [1]: import MDSplus as mds
Error importing MDSplus package: Error finding library: MdsShr
The library is correctly installed, it is listed in /etc/profile, it is located in /usr/local (so there should not be a problem with the new security system of OSX el Captain) and the python interpreted which is called via python or ipython is the same (Anaconda 2.3.0).
How can I diagnose which is the problem?
EDIT:
the solution proposed in https://groups.google.com/a/continuum.io/forum/#!topic/anaconda/PEuOKEM5fdc does not worked for me. It does not seem to be an environment problem and the command
echo $DYLD_LIBRARY_PATH
points to the correct path
EDIT:
which -a ipython
ipython is /Users/vianello/anaconda/bin/ipython
which -a python
python is /Users/vianello/anaconda/bin/python
python is /usr/bin/python
>>> import sys
>>> for x in sys.path: print x
>>>/Users/vianello/anaconda/lib/python2.7/site-packages/pyhht-0.0.1-py2.7.egg
>>>/Users/vianello/anaconda/lib/python2.7/site-packages/MitDevices-0.3-py2.7.egg
>>>/Users/vianello/anaconda/lib/python2.7/site-packages/MDSplus-alpha_7.0.157-py2.7.egg
>>>//anaconda/lib/python2.7/site-packages/MDSplus-alpha_7.0.147-py2.7.egg/MDSplus
>>>/Users/vianello/anaconda/lib/python27.zip
>>>/Users/vianello/anaconda/lib/python2.7
>>>/Users/vianello/anaconda/lib/python2.7/plat-darwin
>>>/Users/vianello/anaconda/lib/python2.7/plat-mac
>>>/Users/vianello/anaconda/lib/python2.7/plat-mac/lib-scriptpackages
>>>/Users/vianello/anaconda/lib/python2.7/lib-tk
>>>/Users/vianello/anaconda/lib/python2.7/lib-old
>>>/Users/vianello/anaconda/lib/python2.7/lib-dynload
>>>/Users/vianello/anaconda/lib/python2.7/site-packages
>>>/Users/vianello/anaconda/lib/python2.7/site-packages/PIL
>>>/Users/vianello/anaconda/lib/python2.7/site-packages/Sphinx-1.3.1-py2.7.egg
>>>/Users/vianello/anaconda/lib/python2.7/site-packages/aeosa
>>>/Users/vianello/anaconda/lib/python2.7/site-packages/lmfit-0.8.3-py2.7.egg
>>>/Users/vianello/anaconda/lib/python2.7/site-packages/setuptools-18.3.2-py2.7.egg
>>>/Users/vianello/anaconda/lib/python2.7/site-packages/suds-0.4-py2.7.egg
in iPython the same give the results
/Users/vianello/anaconda/bin
/Users/vianello/anaconda/lib/python2.7/site-packages/pyhht-0.0.1-py2.7.egg
/Users/vianello/anaconda/lib/python2.7/site-packages/MitDevices-0.3-py2.7.egg
/Users/vianello/anaconda/lib/python2.7/site-packages/MDSplus-alpha_7.0.157-py2.7.egg
/Users/vianello/anaconda/lib/python27.zip
/Users/vianello/anaconda/lib/python2.7
/Users/vianello/anaconda/lib/python2.7/plat-darwin
/Users/vianello/anaconda/lib/python2.7/plat-mac
/Users/vianello/anaconda/lib/python2.7/plat-mac/lib-scriptpackages
/Users/vianello/anaconda/lib/python2.7/lib-tk
/Users/vianello/anaconda/lib/python2.7/lib-old
/Users/vianello/anaconda/lib/python2.7/lib-dynload
/Users/vianello/anaconda/lib/python2.7/site-packages/Sphinx-1.3.1-py2.7.egg
/Users/vianello/anaconda/lib/python2.7/site-packages/lmfit-0.8.3-py2.7.egg
/Users/vianello/anaconda/lib/python2.7/site-packages/setuptools-18.3.2-py2.7.egg
/Users/vianello/anaconda/lib/python2.7/site-packages/suds-0.4-py2.7.egg
/Users/vianello/anaconda/lib/python2.7/site-packages
/Users/vianello/anaconda/lib/python2.7/site-packages/PIL
/Users/vianello/anaconda/lib/python2.7/site-packages/aeosa
/Users/vianello/anaconda/lib/python2.7/site-packages/IPython/extensions
/Users/vianello/.ipython
Thus the MDSplus-alpha_7.0.157-py2.7.egg can be seen both from python and ipython
My first thought is that you are running afoul of the new Apple System Integrity Protection. iPython is launched using a shell script that loads via /bin/bash. That will result in your DYLD_LIBRARY_PATH environment variable being stripped when python launches. You can check this by importing os and then attempting to print the contents of os.environ["DYLD_LIBRARY_PATH"]. If that print throws an exception then you know what the problem is.
A simple fix that works for me is to edit the first line of the ipython script so that the #! calls your python binary directly rather than going via bash. This is not a long term solution as it will have to be redone each time ipython is updated until upstream changes (to be fair I'm not sure why bash is involved).
I have written a report on python and library paths at http://dmtn-001.lsst.io

How to list all modules python 2.7 and 3.4 have installed for MacOSX

How do I make all modules be available for both python 2.7 and 3.4?
I executed:
$ brew install <prerequisites for pygame>
<snip snip>
<snip snip>
$ brew install pygame
$ pip freeze
<snip snip>
pygame==1.9.2a0
<snip snip>
and can see pygame is installed. When I open a new shell/terminal and run python 2.7 and type import pygame, I have no problems:
Python 2.7.9 |Anaconda 2.2.0 (x86_64)| (default, Dec 15 2014, 10:37:34)
[GCC 4.2.1 (Apple Inc. build 5577)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://binstar.org
>>> import pygame
>>>
However, when I run python 3.4 and type import pygame, it says there's no module named pygame.
/usr/local/bin/python3
Python 3.4.2 (default, Dec 11 2014, 17:48:01)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pygame
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'pygame'
>>>
What do I need to do to make 3.4 see/link with all the modules that 2.7 sees?
P.S. I don't know how but pip is running from anaconda, which I think I might have installed when installing numpy or panda or something.
You cannot use modules installed for Python 2 in Python 3. Python 2 and Python 3 are not compatible, that is why most libraries offer a Python 2 and a Python 3 version.
So if you have PyGame installed with pip2, you can only use it with Python 2. To use PyGame in Python 3, you need to install it with pip3. Afterwards you will have both versions on your computer, so that a project that uses your Python 2 interpreter uses the PyGame installed by pip2 and a project that uses Python 3 interpreter uses the pyGame installed by pip3.
Example:
# test.py
import pygame
In the console:
> python3 test.py # imports pygame from /usr/lib/Python3.x/site-packages
> python2 test.py # imports pygame from /usr/lib/Python2.x/site-packages
You should not use Python 2 modules with Python 3 (and vice versa). Instead, install those separately, e.g.
pip3 install pygame
Sometimes, the code bases for a package will be compatible between 2 and 3, but quite often, they are not (especially when shared object modules, *.so files, come into play).
Also, Python 2 will not have the Python 3 packages library in sys.path by default, and vice versa. For a good reason.
All in all, consider Python 2 and Python 3 two different languages, for which you have to do everything separately. They just happen to be very much alike.
(Personal opinion/bit of advocacy: use and write your code in Python 3, unless you have a very good reason no to (unported essential libraries, chance of losing your job). It's relatively straightforward in 99% of the cases to make it Python 2 compatible later on, if really needed.)

Unable to import argparse 1.2.1 for python 2.5 on windows

So, I thought that I had correctly installed the setuptools and argparse 1.2.1.
C:\Python25_64bit>python ez_setup.py argparse
Searching for argparse
Best match: argparse 1.2.1
Processing argparse-1.2.1-py2.5.egg
argparse 1.2.1 is already the active version in easy-install.pth
Using c:\python25_64bit\lib\site-packages\argparse-1.2.1-py2.5.egg
Processing dependencies for argparse
Finished processing dependencies for argparse
C:\Python25_64bit>python
Python 2.5.4 (r254:67916, Dec 23 2008, 15:19:34) [MSC v.1400 64 bit (AMD64)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import argparse
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named argparse
>>> quit()
However, it seems that I can't import it, as you can see above. I also tried directly adding the egg to my Eclipse PyDev projects, but this was to no avail, either. As you can see, nothing is set in my environment. Perhaps, this is the problem?
C:\Python25_64bit>echo %PYTHONPATH%
%PYTHONPATH%
C:\Python25_64bit>echo %PYTHONHOME%
%PYTHONHOME%
I hope you paid attention while installing and noted where argparse was installed.
In your script (run from the commandline or from your IDE) include at the top:
import sys
for p in sys.path:
print repr(p)
to see all the paths python is searching through. Either add the path to argparse using:
sys.path.insert(0, '/your/path/to/argparse')
or move the install to something in the path.
The environment variables can be empty, there are compiled in elements for sys.path as well.

Resources