I am trying to submit a python job to the queue on an SGE cluster with scientific linux os.
I keep getting the error:
ImportError: libg2c.so.0: cannot open shared object file: No such file or directory
When the script uses numpy.
The library doesn't exist on any of the slave nodes and seems to only be installed locally on the head node. I have full access to this cluster but I'm relatively inexperienced at cluster management.
I have tried:
export LD_LIBRARY_PATH=<path_to_lib>:$LD_LIBRARY_PATH
but I get the same error.
Is there a way to use this python without needing to install it on all the nodes?
If the only way is to roll this python on to all the other nodes what is the best way to do it?
Thanks
It turns out the answer was deceptively simple. I had been using the extract command in the terminal. Instead I included
export LD_LIBRARY_PATH=/usr/lib64/:$LD_LIBRARY_PATH
in the module file. Now when I load the module the libraries are loaded with it and everything works fine.
I'm not sure why this works but hopefully this will help someone else.
Ben
Related
I want to adapt Robotframework and RIDE to be able to use them on a USB key. In order to achieve this, I planned to use a Portable Version of Python on which I would install all necessary modules using pip.
The main advantage of Portable Python is that it doesn't required any installation on the user's computer, they just have to use the Console executable provided by Portable Python to access Python and its modules. A simple insertion of the USB key, and the user would have access to the modules.
Let's say I install Portable Python on a USB key, located on E:\. Using E:\PortablePython\Console-Launcher.exe which uses command-lines code with all Python related files in the PATH, I manage to install all necessary packages for RobotFramework and RIDE with pip. For example, we can find the robot.exe in E:\PortablePython\App\Python\Scripts and the python exe is in E:\PortablePython\App\Python\python.exe like a usual installation of Python.
Everything works like a charm here, I can run my tests without any issue. The problems come when the key is moved. If I try to run ride on D:\:
D:\PortablePython>ride.py
Unable to create process using 'E:\PortablePython\App\Python\python.exe "D:\App\Python\Scripts\ride.py" '
It actually works if I write :
D:\PortablePython>D:\App\Python\python.exe D:\PortablePython\App\Python\Scripts\ride.py
But when I try to run tests within RIDE with the robot execution profile, a similar error appears in the console :
Fatal error in launcher: Unable to create process using '"E:\PortablePython\App\Python\python.exe" "D:\App\Python\Scripts\robot.exe" --version': Le fichier spÚcifiÚ est introuvable.
A workaround is to force the reinstallation of RobotFramework on the USB key to have it use D:\PortablePython\App\Python\python.exe instead of E:\PortablePython\App\Python\python.exe.
So here are my questions:
Is there any tweaks available on pip to have packages moveable in a way ? Using relatives paths instead of absolute paths maybe ?
Is it possible to have execution profiles in RIDE use the python exe on D:\ instead of the non-existing one on E:\ ?
Can virtual environments play a role in this case ?
RIDE uses the $HOME/.robotframework/ride or %APPDATA%\Robotframework\ride directories to create a database of keywords documentation (Help), and the settings.cfg. If you want to preserve this information, you should have a copy in the removable memory, and have some script to create a link to it.
This script may be a better solution to setup RIDE before it is launched. That way you can setup all the needed paths or environment variables, like, for example, PATH, PYTHONPATH.
If your code is Open Source, you can propose improvements to RIDE's portability, at https://github.com/HelioGuilherme66/RIDE
It seems I've been using an incomplete version of a Portable Python. I tried to use WinPython instead, which has a script making the installation moveable, and it solved all the issues I mentionned.
First I downloaded all the packages and libraries needed to do my tests, then I applied the make_winpython_movable.bat provided by WinPython. Everything works now when I move the files or the key.
I made an application using GTK3 on Windows (Mingw_x64 installation of GTK) and I cannot really figure out how to make a distribution out of this. According to official documentation of PyGObject, it is possible in some way.
I already tried to make a package using setuptools, but PyGObject documentation is not saying much about this process and I was not able to configure setup correctly to make it work. PyGObject has a lot of dependecies and weird imports, that I do not know how to include.
I also tried Pyinstaller, which claims it has GTK support, and it really can pack it into executable, however it is not working. I tried these two options:
make only one file (.exe), but in this situations, it throws an error, that some file is not found (libpixbufloader-ani.dll)
create a directory with all needed files (libpixbufloader-ani.dll and other libs are included this time), but when running exe, another exeption occurs, this time Struct and 2 other libraries are missing (strangely, there is a folder that contains Struct)
Becouse of the missing files, I tried adding as many paths containing needed libraries as possible to Pyinstaller, but without success.
Does anyone have any experience with packaging GTK appliciations in Python? There is definitely a way to do this, but I am not very experienced with packaging. If needed, I can provide more information.
This is an issue that has been brought up on PyInstaller's GitHub page, as others (including myself) have experienced the same issue that you've mentioned.
The last time I tried the dev version of PyInstaller, the issue still wasn't fixed, but I managed to get a working executable by using PyInstaller to find the dependencies that my Python3/GTK3 app needed, and then I used cx_Freeze to generate the final executable.
I have been trying to dump a database I have created using neo4j Desktop app version 3.3.0 on Mac OS X. The documentation said that I need to use neo4j-admin on command line. This was not installed as part of the Desktop version, so I installed neo4j v3.3.0 with homebrew, and got neo4j-admin.
But while running command:
neo4j-admin dump --database=graph.db --to=~/mydatabase.db
I always get the error:
command failed: database does not exist: graph.db
I have searched here and on google, and the only thing I found has been
Neo4j 3.2.3: dump command does not find database
But the answer there is quite vague and I am not too sure what it means, as I am new to neo4j. I gathered that the desktop version is not compatible with the command-line, and that one needs to stick purely with the command-line version, am I understanding this wrong?
I can find graph.db in:
$NEO4J_HOME/data/databases
But neo4j-admin seems to just refuse to find it, and setting the environment variables did not solve the problem. And I cannot supply a path to the database as neo4j-admin doesn't take paths.
If I know where neo4j-admin is trying to look for the database, then I can hopefully solve the problem by provide a symbolic link to it, but at the moment I have no idea where it is looking.
Any help would be greatly appreciated.
neo4j-admin is probably looking for graph.db in the same directory that your Homebrew Neo4j is installed. You can find out where that is by doing which neo4j-admin, then checking that directory with ls -al to see where Homebrew's actually storing the files.
It'll be somewhere like /usr/local/Cellar/neo4j/<version number>/bin, and by default (at least with Neo4j 3.1.1) it'll expect the database to be in /usr/local/Cellar/neo4j/<version number>/libexec/data/databases/.
I am currently attempting to install the tensorflow object detection app on Windows 7 (employer requirement) and I am failing at a few steps from the end.
Basically I get the following error when I run the installation test command:
ImportError: No module named nets.
I have read some solutions on the subject:
https://github.com/tensorflow/models/issues/729
https://github.com/tensorflow/models/issues/1842
which looks like this:
export PYTHONPATH="$PYTHONPATH:"somepath"/tensorflow/models/slim"
basically meaning that I must set the right path in the PYTHONPATH environmental variable.
Working with Windows, I tried calling this:
SET PYTHONPATH="$PYTHONPATH:C:tensorflow/models/slim
And when it didn't work, I created a PYTHONPATH variable in system-> environmental variables.
I'm still getting the error so I suppose that I am still missing something but due to my lack of knowledge I still can't figure out what.
Would someone familiar with Windows be able to point out what's missing?
Thanks
in linux:
add export export PYTHONPATH=$PYTHONPATH:pwd:pwd/slim to ~/.bashrc
attention:you should keep single quote mark
if you work with windows, i guess it should like this:PYTHONPATH=$PYTHONPATH:'C:/tensorflow/models':'C:/tensorflow/models'/slim
just my guess, you can take a try.
good luck!
If you run the setup.py it will install all the relevant modules for object detection. The other option is download the git directory. cd to the folder and try to run the module from there. You might face protubuf issue. Try to install it before running the code. It's bit complicated to install protobuf in windows. But if you are not using ".pb" file, then you don't need to.
I figured out a way to make it work. I am not writing this as a final answer as it is mostly a workaround and due to lack of understanding from my part I cannot guarantee it will work (and also it might not be best good practice).
Anyway here it is:
As Beta previously suggested, you have to run setup.py, however running it from models folder did not do it for me, I also had to run it from object detection folder.
However there was a problem there, it generated an error saying the BUILD already existed (which was correct) so I had to delete the BUILD file from inside of model.
After that it worked, turns out the path I had set was working fine.
Now if some experts would look into this and explain how and why this workaround worked it might make this a valid solution.
Is there a way to have an easily shareable python install? I have a python install on my PC, and sometimes coworkers want to use and modify my code. To do so, they often need to replicate my python config. Is there a way I could easily share my config with them? From another angle, is there a way I could send my code so it can be run with a clean install of Python?
Have you tried using virtualenv ? I think it will meet your need to share your python environment after you have customized it to run your code.
As #Jeff Mercado mentionned, Portable Python is the way to go for me. I can zip my whole python install into one file and share it. It contains all the libraries I need.