I am using cx_Oracle library in my python script. My code works fine if I directly execute by python script from Linux terminal but when I put it in crontab I am getting following error.
!!DatabaseError: DPI-1047: Oracle Client library cannot be loaded: libclntsh.so: cannot open shared object file: No such file or directory. See https://oracle.github.io/odpi/doc/installation.html for help
DPI-1005: unable to acquire Oracle environment handle
!!Traceback (most recent call last):
import cx_Oracle
!!DatabaseError: DPI-1005: unable to acquire Oracle environment handle
I googled this issue and It seems some environment variable missing when it ran using crontab.
I try to export following in crontab but it does not work.
export LD_LIBRARY_PATH='/usr/lib/oracle/11.2/client64/lib'
If I remove cx_Oracle package other code runs fine. I have only one version of python installed on my machine.
What is your version of Python? Is it 32-bit or 64-bit?
Python 2.6 . 64 bit
What is your version of cx_Oracle?
Version 6.0b1
What is your version of the Oracle client (e.g. Instant Client)? How was it
installed? Where is it installed?
oracle-instantclient11.2-devel-11.2.0.4.0-1.x86_64.rpm
What is your OS and version?
CentOS 6.7
What environment variables did you set? How exactly did you set them?
export LD_LIBRARY_PATH='/usr/lib/oracle/11.2/client64/lib'
Evidently cron doesn't load the bash profile so you will need write a wrapper.
So, write a bash wrapper that exports the variables needed and calls the script. Then call that wrapper from crontab. When that works, you will know that your variables weren't being properly exported for the user whose crontab you edited.
Note: You can add them into /etc/bashrc and it will then be in place for all users, if you have root access.
You can also make a generic wrapper and cron things be sending them through the wrapper.
my_bash_wrapper.sh
#!/bin/bash
. ~/.bash_profile
"$0"
evoke in cron:
0 1 * * * /my/loc/my_bash_wrapper.sh my_python_script arg1 arg2
You should set ORACLE_HOME and LD_LIBRARY_PATH in your bash script.
#!/bin/bash
export ORACLE_HOME=/usr/lib/oracle/<version>/client(64)
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH
It's not working since its looking for the oracle instant client path.
In the shell script, add the export line and point it to your oracle client path. Next, keep writing your shell script. This will work!
#!/bin/sh
export LD_LIBRARY_PATH=/home/<user_name>/opt/oracle/instantclient_19_5:$LD_LIBRARY_PATH
/usr/anaconda3/bin/python /home/<user_name>/test/src/test.py
It may work for some user
I am using Gnome Schedular to schedule python scipts.
For Normal jobs.
~anaconda3/bin/python /path_to_script/target_script.py
For Job which needs environement variable.
. ~/bash_profile ~anaconda3/bin/python /path_to_script/target_script.py
Related
poetry has a feature for creating a shell script to set up the environment and launch the local python package. The following directive in pyproject.toml generates a shell script hercl that has the needed env including python path/libraries to run the hercl.hercl.py main() :
[tool.poetry.scripts]
hercl = "hercl.hercl:main"
I have heard rumors that pip itself has a similar capability. I have been unable to find that feature: does it really exist?
Oh! It looks like pip supports pyproject.toml directly! It is called the Build System Interface It is apparently a newer approach in conjunction with setup.py. I'll get back here on how that goes.
I have setup multiple python environment using Anaconda.
Usually, to run a script "manually", I would open a command line and then type:
activate my-env
python path/to/my/script.py
Fine.
Now I am trying to run a script automatically using a scheduler and I was wondering what the difference was between
Writing a batch which activates the environment and the executes the script (like in the snippet above)
Calling directly the python executable from the environment (within the envs/my-enjv/ directory) like below:
/path/to/envs/my-env/python.exe path/to/my/script.py
Both seem to work fine. Is there any difference?
I don't claim to be an expert but here's my 2 cents.
For small scripts, no, there isn't a difference.
You should notice a difference when calling external modules / packages. conda activate alters the system path to change how the command shell searches for the appropriate capabilities.
If you supply a full path to an interpreter and the full path to an isolated script, then the shell doesn't need to do a lookup as this has priority over the path. This means you could be in a situation where the interpreter can see the script but cannot see dependencies.
If you follow the conda activate process, and the environment is correctly packaged, then the shell will be able to trace any additional resources.
EDIT: The idea behind this is portability. If an admin has been careful in setting up a system, then scripts should have the appropriate visibility - i.e. see everything in it's environment plus everything in the main system installation.
It's possible to full-path every call to an interpreter and a script or package location, but then what happens when you need to move it to another machine? You would need to spend a lot of time setting everything up exactly as it was before. On the other hand, you can follow the package process and the system path will trace everything for you.
Simply checkout the PATH variable in your environment. After conda activation it has been extended by
\Anaconda3;
\Anaconda3\Library\mingw-w64\bin;
\Anaconda3\Library\usr\bin;
\Anaconda3\Library\bin;
\Anaconda3\Scripts;
\Anaconda3\bin;
This doesn't make much of a difference, if you are just using the standard library in your code. However, if you rely on external packages like pandas, it's a prerequisite so that the modules can be found.
I am trying to solve an issue with PyCharm that is literally driving me crazy. I hope I will get some help here.
I am running anaconda with python 3.5 on mac os. I need to use an oracle database and I have installed the cx_Oracle package using the official instructions with "pip install".
I wrote a piece of code to test the connection with the database. If I run the code from the command line, it works fine, but PyCharm does not seem to like it. The GUI seems to not find the package, as there is a red line under the import saying "No module named cx_Oracle". But I can find the module listed with the other packages in "Settings->Project->Project Interpreter". If I run it anyway, in PyCharm, I get the following error:
import cx_Oracle
ImportError: dlopen(/Users/myuser/anaconda/lib/python3.5/site-packages/cx_Oracle.cpython-35m-darwin.so, 2): Library not loaded: #rpath/libclntsh.dylib.12.1
Referenced from: /Users/myuser/anaconda/lib/python3.5/site-packages/cx_Oracle.cpython-35m-darwin.so
Reason: image not found
The project interpreter is the same as the one I am using with the command line, I have checked that with the "which" command.
sys.executable, os.getcwd() and sys.pat, are all the same in both PyCharm and from command line.
The only difference seems to be in os.environ. In order to have the cx_Oracle plugin working from command line I had to set two environment variables DYLD_LIBRARY_PATH and LD_LIBRARY_PATH. If I don't set those I will get the same error with the command line too. Hence in PyCharm I have added the following lines at the beginning of my python script.
os.environ["DYLD_LIBRARY_PATH"]="/path/to/my/library"
os.environ["LD_LIBRARY_PATH"]="/path/to/my/library"
Unfortunately I still get the error if I run the code from PyCharm.
I fix similar problem by following cmd:
ln -s /opt/oracle/instantclient_12_1/libclntsh.dylib.12.1 /usr/local/lib/libclntsh.dylib.12.1
You need to make sure that DYLD_LIBRARY_PATH/LD_LIBRARY_PATH points to the location in which the Oracle client library (libclntsh) is found. Since PyCharm is loading it you need to make sure those environment variables are set prior to launching PyCharm itself, as they do not take effect except at process startup.
Another option is to set the environment variable FORCE_RPATH to some value prior to building cx_Oracle and installing it. This will remove the need for setting the DYLD_LIBRARY_PATH and LD_LIBRARY_PATH environment variables at runtime.
I'm trying to use a custom self-compiled Python version installed under "/opt" from an Upstart script on Ubuntu 14.04. General setup looks like following:
The service configuration file defines a pre-start script which calls some bash script from a user (non-root) directory
This bash script defines a trap error function calling some Python code
The trap error function is defined as
function error_handler()
{
python << END
# python code...
END
}
trap 'error_handler ${LINENO} $?' ERR
Any 3rd party side-packages are installed under "/opt" as well using pip from custom Python installation
PATH env variable for this non-root user includes a binary folder under /opt where custom Python could be found
the ".bashrc" file was modified to be executed for non-interactice log-ins
I have also tried to use a ".bash_profile" file and just called ".bashrc" from it
In the log files under "/var/log/upstart/service.log", I see errors messages saying that any installed 3rd party Python side-package couldn't be found. These packages are installed under "/opt"
The custom Python installation itself works as expected. I can run any Python code and import any of the installed side-packages.
What I have tried so far to get it working as a service:
Update the PATH env varibale for a given user
Call the bash script from the "pre-start" section using the sudo -u user_name command
Modify the PATH env variable inside of the bash script:
PATH=/opt/bin:$PATH
export PATH
function error_handler()
{
PATH="$PATH" python << END
# python code...
END
}
trap 'error_handler ${LINENO} $?' ERR
Set PYTHONHOME and PYTHONPATH env variables
Update the library search path from the Python code using sys.path.append before importing any of the 3rd party side-packages
Here is my question: How can I convince the Startup service to pick up the Python version I want to run and to use packages installed under "/opt"?
Basically, I want to use Python 2.7.9 on Ubuntu 14.04. I couldn't find any pre-compiled Ubuntu 14.04 package so far.
We were able to use our self-compiled Python by adding the setuid call to the service configuration and by a proper usage of the Python virtual environments.
Oleg
I have EPD. I would like to use the ipython shell. I think it's called 'pylab' in your distribution. I would like to be able to append to the system path ($pythonpath) at the moment when the shell loads. Unfortunately, I can not seem to locate the ipy_user_conf.py file that many users on the internet report is where I need to include a line to do that. Please help!!!
I'm going to guess that you're running IPython >= 0.11. (Note the IPython version is displayed when starting up the IPython shell.)
Older versions of IPython (pre-0.11) used ipy_user_conf.py, but IPython's configuration system was overhauled in 0.11. For details, see this overview. If you want to run some code on start up, you can add a python file in your IPython startup directory, which should be here:
~/.ipython/profile_default/startup/
Any python code in that directory gets run on startup, so you can just create a new .py to modify your python path. If the startup directory doesn't exist, you may need to run:
ipython profile create
which creates those directories (plus some other goodies).