I've written a python script using scrapy to crawl a site, and I'm trying to set up a job through jenkins to call the script nightly (this way it's very easy to see the output).
The machine I'm running jenkins on is a bitnami VM inside google compute.
I set up the command to run through the shell command in jenkins, and it's failing with the following error:
Building on master in workspace /opt/bitnami/apps/jenkins/jenkins_home/jobs/Scrape and Import myco/workspace [workspace] $ /bin/sh -xe /opt/bitnami/apache-tomcat/temp/hudson4165433582945317339.sh
+ /usr/local/myco/myscript.py -l /usr/local/myco/results/7.log -o /usr/local/myco/results/7.json -s /usr/local/myco/results/7.stats myspider
Traceback (most recent call last): File "/usr/local/myco/myscript.py", line 5, in <module>
from twisted.internet import reactor File "/usr/local/lib/python2.7/dist-packages/twisted/internet/reactor.py", line 38, in <module>
from twisted.internet import default File "/usr/local/lib/python2.7/dist-packages/twisted/internet/default.py", line 56, in <module>
install = _getInstallFunction(platform) File "/usr/local/lib/python2.7/dist-packages/twisted/internet/default.py", line 44, in _getInstallFunction
from twisted.internet.epollreactor import install File "/usr/local/lib/python2.7/dist-packages/twisted/internet/epollreactor.py", line 24, in <module>
from twisted.internet import posixbase File "/usr/local/lib/python2.7/dist-packages/twisted/internet/posixbase.py", line 23, in <module>
from twisted.internet import error, udp, tcp File "/usr/local/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 29, in <module>
from twisted.internet._newtls import ( File "/usr/local/lib/python2.7/dist-packages/twisted/internet/_newtls.py", line 21, in <module>
from twisted.protocols.tls import TLSMemoryBIOFactory, TLSMemoryBIOProtocol File "/usr/local/lib/python2.7/dist-packages/twisted/protocols/tls.py", line 41, in <module>
from OpenSSL.SSL import Error, ZeroReturnError, WantReadError File "/usr/local/lib/python2.7/dist-packages/OpenSSL/__init__.py", line 8, in <module>
from OpenSSL import rand, crypto, SSL File "/usr/local/lib/python2.7/dist-packages/OpenSSL/rand.py", line 11, in <module>
from OpenSSL._util import ( File "/usr/local/lib/python2.7/dist-packages/OpenSSL/_util.py", line 7, in <module>
binding = Binding() File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 114, in __init__
self._ensure_ffi_initialized() File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 126, in _ensure_ffi_initialized
cls._modules, File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/bindings/utils.py", line 31, in load_library_for_binding
lib = ffi.verifier.load_library() File "/usr/local/lib/python2.7/dist-packages/cffi/verifier.py", line 97, in load_library
return self._load_library() File "/usr/local/lib/python2.7/dist-packages/cffi/verifier.py", line 207, in _load_library
return self._vengine.load_library() File "/usr/local/lib/python2.7/dist-packages/cffi/vengine_cpy.py", line 155, in load_library
raise ffiplatform.VerificationError(error) cffi.ffiplatform.VerificationError: importing '/usr/local/lib/python2.7/dist-packages/cryptography/_Cryptography_cffi_e7d09016xc302a38b.so': /usr/local/lib/python2.7/dist-packages/cryptography/_Cryptography_cffi_e7d09016xc302a38b.so: symbol EC_GFp_nistp521_method, version OPENSSL_1.0.1 not defined in file libcrypto.so.1.0.0 with link time reference Build step 'Execute shell' marked build as failure Finished: FAILURE
I'm perplexed because when I run the same command (as my user and the user jenkins is running under, tomcat) I don't get this error, the script works fine.
I suspect this may have to do with the script being executed inside apache, but I'm at my wits end and googling hasn't turned up any obvious solutions.
Any idea as to how to solve this?
symbol EC_GFp_nistp521_method, version OPENSSL_1.0.1 not defined in file libcrypto.so.1.0.0
It looks like you are running a python compiled with OpenSSL 1.0.1 with a libcrypto from OpenSSL 1.0.0. It might be because you are running with a different python (at least compiled against a different OpenSSL version) but include files from your local python installation which expects the newer OpenSSL.
Related
I'm trying to install the transformers library on HPC. I do:
git clone https://github.com/huggingface/transformers.git
cd transformers
pip install -e . --user
All three of these work as expected, with the last output being:
Successfully installed dataclasses-0.7 numpy-1.19.0 tokenizers-0.8.1rc2 transformers
Then, I try python -c "import transformers" but I get the following error:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/maths/btech/mt1170727/transformers/src/transformers/__init__.py", line 23, in <module>
from .configuration_albert import ALBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, AlbertConfig
File "/home/maths/btech/mt1170727/transformers/src/transformers/configuration_albert.py", line 18, in <module>
from .configuration_utils import PretrainedConfig
File "/home/maths/btech/mt1170727/transformers/src/transformers/configuration_utils.py", line 25, in <module>
from .file_utils import CONFIG_NAME, cached_path, hf_bucket_url, is_remote_url
File "/home/maths/btech/mt1170727/transformers/src/transformers/file_utils.py", line 37, in <module>
import torch
File "/home/soft/PYTHON/3.6.0/ucs4/gnu/447/lib/python3.6/site-packages/torch/__init__.py", line 125, in <module>
_load_global_deps()
File "/home/soft/PYTHON/3.6.0/ucs4/gnu/447/lib/python3.6/site-packages/torch/__init__.py", line 83, in _load_global_deps
ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
File "/home/soft/PYTHON/3.6.0/ucs4/gnu/447/lib/python3.6/ctypes/__init__.py", line 344, in __init__
self._handle = _dlopen(self._name, mode)
OSError: libnvToolsExt.so.1: cannot open shared object file: No such file or directory
I have done as was written in the documentation, and can't see why I'm facing this error. Any help would be great. Thanks...
I made a new environment where I installed geopandas, xarray, and regionmask. When I tried to switch environment using the modular approach (by installing spyder-kernels in the new environment and indicating the new path to the environment in the Python interpreter, as indicated here: https://github.com/spyder-ide/spyder/wiki/Working-with-packages-and-environments-in-Spyder#the-modular-approach), an error persists while starting the kernel:
Traceback (most recent call last):
File "C:\Users\Justin\anaconda3\envs\geo_env\lib\runpy.py", line 193, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\Justin\anaconda3\envs\geo_env\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\Justin\anaconda3\envs\geo_env\lib\site‑packages\spyder_kernels\console\__main__.py", line 11, in
start.main()
File "C:\Users\Justin\anaconda3\envs\geo_env\lib\site‑packages\spyder_kernels\console\start.py", line 288, in main
import_spydercustomize()
File "C:\Users\Justin\anaconda3\envs\geo_env\lib\site‑packages\spyder_kernels\console\start.py", line 39, in import_spydercustomize
import spydercustomize
File "C:\Users\Justin\anaconda3\envs\geo_env\lib\site‑packages\spyder_kernels\customize\spydercustomize.py", line 27, in
from spyder_kernels.comms.frontendcomm import CommError, frontend_request
File "C:\Users\Justin\anaconda3\envs\geo_env\lib\site‑packages\spyder_kernels\comms\frontendcomm.py", line 17, in
from jupyter_client.localinterfaces import localhost
File "C:\Users\Justin\anaconda3\envs\geo_env\lib\site‑packages\jupyter_client\__init__.py", line 4, in
from .connect import *
File "C:\Users\Justin\anaconda3\envs\geo_env\lib\site‑packages\jupyter_client\connect.py", line 21, in
import zmq
File "C:\Users\Justin\anaconda3\envs\geo_env\lib\site‑packages\zmq\__init__.py", line 47, in
from zmq import backend
File "C:\Users\Justin\anaconda3\envs\geo_env\lib\site‑packages\zmq\backend\__init__.py", line 40, in
reraise(*exc_info)
File "C:\Users\Justin\anaconda3\envs\geo_env\lib\site‑packages\zmq\utils\sixcerpt.py", line 34, in reraise
raise value
File "C:\Users\Justin\anaconda3\envs\geo_env\lib\site‑packages\zmq\backend\__init__.py", line 27, in
_ns = select_backend(first)
File "C:\Users\Justin\anaconda3\envs\geo_env\lib\site‑packages\zmq\backend\select.py", line 28, in select_backend
mod = __import__(name, fromlist=public_api)
File "C:\Users\Justin\anaconda3\envs\geo_env\lib\site‑packages\zmq\backend\cython\__init__.py", line 6, in
from . import (constants, error, message, context,
ImportError: DLL load failed while importing error: The specified module could not be found.
How can I debug this?
(Spyder maintainer here) This problem was fixed in this pull request, which is going to be part of Spyder 4.1.0, to be released later in March 2020.
I have created a virtual env on MacOS 10.14, with python 2.7, pip installed py2app, wxpython, etc, ran the Py2app command:
sudo python setup.py py2app
Upon running the app, I see the below error:
(ENV) kevins-MacBook-Pro:Audio_Export_App kevin$ /Users/kevin/python_scripts/Audio_Export_App/dist/Audio_Export_App_V3.3.app/Contents/MacOS/Audio_Export_App_V3.3
Traceback (most recent call last):
File "/Users/kevin/python_scripts/Audio_Export_App/dist/Audio_Export_App_V3.3.app/Contents/Resources/__boot__.py", line 107, in <module>
_run()
File "/Users/kevin/python_scripts/Audio_Export_App/dist/Audio_Export_App_V3.3.app/Contents/Resources/__boot__.py", line 82, in _run
exec(compile(source, path, 'exec'), globals(), globals())
File "/Users/kevin/python_scripts/Audio_Export_App/dist/Audio_Export_App_V3.3.app/Contents/Resources/Audio_Export_App_V3.3.py", line 1, in <module>
import wx
File "wx/__init__.pyc", line 17, in <module>
File "wx/core.pyc", line 12, in <module>
File "wx/_core.pyc", line 14, in <module>
File "wx/_core.pyc", line 10, in __load
ImportError: dlopen(/Users/kevin/python_scripts/Audio_Export_App/dist/Audio_Export_App_V3.3.app/Contents/Resources/lib/python2.7/lib-dynload/wx/_core.so, 2): Symbol not found: __ZThn1216_N10wxTextCtrl3CutEv
Referenced from: /Users/kevin/python_scripts/Audio_Export_App/dist/Audio_Export_App_V3.3.app/Contents/Resources/lib/python2.7/lib-dynload/wx/_core.so
Expected in: /usr/local/lib/libwx_osx_cocoau_core-3.0.0.4.0.dylib
in /Users/kevin/python_scripts/Audio_Export_App/dist/Audio_Export_App_V3.3.app/Contents/Resources/lib/python2.7/lib-dynload/wx/_core.so
2018-10-17 07:58:27.927 Audio_Export_App_V3.3[10303:567300] Audio_Export_App_V3.3 Error
2018-10-17 07:58:28.070 Audio_Export_App_V3.3[10303:567300] GetInputSourceEnabledPrefs user file path = /Users/kevin/Library/Preferences/com.apple.HIToolbox.plist
2018-10-17 07:58:28.070 Audio_Export_App_V3.3[10303:567300] GetInputSourceEnabledPrefs effective user id path = 501
2018-10-17 07:58:28.070 Audio_Export_App_V3.3[10303:567300] GetInputSourceEnabledPrefs user pref content = <CFBasicHash 0x10017c320 [0x7fff9887c8e0]>{type = immutable dict, count = 0,
entries =>
}
I have searched high and low for a solution, but cannot find one.
Anyone able to shed some light on this issue please?
This is an issue with Py2App. It can be solved by manually specifying the wx package during the build process:
python setup.py py2app --packages=wx
Hope that helps.
Source: https://bitbucket.org/ronaldoussoren/py2app/issues/252/py2app-creates-broken-bundles-with
I have been trying to install SerpentAI. I am on a Mac, and I have followed all the steps. I have all the dependencies, but when I use the keyword serpent it gives me this error.
I know I am missing a config file but I don't know where to find it, or how to solve this.
Any command that started with serpent gave me this error.
Here is the error
Traceback (most recent call last):
File "/anaconda3/bin/serpent", line 11, in <module>
load_entry_point('SerpentAI==2018.1.2', 'console_scripts', 'serpent')()
File "/anaconda3/lib/python3.6/site- packages/pkg_resources/__init__.py", line 480, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/anaconda3/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2691, in load_entry_point
return ep.load()
File "/anaconda3/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2322, in load
return self.resolve()
File "/anaconda3/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2328, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/anaconda3/lib/python3.6/site-packages/SerpentAI-2018.1.2-py3.6.egg/serpent/serpent.py", line 11, in <module>
from serpent.utilities import clear_terminal, display_serpent_logo, is_linux, is_macos, is_windows, is_unix, wait_for_crossbar
File "/anaconda3/lib/python3.6/site-packages/SerpentAI-2018.1.2-py3.6.egg/serpent/utilities.py", line 8, in <module>
from serpent.config import config
File "/anaconda3/lib/python3.6/site-packages/SerpentAI-2018.1.2-py3.6.egg/serpent/config.py", line 18, in <module>
raise Exception("Configuration file not found at: 'config/config.yml'...")
Exception: Configuration file not found at: 'config/config.yml'...
So I think I found the problem, you can clone the GitHub repo, and it will have the config files. Then when you run pip install it will clone a version, and that version won't have the config files.
I had the same issue when trying to run serpent vi sual_debugger after following all installation guide steps.
Try to run serpent setup, it might solve the problem you have. It helped in my case.
Using cx_freeze under ubuntu 12.04. The generation of the package is
done without any errors but when I try to run the package as a
standalone, I get the following error:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/cx_Freeze/initscripts /Console.py",
line 27, in <module>
exec code in m.__dict__
m m File "Setup_cx_freeze.py", line 3, in <module>
File "/usr/local/lib/python2.7/dist-packages/cx_Freeze/__init__.py",
line 4, in <module>
from cx_Freeze.dist import *
File "/usr/local/lib/python2.7/dist-packages/cx_Freeze/dist.py",
line 3, in <module>
import distutils.command.install
File "/usr/lib/python2.7/distutils/command/install.py", line 21, in <module>
from site import USER_BASE
File "/usr/lib/python2.7/site.py", line 562, in <module>
main()
File "/usr/lib/python2.7/site.py", line 540, in main
abs__file__()
File "/usr/lib/python2.7/site.py", line 100, in abs__file__
m.__file__ = os.path.abspath(m.__file__)
File "/usr/lib/python2.7/email/__init__.py", line 79, in __getattr__
__import__(self.__name__)
ImportError: No module named image
I think what to focus on is the last sentence. "ImportError: No module named image."
The application runs without any problems in pure python environment
So why do I get the Error when running it as a "freeze" package? The
fault is somehow introduced when generating the freeze-package.
I ended up fixing this by explicitly including the package "email.mime" in the setup.py script build exe options.
Missing 'image' is just first part of the problem. Try to add this to your code:
from email.mime.image import MIMEImage
from email.mime.audio import MIMEAudio
from email.mime.message import MIMEMessage
This resolved problems with 'image', 'audio' and 'message'.
Resolved this by installing Python Imaging Library: http://www.pythonware.com/products/pil/
1) python setup.py build
2) sudo python setup.py install