I have a bundle com.elsten.bliss.platform. Here are the wirings before update:
g! inspect capability osgi.wiring.package 59
com.elsten.bliss.platform [59] provides:
----------------------------------------
osgi.wiring.package; com.elsten.bliss.client.impl 1.1.0 required by:
com.elsten.bliss.ui [33]
osgi.wiring.package; com.elsten.bliss.licence 1.0.1 required by:
com.elsten.bliss.ui [33]
osgi.wiring.package; com.elsten.bliss.main 2.0.5 required by:
com.elsten.bliss.ui.systemtray [23]
com.elsten.bliss.api [40]
osgi.wiring.package; com.elsten.bliss.music.policy.config 1.0.0 required by:
com.elsten.bliss.ui [33]
com.elsten.bliss.configuration.file [72]
osgi.wiring.package; com.elsten.bliss.music.policy.coverartpolicy 1.1.0 required by:
com.elsten.bliss.ui [33]
osgi.wiring.package; com.elsten.bliss.music.policy.coverartpolicy.assessment 1.0.0 required by:
com.elsten.bliss.ui [33]
osgi.wiring.package; com.elsten.bliss.music.policy.coverartpolicy.response 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.policy.fileorg 1.1.0 required by:
com.elsten.bliss.ui [33]
osgi.wiring.package; com.elsten.bliss.music.policy.fileorg.pattern 1.0.0 required by:
com.elsten.bliss.ui [33]
osgi.wiring.package; com.elsten.bliss.music.policy.fileorg.pattern.token 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.policy.tag.auto 1.0.0 required by:
com.elsten.bliss.untagged.acoustid [52]
osgi.wiring.package; com.elsten.bliss.music.policy.tagsync 1.0.0 required by:
jdbm-trunk-patched [20]
com.elsten.bliss.ui [33]
osgi.wiring.package; com.elsten.bliss.music.policy.tagsync.allowablegenres 1.0.0 required by:
com.elsten.bliss.ui [33]
osgi.wiring.package; com.elsten.bliss.music.policy.tagsync.consolidateartist 1.0.1 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.policy.tagsync.response 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.policy.tagsync.trackartist 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.policy.tagsync.tracknumberformat 1.0.0 required by:
jdbm-trunk-patched [20]
osgi.wiring.package; com.elsten.bliss.music.policy2 1.0.1 required by:
com.elsten.bliss.ui [33]
com.elsten.bliss.configuration.file [72]
osgi.wiring.package; com.elsten.bliss.platform.storage.bus 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.platform.storage.file.jaudiotagger 1.0.0 required by:
jdbm-trunk-patched [20]
osgi.wiring.package; com.elsten.bliss.platform.tagindex 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.platform.util2 1.0.0 required by:
com.elsten.bliss.web.debug [14]
osgi.wiring.package; com.elsten.bliss.untagged2 1.0.0 required by:
com.elsten.bliss.untagged.acoustid [52]
com.elsten.bliss.ui [33]
osgi.wiring.package; com.elsten.bliss.platform.activity 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.platform2 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.platform.storage.file 1.0.0 required by:
com.elsten.bliss.configuration.file [72]
osgi.wiring.package; com.elsten.bliss.music.conceptual 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.tagged.mapper 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.platform.storage.file.jaudiotagger.tag 28.1.2 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.policy.tagsync.regex 28.1.2 [UNUSED]
osgi.wiring.package; com.elsten.bliss.client.platform 1.0.0 [UNUSED]
And services:
g! inspect capability service 59
com.elsten.bliss.platform [59] provides:
----------------------------------------
service; com.elsten.bliss.client.Client with properties:
service.id = 31
Used by:
com.elsten.bliss.ui [33]
service; com.elsten.bliss.licence.PerFixLicenceStore with properties:
service.id = 32
Used by:
com.elsten.bliss.ui [33]
service; com.elsten.bliss.music.policy2.PerFixLicensedPolicyCommandExecutor with properties:
service.id = 33
service; com.elsten.bliss.main.Platform with properties:
component.id = 5
component.name = com.elsten.bliss.main.PlatformImpl
coverArtLookupStrategy.target = (rootLookupStrategy=true)
informationSource.target = (getTags=true)
service.id = 34
Used by:
com.elsten.bliss.api [40]
com.elsten.bliss.ui.systemtray [23]
Now I update, which updates the bundle and a number of bundles besides. I perform this with a management agent coded by myself which uses OBR to update the new bundle versions and then calls refresh.
After this occurs, I set a breakpoint in com.elsten.bliss.main.PlatformImpl so I can test the wirings again. I expect that the old com.elsten.bliss.main.Platform has been removed. However:
g! lb platform
START LEVEL 1
ID|State |Level|Name
59|Active | 1|com.elsten.bliss.platform (28.1.2)
84|Resolved | 1|com.elsten.bliss.platform (28.1.3)
Now there are two versions! So I print the wiring (at this stage the services are not wired so no point in including them):
g! inspect capability osgi.wiring.package 59
com.elsten.bliss.platform [59] provides:
----------------------------------------
osgi.wiring.package; com.elsten.bliss.client.impl 1.1.0 required by:
com.elsten.bliss.ui [33]
osgi.wiring.package; com.elsten.bliss.licence 1.0.1 required by:
com.elsten.bliss.ui [33]
com.elsten.bliss.platform [84]
osgi.wiring.package; com.elsten.bliss.main 2.0.5 required by:
com.elsten.bliss.ui.systemtray [23]
com.elsten.bliss.api [40]
osgi.wiring.package; com.elsten.bliss.music.policy.config 1.0.0 required by:
com.elsten.bliss.ui [33]
com.elsten.bliss.configuration.file [72]
osgi.wiring.package; com.elsten.bliss.music.policy.coverartpolicy 1.1.0 required by:
com.elsten.bliss.ui [33]
osgi.wiring.package; com.elsten.bliss.music.policy.coverartpolicy.assessment 1.0.0 required by:
com.elsten.bliss.ui [33]
osgi.wiring.package; com.elsten.bliss.music.policy.coverartpolicy.response 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.policy.fileorg 1.1.0 required by:
com.elsten.bliss.ui [33]
osgi.wiring.package; com.elsten.bliss.music.policy.fileorg.pattern 1.0.0 required by:
com.elsten.bliss.ui [33]
osgi.wiring.package; com.elsten.bliss.music.policy.fileorg.pattern.token 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.policy.tag.auto 1.0.0 required by:
com.elsten.bliss.untagged.acoustid [52]
osgi.wiring.package; com.elsten.bliss.music.policy.tagsync 1.0.0 required by:
com.elsten.bliss.ui [33]
osgi.wiring.package; com.elsten.bliss.music.policy.tagsync.allowablegenres 1.0.0 required by:
com.elsten.bliss.ui [33]
osgi.wiring.package; com.elsten.bliss.music.policy.tagsync.consolidateartist 1.0.1 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.policy.tagsync.response 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.policy.tagsync.trackartist 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.policy.tagsync.tracknumberformat 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.policy2 1.0.1 required by:
com.elsten.bliss.ui [33]
com.elsten.bliss.configuration.file [72]
osgi.wiring.package; com.elsten.bliss.platform.storage.bus 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.platform.storage.file.jaudiotagger 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.platform.tagindex 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.platform.util2 1.0.0 required by:
com.elsten.bliss.web.debug [14]
osgi.wiring.package; com.elsten.bliss.untagged2 1.0.0 required by:
com.elsten.bliss.untagged.acoustid [52]
com.elsten.bliss.ui [33]
osgi.wiring.package; com.elsten.bliss.platform.activity 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.platform2 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.platform.storage.file 1.0.0 required by:
com.elsten.bliss.configuration.file [72]
osgi.wiring.package; com.elsten.bliss.music.conceptual 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.tagged.mapper 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.platform.storage.file.jaudiotagger.tag 28.1.2 [UNUSED]
osgi.wiring.package; com.elsten.bliss.client.platform 1.0.0 [UNUSED]
And the new bundle:
g! inspect capability osgi.wiring.package 84
com.elsten.bliss.platform [84] provides:
----------------------------------------
osgi.wiring.package; com.elsten.bliss.client.impl 1.1.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.main 2.0.5 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.policy.config 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.policy.coverartpolicy 1.1.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.policy.coverartpolicy.assessment 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.policy.coverartpolicy.response 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.policy.fileorg 1.1.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.policy.fileorg.pattern 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.policy.fileorg.pattern.token 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.policy.tag.auto 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.policy.tagsync 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.policy.tagsync.allowablegenres 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.policy.tagsync.consolidateartist 1.0.1 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.policy.tagsync.response 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.policy.tagsync.trackartist 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.policy.tagsync.tracknumberformat 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.policy2 1.0.1 [UNUSED]
osgi.wiring.package; com.elsten.bliss.platform.storage.bus 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.platform.storage.file.jaudiotagger 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.platform.tagindex 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.platform.util2 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.untagged2 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.platform.activity 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.platform2 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.platform.storage.file 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.conceptual 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.tagged.mapper 1.0.0 [UNUSED]
osgi.wiring.package; com.elsten.bliss.platform.storage.file.jaudiotagger.tag 28.1.3 [UNUSED]
osgi.wiring.package; com.elsten.bliss.music.policy.tagsync.regex 28.1.3 required by:
com.elsten.bliss.platform [59]
osgi.wiring.package; com.elsten.bliss.client.platform 1.0.0 [UNUSED]
So the new bundle is barely being used for anything.
My suspicion is that it's something to do with the withdrawal of the com.elsten.bliss.licence package. This was removed, but note that this always existed (before the update):
g! lb licence
START LEVEL 1
ID|State |Level|Name
69|Active | 1|com.elsten.bliss.licence (1.0.1)
g! inspect capability osgi.wiring.package 69
com.elsten.bliss.licence [69] provides:
---------------------------------------
osgi.wiring.package; com.elsten.bliss.licence 1.0.1 [UNUSED]
So this is an interesting case: I had a split package before update which I didn't know about. However, dependencies for that split package were wired to a bundle that got updated. Subsequent to the bundle update, they are still wired to the old bundle. So does this stop those bundles being re-wired to the extant com.elsten.bliss.licence?
A new idea: before update, ensure that all bundles except for the bootstrap bundle are not started. Then, start Felix with just the bootstrap bundle, check the package wiring, then perform the update.
The results: another pointer to the split com.elsten.bliss.licence package being the problem. After the update, there are again two com.elsten.bliss.platform bundles. However, there is only one wiring on the old version:
osgi.wiring.package; com.elsten.bliss.licence 1.0.1 required by:
com.elsten.bliss.platform [84]
The new version is now dependent on the old version because of the removed package!
Any advice (including don't have split packages you dimwit!) accepted ;)
Not sure I understand. You have the bundle installed twice (59 and 84). So you did not update 59 with a new jar. So it seems you did not do a bundle update on 59. You just installed the bundle again (84). So this would explain why 59 is still present and probably still in use after the refresh.
Related
When running
bin/elasticsearch-plugin install ingest-attachment
I see:
org.elasticsearch.plugins.PluginCli.main(PluginCli.java:48) Caused by: java.lang.IllegalStateException: jar hell! class: org.slf4j.helpers.BasicMarker
jar1: /home/azureuser/elassandra-6.2.3.38/plugins/.installing-2974073780624375496/slf4j-api-1.6.2.jar
jar2: /home/azureuser/elassandra-6.2.3.38/lib/slf4j-api-1.7.25.jar
How do I resolve this and install an elasticsearch plugin for ingest-attachment that is compatible with elassandra?
Found the solution :)
From:
https://jarcasting.com/artifacts/com.strapdata.elasticsearch.plugin/ingest-attachment/
or even better try the official site:
https://search.maven.org/artifact/com.strapdata.elasticsearch.plugin/ingest-attachment/6.2.3.31/jar
and click downloads and then zip
Downloaded: ingest-attachment-6.2.3.31.zip
Then run:
bin/elasticsearch-plugin install file:"/home/sampleuser/ingest-attachment-6.2.3.31.zip"
I was trying to run dask-on-yarn with my research group's Hadoop cluster.
I tried each of the following instructions:
dd.read_parquet('hdfs://file.parquet', engine='fastparquet')
dd.read_parquet('hdfs://file.parquet', engine='pyarrow')
dd.read_csv('hdfs://file.csv')
Each time, the following error message occurs:
~/miniconda3/envs/dask/lib/python3.8/site-packages/fsspec/core.py in get_fs_token_paths(urlpath, mode, num, name_function, storage_options, protocol)
521 path = cls._strip_protocol(urlpath)
522 update_storage_options(options, storage_options)
--> 523 fs = cls(**options)
524
525 if "w" in mode:
~/miniconda3/envs/dask/lib/python3.8/site-packages/fsspec/spec.py in __call__(cls, *args, **kwargs)
52 return cls._cache[token]
53 else:
---> 54 obj = super().__call__(*args, **kwargs)
55 # Setting _fs_token here causes some static linters to complain.
56 obj._fs_token_ = token
~/miniconda3/envs/dask/lib/python3.8/site-packages/fsspec/implementations/hdfs.py in __init__(self, host, port, user, kerb_ticket, driver, extra_conf, **kwargs)
42 AbstractFileSystem.__init__(self, **kwargs)
43 self.pars = (host, port, user, kerb_ticket, driver, extra_conf)
---> 44 self.pahdfs = HadoopFileSystem(
45 host=host,
46 port=port,
~/miniconda3/envs/dask/lib/python3.8/site-packages/pyarrow/hdfs.py in __init__(self, host, port, user, kerb_ticket, driver, extra_conf)
38 _maybe_set_hadoop_classpath()
39
---> 40 self._connect(host, port, user, kerb_ticket, extra_conf)
41
42 def __reduce__(self):
~/miniconda3/envs/dask/lib/python3.8/site-packages/pyarrow/io-hdfs.pxi in pyarrow.lib.HadoopFileSystem._connect()
~/miniconda3/envs/dask/lib/python3.8/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()
OSError: Getting symbol hdfsNewBuilderfailed
How should I resolve this problem?
My Environment
Here are my packages in this conda env:
# Name Version Build Channel
_libgcc_mutex 0.1 main
abseil-cpp 20200225.2 he1b5a44_0 conda-forge
arrow-cpp 0.17.1 py38h1234567_9_cpu conda-forge
attrs 19.3.0 py_0
aws-sdk-cpp 1.7.164 hc831370_1 conda-forge
backcall 0.2.0 py_0
blas 1.0 mkl
bleach 3.1.5 py_0
bokeh 2.1.1 py38_0
boost-cpp 1.72.0 h7b93d67_1 conda-forge
brotli 1.0.7 he6710b0_0
brotlipy 0.7.0 py38h7b6447c_1000
bzip2 1.0.8 h7b6447c_0
c-ares 1.15.0 h7b6447c_1001
ca-certificates 2020.6.24 0
certifi 2020.6.20 py38_0
cffi 1.14.0 py38he30daa8_1
chardet 3.0.4 py38_1003
click 7.1.2 py_0
cloudpickle 1.4.1 py_0
conda-pack 0.4.0 py_0
cryptography 2.9.2 py38h1ba5d50_0
curl 7.71.0 hbc83047_0
cytoolz 0.10.1 py38h7b6447c_0
dask 2.19.0 py_0
dask-core 2.19.0 py_0
dask-yarn 0.8.1 py38h32f6830_0 conda-forge
decorator 4.4.2 py_0
defusedxml 0.6.0 py_0
distributed 2.19.0 py38_0
entrypoints 0.3 py38_0
fastparquet 0.3.2 py38heb32a55_0
freetype 2.10.2 h5ab3b9f_0
fsspec 0.7.4 py_0
gflags 2.2.2 he6710b0_0
glog 0.4.0 he6710b0_0
grpc-cpp 1.30.0 h9ea6770_0 conda-forge
grpcio 1.27.2 py38hf8bcb03_0
heapdict 1.0.1 py_0
icu 67.1 he1b5a44_0 conda-forge
idna 2.10 py_0
importlib-metadata 1.7.0 py38_0
importlib_metadata 1.7.0 0
intel-openmp 2020.1 217
ipykernel 5.3.0 py38h5ca1d4c_0
ipython 7.16.1 py38h5ca1d4c_0
ipython_genutils 0.2.0 py38_0
jedi 0.17.1 py38_0
jinja2 2.11.2 py_0
jpeg 9b h024ee3a_2
json5 0.9.5 py_0
jsonschema 3.2.0 py38_0
jupyter_client 6.1.3 py_0
jupyter_core 4.6.3 py38_0
jupyterlab 2.1.5 py_0
jupyterlab_server 1.1.5 py_0
krb5 1.18.2 h173b8e3_0
ld_impl_linux-64 2.33.1 h53a641e_7
libcurl 7.71.0 h20c2e04_0
libedit 3.1.20191231 h7b6447c_0
libevent 2.1.10 hcdb4288_1 conda-forge
libffi 3.3 he6710b0_1
libgcc-ng 9.1.0 hdf63c60_0
libgfortran-ng 7.3.0 hdf63c60_0
libllvm9 9.0.1 h4a3c616_0
libpng 1.6.37 hbc83047_0
libprotobuf 3.12.3 hd408876_0
libsodium 1.0.18 h7b6447c_0
libssh2 1.9.0 h1ba5d50_1
libstdcxx-ng 9.1.0 hdf63c60_0
libtiff 4.1.0 h2733197_1
llvmlite 0.33.0 py38hd408876_0
locket 0.2.0 py38_1
lz4-c 1.9.2 he6710b0_0
markupsafe 1.1.1 py38h7b6447c_0
mistune 0.8.4 py38h7b6447c_1000
mkl 2020.1 217
mkl-service 2.3.0 py38he904b0f_0
mkl_fft 1.1.0 py38h23d657b_0
mkl_random 1.1.1 py38h0573a6f_0
msgpack-python 1.0.0 py38hfd86e86_1
nbconvert 5.6.1 py38_0
nbformat 5.0.7 py_0
ncurses 6.2 he6710b0_1
notebook 6.0.3 py38_0
numba 0.50.1 py38h0573a6f_0
numpy 1.18.5 py38ha1c710e_0
numpy-base 1.18.5 py38hde5b4d6_0
olefile 0.46 py_0
openssl 1.1.1g h7b6447c_0
packaging 20.4 py_0
pandas 1.0.5 py38h0573a6f_0
pandoc 2.9.2.1 0
pandocfilters 1.4.2 py38_1
parquet-cpp 1.5.1 2 conda-forge
parso 0.7.0 py_0
partd 1.1.0 py_0
pexpect 4.8.0 py38_0
pickleshare 0.7.5 py38_1000
pillow 7.1.2 py38hb39fc2d_0
pip 20.1.1 py38_1
prometheus_client 0.8.0 py_0
prompt-toolkit 3.0.5 py_0
protobuf 3.12.3 py38he6710b0_0
psutil 5.7.0 py38h7b6447c_0
ptyprocess 0.6.0 py38_0
pyarrow 0.17.1 py38h1234567_9_cpu conda-forge
pycparser 2.20 py_0
pygments 2.6.1 py_0
pyopenssl 19.1.0 py38_0
pyparsing 2.4.7 py_0
pyrsistent 0.16.0 py38h7b6447c_0
pysocks 1.7.1 py38_0
python 3.8.3 hcff3b4d_2
python-dateutil 2.8.1 py_0
python_abi 3.8 1_cp38 conda-forge
pytz 2020.1 py_0
pyyaml 5.3.1 py38h7b6447c_1
pyzmq 19.0.1 py38he6710b0_1
re2 2020.07.01 he1b5a44_0 conda-forge
readline 8.0 h7b6447c_0
requests 2.24.0 py_0
send2trash 1.5.0 py38_0
setuptools 47.3.1 py38_0
six 1.15.0 py_0
skein 0.8.0 py38h32f6830_1 conda-forge
snappy 1.1.8 he6710b0_0
sortedcontainers 2.2.2 py_0
sqlite 3.32.3 h62c20be_0
tbb 2020.0 hfd86e86_0
tblib 1.6.0 py_0
terminado 0.8.3 py38_0
testpath 0.4.4 py_0
thrift 0.13.0 py38he6710b0_0
thrift-cpp 0.13.0 h62aa4f2_2 conda-forge
tk 8.6.10 hbc83047_0
toolz 0.10.0 py_0
tornado 6.0.4 py38h7b6447c_1
traitlets 4.3.3 py38_0
typing_extensions 3.7.4.2 py_0
urllib3 1.25.9 py_0
wcwidth 0.2.5 py_0
webencodings 0.5.1 py38_1
wheel 0.34.2 py38_0
xz 5.2.5 h7b6447c_0
yaml 0.2.5 h7b6447c_0
zeromq 4.3.2 he6710b0_2
zict 2.0.0 py_0
zipp 3.1.0 py_0
zlib 1.2.11 h7b6447c_3
zstd 1.4.4 h0b5b093_3
The Hadoop cluster is running version Hadoop 2.7.0-mapr-1607.
The Cluster object is created with:
# Create a cluster where each worker has two cores and eight GiB of memory
cluster = YarnCluster(
environment='conda-env-packed-for-worker-nodes.tar.gz',
worker_env={
# See https://github.com/dask/dask-yarn/pull/30#issuecomment-434001858
'ARROW_LIBHDFS_DIR': '/opt/mapr/hadoop/hadoop-0.20.2/c++/Linux-amd64-64/lib',
},
)
Suspected Cause
I suspect that the version mismatch between the hadoop-0.20.2 in the ARROW_LIBHDFS_DIR environmental variable and the hadoop CLI version Hadoop 2.7.0 might be a cause of the problem.
I had to manually specify pyarrow to use this file (using this setup: https://stackoverflow.com/a/62749053/1147061). The required file, libhdfs.so, is not provided under /opt/mapr/hadoop/hadoop-2.7.0/. Installing libhdfs3 via conda install -c conda-forge libhdfs3 does not resolve the requirement, either.
Might this be the problem?
(a part answer)
To use libhdfs3 (which is poorly maintained these days), you would need to call
dd.read_csv('hdfs://file.csv', storage_options={'driver': 'libhdfs3'})
and, of course, install libhdfs3. This will not help with the hadoop library option, they are independent code paths.
I also suspect that getting the JNI libhdfs (without the "3") working is a case of locating the right .so file.
$ composer update
Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.
Problem 1
- jenssegers/mongodb v3.3.1 requires mongodb/mongodb ^1.0.0 -> satisfiable by mongodb/mongodb[1.2.0, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.1.0, 1.1.1, 1.1.2].
- jenssegers/mongodb v3.3.0 requires mongodb/mongodb ^1.0.0 -> satisfiable by mongodb/mongodb[1.2.0, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.1.0, 1.1.1, 1.1.2].
- jenssegers/mongodb v3.3.1 requires mongodb/mongodb ^1.0.0 -> satisfiable by mongodb/mongodb[1.2.0, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.1.0, 1.1.1, 1.1.2].
- mongodb/mongodb 1.2.0 requires ext-mongodb ^1.3.0 -> the requested PHP extension mongodb is missing from your system.
- mongodb/mongodb 1.1.2 requires ext-mongodb ^1.2.0 -> the requested PHP extension mongodb is missing from your system.
- mongodb/mongodb 1.1.1 requires ext-mongodb ^1.2.0 -> the requested PHP extension mongodb is missing from your system.
- mongodb/mongodb 1.1.0 requires ext-mongodb ^1.2.0 -> the requested PHP extension mongodb is missing from your system.
- mongodb/mongodb 1.0.5 requires ext-mongodb ^1.1.0 -> the requested PHP extension mongodb is missing from your system.
- mongodb/mongodb 1.0.4 requires ext-mongodb ^1.1.0 -> the requested PHP extension mongodb is missing from your system.
- mongodb/mongodb 1.0.3 requires ext-mongodb ^1.1.0 -> the requested PHP extension mongodb is missing from your system.
- mongodb/mongodb 1.0.2 requires ext-mongodb ^1.1.0 -> the requested PHP extension mongodb is missing from your system.
- mongodb/mongodb 1.0.1 requires ext-mongodb ^1.1.0 -> the requested PHP extension mongodb is missing from your system.
- mongodb/mongodb 1.0.0 requires ext-mongodb ^1.1.0 -> the requested PHP extension mongodb is missing from your system.
- mongodb/mongodb 1.2.0 requires ext-mongodb ^1.3.0 -> the requested PHP extension mongodb is missing from your system.
- Installation request for jenssegers/mongodb ^3.3 -> satisfiable by jenssegers/mongodb[v3.3.0, v3.3.1].
To enable extensions, verify that they are enabled in your .ini files:
- /usr/local/etc/php/7.1/php.ini
- /usr/local/etc/php/7.1/conf.d/php-memory-limits.ini
You can also run `php --ini` inside terminal to see which files are used by PHP in CLI mode.
The problem is, that the mongodb extension is not installed our outdated.
Check the documentation to install mongodb for your operating system: https://secure.php.net/manual/en/mongodb.setup.php
and make sure the extension is enabled in your php.ini by checking for:
extension=mongodb.so
If the system you are using is different from where the code will run and that is why you don't have the extension installed you can also choose to ignore the platform requirements.
I am new with Web API 2. I am trying to create an application using VS 2013 Update 4, ASP.Net 4.5, while trying to install "Microsoft.AspNet.Odata" Nuget Package,l getting following error:
Attempting to resolve dependency 'Microsoft.AspNet.WebApi.Client (≥ 5.2.2)'.
Attempting to resolve dependency 'Newtonsoft.Json (≥ 6.0.4)'.
Attempting to resolve dependency 'Microsoft.AspNet.WebApi.Core (≥ 5.2.2 && < 5.3.0)'.
Attempting to resolve dependency 'Microsoft.Extensions.DependencyInjection.Abstractions (≥ 1.0.0 && < 2.0.0)'.
Attempting to resolve dependency 'Microsoft.Extensions.DependencyInjection (≥ 1.0.0 && < 2.0.0)'.
Attempting to resolve dependency 'Microsoft.OData.Core (≥ 7.0.0 && < 8.0.0)'.
Attempting to resolve dependency 'Microsoft.Spatial (= 7.0.0)'.
Attempting to resolve dependency 'Microsoft.OData.Edm (= 7.0.0)'.
Installing 'Microsoft.Extensions.DependencyInjection.Abstractions 1.0.0'.
Successfully installed 'Microsoft.Extensions.DependencyInjection.Abstractions 1.0.0'.
Installing 'Microsoft.Extensions.DependencyInjection 1.0.0'.
Successfully installed 'Microsoft.Extensions.DependencyInjection 1.0.0'.
Installing 'Microsoft.Spatial 7.0.0'.
Successfully installed 'Microsoft.Spatial 7.0.0'.
Installing 'Microsoft.OData.Edm 7.0.0'.
Successfully installed 'Microsoft.OData.Edm 7.0.0'.
Installing 'Microsoft.OData.Core 7.0.0'.
Successfully installed 'Microsoft.OData.Core 7.0.0'.
Installing 'Microsoft.AspNet.OData 6.0.0'.
Successfully installed 'Microsoft.AspNet.OData 6.0.0'.
Adding 'Microsoft.Extensions.DependencyInjection.Abstractions 1.0.0' to WebAPI_Empty.
Uninstalling 'Microsoft.Extensions.DependencyInjection.Abstractions 1.0.0'.
Successfully uninstalled 'Microsoft.Extensions.DependencyInjection.Abstractions 1.0.0'.
Install failed. Rolling back...
Could not install package 'Microsoft.Extensions.DependencyInjection.Abstractions 1.0.0'. You are trying to install this package into a project that targets '.NETFramework,Version=v4.5', but the package does not contain any assembly references or content files that are compatible with that framework. For more information, contact the package author.
Any help to proceed are appreciated.
Thanks!!
You have to downgrade the OData package to a previous version. Right now, the current version is 6.1.0, but the most recent version that I found that works is 5.10.0
Install-Package Microsoft.AspNet.OData -Version 5.10.0
See package versions at https://www.nuget.org/packages/Microsoft.AspNet.OData/
I am a newbie to Scrapy and going through the tutorials.
Ran this command and got some error.
C:\Users\Sandra\Anaconda>scrapy shell 'http://scrapy.org'
In particular what is this URLError: <urlopen error [Errno 10051] A socket operation was attempted to an unreachable network>
Full Error message:
2015-08-20 23:35:08 [scrapy] INFO: Scrapy 1.0.3 started (bot: scrapybot)
2015-08-20 23:35:08 [scrapy] INFO: Optional features available: ssl, http11, boto
2015-08-20 23:35:08 [scrapy] INFO: Overridden settings: {'LOGSTATS_INTERVAL': 0}
2015-08-20 23:35:10 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, CoreStats, SpiderState
2015-08-20 23:35:10 [boto] DEBUG: Retrieving credentials from metadata server.
2015-08-20 23:35:10 [boto] ERROR: Caught exception reading instance data
Traceback (most recent call last):
File "C:\Users\Sandra\Anaconda\lib\site-packages\boto\utils.py", line 210, in retry_url
r = opener.open(req, timeout=timeout)
File "C:\Users\Sandra\Anaconda\lib\urllib2.py", line 431, in open
response = self._open(req, data)
File "C:\Users\Sandra\Anaconda\lib\urllib2.py", line 449, in _open
'_open', req)
File "C:\Users\Sandra\Anaconda\lib\urllib2.py", line 409, in _call_chain
result = func(*args)
File "C:\Users\Sandra\Anaconda\lib\urllib2.py", line 1227, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "C:\Users\Sandra\Anaconda\lib\urllib2.py", line 1197, in do_open
raise URLError(err)
URLError: <urlopen error [Errno 10051] A socket operation was attempted to an unreachable network>
2015-08-20 23:35:10 [boto] ERROR: Unable to read instance data, giving up
2015-08-20 23:35:10 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddlewar
e, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddlewar
e, ChunkedTransferMiddleware, DownloaderStats
2015-08-20 23:35:10 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthM
iddleware, DepthMiddleware
2015-08-20 23:35:10 [scrapy] INFO: Enabled item pipelines:
2015-08-20 23:35:10 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
Traceback (most recent call last):
File "C:\Users\Sandra\Anaconda\Scripts\scrapy-script.py", line 5, in <module>
sys.exit(execute())
File "C:\Users\Sandra\Anaconda\lib\site-packages\scrapy\cmdline.py", line 143, in execute
_run_print_help(parser, _run_command, cmd, args, opts)
File "C:\Users\Sandra\Anaconda\lib\site-packages\scrapy\cmdline.py", line 89, in _run_print_help
func(*a, **kw)
File "C:\Users\Sandra\Anaconda\lib\site-packages\scrapy\cmdline.py", line 150, in _run_command
cmd.run(args, opts)
File "C:\Users\Sandra\Anaconda\lib\site-packages\scrapy\commands\shell.py", line 63, in run
shell.start(url=url)
File "C:\Users\Sandra\Anaconda\lib\site-packages\scrapy\shell.py", line 44, in start
self.fetch(url, spider)
File "C:\Users\Sandra\Anaconda\lib\site-packages\scrapy\shell.py", line 81, in fetch
url = any_to_uri(request_or_url)
File "C:\Users\Sandra\Anaconda\lib\site-packages\w3lib\url.py", line 232, in any_to_uri
return uri_or_path if u.scheme else path_to_file_uri(uri_or_path)
File "C:\Users\Sandra\Anaconda\lib\site-packages\w3lib\url.py", line 213, in path_to_file_uri
x = moves.urllib.request.pathname2url(os.path.abspath(path))
File "C:\Users\Sandra\Anaconda\lib\nturl2path.py", line 58, in pathname2url
raise IOError, error
Error: Bad path: C:\Users\Sandra\Anaconda\'http:\scrapy.org'
Here is list of packages installed:
# packages in environment at C:\Users\Sandra\Anaconda:
#
_license 1.1 py27_0
alabaster 0.7.3 py27_0
anaconda 2.3.0 np19py27_0
argcomplete 0.8.9 py27_0
astropy 1.0.3 np19py27_0
babel 1.3 py27_0
backports.ssl-match-hostname 3.4.0.2
bcolz 0.9.0 np19py27_0
beautiful-soup 4.3.2 py27_1
beautifulsoup4 4.3.2
binstar 0.11.0 py27_0
bitarray 0.8.1 py27_1
blaze 0.8.0
blaze-core 0.8.0 np19py27_0
blz 0.6.2 np19py27_1
bokeh 0.9.0 np19py27_0
boto 2.38.0 py27_0
bottleneck 1.0.0 np19py27_0
cdecimal 2.3 py27_1
certifi 14.05.14 py27_0
cffi 1.1.2 py27_0
characteristic 14.3.0
clyent 0.3.4 py27_0
colorama 0.3.3 py27_0
conda 3.16.0 py27_0
conda-build 1.14.0 py27_0
conda-env 2.4.2 py27_0
configobj 5.0.6 py27_0
crcmod 1.7
cryptography 0.9.3 py27_0
cssselect 0.9.1 py27_0
cython 0.22.1 py27_0
cytoolz 0.7.3 py27_0
datashape 0.4.5 np19py27_0
decorator 3.4.2 py27_0
docopt 0.6.2
docutils 0.12 py27_1
dynd-python 0.6.5 np19py27_0
enum34 1.0.4 py27_0
fastcache 1.0.2 py27_0
filechunkio 1.6
flask 0.10.1 py27_1
funcsigs 0.4 py27_0
futures 3.0.2 py27_0
gcs-oauth2-boto-plugin 1.9
gevent 1.0.1 py27_0
gevent-websocket 0.9.3 py27_0
google-api-python-client 1.4.0
google-apitools 0.4.3
greenlet 0.4.7 py27_0
grin 1.2.1 py27_2
gsutil 4.12
h5py 2.5.0 np19py27_1
hdf5 1.8.15.1 2
httplib2 0.9.1
idna 2.0 py27_0
ipaddress 1.0.7 py27_0
ipython 3.2.0 py27_0
ipython-notebook 3.2.0 py27_0
ipython-qtconsole 3.2.0 py27_0
itsdangerous 0.24 py27_0
jdcal 1.0 py27_0
jedi 0.8.1 py27_0
jinja2 2.7.3 py27_2
jsonschema 2.4.0 py27_0
launcher 1.0.0 1
llvmlite 0.5.0 py27_0
lxml 3.4.4 py27_0
markupsafe 0.23 py27_0
matplotlib 1.4.3 np19py27_1
menuinst 1.0.4 py27_0
mistune 0.5.1 py27_1
mock 1.0.1 py27_0
mrjob 0.4.4
multipledispatch 0.4.7 py27_0
networkx 1.9.1 py27_0
nltk 3.0.3 np19py27_0
node-webkit 0.10.1 0
nose 1.3.7 py27_0
numba 0.19.1 np19py27_0
numexpr 2.4.3 np19py27_0
numpy 1.9.2 py27_0
oauth2client 1.4.7
odo 0.3.2 np19py27_0
openpyxl 1.8.5 py27_0
pandas 0.16.2 np19py27_0
patsy 0.3.0 np19py27_0
pattern 2.6
pbs 0.110
pep8 1.6.2 py27_0
pillow 2.8.2 py27_0
pip 7.1.0 py27_1
ply 3.6 py27_0
protorpc 0.10.0
psutil 2.2.1 py27_0
py 1.4.27 py27_0
pyasn1 0.1.7 py27_0
pyasn1-modules 0.0.5
pycosat 0.6.1 py27_0
pycparser 2.14 py27_0
pycrypto 2.6.1 py27_3
pyflakes 0.9.2 py27_0
pygments 2.0.2 py27_0
pyopenssl 0.15.1 py27_1
pyparsing 2.0.3 py27_0
pyqt 4.10.4 py27_1
pyreadline 2.0 py27_0
pytables 3.2.0 np19py27_0
pytest 2.7.1 py27_0
python 2.7.9 1
python-dateutil 2.4.2 py27_0
python-gflags 2.0
pytz 2015.4 py27_0
pywin32 219 py27_0
pyyaml 3.11 py27_1
pyzmq 14.7.0 py27_0
queuelib 1.2.2 py27_0
requests 2.7.0 py27_0
retry-decorator 1.0.0
rodeo 0.2.3
rope 0.9.4 py27_1
rsa 3.1.4
runipy 0.1.3 py27_0
scikit-image 0.11.3 np19py27_0
scikit-learn 0.16.1 np19py27_0
scipy 0.15.1 np19py27_0
scrapy 1.0.3
seaborn 0.5.1 np19py27_0
service-identity 14.0.0
setuptools 18.1 py27_0
simplejson 3.6.5
six 1.9.0 py27_0
snowballstemmer 1.2.0 py27_0
sockjs-tornado 1.0.1 py27_0
socksipy-branch 1.1
sphinx 1.3.1 py27_0
sphinx-rtd-theme 0.1.7
sphinx_rtd_theme 0.1.7 py27_0
spyder 2.3.5.2 py27_0
spyder-app 2.3.5.2 py27_0
sqlalchemy 1.0.5 py27_0
ssl_match_hostname 3.4.0.2 py27_0
statsmodels 0.6.1 np19py27_0
sympy 0.7.6 py27_0
tables 3.2.0
toolz 0.7.2 py27_0
tornado 4.2 py27_0
twisted 15.3.0 py27_0
ujson 1.33 py27_0
unicodecsv 0.9.4 py27_0
uritemplate 0.6
w3lib 1.12.0 py27_0
werkzeug 0.10.4 py27_0
wheel 0.24.0 py27_0
xlrd 0.9.3 py27_0
xlsxwriter 0.7.3 py27_0
xlwings 0.3.5 py27_0
xlwt 1.0.0 py27_0
zlib 1.2.8 0
zope.interface 4.1.2 py27_1
That particular error message is being generated by boto (boto 2.38.0 py27_0), which is used to connect to Amazon S3. Scrapy doesn't have this enabled by default.
If you're just going through the tutorial, and haven't done anything other than what you've been instructed to do, then it could be a configuration problem. Launching Scrapy with the shell argument from the command will still use the configuration and the associated settings file. By default, Scrapy will look in:
/etc/scrapy.cfg or c:\scrapy\scrapy.cfg (system-wide),
~/.config/scrapy.cfg ($XDG_CONFIG_HOME) and ~/.scrapy.cfg ($HOME) for global (user-wide) settings, and
scrapy.cfg inside a scrapy project’s root (see next section).
EDIT:
In reply to the comments, this appears to be a bug with Scrapy when boto is present (bug here).
In response "how to disable the Download handler", add the following to your settings.py file:
DOWNLOAD_HANDLERS : {
's3': None,
}
Your settings.py file should be in the root of your Scrapy project folder, (one level deeper than your scrapy.cfg file).
If you've already got DOWNLOAD_HANDLERS in your settings.py file, just add a new entry for 's3' with a None value.
EDIT 2:
I'd highly recommend looking at setting up virtual environments for your projects. Look into virtualenv, and it's usage. I'd make this recommendation regardless of packages used for this project, but doubly so with your extreme number of packages.
Maybe you should use double-quote (") instead of single-quote (').
My Python version is 2.7.10 on win32. Scrapy version is 1.0.3.