How do I add the lz4 native libraries for use by Spark workers?
I have tried to add them via both LD_LIBRARY_PATH and ( as shown - but no accepted or even upvoted answer - in Apache Spark Native Libraries ) - in SPARK_LIBRARY_PATH. They are not working: we get:
java.lang.RuntimeException: native lz4 library not available
at org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165)
at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1201)
at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1094)
at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.<init>(SequenceFile.java:1444)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:277)
at BIDMat.HDFSIO.writeThing(HDFSIO.scala:96)
Here is the LD_LIBRARY_PATH
$echo $LD_LIBRARY_PATH
/usr/local/Cellar/lz4/r131/lib:/usr/local/Cellar/hadoop/2.7.2/libexec/lib:
12:15:35/BIDMach_Spark $ll /usr/local/Cellar/lz4/r131/lib
and the contents of the lz4 related entry:
$ll /usr/local/Cellar/lz4/r131/lib
total 528
-r--r--r-- 1 macuser admin 71144 Sep 21 2015 liblz4.a
drwxr-xr-x 7 macuser admin 238 Sep 21 2015 .
drwxr-xr-x 3 macuser admin 102 Jun 13 10:41 pkgconfig
-r--r--r-- 1 macuser admin 64120 Jun 13 10:41 liblz4.dylib
-r--r--r-- 1 macuser admin 64120 Jun 13 10:41 liblz4.1.dylib
-r--r--r-- 1 macuser admin 64120 Jun 13 10:41 liblz4.1.7.1.dylib
Update your hadoop jars and should work perfectly fine.
Related
Welcome,
I would like to add custom resources - a zip file to my project
Gradle is somehow changing my zip file and I don't know how and why
Sources:
total 11576
drwxr-xr-x 5 user staff 160 Nov 1 00:39 .
drwxr-xr-x 7 user staff 224 Oct 31 14:17 ..
-rw-r--r-- 1 user staff 5900938 Oct 31 15:33 Custom.zip
-rw-r--r-- 1 user staff 16452 Nov 1 00:35 swagger.yaml.zip
-rw-r--r-- 1 user staff 12 Nov 1 00:39 test.txt
Build directory
total 20880
drwxr-xr-x 5 user staff 160 Nov 1 00:39 .
drwxr-xr-x 7 user staff 224 Nov 1 00:39 ..
-rw-r--r-- 1 user staff 10652091 Nov 1 00:39 Custom.zip
-rw-r--r-- 1 user staff 29355 Nov 1 00:39 swagger.yaml.zip
-rw-r--r-- 1 user staff 12 Nov 1 00:39 test.txt
Zip files in build resources have more bytes and are not valid zips anymore, simple text file is not changed
Environment
Gradle 7.5
------------------------------------------------------------
Build time: 2022-07-14 12:48:15 UTC
Revision: c7db7b958189ad2b0c1472b6fe663e6d654a5103
Kotlin: 1.6.21
Groovy: 3.0.10
Ant: Apache Ant(TM) version 1.10.11 compiled on July 10 2021
JVM: 17.0.4.1 (Azul Systems, Inc. 17.0.4.1+1-LTS)
OS: Mac OS X 12.6 aarch64
I'm following the Anchor docs here, but I keep getting this error...
BPF SDK path does not exist: /Users/herbie/.cargo/bin/sdk/bpf: No such file or directory (os error 2)
I ran ls -al /Users/herbie/.cargo/bin and got this output:
total 239152
drwxr-xr-x 17 herbie staff 544 31 Jan 16:55 .
drwxr-xr-x 9 herbie staff 288 13 Dec 11:58 ..
-rwxr-xr-x 1 herbie staff 12574724 31 Jan 16:49 anchor
-rwxr-xr-x 12 herbie staff 8521112 31 Jan 16:55 cargo
-rwxr-xr-x 1 herbie staff 7578989 14 Dec 14:05 cargo-build-bpf
-rwxr-xr-x 12 herbie staff 8521112 31 Jan 16:55 cargo-clippy
-rwxr-xr-x 12 herbie staff 8521112 31 Jan 16:55 cargo-fmt
-rwxr-xr-x 12 herbie staff 8521112 31 Jan 16:55 cargo-miri
-rwxr-xr-x 12 herbie staff 8521112 31 Jan 16:55 clippy-driver
-rwxr-xr-x 12 herbie staff 8521112 31 Jan 16:55 rls
-rwxr-xr-x 12 herbie staff 8521112 31 Jan 16:55 rust-gdb
-rwxr-xr-x 12 herbie staff 8521112 31 Jan 16:55 rust-lldb
-rwxr-xr-x 12 herbie staff 8521112 31 Jan 16:55 rustc
-rwxr-xr-x 12 herbie staff 8521112 31 Jan 16:55 rustdoc
-rwxr-xr-x 12 herbie staff 8521112 31 Jan 16:55 rustfmt
-rwxr-xr-x 12 herbie staff 8521112 31 Jan 16:55 rustup
Haven't found much online, and have never heard of BPF before...
It's unclear when you're getting the error during the installation, but here's a few things to try:
be sure that you're using an up-to-date version of Rust stable with rustup update stable
check that you're using the Solana CLI version designated in the docs using solana -V
run cargo build-bpf on the hello world Rust application: https://github.com/solana-labs/example-helloworld/tree/master/src/program-rust
For more reference, BPF is the bytecode format used by on-chain programs with Solana. You can find some more info at the links contained within https://docs.solana.com/developing/on-chain-programs/overview#berkeley-packet-filter-bpf
Try removing the solana cache before running your code. It worked for me. Basically, the BPF SDK hasn't been installed accurately.
rm -rf ~/.cache/solana/*
After deleting the solana cache run. It should download the BPF SDK again
solana build
My brew installed elasticsearch-full started failing with error:
Unrecognized VM option 'UseConcMarkSweepGC'
Error: Could not create the Java Virtual Machine.
macOS 10.15.7
My config ended up having some old settings, so I had to comment out the lines under GC configuration:
# /usr/local/etc/elasticsearch/jvm.options
## GC configuration
#-XX:+UseConcMarkSweepGC
#-XX:CMSInitiatingOccupancyFraction=75
#-XX:+UseCMSInitiatingOccupancyOnly
There should be default configuration files in /usr/local/etc/elasticsearch.
I suggest to backup your current configuration files, replace it with the defaults (which also should be up2date) and apply your special configuration if required.
cd /usr/local/etc/elasticsearch
cp elasticsearch.yml elasticsearch.yml.bck
cp elasticsearch.yml.default elasticsearch.yml
cp jvm.options jvm.options.bck
cp jvm.options.default jvm.options
cp log4j2.properties log4j2.properties.bck
cp log4j2.properties.default log4j2.properties
eddie[elasticsearch]$ ls -la
total 112
drwxr-xr-x 12 eddie admin 384 Dec 7 12:18 .
drwxrwxr-x 39 eddie admin 1248 Dec 6 15:22 ..
-rw-rw---- 1 eddie admin 199 Oct 27 14:16 elasticsearch.keystore
-rw-r----- 1 eddie admin 2872 Dec 7 12:18 elasticsearch.yml
-rw-r----- 1 eddie admin 2872 Dec 6 15:22 elasticsearch.yml.default
-rw-r----- 1 eddie admin 3058 Dec 7 12:18 elasticsearch.yml.bck
-rw-r----- 1 eddie admin 2357 Dec 7 12:17 jvm.options
-rw-r----- 1 eddie admin 2357 Dec 6 15:22 jvm.options.default
-rw-r----- 1 eddie admin 2892 Dec 7 12:17 jvm.options.bck
-rw-r----- 1 eddie admin 11021 Dec 7 12:18 log4j2.properties
-rw-r----- 1 eddie admin 11021 Dec 6 15:22 log4j2.properties.default
-rw-r----- 1 eddie admin 3992 Dec 7 12:18 log4j2.properties.bck
I maintain some legacy code that runs on RH Linux that sends inserts over the network to a client's Sybase. We were using Sybase 12.5 libraries and have just migrated to use Sybase 15.0 client libraries.
My application logs the time at which it sends the insert over the network and also the time it get the acknowledgment back from the target Sybase. When using 12.5 libraries the time was ~5 ms, now with the 15.5 libraries it's roughly 50 ms.
The only change I've made on the application side is to specify the location of the interfaces file on the command line. Previously the file was located in the default location - the location of the Sybase installation. Now it's located where the application is deployed, hence the need to specify the location explicitly.
Would anyone have any idea what is causing the dramatic change in speed, or have hints on where I could look or ideas on how to trace the root cause?
Please forgive the lack of technical details. I'm not a DB admin but a developer using a compiled library to connect to Sybase and don't have access to the nitty-gritty internals. That being said, I'm using the same internal library in both cases, it's only the Sybase librairies that are different.
My Sybase 12.5 and 15 installations look like this:
$ ls -l /opt/sybase/
total 48
-rw-r--r-- 1 root root 555 Jul 2 2019 ASE150.csh
-rw-r--r-- 1 root root 259 Jul 2 2019 ASE150.env
-rw-r--r-- 1 root root 388 Jul 2 2019 ASE150.sh
drwxr-xr-x 10 root root 4096 Feb 2 2017 OCS-15_0
-rw-r--r-- 1 root root 555 Jul 2 2019 SYBASE.csh
-rw-r--r-- 1 root root 259 Jul 2 2019 SYBASE.env
-rw-r--r-- 1 root root 388 Jul 2 2019 SYBASE.sh
drwxr-xr-x 58 root root 4096 Jul 2 2019 charsets
drwxr-xr-x 3 root root 4096 Jul 2 2019 collate
drwxr-xr-x 2 root root 4096 Nov 23 20:55 config
-rw-r--r-- 1 root root 1239 Jul 2 2019 interfaces
drwxr-xr-x 5 root root 4096 Nov 23 20:55 locales
$ ls -l ~/12_5/sybase/
total 28
drwxrwxr-x 4 oadc oadc 4096 Nov 29 2017 OCS-12_5
drwxrwxr-x 58 oadc oadc 4096 Nov 29 2017 charsets
drwxrwxr-x 2 oadc oadc 4096 Mar 16 09:45 config
drwxrwxr-x 2 oadc oadc 4096 Mar 16 09:45 include
-r-xr-xr-x 1 oadc oadc 1184 Mar 16 09:45 interfaces
drwxrwxr-x 2 oadc oadc 4096 Mar 16 09:45 lib
drwxrwxr-x 5 oadc oadc 4096 Mar 16 09:45 locales
EDIT
After some more digging it looks like the libraries under OCS-12-5 are not actually for 12_5 but for 15_5!
$ strings sybase/OCS-12_5lib/libsybct*.a | grep "Sybase Client-Library"
Sybase Client-Library/15.5/P/DRV.15.5.0/Linux x86_64/Linux 2.6.9-55.ELsmp x86_64/BUILD1550-003/64bit/OPT/Mon Oct 5 23:16:48 2009
Sybase Client-Library/15.5/P/DRV.15.5.0/Linux x86_64/Linux 2.6.9-55.ELsmp x86_64 Native Threads/BUILD1550-003/64bit/OPT/Tue Oct 6 00:06:57 2009
Which means that my assumption that 12.5 was faster than 15.0 is wrong. What is actually happening is that 15.5 is faster than 15.0. Which makes more sense.
I'm not going to go hunt down the idiot that submitted these files into a directory labelled OCS-12-5 ...
After some more digging it looks like the libraries under OCS-12-5 are not actually for 12_5 but for 15_5!
$ strings sybase/OCS-12_5lib/libsybct*.a | grep "Sybase Client-Library" Sybase Client-Library/15.5/P/DRV.15.5.0/Linux x86_64/Linux 2.6.9-55.ELsmp x86_64/BUILD1550-003/64bit/OPT/Mon Oct 5 23:16:48 2009 Sybase Client-Library/15.5/P/DRV.15.5.0/Linux x86_64/Linux 2.6.9-55.ELsmp x86_64 Native Threads/BUILD1550-003/64bit/OPT/Tue Oct 6 00:06:57 2009
Which means that my assumption that 12.5 was faster than 15.0 is wrong. What is actually happening is that 15.5 is faster than 15.0. Which makes more sense.
I've updated the question with this new information.
I Want to use perl module DBD::Oracle on Mac OS X 10.8.
I installed DBI through CPAN.
Downloaded the Oracle instant client 11.2 (basic, sqlplus and jdk).
Extracted it to /usr/local/oracle.
$ ls /usr/local/oracle/instantclient_11_2/
BASIC_README libnnz11.dylib ojdbc6.jar
SQLPLUS_README libocci.dylib.11.1 sdk
adrci libociei.dylib sqlplus
genezi libocijdbc11.dylib uidrvci
glogin.sql libsqlplus.dylib xstreams.jar
libclntsh.dylib libsqlplusic.dylib
libclntsh.dylib.11.1 ojdbc5.jar
Then installed DBD::Oracle.
Now when I want to use DBD::Oracle it gives an error.
install_driver(Oracle) failed: Can't load '/Library/Perl/5.12/darwin-thread-multi-
2level/auto/DBD/Oracle/Oracle.bundle' for module DBD::Oracle:
dlopen(/Library/Perl/5.12/darwin-thread-multi-2level/auto/DBD/Oracle/Oracle.bundle, 1):
Library not loaded: /ade/b/2649109290/oracle/rdbms/lib/libclntsh.dylib.11.1
Referenced from: /Library/Perl/5.12/darwin-thread-multi-
2level/auto/DBD/Oracle/Oracle.bundle
Reason: image not found at /System/Library/Perl/5.12/darwin-thread-multi-
2level/DynaLoader.pm line 204.
I have DYLD_LIBRARY_PATH=/usr/local/oracle/instaclient_11_2
I have not a clue what I am doing wrong.
SOLVED:
I got the same error when trying to run sqlplus. I added my oracle client directory to my global PATH variable and it is working now.
Saw they did the same in this tutorial: http://www.janhellevik.no/?p=521
Check this one out: perl DBD::Oracle Module installtion
Above information is about Linux Environment. But you might get some clues on how to go about configuring DBD::Oracle on MAC OSX.
On linux, you need "Basic, SQLPLUS and Devel" binaries of the oracle instant client. Here is the directory listing on my Linux box:
# pwd
/usr/lib/oracle/11.2/client64/lib
# ls -l
total 185232
-rw-r--r-- 1 root root 368 Sep 17 2011 glogin.sql
lrwxrwxrwx 1 root root 17 Jul 9 2012 libclntsh.so -> libclntsh.so.11.1
-rw-r--r-- 1 root root 52761218 Sep 17 2011 libclntsh.so.11.1
-rw-r--r-- 1 root root 7955322 Sep 17 2011 libnnz11.so
lrwxrwxrwx 1 root root 15 Jul 9 2012 libocci.so -> libocci.so.11.1
-rw-r--r-- 1 root root 1971762 Sep 17 2011 libocci.so.11.1
-rw-r--r-- 1 root root 118408281 Sep 17 2011 libociei.so
-rw-r--r-- 1 root root 164836 Sep 17 2011 libocijdbc11.so
-rw-r--r-- 1 root root 1503303 Sep 17 2011 libsqlplusic.so
-rw-r--r-- 1 root root 1477446 Sep 17 2011 libsqlplus.so
-rw-r--r-- 1 root root 2095661 Sep 17 2011 ojdbc5.jar
-rw-r--r-- 1 root root 2714016 Sep 17 2011 ojdbc6.jar
-rw-r--r-- 1 root root 300666 Sep 17 2011 ottclasses.zip
-rw-r--r-- 1 root root 66779 Sep 17 2011 xstreams.jar
and here is the list of oracle instant client RPMs that I installed:
# rpm -qa | grep -i oracle
oracle-instantclient11.2-basic-11.2.0.3.0-1
oracle-instantclient11.2-devel-11.2.0.3.0-1
oracle-instantclient11.2-sqlplus-11.2.0.3.0-1
Hope this helps.