Why netlib-java native blas/lapack libraries doesn't give performance improvement? - maven

Question

I am using this piece of code to calculate spark recommendations:
SparkSession spark = SparkSession
.builder()
.appName("SomeAppName")
.config("spark.master", "local[" + args[2] + "]")
.config("spark.local.dir",args[4])
.getOrCreate();
JavaRDD<Rating> ratingsRDD = spark
.read().textFile(args[0]).javaRDD()
.map(Rating::parseRating);
Dataset<Row> ratings = spark.createDataFrame(ratingsRDD, Rating.class);
ALS als = new ALS()
.setMaxIter(Integer.parseInt(args[3]))
.setRegParam(0.01)
.setUserCol("userId")
.setItemCol("movieId")
.setRatingCol("rating").setImplicitPrefs(true);
ALSModel model = als.fit(ratings);
model.setColdStartStrategy("drop");
Dataset<Row> rowDataset = model.recommendForAllUsers(50);
These are maven dependencies to make this piece of code work:
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.4.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.4.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.11</artifactId>
<version>2.4.0</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.11.8</version>
</dependency>
Calculating recommendations with this code takes ~70sec for my data file. This code produces following warning:
WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK
WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeRefLAPACK
Now I try to enable netlib-java by adding this dependency in maven:
<dependency>
<groupId>com.github.fommil.netlib</groupId>
<artifactId>all</artifactId>
<version>1.1.2</version>
<type>pom</type>
</dependency>
to avoid crashing of this new environment I had to do this extra trick:
LD_PRELOAD=/usr/lib64/libopenblas.so
Now it also works, it gives no warnings, but it works slower and it takes ~170sec on average to perform the same calculation. I am running this on CentOS.
Shouldn't it be faster with native libraries? Is it possible to make it faster?

Answer 1

first you can check you centos version ,for centos 6 may not using the native libraries, check this
As far as I konw , the ALS algorithm has been improved since 2.0 version , you can check
Highlights in 2.2
And the source code from 2.2 as blow :
so the native libraries has not help!