I've installed databricks-connect on Windows 10 with the instructions here: https://docs.databricks.com/dev-tools/databricks-connect.html
After running databricks-connect configure and entering all values, i'm running databricks-connect test. This is the output I'm getting, and it hangs:
* PySpark is installed at c:\users\user\.conda\envs\myenv\lib\site-packages\pyspark
* Checking SPARK_HOME
* Checking java version
java version "1.8.0_251"
Java(TM) SE Runtime Environment (build 1.8.0_251-b08)
Java HotSpot(TM) 64-Bit Server VM (build 25.251-b08, mixed mode)
* Skipping scala command test on Windows
* Testing python command
The system cannot find the path specified.
Digging a bit deeper, it seems that the underlying pyspark package fails to initialize. It fails on this line:
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
When I try to run this manually, it hangs. I guess this is a problem is either the local Spark or the required Hadoop (and winutils.exe) installations, but databricks-connect requires a fresh pyspark installation (doc says to uninstall pyspark prior to installation).
Would be happy for any references for:
Fixing the databricks-connect issue
Fixing the underlying pyspark installation issue
Related
I just switch to try develop on Linux for a while.
But one thing I couldn't understand how to solve it, as I mentioned in topic.
I'm going to develop a project that need Java SDK, then I installed the SDK with command:
#bash
sudo apt install openjdk-17-jdk-headless
After installation finished, I could run command java -version and javac -version to see the result of installation and it looks OK.
openjdk version "17.0.3" 2022-04-19
OpenJDK Runtime Environment (build 17.0.3+7-Ubuntu-0ubuntu0.22.04.1)
OpenJDK 64-Bit Server VM (build 17.0.3+7-Ubuntu-0ubuntu0.22.04.1, mixed mode, sharing)
And then when I open IDE (Intellij IDEA) and it has feature of integrated terminal. (see picture)
But when I try to test java -version it was shown the error:
bash: java: command not found
As you can see from picture, I run java -version on terminal it return output properly, but when I run the same command java -version, it output with error message, like it couldn't find the java SDK that I installed.
Could anyone help me to describe the reason and solutions to solve the issue?
P.S. This is same on VS Codium.
I found the answer.
Solution:
Install the IDE app with the file from official website instead of using package manager of OS.
Description:
The cause is I installed the IDE via Package Manager of OS (here I'm using Pop_OS, install via Pop!_Shop)
And its package manager has some strict rules for security purpose.
I'm not sure, Pop_OS using which package manager, but I'll raise Flatpak as reference: https://docs.flatpak.org/en/latest/sandbox-permissions.html
And on blacklisted directories, my installed java command is under /usr/bin/ which is in blacklist.
base) stephen#stephen-Aspire-5250:~$ java --version
java 13.0.1 2019-10-15
Java(TM) SE Runtime Environment (build 13.0.1+9)
Java HotSpot(TM) 64-Bit Server VM (build 13.0.1+9, mixed mode, sharing)
(base) stephen#stephen-Aspire-5250:~$ find . -name h2o.jar
./R/x86_64-pc-linux-gnu-library/3.4/h2o/java/h2o.jar
then from R
> h2o.init()
H2O is not running yet, starting it now...
Error in .h2o.checkJava() :
Cannot find Java. Please install the latest JRE from
http://www.oracle.com/technetwork/java/javase/downloads/index.html
I have a feeling conda is messing up the landscape, but don't know how to fix.
I started h2o from terminal with:
java -jar ~/R/x86_64-pc-linux-gnu-library/3.4/h2o/java/h2o.jar
and then the h2o.init() from R works. Still, don't why h2o cannot find java, which is on my PATH.
H2O gets its path to java runtime from JAVA_HOME environment variable, so make sure to set it properly for/from R if you have to use h2o.init() rather than system shell/bash (e.g. with java -Xmx1g -jar ./h2o.jar).
More info
After several years of experience with H2O in Ubuntu/Centos/RHEL I now start H2O only from bash (issuing richly parameterized commands at H2O docker container startup), rather than with R or python API functions (it led to all sorts of problems, such as using all server CPU cores that yielded huge performance degradation for the inexperienced users or exposing passwordless REST API with root file access over standard H2O port to the entire corporate network...).
As a side note, Java 13 is supported by latest H2O versions, but I would still recommend using LTS versions, currently 11, for security reasons. The same of course applies to Ubuntu itself.
I am trying to install spark in windows 8.1 and getting below error.
C:>cd Program Files\spark-1.5.0\bin
C:\Program Files\spark-1.5.0\bin>spark-shell
'C:\Program' is not recognized as an internal or external command,
operable program or batch file.
C:\Program Files\spark-1.5.0\bin>
and done prerequisites,please refer below commands,
C:\Users\Anbu>java -version
java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)
C:\Users\Anbu>scala -version
Scala code runner version 2.11.8 -- Copyright 2002-2016, LAMP/EPFL
C:\Users\Anbu>
set the system properties also.
so where i missed? why this error coming?
Any one please help me to install spark.
Thanks,
Anbu k
It seems you downloaded source code package of Spark.
For installations you need built package of Spark. You can download pre-built package of Spark from Sparks's downloads page by choosing a specific package type(as shown in below image). So download pre-built package and check the installation. It should work.
While trying to install Informix's JDBC driver, I get this error:
java -cp /home/ics/sandbox/jdbc/setup.jar run -console
The wizard cannot continue because of the following error:
could not load wizard specified in /wizard.inf (104)
I have pointed to a newer Java from Sun using:
export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_05
export PATH=$JAVA_HOME/bin:$PATH
java -version
java version "1.8.0_05"
Java(TM) SE Runtime Environment (build 1.8.0_05-b13)
Java HotSpot(TM) Server VM (build 25.5-b02, mixed mode)
Pointing to a newer Java and, as IBM/Informix support told me, getting away from OpenJDK should allow the installer to run, but that does not work. I also saw using Sun's or IBM's java elsewhere, when I searched for posts on the specific error. On another Centos System, OpenJDK is installed, and I can install the JDBC driver successfully
I also tried removing tty settings from my environment, which also did not work.
Here is the so post where this error is mentioned.
If anyone has heard of a solution, I'd love to hear it, 'cause without the jdbc driver, there's no Clojure database work with Informix, and, when working, it works well. And, I have thought of tarring and zipping the good install and moving it, but that's kind of cheating.
This problem occurs on
cat /etc/redhat-release
CentOS release 6.4 (Final)
The installed java version, not my workaround newer version is
java -version
java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.11.90) (rhel-1.62.1.11.11.90.el6_4-i386)
OpenJDK Server VM (build 20.0-b12, mixed mode)
Until an answer arrives that allows an install, I took #Michał Niklas 's suggestion and manually installed the driver under /opt/ on the new system. This worked.
I am still going to pursue the cause of this problem. I edited the OP to reflect #ceinmart 's suggestion to remove tty settings. This did not work, but I found it a useful suggestion.
I am trying to get Elasticsearch to run but unable to. Running ./bin/elasticsearch yields the following error:
Initialization Failed ...
- MissingResourceException[Can't find bundle for base name org.elasticsearch.common.joda.time.format.messages, locale en]
I'm running Mac OSX 10.8.2 and java -version yields:
java version "1.7.0_25"
Java(TM) SE Runtime Environment (build 1.7.0_25-b15)
Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)
Any ideas on what the problem is? I also trying running the ./bin/elasticsearch.in.sh script before but nothing has worked
It turns out the file on the elasticsearch website was messed up or had a bad download. I downloaded the tar.gz version and it worked.
I had the similar issue with both Elasticsearch 1.7 & Elasticsearch 2.0 on Mac OSx 10.10 Yosemite. I messed around with installing Java 8's JRE without success.
I ended up installing the Java 8 JDK in addition to the JRE based on this answer.
Additionally here's Elasticsearch's instructions