I am trying to run pyspark on my local machine and somehow I can not. I have downloaded the latest version of the Spark and set the following path in .bash_profile:
export PATH="/Users/user/Downloads/spark-1.3.1-bin-hadoop2.6/bin:$PATH"
When I run this command:
IPYTHON_OPTS="notebookâ pyspark
Nothing happens. I have done everything based on instructions and what I see as the output of that command is this symbol followed by nothing:
>
Can anyone please help me?
Thanks
Related
I want to installed the hadoop(3.2) in my linux system which has installed the hadoop(2.7).When I execute hadoop , I can only get the information of hadoop 2.7 ,even if I change the environment variable. And the most confused thing is when I run echo $HADOOP_HOME , sometimes I can get the path of hadoop 2.7,sometimes hadoop 3.2. I hope someone can help me.
enter image description here
enter image description here
I have downloaded Hadoop 3.1.0 and Hive is 2.1.0 in my Windows 10. I have Hadoop running properly using start-all.cmd command from terminal. When I try to run 'hive' from command prompt it gives following messages and errors which are in the attached screenshot. I am using Derby 10.12.1.1 with hive. Following this tutorial from Youtube.
Have also tried reinstalling Hive but still not working. I have already spent a lot of my time dealing with this problem but got no success.
Any sort of help will be appreciated. Thank You.
I am trying to install ozzie on my Unix machine, however i am getting error while executing the distro file.
I have tried making changes into pom.xml of oozie, but that also did not worked.
I hvae followed below links :
https://gauravkohli.com/2014/08/26/apache-oozie-installation-on-hadoop-2-4-1/
https://milindjagre.wordpress.com/2016/01/05/oozie-4-1-0-installation-on-hadoop-2-6-0-on-ubuntu-14-04/
Any help or installation guide will be great help.
I tried running spark-1.5.1-bin-hadoop2.6 distribution (and newer versions of Spark with same results) on Windows using Cygwin.
When trying to execute spark-shell script in the bin folder, I get below output:
Error: Could not find or load main class org.apache.spark.launcher.Main
I tried to set CLASSPATH to the location of lib/spark-assembly-1.5.1-hadoop2.6.0.jar but to no avail.
(FYI: I am able to run the same distribution fine on my MAC with no extra setup steps required)
Please assist in finding resolution for Cygwin execution on Windows.
I ran into and solved a similar problem with cywin on Windows 10 and spark-1.6.0.
build with Maven (maybe you're past this step)
mvn -DskipTests package
make sure JAVA_HOME is set to a JDK
$ export JAVA_HOME="C:\Program Files\Java\jdk1.8.0_60"
$ ls "$JAVA_HOME"
bin include LICENSE THIRDPARTYLICENSEREADME.txt ....
use the Windows batch file. Launch from PowerShell or CommandPrompt if you have terminal problems with cygwin.
$ chmod a+x bin/spark-shell.cmd
$ ./bin/spark-shell.cmd
My solution to the problem was to move the Spark installation into a path that didn't have spaces in it. Under Program Files I got the above error, but moving it directly under C:\ and running spark-shell.bat file cleared it up.
I am installing predictionIO from source code. I have downloaded and done the predictionIO installation successfully. I am now trying to install the dependencies (Spark, Elasticsearch, HBase) but I am running into errors for each of them. Below are the issues I am facing when I execute pio status:
1 - Unable to locate a proper Apache Spark installation
2 - It is also unable to find metadata files.
I have not changed any default settings. I'm using windows 8.1. On localhost, I have running IIS. On 127.0.0.1:8888 I run ipython notebook.
Please help on how I can get predictionIO up and running on my machine.
Thanks
If you are on Windows, you can install with Vagrant.
http://docs.prediction.io/community/projects/#vagrant-installation-for-predictionio
I believe the discussion has moved to the google group.
https://groups.google.com/forum/#!searchin/predictionio-user/SAS/predictionio-user/ZamBr1ZaQ3o/fyNkXh3zsv0J
https://groups.google.com/forum/#!searchin/predictionio-user/SAS/predictionio-user/0awaASUR8lE/JkLtPeRrNt4J
This is the relevant thread.
https://groups.google.com/forum/#!searchin/predictionio-user/SAS/predictionio-user/0awaASUR8lE/JkLtPeRrNt4J
Moreover, PredictionIO docs had a few errors. Below are some of them and their corrected versions.
1 - Actual line: PATH=$PATH:/home/yourname/predictionio/bin; export PATH
Corrected Version PATH=$PATH:/home/yourname/PredictionIO/bin; export PATH
2 - Actual Line: $ pio eventserver
Corrected Version: $ pio eventserver --ip 0.0.0.0
3 - Actual Line pio template get PredictionIO/templates-scala-parallel-recommendation MyRecommendation
Corrected Version pio template get PredictionIO/template-scala-parallel-recommendation MyRecommendation