Getting error when parsing spark driver host in hadoop - hadoop

Question

I am trying to run Spark-1.0.1 against Apache Hadoop 2.2.0 YARN cluster. Both are deployed on my Windows 7 machine. When I am trying to run JavaSparkPI sample I am getting parsing exception on Hadoop side. On the Spark side, all the parameters looks ok and there is no extra character after port's 5 digits. Can anybody help please...
Exception in thread "main" java.lang.NumberFormatException: For input string: "57831'"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:492)
at java.lang.Integer.parseInt(Integer.java:527)
at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:229)
at scala.collection.immutable.StringOps.toInt(StringOps.scala:31)
at org.apache.spark.util.Utils$.parseHostPort(Utils.scala:544)
at org.apache.spark.deploy.yarn.ExecutorLauncher.waitForSparkMaster(ExecutorLauncher.scala:163)
at org.apache.spark.deploy.yarn.ExecutorLauncher.run(ExecutorLauncher.scala:101)
at org.apache.spark.deploy.yarn.ExecutorLauncher$$anonfun$main$1.apply$mcV$sp(ExecutorLauncher.scala:263)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:53)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:52)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:52)
at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ExecutorLauncher.scala:262)
at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ExecutorLauncher.scala)
14/08/11 09:00:38 INFO yarn.Client: Command for starting the Spark ApplicationMaster:
List(%JAVA_HOME%/bin/java, -server, -Xmx512m, -Djava.io.tmpdir=%PWD%/tmp,
-Dspark.tachyonStore.folderName=\"spark-80c61976-f671-41b9-96a0-0c7c5c317fdb\",
-Dspark.yarn.secondary.jars=\"\",
-Dspark.driver.host=\"W01B62GR.UBSPROD.MSAD.UBS.NET\",
-Dspark.app.name=\"JavaSparkPi\",
-Dspark.jars=\"file:/N:/Nick/Spark/spark-1.0.1-bin-hadoop2/bin/../lib/spark-examples-1.0.1-hadoop2.2.0.jar\",
-Dspark.fileserver.uri=\"http://139.149.169.172:57836\",
-Dspark.executor.extraClassPath=\"N:\Nick\Spark\spark-1.0.1-bin-hadoop2\lib\spark-examples-1.0.1-hadoop2.2.0.jar\",
-Dspark.master=\"yarn-client\", -Dspark.driver.port=\"57831\",
-Dspark.driver.extraClassPath=\"N:\Nick\Spark\spark-1.0.1-bin-hadoop2\lib\spark-examples-1.0.1-hadoop2.2.0.jar\",
-Dspark.httpBroadcast.uri=\"http://139.149.169.172:57835\",
-Dlog4j.configuration=log4j-spark-container.properties,
org.apache.spark.deploy.yarn.ExecutorLauncher, --class, notused, --jar , null,
--args 'W01B62GR.UBSPROD.MSAD.UBS.NET:57831' ,
--executor-memory, 1024, --executor-cores, 1,
--num-executors , 2, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)

Answer 1

The error looks pretty clear there: 57831' is not a number. 57831 is. Look at your argument:
'W01B62GR.UBSPROD.MSAD.UBS.NET:57831'
The ' should not be there. If you mean those weren't in your original argument, show your command line. I am not sure this will work on Windows without Cygwin.

Answer 2

i got same problem when running spark pi . The issue is that you are not passing the arguments in correct order. Ensure you list all the options (like --master) before and then specify the job you are running .
http://spark.apache.org/docs/latest/running-on-yarn.html