Spark : Execute python script with Spark based on Hadoop Multinode - hadoop

Question

I'm looking for use Spark based on Hadoop Multinodes and I have a question about my pythonic script with cluster mode.
My Configuration :
I have into my Hadoop Cluster :
1 Namenode (master)
2 Datanodes (slaves)
So I would like to execute my script in Python in order to use this cluster. I know that Spark could be used as standalone mode, but I would like to use my nodes.
My python script :
It's a very simple script which let to count words in my text.
import sys
from pyspark import SparkContext
sc = SparkContext()
lines = sc.textFile(sys.argv[1])
words = lines.flatMap(lambda line: line.split(' '))
words_with_1 = words.map(lambda word: (word, 1))
word_counts = words_with_1.reduceByKey(lambda count1, count2: count1 + count2)
result = word_counts.collect()
for (word, count) in result:
print word.encode("utf8"), count
My Spark command :
In order to use Spark, I do :
time ./bin/spark-submit --master spark://master:7077 /home/hduser/count.py /data.txt
But, this command lets to execute Spark in standalone mode right ?
How I can execute Spark using my Hadoop Cluster (e.g yarn) and make parallel and distributed computing on my cluster ?
I tried :
time ./bin/spark-submit --master yarn /home/hduser/count.py /data.txt
And I get issues :
2018-03-15 10:13:14 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-03-15 10:13:15 INFO SparkContext:54 - Running Spark version 2.3.0
2018-03-15 10:13:15 INFO SparkContext:54 - Submitted application: count.py
2018-03-15 10:13:15 INFO SecurityManager:54 - Changing view acls to: hduser
2018-03-15 10:13:15 INFO SecurityManager:54 - Changing modify acls to: hduser
2018-03-15 10:13:15 INFO SecurityManager:54 - Changing view acls groups to:
2018-03-15 10:13:15 INFO SecurityManager:54 - Changing modify acls groups to:
2018-03-15 10:13:15 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hduser); groups with view permissions: Set(); users with modify permissions: Set(hduser)$
2018-03-15 10:13:16 INFO Utils:54 - Successfully started service 'sparkDriver' on port 40388.
2018-03-15 10:13:16 INFO SparkEnv:54 - Registering MapOutputTracker
2018-03-15 10:13:16 INFO SparkEnv:54 - Registering BlockManagerMaster
2018-03-15 10:13:16 INFO BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2018-03-15 10:13:16 INFO BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
2018-03-15 10:13:16 INFO DiskBlockManager:54 - Created local directory at /tmp/blockmgr-b131528e-849e-4ba7-94fe-c552572f12fc
2018-03-15 10:13:16 INFO MemoryStore:54 - MemoryStore started with capacity 413.9 MB
2018-03-15 10:13:16 INFO SparkEnv:54 - Registering OutputCommitCoordinator
2018-03-15 10:13:17 INFO log:192 - Logging initialized #5400ms
2018-03-15 10:13:17 INFO Server:346 - jetty-9.3.z-SNAPSHOT
2018-03-15 10:13:17 INFO Server:414 - Started #5667ms
2018-03-15 10:13:17 INFO AbstractConnector:278 - Started ServerConnector#4f835332{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-03-15 10:13:17 INFO Utils:54 - Successfully started service 'SparkUI' on port 4040.
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler#2f867b0c{/jobs,null,AVAILABLE,#Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler#2a0105b7{/jobs/json,null,AVAILABLE,#Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler#3fd04590{/jobs/job,null,AVAILABLE,#Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler#2637750b{/jobs/job/json,null,AVAILABLE,#Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler#439f0c7{/stages,null,AVAILABLE,#Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler#3978d915{/stages/json,null,AVAILABLE,#Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler#596dc76d{/stages/stage,null,AVAILABLE,#Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler#7054d173{/stages/stage/json,null,AVAILABLE,#Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler#47b526bb{/stages/pool,null,AVAILABLE,#Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler#7896fc75{/stages/pool/json,null,AVAILABLE,#Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler#2fd54632{/storage,null,AVAILABLE,#Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler#79dcd5f2{/storage/json,null,AVAILABLE,#Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler#1732b48c{/storage/rdd,null,AVAILABLE,#Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler#5888874b{/storage/rdd/json,null,AVAILABLE,#Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler#5de9bebe{/environment,null,AVAILABLE,#Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler#428593b4{/environment/json,null,AVAILABLE,#Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler#4011c9bc{/executors,null,AVAILABLE,#Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler#5cbfbc2a{/executors/json,null,AVAILABLE,#Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler#4c33f54d{/executors/threadDump,null,AVAILABLE,#Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler#22c5d74c{/executors/threadDump/json,null,AVAILABLE,#Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler#6cd7b681{/static,null,AVAILABLE,#Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler#5ee342f2{/,null,AVAILABLE,#Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler#4d68a347{/api,null,AVAILABLE,#Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler#1e878af1{/jobs/job/kill,null,AVAILABLE,#Spark}
2018-03-15 10:13:17 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler#590aa379{/stages/stage/kill,null,AVAILABLE,#Spark}
2018-03-15 10:13:17 INFO SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://master:4040
2018-03-15 10:13:19 INFO RMProxy:98 - Connecting to ResourceManager at master/172.30.10.64:8050
2018-03-15 10:13:20 INFO Client:54 - Requesting a new application from cluster with 3 NodeManagers
2018-03-15 10:13:20 INFO Client:54 - Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
2018-03-15 10:13:20 INFO Client:54 - Will allocate AM container, with 896 MB memory including 384 MB overhead
2018-03-15 10:13:20 INFO Client:54 - Setting up container launch context for our AM
2018-03-15 10:13:20 INFO Client:54 - Setting up the launch environment for our AM container
2018-03-15 10:13:20 INFO Client:54 - Preparing resources for our AM container
2018-03-15 10:13:24 WARN Client:66 - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
2018-03-15 10:13:29 INFO Client:54 - Uploading resource file:/tmp/spark-bbfad5cb-3d29-4f45-a1a9-2e37f2c76606/__spark_libs__580552500091841387.zip -> hdfs://master:54310/user/hduser/.sparkStaging/application_1521023754917_0007/__s$
2018-03-15 10:13:33 INFO Client:54 - Uploading resource file:/usr/local/spark/python/lib/pyspark.zip -> hdfs://master:54310/user/hduser/.sparkStaging/application_1521023754917_0007/pyspark.zip
2018-03-15 10:13:33 INFO Client:54 - Uploading resource file:/usr/local/spark/python/lib/py4j-0.10.6-src.zip -> hdfs://master:54310/user/hduser/.sparkStaging/application_1521023754917_0007/py4j-0.10.6-src.zip
2018-03-15 10:13:34 INFO Client:54 - Uploading resource file:/tmp/spark-bbfad5cb-3d29-4f45-a1a9-2e37f2c76606/__spark_conf__7840630163677580304.zip -> hdfs://master:54310/user/hduser/.sparkStaging/application_1521023754917_0007/__$
2018-03-15 10:13:34 INFO SecurityManager:54 - Changing view acls to: hduser
2018-03-15 10:13:34 INFO SecurityManager:54 - Changing modify acls to: hduser
2018-03-15 10:13:34 INFO SecurityManager:54 - Changing view acls groups to:
2018-03-15 10:13:34 INFO SecurityManager:54 - Changing modify acls groups to:
2018-03-15 10:13:34 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hduser); groups with view permissions: Set(); users with modify permissions: Set(hduser)$
2018-03-15 10:13:34 INFO Client:54 - Submitting application application_1521023754917_0007 to ResourceManager
2018-03-15 10:13:34 INFO YarnClientImpl:251 - Submitted application application_1521023754917_0007
2018-03-15 10:13:34 INFO SchedulerExtensionServices:54 - Starting Yarn extension services with app application_1521023754917_0007 and attemptId None
2018-03-15 10:13:35 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:35 INFO Client:54 -
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1521105214408
final status: UNDEFINED
tracking URL: http://master:8088/proxy/application_1521023754917_0007/
user: hduser
2018-03-15 10:13:36 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:37 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:38 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:39 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:40 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:41 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:42 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:43 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:44 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:45 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:46 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:47 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:48 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:49 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:50 INFO Client:54 - Application report for application_1521023754917_0007 (state: ACCEPTED)
2018-03-15 10:13:51 INFO Client:54 - Application report for application_1521023754917_0007 (state: FAILED)
2018-03-15 10:13:51 INFO Client:54 -
client token: N/A
diagnostics: Application application_1521023754917_0007 failed 2 times due to AM Container for appattempt_1521023754917_0007_000002 exited with exitCode: -103
For more detailed output, check application tracking page:http://master:8088/cluster/app/application_1521023754917_0007Then, click on links to logs of each attempt.
Diagnostics: Container [pid=9363,containerID=container_1521023754917_0007_02_000001] is running beyond virtual memory limits. Current usage: 147.7 MB of 1 GB physical memory used; 2.1 GB of 2.1 GB virtual memory used. Killing cont$
Dump of the process-tree for container_1521023754917_0007_02_000001 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 9369 9363 9363 9363 (java) 454 16 2250776576 37073 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -server -Xmx512m -Djava.io.tmpdir=/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1521023754917_0007/co$
|- 9363 9361 9363 9363 (bash) 0 0 12869632 742 /bin/bash -c /usr/lib/jvm/java-8-openjdk-amd64/bin/java -server -Xmx512m -Djava.io.tmpdir=/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1521023754917_0$
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Failing this attempt. Failing the application.
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1521105214408
final status: FAILED
tracking URL: http://master:8088/cluster/app/application_1521023754917_0007
user: hduser
2018-03-15 10:13:51 INFO Client:54 - Deleted staging directory hdfs://master:54310/user/hduser/.sparkStaging/application_1521023754917_0007
2018-03-15 10:13:51 ERROR SparkContext:91 - Error initializing SparkContext.
org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:89)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:63)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:500)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:748)
2018-03-15 10:13:51 INFO AbstractConnector:318 - Stopped Spark#4f835332{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-03-15 10:13:51 INFO SparkUI:54 - Stopped Spark web UI at http://master:4040
2018-03-15 10:13:51 WARN YarnSchedulerBackend$YarnSchedulerEndpoint:66 - Attempted to request executors before the AM has registered!
2018-03-15 10:13:51 INFO YarnClientSchedulerBackend:54 - Shutting down all executors
2018-03-15 10:13:51 INFO YarnSchedulerBackend$YarnDriverEndpoint:54 - Asking each executor to shut down
2018-03-15 10:13:51 INFO SchedulerExtensionServices:54 - Stopping SchedulerExtensionServices
(serviceOption=None,
services=List(),
started=false)
2018-03-15 10:13:51 INFO YarnClientSchedulerBackend:54 - Stopped
2018-03-15 10:13:51 INFO MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
2018-03-15 10:13:51 INFO MemoryStore:54 - MemoryStore cleared
2018-03-15 10:13:51 INFO BlockManager:54 - BlockManager stopped
2018-03-15 10:13:51 INFO BlockManagerMaster:54 - BlockManagerMaster stopped
2018-03-15 10:13:51 WARN MetricsSystem:66 - Stopping a MetricsSystem that is not running
2018-03-15 10:13:51 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
2018-03-15 10:13:52 INFO SparkContext:54 - Successfully stopped SparkContext
Traceback (most recent call last):
File "/home/hduser/count.py", line 4, in <module>
sc = SparkContext()
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/context.py", line 118, in __init__
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/context.py", line 180, in _do_init
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/context.py", line 270, in _initialize_context
File "/usr/local/spark/python/lib/py4j-0.10.6-src.zip/py4j/java_gateway.py", line 1428, in __call__
File "/usr/local/spark/python/lib/py4j-0.10.6-src.zip/py4j/protocol.py", line 320, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:89)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:63)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:500)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:748)
2018-03-15 10:13:52 INFO ShutdownHookManager:54 - Shutdown hook called
2018-03-15 10:13:52 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-bbfad5cb-3d29-4f45-a1a9-2e37f2c76606
2018-03-15 10:13:52 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-f5d31d54-e456-4fcb-bf48-9f950233ad4b
I'm getting all the time FAILED when I want to use my cluster with Spark
Finally I tried :
time ./bin/spark-submit --master yarn --deploy-mode cluster /home/hduser/count.py /data.txt
But I get one more time issues.
I don't understand something ? I'm very new with Big Data so it's possible :/
EDIT :
This is what I obtain with : yarn application -status application_1521023754917_0007
18/03/15 10:52:07 INFO client.RMProxy: Connecting to ResourceManager at master/172.30.10.64:8050
18/03/15 10:52:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Application Report :
Application-Id : application_1521023754917_0007
Application-Name : count.py
Application-Type : SPARK
User : hduser
Queue : default
Start-Time : 1521105214408
Finish-Time : 1521105231067
Progress : 0%
State : FAILED
Final-State : FAILED
Tracking-URL : http://master:8088/cluster/app/application_1521023754917_0007
RPC Port : -1
AM Host : N/A
Aggregate Resource Allocation : 16329 MB-seconds, 15 vcore-seconds
Diagnostics : Application application_1521023754917_0007 failed 2 times due to AM Container for appattempt_1521023754917_0007_000002 exited with exitCode: -103
For more detailed output, check application tracking page:http://master:8088/cluster/app/application_1521023754917_0007Then, click on links to logs of each attempt.
Diagnostics: Container [pid=9363,containerID=container_1521023754917_0007_02_000001] is running beyond virtual memory limits. Current usage: 147.7 MB of 1 GB physical memory used; 2.1 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1521023754917_0007_02_000001 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 9369 9363 9363 9363 (java) 454 16 2250776576 37073 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -server -Xmx512m -Djava.io.tmpdir=/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1521023754917_0007/container_1521023754917_0007_02_000001/tmp -Dspark.yarn.app.container.log.dir=/usr/local/hadoop-2.7.5/logs/userlogs/application_1521023754917_0007/container_1521023754917_0007_02_000001 org.apache.spark.deploy.yarn.ExecutorLauncher --arg master:40388 --properties-file /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1521023754917_0007/container_1521023754917_0007_02_000001/__spark_conf__/__spark_conf__.properties
|- 9363 9361 9363 9363 (bash) 0 0 12869632 742 /bin/bash -c /usr/lib/jvm/java-8-openjdk-amd64/bin/java -server -Xmx512m -Djava.io.tmpdir=/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1521023754917_0007/container_1521023754917_0007_02_000001/tmp -Dspark.yarn.app.container.log.dir=/usr/local/hadoop-2.7.5/logs/userlogs/application_1521023754917_0007/container_1521023754917_0007_02_000001 org.apache.spark.deploy.yarn.ExecutorLauncher --arg 'master:40388' --properties-file /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1521023754917_0007/container_1521023754917_0007_02_000001/__spark_conf__/__spark_conf__.properties 1> /usr/local/hadoop-2.7.5/logs/userlogs/application_1521023754917_0007/container_1521023754917_0007_02_000001/stdout 2> /usr/local/hadoop-2.7.5/logs/userlogs/application_1521023754917_0007/container_1521023754917_0007_02_000001/stderr
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Failing this attempt. Failing the application.

Answer 1

for me this spark submit run the python on all spark nodes:
spark-submit --master yarn
--deploy-mode cluster
--num-executors 1
--driver-memory 2g
--executor-memory 1g
--executor-cores 1
hdfs://<host>:<port>/home/hduser/count.py /data.txt
The Spark environment need to be extended with:
export PYSPARK_PYTHON=/opt/bin/python
Furthermore the py file need to be located on hdfs so that all spark nodes in the cluster can read it. The py file need to be accessible for the spark user.

Answer 2

you have to first locate the py scrip in the HDFS location. With the name node correct URL
like hdfs dfs -ls hdfs://hostname:1543/
if you see your file is reflecting in the screen then this is the correct path.
Next execute
/bin/spark-submit --master yarn hdfs://COMPLETEHOSTNAME:1543/count.py /data.txt
it will surely work.