let's consider a following scenario:
[user#host ~]$ yarn applicationattempt -list application_1620157095390_0392
Total number of application attempts :1
ApplicationAttempt-Id State AM-Container-Id
appattempt_1620157095390_0392_000001 FINISHED container_1620157095390_0392_01_000001
[user#host ~]$ yarn container -list appattempt_1620157095390_0392_000001
Total number of containers :0
Container-Id Start Time Finish Time State Host Node Http Address LOG-URL
Could someone explain me why it does not show any container? I am sure that for this app some containers were executed.
My version of hadoop is 3.
Related
I'm new with Apache Hadoop and I'm trying to install in Alpine (docker container) in pseudo-distribuited mode Apache Hadoop 2.9.1 but I got this error when I run start-dfs.sh
localhost: /usr/local/hadoop/sbin/hadoop-daemon.sh: line 131: 883 Aborted (core dumped) nohup nice -n $HADOOP_NICENESS $hdfsScript --config $HADOOP_CONF_DIR $command "$#" > "$log" 2>&1 < /dev/null
The NameNode and SecondaryNameNode start succesfully but the DataNode no.
I had the exact same problem and also every version >2.9.1 ended in a quick core dump of a the DataNode in Docker.
The comment from #OneCricketeer actually lead me to the right direction which should have been an acceptable answer - so here is a quick heads up for future users:
Apparently, components of the DataNode won't work well with Alpine/Musl and switching to e.g. an Ubuntu-based parent image like 8-jdk solves this problem.
Here is a link to the Dockerfile I am currently using.
My hive-server2 list a few running jobs, so I can find the various query_id.
But there is not yarn-application information in the Yarn 8088 pages.
My question is how to kill the running job.
If you are using Yarn as resource manager, you can find all running jobs by running the following in shell:
yarn application -list -appStates ALL
You can change ALL to RUNNING etc. depending on what application state you are interested in seeing.
An alternative command to the above to see running applications is:
mapred job -list
In order to kill a specific application/job, with YARN you can run:
yarn application -kill <application_id>
Or otherwise:
mapred job -kill <job_id>
I want to get top 3 yarn applications in running state from all clusters
output : AppicationId, Application Type, Memory occupied, start time
I am setting up a hadoop testbed that has have two nodes / servers (A, B). Server B contains the docker daemon and other hadoop related services such as Data Node, Secondary name node and Node manager while server A has resource manager and name node. When a container is spawned / launched on Server B using DCE (Docker container executor), I want to attach a volume to it.
Can somebody kindly suggest on how I could do this in the DCE environment?
According to me we can add volume to Docker Container in 2 ways
1) Specify Volume in Docker file
FROM ubuntu
RUN mkdir /myvol
RUN echo "hello world" > /myvol/greeting
VOLUME /myvol
2) Specify the Volume during run time
docker run -it v /hdfs/foldername dockerrepositoryyname:version /bin/bash
For more details refer https://docs.docker.com/engine/reference/builder/
Hope this Help!!!...
I've setup a Mesos cluster using the CloudFormation templates from Mesosphere. Things worked fine after cluster launch.
I recently noticed that none of the slave nodes are listed in the Mesos dashboard. EC2 console shows the slaves are running & pass health checks. I restarted nodes on cluster but that didn't help.
I ssh'ed into one of the slaves and noticed mesos-slave services are not running. Executed sudo systemctl status dcos-mesos-slave.service but that couldn't start the service.
Looked in /var/log/mesos/ and tail -f mesos-slave.xxx.invalid-user.log.ERROR.20151127-051324.31267 and saw the following...
F1127 05:13:24.242182 31270 slave.cpp:4079] CHECK_SOME(state::checkpoint(path, bootId.get())): Failed to create temporary file: No space left on device
But the output of df -h and free show there is plenty of disk space left.
Which leads me to wonder, why is it complaining about no disk space?
Ok I figured it out.
When running Mesos for a long time or under frequent load, the /tmp folder won't have any disk space left since Mesos uses the /tmp/mesos/ as the work_dir. You see, the filesystem can only hold a certain number of file references(inodes). In my case, slaves were collecting large number of file chuncks from image pulls in /var/lib/docker/tmp.
To resolve this issue:
1) Remove files under /tmp
2) Set a different work_dir location
It is good practice to run
docker rmi -f $(docker images | grep "<none>" | awk "{print \$3}")
this way you will free space by deleting unused docker images