I know that i can find the map/reduce task log inside: /usr/local/hadoop/logs/userlogs/.
Are there a friendly way to see it?
For example, when i clicked http://127.0.0.1:8088/cluster/, I can see all jobs executed in the cluster. Then i clicked in a FINISHED job. But now, when i try to click in Tracking URL: History it gives me an error, Why can i see the task logs from here?
I would like to see the stderr, stdout and syslog from each task.
Try using Job Browser from HUE
or Use the command
yarn logs -applicationId [OPTIONS]
general options are:
-appOwner AppOwner (assumed to be current user if
not specified)
-containerId ContainerId (must be specified if node
address is specified)
-nodeAddress NodeAddress in the format nodename:port
(must be specified if container id is
specified)
Example: yarn logs -applicationId application_1414530900704_0007
Related
I have to run multiple spark job one by one in a sequence, So I am writing a shell script. One way I can do is to check success file in output folder for job status, but i wanna know that is there any other way to check the status of spark-submit job using unix script, where I am running my jobs.
You can use command
yarn application -status <APPLICATIOM ID>
where <APPLICATIOM ID> is your application ID and check for line like:
State : RUNNING
This will give you the status of your application
To check the list of application, run via yarn you can use command
yarn application --list
You can add also -appTypes to limit the listing based on the application type
Using scontrol makes it very easy to get the WorkDir, StdErr, StdOut and Command of a running and queued job. Is it possible to access these informations after a job is completed? I could not find any hints when looking in the documentation of sacct.
sacct will not report that information. If your cluster has the elasticsearch plugin configured, that information will be stored in elasticsearch as well as the full script submitted
The jobcomp/filetxt JobCompType plugin will also store the WorkDir but not the other fields.
You can also use a SlurmctldEpilog to store all the data you want in a file.
I am running the hadoop in my local system but while running ./start-all.sh command its running all functionality except Name Node while running it's getting connection refused and in log file prints below exception
java.io.ioexception : there appears to be a gap in the edit log, we expected txid 1, but got txid 291.
Can You please help me.
Start namenode with recover flag enabled. Use the following command
./bin/hadoop namenode -recover
The metadata in Hadoop NN consists of:
fsimage: contains the complete state of the file system at a point in time
edit logs: contains each file system change (file creation/deletion/modification) that was made after the most recent fsimage.
If you list all files inside your NN workspace directory, you'll see files include:
fsimage_0000000000000000000 (fsimage)
fsimage_0000000000000000000.md5
edits_0000000000000003414-0000000000000003451 (edit logs, there're many ones with different name)
seen_txid (a separated file contains last seen transaction id)
When NN starts, Hadoop will load fsimage and apply all edit logs, and meanwhile do a lot of consistency checks, it'll abort if the check failed. Let's make it happen, I'll rm edits_0000000000000000001-0000000000000000002 from many of my edit logs in my NN workspace, and then try to sbin/start-dfs.sh, I'll get error message in log like:
java.io.IOException: There appears to be a gap in the edit log. We expected txid 1, but got txid 3.
So your error message indicates that your edit logs is inconsitent(may be corrupted or maybe some of them are missing). If you just want to play hadoop on your local and don't care its data, you could simply hadoop namenode -format to re-format it and start from beginning, otherwise you need to recovery your edit logs, from SNN or somewhere you backed up before.
I am new in spark and i want to print logs on console using apache spark in yarn cluster mode.
You need to check the value in log4j.properties file. In my case i have this file in /etc/spark/conf.dist directory
log4j.rootCategory=INFO,console
INFO - prints the all the logs on the console. You can change the value to ERROR, WARN to limit the information you would like to see on the console as sparks logs can be overwhelming
I'd like to list the nodes on which the containers are running for a particular MR job.
I only have the application_id.
Is it possible to do it with Hadoop REST API and/or through command line?
This can be done using the yarn command.
Run yarn applicationattempt -list <Application Id> to get an app attempt id
Run yarn container -list <Application Attempt Id> to get the container ids
Run yarn container -status <Container Id> to get the host for any particular container.
If you want this in a bash script or want to get every host for an application with a large number of containers you will probably want to parse out the attempt/container id and host, but this is at least a start.
You can find them using Resource manager UI. Find your application by ID among the existing applications and click on the link with ID you have. You will see your application stats. Fint the tracking URL and click on the link 'History'. There you'll be able to find the tasks in your map operation and recude optration. You can open each task and see the information, to which node it was assigned for, nubmer of attempts, logs for each task and attempts and lots of other usefull information.
For getting the information about the container status from command line you can use yarn container -status command from bash