My program, which I've run numerous times on different clusters suddenly stops. The log:
15/04/20 19:19:59 INFO scheduler.TaskSetManager: Finished task 12.0 in stage 15.0 (TID 374) in 61 ms on ip-XXX.compute.internal (16/24)
15/04/20 19:19:59 INFO storage.BlockManagerInfo: Added rdd_44_14 in memory on ip-XXX.compute.internal:37999 (size: 16.0 B, free: 260.6 MB)
Killed
What does "Killed" mean and why does it occur? There's no other errors.
"Killed" usually means that the OS has terminated the process by sending a SIGKILL signal. This is an unblockable signal that terminates a process immediately. It's often used as an OOM (out-of-memory) process killer -- if the OS decides that memory resources are getting dangerously low, it can pick a process to kill to try to free some memory.
Without more information, it's impossible to tell whether your process was killed because of memory problems or for some other reason. The kind of information you might be able to provide to help diagnose what's going on includes: how long was the process running before it was killed? can you enable and provide more verbose debug output from the process? is the process termination associated with any particular pattern of communication or processing activity?
Try setting yarn.nodemanager.vmem-check-enabled to false in your program's Spark config, something like this:
val conf = new SparkConf().setAppName("YourProgramName").set("yarn.nodemanager.vmem-check-enabled","false")
val sc = new SparkContext(conf)
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-avoid-being-killed-by-YARN-node-manager-td22199.html
maybe the vm problem
ensure you have swap partition.
ensure vm.swappiness is not zero.
Related
I am running a Go process under a cgroup with an OOM Killer, and I want to print the heap memory info by such as command "go tool pprofhttp://localhost:6060/debug/pprof/heap" or any other way which can figure out the top memory consumers when the process is killed (Btw, it is better to print the heap info by the Go process itself such as call some method by the Go process), so I can know top memory consumers and fix it next time.
I have a program, written on C/C++ by myself, that is killed by Linux. The message "killed" appears. Willing to dig out the problem I observed inside the file /var/log/kern.log:
Out of memory: Kill process 3915 (my_proj) score 236 or sacrifice child
Killed process 3915 (my_proj) total-vm:5503376kB, anon-rss:3857420kB, file-rss:40kB
I do not know how to read this information and if there is some useful information to understand why this killed happened. Can you help me?
You are victim of the Linux OOM Killer.
You can tune the way that the OOM killer handles OOM conditions with certain processes. for example, your my_proj process 3915 that was killed earlier.
If you want it not to be killed by the OOM killer, you can :
echo -15 > /proc/3915/oom_adj
Here is what I did:
Ran a q process with a limiting vmem argument
(say in a 100GB system, running vmem of 50GB)
Logged a unix top command
After the entire process was completed, I was trying to analyse the memory usage. I saw that the process %age memory usage crossed 90% mark. I believed that vmem restricts the memory consumption. But it seems that my process used more than 90GB memory at times.
How can this be explained? Am I missing something?
As #terrylynch said, command line parameter "-w" is used to set memory cap in kdb/q process. I checked and found that "-vmem" is not a kdb command line parameter.
vmem, in this context, is used in unix to manage memory of processes.
This is purely academic question, I don't really need to know this information for anything, but I would like to understand kernel a bit more :-)
According to kernel documentation http://www.tldp.org/LDP/tlk/kernel/processes.html processes in linux kernel have following states:
Running
The process is either running (it is the current process in the
system) or it is ready to run (it is waiting to be assigned to one of
the system's CPUs).
Waiting
The process is waiting for an event or for a resource. Linux
differentiates between two types of waiting process; interruptible and
uninterruptible. Interruptible waiting processes can be interrupted by
signals whereas uninterruptible waiting processes are waiting directly
on hardware conditions and cannot be interrupted under any
circumstances.
Stopped
The process has been stopped, usually by receiving a signal. A process
that is being debugged can be in a stopped state.
Zombie
This is a halted process which, for some reason, still has a
task_struct data structure in the task vector. It is what it sounds
like, a dead process.
As you can see, when I take a snapshot of processes state, using command like ps, I can see, if it's in Running state, that process either was literally Running or just waiting to be assigned to some CPU by kernel.
In my opinion, these 2 states (that are actually both represented by 1 state in task_struct) are quite different.
Why there is no state like "Ready" that would mean the process is "ready to run" but wasn't assigned to any CPU so far, so that the task_struct would be more clear about the real state? Is it even possible to retrieve this information, or is it secret for whatever reason which process is "literally running" on the CPU?
The struct task_struct contains a long to represent current state:
volatile long state; /* -1 unrunnable, 0 runnable, >0 stopped */
This simply indicates if a process is 'runnable'.
To see the currently executing process you should look at the runqueue. Specifically a struct rq (as defined in kernel/sched/sched.h) contains:
struct task_struct *curr, *idle, *stop;
The pointer *curr is the currently running process on this runqueue (there exists a runqueue per CPU).
You should consult files under kernel/sched/ to see how the Kernel determines which processes should be scheduled according to the different scheduling algorithms if you are interested in exactly how it arrives at the running state.
This is not a linux-kernel answer but a more general about scheduling ^^
A core part of any OS is the Scheduler: http://en.wikipedia.org/wiki/Process_scheduler
Many of them work giving every process a time slice of execution and letting each of them do a little bit of work before switching (referred as a context switch) to another process.
Since the length of a time slice is in the order of milliseconds by the time the information you requested is shown, the state has surely changed so differentiate between "Really Running" and "Ready-but-not-really-running" could result (most of the time) in inaccurate informations.
Today I found a very strange problem.
I ran Redhat Enterprise Linux 6, and the CPU was Intel E31275 (4 cores, 8 threads). I found one kernel thread (I called it as my_thread) didn't work correctly.
With "ps" command, I found the status of my_thread was always running:
ps ax
5545 ? R 3:14 [my_thread]
15774 ttyS0 Ss 0:00 -bash
...
But its running time was always 3:14. Since it ws running, why didn't the total time increase?
From the proc file /proc/5545/sched, I found the all statistics including wakeups count (se.nr_wakeups) for this thread was always the same, too.
From /proc/5545/stack, I found this thread called this function and never returned:
interruptible_sleep_on_timeout(&q, 3*HZ);
In theory this function would return every 3 seconds if no other threads woke up the thread. Each time after the function returned, se.nr_wakeups in /proc/5545/sched would be increased by 1. But this never happened after I found the thread had some problems.
Does any one have some ideas? Is it possible that interruptible_sleep_on_timeout() never returns?
Update:
I find the problem won't occur if I set CPU affinity for this thread. If I pin it to a dedicated core, then everything is OK. Are there any problems with SMP scheduling?
Update again:
After I disalbe hyperthread in BIOS, I have not seen such a problem until now.
First off, R indicates the thread is not in running state but runnable. That is, it does not mean it runs, it means it is in a state the scheduler is allowed to pick it for running. There is a big difference between the two.
In a similar sense interruptible_sleep_on_timeout(&q, 3*HZ); will not run the thread after 3 jiffies, but rather make it available for running after 3 jiffies - and indeed you see it in "ps" as available for running, so possibly the timeout has indeed occurred.
Since you did not say anything about the kernel thread in question I don't even know if it is in your own code or standard kernel code so I cannot really answer in detail.
One possible reason for the situation you described is that some other thread (user or kernel) has higher priority then your thread and so the scheduler never picks it for running. If so, it is not probably a thread running in real time priority (SCHED_FIFO or SCHED_RR).