I am working on CPU-IDLE in Linux. I have the question who spawns idle tasks per CPU core i.e. if there are 4 arm CPU cores, who spawns the per CPU idle task? Where the code for the same is located for creating idle tasks? I understand that start_kernel in init/main.c is run by init process.
Can someone please point out the location of the code where the idle tasks are created per CPU core for CPU idle?
Regards,
Snu
Using the 5.3 kernel as a reference:
start_kernel() in "init/main.c" calls arch_call_rest_init().
arch_call_rest_init() in "init/main.c" calls rest_init().
rest_init() in "init/main.c" calls kernel_thread() to create the init process with kernel_init as the thread function.
kernel_init() in "init/main.c" calls kernel_init_freeable().
kernel_init_freeable() in "init/main.c" calls smp_init().
smp_init() in "kernel/smp.c" calls idle_threads_init().
idle_threads_init() in "kernel/smpboot.c" calls idle_init(cpu) for each CPU apart from the boot CPU.
idle_cpu(cpu) in "kernel/smpboot.c" calls fork_idle(cpu).
fork_idle(cpu) in "kernel/fork.c" clones the init process thread and calls init_idle(task, cpu).
init_idle(task, cpu) in "kernel/sched/core.c" sets up the idle thread for the CPU.
Related
I have been trying to use subprocess32.Popen but this causes my system to crash (CPU 100%). So, I have the following code:
import subprocess32 as subprocess
for i in some_iterable:
output = subprocess.Popen(['/path/to/sh/file/script.sh',i[0],i[1],i[2],i[3],i[4],i[5]],shell=False,stdin=None,stdout=None,stderr=None,close_fds=True)
Before this, I had the following:
import subprocess32 as subprocess
for i in some_iterable:
output subprocess.check_output(['/path/to/sh/file/script.sh',i[0],i[1],i[2],i[3],i[4],i[5]])
.. and I had no problems with this - except that it was dead slow.
With Popen I see that this is fast - but my CPU goes too 100% in a couple of secs and the system crashes - forcing a hard reboot.
I am wondering what it is I am doing which is making Popen to crash?
On Linux,Python2.7 if that helps at all.
Thanks.
The problem is that you are trying to start 2 millon processes at once, which is blocking your system.
A solution would be to use a Pool to limit the maximum number of processes that can run at a time, and wait for each process to finish. For this cases where you're starting subprocesses and waiting for them (IO bound), a thread pool from the multiprocessing.dummy module would do:
import multiprocessing.dummy as mp
import subprocess32 as subprocess
def run_script(args):
args = ['/path/to/sh/file/script.sh'] + args
process = subprocess.Popen(args, close_fds=True)
# wait for exit and return exit code
# or use check_output() instead of Popen() if you need to process the output.
return process.wait()
# use a pool of 10 to allow at most 10 processes to be alive at a time
threadpool = mp.Pool(10)
# pool.imap or pool.imap_unordered should be used to avoid creating a list
# of all 2M return values in memory
results = threadpool.imap_unordered(run_script, some_iterable)
for result in results:
... # process result if needed
I've left out most of the arguments to Popen because you are using the default values anyway. The size of the pool should probably be in the range of your available CPU cores if your script is doing comutational work, if it's doing mostly IO (network access, writing files, ...) then probably more.
I run cat /proc/interrupts on CentOS 6.5 with a 2.6.32-431.el6.x86_64 kernel. The result is
CPU0 CPU1 CPU2 CPU3
0: 31039 0 0 0 IO-APIC-edge timer
// content omitted
LOC: 211509915 178638855 154577696 153050202 Local timer interrupts
// content omitted
Then I run cat /proc/interrupts several times. But the count 31039 of IO-APIC-edge timer interrupt does not change. My first question is whether IO-APIC-edge timer represents the global timer which interrupts HZ times every second. If yes, why its count does not change HZ times every second?
I run grep CONFIG_HZ /boot/*config*, it shows CONFIG_HZ=1000.
My second question is why only CPU0 receives the timer interrupts?
timer is the good old ISA timer interrupt; it is used only when booting, until the kernel has detected and initialized the local APIC timers.
Every CPU (core) uses a HZ timer for scheduling.
However, with CONFIG_NO_HZ_IDLE or even CONFIG_NO_HZ, that timer is disabled when it is not needed.
In this case, only one CPU needs a timer for timekeeping.
On a SMP machine with local APIC, global timer is only used during boot time. After local APIC is setup up, local timer interrupts both call update_process_times and update jiffies. The global timer is unused. All CPUs perform update_process_times. But only one CPU updates jiffies.
Answer to my first question: IO-APIC-edge timer represents the global timer. But it is only used during boot time. Since it is unused after boot time, its count does not change HZ times every second.
Answer to my second question: Only one CPU handles the interrupts, other CPUs ignores it:
if cpuid == cpu_for_global_timer
handle it
else
ignore it
For details, refer to http://yaojingguo.github.io/Linux-Kernel-Time.html
When I'm running wmic query via command line, I'm detected a line with ThreadCount value.
I don't know about the meaning of ThreadCount.
I'm running this wmic query:
wmic process where (Caption like '%explorer%') get * /format:list
Output of above query:
Caption=explorer.exe
CommandLine=C:\Windows\Explorer.EXE
CreationClassName=Win32_Process
CreationDate=20140725092933.908032+330
CSCreationClassName=Win32_ComputerSystem
CSName=DIGITALFOX
Description=explorer.exe
ExecutablePath=C:\Windows\Explorer.EXE
ExecutionState=
Handle=1820
HandleCount=856
InstallDate=
KernelModeTime=50388323
MaximumWorkingSetSize=1380
MinimumWorkingSetSize=200
Name=explorer.exe
OSCreationClassName=Win32_OperatingSystem
OSName=Microsoft Windows 7 Ultimate |C:\Windows|\Device\Harddisk0\Partition2
OtherOperationCount=90378
OtherTransferCount=2089300
PageFaults=63847
PageFileUsage=32724
ParentProcessId=1776
PeakPageFileUsage=70672
PeakVirtualSize=284794880
PeakWorkingSetSize=42564
Priority=8
PrivatePageCount=33509376
ProcessId=1820
QuotaNonPagedPoolUsage=48
QuotaPagedPoolUsage=388
QuotaPeakNonPagedPoolUsage=53
QuotaPeakPagedPoolUsage=490
ReadOperationCount=1543
ReadTransferCount=4529679
SessionId=1
Status=
TerminationDate=
ThreadCount=30
UserModeTime=34008218
VirtualSize=235257856
WindowsVersion=6.1.7600
WorkingSetSize=33030144
WriteOperationCount=6
WriteTransferCount=696
What is the meaning of ThreadCount in above data?
About Processes and Threads
Each process provides the resources needed to execute a program. A process has a virtual address space, executable code, open handles to system objects, a security context, a unique process identifier, environment variables, a priority class, minimum and maximum working set sizes, and at least one thread of execution. Each process is started with a single thread, often called the primary thread, but can create additional threads from any of its threads.
A thread is the entity within a process that can be scheduled for execution. All threads of a process share its virtual address space and system resources. In addition, each thread maintains exception handlers, a scheduling priority, thread local storage, a unique thread identifier, and a set of structures the system will use to save the thread context until it is scheduled. The thread context includes the thread's set of machine registers, the kernel stack, a thread environment block, and a user stack in the address space of the thread's process. Threads can also have their own security context, which can be used for impersonating clients.
ms-help://MS.MSSDK.1033/MS.WinSDK.1033/dllproc/base/about_processes_and_threads.htm
In this the Threadcount is the No of threads that process is currently using.
In your situation the process explorer is using 30 threads.
Thread count is used for avoiding orphan threads so before closing the process thread count should be zero.
I have a worker role that runs a number of parallel background workers. These workers run tasks that last from one minute to 5 hours and use quite a lot of memory.
I would like to delay the start of a new worker by testing the current level of memory consumption. Something like this:
while (memoryAvailable < 50%) {
Thread.Sleep( 1000 * 60 * 10 ); // 10 minutes
}
Can I test for available memory within a worker role?
Also, can I automate a reboot of the instance if memory drops below a certain amount?
Since your worker role instances are Windows Server 2012, you can just set up an appropriate perf counter during role startup ( OnStart() ) with whichever pertinent Memory counters you're interested in, and set up a task to observe the perf counter periodically. When available memory drops below your threshold (or committed bytes exceeds your threshold), you can easily recycle the role instance:
RoleEnvironment.RequestRecycle();
How do I check to see if a Win32 thread is running or in suspended state?
I can't find any Win32 API which gives the state of a thread. So how do I get the thread state?
I think - originally - this information was not provided because any API that provided this info would be misleading and useless.
Consider two possible cases - the current thread has suspended the thread-of-interest. Code in the current thread knows about the suspended state and should be able to share it so theres no need for the kernel team to add an API.
The 2nd case, some other / a 3rd thread in the system has suspended the thread of interest (and theres no way to track which thread that was). Now you have a race condition - that other thread could, at any time - unsuspend the thread of interest and the information gleaned from the API is useless - you have a value indicating the thread is suspended when it is in fact, not.
Moral of the story - if you want to know that a thread is suspended - suspend it: The return value from SuspendThread is the previous suspend count of the thread. And now you DO know something useful - The thread WAS AND STILL IS suspended - which is useful. Or that it WASN't (but now is) suspended. Either way, the thread's state is now deterministically known so you can in theory make some intelligent choices based on that - whether to ResumeThread, or keep it suspended.
You can get this information by calling NtQuerySystemInformation() with the value for SystemProcessesAndThreadsInformation (integer value 5).
If you want an example of what you can do with this information take a look at Thread Status Monitor.
WMI's Win32_Thread class has a ThreadState property, where 5: "Suspended Blocked" and 6:Suspended Ready.
You will need the Thread's Id to get the right instance directly (the WMI object's Handle property is the thread id).
EDIT: Given this PowerShell query:
gwmi win32_thread | group ThreadState
gives
Count Name Group
----- ---- -----
6 2 {, , , ...}
966 5 {, , , ...}
WMI has a different definition of "Suspended" to Win32.
In Windows 7, you can use QueryUmsThreadInformation. (UMS stands for User mode scheduling).
See here for UmsThreadIsSuspended.
you could get thread suspend count with code like this:
DWORD GetThreadSuspendCount(HANDLE hThread) {
DWORD dwSuspendCount = SuspendThread(hThread);
ResumeThread(hThread);
return dwSuspendCount;
}
but, as already said - it is not accurate.
Moreover, suspending a thread is evil.
YES: it IS possible to get the thread state and determine if it is suspended.
And NO: You don't need Windows 7 to do that.
I published my working class here on Stackoverflow: How to get thread state (e.g. suspended), memory + CPU usage, start time, priority, etc
This class requires Windows 2000 or higher.
I think the state here is referred to as
If thread is in thread proc , doing some processing Or
Waiting for event
This can be taken care of by using variable which can tell that if a thread is actually running or waiting for event to happen.
These scenarios appear when considering threadpools, having some n threads and based on each thread running status , tasks can be assigned to idle threads.