I have a massive number of shell commands being executed with root/admin priveleges through Authorization Services' "AuthorizationExecuteWithPrivileges" call. The issue is that after a while (10-15 seconds, maybe 100 shell commands) the program stops responding with this error in the debugger:
couldn't fork: errno 35
And then while the app is running, I cannot launch any more applications. I researched this issue and apparently it means that there are no more threads available for the system to use. However, I checked using Activity Monitor and my app is only using 4-5 threads.
To fix this problem, I think what I need to do is separate the shell commands into a separate thread (away from the main thread). I have never used threading before, and I'm unsure where to start (no comprehensive examples I could find)
Thanks
As Louis Gerbarg already pointed out, your question has nothing to do with threads. I've edited your title and tags accordingly.
I have a massive number of shell commands being executed with root/admin priveleges through Authorization Services' "AuthorizationExecuteWithPrivileges" call.
Don't do that. That function only exists so you can restore the root:admin ownership and the setuid mode bit to the tool that you want to run as root.
The idea is that you should factor out the code that should run as root into a completely separate program from the part that does not need to run as root, so that the part that needs root can have it (through the setuid bit) and the part that doesn't need root can go without it (through not having setuid).
A code example is in the Authorization Services Programming Guide.
The issue is that after a while (10-15 seconds, maybe 100 shell commands) the program stops responding with this error in the debugger:
couldn't fork: errno 35
Yeah. You can only run a couple hundred processes at a time. This is an OS-enforced limit.
It's a soft limit, which means you can raise it—but only up to the hard limit, which you cannot raise. See the output of limit and limit -h (in zsh; I don't know about other shells).
You need to wait for processes to finish before running more processes.
And then while the app is running, I cannot launch any more applications.
Because you are already running as many processes as you're allowed to. That x-hundred-process limit is per-user, not per-process.
I researched this issue and apparently it means that there are no more threads available for the system to use.
No, it does not.
The errno error codes are used for many things. EAGAIN (35, “resource temporarily unavailable”) may mean no more threads when set by a system call that starts a thread, but it does not mean that when set by another system call or function.
The error message you quoted explicitly says that it was set by fork, which is the system call to start a new process, not a new thread. In that context, EAGAIN means “you are already running as many processes as you can”. See the fork manpage.
However, I checked using Activity Monitor and my app is only using 4-5 threads.
See?
To fix this problem, I think what I need to do is separate the shell commands into a separate thread (away from the main thread).
Starting one process per thread will only help you run out of processes much faster.
I have never used threading before …
It sounds like you still haven't, since the function you're referring to starts a process, not a thread.
This is not about threads (at least not threads in your application). This is about system resources. Each of those forked processes is consuming at least 1 kernel thread (maybe more), some vnodes, and a number of other things. Eventually the system will not allow you to spawn more processes.
The first limits you hit are administrative limits. The system can support more, but it may causes degraded performance and other issues. You can usually raise these through various mecahanisms, like sysctls. In general doing that is a bad idea unless you have a particular (special) work load that you know will benefit from specific tweaks.
Chances are raising those limits will not fix your issues. While adjusting those limits may make you run a little longer, in order to actually fix it you need to figure out why the resources are not being returned to the system. Based on what you described above I would guess that your forked processes are never exiting.
Related
I have a server running in an eventmachine reactor which listens to heartbeats from users to tell if they are online. It marks the users as online and offline appropriately, when it starts/stops receiving the heartbeat.
I want to wrap it all in an ensure block to mark all currently online users offline when it exits, but I'm unsure how reliable that would be.
Under what conditions could a process exit without running the ensure blocks wrapping the current execution context?
Quite a few, for example:
being killed with kill -9
segmentation faults etc (eg bugs in ruby itself or in native extensions)
power failures
the system as a whole crashing (eg kernel/driver bugs, hardware failures etc)
A network failure wouldn't stop your ensure block from running but might mean that it can't update whatever datastore stores these statuses.
Our group has long running processes which run daily. The processes are typically started at 9pm on any given day and run until 7pm the next day. Thus they typically run 22hrs/day. They are started by scheduled tasks on servers under a particular generic user ID, and they start and run regardless of whether or not that user ID is logged on. Thus, they are windowless console executables.
The tasks orchestrate computations running on a large server farm. Generally these controlling tasks run uninterrupted for the full 22hrs/day. However, we often have a need to stop and restart these processes. Because they control a multitude of tasks running on our server farm, it is important that they be shut down cleanly, so that they can stop and shut down all the server farm processes. Which brings me to our problem.
The controlling process has been programmed to respond to ctrl-C and ctrl-break signals. This works fine when the process is manually started in a console where we have access to the console and can "type" ctrl-c or ctrl-break in the console window. However, as mentioned, the processes typically run as windowless scheduled tasks. Hence we cannot "type" anything into a non-existent console window. Because they are console processes that execute without a logon process, the also must be able to execute in a completely windowless environment. So, how do we set up the process to listen for a shut-down signal?
While the process does indeed listen for a ctrl-C and ctrl-break signal, I can see no way to send that signal to a process. This seems to be a fundamental problem in Windows, or am I wrong? I am aware of SendSignal.exe, but so far have been unable to get it to work. It fails as follows:
>SendSignal 26320
Sending signal to process 26320...
CreateRemoteThread failed with 0x00000005.
StartRemoteThread failed with 0x00000005.
0x00000005 == Access is denied.
Trying "taskkill" without -F results in:
>taskkill /PID 24840
ERROR: The process with PID 24840 could not be terminated.
Reason: This process can only be terminated forcefully (with /F option).
All other "kill" functions kill the process immediately rather than sending a signal.
One possible solution would be a file-watch based solution: create a watch for some modification of a specific file. But this is a hack and we would prefer to do it with appropriate signaling. Has anyone solved this issue? It seems to be so very basic a functionality, and it is certainly trivial to do it in a Unix environment. Surely Microsoft has provided SOME mechanism to allow clean shut down of a windowless executable?
I am aware of the thread below, whose question is virtually identical (save for the specification of why the answer is necessary, i.e. why one needs to be able to do this for a windowless, console-less process), but there is no answer there excpet for "use SendSignal", which, as I said, does not work for us:
Can I send a ctrl-C (SIGINT) to an application on Windows?
There are other similar questions, but no answers as yet.
Any help appreciated.
[Upgrading #Anon's comment to an answer for visibility]
windows-kill worked perfectly and managed to resolve access denial issues faced with SendSignal. A privileged user would have to run it as well of course.
windows-kill also supports both ctrl-c and ctrl-break signals.
What system process is responsible for executing system call, when user process calls ‘system call’ and the CPU switches to supervisor mode?
Are system calls scheduled by thread scheduler (can CPU switch to executing another system call after getting interrupt)?
What system process is responsible for executing system call?
The system call wrapper(the function you call to perform the system call, yeah it's just a wrapper, not the actually System call) will take the parameters, pass them to the approperiate registers(or on stack, depends on implementation), next it will put the system call number you're requesting in the eax (assuming x86) and finally will call INT 0x80 assembly instruction which is basically telling the OS that it received an interrupt and this interrupt is a system call that needs to be served, which system call to serve is available in the eax and the parameters are in the registers.
(modern implementations stopped using INT because it's expensive in performance and now use SYSENTER and SYSEXIT; the above is still almost the same though)
From the perspective of the scheduler, it makes no difference if you perform a system call or not; the thing is, once you ask the OS for a service(via the x86 instruction INT or SYSENTER and SYSEXIT ) the CPU mode flag will change to a privileged set, then the kernel will perform the task you asked for on behalf of your process and once done, it sets the flag back and returns the execution to the next instruction.
So, from a scheduler point of view, the OS will see no difference when you execute a system call or anything else.
Few notes:
-What I mentioned above is a general description, I am not sure if Windows applies this but if it doesn't, it should be doing something of similar fashion.
-Many System Calls perform blocking tasks(like I/O handling); to make better CPU utilization if your process asks for a blocking system call, the scheduler will let your process wait in the wait-queue till what it requested is ready, meanwhile other processes run on the CPU BUT do not confuse this with anything, the OS did not 'schedule system calls'.
The scheduler's task is to organize tasks, and from its perspective the system call is just a routine that the process is executing.
A final note, some system calls are atomic which means they should be performed without any interruption to their execution, these system calls if interrupted, will be be asked to restart execution once the interrupt's cause is over; still this is far from the scheduling concept.
First question: it depends. Some system calls go to services which are already running (say a network call) as a process. Some system calls result in a new process getting created and then getting scheduled for execution.
Last question: yes windows is a multiprocessing system. The process scheduler handles when a thread runs, for how long, and hardware interrupts can end up causing the running process to release the CPU or a idle process that the hardware is now ready for to get the CPU.
In windows (at least > Win 7 but I think in the past it was true too) a lot of the system services run in processes called svchost. A good application for seeing what is running were is Process Explorer from sys internals. It is like task manager on steroids and will show you all the threads that a given process owns. For finer grained "I called this dos command what happened" details you'd probably want to use a debugging tool where you can step through your call. Generally though you don't have to concern yourself with these things, you make a system call the system knows you aren't ready to continue processing until whatever process is handling that request has returned. Your request might get the CPU right after your process releases it, it might get the CPU 2 days from now but as far as the OS is concerned (or your program should be concerned) it doesn't matter, execution stops and waits for a result unless you are running multithreaded and then it gets really complicated.
I would like to reserve one core for my application. On my searches I could find dwProcessAffinityMask to limit my process to run on the cores I want. But this does not
prevent threads of other processes to run on "my" core as well.
Is there a way to disallow a specific core/processor to be used by any (system-wide) process/thread except my process/thread?
Even if it was possible to set the SystemAffinityMask, this won't help because this would also prohibit the execution of my process/thread on that processor/core.
If your goal is to ensure that your process gets to run in a timely manner, just set a high priority for your process (for instance HIGH_PRIORITY_CLASS) using SetPriorityClass. Unless the system is running other equally high-priority work (of which there is little on a typical machine), your work will get to run immediately when it's ready to execute.
Here: http://download.oracle.com/docs/html/A95907_01/diff_uni.htm#1077398
I found that on Windows Oracle is thread based, while on Unix this is process based. Why it is like that?
What's more, there are many Oracle processes http://www.adp-gmbh.ch/ora/concepts/processes/index.html regardless the system.
Why log writer and db writer are implemented as processes... and the query execution is done using threads (windows) or processes (unix).
Oracle makes use of a SGA shared memory area to store information that is (and has to be) accessible to all sessions/transactions. For example, when a row is locked, that lock is in memory (as an attribute of the row) and all the other transactions need to see it is locked.
In windows a thread cannot access another process's memory
threads cannot access memory that
belongs to another process, which
protects a process from being
corrupted by another process.
As such, in Windows Oracle must be a single process with multiple threads.
On OS's supporting the sharing of memory between processes then it is less work for Oracle to work as a multi-process architecture and leave the process management to the OS.
Oracle runs a number of background threads/processes to do work that is (or can be) asynchronous to the other processes. That way those can continue even when other processes/threads are blocked or busy.
See this answer I posted earlier on in similar vein to this question 'What is process and thread?'. Windows makes extensive use of threads in this fashion. Unlike *nix/Linux based systems which are based on threads. And see here also, this link is a direct link(which is embedded in the first link I have given) to the explanation I gave on how Linux time divisions threads and processes.
Hope this helps,
Best regards,
Tom.