I have a server written in python that would use a lot of RES memory when occasionally certain input comes in. It'd be annoying to have that python script continuously occupying that much RAM because we have a lot of other things running on the same machine. I did some research and found that the only sure way to release those memory back to the OS is to exit the process. So I am hoping to be able to restart the python script when it detects itself using too much memory after processing each input.
I tried the following trick to reload the process, and but found the process still uses as much RAM after reloading. No cleanup is done.
os.execl(sys.executable, sys.executable, * sys.argv)
Is there another clean way to restart a python script without inheriting all this RAM usage?
I'm actually in a similar situation myself. I haven't come up with a solution yet, but what you might be able to do is make a bash script or .bat file that restarts the script when it finishes. This would most likely not inherit all that RAM because python itself exits and then starts again.
os.execl(sys.executable, sys.executable, * sys.argv)
This doesn't work because when you call os.execl it spawns a new process within the original python script and waits until that finishes, and then exits. That's why it "inherits" the RAM because it's still running in the background and hasn't exited yet.
Related
I'm developing a program running on embedded Linux (Debian Buster), and I found the program sometimes has performance issues. After some debugging process, I doubt the issue might not be in my program. Instead, somehow the OS start doing memory swap and my program was swapped to the file system.
Therefore, I use the code here to verify. And it turns out my program occupied much less physical memory after about 500 seconds, and it matches the hypothesis.
Now I want to find which process suddenly takes lots of memory at that point, but I don't know how.
Is there anyway to keep monitoring memory usage of all processes (or the top 10) of the system and dump to a log file? Any tools or commands would be good.
Thanks.
I'm developing a program running on embedded Linux
It will be helpful, if you could specify which embedded Linux you are working on.
Based on that, there are tools that someone could suggest.
For Linux, I would say, you could use:
top -p [PID]
you can get PID by:
ps [options]
I am not sure if there is a problem while using the command line?
dump to a log file
I think you could use grep to dump the terminal output to a log file you can create using touch command.
I'm trying to reduce the amount of forking our Ruby/Rails app does. We shell out a lot with backticks, and each of these forks the entire process, which can cause a huge memory bloat.
I'm going through, identifying the ones that get called the most, and trying to replace them with code which achieves the same thing without making a shell call. However, in some cases I suspect it might still be forking under the hood anyway.
Is there a way to detect or log whenever a process forks? I'm using ubuntu 14.04. A log would be ideal as I can then keep an eye on it when I run the amended code.
I'm developing an app. The operating system I'm using is linux. I need to run if possible a ruby script on the file created in the directory. I need to keep this script always running. The first thing I thought about is inotify:
The inotify API provides a mechanism for monitoring file system events. Inotify can be used to monitor individual files, or to monitor directories.
It's exactly what I need, then I found "rb-inotify", a wrapper fir inotify.
Do you think there is a better way of doing what I need than using inotify? Also, I really don't understand the way that I have to use rb-inotify.
I just create, for example, a rb file with:
notifier = INotify::Notifier.new
notifier.watch("directory/to/check",:create) do |event|
#do task with event.name file
end
notifier.run
Then I just ruby myRBNotifier.rb, and it will stay looping for ever. How do I stop it? Any idea? Is this a good approach?
I'd recommend looking at god. It's designed for this sort of task, and makes it pretty easy to build a monitoring system for background and daemon apps.
As for the main code itself, inotify isn't cross-platform, so if you have a possibility you'll need to run on Windows or Mac OS then you'll need a different solution. It's not too hard to write a little piece of code that checks your target directory periodically for a change. If you need to know what changed, read and cache the directory entries then compare them the next time your code runs. Use sleep between runs to wait some period of time before looping.
The old-school method of doing similar things is to use cron to fire off a job at regular intervals. That job can be your script that checks whether the file list changed by comparing it to the cached version, then acting as needed if something is different.
Just run your script in the background with
ruby myRBNotifier.rb &
When you need to stop it, find the process id and use kill on it:
ps ux
kill [whatever pid your process gets from the OS]
Does that answer your question?
If you're running on a mac/unix machine, look at the launchctl man page. You can set up a process to run and execute a ruby script whenever a file changes. It's highly configurable.
I have a Perl program on Windows that needs to execute cleanup actions on exit. I wrote a signal handler using sigtrap, but it doesn't always work. I can intercept Ctrl-C, but if the machine is rebooted or the program is killed some other way, neither the signal handler nor the END block are run. I've read that Windows doesn't really have signals, and signal handling on windows is sort of a hack in Perl. My question is, how can I handle abnormal termination the Windows way? I want to run my cleanup code regardless of how or why the program terminates (excluding events that can't be caught). I've read that Windows uses events instead of signals, but I can't find information on how to deal with Windows events in Perl.
Unfortunately, I don't have the authority to install modules from CPAN, so I'll have to use vanilla ActiveState Perl. And to make things even more interesting, most of the machines I'm using only have Perl 5.6.1.
Edit: I would appreciate any answers, even if they require CPAN modules or newer versions of Perl. I want to learn about Windows event handling in Perl, and any information would be welcome.
In all operating systems, you can always abruptly terminate any program. Think of kill -9 command in Unix/Linux. You do that on any program, and it stops instantly. No way to trap it. No way for the program to request a few more operating system cycles for a clean up.
I'm not up on the difference between Unix and Windows signals, but you can imagine why each OS must allow what we call in Unix SIGKILL - a sure and immediate way to kill any program.
Imagine you have a buggy program that intercepts a request to terminate (a SIGTERM in Unix), and it enters a cleanup phase. Instead of cleaning up, the program instead gets stuck in a loop that requests more and more memory. If you couldn't pull the SIGKILL emergency cord, you'd be stuck.
The ultimate SIGKILL, of course is the plug in the wall. Pull it, and the program (along with everything else) comes to a screeching halt. There's no way your program can say "Hmm... the power is out and the machines has stopped running... Better start up the old cleanup routine!"
So, there's no way you can trap every program termination signal, and, your program will have to account for that. What you can do is see if your program needs to do a cleanup before running. On Windows, you can put an entry in the registry when your program starts up, and remove it when it shuts down and does a cleanup. In Unix, you can put a file or directory name starting wit a period in the $ENV{HOME} directory.
Back in the 1980s, I wrote accounting software for a very proprietary OS. When the user pressed the ESCAPE button, we were suppose return immediately to the main menu. If the user was entering an order, and took stuff out of inventory, the transaction would be incomplete, and inventory would be showing the items as being sold even though the order was incomplete. The solution was to check for these incomplete orders the next time someone entered an order, and back out the changes in inventory before entering the new order. Your program may have to do something similar.
My application creates a suspended process, gets process's information via VirtualQueryEx() ,but fails getting process's module information using EnumProcessModules().
The task above is completed ONLY if the process is NOT created suspended and a breakpoint is hit in the debugger(so the program runs, before the call is executed).
I'm trying to write a very decent disassembler and for that I would need to run a target process suspended, but EnumProcessModules() does not work on suspended processes.
Is there an alternative?
I dealt with something like this several years ago. If I remember right, what I ended up doing was creating the task suspended, then GetThreadContext, set its trap flag, SetThreadContext, resume the thread (which runs one instruction), then use EnumProcessModules.
Of course, there may be other ways to handle this, but at least if memory serves, that's what I came up with at the time and I seem to recall its working.