I'm trying to reduce the amount of forking our Ruby/Rails app does. We shell out a lot with backticks, and each of these forks the entire process, which can cause a huge memory bloat.
I'm going through, identifying the ones that get called the most, and trying to replace them with code which achieves the same thing without making a shell call. However, in some cases I suspect it might still be forking under the hood anyway.
Is there a way to detect or log whenever a process forks? I'm using ubuntu 14.04. A log would be ideal as I can then keep an eye on it when I run the amended code.
Related
I have a server on which we execute multiple bash scripts to automate tasks (like copying files to other servers, kicking off backups, etc). It has been working for some months, but today it started to get erratic.
What is happening, is that the script gets 'stuck' for a while, and after that, it runs with no problem. If I copy and paste the commands one by one on the terminal, it works, so is not something on the script itself, but it seems something that is preventing the bash interpreter (if that makes sense).
Another weird behavior is that the same script will run with no issues eventually. However, as we use Jenkins for automation, the scripts are re-created every time a new job starts.
For example, I created a new script, tst.sh, which only contains an echo. If I try to run it directly, it gets stuck for a while. I tried to debug it with bash -xeav but it does not print my script code, which means that it is not reading it. After a while, the script ran, with no changes. However, creating one script, with the same content and a different name, resurfaces the issue.
My hypothesis is that something prevents the script to be read, and just waits until whatever is blocking it to finish. However, I did not see any process holding the file, which means that it may not the case.
Is there any other thing I should try? My knowledge in bash is pretty basic, so I don't know if there is a flag that may help me on debugging this internally.
I am working on RHEL 8.85, the bash version is GNU bash, version 4.4.20(1)-release (x86_64-redhat-linux-gnu)
UPDATES BASED ON THE COMMENTS
Server resources are OK, no usage for them.
Hardware for the server also works fine, the ops team has not reached out with any known issue at least
Reboot makes the issue disappear, however, it reappears after 5 minutes or so
The issue seems that is not related to bash profiles and such.
Issue solved, posting this as an answer so people can find it quicker.
Turns out, as multiple users suggested in the comments (thanks to all!!) the problem was caused by a security monitor, which analyzed each of the scripts that were executed. The team changed some settings on that end to prevent it from happening, and so far is working.
I want to run a Julia script from window command line, but it seems everytime I run > Julia code.jl, a new instance of Julia is created and the initiation time (package loading, compiling?) is quite long.
Is there a way for me to skip this initiation time by running the script on the current REPL/Julia instance? (which usually saves me 50% of running time).
I am using Julia 1.0.
Thank you,
You can use include:
julia> include("code.jl")
There are several possible solutions. All of them involve different ways of sending commands to a running Julia session. The first few that come to my mind are:
use sockets as explained in https://docs.julialang.org/en/v1/manual/networking-and-streams/#A-simple-TCP-example-1
set up a HTTP server e.g. using https://github.com/JuliaWeb/HTTP.jl
use named pipes, as explained in Named pipe does not wait until completion in bash
communicate e.g. through the file system (e.g. make Julia scan some folder for .jl files and if it finds them there they get executed and moved to another folder or deleted) - this is probably simplest to implement correctly
In all the solutions you can send the command to Julia by executing some shell command.
No matter which approach you prefer the key challenge is sanitizing the code to handle errors properly (i.e. a situation when you sent some command to the Julia session and it crashes or when you send requests faster than Julia is able to handle them). This is especially important if you want the Julia server to be detached from the terminal.
As a side note: when using the Distributed module from stdlib in Julia for multiprocessing you actually do a very similar thing (but the communication is Julia to Julia) so you can also have a look how this module is implemented to get the feeling how it can be done.
We have several jenkins pipeline jobs that are taking much longer to complete than we expect. Specific steps seem to "hang" for unwarranted periods of time. Running those same steps manually on another system runs significantly faster.
One example job is a step that uses Ruby to recurse through a bunch of directories and performs a shell command on each file in those directories. Running on our Ubuntu 14.04 Jenkins system takes about 50 minutes. Running the same command on my desktop Mac runs in about 10 seconds.
I did some experimentation on the Jenkins builder by running the Ruby command at the command prompt and had the same slow result that Jenkins had. I also removed Ruby from the equation by batching up each of the individual shell commands Ruby would have run and put them in a shell script to run each shell command sequentially. That took a long time as well.
I've read some posts about STDERR blocking may be the reason. I've then done some experimentation with redirecting STDERR and STDOUT to /dev/null and the commands will finish in about 20 seconds. That is what I would expect.
My questions are:
1. Would these slowdowns in execution time be the result of some I/O blocking?
2. What is the best way to fix this? Some cases I may want the output so redirecting to /dev/null is probably not going to work. Is there a kernel or OS level change I can make?
Running on Ubuntu 14.04 Amazon EC2 instance R3.Large.
Ubuntu 14.04.5 LTS (GNU/Linux 3.13.0-108-generic x86_64)
ruby 2.1.5p273 (2014-11-13 revision 48405) [x86_64-linux]
Yes. Transferring huge amounts of data between slave and master leads to performance problems indeed. This applies for storing build artifacts as well as for massive amounts of console output.
For console output, the performance penalty is particularly big if you use the timestamper plugin. If enabled for your job, try disabling that first.
Otherwise, I'd avoid huge amounts of console output in general. Try to restrict console output to very high job-level information that (in case of failure) provides links to further "secondary" logfile data.
Using I/O redirection (as you already did) is the proper way to accomplish that, e.g.
mycommand 2>mycommand.stderr.txt 1>mycommand.stdout.txt
This will always work (except for very special cases where you may need to select command-specific options to redirect a console output stream that a command explicitly creates on its own).
It must be obvious, but I cant get a usecase of Delayed Job, cause due to ruby`s Gargabe Collector specific, it doesnt free memory back to OS. And once delayed job process will take all memory anyway. And the only way is to restart delayed job process.
But if I restart delayed job process and there is currenlty running task - it will never be completed. Probably, there is some workaround to restart that task later, but this approach seems ugly to me.
I tried real jobs and some simple computatuin without any variables, symbols or links so I dont think that "my code leaks". Still, every new job increases memory of delayed_job process.
May be I use Delayed job for something that its not designed? Or it could be environment problem (besides, tried on local machine and on VPS) ?
Tested on: Ubuntu 14.04 and Debian 6 (both x86), Rails 3.2, delayed_job 4.0.2, delayed_job_active_record 4.0.1, ruby 2.1.2
I could give some code examples, but, as I mentioned, I tried both: real job and simple computation. So I won`t if it is not significant and my mistakes are fundamental.
Due to my conditions - my tasks can be executed for couple of minutes, read and write about 100K records to database and require a lot of computation, tasks cant be interrupted, and number of tasks limited by 10-20 dayli, may be - I only guess to use Resque, because it forks process everytime, so there should be no problems with accumulating memory with time.
So do I realy do something wrong or this is a nature of DJ - to occupie all memory or require a restart - and if I cant restart it, I shouldnt use its approach ?
Everything I read on the internet (not so much, by the way) tells that its rubys GC trouble that it doesnt free memory back to OS, and some advises to profile code for unlinked objects (it sounds the most realistic to my case, but, I tried a lot with code that doesnt create any objects, and I explicitly set everything to nil and call GC.start)
I have a server written in python that would use a lot of RES memory when occasionally certain input comes in. It'd be annoying to have that python script continuously occupying that much RAM because we have a lot of other things running on the same machine. I did some research and found that the only sure way to release those memory back to the OS is to exit the process. So I am hoping to be able to restart the python script when it detects itself using too much memory after processing each input.
I tried the following trick to reload the process, and but found the process still uses as much RAM after reloading. No cleanup is done.
os.execl(sys.executable, sys.executable, * sys.argv)
Is there another clean way to restart a python script without inheriting all this RAM usage?
I'm actually in a similar situation myself. I haven't come up with a solution yet, but what you might be able to do is make a bash script or .bat file that restarts the script when it finishes. This would most likely not inherit all that RAM because python itself exits and then starts again.
os.execl(sys.executable, sys.executable, * sys.argv)
This doesn't work because when you call os.execl it spawns a new process within the original python script and waits until that finishes, and then exits. That's why it "inherits" the RAM because it's still running in the background and hasn't exited yet.