My routine is supposed to spin 10 child processes from the same go executable binary (os.Args[0]), adding some command line arguments that are valid. All processes should live for a number of seconds, which is specified in one of the arguments.
func spinChildProcesses() {
cmdParts := make([]string, 4)
cmdParts[0] = "-c"
cmdParts[1] = os.Args[0]
cmdParts[2] = "--duration"
cmdParts[3] = "10000"
for i := 0; i < 10; i++ {
proc := exec.Command("bash", cmdParts...)
go proc.Start()
}
}
func main() {
# not showing code that parses duration arg
# create 10 child subprocesses
go spinChildProcesses()
// set a duration to the process and terminate
time.Sleep(time.Second * time.Duration(duration))
fmt.Println(" - process terminating normaly")
}
When the above is run, looking at OS level I can see the arguments are not carried out. Only the root process has the arguments which I typed:
ps -ef | grep my-test-pr
root 3806 14446 0 15:23 pts/1 00:00:00 ./my-test-pr --duration 10000
root 3810 3806 0 15:23 pts/1 00:00:00 ./my-test-pr
root 3811 3806 0 15:23 pts/1 00:00:00 ./my-test-pr
root 3813 3806 0 15:23 pts/1 00:00:00 ./my-test-pr
root 3814 3806 0 15:23 pts/1 00:00:00 ./my-test-pr
root 3818 3806 0 15:23 pts/1 00:00:00 ./my-test-pr
root 3823 3806 0 15:23 pts/1 00:00:00 ./my-test-pr
root 3824 3806 0 15:23 pts/1 00:00:00 ./my-test-pr
root 3829 3806 0 15:23 pts/1 00:00:00 ./my-test-pr
root 3836 3806 0 15:23 pts/1 00:00:00 ./my-test-pr
root 3840 3806 0 15:23 pts/1 00:00:00 ./my-test-pr
Any idea why and how to ensure the arguments are passed to the children processes ?
The -c bash flag takes a single string argument to interpret. Since the argument to -c is only the string os.Args[0], that is all bash is executing, and the rest of the args are being ignored.
To provide the arguments to your binary to be executed by bash -c, join them into a single string:
var args []string
args = append(args, os.Args[0])
args = append(args, "--duration")
args = append(args, "10000")
for i := 0; i < 10; i++ {
proc := exec.Command("/bin/bash", "-c", stringsJoin(args, " "))
go proc.Start()
}
Or simply exec your binary directly without the extra shell.
Related
I created multiple subshells
$ ps -f
UID PID PPID C STIME TTY TIME CMD
501 2659 2657 0 8:22AM ttys000 0:00.15 -bash
501 2776 2659 0 8:23AM ttys000 0:00.02 bash
501 2778 2776 0 8:23AM ttys000 0:00.09 bash
501 3314 2778 0 9:13AM ttys000 0:00.26 bash
501 8884 3314 0 4:41PM ttys000 0:00.03 /bin/bash
501 8891 8884 0 4:41PM ttys000 0:00.01 /bin/bash
501 8899 8891 0 4:41PM ttys000 0:00.02 /bin/bash
501 423 408 0 7:16AM ttys001 0:00.22 -bash
501 8095 423 0 3:52PM ttys001 0:00.15 ssh root#www.****.com
501 8307 8303 0 4:05PM ttys002 0:00.17 -bash
I'd like to jump back the most top one, but have to try exit one by one
$ ps -f
UID PID PPID C STIME TTY TIME CMD
501 2659 2657 0 8:22AM ttys000 0:00.17 -bash
501 423 408 0 7:16AM ttys001 0:00.22 -bash
501 8095 423 0 3:52PM ttys001 0:00.15 ssh root#***.com
501 8307 8303 0 4:05PM ttys002 0:00.17 -bash
I checked there are 3 bashes left, so I continue,
$ exit
logout
Saving session...completed.
[Process completed]
Sad, it's the most cases I encounter, How could I jump to the top from arbitrary depth of subshells?
I've developed a bash script which is causing an anomalous memory usage when looping over a great number of files. After some time, all memory is exhausted and system starts swapping, so that it becomes unusable.
After having put some sentinels around the code, I think that function which is causing the issue is _do_cmd(), which is included in the following simplified script.
#!/bin/bash
WORKINGDIR=$(dirname "$0")
SCRIPT=$(basename $0)
LOGFILE=$WORKINGDIR/test-log.txt
FILELIST=$WORKINGDIR/file.list
INDIR=/media/data/incoming
OUTDIR=$WORKINGDIR/Foto/copied
_log() {
echo -e "[$(date +"%Y-%m-%d %H:%M:%S")]: $*" >> $LOGFILE
}
_do_cmd() {
local LOGFILE_TMP="$LOGFILE.tmp"
exec 2> >(tee -a $LOGFILE_TMP)
_log " ACTION: $#"
"$#"
ret=$?
if [[ $ret -ne 0 ]]; then
_log "ERROR: Return code $ret"
grep -v "frame= " "$LOGFILE_TMP" >> $LOGFILE
rm -f $LOGFILE_TMP
exit $ret
fi
if [ -f $LOGFILE_TMP ]; then rm $LOGFILE_TMP; fi
}
while read F
do
echo "Before: $(ps -ef | grep $SCRIPT | wc -l)"
FILE=$(basename $F)
_do_cmd cp "$INDIR/$FILE" "$OUTDIR"
echo "After: $(ps -ef | grep $SCRIPT | wc -l)"
done < $FILELIST
When I run the script, I see an output like the following one:
$ ./test-mem.sh
Before: 3
After: 4
Before: 4
After: 5
Before: 5
After: 6
Before: 6
After: 7
Before: 7
After: 8
Before: 8
After: 9
Before: 9
After: 10
Before: 10
After: 11
Before: 11
After: 12
Before: 12
After: 13
Before: 13
After: 14
Before: 14
After: 15
Before: 15
After: 16
Before: 16
After: 17
Before: 17
After: 18
Before: 18
After: 19
Before: 19
After: 20
Before: 20
After: 21
^C
Looking at running processes during the execution, I find that number of instances of my script constantly grows during the execution:
$ watch -n 1 "ps -ef | grep test-mem.sh"
Every 1,0s: ps -ef | grep test-mem.sh Wed Apr 4 10:23:32 2018
user 4117 4104 0 10:23 pts/1 00:00:00 watch -n 1 ps -ef | grep test-mem.sh
user 4877 1309 11 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 4885 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 4899 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 4913 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 4927 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 4941 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 4955 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 4969 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 4983 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 4997 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 5011 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 5025 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 5043 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 5057 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 5071 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 5085 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 5099 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 5113 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 5127 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 5141 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 5155 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 5169 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 5183 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 5304 4117 0 10:23 pts/1 00:00:00 watch -n 1 ps -ef | grep test-mem.sh
user 5305 5304 0 10:23 pts/1 00:00:00 sh -c ps -ef | grep test-mem.sh
user 5307 5305 0 10:23 pts/1 00:00:00 grep test-mem.sh
Function _do_cmd() has the purpose of running a command, capturing its error output and, only in case of an error, store it to the log file and exit to shell.
Can anybody help me to understand why after every _do_cmd execution I have a new instance of test-mem.sh running in the system?
Thanks in advance.
Ok, I've found a solution.
Using this reviewed function, "Before" and "After" values remain stable and memory usage stops from increasing more and more.
_do_cmd() {
local LOGFILE_TMP="$LOGFILE.tmp"
_log " ACTION: $#"
"$#" 2> >(tee -a $LOGFILE_TMP)
ret=$?
if [[ $ret -ne 0 ]]; then
_log "ERROR: Return code $ret"
grep -v "frame= " "$LOGFILE_TMP" >> $LOGFILE
rm -f $LOGFILE_TMP
exit $ret
fi
if [ -f $LOGFILE_TMP ]; then rm $LOGFILE_TMP; fi
}
The issue was probably due to my exec command usage, which left the process substitution running in background.
I'm using golang to call pppd and then kill it after a while. However I got a lot of defunct proccesses in this way.
This is how I run pppd
exec.Command("sh", "-c", "pppd call vpn").CombinedOutput()
This is how I kill it.
exec.Command("sh", "-c", "pkill pppd").CombinedOutput()
Then I got a lot of this
root 31541 23536 0 10:54 ? 00:00:00 [pppd] <defunct>
root 31929 23356 0 10:55 ? 00:00:00 [pptpgw] <defunct>
root 31933 23356 0 10:55 ? 00:00:00 [pptpcm] <defunct>
root 31940 23356 0 10:55 ? 00:00:00 [pppd] <defunct>
root 31993 23536 0 10:55 ? 00:00:00 [pptpgw] <defunct>
root 31997 23536 0 10:55 ? 00:00:00 [pptpcm] <defunct>
root 31998 23536 0 10:55 ? 00:00:00 [pppd] <defunct>
root 32012 23356 0 10:55 ? 00:00:00 [pptpgw] <defunct>
root 32016 23356 0 10:55 ? 00:00:00 [pptpcm] <defunct>
root 32017 23356 0 10:56 ? 00:00:00 [pppd] <defunct>
root 32070 23536 0 10:56 ? 00:00:00 [pptpgw] <defunct>
root 32074 23536 0 10:56 ? 00:00:00 [pptpcm] <defunct>
root 32075 23536 0 10:56 ? 00:00:00 [pppd] <defunct>
root 32083 23356 0 10:56 ? 00:00:00 [pptpgw] <defunct>
root 32087 23356 0 10:56 ? 00:00:00 [pptpcm] <defunct>
root 32089 23356 0 10:56 ? 00:00:00 [pppd] <defunct>
root 32131 23536 0 10:57 ? 00:00:00 [pptpgw] <defunct>
root 32135 23536 0 10:57 ? 00:00:00 [pptpcm] <defunct>
root 32148 23536 0 10:57 ? 00:00:00 [pppd] <defunct>
root 32160 23356 0 10:57 ? 00:00:00 [pptpgw] <defunct>
root 32164 23356 0 10:57 ? 00:00:00 [pptpcm] <defunct>
root 32165 23356 0 10:57 ? 00:00:00 [pppd] <defunct>
root 32177 23536 0 10:57 ? 00:00:00 [pptpgw] <defunct>
root 32181 23536 0 10:57 ? 00:00:00 [pptpcm] <defunct>
How can I avoid defunct processes.
These "zombie" processes are created when a process has finished, but the parent has not read their exit status via the wait system call.
I would guess that all you need to do is call (*Cmd).Wait() on every command structure you create. Obviously This will be less straight forward than you may like, since you probably don't want to call Wait on the first command until after the second command is finished.
EDIT: As is pointed out in the comments, (*Cmd).CombinedOutput() calls (*Cmd).Run(), which calls (*Cmd).Wait()... So the above is wrong. The real answer in this case is that for some reason sh isn't cleaning up, and so the solution is to cut out the midle man and do the call like so:
exec.Command("pppd", "call", "vpn").CombinedOutput()
That'll teach me to read the docs a little closer next time...
A simpler way to cancel your command would be to use exec.CommandContext. e.g.
ctx, cancel := context.WithCancel(context.Background())
exec.CommandContext(ctx, "pppd", "call", "vpn").CombinedOutput()
// in some other goroutine...
cancel()
Maybe this would solve your zombie problem?
run subprocess in a new thread
go exec.Command("sh", "-c", "pppd call vpn").CombinedOutput()
kill subprocess
exec.Command("pkill", "pppd").CombinedOutput().CombinedOutput()
I have one process (PID1) that does:
exec 3<>/dev/tcp/127.0.0.1/12713
And when I do:
$ ls -lh /proc/self/fd/
lrwx------ 1 0 0 64 Mar 24 12:19 0 -> /dev/pts/9
lrwx------ 1 0 0 64 Mar 24 12:19 1 -> /dev/pts/9
lrwx------ 1 0 0 64 Mar 24 12:19 2 -> /dev/pts/9
lrwx------ 1 0 0 64 Mar 24 12:20 255 -> /dev/pts/9
lrwx------ 1 0 0 64 Mar 24 12:19 3 -> socket:[83968639]
Now let's say I have a second process PID2, is it possible to read the socket opened through the PID1?
I have tried:
exec 1>/proc/PID1/fd/3
but i get the error message: No such device or address
My scenario has the PID1 writing to the socket and PID2 reading it. (basically for experimentation with the file descriptors)
I am experimenting with TCL command exec in tclsh and here are my results:
% set show_me_dir "ls"
ls
% exec $show_me_dir
VboxSharedFolder
% set show_me_dir "ls -la"
ls -la
% exec $show_me_dir
couldn't execute "ls -la": no such file or directory
% set show_me_dir {ls -la}
ls -la
% exec $show_me_dir
couldn't execute "ls -la": no such file or directory
% ls -la
total 141
d---------+ 1 wakatana Domain Users 0 Jan 22 19:12 .
d---------+ 1 wakatana Domain Users 0 Apr 16 2014 ..
----------+ 1 wakatana Domain Users 20214 Jan 23 18:43 .bash_history
----------+ 1 wakatana Domain Users 1494 Apr 15 2014 .bash_profile
----------+ 1 wakatana Domain Users 7593 Jan 22 19:03 .bashrc
d---------+ 1 wakatana Domain Users 0 Jan 15 14:56 VboxSharedFolder
%
Can somebody please explain how can I execute command with arguments?
Edit:
The following example from Expanding a list of parameters in Tcl and eval article was big eye opener of what is going on here:
The variable $action is only expanded into the string "piemiddle apple" AFTER the command line has been split into its individual parameters:
% set action {piemiddle apple}
% set $action
can't read "piemiddle apple": no such variable
Result: set command "sees" one argument, equivalent to:
% set {piemiddle apple}
The expand operator allows you to specify that a variable is to be expanded BEFORE the command line is split into individual parameters:
% set action {piemiddle apple}
% set {*}$action
apple
Result: set command "sees" two arguments, equivalent to:
% set piemiddle apple
In earlier versions of Tcl, the eval command was the recommended alternative and it remains available today.
% set action {piemiddle apple}
% eval set $action
apple
Another examples which proves functionality of expansion operator:
% set {*}"name Linus"
Linus
% puts $name
Linus
%
%
% set distro Unbuntu
Unbuntu
% set {*}"linux $distro"
Unbuntu
% puts $linux
Unbuntu
%
%
Finally the discovery that exec needs command as it's first argument and first command option as it's second argument etc.
% exec "ls" "-la"
total 137
d---------+ 1 wakatana Domain Users 0 Jan 22 19:12 .
d---------+ 1 wakatana Domain Users 0 Apr 16 2014 ..
----------+ 1 wakatana Domain Users 20214 Jan 23 18:43 .bash_history
----------+ 1 wakatana Domain Users 1494 Apr 15 2014 .bash_profile
----------+ 1 wakatana Domain Users 7593 Jan 22 19:03 .bashrc
d---------+ 1 wakatana Domain Users 0 Jan 15 14:56 VboxSharedFolder
%
%
% exec "ls -la"
couldn't execute "ls -la": no such file or directory
The safest way to build a command for exec is to use Tcl's list. For example:
% set tcl_version
8.5
% set cmd [list ls -l tmp]
ls -l tmp
% eval exec $cmd
total 32
-rw-r--r-- 1 pynexj staff 1176 Jan 23 23:24 file.txt
-rw-r--r-- 1 pynexj staff 1176 Jan 23 23:24 foo-1.dat
-rw-r--r-- 1 pynexj staff 1176 Jan 23 23:24 foo-2.dat
-rw-r--r-- 1 pynexj staff 1176 Jan 23 23:24 foo-3.dat
% exec {*}$cmd
total 32
-rw-r--r-- 1 pynexj staff 1176 Jan 23 23:24 file.txt
-rw-r--r-- 1 pynexj staff 1176 Jan 23 23:24 foo-1.dat
-rw-r--r-- 1 pynexj staff 1176 Jan 23 23:24 foo-2.dat
-rw-r--r-- 1 pynexj staff 1176 Jan 23 23:24 foo-3.dat
%
Note that {*} is a new syntax of Tcl 8.5 which can help reduce the uses of eval.
As example for ls command you can do:
exec {*}ls -lsa {*}[glob *.cpp]
Please have a look at What does {*} do in TCL?