I'm using golang to call pppd and then kill it after a while. However I got a lot of defunct proccesses in this way.
This is how I run pppd
exec.Command("sh", "-c", "pppd call vpn").CombinedOutput()
This is how I kill it.
exec.Command("sh", "-c", "pkill pppd").CombinedOutput()
Then I got a lot of this
root 31541 23536 0 10:54 ? 00:00:00 [pppd] <defunct>
root 31929 23356 0 10:55 ? 00:00:00 [pptpgw] <defunct>
root 31933 23356 0 10:55 ? 00:00:00 [pptpcm] <defunct>
root 31940 23356 0 10:55 ? 00:00:00 [pppd] <defunct>
root 31993 23536 0 10:55 ? 00:00:00 [pptpgw] <defunct>
root 31997 23536 0 10:55 ? 00:00:00 [pptpcm] <defunct>
root 31998 23536 0 10:55 ? 00:00:00 [pppd] <defunct>
root 32012 23356 0 10:55 ? 00:00:00 [pptpgw] <defunct>
root 32016 23356 0 10:55 ? 00:00:00 [pptpcm] <defunct>
root 32017 23356 0 10:56 ? 00:00:00 [pppd] <defunct>
root 32070 23536 0 10:56 ? 00:00:00 [pptpgw] <defunct>
root 32074 23536 0 10:56 ? 00:00:00 [pptpcm] <defunct>
root 32075 23536 0 10:56 ? 00:00:00 [pppd] <defunct>
root 32083 23356 0 10:56 ? 00:00:00 [pptpgw] <defunct>
root 32087 23356 0 10:56 ? 00:00:00 [pptpcm] <defunct>
root 32089 23356 0 10:56 ? 00:00:00 [pppd] <defunct>
root 32131 23536 0 10:57 ? 00:00:00 [pptpgw] <defunct>
root 32135 23536 0 10:57 ? 00:00:00 [pptpcm] <defunct>
root 32148 23536 0 10:57 ? 00:00:00 [pppd] <defunct>
root 32160 23356 0 10:57 ? 00:00:00 [pptpgw] <defunct>
root 32164 23356 0 10:57 ? 00:00:00 [pptpcm] <defunct>
root 32165 23356 0 10:57 ? 00:00:00 [pppd] <defunct>
root 32177 23536 0 10:57 ? 00:00:00 [pptpgw] <defunct>
root 32181 23536 0 10:57 ? 00:00:00 [pptpcm] <defunct>
How can I avoid defunct processes.
These "zombie" processes are created when a process has finished, but the parent has not read their exit status via the wait system call.
I would guess that all you need to do is call (*Cmd).Wait() on every command structure you create. Obviously This will be less straight forward than you may like, since you probably don't want to call Wait on the first command until after the second command is finished.
EDIT: As is pointed out in the comments, (*Cmd).CombinedOutput() calls (*Cmd).Run(), which calls (*Cmd).Wait()... So the above is wrong. The real answer in this case is that for some reason sh isn't cleaning up, and so the solution is to cut out the midle man and do the call like so:
exec.Command("pppd", "call", "vpn").CombinedOutput()
That'll teach me to read the docs a little closer next time...
A simpler way to cancel your command would be to use exec.CommandContext. e.g.
ctx, cancel := context.WithCancel(context.Background())
exec.CommandContext(ctx, "pppd", "call", "vpn").CombinedOutput()
// in some other goroutine...
cancel()
Maybe this would solve your zombie problem?
run subprocess in a new thread
go exec.Command("sh", "-c", "pppd call vpn").CombinedOutput()
kill subprocess
exec.Command("pkill", "pppd").CombinedOutput().CombinedOutput()
Related
I am trying to list some selective files but want to exclude atop_20210428, but the following extended glob atop_20210#(3|4)*[0-4]*!(8)* does not exclude the file atop_20210428, what is the correction required in that?.
[root#server atop]# ls -lh atop_20210#(3|4)*[0-4]*!(8)*
-rw-r--r-- 1 root root 81M Mar 31 00:00 atop_20210330
-rw-r--r-- 1 root root 80M Apr 1 00:00 atop_20210331
-rw-r--r-- 1 root root 79M Apr 2 00:00 atop_20210401
-rw-r--r-- 1 root root 82M Apr 3 00:00 atop_20210402
-rw-r--r-- 1 root root 82M Apr 4 00:00 atop_20210403
-rw-r--r-- 1 root root 81M Apr 5 00:00 atop_20210404
-rw-r--r-- 1 root root 80M Apr 6 00:00 atop_20210405
-rw-r--r-- 1 root root 81M Apr 7 00:00 atop_20210406
-rw-r--r-- 1 root root 81M Apr 8 00:00 atop_20210407
-rw-r--r-- 1 root root 81M Apr 9 00:00 atop_20210408
-rw-r--r-- 1 root root 80M Apr 10 00:00 atop_20210409
-rw-r--r-- 1 root root 81M Apr 11 00:00 atop_20210410
-rw-r--r-- 1 root root 80M Apr 12 00:00 atop_20210411
-rw-r--r-- 1 root root 82M Apr 13 00:00 atop_20210412
-rw-r--r-- 1 root root 81M Apr 14 00:00 atop_20210413
-rw-r--r-- 1 root root 81M Apr 15 00:00 atop_20210414
-rw-r--r-- 1 root root 79M Apr 16 00:00 atop_20210415
-rw-r--r-- 1 root root 78M Apr 17 00:00 atop_20210416
-rw-r--r-- 1 root root 80M Apr 18 00:00 atop_20210417
-rw-r--r-- 1 root root 78M Apr 19 00:00 atop_20210418
-rw-r--r-- 1 root root 81M Apr 20 00:00 atop_20210419
-rw-r--r-- 1 root root 79M Apr 21 00:00 atop_20210420
-rw-r--r-- 1 root root 82M Apr 22 00:00 atop_20210421
-rw-r--r-- 1 root root 81M Apr 23 00:00 atop_20210422
-rw-r--r-- 1 root root 82M Apr 24 00:00 atop_20210423
-rw-r--r-- 1 root root 81M Apr 25 00:00 atop_20210424
-rw-r--r-- 1 root root 83M Apr 26 00:00 atop_20210425
-rw-r--r-- 1 root root 83M Apr 27 00:00 atop_20210426
-rw-r--r-- 1 root root 83M Apr 28 00:00 atop_20210427
-rw-r--r-- 1 root root 29M Apr 28 08:35 atop_20210428
I did turn on extglob already:
[root#server atop]# shopt -p extglob
shopt -s extglob
[root#server atop]#
[root#server atop]# shopt | grep extglob
extglob on
* is matching everything, so *!(8)* is always going to match everything - first !(8) will not match anything (match empty), then * will match everything.
atop_20210 #(3|4) * [0-4] * !(8) *
atop_20210 4 2 8
Why all the *? Remove them. You want to just match what you want to match, not to match anything in between. Just:
atop_20210#(3|4)[0-4]!(8)
I created multiple subshells
$ ps -f
UID PID PPID C STIME TTY TIME CMD
501 2659 2657 0 8:22AM ttys000 0:00.15 -bash
501 2776 2659 0 8:23AM ttys000 0:00.02 bash
501 2778 2776 0 8:23AM ttys000 0:00.09 bash
501 3314 2778 0 9:13AM ttys000 0:00.26 bash
501 8884 3314 0 4:41PM ttys000 0:00.03 /bin/bash
501 8891 8884 0 4:41PM ttys000 0:00.01 /bin/bash
501 8899 8891 0 4:41PM ttys000 0:00.02 /bin/bash
501 423 408 0 7:16AM ttys001 0:00.22 -bash
501 8095 423 0 3:52PM ttys001 0:00.15 ssh root#www.****.com
501 8307 8303 0 4:05PM ttys002 0:00.17 -bash
I'd like to jump back the most top one, but have to try exit one by one
$ ps -f
UID PID PPID C STIME TTY TIME CMD
501 2659 2657 0 8:22AM ttys000 0:00.17 -bash
501 423 408 0 7:16AM ttys001 0:00.22 -bash
501 8095 423 0 3:52PM ttys001 0:00.15 ssh root#***.com
501 8307 8303 0 4:05PM ttys002 0:00.17 -bash
I checked there are 3 bashes left, so I continue,
$ exit
logout
Saving session...completed.
[Process completed]
Sad, it's the most cases I encounter, How could I jump to the top from arbitrary depth of subshells?
I've developed a bash script which is causing an anomalous memory usage when looping over a great number of files. After some time, all memory is exhausted and system starts swapping, so that it becomes unusable.
After having put some sentinels around the code, I think that function which is causing the issue is _do_cmd(), which is included in the following simplified script.
#!/bin/bash
WORKINGDIR=$(dirname "$0")
SCRIPT=$(basename $0)
LOGFILE=$WORKINGDIR/test-log.txt
FILELIST=$WORKINGDIR/file.list
INDIR=/media/data/incoming
OUTDIR=$WORKINGDIR/Foto/copied
_log() {
echo -e "[$(date +"%Y-%m-%d %H:%M:%S")]: $*" >> $LOGFILE
}
_do_cmd() {
local LOGFILE_TMP="$LOGFILE.tmp"
exec 2> >(tee -a $LOGFILE_TMP)
_log " ACTION: $#"
"$#"
ret=$?
if [[ $ret -ne 0 ]]; then
_log "ERROR: Return code $ret"
grep -v "frame= " "$LOGFILE_TMP" >> $LOGFILE
rm -f $LOGFILE_TMP
exit $ret
fi
if [ -f $LOGFILE_TMP ]; then rm $LOGFILE_TMP; fi
}
while read F
do
echo "Before: $(ps -ef | grep $SCRIPT | wc -l)"
FILE=$(basename $F)
_do_cmd cp "$INDIR/$FILE" "$OUTDIR"
echo "After: $(ps -ef | grep $SCRIPT | wc -l)"
done < $FILELIST
When I run the script, I see an output like the following one:
$ ./test-mem.sh
Before: 3
After: 4
Before: 4
After: 5
Before: 5
After: 6
Before: 6
After: 7
Before: 7
After: 8
Before: 8
After: 9
Before: 9
After: 10
Before: 10
After: 11
Before: 11
After: 12
Before: 12
After: 13
Before: 13
After: 14
Before: 14
After: 15
Before: 15
After: 16
Before: 16
After: 17
Before: 17
After: 18
Before: 18
After: 19
Before: 19
After: 20
Before: 20
After: 21
^C
Looking at running processes during the execution, I find that number of instances of my script constantly grows during the execution:
$ watch -n 1 "ps -ef | grep test-mem.sh"
Every 1,0s: ps -ef | grep test-mem.sh Wed Apr 4 10:23:32 2018
user 4117 4104 0 10:23 pts/1 00:00:00 watch -n 1 ps -ef | grep test-mem.sh
user 4877 1309 11 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 4885 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 4899 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 4913 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 4927 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 4941 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 4955 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 4969 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 4983 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 4997 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 5011 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 5025 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 5043 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 5057 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 5071 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 5085 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 5099 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 5113 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 5127 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 5141 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 5155 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 5169 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 5183 4877 0 10:23 pts/0 00:00:00 /bin/bash ./test-mem.sh
user 5304 4117 0 10:23 pts/1 00:00:00 watch -n 1 ps -ef | grep test-mem.sh
user 5305 5304 0 10:23 pts/1 00:00:00 sh -c ps -ef | grep test-mem.sh
user 5307 5305 0 10:23 pts/1 00:00:00 grep test-mem.sh
Function _do_cmd() has the purpose of running a command, capturing its error output and, only in case of an error, store it to the log file and exit to shell.
Can anybody help me to understand why after every _do_cmd execution I have a new instance of test-mem.sh running in the system?
Thanks in advance.
Ok, I've found a solution.
Using this reviewed function, "Before" and "After" values remain stable and memory usage stops from increasing more and more.
_do_cmd() {
local LOGFILE_TMP="$LOGFILE.tmp"
_log " ACTION: $#"
"$#" 2> >(tee -a $LOGFILE_TMP)
ret=$?
if [[ $ret -ne 0 ]]; then
_log "ERROR: Return code $ret"
grep -v "frame= " "$LOGFILE_TMP" >> $LOGFILE
rm -f $LOGFILE_TMP
exit $ret
fi
if [ -f $LOGFILE_TMP ]; then rm $LOGFILE_TMP; fi
}
The issue was probably due to my exec command usage, which left the process substitution running in background.
My routine is supposed to spin 10 child processes from the same go executable binary (os.Args[0]), adding some command line arguments that are valid. All processes should live for a number of seconds, which is specified in one of the arguments.
func spinChildProcesses() {
cmdParts := make([]string, 4)
cmdParts[0] = "-c"
cmdParts[1] = os.Args[0]
cmdParts[2] = "--duration"
cmdParts[3] = "10000"
for i := 0; i < 10; i++ {
proc := exec.Command("bash", cmdParts...)
go proc.Start()
}
}
func main() {
# not showing code that parses duration arg
# create 10 child subprocesses
go spinChildProcesses()
// set a duration to the process and terminate
time.Sleep(time.Second * time.Duration(duration))
fmt.Println(" - process terminating normaly")
}
When the above is run, looking at OS level I can see the arguments are not carried out. Only the root process has the arguments which I typed:
ps -ef | grep my-test-pr
root 3806 14446 0 15:23 pts/1 00:00:00 ./my-test-pr --duration 10000
root 3810 3806 0 15:23 pts/1 00:00:00 ./my-test-pr
root 3811 3806 0 15:23 pts/1 00:00:00 ./my-test-pr
root 3813 3806 0 15:23 pts/1 00:00:00 ./my-test-pr
root 3814 3806 0 15:23 pts/1 00:00:00 ./my-test-pr
root 3818 3806 0 15:23 pts/1 00:00:00 ./my-test-pr
root 3823 3806 0 15:23 pts/1 00:00:00 ./my-test-pr
root 3824 3806 0 15:23 pts/1 00:00:00 ./my-test-pr
root 3829 3806 0 15:23 pts/1 00:00:00 ./my-test-pr
root 3836 3806 0 15:23 pts/1 00:00:00 ./my-test-pr
root 3840 3806 0 15:23 pts/1 00:00:00 ./my-test-pr
Any idea why and how to ensure the arguments are passed to the children processes ?
The -c bash flag takes a single string argument to interpret. Since the argument to -c is only the string os.Args[0], that is all bash is executing, and the rest of the args are being ignored.
To provide the arguments to your binary to be executed by bash -c, join them into a single string:
var args []string
args = append(args, os.Args[0])
args = append(args, "--duration")
args = append(args, "10000")
for i := 0; i < 10; i++ {
proc := exec.Command("/bin/bash", "-c", stringsJoin(args, " "))
go proc.Start()
}
Or simply exec your binary directly without the extra shell.
I'm running a validation software and I want all of the output sent to a text file and have the results of multiple files placed/appended to the same file. I thought my code was working, but I just discovered I'm only getting the results from 1 file output to the text file.
java -jar /Applications/epubcheck-3.0.1/epubcheck-3.0.1.jar ~/Desktop/Validator/*.epub 2>&1 | tee -a ~/Desktop/Validator/EPUBCHECK3_results.txt
open ~/Desktop/Validator/EPUBCHECK3_results.txt
EDIT
When I run the same .jar file using Windows command line it will process a batch of files and appeand the results appropriately. I would just do this, but it would mean having to switch work stations and transferring files to validate them. I would like to get this running through the Unix shell on my Mac system so that I don't have to do unnecessary work. Command line that IS working below:
FOR /f %%1 in ('dir /b "C:\Users\scrawfo\Desktop\epubcheck\drop epubs here\*.epub"') do (
echo %%1 >> epubcheck.txt
java -jar "C:\Users\scrawfo\Desktop\epubcheck\epubcheck-3.0.jar" "C:\Users\scrawfo\Desktop\epubcheck\drop epubs here\%%1" 2>> epubcheck.txt
echo. >> epubcheck.txt)
notepad epubcheck.txt
del epubcheck.txt
syntax provided by you is correct there might be some problem with java output or something Try Executing it without redirection
cat test
Output:-
This is Test File ...............
Next Executed Command with same syntax
ps l 2>&1 | tee -a test
Output:-
F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME
COMMAND 4 0 3287 1 20 0 4060 572 n_tty_ Ss+ tty2
0:00 /sbin/mingetty /dev/tty2 4 0 3289 1 20 0 4060 572
n_tty_ Ss+ tty3 0:00 /sbin/mingetty /dev/tty3 4 0 3291
1 20 0 4060 576 n_tty_ Ss+ tty4 0:00 /sbin/mingetty
/dev/tty4 4 0 3295 1 20 0 4060 576 n_tty_ Ss+ tty5
0:00 /sbin/mingetty /dev/tty5 4 0 3297 1 20 0 4060 572
n_tty_ Ss+ tty6 0:00 /sbin/mingetty /dev/tty6 4 0 19086
1 20 0 4060 572 n_tty_ Ss+ tty1 0:00 /sbin/mingetty
/dev/tty1 4 0 20837 20833 20 0 108432 2148 wait Ss pts/0
0:00 -bash 4 0 21471 20837 20 0 108124 1036 - R+ pts/0
0:00 ps l 0 0 21472 20837 20 0 100908 664 pipe_w S+ pts/0
0:00 tee -a test
Checked File
cat test
Output:-(Appended properly)
This is Test File ...............
F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND 4 0
3287 1 20 0 4060 572 n_tty_ Ss+ tty2 0:00
/sbin/mingetty /dev/tty2 4 0 3289 1 20 0 4060 572
n_tty_ Ss+ tty3 0:00 /sbin/mingetty /dev/tty3 4 0 3291
1 20 0 4060 576 n_tty_ Ss+ tty4 0:00 /sbin/mingetty
/dev/tty4 4 0 3295 1 20 0 4060 576 n_tty_ Ss+ tty5
0:00 /sbin/mingetty /dev/tty5 4 0 3297 1 20 0 4060 572
n_tty_ Ss+ tty6 0:00 /sbin/mingetty /dev/tty6 4 0 19086
1 20 0 4060 572 n_tty_ Ss+ tty1 0:00 /sbin/mingetty
/dev/tty1 4 0 20837 20833 20 0 108432 2148 wait Ss pts/0
0:00 -bash 4 0 21471 20837 20 0 108124 1036 - R+ pts/0
0:00 ps l 0 0 21472 20837 20 0 100908 664 pipe_w S+ pts/0
0:00 tee -a test