I'd like to run both traceroute -w2 and traceroute6 -w2, sequentially, in a shell script, to try out multiple different hosts.
A naive approach may just use a temporary variable to gather all the hosts within (e.g., set HOSTS to ordns.he.net one.one.one.one google-public-dns-a.google.com), and then just pipe it individually to each command, like, echo $HOSTS | xargs -n1 traceroute -w2 et al, but this would work differently in tcsh than in bash, and may be prone to mistakes if you want to add more commands (as you'd be adding those as code as opposed to a list of things to do), and I'm thinking there's some better way to join together the list of commands (e.g., a command name with a single parameter) with the list of arguments (e.g., hostnames in our example), for the shell to execute every possible combination.
I've tried doing some combination of xargs -n1 (for hosts) and xargs -n2 (for commands with one parameter) piping into each other, but it didn't really make much sense and didn't work.
I'm looking for a solution that doesn't use any GNU tools and would work in a base OpenBSD install (if necessary, perl is part of the base OpenBSD, so, it's available as well).
If you have perl:
perl -e 'for(#ARGV){ print qx{ traceroute -w2 -- $_; traceroute6 -w2 -- $_ } }' google.com debian.org
As for a better way to join together the list of commands (e.g., a command name with a single parameter) with the list of arguments (e.g., hostnames) the answer could be GNU Parallel, which is built for doing just that:
parallel "{1}" -w2 -- "{2}" ::: traceroute traceroute6 ::: google.com debian.org
If you want special arguments connected with each command you can do:
parallel eval "{1}" -- "{2}" ::: "traceroute -a -w2" "traceroute6 -w2" ::: google.com debian.org
The eval is needed because GNU Parallel quotes all input, and while you normally want that, we do not want that in this case.
But since that is a GNU tool, it is out of scope for your question. It is only included here for other people that read your question and who do not have that limitation.
Keeping it simple:
#!/bin/sh
set -- host1 host2 host3 host4 ...
for host do traceroute -w2 -- "$host"; done
for host do traceroute6 -w2 -- "$host"; done
With GNU Parallel, the final solution for the problem at stake would be something like the following snippet, using tcsh syntax, and OS X traceroute and traceroute6:
( history 1 ; parallel --keep-order -j4 eval \{1} -w1 -- \{2} '2>&1' ::: "traceroute -a -f1" traceroute6 ::: ordns.he.net ns{1,2,3,4,5}.he.net one.one.one.one google-public-dns-a.google.com resolver1.opendns.com ; history 1 ) | & mail -s "traceroute: ordns.he.net et al from X" receipient#example.org -f sender#example.org
Related
Is there a quick, easy, and efficient way of running iterations in this for loop in parallel?
for i in `seq 1 5000`; do
repid="$(printf "%05d" "$i")"
inp="${repid}.inp"
out="${repid}.out"
/command "$inp" "$out"
done
If you want to take advantage of all your lovely CPU cores that you paid Intel so handsomely for, turn to GNU Parallel:
seq -f "%05g" 5000 | parallel -k echo command {}.inp {}.out
If you like the look of that, run it again without the -k (which keeps the output in order) and without the echo. You may need to enclose the command in single quotes:
seq -f "%05g" 5000 | parallel '/command {}.inp {}.out'
It will run 1 instance per CPU core in parallel, but, if you want say 32 in parallel, use:
seq ... | parallel -j 32 ...
If you want an "estimated time of arrival", use:
parallel --eta ...
If you want a progress meter, use:
parallel --progress ...
If you have bash version 4+, it can zero-pad brace expansions. And if your ARGMAX is big enough, so you can more simply use:
parallel 'echo command {}.inp {}.out' ::: {00001..05000}
You can check your ARGMAX with:
sysctl -a kern.argmax
and it tells you how many bytes long your parameter list can be. You are going to need 5,000 numbers at 5 digits plus a space each, so 30,000 minimum.
If you are on macOS, you can install GNU Parallel with homebrew:
brew install parallel
for i in `seq 1 5000`; do
repid="$(printf "%05d" "$i")"
inp="${repid}.inp"
out="${repid}.out"
/command "$inp" "$out" &
done
Is there a more idiomatic way of doing the following:
cat some_lines.txt | while read x; do ./process_line.sh $x; done
ie. applying a script to each line of stdin?
I could include the while read x; boilerplate in the script itself, but that doesn't really feel right either.
If you're running an external process and have GNU xargs, consider:
xargs -n1 -d $'\n' ./process_line.sh <some_lines.txt
If you don't like the while read loop's verbosity, and are running a shell function (where a fork() isn't natively needed, and thus where using an external tool like xargs or GNU parallel has a substantial performance cost), you can avoid it by wrapping the loop in a function:
for_each_line() {
local line
while IFS= read -r line; do
"$#" "$line" </dev/null
done
}
...can be run as:
process_line() {
echo "Processing line: $1"
}
for_each_line process_line <some_lines.txt
GNU Parallel is made for this kind of tasks - provided there is no problem in running the processing in parallel:
cat some_lines.txt | parallel ./process_line.sh {}
By default it will run one job per cpu-core. This can be adjusted with --jobs.
There is an overhead of running it through GNU Parallel in the order of 5 ms per job. One of the benefits you get is that you are guaranteed the output from the different jobs are not jumbled together and you can therefore use use the output as if the jobs had not been run in parallel:
cat some_lines.txt | parallel ./process_line.sh {} | do_post_processing
GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to.
If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:
GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:
Installation
For security reasons you should install GNU Parallel with your package manager, but if GNU Parallel is not packaged for your distribution, you can do a personal installation, which does not require root access. It can be done in 10 seconds by doing this:
(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash
For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README
Learn more
See more examples: http://www.gnu.org/software/parallel/man.html
Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html
Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel
I've written a pretty long, and moderately complex Bash script that enables me to start my Node server with chosen options very easily...the problem is that it's not working correctly.
The part that is giving me trouble is here...
if netstat -an | grep ":$REQUESTED_PORT" > /dev/null
then
SERVICE_PIDS_STRING=`lsof -i tcp:$REQUESTED_PORT -t`
OLD_IFS="$IFS"
IFS='
'
read -a SERVICE_PIDS <<< "${SERVICE_PIDS_STRING}"
IFS="$OLD_IFS"
printf 'Port is in use by the following service(s)...\n\n-------------------\n\nProcess : PID\n\n'
for PID in "${SERVICE_PIDS[#]}"
do
PROCESS_NAME=`ps -p $PID -o comm=`
printf "$PROCESS_NAME : $PID\n"
done
printf "\n-------------------\n\nPlease kill the procceses utilizing port $REQUESTED_PORT and run this script again...exiting.\n"
exit
The intended function of this script is to use netstat to test if the requested port is busy. If so, it reports the PIDs utilizing the port so that the user can kill them if they wish.
I'm fairly certain this is a problem with the way I'm using netstat. Occasionally, the netstat if statement will trigger, even though there is nothing using the port. lsof is working correctly, and doesn't report any PIDs using the port.
However, when the last time the script made this error, I declared the REQUESTED_PORT and then ran netstat -an | grep ":$REQUESTED_PORT". The shell did not report anything.
What is the problem with this condition that causes it to fire at inappropriate times?
EDIT
I should also mention that this machine is running Debian Jessie.
You're searching an awful lot of text, and your desired number could show up anywhere. Better to narrow the search down; and you can grab your PIDs and process names in the same step. Some other optimizations follow:
# upper case variable names should be reserved for the shell
if service_pids_string=$(lsof +c 15 -i tcp:$requested_port)
then
# make an array with newline separated string containing spaces
# note we're only setting IFS for this one command
IFS=$'\n' read -r -d '' -a service_pids <<< "$service_pids_string"
# remove the first element containing column headers
service_pids=("${service_pids[#]:1}")
printf 'Port is in use by the following service(s)...\n\n-------------------\n\nProcess : PID\n\n'
for pid in "${service_pids[#]}"
do
# simple space-separated text to array
pid=($pid)
echo "${pid[0]} : ${pid[1]}"
done
# printf should be passed variables as parameters
printf "\n-------------------\n\nPlease kill the procceses utilizing port %s and run this script again...exiting.\n" $requested_port
fi
You should run your script through shellcheck.net; it will probably find other potential issues that I haven't.
I have written a script that accept two files as input. I want to run all in parallel at the same time on different CPUs.
inputs:
x00.A x00.B
x01.A x01.B
...
x30.A x30.B
instead of running 30 times:
./script x00.A x00.B
./script x01.A x01.B
...
./script x30.A x30.B
I wanted to use paste and seq to generate and send them to the script.
paste & seq | xargs -n1 -P 30 ./script
But I do not know how to combine letters and numbers using paste and seq commands.
for num in $(seq -f %02.f 0 30); do
./script x$num.A x$num.B &
done
wait
Although I personally prefer to not use GNU seq or BSD jot but (ksh/bash) builtins:
num=-1; while (( ++num <= 30 )); do
./script x$num.A x$num.B &
done
wait
The final wait is just needed to make sure they all finish, after having run spread across your available CPU cores in the background. So, if you need the output of ./script, you must keep the wait.
Putting them into the background with & is the simplest way for parallelism. If you really want to exercise any sort of control over lots of backgrounded jobs like that, you will need some sort of framework like GNU Parallel instead.
You can use pure bash for generating the sequence:
printf "%s %s\n" x{00..30}.{A..B} | xargs -n1 -P 30 ./script
Happy holidays!
I am, like many non-engineers or non-mathematicians who try writing algorithms, an intuitive. My exact psychological typology makes it quite difficult for me to learn anything serious like computers or math. Generally, I prefer audio, because I can engage my imagination more effectively in the learning process.
That said, I am trying to write a shell script that will help me master Linux. To that end, I copied and pasted a list of Linux commands from the O'Reilly website's index to the book Python In a Nutshell. I doubt they'll mind, and I thank them for providing it. These are the textfile `massivelistoflinuxcommands,' not included fully below in order to save space...
OK, now comes the fun part. How do I get this script to work?
#/bin/sh
read -d 'massivelistoflinuxcommands' commands <<EOF
accept
bison
bzcmp
bzdiff
bzgrep
bzip2
bzless
bzmore
c++
lastb
lastlog
strace
strfile
zmore
znew
EOF
for i in $commands
do
$i --help | less | cat > masterlinuxnow
text2wave masterlinuxnow -o ml.wav
done
It really helps when you include error messages or specific ways that something deviates from expected behavior.
However, your problem is here:
read -d 'massivelistoflinuxcommands' commands <<EOF
It should be:
read -d '' commands <<EOF
The delimiter to read causes it to stop at the first character it finds that matches the first character in the string, so it stops at "bzc" because the next character is "m" which matches the "m" at the beginning of "massive..."
Also, I have no idea what this is supposed to do:
$i --help | less | cat > masterlinuxnow
but it probably should be:
$i --help > masterlinuxnow
However, you should be able to pipe directly into text2wave and skip creating an intermediate file:
$i --help | text2wave -o ml.wav
Also, you may want to prevent each file from overwriting the previous one:
$i --help | text2wave -o ml-$i.wav
That will create files named like "ml-accept.wav" and "ml-bison.wav".
I would point out that if you're learning Linux commands, you should prioritize them by frequency of use and/or applicability to a beginner. For example, you probably won't be using bison right away`.
The first problem here is that not every command has a --help option!! In fact the very first command, accept, has no such option! A better approach might be executing man on each command since a manual page is more likely to exist for each of the commands. Thus change;
$i --help | less | cat > masterlinuxnow
to
man $i >> masterlinuxnow
note that it is essential you use the append output operator ">>" instead of the create output operator ">" in this loop. Using the create output operator will recreate the file "masterlinuxnow" on each iteration thus containing only the output of the last "man $i" processed.
you also need to worry about whether the command exists on your version of linux (many commands are not included in the standard distribution or may have different names). Thus you probably want something more like this where the -n in the head command should be replace by the number of lines you want, so if you want only the first 2 lines of the --help output you would replace -n with -2:
if [ $(which $i) ]
then
$i --help | head -n >> masterlinuxnow
fi
and instead of the read command, simply define the variable commands like so:
commands="
bison
bzcmp
bzdiff
bzgrep
bzip2
bzless
bzmore
c++
lastb
lastlog
strace
strfile
zmore
znew
"
Putting this all together, the following script works quite nicely:
commands="
bison
bzcmp
bzdiff
bzgrep
bzip2
bzless
bzmore
c++
lastb
lastlog
strace
strfile
zmore
znew
"
for i in $commands
do
if [ $(which $i) ]
then
$i --help | head -1 >> masterlinuxnow 2>/dev/null
fi
done
You're going to learn to use Linux by listening to help descriptions? I really think that's a bad idea.
Those help commands usually list every obscure option to a command, including many that you will never use-- especially as a beginner.
A guided tutorial or book would be much better. It would only present the commands and options that will be most useful. For example, that list of commands you gave has many that I don't know-- and I've been using Linux/Unix extensively for 10 years.