I'm trying to create a bash script that builds up a command to execute (that includes arguments). The name of the command executable is ms (which lives in the ms directory) and it takes a bunch of parameters that I compute and store in a string. When it comes time to execute the command, I try:
GENETREES=$(../ms/${x:1})
but am getting the error message:
./simulate.sh: line 21: ../ms/ms 6 1 -T -I 6 1 1 1 1 1 1 -ej 0.059851352500000010170566611122922040522098541259765625 1 4 -es 0.059851352500000010170566611122922040522098541259765625 4 0.3457841801761454281205487859551794826984405517578125 -ej 0.059851352500000010170566611122922040522098541259765625 2 3 -es 0.059851352500000010170566611122922040522098541259765625 3 0.54870128110803395582451003065216355025768280029296875 -ej 0.0897770262499999471828004971030168235301971435546875 7 9 -es 0.089777026250000002693951728360843844711780548095703125 3 0.8097582153199012200417428175569511950016021728515625 -ej 0.119702699999999995217336845598765648901462554931640625 3 4 -ej 0.125827642499999947656164067666395567357540130615234375 9 10 -es 0.1258276425000000031673152989242225885391235351562500 4 0.28069295861466903030390085405088029801845550537109375 -ej 0.13195258499999995560614252099185250699520111083984375 8 10 -ej 0.1817980399999999663318561715641408227384090423583984375 6 5 -ej 0.2525933399999999717788767839010688476264476776123046875 10 4 -ej 0.41145434999999996872332985731191001832485198974609375 4 5 : File name too long
I think bash thinks that I intended all those command parameters to be a part of the executable name. But this is not my intent.
What am I doing wrong?
Thanks.
Update - More info requested on how x is constructed
MSOUT=$(java -jar ./NetworkSearchGen.jar ms $3 $TRUENETWORK $4)
OIFS=$IFS
IFS='}'
first="true"
for x in $MSOUT
do
if [ $first = "true" ]; then
echo "$x"
else
GENETREES=$(../ms/${x:1})
fi
first="false"
done
It was setting IFS that screwed things up. Making IFS=" " just before executing got the script working again.
Related
I need to know if Bash has some solution for my case. I need after some conditions to do a "double return". I mean, to perform a return of a function and also return the parent function to skip the rest of the code of that parent function.
I know that I can do a conditional using function return values to achieve this. But I'd like to know if in Bash exist something like "break 2" for functions. I don't want if possible to modify the code of the parent function because as you can imagine, in my real script there are dozens of functions and I don't want to modify all of them.
Example:
#!/bin/bash
function sublevelone() {
echo "sublevelone"
# Return 2, or break 2 or something :D
}
function main() {
sublevelone
echo "This is the part of the code to being avoid executed"
}
main
I don't know what the bash experts will think, but this works at least for simple cases:
multireturn(){
[ -n "$1" ] && poplevel="$1"
if [ "$poplevel" -ge 0 ]; then
trap multireturn DEBUG
shopt -s extdebug
(( poplevel-- ))
return 2
else
shopt -u extdebug
trap - DEBUG
return 0
fi
}
This makes use of the DEBUG trap and the extdebug flag:
extdebug
If set at shell invocation, arrange to execute the
debugger profile before the shell starts, identical to
the --debugger option. If set after invocation, behav-
ior intended for use by debuggers is enabled:
1. The -F option to the declare builtin displays the
source file name and line number corresponding to
each function name supplied as an argument.
2. If the command run by the DEBUG trap returns a
non-zero value, the next command is skipped and
not executed.
3. If the command run by the DEBUG trap returns a
value of 2, and the shell is executing in a sub-
routine (a shell function or a shell script exe-
cuted by the . or source builtins), the shell
simulates a call to return.
4. BASH_ARGC and BASH_ARGV are updated as described
in their descriptions above.
5. Function tracing is enabled: command substitu-
tion, shell functions, and subshells invoked with
( command ) inherit the DEBUG and RETURN traps.
6. Error tracing is enabled: command substitution,
shell functions, and subshells invoked with (
command ) inherit the ERR trap.
Example usage:
#!/bin/bash
multireturn(){
[ -n "$1" ] && poplevel="$1"
if [ "$poplevel" -ge 0 ]; then
trap multireturn DEBUG
shopt -s extdebug
(( poplevel-- ))
return 2
else
shopt -u extdebug
trap - DEBUG
return 0
fi
}
# define 8 levels of function calls
# (level N prints output, calls level N+1, then prints more output)
for i in $(seq 1 8); do
eval \
'level'$i'(){
echo -n " '$i'"
level'$((i+1))'
echo -n "('$i')"
}'
done
# final level calls multireturn
level9(){
echo -n " 9"
multireturn $n
echo -n "(9)"
}
# test various skip amounts
for i in $(seq 0 10); do
echo -n "$i:"
n=$i
level1
echo .
done
echo
echo done
Result:
0: 1 2 3 4 5 6 7 8 9(9)(8)(7)(6)(5)(4)(3)(2)(1).
1: 1 2 3 4 5 6 7 8 9(8)(7)(6)(5)(4)(3)(2)(1).
2: 1 2 3 4 5 6 7 8 9(7)(6)(5)(4)(3)(2)(1).
3: 1 2 3 4 5 6 7 8 9(6)(5)(4)(3)(2)(1).
4: 1 2 3 4 5 6 7 8 9(5)(4)(3)(2)(1).
5: 1 2 3 4 5 6 7 8 9(4)(3)(2)(1).
6: 1 2 3 4 5 6 7 8 9(3)(2)(1).
7: 1 2 3 4 5 6 7 8 9(2)(1).
8: 1 2 3 4 5 6 7 8 9(1).
9: 1 2 3 4 5 6 7 8 9.
10: 1 2 3 4 5 6 7 8 9
done
This is kinda whacky but if you use parentheses to define levelone, it will execute the function in a subshell and then you can exit out of that shell from an inner function. That said, I think it's more appropriate to use return to send back value that you check for in the parent function.
#!/bin/bash
function leveltwo() {
echo "two"
exit
}
function levelone() (
echo "one"
leveltwo
echo "three"
)
levelone
echo "four"
Will print:
one
two
four
Why don't you return an exit status from the function and add an if-statement to main like you normally would in any other programming/scripting language?
#!/bin/bash
function sublevelone() {
echo "sublevelone"
[ 0 -eq 1 ]
return $? # Returns 1 in this case because 0 != 1
}
function main() {
sublevelone
[ $? -eq 0 ] || return 1 # Return in case of error
echo "This is the part of the code to being avoid executed"
}
main
exit 0
Just a short version of #jhnc answer for exactly 2-level return from a function:
trap 'trap "shopt -u extdebug; trap - DEBUG; return 0" DEBUG; return 2' DEBUG
shopt -s extdebug
return
I'm attempting to use a simple bash script to sequentially run a batch of MPI jobs. This script works perfectly when running serial code (I am using Fortran 90), but for some reason bash breaks out of the loop when I attempt to execute MPI code.
I already found a work-around to the problem. I just wrote essentially the exact same script in Perl and it worked like a charm. I just really want to understand the issue here because I prefer the simplicity of bash and it perfectly fits my own scripting needs in almost all other cases.
I've tried running the MPI code as a background process and using wait with the same result. If I run the jobs in the background without using wait, bash does not break out of the loop, but it stacks up jobs until eventually crashing. The goal is to run the executable sequentially for each parameter set anyway, I just wanted to note that the loop is not broken in that case.
Bash Script, interp.sh: Usage --> $ ./interp.sh inputfile
#!/bin/bash
PROG=$1
IFILE=$2
kount=0 # Counter variable for looping through input file
sys=0 # Counter variable to store how many times model has been run
while IFS="\n" read -r line
do
kount=$(( $kount + 1 ))
if [ $(( kount % 2 )) -eq 1 ] # if kount is even, then expect headers
then
unset name defs
sys=$(( $sys + 1 ))
name=( $line ) # parse headers
defs=${#name[*]}
k=$(( $defs - 1 ))
else # if count is odd, then expect numbers
unset vals
vals=( $line ) # parse parameters
for i in $( seq 0 $k )
do
# Define variables using header names and set their values
printf -v "${name[i]}" "${vals[i]}"
done
# Print input variable values
echo $a $b $c $d $e $nPROC
# Run executable
mpiexec -np $nPROC --oversubscribe --hostfile my_hostfile $PROG
fi
done < $IFILE
Input file, input.dat:
a b c d e nPROC
1 2 3 4 5 2
nPROC
3
nPROC
4
nPROC
5
nPROC
6
nPROC
7
nPROC
8
Sample MPI f90 code, main.f90:
program main
use mpi
implicit none
integer :: i, ierr, myID, nPROC
integer, parameter :: foolen = 100000
double precision, dimension(0:foolen) :: foo
call MPI_INIT(ierr)
call MPI_COMM_SIZE(MPI_COMM_WORLD, nPROC, ierr)
call MPI_COMM_RANK(MPI_COMM_WORLD, myID, ierr)
if ( myID .eq. 0 ) then
do i=0,foolen
foo(i) = i
end do
else
do i=0,foolen
foo(i) = i
end do
end if
call MPI_FINALIZE(ierr)
end program
Sample makefile:
COMP=mpif90
EXT=f90
CFLAGs=-Wall -Wextra -Wimplicit-interface -fPIC -fmax-errors=1 -g -fcheck=all \
-fbacktrace
MPIflags=--oversubscribe --hostfile my_hostfile
PROG=main.x
INPUT=input.dat
OUTPUT=output
OBJS=main.o
$(PROG): $(OBJS)
$(COMP) $(CFLAGS) -o $(PROG) $(OBJS) $(LFLAGS)
main.o: main.f90
$(COMP) -c $(CFLAGs) main.f90
%.o: %.f90
$(COMP) -c $(CFLAGs) $<
run:
make && make clean
./interp.sh $(PROG) $(INPUT)
clean:
rm -f *.o DONE watch
my_hostfile
localhost slots=4
Note that if the mpiexec line is commented out, the script runs as expected. The output looks like this:
1 2 3 4 5 2
1 2 3 4 5 3
1 2 3 4 5 4
1 2 3 4 5 5
1 2 3 4 5 6
1 2 3 4 5 7
1 2 3 4 5 8
These are the parameter values which are supposed to be passed to the MPI code in each loop. However, when mpiexec is called in the script, only the first set of parameters is read and passed.
I apologize if all that is a bit excessive, I just wanted to provide all that is needed for testing. Any help solving the issue in bash or explanation of why this happens would be greatly appreciated!
mpiexec is consuming the stdin thus reading all remaining lines in the loop. So after the first loop stdin is empty and the loop breaks.
This is an issue that occurs not only with loops calling mpiexec from whithin but also with loops other commands that consumes stdin by default such as ssh.
The general solution is to use < /dev/null so that the offending command won't consume stdin but the /dev/null instead. Some commands have special flags to replace the redirect command such as ssh -n
so the solution in this case would be to add the redirect at the end of the line where mpiexec is called:
mpiexec -np $nPROC --oversubscribe --hostfile my_hostfile $PROG < /dev/null
there are some issues to pay attention to in the case of mpiexec related to Standard I/O detailed here: https://www.open-mpi.org/doc/v3.0/man1/mpiexec.1.php#toc14
Value "4" below is the number of CPU threads. Idea is to run the tasks in batch of 4 and wait until the current batch is finished before starting the next batch.
Syntax 1:
while read something; do
((++i%4==0)) && wait
(
task using something as input;
)
done < input_file.txt
Syntax 2:
while read something; do
((i=i%4)); ((i++==0)) && wait
(
task using something as input;
)
done < input_file.txt
To me they both work the same except the second one is longer. But when running in the cloud (AWS ubuntu 14.04), only syntax 1 worked. The syntax2 threw a generic syntax error at "((i=i%4));" step and it became a mystery.
"The second one is longer" doesn't help since you used pseudocode.
Maybe this will help:
while read x; do ((i=++i%4)) || wait; sleep $x & done < input_file.txt
My input_file.txt:
10
9
8
7
6
5
4
3
2
1
The goal of code is that, I want to make a random tcp traffic using iperf and capture it over 5, 10, 15, 20 seconds using tcpdump. In addition, capturing the throughput is also important for me. My problem is that, I would like to execute code1, code2, code3 and code4 for 5, 10, 15 and 20 seconds in bash. However I don't know how to put the mentioned condition for it. Here is my code:
for Test_duration in 5 10 15 20
do
echo “Test performing with $Test_duration duration”
sudo tcpdump -G 10 -W 2 -w /tmp/scripttest_$Test_duration -i h1-eth0 &
while true; do
#code1
time2=$(($RANDOM%20+1))&
pksize2=$(($RANDOM%1000+200))&
iperf -c 10.0.0.2 -t $time2 -r -l $pksize2 >> /media/sf_sharedsaeed/throughtput/iperthroughput_host2_$Test_duration.txt &\
#code2
time3=$(($RANDOM%20+1))&
pksize3=$(($RANDOM%1000+200))&
iperf -c 10.0.0.3 -t $time3 -r -l $pksize3 >> /media/sf_sharedsaeed/throughtput/iperthroughput_host3_$Test_duration.txt &\
#code3
time4=$(($RANDOM%20+1))&
pksize4=$(($RANDOM%1000+200))&
iperf -c 10.0.0.4 -t $time4 -r -l $pksize4 >> /media/sf_sharedsaeed/throughtput/iperthroughput_host4_$Test_duration.txt &\
#code4
time5=$(($RANDOM%20+1))&
pksize5=$(($RANDOM%1000+200))&
iperf -c 10.0.0.5 -t $time5 -r -l $pksize5 >> /media/sf_sharedsaeed/throughtput/iperthroughput_host5_$Test_duration.txt &\
done
done
Another constraint is that, code1, code2, code3 and code4 should be executed at the same time so, I used &.
Please help me what should I replace instead of while true; to have periodic execution of codes. Can any body help me?
You could do that by using a background subshell that creates a simple file lock on expiration that you detect from your while loops. Here is an example based on a simplified version of your code:
for Test_duration in 5 10 15 20
do
# TIMEOUT_LOCK will be your file lock
rm -f TIMEOUT_LOCK
# next command will run in a parallel subshell at the background
(sleep $Test_duration; touch TIMEOUT_LOCK) &
echo “Test performing with $Test_duration duration”
while true; do
# check whether current timeout (5 or 10 or 15 ...) has occured
if [ -f TIMEOUT_LOCK ]; then rm -f TIMEOUT_LOCK; break; fi
# do your stuff here - I'm just outputing dots and sleeping
echo -n "."
sleep 1
done
echo ""
done
The output of this code is:
“Test performing with 5 duration”
.....
“Test performing with 10 duration”
..........
“Test performing with 15 duration”
...............
“Test performing with 20 duration”
....................
I used homebrew to install GNU parallel on my mac so I can run some tests remotely on my University's servers. I was quickly running through the tutorials, but when I ran
parallel -S <username>#$SERVER1 echo running on ::: <username>#$SERVER1
I got the message
parallel: Warning: Could not figure out number of cpus on <username#server> (). Using 1.
Possibly related, I never added parallel to my path and got the warning that "parallel" wasn't a recognized command, but parallel ran anyways and still echo'd correctly. This particular server has 16 cores, how can I get parallel to recognize them?
GNU Parallel is less tested on OS X as I do not have access to an OS X installation, so you have likely found a bug.
GNU Parallel has since 20120322 used these to find the number of CPUs:
sysctl -n hw.physicalcpu
sysctl -a hw 2>/dev/null | grep [^a-z]physicalcpu[^a-z] | awk '{ print \$2 }'
And the number of cores:
sysctl -n hw.logicalcpu
sysctl -a hw 2>/dev/null | grep [^a-z]logicalcpu[^a-z] | awk '{ print \$2 }'
Can you test what output you get from those?
Which version of GNU Parallel are you using?
As a work around you can force GNU Parallel to detect 16 cores:
parallel -S 16/<username>#$SERVER1 echo running on ::: <username>#$SERVER1
Since version 20140422 you have been able to export your path to the remote server:
parallel --env PATH -S 16/<username>#$SERVER1 echo running on ::: <username>#$SERVER1
That way you just need to add the dir where parallel lives on the server to your path on local machine. E.g. parallel on the remote server is in /home/u/user/bin/parallel:
PATH=$PATH:/home/u/user/bin parallel --env PATH -S <username>#$SERVER1 echo running on ::: <username>#$SERVER1
Information for Ole
My iMac (OSX MAvericks on Intel core i7) gives the following, which all looks correct:
sysctl -n hw.physicalcpu
4
sysctl -a hw
hw.ncpu: 8
hw.byteorder: 1234
hw.memsize: 17179869184
hw.activecpu: 8
hw.physicalcpu: 4
hw.physicalcpu_max: 4
hw.logicalcpu: 8
hw.logicalcpu_max: 8
hw.cputype: 7
hw.cpusubtype: 4
hw.cpu64bit_capable: 1
hw.cpufamily: 1418770316
hw.cacheconfig: 8 2 2 8 0 0 0 0 0 0
hw.cachesize: 17179869184 32768 262144 8388608 0 0 0 0 0 0
hw.pagesize: 4096
hw.busfrequency: 100000000
hw.busfrequency_min: 100000000
hw.busfrequency_max: 100000000
hw.cpufrequency: 3400000000
hw.cpufrequency_min: 3400000000
hw.cpufrequency_max: 3400000000
hw.cachelinesize: 64
hw.l1icachesize: 32768
hw.l1dcachesize: 32768
hw.l2cachesize: 262144
hw.l3cachesize: 8388608
hw.tbfrequency: 1000000000
hw.packages: 1
hw.optional.floatingpoint: 1
hw.optional.mmx: 1
hw.optional.sse: 1
hw.optional.sse2: 1
hw.optional.sse3: 1
hw.optional.supplementalsse3: 1
hw.optional.sse4_1: 1
hw.optional.sse4_2: 1
hw.optional.x86_64: 1
hw.optional.aes: 1
hw.optional.avx1_0: 1
hw.optional.rdrand: 0
hw.optional.f16c: 0
hw.optional.enfstrg: 0
hw.optional.fma: 0
hw.optional.avx2_0: 0
hw.optional.bmi1: 0
hw.optional.bmi2: 0
hw.optional.rtm: 0
hw.optional.hle: 0
hw.cputhreadtype: 1
hw.machine = x86_64
hw.model = iMac12,2
hw.ncpu = 8
hw.byteorder = 1234
hw.physmem = 2147483648
hw.usermem = 521064448
hw.pagesize = 4096
hw.epoch = 0
hw.vectorunit = 1
hw.busfrequency = 100000000
hw.cpufrequency = 3400000000
hw.cachelinesize = 64
hw.l1icachesize = 32768
hw.l1dcachesize = 32768
hw.l2settings = 1
hw.l2cachesize = 262144
hw.l3settings = 1
hw.l3cachesize = 8388608
hw.tbfrequency = 1000000000
hw.memsize = 17179869184
hw.availcpu = 8
sysctl -n hw.logicalcpu
8