I am trying to write a bash script that echoes values of 2 variables in parallel as part of an experiment before writing a shell script that generates files with numbers appended to them in parallel for a project of mine.
Here is the shell script:
#!/bin/bash
value1=0
value2=1
for i in $(seq 1 2); do
echo "Value 1 : " $((++value1)) &
echo "Value 2 : " $((++value2))
wait
echo "Wait"
done
And here is the output I get:
Value 2 : 2
Value 1 : 1
Wait
Value 2 : 3
Value 1 : 1
Wait
I know about GNU parallel and xargs but I don't want to use them.
I would like to know why 'value2' gets printed first and why 'value1' never gets incremented.
value1 is incremented, but in a separate process. value1 in the original process is not modified. There is no guarantee which value you will see printed first; the two echo statements occur in two different processes which are scheduled at the whim of the operating system.
Related
I'd like to see a run counter at the top of my watch output.
Ex: this command should print a count value which increments every 2 seconds, in addition to the output of my main command, which, in this case is just echo "hello" for the purposes of this demonstration:
export COUNT=0 && watch -n 2 'export COUNT=$((COUNT+1)); echo "Count = $COUNT" \
&& echo "hello"'
But, all it outputs is this, with the count always being 1 and never changing:
Count = 1
hello
How can I get this Count variable to increment every 2 seconds when watch runs the command?
Thanks #Inian for pointing this out in the comments. I've consulted the cross-site duplicate, slightly modified it, and come up with this:
count=0; while sleep 2 ; do clear; echo "$((count++))"; echo "hello" ; done
Replace echo "hello" with the real command you want to run once every sleep n second now, and it works perfectly.
There's no need to use watch at all, and, as a matter of fact, no way to use watch in this way either unless you write the variable to a file rather than to RAM, which I'd like to avoid since that's unnecessary wear and tear write/erase cycles on a solid state drive (SSD).
Now I can run this command to repeatedly build my asciidoctor document so I can view it in a webpage, hitting only F5 each time I make and save a change to view the updated HTML page.
count=0; while sleep 2 ; do clear; echo "$((count++))"; \
asciidoctor -D temp README.adoc ; done
Sample output:
96
asciidoctor: ERROR: README.adoc: line 6: level 0 sections can only be used when doctype is book
Final answer:
And, here's a slightly better version which prints count = 2 instead of just 2, and which also runs first and then sleeps, rather than sleeping first and then running:
count=1; while true ; do clear; echo "count = $((count++))"; \
asciidoctor -D temp README.adoc; sleep 2 ; done
...but, it looks like it's not just updating the file if it's changed, it's rewriting constantly, wearing down my disk anyway. So, I'd need to write a script and only run this if the source file has changed. Oh well...for the purposes of my original question, this answer is done.
I am learning DEBUG signal of bash.
The followings are my test code to regenerate the phenomenon of my question. So it does not have much meaning. Please don't care details.
It prepares two traps, one is called by EXIT signal to clean up temporal script originally but it is a dummy function here. And the second one is called by DEBUG signal to calculate the line where the debugger is scanning.
My question is DEBUG signal may be dispached at clean_up_debugger with LINENO = 0. Why is the LINENO 0 at that time? I add detail of my question after following output by bash -x. Please tell me why it happens.
Thank you very much.
#!/bin/bash
# file name is debug.working
source "bash_debugger_functions.sh"
trap clean_up_debugger_func EXIT
no_of_line_until_here=${LINENO} # *** no_of_line_until_here is 12 ***
trap "show_line_scanned \$(( \${LINENO} - ${no_of_line_until_here} - 1 ))" DEBUG
#!/bin/bash
echo "echo_sring = $1"
The following is a library file
#!/bin/bash
# file name is 'bash_debugger_functions.sh'
clean_up_debugger_func() {
echo "dummy"
}
show_line_scanned() {
echo "At line $1"
}
The following is a part of output by bash -x debug.working
+ (debug.working:17): echo 'echo_sring = test_message'
++ (debug.working:1): show_line_scanned -12
Just after "echo 'echo_sring = test_message'" is called, show_line_scanned is called with negative value, -12. no_of_line_until_here is +12. So it seems LINENO is 0 at that time. I don't know why the show_line_scanned is called here because I supposed that DEBUG signal is dispatched at each line but there is no new line after "echo "echo_sring = $1"". And I would like to know why LINENO is 0 here.
Please teach me the mechanism here.
I supposed that DEBUG signal is dispatched at each line but there is no new line after "echo "echo_sring = $1"".
Still there is a line executed after the last line of your debug.working script, because you set up the trap clean_up_debugger_func EXIT command, and for that clean_up_debugger_func command you get the DEBUG command dispatch which puzzles you.
And I would like to know why LINENO is 0 here.
The execution of the script has just ended at this time, and man bash states about LINENO:
When not in a
script or function, the value substituted is not guaranteed to
be meaningful.
This potentially is a stupid question or an impossible request. Anyway, I'm writing a little script to launch a program chosen from a list of installed software. Basically, my script presents me a numbered list, in which the programs are listed alphabetically, and I input the number corresponding to the program I want to launch. The variable in which my choice is stored is sent to a case, which launches the corresponding software, having its location stored in each case's command list.
i=1
echo -e "Which program to launch?\n"$((i++))". Program 1\n"$((i++))". Program 2\n"
read choice
case $choice in
1) path to program 1
2) path to program 2
esac
As you see, in the echo I've used a variable which gets incremented every time a new program is listed. This, to avoid having to manually write static numbers I have to personally shift every time a new program is installed and has to be inserted in the list between two existing programs. All I need to do is copy the universal $((i++)) index and the list adjusts itself.
The problem is I don't know how to implement this in the case cycle. Supposing I install a Program 3 which has to be alphabetically put between the two existing ones, the echo gets modified this way
echo -e "Which program to launch?\n"$((i++))". Program 1\n"$((i++))". Program 3\n"$((i++))". Program 2\n"
But in the case, I manually have to change the 2) before the second program into a 3).
case $choice in
1) path to program 1
2) path to program 3
3) path to program 2
esac
This may not be a problem in his example, but it is now that I have dozens of programs, and I have to change the 5 into a 6, the 6 into a 7 and so on until more than 20.
How can I automatize the case numbering, so that the cycle understands on its own that has to execute the n-th case if the variable value is n?
Bash already has select which does what you need:
#!/bin/bash
select choice in ls date 'ls /' ; do
$choice
break
done
If you want to present something different to what you run, you can use an associative array:
#!/bin/bash
declare -A choices=(
[show files]=ls
[show date]=date
[list root dir]='ls /'
)
select choice in "${!choices[#]}" exit ; do
[[ $choice == exit ]] && break
${choices[$choice]}
done
exit is handled outsice of the associative array as we want to keep it last, but associative arrays are unordered.
I would like to write a script to execute the steps outlined below. If someone can provide simple examples on how to modify files and search through folders using a script (not necessarily solving my problem below), I will greatly appreciate it.
submit job MyJob in currentDirectory using myJobShellFile.sh to a queue
upon completion of MyJob, goto to currentDirectory/myJobDataFolder.
In myJobDataFolder, there are folders
myJobData.0000 myJobData.0001 myJobData.0002 myJobData.0003
I want to find the maximum number maxIteration of all the listed folders. Here it would be maxIteration=0003.\
In file myJobShellFile.sh, at the last line says
mpiexec ./main input myJobDataFolder
I want to append this line to
'mpiexec ./main input myJobDataFolder 0003'
I want to submit MyJob to the que while maxIteration < 10
Upon completion of MyJob, find the new maxIteration and change this number in myJobShellFile.sh and goto step 4.
I think people write python scripts typically to do this stuff, but am having a hard time finding out how. I probably don't know the correct terminology for this procedure. I am also aware that the script will vary slightly depending on the queing system, but any help will be greatly appreciated.
Quite a few aspects of your question are unclear, such as the meaning of “submit job MyJob in currentDirectory using myJobShellFile.sh to a que”, “append this line to
'mpiexec ./main input myJobDataFolder 0003'”, how you detect when a job is done, relevant parts of myJobShellFile.sh, and some other details. If you can list the specific shell commands you use in each iteration of job submission, then you can post a better question, with a bash tag instead of python.
In the following script, I put a ### at the end of any line where I am guessing what you are talking about. Lines ending with ### may be irrelevant to whatever you actually do, or may be pseudocode. Anyway, the general idea is that the script is supposed to do the things you listed in your items 1 to 5. This script assumes that you have modified myJobShellFile.sh to say
mpiexec ./main input $1 $2
instead of
mpiexec ./main input
because it is simpler to use parameters to modify what you tell mpiexec than it is to keep modifying a shell script. Also, it seems to me you would want to increment maxIter before submitting next job, instead of after. If so, remove the # from the t=$((1$maxIter+1)); maxIter=${t#1} line. Note, see the “Parameter Expansion” section of man bash re expansion of the ${var#txt} form, and the “Arithmetic Expansion” section re $((expression)) form. The 1$maxIter and similar forms are used to change text like 0018 (which is not a valid bash number because 8 is not an octal digit) to 10018.
#!/bin/sh
./myJobShellFile.sh MyJob ###
maxIter=0
while true; do
waitforjobcompletion ###
cd ./myJobDataFolder
maxFile= $(ls myJobData* | tail -1)
maxIter= ${maxFile#myJobData.} #Get max extension
# If you want to increment maxIter, uncomment next line
# t=$((1$maxIter+1)); maxIter=${t#1}
cd ..
if [[ 1$maxIter -lt 11000 ]] ; then
./myJobShellFile.sh MyJobDataFolder $maxIter
else
break
fi
done
Notes: (1) To test with smaller runs than 1000 submissions, replace 11000 by 10000+n; for example, to do 123 runs, replace it with 10123. (2) In writing the above script, I assumed that not-previously-known numbers of output files appear in the output directory from time to time. If instead exactly one output file appears per run, and you just want to do one run per value for the values 0000, 0001, 0002, 0999, 1000, then use a script like the following. (For testing with a smaller number than 1000, replace 1000 with (eg) 0020. The leading zeroes in these numbers tell bash to fill the generated numbers with leading zeroes.)
#!/bin/sh
for iter in {0000..1000}; do
./myJobShellFile.sh MyJobDataFolder $iter
waitforjobcompletion ###
done
(3) If the system has a command that sleeps while it waits for a job to complete on the supercomputing resource, it is reasonable to use that command in place of waitforjobcompletion in the above scripts. Otherwise, if the system has a command jobisrunning that returns true if a job is still running, replace waitforjobcompletion with something like the following:
while jobisrunning ; do sleep 15; done
This will run the jobisrunning command; if it returns true, the shell will sleep for 15 seconds and then retest. Here is an example that illustrates waiting for a file to appear and then for it to go away:
while [ ! -f abc ]; do sleep 3; echo no abc; done
while ls abc >/dev/null 2>&1; do sleep 3; echo an abc; done
The second line's test could be [ -f abc ] instead; I showed a longer example to illustrate how to suppress output and error messages by routing them to /dev/null. (4) To reverse the sense of a while statement's test, replace the word while with until. For example, while [ ! -f abc ]; ... is equivalent to until [ -f abc ]; ....
I'm trying to run a piece of code on a large computer cluster in order to analyze different parts of the data.
I created 2 loops to assign the jobs to different nodes and the cpu's that the nodes contain.
The analysis function I wrote, 'chnJob()', just needs to take an index to know what part of the data it needs to analyze (it's the shell variable called 'chn' in this case).
the loop is like this:
for NODE in $NODES; do # Loop through nodes
for job_idx in {1..$PROCS_PER_NODE}; do # Loop through jobs per node (8 per node)
echo "this is the channel $chn"
ssh $NODE "matlab -nodisplay -nodesktop -nojvm -nosplash -r 'cd $WORK_DIR; chnJob($chn); quit'" &
let chn++
sleep 2
done
done
Even though I see that chn variable is being incremented properly, the value of chn that is passed to the matlab function is always the last value of the chn.
This is probably because matlab takes a lot of time to open on each node and bash finishes the loops by then. So the value that is being passed to each matlab instance is only the last value.
Is there a way to circumvent that? Can I 'bake' the value of that variable when I'm calling the function?
Or is the problem entirely different?
I don't think that's what's happening. Can you try running this:
cnt=0
for a in 1 2; do
for b in 1 2; do
echo --- $cnt
ssh somehost "echo result: '$cnt'" &
let cnt++
done
done
Replace somehost with some host where you have sshd running. This prints numbers 0 - 3 getting back from echo result: '$cnt' getting executed remotely. Thus, executing itself works OK.
One thing that I can suggest is for you to move your command (matlab ...) into some script in a known folder, then run that script in the above loops by giving a full path to that script. Something like:
ssh $NOTE "/path/to/script.sh $cnt"
In the script, $1 will give you the value you want (i.e. $cnt from the loop). You can use echo $1 >> /tmp/values at the beginning of your script to collect all the values in file /tmp/values. Of course, rm /tmp/values before you start. This will confirm whether you are getting all the values as you want them.
Bash can't handle variables in brace range expressions. They have to be literals: {1..10}. Because of the way you have it now, the inner loop is always executed exactly once per iteration of the outer loop instead of eight times (or whatever the value of PROCS_PER_NODE is). As a result, chn goes from its initial value to that plus NODES when it should go from Original_chn to NODES * PROCS_PER_NODE.
Use a C-style for loop instead:
for ((job_idx=1; job_idx<=$PROCS_PER_NODE; job_idx++))
You could increment both job_idx and chn in the for (if that doesn't give you off-by-one problems):
for ((job_idx=1; job_idx<=$PROCS_PER_NODE; job_idx++, chn++))
If $PBS_NODEFILE contains the filename with the list of nodes (one per line) then this should work:
seq 1 100 | parallel --slf $PBS_NODEFILE "matlab -nodisplay -nodesktop -nojvm -nosplash -r 'cd $WORK_DIR; chnJob({}); quit'"
Learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1