I'd like to see a run counter at the top of my watch output.
Ex: this command should print a count value which increments every 2 seconds, in addition to the output of my main command, which, in this case is just echo "hello" for the purposes of this demonstration:
export COUNT=0 && watch -n 2 'export COUNT=$((COUNT+1)); echo "Count = $COUNT" \
&& echo "hello"'
But, all it outputs is this, with the count always being 1 and never changing:
Count = 1
hello
How can I get this Count variable to increment every 2 seconds when watch runs the command?
Thanks #Inian for pointing this out in the comments. I've consulted the cross-site duplicate, slightly modified it, and come up with this:
count=0; while sleep 2 ; do clear; echo "$((count++))"; echo "hello" ; done
Replace echo "hello" with the real command you want to run once every sleep n second now, and it works perfectly.
There's no need to use watch at all, and, as a matter of fact, no way to use watch in this way either unless you write the variable to a file rather than to RAM, which I'd like to avoid since that's unnecessary wear and tear write/erase cycles on a solid state drive (SSD).
Now I can run this command to repeatedly build my asciidoctor document so I can view it in a webpage, hitting only F5 each time I make and save a change to view the updated HTML page.
count=0; while sleep 2 ; do clear; echo "$((count++))"; \
asciidoctor -D temp README.adoc ; done
Sample output:
96
asciidoctor: ERROR: README.adoc: line 6: level 0 sections can only be used when doctype is book
Final answer:
And, here's a slightly better version which prints count = 2 instead of just 2, and which also runs first and then sleeps, rather than sleeping first and then running:
count=1; while true ; do clear; echo "count = $((count++))"; \
asciidoctor -D temp README.adoc; sleep 2 ; done
...but, it looks like it's not just updating the file if it's changed, it's rewriting constantly, wearing down my disk anyway. So, I'd need to write a script and only run this if the source file has changed. Oh well...for the purposes of my original question, this answer is done.
Related
This question already has answers here:
Variable in Bash Script that keeps it value from the last time running
(3 answers)
Closed 7 years ago.
TL;DR: How do you set up a variable upon first run of a bash script and modify it upon further runs (something like a static variable in a function in C)?
Background info:
The following line in Bash is for playing "Command line Russian Roulette" (a much safer version of https://stackoverflow.com/a/575464/3696619):
[ $[ $RANDOM % 6 ] == 0 ] && echo *Boom* || echo *Click*
However, it doesn't really work like normal Russian Roulette (for which, within the first 6 tries, you are guaranteed to have a bullet in one of them).
Is there any way to make sure that this occurs (i.e. a *Boom* is guaranteed within the first 6 tries). The code should be repeatable (i.e. exact same code should be able to be copy pasted again and again, not parts of it) and it should work properly.
My idea is: Set up a variable that is set to $[$RANDOM % 6] initially and then decremented each time the code is run. If the variable reaches 0, the gun goes *Boom*, otherwise, it *Click*s. However, I am unable to figure out how to set up the variable only upon first run, and not on further runs.
Warning: Execute the code from https://stackoverflow.com/a/575464/3696619 only at your own risk.
Short answer: No, bash doesn't have "static" variables.
Longer: When your "C" program finishes its run, the "static" variable is lost. The same is applied for the bash script. So, you must decide what you want:
run the script once, simulate the revolver (and you can use global variable to hold the revolver status)
want preserve the revolver's cylinder status between each script run, so you need use some external storage to preserve the data. (for example file).
From you question i guessed the second version, so you must use file to hold the revolver status. For the game, you should divide the problem to different parts:
roll - (roll the revolver's cylinder (random bullet position))
shoot - (move the cylinder by one position and check the bullet)
I would do this as the following:
define the revolver for example as /tmp/revolver.
It will have 6 lines, in each line could be 2 values: 0 - no bullet, 1 - bullet
the hammer is in the 1st line - so, if the bullet is in the first line (e.g. the 1st line has value 1) the bullet will fire.
each "roll" ensures than exactly ONE bullet is in the cylinder
when shooting - once the bullet is fired, will not fire again, so any number of subsequent shots will not fire again
the roll "command". Defined as an bash function and saved as ./roll command.
revolver="/tmp/revolver"
roll() {
cyl=(0 0 0 0 0 0) # empty cylinder
cyl[$(($RANDOM % 6))]=1 # one bullet at random position
printf "%d\n" "${cyl[#]}" >"$revolver" # save
}
roll #do the roll
now, you can do bash roll (or after the chmod 755 roll) simply ./roll and your revolver is loaded with one bullet. Run the ./roll few times and after each ./roll check the bullet position with cat /tmp/revolver.
the shoot command should:
rotate the lines by one position (as in the real revolver)
and cock the hammer - e.g. check the value of the 1st line
revolver="/tmp/revolver"
rollone() {
at_hammer=$1 # store what is under the hammer
shift # shift the rest by one position
printf "%d\n" "$#" 0 > "$revolver" # save new cylinder the status
# note, we adding to the last position 0,
# because when the bullet is fired it will not fire again
# out digital revolver is not jamming
echo $at_hammer # return the bullet yes/no
}
shoot() {
cyl=($(<"$revolver")) #load the revolver status
return $(rollone "${cyl[#]}") #make the shoot, roll the cylinder and return the status
}
boom() { echo "Boom"; } #the "boom" action
click() { echo "Click"; } #the "click" action
shoot && click || boom #the actual shot
Now you can play the game:
a. ./roll - load the revolver with one bullet and roll the cylinder
b. ./shoot - any number of times
The whole game as script.
Variant A - roll once and shooting multiple times i
./roll
while :
do
./shoot
done
this will output something like:
Click
Click
Boom
Click
Click
... and forever ...
Click
Variant B - roll (e.g. reload with 1 bullet) between each shot
while :
do
./roll
./shoot
done
this will prints e.g.:
Click
Click
Boom
Boom
Click
Click
Click
Boom
Click
Click
Click
Click
Click
Click
Click
Click
Click
Boom
Click
... etc ...
Also, you could extend/modify the scripts with one more command: load and redefine your revolver as:
load - will load one (or more) bullet(s) into the cylinder
roll - will rotate the cylinder by the random number of positions (but NOT reloads the bullet) - e.g. after the fired bullet the roll will rotate only empty cylinder
shoot - fire the gun (no modification needed).
I don't think there's a different way for persisting the value without using a file. So, if you're going to use a file you could have the command below:
f=/tmp/a; test -f $f || echo $(($RANDOM%6))>$f; test $(<$f) == "0" && echo *Boom* && rm $f; test -f $f && echo *Click* && echo $(($(<$f)-1))>$f
What actually happens is:
Declare a variable $f where to store the name of the file that we use to store the variable.
f=/tmp/a;
Generate a random number between 0 and 5 and store it in the file:
test -f $f || echo $(($RANDOM%6))>$f;
If what we have in the file is 0 then print Boom and remove the file:
test $(<$f) == "0" && echo *Boom* && rm $f;
If the file is still there, print Click and decrement the value by 1:
test -f $f && echo *Click* && echo $(($(<$f)-1))>$f
I have a bash script which get details from many servers. It works good but i want that the lines get updated and not get written new.
while [ true ] ; do
for i in $(seq ${#details[#]}); do
.... more code (irrelevant)
echo ${server[$i]}
echo $stat1
echo $stat2
echo $stat3
echo $stat4
done
done
How can i do, that all lines get constantly updated into same line?
I try with echo -ne but this makes that everything is in one long line.
I want that the line keep the place and just get updated with new value.
Would be great if somebody knows a trick.
Thank you!
UPDATE 1
#cbuckley:
Thanks for your answer, but its not working correctly. In this way with -ne i tryed it already. Result is (it always create new lines):
10.0.0.2
100310.0.0.1
72710.0.0.3
368310.0.0.2
100310.0.0.1
72710.0.0.3
Should be
10.0.0.1
17
1003
10.0.0.2
319
727
10.0.0.3
157
3683
values under IP should get updated constantly. (i think this should normaly work with -ne, but in my case it dont).
If you've already outputted across multiple lines, you can't remove those lines without clearing the screen. You have two options:
Using watch
You can write a script that outputs the stats once, and then use watch to repeatedly that script:
watch -n 10 ./script.sh # calls script every 10 seconds.
Clearing the screen
If that is not suitable, you'll need to clear the screen yourself:
while [ true ] ; do
clear # clear the screen
for i in $(seq ${#details[#]}); do
# ...
done
sleep 10 # don't update the screen too often
done
However, at this point, you've pretty much implemented a basic version of watch anyway.
You might want to try using sed with the -i option to edit a file 'in place' (i.e. to change the existing file instead of writing a new file)
I would like to write a script to execute the steps outlined below. If someone can provide simple examples on how to modify files and search through folders using a script (not necessarily solving my problem below), I will greatly appreciate it.
submit job MyJob in currentDirectory using myJobShellFile.sh to a queue
upon completion of MyJob, goto to currentDirectory/myJobDataFolder.
In myJobDataFolder, there are folders
myJobData.0000 myJobData.0001 myJobData.0002 myJobData.0003
I want to find the maximum number maxIteration of all the listed folders. Here it would be maxIteration=0003.\
In file myJobShellFile.sh, at the last line says
mpiexec ./main input myJobDataFolder
I want to append this line to
'mpiexec ./main input myJobDataFolder 0003'
I want to submit MyJob to the que while maxIteration < 10
Upon completion of MyJob, find the new maxIteration and change this number in myJobShellFile.sh and goto step 4.
I think people write python scripts typically to do this stuff, but am having a hard time finding out how. I probably don't know the correct terminology for this procedure. I am also aware that the script will vary slightly depending on the queing system, but any help will be greatly appreciated.
Quite a few aspects of your question are unclear, such as the meaning of “submit job MyJob in currentDirectory using myJobShellFile.sh to a que”, “append this line to
'mpiexec ./main input myJobDataFolder 0003'”, how you detect when a job is done, relevant parts of myJobShellFile.sh, and some other details. If you can list the specific shell commands you use in each iteration of job submission, then you can post a better question, with a bash tag instead of python.
In the following script, I put a ### at the end of any line where I am guessing what you are talking about. Lines ending with ### may be irrelevant to whatever you actually do, or may be pseudocode. Anyway, the general idea is that the script is supposed to do the things you listed in your items 1 to 5. This script assumes that you have modified myJobShellFile.sh to say
mpiexec ./main input $1 $2
instead of
mpiexec ./main input
because it is simpler to use parameters to modify what you tell mpiexec than it is to keep modifying a shell script. Also, it seems to me you would want to increment maxIter before submitting next job, instead of after. If so, remove the # from the t=$((1$maxIter+1)); maxIter=${t#1} line. Note, see the “Parameter Expansion” section of man bash re expansion of the ${var#txt} form, and the “Arithmetic Expansion” section re $((expression)) form. The 1$maxIter and similar forms are used to change text like 0018 (which is not a valid bash number because 8 is not an octal digit) to 10018.
#!/bin/sh
./myJobShellFile.sh MyJob ###
maxIter=0
while true; do
waitforjobcompletion ###
cd ./myJobDataFolder
maxFile= $(ls myJobData* | tail -1)
maxIter= ${maxFile#myJobData.} #Get max extension
# If you want to increment maxIter, uncomment next line
# t=$((1$maxIter+1)); maxIter=${t#1}
cd ..
if [[ 1$maxIter -lt 11000 ]] ; then
./myJobShellFile.sh MyJobDataFolder $maxIter
else
break
fi
done
Notes: (1) To test with smaller runs than 1000 submissions, replace 11000 by 10000+n; for example, to do 123 runs, replace it with 10123. (2) In writing the above script, I assumed that not-previously-known numbers of output files appear in the output directory from time to time. If instead exactly one output file appears per run, and you just want to do one run per value for the values 0000, 0001, 0002, 0999, 1000, then use a script like the following. (For testing with a smaller number than 1000, replace 1000 with (eg) 0020. The leading zeroes in these numbers tell bash to fill the generated numbers with leading zeroes.)
#!/bin/sh
for iter in {0000..1000}; do
./myJobShellFile.sh MyJobDataFolder $iter
waitforjobcompletion ###
done
(3) If the system has a command that sleeps while it waits for a job to complete on the supercomputing resource, it is reasonable to use that command in place of waitforjobcompletion in the above scripts. Otherwise, if the system has a command jobisrunning that returns true if a job is still running, replace waitforjobcompletion with something like the following:
while jobisrunning ; do sleep 15; done
This will run the jobisrunning command; if it returns true, the shell will sleep for 15 seconds and then retest. Here is an example that illustrates waiting for a file to appear and then for it to go away:
while [ ! -f abc ]; do sleep 3; echo no abc; done
while ls abc >/dev/null 2>&1; do sleep 3; echo an abc; done
The second line's test could be [ -f abc ] instead; I showed a longer example to illustrate how to suppress output and error messages by routing them to /dev/null. (4) To reverse the sense of a while statement's test, replace the word while with until. For example, while [ ! -f abc ]; ... is equivalent to until [ -f abc ]; ....
I'm trying to run a piece of code on a large computer cluster in order to analyze different parts of the data.
I created 2 loops to assign the jobs to different nodes and the cpu's that the nodes contain.
The analysis function I wrote, 'chnJob()', just needs to take an index to know what part of the data it needs to analyze (it's the shell variable called 'chn' in this case).
the loop is like this:
for NODE in $NODES; do # Loop through nodes
for job_idx in {1..$PROCS_PER_NODE}; do # Loop through jobs per node (8 per node)
echo "this is the channel $chn"
ssh $NODE "matlab -nodisplay -nodesktop -nojvm -nosplash -r 'cd $WORK_DIR; chnJob($chn); quit'" &
let chn++
sleep 2
done
done
Even though I see that chn variable is being incremented properly, the value of chn that is passed to the matlab function is always the last value of the chn.
This is probably because matlab takes a lot of time to open on each node and bash finishes the loops by then. So the value that is being passed to each matlab instance is only the last value.
Is there a way to circumvent that? Can I 'bake' the value of that variable when I'm calling the function?
Or is the problem entirely different?
I don't think that's what's happening. Can you try running this:
cnt=0
for a in 1 2; do
for b in 1 2; do
echo --- $cnt
ssh somehost "echo result: '$cnt'" &
let cnt++
done
done
Replace somehost with some host where you have sshd running. This prints numbers 0 - 3 getting back from echo result: '$cnt' getting executed remotely. Thus, executing itself works OK.
One thing that I can suggest is for you to move your command (matlab ...) into some script in a known folder, then run that script in the above loops by giving a full path to that script. Something like:
ssh $NOTE "/path/to/script.sh $cnt"
In the script, $1 will give you the value you want (i.e. $cnt from the loop). You can use echo $1 >> /tmp/values at the beginning of your script to collect all the values in file /tmp/values. Of course, rm /tmp/values before you start. This will confirm whether you are getting all the values as you want them.
Bash can't handle variables in brace range expressions. They have to be literals: {1..10}. Because of the way you have it now, the inner loop is always executed exactly once per iteration of the outer loop instead of eight times (or whatever the value of PROCS_PER_NODE is). As a result, chn goes from its initial value to that plus NODES when it should go from Original_chn to NODES * PROCS_PER_NODE.
Use a C-style for loop instead:
for ((job_idx=1; job_idx<=$PROCS_PER_NODE; job_idx++))
You could increment both job_idx and chn in the for (if that doesn't give you off-by-one problems):
for ((job_idx=1; job_idx<=$PROCS_PER_NODE; job_idx++, chn++))
If $PBS_NODEFILE contains the filename with the list of nodes (one per line) then this should work:
seq 1 100 | parallel --slf $PBS_NODEFILE "matlab -nodisplay -nodesktop -nojvm -nosplash -r 'cd $WORK_DIR; chnJob({}); quit'"
Learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
I'm writing shell scripts where quite regularly some stuff is written
to a file, after which an application is executed that reads that file. I find that through our company the network latency differs vastly, so a simple sleep 2 for example will not be robust enough.
I tried to write a (configurable) timeout loop like this:
waitLoop()
{
local timeout=$1
local test="$2"
if ! $test
then
local counter=0
while ! $test && [ $counter -lt $timeout ]
do
sleep 1
((counter++))
done
if ! $test
then
exit 1
fi
fi
}
This works for test="[ -e $somefilename ]". However, testing existence is not enough, I sometimes need to test whether a certain string was written to the file. I tried
test="grep -sq \"^sometext$\" $somefilename", but this did not work. Can someone tell me why?
Are there other, less verbose options to perform such a test?
You can set your test variable this way:
test=$(grep -sq "^sometext$" $somefilename)
The reason your grep isn't working is that quotes are really hard to pass in arguments. You'll need to use eval:
if ! eval $test
I'd say the way to check for a string in a text file is grep.
What's your exact problem with it?
Also you might adjust your NFS mount parameters, to get rid of the root problem. A sync might also help. See NFS docs.
If you're wanting to use waitLoop in an "if", you might want to change the "exit" to a "return", so the rest of the script can handle the error situation (there's not even a message to the user about what failed before the script dies otherwise).
The other issue is using "$test" to hold a command means you don't get shell expansion when actually executing, just evaluating. So if you say test="grep \"foo\" \"bar baz\"", rather than looking for the three letter string foo in the file with the seven character name bar baz, it'll look for the five char string "foo" in the nine char file "bar baz".
So you can either decide you don't need the shell magic, and set test='grep -sq ^sometext$ somefilename', or you can get the shell to handle the quoting explicitly with something like:
if /bin/sh -c "$test"
then
...
Try using the file modification time to detect when it is written without opening it. Something like
old_mtime=`stat --format="%Z" file`
# Write to file.
new_mtime=$old_mtime
while [[ "$old_mtime" -eq "$new_mtime" ]]; do
sleep 2;
new_mtime=`stat --format="%Z" file`
done
This won't work, however, if multiple processes try to access the file at the same time.
I just had the exact same problem. I used a similar approach to the timeout wait that you include in your OP; however, I also included a file-size check. I reset my timeout timer if the file had increased in size since last it was checked. The files I'm writing can be a few gig, so they take a while to write across NFS.
This may be overkill for your particular case, but I also had my writing process calculate a hash of the file after it was done writing. I used md5, but something like crc32 would work, too. This hash was broadcast from the writer to the (multiple) readers, and the reader waits until a) the file size stops increasing and b) the (freshly computed) hash of the file matches the hash sent by the writer.
We have a similar issue, but for different reasons. We are reading s file, which is sent to an SFTP server. The machine running the script is not the SFTP server.
What I have done is set it up in cron (although a loop with a sleep would work too) to do a cksum of the file. When the old cksum matches the current cksum (the file has not changed for the determined amount of time) we know that the writes are complete, and transfer the file.
Just to be extra safe, we never overwrite a local file before making a backup, and only transfer at all when the remote file has two cksums in a row that match, and that cksum does not match the local file.
If you need code examples, I am sure I can dig them up.
The shell was splitting your predicate into words. Grab it all with $# as in the code below:
#! /bin/bash
waitFor()
{
local tries=$1
shift
local predicate="$#"
while [ $tries -ge 1 ]; do
(( tries-- ))
if $predicate >/dev/null 2>&1; then
return
else
[ $tries -gt 0 ] && sleep 1
fi
done
exit 1
}
pred='[ -e /etc/passwd ]'
waitFor 5 $pred
echo "$pred satisfied"
rm -f /tmp/baz
(sleep 2; echo blahblah >>/tmp/baz) &
(sleep 4; echo hasfoo >>/tmp/baz) &
pred='grep ^hasfoo /tmp/baz'
waitFor 5 $pred
echo "$pred satisfied"
Output:
$ ./waitngo
[ -e /etc/passwd ] satisfied
grep ^hasfoo /tmp/baz satisfied
Too bad the typescript isn't as interesting as watching it in real time.
Ok...this is a bit whacky...
If you have control over the file: you might be able to create a 'named pipe' here.
So (depending on how the writing program works) you can monitor the file in an synchronized fashion.
At its simplest:
Create the named pipe:
mkfifo file.txt
Set up the sync'd receiver:
while :
do
process.sh < file.txt
end
Create a test sender:
echo "Hello There" > file.txt
The 'process.sh' is where your logic goes : this will block until the sender has written its output. In theory the writer program won't need modifiying....
WARNING: if the receiver is not running for some reason, you may end up blocking the sender!
Not sure it fits your requirement here, but might be worth looking into.
Or to avoid synchronized, try 'lsof' ?
http://en.wikipedia.org/wiki/Lsof
Assuming that you only want to read from the file when nothing else is writing to it (ie, the writing process has finished) - you could check whether nothing else has file handle to it ?