I have a bash script which get details from many servers. It works good but i want that the lines get updated and not get written new.
while [ true ] ; do
for i in $(seq ${#details[#]}); do
.... more code (irrelevant)
echo ${server[$i]}
echo $stat1
echo $stat2
echo $stat3
echo $stat4
done
done
How can i do, that all lines get constantly updated into same line?
I try with echo -ne but this makes that everything is in one long line.
I want that the line keep the place and just get updated with new value.
Would be great if somebody knows a trick.
Thank you!
UPDATE 1
#cbuckley:
Thanks for your answer, but its not working correctly. In this way with -ne i tryed it already. Result is (it always create new lines):
10.0.0.2
100310.0.0.1
72710.0.0.3
368310.0.0.2
100310.0.0.1
72710.0.0.3
Should be
10.0.0.1
17
1003
10.0.0.2
319
727
10.0.0.3
157
3683
values under IP should get updated constantly. (i think this should normaly work with -ne, but in my case it dont).
If you've already outputted across multiple lines, you can't remove those lines without clearing the screen. You have two options:
Using watch
You can write a script that outputs the stats once, and then use watch to repeatedly that script:
watch -n 10 ./script.sh # calls script every 10 seconds.
Clearing the screen
If that is not suitable, you'll need to clear the screen yourself:
while [ true ] ; do
clear # clear the screen
for i in $(seq ${#details[#]}); do
# ...
done
sleep 10 # don't update the screen too often
done
However, at this point, you've pretty much implemented a basic version of watch anyway.
You might want to try using sed with the -i option to edit a file 'in place' (i.e. to change the existing file instead of writing a new file)
Related
I'd like to see a run counter at the top of my watch output.
Ex: this command should print a count value which increments every 2 seconds, in addition to the output of my main command, which, in this case is just echo "hello" for the purposes of this demonstration:
export COUNT=0 && watch -n 2 'export COUNT=$((COUNT+1)); echo "Count = $COUNT" \
&& echo "hello"'
But, all it outputs is this, with the count always being 1 and never changing:
Count = 1
hello
How can I get this Count variable to increment every 2 seconds when watch runs the command?
Thanks #Inian for pointing this out in the comments. I've consulted the cross-site duplicate, slightly modified it, and come up with this:
count=0; while sleep 2 ; do clear; echo "$((count++))"; echo "hello" ; done
Replace echo "hello" with the real command you want to run once every sleep n second now, and it works perfectly.
There's no need to use watch at all, and, as a matter of fact, no way to use watch in this way either unless you write the variable to a file rather than to RAM, which I'd like to avoid since that's unnecessary wear and tear write/erase cycles on a solid state drive (SSD).
Now I can run this command to repeatedly build my asciidoctor document so I can view it in a webpage, hitting only F5 each time I make and save a change to view the updated HTML page.
count=0; while sleep 2 ; do clear; echo "$((count++))"; \
asciidoctor -D temp README.adoc ; done
Sample output:
96
asciidoctor: ERROR: README.adoc: line 6: level 0 sections can only be used when doctype is book
Final answer:
And, here's a slightly better version which prints count = 2 instead of just 2, and which also runs first and then sleeps, rather than sleeping first and then running:
count=1; while true ; do clear; echo "count = $((count++))"; \
asciidoctor -D temp README.adoc; sleep 2 ; done
...but, it looks like it's not just updating the file if it's changed, it's rewriting constantly, wearing down my disk anyway. So, I'd need to write a script and only run this if the source file has changed. Oh well...for the purposes of my original question, this answer is done.
I'm trying to fetch the JSON output by an internal API and add 100 to a parameter value between cURL requests. I need to loop through because it restricts the maximum number of results per request to 100. I was told to "increment and you should be able to get what you need".
Anyway, here's what I wrote:
#!/bin/bash
COUNTER=100
until [ COUNTER -gt 30000 ]; do
curl -vs "http://example.com/locations/city?limit=100&offset=$COUNTER" >> cities.json
let COUNTER=COUNTER+100
done
The problem is that I get a bunch of weird messages in the terminal and the file I'm trying to redirect the output too still contains it's original 100 objects. I feel like I'm probably missing something terrifically obvious. Any thoughts? I did use a somewhat old tutorial on the until loop, so maybe it's a syntax issue?
Thank you in advance!
EDIT: I'm not opposed to a completely alternate method, but I had hoped this would be somewhat straightforward. I figured my lack of experience was the main limiter.
You might find you can do this faster, and pretty easily with GNU Parallel:
parallel -k curl -vs "http://example.com/locations/city?limit=100\&offset={}" ::: $(seq 100 100 30000) > cities.json
If you want to overwrite the file's content only once, for your entire loop...
#!/bin/bash
# ^-- NOT /bin/sh, as this uses bash-only syntax
for (( counter=100; counter<=30000; counter+=100 )); do
curl -vs "http://example.com/locations/city?limit=100&offset=$counter"
done >cities.json
This is actually more efficient than putting >>cities.json on each curl command, as it only opens the output file once, and has the side effect (which you appear to want) of clearing the file's former contents when the loop is started.
I would like to write a script to execute the steps outlined below. If someone can provide simple examples on how to modify files and search through folders using a script (not necessarily solving my problem below), I will greatly appreciate it.
submit job MyJob in currentDirectory using myJobShellFile.sh to a queue
upon completion of MyJob, goto to currentDirectory/myJobDataFolder.
In myJobDataFolder, there are folders
myJobData.0000 myJobData.0001 myJobData.0002 myJobData.0003
I want to find the maximum number maxIteration of all the listed folders. Here it would be maxIteration=0003.\
In file myJobShellFile.sh, at the last line says
mpiexec ./main input myJobDataFolder
I want to append this line to
'mpiexec ./main input myJobDataFolder 0003'
I want to submit MyJob to the que while maxIteration < 10
Upon completion of MyJob, find the new maxIteration and change this number in myJobShellFile.sh and goto step 4.
I think people write python scripts typically to do this stuff, but am having a hard time finding out how. I probably don't know the correct terminology for this procedure. I am also aware that the script will vary slightly depending on the queing system, but any help will be greatly appreciated.
Quite a few aspects of your question are unclear, such as the meaning of “submit job MyJob in currentDirectory using myJobShellFile.sh to a que”, “append this line to
'mpiexec ./main input myJobDataFolder 0003'”, how you detect when a job is done, relevant parts of myJobShellFile.sh, and some other details. If you can list the specific shell commands you use in each iteration of job submission, then you can post a better question, with a bash tag instead of python.
In the following script, I put a ### at the end of any line where I am guessing what you are talking about. Lines ending with ### may be irrelevant to whatever you actually do, or may be pseudocode. Anyway, the general idea is that the script is supposed to do the things you listed in your items 1 to 5. This script assumes that you have modified myJobShellFile.sh to say
mpiexec ./main input $1 $2
instead of
mpiexec ./main input
because it is simpler to use parameters to modify what you tell mpiexec than it is to keep modifying a shell script. Also, it seems to me you would want to increment maxIter before submitting next job, instead of after. If so, remove the # from the t=$((1$maxIter+1)); maxIter=${t#1} line. Note, see the “Parameter Expansion” section of man bash re expansion of the ${var#txt} form, and the “Arithmetic Expansion” section re $((expression)) form. The 1$maxIter and similar forms are used to change text like 0018 (which is not a valid bash number because 8 is not an octal digit) to 10018.
#!/bin/sh
./myJobShellFile.sh MyJob ###
maxIter=0
while true; do
waitforjobcompletion ###
cd ./myJobDataFolder
maxFile= $(ls myJobData* | tail -1)
maxIter= ${maxFile#myJobData.} #Get max extension
# If you want to increment maxIter, uncomment next line
# t=$((1$maxIter+1)); maxIter=${t#1}
cd ..
if [[ 1$maxIter -lt 11000 ]] ; then
./myJobShellFile.sh MyJobDataFolder $maxIter
else
break
fi
done
Notes: (1) To test with smaller runs than 1000 submissions, replace 11000 by 10000+n; for example, to do 123 runs, replace it with 10123. (2) In writing the above script, I assumed that not-previously-known numbers of output files appear in the output directory from time to time. If instead exactly one output file appears per run, and you just want to do one run per value for the values 0000, 0001, 0002, 0999, 1000, then use a script like the following. (For testing with a smaller number than 1000, replace 1000 with (eg) 0020. The leading zeroes in these numbers tell bash to fill the generated numbers with leading zeroes.)
#!/bin/sh
for iter in {0000..1000}; do
./myJobShellFile.sh MyJobDataFolder $iter
waitforjobcompletion ###
done
(3) If the system has a command that sleeps while it waits for a job to complete on the supercomputing resource, it is reasonable to use that command in place of waitforjobcompletion in the above scripts. Otherwise, if the system has a command jobisrunning that returns true if a job is still running, replace waitforjobcompletion with something like the following:
while jobisrunning ; do sleep 15; done
This will run the jobisrunning command; if it returns true, the shell will sleep for 15 seconds and then retest. Here is an example that illustrates waiting for a file to appear and then for it to go away:
while [ ! -f abc ]; do sleep 3; echo no abc; done
while ls abc >/dev/null 2>&1; do sleep 3; echo an abc; done
The second line's test could be [ -f abc ] instead; I showed a longer example to illustrate how to suppress output and error messages by routing them to /dev/null. (4) To reverse the sense of a while statement's test, replace the word while with until. For example, while [ ! -f abc ]; ... is equivalent to until [ -f abc ]; ....
I'm working with lots of log files, and most log files have lots of repeating strings that are logged multiple times.
To make the logs easily viewable for others who don't have much to do with such things (for myself also), i've wanted to make a script that rips out some text lines that can cause a 'false alarm' to other people. ("Hey admin, i have these errors here multiple times"; > "Sigh, these errors don't mean anything" kind of way)
Is there some bash code with grep, cat or awk that can get rid of lots of different text lines, without having to go through the document over and over again for each line to be removed? (basically remove all garbage lines in one swoop)
Example, i'll mark the lines that i want removed in bold:
One thing I don't know why
It doesn’t even matter how hard you try
Keep that in mind, I designed this
rhyme
To explain in due time
All I know
time is a valuable thing
Watch it fly by as the pendulum swings
Watch it count down to the end of the
day
The clock ticks life away
It’s so unreal
Didn’t look out below
Watch the time go right out the window
Trying to hold on but didn’t even know
Wasted it all just to
Watch you go
Sorry about the Linkin Park Lyrics, listening to the Radio while trying to solve a problem gives some bad examples sometimes :P
Are all these lines removable in one command?
Many thanks if somebody knows how.
grep -v "<string1>\|<string2>\|<stringN>" /path/to/file
It removes the lines provided in not_wanted array.
#!/bin/bash
exec < example.txt
not_wanted[0]="It doesn’t even matter how hard you try"
not_wanted[1]="time is a valuable thing"
not_wanted[2]="The clock ticks life away"
not_wanted[3]="It’s so unreal"
not_wanted[4]="Trying to hold on but didn’t even know"
while read line; do
for i in "${not_wanted[#]}"; do
if [ "$line" == "$i" ]; then unset line; break; fi
done
if [ "$line" ]; then echo "$line"; fi
done
Put the lines you don't want in a file, then
grep -v -f not.wanted filename > smaller.file
I'm writing shell scripts where quite regularly some stuff is written
to a file, after which an application is executed that reads that file. I find that through our company the network latency differs vastly, so a simple sleep 2 for example will not be robust enough.
I tried to write a (configurable) timeout loop like this:
waitLoop()
{
local timeout=$1
local test="$2"
if ! $test
then
local counter=0
while ! $test && [ $counter -lt $timeout ]
do
sleep 1
((counter++))
done
if ! $test
then
exit 1
fi
fi
}
This works for test="[ -e $somefilename ]". However, testing existence is not enough, I sometimes need to test whether a certain string was written to the file. I tried
test="grep -sq \"^sometext$\" $somefilename", but this did not work. Can someone tell me why?
Are there other, less verbose options to perform such a test?
You can set your test variable this way:
test=$(grep -sq "^sometext$" $somefilename)
The reason your grep isn't working is that quotes are really hard to pass in arguments. You'll need to use eval:
if ! eval $test
I'd say the way to check for a string in a text file is grep.
What's your exact problem with it?
Also you might adjust your NFS mount parameters, to get rid of the root problem. A sync might also help. See NFS docs.
If you're wanting to use waitLoop in an "if", you might want to change the "exit" to a "return", so the rest of the script can handle the error situation (there's not even a message to the user about what failed before the script dies otherwise).
The other issue is using "$test" to hold a command means you don't get shell expansion when actually executing, just evaluating. So if you say test="grep \"foo\" \"bar baz\"", rather than looking for the three letter string foo in the file with the seven character name bar baz, it'll look for the five char string "foo" in the nine char file "bar baz".
So you can either decide you don't need the shell magic, and set test='grep -sq ^sometext$ somefilename', or you can get the shell to handle the quoting explicitly with something like:
if /bin/sh -c "$test"
then
...
Try using the file modification time to detect when it is written without opening it. Something like
old_mtime=`stat --format="%Z" file`
# Write to file.
new_mtime=$old_mtime
while [[ "$old_mtime" -eq "$new_mtime" ]]; do
sleep 2;
new_mtime=`stat --format="%Z" file`
done
This won't work, however, if multiple processes try to access the file at the same time.
I just had the exact same problem. I used a similar approach to the timeout wait that you include in your OP; however, I also included a file-size check. I reset my timeout timer if the file had increased in size since last it was checked. The files I'm writing can be a few gig, so they take a while to write across NFS.
This may be overkill for your particular case, but I also had my writing process calculate a hash of the file after it was done writing. I used md5, but something like crc32 would work, too. This hash was broadcast from the writer to the (multiple) readers, and the reader waits until a) the file size stops increasing and b) the (freshly computed) hash of the file matches the hash sent by the writer.
We have a similar issue, but for different reasons. We are reading s file, which is sent to an SFTP server. The machine running the script is not the SFTP server.
What I have done is set it up in cron (although a loop with a sleep would work too) to do a cksum of the file. When the old cksum matches the current cksum (the file has not changed for the determined amount of time) we know that the writes are complete, and transfer the file.
Just to be extra safe, we never overwrite a local file before making a backup, and only transfer at all when the remote file has two cksums in a row that match, and that cksum does not match the local file.
If you need code examples, I am sure I can dig them up.
The shell was splitting your predicate into words. Grab it all with $# as in the code below:
#! /bin/bash
waitFor()
{
local tries=$1
shift
local predicate="$#"
while [ $tries -ge 1 ]; do
(( tries-- ))
if $predicate >/dev/null 2>&1; then
return
else
[ $tries -gt 0 ] && sleep 1
fi
done
exit 1
}
pred='[ -e /etc/passwd ]'
waitFor 5 $pred
echo "$pred satisfied"
rm -f /tmp/baz
(sleep 2; echo blahblah >>/tmp/baz) &
(sleep 4; echo hasfoo >>/tmp/baz) &
pred='grep ^hasfoo /tmp/baz'
waitFor 5 $pred
echo "$pred satisfied"
Output:
$ ./waitngo
[ -e /etc/passwd ] satisfied
grep ^hasfoo /tmp/baz satisfied
Too bad the typescript isn't as interesting as watching it in real time.
Ok...this is a bit whacky...
If you have control over the file: you might be able to create a 'named pipe' here.
So (depending on how the writing program works) you can monitor the file in an synchronized fashion.
At its simplest:
Create the named pipe:
mkfifo file.txt
Set up the sync'd receiver:
while :
do
process.sh < file.txt
end
Create a test sender:
echo "Hello There" > file.txt
The 'process.sh' is where your logic goes : this will block until the sender has written its output. In theory the writer program won't need modifiying....
WARNING: if the receiver is not running for some reason, you may end up blocking the sender!
Not sure it fits your requirement here, but might be worth looking into.
Or to avoid synchronized, try 'lsof' ?
http://en.wikipedia.org/wiki/Lsof
Assuming that you only want to read from the file when nothing else is writing to it (ie, the writing process has finished) - you could check whether nothing else has file handle to it ?