I have a function which uploads a file to a server, I want to measure how long that part of the script takes and display the result for use un the output,
eg:
echo "$file uploaded in XXXM:XXs"
I can display the number of seconds, with:
my_upload_stuff here
echo "$file uploaded in $SECONDS"
Which as I understand displays the number of seconds since the script has started (which is fine for what I need) But this is as far as I can get.
Pulling my remaining hair out trying to figure this, seems to be way harder than I would imagine. Been all around the houses but nothing seems to work - I confess my bash skills are newbie...
You could use the command time if installed (but you'd have to use the external command /usr/bin/time and so you'd have to install it if it's not already installed ... because the built-in bash command time hasn't got the options to produce the desired output).
Or alternatively, you can perform the calculation explicitely, like that:
t1=$(date +%s)
<put upload_command here>
t2=$(date +%s)
echo "Time elapsed for upload: $(( ( t2 - t1 ) / 60 )) minutes and $(( ( t2 - t1 ) % 60 )) seconds"
Related
I am trying to create my first bash script. The goal of this script is to check at what rate my public IP changes. It is a fairly straight forward script. First it checks if the new address is different from the old one. If so then it should update the old one to the new one and print out the date along with the new IP address.
At this point I have created a simple script in order to accomplish this. But I have two main problems.
First the script keeps on printing out the IP even tough it hasn't changed and I have updated the PREV_IP with the CUR_IP.
My second problem is that I want the output to direct to a file instead of outputting it into the terminal.
The interval is currently set to 1 second for test purposes. This will change to a higher interval in the final product.
#!/bin/bash
while true
PREV_IP=00
do
CUR_IP=$(curl https://ipinfo.io/ip)
if [ $PREV_IP != "$CUR_IP" ]; then
PREV_IP=$CUR_IP
"$(date)"
echo "$CUR_IP"
sleep 1
fi
done
I also get a really weird output. I have edited my public IP to xx.xxx.xxx.xxx:
Sat 20 Mar 09:45:29 CET 2021
xx.xxx.xxx.xxx
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
while true
PREV_IP=00
do
is the reason you are seeing ip each loop. It's the same as while true; PREV_IP=00; do. The exit status of true; PREV_IP=00 is the exit status of last command - the exit status of assignment is 0 (success) - so the loop will always execute. But PREV_IP will be reset to 00 each loop... This is a typo and you meant to set prev_ip once, before the loop starts.
"$(date)"
will try execute the output of date command, as a next command. So it will print:
$ "$(date)"
bash: sob, 20 mar 2021, 10:57:02 CET: command not found
And finally, to silence curl, read man curl first and then find out about -s. I use -sS so errors are also visible.
Do not use uppercase variables in your scripts. Prefer lower case variables. Check you scripts with http://shellcheck.net . Quote variable expansions.
I would sleep each loop. Your script could look like this:
#!/bin/bash
prev=""
while true; do
cur=$(curl -sS https://ipinfo.io/ip)
if [ "$prev" != "$cur" ]; then
prev="$cur"
echo "$(date) $cur"
fi
sleep 1
done
that I want the output to direct to a file instead of outputting it into the terminal.
Then research how redirection works in shell and how to use it. The simplest would be to redirect echo output.
echo "$(date) $cur" >> "a_file.txt"
The interval is currently set to 1 second for test purposes. This will change to a higher interval in the final product.
You are still limited with the time it takes to connect to https://ipinfo.io/ip. And from ipinfo.io documentation:
Free usage of our API is limited to 50,000 API requests per month.
And finally, I wrote a script where I tried to use many public services as I found ,get_ip_external for getting external ip address. You may take multiple public services for getting ipv4 address and choose a random/round-robin one so that rate-limiting don't kick that fast.
I am getting a GC overhead issue when i am running a count for a date range as it has a huge data to pull, so need a logic to run the query for a specific date range (for example to run query for every 30 days without missing any data ) and sum it up all at the end.
I have tried to run the query for every 30 days, but in this approach ,there might be some chances to miss the data count for few days.
currently , I wrote the below code and am able to run the query successfully,but its very time consuming process, so need a help to change this code for every month or some specific date range instead of running below.
while [ ${PART_START_DATE} -le ${RUN_START_DAY} ]
do
fb_TEST=$(($fb_TEST+$(hive -S -e "use ${DATABASE};set hive.cli.print.header=false;select count(*) from fb_wrk_tab where date = '${PART_START_DATE}';")))
PART_START_DATE=`date -d "${PART_START_DATE} 1 days" +%Y%m%d`
echo "fbwrk_TEST count is"$fb_TEST >> ${LOG_FILE}
done
I have a simple script that I need to run every 15 minutes everyday (until I get to the last record in my database) giving it greater argument. I know how to do this with the constant argument - example:
*/15 * * * * ./my_awesome_script 1
But I need this, let's say, we start from 8:00 AM:
at 8:00 it should run ./my_awesome_script 1
at 8:15 it should run ./my_awesome_script 2
at 8:30 it should run ./my_awesome_script 3
at 8:45 it should run ./my_awesome_script 4
at 9:00 it should run ./my_awesome_script 5
...
How to make something like this?
I came up with temporary solution:
#!/bin/bash
start=$1
stop=$2
for i in `seq $start $stop`
do
./my_awesome_script $i
sleep 900
done
Writing a wrapper script is pretty much necessary (for sanity's sake). The script can record in a file the previous value of the number and increment it and record the new value ready for next time. Then you don't need the loop. How are you going to tell when you've reached the end of the data in the database? You need to know about how you want to handle that, too.
New cron entry:
*/15 * * * * ./wrap_my_awesome_script
And wrap_my_awesome_script might be:
crondir="$HOME/cron"
counter="$crondir/my_awesome_script.counter"
[ -d "$crondir" ] || mkdir -p "$crondir"
[ -s "$counter" ] || echo 0 > "$counter"
count=$(<"$counter")
((count++))
echo "$count" > $counter
"$HOME/bin/my_awesome_script" "$count"
I'm not sure why you use ./my_awesome_script; it likely means your script is in your $HOME directory. I'd keep it in $HOME/bin and use that name in the wrapper script — as shown.
Note the general insistence on putting material in some sub-directory of $HOME rather than directly in $HOME. Keeping your home directory uncluttered is generally a good idea. You can place the files and programs where you like, of course, but I recommend being as organized as possible. If you aren't organized then, in a few years time, you'll wish you had been.
I have a bash script that is sending me a text daily, for 100 days.
#! /bin/bash
EMAIL="my-phone-gateway#address.net"
MESSAGE="message_content.txt"
mail $EMAIL < $MESSAGE
Using crontab, I can have the static $MESSAGE sent to me every day.
Other than hard-coding 100 days of texts ;)
How could I implement a variable counter such that I can have my texts say:
"Today is Day #1" on the first day, "Today is Day #2" on the second day, etc. ?
Note: The location of the requested text within the $MESSAGE file doesn't matter. Last line, first line, middle, etc.
The only requirement for an answer here is that I know what day it is relative to the first, where the first day is the day the script was started.
Of course, bonus awesome points for the cleanest, simplest, shortest solution :)
For our nightly build systems, I wrote a C program that does the calculation (using local proprietary libraries that store dates as a number of days since a reference date). Basically, given a (non-changing) reference date, it reports the number of days since the reference date. So, the cron script would have a hard-wired first day in it, and the program would report the number of days since then.
The big advantage of this system is that the reference date doesn't change (very often), so the script doesn't change (very often), and there are no external files to store information in.
There probably are ways to achieve the same effect with standard Unix tools, but I've not sat down and worked out the portable solution. I'd probably think it terms of using Perl. (The C program only works up to 2999 CE; I left a note in the code for people to contact me about 50 years before it becomes a problem for the Y3K fix. It is probably trivial.)
You could perhaps work in terms of Unix timestamps...
Create a script 'days_since 1234567890' which treats the number as the reference date, gets the current time stamp (from date with appropriate format specification; on Linux, date '+%s' would do that job, and it works on Mac OS X too), takes the difference and divides by 86,400 (the number of seconds in a day).
refdate=1234567890
bc <<EOF
scale=0
($(date '+%s') - $refdate) / 86400
EOF
An example:
$ timestamp 1234567890
1234567890 = Fri Feb 13 15:31:30 2009
$ timestamp
1330027280 = Thu Feb 23 12:01:20 2012
$ refdate=1234567890
$ bc <<EOF
> scale=0
> ($(date '+%s') - $refdate) / 86400
> EOF
1104
$
So, if the reference date was 13th Feb 2009, today is day 1104. (The program bc is the calculator; its name has nothing to do with Anno Domini or Before Christ. The program timestamp is another homebrew of mine that prints timestamps according to a format that can be specified; it is a specialized variant of date originally written in the days before date had the functionality, by which I mean in the early 1980s.)
In a Perl one-liner (assuming you specify the reference date in your script):
perl -e 'printf "%d\n", int((time - 1234567890)/ 86400)'
or:
days=$(perl -e 'printf "%d\n", int((time - 1234567890)/ 86400)')
The only way to accomplish this would be to store the date in a file, and read from that file each day. I would suggest storing the epoch time.
today=$(date +%s)
time_file="~/.first_time"
if [[ -f $time_file ]]; then
f_time=$(< "$time_file")
else
f_time=$today
echo "$f_time" > "$time_file"
fi
printf 'This is day: %s\n' "$((($today - $f_time) / 60 / 60 / 24))"
Considering that your script is running only once a day, something like this should work:
#!/bin/bash
EMAIL="my-phone-gateway#address.net"
MESSAGE="message_content.txt"
STFILE=/tmp/start.txt
start=0
[ -f $STFILE ] && start=$(<$STFILE)
start=$((start+1))
MESSAGE=${MESSAGE}$'\n'"Today is Day #${start}"
echo "$start" > $STFILE
mail $EMAIL < $MESSAGE
A simple answer would be to export the current value to an external file, and read that back in again later.
So, for example, make a file called "CurrentDay.dat" that has the number 1 in it.
Then, in your bash script, read in the number and increment it.
e.g. your bash script could be:
#!/bin/bash
#Your stuff here.
DayCounter=$(<CurrentDay.dat)
#Use the value of DayCounter (i.e. $DayCounter) in your message.
DayCounter=$((DayCounter + 1))
echo $DayCounter > CurrentDay.dat
Of course, you may need to implement some additional checks to avoid something going wrong, but that should work as is.
I need to examine the output of a certain script 1000s of times on a unix platform and check if any of it has changed from before.
I've been doing this:
(script_stuff) | md5sum
and storing this value. I actually don't really need "md5", JUST a simple hash function which I can compare against a stored value to see if its changed. Its okay if there are an occassional false positive.
Is there anything better than md5sum that works faster and generates a fairly usable hash value? The script itself generates a few lines of text - maybe 10-20 on average to max 100 or so.
I had a look at fast md5sum on millions of strings in bash/ubuntu - that's wonderful, but I can't compile a new program. Need a system utility... :(
Additional "background" details:
I've been asked to monitor the DNS record of a set of 1000 or so domains and immediately call certain other scripts if there has been any change. I intend to do a dig xyz +short statement and hash its output and store that, and then check it against a previously stored value. Any change will trigger the other script, otherwise it just goes on. Right now, we're planning on using cron for a set of these 1000, but can think completely diffeerently for "seriously heavy" usage - ~20,000 or so.
I have no idea what the use of such a system would be, I'm just doing this as a job for someone else...
The cksum utility calculates a non-cryptographic CRC checksum.
How big is the output you're checking? A hundred lines max. I'd just save the entire original file then use cmp to see if it's changed. Given that a hash calculation will have to read every byte anyway, the only way you'll get an advantage from a checksum type calculation is if the cost of doing it is less than reading two files of that size.
And cmp won't give you any false positives or negatives :-)
pax> echo hello >qq1.txt
pax> echo goodbye >qq2.txt
pax> cp qq1.txt qq3.txt
pax> cmp qq1.txt qq2.txt >/dev/null
pax> echo $?
1
pax> cmp qq1.txt qq3.txt >/dev/null
pax> echo $?
0
Based on your question update:
I've been asked to monitor the DNS record of a set of 1000 or so domains and immediately call certain other scripts if there has been any change. I intend to do a dig xyz +short statement and hash its output and store that, and then check it against a previously stored value. Any change will trigger the other script, otherwise it just goes on. Right now, we're planning on using cron for a set of these 1000, but can think completely diffeerently for "seriously heavy" usage - ~20,000 or so.
I'm not sure you need to worry too much about the file I/O. The following script executed dig microsoft.com +short 5000 times first with file I/O then with output to /dev/null (by changing the comments).
#!/bin/bash
rm -rf qqtemp
mkdir qqtemp
((i = 0))
while [[ $i -ne 5000 ]] ; do
#dig microsoft.com +short >qqtemp/microsoft.com.$i
dig microsoft.com +short >/dev/null
((i = i + 1))
done
The elapsed times at 5 runs each are:
File I/O | /dev/null
----------+-----------
3:09 | 1:52
2:54 | 2:33
2:43 | 3:04
2:49 | 2:38
2:33 | 3:08
After removing the outliers and averaging, the results are 2:49 for the file I/O and 2:45 for the /dev/null. The time difference is four seconds for 5000 iterations, only 1/1250th of a second per item.
However, since an iteration over the 5000 takes up to three minutes, that's how long it will take maximum to detect a problem (a minute and a half on average). If that's not acceptable, you need to move away from bash to another tool.
Given that a single dig only takes about 0.012 seconds, you should theoretically do 5000 in sixty seconds assuming your checking tool takes no time at all. You may be better off doing something like this in Perl and using an associative array to store the output from dig.
Perl's semi-compiled nature means that it will probably run substantially faster than a bash script and Perl's fancy stuff will make the job a lot easier. However, you're unlikely to get that 60-second time much lower just because that's how long it takes to run the dig commands.