I have a date that is inputted into a bash script through 3 separate command line arguments. The user could put in either 1 or 2 digits for the month and day and 4 digits for the year (eg. 2014 01 01 or 2014 1 1). But my script needs two digits to run. I was thinking of using an if statement to handle this. It would basically say "if the amount of digits in the month is less than 2 then put a leading zero in front of it". Though, I am unsure how to determine the amount of digits in bash. I am rather new to scripting so any help would be greatly appreciated!
You could solve this using the printf command:
month=5
month2=`printf %02d $month`
The %02d format specifier formats $month as two decimal digits with a leading zero if necessary.
For any bash variable $x (including parameters $1, $2, etc.), ${#x} is the length of $x in characters.
Related
I am taking a rather large file which is basically a list of products sold in various quantities and I want to add a fixed number to every existing dollar amount mentioned in the file (so everything with a dollar sign in front of it, to make things even more confusing .) The file contents are very predictable and are arranged as such:
16-point printed 2 side three and a half by two matte finish no round corners turn around 2-4 business days one set
250 $9.40 500 $11.05 750 $13.58 1000 $14.40 2500 $33.25 5000 $43.00 10000 $73.00 15000 $108.00 20000 $140.00 25000 $172.50
and that goes on until forever and a day. All I want to do is add lets say 5 bucks to each dollar amount, and spit out a new file. I am pretty sure that I want to evaluate it one word at a time, but entire lines can be totally skipped, due to quantities/values only appearing every second line. NOt all items have the same number of quantities so a fixed loop can't work.
I have been able to read and regurgitate the file, and I have played with reading single characters at a time. but seeing as this script will be useful now and down the road, I want to do it right and I'm new to BASH.
I'm sure it will be a combination of
#!/bin/sh
while read -r line; do
for word in $line; do
echo -n "'$word'"
done
done < "textfile.txt.bak"
and
n=1
while IFS= read -r "variable$n"; do
n=$((n + 1))
done < textfile.txt
for ((i=1; i<n; i++)); do
var="variable$i"
printf '%s\n' "${!var}"
done
I know I should be searching for the '/$' and reading in the numbers that follow, convert string to a real number, add 5, and then convert back to string, print a $ and the string, rinse wash repeat until next year. I'm pulling my hair out trying to find the best way to approach it in Bash.
PLEASE HELP!
RJM
While possible in bash, I'd recommend to use something else. For example, in Perl it boils down to:
perl -pe 's/\$\K([0-9.]+)/sprintf "%.2f", $1+5/ge if /\$[0-9]/' -- file
-p processes the input line by line, printing eachline after processing.
... if /\$[0-9]/ runs the ... part if the current line contains a dollar sign followed by a digit.
/g does a global replacement, i.e. all the possible occurrences.
/e interprets the replacement as code.
\K forgets what matched so far, in other words it only replaces the number, not the dollar sign.
It's pretty gnarly in bash due to the integer-only arithmetic
while read -ra words; do
for i in "${!words[#]}"; do
if [[ ${words[i]} =~ ^\$([0-9]+)(\.[0-9]+)? ]]; then
printf -v words[i] '$%d%s' $((BASH_REMATCH[1] + 5)) "${BASH_REMATCH[2]}"
fi
done
echo "${words[*]}"
done < file
16-point printed 2 side three and a half by two matte finish no round corners turn around 2-4 business days one set
250 $14.40 500 $16.05 750 $18.58 1000 $19.40 2500 $38.25 5000 $48.00 10000 $78.00 15000 $113.00 20000 $145.00 25000 $177.50
To send to a new file, change the last line to
done < file > new.file
You can also use awk for tnis task.
awk 'NR%2==0{for(i=2;i<=NF;i+=2){split($i,a,"$");$i=sprintf("%s%.2f","$",a[2]+5)}}1' file
I'm trying to get the week number of last week. The following command normally had work, but now I'm getting error.
lastweeknumber=$((`date +%V`-1))
bash: 09: value too great for base (error token is "09")
This week number is 09, so I've tried to convert to decimal adding 10# like this $(10#(date +%V)) but it's not working.
How to fix this?
Consider the following, which uses bash's built-in functionality in place of the external date command, and thus requires a recent shell release but is much faster to run (and will behave consistently without depending on a specific version of date).
With that done, though, there's still a need to strip the leading 0 -- which a parameter expansion will do just fine:
printf -v seconds_now '%(%s)T' -1
printf -v weeknum_lastweek '%(%V)T' "$(( seconds_now - (60 * 60 * 24 * 7) ))"
echo "The index of last week is ${weeknum_lastweek#0}"
It is because date +%V returns 09 and shell is interpreting any value starting with 0 as an octal number. Note that 09 is an invalid octal number hence you get that error value too great for base.
You can just force module 10 arithmetic in (( ... )):
echo $(( 10#$(date +%V) - 1 ))
8
Another way that handles wrapping around year correctly:
lastweeknumber=$(date -d "1 week ago" +%V)
I've got a series of files that are namedHHMMSSxxxxxxxxxxxxxxxx.mp3, where HH,MM, and SS are parts of a timestamp and the x's are unique per file.
The timestamp follows a 24 hour form (where 10am is 100000, 12pm is 120000, 6pm is 180000, 10pm is 220000, etc). I'd like to shift each down by 10 hours, so that 10am is 000000, 12pm is 020000, etc.
I know basic BASH commands for renaming and moving, etc, but I can't figure out how to do the modular arithmetic on the filenames.
Any help would be very much appreciated.
#!/bin/bash
for f in *.mp3
do
printf -v newhour '%02d' $(( ( 10#${f:0:2} + 14 ) % 24 ))
echo mv "$f" "$newhour${f:2}"
done
Remove the echo to make it functional.
Explanation:
printf -v newhour '%02d' - this is like sprintf(), the value is stored in the named variable
$(( ( 10#${f:0:2} + 14 ) % 24 )) - 10# forces the number to base 10 (e.g. 08 would otherwise be considered an invalid octal), ${f:0:2} extracts the first two characters (the hour), the rest does the math
"$newhour${f:2}" - prepend the new hour before the substring of the original name, starting at the third character
The easiest way is probably to extract the timestamp and use date to turn it into a number of seconds, do normal math on the result, then convert it back. date -d datestring +format lets you do these conversions.
I'm running a pretty simple bash script in ubuntu but have come up with a problem.
If needed I'll post the whole script, but I've narrowed down the problem.
Basically, I want to run some code every 15 seconds, so I started with this:
time=`date +%S`
time2=$((time%15))
if [ $time2 -eq 0 ]
then
etc, etc, etc....
The problem comes up when time is 08 seconds. The script terminates with Illegal number: 08.
Adding to that, when using:
time2=$(($time%15))
instead of the illegal number error, it would terminate with Arithmetic expression: expecting EOF: "08%15"
I'm guessing 08 isn't interpreted as a number. Or there's some base issue, like it thinks it's in octal or something. Any help?
Shortest solution:
time2=$(( ${time#0} % 15 ))
${var#glob} means "$var with glob removed from the beginning if present".
Try using the following flags instead
date +%-S
It says given the -, it won't pad. It has problems with the base, interpreting it as an octal integer.
Anyway, if you want to do something every 15 seconds, i find this one easier to follow:
while ((1)); do
echo do something now...
sleep 15s
done
Force Bash to interpret the number in decimal, no matter how many padded zeros:
time2=$((10#$time % 15))
you're right, it was interpreting it as octal. bourne shells do that for any number with a leading 0 in an Arithmetic Substition:
#~ $ echo $(( 010 ))
8
#~ $ echo $(( 0100 ))
64
#~ $ echo $(( 10#0100 ))
100
#~ $ echo $(( 40#lolwut ))
2213236429
look in the manpage for 'base#' to see all the details about this '#-forcing' thing. you can get pretty ridiculous with it if you want to
Since you're only interested in "every fifteen seconds" rather than running things on the minute exactly you could use date +%s (lowercase s) which will give you the number of seconds since the start of the epoch. This won't have a leading 0 so your script should run fine.
However, I would wonder about your code in a wider context. If the system is running very slow for some reason it could be possible for the script only be run second 14 and then second 16 meaning it will miss an execution.
It might be worth touching a file when you do whatever it is the script does and then performing your action when then last modified date of the file is 15 or more seconds ago.
That does look like it's interpreting it as octal.
Try date +%S | sed -e 's/^0//'
I have a set of mail logs: mail.log mail.log.0 mail.log.1.gz mail.log.2.gz
each of these files contain chronologically sorted lines that begin with timestamps like:
May 3 13:21:12 ...
How can I easily grab every log entry after a certain date/time and before another date/time using bash (and related command line tools) without comparing every single line? Keep in mind that my before and after dates may not exactly match any entries in the logfiles.
It seems to me that I need to determine the offset of the first line greater than the starting timestamp, and the offset of the last line less than the ending timestamp, and cut that section out somehow.
Convert your min/max dates into "seconds since epoch",
MIN=`date --date="$1" +%s`
MAX=`date --date="$2" +%s`
Convert the first n words in each log line to the same,
L_DATE=`echo $LINE | awk '{print $1 $2 ... $n}'`
L_DATE=`date --date="$L_DATE" +%s`
Compare and throw away lines until you reach MIN,
if (( $MIN > $L_DATE )) ; then continue ; fi
Compare and print lines until you reach MAX,
if (( $L_DATE <= $MAX )) ; then echo $LINE ; fi
Exit when you exceed MAX.
if (( $L_DATE > $MAX )) ; then exit 0 ; fi
The whole script minmaxlog.sh looks like this,
#!/usr/bin/env bash
MIN=`date --date="$1" +%s`
MAX=`date --date="$2" +%s`
while true ; do
read LINE
if [ "$LINE" = "" ] ; then break ; fi
L_DATE=`echo $LINE | awk '{print $1 " " $2 " " $3 " " $4}'`
L_DATE=`date --date="$L_DATE" +%s`
if (( $MIN > $L_DATE )) ; then continue ; fi
if (( $L_DATE <= $MAX )) ; then echo $LINE ; fi
if (( $L_DATE > $MAX )) ; then break ; fi
done
I ran it on this file minmaxlog.input,
May 5 12:23:45 2009 first line
May 6 12:23:45 2009 second line
May 7 12:23:45 2009 third line
May 9 12:23:45 2009 fourth line
June 1 12:23:45 2009 fifth line
June 3 12:23:45 2009 sixth line
like this,
./minmaxlog.sh "May 6" "May 8" < minmaxlog.input
Here one basic idea of how to do it:
Examine the datestamp on the file to see if it is irrelevent
If it could be relevent, unzip if necessary and examine the first and last lines of the file to see if it contains the start or finish time.
If it does, use a recursive function to determine if it contains the start time in the first or second half of the file. Using a recursive function I think you could find any date in a million line logfile with around 20 comparisons.
echo the logfile(s) in order from the offset of the first entry to the offset of the last entry (no more comparisons)
What I don't know is: how to best read the nth line of a file (how efficient is it to use tail n+**n|head 1**?)
Any help?
You have to look at every single line in the range you want (to tell if it's in the range you want) so I'm guessing you mean not every line in the file. At a bare minimum, you will have to look at every line in the file up to and including the first one outside your range (I'm assuming the lines are in date/time order).
This is a fairly simple pattern:
state = preprint
for every line in file:
if line.date >= startdate:
state = print
if line.date > enddate:
exit for loop
if state == print:
print line
You can write this in awk, Perl, Python, even COBOL if you must but the logic is always the same.
Locating the line numbers first (with say grep) and then just blindly printing out that line range won't help since grep also has to look at all the lines (all of them, not just up to the first outside the range, and most likely twice, one for the first line and one for the last).
If this is something you're going to do quite often, you may want to consider shifting the effort from 'every time you do it' to 'once, when the file is stabilized'. An example would be to load up the log file lines into a database, indexed by the date/time.
That takes a while to get set up but will result in your queries becoming a lot faster. I'm not necessarily advocating a database - you could probably achieve the same effect by splitting the log files into hourly logs thus:
2009/
01/
01/
0000.log
0100.log
: :
2300.log
02/
: :
Then for a given time, you know exactly where to start and stop looking. The range 2009/01/01-15:22 through 2009/01/05-09:07 would result in:
some (the last bit) of the file 2009/01/01/1500.txt.
all of the files 2009/01/01/1[6-9]*.txt.
all of the files 2009/01/01/2*.txt.
all of the files 2009/01/0[2-4]/*.txt.
all of the files 2009/01/05/0[0-8]*.txt.
some (the first bit) of the file 2009/01/05/0900.txt.
Of course, I'd write a script to return those lines rather than trying to do it manually each time.
Maybe you can try this:
sed -n "/BEGIN_DATE/,/END_DATE/p" logfile
It may be possible in a Bash environment but you should really take advantage of tools that have more built-in support for working with Strings and Dates. For instance Ruby seems to have the built in ability to parse your Date format. It can then convert it to an easily comparable Unix Timestamp (a positive integer representing the seconds since the epoch).
irb> require 'time'
# => true
irb> Time.parse("May 3 13:21:12").to_i
# => 1241371272
You can then easily write a Ruby script:
Provide a start and end date. Convert those to this Unix Timestamp Number.
Scan the log files line by line, converting the Date into its Unix Timestamp and check if that is in the range of the start and end dates.
Note: Converting to a Unix Timestamp integer first is nice because comparing integers is very easy and efficient to do.
You mentioned "without comparing every single line." Its going to be hard to "guess" at where in the log file the entries starts being too old, or too new without checking all the values in between. However, if there is indeed a monotonically increasing trend, then you know immediately when to stop parsing lines, because as soon as the next entry is too new (or old, depending on the layout of the data) you know you can stop searching. Still, there is the problem of finding the first line in your desired range.
I just noticed your edit. Here is what I would say:
If you are really worried about efficiently finding that start and end entry, then you could do a binary search for each. Or, if that seems like overkill or too difficult with bash tools you could have a heuristic of reading only 5% of the lines (1 in every 20), to quickly get a close to exact answer and then refining that if desired. These are just some suggestions for performance improvements.
I know this thread is old, but I just stumbled upon it after recently finding a one line solution for my needs:
awk -v ts_start="2018-11-01" -v ts_end="2018-11-15" -F, '$1>=ts_start && $1<ts_end' myfile
In this case, my file has records with comma separated values and the timestamp in the first field. You can use any valid timestamp format for the start and end timestamps, and replace these will shell variables if desired.
If you want to write to a new file, just use normal output redirection (> newfile) appended to the end of above.