I have an auto-generated file each day that gets called by a shell script.
But, the problem I'm facing is that the auto-generated file has a form of:
FILE_MM_DD.dat
... where MM and DD are 2-digit month and day-of-the-month strings.
I did some research and banged it at on my own, but I don't know how to create these custom strings using only shell scripting.
To be clear, I am aware of the DATE function in Bash, but what I'm looking for is the equivalent of the SPRINTF function in C.
In Bash:
var=$(printf 'FILE=_%s_%s.dat' "$val1" "$val2")
or, the equivalent, and closer to sprintf:
printf -v var 'FILE=_%s_%s.dat' "$val1" "$val2"
If your variables contain decimal values with leading zeros, you can remove the leading zeros:
val1=008; val2=02
var=$(printf 'FILE=_%d_%d.dat' $((10#$val1)) $((10#$val2)))
or
printf -v var 'FILE=_%d_%d.dat' $((10#$val1)) $((10#$val2))
The $((10#$val1)) coerces the value into base 10 so the %d in the format specification doesn't think that "08" is an invalid octal value.
If you're using date (at least for GNU date), you can omit the leading zeros like this:
date '+FILE_%-m_%-d.dat'
For completeness, if you want to add leading zeros, padded to a certain width:
val1=8; val2=2
printf -v var 'FILE=_%04d_%06d.dat' "$val1" "$val2"
or with dynamic widths:
val1=8; val2=2
width1=4; width2=6
printf -v var 'FILE=_%0*d_%0*d.dat' "$width1" "$val1" "$width2" "$val2"
Adding leading zeros is useful for creating values that sort easily and align neatly in columns.
Why not using the printf program from coreutils?
$ printf "FILE_%02d_%02d.dat" 1 2
FILE_01_02.dat
Try:
sprintf() { local stdin; read -d '' -u 0 stdin; printf "$#" "$stdin"; }
Example:
$ echo bar | sprintf "foo %s"
foo bar
Related
Is it possible to format currency in Bash?
Example data is received as
19366
Data to be displayed as
$193,66
Thanks.
Simply handle your value as a text string, instead of a number, and insert a dollar sign and a comma at the correct positions:
$ v=19366
$ printf '$%s,%s\n' "${v:0: -2}" "${v: -2}"
$193,66
${v:offset:length) expands as the substring of $v that starts at character offset (counting from 0) and which length is length. But negative offsets and lengths can be used to refer to the end of the string.
${v:0:-2} expands as the substring of $v that starts at the beginning (0) and which length is the number of remaining characters minus two (-2). In our example this is 193.
${v: -2} expands as the substring of $v that starts two characters before the end (-2) and which length (not specified) is the number of remaining characters. In our example this is 66. Note the space between : and -2, it is needed to avoid another interpretation by the shell (providing default value 2 if v is unset or null).
Preamble In your request, you use coma , as decimal separator (radix mark). For locale support, see second part of my answer.
1. Pseudo floating poing using integer as strings
I often use this kind of pseudo float:
amount=123456
amount=00$amount # avoid bad length error
printf '$%.2f\n' ${amount::-2}.${amount: -2}
$1234.56
for amount in 0 1 12 123 1234 12345;do
amount=00$amount
printf '$%.2f\n' ${amount::-2}.${amount: -2}
done
$0.00
$0.01
$0.12
$1.23
$12.34
$123.45
As a function:
int2amount() {
if [[ $1 == -v ]]; then
local -n _out="$2"
shift 2
else
local _out
fi
local _amount=00$(($1))
printf -v _out $'$%\47.2f' ${_amount::-2}.${_amount: -2}
[[ ${_out#A} != _out=* ]] || echo "$_out"
}
Then
int2amount 123456
$1’234.56
int2amount -v var 1234567
echo $var
$12’345.67
2. Remark regarding locale, decimal separator and thousand separators
In your request, your radix mark is a coma ,. This depend on your locale configuration. U could hit something like:
set | grep ^LC\\\|^LANG
to show how this is configured on your host.
As there are many issues regarding locales, I've asked How to determine which character is used as decimal separator or thousand separator under current locale as separated question.
Try:
for locvar in C en_US.UTF-8 de_DE.UTF-8 ;do
LC_NUMERIC=$locvar int2amount 1234567
done
$12345.67
$12,345.67
bash: line 1: printf: 0012345.67: invalid octal number
$12.345,00
Error because unsing de_DE locale configuration, you have to use a coma as separator (Decimal separator at wikipedia).
This is already know to produce issues using bc: How do I change the decimal separator in the printf command in bash?
Final function unsing variable decimal separator
int2amount () {
if [[ $1 == -v ]]; then
local -n _out="$2"
shift 2
else
local _out
fi
local _amount=00$(($1)) _decsep
printf -v _decsep %.1f 1
_decsep=${_decsep:1:1}
printf -v _out '$%'\''.2f' ${_amount::-2}${_decsep}${_amount: -2}
[[ ${_out#A} != _out=* ]] || echo "$_out"
}
for locvar in C en_US.UTF-8 de_DE.UTF-8 ;do
LC_NUMERIC=$locvar int2amount 1234567
done
$12345.67
$12,345.67
$12.345,67
Note about LC_ALL: If in your environment, a variable $LC_ALL is defined, all demos using LC_NUMERIC won't work because LC_ALL is over. You have to unset LC_ALL or use:
LC_ALL=$locvar LC_NUMERIC=$locvar int2amount 1234567
in last demo.
You can use printf
amount="240570.578"
printf "%'.2f\n" $amount
> 240,570.58
printf does have a thousands grouping format specifier flag, however the character used to denote the groups (non-monetary grouping character) depends on locale (LC_NUMERIC).
The C or POSIX locale uses no grouping character. Therefore you can't do this portably with printf.
printf "%'d\n" 19366
Works if the current locale supports the comma grouping character.
In my bashrc, I use the following function to add thousands groupings to any integer, using comma (,) and preserving a non numeric prefix (like $, or - for negative numbers). It doesn't depend on locale, but does require rev.
commafy ()
{
printf %s "${1%%[0-9]*}"
printf '%s\n' "${1##*[!0-9]}" |
rev |
sed -E 's/[0-9]{3}/&,/g; s/,$//' |
rev
}
Example:
commafy '$19366'
# gives
$19,366
You could slightly simplify this too:
printf %s \$
printf '%s\n' 19366 |
rev |
sed -E 's/[0-9]{3}/&,/g; s/,$//' |
rev
Simplistically -
$: sed -E 's/([0-9]*)([0-9][0-9])$/$\1,\2/'<<<"19366"
$193,66
How to get a shift between two characters in bash?
For instance, in C++ we have:
'c'-'a'=2
Are there any elegant solutions?
Define ord to get the ASCII value of each character (from Unix & Linux Stack Exchange, Bash FAQ):
ord() { LC_CTYPE=C printf '%d' "'$1"; }
(note that the ' is not a typo! It is required for printf to treat a character as a number1)
Then you can subtract one from the other:
$ echo "$(( "$(ord c)" - "$(ord a)" ))"
2
If you wanted to put this in a function, you could:
diff_ord() { echo "$(( "$(ord $1)" - "$(ord $2)" ))"; }
Then call it like:
$ diff_ord c a
2
If the leading character is a single-quote or double-quote, the value shall be the numeric value in the underlying codeset of the character following the single-quote or double-quote.
I have this variable:
A="Some variable has value abc.123"
I need to extract this value i.e abc.123. Is this possible in bash?
Simplest is
echo "$A" | awk '{print $NF}'
Edit: explanation of how this works...
awk breaks the input into different fields, using whitespace as the separator by default. Hardcoding 5 in place of NF prints out the 5th field in the input:
echo "$A" | awk '{print $5}'
NF is a built-in awk variable that gives the total number of fields in the current record. The following returns the number 5 because there are 5 fields in the string "Some variable has value abc.123":
echo "$A" | awk '{print NF}'
Combining $ with NF outputs the last field in the string, no matter how many fields your string contains.
Yes; this:
A="Some variable has value abc.123"
echo "${A##* }"
will print this:
abc.123
(The ${parameter##word} notation is explained in §3.5.3 "Shell Parameter Expansion" of the Bash Reference Manual.)
Some examples using parameter expansion
A="Some variable has value abc.123"
echo "${A##* }"
abc.123
Longest match on " " space
echo "${A% *}"
Some variable has value
Longest match on . dot
echo "${A%.*}"
Some variable has value abc
Shortest match on " " space
echo "${A%% *}"
some
Read more Shell-Parameter-Expansion
The documentation is a bit painful to read, so I've summarised it in a simpler way.
Note that the '*' needs to swap places with the ' ' depending on whether you use # or %. (The * is just a wildcard, so you may need to take off your "regex hat" while reading.)
${A% *} - remove shortest trailing * (strip the last word)
${A%% *} - remove longest trailing * (strip the last words)
${A#* } - remove shortest leading * (strip the first word)
${A##* } - remove longest leading * (strip the first words)
Of course a "word" here may contain any character that isn't a literal space.
You might commonly use this syntax to trim filenames:
${A##*/} removes all containing folders, if any, from the start of the path, e.g.
/usr/bin/git -> git
/usr/bin/ -> (empty string)
${A%/*} removes the last file/folder/trailing slash, if any, from the end:
/usr/bin/git -> /usr/bin
/usr/bin/ -> /usr/bin
${A%.*} removes the last extension, if any (just be wary of things like my.path/noext):
archive.tar.gz -> archive.tar
How do you know where the value begins? If it's always the 5th and 6th words, you could use e.g.:
B=$(echo "$A" | cut -d ' ' -f 5-)
This uses the cut command to slice out part of the line, using a simple space as the word delimiter.
As pointed out by Zedfoxus here. A very clean method that works on all Unix-based systems. Besides, you don't need to know the exact position of the substring.
A="Some variable has value abc.123"
echo "$A" | rev | cut -d ' ' -f 1 | rev
# abc.123
More ways to do this:
(Run each of these commands in your terminal to test this live.)
For all answers below, start by typing this in your terminal:
A="Some variable has value abc.123"
The array example (#3 below) is a really useful pattern, and depending on what you are trying to do, sometimes the best.
1. with awk, as the main answer shows
echo "$A" | awk '{print $NF}'
2. with grep:
echo "$A" | grep -o '[^ ]*$'
the -o says to only retain the matching portion of the string
the [^ ] part says "don't match spaces"; ie: "not the space char"
the * means: "match 0 or more instances of the preceding match pattern (which is [^ ]), and the $ means "match the end of the line." So, this matches the last word after the last space through to the end of the line; ie: abc.123 in this case.
3. via regular bash "indexed" arrays and array indexing
Convert A to an array, with elements being separated by the default IFS (Internal Field Separator) char, which is space:
Option 1 (will "break in mysterious ways", as #tripleee put it in a comment here, if the string stored in the A variable contains certain special shell characters, so Option 2 below is recommended instead!):
# Capture space-separated words as separate elements in array A_array
A_array=($A)
Option 2 [RECOMMENDED!]. Use the read command, as I explain in my answer here, and as is recommended by the bash shellcheck static code analyzer tool for shell scripts, in ShellCheck rule SC2206, here.
# Capture space-separated words as separate elements in array A_array, using
# a "herestring".
# See my answer here: https://stackoverflow.com/a/71575442/4561887
IFS=" " read -r -d '' -a A_array <<< "$A"
Then, print only the last elment in the array:
# Print only the last element via bash array right-hand-side indexing syntax
echo "${A_array[-1]}" # last element only
Output:
abc.123
Going further:
What makes this pattern so useful too is that it allows you to easily do the opposite too!: obtain all words except the last one, like this:
array_len="${#A_array[#]}"
array_len_minus_one=$((array_len - 1))
echo "${A_array[#]:0:$array_len_minus_one}"
Output:
Some variable has value
For more on the ${array[#]:start:length} array slicing syntax above, see my answer here: Unix & Linux: Bash: slice of positional parameters, and for more info. on the bash "Arithmetic Expansion" syntax, see here:
https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Arithmetic-Expansion
https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Shell-Arithmetic
You can use a Bash regex:
A="Some variable has value abc.123"
[[ $A =~ [[:blank:]]([^[:blank:]]+)$ ]] && echo "${BASH_REMATCH[1]}" || echo "no match"
Prints:
abc.123
That works with any [:blank:] delimiter in the current local (Usually [ \t]). If you want to be more specific:
A="Some variable has value abc.123"
pat='[ ]([^ ]+)$'
[[ $A =~ $pat ]] && echo "${BASH_REMATCH[1]}" || echo "no match"
echo "Some variable has value abc.123"| perl -nE'say $1 if /(\S+)$/'
I am starting to use the printf instead of echo. My first forray into printf %s is this:
#!/bin/bash
danny=$(tail -1 /come/and/play/with/US.log| ~/walt/convert_gm_est)
printf "%s" $danny
08:02.020ZINFO<-casper/casperbox001wowSYSSTATS[sz=21,tag=0,aux=0]process_log2221397800
It eliminates all the spaces in the string. So I added a space after the format specifier. the string looks nicer with the spaces plus I could rip out the time with awk very easily. I don't see anything about this on the interwebs. I tried a "%s\s" and that did not work. Is this standard procedure for using format specifiers - use a space after %s? Is this the way to do it or am I missing something?
#!/bin/bash
danny=$(tail -1 /come/and/play/with/US.log| ~/walt/convert_gm_est)
printf "%s " $danny
08:02.020Z INFO <- casper/casperbox001wow SYSSTATS[sz=21, tag=0, aux=0] process_log 2221397 80 0 casper#casperbox001wow:~$
When the shell evaluates:
printf "%s" $danny
the shell will expand the value of the variable danny and then split it into words. It will also expand globs in those words. Once that is done, the expression will look something like this (quotes added for clarification):
printf '%s' '08:02.020Z' 'INFO' '<-' 'casper/casperbox001' 'wow0' 'SYSSTATS'...
printf repeats its format string until all of the arguments are consumed. So using the format string %s causes the arguments to be concatenated without intervening spaces.
You probably meant to quote $danny so that it would be presented as a single argument to printf:
printf "%s" "$danny"
I have a text stream that looks like this:
----------------------------------------
s123456789_9780
heartbeat:test # 1344280205000000: '0'
heartbeat:test # 1344272490000000: '0'
Those long numbers are timestamps in microseconds. I would like to run this output through some sort of pipe that will change those timestamps to a more human-understandable date.
I have a date command that can do that, given just the timestamp (with the following colon):
$ date --date=#$(echo 1344272490000000: | sed 's/.......$//') +%Y/%d/%m-%H:%M:%S
2012/06/08-10:01:30
I would like to end up with something like this:
----------------------------------------
s123456789_9780
heartbeat:test # 2012/06/08-12:10:05: '0'
heartbeat:test # 2012/06/08-10:01:30: '0'
I don't think sed will allow me to match the timestamp and replace it with the value of calling a shell function on it (although I'd love to be shown wrong). Perhaps awk can do it? I'm not very familiar with awk.
The other part that seems tricky to me is letting the lines that don't match through without modification.
I could of course write a Python program that would do this, but I'd rather keep this in shell if possible (this is generated inside a shell script, and I'd rather not have dependencies on outside files).
This might work for you (GNU sed):
sed '/# /!b;s//&\n/;h;s/.*\n//;s#\(.\{10\}\)[^:]*\(:.*\)#date --date=#\1 +%Y/%d/%m-%H:%M:%S"\2"#e;H;g;s/\n.*\n//' file
Explanation:
/# /!b bail out and just print any lines that don't contain an # followed by a space
s//&\n/ insert a newline after the above pattern
h copy the pattern space (PS) to the hold space (HS)
s/.*\n// delete upto and including the # followed by a space
s#\(.\{10\}\)[^:]*\(:.*\)#date --date=#\1 +%Y/%d/%m-%H:%M:%S"\2"#e from whats remaining in the PS, make a back reference of the first 10 characters and from the : to the end of the string. Have these passed in to the date command and evaluate the result into the PS
H append the PS to the HS inserting a newline at the same time
g copy the HS into the PS
s/\n.*\n// remove the original section of the string
Bash with a little sed, preserving the whitespace of the input:
while read -r; do
parts=($REPLY)
if [[ ${parts[0]} == "heartbeat:test" ]]; then
dateStr=$(date --date=#${parts[2]%000000:} +%Y/%d/%m-%H:%M:%S)
REPLY=$(echo "$REPLY" | sed "s#[0-9]\+000000:#$dateStr#")
fi
printf "%s\n" "$REPLY"
done
How about:
while read s1 at tm s2
do
tm=${tm%000000:}
echo $s1 $at $(date --date #$tm +%Y/%d/%m-%H:%M:%S)
done < yourfile
I would also like to see a sed solution, but it is a bit beyond my sed-fu. As awk supports strftime it is fairly straight forward here:
awk '
/^ *heartbeat/ {
gsub(".{7}$", "", $3)
$3 = strftime("%Y/%d/%m-%T", $3)
print " ", $1, $3
}
$0 !~ /heartbeat/' file
Output:
s123456789_9780
heartbeat:test 2012/06/08-21:10:05
heartbeat:test 2012/06/08-19:01:30
$3 is the microsecond field. gsub converts the timestamp to seconds.
The $0 !~ makes sure non-heartbeat lines are printed ({ print } implicitly is the default block).
This does it mostly within bash using your date command:
#!/bin/bash
IFS=$
while read a ; do
case "$a" in
*" # "[0-9]*) pre=${a% # *}
a=${a#$pre # }
post=${a##*:}
a=${a%??????:$post}
echo "$pre$(date --date=#$a +%Y/%d/%m-%H:%M:%S):$post"
;;
*) echo "$a" ;;
esac
done <<.
----------------------------------------
s123456789_9780
heartbeat:test # 1344280205000000: '0'
heartbeat:test # 1344272490000000: '0'
.