I am writing a script in bash using awk (mawk) and I would like to return the output of a system command to awk.
I have a variable called created that is in ISO format: 2013-12-26T17:03:05Z and I want to convert it from ISO to epoch. I can run this line created_epoch=system("date +\%s -d " created) and it prints the output to the terminal, but the variable created_epoch is equal to 0, not the epoch time.
Does anyone know how I can convert an ISO format to epoch in awk?
Thanks!
Try using getline into a variable from a pipe instead of system
cmd="date +\%s -d "created; cmd | getline current_time
https://www.gnu.org/software/gawk/manual/html_node/Getline_002fVariable_002fPipe.html#Getline_002fVariable_002fPipe
The system command returns the exit status returned by the command that was executed as its value.
http://www.delorie.com/gnu/docs/gawk/gawk_137.html
The system command in awk, as in C, returns the exit status of a command, and not the output of said command. As others have suggested, your best bet is using getline from a pipe
I ran into a similar problem a while back and rolled my own exec function, here it is:
# get the output of a command
function exec(cmd, data) {
while ((cmd | getline data) > 0) printf("%s", data);
close(cmd);
}
Then using the function defined above, you should do something like this:
epoch = exec("date +%s -d \"<ISO DATE>\"")
where <ISO DATE> is an ISO conformant date/timestamp.
Example:
# helper.awk is where `exec` is defined
awk '#include "helper.awk"; BEGIN
{
epoch = exec("gdate -d \"2014-02-14T12:30\" +%s");
print epoch
}'
# 1392406200
Do it like this:
created_epoch=$(date +\%s -d "$created")
echo $created_epoch
1388077385
Related
I've written my first script, one in which I want to know if 2 files have the same values in a specific column.
Both files are WEKA machine-learning prediction outputs for different algorithms, hence they have to be in the same format, but the prediction column would be different.
Here's the code I've written based on the tutorial presented in https://linuxconfig.org/bash-scripting-tutorial-for-beginners:
#!/bin/bash
lineasdel1=$(wc -l $1 | awk '{print $1}')
lineasdel2=$(wc -l $2 | awk '{print $1}')
if [ "$lineasdel1" != "$lineasdel2" ]; then
echo "Files $1 and $2 have different number of lines, unable to perform"
exit 1
fi
function quitalineasraras {
awk '$1!="==="&&NF>0'
}
function acomodo {
awk '{gsub(/^ +| +$/, ""); gsub(/ +0/, " W 0"); gsub(/ +1$/, " W 1"); gsub(/ +/, "\t") gsub(/\+\tW/, "+"); print}'
}
function procesodel1 {
quitalineasraras "$1" | acomodo
}
function procesodel2 {
quitalineasraras "$2" | acomodo
}
el1procesado=$(procesodel1)
el2procesado=$(procesodel2)
function pegar {
paste <(echo "$el1procesado") <(echo "$el2procesado")
}
function contarintersec {
awk 'BEGIN {FS="\t"} $3==$8 {n++} END {print n}'
}
unido=$(pegar)
interseccion=$(contarintersec $unido)
echo "Estos 2 archivos tienen $interseccion coincidencias."
I ran all individual codes of all functions in the terminal and verified they work successfully (I'm using Linux Mint 19.2). Script's permissions also have been changed to make it executable. Paste command also is supposed to work with that variable syntax.
But when I run it via:
./script.sh file1 file2
if both files have the same number of lines, and I press enter, no output is obtained; instead, the terminal opens an empty line with cursor waiting for something. In order to write another command, I've got to press CTRL+C.
If both files have different number of lines the error message prints successfully, so I think the problem has something to do with the functions, with the fact that awk has different syntax for some chores, or with turning the output of functions into variables.
I know that I'm missing something, but can't come up with what could be.
Any help will be appreciated.
what could be.
function quitalineasraras {
awk '$1!="==="&&NF>0'
}
function procesodel1 {
quitalineasraras "$1" | acomodo
}
el1procesado=$(procesodel1)
The positional variables $1 are set for each function separately. The "$1" inside procesodel1 expands to empty. The quitalineasraras is passed one empty argument "".
The awk inside quitalineasraras is passed only the script without the filename, so it reads the input for standard input, ie. it waits for the input on standard input.
The awk inside quitalineasraras without any file arguments makes your script seem to wait.
I am trying to convert ProcessedTime": 1545166011.794351 to a date using date function in bash script, what is the best way to convert it ?
date -d #154166011.794351
Date command should work fine.
With GNU awk:
echo 154166011.794351 | awk '{print strftime("%Y-%m-%d %H:%M:%S",$0)}'
Output:
1974-11-20 08:53:31
Source: https://unix.stackexchange.com/a/168317/74329
I am trying to remove the new line character for a date function and have it include spaces. I am saving the variables using this:
current_date=$(date "+%m/%d/ AT %y%H:%M:%S" )
I can see that this is the right format I need by doing a echo $current_date.
However, when I need to use this variable it does not act the way I would like it.
awk '(++n==47) {print "1\nstring \nblah '$current_date' blah 2; n=0} (/blah/) {n=0} {print}' input file > output file
I need the date to stay in the current line of text and continue with no newline unless specified.
Thanks in advance.
Rather than attempting to insert the variable into the command string as you are doing, you can pass it to awk like this:
awk -v date="$(date "+%m/%d/ AT %y%H:%M:%S")" '# your awk one-liner here' input_file
You can then use the variable date as an awk variable within the script:
print "1\nstring \nblah " date " blah 2";
As an aside, it looks like your original print statement was broken, as there were double quotes missing from the end of it.
when i store the output of history command via ssh in a file i get something like this
ssh -i private_key user#ip 'export HISTFILE=~/.bash_history; export HISTTIMEFORMAT="%D-%T "; set -o history; history' > myfile.txt
OUTPUT
#1337431451
command
as far as ive learnt this hash string represents a timestamp. how do i change this to a string of my desired format
P.S- using history in ssh is not outputting with timestamps. Tried almost everything. So i guess the next best thing to do would be to convert these # timestamps to a readable date time format myself. How do i go about it?
you can combine rows with paste command:
paste -sd '#\n' .bash_history
and convert date with strftime in awk:
echo 1461136015 | awk '{print strftime("%d/%m/%y %T",$1)}'
as a result bash history with timestamp can be parsed by the next command:
paste -sd '#\n' .bash_history | awk -F"#" '{d=$2 ; $2="";print NR" "strftime("%d/%m/%y %T",d)" "$0}'
which converts:
#1461137765
echo lala
#1461137767
echo bebe
to
1 20/04/16 10:36:05 echo lala
2 20/04/16 10:36:07 echo bebe
also you can create script like /usr/local/bin/fhistory with content:
#!/bin/bash
paste -sd '#\n' $1 | awk -F"#" '{d=$2 ; $2="";print NR" "strftime("%d/%m/%y %T",d)" "$0}'
and quickly parse bash history file with next command:
fhistory .bash_history
Interesting question: I have tried it but found no simple and clean solution to access the history in a non-interactive shell. However, the format of the history file is simple, and you can write a script to parse it. The following python script might be interesting. Invoke it with ssh -i private_key user#ip 'path/to/script.py .bash_history':
#! /usr/bin/env python3
import re
import sys
import time
if __name__ == '__main__':
pattern = re.compile(br'^#(\d+)$')
out = sys.stdout.buffer
for pathname in sys.argv[1:]:
with open(pathname, 'rb') as f:
for line in f:
timestamp = 0
while line.startswith(b'#'):
match = pattern.match(line)
if match: timestamp, = map(int, match.groups())
line = next(f)
out.write(time.strftime('%F %T ', time.localtime(timestamp)).encode('ascii'))
out.write(line)
Using just Awk and in a slightly more accurate way:
awk -F\# '/^#1[0-9]{9}$/ { if(cmd) printf "%5d %s %s\n",n,ts,cmd;
ts=strftime("%F %T",$2); cmd=""; n++ }
!/^#1[0-9]{9}$/ { if(cmd)cmd=cmd " " $0; else cmd=$0 }' .bash_history
This parses only lines starting with something that looks like a timestamp (/^#1[0-9]{9}$/), compiles all subsequent lines up until the next timestamp, combines multi-line commands with " " (1 space) and prints the commands in a format similar to history including a numbering.
Note that the numbering does not (necessarily) match if there are multi-line commands.
Without the numbering and breaking up multi-line commands with a newline:
awk -F\# '/^#1[0-9]{9}$/ { if(cmd) printf "%s %s\n",ts,cmd;
ts=strftime("%F %T",$2); cmd="" }
!/^#1[0-9]{9}$/ { if(cmd)cmd=cmd "\n" $0; else cmd=$0 }' .bash_history
Finally, a quick and dirty solution using GNU Awk (gawk) to also sort the list:
gawk -F\# -v histtimeformat="$HISTTIMEFORMAT" '
/^#1[0-9]{9}$/ { i=$2 FS NR; cmd[i]="" }
!/^#1[0-9]{9}$/ { if(cmd[i]) cmd[i]=cmd[i] "\n" $0; else cmd[i]=$0 }
END { PROCINFO["sorted_in"] = "#ind_str_asc"
for (i in cmd) { split(i,arr)
print strftime(histtimeformat,arr[1]) cmd[i]
}
}'
I have strftime format of time, let's say (%Y-%m-%d %H:%M:%S) and a file which should contain this kind of data e.g. (2012-02-11 17:15:00). I need to check if given pattern actually matches the data.
How to approach this? awk, date?
EDIT:
More info: The user enters the strftime format, let's say on input. Then he enters a file which should contain those dates. I need to make sure, that those data are valid (he didn't make a mistake). So I need to check the rows in the input file and see, if there are data that matches the given pattern. Example:
user enters strftime format: (%Y-%m-%d %H:%M:%S)
input file: (2012-02-11 17:15:00) long sentence
VALID
user enters strftime format: Date[%Y.%m.%d %H:%M:%S]
input file: Date-2012.02.11 17:15:00- long sentence
INVALID
If you allow an external helper binary, I've written dateutils to batch process date and time data.
dconv -q -i '(%Y-%m-%d %H:%M:%S)' <<EOF
not a match: 2012-04-10 12:00:00
a match: (2012-04-10 13:00:00)
EOF
will give
2012-04-10T13:00:00
-i is the input format, -q suppresses warnings. And dconv tries to convert input lines to output lines (in this case it converts matching lines to ISO standard format.
So using this, a file matches completely if the number of input lines equals the number of output lines.
If you want to check current datetime:
echo "(2012-02-11 17:15:00)" | grep "$(date "+%Y-%m-%d %H:%M:%S")"
If some other you need GNU date (-d option). This works for me:
echo "(2012-02-11 17:15:00)" |
grep "$(date -d "2012-02-11 17:15:00" "+%Y-%m-%d %H:%M:%S")"
I would take a brute force approach to this: replace any %X specifier with a corresponding regular expression, then you can filter out lines that don't match the resulting generated regex:
user_format="%Y-%m-%d"
awk -v fmt_string="$user_format" '
BEGIN {
gsub(/[][(){}?|*+.]/ "\\&", fmt_string) # protect any regex-special chars
gsub(/%Y/, "([0-9]{4})", fmt_string)
gsub(/%m/, "(0[1-9]|1[012])", fmt_string)
gsub(/%d/, "(0[1-9]|[12][0-9]|3[01])", fmt_string)
# and so on
}
$0 !~ "^" fmt_string {print "line " NR " does not match: " $0}
' filename