I'm writing a bash script that reads and parses data from log lines, and it's trying to convert the format. The script is long, but I'll focus on the error:
if [[ $line == *"Marked: Malicious, To:"* ]]; then
QUEUE_ID=`awk ' {print $13}' <<< $line | sed 's/,//g'` # QUEUE ID... index of the array
F_TIME["X$QUEUE_ID"]=`awk '{print $3}' <<< $line ` # Hour
F_DAY["X$QUEUE_ID"]=`awk '{print $2}' <<< $line ` # Day
F_MONTH["X$QUEUE_ID"]=`awk '{print $1}' <<< $line ` # Month
#Procesing and obtaining the diff
F_FULLTIME["X$QUEUE_ID"]="${F_DAY["X$QUEUE_ID"]}-${F_MONTH["X$QUEUE_ID"]}-$year-${F_TIME["X$QUEUE_ID"]}" # creating the time in desired format
s=`date -j -f "%d-%b-%Y-%H:%M:%S" "${F_FULLTIME[X$QUEUE_ID]}" +%s` # ERROR LINE!! Doesn't work!
#s=`date -j -f "%d-%b-%Y-%H:%M:%S" "2-Mar-2016-22:24:33" +%s` # Test line
echo ".....${F_FULLTIME["X$QUEUE_ID"]}....2-Mar-2016-22:24:33...."
echo "time $s ...\n";
fi
Test process: I comment the test line (marked on the script), and I try to do conversion to variable s. This is the output for each key:
Failed conversion of ``2-Mar-2016-20:30:03'' using format ``%d-%b-%Y-%H:%M:%S''
date: illegal time format
usage: date [-jnu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
[-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
.....2-Mar-2016-20:30:03....2-Mar-2016-22:24:33....
Date is not taking the format. And as you can see, desired format and the format I have on the variable are pretty much same.
Test #2: I comment the error line, and I remove comment the test line. This is the result:
.....2-Mar-2016-22:06:10....2-Mar-2016-22:24:33....
time 1456953873 ...
Works perfectly. And of course text is aparently same as my variable on the first try.
The code looks OK, although overly complex. Try this simplified version, which should be a little easier to debug. We're going to work with individual variables as much as possible, rather than array entries.
if [[ $line == *"Marked: Malicious, To:"* ]]; then
# run `awk` once
fields=$(awk '{print $1, $2, $3, $13}' <<< "$line")
read f_month f_day f_time queue_id <<< "$fields"
queue_id=${queue_id//,/}
# Is the X really necessary?
# F_TIME["X$queue_id"]=$f_time
# F_DAY["X$queue_id"]=$f_day
# F_MONTH["X$queue_id"]=$f_month
#Procesing and obtaining the diff
f_fulltime="$f_day-$f-month-$year-$f_time"
# F_FULLTIME["X$queue_id"]=$f_fulltime
s=$(date -j -f "%d-%b-%Y-%H:%M:%S" "$f_fulltime" )
fi
Some questions to ask:
What are the exact values of queue_id, f_time, f_day, and f_month? Use, e.g., printf '%s' "$queue_id" | hexdump -C to see if
there are any hidden characters from $line that are confusing date.
Can you reproduce the problem by hardcoding the values of the pieces instead of just f_fulltime itself?
Can you reproduce the problem by hard-coding the value of line just prior to the call to awk?
Well.. I've clean the script.., Remove extra variables, declared arrays, and I found where the error comes.. but not sure why. The error comes on the variable $year.
This is defined as:
year=`date +"%Y"`
Then.. when I tried this conversion:
s_fulltime["X$queue_id"]="$day-$month-$year-$time" >> date: illegal time format
But.. if I harcode "2016" then..
year="2016"
All works fine..
Why?
Related
I'm relatively new to bash scripting and I would like someone to explain this properly, thank you. Here is my code:
#! /bin/bash
echo "first arg: $1"
echo "first arg: $2"
var="$( grep -rnw $1 -e $2 | cut -d ":" -f1 )"
var2=$( grep -rnw $1 -e $2 | cut -d ":" -f1 | awk '{print substr($0,length,1)}')
echo "$var"
echo "$var2"
The problem I have is with the output, the script I'm trying to write is a c++ function searcher, so upon launching my script I have 2 arguments, one for the directory and the second one as the function name. This is how my output looks like:
first arg: Projekt
first arg: iseven
Projekt/AX/include/ax.h
Projekt/AX/src/ax.cpp
h
p
Now my question is: how do can I save the line by line output as a variable, so that later on I can use var as a path, or to use var2 as a character to compare. My plan was to use IF() statements to determine the type, idea: IF(last_char == p){echo:"something"}What I've tried was this question: Capturing multiple line output into a Bash variable and then giving it an array. So my code looked like: "${var[0]}". Please explain how can I use my line output later on, as variables.
I'd use readarray to populate an array variable just in case there's spaces in your command's output that shouldn't be used as field separators that would end up messing up foo=( ... ). And you can use shell parameter expansion substring syntax to get the last character of a variable; no need for that awk bit in your var2:
#!/usr/bin/env bash
readarray -t lines < <(printf "%s\n" "Projekt/AX/include/ax.h" "Projekt/AX/src/ax.cpp")
for line in "${lines[#]}"; do
printf "%s\n%s\n" "$line" "${line: -1}" # Note the space before the -1
done
will display
Projekt/AX/include/ax.h
h
Projekt/AX/src/ax.cpp
p
printf is printing my variables in the wrong order and newline isn't working. after iterating on various printf statements, it still doesn't working and i'm not quite sure what's wrong.
this is my current code:
cat ~/data/quotes.csv | while IFS=, read author quote; do
author_s=$(echo $author | cut -d'"' -f 2) # removes quotations - e.g. turns "Jonathan Kozol" to Jonathan Kozol
printf "$quote\n\t~$author_s"
done | sort -R | tail -1 # print one random line from quotes.csv
this is my output (i'm also not sure what's causing the error):
-bash: printf: `w': invalid format character
~ Jonathan Kozol"Don't compromise yourself. You are all you've got."
however, i'd like to end up with something like this:
"Don't compromise yourself. You are all you've got."
~ Jonathan Kozol
also, when i try printing the variables $author_s and $quote on separate lines
e.g.
printf "$quote\n"
printf "$author_s"
author doesn't print
The first part of quotes.csv:
"Author","Quote"
"Thomas Edison","Genius is one percent inspiration and ninety-nine percent perspiration."
"Yogi Berra","You can observe a lot just by watching."
"Abraham Lincoln","A house divided against itself cannot stand."
"Johann Wolfgang von Goethe","Difficulties increase the nearer we get to the goal."
"Byron Pulsifer","Fate is in your hands and no one elses"
"Lao Tzu","Be the chief but never the lord."
"Carl Sandburg","Nothing happens unless first we dream."
"Aristotle","Well begun is half done."
"Yogi Berra","Life is a learning experience, only if you learn."
"Margaret Sangster","Self-complacency is fatal to progress."
"Buddha","Peace comes from within. Do not seek it without."
The first argument to printf should contain the format string. Your particular format would be "%s\n\t~ %s\n":
The first %s is the actual quote
\n\t a newline and a tab
~ %s\n a tilde, the author and a newline
Example:
#!/bin/bash
while IFS=, read -r author quote
do
author="${author%\"}" # remove first "
author="${author#\"}" # remove last "
printf "%s\n\t~ %s\n" "$quote" "$author"
done < quotes.csv
In order to select a random quote you could use shuf:
#!/bin/bash
tail -n +2 quotes.csv | shuf -n1 | while IFS=, read -r author quote
do
author="${author%\"}" # remove first "
author="${author#\"}" # remove last "
printf "%s\n\t~ %s\n" "$quote" "$author"
done
Here tail -n +2 quotes.csv skips the first line in the file ("Author","Quote") and shuf -n1 picks one random line.
The second example again, but using process substitution instead:
#!/bin/bash
while IFS=, read -r author quote
do
author="${author%\"}" # remove first "
author="${author#\"}" # remove last "
printf "%s\n\t~ %s\n" "$quote" "$author"
done < <(shuf -n1 <(tail -n +2 quotes.csv))
I am trying to figure out how to parse a file with ISO 8601-formatted time stamps, add one second and then output them to a file.
All the examples I have found don't really tell me how to do it with ISO 8601 date/time strings.
Example:
read a csv of times like: "2017-02-15T18:47:59" (some are correct, others are not)
and spit out in a new file "2017-02-15T18:48:00"
mainly just trying to correct a bunch of dates that have 59 seconds at the end to round up to the 1 second mark.
This is my current progress:
#!/bin/bash
while IFS='' read -r line || [[ -n "$line" ]]; do
# startd=$(date -j -f '%Y%m%d' "$line" +'%Y%m%d');
# echo "$startd";
startd=$(date -j -u -f "%a %b %d %T %Z %Y" $line)
#startd=$(date -j -f '%Y%m%d' "$line" +'%Y%m%d');
echo "$startd";
done < "$1"
Any help would be appreciated
jm666's helpful perl answer will be much faster than your shell loop-based approach.
That said, if you want to make your bash code work on macOS, with its BSD date implementation, here's a solution:
# Define the input date format, which is also used for output.
fmt='%Y-%m-%dT%H:%M:%S'
# Note: -j in all date calls below is needed to suppress setting the
# system date.
while IFS= read -r line || [[ -n "$line" ]]; do
# Parse the line at hand using input format specifier (-f) $fmt,
# and output-format specifier (+) '%s', which outputs a Unix epoch
# timestamp (in seconds).
ts=$(date -j -f "$fmt" "$line" +%s)
# See if the seconds-component (%S) is 59...
if [[ $(date -j -f %s "$ts" +%S) == '59' ]]; then
# ... and, if so, add 1 second (-v +1S).
line=$(date -j -f %s -v +1S "$ts" +"$fmt")
fi
# Output the possibly adjusted timestamp.
echo "$line"
done < "$1"
Note that input dates such as 2017-02-15T18:47:59 are interpreted as local time, because they contain no time-zone information.
This could do the job
perl -MTime::Piece -nlE '$f=q{%Y-%m-%dT%H:%M:%S};$t=Time::Piece->strptime($_,$f)+1;say $t->strftime($f)' < dates.txt
if the dates.txt contains
2017-02-15T18:47:59
2016-02-29T23:59:59
2017-02-28T23:59:59
2015-12-31T23:59:59
2010-10-10T10:10:10
the above produces
2017-02-15T18:48:00
2016-03-01T00:00:00
2017-03-01T00:00:00
2016-01-01T00:00:00
2010-10-10T10:10:11
There's a file with some lines containing some text and either date or time stamp:
...
string1-20141001
string2-1414368000000
string3-1414454400000
...
I want to quickly convert time stamps to dates, like this:
$ date -d #1414368000 +"%Y%m%d"
20141027
and I want to do this dynamically with sed or some similar command line tool. For testing I unsuccessfully use this:
$ echo "something-1414454400000" | sed "s/-\(..........\)...$/-$(date -d #\\1 +'%Y%m%d')/"
date: invalid date '#\\1'
something-
but echoing seems to be working:
$ echo "something-1414454400000" | sed "s/-\(..........\)...$/-$(echo \\1)/"
something-1414454400
so what could be done?
It's interesting what's happening here. Some pointers:
Always single-quote your regex for sed, if possible, when using BASH (etc), especially if using special characters like$. This is why date is being run (with -d #\\1) before sed even gets involved.
Your "working" echo example isn't, actually (I believe): echo \\1 produces \1 (and as above, will do so before sed even gets invoked). This then happens to valid sed replacement syntax, so will substitute your group on the LHS, which is why the output looks about right.
Note that by using -r, you can use easier / more advanced regex syntax.
Hard to say exactly what to do without a bit more context, but to fix the immediate problems, try something like:
echo "something-1414454400000" | sed -re 's/-([0-9]{10,}).+/-$(date -d #\1 +"%Y%m%d")/'
which produces: $(date -d #1414454400) (which you can then pipe to sh)
Or for a more complete solution, you can change the regex to produce a shell command directly, and pipe it:
echo "something-1414454400000" | sed -re 's/(.*-)([0-9]{10,10}).+/echo \1$(date -d #\2 \"+%Y%M%d\")/' | sh
..producing something-20140028
You can do this in BASH:
while read -r p; do
if [[ "$p" =~ ^(.+-)([0-9]{10}).{3}$ ]]; then
echo -n "${BASH_REMATCH[1]}"
date -d "#${BASH_REMATCH[2]}" +"%Y%m%d"
else
echo "$p"
fi
done < file
OUTPUT:
string1-20141001
string2-20141026
string3-20141027
awk -F- 'BEGIN { OFS=FS }
$2 ~ /^[0-9]{13}$/ {
"date -d#" $2/1000 " +%Y%m%d " | getline t; $2=t }1'
Just try this command. I have checked it. It is working on your inputs.
cat file | sed -E "s,(.*)-(.*),\1-`date -d #1414368000 +'%Y%m%d'`,g"
So, everything works fine in the code, except for one tiny little thing.
This part:
if [ "$LIMITHOURS" -gt "0" -a "$LIMITHOURS" -lt "24" ]; then
x=$(($LIMITHOURS*60*60))
fi
SDATE=$( echo "01/jan/2003:11:00:06 +0100"| sed 's/[/]/ /g' |sed 's/:/ /')
EDATE=$(date --date "$SDATE - $x seconds" +"%d%m%Y%H%M%S")
#echo "$SDATE"
#echo "$EDATE"
while read LINE; do
CDATE=$( awk '{print $4}'| sed 's/[[]//' | sed 's/[/]//g' |sed 's/://g' )
DATE=$(date --date "$CDATE" +"%d%m%Y%H%M%S")
#echo "$CDATE"
done < "$FILENAME"
When I try to run the script, I get the error message "date: Argument list too long
" and I know that the problem is in the while loop, with:
DATE=$(date --date "$CDATE" +"%d%m%Y%H%M%S")
Anyone who know any solution for this? I want the date format in ddmmYYYYHHMMSS, eg. 23102002120022
You can find rest of the script here: http://pastebin.com/PMk2QDre
This code:
while read LINE; do
CDATE=$( awk '{print $4}'| sed 's/[[]//' | sed 's/[/]//g' |sed 's/://g' )
DATE=$(date --date "$CDATE" +"%d%m%Y%H%M%S")
#echo "$CDATE"
done < "$FILENAME"
will read one line from $FILENAME into the variable LINE, but then the first call to awk is reading the rest of the lines. The resulting CDATE value is probably too large to fit in a single command line, never mind it containing too many dates. You probably wanted
echo "$LINE" | awk '{print $4}' | ...
A simpler way to strip the undesirable characters from LINE, however, is
CDATE=${LINE//[\/[:]}