I have the following line:
for custom_field in $(jq -r '.issues[] | .fields.created' 1.json
); do
echo $custom_field
done
Output:
2018-03-06T21:24:41.000+0000
2018-03-06T22:48:47.000+0000
How to compare current datetime with each output and if it's older than 3 hours to print "old"?
Given input like
{ "issues":
[{"id": 1, "fields": {"created": "2018-03-06T21:24:41.000+0000"}},
{"id": 2, "fields": {"created": "2018-03-06T22:48:47.000+0000"}},
{"id": 3, "fields": {"created": "2018-03-09T22:48:47.000+0000"}}]}
you can use the built-in date manipulation functions to print the old records with something like
jq -r '(now-3600*3) as $when | .issues[] |
select(.fields.created | strptime("%Y-%m-%dT%H:%M:%S.000+0000") | mktime < $when) |
[.id, .fields.created, "old"]' 1.json
where the last line probably needs tweaking to produce exactly the output you want.
It is much easier to convert first and subtract the three hours.
Example below converts to seconds and prints true if condition is met.
date_in_seconds=$(date -d $custom_field +"%s");
[ ${date_in_seconds} -gt 259200 ] && echo true;
For non GNU versions of the date command you can use the following;
date_in_seconds=$(date -j -f '%Y-%m-%d %H:%M:%S' "2016-02-22 20:22:14" '+%s')
Keep in mind that the EPOCH will rollover 1 Jan 2036.
Related
I have my command below and I want to have the result in the same line with delimeters. My command:
Array=("GET" "POST" "OPTIONS" "HEAD")
echo $(date "+%Y-%m-%d %H:%M")
for i in "${Array[#]}"
do
cat /home/log/myfile_log | grep "$(date "+%d/%b/%Y:%H")"| awk -v last5=$(date --date="-5 min" "+%M") -F':' '$3>=last5 && $3<last5+5{print}' | egrep -a "$i" | wc -l
done
Results is:
2019-01-01 13:27
1651
5760
0
0
I want to have the result below:
2019-01-01 13:27,1651,5760,0,0
It looks (to me) like the overall objective is to scan /home/log/myfile.log for entries that have occurred within the last 5 minutes and which match one of the 4 entries in ${Array[#]}, keeping count of the matches along the way and finally printing the current date and the counts to a single line of output.
I've opted for a complete rewrite that uses awk's abilities of pattern matching, keeping counts and generating a single line of output:
date1=$(date "+%Y-%m-%d %H:%M") # current date
date5=$(date --date="-5 min" "+%M") # date from 5 minutes ago
awk -v d1="${date1}" -v d5="${date5}" -F":" '
BEGIN { keep=0 # init some variables
g=0
p=0
o=0
h=0
}
$3>=d5 && $3<d5+5 { keep=1 } # do we keep processing this line?
!keep { next } # if not then skip to next line
/GET/ { g++ } # increment our counters
/POST/ { p++ }
/OPTIONS/ { o++ }
/HEAD/ { h++ }
{ keep=0 } # reset keep flag for next line
# print results to single line of output
END { printf "%s,%s,%s,%s,%s\n", d1, g, p, o, h }
' <(grep "$(date '+%d/%b/%Y:%H')" /home/log/myfile_log)
NOTE: The OP may need to revisit the <(grep "$(date ...)" /home/log/myfile.log) to handle timestamp periods that span hours, days, months and years, eg, 14:59 - 16:04, 12/31/2019 23:59 - 01/01/2020 00:04, etc.
Yeah, it's a bit verbose but a bit easier to understand; OP can rewrite/reduce as sees fit.
At the moment, I have a while-loop that takes a starting date, runs a python script with the day as the input, then takes the day + 1 until a certain due date is reached.
day_start=2016-01-01
while [ "$day_start"!=2018-01-01 ] ;
do
day_end=$(date +"%Y-%m-%d" -d "$day_start + 1 day")
python script.py --start="$day_start" --end="$day_end";
day_start=$(date +"%Y-%m-%d" -d "$day_start + 1 day")
done
I would like to do the same thing, but now to pick a random day between 2016-01-01 and 2018-01-01 and repeat until all days have been used once. I think it should be a for-loop instead of this while loop, but I have trouble to specify the for-loop over this date-range in bash. Does anyone have an idea how to formulate this?
It can take quite a long time if you randomly choose the dates because of the Birthday Problem. (You'll hit most of the dates over and over again but the last date can take quite some time).
The best idea I can give you is this:
Create all dates as before in a while loop (only the day_start-line)
Output all dates into a temporary file
Use sort -R on this file ("shuffles" the contents and prints the result)
Loop over the output from sort -R and you'll have dates randomly picked until all were reached.
Here's an example script which incorporates my suggestions:
#!/bin/bash
day_start=2016-01-01
TMPFILE="$(mktemp)"
while [ "$day_start" != "2018-01-01" ] ;
do
day_start=$(date +"%Y-%m-%d" -d "$day_start + 1 day")
echo "${day_start}"
done > "${TMPFILE}"
sort -R "${TMPFILE}" | while read -r day_start
do
day_end=$(date +"%Y-%m-%d" -d "$day_start + 1 day")
python script.py --start="$day_start" --end="$day_end";
done
rm "${TMPFILE}"
By the way, without the spaces in the while [ "$day_start" != "2018-01-01" ];, bash won't stop your script.
Fortunately, from 16 to 18 there was no leap year (or was it, and it just works because of that)?
Magic number: 2*365 = 730
The i % 100, just to have less output.
for i in {0..730}; do nd=$(date -d "2016/01/01"+${i}days +%D); if (( i % 100 == 0 || i == 730 )); then echo $nd ; fi; done
01/01/16
04/10/16
07/19/16
10/27/16
02/04/17
05/15/17
08/23/17
12/01/17
12/31/17
With the format instruction (here +%D), you might transform the output to your needs, date --help helps.
In a better readable format, and with +%F:
for i in {0..730}
do
nd=$(date -d "2016/01/01"+${i}days +%F)
echo $nd
done
2016-01-01
2016-04-10
2016-07-19
...
For a random distribution, use shuf (here, for bevity, with 7 days):
for i in {0..6}; do nd=$(date -d "2016/01/01"+${i}days +%D); echo $nd ;done | shuf
01/04/16
01/07/16
01/05/16
01/01/16
01/03/16
01/06/16
01/02/16
I have 52 json files (r$i.json) containing each 25 results (0 to 24). I'd like to create a json file with a special name for each of these results. The name would be composed according to the content of each of these results : YYYYMMDDHHMMSS_company_jobtitle.json
the command generating names work fine :
#!bin/bash
for ((j=0;j<=24;j++))
do
datein=$(jq <"r1.json" ".results[$j].date" | sed 's/"//g')
dateout=$(date -d "${datein}" +"%Y%m%d%H%M%S")
company=$(jq <"r1.json" ".results[$j].company" | sed 's/,//g;s/"//g;s/ //g')
job=$(jq <"r1.json" ".results[$j].jobtitle" | sed 's/,//g;s/"//g;s/ //g')
jq <"r1.json" ".results[$j]" > ${dateout}_${company}_${job}.json
done
Now when I replace r1 by r$i and add ((i=1;i<=52;j++)) it doesn't work... So I guess my problem comes from nested loop syntax in jq...
r1.json would look like that :
{
"radius" : 25,
"totalResults" : 1329,
"results" : [
{
"jobtitle" : "job1",
"company" : "company1,
"date" : "Sun, 01 Sep 2015 07:59:58 GMT",
}
,
{
"jobtitle" : "job2",
"company" : "company2",
"date" : "Sun, 02 Sep 2015 07:59:58 GMT",
}
,
|...]
{
"jobtitle" : "job25",
"company" : "company25,
"date" : "Sun, 25 Sep 2015 07:59:58 GMT",
}
]
}
You should respect the bash syntax in your fors:
for (( i=0; i<5; i++ ))
((i=1,i< =52,j++)) won't work, use ; instead of ,.
1) You wrote that your i-loop used ((i=1;i< =52;j++)); that should be ((i-1; i<=52; i++))
2) We can't see exactly what you did with respect to r1 and r$i, so if (1) doesn't resolve your difficulty, maybe you should double-check that what you did is actually what is needed. Should you change "> $outputname" to ">> $outputname"?
3) I suspect that rather than using s/"//g, it might be better to use the -r option of jq; you might also consider avoiding sed altogether (jq 1.5 has sub and gsub functions).
4) As I said, it would be better to get rid of all the backticks.
Finally I found the solution, and my issue didn't come from jq but from the syntax I was using for nested loops... Here it is :
for ((i=1;i<=kdr;i++))
do
for ((j=0;j<=24;j++))
do
datein=$(jq <"r$i.json" ".results[$j].date" | sed 's/"//g')
dateout=$(date -d "${datein}" +"%Y%m%d%H%M%S")
company=$(jq <"r$i.json" ".results[$j].company" | sed 's/,//g;s/"//g;s/ //g')
job=$(jq <"r$i.json" ".results[$j].jobtitle" | sed 's/,//g;s/"//g;s/ //g')
jq <"r$i.json" ".results[$j]" > ${dateout}_${company}_${job}.json
done
done
I am completely new to shell scripting.
I need to change the format of given date to customized format like i have date in a variable with format as MM/DD/YY HH:MM:SS but i want date in a format as MM/DD/YYY HH:MM:SS which is having four digits of the year.
We can change the sys date format but i need the same in a variable.
My code as below
START_DATE="12/20/14 05:59:01"
yr=`echo $START_DATE | cut -d ' ' -f1 | cut -d '/' -f3`
yr_len=`echo $yr | wc -c`
if [ $yr_len -lt 4 ]
then
tmp_yr="20${yr}";
else
1=1;
fi
ln=`echo $tmp_yr|wc -c`
After this i strucked in reframe the same date in wanted format.
Can some one please help me
Regards,
Sai.
Using GNU date:
date -d'02/16/15 09:16:04' "+%m/%d/%Y %T"
produces
02/16/2015 09:16:04
which is what you want. See man date for details about the formatting, or this question for a number of great examples.
One option may be using the date/time functions inside awk. Here is a oneliner:
echo '02/16/15 09:16:04' | sed 's\[/:]\ \g' | awk '{d0=$3+2000FS$1FS$2FS$4FS$5FS$6; d1=mktime(d0);print strftime("%m/%d/%Y %T", d1) }'
output is:
02/16/2015 09:16:04
You can find more strftime formats in https://www.gnu.org/software/gawk/manual/html_node/Time-Functions.html
I have this awk script that runs through a file and counts every occurrence of a given date. The date format in the original file is the standard date format, like this: Thu Mar 5 16:46:15 EST 2009 I use awk to throw away the weekday, time, and timezone, and then do my counting by pumping the dates into an associative array with the dates as indices.
In order to get the output to be sorted by date, I converted the dates to a different format that I could sort with bash sort.
Now, my output looks like this:
Date Count
03/05/2009 2
03/06/2009 1
05/13/2009 7
05/22/2009 14
05/23/2009 7
05/25/2009 7
05/29/2009 11
06/02/2009 12
06/03/2009 16
I'd really like the output to have more human readable dates, like this:
Mar 5, 2009
Mar 6, 2009
May 13, 2009
May 22, 2009
May 23, 2009
May 25, 2009
May 29, 2009
Jun 2, 2009
Jun 3, 2009
Any suggestions for a way I could do this? If I could do this on the fly when I output the count values that would be best.
UPDATE:
Here's my solution incorporating ghostdog74's example code:
grep -i "E[DS]T 2009" original.txt | awk '{printf "%s %2.d, %s\r\n",$2,$3,$6}' >dates.txt #outputs dates for counting
date -f dates.txt +'%Y %m %d' | awk ' #reformat dates as YYYYMMDD for future sort
{++total[$0]} #pump dates into associative array
END {
for (item in total) printf "%s\t%s\r\n", item, total[item] #output dates as yyyy mm dd with counts
}' | sort -t \t | awk ' #send to sort, then to cleanup
BEGIN {printf "%s\t%s\r\n","Date","Count"}
{t=$1" "$2" "$3" 0 0 0" #cleanup using example by ghostdog74
printf "%s\t%2.d\r\n",strftime("%b %d, %Y",mktime(t)),$4
}'
rm dates.txt
Sorry this looks so messy. I've tried to put clarifying comments in.
Use awk's sort and date's stdin to greatly simplify the script
Date will accept input from stdin so you can eliminate one pipe to awk and the temporary file. You can also eliminate a pipe to sort by using awk's array sort and as a result, eliminate another pipe to awk. Also, there's no need for a coprocess.
This script uses date for the monthname conversion which would presumably continue to work in other languages (ignoring the timezone and month/day order issues, though).
The end result looks like "grep|date|awk". I have broken it into separate lines for readability (it would be about half as big if the comments were eliminated):
grep -i "E[DS]T 2009" original.txt |
date -f - +'%Y %m %d' | #reformat dates as YYYYMMDD for future sort
awk '
BEGIN { printf "%s\t%s\r\n","Date","Count" }
{ ++total[$0] #pump dates into associative array }
END {
idx=1
for (item in total) {
d[idx]=item;idx++ # copy the array indices into the contents of a new array
}
c=asort(d) # sort the contents of the copy
for (i=1;i<=c;i++) { # use the contents of the copy to index into the original
printf "%s\t%2.d\r\n",strftime("%b %e, %Y",mktime(d[i]" 0 0 0")),total[d[i]]
}
}'
I get testy when I see someone using grep and awk (and sed, cut, ...) in a pipeline. Awk can fully handle the work of many utilities.
Here's a way to clean up your updated code to run in a single instance of awk (well, gawk), and using sort as a co-process:
gawk '
BEGIN {
IGNORECASE = 1
}
function mon2num(mon) {
return(((index("JanFebMarAprMayJunJulAugSepOctNovDec", mon)-1)/3)+1)
}
/ E[DS]T [[:digit:]][[:digit:]][[:digit:]][[:digit:]]/ {
month=$2
day=$3
year=$6
date=sprintf("%4d%02d%02d", year, mon2num(month), day)
total[date]++
human[date] = sprintf("%3s %2d, %4d", month, day, year)
}
END {
sort_coprocess = "sort"
for (date in total) {
print date |& sort_coprocess
}
close(sort_coprocess, "to")
print "Date\tCount"
while ((sort_coprocess |& getline date) > 0) {
print human[date] "\t" total[date]
}
close(sort_coprocess)
}
' original.txt
if you are using gawk
awk 'BEGIN{
s="03/05/2009"
m=split(s,date,"/")
t=date[3]" "date[2]" "date[1]" 0 0 0"
print strftime("%b %d",mktime(t))
}'
the above is just an example, as you did not show your actual code and so cannot incorporate it into your code.
Why don't you prepend your awk-date to the original date? This yields a sortable key, but is human readable.
(Note: to sort right, you should make it yyyymmdd)
If needed, cut can remove the prepended column.
Gawk has strftime(). You can also call the date command to format them (man). Linux Forums gives some examples.