Shell script to fetch log (json format) file between date and timestamp - shell

Question

The log file folder structure is \Mainfolder\folder1\year(2020)\month(07)\date(24)*.json.
Ex: \Mainfolder\folder1\2020\07\24\filename.json.
The .json file is getting created every hour, like 00:00:00_00:59:59.json, 01:00:00_01:59:59.json and so on.
I have to search under the .json file with following inputs.
My current inputs are keyword, start date. Currently I'm taking that Date, and keyword and able to get the output in a file.
Current script for your reference:
#!/bin/bash
set +x
DTE=$(date "+%d-%m-%Y-v%H%m%s")
Date=$1 #yyyy/mm/dd
Keyword=$2 #keyword in string
Start_Time=$3 #hh:mm
End_Time=$4 #hh:mm
BKT=bucketpath/mainfolder/
output=$(gsutil cat -h gs://bucketpath/mainfolder/"$Date"/* | egrep "$Keyword")
echo $output >> $"/tmp/folder/logoutput-$DTE"
gsutil cp -r /tmp/folder/logoutput-$DTE gs://bucketpath/mainfolder/
I have to add end date, Start_Time & End_Time and search in the .json file and get the output in a file like above.
I tried to use awk & sed, but i'm unable to get the output.
Could anyone help me on this script please.
Thanks in advance.

Answer 1

I prepared following script to collect the logs between date and timestamp along with keyword. My log file is in .json format.
The reason for posting here is, it might help someone who is looking for similar script.
#!/bin/bash
set +x
DTE=$(date "+%d-%m-%Y-v%H%m%s")
startdate=$1
enddate=$2
start_Time=$3
end_Time=$4
keyword=$5
BKT=storage/folder
i=$start_time
i1=$(sed 's/.\{3\}$//' <<< "$i")
j=$end_time
j1=$(sed 's/.\{3\}$//' <<< "$j")
curr="$startdate"
while true; do
echo "$curr"
[ "$curr" \< "$enddate" ] || break
output=$(gsutil cat -h gs://storage/folder/"$curr"/"$i1:00:00_$j1:59:59*" | sed -n '/"timestamp":"[^"]*T'$i':/,/"timestamp":"[^"]*T'$j':/p' | grep "$keyword")
echo $output >> $"/tmp/folder/mylog-$DTE"
curr=$( date +%Y/%m/%d --date "$curr +1 day" )
done
gsutil cp -r /tmp/folder/mylog-$DTE gs://storage/folder/