I would like to generate an iso-style time info (YYYY-MM-DDTmm:hh:ss) on
a bsd-ish system with an older ls command (man states it's a
BSD-ls) and no --time-style option. The best I get is
$ ls -lT log.out
-rw-r--r-- 1 admin4 staff 49152 Jul 17 09:38:38 2018 log.out
Is there a simple way to transform this time information within a
bash-script into iso-style?
$ timestamp="$(ls -lT log.out | iso-timestamp-gen)"
Your help will be highly appreciated
first, cut out date part:
DATE=`echo - | cut -f9-12 -d' '`;
then convert it to timestamp using date:
echo `date --date="$DATE" +"%s"`;
the iso-timestamp-gen file:
#!/bin/bash
DATE=`echo - | cut -f9-12 -d' '`;
echo `date --date="$DATE" +"%s"`;
To solve our problem
timestamp="$(ls -lT log.out | iso-timestamp-gen)"
#Hamidreza pointed us into the direktion of the date command: One
of our engineers came up with this slender inline solution:
timestamp=$(date -j -f '%b %e %T %Y' \
"$(ls -lT $f | awk '{ print $6, $7, $8, $9 }')" \
+'%F_%T')
where the variable f contains the name of the file to be considered.
In line 2 awk extracts the date-part (fields 6 upto 9) from the output of ls -lT.
Together with this "DateToConvert" date is fed a matching
input format "%b %e %T %Y"
and the requested output format "%F_%T".
We looked up the conversion specifications in
strftime(3).
Up to now neither of us was aware that you could use date not only for
reading and setting the system clock but also for converting dates from
one format to another format:
date -j -f "${InputFormat}" "${DateToConvert}" +"${OutputFormat}"
With some embarassment we confessed this to each other and
decided to make good use of the new insight.
A couple of hours later we added the following new helper to our toolbox.
It extracts the date from the output of ls -lT and converts it to
other formats. So far we included only the iso-format and a format
which is handy when using the touch(1) command. This list may be expanded in the future.
#!/bin/bash
#
# conv-date-of-llT.sh
# convert date output of ls -lT ($OSTYPE == darwin14)
# - to iso-format YYYY-MM-DD_hh:mm:ss or
# - to "touch" format: [[CC]YY]MMDDhhmm[.SS]
#
# hppo, 2018-07-20
# strip $0 of path and ext
Arg0="${0##*/}" # peel off path
Arg0="${Arg0%%.*}" # peel off ext
USAGE="${Arg0} [ -iso | -touch ]"
# select output format
case "fmt$1" in
fmt|fmt-iso|fmtiso) # YYYY-MM-DD_hh:mm:ss
OutputFormat="%F_%T" ;;
fmt-touch|fmttouch) # YYYYMMDDhhmm
OutputFormat="%Y%m%d%H%M" ;;
*)
1>&2 echo -e "${Arg0}: no such output format '$1'\nUsage: ${USAGE}"; exit 1 ;;
esac
# input: output of "ls -lT file" on darwin-14
# -rwxr-xr-x 1 admin4 staff 387 Jul 17 01:38:24 2018 file
# 1 2 3 4 5 6 7 8 9 10
#
# Field 6 - 9: month(%b) day(%e) time(%T) year(%Y), cf strftime(3)
InputFormat="%b %e %T %Y"
# assume stdin is fed by output of ls -lT
DateToConvert="$(awk '{ print $6, $7, $8, $9 }')"
date -j -f "${InputFormat}" "${DateToConvert}" +"${OutputFormat}"
Related
I have my command below and I want to have the result in the same line with delimeters. My command:
Array=("GET" "POST" "OPTIONS" "HEAD")
echo $(date "+%Y-%m-%d %H:%M")
for i in "${Array[#]}"
do
cat /home/log/myfile_log | grep "$(date "+%d/%b/%Y:%H")"| awk -v last5=$(date --date="-5 min" "+%M") -F':' '$3>=last5 && $3<last5+5{print}' | egrep -a "$i" | wc -l
done
Results is:
2019-01-01 13:27
1651
5760
0
0
I want to have the result below:
2019-01-01 13:27,1651,5760,0,0
It looks (to me) like the overall objective is to scan /home/log/myfile.log for entries that have occurred within the last 5 minutes and which match one of the 4 entries in ${Array[#]}, keeping count of the matches along the way and finally printing the current date and the counts to a single line of output.
I've opted for a complete rewrite that uses awk's abilities of pattern matching, keeping counts and generating a single line of output:
date1=$(date "+%Y-%m-%d %H:%M") # current date
date5=$(date --date="-5 min" "+%M") # date from 5 minutes ago
awk -v d1="${date1}" -v d5="${date5}" -F":" '
BEGIN { keep=0 # init some variables
g=0
p=0
o=0
h=0
}
$3>=d5 && $3<d5+5 { keep=1 } # do we keep processing this line?
!keep { next } # if not then skip to next line
/GET/ { g++ } # increment our counters
/POST/ { p++ }
/OPTIONS/ { o++ }
/HEAD/ { h++ }
{ keep=0 } # reset keep flag for next line
# print results to single line of output
END { printf "%s,%s,%s,%s,%s\n", d1, g, p, o, h }
' <(grep "$(date '+%d/%b/%Y:%H')" /home/log/myfile_log)
NOTE: The OP may need to revisit the <(grep "$(date ...)" /home/log/myfile.log) to handle timestamp periods that span hours, days, months and years, eg, 14:59 - 16:04, 12/31/2019 23:59 - 01/01/2020 00:04, etc.
Yeah, it's a bit verbose but a bit easier to understand; OP can rewrite/reduce as sees fit.
I have a string from conf file (lets call for example date1):
#!/bin/bash
# it is example
date1="201605250925"
datenow="$(date +%Y%m%d%H%M -d "+1hour")"
date2=$(date +%Y%m%d%H%M -d "$date1 + 1hour")
# NOT WORK?
echo "$date1 --> $date2"
# WORK!
echo "$date1 --> $datenow"
I need to add 1 hour. But getting error like this:
date: invalid date `201605250925 + 1hour'
But its work for datenow.
How can I user addhour for custom date format from string?
You need a format that meets the command date expectations, something like:
2016-05-25 09:25
The space denote the start of time and the time format is HH:MM.
That comes from then international ISO 8601, but using an space instead of a T.
If the format is fixed, we can use bash internal capacities (no external command except date used) to change it like this:
#!/bin/bash
d1="201605250925"
dc="${d1:0:8} ${d1:8:2}:${d1:10:2}+0"
d2=$(date +'%Y%m%d %H:%M' -ud "$dc + 1 hour" )
echo "$d2"
Or POSIXly (dash) with no call needed to sed, awk or cut (faster):
#!/bin/dash
d1="201605250925"
dt=${d1##????????}
dc="${d1%%"$dt"} ${dt%%??}:${dt##??}+0"
d2=$(date -ud "$dc + 1 hour" +'%Y%m%d %H:%M')
echo "$d2"
20160525 10:25
The inclusion of a +0 after the time in dc: 20160525 09:25+0
will ensure that date will interpret the time as with offset 0 (UTC).
The use of the option -u to date will ensure that the value read in UTC also change in UTC, avoiding any Daylight correction or local time change.
If you want to keep the same format as your input string you could use cut or sed to split it:
cut
d1="201605250925"
ds=$(echo $d1 | cut --output-delimiter=" " -c 1-8,9-12); \
d2=$(date '+%Y%m%d%H%M' -ud "$ds + 1hour")
sed
d1="201605250925"
ds=$(echo $d1 | sed 's/\(.*\)\(....\)$/\1 \2/g'); \
d2=$(date '+%Y%m%d%H%M' -ud "$ds + 1hour")
This way the date utility can make sense of the date and time.
Result:
$ echo $d2
201605251025
Trying to pull the last 5 minutes of logs with (grep matches)
so i do a tac syslog.log | sed / date -d "5 minutes ago"
every line on the log shows this format
Jun 14 14:03:58
Jul 3 08:04:35
so i really want to get the check of data from
Jul 4 08:12
Jul 4 08:17
i tried this method but KINDA works (though its still going through every day from this that 08:12: through 08:17: fits in)
e=""
for (( i = 5; i >= 0; i-- ))
do
e='-e /'`date +\%R -d "-$i min"`':/p '$e;
done
tac /var/log/syslog.log | sed -n $e
e=""
for (( i = 5; i >= 0; i-- ))
do
if [[ -z $e ]]
then e=`date +\%R -d "-$i min"`
else e=$e'\|'`date +\%R -d "-$i min"`
fi
done
re=' \('$e'\):'
tac /var/log/syslog.log | sed -n -e "/$re/p" -e "/$re/!q"
This creates a single regular expression listing all the times from the last 5 minutes, connected with \|. It prints the lines that matches them. Then it uses the ! modifier to quit on the first line that doesn't match the RE.
If you know the format of the dates then why not do:
tac syslog.log | awk '/Jul 4 08:17/,/Jul 4 08:12/ { print } /Jul 4 08:11/ {exit}'
/ .. /,/ .. / is regex range. It will print everything in this range. So as soon as you see /Jul 4 08:11/ on your line that would mean your 5 minutes window has been captured, you exit perusing the file.
So it didnt really work for the above method But i think i got it to work
if i see this i added a RANGE for the {exit}
awk '/'"$dtnow"'/,/'"$dt6min"'/ { print } /'"$dt7min"'/,/'"$dt11min"'/ {exit}'
Seems to work im testing it again
OK Finally looks like it really works this time (where it exits after the hour using SED instead of awk finally got it to work running through some tests.
tac /var/log/syslog.log | sed -e "$( date -d '-1 hour -6 minutes' '+/^%b %e %H:/q;'
date -d '-1 day -6 minutes' '+/^%b %e /q;'
date -d '-1 month -6 minutes' '+/^%b /q;'
for ((o=0;o<=5;o++)) do date -d "-$o minutes" '+/^%b %e %R:/p;'; done ; echo d)"
It works if log entries begins from "May 14 11:41". Variable LASTMINUTES is used to set the last n minutes in the log:
cat log | awk 'BEGIN{ LASTMINUTES=30; for (L=0;L<=LASTMINUTES;L++) TAB[strftime("%b %d %H:%M",systime()-L*60)] } { if (substr($0,0,12) in TAB) print $0 }'
To run the above script you need gawk which can be installed by:
apt-get install gawk
or
yum install gawk
I'm trying to store the files listing into an array and then loop through the array again.
Below is what I get when I run ls -ls command from the console.
total 40
36 -rwxrwxr-x 1 amit amit 36720 2012-03-31 12:19 1.txt
4 -rwxrwxr-x 1 amit amit 1318 2012-03-31 14:49 2.txt
The following bash script I've written to store the above data into a bash array.
i=0
ls -ls | while read line
do
array[ $i ]="$line"
(( i++ ))
done
But when I echo $array, I get nothing!
FYI, I run the script this way: ./bashscript.sh
I'd use
files=(*)
And then if you need data about the file, such as size, use the stat command on each file.
Try with:
#! /bin/bash
i=0
while read line
do
array[ $i ]="$line"
(( i++ ))
done < <(ls -ls)
echo ${array[1]}
In your version, the while runs in a subshell, the environment variables you modify in the loop are not visible outside it.
(Do keep in mind that parsing the output of ls is generally not a good idea at all.)
Here's a variant that lets you use a regex pattern for initial filtering, change the regex to be get the filtering you desire.
files=($(find -E . -type f -regex "^.*$"))
for item in ${files[*]}
do
printf " %s\n" $item
done
This might work for you:
OIFS=$IFS; IFS=$'\n'; array=($(ls -ls)); IFS=$OIFS; echo "${array[1]}"
Running any shell command inside $(...) will help to store the output in a variable. So using that we can convert the files to array with IFS.
IFS=' ' read -r -a array <<< $(ls /path/to/dir)
You may be tempted to use (*) but what if a directory contains the * character? It's very difficult to handle special characters in filenames correctly.
You can use ls -ls. However, it fails to handle newline characters.
# Store la -ls as an array
readarray -t files <<< $(ls -ls)
for (( i=1; i<${#files[#]}; i++ ))
{
# Convert current line to an array
line=(${files[$i]})
# Get the filename, joining it together any spaces
fileName=${line[#]:9}
echo $fileName
}
If all you want is the file name, then just use ls:
for fileName in $(ls); do
echo $fileName
done
See this article or this this post for more information about some of the difficulties of dealing with special characters in file names.
My two cents
The asker wanted to parse output of ls -ls
Below is what I get when I run ls -ls command from the console.
total 40
36 -rwxrwxr-x 1 amit amit 36720 2012-03-31 12:19 1.txt
4 -rwxrwxr-x 1 amit amit 1318 2012-03-31 14:49 2.txt
But there are few answer addressing this parsing operation.
ls's output
Before trying to parse something, we have to ensure command output is consistant, stable and easy to parse as possible
In order to ensure output wont be altered by some alias you may prefer to specify full path of command: /bin/ls.
Avoid variations of output due to locales, prefix your command by LANG=C LC_ALL=C
Use --time-style command switch to use UNIX EPOCH more easier to parse time infos.
Use -b switch for holding special characters
So we will prefer
LANG=C LC_ALL=C /bin/ls -lsb --time-style='+%s.%N'
to just
ls -ls
Full bash sample
#!/bin/bash
declare -a bydate=() bysize=() byname=() details=()
declare -i cnt=0 vtotblk=0 totblk
{
read -r _ totblk # ignore 1st line
while read -r blk perm lnk usr grp sze date file;do
byname[cnt]="${file//\\ / }"
details[cnt]="$blk $perm $lnk $usr $grp $sze $date"
bysize[sze]+="$cnt "
bydate[${date/.}]+="$cnt "
cnt+=1 vtotblk+=blk
done
} < <(LANG=C LC_ALL=C /bin/ls -lsb --time-style='+%s.%N')
From there, you could easily sort by dates, sizes of names (sorted by ls command).
echo "Path '$PWD': Total: $vtotblk, sorted by dates"
for dte in ${!bydate[#]};do
printf -v msec %.3f .${dte: -9}
for idx in ${bydate[dte]};do
read -r blk perm lnk usr grp sze date <<<"${details[idx]}"
printf ' %11d %(%a %d %b %T)T%s %s\n' \
$sze "${date%.*}" ${msec#0} "${byname[idx]}"
done
done
echo "Path '$PWD': Total: $vtotblk, sorted by sizes"
for sze in ${!bysize[#]};do
for idx in ${bysize[sze]};do
read -r blk perm lnk usr grp sze date <<<"${details[idx]}"
printf -v msec %.3f .${date#*.}
printf ' %11d %(%a %d %b %T)T%s %s\n' \
$sze "${date%.*}" ${msec#0} "${byname[idx]}"
done
done
echo "Path '$PWD': Total: $vtotblk, sorted by names"
for((idx=0;idx<cnt;idx++));{
read -r blk perm lnk usr grp sze date <<<"${details[idx]}"
printf -v msec %.3f .${date#*.}
printf ' %11d %(%a %d %b %T)T%s %s\n' \
$sze "${date%.*}" ${msec#0} "${byname[idx]}"
}
( Accessory, you could check if total block printed by ls match total block by lines:
(( vtotblk == totblk )) ||
echo "WARN: Total blocks: $totblk != Block count: $vtotblk" >&2
Of course, this could be inserted before first echo "Path...;)
Here is an output sample. (Note: there is a filename with a newline)
Path '/tmp/so': Total: 16, sorted by dates
0 Sun 04 Sep 10:09:18.221 2.txt
247 Mon 05 Sep 09:11:50.322 Filename with\nsp\303\251cials characters
13 Mon 05 Sep 10:12:24.859 1.txt
1313 Mon 05 Sep 11:01:00.855 parseLs.00
1913 Thu 08 Sep 08:20:20.836 parseLs
Path '/tmp/so': Total: 16, sorted by sizes
0 Sun 04 Sep 10:09:18.221 2.txt
13 Mon 05 Sep 10:12:24.859 1.txt
247 Mon 05 Sep 09:11:50.322 Filename with\nsp\303\251cials characters
1313 Mon 05 Sep 11:01:00.855 parseLs.00
1913 Thu 08 Sep 08:20:20.836 parseLs
Path '/tmp/so': Total: 16, sorted by names
13 Mon 05 Sep 10:12:24.859 1.txt
0 Sun 04 Sep 10:09:18.221 2.txt
247 Mon 05 Sep 09:11:50.322 Filename with\nsp\303\251cials characters
1913 Thu 08 Sep 08:20:20.836 parseLs
1313 Mon 05 Sep 11:01:00.855 parseLs.00
And if you want to format characters (with care: there could be some issues, if you don't know who create content of path). But if folder is your, you could:
echo "Path '$PWD': Total: $vtotblk, sorted by dates, with special chars"
printf -v spaces '%*s' 37 ''
for dte in ${!bydate[#]};do
printf -v msec %.3f .${dte: -9}
for idx in ${bydate[dte]};do
read -r blk perm lnk usr grp sze date <<<"${details[idx]}"
printf ' %11d %(%a %d %b %T)T%s %b\n' $sze \
"${date%.*}" ${msec#0} "${byname[idx]//\\n/\\n$spaces}"
done
done
Could output:
Path '/tmp/so': Total: 16, sorted by dates, with special chars
0 Sun 04 Sep 10:09:18.221 2.txt
247 Mon 05 Sep 09:11:50.322 Filename with
spécials characters
13 Mon 05 Sep 10:12:24.859 1.txt
1313 Mon 05 Sep 11:01:00.855 parseLs.00
1913 Thu 08 Sep 08:20:20.836 parseLs
Isn't these 2 code lines, either using scandir or including the dir pull in the declaration line, supposed to work?
src_dir="/3T/data/MySQL";
# src_ray=scandir($src_dir);
declare -a src_ray ${src_dir/*.sql}
printf ( $src_ray );
In the conversation over at https://stackoverflow.com/a/9954738/11944425
the behavior can be wrapped into a convenience function which applies some action to entries of the directory as string values.
#!/bin/bash
iterfiles() {
i=0
while read filename
do
files[ $i ]="$filename"
(( i++ ))
done < <( ls -l )
for (( idx=0 ; idx<${#files[#]} ; idx++ ))
do
$# "${files[$idx]}" &
wait $!
done
}
where $# is the complete glob of arguments passed to the function! This lets the function have the utility to take an arbitrary command as a partial function of sorts to operate on the filename:
iterfiles head -n 1 | tee -a header_check.out
When a script needs to iterate over files, returning an array of them is not possible. The workaround is to define the array outside of the function scope (and possibly unset it later) — modifying it inside the function's scope. Then, after the function is called by a script, the array variable becomes available. For instance, the mutation on files demonstrates how this could be done.
declare -a files # or just `files= ` (nothing)
iterfiles() {
# ...
files=...
}
Extending the conversation above, #Jean-BaptistePoittevin pointed out a valuable detail.
#!/bin/bash
# Adding a section to unset certain variable names that
# may already be active in the shell.
unset i
unset files
unset omit
i=0
omit='^([\n]+)$'
while read file
do
files[ $i ]="$file"
(( i++ ))
done < <(ls -l | grep -Pov ${omit} )
Note: This can be tested using echo ${files[0]} or for entry in ${files[#]}; do ... ; done
Often times, the circumstance could require an absolute path in double quotes, where the file (or ancestor directories) have spaces or unusual characters in the name. find is one answer here. The simplest usage might look like the above one, except done < <(ls -l ... ) is replaced with:
done < <(find /path/to/directory ! -path /path/to/directory -type d)
Its convenient when you need absolute paths in double quotes as an iterable collection to use a recipe like the one below. When export is not used, the shell does not update the environment namespace to include it in the find subshell:
#!/bin/bash
export DIRECTORY="$PWD" # For example
declare -a files
i=0
while read filename; do
files[ $i ]="$filename"
done < <(find $DIRECTORY ! -path $DIRECTORY -type d)
for (( idx=0; idx<${#files[#]}; idx++ )); do
# Make a templated string for macro script generation
quoted_path="\"${files[$idx]}\""
if [[ "$(echo $quoted_path | grep some_substring | wc -c)" != "0" ]]; then
echo "mv $quoted_path /some/other/watched/folder/" >> run_nightly.sh
fi
done
Upon running this, ./run_nightly.sh will be populated with bulk commands to move a quoted path to /some/other/watched/folder/. This kind of scripting pattern will make it possible to supercharge your scripts.
simply you can use this below for loop (do not forget to quote to handle filenames with spaces)
declare -a arr
arr=()
for file in "*.txt"
do
arr=(${arr[*]} "$file")
done
Run
for file in ${arr[*]}
do
echo "<$file>"
done
to test.
I need to create a bash shell script starting with a day and then loop through each subsequent day formatting that output as %Y_%m_d
I figure I can submit a start day and then another param for the number of days.
My issue/question is how to set a DATE (that is not now) and then add a day.
so my input would be 2010_04_01 6
my output would be
2010_04_01
2010_04_02
2010_04_03
2010_04_04
2010_04_05
2010_04_06
[radical#home ~]$ cat a.sh
#!/bin/bash
START=`echo $1 | tr -d _`;
for (( c=0; c<$2; c++ ))
do
echo -n "`date --date="$START +$c day" +%Y_%m_%d` ";
done
Now if you call this script with your params it will return what you wanted:
[radical#home ~]$ ./a.sh 2010_04_01 6
2010_04_01 2010_04_02 2010_04_03 2010_04_04 2010_04_05 2010_04_06
Very basic bash script should be able to do this:
#!/bin/bash
start_date=20100501
num_days=5
for i in `seq 1 $num_days`
do
date=`date +%Y/%m/%d -d "${start_date}-${i} days"`
echo $date # Use this however you want!
done
Output:
2010/04/30
2010/04/29
2010/04/28
2010/04/27
2010/04/26
Note: NONE of the solutions here will work with OS X. You would need, for example, something like this:
date -v-1d +%Y%m%d
That would print out yesterday for you. Or with underscores of course:
date -v-1d +%Y_%m_%d
So taking that into account, you should be able to adjust some of the loops in these examples with this command instead. -v option will easily allow you to add or subtract days, minutes, seconds, years, months, etc. -v+24d would add 24 days. and so on.
#!/bin/bash
inputdate="${1//_/-}" # change underscores into dashes
for ((i=0; i<$2; i++))
do
date -d "$inputdate + $i day" "+%Y_%m_%d"
done
Very basic bash script should be able to do this.
Script:
#!/bin/bash
start_date=20100501
num_days=5
for i in seq 1 $num_days
do
date=date +%Y/%m/%d -d "${start_date}-${i} days"
echo $date # Use this however you want!
done
Output:
2010/04/30
2010/04/29
2010/04/28
2010/04/27
2010/04/26
You can also use cal, for example
YYYY=2014; MM=02; for d in $(cal $MM $YYYY | grep "^ *[0-9]"); do DD=$(printf "%02d" $d); echo $YYYY$MM$DD; done
(originally posted here on my commandlinefu account)
You can pass a date via command line option -d to GNU date handling multiple input formats:
http://www.gnu.org/software/coreutils/manual/coreutils.html#Date-input-formats
Pass starting date as command line argument or use current date:
underscore_date=${1:-$(date +%y_%m_%d)}
date=${underscore_date//_/-}
for days in $(seq 0 6);do
date -d "$date + $days days" +%Y_%m_%d;
done
you can use gawk
#!/bin/bash
DATE=$1
num=$2
awk -vd="$DATE" -vn="$num" 'BEGIN{
m=split(d,D,"_")
t=mktime(D[1]" "D[2]" "D[3]" 00 00 00")
print d
for(i=1;i<=n;i++){
t+=86400
print strftime("%Y_%m_%d",t)
}
}'
output
$ ./shell.sh 2010_04_01 6
2010_04_01
2010_04_02
2010_04_03
2010_04_04
2010_04_05
2010_04_06
2010_04_07