I have a list of Git timestamps in the format Mon Jan 1 01:01:01 2013 +0500. I need sort them in the shell somehow and have no clue how to approach this. So far I've created two arrays - one for months and one for days.
Any suggestions?
Thanks.
EDIT: This is not a git log that I'm going through, this is just a bunch of git timestamps that I have pulled out from different repos.
You can use date to convert to a format that's easier to sort, such as epoch. I'll assume you have a file called dates.in, with one date per line.
#!/bin/bash
while read d; do
date -d "$d" +%s
done <dates.in | sort | \
while read d; do
date -d "#$d"
done
Related
I am stuck in one situation where I am having a bunch of files and I need to pick the oldest one on the basis of time present in name only. Not on basis of the timestamp as I am doing SCP from one system to another so timestamp would be same for all the files once SCP runs
I have files like
UAT-2019-03-21-16-31.csv
UAT-2019-03-21-17-01.csv
AIT-2019-03-21-17-01.csv
Here, 2019 represents the year, 03 the month, 21 the day, 16 the hours in 24-hour format and 31 represent the minutes.
I need to pick the UAT-2019-03-21-16-31.csv file from the above files first.
How can I do in shell scripting.
I tried doing ls -1 but it will sort alphabetically, that means AIT-2019-03-21-17-01.csv will be picked first, but I need according to time mentioned in the file name
You can try this
ls -1 | sort -t"-" -k2 -k3 -k4 -k5 -k6 | head -n1
Output :
UAT-2019-03-21-16-31.csv
Curious about alternatives answer as I know that parsing ls output is not ideal.
The best and efficient way to do this would be to convert the filename time stamp to epoch time and find the oldest among them.
You need to write a script that does below in order:
Get all the filename timestamp into a variable.
Convert all filename timestamp to epoch time.
Find the oldest and get the filename.
Command to convert the filename timestamp to epoch time would be
date -d"2019-03-21T17:01" +%s
date -d"YYYY-MM-DDTHH:MM" +%s
You can try these steps in script
Hope so this helps you to start writing the script.
I work with Postfix and Dovecot for SMTP and IMAP. They are on the latest CentOS 7 available versions and the messages are stored in Maildir format.
We have made an agreement with Google and our mailboxes will be transfered to them very soon.
We have this mailserver infrastructure since the nineties. So, some messages have an old "Date" header format, with the year field as "yy". Since Google demands it to be "yyyy", they told me that I need to convert this information on every needed message before the Imap import to their cloud. This follows the RFC2060 and 3501.
This is an university and theses old messages contains research data that should be preserved.
Here is an example:
date: Thu, 20 Apr 17 15:45:15 +0000
should be:
date: Thu, 20 Apr 2017 15:45:15 +0000
I've been looking for script to perform this fix by keeping the header, the date, and only fix the year in every needed file and without changing the file timestamp (some mail clients use this as sorting method). But I haven't found any.
So, is there anyone that can help me?
Thank you.
You cannot modify a file without changing its time stamp; but you can keep the original time stamp and apply it back with touch, as indicated in a separate answer.
Finding the broken Date: headers isn't too hard, either, especially if the messages are sent by a small set of clients which are all uniformly broken in the same way. You can find many, many different violations of the RFCs in the wild, though, so probably perform a test run to extract all Date: headers which aren't in one of the expected formats before you go ahead with modifications.
find Maildir -type f -exec sh -c 'for f; do
sed -n "/^\$/q;/^[Dd][Aa][Tt][Ee]:/p" "$f"; done' _ {} +
The -exec ... + is a GNU extension which mimics xargs in that it will pass as many of the found files as possible as arguments to the process started by -exec.
You can augment the regex after [Dd][Aa][Tt][Ee]: to search for date headers matching a particular erroneous Date: format.
If you can verify that all the erroneous messages are similar to your sample,
sed -i '1,/^$/!b;s/^\([Dd][Aa][Tt][Ee]: [A-Z][a-z][a-z], [ 0-3][0-9] [A-Z][a-z][a-z] \)\([7-9][0-9] \)/\119\2/;s/^\([Dd][Aa][Tt][Ee]: [A-Z][a-z][a-z], [ 0-3][0-9] [A-Z][a-z][a-z] \)\([01][0-9] \)/\120\2/'
might be at least a good start towards fixing the erroneous messages.
Pulling everything together, the final script might look something like
find Maildir -type f -exec sh -c 'for f; do
timestamp=$(stat -c "%y" "$f")
sed -i "1,/^\$/!b;s/^\(Date: [A-Z][a-z][a-z], [ 0-3][0-9] [A-Z][a-z][a-z] \)\([7-9][0-9] \)/\119\2/;s/^\(Date: [A-Z][a-z][a-z], [ 0-3][0-9] [A-Z][a-z][a-z] \)\([01][0-9] \)/\120\2/" "$f"
touch -d "$timestamp" "$f"
done' _ {} +
My prediction is that your final sed script will need to be quite a lot more complex if you need to deal with decades of buggy mail clients from the strongholds of intellectual creativity like Lotus, Yahoo!, and Microsoft. The peskiest are perhaps the ones which have incorrectly been localized - you can probably guess that Märtz is March, but good luck with marraskuu or 十一月...
You can use touch command to restore the timestamp of the file after the modification.
Suppose the name of the file is input_file.
orinal_time=$(stat -c "%y"input_file) #store the orignal timestamp
#do yy->yyyy conversion, this is just example. key here is %Y for conversion
date -d"Thu, 20 Apr 17 15:45:15 +0000" "+%a,%d %b %Y %T %z" >input_file.tmp && mv input_file.tmp input_file
#restore the orignal timestamp
touch -d "$orinal_time" input_file
i try to delete all days of a csv file which not matched last days. But I find not the right solution.
date,price
2018-07-02,162.17
2018-06-29,161.94
2018-06-28,162.22
2018-06-27,162.32
2018-06-12,163.01
2018-06-11,163.53
2018-05-31,164.87
2018-05-30,165.59
2018-05-29,165.42
2018-05-25,165.96
2018-05-02,164.94
2018-04-30,166.16
2018-04-27,166.69
The output I want become
date,price
2018-06-29,161.94
2018-05-31,164.87
2018-04-30,166.16
I try it with cut + grep
cut -d, -f1 file.csv | grep -E "28|29|30"
Work but bring nothing when combine -f1,2.
I find csvkit which seem to me the right tool, but I find not the solution for multiple grep.
csvgrep -c 1 -m 30 file.csv
Bring me the right result but how can combine multiple search option? I try -m 28,29,30 and -m 28 -m 29 -m 30 all work not. Best it work with last day of every month.
Maybe one have here a idea.
Thank you and nice Sunday
Silvio
You want to get all records of the LAST day of the month. But months vary in length (28-29-30-31).
I don't see why you used cut to extract the first field (the date part), because the data in the second field does not look like dates at all (xx-xx).
I suggest to use grep directly to display the lines that matches the following pattern mm-dd; where mm is the month number, and dd is the last day of the month.
This command should do the trick:
grep -E "01-31|02-(28|29)|03-31|04-30|05-31|06-30|07-31|08-30|09-31|10-30|11-31|12-30" file.csv
This command will give the following output:
2018-05-31,164.87
2018-04-30,166.16
I want to pull some information from a website from past 4 years and each file is date based, like http://ransompull.com/pullme/2013-04-06/example.2013-04-06.txt
and it is the starting file and it ends today, so i want to pull all the txt files from last 4 years.
What I tried:
DATE=`date +%Y`
MONTH='01'
DAY='1'
for i in range(1,31);
for j in range(01,12):
do wget http://ransompull.com/pullme/$DATE$i/example.$DATE$i.txt;
done
done
But this seems to wrong as iterating over month and date is not feasible as it is not giving desired output.Any suggestions on how to pull all data from
http://ransompull.com/pullme/2013-04-06/example.2013-04-06.txt
to
http://ransompull.com/pullme/2017-08-10/example.2017-08-10.txt
Instead of counting years, months and days,
you could just count days relative to the start date.
If you have the GNU implementation of the date command,
you can use it to compute the relative date, for example:
date +%F -d '2013-04-06 + 1000 days'
This outputs 2016-01-01.
You can create a loop, generating dates by incrementing the number of days from start, until you reach the end:
start=2013-04-06
end=2017-08-10
date=$start
days=0
while [ "$date" != "$end" ]; do
date=$(date +%F -d "$start + $days days")
wget http://ransompull.com/pullme/$date/example.$date.txt
((days++))
done
try this:
$startdate=get-date 2017-08-11
$enddate=$startdate.AddYears(-4)
0..($startdate - $enddate).Days | %{wget ("http://ransompull.com/pullme/{0:yyyy-MM-dd}/example.{0:yyyy-MM-dd}.txt" -f $startdate.AddDays(-$_))}
I'm trying to put together a bash/sh script that gets the UTC time of the last commit from a svn repo (the other VCSs are easy)
I understand that i can just do svn propget --revprop -rHEAD svn:date and get it rather easily, but there is no guarantee that the svn checkout will be online, so I'd prefer an offline version, if possible.
Maybe something to do with getting the UTC time from svn info? (by screwing with the timezones)
Summary: How can i get the UTC time of a svn commit, while not having access to the server?
Thanks
You can use svn log -r COMMITTED and extract date info from it. This is valid for offline copies.
svn log -r COMMITTED | sed -nE 's/^r.*\| ([0-9]{4}-[0-9]{2}-[0-9]{2} \S+ \S+).*/\1/p' | xargs -i -- date -ud '{}' '+%s'
The -u option makes date show UTC time instead.
Actually we don't need to use xargs:
date -ud "$(exec svn log -r COMMITTED | sed -nE 's/^r.*\| ([0-9]{4}-[0-9]{2}-[0-9]{2} \S+ \S+).*/\1/p')" '+%s'
UPDATE: I got the wrong command. The command above won't work offline. Here's the right one:
date -ud "$(svn info | sed -nE 's/^Last Changed Date: (\S+ \S+ \S+).*/\1/p')" '+%s'
I'm silly. As soon as i actually realised i just need to convert one timezone to another, it was obvious:
date -u +[format] -d $(svn info | <some grepping and cutting here>)
In my case, this is:
date -u +"%Y%m%d-%H%M" -d "$(svn info | grep 'Date' | cut -d' ' -f4-6)"
Of course, my solution probably isn't optimal, and if someone knows a better way, that'd be much appreciated :)
It turns out that the xml output of "svn info" has a zulu timestamp:
$ svn info --xml | grep date
<date>2015-04-30T15:38:49.371762Z</date>
So your bash command might be:
svn info --xml | grep -oP '(?<=<date>).*?(?=</date>)'
I just stumbled on this post. Ended up figuring out that svn uses env variable TZ, so for example:
TZ= svn log
will log dates in UTC