So a website that I'm using has a websocket and they provide the broadcast time in the following manner:
"broadcasted_at":1574325570.71308
What is this time format and how do they generate it?
Unix epoch time ... the number of seconds that have elapsed since the Unix epoch, that is the time 00:00:00 UTC on 1 January 1970
now : 1574327074 : Thu Nov 21 03:04:34 2019
start of day : 1574316000 : Thu Nov 21 00:00:00 2019
1574325570 : 1574325570 : Thu Nov 21 02:39:30 2019
convert online : https://www.epochconverter.com/
... or download code (to build) to have command line program to perform the conversion https://github.com/darrenjs/c_dev_utils
I'm guessing the fractional part is the number of microseconds within the current second.
… and how do they generate it?
I don’t know, of course, what language or libraries your website is using. So this is just an example. To generate a value like 1574325570.71308 in Java:
Instant now = Instant.now();
double epochSeconds = now.getEpochSecond()
+ (double) now.getNano() / (double) TimeUnit.SECONDS.toNanos(1);
String result = String.format(Locale.ROOT, "%f", epochSeconds);
System.out.println("result: " + result);
When I ran this snippet just now (2019-12-15T11:18:01.562699Z), the output was:
result: 1576408681.562699
If you want exactly 5 decimals always another way is to use a DateTimeFormatter:
DateTimeFormatter formatter = new DateTimeFormatterBuilder()
.appendValue(ChronoField.INSTANT_SECONDS)
.appendPattern(".SSSSS")
.toFormatter();
String result = formatter.format(now);
result: 1576408681.56269
Related
I have an application that runs the old version of the spring application. The application has the function to create date objects using Date.parse as follows
Date getCstTimeZoneDateNow() {
String dateFormat = "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"
def zonedDateString = new Date().format(dateFormat, TimeZone.getTimeZone('CST'))
Date date = Date.parse(dateFormat, zonedDateString)
return date // Tue Oct 18 20:36:12 EDT 2022 (in Date)
}
However, the code above is deprecated. I need to produce the same result.
I read other posts and it seems like Calender or SimpleDateFormatter is preferred.
And I thought SimpleDateFormatter has more capabilities.
This post helped me understand more about what is going on in the following code
SimpleDateFormat parse loses timezone
Date getCstTimeZoneDateNow() {
Date now = new Date()
String pattern = "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"
SimpleDateFormat sdf = new SimpleDateFormat()
sdf.setTimeZone(TimeZone.getTimeZone('CST'))
// cstDateTime prints times in cst
String cstDateTime = sdf.format(now) // 2022-10-18T20:36:12.088Z (in String)
// JVM current time
Date date = sdf.parse(cstDateTime) // Tue Oct 18 21:36:12 EDT 2022 (in Date)
return date
}
Here my goal is to return the date object that is in the format of Tue Oct 18 20:36:12 EDT 2022
The format is good. However, like the post says, when I do sdf.parse(), it prints in JVM time.
This means, the format is good but the time zone is off.
How can I get the exact same result as before?
It does not have to use SimpleDateFormatter. Could be anything.
Thank you so much for reading and for your time.
Perhaps the important thing is, that the Date is always neutral to the timezone. Given example shows what is to be expected to work from the Java specs:
def format = new SimpleDateFormat()
format.setTimeZone(TimeZone.getTimeZone("CST"))
println new Date()
def date = format.parse(format.format(new Date()))
printf "parsed to %s%n", date
printf "formatted to %s (%s)%n", format.format(date), format.getTimeZone().getDisplayName()
In the output, notice when using the Format and when the toString(), a different time is shown accordingly, which is perfectly fine, since first we format and then parse again in the same format, thus the same time-zone. Later, we use the Date.toString() to output the date, this time using the system default time-zone which is always used when Date.toString() is called. In the output, the time-zone shift is reflected:
Thu Oct 20 09:22:58 EDT 2022
parsed to Thu Oct 20 09:22:00 EDT 2022
formatted to 10/20/22 8:22 AM (Central Standard Time)
I need to move away mails older than given time - let it be 24h = 86400s. I use old good procmail for multiple other purposes on that machine, so I wanted to use is as well for this purpose. It also behaves well under the load (~1 000 000 small automated messages per day).
It took me a while to get to this ugly solution (excerpt from bigger procmailrc file):
Grab Date: field using formail
Grab current date in UNIX format (seconds)
bash convert the mail date to unix format
compare values using bash
return result to procmail using exit code.
Together:
MAILDATE_RFC=`formail -zxDate:`
DATE_UNIX=`date "+%s"`
:0
* ? MAILDATE_UNIX=`date -d "$MAILDATE_RFC" "+%s"` ; if ( (( ($DATE_UNIX-$MAILDATE_UNIX) > 86400)) ) then exit 0; else exit 1; fi
! account_for_outdated_mails
In this case I need to use the "Date:" field, as this contains the local time at which the mail was generated (it can take multiple days to get to my machine). We are 100% sure that "Date:" field exists and contains RFC-style date (those are automated messages in separated mail network).
My solution looks pretty ugly:
Getting the comparison result from bash using exit codes looks pretty bad. Might be inefficient as well.
I would like to calculate the MAILDATE_RFC still in procmail but it seems I cannot use any variable as the argument to generate another variable:
MAILDATE_UNIX=`date -d "$MAILDATE_RFC" "+%s"`
does not work.
The only optimization I am aware of would be to push the whole process of getting MAILDATE_RFC, MAILDATE_UNIX and DATE_UNIX processed in bash script and doing it in one bash session instead of 3.
My question: Is there a better way to do it? Maybe more efficient?
What you say doesn't work actually does. Here's a quick demo.
testing.rc:
DEFAULT=/dev/null
SHELL=/bin/sh
VERBOSE=yes
MAILDATE_RFC=`formail -zxDate:`
MAILDATE_UNIX=`date -d "$MAILDATE_RFC" "+%s"`
NOW=`date +%s`
:0
* 86400^0 ^
* $ -$NOW^0 ^
* $ $MAILDATE_UNIX^0 ^
{ LOG="score: $=
" }
Test run, in a fresh Ubuntu 20.04 Docker image:
tripleee#bash$ procmail -m testing.rc <<\:
> Subject: demo message
> Date: Fri, 10 Jun 2022 06:20:36 +0000
>
> Try me
> :
procmail: [263] Fri Jun 10 06:21:23 2022
procmail: Executing "formail,-zxDate:"
procmail: [263] Fri Jun 10 06:21:23 2022
procmail: Assigning "MAILDATE_RFC=Fri, 10 Jun 2022 06:20:36 +0000"
procmail: Executing "date,-d,Fri, 10 Jun 2022 06:20:36 +0000,+%s"
procmail: Assigning "MAILDATE_UNIX=1654842036"
procmail: Executing "date,+%s"
procmail: Assigning "NOW=1654842083"
procmail: Score: 86400 86400 "^"
procmail: Score: -1654842083 -1654755683 "^"
procmail: Score: 1654842036 86353 "^"
procmail: Assigning "LOG=score: 86353
"
score: 86353
procmail: Assigning "LASTFOLDER=/dev/null"
procmail: Opening "/dev/null"
Folder: /dev/null 68
This also demonstrates how to use scoring to do the calculation. It's perhaps somewhat intimidating, but saves an external process, and so should be more efficient than doing the calculation in Bash.
In some more detail, 123^0 regex says to add 123 to the score just once if the message matches the regex regex (in the recipe above, we use the regex ^ which of course always matches; every message contains a beginning. You could change the 0 to e.g. 1 to say to add for every match, or etc - see the procmailsc man page for proper documentation). The $ modifier says to expand any variables in the recipe itself.
If you are not using GNU date, you don't have date -d; in that case, probably refer to your platform's man page for how to calculate a date stamp for an arbitrary date. How to convert date string to epoch timestamp with the OS X BSD `date` command? has a discussion for MacOS, which should also work for any other *BSD platform.
If you really wanted to make this more efficient, and can be sure that the Date: header really always uses the RFC-mandated format, you could even parse the date in Procmail. Something like
:0
* ^Date: [A-Z][a-z][a-z], \/[ 0-9][0-9] [A-Z][a-z][a-z] [0-9][0-9][0-9][0-9]
{
date=$MATCH
:0
* date ?? ^\/[ 0-9][0-9]
{ dd=$MATCH }
:0
* date ?? ^[ 0-9][0-9] \/[A-Z][a-z][a-z]
{ mon=$MATCH }
* date ?? [A-Z][a-z][a-z] \/[0-9][0-9][0-9][0-9]
{ yyyy=$MATCH }
:0
* mon ?? 1^0 ^Jan
* mon ?? 2^0 ^Feb
* mon ?? 3^0 ^Mar
* mon ?? 4^0 ^Apr
* mon ?? 5^0 ^May
* mon ?? 6^0 ^Jun
* mon ?? 7^0 ^Jul
* mon ?? 8^0 ^Aug
* mon ?? 9^0 ^Sep
* mon ?? 10^0 ^Oct
* mon ?? 11^0 ^Nov
* mon ?? 12^0 ^Dec
{ mm=$= }
}
The \/ token in a regex says to save the matched text after it into the special variable MATCH. We then copy that variable to date and perform additional matching to extract its parts.
Performing the necessary arithmetic to convert this into seconds since January 1, 1970 should be doable at this point, I hope. If you need complete per-day accuracy, you would also need to extract the time and the time zone and adjust to the correct day if it's not in your preferred time zone, or perhaps UTC (that would be +0000 at the very end); but this is just a sketch, anyway, because I think I have a better idea altogether.
Namely, save the messages to the correct folder as they arrive, then just forward or discard or archive older folders when you no longer need them.
MAILDATE_RFC=`formail -czxDate:`
MAILDATE=`date -d "$MAILDATE_RFC" +%F`
:0:
inbox-$MAILDATE
This will save to an mbox file named like inbox-2022-06-10 based on the extracted Date: header. (Again, you could avoid the external processes if you really wanted to squeeze out the last bit of performance, using the date parsing sketch above. And again, if you can't have a message from a different time zone land in the previous or next day's folder, you need to recalculate the date for your time zone.)
My client generates a timestamp using Date.now(), generating a unix time in milliseconds. But when I parse it later on the server, I get a very different date. I can reproduce this discrepancy with Jsfiddle. My code is:
const parseMoment = moment('1603551654605', 'x').format();
const parseDate = new Date(1603551654605).toDateString();
alert(`moment: ${parseMoment}, date: ${parseDate}`);
The output I get is: moment: 2020-11-16T00:00:00+01:00, date: Sat Oct 24 2020
The correct output is what moment gives me here, 11/16. But moment on my server actually gives me the incorrect October date. I need to reliably parse my unix timestamp back to the correct (later) date.
The unix timestamp converts to Saturday, 24 October 2020 15:00:54.605 (GMT)
momentJS does't require any second parameters while creating a date from milliseconds (DOCS).
Using the correct conversion, both moment() and new Date() show the same result:
const parseMoment = moment(1603551654605).format();
const parseDate = new Date(1603551654605).toDateString()
console.log(`moment\t: ${parseMoment}`);
console.log(`date\t: ${parseDate}`);
<script src="https://cdnjs.cloudflare.com/ajax/libs/moment.js/2.29.1/moment.min.js"></script>
moment : 2020-10-24T17:00:54+02:00
date : Sat Oct 24 2020
I'm curious how to parse these weird timestamps in Ruby:
566455139129676
566455199011666
566455199892825
566455259010949
566455319010859
566455335000847
566455336000936
566455336127533
566455347898055
Here's what I know about the format of these timestamps:
The value is a 64-bit integer in little-endian format containing the
number of microseconds since Julian day: Jan 01 2000 00:00:00 in the
UTC timezone.
For the curious, these are Vertica's internal representation of timestamps: https://my.vertica.com/docs/9.0.x/HTML/index.htm#Authoring/AdministratorsGuide/BinaryFilesAppendix/ColumnDefinitions.htm
Any help would be most appreciated.
Not really sure what endianness would have to do with anything here. You've got an integer already.
So Epoch is Jan 01, 2000? Basically you just need to account for diff between epochs, 1969-12-31 and theirs...
epoch = Time.new(2000, 1, 1)
epochOffset = epoch.to_i - Time.at(0).to_i;
ts = Time.at((566455347898055 / 1000000) + epochOffset)
print ts.strftime "%Y-%m-%d %H:%M:%S %z";
> 2017-12-13 04:42:27 +0000
I am taking the MSG file and changing the date to match the local date in outlook. (same Sent date when you open the message in outlook).
But the times seem to be off 1 hour or many hours. I've dumped out the times to investigate further:-
I'm using Aspose to open the msg file.
c#
1.msg Shows: 05/01/2011 00:46 in outlook 2016. GMT
05/01/2011 00:46:07, Kind = Utc
TimeZone offset = -08:00:00
Actual UTC Time is 05/01/2011 00:46:07
Message date to universal time + timezoneoffset: 04/01/2011 16:46:07
timezoneoffset: -08:00:00
calculated universal msg date: 04/01/2011 16:46:07
output 04/01/2011 08:46 AM
2.msg Shows 20/06/2016 16:25 in outlook 2016. GMT
20/06/2016 16:25:23, Kind = Local
TimeZone offset = 02:00:00
Actual UTC Time is 20/06/2016 15:25:23
Message date to universal time + timezoneoffset: 20/06/2016 17:25:23
timezoneoffset: 02:00:00
calculated universal msg date: 20/06/2016 17:25:23
output 06/20/2016 07:25 PM
any ideas on how to correct this to show same as outlook? I see some times are UTC and some are Local, is there a solution anyone know of?
MailMessage msg = MailMessage.Load(inFile);
MemoryStream ms = new MemoryStream();
Console.WriteLine(msg.Date.ToString() + ", Kind = " + msg.Date.Kind);
Console.WriteLine("TimeZone offset = " + msg.TimeZoneOffset);
Console.WriteLine("Actual UTC Time is " + msg.Date.ToUniversalTime().ToString());
Console.WriteLine("Message date to universal time + timezoneoffset: " + (msg.Date.ToUniversalTime() +
msg.TimeZoneOffset).ToString());
Console.WriteLine("timezoneoffset: " + msg.TimeZoneOffset.ToString());
// do calculation.
TimeZone localZone = TimeZone.CurrentTimeZone;
TimeSpan ts = localZone.GetUtcOffset(msg.Date);
msg.Date = msg.Date + msg.TimeZoneOffset;
Console.WriteLine("calculated universal msg date: " +msg.Date.ToUniversalTime().ToString());
if (msg.Date.Second >= 30) // for rounding up to match outlook
{
// ... Days, hours, minutes, seconds, milliseconds.
TimeSpan span = new TimeSpan(0, 0, 0, 31, 0);
msg.Date = msg.Date + span;
}
MhtMessageFormatter mhtlFormat = new MhtMessageFormatter();
mhtlFormat.DateTimeFormat = "ddd MM/dd/yyyy hh:mm tt";
mhtlFormat.Format(msg);
MhtSaveOptions mhtSaveOptions = new MhtSaveOptions();
mhtSaveOptions.MhtFormatOptions = MhtFormatOptions.None;
mhtSaveOptions.MhtFormatOptions = mhtSaveOptions.MhtFormatOptions | MhtFormatOptions.HideExtraPrintHeader;
Thanks,
Lee.
As assisted on your thread in Aspose.Email forum, we believe your query has already met an answer. If you still find any issue or have confusion, you can inquire further on respective thread.
I work with Aspose as Developer Evangelist.