I'm using Laravel 5.8 and I like to auto-generate Invoices should be generated auto from Monday - Sunday.
every Sunday night at 23:59:00
Let's say I have
"23:59:00 according to the region time zone".
Every store is related to a region, from the region table you'll find the time zone.
Questions
How should I set the CRON JOB so I can generate invoices automatically according to the list of timezone I will get from the table?.
I have two suggestions;
if you want it to be static, define each command with the corresponding timezone from your database. Different timezones may have same time such as Europe/Amsterdam and Europe/Berlin
$schedule->command('generate:report')->timezone('one-of-the-timezone')->at('23:59');
$schedule->command('generate:report')->timezone('another-timezone')->at('23:59');
$schedule->command('generate:report')->timezone('yet-another-timezone')->at('23:59');
$schedule->command('generate:report')->timezone('some-other-timezone')->at('23:59');
If you want it to be dynamic, make it run every 59. min hourly and try to match it with the timezones via using hour.
$schedule->command('generate:report')->hourlyAt(59);
Inside your command class;
public function handle(TimezoneRepository $timezoneRepository)
{
$timezones = $timezoneRepository->getUniqueTimezones(); // the unique timezones in your database - cache if you want
foreach ($timezones as $timezone) {
$date = Carbon::now($timezone); // Carbon::now('Europe/Moscow'), Carbon::now('Europe/Amsterdam') etc..
if ($date->hour === 23) { // you are in that timezone
$currentTimezone = $date->getTimezone();
dispatch(new ReportMaker($currentTimezone)); // dispatch your report maker job
}
}
}
With the dynamic one, you will hit to multiple timezones at one iteration(when generate:report is executed) as i said at then beginning.
one of the possible flaw may be; if the execution of getting timezones etc takes more than 1 minute you may be in 00:00 instead of 23:59. It is better to calculate your report asynchronous and cache the list of timezones to not face problems while executing this loop.
Another possible flaw;
According to wiki;
A few zones are offset by 30 or 45 minutes (e.g. Newfoundland Standard Time is UTC−03:30, Nepal Standard Time is UTC+05:45, Indian Standard Time is UTC+05:30 and Myanmar Standard Time is UTC+06:30).
If you want to cover any of these, then it is better to execute the command like this
$schedule->command('generate:report')->cron('14,29,44,59 * * * *');
and make both hour and minute comparison such as;
$date->hour === 23 && $date->hour === 59
You can use the timezone() method in your task Schedulding
Using the timezone method, you may specify that a scheduled task's
time should be interpreted within a given timezone:
$schedule->command('report:generate')
->timezone('America/New_York')
->at('02:00')
want to Display the time along with milliseconds with the time duration like 1200 ms, 1500 ms on the Dashboard , Is there a way to define these time duration to display on chart?
enter image description here
I'm not sure if this would work for you. But what I do when sharing a report from an App Insights or Log Analytics query, is use the render * with(title="") syntax to add a title. This allows me to add context to what it is I am sharing. For you this could be the place you specify the units of the X and Y axis.
See my example below for including titles.
let start = ago(3h);
let end = now();
let timegrain = 1m;
pageViews
| where
timestamp > start and timestamp < end
| summarize
Avg_Duration_Seconds = avg(duration)/1000
by
bin(timestamp,timegrain)
| render timechart with(title="AVG PageView Duration(seconds) - Last 3 Hours, By Minute")
You'd need to change the units in whatever chart that is however you got it on the dashboard.
If you did that query in workbooks, you'd go into chart settings in that step of the workbook and tell it the y-axis is milliseconds, and it would add the appropriate units for you in the labels. If you are using metrics, the metrics should already know the units and show the right labels.
You'll need to clarify your question about exactly how / where you created that chart for anyone to help you any further.
my issue is that I want to be able to get two time stamps and compare if the second (later taken) one is less than 59 minutes away from the first one.
Following this thread Compare two dates with JavaScript
the date object may do the job.
but first thing i am not happy with is that it takes the time from my system.
is it possible to get the time from some public server or something?
cause there always is a chance that the system clock gets manipulated within the time stamps, so that would be too unreliable.
some outside source would be great.
then i am not too sure how to get the difference between 2 times (using 2 date objects).
many issue that may pop up:
time being something like 3:59 and 6:12
so just comparing minutes would give the wrong idea.
so we consider hours too.
biut there the issue with the modulo 24.
day 3 23:59 and day 4 0:33 wouldnt be viewed proper either.
so including days too.
then the modulo 30 thing, even though that on top changes month for month.
so month and year to be included as well.
so we would need the whole date, everything from current year to second (because second would be nice too, for precision)
and comparing them would require tons of if clauses for year, month, etc.
do the date objects have some predfeined date comparision function that actually keeps all these things in mind (havent even mentioned leap years yet, have I)?
time would be very important cause exactly at the 59 minutes mark (+-max 5 seconds wouldnt matter but getting rmeitely close to 60 is forbidden)
a certain function would have to be used that without fail closes a website.
script opens website at mark 0 min, does some stuff rinse and repeat style and closes page at 59 min mark.
checking the time like every few seconds would be smart.
Any good ideas how to implement such a time comparision that doesnt take too more computer power yet is efficient as in new month starting and stuff doesnt mess it up?
You can compare the two Date times, but when creating a date time there is a parameter of DateTime(value) which you can use.
You can use this API to get the current UTC time which returns a example JSON array like this:
{
"$id":"1",
"currentDateTime":"2019-11-09T21:12Z",
"utcOffset":"00:00:00",
"isDayLightSavingsTime":false,
"dayOfTheWeek":"Saturday",
"timeZoneName":"UTC",
"currentFileTime":132178075626292927,
"ordinalDate":"2019-313",
"serviceResponse":null
}
So you can use either the currentFileTime or the currentDateTime return from that API to construct your date object.
Example:
const date1 = new Date('2019-11-09T21:12Z') // time when I started writing this answer
const date2 = new Date('2019-11-09T21:16Z') // time when I finished writing this answer
const diff = new Date(date2-date1)
console.log(diff.toTimeString()) // time it took me to write this
Please keep in mind that due to network speeds, the time API will be a little bit off (by a few milliseconds)
Context
I have multiple servers listening to a specific collection (/items). Each of them use NTS for time calibration and the ".info/serverTimeOffset" to measure the expected time difference with Firebase. It is consistently around 20ms.
I have many clients pushing items to the collection with the specific field:
{
...
created: Firebase.database.ServerValue.TIMESTAMP
}
What is expected:
When the server receives the item from Firebase and subtracts the item.created with the Firebase expected time (Date.now() + offset), this value should be positive and probably around 10ms (time for the item to be sent from Firebase to the server).
What is happening:
When the server receives the items, the item.created field is superior to the Firebase expected time. Like it was created in the future. Usually the difference is around -5ms
Question:
What is the Firebase.database.ServerValue.TIMESTAMP set to ? and how is it related to the ".info/serverTimeOffset" ?
The 27th September 2016 at 1am UTC, that difference jumped from -5ms to around -5000ms like a kind of re-calibration happened (it lasted until I reset the .info/serverTimeOffset) Did someone experienced something similar?
Retrieving data from mongo takes too long, even for small datasets. For bigger datasets we get out of memory errors of the javascript engine. We've tried several schema designs and several ways to retrieve data. How do we optimize mongoDB/mapReduce function/MongoWire to retrieve more data quicker?
We're not very experienced with MongoDB yet and are therefore not sure whether we're missing optimization steps or if we're just using the wrong tools.
1. Background
For graphing and playback purposes we want to store changes for several objects over time. Currently we have tens of objects per project, but expectations are we need to store thousands of objects. The objects may change every second or not change for long periods of time. A Delphi backend writes to and reads from MongoDB through MongoWire and SuperObjects, the data is displayed in a web frontend.
2. Schema design
We're storing the object changes in minute-second-millisecond objects in a record per hour. The schema design is like described here. Sample:
o: object1,
dt: $date,
v: {0: {0:{0: {speed: 8, rate: 0.8}}}, 1: {0:{0: {speed: 9}}}, …}
We've put indexes on {dt: -1, o: 1} and {o:1}.
3. Retrieving data
We use a mapReduce to construct a new date based on the minute-second-millisecond objects and to put the object back in v:
o: object1,
dt: $date,
v: {speed: 8, rate:0.8}
An average document is about 525 kB before the mapReduce function and has had ~29000 updates. After mapReduce of such a document, the result is about 746 kB.
3.1 Retrieving data from through mongo shell with mapReduce
We're using the following map function:
function mapF(){
for (var i = 0; i < 3600; i++){
var imin = Math.floor(i / 60);
var isec = (i % 60);
var min = ''+imin;
var sec = ''+isec;
if (this.v.hasOwnProperty(min) && this.v[min].hasOwnProperty(sec)) {
for (var ms in this.v[min][sec]) {
if (imin !== 0 && isec !== 0 && ms !== '0' && this.v[min][sec].hasOwnProperty(ms)) {// is our keyframe
var currentV = this.v[min][sec][ms];
//newT is new date computed by the min, sec, ms above
if (toDate > newT && newT > fromDate) {
if (fields && fields.length > 0) {
for (var p = 0, length = fields.length; p < length; p++){
//check if field is present and put it in newV
}
if (newV) {
emit(this.o, {vs: [{o: this.o, dt: newT, v: newV}]});
}
} else {
emit(this.o, {vs: [{o: this.o, dt: newT, v: currentV}]});
}
}
}
}
}
}
};
The reduce function basically just passes the data on. The call to mapReduce:
db.collection.mapReduce( mapF,reduceF,
{out: {inline: 1},
query: {o: {$in: objectNames]}, dt: {$gte: keyframeFromDate, $lt: keyframeToDate}},
sort: {dt: 1},
scope: {toDate: toDateWithinKeyframe, fromDate: fromDateWithinKeyframe, fields: []},
jsMode: true});
Retrieving 2 objects over 1 hour: 2,4 seconds.
Retrieving 2 objects over 5 hour: 8,3 seconds.
For this method we would have to write js and bat files runtime and read the json data back in. We have not measured times fort his yet, because frankly, we don’t like the idea very much.
Another problem with this method is that we get out of memory errors of the v8 javascript engine when we try to retrieve data for longer periods and/or more objects. Using a pc with more RAM works to some extend in preventing out of memory, but it doesn't make retrieving data faster.
This article mentions splitVector, which we might use to devide the workload. But we're not sure on how to use the keyPattern and maxChunkSizeBytes options. Can we use a keyPattern for both o and dt?
We might use multiple collections, but our dataset isn’t that big to start with at the moment, so we’re worried about how much collections we’d need.
3.2 Retrieving data through mongoWire with mapReduce
For retrieving data through mongoWire with mapReduce, we use the same mapReduce functions as above. We use the following Delphi code to start te query:
FMongoWire.Get('$cmd',BSON([
'mapreduce', ‘collection’,
'map', bsonJavaScriptCodePrefix + FMapVCRFunction.Text,
'reduce', bsonJavaScriptCodePrefix + FReduceVCRFunction.Text,
'out', BSON(['inline', 1]),
'query', mapquery,
'sort', BSON(['dt', -1]),
'scope', scope
]));
Retrieving data with this method is about 3-4 times (!) slower. And then the data has to be translated from BSON (IBSONDocument to JSON (SuperObject), which is a major time consuming part in this method. For retrieving raw data we use TMongoWireQuery which translates the BSONdocument in parts, while this mapReduce function uses TMongoWire directly and tries to translate the complete result. This might explain why this takes so long, while normally it's quite fast. If we can reduce the time it takes for the mapReduce to return results, this might be a next step for us to focus on.
3.3 Retrieving raw data and parsing in Delphi
Retrieving raw data to Delphi takes a bit longer then the previous method, but probably because of the use of TMongoWireQuery, the translation from BSON to JSON is much quicker.
4. Questions
Can we do further optimizations on our schema design?
How can we make the mapReduce function faster?
How can we prevent the out of
memory errors of the v8 engine? Can someone give more information on
the splitVector function?
How can we best use of mapReduce from Delphi? Can we use
MongoWireQuery in stead of MongoWire?
5. Specs
MongoDB 3.0.3
MongoWire from 2015 (recently updated)
Delphi 2010 (got XE5 as well)
4GB RAM (tried on 8GB RAM as well, less out of memory, but reading times are about the same)
Phew what a question! First up: I'm not an expert at MongoDB. I wrote TMongoWire as a way to get to know MongoDB a little. Also I really (really) dislike when wrappers have a plethora of overloads to do the same thing but for all kinds of specific types. A long time ago programmers didn't have generics, but we did have Variant. So I built a MongoDB wrapper (and IBSONDocument) based around variants. That said, I apparently made something people like to use, and by keeping it simple performs quite well. (I haven't been putting much time in it lately, but on the top of the list is catering for the new authentication schemes since version 3.)
Now, about your specific setup. You say you use mapreduce to get from 500KB to 700KB? I think there's a hint there you're using the wrong tool for the job. I'm not sure what the default mongo shell does differently than when you do the same over TMongoWire.Get, but if I assume mapReduce assembles the response first before sending it over the wire, that's where the performance gets lost.
So here's my advice: you're right with thinking about using TMongoWireQuery. It offers a way to process data faster as the server will be streaming it in, but there's more.
I strongly suggest to use an array to store the list of seconds. Even if not all seconds have data, store null on the seconds without data so each minute array has 60 items. This is why:
One nicety that turned up in designing TMongoWireQuery, is the assumption you'll be processing a single (BSON) document at a time, and that the contents of the documents will be roughly similar, at least in the value names. So by using the same IBSONDocument instance when enumerating the response, you actually save a lot of time by not having to de-allocate and re-allocate all those variants.
That goes for simple documents, but would actually be nice to have on arrays as well. That's why I created IBSONDocumentEnumerator. You need to pre-load an IBSONDocument instance with an IBSONDocumentEnumerator in the place where you're expecting the array of documents, and you need to process the array in roughly the same way as with TMongoWireQuery: enumerate it using the same IBSONDocument instance, so when subsequent documents have the same keys, time is saved not having to re-allocate them.
In your case though, you would still need to pull the data of an entire hour through the wire just to select the seconds you need. As I said before, I'm not a MongoDB expert, but I suspect there could be a better way to store data like this. Either with a separate document per second (I guess this would let the indexes do more of the work, and MongoDB can take that insert-rate), or with a specific query construction so that MongoDB knows to shorten the seconds array into just that data you're requesting (is that what $splice does?)
Here's an example of how to use IBSONDocumentEnumerator on documents like {name:"fruit",items:[{name:"apple"},{name:"pear"}]}
q:=TMongoWireQuery.Create(db);
try
q.Query('test',BSON([]));
e:=BSONEnum;
d:=BSON(['items',e]);
d1:=BSON;
while q.Next(d) do
begin
i:=0;
while e.Next(d1) do
begin
Memo1.Lines.Add(d['name']+'#'+IntToStr(i)+d1['name']);
inc(i);
end;
end;
finally
q.Free;
end;