I'm trying to publish a XML message using the tibco adb adapter which monitors a table.
There is a column in the table that stores a date as a long value in milliseconds- Is there a way we can convert this long value to a date as a part of the XML marshalling ?
-tia
You can translate the milliseconds to a datetime xml value with the following x-path:
tib:add-to-dateTime(tib:translate-timezone('1970-01-01T00:00:00Z', tib:get-timezone-from-dateTime(current-dateTime())), 0, 0, 0, 0, 0, (<your input in ms> div 1000))
What this does is:
1) Find your timezone:
tib:get-timezone-from-dateTime(current-dateTime())
2) Move the base date (1970-01-01) to the correct timezone:
tib:translate-timezone('1970-01-01T00:00:00Z' <result above>)
3) Add the number of seconds in your input epoch:
tib:add-to-dateTime(<result above>, 0, 0, 0, 0, 0, (<your input in ms> div 1000))
Notice the downside to this solution is losing the milliseconds precision. The final datetime is in seconds.
If you need to keep the milliseconds in the final result, you are better off delegating this to an external Java library (as someone already suggested.) You can easily make a Java method available in the Designer mapper by importing it through a Java Custom Function resource.
I had to write a custom java process to get this done - Just in case someone finds it useful
I have ran into this quite a bit. We use the option in Adb to write all of the data to the other table. What you can do then do the following
Option 1
1. Change the data type of the other table to be varchar
2. Update the trigger to do e conversion for you
3. Update the scheme in bw.
- this will make it so you cannot use the GUI anymore for updates to Adb
Option 2
An alternate patter that we use for Adb is to use Adb as a trigger for an action. So instead of using all the information from Adb we use the key. We do a callback to a stored procedure to retrieve the rest of it. This is where we fix the formatting issue.
So if you are faimiliar with doing a db call in bw the second option is by far the easiest and less error prone. We have been using Tibco for about 3 years and have found some limitation to the active database adapter. Behind the scenes it is just a table pooler with an XML serialized and places the messages on a queue/topic
Option 3
The other option I have used is to create a pretty complicated mapper in your BW process. Search for a space and replace it with a "t".
Hope this helps
This works for me, and taking the resulting date time without the 'Z' in the end and adding the milliseconds I have my precision back. Now just to work out the daylight saving time ...
concat(
substring(
tib:add-to-dateTime(tib:translate-timezone('1970-01-01T00:00:00Z', tib:get-timezone-from-dateTime(current-dateTime())), 0, 0, 0, 0, 0, ($Element/root/s_date_in_ms div 1000))
,1,19)
,
".",
substring($Element/root/s_date_in_ms,12,3)
)
2014-11-10T23:02:28.858
tib:add-to-dateTime('1970-01-01T00:00:00', 0, 0, 0, 0,0, (<<timeinmillisecond>> div 1000))
Related
Summary
I wish to be able to measure time elapsed in milliseconds, on the GPU, of running the entire graphics pipeline. The goal: To be able to save benchmarks before/after optimizing the code (next step would be mipmapping textures) to see improvements. This was really simple in OpenGL, but I'm new to Vulkan, and could use some help.
I have browsed related existing answers (here and here), but they aren't really of much help. And I cannot find code samples anywhere, so I dare ask here.
Through documentation pages I have found a couple of functions that I think I should be using, so I have in place something like this:
1: Creating query pool
void CreateQueryPool()
{
VkQueryPoolCreateInfo createInfo{};
createInfo.sType = VK_STRUCTURE_TYPE_QUERY_POOL_CREATE_INFO;
createInfo.pNext = nullptr; // Optional
createInfo.flags = 0; // Reserved for future use, must be 0!
createInfo.queryType = VK_QUERY_TYPE_TIMESTAMP;
createInfo.queryCount = mCommandBuffers.size() * 2; // REVIEW
VkResult result = vkCreateQueryPool(mDevice, &createInfo, nullptr, &mTimeQueryPool);
if (result != VK_SUCCESS)
{
throw std::runtime_error("Failed to create time query pool!");
}
}
I had the idea of queryCount = mCommandBuffers.size() * 2 to have space for a separate query timestamp before and after rendering, but I have no clue whether this assumption is correct or not.
2: Recording command buffers
// recording command buffer i:
vkCmdWriteTimestamp(mCommandBuffers[i], VK_PIPELINE_STAGE_TOP_OF_PIPE_BIT, mTimeQueryPool, i);
// render pass ...
vkCmdWriteTimestamp(mCommandBuffers[i], VK_PIPELINE_STAGE_BOTTOM_OF_PIPE_BIT, mTimeQueryPool, i);
vkCmdCopyQueryPoolResults(/* many parameters here */);
I'm looking for a couple of clarifications:
What is the concequence of writing to the same query index? Do I need two separate query pools - one for before render time and one for after render time?
How should I handle synchronization? I assume having a separate query for each command buffer.
For the destination buffer containing the query result, is it good enough to store somewhere with "host visible bit", or do I need staging memory for "device visible only"? I'm a bit lost on this one as well.
I have not been able to find any online examples of how to measure render time, but I just assume it's such a common task that surely there must be an example out there somewhere.
So, thanks to #karlschultz, I managed to get something working. So in case other people will be looking for the same answer, I decided to post my findings here. For the Vulkan experts out there: Please let me know if I make obvious mistakes, and I will correct them here!
Query Pool Creation
I fill out a VkQueryPoolCreateInfo struct as described in my question, and let its queryCount field equal twice the number of command buffers, to store space for a query before and after rendering.
Important here is to reset all entries in the query pool before using the queries, and to reset a query after writing to it. This necessitates a few changes:
1) Asking graphics queue if timestamps are supported
When picking the graphics queue family, the struct VkQueueFamilyProperties has a field timestampValidBits which must be greater than 0, otherwise the queue family cannot be used for timestamp queries!
2) Determining the timestamp period
The physical device contains a special value which indicates the number of nanoseconds it takes for a timestamp query to be incremented by 1. This is necessary to interpret the query result as e.g. nanoseconds or milliseconds. That value is a float, and can be retrieved by calling vkGetPhysicalDeviceProperties and looking at the field VkPhysicalDeviceProperties.limits.timestampPeriod.
3) Asking for query reset support
During logical device creation, one must fill out a struct and add it to the pNext chain to enable the host query reset feature:
VkDeviceCreateInfo createInfo{};
VkPhysicalDeviceHostQueryResetFeatures resetFeatures;
resetFeatures.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_HOST_QUERY_RESET_FEATURES;
resetFeatures.pNext = nullptr;
resetFeatures.hostQueryReset = VK_TRUE;
createInfo.pNext = &resetFeatures;
4) Recording command buffers
Timestamp queries should be outside the scope of the render pass, as seen below. It is not possible to measure running time of a single shader (e.g. fragment shader), only the entire pipeline or whatever is outside the scope of the render pass, due to (potential) temporal overlap of pipeline stages.
vkCmdWriteTimestamp(mCommandBuffers[i], VK_PIPELINE_STAGE_TOP_OF_PIPE_BIT, mTimeQueryPool, i * 2);
vkCmdBeginRenderPass(/* ... */);
// render here...
vkCmdEndRenderPass(mCommandBuffers[i]);
vkCmdWriteTimestamp(mCommandBuffers[i], VK_PIPELINE_STAGE_BOTTOM_OF_PIPE_BIT, mTimeQueryPool, i * 2 + 1);
5) Retrieving query result
We have two methods for this: vkCmdCopyQueryPoolResults and vkGetQueryPoolResults. I chose to go with the latter since is greatly simplifies the setup and does not require synchronization with GPU buffers.
Given that I have a swapchain index (in my scenario same is command buffer index!), I have a setup like this:
void FetchRenderTimeResults(uint32_t swapchainIndex)
{
uint64_t buffer[2];
VkResult result = vkGetQueryPoolResults(mDevice, mTimeQueryPool, swapchainIndex * 2, 2, sizeof(uint64_t) * 2, buffer, sizeof(uint64_t),
VK_QUERY_RESULT_64_BIT);
if (result == VK_NOT_READY)
{
return;
}
else if (result == VK_SUCCESS)
{
mTimeQueryResults[swapchainIndex] = buffer[1] - buffer[0];
}
else
{
throw std::runtime_error("Failed to receive query results!");
}
// Queries must be reset after each individual use.
vkResetQueryPool(mDevice, mTimeQueryPool, swapchainIndex * 2, 2);
}
The variable mTimeQueryResults refers to an std::vector<uint64_t> which contains a result for each swapchain. I use it to calculate an average rendering time each second by using the timestamp period determined in step 2).
And one must not forget to cleanup to query pool by calling vkDestroyQueryPool.
There are a lot of details omitted, and for a total Vulkan noob like me this setup was frightening and took several days to figure out. Hopefully this will spare someone else the headache.
More info in documentation.
Writing to the same query index is bad because you are overwriting your "before" timestamp with the "after" timestamp at the same query index. You might want to change the last parameter in your write timestamp calls to i * 2 for the "before" call and to i * 2 + 1 for the "after". You are already allocating 2 timestamps for each command buffer, but only using half of them. This scheme ends up producing a pair of before/after timestamps for each command buffer i.
I don't have any experience using vkCmdCopyQueryPoolResults(). If you can idle your queue, then after idle, call vkGetQueryPoolResults() which will probably be much easier for what you are doing here. It copies the query results back into host memory and you don't have to mess with synchronizing writes to another buffer and then mapping/reading it back.
I have an incoming stream of events, each of which already has an associated sessionId from another process.
All I wish to do is combine these events into a single session object using a custom CombineFn.
During development, I'm using a bounded dataset that reads from a file and the following code seems to work:
input.apply(ParDo.named("ParseEvent").of(new ParseEventFn()))
.setCoder(KvCoder.of(StringUtf8Coder.of(), AvroCoder.of(Event.class)))
.apply(GroupByKey.<String, Event>create())
.apply(Combine.groupedValues(new SessionAccumulator()))
The above code (with input/output handling) will output a series of sessions with multiple events in each.
{sessionId: 1, events: [event1,event2,event3]}
{sessionId: 2, events: [event4,event5]}
But in order for this to work on an unbounded dataset, I need to apply a Windowing function, which in this case is a SessionWindow.
input.apply(ParDo.named("ParseEvent").of(new ParseEventFn()))
.setCoder(KvCoder.of(StringUtf8Coder.of(), AvroCoder.of(Event.class)))
.apply(Window.<KV<String, Event>>into(Sessions.withGapDuration(Duration.standardMinutes(30))))
.apply(GroupByKey.<String, Event>create())
.apply(Combine.groupedValues(new SessionAccumulator()))
In that case the only new code is the Windowing function, and rather than rolling up the events, I get each event in it's own session, like this:
{sessionId: 1, events: [event1]}
{sessionId: 1, events: [event2]}
{sessionId: 1, events: [event3]}
{sessionId: 2, events: [event4]}
{sessionId: 2, events: [event5]}
Any idea why this is happening?
EDIT: I should add, the ParseEventFn is applying a timestamp to the PCollection using context.outputWithTimestamp(), and that timestamp seems to be correct.
Digging into it further, in my case the issue was that my core assumption that the timestamps were correct, was wrong.
The timestamps I was applying before the windowing were wrong.
The Windowing was doing exactly what it should, but I had set my timestamps too far apart and it was creating separate sessions for each event.
Oops
In your case, you could possibly write your own WindowFn. If you set the keys to be the session IDs then a large gap duration also works, but it doesn't reflect the nature of your data and computation quite as well.
The ingredients to your WindowFn would be:
your own subclass of BoundedWindow, in this case you would make a window type that contained the session ID in a field
assignWindows, where you would assign each element to a window identified by the session ID. The length of the window still matters, as it controls when the window expires and is garbage collected.
mergeWindows, where you would merge all windows that have the same session ID. They wouldn't have to fall within any particular gap duration.
Another thing you'll need to be careful of is that the watermark that governs the garbage collection of these windows is determined by the source of your unbounded stream of events. So setting the timestamps in your ParDo.of(new ParseEventFn()) will be too late to influence the watermark. You may have data dropped that you'd like to keep.
Retrieving data from mongo takes too long, even for small datasets. For bigger datasets we get out of memory errors of the javascript engine. We've tried several schema designs and several ways to retrieve data. How do we optimize mongoDB/mapReduce function/MongoWire to retrieve more data quicker?
We're not very experienced with MongoDB yet and are therefore not sure whether we're missing optimization steps or if we're just using the wrong tools.
1. Background
For graphing and playback purposes we want to store changes for several objects over time. Currently we have tens of objects per project, but expectations are we need to store thousands of objects. The objects may change every second or not change for long periods of time. A Delphi backend writes to and reads from MongoDB through MongoWire and SuperObjects, the data is displayed in a web frontend.
2. Schema design
We're storing the object changes in minute-second-millisecond objects in a record per hour. The schema design is like described here. Sample:
o: object1,
dt: $date,
v: {0: {0:{0: {speed: 8, rate: 0.8}}}, 1: {0:{0: {speed: 9}}}, …}
We've put indexes on {dt: -1, o: 1} and {o:1}.
3. Retrieving data
We use a mapReduce to construct a new date based on the minute-second-millisecond objects and to put the object back in v:
o: object1,
dt: $date,
v: {speed: 8, rate:0.8}
An average document is about 525 kB before the mapReduce function and has had ~29000 updates. After mapReduce of such a document, the result is about 746 kB.
3.1 Retrieving data from through mongo shell with mapReduce
We're using the following map function:
function mapF(){
for (var i = 0; i < 3600; i++){
var imin = Math.floor(i / 60);
var isec = (i % 60);
var min = ''+imin;
var sec = ''+isec;
if (this.v.hasOwnProperty(min) && this.v[min].hasOwnProperty(sec)) {
for (var ms in this.v[min][sec]) {
if (imin !== 0 && isec !== 0 && ms !== '0' && this.v[min][sec].hasOwnProperty(ms)) {// is our keyframe
var currentV = this.v[min][sec][ms];
//newT is new date computed by the min, sec, ms above
if (toDate > newT && newT > fromDate) {
if (fields && fields.length > 0) {
for (var p = 0, length = fields.length; p < length; p++){
//check if field is present and put it in newV
}
if (newV) {
emit(this.o, {vs: [{o: this.o, dt: newT, v: newV}]});
}
} else {
emit(this.o, {vs: [{o: this.o, dt: newT, v: currentV}]});
}
}
}
}
}
}
};
The reduce function basically just passes the data on. The call to mapReduce:
db.collection.mapReduce( mapF,reduceF,
{out: {inline: 1},
query: {o: {$in: objectNames]}, dt: {$gte: keyframeFromDate, $lt: keyframeToDate}},
sort: {dt: 1},
scope: {toDate: toDateWithinKeyframe, fromDate: fromDateWithinKeyframe, fields: []},
jsMode: true});
Retrieving 2 objects over 1 hour: 2,4 seconds.
Retrieving 2 objects over 5 hour: 8,3 seconds.
For this method we would have to write js and bat files runtime and read the json data back in. We have not measured times fort his yet, because frankly, we don’t like the idea very much.
Another problem with this method is that we get out of memory errors of the v8 javascript engine when we try to retrieve data for longer periods and/or more objects. Using a pc with more RAM works to some extend in preventing out of memory, but it doesn't make retrieving data faster.
This article mentions splitVector, which we might use to devide the workload. But we're not sure on how to use the keyPattern and maxChunkSizeBytes options. Can we use a keyPattern for both o and dt?
We might use multiple collections, but our dataset isn’t that big to start with at the moment, so we’re worried about how much collections we’d need.
3.2 Retrieving data through mongoWire with mapReduce
For retrieving data through mongoWire with mapReduce, we use the same mapReduce functions as above. We use the following Delphi code to start te query:
FMongoWire.Get('$cmd',BSON([
'mapreduce', ‘collection’,
'map', bsonJavaScriptCodePrefix + FMapVCRFunction.Text,
'reduce', bsonJavaScriptCodePrefix + FReduceVCRFunction.Text,
'out', BSON(['inline', 1]),
'query', mapquery,
'sort', BSON(['dt', -1]),
'scope', scope
]));
Retrieving data with this method is about 3-4 times (!) slower. And then the data has to be translated from BSON (IBSONDocument to JSON (SuperObject), which is a major time consuming part in this method. For retrieving raw data we use TMongoWireQuery which translates the BSONdocument in parts, while this mapReduce function uses TMongoWire directly and tries to translate the complete result. This might explain why this takes so long, while normally it's quite fast. If we can reduce the time it takes for the mapReduce to return results, this might be a next step for us to focus on.
3.3 Retrieving raw data and parsing in Delphi
Retrieving raw data to Delphi takes a bit longer then the previous method, but probably because of the use of TMongoWireQuery, the translation from BSON to JSON is much quicker.
4. Questions
Can we do further optimizations on our schema design?
How can we make the mapReduce function faster?
How can we prevent the out of
memory errors of the v8 engine? Can someone give more information on
the splitVector function?
How can we best use of mapReduce from Delphi? Can we use
MongoWireQuery in stead of MongoWire?
5. Specs
MongoDB 3.0.3
MongoWire from 2015 (recently updated)
Delphi 2010 (got XE5 as well)
4GB RAM (tried on 8GB RAM as well, less out of memory, but reading times are about the same)
Phew what a question! First up: I'm not an expert at MongoDB. I wrote TMongoWire as a way to get to know MongoDB a little. Also I really (really) dislike when wrappers have a plethora of overloads to do the same thing but for all kinds of specific types. A long time ago programmers didn't have generics, but we did have Variant. So I built a MongoDB wrapper (and IBSONDocument) based around variants. That said, I apparently made something people like to use, and by keeping it simple performs quite well. (I haven't been putting much time in it lately, but on the top of the list is catering for the new authentication schemes since version 3.)
Now, about your specific setup. You say you use mapreduce to get from 500KB to 700KB? I think there's a hint there you're using the wrong tool for the job. I'm not sure what the default mongo shell does differently than when you do the same over TMongoWire.Get, but if I assume mapReduce assembles the response first before sending it over the wire, that's where the performance gets lost.
So here's my advice: you're right with thinking about using TMongoWireQuery. It offers a way to process data faster as the server will be streaming it in, but there's more.
I strongly suggest to use an array to store the list of seconds. Even if not all seconds have data, store null on the seconds without data so each minute array has 60 items. This is why:
One nicety that turned up in designing TMongoWireQuery, is the assumption you'll be processing a single (BSON) document at a time, and that the contents of the documents will be roughly similar, at least in the value names. So by using the same IBSONDocument instance when enumerating the response, you actually save a lot of time by not having to de-allocate and re-allocate all those variants.
That goes for simple documents, but would actually be nice to have on arrays as well. That's why I created IBSONDocumentEnumerator. You need to pre-load an IBSONDocument instance with an IBSONDocumentEnumerator in the place where you're expecting the array of documents, and you need to process the array in roughly the same way as with TMongoWireQuery: enumerate it using the same IBSONDocument instance, so when subsequent documents have the same keys, time is saved not having to re-allocate them.
In your case though, you would still need to pull the data of an entire hour through the wire just to select the seconds you need. As I said before, I'm not a MongoDB expert, but I suspect there could be a better way to store data like this. Either with a separate document per second (I guess this would let the indexes do more of the work, and MongoDB can take that insert-rate), or with a specific query construction so that MongoDB knows to shorten the seconds array into just that data you're requesting (is that what $splice does?)
Here's an example of how to use IBSONDocumentEnumerator on documents like {name:"fruit",items:[{name:"apple"},{name:"pear"}]}
q:=TMongoWireQuery.Create(db);
try
q.Query('test',BSON([]));
e:=BSONEnum;
d:=BSON(['items',e]);
d1:=BSON;
while q.Next(d) do
begin
i:=0;
while e.Next(d1) do
begin
Memo1.Lines.Add(d['name']+'#'+IntToStr(i)+d1['name']);
inc(i);
end;
end;
finally
q.Free;
end;
I am trying to call liveReport service of kaltura but I m not sure how to specify time in the request filters. It asks time to be in int format but when i canver time in milli seconds I get long value, but API is expecting int value.
can anyone provide help in this?
If you are referring to livereports.getreport service then fromTime & toTime properties expect an integer timestamp in seconds. You can also provide a timestamp that is lower then 315360000 seconds and it will be treated relative to NOW, so -1440 will be converted to NOW - 1440
Seems like other report services are expecting an integer that is created using conversion of date format YYYYMMDD to integer, so for May 18th, 2015 you would send 20150318.
I want to check the performance of API using Apache Jmeter 2.11. The response data for HTTP request is as follows:
{
"global_id": 11111,
"name": "IMG_001.JPG",
"width": "1111",
"height": "1111",
"time_taken": {
"segment_1_time": 1,
"segment_2_time": 1,
"segment_3_time": 27,
"segment_4_time": 1,
"segment_5_time": 56,
"segment_6_time": 8,
"total_time": 94
}
}
Thread Properties:
Number of threads: 1,
Ramp-up period(in seconds): 1,
Loop Count: 50
I want to calculate the aggregate mean time taken by each segment for all the responses i.e. Mean time taken by segment_1, Mean time taken by segment_2, Mean time taken by segment_3, Mean time taken by segment_4, Mean time taken by segment_5 and Mean time taken by segment_6.
What are the post processors and javascript required for calculating the mean segments' times?
I think you need to use regular expression extractor with beanshell post processor,
First, you need to extract your segment value using regular expression extractor with proper Regex like segment_1_time":(.*?), etc.
Next,
use beanshell post processor to calculate the Total value of each segment time,
var CurrentSegmentValue=vars.get("SEGMENT1");//Current segment value from Regular expression extractor
log.info("CurrentSegmentValue = "+CurrentSegmentValue);
/* You need to use this logic to calculate other segment values like SEGMENT2,3,4,5,6 etc
* but remember this method makes performace bottle neck when you have tested with higher number of threads
* and result may be differ see any appropriate web links for Jmeter performance tuning and suggested to use 64 bit JVM
*/
var totalvalueString=vars.get("totalValue");
if(totalvalueString==null)
{
vars.put("totalValue","0");
}
int totalValue= java.lang.Integer.parseInt(CurrentSegmentValue)+java.lang.Integer.parseInt(totalvalueString);
vars.put("totalValue",""+totalValue);
log.info("Final Total Value:"+totalValue);
//Going to next loop
/*Divide total value with number of threads you will get your mean time*/
In above code each segment value from response is saved and added with previous value.
Finally you can divide it with number of threads to get mean
Hopes this may help you and refer this link for more information about bean shell scripting
If you are open to try other test tool, you may want to look at NetGend. Here is a blog that shows how this scenario is implemented on NetGend platform - so simple that you don't need a lot of programming background to understand it.
Good luck.