Good Morning all, I am currently building a tool which scrapes a number of figures from various logs on several different servers using AUX and Shell scripts on a minute by minute basis. Currently I have scripts that produce HTML files also on a minute by minute basis so that I can look back and see the system health for any given minute. So essentially these HTML files as being used as the archiving medium, which obviously is very strange.
Taking this tool forward I am looking to build a front end on a DOJO platform which will support widgets and graphs for each of the different data sets that is gathered from the logs. I haven't used DOJO much in the past and was hoping that somebody more experienced in the technology would be able to tell me what storage medium I should be using for these simple metrics that are produced by the scripts every minute. I have heard of the JSON method but am not sure.
Thanks for your time!
Jonny
Just to follow up on this in the rare chance it may be stumbled upon by another user. I decided that JSON was definitely the way forward. Each minute I have a system that produces a JSON files like the below.
{
"DataDTS" : "2014.05.02-09:00",
"System" : "Performance Test",
"SourceServer" : "gb02qws122debx7",
"OldestData" : "2014.05.02-09:00",
"MetricData" : {
"FredHopper Direct" : {
"Small" : "FH",
"Type" : "ByMinute",
"HeaderLevels" : 1,
"Data": { "Failures":0, "Avg":"0.318", "Min":"0.306", "Max":"0.340" }
},
"Performance By Instance" : {
"Small" : "RATE",
"Type" : "ByMinute",
"HeaderLevels" : 2,
"Sections" : {
"102.01" : { "Attributes" : ["(20)"], "Data": { "Req":"999", "Avg Resp":"0.254" } },
"102.02" : { "Attributes" : ["(20)"], "Data": { "Req":"144", "Avg Resp":"0.376" } },
"103.01" : { "Attributes" : ["(20)"], "Data": { "Req":"168", "Avg Resp":"0.199" } },
"103.02" : { "Attributes" : ["(20)"], "Data": { "Req":"152", "Avg Resp":"0.283" } },
"Unallocated" : { "Attributes": [], "Data": {"Req":"87", "Avg Resp":"0.154"} },
"qws122" : { "Attributes": [], "Data": {"Tot Req":"460", "Rate PPS":"7.7"} }
}
},
"HTML Status" : {
"Small" : "HTML",
"Type" : "Status",
"HeaderLevels" : 1,
"Data" : [
{ "Code":"200", "Req":447 },
{ "Code":"206", "Req":1 },
{ "Code":"301", "Req":7 },
{ "Code":"302", "Req":5 },
{ "Code":"TOT", "Req":460 }
]
},
"Page Breakdown" : {
"Small" : "PB",
"Type" : "Status",
"HeaderLevels" : 2,
"Sections" : {
"Top 15 Average Response Times" : {
"Attributes": [],
"Data" : [
{ "URL":"men", "Req":1, "Avg Resp":"3.597", "Cached":"0.0%" },
{ "URL":"jacques-vert", "Req":1, "Avg Resp":"3.335", "Cached":"0.0%" },
{ "URL":"OrderOKView", "Req":1, "Avg Resp":"1.615", "Cached":"0.0%" },
{ "URL":"warehouse", "Req":1, "Avg Resp":"1.050", "Cached":"0.0%" }
]
},
"Top 15 Requests" : {
"Attributes": [],
"Data" : [
{ "URL":"Navigate", "Req":220, "Avg Resp":"0.372", "Cached":"26.4%" },
{ "URL":"prod", "Req":42, "Avg Resp":"0.186", "Cached":"13.5%" },
{ "URL":"sureroute-test-object.html", "Req":24, "Avg Resp":"0.001", "Cached":"0.0%" },
{ "URL":"TopCategories1", "Req":8, "Avg Resp":"0.032", "Cached":"0.0%" }
]
}
}
},
"App server CPU" : {
"Small" : "CPU",
"Type" : "ByMinute",
"HeaderLevels" : 2,
"Sections" : {
"qap302" : { "Attributes" : ["Ent:7.0", "Max:8.0", "Mem:32768MB", "Wgt:200"], "Data": { "usr%":"16.8", "iow%":"1.4", "phy":"1.8", "ent%":"25.1" } },
"qap312" : { "Attributes" : ["Ent:7.0", "Max:8.0", "Mem:32768MB", "Wgt:200"], "Data": { "usr%":"8.1", "iow%":"0.1", "phy":"0.8", "ent%":"11.0" } }
}
}
}
Related
I need to return AppointmentTypes as a FHIR resource. Unfortunately, I couldn't find it as an official FHIR resource format.
My best guess would be to create a Basic resource, like this:
{
"resourceType": "Basic",
"id" : "id-of-appointment-type",
"identifier" : [
{
"use" : "secondary",
"system" : "http://myUrl/myIdentifier",
"value" : "7"
}
],
"code" : {
"coding": [
{
"system": "http://myUrl/appointment-type",
"code": "appointment-type"
}
]
},
"text" : {
"status" : "generated",
"div" : "<div xmlns=\"http://www.w3.org/1999/xhtml\">AppointmentType</div>"
},
"extension": [
{
"url": "http://myUrl/appointment-type-name",
"valueString": "New Patient"
},
{
"url": "http://myUrl/appointment-type-availability",
"valueBoolean": true
}
],
"meta" : {
"lastUpdated" : "2020-05-27T00:00:00.000Z"
}
}
Would this be the right way to create the AppointmentType resource?
I don't see any obvious issues, but did you evaluate using CodeSystem? You can define properties on CodeSystem codes which would be able to distinguish available from non-available appointment types - and that would work better with Appointment, where 'type' is expected to be a code.
I am setting the return to true, but when i complete the payment on square, the app does not redirect on its own.
{
"amount_money": {
"amount" : "100",
"currency_code" : "USD"
},
"auto_return":true,
"callback_url" : "https://floating-inlet-19449.herokuapp.com/redirect",
"client_id" : "sq0idp-U8x6mJyLFtHuhCfv9sqL5g",
"version": "1.3",
"notes": "notes for the transaction",
"options" : {
"supported_tender_types" : ["CREDIT_CARD"]
}
auto_return should be nested in the options object, like this:
{
"amount_money": { "amount" : 100, "currency_code" : "USD" },
"callback_url" : "https://floating-inlet-19449.herokuapp.com/redirect",
"client_id" : "sq0idp-U8x6mJyLFtHuhCfv9sqL5g",
"version": "1.3",
"notes": "notes for the transaction",
"options" : {
"supported_tender_types" : ["CREDIT_CARD"],
"auto_return": true
}
How can I add an extension that has multiple fields ?
For example if I have a extension related to visit motives that is structured like this in my application :
"visit_motive":
{
"id": "1",
"label": "Visit motive name",
"color": "#000000",
"duration": 5
}
I have something like this for the moment :
"extension": [
{
"url": "https://api.test.com/fhir/StructureDefinition/schedule-visit_motives",
"valueIdentifier": "visit_motive1_id",
"valueString" : "visit motive name",
"valueString" : "#000",
"valueInteger" : 5,
},
{
"url": "https://api.test.com/fhir/StructureDefinition/schedule-visit_motives",
"valueIdentifier": "visit_motive2_id",
"valueString" : "visit motive name 2",
"valueString" : "#111",
"valueInteger" : 10,
}
]
But I'm pretty sure it is not correct since I can't name the fields since I have to precise value[x] each time.
How can I do it ?
You would do that with a complex extension, which is basically an extension containing extensions. You can see more information about extensions in the spec. Scroll down a bit for the complex one. Or view an example of the structure of a complex extension here.
The use of your extension would look something like this:
"extension" : [{
"url" : "https://api.test.com/fhir/StructureDefinition/schedule-visit_motives",
"extension" : [{
"url" : "id",
"valueIdentifier": "visit_motive1_id"
}, {
"url" : "label",
"valueString" : "visit motive name"
}, {
"url" : "color",
"valueString" : "#000"
}, {
"url" : "duration",
"valueInteger" : 5,
}]
}]
I have a collection of user documents, where each user can have an arbitrary set of properties. Each user is associated to an app document. Here is an example user:
{
"appId": "XXXXXXX",
"properties": [
{ "name": "age", "value": 30 },
{ "name": "gender", "value": "female" },
{ "name": "alive", "value": true }
]
}
I would like to be able to find/count users based on the values of their properties. For example, find me all users for app X that have property Y > 10 and Z equals true.
I have a compound, multikey index on this collection db.users.ensureIndex({ "appId": 1, "properties.name": 1, "properties.value": 1}). This index is working well for single condition queries, ex:
db.users.find({
appId: 'XXXXXX',
properties: {
$elemMatch: {
name: 'age',
value: {
$gt: 10
}
}
}
})
The above query completes in < 300ms with a collection of 1M users. However, when I try and add a second condition, the performance degrades considerably (7-8s), and the explain() output indicates that the whole index is being scanned to fulfill the query ("nscanned" : 2752228).
Query
db.users.find({
appId: 'XXXXXX',
properties: {
$all: [
{
$elemMatch: {
name: 'age',
value: {
$gt: 10
}
}
},
{
$elemMatch: {
name: 'alive',
value: true
}
}
]
}
})
Explain
{
"cursor" : "BtreeCursor appId_1_properties.name_1_properties.value_1",
"isMultiKey" : true,
"n" : 256,
"nscannedObjects" : 1000000,
"nscanned" : 2752228,
"nscannedObjectsAllPlans" : 1018802,
"nscannedAllPlans" : 2771030,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 21648,
"nChunkSkips" : 0,
"millis" : 7425,
"indexBounds" : {
"appId" : [
[
"XXXXX",
"XXXXX"
]
],
"properties.name" : [
[
{
"$minElement" : 1
},
{
"$maxElement" : 1
}
]
],
"properties.value" : [
[
{
"$minElement" : 1
},
{
"$maxElement" : 1
}
]
]
},
"filterSet" : false
}
I assume this is because Mongo is unable to create suitable bounds since I am looking for both boolean and integer values.
My question is this: Is there a better way to structure my data, or modify my query to improve performance and take better advantage of my index? Is it possible to instruct mongo to treat each condition separately, generate appropriate bounds, and then perform the intersection of the results, instead of scanning all documents? Or is mongo just not suited for this type of use case?
I know this is an old question, but I think it would be much better to structure your data without the "name" and "value" tags:
{
"appId": "XXXXXXX",
"properties": [
{ "age": 30 },
{ "gender: "female" },
{ "alive": true }
]
}
I'm trying to get a jsTree working with on demand loading of subnodes. My code is this:
jQuery('#introspection_tree').jstree({
"json_data" : {
"ajax" : {
url : "http://localhost/introspection/introspection/product"
}
},
"plugins" : [ "themes", "json_data", "ui" ]
});
The json returned from the call is
[
{
"data": "Kit 1",
"attr": {
"id": "1"
},
"children": [
[
{
"data": "Hardware",
"attr": {
"id": "2"
},
"children": [
]
}
],
[
{
"data": "Software",
"attr": {
"id": "3"
},
"children": [
]
}
]
]
}
.....
]
Each element could have a lot of children, the tree is going to be big. Currently this is loading the whole tree at once, which could take some time. What do I have to do to implement on-demand-loading of child-nodes when they are opened by the user?
Thanks in advance.
Irishka pointed me in the right direction, but does not fully resolve my problem. I fiddled around with her answer and came up with this. Using two different server functions is done only for clarity. The first one lists all products at top level, the second one lists all children of a given productid:
jQuery("#introspection_tree").jstree({
"plugins" : ["themes", "json_data", "ui"],
"json_data" : {
"ajax" : {
"type": 'GET',
"url": function (node) {
var nodeId = "";
var url = ""
if (node == -1)
{
url = "http://localhost/introspection/introspection/product/";
}
else
{
nodeId = node.attr('id');
url = "http://localhost/introspection/introspection/children/" + nodeId;
}
return url;
},
"success": function (new_data) {
return new_data;
}
}
}
});
The json data returned from the functions is like this (notice the state=closed in each node):
[
{
"data": "Kit 1",
"attr": {
"id": "1"
},
"state": "closed"
},
{
"data": "KPCM 049",
"attr": {
"id": "4"
},
"state": "closed"
},
{
"data": "Linux BSP",
"attr": {
"id": "8"
},
"state": "closed"
}
]
No static data is needed, the tree is now fully dynamic on each level.
I guess it would be nice to display by default first level nodes and then the children will be loaded on demand. In that case the only thing you have to modify is to add "state" : "closed" to the nodes whose child nodes are going to be loaded on demand.
You might wish to send node's id in ajax call so you modify your code
"json_data": {
//root elements to be displayed by default on the first load
"data": [
{
"data": 'Kit 1',
"attr": {
"id": 'kit1'
},
"state": "closed"
},
{
"data": 'Another node of level 1',
"attr": {
"id": 'kit1'
},
"state": "closed"
}
],
"ajax": {
url: "http://localhost/introspection/introspection/product",
data: function (n) {
return {
"nodeid": $.trim(n.attr('id'))
}
}
}
}
From jsTree documentation
NOTE:
If both data and ajax are set the initial tree is rendered from the data string. When opening a closed node (that has no loaded children) an AJAX request is made.
you need to set root elements as tree data on page load and then you will be able to retrieve their children with an ajax request
$("#introspection_tree").jstree({
"plugins": ["themes", "json_data", "ui"],
"json_data": {
//root elements
"data": [{"data": 'Kit 1', "attr": {"id": 'kit1'}} /*, ... */], //the 'id' can not start with a number
"ajax": {
"type": 'POST',
"data": {"action": 'getChildren'},
"url": function (node) {
var nodeId = node.attr('id'); //id="kit1"
return 'yuorPathTo/GetChildrenScript/' + nodeId;
},
"success": function (new_data) {
//where new_data = node children
//e.g.: [{'data':'Hardware','attr':{'id':'child2'}}, {'data':'Software','attr':{'id':'child3'}}]
return new_data;
}
}
}
});
See my answer to a similar question here (the old part) for more details
I spended hours on this problem. Finally i got it that way:
$("#resourceTree").jstree({
"types": {
"default": {
"icon": "fa fa-folder-open treeFolderIcon",
}
},
"plugins": ["json_data", "types", "wholerow", "search"],
"core": {
"multiple": false,
"data": {
"url" : function(node){
var url = "rootTree.json";
if(node.id === "specialChildSubTree")
url = "specialChildSubTree.json";
return url;
},
"data" : function(node){
return {"id" : node.id};
}
}
},
});
rootTree.json:
[
{
"text": "Opened root folder",
"state": {
"opened": true
},
"children": [
{
"id" : "specialChildSubTree",
"state": "closed",
"children":true
}
]
}
]
specialChildSubTree.json:
[
"Child 1",
{
"text": "Child 2",
"children": [
"One more"
]
}
]
So i mark the node that become the parent of the ajax loaded subtree with an id, i watch for in the core configuration.
NOTE:
That node must have the "state" : "closed" parameter and it must have
the parameter "children" : true.
I am using jsTree.js in version 3.3.3
Above solution is all fine. Here I am also providing similar working solution and very simple for lazy loading of nodes using ajax call vakata. When your API works like
https://www.jstree.com/fiddle/?lazy
and for getting any child nodes
https://www.jstree.com/fiddle/?lazy&id=2
for explanation and for complete solution you can have a look at https://everyething.com/Example-of-jsTree-with-lazy-loading-and-AJAX-call
<script type="text/javascript">
$(function () {
$('#SimpleJSTree').jstree({
'core' : {
'data' : {
'url' : "https://www.jstree.com/fiddle/?lazy",
'data' : function (node) {
return { 'id' : node.id };
}
}
}
});
});
</script>