I have a kql-query which calculates number of uploaded BLOBS in Azure storage since last 24 hours.
The query blow returns a number as expected when run in Azure log analytics.
StorageBlobLogs
| where TimeGenerated > ago(1d) and OperationName has "PutBlob" and StatusText contains "success" a
| distinct Uri
| summarize count()
I want now to visualise this information in a timechart to get some detailed view. Have tried to add "render timechart" to the query chain as follows
StorageBlobLogs
| where TimeGenerated > ago(1d) and OperationName has "PutBlob" and StatusText contains "success" a
| distinct Uri
| summarize count()
| render timechart
When executing the query however, i am getting the error message;
Failed to create visualization
The Stacked bar chart can't be created as you are missing a column of one of the following types: int, long, decimal or real
Any tips to how this can be accomplished?
if you wish to look at the data aggregated at an hourly resolution (for example) and rendered as a timechart, you could try this:
StorageBlobLogs
| where TimeGenerated > ago(1d) and OperationName has "PutBlob" and StatusText contains "success"
| summarize dcount(Uri) by bin(TimeGenerated, 1h)
| render timechart
Related
I have created an alert policy in GCP MOnitoring which will notify me when a certain kind of log message stops appearing (a dead man's switch). I have create a logs-based metric with a label, "client", which I use to group the metric and get a timeseries per client. I have been using "absence of data" as the trigger for the alert. This has all been working well, until...
After a recent change, the logs now also com from different resources, so there is a need to combine the metric across those resources. I can achieve this using QML
{ fetch gce_instance::logging.googleapis.com/user/ping
| group_by [metric.client], sum(val())
| every 30m
; fetch global::logging.googleapis.com/user/ping
| group_by [metric.client], sum(val())
| every 30m }
| union
Notice that I need to align the two series with the same bucket size (30m) to be able to join them, which makes sense. I notice that the value for a timeseries is "undefined" in those buckets where the metric data was absent (by downloading a CSV of the query).
To create an alert using this query, I tried something like this:
{ fetch gce_instance::logging.googleapis.com/user/ping
| group_by [metric.client], sum(val())
| every 30m
; fetch global::logging.googleapis.com/user/ping
| group_by [metric.client], sum(val())
| every 30m }
| union
| absent_for 1h
If I look at the CSV output for this query it doesn't reflect the absence of metric data for a timeseries, and this is presumably because a value of "undefined" doesn't qualify as absent data.
Is there a way to detect for absence of data for a "unioned" metric (and therefore aligned) across multiple resources?
Update 1
I have tried this, which seems to get me some of the way there. I'd really appreciate comments on this approach.
{
fetch gce_instance::logging.googleapis.com/user/ping
| group_by [metric.client], sum(val())
;
fetch global::logging.googleapis.com/user/ping
| group_by [metric.client], sum(val())
}
| union
| absent_for 1h
I have settled on a solution as follows,
{
fetch gce_instance::logging.googleapis.com/user/ping
| group_by [metric.client]
;
fetch global::logging.googleapis.com/user/ping
| group_by [metric.client]
}
| union
| absent_for 1h
| every 30m
Note:
group_by [metric.client] conforms the tables from different resource, which allows the union to work
absent_for does align input timeseries using the default period or one specified by a following every
I found it really hard to debug these MQL queries, in particular to confirm that absent_for was going to trigger an alert. I realised that I could use value [active] to show a plot of the active column (which absent_for produces) and that gave me confidence that my alert was actually going to work.
{
fetch gce_instance::logging.googleapis.com/user/ping
| group_by [metric.client]
;
fetch global::logging.googleapis.com/user/ping
| group_by [metric.client]
}
| union
| absent_for 1h
| value [active]
i am trying to get Throughput(Volume) metrics by using below query
requests
// additional filters can be applied here
| where timestamp > ago(24h)
| where client_Type != "Browser"
| summarize count() by bin(timestamp, 5m)
| extend request='Volume'
// render result in a chart
| render timechart
So my question is for Volume do we use Count() or sum(itemCount) ? Which one of these is more Accurate to get Volume(Throughput) details per interval ??
The right way is to use sum(itemCount). Then this metric will be correct for sampled applications (by default adaptive sampling will kick in when number of telemetry items exceeds 5/sec).
What is the best way to track what requests were made to LUIS and QnA Maker and what the response's were?
I don't want to log the utterances and responses in any DB, I need something like AppInsights.
It's certainly possible to do this, and there are a lot of built-in pieces to do so. You don't mention what language/environment you're using (.net or node), but here are some starting points to look at:
Add telemetry to your bot (this link will take you straight to the section on LUIS & QnA Maker)
Add telemetry to your QnAMaker bot
The equivalent is possible in node, I'd imagine, if that's your language of choice. If so, this example might be useful.
For Qna Maker, as long as you enabled Application Insights when you created your QnA Maker service, all you need to do is create and run the queries in Logs. This page gives many examples of analytics you can run.
For LUIS, the link Hilton provided works for C#, but for node you need a different approach. This example will show you how to create a LUIS App Insights helper in node and send the traces to Application insights. It doesn't give you an example queries, but here are a few I use and have found useful. If there's some specific metric you're looking for, let me know.
Pie chart of intents over the last 30 days
requests
| where url endswith "messages"
| where timestamp > ago(30d)
| project timestamp, duration, performanceBucket, resultCode, url, id
| parse kind = regex url with *"(?i)http://"botName".azurewebsites.net/api/messages"
| join kind= inner (
traces | extend id = operation_ParentId
) on id
| where message == "LUIS"
| extend topIntent = tostring(customDimensions.LUIS_luisResponse_luisResult_topScoringIntent_intent)
| where topIntent != "None"
| where topIntent != ""
| summarize count() by topIntent
| order by count_ desc
| render piechart
List of queries with results below 0.5 confidence score
requests
| where url endswith "messages"
| where timestamp > ago(30d)
| project timestamp, duration, performanceBucket, resultCode, url, id
| parse kind = regex url with *"(?i)http://"botName".azurewebsites.net/api/messages"
| join kind= inner (
traces | extend id = operation_ParentId
) on id
| where message == "LUIS"
| extend topIntent = tostring(customDimensions.LUIS_luisResponse_luisResult_topScoringIntent_intent)
| extend score = todouble(customDimensions.LUIS_luisResponse_luisResult_topScoringIntent_score)
| extend utterance = tostring(customDimensions.LUIS_luisResponse_text)
| order by timestamp desc nulls last
| project timestamp, botName, topIntent, score, utterance, performanceBucket, duration, resultCode
| where score < 0.5
Average number of messages per conversation
requests
| where url endswith "messages"
| where timestamp > ago(30d)
| project timestamp, url, id
| parse kind = regex url with *"(?i)http://"botName".azurewebsites.net/api/messages"
| join kind= inner (
traces | extend id = operation_ParentId
) on id
| where message == "LUIS"
| extend convID = tostring(customDimensions.LUIS_botContext_conversation_id)
| order by timestamp desc nulls last
| project timestamp, botName, convID
| summarize messages=count() by conversation=convID
| summarize conversations=count(), messageAverage=avg(messages)
I am a newbie in UIPath.
I have a DataTable with these headers:
1.) Date
2.) Error
I want to extract a Distinct Date for every error, and use this code:
dtQuery = ExtractDataTable.DefaultView.ToTable(True,{"Date","Error"})
With this, I get my desired result. My problem is how can I append (a new Column, "Count") EACH COUNT of DISTINCT VALUES given? For Example:
DATE | ERROR | COUNT
2/27/2019 | Admin Query String |
2/27/2019 | 404 Shield |
2/26/2019 | 404 Shield |
2/25/2019 | 404 Shield |
2/25/2019 | Admin Query String |
I tried to use ADD DATA COLUMN ACTIVITY with these properties:
Column Name = "COUNT"
Data Table = dtQuery
DefaultValue = ExtractDataTable.DefaultView.ToTable(True,{"Date","Error"}).Rows.Count
But by using this, it gives me this:
DATE | ERROR | COUNT
2/27/2019 | Admin Query String | 5
2/27/2019 | 404 Shield | 5
2/26/2019 | 404 Shield | 5
2/25/2019 | 404 Shield | 5
2/25/2019 | Admin Query String | 5
Thanks in advance! Happy coding!
After hours of research, here is what I learned.
I can iterate on each item of the datatable by using FOR EACH ROW Activity.
So for every row item of my dtQuery, I add ASSIGN Activity that looks like this:
row(2) = [item i want to add]
But that doesn't answer my question. I want to know the count of each unique item with 2 criteria - They are same DATE and ERROR.
Maybe I can code directly on the Excel File?
So I researched for Excel Formula that looks like "Select Distinct Col1....etc."
I found this video tutorial, hope it might help: Countif
But its only for a single criterion, so I found this: Countifs
So to wrap it up,
For Each Row Image
1.) I loop inside dtQuery using For Each Row Activity
2.) Inside loop, I add Assign Activity with this code
row(2) = "=COUNTIFS('LookUp Sheet'!B:B,'Result Sheet'!A" & indexerRow + 2 & ",'LookUp Sheet'!D:D,'Result Sheet'!B" & indexerRow + 2 & ")"
Hope this help others who will be stumbling upon the same problem. Happy Automating! ^_^
I am pretty new to YUI and need some help.
I have a JSON response like this:
{
"Results":[
{
"alpha":57.935,
"beta:{
"delta":2.975,
"omega":1.431
},
"gamma":{
"theta":"0.339",
"lambda":"1.195"
}
},
{
"alpha":87,
"beta":{
"lambda":2.680,
"kappa":0.714
},
"gamma":{
"zeta":"0.288",
"epsilon":"0.289"
}
}
]
}
I would like to have a datatable with nested columns where:
1) alpha, beta and gamma are parent columns.
2) beta and gamma each have two columns formed of the JSON key-value pair (e.g., delta => 2.975).
3) The number of rows, i.e., total key-value pairs, is dynamic.
Basically, something like this:
----------------------------------------------
| alpha | beta | gamma |
----------------------------------------------
| 57.935 | delta | 2.975 | theta | 0.339 |
----------------------------------------------
| | omega | 1.431 | lambda | 1.195 |
----------------------------------------------
| 87.435 | lambda | 2.680 | zeta | 0.288 |
----------------------------------------------
| | kappa | 0.714 | epsilon | 0.289 |
----------------------------------------------
I have been able to generate non-nested, simple JSON responses.
My problems:
1) I have the object for each JSON child ({theta:0.339}, etc.). Both child columns will need data from this same object. How do I use it without modifying it? Should I use the same 'keyName' for both child columns in myColumnDefs?
2) How to create more than one rows where alpha td is empty?
Any help will be appreciated !
This is not an easy problem to solve. Barring your ability to format the JSON into individual rows before its sent to the client, you can hack together a solution using some column configurations, formatters, and a custom bodyView modelList attribute setter that flattens the data for display.
http://jsbin.com/3/efigim/1/edit?javascript,live
This would likely involve some breakage of table row -> data record associations since the bodyView's modelList contains its own Models for the rows rather than sharing a clientId. This may or may not get in your way, depending on whether you need additional features.
But since the DataTable's data ModelList preserves the objects for beta and gamma values--only the view's representation is customized--you might be fine.
YMMV, HTH