LUIS Utterance and response tracking - botframework

Question

What is the best way to track what requests were made to LUIS and QnA Maker and what the response's were?
I don't want to log the utterances and responses in any DB, I need something like AppInsights.

Answer 1

It's certainly possible to do this, and there are a lot of built-in pieces to do so. You don't mention what language/environment you're using (.net or node), but here are some starting points to look at:
Add telemetry to your bot (this link will take you straight to the section on LUIS & QnA Maker)
Add telemetry to your QnAMaker bot
The equivalent is possible in node, I'd imagine, if that's your language of choice. If so, this example might be useful.

Answer 2

For Qna Maker, as long as you enabled Application Insights when you created your QnA Maker service, all you need to do is create and run the queries in Logs. This page gives many examples of analytics you can run.
For LUIS, the link Hilton provided works for C#, but for node you need a different approach. This example will show you how to create a LUIS App Insights helper in node and send the traces to Application insights. It doesn't give you an example queries, but here are a few I use and have found useful. If there's some specific metric you're looking for, let me know.
Pie chart of intents over the last 30 days
requests
| where url endswith "messages"
| where timestamp > ago(30d)
| project timestamp, duration, performanceBucket, resultCode, url, id
| parse kind = regex url with *"(?i)http://"botName".azurewebsites.net/api/messages"
| join kind= inner (
traces | extend id = operation_ParentId
) on id
| where message == "LUIS"
| extend topIntent = tostring(customDimensions.LUIS_luisResponse_luisResult_topScoringIntent_intent)
| where topIntent != "None"
| where topIntent != ""
| summarize count() by topIntent
| order by count_ desc
| render piechart
List of queries with results below 0.5 confidence score
requests
| where url endswith "messages"
| where timestamp > ago(30d)
| project timestamp, duration, performanceBucket, resultCode, url, id
| parse kind = regex url with *"(?i)http://"botName".azurewebsites.net/api/messages"
| join kind= inner (
traces | extend id = operation_ParentId
) on id
| where message == "LUIS"
| extend topIntent = tostring(customDimensions.LUIS_luisResponse_luisResult_topScoringIntent_intent)
| extend score = todouble(customDimensions.LUIS_luisResponse_luisResult_topScoringIntent_score)
| extend utterance = tostring(customDimensions.LUIS_luisResponse_text)
| order by timestamp desc nulls last
| project timestamp, botName, topIntent, score, utterance, performanceBucket, duration, resultCode
| where score < 0.5
Average number of messages per conversation
requests
| where url endswith "messages"
| where timestamp > ago(30d)
| project timestamp, url, id
| parse kind = regex url with *"(?i)http://"botName".azurewebsites.net/api/messages"
| join kind= inner (
traces | extend id = operation_ParentId
) on id
| where message == "LUIS"
| extend convID = tostring(customDimensions.LUIS_botContext_conversation_id)
| order by timestamp desc nulls last
| project timestamp, botName, convID
| summarize messages=count() by conversation=convID
| summarize conversations=count(), messageAverage=avg(messages)