How do you identify questions that the bot could not answer - botframework

Question

My organisation is starting to experiment with the Microsoft bot framework. One of the questions our enterprise architect has asked is as follows:
How do we identify questions that the bot was unable to answer?
I've checked the documentation but I'm still unclear. Can anyone elaborate on the techniques that they use to identify unanswered questions? We feel this is important as it identifies opportunities for further growth.

Answer 1

You can achieve this using a number of techniques. Essentially, what you are trying to do is to store any questions the Bot has not been able to provide an answer for analysis.
You can do this by using the scoring mechanism in the QnAMaker. For example, if the QnAMaker returns a score of zero, an answer doesn't exist, so we need to write that question back to storage for analysis.
You can use a number of storage solutions for this in the Azure stack, such as Application Insights, Cosmos, Blob, SharePoint Lists etc.
In the example below (code trimmed for brevity), I'm using Application Insights to store this information. I have imported the botbuilder-applicationinsights package and have created a simple custom event to capture any responses that score zero against the QnAMaker.
const {
ApplicationInsightsTelemetryClient,
ApplicationInsightsWebserverMiddleware
} = require('botbuilder-applicationinsights');
const {
MessageFactory,
CardFactory
} = require('botbuilder');
const {
QnAServiceHelper
} = require('../helpers/qnAServiceHelper');
const {
CardHelper
} = require('../helpers/cardHelper');
const {
FunctionDialogBase
} = require('./functionDialogBase');
// Setup Application Insights
settings = require('../settings').settings;
const appInsightsClient = new ApplicationInsightsTelemetryClient(settings.instrumentationKey);
class QnADialog extends FunctionDialogBase {
constructor() {
super('qnaDialog');
}
async processAsync(oldState, activity) {
var newState = null;
var query = activity.text;
var qnaResult = await QnAServiceHelper.queryQnAService(query, oldState);
var qnaAnswer = qnaResult[0].answer;
var qnaNonResponse = qnaResult[0].score;
var prompts = null;
if (qnaResult[0].context != null) {
prompts = qnaResult[0].context.prompts;
}
var outputActivity = null;
if (prompts == null || prompts.length < 1) {
outputActivity = MessageFactory.text(qnaAnswer);
} else {
var newState = {
PreviousQnaId: qnaResult[0].id,
PreviousUserQuery: query
}
outputActivity = CardHelper.GetHeroCard(qnaAnswer, prompts);
}
if (qnaNonResponse === 0) {
const {
NonResponseCard
} = require('../dialogs/non-response');
const quicknonresponseCard = CardFactory.adaptiveCard(NonResponseCard);
outputActivity = ({
attachments: [quicknonresponseCard]
});
console.log("Cannot find QnA response for" + " " + query);
appInsightsClient.trackEvent({
name: "Non-response",
properties: {
question: query
}
});
}
return ([newState, outputActivity, null]);
}
}
module.exports.QnADialog = QnADialog;
I can then hook up the query I might use in Application Insights in Power Bi to surface those non-answered questions.
There are multiple ways to achieve this, but this was one I ended up going with.

Answer 2

Depending of the size and the complexity of your model you will want to use LUIS or qnamaker. If your mother is very simple qnamaker will works. for something a bit more complex especially if you want to make use of entities LUIS is definitely the way to go. Each of them have their own technique and #steviebleeds describe how to do it on qnamaker. For Louis you are going to look at your confidence threshold and you should record that have below the confidence threshold you have set. each time you get a prediction from Lewis it send you a list of intent each of them having a confidence percentage on the predictions. You should assess this confidence percentage and decide depending of your fresh hold if you want or not to answer you users. You also want to look at all questions that have return none intent.