I have built a chatbot that answers questions related to Sustainable Development Goals (SDGs) using QnA Maker service. As a part of that, the main functionality of the bot is to answer different SDGs like what is SDG 1, what is SDG 2.... what is SDG 17.
How can I get accurate predictions from QnA Maker in this scenario where the questions are so similar as currently, the QnA Maker is always returning the answer for what is SDG 1 when I ask questions like what is SDG 16, 15, 14, etc even though the knowledge base has exact same questions
I'm not surprised. Looking at the just the characters of those questions, if your QnA KB is trained to recognize 'What is SDG 1' and does, to a high lvl of certainity, then 'What is SDG 1x' is going to be recognized simply based on the percentage of matching characters. To QnA maker:
'What is SDG 1x' ALL look like 'What is SDG 1'. You need to go into your QnA KB, and train it so that questions like 'What is SDG 19' has a 100% certainty. You can check this by looking at the 'inspect' element of the 'test' feature:
As you can see from my image 'One' was a possible anser to this question, but 'One' is what I have as the answer to 'What is SDG 1'. (ignore the other answer, i do alot of testing on this KB). If you go into inspect like this, and you see that the wrong answer is selected, you can simply chose the right one, then retrain your KB.
I did this, choosing the wrong answer repeated, then retraining, until I got my KB (despite having a perfect answer already), to respond with 100% certainty with the wrong answer (shown below):
You're going to have to do something similar, but with the correct answer.
It's just not only that question, but there are also many other similar questions like what is the agenda of sdg 11, what is the mandate of sdg 11 etc. for each sdg which are currently predicted as sdg 1 by QnA Maker
The idea that one question -> one answer is a good one, but you're going to have to work a bit more if all the questions look the same. Additionally, if you think this is going to come up when working with customers, you can code into your bot to return the top 3 or 5 answers, and have your bot do a follow up question like "I'm not sure I understood, did you mean '1' or '17' or '19'?", then have the user select which one they meant.
The issue was resolved by using the parameter "Top" from QnAMakerOptions Class when calling the GetAnswersAsync method. I kept the value of the parameter "Top" to be 20.
If there are very similar questions in a Knowledgebase, Azure Search does not return relevant QnA pairs in the default list of results. These results are re-ranked by the final QnA Maker ranker if the parameter "top" is used.
Related
My scenario is that our marketing department have various (FAQ related) URL's , each of which contains approximately 20 question and answer pairs, per URL.
I have imported these URL's into QnA maker and (importantly) the marketing department have finessed and tidied the answers held in the knowledge base (* reason below)
The marketing department still use a process where they add new question and answers to the relevant URL when needed, so - for example URL "X" might contain 20 Question and Answer pairs currently , all of which have been imported (and tidied/finessed) in the KB.
They want to add a new question and answer pair to the URL (i.e. make the change on the website), and then they want to "Refresh" that URL into the Knowledge Base. The problem is (I believe), that the original 20 question and answers pairs on the web url will actually over-write the "finessed" ones in the KB.
Is there any way to avoid this? Ideally just (somehow) bring in the "newly added" Question and Answer pair? (without having to copy/paste the content from the web to the KB?)
(*)the reason this needed doing was simply that, on the source URL's the answers were quite verbose, and included images, which work fine on the actual website, but when these answers are imported , they didn't look elegant within the bot , so they have been edited (within QnA maker) so they now look good when yielded by the bot)
Importing URLs into QnA Maker that only contain new content will always overwrite a knowledgebase with the content found within. This is true whether that is from doc linked from a URL, an uploaded doc, or some other method. If you are wanting to add new questions to an existing knowledgebase, then you first need to export that KB, make updates to the content contained within it (i.e. add your new questions to the old), and then upload/provide a link to that new data set.
If you are wanting to do this via REST API, then you should look over the QnA Maker REST API Reference docs. By using calls like GET, LIST, and UPDATE, you should be able to achieve what you are wanting to do. There are samples you can reference, as well, to help you on your way.
We are uploading a PDF with semi structured Question & Answers. QnA maker is merging same question if they are successive. If there is some other question exist between them, then QnA maker is not merging same questions. For example.
Q Machine was not able to be started
Answer 1
Q Machine was not able to be started
Answer 2
Q Burning plastic smell on machine
Answer 3
Now the QnA Maker will train it like this
Q Machine was not able to be started
Answer 1
Answer 2
Q Burning plastic smell on machine
Answer 3
Why is QnA is behaving like this and how to separate same questions. Help is required.
This is expected behavior. QnA works on the 'one question (and related, similar questions)' to one answer idea, and expects unique questions for the queries. The QnA documentation states:
A knowledge base consists of question and answer (QnA) sets. Each set has one answer and a set contains all the information associated with that answer. An answer can loosely resemble a database row or a data structure instance.
The required settings in a question-and-answer (QnA) set are:
a question - text of user query, used to QnA Maker's machine-learning, to align with text of user's question with different wording but the same answer
the answer - the set's answer is the response that's returned when a user query is matched with the associated question
Each set is represented by an ID.
The optional settings for a set include:
Alternate forms of the question - this helps QnA Maker return the correct answer for a wider variety of question phrasings
Metadata: Metadata are tags associated with a QnA pair and are represented as key-value pairs. Metadata tags are used to filter QnA pairs and limit the set over which query matching is performed.
Multi-turn prompts, used to continue a multi-turn conversation
QnA Maker doesn't differentiate between the two questions because it isn't two questions. It's literally the same question with two different answers.
This particular case would be a good use of QnAMaker's multi-turn prompt feature, where, after the customer has put in the query 'The machine won't start', QnA can follow up with a prompt that says "Which machine did you mean? Machine A or Machine B", and whichever they choose leads to the correct answer. I would look into Multi-Turn Conversations for your knowledgebase.
I have been working with the Microsoft Bot Framework v4 and QnA Maker(GA). A problem that I have come across is when the user types a keyword like 'leave absence'. There are 10+ kind of leave absence questions. The QnAMaker will send back the one with the highest score no matter what kind leave it is (not the right answer).
I have a tree to answer question that looks something like this:
Leave of absence
Parental leave
Maternity leave
Care leave
etc.
Each kind can have one or more related questions and a leave can also have a sub-leave.
When the user ask 'leave absence', the bot should answer: 'Which kind of leave absence' and after the user can ask a question about it.
When the user ask 'How many days can I have for a parental leave', the bot should answer straight from the QnA: 'You can have 10 free days'.
My question is, how can I implement this in v4 so the user can receive the right answer? Is LUIS a option for this? Any suggestions?
Thank you.
Its difficult if you have question after question to ask the user. For this, you may need to have a separate Dialog class with a
List<string>
for the set of questions built on runtime of course. At the end it could return back to the original Dialog class. I have implemented something similar for job openings on different posts. Each post having its own set of questions. The control remains in this QuestionnaireDialog (the separate Dialog class) asking next question once the user answers the current question. I don't think QnA Maker will help on this. I have not seen QnA maker much nor v4. I have done the above on v3 and the intent-response mapping was in a database table.
My suggestion is to flatten your structure if possible from multiple levels to just 2-level to avoid the tree.
For eg:
Leaves --> Care Leave --> Medical Care Leave
--> Family Care Leave
Change the structure to
Leaves --> Medical Care Leave
--> Family Care Leave
So that you could manage it with LUIS entities. Simply asking about leaves will bring a response that will have all the type of leaves available and asking specifically about a leave type will bring a different response specific to the type. Again I have done something similar without QnA maker in v3. If you can't flatten the structure then you will probably have to bring in a mixture of the two approaches because you want to respond to user's specific leave type query (LUIS entities) and take user through a questionnaire.
Recently I created a LUIS App for testing and see how it works.
As you can see in the next image, the Intent "Actividades" (sorry about spanish words) has all that utterances.
When I train LUIS and then test with words like "act" or "acti" it returns the Intent Actividades, which is fine. But, if I test the words "actividade" or "activida" (for example) it returns a None Intent (which has no utterrances).
Why does LUIS do this?
Publish the app, test the endpoint, then go to "Review endpoint utterances" in Luis.ai. The utterances that are mapped to None are probably there, indicated that LUIS wants to know which intent they should be mapped to. Map them to the correct intent, train, and publish again. Test the corrected version. It should be correct now.
I suggest you note Microsoft's advice that you should always populate the 'None' intent with a collection of utterances that are completely off-topic, and the count of these should amount to at least 10% of all the other utterances... and as much as 20%. That's referred to in a number of places. They say if you leave None unpopulated, you'll possibly get the kind of response you describe here.
We are developing Library BOT using Microsoft BOT.
Here We have created one Intent BookSearch, and Entity BookName, BookAuthor.
We trained LUIS with Simple question,but he works only matching questions.
Ex. I trained LUIS like "I need book", so its works properly
But with Same question we write "I need a book", its doesn't understand to match with book intent.
Can anyone help us here? Like that so many scenario where we found LUIS only works with exact matching questions.
One More Problem, We have Book name, with Three Work, unable to tag three words as a bookname entity.
It sounds like your model just needs more training with a variety of sentence structures.
LUIS will match the exact intent when it's been trained but needs more examples to get better with novel utterances. So "I need book" vs "I need a book" should be pretty easy for it to learn with more properly labeled utterances.
As for the title with three words, highlighting them all by clicking and dragging across all three is possible.
You have to write more possible questions which user can ask regarding book?
Let me give an example to explain this...
I need book
I need a book
What latest book you have?
Can you recommend a book to me
I am looking for book
I am looking for a book
In above examples - Intent should be - FindBook
I am looking for C# book?
In above example - Intent should be - FindBook, here user has mentioned the Subject (C#) as well. Subject will be entity.
I am looking for C# book written by Joseph Albahari, Ben Albahari
In above example - Intent Should be one - FindBook, here user has mentioned the Subject and Writer.
Subject and Writer will be entity.
You have to train your model and feed more possible questions, then only LUIS will work perfectly.
You can highlight full sentence as well.