Rasa nlu parse request giving wrong intent result - rasa-nlu

Question

Rasa NLU version (e.g. 0.7.3): 0.10.0a6
Used backend / pipeline (mitie, spacy_sklearn, ...): ["nlp_spacy", "tokenizer_spacy", "intent_featurizer_spacy","ner_crf", "ner_synonyms", "intent_classifier_sklearn","ner_spacy"]
Operating system (windows, osx, ...): Windows server 2012 R2
Issue: I have installed Rasa nlu 0.10.0a6 version. My confi_spacy file looks like.
{
"project":"Project",
"pipeline" : ["nlp_spacy", "tokenizer_spacy", "intent_featurizer_spacy","ner_crf", "ner_synonyms", "intent_classifier_sklearn","ner_spacy"],
"path" : "./projects",
"cors_origins":["*"],
"data" : "./data/examples/rasa/People.json"
}
and my data file looks like.
{
"rasa_nlu_data": {
"regex_features": [
{
"name": "zipcode",
"pattern": "[0-9]{5}"
}
],
"entity_synonyms": [
{
"value": "chinese",
"synonyms": ["Chinese", "Chines", "chines"]
},
{
"value": "vegetarian",
"synonyms": ["veggie", "vegg"]
}
],
"common_examples": [
{
"text": "hey",
"intent": "greet",
"entities": []
},
{
"text": "howdy",
"intent": "greet",
"entities": []
},
{
"text": "hey there",
"intent": "greet",
"entities": []
},
{
"text": "hello",
"intent": "greet",
"entities": []
},
{
"text": "hi",
"intent": "greet",
"entities": []
},
{
"text": "good morning",
"intent": "greet",
"entities": []
},
{
"text": "good evening",
"intent": "greet",
"entities": []
},
{
"text": "dear sir",
"intent": "greet",
"entities": []
},
{
"text": "yes",
"intent": "affirm",
"entities": []
},
{
"text": "yep",
"intent": "affirm",
"entities": []
},
{
"text": "yeah",
"intent": "affirm",
"entities": []
},
{
"text": "indeed",
"intent": "affirm",
"entities": []
},
{
"text": "that's right",
"intent": "affirm",
"entities": []
},
{
"text": "ok",
"intent": "affirm",
"entities": []
},
{
"text": "great",
"intent": "affirm",
"entities": []
},
{
"text": "right, thank you",
"intent": "affirm",
"entities": []
},
{
"text": "correct",
"intent": "affirm",
"entities": []
},
{
"text": "great choice",
"intent": "affirm",
"entities": []
},
{
"text": "sounds really good",
"intent": "affirm",
"entities": []
},
{
"text": "i'm looking for a place to eat",
"intent": "restaurant_search",
"entities": []
},
{
"text": "I want to grab lunch",
"intent": "restaurant_search",
"entities": []
},
{
"text": "I am searching for a dinner spot",
"intent": "restaurant_search",
"entities": []
},
{
"text": "i'm looking for a place in the north of town",
"intent": "restaurant_search",
"entities": [
{
"start": 31,
"end": 36,
"value": "north",
"entity": "location"
}
]
},
{
"text": "show me chinese restaurants",
"intent": "restaurant_search",
"entities": [
{
"start": 8,
"end": 15,
"value": "chinese",
"entity": "cuisine"
}
]
},
{
"text": "show me chines restaurants",
"intent": "restaurant_search",
"entities": [
{
"start": 8,
"end": 14,
"value": "chinese",
"entity": "cuisine"
}
]
},
{
"text": "show me a mexican place in the centre",
"intent": "restaurant_search",
"entities": [
{
"start": 31,
"end": 37,
"value": "centre",
"entity": "location"
},
{
"start": 10,
"end": 17,
"value": "mexican",
"entity": "cuisine"
}
]
},
{
"text": "i am looking for an indian spot called olaolaolaolaolaola",
"intent": "restaurant_search",
"entities": [
{
"start": 20,
"end": 26,
"value": "indian",
"entity": "cuisine"
}
]
}, {
"text": "search for restaurants",
"intent": "restaurant_search",
"entities": []
},
{
"text": "anywhere in the west",
"intent": "restaurant_search",
"entities": [
{
"start": 16,
"end": 20,
"value": "west",
"entity": "location"
}
]
},
{
"text": "anywhere near 18328",
"intent": "restaurant_search",
"entities": [
{
"start": 14,
"end": 19,
"value": "18328",
"entity": "location"
}
]
},
{
"text": "I am looking for asian fusion food",
"intent": "restaurant_search",
"entities": [
{
"start": 17,
"end": 29,
"value": "asian fusion",
"entity": "cuisine"
}
]
},
{
"text": "I am looking a restaurant in 29432",
"intent": "restaurant_search",
"entities": [
{
"start": 29,
"end": 34,
"value": "29432",
"entity": "location"
}
]
},
{
"text": "I am looking for mexican indian fusion",
"intent": "restaurant_search",
"entities": [
{
"start": 17,
"end": 38,
"value": "mexican indian fusion",
"entity": "cuisine"
}
]
},
{
"text": "central indian restaurant",
"intent": "restaurant_search",
"entities": [
{
"start": 0,
"end": 7,
"value": "central",
"entity": "location"
},
{
"start": 8,
"end": 14,
"value": "indian",
"entity": "cuisine"
}
]
},
{
"text": "bye",
"intent": "goodbye",
"entities": []
},
{
"text": "goodbye",
"intent": "goodbye",
"entities": []
},
{
"text": "good bye",
"intent": "goodbye",
"entities": []
},
{
"text": "stop",
"intent": "goodbye",
"entities": []
},
{
"text": "end",
"intent": "goodbye",
"entities": []
},
{
"text": "farewell",
"intent": "goodbye",
"entities": []
},
{
"text": "Bye bye",
"intent": "goodbye",
"entities": []
},
{
"text": "have a good one",
"intent": "goodbye",
"entities": []
}
]
}
}
Using above config and json data I have trained Rasa using below HTTP end point
/train?project=Project
After training data there in one Project folder created with trained data.
And I started Rasa server with below command.
python -m rasa_nlu.server -c config_spacy.json
Now server started with port 5000.
When I can execute '/parse?q=hello&project=Project' parse end point am getting below response.
{
"intent": {
"name": "greet",
"confidence": 0.6409561289105246
},
"entities": [],
"intent_ranking": [
{
"name": "greet",
"confidence": 0.6409561289105246
},
{
"name": "goodbye",
"confidence": 0.16788352870824252
},
{
"name": "restaurant_search",
"confidence": 0.10908268742176423
},
{
"name": "affirm",
"confidence": 0.08207765495946878
}
],
"text": "hello"
}
When I can execute '/parse?q=Great choice&project=Project' parse end point am getting below response.
{
"intent": {
"name": "affirm",
"confidence": 0.7718580601897227
},
"entities": [],
"intent_ranking": [
{
"name": "affirm",
"confidence": 0.7718580601897227
},
{
"name": "goodbye",
"confidence": 0.11611828257295627
},
{
"name": "greet",
"confidence": 0.07060567364272623
},
{
"name": "restaurant_search",
"confidence": 0.04141798359459499
}
],
"text": "Great choice"
}
When I can execute '/parse?q=Great choice&Project=Project' parse end point am getting below response.
{
"intent": {
"name": "None",
"confidence": 1
},
"entities": [],
"text": "Great choice"
}
When I can execute "/parse?q=Book a cab&project=Project" parse end point am getting below response.
{
"intent": {
"name": "goodbye",
"confidence": 0.40930529216955336
},
"entities": [],
"intent_ranking": [
{
"name": "goodbye",
"confidence": 0.40930529216955336
},
{
"name": "restaurant_search",
"confidence": 0.31818118919270977
},
{
"name": "greet",
"confidence": 0.20524111006007764
},
{
"name": "affirm",
"confidence": 0.06727240857765926
}
],
"text": "Book a cab"
}
In this way for each request it is responding with proper results some times and sometime not. If you can observe Parse_reponce2.txt and Parse_reponce3.txt in these both the response I just changed from small 'p' to capital 'P' in place 'Project' because of this change, i got different results for each request.
And in trained data there is no 'Book a cab' text or any related intent to it. But when I send parse using this text I am not getting None intent, it is returning intent result. for any parse request with any random am not getting None intent.
Is this my training issue or what went wrong in it.
Please let me know how can I get proper intent results as well as entities result.
Content of configuration file (if used & relevant):
{
"project":"Project",
"pipeline" : ["nlp_spacy", "tokenizer_spacy", "intent_featurizer_spacy","ner_crf", "ner_synonyms", "intent_classifier_sklearn","ner_spacy"],
"path" : "./projects",
"cors_origins":["*"],
"data" : "./data/examples/rasa/People.json"
}

Answer 1

URL parameters are case sensitive, which is why the two great choice examples had different outputs. In the 2nd case Rasa wasn't finding a project/model to parse against.
Rasa NLU will always return a matched intent. So in the last example you can see that it returned an intent, but the confidence was low. Handling this is what would be called a fallback or out of scope. The two primary ways discussed for handling fallback is to implement logic that takes over when the confidence falls below a certain threshold or to train an actual fallback intent that has all the non-intent examples you want to catch.

Answer 2

I had the same issue with using Rasa NLU where I had around 120 examples for 5 different intents, and 5-7 entities as well. Here, you seem to have used the spacy-sklearn pipeline.
spaCy typically requires more data to train and detect intents and (more so) entities. The docs say 500-1000 examples would be considered "decent and good" for the library.
In my case I changed my pipeline to MITIE-sklearn and I got a decent model trained with only 80 examples and same number of intents as before.
Also spaCy tends to be faster as you would have noted, but MITIE does take around 6 minutes for an 80-example model.