Sinhala entity classifications - rasa-nlu

Question

UserWarning: Misaligned entity annotation in message ‘කමල්’ with intent ‘username’. Make sure the start and end values of entities in the training data match the token boundaries (e.g. entities don’t include trailing whitespaces or punctuation).Make sure the start and end values of entities in the training data match the token boundaries (e.g. entities don’t include trailing whitespaces or punctuation).
I am using Sinhala language and also I am using rasa open source .
This is my nlu part
{
"text": "කමල්",
"intent": "username",
"entities": [
{
"start": 1,
"end": 8,
"value": "කමල්",
"entity": "uname"
}
]
},
Config.yml
# Configuration for Rasa NLU.
# https://rasa.com/docs/rasa/nlu/components/
language: en
pipeline:
- name: WhitespaceTokenizer
- name: RegexFeaturizer
- name: LexicalSyntacticFeaturizer
- name: CountVectorsFeaturizer
- name: CountVectorsFeaturizer
analyzer: "char_wb"
min_ngram: 1
max_ngram: 4
- name: DIETClassifier
epochs: 100
- name: EntitySynonymMapper
- name: ResponseSelector
epochs: 100
# Configuration for Rasa Core.
# https://rasa.com/docs/rasa/core/policies/
policies:
- name: MemoizationPolicy
- name: TEDPolicy
max_history: 5
epochs: 100
- name: MappingPolicy

Answer 1

You're using the WhitespaceTokenizer which will split the text into tokens if there is a space between characters. It seems that the text that you provide (apologies, I do not recognize the language) does not split tokens using space. That is why the entire text is seen as a single token.
It looks like you may need to find another tokenizer for your language. I don't know which tokenizer might apply though. Feel free to contribute to the discussion here.