SUTime custom date parsing - stanford-nlp

Question

Need to parse a sentence like "Bob was born on Jan fifteen nineteen seventy nine." and extract the date. How do I create a new rule to handle the date expression?
If I use "Bob was born on Jan fifteenth nineteen seventy nine." the parser extracts the correct date 01/15/1979. Simply changing "fifteenth" to "fifteen" leads to incorrect parsing.
val input = "Bob was born on Jan fifteenth nineteen seventy nine."
val document = new CoreDocument(input)
val props = new Properties()
val annotators_ner = "tokenize,ssplit,pos,lemma,ner"
props.setProperty("annotators", annotators_ner)
val pipeline = new StanfordCoreNLP(props)
pipeline.annotate(document)
val writer = new StringWriter
pipeline.jsonPrint(document.annotation(), writer);
val json = writer.toString()
println(json)
The json excerpt below shows correct entity mentions.
"entitymentions": [
{
"docTokenBegin": 0,
"docTokenEnd": 1,
"tokenBegin": 0,
"tokenEnd": 1,
"text": "Bob",
"characterOffsetBegin": 0,
"characterOffsetEnd": 3,
"ner": "PERSON"
},
{
"docTokenBegin": 4,
"docTokenEnd": 9,
"tokenBegin": 4,
"tokenEnd": 9,
"text": "Jan fifteenth nineteen seventy nine",
"characterOffsetBegin": 16,
"characterOffsetEnd": 51,
"ner": "DATE",
"normalizedNER": "1979-01-15",
"timex": {
"tid": "t1",
"type": "DATE",
"value": "1979-01-15"
}
}
]
Changing the input as follows.
val input = "Bob was born on Jan fifteen nineteen seventy nine."
Leads to the following output for entity mentions.
"entitymentions": [
{
"docTokenBegin": 0,
"docTokenEnd": 1,
"tokenBegin": 0,
"tokenEnd": 1,
"text": "Bob",
"characterOffsetBegin": 0,
"characterOffsetEnd": 3,
"ner": "PERSON"
},
{
"docTokenBegin": 4,
"docTokenEnd": 7,
"tokenBegin": 4,
"tokenEnd": 7,
"text": "Jan fifteen nineteen",
"characterOffsetBegin": 16,
"characterOffsetEnd": 36,
"ner": "DATE",
"normalizedNER": "1519-01",
"timex": {
"tid": "t1",
"type": "DATE",
"value": "1519-01"
}
},
{
"docTokenBegin": 7,
"docTokenEnd": 9,
"tokenBegin": 7,
"tokenEnd": 9,
"text": "seventy nine",
"characterOffsetBegin": 37,
"characterOffsetEnd": 49,
"ner": "NUMBER",
"normalizedNER": "79.0"
}
]

Answer 1

This is complicated because I think the rules are recognizing "Jan fifteen nineteen" as January 1519. One thing you could try is adding some post-processing rules if you find a common pattern such as "DATE" followed by "NUMBER" is actually a written out "DATE". But you could imagine scenarios such as "In Jan Fifteen Nineteen Seventy Nine people visited".
There is a full write up about how to add TokensRegex rules to the ner pipeline here: https://stanfordnlp.github.io/CoreNLP/ner.html
and info on writing tokensregex rules here: https://stanfordnlp.github.io/CoreNLP/tokensregex.html