Check for words in input text from words collection - stanford-nlp

Question

I have collection of Noun Phrases say around 10,000 words. I want to check every new input text data for these NP collection and extract those sentences that contains any of these NP. I don't want to run loops for every word because it makes my code dead slow. I am using Java and Stanford CoreNLP.

Answer 1

A quick and easy way to do this is to use RegexNER to identify all examples of anything in your dictionary, and then check for non "O" NER tags in the sentence.
package edu.stanford.nlp.examples;
import edu.stanford.nlp.ling.*;
import edu.stanford.nlp.pipeline.*;
import edu.stanford.nlp.util.*;
import java.util.*;
import java.util.stream.Collectors;
public class FindSentencesWithPhrase {
public static boolean checkForNamedEntity(CoreMap sentence) {
for (CoreLabel token : sentence.get(CoreAnnotations.TokensAnnotation.class)) {
if (token.ner() != null && !token.ner().equals("O")) {
return true;
}
}
return false;
}
public static void main(String[] args) {
Properties props = new Properties();
props.setProperty("annotators", "tokenize,ssplit,pos,lemma,regexner");
props.setProperty("regexner.mapping", "phrases.rules");
StanfordCoreNLP pipeline = new StanfordCoreNLP(props);
String exampleText = "This sentence contains the phrase \"ice cream\"." +
"This sentence is not of interest. This sentences contains pizza.";
Annotation ann = new Annotation(exampleText);
pipeline.annotate(ann);
for (CoreMap sentence : ann.get(CoreAnnotations.SentencesAnnotation.class)) {
if (checkForNamedEntity(sentence)) {
System.out.println("---");
System.out.println(sentence.get(CoreAnnotations.TokensAnnotation.class).
stream().map(token -> token.word()).collect(Collectors.joining(" ")));
}
}
}
}
The file "phrases.rules" should look like this:
ice cream PHRASE_OF_INTEREST MISC 1
pizza PHRASE_OF_INTEREST MISC 1