About the Stanford CoreNLP in chinese model - stanford-nlp

Question

How to use the chinese model, and I download the "stanford-corenlp-3.5.2-models-chinese.jar" in my classpath and I copy
<dependency>
<groupId>edu.stanford.nlp</groupId>
<artifactId>stanford-corenlp</artifactId>
<version>3.5.2</version>
<classifier>models-chinese</classifier>
</dependency>
to pom.xml file. In additional, my input.txt is
因出席中國大陸閱兵引發爭議的國民黨前主席連戰今晚金婚宴，立法院長王金平說，已向連戰恭喜，等一下回南部。
連戰夫婦今晚的50週年金婚紀念宴，正值連戰赴陸出席閱兵引發爭議之際，社會關注會否受到影響。
包括國民黨主席朱立倫、副主席郝龍斌等人已分別對外表示另有行程，無法出席。
then I compile the program using the code
java -cp "*" -Xmx1g edu.stanford.nlp.pipeline.StanfordCoreNLP -props StanfordCoreNLP-chinese.properties -annotators segment,ssplit -file input.txt
and the result is as follows. But it gives the following error and how do i solve this problem?
C:\stanford-corenlp-full-2015-04-20>java -cp "*" -Xmx1g edu.stanford.nlp.pipelin
e.StanfordCoreNLP -props StanfordCoreNLP-chinese.properties -annotators segment,
ssplit -file input.txt
Registering annotator segment with class edu.stanford.nlp.pipeline.ChineseSegmen
terAnnotator
Adding annotator segment
Loading Segmentation Model ... Loading classifier from edu/stanford/nlp/models/s
egmenter/chinese/ctb.gz ... Loading Chinese dictionaries from 1 file:
edu/stanford/nlp/models/segmenter/chinese/dict-chris6.ser.gz
Done. Unique words in ChineseDictionary is: 423200.
done [22.9 sec].
Ready to process: 1 files, skipped 0, total 1
Processing file C:\stanford-corenlp-full-2015-04-20\input.txt ... writing to C:\
stanford-corenlp-full-2015-04-20\input.txt.xml {
Annotating file C:\stanford-corenlp-full-2015-04-20\input.txt Adding Segmentat
ion annotation ... INFO: TagAffixDetector: useChPos=false | useCTBChar2=true | u
sePKChar2=false
INFO: TagAffixDetector: building TagAffixDetector from edu/stanford/nlp/models/s
egmenter/chinese/dict/character_list and edu/stanford/nlp/models/segmenter/chine
se/dict/in.ctb
Loading character dictionary file from edu/stanford/nlp/models/segmenter/chinese
/dict/character_list
Loading affix dictionary from edu/stanford/nlp/models/segmenter/chinese/dict/in.
ctb
?]?X?u????j???\?L??o??????????e?D?u?s???????B?b?A??k?|???????????A?w?V?s?????A?
??#?U?^?n???C
?s?????????50?g?~???B?????b?A????s??u???X?u?\?L??o???????A???|???`?|?_????v?T?C
?]?A?????D?u?????B??D?u?q?s?y???H?w???O??~???t????{?A?L?k?X?u?C
--->
[?, ], ?, X, ?u????j???, \, ?L??o??????????e?D?u?s???????B?b?A??k?|???????????A?
w?V?s?????A???#?U?^?n???C, , , , ?s?????????, 50, ?, g?, ~, ???B?????b?A????s??u
???X?u?, \, ?L??o???????A???, |, ???, `, ?, |, ?_????v?T?C, , , , ?, ], ?, A????
?D?u???, ??, B??D?u?q, ?, s?y???H?w???O??, ~, ???t????, {, ?, A?L?k?X?u?C]
}
Processed 1 documents
Skipped 0 documents, error annotating 0 documents
Annotation pipeline timing information:
ChineseSegmenterAnnotator: 0.1 sec.
TOTAL: 0.1 sec. for 34 tokens at 485.7 tokens/sec.
Pipeline setup: 0.0 sec.
Total time for StanfordCoreNLP pipeline: 0.1 sec.

Answer 1

I edited your question to change the command to the one that you actually used to produce the output shown. It looks like you worked out that the former command:
java -cp "*" -Xmx1g edu.stanford.nlp.pipeline.StanfordCoreNLP -annotators tokenize,ssplit,pos,lemma,ner,parse,dcoref -file input.txt input.xml
ran the English analysis pipeline, and that didn't work very well for Chinese text....
The CoreNLP support of Chinese in v3.5.2 is still a little rough, and will hopefully be a bit smoother in the next release. But from here you need to:
Specify a properties file for Chinese, giving appropriate models. (If no properties file is specified, CoreNLP defaults to English): -props StanfordCoreNLP-chinese.properties
At present, word segmentation of Chinese is not the annotator tokenize, but segment, specified as a custom annotator in StanfordCoreNLP-chinese.properties. (Maybe we'll unify the two in a future release...)
The current dcoref annotator only works for English. There is Chinese coreference, but it is not fully integrated into the pipeline. If you want to use it, you currently have to write some code, as explained here. So let's delete it. (Again, this should be better integrated in the future).
At that point, things run, but the ugly stderr output you show is that by default the segmenter has VERBOSE turned on, but your output character encoding is not right for our Chinese output. We should have VERBOSE off by default, but you can turn it off with: -segment.verbose false
We have no Chinese lemmatizer, so may as well delete that annotator.
Also, CoreNLP needs more than 1GB of RAM. Try 2GB.
At this point, all should be good! With the command:
java -cp "*" -Xmx2g edu.stanford.nlp.pipeline.StanfordCoreNLP -props StanfordCoreNLP-chinese.properties -annotators segment,ssplit,pos,ner,parse -segment.verbose false -file input.txt
you get the output in input.txt.xml. (I'm not posting it, since it's a couple of thousand lines long....)
Update for CoreNLP v3.8.0: If using the (current in 2017) CoreNLP v3.8.0, then there are some changes/progress: (i) We now use the annotator tokenize for all languages and it doesn't require loading a custom annotator for Chinese; (ii) verbose segmentation is correctly turned off by default; (iii) [negative progress] the requirements require the lemma annotator prior to ner, even though it is a no-op for Chinese; and (iv) coreference is now available for Chinese, invoked as coref, which requires the prior annotator `mentions, and its statistical models require considerable memory. Put that all together, and you're now good with this command:
java -cp "*" -Xmx3g edu.stanford.nlp.pipeline.StanfordCoreNLP -props StanfordCoreNLP-chinese.properties -annotators tokenize,ssplit,pos,lemma,ner,parse,mention,coref -file input.txt