StanfordNLP Training Steps Verification and loadClassifier check - stanford-nlp

Question

I need help verifying the training steps below and can I add my classifier to -loadClassifier list?
-loadClassifier sample-ner-model.ser.gz, classifiers/english.all.3class.distsim.crf.ser.gz,classifiers/english.conll.4class.distsim.crf.ser.gz,classifiers/english.muc.7class.distsim.crf.ser.gz \
sample.txt
The fate of Lehman Brothers, the beleaguered investment bank, hung in the balance on Sunday as Federal Reserve officials and the leaders of major financial institutions continued to gather in emergency meetings trying to complete a plan to rescue the stricken bank. Several possible plans emerged from the talks, held at the Federal Reserve Bank of New York and led by Timothy R. Geithner, the president of the New York Fed, and Treasury Secretary Henry M. Paulson Jr.
Step 1 Tokenize
java -cp stanford-ner.jar edu.stanford.nlp.process.PTBTokenizer sample.txt > sample.tok
The
fate
of
Lehman
Brothers
,
the
beleaguered
investment
bank
,
hung
in
the
balance
. . .
president
of
the
New
York
Fed
,
and
Treasury
Secretary
Henry
M.
Paulson
Jr.
.
Step 2 Classify
Need a better command to replace EOL "\n" with "\tO\n" . Perl chomp not working. Edited sample.tzv manually.
perl -ne 'chomp; print "$_\tO"' sample.tok > sample.tsv
The 0
fate 0
of 0
Lehman 0
Brothers 0
, 0
the 0
beleaguered 0
investment 0
bank 0
, 0
hung 0
in 0
the 0
balance 0
. . .
president 0
of 0
the 0
New 0
York 0
Fed 0
, 0
and 0
Treasury 0
Secretary 0
Henry 0
M. 0
Paulson 0
Jr. 0
. 0
Step 3 Adjust Properties (sample.prop)
# location of the training file
trainFile = sample.tsv
# location where you would like to save (serialize) your
# classifier; adding .gz at the end automatically gzips the file,
# making it smaller, and faster to load
serializeTo = sample-ner-model.ser.gz
. . .
useTypeySequences=true
wordShape=chris2useLC
Step 4 Modify Gold Standard (sample.tsv)
The 0
fate 0
of 0
Lehman ORG
Brothers ORG
, 0
the 0
beleaguered 0
investment 0
bank 0
, 0
hung 0
in 0
the 0
balance 0
. . .
president 0
of 0
the 0
New ORG
York ORG
Fed ORG
, 0
and 0
Treasury PERS
Secretary PERS
Henry PERS
M. PERS
Paulson PERS
Jr. PERS
. 0
Step 4 Train
java -cp stanford-ner.jar edu.stanford.nlp.ie.crf.CRFClassifier -prop sample.prop
Step 5 Test and Verify
java -cp stanford-ner.jar edu.stanford.nlp.ie.crf.CRFClassifier -loadClassifier sample-ner-model.ser.gz -testFile sample.tsv
Production Maybe:
java -mx1g edu.stanford.nlp.ie.NERClassifierCombiner -textFile sample.txt -ner.model \
-loadClassifier classifiers/english.all.3class.distsim.crf.ser.gz,classifiers/english.conll.4class.distsim.crf.ser.gz,classifiers/english.muc.7class.distsim.crf.ser.gz \
-outputFormat tabbedEntities -textFile sample.txt > sampleNew.tsv

Answer 1

This seems correct to me.
Yes, if you build a new model with Stanford CoreNLP you can just add it into the list.
Note that the models are run in order, and earlier NER taggers in the list tag first, and then later models cannot overwrite the tags (e.g. ORG, PER) written by previous ones (except O of course). So basically where you put the models matters, closer to the front takes priority.
Also ner.combinationMode = HIGH_RECALL will allow every classifier in the list to apply all of their tags. ner.combinationMode = NORMAL means only the first classifier that applies a tag (e.g. ORG, PER) can apply it. You can set ner.combinationMode in the .prop file.