Stanford NER custom model accuracy testing - stanford-nlp

Question

I am working on Entity extraction using a custom model. I trained my CRF based model on a large dataset as
java -Xmx16g stanford-ner.jar edu.stanford.nlp.ie.crf.CRFClassifier -prop ner.prop
using these features
Property file (ner.prop)
trainFile = training_data_IOB.tsv
#serializeTo = ner-model.ser.gz
map = word=0,answer=1
useClassFeature=true
useWord=true
qnSize=10
entitySubclassification=IOB1
retainEntitySubclassification=true
mergeTags=true
useNGrams=true
noMidNGrams=true
maxNGramLeng=6
usePrev=true
useNext=true
useSequences=true
usePrevSequences=true
maxLeft=1
useTypeSeqs=true
useTypeSeqs2=true
useTypeySequences=true
wordShape=chris2useLC
useDisjunctive=true
useGazettes=true
gazette=gazetter.txt
sloppyGazette=true
Training file (training_data_IOB.tsv)
Thousands O
of O
demonstrators O
have O
marched O
through O
London B-LOC
to O
protest O
the O
war O
in O
Iraq B-LOC
... ...
Gazette file(gazetter.txt)
B-LOC Iraq
B-LOC Afghanistan
B-ORG Congressional
B-LOC Bangladesh
B-LOC Canada
B-ORG ...
the new model is created as ner-model.ser.gz and working quite well.
Now my question is, How I can calculate its percentage accuracy on any unseen(new) data without any manual counting and calculations??
I'm new in this field so kindly post a detailed descriptive answer. Thanks for your time.

Answer 1

If you create a conll file with the gold tags for a test set you can use this command and it will output scores (this example runs our model, replace with your custom model):
java -Xmx2g edu.stanford.nlp.ie.crf.CRFClassifier -loadclassifier edu/stanford/nlp/models/ner/english.all.3class.distsim.crf.ser.gz -testFile testData.conll