Concurrent requests to Stanford CoreNLP server don't scale - stanford-nlp

Question

I am running the local Stanford CoreNLP server and I am trying to simulate the load by creating simultaneous POST requests. I noticed that the processing time increases linearly with the number of "users" that send requests to the server. The threads option is set to 8. Am I missing something? I feel like going from 1 user/process to 2 should not have such an impact...
Stanford CoreNLP with this command:
nohup java -mx10g -Dorg.slf4j.simpleLogger.defaultLogLevel=error -cp "${USER_HOME}/stanford-corenlp/*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -threads 8 -serverProperties "${USER_HOME}/NLP/dependencyparser/dependencyparser/tools.properties" -port 9000 -timeout 300000 -preload > ${USER_HOME}/log/StanfordCoreNLPServer.log 2>&1 & disown
To simulate simultaneous requests I am using multiprocessing in python to send requests asynchronously. Each process will send the same request 5 times.
import requests
import pandas as pd
import numpy as np
import multiprocessing
import time
def send_stanford_request(batch_id):
all_data = ['This is sample Sentence 1. \nThis is sample Sentence 2. \nThis is sample Sentence 3']
num_requests=5
t0 = time.time()
for ii in range(num_requests):
rr = requests.post('http://[::]:9000/?properties={"outputFormat":"text"}', all_data)
t1 = time.time()
time_batch = t1 - t0
out_res = {'batch_id':batch_id,'total_batch_time':time_batch,
'batch_time_per_req':time_batch/num_requests}
return out_res
Here I create multiple processes that send requests at the same time:
def multi_requests(nprocs):
tt0 = time.time()
process = multiprocessing.Pool(processes=nprocs)
out_data = process.map(send_stanford_request,list(range(1,nprocs+1)))
process.close()
tt1 = time.time()
full_run_time = tt1-tt0
print ("Processing complete with {} processes".format(nprocs))
print ("Total time: {}".format((tt1-tt0)))
return out_data,full_run_time
The main program:
if __name__=="__main__":
total_time_list = []
for nreq in range(1,11,1):
out_nreq,full_run_time = multi_requests(nreq)
total_time_list.append(full_run_time)
print(total_time_list)
Output:
[25.180917024612427, 50.08782601356506, 75.14966297149658,
100.1421709060669, 125.16093802452087, 150.2395520210266,
175.24192595481873, 200.2490758895874, 225.28618001937866,
250.2914171218872]
Properies file:
annotators = tokenize,ssplit,pos,lemma,ner,parse,mention,dcoref
depparse.extradependencies = MAXIMAL
depparse.model = edu/stanford/nlp/models/parser/nndep/english_SD.gz
outputExtension = .out
parse.model = edu/stanford/nlp/models/lexparser/englishPCFG.ser.gz
ssplit.eolonly = true
ssplit.newlineIsSentenceBreak = always

Answer 1

I'm not getting the same result when I do this:
import multiprocessing
import time
from stanza.server import CoreNLPClient
filename="1000.txt"
lines = open(filename).readlines()
def annotate(batch_id):
with CoreNLPClient(annotators=['tokenize','ssplit','pos','lemma','depparse','ner'],
start_server=False,
timeout=60000) as client:
# submit the request to the server
for i, line in enumerate(lines):
ann = client.annotate(line)
def multi_requests(nprocs):
tt0 = time.time()
process = multiprocessing.Pool(processes=nprocs)
out_data = process.map(annotate,list(range(1,nprocs+1)))
process.close()
tt1 = time.time()
full_run_time = tt1-tt0
print ("Processing complete with {} processes".format(nprocs))
print ("Total time: {}".format((tt1-tt0)))
return out_data,full_run_time
for i in range(5):
multi_requests(i+1)
I started my server locally with this:
java -Xmx5G edu.stanford.nlp.pipeline.StanfordCoreNLPServer -port 9000 -timeout 60000 -threads 5 -maxCharLength 100000 -quiet True -preload tokenize,ssplit,pos,lemma,parse,ner,depparse -parse.model edu/stanford/nlp/models/lexparser/englishPCFG.ser.gz
This is the timing information I get:
Processing complete with 1 processes
Total time: 57.21682000160217
Processing complete with 2 processes
Total time: 61.54398465156555
Processing complete with 3 processes
Total time: 68.56325793266296
Processing complete with 4 processes
Total time: 77.91925978660583
Processing complete with 5 processes
Total time: 86.51665258407593
The server is going to get a little slower when there are multiple requests. Certainly not 5x, though.
When I ran your code, I got:
Processing complete with 1 processes
Total time: 0.362884521484375
Processing complete with 2 processes
Total time: 0.40112733840942383
Processing complete with 3 processes
Total time: 0.4257957935333252
Processing complete with 4 processes
Total time: 0.4603590965270996
Processing complete with 5 processes
Total time: 0.4904172420501709
I don't know how you got 25s per thread when running 3 short sentences worth of text.