Akka HTTP REST API for producing to Kafka Performance - performance

Question

I'm building an API with Akka that should produce to a Kafka bus. I have been load testing the application using Gatling. Noticed that when more than 1000 users are created in Gatling, the API starts to struggle. On average, about 170 requests per second are handled, which seems like very little to me.
The API's main entry point is this:
import akka.actor.{Props, ActorSystem}
import akka.http.scaladsl.Http
import akka.http.scaladsl.model._
import akka.pattern.ask
import akka.http.scaladsl.server.Directives
import akka.http.scaladsl.unmarshalling.Unmarshaller
import akka.stream.ActorMaterializer
import com.typesafe.config.{Config, ConfigFactory}
import play.api.libs.json.{JsObject, Json}
import scala.concurrent.{Future, ExecutionContext}
import akka.http.scaladsl.server.Directives._
import akka.util.Timeout
import scala.concurrent.duration._
import ExecutionContext.Implicits.global
case class PostMsg(msg:JsObject)
case object PostSuccess
case class PostFailure(msg:String)
class Msgapi(conf:Config) {
implicit val um:Unmarshaller[HttpEntity, JsObject] = {
Unmarshaller.byteStringUnmarshaller.mapWithCharset { (data, charset) =>
Json.parse(data.toArray).asInstanceOf[JsObject]
}
}
implicit val system = ActorSystem("MsgApi")
implicit val timeout = Timeout(5 seconds)
implicit val materializer = ActorMaterializer()
val router = system.actorOf(Props(new RouterActor(conf)))
val route = {
path("msg") {
post {
entity(as[JsObject]) {obj =>
if(!obj.keys.contains("key1") || !obj.keys.contains("key2") || !obj.keys.contains("key3")){
complete{
HttpResponse(status=StatusCodes.BadRequest, entity="Invalid json provided. Required fields: key1, key2, key3 \n")
}
} else {
onSuccess(router ? PostMsg(obj)){
case PostSuccess => {
complete{
Future{
HttpResponse(status = StatusCodes.OK, entity = "Post success")
}
}
}
case PostFailure(msg) =>{
complete{
Future{
HttpResponse(status = StatusCodes.InternalServerError, entity=msg)
}
}
}
case _ => {
complete{
Future{
HttpResponse(status = StatusCodes.InternalServerError, entity = "Unknown Server error occurred.")
}
}
}
}
}
}
}
}
}
def run():Unit = {
Http().bindAndHandle(route, interface = conf.getString("http.host"), port = conf.getInt("http.port"))
}
}
object RunMsgapi {
def main(Args: Array[String]):Unit = {
val conf = ConfigFactory.load()
val api = new Msgapi(conf)
api.run()
}
}
The router actor is as follows:
import akka.actor.{ActorSystem, Props, Actor}
import akka.http.scaladsl.server.RequestContext
import akka.routing.{Router, SmallestMailboxRoutingLogic, ActorRefRoutee}
import com.typesafe.config.Config
import play.api.libs.json.JsObject
class RouterActor(conf:Config) extends Actor{
val router = {
val routees = Vector.tabulate(conf.getInt("kafka.producer-number"))(n => {
val r = context.system.actorOf(Props(new KafkaProducerActor(conf, n )))
ActorRefRoutee(r)
})
Router(SmallestMailboxRoutingLogic(), routees)
}
def receive = {
case PostMsg(msg) => {
router.route(PostMsg(msg), sender())
}
}
}
And finally, the kafka producer actor:
import akka.actor.Actor
import java.util.Properties
import com.typesafe.config.Config
import kafka.message.NoCompressionCodec
import kafka.utils.Logging
import org.apache.kafka.clients.producer._
import play.api.libs.json.JsObject
import scala.concurrent.duration._
import scala.concurrent.{ExecutionContext, Future, Await}
import ExecutionContext.Implicits.global
import scala.concurrent.{Future, Await}
import scala.util.{Failure, Success}
class KafkaProducerActor(conf:Config, id:Int) extends Actor with Logging {
var topic: String = conf.getString("kafka.topic")
val codec = NoCompressionCodec.codec
val props = new Properties()
props.put("bootstrap.servers", conf.getString("kafka.bootstrap-servers"))
props.put("acks", conf.getString("kafka.acks"))
props.put("retries", conf.getString("kafka.retries"))
props.put("batch.size", conf.getString("kafka.batch-size"))
props.put("linger.ms", conf.getString("kafka.linger-ms"))
props.put("buffer.memory", conf.getString("kafka.buffer-memory"))
props.put("key.serializer", conf.getString("kafka.key-serializer"))
props.put("value.serializer", conf.getString("kafka.value-serializer"))
val producer = new KafkaProducer[String, String](props)
def receive = {
case PostMsg(msg) => {
// push the msg to Kafka
try{
val res = Future{
producer.send(new ProducerRecord[String, String](topic, msg.toString()))
}
val result = Await.result(res, 1 second).get()
sender ! PostSuccess
} catch{
case e: Exception => {
println(e.printStackTrace())
sender ! PostFailure("Kafka push error")
}
}
}
}
}
The idea being that in application.conf I can easily specify how many producers there should be, allowing better horizontal scaling.
Now, however, it seems that the api or router is actually the bottleneck. As a test, I disabled the Kafka producing code, and replaced it with a simple: sender ! PostSuccess. With 3000 users in Gatling, I still had 6% of requests failing due to timeouts, which seems like a very long time to me.
The Gatling test I am executing is the following:
import io.gatling.core.Predef._ // 2
import io.gatling.http.Predef._
import scala.concurrent.duration._
class BasicSimulation extends Simulation { // 3
val httpConf = http // 4
.baseURL("http://localhost:8080") // 5
.acceptHeader("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8") // 6
.doNotTrackHeader("1")
.acceptLanguageHeader("en-US,en;q=0.5")
.acceptEncodingHeader("gzip, deflate")
.userAgentHeader("Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0")
.header("Content-Type", "application/json")
val scn = scenario("MsgLoadTest")
.repeat(100)(
pace(2 seconds)
.exec(http("request_1")
.post("/msg").body(StringBody("""{ "key1":"something", "key2": "somethingElse", "key3":2222}""")).asJSON)
)
setUp( // 11
scn.inject(rampUsers(3000) over (5 seconds)) // 12
).protocols(httpConf) // 13
}
update
Following some pointers from cmbaxter, I tried some things (see discussion in comments), and profiled the application using visualvm during the gatling load test. I don't quite know how to interpret these results though. It seems that a lot of time is spent in the ThreadPoolExecutor, but this might be ok?
Two screenshots from the profiling are below:

Answer 1

To exclude the Kafka producer, I removed the logic from the Actor. I was still getting performance issues. So, as a final test, I reworked the API to simply give a direct answer when a POST came in:
val route = {
path("msg") {
post {
entity(as[String]) { obj =>
complete(
HttpResponse(status = StatusCodes.OK, entity = "OK")
)
}
}
}
}
and I implemented the same route in Spray, to compare performance. The results were clear. Akka HTTP (at least in this current test setup) does not come close to Spray's performance. Perhaps there is some tweaking that can be done for Akka HTTP? I have attached two screenshots of response time graphs for 3000 concurrent users in Gatling, making a post request.
Akka HTTP
Spray

Answer 2

I would eliminate the KafkaProducerActor and router completely and call a Scala wrapped version of producer.send directly. Why create a possible bottleneck if not necessary? I could very well imagine the global execution context or the actor system becoming a bottleneck in your current setup.
Something like this should do the trick:
class KafkaScalaProducer(val producer : KafkaProducer[String, String](props)) {
def send(topic: String, msg : String) : Future[RecordMetadata] = {
val promise = Promise[RecordMetadata]()
try {
producer.send(new ProducerRecord[String, String](topic, msg), new Callback {
override def onCompletion(md : RecordMetadata, e : java.lang.Exception) {
if (md == null) promise.success(md)
else promise.failure(e)
}
})
} catch {
case e : BufferExhaustedException => promise.failure(e)
case e : KafkaException => promise.failure(e)
}
promise.future
}
def close = producer.close
}
(note: I have not actually tried this code. It should be interpreted as pseudo-code)
I would then simply transform the result of the future to a HttpResponse.
After that it's a question of tweaking configuration. Your bottleneck is now either the Kafka Producer or Akka Http.