I'm working on a simple micro service written in Spring Boot. This service will act as a proxy towards another resources that have a hard concurrent connection limit and the requests take a while to process.
I would like to impose a hard limit on concurrent connections allowed to my micro service and rejecting any with either a 503 or on tcp/ip level. I've tried to look into different configurations that can be made for Jetty/Tomcat/Undertow but haven't figured out yet something completely convincing.
I found some settings regulating thread pools:
server.tomcat.max-threads=0 # Maximum amount of worker threads.
server.undertow.io-threads= # Number of I/O threads to create for the worker.
server.undertow.worker-threads= # Number of worker threads.
server.jetty.acceptors= # Number of acceptor threads to use.
server.jetty.selectors= # Number of selector threads to use.
But if understand correctly these are all configuring thread pool sizes and will just result in connections being queued on some level.
This seems like really interesting, but this hasn't been merged in yet and is targeted for Spring Boot 1.5 , https://github.com/spring-projects/spring-boot/pull/6571
Am I out of luck using a setting for now? I could of course implement a filter but would rather block it on an earlier level and not have to reinvent the wheel. I guess using apache or something else in front is also an option, but still that feels like an overkill.
Try to look at EmbeddedServletContainerCustomizer
this gist could give you and idea how to do that.
TomcatEmbeddedServletContainerFactory factory = ...;
factory.addConnectorCustomizers(connector ->
((AbstractProtocol) connector.getProtocolHandler()).setMaxConnections(10000));
Related
I am starting a project using spring webflux reactive stack which by default uses Reactor Netty as the server. Pls correct me if i'm wrong, but i read that Netty can only have maximum number of event loops as the amount of processors on the instance.
This means that if a request gets blocked for a second (which should not be the use case i know, just for example), we would only be able to get max 1 Transaction Per Second if there is only 1 processor on instance.
I am wondering how scalable Netty is compared to servlet container like Tomcat? What are the pros and cons of using Netty vs Tomcat?
I also want to know the ways to optimize Netty configurations to make sure it is production ready.
This means that if a request gets blocked for a second (which should not be the use case i know, just for example)
the whole purpose of this stack is to scale hugely on a limited amount of resources (here, threads). This is all built on the critical requirement that every step is asynchronous and non-blocking.
So your "just for example" doesn't make any sense. Yes, if you block for one second that CPU will only process that single request during that second. That is also completely wrong of you to do so, and everything in the stack is made to help you avoid blocking.
When using a classical Tomcat approach, you can give your server a maximum number of threads it can use to handle web requests from users. Using the Reactive Programming paradigm, and Reactor in Spring 5, we are able to scale better vertically, making sure we are blocked minimally.
It seems to me that it makes this less manageable than the classical Tomcat approach, where you simply define the max number of concurrent requests. When you have a max number of concurrent requests, it's easier to estimate the maximum memory your application will need and scale accordingly. When you use Spring 5's Reactive Programming this seems like more of a hassle.
When I talk about these new technologies to sysadmin friends, they reply with worry about applications running out of RAM, or even threads on the OS level. So how can we deal with this better?
No blocking I/O at ALL
First of all, if you don't have any blocking operation then you should not worry at all about How much Thread should I provide for managing concurrency. In that case, we have only one worker which process all connections asynchronously and nonblockingly. And in that case, we may easily scale connection-servant workers which process all connections without contention and coherence (each worker has its own queue of received connections, each worker works on its own CPU) and we may scale application better in that case (shared nothing design).
Summary: in that case you manage max number of webthread identically as previously, by configuration application-container (Tomcat, WebSphere, etc) or similar in case of non-Servlet servers like Netty, or hybrid Undertow. The benefit - you may process muuuuuuch more users requests but with the same resources consumption.
Blocking Database and Non-Blocking Web API (such as WebFlux over Netty).
In case we should deal somehow with blocking I/O, for an instant communication with DB over blocking JDBC, the most appropriate way to keep your app scalable and efficient as possible we should use dedicated thread-pool for I/O.
Thread-pool requirements
First of all, we should create thread-pool with exactly the same amount of workers as available connections in JDBC connections-pool. Hence, we will have exactly the same amount of thread which will be blockingly wait for the response and we utilize our resources as efficiently as it possible, so no more memory will be consumed for Thread stack as it actually needed (In other word Thread per Connection model).
How to configure thread-pool accordingly to size of connection-pool
Since access to properties is varying for a particular database and JDBC driver, we may always externalize that configuration on a particular property, which in turn means that it may be configured by devops or sysadmin.
A configuration of Threadpool (in our example it is configuring of Scheduler of Project Reactor 3) may looks like next:
#Configuration
public class ReactorJdbcSchedulerConfig {
#Value("my.awasome.scheduler-size")
int schedulerSize;
#Bean
public Scheduler jdbcScheduler() {
return Schedulers.fromExecutor(new ForkJoinPool(schedulerSize));
// similarly
// ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
// taskExecutor.setCorePoolSize(schedulerSize);
// taskExecutor.setMaxPoolSize(schedulerSize);
// taskExecutor.setQueueCapacity(schedulerSize);
// taskExecutor.initialize();
// return Schedulres.fromExecutor(taskExecutor);
}
}
...
#Autowire
Scheduler jdbcScheduler;
public Mono myJdbcInteractionIsolated(String id) {
return Mono.fromCallable(() -> jpaRepo.findById(id))
.subscribeOn(jdbcScheduler)
.publishOn(Schedulers.single());
}
...
As it might be noted, with that technique, we may delegate our shared thread-pool configuration to an external team (sysadmins for an instance) and allows them to manage consumption of memory which is used for created Java Threads.
Keep your blocking I/O thread pool only for I/O work
This statement means that I/O thread should be only for operations which are blockingly waiting. In turn, it means that after the thread has done his awaiting the response, you should move result processing to another thread.
That is why in the above code-snippet I put .publishOn right after .subscribeOn.
So, to summarize, with that technique we may allow external team managing application sizing by controlling thread-pool size to connection-pool size accordingly. All results processing will be executed within one thread and there will be no redundant, uncontrolled memory consumption hence.
Finally, Blocking API (Spring MVC) and blocking I/O (Database access)
In that case, there is no need for reactive paradigm at all since you don't get any profit from that. First of all, Reactive Programming requires particular mind shifting, especially in the understanding of the usage of functional techniques with Reactive libraries such as RxJava or Project Reactor. In turn for non-prepared users, it gives more complexity and causes more "What ****** is going on here???". So, in case of blocking operations from both ends, you should think twice do you really need Reactive Programming here.
Also, there is no magic for free. Reactive Extensions comes with a lot of internal complexity and using all that magical .map, .flatMap, etc., you may lose in overall performance and memory consumption instead of winning like in case of end-to-end non-blocking, async communication.
That means that old good imperative programming will be more suitable here and it will much easier to control your application sizing in memory using old good Tomcat configuration management.
Can you try this :
public class AsyncConfig implements AsyncConfigurer {
#Override
public Executor getAsyncExecutor() {
ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
taskExecutor.setCorePoolSize(15);
taskExecutor.setMaxPoolSize(100);
taskExecutor.setQueueCapacity(100);
taskExecutor.initialize();
return taskExecutor;
}
}
This works for async in spring 4 but I'm not sure it'll works in spring 5 with reactive.
I'm new to Vert.x and I would like to implement a pool of worker verticles to make database queries using BoneCP. However, I'm a little bit confused about how to 'call' them to work and how to share the BoneCP connection pool between them.
I saw in Vertx DeploymentManager source that the start(Future) method is called synchronously and then the verticle is kept in memory until undeployed. After the start method completes, what's the correct way of calling methods on the worker verticle? If I deploy many instances of the verticle (using DeploymentOptions.setInstances()), will Vertx do load balancing between them?
I saw that Vert.x comes with a JDBC client and a worker pool, but it has limited datatypes I can work with because it uses the EventBus and serializes all data returned by the database. I need to work with many different datatypes (including dates, BigDecimals and binary objects) and I would like to avoid serialization as much as possible, but instead make queries in the worker verticle, process the results and return an object via a Future or AsyncResult (I believe this is done on-heap, so no serialization needed; is this correct?).
Please help me to sort out all these questions :) I will appreciate a lot if you give me examples of how can I make this work!
Thanks!
I'll try to answer your questions one by one.
how to 'call' them to work
You call your worker verticles using the EventBus. That's the proper way to communicate between them. Please see this example:
https://github.com/vert-x3/vertx-examples/blob/master/core-examples/src/main/java/io/vertx/example/core/verticle/worker/MainVerticle.java#L27
how to share the BoneCP connection pool between them.
Don't. Instead, create a small connection pool for each. Otherwise, it will cause unexpected behavior.
config.setMinConnectionsPerPartition(1);
config.setMaxConnectionsPerPartition(5);
config.setPartitionCount(1);
will Vertx do load balancing between them
No. That's the reason #Jochen Bedersdorfer and I suggest to use EventBus. You can have a reference to your worker verticle, as you suggested, but then you're stuck with 1:1 configuration.
return an object via a Future or AsyncResult (I believe this is done
on-heap, so no serialization needed; is this correct?)
This is correct. But again, you're stuck with 1:1 mapping then. Which is a lot worse in terms of performance that serialization (that's using buffers).
If you still do need something like that, maybe you shouldn't use worker verticles at all, but something like .executeBlocking:
https://github.com/vert-x3/vertx-examples/blob/master/core-examples/src/main/java/io/vertx/example/core/execblocking/ExecBlockingExample.java#L25
In your start(...) method, register event listeners with event bus as this is how you interact with verticles (worker or not).
Yes, if you deploy many instances, Vert.x will use round-robin to send messages to those instances.
For what you describe, Vert.x might not be the best fit, since it works best with asynchronous I/O.
You might be better off using standard Java concurrency tools to manage the load, i.e. Executor and friends.
I'm creating a client-server relationship whereby a single client will be connected to an arbitrary number of servers using persistent TCP connections. The actual number of servers is as-of-yet undetermined, but the design goal is to shoot for 1000.
I found an example using direct Java NIO that nearly completely matches my mental model of how this could work:
http://drdobbs.com/jvm/184406242
In general, it opens up all of the channels and adds them to a single thread monitoring java.nio.channels.Selector. The use of the Selector, in particular, is what allows this to scale far better than using the standard thread-per-channel.
I would rather use a (slightly) higher level socket framework like Netty, than direct Java NIO. Unfortunately, I have not been able to determine how Netty would handle a case like this. That is, the examples and discussions I've found all tend to center around the server side, with accepting scores of concurrent connections.
But what about doing this from the client side? If I create a large number of channels and just wait on their events, how is Netty going to handle this at the back-end?
This isn't a direct answer to your question but I hope it is helpful nonetheless. Below, I describe a way for you to determine the answer that you are looking for. This is something that I recently did myself for an upcoming project.
Compared to OIO (Old IO) the asynchronous nature of the Netty framework and NIO will indeed provide much better memory and CPU usage characteristics for your application. The way buffers are handled in Netty will also be of benefit as it will help you to avoid copying byte buffers. The point is that all of the thread pool and NIO details will be handled for you allowing you to focus on your business logic. You mentioned the NIO Selector and you will benefit from that; the nice thing about Netty is that you get the benefits without having to worry about that implementation yourself because it is already done for you.
My understanding of the client side is that it is very similar to the server side and should provide you with commensurate performance gains (as long as your business logic doesn't introduce any performance issues).
My advice would be to throw together a prototype that more or less does what you want. Leave out any time consuming details and just add in the basic Netty handlers that you need to make something that works.
Then I would use jmeter to invoke your client to apply load to the server and client. Using something like jconsole or jvisualvm will show you the performance characteristics of the client and server under load. You could also try jprobe. You can add a listener in jmeter that will indicate the throughput. I would advise to use jmeter in server mode, the client on another machine and the server on yet another. This is a bit of up front work but if you decide to move forward you will have these tools ready to go for further testing as your proceed.
I suspect a decent Netty implementation that doesn't introduce any extraneous poorly performing components will give you the performance characteristics you are looking for, but, the only way to know for sure is to measure the system under the expected load.
You need to define what the expected load looks like and the desired performance characteristics under such load. Given these inputs you can measure your system to find out if it will meet your expectations. I personally don't think anyone can tell you if it will behave in the desired manner. You have to measure it. It's the only reliable way to know if the system can meet your needs.
I would rather use a (slightly) higher level socket framework like Netty, than direct Java NIO.
This is the correct approach. You can try implementing your own NIO server and client but why do that when you have the benefit of a highly refined framework at your fingertips already?
Netty will use up to x worker threads that handle the work for you. Each worker thread will have one Selector that is used to register Channels to it. The number of used workers is configurable and by default 2 * cpu-count.
As you can see in the example from Netty's doc [http://netty.io/docs/stable/guide/html/#start.9][1] you can control exactly the number of worker threads (meaning the number of underlying selectors) on the Client side.
Netty solves a numbers of issues that are very hard to handle in a simple way such as NIO vs SSL, and have a lot of default encoder/decoder for Zip... etc.
I started using Netty a few week ago and it was quite fast to came into. (I recommend dowloading the project with all the example code inside, there is a lot of documentation in it that can not be found on the url above.
ChannelFactory factory = new NioClientSocketChannelFactory(
Executors.newCachedThreadPool(),
Executors.newCachedThreadPool());
ClientBootstrap bootstrap = new ClientBootstrap(factory);
bootstrap.setPipelineFactory(new ChannelPipelineFactory() {
public ChannelPipeline getPipeline() {
return Channels.pipeline(new TimeClientHandler());
}
});
bootstrap.setOption("tcpNoDelay", true);
bootstrap.setOption("keepAlive", true);
bootstrap.connect(new InetSocketAddress(host, port));
Good luck,
Renaud
I have a couple of questions
1) How can we define in weblogic configuration how many concurrent users are allowed or can be allowed at a time to a particular application?
2) how can we tell how may threads are being used in a weblogic at a time?
3) How many max jdbc connections should I set so that users are not blocked due to all connections used up. How to keep a balance between number of concurrent user/threads allowed to jdbc connections max?
Thanks
It is different in each use case scenario.
But usually WLS 1 instance can cover 50~100 active user per instance.
The instance has 2 CPU and 1~1.5GB heap.
This document will be useful to your question:
"Planning Number Of Instance And Thread In Web Application Server"
1) You can user Work Managers to do this for managing requests. However, restricting the number of concurrent users will vary application to application. If it is a web app, use the work managers with a max constraint equal to the number of users you want to restrict it to. However, be sure you figure out how to handle overflow - what will you do when you get 100 requests but have a 5-user restriction? Is this synchronous or asynchronous processing?
2) Ideally you would want a 1:1 ratio of threads to connections in the pool. This guarantees that no thread (User Request) is waiting for a connection. I would suggest trying this. You can monitor the JDBC connection pools using the WebLogic console and adding fields to the columns under the 'Monitoring' tab for the connection. If you have a high number of waiters, and/or a high wait time then you would want to increase the number of connections in the pool. You could start with a 1:0.75 ratio of threads:connections, do performance/load testing and adjust based on your findings. It really depends on how well you manage the connections. Do you release the connection immediately after you get the data from the database, or do you proceed with application logic and release the connection at the end of the method/logic? If you hold the connection for a long time you will likely need closer to a 1:1 ratio.
1) If to each user you assign a session, then you can control the max number of sessions in your webapp weblogic descriptor, for example adding the following constraint :
<session-descriptor> <max-in-memory-sessions>12</max-in-memory-sessions> </session-descriptor>
It's more effective (if you mean 1 user = 1session) than limiting the number of requests by work managers.
Another way, when you can't predict the size of sessions and the number of users, is to adjust memory overloading parameters and set :
weblogic.management.configuration.WebAppContainerMBean.OverloadProtectionEnabled.
More info here :
http://download.oracle.com/docs/cd/E12840_01/wls/docs103/webapp/sessions.html#wp150466
2) Capacity of threads is managed by WebLogic through work managers. By default, just one exists : default with unllimited number of threads (!!!).
3) Usually, adapting the number of JDBC connections to the number of threads is the more effective.
The following page could surely be of great interest :
http://download.oracle.com/docs/cd/E11035_01/wls100/config_wls/overload.html
As far as i know you have to control these kind of things in
weblogic-xml-jar.xml
or
weblogic.xml
if you look for weblogic-xml-jar.xml commands you can find your desire .