I am writing below code to override the thread pool. But it is not working properly. What is the correct way to set override the thread pool in spring boot app startup. Note that i don't have a control in my code over the Server instance, so instantiating a Server is not a solution for my need.
#Bean
public EmbeddedServletContainerCustomizer getContainerCustomizer() {
return (configurableEmbeddedServletContainer) -> {
if (configurableEmbeddedServletContainer instanceof JettyEmbeddedServletContainerFactory) {
((JettyEmbeddedServletContainerFactory)configurableEmbeddedServletContainer).addServerCustomizers((server) -> {
QueuedThreadPool newPool = new QueuedThreadPool(10);
QueuedThreadPool oldPool = server.getBean(QueuedThreadPool.class);
server.updateBean(oldPool, newPool);
});
}
};
}
When i execute the code, it was throwing below error
Exception in thread "main"
java.util.concurrent.RejectedExecutionException: org.eclipse.jetty.io.ManagedSelector#1ee0005 id=0 keys=0 selected=0
at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:377)
at org.eclipse.jetty.io.SelectorManager.execute(SelectorManager.java:125)
at org.eclipse.jetty.io.SelectorManager.doStart(SelectorManager.java:255)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:260)
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:244)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.server.Server.doStart(Server.java:384)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
I tried by sample main code also, and it is also giving same error.
public class FileServer
{
public static void main(String[] args) throws Exception
{
Server server = new Server(9090);
QueuedThreadPool newPool = new QueuedThreadPool(10);
QueuedThreadPool oldPool = server.getBean(QueuedThreadPool.class);
server.updateBean(oldPool, newPool);
ResourceHandler resource_handler = new ResourceHandler();
resource_handler.setDirectoriesListed(true);
resource_handler.setWelcomeFiles(new String[]{ "index.html" });
resource_handler.setResourceBase(".");
HandlerList handlers = new HandlerList();
handlers.setHandlers(new Handler[] { resource_handler, new DefaultHandler() });
server.setHandler(handlers);
server.start();
server.join();
}
}
Here is an example of configuring a thread pool in Jetty with properties and a different type of thread pool. The thread pool in my case is InstrumentedQueuedThreadPool.
#Configuration
public class CustomJettyConfiguration {
#Bean
public JettyEmbeddedServletContainerFactory jettyEmbeddedServletContainerFactory(
#Value("${server.port:8080}") final String port,
#Value("${jetty.threadPool.maxThreads:600}") final String maxThreads,
#Value("${jetty.threadPool.minThreads:10}") final String minThreads,
#Value("${jetty.threadPool.idleTimeout:5000}") final String idleTimeout) {
final JettyEmbeddedServletContainerFactory factory =
new JettyEmbeddedServletContainerFactory(Integer.valueOf(port));
// Tweak the connection pool used by Jetty to handle incoming HTTP connections
InstrumentedQueuedThreadPool instThreadPool =
new InstrumentedQueuedThreadPool(registry);
instThreadPool.setPrefix("jetty");
instThreadPool.setMaxThreads(Integer.valueOf(maxThreads));
instThreadPool.setMinThreads(Integer.valueOf(minThreads));
instThreadPool.setIdleTimeout(Integer.valueOf(idleTimeout));
factory.setThreadPool(instThreadPool);
...
return factory;
}
}
In spring boot 2.x, the JettyEmbeddedServletContainerFactory has been replaced by org.springframework.boot.web.embedded.jetty.JettyServletWebServerFactory.
You need to customize JettyServletWebServerFactory and set your own thread pool.
#Component
public class JettyConfiguration implements WebServerFactoryCustomizer<JettyServletWebServerFactory> {
#Override
public void customize(JettyServletWebServerFactory factory) {
// customize your thread pool here
QueuedThreadPool qtp = new QueuedThreadPool(80, 500);
qtp.setName("jettyThreadPool");
factory.setThreadPool(qtp);
}
}
Related
I am using spring inbound channel adapter to poll files from sftp server. Application needs to poll from multiple directories from single sftp server. Since Inbound channel adapter does not allow to poll multiple directories I tried creating multiple beans of same type with different values. Since number of directories can increase in future, I want to control it from application properties and want to register beans dynamically.
My code -
#Override
public void postProcessBeanFactory(ConfigurableListableBeanFactory beanFactory) throws BeansException {
beanFactory.registerSingleton("sftpSessionFactory", sftpSessionFactory(host, port, user, password));
beanFactory.registerSingleton("sftpInboundFileSynchronizer",
sftpInboundFileSynchronizer((SessionFactory) beanFactory.getBean("sftpSessionFactory")));
}
public SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory(String host, String port, String user, String password) {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost(host);
factory.setPort(Integer.parseInt(port));
factory.setUser(user);
factory.setPassword(password);
factory.setAllowUnknownKeys(true);
return new CachingSessionFactory<>(factory);
}
private SftpInboundFileSynchronizer sftpInboundFileSynchronizer(SessionFactory sessionFactory) {
SftpInboundFileSynchronizer fileSynchronizer = new SftpInboundFileSynchronizer(sessionFactory);
fileSynchronizer.setDeleteRemoteFiles(true);
fileSynchronizer.setPreserveTimestamp(true);
fileSynchronizer.setRemoteDirectory("/mydir/subdir);
fileSynchronizer.setFilter(new SftpSimplePatternFileListFilter("*.pdf"));
return fileSynchronizer;
}
#Bean
#InboundChannelAdapter(channel = "sftpChannel", poller = #Poller(fixedDelay = "2000"))
public MessageSource<File> sftpMessageSource(String s) {
SftpInboundFileSynchronizingMessageSource source = new SftpInboundFileSynchronizingMessageSource(
(AbstractInboundFileSynchronizer<ChannelSftp.LsEntry>) applicationContext.getBean("sftpInboundFileSynchronizer"));
source.setLocalDirectory(new File("/dir/subdir"));
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(new AcceptOnceFileListFilter<>());
source.setMaxFetchSize(Integer.parseInt(maxFetchSize));
source.setAutoCreateLocalDirectory(true);
return source;
}
#Bean
#ServiceActivator(inputChannel = "sftpChannel")
public MessageHandler handler() {
return message -> {
LOGGER.info("Payload - {}", message.getPayload());
};
}
This code works fine. But If I create sftpMessageSource dynamically, then #InboundChannelAdapter annotation won't work. Please suggest a way to dynamically create sftpMessageSource and handler beans also and add respective annotations.
Update:
Following Code Worked :
#PostConstruct
void init() {
int index = 0;
for (String directory : directories) {
index++;
int finalI = index;
IntegrationFlow flow = IntegrationFlows
.from(Sftp.inboundAdapter(sftpSessionFactory())
.preserveTimestamp(true)
.remoteDirectory(directory)
.autoCreateLocalDirectory(true)
.localDirectory(new File("/" + directory))
.localFilter(new AcceptOnceFileListFilter<>())
.maxFetchSize(10)
.filter(new SftpSimplePatternFileListFilter("*.pdf"))
.deleteRemoteFiles(true),
e -> e.id("sftpInboundAdapter" + finalI)
.autoStartup(true)
.poller(Pollers.fixedDelay(2000)))
.handle(handler())
.get();
this.flowContext.registration(flow).register();
}
}
#Bean
public SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory() {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost(host);
factory.setPort(Integer.parseInt(port));
factory.setUser(user);
factory.setPassword(password);
factory.setAllowUnknownKeys(true);
return new CachingSessionFactory<>(factory);
}
Annotations in Java are static. You can't add them at runtime for created objects. Plus the framework reads those annotation on application context startup. So, what you are looking for is just not possible with Java as language per se.
You need consider to switch to Java DSL in Spring Integration to be able to use its "dynamic flows": https://docs.spring.io/spring-integration/docs/5.3.1.RELEASE/reference/html/dsl.html#java-dsl-runtime-flows.
But, please, first of all study more what Java can do and what cannot.
From reading the spring-boot docs, it seems like the standard way to customize the Jetty server is to implement a class like the following:
#Component
public class JettyServerCustomizer
implements WebServerFactoryCustomizer<JettyServletWebServerFactory> {
#Autowired
private ServerProperties serverProperties;
#Override
public void customize(final JettyServletWebServerFactory factory) {
factory.addServerCustomizers((server) -> {
// Customize
});
}
}
I'm specifically interested in modifying the SSLContextFactory.
Tracing through the spring-boot code, right before the customizers are called, ssl is configured:
if (getSsl() != null && getSsl().isEnabled()) {
customizeSsl(server, address);
}
for (JettyServerCustomizer customizer : getServerCustomizers()) {
customizer.customize(server);
}
customizeSsl is a private method so cannot be overridden easily:
private void customizeSsl(Server server, InetSocketAddress address) {
new SslServerCustomizer(address, getSsl(), getSslStoreProvider(), getHttp2()).customize(server);
}
One option is to create the context factory and connector ourselves in the customizer, and then overwrite the connectors on the server. This would probably work but it feels like we are re-creating a bunch of code that spring-boot is already doing just to be able to call a method on the SSLContextFactory.
It seems like if we could somehow provider our own SslServerCustomizer then we could do the custom configuration we want.
Does anyone know of a better way to do this?
On my case it works just fine as:
#SpringBootApplication
#ComponentScan(basePackages = { "org.demo.jetty.*" })
public class DemoWebApplication {
public static void main(String[] args) {
SpringApplication.run(DemoWebApplication.class, args);
}
#Bean
public ConfigurableServletWebServerFactory webServerFactory() {
JettyServletWebServerFactory factory = new JettyServletWebServerFactory();
factory.setContextPath("/demo-app");
factory.addServerCustomizers(getJettyConnectorCustomizer());
return factory;
}
private JettyServerCustomizer getJettyConnectorCustomizer() {
return server -> {
final HttpConfiguration httpConfiguration = new HttpConfiguration();
httpConfiguration.setSecureScheme("https");
httpConfiguration.setSecurePort(44333);
SslContextFactory.Server sslContextFactory = new SslContextFactory.Server();
sslContextFactory.setKeyStoreType("PKCS12");
sslContextFactory.setKeyStorePath("C:/jetty-demo/demo_cert.p12");
sslContextFactory.setKeyStorePassword("*****");
sslContextFactory.setKeyManagerPassword("****");
final HttpConfiguration httpsConfiguration = new HttpConfiguration(httpConfiguration);
httpsConfiguration.addCustomizer(new SecureRequestCustomizer());
ServerConnector httpsConnector = new ServerConnector(server,
new SslConnectionFactory(sslContextFactory, HttpVersion.HTTP_1_1.asString()),
new HttpConnectionFactory(httpsConfiguration));
httpsConnector.setPort(44333);
server.setConnectors(new Connector[] { httpsConnector });
server.setStopAtShutdown(true);
server.setStopTimeout(5_000);
};
}
}
You can define also a HTTP connector and add it to the customized section
...
ServerConnector connector = new ServerConnector(server);
connector.addConnectionFactory(new HttpConnectionFactory(httpConfiguration));
connector.setPort(8081);
server.setConnectors(new Connector[]{connector, httpsConnector});
...
Using Spring FTP Integration and Annotation configuration, I downloaded files from the FTP server. After downloaded still our application is trigger to connect the server and find the any newly added files, if any files added it will download from the server. But I don't need to maintain the FTP server session alive and disconnect the server after first connection or first time downloaded.
Code :
public class FtpServices {
#Bean(name="ftpSessionFactory")
public DefaultFtpSessionFactory ftpSessionFactory() {
System.out.println("session");
DefaultFtpSessionFactory sf = new DefaultFtpSessionFactory();
sf.setHost("localhost");
sf.setPort(21);
sf.setUsername("user");
sf.setPassword("password");
return sf;
}
#Bean
public FtpInboundFileSynchronizer ftpInboundFileSynchronizer() {
System.out.println("2");
FtpInboundFileSynchronizer fileSynchronizer = new FtpInboundFileSynchronizer(ftpSessionFactory());
fileSynchronizer.setDeleteRemoteFiles(false);
fileSynchronizer.afterPropertiesSet();
fileSynchronizer.setRemoteDirectory("/test/");
// fileSynchronizer.setFilter(new FtpSimplePatternFileListFilter("*.docx"));
fileSynchronizer.setFilter(filter);
return fileSynchronizer;
}
#Bean()
#InboundChannelAdapter(value="ftpChannel", poller = #Poller(fixedDelay = "50", maxMessagesPerPoll = "1"))
public FtpInboundFileSynchronizingMessageSource ftpMessageSource() {
System.out.println(3);
FtpInboundFileSynchronizingMessageSource source =
new FtpInboundFileSynchronizingMessageSource(ftpInboundFileSynchronizer());
source.setLocalDirectory(new File("D:/Test-downloaded/"));
//source.stop();
return source;
}
#Bean
#ServiceActivator(inputChannel = "ftpChannel", requiresReply="false")
public MessageHandler handler() {
System.out.println(4);
MessageHandler handler = new MessageHandler() {
#Override
public void handleMessage(Message<?> message) throws MessagingException {
System.out.println(message.getPayload()+" #ServiceActivator");
System.out.println(" Message Header :"+message.getHeaders());
}
};
return handler;
}
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata defaultPoller() {
PollerMetadata pollerMetadata = new PollerMetadata();
pollerMetadata.setTrigger(triggerOnlyOnce());
return pollerMetadata;
}
}
and also I override the AbtractFTPSessionFactory.java to test FTP server connection and disconnection process.
protected void postProcessClientAfterConnect(T t) throws IOException {
System.out.println("After connect");
}
protected void postProcessClientBeforeConnect(T client) throws IOException {
System.out.println("Before connect");
}
Console :
INFO : org.springframework.context.support.DefaultLifecycleProcessor - Starting beans in phase -2147483648
INFO : org.springframework.context.support.DefaultLifecycleProcessor - Starting beans in phase 0
Before connect
After connect
D:\Test-downloaded\demo 1.txt #ServiceActivator
Message Header :{id=e4a1fd7f-0bbf-9692-f70f-b0ac68b4dec4, timestamp=1477317086272}
D:\Test-downloaded\demo.txt #ServiceActivator
Message Header :{id=9115ee92-12b4-bf1f-d592-9c13bf7a27fa, timestamp=1477317086324}
Before connect
After connect
Before connect
After connect
Before connect
After connect
Before connect
After connect
Before connect
After connect
Before connect
After connect
Thanks.
That is really a purpose of any #InboundChannelAdapter: poll the target system for new data periodically.
To do that once we sometimes suggest OnlyOnceTrigger:
public class OnlyOnceTrigger implements Trigger {
private final AtomicBoolean done = new AtomicBoolean();
#Override
public Date nextExecutionTime(TriggerContext triggerContext) {
return !this.done.getAndSet(true) ? new Date() : null;
}
}
But this might not work for your case, because there might not be desired files in the source FTP directory yet.
Therefore we have to poll until you will receive required files and .stop() an adapter when that condition is met.
For this purpose you can use any downstream logic to determine the state or consider to implement AbstractMessageSourceAdvice to be injected to the PollerMetadata of the #Poller: http://docs.spring.io/spring-integration/reference/html/messaging-channels-section.html#conditional-pollers
I have the following configuration
spring.rabbitmq.listener.prefetch=1
spring.rabbitmq.listener.concurrency=1
spring.rabbitmq.listener.retry.enabled=true
spring.rabbitmq.listener.retry.max-attempts=3
spring.rabbitmq.listener.retry.max-interval=1000
spring.rabbitmq.listener.default-requeue-rejected=false //I have also changed it to true but the same behavior still happens
and in my listener I throw the exception AmqpRejectAndDontRequeueException to reject the message and enforce rabbit not to try to redeliver it...But rabbit redilvers it for 3 times then finally route it to dead letter queue.
Is that the standard behavior according to my provided configuration or do I miss something?
You have to configure the retry policy to not retry for that exception.
You can't do that with properties, you have to configure the retry advice yourself.
I'll post an example later if you need help with that.
requeue-rejected is at the container level (below retry on the stack).
EDIT
#SpringBootApplication
public class So39853762Application {
public static void main(String[] args) throws Exception {
ConfigurableApplicationContext context = SpringApplication.run(So39853762Application.class, args);
Thread.sleep(60000);
context.close();
}
#RabbitListener(queues = "foo")
public void foo(String foo) {
System.out.println(foo);
if ("foo".equals(foo)) {
throw new AmqpRejectAndDontRequeueException("foo"); // won't be retried.
}
else {
throw new IllegalStateException("bar"); // will be retried
}
}
#Bean
public ListenerRetryAdviceCustomizer retryCustomizer(SimpleRabbitListenerContainerFactory containerFactory,
RabbitProperties rabbitPropeties) {
return new ListenerRetryAdviceCustomizer(containerFactory, rabbitPropeties);
}
public static class ListenerRetryAdviceCustomizer implements InitializingBean {
private final SimpleRabbitListenerContainerFactory containerFactory;
private final RabbitProperties rabbitPropeties;
public ListenerRetryAdviceCustomizer(SimpleRabbitListenerContainerFactory containerFactory,
RabbitProperties rabbitPropeties) {
this.containerFactory = containerFactory;
this.rabbitPropeties = rabbitPropeties;
}
#Override
public void afterPropertiesSet() throws Exception {
ListenerRetry retryConfig = this.rabbitPropeties.getListener().getRetry();
if (retryConfig.isEnabled()) {
RetryInterceptorBuilder<?> builder = (retryConfig.isStateless()
? RetryInterceptorBuilder.stateless()
: RetryInterceptorBuilder.stateful());
Map<Class<? extends Throwable>, Boolean> retryableExceptions = new HashMap<>();
retryableExceptions.put(AmqpRejectAndDontRequeueException.class, false);
retryableExceptions.put(IllegalStateException.class, true);
SimpleRetryPolicy policy =
new SimpleRetryPolicy(retryConfig.getMaxAttempts(), retryableExceptions, true);
ExponentialBackOffPolicy backOff = new ExponentialBackOffPolicy();
backOff.setInitialInterval(retryConfig.getInitialInterval());
backOff.setMultiplier(retryConfig.getMultiplier());
backOff.setMaxInterval(retryConfig.getMaxInterval());
builder.retryPolicy(policy)
.backOffPolicy(backOff)
.recoverer(new RejectAndDontRequeueRecoverer());
this.containerFactory.setAdviceChain(builder.build());
}
}
}
}
NOTE: You cannot currently configure the policy to retry all exceptions, "except" this one - you have to classify all exceptions you want retried (and they can't be a superclass of AmqpRejectAndDontRequeueException). I have opened an issue to support this.
The other answers posted here didn't work me when using Spring Boot 2.3.5 and Spring AMQP Starter 2.2.12, but for these versions I was able to customize the retry policy to not retry AmqpRejectAndDontRequeueException exceptions:
#Configuration
public class RabbitConfiguration {
#Bean
public RabbitRetryTemplateCustomizer customizeRetryPolicy(
#Value("${spring.rabbitmq.listener.simple.retry.max-attempts}") int maxAttempts) {
SimpleRetryPolicy policy = new SimpleRetryPolicy(maxAttempts, Map.of(AmqpRejectAndDontRequeueException.class, false), true, true);
return (target, retryTemplate) -> retryTemplate.setRetryPolicy(policy);
}
}
This lets the retry policy skip retries for AmqpRejectAndDontRequeueExceptions but retries all other exceptions as usual.
Configured this way, it traverses the causes of an exception, and skips retries if it finds an AmqpRejectAndDontRequeueException.
Traversing the causes is needed as org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter#invokeHandler wraps all exceptions as a ListenerExecutionFailedException
I have simple code which connects to broker URL using TCP . When i set transaction manager my code completes without exception and no messages are consumed but when i don't set traction manager it works fine.
Following is code , i am not getting what is missing or is there any configuration needs to be done on broker host.
import org.apache.activemq.ActiveMQConnectionFactory;
import org.apache.camel.CamelContext;
import org.apache.camel.ConsumerTemplate;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.component.jms.JmsComponent;
import org.apache.camel.component.jms.JmsConfiguration;
import org.apache.camel.impl.DefaultCamelContext;
import org.springframework.jms.connection.JmsTransactionManager;
public class JmsCamelComponent {
public void getMessage() {
CamelContext context = new DefaultCamelContext();
ActiveMQConnectionFactory jmsConnectionFactory = new ActiveMQConnectionFactory(
"tcp://10.10.3.140:61616?jms.redeliveryPolicy.maximumRedeliveries=10");
JmsTransactionManager jmsTransactionManager = new JmsTransactionManager(jmsConnectionFactory);
JmsConfiguration config = new JmsConfiguration(jmsConnectionFactory);
config.setConnectionFactory(jmsConnectionFactory);
JmsComponent jms = new JmsComponent(config);
jms.setTransactionManager(jmsTransactionManager);
jms.setTransactionTimeout(1000000);
try {
System.out.println("Creating and adding route...");
context.addComponent("jms", jms);
context.addRoutes(new RouteBuilder() {
public void configure() {
System.out.println("Inside configure...");
from("jms:topic:prathamq?clientId=411&durableSubscriptionName=parag").to("file://test2");
System.out.println("after getting message");
}
});
ConsumerTemplate template = context.createConsumerTemplate();
context.start();
System.out.println("Context started...");
Thread.sleep(1000);
System.out.println(" " + template.toString());
} catch (Exception e) {
// TODO Auto-generated catch block
System.out.println("In catch block");
e.printStackTrace();
}
}
public static void main(String[] args) {
(new JmsCamelComponent()).getMessage();
}
}
Following line i set or unset "jms.setTransactionManager(jmsTransactionManager);"
Please help.
Thanks in adavnce ,
Sohan
I've re-produced the problem (after a few false starts) and got a version that works:
public void getMessage() throws Exception {
ActiveMQConnectionFactory jmsConnectionFactory = new ActiveMQConnectionFactory("tcp://localhost:61616");
JmsTransactionManager jmsTransactionManager = new JmsTransactionManager(jmsConnectionFactory);
SimpleRegistry registry = new SimpleRegistry();
registry.put("transactionManager", jmsTransactionManager);
JmsConfiguration config = new JmsConfiguration(jmsConnectionFactory);
config.setConnectionFactory(jmsConnectionFactory);
CamelContext context = new DefaultCamelContext(registry);
JmsComponent jms = new JmsComponent();
jms.setTransactionManager(jmsTransactionManager);
context.getRegistry();
context.addComponent("jms", jms);
context.addRoutes(new RouteBuilder() {
public void configure() {
from("activemq:topic:matt?clientId=411&durableSubscriptionName=matt")
.log("GOT A MESSAGE");
}
});
context.start();
System.out.println("Context started...");
Thread.sleep(10000000);
}
Apart from increasing the thread sleep time to make sure the app doesn't exit before its had chance to do anything, the root cause was lack of a platform transaction manager in the registry. You need to create/get a registry and add the transaction manager to it.
If you're using Spring, you can just use the Spring context. In this example, I just create a SimpleRegistry to get the code to work.
If you take your failing code and add an explicit .transacted() in the route, you can see it complains about not finding a platform transaction manager. Why it doesn't complain without an explicit .transacted() call I don't know but that appears to be the underlying problem.