I have a issue here. When I am using this as my code (below) in spring batch configuration, my code ends successfully.
#Bean(name = "myStep")
public Step myStep() {
int cores = Runtime.getRuntime().availableProcessors();
int maxAndQueueSize = cores * 2;
return stepBuilderFactory.get("myStep").<A, B> chunk(CHUNKS)
.reader(myItemReader(entityManagerFactory)).processor(myProcessor())
.writer(myWriter()).listener(myListener()).throttleLimit(maxAndQueueSize).allowStartIfComplete(true).build();
}
But when I modify this code (for async writing) by adding taskExecutor(asyncTaskExecutor()), my aim is fulfilled but code is still running in the eclipse. Seems its a thread closing related issue. Please help how can I close my code gracefully?
#Bean(name = "myStep")
public Step myStep() {
int cores = Runtime.getRuntime().availableProcessors();
int maxAndQueueSize = cores * 2;
return stepBuilderFactory.get("myStep").<A, B> chunk(CHUNKS)
.reader(myItemReader(entityManagerFactory)).processor(myProcessor())
.writer(myWriter()).listener(myListener()).taskExecutor(asyncTaskExecutor()).throttleLimit(maxAndQueueSize).allowStartIfComplete(true).build();
}
This is the asyncTaskExecutor()
#Bean(name = "asyncTaskExecutor")
public AsyncTaskExecutor asyncTaskExecutor() {
int cores = Runtime.getRuntime().availableProcessors();
int maxAndQueueSize = cores * 2;
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(cores);
executor.setMaxPoolSize(maxAndQueueSize);
executor.setQueueCapacity(maxAndQueueSize);
executor.setThreadNamePrefix("asyncExecutor-");
executor.initialize();
return executor;
}
Related
I've set up a spring-reactive server and an use a while loop inside of a Scheduled job to make fire and forget requests from a httpClient (backend just returns the same simple string via tomcat) it is configured as follows:
public ConnectionProvider getConnectionProvider(){
int maxConnections = 20;
ConnectionProvider connProvider = ConnectionProvider
.builder("webclient-conn-pool")
.maxConnections(maxConnections)
.maxIdleTime(Duration.of(20, ChronoUnit.SECONDS))
.maxLifeTime(Duration.of(1000, ChronoUnit.SECONDS))
.pendingAcquireMaxCount(2000)
.pendingAcquireTimeout(Duration.ofMillis(20000))
.build();
return connProvider;
}
public HttpClient getHttpClient(){
return HttpClient
.create(getConnectionProvider());
//.secure(sslContextSpec -> sslContextSpec.sslContext(webClientSslHelper.getSslContext()))
/*.tcpConfiguration(tcpClient -> {
LoopResources loop = LoopResources.create("webclient-event-loop",
selectorThreadCount, workerThreadCount, Boolean.TRUE);
return tcpClient
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 10000)
.option(ChannelOption.TCP_NODELAY, true);*/
}
I've also created a job that uses #Autowired kafkaTemplate and makes a while loop that sends a short text string
For http / tomcat calls I am getting around 65 requests a second
For kafka I am maxing out around 50000 requests with a half second lag
application.props
spring.kafka.bootstrap-servers=PLAINTEXT://localhost:9092,PLAINTEXT://localhost:9093
host.name=localhost
Job to make calls
#Async
#Scheduled(fixedDelay = 15000)
public void scheduleTaskUsingCronExpression() {
generateCalls();
}
private void generateCalls() {
try{
int i = 0;
System.out.println("start");
long startTime = System.currentTimeMillis();
while(i <= 5000){
//Thread.sleep(5);
String message = "Test Message sadg sad-";
kafkaTemplate.send(TOPIC, message + i);
i++;
}
long endTime = System.currentTimeMillis();
System.out.println((endTime - startTime));
System.out.println("done");
}
catch(Exception e){
e.printStackTrace();
}
System.out.println("RUNNING");
}
Kakfa partition config
#Bean
public KafkaAdmin kafkaAdmin() {
//String bootstrapAddress = "localhost:29092";
String bootstrapAddress = "localhost:9092";
Map<String, Object> configs = new HashMap<>();
configs.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress);
return new KafkaAdmin(configs);
}
#Bean
public NewTopic testTopic() {
return new NewTopic("test-topic", 6, (short) 1);
}
Kafka consumer consuming messge
#KafkaListener(topics = "test-topic", groupId = "one", concurrency = "6" )
public void listenGroupFoo(String message) {
if(message.indexOf("-0") != -1){
startTime = new Date().getTime();
System.out.println("Starting Message in group foo: " + message);
}
else if(message.indexOf("-100000") != -1){
endTime = new Date().getTime();
System.out.println("Received Message in group foo: " + message);
System.out.println(endTime - startTime);
}
}
For hardware I have a 10900k with 64gb ram
5ghz clock speed
970 Evo single nvme disk
10 core 20 thread
All requests are from the same local server to the same local server
I can add more local servers for testing if needed
Is there a better way to organize / optimize the code to make a massive number of requests?
Theories:
Multiple Threads?
Changing configurations of servers such as tomcat configs (receiving or sending side)?
Not use the kafkaTemplate that is autowired or creating multiple?
Modify Hardware to have multiple disks?
Improve server receiving http to allow more connections?
Not use a job to make the requests?
Anything else anyone can think of to help?
We have recently moved form jedis to using lettuce in our production services. However we have hit a roadblock while creating redis distributed locks
We are using non clustered setup of aws elasticache with one master and 2 read repilcas
Configs :
Spring-boot : 2.2.5
spring-boot-starter-data-redis : 2.2.5
spring-data-redis : 2.2.5
spring-integration-redis : 5.2.4
redis : 5.0.6
#Bean
public LettuceConnectionFactory redisConnectionFactory() {
GenericObjectPoolConfig poolingConfig = new GenericObjectPoolConfig();
poolingConfig.setMaxIdle(Integer.valueOf(maxConnections));
poolingConfig.setMaxTotal(Integer.valueOf(maxIdleConnections));
poolingConfig.setMinIdle(Integer.valueOf(minIdleConnections));
poolingConfig.setMaxWaitMillis(-1);
final SocketOptions socketOptions = SocketOptions.builder().connectTimeout(Duration.ofSeconds(10)).build();
final ClientOptions clientOptions = ClientOptions.builder().socketOptions(socketOptions).build();
LettucePoolingClientConfiguration clientOption = LettucePoolingClientConfiguration.builder()
.poolConfig(poolingConfig).readFrom(ReadFrom.REPLICA_PREFERRED)
.commandTimeout(Duration.ofMillis(Long.valueOf(commandTimeout)))
.clientOptions(clientOptions).useSsl().build();
RedisStaticMasterReplicaConfiguration redisStaticMasterReplicaConfiguration = new RedisStaticMasterReplicaConfiguration(
primaryEndPoint, Integer.valueOf(port));
redisStaticMasterReplicaConfiguration.addNode(readerEndPoint, Integer.valueOf(port));
redisStaticMasterReplicaConfiguration.setPassword(password);
/*
* LettuceClientConfiguration clientConfig = LettuceClientConfiguration
* .builder() .useSsl()
*
* .readFrom(new ReadFrom() {
*
* #Override public List<RedisNodeDescription> select(Nodes nodes) {
* List<RedisNodeDescription> allNodes = nodes.getNodes(); int ind =
* Math.abs(index.incrementAndGet() % allNodes.size()); RedisNodeDescription
* selected = allNodes.get(ind);
* //logger.info("Selected random node {} with uri {}", ind, selected.getUri());
* List<RedisNodeDescription> remaining = IntStream.range(0, allNodes.size())
* .filter(i -> i != ind) .mapToObj(allNodes::get).collect(Collectors.toList());
* return Stream.concat( Stream.of(selected), remaining.stream()
* ).collect(Collectors.toList()); } }) .build();
*/
return new LettuceConnectionFactory(redisStaticMasterReplicaConfiguration, clientOption);
}
#Bean
public StringRedisTemplate stringRedisTemplate() {
return new StringRedisTemplate(redisConnectionFactory());
}
LOCKING SERVICE
#Service
public class RedisLockService {
#Autowired
RedisConnectionFactory redisConnectionFactory;
private static final Logger LOGGER = LoggerFactory.getLogger(RedisLockService.class);
public Lock obtainLock(String registryKey,String redisKey,Long lockExpiry){
try{
RedisLockRegistry registry = new RedisLockRegistry(redisConnectionFactory, registryKey, lockExpiry);
Lock lock = registry.obtain(redisKey);
if(lock.tryLock()==false)
{
LOGGER.info("Lock already made");
return null;
}
else
return lock;
}catch (Exception e) {
LOGGER.warn("Unable to acquire lock: ", e);
return null;
}
}
public void unLock(Lock lock) {
if(lock!=null)
lock.unlock();
}
}
error We are getting while trying to call obtainLock function
.RedisSystemException: Error in execution; nested exception is io.lettuce.core.RedisCommandExecutionException: ERR Error running script (call to f_8426c8df41c64d8177dce3ecbbe9146ef3759cd2): #user_script:6: #user_script: 6: -READONLY You can't write against a read only replica.
at org.springframework.integration.redis.util.RedisLockRegistry$RedisLock.rethrowAsLockException(RedisLockRegistry.java:224)
at org.springframework.integration.redis.util.RedisLockRegistry$RedisLock.tryLock(RedisLockRegistry.java:276)
at org.springframework.integration.redis.util.RedisLockRegistry$RedisLock.tryLock(Re
you need to connect to the master read/write node of Redis
I have InboundChannelAdapter configured with S3StreamingMessageSource.
I forced Poller to use taskExecutor with only 1 thread. But I see the same file is being picked up by the same thread 3 times with 3-4 seconds interval. Even though poller interval is 10 seconds. I've specified Composite filter which consists of pattern filter and acceptoncefilter. But no result, file is always picked up 3 times.
String prefix = "some_prefix";
String channel = "some_channel"
Pattern filePattern = Pattern.compile(
"^" + prefix + "some_file_name_pattern");
#Bean
#InboundChannelAdapter(value = channel,
poller = #Poller(fixedDelay = "10000", taskExecutor = "threadPoolTaskExecutor"))
public MessageSource<InputStream> createS3InboundStreamingMessageSource() {
S3StreamingMessageSource messageSource = new S3StreamingMessageSource(template());
messageSource.setRemoteDirectory(bucketName);
CompositeFileListFilter<S3ObjectSummary> compositeFileListFilter = new ChainFileListFilter<>();
compositeFileListFilter.addFilter(new S3PersistentAcceptOnceFileListFilter(
new SimpleMetadataStore(), prefix));
compositeFileListFilter.addFilter(new S3RegexPatternFileListFilter(filePattern));
messageSource.setFilter(compositeFileListFilter);
return messageSource;
}
#Transformer(inputChannel = channel,"another_channel")
public Message<S3ObjectInputStream> enrich(Message<S3ObjectInputStream> s3ObjectInputStreamMessage) {
S3ObjectInputStream s3ObjectInputStream = s3ObjectInputStreamMessage.getPayload();
URI zipUri = s3ObjectInputStream.getHttpRequest().getURI();
LOGGER.info("Picking up file : {}", zipUri.getPath());
...
}
private S3RemoteFileTemplate template() {
S3SessionFactory sessionFactory = new S3SessionFactory(amazonS3);
return new S3RemoteFileTemplate(sessionFactory);
}
#Bean
public TaskExecutor threadPoolTaskExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setMaxPoolSize(1);
executor.setThreadNamePrefix("single_thread_task_executor");
executor.initialize();
return executor;
}
I see that the app comes to #Transformer 3 times. Would really appreciate any help.
I am using Spring Cloud library to poll SQS. How can I set poll interval?
#Bean
#Primary
public AmazonSQSAsync amazonSQSAsync() {
return AmazonSQSAsyncClientBuilder.standard().
withCredentials(awsCredentialsProvider()).
withClientConfiguration(clientConfiguration()).
build();
}
#Bean
#ConfigurationProperties(prefix = "aws.queue")
public SimpleMessageListenerContainer simpleMessageListenerContainer(AmazonSQSAsync amazonSQSAsync) {
SimpleMessageListenerContainer simpleMessageListenerContainer = new SimpleMessageListenerContainer();
simpleMessageListenerContainer.setAmazonSqs(amazonSQSAsync);
simpleMessageListenerContainer.setMessageHandler(queueMessageHandler());
simpleMessageListenerContainer.setMaxNumberOfMessages(10);
simpleMessageListenerContainer.setTaskExecutor(threadPoolTaskExecutor());
return simpleMessageListenerContainer;
}
#Bean
public QueueMessageHandler queueMessageHandler() {
QueueMessageHandlerFactory queueMessageHandlerFactory = new QueueMessageHandlerFactory();
queueMessageHandlerFactory.setAmazonSqs(amazonSQSAsync());
QueueMessageHandler queueMessageHandler = queueMessageHandlerFactory.createQueueMessageHandler();
return queueMessageHandler;
}
#Bean
public ThreadPoolTaskExecutor threadPoolTaskExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(10);
executor.setMaxPoolSize(10);
executor.setThreadNamePrefix("oaoQueueExecutor");
executor.initialize();
return executor;
}
Call setWaitTimeOut(N) function of the base class AbstractMessageListenerContainer in package org.springframework.cloud.aws.messaging.listener. N is the long poll timeout in seconds.
For example if you want to wait for 5 sec before it returns, use below line of code in your queueMessageHandler() function. The default is 1 sec if you don't call this function. The maximum long polling timeout is 20 sec, so the max you can give to this function is 20 which means "wait for 20 seconds"
simpleMessageListenerContainer.setWaitTimeOut (5);
The source code is here: https://github.com/spring-cloud/spring-cloud-aws/blob/master/spring-cloud-aws-messaging/src/main/java/org/springframework/cloud/aws/messaging/listener/AbstractMessageListenerContainer.java
/**
* Configures the wait timeout that the poll request will wait for new message to arrive if the are currently no
* messages on the queue. Higher values will reduce poll request to the system significantly.
*
* #param waitTimeOut
* - the wait time out in seconds
*/
public void setWaitTimeOut(Integer waitTimeOut) {
this.waitTimeOut = waitTimeOut;
}
I have the following ThreadPoolTaskExecutor thats gets created with the expected core/max pool size configurations.
#Slf4j
#Configuration
public class ThreadPoolConfig {
#Value("${corePoolSize}")
private Integer corePoolSize;
#Value("${queueCapacity}")
private Integer queueCapacity;
#Value("${maxPoolSize}")
private Integer maxPoolSize;
#Bean(name="myThreadPoolTaskExecutor")
public ThreadPoolTaskExecutor myThreadPoolTaskExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setBeanName("myThreadPoolTaskExecutor");
executor.setCorePoolSize(corePoolSize);
executor.setQueueCapacity(queueCapacity);
executor.setMaxPoolSize(maxPoolSize);
executor.setThreadNamePrefix("my_thread_");
executor.setWaitForTasksToCompleteOnShutdown(true);
executor.initialize();
log.debug("threadPoolTaskExecutor CorePoolSize is : " + executor.getCorePoolSize());
log.debug("threadPoolTaskExecutor MaxPoolSize is : " + executor.getMaxPoolSize());
return executor;
}
}
When my #scheduled method runs the max pool size is set to the DEFAULT value of 2147483647 and I don't understand why it's not using the configured ThreadPoolTaskExecutor above:
#EnableScheduling
public class SchedulingConfig {
}
#Component
public class Scheduler {
#Autowired
#Qualifier("myThreadPoolTaskExecutor")
private ThreadPoolTaskExecutor threadPoolTaskExecutor;
#Scheduled(fixedRateString = "${fixedRate}")
public void invokeScheduledThread() {
while (threadPoolTaskExecutor.getActiveCount() <= threadPoolTaskExecutor.getMaxPoolSize()) {
log.debug("Active Thread Pool count is : " + threadPoolTaskExecutor.getActiveCount() + ", Max Thread Pool count is : " + threadPoolTaskExecutor.getMaxPoolSize() + " on the scheduled Thread : " + Thread.currentThread().getName());
//call a service to retrieve some items to process
threadPoolTaskExecutor.execute(Some Object that implements runnable);
}
}
}
Output:
Active Thread Pool count is : 0, Max Thread Pool count is : 2147483647 on the scheduled Thread : task-scheduler-1
I put a break point into the initialise() method of org.springframework.scheduling.concurrent.ExecutorConfigurationSupport
and it looks like the method is getting invoked 3 times, twice with a ThreadName Prefix of "my_thread_"
which is expected and finally once for a Bean called "taskScheduler" with a ThreadName Prefix of "task-scheduler-".
Does anyone know why I can't use my own ThreadPoolTaskExecutor within the Scheduler class?
I wanted to use a default #Scheduler to run on a single thread every x number of seconds and create X number of Threads using my own ThreadPoolTaskExecutor.
Use ThreadPoolTaskScheduler instead of ThreadPoolTaskExecutor.
For example:
#Configuration
public class SpringSchedulerConfig {
private static final int THREAD_POOL_SIZE = 5;
#Bean
public ThreadPoolTaskScheduler getScheduler() {
ThreadPoolTaskScheduler threadPoolTaskScheduler = new ThreadPoolTaskScheduler();
//we want every Job in a separate Thread.
threadPoolTaskScheduler.setPoolSize(THREAD_POOL_SIZE);
return threadPoolTaskScheduler;
}
}