I use
Tomcat 8.0.26
Spring Boot 1.2.6.RELEASE
Spring 4.2.1.RELEASE
Spring Batch 3.0.5.RELEASE
In my application I have a following Spring Batch config:
#Configuration
#EnableBatchProcessing
public class ReportJobConfig {
public static final String REPORT_JOB_NAME = "reportJob";
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
private ReportService reportService;
#Bean(name = REPORT_JOB_NAME)
public Job reportJob() {
//#formatter:off
return jobBuilderFactory
.get(REPORT_JOB_NAME)
.flow(createRequestStep())
.on("*").to(retriveInfoStep())
.on("*").to(notifyAdminStep())
.end().build();
//#formatter:on
}
#Bean
public Step createRequestStep() {
return stepBuilderFactory.get("createRequest").tasklet(new CreateRequestTasklet(reportService)).build();
}
#Bean
public Step retrivePHIStep() {
return stepBuilderFactory.get("retriveInfo").tasklet(new RetriveInfoTasklet(reportService)).build();
}
#Bean
public Step notifyAdminStep() {
return stepBuilderFactory.get("notifyAdmin").tasklet(new NotifyAdminTasklet()).build();
}
}
This is how I run the job:
#Service
public class ReportJobServiceImpl implements ReportJobService {
final static Logger logger = LoggerFactory.getLogger(ReportJobServiceImpl.class);
#Autowired
#Qualifier(ReportJobConfig.REPORT_JOB_NAME)
private Job reportJob;
#Autowired
private JobLauncher jobLauncher;
#Override
public void runReportJob(String messageContent) throws JobExecutionAlreadyRunningException, JobRestartException,
JobInstanceAlreadyCompleteException, JobParametersInvalidException {
Map<String, JobParameter> parameters = new HashMap<>();
JobParameter reportIdParameter = new JobParameter(messageContent);
parameters.put(REPORT_ID, reportIdParameter);
jobLauncher.run(reportJob, new JobParameters(parameters));
}
}
Batch properties:
batch.jdbc.driver=com.mysql.jdbc.Driver
batch.jdbc.url=jdbc:mysql://localhost/database
batch.jdbc.user=user
batch.jdbc.password=password
batch.jdbc.testWhileIdle=true
batch.jdbc.validationQuery=SELECT 1
batch.drop.script=classpath:/org/springframework/batch/core/schema-drop-mysql.sql
batch.schema.script=classpath:/org/springframework/batch/core/schema-mysql.sql
batch.business.schema.script=classpath:/business-schema-mysql.sql
batch.database.incrementer.class=org.springframework.jdbc.support.incrementer.MySQLMaxValueIncrementer
batch.database.incrementer.parent=columnIncrementerParent
batch.lob.handler.class=org.springframework.jdbc.support.lob.DefaultLobHandler
batch.grid.size=50
batch.jdbc.pool.size=6
batch.verify.cursor.position=true
batch.isolationlevel=ISOLATION_SERIALIZABLE
batch.table.prefix=BATCH_
I deploy this application to Tomcat 8, perform some jobs and then undeploy application via Tomcat Web Application Manager.
With Java VisualVM tool I compared memory snapshots before and after and see that there are a lot of Spring Batch(org.springframework.batch.*) related objects still exist in memory:
Also, I run 1000 reportJob and got a huge memory consumption on my machine.. I have no idea what can be wrong right now..
What could be causing this issue ?
UPDATED
I have consumed ~1000 messages from AWS SQS queue. My JMS listener configured to consume 1 message at a time. During the execution I had a following heap histogram:
I really don't understand why do I need for example to have in memory 7932 instances of StepExecution.. or 5285 of JobExecution objects. Where is my mistake ?
Related
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration(classes = { ApplicationBoot.class })
#SpringBootTest
#DirtiesContext(classMode=ClassMode.AFTER_CLASS)
public class ControlAdaptorTest {
#Autowired
private ControlAdaptorTest controlAdaptorTest;
#EndpointInject("mock:controlAdaptor")
protected MockEndpoint mockEndpoint;
#Autowired
private ProducerTemplate template;
#MockBean
private SupporTestss supporTestss;
private final String invalidMappingResultOrderDTOstr ="somejson";
#Test
public void testToMockTheEndPointsWithInvalidDTOjson() throws Exception {
List version=new ArrayList<>();
version.add("51");
template.sendBody(mockEndpoint, invalidMappingResultOrderDTOstr );
Mockito.when(supporTestss.getVersions(validResultOrderDTOstr)).thenReturn(version);
//Getting error in this line. mockEndpoint.getExchanges() this is return empty
controlAdaptorTest.process(mockEndpoint.getExchanges().get(0));
mockEndpoint.assertIsSatisfied();
}
}
//Below are configuration in my projectyour text`
/**From java 8 to java 11, spring boot 1. to 2.6.7, org.apache.camel version also changed to 2. to 3.16.0.
As CamelTestSupport is deprecated. trying different ways of mock the enpoint tried exchange is empty because of that im getting array index out of bound. Please help me in this. */``
I couldn't find any example to swtich between kafka cluster .
Anyone has implmeneted this class ABSwitchCluster from Spring Kafka.
https://docs.spring.io/spring-kafka/reference/html/
I tried with below code, but its not switching cluster.
#RestController
public class ApacheKafkaWebController {
#Autowired
ConsumerKakfaConfiguration configuration;
#Autowired
private KafkaListenerEndpointRegistry registry;
#Autowired
private ABSwitchCluster switcher;
#GetMapping(value = "/switch")
public String producer() {
registry.stop();
switcher.secondary();
registry.start();
return "switched!";
}
}
and swticher bean here:
#Bean
public ABSwitchCluster switcher() {
return new ABSwitchCluster("127.0.0.1:9095", "127.0.0.1:9096");
}
Could you please tell me am I missing anything here?, still its running in 9095 port.
See this answer and this test.
Basically, you switch the cluster and reset the connections by stopping and starting listener containers and resetting the producer factory.
I want to test a job that I used to load as a SpringBootTest and SpringJunit4Runner. As I upgraded to JUnit 5 the jobLauncherTestUtils class no longer loads. The project is a Spring Batch application using Spring Boot 2.2.0.RELEASE. My main configuration is called AppConfig and I have configured the step and job as beans that I can autowire in the test class. However, the application context which used to load now longer loads even. The error tells me the job is not added to the jobLauncherTestUtils. I do not understand why the job can no longer be loaded when it could before. I'd appreciate some help in fixing this issue
src/main/com/indigo/search/config/AppConfig
#Bean
public Step orderIntakeStep() {
return stepBuilderFactory.get("orderIntakeStep")
.<Order, Order>chunk(30)
.reader(orderReader())
.processor(orderProcessor())
.writer(orderWriter())
.build();
}
#Bean(name = "orderIntakeJob")
public Job orderIntakeJob() {
return jobBuilderFactory.get("orderIntakeJob")
.incrementer(new RunIdIncrementer())
.flow(orderIntakeStep())
.end()
.build();
}
...
}
#ExtendWith(SpringExtension.class)
#SpringBatchTest
#Transactional(propagation = Propagation.NOT_SUPPORTED)
class OrderIntakeJobTest {
#Autowired
private JobLauncherTestUtils jobLauncherTestUtils;
#Autowired
private JobRepositoryTestUtils jobRepositoryTestUtils;
#Autowired
private JobLauncher jobLauncher;
#Autowired
Job orderIntakeJob;
...
#Before
public void initJob(){
jobLauncherTestUtils.setLauncher(jobLauncher);
jobLauncherTestUtils.setJobLauncher(jobLauncher);
jobLauncherTestUtils.setJobRepository(jobRepository);
jobLauncherTestUtils.setJob(orderIntakeJob);
j
}
From what you shared, there is nothing that imports a configuration class that contains the job under test in OrderIntakeJobTest. You should have something like:
#ExtendWith(SpringExtension.class)
#SpringBatchTest
#Transactional(propagation = Propagation.NOT_SUPPORTED)
#ContextConfiguration(classes = MyJobConfiguration.class) // this is where the job under test is defined
class OrderIntakeJobTest {
// ...
}
Hi I have been working in Spring batch recently and need some help.
1) I want to run my Job using multiple threads, hence I have used TaskExecutor as below,
#Bean
public TaskExecutor taskExecutor() {
SimpleAsyncTaskExecutor taskExecutor = new SimpleAsyncTaskExecutor();
taskExecutor.setConcurrencyLimit(4);
return taskExecutor;
}
#Bean
public Step myStep() {
return stepBuilderFactory.get("myStep")
.<MyEntity,AnotherEntity> chunk(1)
.reader(reader())
.processor(processor())
.writer(writer())
.taskExecutor(taskExecutor())
.throttleLimit(4)
.build();
}
but, while executing in can see below line in console.
o.s.b.c.l.support.SimpleJobLauncher : No TaskExecutor has been set, defaulting to synchronous executor.
What does this mean? However, while debugging I can see four SimpleAsyncExecutor threads running. Can someone shed some light on this?
2) I don't want to run my Batch application with the metadata tables that spring batch creates. I have tried adding spring.batch.initialize-schema=never. But it didn't work. I also saw some way to do this by using ResourcelessTransactionManager, MapJobRepositoryFactoryBean. But I have to make some database transactions for my job. So will it be alright if I use this?
Also I was able to do this by extending DefaultBatchConfigurer and overriding:
#Override
public void setDataSource(DataSource dataSource) {
// override to do not set datasource even if a datasource exist.
// initialize will use a Map based JobRepository (instead of database)
}
Please guide me further. Thanks.
Update:
My full configuration class here.
#EnableBatchProcessing
#EnableScheduling
#Configuration
public class MyBatchConfiguration{
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Autowired
public DataSource dataSource;
/* #Override
public void setDataSource(DataSource dataSource) {
// override to do not set datasource even if a datasource exist.
// initialize will use a Map based JobRepository (instead of database)
}*/
#Bean
public Step myStep() {
return stepBuilderFactory.get("myStep")
.<MyEntity,AnotherEntity> chunk(1)
.reader(reader())
.processor(processor())
.writer(writer())
.taskExecutor(executor())
.throttleLimit(4)
.build();
}
#Bean
public Job myJob() {
return jobBuilderFactory.get("myJob")
.incrementer(new RunIdIncrementer())
.listener(listener())
.flow(myStep())
.end()
.build();
}
#Bean
public MyJobListener myJobListener()
{
return new MyJobListener();
}
#Bean
public ItemReader<MyEntity> reader()
{
return new MyReader();
}
#Bean
public ItemWriter<? super AnotherEntity> writer()
{
return new MyWriter();
}
#Bean
public ItemProcessor<MyEntity,AnotherEntity> processor()
{
return new MyProcessor();
}
#Bean
public TaskExecutor taskExecutor() {
SimpleAsyncTaskExecutor taskExecutor = new SimpleAsyncTaskExecutor();
taskExecutor.setConcurrencyLimit(4);
return taskExecutor;
}}
In the future, please break this up into two independent questions. That being said, let me shed some light on both questions.
SimpleJobLauncher : No TaskExecutor has been set, defaulting to synchronous executor.
Your configuration is configuring myStep to use your TaskExecutor. What that does is it causes Spring Batch to execute each chunk in it's own thread (based on the parameters of the TaskExecutor). The log message you are seeing has nothing to do with that behavior. It has to do with launching your job. By default, the SimpleJobLauncher will launch the job on the same thread it is running on, thereby blocking that thread. You can inject a TaskExecutor into the SimpleJobLauncher which will cause the job to be executed on a different thread from the JobLauncher itself. These are two separate uses of multiple threads by the framework.
I don't want to run my Batch application with the metadata tables that spring batch creates
The short answer here is to just use an in memory database like HSQLDB or H2 for your metadata tables. This provides a production grade data store (so that concurrency is handled correctly) without actually persisting the data. If you use the ResourcelessTransactionManager, you are effectively turning transactions off (a bad idea if you're using a database in any capacity) because that TransactionManager doesn't actually do anything (it's a no-op implementation).
I am trying to register the # of spring boot apps I've in my pvt cloud environment. Logic is to use Counter metric to increment during startUp and decrement during shut down. All the different deployments will publish to the same metricPreFix(--assumption). Following is the graph I get in Graphite:
#application.properties
spring.metrics.export.delay-millis=100
Why do I see the value to come down to 0 even when the app is running? I have tried with 2 different implementations with same result. Can someone please point out the gap in my understanding? PFB the code
#Component
public class AppStartupBean implements CommandLineRunner {
private static final String appMetricName = "MyApp.currentCount.GraphOne";
private static final String metricName = "MyApp.currentCount.GraphTwo";
#Autowired
DropwizardMetricServices dwMetricService;
#Autowired
private MetricRegistry registry;
#Override
public void run(String... arg0) throws Exception {
dwMetricService.increment(appMetricName);
Counter counter = registry.counter(metricName);
counter.inc();
}
}
The configuration for DropwizardMetricServices was wrong. I was using
#Bean
public DropwizardMetricServices dwMetricService(MetricRegistry registry) {
return new DropwizardMetricServices(registry);
}
Instead we should just #Autowire DropwizardMetricServices as needed. PFB
When Dropwizard metrics are in use, the default CounterService and
GaugeService are replaced with a DropwizardMetricServices, which is a
wrapper around the MetricRegistry (so you can #Autowired one of those
services and use it as normal).