Spring Batch - Read multiple files from S3 - spring

Like for reading single file in spring batch from s3, we use
#Bean
public FlatFileItemReader<Map<String, Object>> itemReader() {
FlatFileItemReader<Map<String, Object>> reader = new FlatFileItemReader<>();
reader.setLineMapper(new JsonLineMapper());
reader.setRecordSeparatorPolicy(new JsonRecordSeparatorPolicy());
reader.setResource(resourceLoader.getResource("s3://" + amazonS3Bucket + "/" + file));
return reader;
}
But if i want to read all the files from some specific folder/key then is there something to MultiResourceItemReader, like below(which we use for local filesystem)
MultiResourceItemReader<UserData> reader = new MultiResourceItemReader<>();
reader.setResources(resources);

No, it is up to you to create the Resource array and pass it to the MultiResourceItemReader.

Create a MultiResourceItemReader like this,
#Autowired
private AmazonS3 s3;
#Autowired
private ResourceLoader resourceLoader;
public MultiResourceItemReader<String> fileItemReader() throws Exception {
List<Resource> resourceList = new ArrayList<>();
String s3ResponseFilePath = "s3://bucket/path/"; //put your s3 path here
//TODO: warn: this functn can only return max 1000 objects
s3objects = s3.listObjects("bucket", s3ResponseFilePath).getObjectSummaries();
for(S3ObjectSummary it:s3objects)
resourceList.add(resourceLoader.getResource( "s3://" + s3Config.getBucket() + "/" + it.getKey()));
Resource[] resources = resourceList.toArray(new Resource[resourceList.size()]);
MultiResourceItemReader<String> reader = new MultiResourceItemReader<>();
reader.setResources(resources);
reader.setDelegate(flatFileItemReader());
return reader;
}
This reader will need a delegate and lineMapper, you can implement it like this,
private FlatFileItemReader<String> flatFileItemReader() throws Exception {
FlatFileItemReader<String> reader = new FlatFileItemReader<>();
JsonLineMapper lineMapper = new JsonLineMapper();
reader.setLineMapper(lineMapper);
reader.afterPropertiesSet();
return reader;
}
public class JsonLineMapper implements LineMapper<String> {
private ObjectMapper mapper = new ObjectMapper();
#Override
public String mapLine(String s, int i) throws Exception {
return s;
}
}

Related

JwtAuthenticationToken is not in the allowlist, Jackson issue

I have created my authorization server using org.springframework.security:spring-security-oauth2-authorization-server:0.2.2 and my client using org.springframework.boot:spring-boot-starter-oauth2-client. The users are able to sign in and out successfully, however, while testing I noticed that if I log in successfully then restart the client (but not the server) without signing out and try to login in again the server throws the following error in an endless loop of redirects
java.lang.IllegalArgumentException: The class with org.springframework.security.oauth2.server.resource.authentication.JwtAuthenticationToken and name of org.springframework.security.oauth2.server.resource.authentication.JwtAuthenticationToken is not in the allowlist. If you believe this class is safe to deserialize, please provide an explicit mapping using Jackson annotations or by providing a Mixin. If the serialization is only done by a trusted source, you can also enable default typing. See https://github.com/spring-projects/spring-security/issues/4370 for details
I tried to follow this link https://github.com/spring-projects/spring-security/issues/4370 but the solution on it did not work for me. I also tried a different solution described in this link https://github.com/spring-projects/spring-authorization-server/issues/397#issuecomment-900148920 and modified my authorization server code as follows:-
Here is my Jackson Configs
#Configuration
public class JacksonConfiguration {
/**
* Support for Java date and time API.
*
* #return the corresponding Jackson module.
*/
#Bean
public JavaTimeModule javaTimeModule() {
return new JavaTimeModule();
}
#Bean
public Jdk8Module jdk8TimeModule() {
return new Jdk8Module();
}
/*
* Support for Hibernate types in Jackson.
*/
#Bean
public Hibernate5Module hibernate5Module() {
return new Hibernate5Module();
}
/*
* Module for serialization/deserialization of RFC7807 Problem.
*/
#Bean
public ProblemModule problemModule() {
return new ProblemModule();
}
/*
* Module for serialization/deserialization of ConstraintViolationProblem.
*/
#Bean
public ConstraintViolationProblemModule constraintViolationProblemModule() {
return new ConstraintViolationProblemModule();
}
/**
* To (de)serialize a BadCredentialsException, use CoreJackson2Module:
*/
#Bean
public CoreJackson2Module coreJackson2Module() {
return new CoreJackson2Module();
}
#Bean
#Primary
public ObjectMapper objectMapper() {
ObjectMapper mapper = new ObjectMapper();
mapper.registerModule(coreJackson2Module());
mapper.registerModule(javaTimeModule());
mapper.registerModule(jdk8TimeModule());
mapper.registerModule(hibernate5Module());
mapper.registerModule(problemModule());
mapper.registerModule(constraintViolationProblemModule());
return mapper;
}
}
and here is my Authorization server config
#Configuration(proxyBeanMethods = false)
public class AuthServerConfig {
private final DataSource dataSource;
private final AuthProperties authProps;
private final PasswordEncoder encoder;
public AuthServerConfig(DataSource dataSource, AuthProperties authProps, PasswordEncoder encoder) {
this.dataSource = dataSource;
this.authProps = authProps;
this.encoder = encoder;
}
#Bean
public JdbcTemplate jdbcTemplate() {
return new JdbcTemplate(dataSource);
}
#Bean
#Order(Ordered.HIGHEST_PRECEDENCE)
public SecurityFilterChain authServerSecurityFilterChain(HttpSecurity http) throws Exception {
OAuth2AuthorizationServerConfigurer<HttpSecurity> authorizationServerConfigurer =
new OAuth2AuthorizationServerConfigurer<>();
authorizationServerConfigurer.tokenRevocationEndpoint(tokenRevocationEndpoint -> tokenRevocationEndpoint
.revocationResponseHandler((request, response, authentication) -> {
Assert.notNull(request, "HttpServletRequest required");
HttpSession session = request.getSession(false);
if (!Objects.isNull(session)) {
session.removeAttribute("SPRING_SECURITY_CONTEXT");
session.invalidate();
}
SecurityContextHolder.getContext().setAuthentication(null);
SecurityContextHolder.clearContext();
response.setStatus(HttpStatus.OK.value());
})
);
RequestMatcher endpointsMatcher = authorizationServerConfigurer.getEndpointsMatcher();
http
.requestMatcher(endpointsMatcher)
.authorizeRequests(authorizeRequests -> authorizeRequests.anyRequest().authenticated())
.csrf(csrf -> csrf.ignoringRequestMatchers(endpointsMatcher))
.apply(authorizationServerConfigurer);
return http.formLogin(Customizer.withDefaults()).build();
}
#Bean
public RegisteredClientRepository registeredClientRepository(JdbcTemplate jdbcTemplate, TokenSettings tokenSettings) {
JdbcRegisteredClientRepository clientRepository = new JdbcRegisteredClientRepository(jdbcTemplate);
RegisteredClient webClient = RegisteredClient.withId("98a9104c-a9c7-4d7c-ad03-ec61bcfeab36")
.clientId(authProps.getClientId())
.clientName(authProps.getClientName())
.clientSecret(encoder.encode(authProps.getClientSecret()))
.clientAuthenticationMethod(ClientAuthenticationMethod.CLIENT_SECRET_BASIC)
.clientAuthenticationMethod(ClientAuthenticationMethod.CLIENT_SECRET_POST)
.authorizationGrantType(AuthorizationGrantType.AUTHORIZATION_CODE)
.authorizationGrantType(AuthorizationGrantType.CLIENT_CREDENTIALS)
.authorizationGrantType(AuthorizationGrantType.REFRESH_TOKEN)
.redirectUri("http://127.0.0.1:8000/login/oauth2/code/web-client")
.scope(OidcScopes.OPENID)
.scope(OidcScopes.PROFILE)
.tokenSettings(tokenSettings)
.build();
clientRepository.save(webClient);
return clientRepository;
}
#Bean
public OAuth2AuthorizationService authorizationService(JdbcTemplate jdbcTemplate,
RegisteredClientRepository registeredClientRepository,
ObjectMapper objectMapper) {
JdbcOAuth2AuthorizationService authorizationService =
new JdbcOAuth2AuthorizationService(jdbcTemplate, registeredClientRepository);
JdbcOAuth2AuthorizationService.OAuth2AuthorizationRowMapper rowMapper = new JdbcOAuth2AuthorizationService.OAuth2AuthorizationRowMapper(registeredClientRepository);
ClassLoader classLoader = JdbcOAuth2AuthorizationService.class.getClassLoader();
objectMapper.registerModules(SecurityJackson2Modules.getModules(classLoader));
objectMapper.registerModule(new OAuth2AuthorizationServerJackson2Module());
// You will need to write the Mixin for your class so Jackson can marshall it.
// objectMapper.addMixIn(UserPrincipal .class, UserPrincipalMixin.class);
rowMapper.setObjectMapper(objectMapper);
authorizationService.setAuthorizationRowMapper(rowMapper);
return authorizationService;
}
#Bean
public OAuth2AuthorizationConsentService authorizationConsentService(JdbcTemplate jdbcTemplate,
RegisteredClientRepository registeredClientRepository) {
return new JdbcOAuth2AuthorizationConsentService(jdbcTemplate, registeredClientRepository);
}
#Bean
public JWKSource<SecurityContext> jwkSource() {
RSAKey rsaKey = generateRsa();
JWKSet jwkSet = new JWKSet(rsaKey);
return (jwkSelector, securityContext) -> jwkSelector.select(jwkSet);
}
private static RSAKey generateRsa() {
KeyPair keyPair = generateRsaKey();
RSAPublicKey publicKey = (RSAPublicKey) keyPair.getPublic();
RSAPrivateKey privateKey = (RSAPrivateKey) keyPair.getPrivate();
return new RSAKey.Builder(publicKey)
.privateKey(privateKey)
.keyID(UUID.randomUUID().toString())
.build();
}
private static KeyPair generateRsaKey() {
KeyPair keyPair;
try {
KeyPairGenerator keyPairGenerator = KeyPairGenerator.getInstance("RSA");
keyPairGenerator.initialize(2048);
keyPair = keyPairGenerator.generateKeyPair();
} catch (Exception ex) {
throw new IllegalStateException(ex);
}
return keyPair;
}
#Bean
public ProviderSettings providerSettings() {
return ProviderSettings.builder()
.issuer(authProps.getIssuerUri())
.build();
}
#Bean
public TokenSettings tokenSettings() {
return TokenSettings.builder()
.accessTokenTimeToLive(Duration.ofDays(1))
.refreshTokenTimeToLive(Duration.ofDays(1))
.build();
}
}
But am still facing the same issue.
How do I solve this? Any assistance is highly appreciated.
After trying out different solutions this was how I was able to solve it.
I changed my OAuth2AuthorizationService bean to look like this.
#Bean
public OAuth2AuthorizationService authorizationService(JdbcTemplate jdbcTemplate,
RegisteredClientRepository registeredClientRepository) {
JdbcOAuth2AuthorizationService authorizationService =
new JdbcOAuth2AuthorizationService(jdbcTemplate, registeredClientRepository);
JdbcOAuth2AuthorizationService.OAuth2AuthorizationRowMapper rowMapper =
new JdbcOAuth2AuthorizationService.OAuth2AuthorizationRowMapper(registeredClientRepository);
JdbcOAuth2AuthorizationService.OAuth2AuthorizationParametersMapper oAuth2AuthorizationParametersMapper =
new JdbcOAuth2AuthorizationService.OAuth2AuthorizationParametersMapper();
ObjectMapper objectMapper = new ObjectMapper();
ClassLoader classLoader = JdbcOAuth2AuthorizationService.class.getClassLoader();
List<Module> securityModules = SecurityJackson2Modules.getModules(classLoader);
objectMapper.registerModules(securityModules);
objectMapper.registerModule(new OAuth2AuthorizationServerJackson2Module());
objectMapper.addMixIn(JwtAuthenticationToken.class, JwtAuthenticationTokenMixin.class);
rowMapper.setObjectMapper(objectMapper);
oAuth2AuthorizationParametersMapper.setObjectMapper(objectMapper);
authorizationService.setAuthorizationRowMapper(rowMapper);
authorizationService.setAuthorizationParametersMapper(oAuth2AuthorizationParametersMapper);
return authorizationService;
}
and here is my JwtAuthenticationTokenMixin configurations
#JsonTypeInfo(use = JsonTypeInfo.Id.CLASS)
#JsonDeserialize(using = JwtAuthenticationTokenDeserializer.class)
#JsonAutoDetect(
fieldVisibility = JsonAutoDetect.Visibility.ANY,
getterVisibility = JsonAutoDetect.Visibility.NONE,
isGetterVisibility = JsonAutoDetect.Visibility.NONE)
#JsonIgnoreProperties(ignoreUnknown = true)
public abstract class JwtAuthenticationTokenMixin {}
class JwtAuthenticationTokenDeserializer extends JsonDeserializer<JwtAuthenticationToken> {
#Override
public JwtAuthenticationToken deserialize(JsonParser parser, DeserializationContext context) throws IOException {
ObjectMapper mapper = (ObjectMapper) parser.getCodec();
JsonNode root = mapper.readTree(parser);
return deserialize(parser, mapper, root);
}
private JwtAuthenticationToken deserialize(JsonParser parser, ObjectMapper mapper, JsonNode root)
throws JsonParseException {
JsonNode principal = JsonNodeUtils.findObjectNode(root, "principal");
if (!Objects.isNull(principal)) {
String tokenValue = principal.get("tokenValue").textValue();
long issuedAt = principal.get("issuedAt").longValue();
long expiresAt = principal.get("expiresAt").longValue();
Map<String, Object> headers = JsonNodeUtils.findValue(
principal, "headers", JsonNodeUtils.STRING_OBJECT_MAP, mapper);
Map<String, Object> claims = new HashMap<>();
claims.put("claims", principal.get("claims"));
Jwt jwt = new Jwt(tokenValue, Instant.ofEpochMilli(issuedAt), Instant.ofEpochMilli(expiresAt), headers, claims);
return new JwtAuthenticationToken(jwt);
}
return null;
}
}
abstract class JsonNodeUtils {
static final TypeReference<Set<String>> STRING_SET = new TypeReference<Set<String>>() {
};
static final TypeReference<Map<String, Object>> STRING_OBJECT_MAP = new TypeReference<Map<String, Object>>() {
};
static String findStringValue(JsonNode jsonNode, String fieldName) {
if (jsonNode == null) {
return null;
}
JsonNode value = jsonNode.findValue(fieldName);
return (value != null && value.isTextual()) ? value.asText() : null;
}
static <T> T findValue(JsonNode jsonNode, String fieldName, TypeReference<T> valueTypeReference,
ObjectMapper mapper) {
if (jsonNode == null) {
return null;
}
JsonNode value = jsonNode.findValue(fieldName);
return (value != null && value.isContainerNode()) ? mapper.convertValue(value, valueTypeReference) : null;
}
static JsonNode findObjectNode(JsonNode jsonNode, String fieldName) {
if (jsonNode == null) {
return null;
}
JsonNode value = jsonNode.findValue(fieldName);
return (value != null && value.isObject()) ? value : null;
}
}
you don't need to create a Mixin, because it's all ready created by authorization springboot module. juste
#Bean
public OAuth2AuthorizationService authorizationService(JdbcTemplate jdbcTemplate, RegisteredClientRepository registeredClientRepository) {
JdbcOAuth2AuthorizationService authorizationService = new JdbcOAuth2AuthorizationService(jdbcTemplate, registeredClientRepository);
JdbcOAuth2AuthorizationService.OAuth2AuthorizationRowMapper rowMapper = new JdbcOAuth2AuthorizationService.OAuth2AuthorizationRowMapper(registeredClientRepository);
ClassLoader classLoader = JdbcOAuth2AuthorizationService.class.getClassLoader();
ObjectMapper objectMapper = new ObjectMapper();
objectMapper.registerModules(new CoreJackson2Module());
objectMapper.registerModules(SecurityJackson2Modules.getModules(classLoader));
objectMapper.registerModule(new OAuth2AuthorizationServerJackson2Module());
rowMapper.setObjectMapper(objectMapper);
authorizationService.setAuthorizationRowMapper(rowMapper);
return authorizationService;
}
i think you miss this line and is where the token mixin is registered
objectMapper.registerModules(new CoreJackson2Module());

Saving file information in Spring batch MultiResourceItemReader

I have a directory having text files. I want to process files and write data into db. I did that by using MultiResourceItemReader.
I have a scenario like whenever file is coming, the first step is to save file info, like filename, record count in file in a log table(custom table).
Since i used MultiResourceItemReader, It's loading all files once and the code which i wrote is executing once in server startup. I tried with getCurrentResource() method but its returning null.
Please refer below code.
NetFileProcessController.java
#Slf4j
#RestController
#RequestMapping("/netProcess")
public class NetFileProcessController {
#Autowired
private JobLauncher jobLauncher;
#Autowired
#Qualifier("netFileParseJob")
private Job job;
#GetMapping(path = "/process")
public #ResponseBody StatusResponse process() throws ServiceException {
try {
Map<String, JobParameter> parameters = new HashMap<>();
parameters.put("date", new JobParameter(new Date()));
jobLauncher.run(job, new JobParameters(parameters));
return new StatusResponse(true);
} catch (Exception e) {
log.error("Exception", e);
Throwable rootException = ExceptionUtils.getRootCause(e);
String errMessage = rootException.getMessage();
log.info("Root cause is instance of JobInstanceAlreadyCompleteException --> "+(rootException instanceof JobInstanceAlreadyCompleteException));
if(rootException instanceof JobInstanceAlreadyCompleteException){
log.info(errMessage);
return new StatusResponse(false, "This job has been completed already!");
} else{
throw new ServiceException(errMessage);
}
}
}
}
BatchConfig.java
#Configuration
#EnableBatchProcessing
public class BatchConfig {
private JobBuilderFactory jobBuilderFactory;
#Autowired
public void setJobBuilderFactory(JobBuilderFactory jobBuilderFactory) {
this.jobBuilderFactory = jobBuilderFactory;
}
#Autowired
StepBuilderFactory stepBuilderFactory;
#Value("file:${input.files.location}${input.file.pattern}")
private Resource[] netFileInputs;
#Value("${net.file.column.names}")
private String netFilecolumnNames;
#Value("${net.file.column.lengths}")
private String netFileColumnLengths;
#Autowired
NetFileInfoTasklet netFileInfoTasklet;
#Autowired
NetFlatFileProcessor netFlatFileProcessor;
#Autowired
NetFlatFileWriter netFlatFileWriter;
#Bean
public Job netFileParseJob() {
return jobBuilderFactory.get("netFileParseJob")
.incrementer(new RunIdIncrementer())
.start(netFileStep())
.build();
}
public Step netFileStep() {
return stepBuilderFactory.get("netFileStep")
.<NetDetailsDTO, NetDetailsDTO>chunk(1)
.reader(new NetFlatFileReader(netFileInputs, netFilecolumnNames, netFileColumnLengths))
.processor(netFlatFileProcessor)
.writer(netFlatFileWriter)
.build();
}
}
NetFlatFileReader.java
#Slf4j
public class NetFlatFileReader extends MultiResourceItemReader<NetDetailsDTO> {
public netFlatFileReader(Resource[] netFileInputs, String netFilecolumnNames, String netFileColumnLengths) {
setResources(netFileInputs);
setDelegate(reader(netFilecolumnNames, netFileColumnLengths));
}
private FlatFileItemReader<NetDetailsDTO> reader(String netFilecolumnNames, String netFileColumnLengths) {
FlatFileItemReader<NetDetailsDTO> flatFileItemReader = new FlatFileItemReader<>();
FixedLengthTokenizer tokenizer = CommonUtil.fixedLengthTokenizer(netFilecolumnNames, netFileColumnLengths);
FieldSetMapper<NetDetailsDTO> mapper = createMapper();
DefaultLineMapper<NetDetailsDTO> lineMapper = new DefaultLineMapper<>();
lineMapper.setLineTokenizer(tokenizer);
lineMapper.setFieldSetMapper(mapper);
flatFileItemReader.setLineMapper(lineMapper);
return flatFileItemReader;
}
/*
* Mapping column data to DTO
*/
private FieldSetMapper<NetDetailsDTO> createMapper() {
BeanWrapperFieldSetMapper<NetDetailsDTO> mapper = new BeanWrapperFieldSetMapper<>();
try {
mapper.setTargetType(NetDetailsDTO.class);
} catch(Exception e) {
log.error("Exception in mapping column data to dto ", e);
}
return mapper;
}
}
I am stuck on this scenario, Any help appreciated
I don't think MultiResourceItemReader is appropriate in your case. I would run a job per file for all the reasons of making one thing do one thing and do it well:
Your preparatory step will work by design
It would be easier to run multiple jobs in parallel and improve your file ingestion throughput
In case of failure, you would only restart the job for the failed file
EDIT: add an example
Resource[] netFileInputs = ... // same code that looks for file as currently in your reader
for (Resource netFileInput : netFileInputs) {
Map<String, JobParameter> parameters = new HashMap<>();
parameters.put("netFileInput", new JobParameter(netFileInput.getFilename()));
jobLauncher.run(job, new JobParameters(parameters));
}

Springboot Batch get CSV from rest API

I have a scenario for Springboot batch application where I need to read the CSV-11 and write to another CSV-2, the problem arises when I need to get the CSV-1 from a REST endpoint, for e.g. when I initiate the job it should fetch a CSV-1 from an endpoint and then continue the batch processing to write to CSV-2.
But it seems like to process a CSV-1 we need to feed the 'Resource' in advance while application startup [I am fairly new to batch so not sure if it's 100% correct].
Can anyone please guide me the right approach to resolve this?
EDIT : Adding code, I was able to resolve it but need advice if it's the right way to do things (please ignore the hard-coded data).
#Configuration
#EnableBatchProcessing
public class BatchConfig {
#Autowired
private JobBuilderFactory jobbuilder;
#Autowired
private StepBuilderFactory stepbuilder;
#Autowired
private CSVProcessor processor;
#Autowired
private CustomSkipPolicy customSkipPolicy;
#Autowired
private CSVResponse response;
#Bean(name="csv")
public Job job_csv() throws Exception {
Step step = stepbuilder
.get("csv-step")
.<Person, Person>chunk(5)
.reader(new CSVReader(response.getResource()))
.processor(processor)
.writer(new CSVWriter().write(response.getFilename()))
.faultTolerant()
.skipPolicy(customSkipPolicy)
.listener(new StepListener())
.build();
return jobbuilder
.get("csv-job")
.incrementer(new RunIdIncrementer())
.listener(new JobListener())
.flow(step)
.end()
.build();
}
}
#Slf4j
public class CSVReader extends FlatFileItemReader<Person> {
public CSVReader(Resource resource) throws Exception {
super();
setResource(resource);
setStrict(false);
setLinesToSkip(1);
doOpen();
DelimitedLineTokenizer dlt = new DelimitedLineTokenizer();
dlt.setNames(new String[] {"id","first_name","last_name","email","gender","ip_address","dob"});
dlt.setDelimiter(",");
dlt.setStrict(false);
BeanWrapperFieldSetMapper<Person> fsp = new BeanWrapperFieldSetMapper<>();
fsp.setTargetType(Person.class);
DefaultLineMapper<Person> dlp = new DefaultLineMapper<>();
dlp.setLineTokenizer(dlt);
dlp.setFieldSetMapper(fsp);
setLineMapper(dlp);
}
}
public class CSVWriter {
private static final String DATA_PROCESSED = "C:/data/processed";
public FlatFileItemWriter<Person> write(String filename) throws Exception {
FlatFileItemWriter<Person> writer = new FlatFileItemWriter<>();
writer.setResource(resource(filename));
writer.setHeaderCallback(new Header());
BeanWrapperFieldExtractor<Person> fe = new BeanWrapperFieldExtractor<>();
fe.setNames(new String[] { "id", "first_name", "last_name", "age" });
DelimitedLineAggregator<Person> dla = new DelimitedLineAggregator<>();
dla.setDelimiter(",");
dla.setFieldExtractor(fe);
writer.setLineAggregator(dla);
return writer;
}
private Resource resource(String filename) throws IOException {
String processed_file = filename.replace(".csv", "").concat("_PROCESSED").concat(".csv");
if (Files.notExists(Paths.get(DATA_PROCESSED), new LinkOption[] { LinkOption.NOFOLLOW_LINKS })) {
Files.createDirectory(Paths.get(DATA_PROCESSED));
}
if (Files.notExists(Paths.get(DATA_PROCESSED + "/" + processed_file),
new LinkOption[] { LinkOption.NOFOLLOW_LINKS })) {
Files.createFile(Paths.get(DATA_PROCESSED + "/" + processed_file));
}
return new FileSystemResource(DATA_PROCESSED + "/" + processed_file);
}
}
#Component
#Slf4j
public class CSVResponse {
private static final String DATA_RECEIVE = "C:/data/receive";
private String filename;
public Resource getResource() throws IOException {
ResponseEntity<Resource> resp = new RestTemplate().getForEntity("http://localhost:8081/csv", Resource.class);
log.info("File Received : "+resp.getBody().getFilename());
setFilename(resp.getBody().getFilename());
Path path = Paths.get(DATA_RECEIVE);
if(!Files.exists(path, new LinkOption[]{ LinkOption.NOFOLLOW_LINKS})) {
Files.createDirectory(path);
}
Files.copy(resp.getBody().getInputStream(), Paths.get(DATA_RECEIVE + "/" +resp.getBody().getFilename()), StandardCopyOption.REPLACE_EXISTING);
return resp.getBody();
}
public String getFilename() {
return filename;
}
public void setFilename(String filename) {
this.filename = filename;
}
}
You can implement item reader with StepExecutionListener interface, and you have to implement the fetching CSV-1 file via RestTemplate in beforeStep() method of the item reader.
You can refer how to use RestTemplate from https://www.baeldung.com/rest-template

The FlatFileItemReader read only one line from the CSV file - Spring Batch

I'm creating a Spring Batch Job to populate Data into a Database table from a given CSV file.
I created a customized FlatFileItemReader.
my problem is that the read() method is invoked only one time so only the first line of my CSV file is inserted into the database.
#Configuration
#EnableBatchProcessing
public class SpringBatchConfig {
private MultipartFile[] files;
#Bean
public Job job(JobBuilderFactory jobBuilderFactory, StepBuilderFactory stepBuilderFactory,
ItemReader<MyModelEntity> itemReader,
ItemWriter<MyModelEntity> itemWriter) {
Step step = stepBuilderFactory.get("Load-CSV-file_STP")
.<MyModelEntity, MyModelEntity > chunk(12)
.reader(itemReader)
.writer(itemWriter).build();
return jobBuilderFactory.get("Load-CSV-Files").
incrementer(new RunIdIncrementer()) /
.start(step)
.build();
}
#Bean
ItemReader<MyModelEntity> myModelCsvReader() throws Exception {
return new MyModelCsvReader();
}
}
the myModelCsvReader
#Component
#StepScope
public class MyModelCsvReader implements ItemReader<MyModelEntity>{
#Value("#{jobParameters['SDH']}")
private String sdhPath;
private boolean batchJobState= false;
#Autowired
MyModelFieldSetMapper myModelFieldSetMapper;
public LineMapper<MyModelEntity> lineMapper() throws Exception {
DefaultLineMapper<MyModelEntity> defaultLineMapper = new
DefaultLineMapper<MyModelEntity>();
DelimitedLineTokenizer lineTokenizer = new DelimitedLineTokenizer();
lineTokenizer.setDelimiter(",");
lineTokenizer.setStrict(false);
lineTokenizer.setNames(new String[]
{
"clientId","ddId","institName","progName",
"qual","startDate","endDate","eType", "country","comments"
});
defaultLineMapper.setLineTokenizer(lineTokenizer);
defaultLineMapper.setFieldSetMapper(myModelFieldSetMapper);
return defaultLineMapper;}
#Override
public MyModelEntity read()
throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException {
//if(!batchJobState )
{
FlatFileItemReader<MyModelEntity> flatFileItemReader = new
FlatFileItemReader<MyModelEntity>();
flatFileItemReader.setMaxItemCount(2000);
flatFileItemReader.setResource(new UrlResource("file:\\"+sdhPath));
flatFileItemReader.setName("CSV-Reader");
flatFileItemReader.setLinesToSkip(1);
flatFileItemReader.setLineMapper(lineMapper());
flatFileItemReader.open(new ExecutionContext());
batchJobState=true;
return flatFileItemReader.read();
}
// return null;
}
}
the FieldSetMapper Implementation
#Component
public class MyModelFieldSetMapper implements FieldSetMapper<MyModelEntity> {
//private SiteService siteService =BeanUtil.getBean(SiteServiceImpl.class);
#Autowired
private SiteService siteService;
#Override
public MyModelEntity mapFieldSet(FieldSet fieldSet ) throws BindException {
if(fieldSet == null){
return null;
}
MyModelEntity educationHistory = new MyModelEntity();
// setting MyModelAttributes Values
return myModel;
}
}
any conribution is welcomed . thanks
// thereader after extending FlatFileItemReader
#Component
#StepScope
public class CustomUserItemReader extends FlatFileItemReader<User> {
#Value("#{jobParameters['UserCSVPath']}")
private String UserCSVPath;
private boolean batchJobState;
public CustomUserItemReader() throws Exception {
super();
setResource(new UrlResource("file:\\"+UserCSVPath));
setLineMapper(lineMapper());
afterPropertiesSet();
setStrict(false);
}
public LineMapper<User> lineMapper() throws Exception {
DefaultLineMapper<User> defaultLineMapper =
new DefaultLineMapper<User>();
DelimitedLineTokenizer lineTokenizer = new DelimitedLineTokenizer();
lineTokenizer.setDelimiter(",");
lineTokenizer.setStrict(false);
lineTokenizer.setNames(new String[]{"name", "dept",
"salary","endDate"});
defaultLineMapper.setLineTokenizer(lineTokenizer);
defaultLineMapper.setFieldSetMapper(new CustomUserFieldSetMapper());
//defaultLineMapper.setFieldSetMapper(fieldSetMapper);
return defaultLineMapper;}
#Override
public User read()
throws Exception, UnexpectedInputException, ParseException,
NonTransientResourceException {
//if(!batchJobState )
{
flatFileItemReader).size())
// flatFileItemReader.setMaxItemCount(2000);
this.setResource(new UrlResource("file:\\"+UserCSVPath));
this.setName("CSV-Reader");
this.setLinesToSkip(1);
//flatFileItemReader.setLineMapper(lineMapper());
this.open(new ExecutionContext());
User e = this.read();
batchJobState = true;
return e ;
}
// return null;
}
public
String getUserCSVPath() {
return UserCSVPath;
}
public
void setUserCSVPath(String userCSVPath) {
UserCSVPath = userCSVPath;
}
}
Thanks for all your suggestions, even if i have implemented ItemReader<>. I fixed the problem by moving the instantiation of the FlatFileItemReader Out from the read() method.
that was creating a new FlatFileItemReader in each loop and reading only the first line for each object created .
thanks

Spring batch accessing job parameter returning null value even though stepscope is used

I have read this forum with lots of examples on how to use jobparameters but still got stuck as i am getting null value when retrieving jobparameter
Below is my code so far
Controller :
#RestController
#CrossOrigin
#RequestMapping("/bulkimport/api/")
public class BatchController {
#Autowired
public JobLauncher jobLauncher;
#Autowired
public Job bulkImportJob;
#RequestMapping(value = "/bulkProcessjob", method = RequestMethod.POST)
public void batchImportProcess(#RequestHeader(value = "X-Auth-Token") String jwtToken,
#RequestParam("excelfile") final MultipartFile excelfile
) throws IOException{
String path = new ClassPathResource("importFileFolder/").getURL().getPath();
File fileToImport = new File(path + excelfile.getOriginalFilename());
try {
Map<String, JobParameter> jobParametersMap = new HashMap<String, JobParameter>();
jobParametersMap.put("processName", new JobParameter("processName"));
jobParametersMap.put("operation", new JobParameter("operation"));
jobParametersMap.put("filePath", new JobParameter(fileToImport.getAbsolutePath()));
System.out.println(fileToImport.getAbsolutePath()+"filePath:::::::::::");
jobLauncher.run(bulkImportJob, new JobParameters(jobParametersMap));
}catch(Exception e){
}
}
}
Config class:
public String filePath;
#Bean(name = "bulkImportJob")
public Job bulkImportJob() {
return jobBuilderFactory.get("bulkImportJob").incrementer(new RunIdIncrementer()).flow(bulkImportProcessStep()).end().build();
}
#Bean
public Step bulkImportProcessStep() {
return stepBuilderFactory.get("bulkImportProcessStep").<String, String> chunk(1).reader(validateFileHeader(filePath)).writer(new TempWriter()).build();
}
#Bean
#StepScope
BulkUploadExcelHeader validateFileHeader(#Value("#{jobParameters[filePath]}") String filePath) {
System.out.println("filePath value::::::::::::::::::::::::::::::::"+filePath);
return new BulkUploadExcelHeader(filePath);
}
in above print statement filePath value is printed as null
can anyone please suggest what is wrong in above code
Most probably you need to enclose filePath with single quotes:
validateFileHeader(#Value("#{jobParameters['filePath']}")

Resources