I am working on creating excel file from data, for that I have created job. I want to set hashmap to the jobparameter so that I can use it in MyReader class, I have created CustomJobParameter Class.
Below code you can find to get the job parameters :
Get Job Parameters :
public JobParameters createJobParam (MyRequest request) {
final JobParameters parameters = new JobParametersBuilder()
.addString("MyParam1", request.getReportGenerationJobId())
.addString("MyParam2", request.getSessionId())
.addLong("time", System.currentTimeMillis())
.addParameter(
"MyObject",
new MyUtils.CustomJobParameter(request.getHsSlideArticles())
)
.toJobParameters();
return JobParameters;
}
CustomJobParameter Class written in MyUtils class:
public static class CustomJobParameter<T extends Serializable> extends JobParameter {
private HashMap customParam;
public CustomJobParameter (HashMap slideArticles) {
super("");
this.customParam = customParam;
}
public HashMap getValue () {
return customParam;
}
}
But while I am setting using custom parameters, it setting blank string, not object I am passing.
How can I pass the hashmap to my reader.
According to the documentation for JobParameter, a JobParameter can only be String, Long, Date, and Double.
https://docs.spring.io/spring-batch/docs/current/api/org/springframework/batch/core/JobParameter.html
Domain representation of a parameter to a batch job. Only the following types can be parameters: String, Long, Date, and Double.
The identifying flag is used to indicate if the parameter is to be
used as part of the identification of a job instance.
Therefore you can not extend JobParameter and expect it to work with HashMap.
However there is another option, JobParameters:
https://docs.spring.io/spring-batch/docs/current/api/org/springframework/batch/core/JobParameters.html
https://docs.spring.io/spring-batch/docs/current/api/org/springframework/batch/core/JobParametersBuilder.html
You could create a Map<String, JobParameter> instead :
Example:
new JobParameters(Maps.newHashMap("yearMonth", new JobParameter("2021-07")))
and then use JobParametersBuilder addJobParameters in your createJobParam to simply add all your Map<String, JobParameter> records:
addJobParameters(JobParameters jobParameters) //Copy job parameters into the current state.
So your method will look like:
public JobParameters createJobParam (MyRequest request) {
final JobParameters parameters = new JobParametersBuilder()
.addString("MyParam1", request.getReportGenerationJobId())
.addString("MyParam2", request.getSessionId())
.addLong("time", System.currentTimeMillis())
.addParameters(mayHashMapThatHas<String,JobParameter>)
.toJobParameters();
return JobParameters;
}
Related
I have to set the line of a record in a csv file within the record object when processing it before passing it to the writer .
What i want to achieve is to declare a global variable lets say line = 1 and each time an itemProcessor runs it should assign its value to the current item , and increment its value by 1 line++.
How can i share a variable over multiple itemProcessor runs ?
You can leverage the ExecutionContext to save the line variable of either the StepExecution if you only need the ExecutionContext in the single Step or JobExecution if you need the ExecutionContext in other Steps of the Job.
Inject either execution in your ItemProcessor:
#Value("#{stepExecution.jobExecution}") JobExecution jobExecution
or
#Value("#{stepExecution}") StepExecution stepExecution
Taking JobExecution as an example:
#StepScope
#Bean
public MyItemProcessor myItemProcessor(#Value("#{stepExecution.jobExecution}") JobExecution jobExecution) {
return new MyItemProcessor(jobExecution);
}
#Bean
public Step myStep() {
return stepBuilderFactory.get("myStep")
...
.processor(myItemProcessor(null))
...
.build()
}
public class MyItemProcessor implements ItemProcessor<MyItem, MyItem> {
private ExecutionContext executionContext;
public MyItemProcessor() {
this.executionContext = jobExecution.getExecutionContext();
}
#Override
public MyItem process(MyItem item) throws Exception {
// get the line from previous item processors, if exists, otherwise start with 0
int line = executionContext.getInt("myLineKey", 0);
item.setLine(line++);
// save the line for other item processors
item.put("myLineKey", line);
return item;
}
}
We're creating a spring batch app that reads data from a database and writes in another database. In this process, we need to dynamically set the parameter to the SQL as we have parameters that demands data accordingly.
For this, We created a JdbcCursorItemReader Reader with #StepScope as I've found in other articles and tutorials. But was not successful. The chunk reader in our Job actually uses Peekable reader which internally uses the JdbcCursorItemReader object to perform the actual read operation.
When the job is triggered, we get the error - "jobParameters cannot be found on object of type BeanExpressionContext"
Please let me know what is that I am doing wrongly in the bean configuration below.
#Bean
#StepScope
#Scope(proxyMode = ScopedProxyMode.TARGET_CLASS)
public JdbcCursorItemReader<DTO> jdbcDataReader(#Value() String param) throws Exception {
JdbcCursorItemReader<DTO> databaseReader = new JdbcCursorItemReader<DTO>();
return databaseReader;
}
// This class extends PeekableReader, and sets JdbcReader (jdbcDataReader) as delegate
#Bean
public DataPeekReader getPeekReader() {
DataPeekReader peekReader = new DataPeekReader();
return peekReader;
}
// This is the reader that uses Peekable Item Reader (getPeekReader) and also specifies chunk completion policy.
#Bean
public DataReader getDataReader() {
DataReader dataReader = new DataReader();
return dataReader;
}
// This is the step builder.
#Bean
public Step readDataStep() throws Exception {
return stepBuilderFactory.get("readDataStep")
.<DTO, DTO>chunk(getDataReader())
.reader(getDataReader())
.writer(getWriter())
.build();
}
#Bean
public Job readReconDataJob() throws Exception {
return jobBuilderFactory.get("readDataJob")
.incrementer(new RunIdIncrementer())
.flow(readDataStep())
.end()
.build();
}
Please let me know what is that I am doing wrongly in the bean configuration below.
Your jdbcDataReader(#Value() String param) is incorrect. You need to specify a Spel expression in the #Value to specify which parameter to inject. Here is an example of how to pass a job parameter to a JdbcCursorItemReader:
#Bean
#StepScope
public JdbcCursorItemReader<DTO> jdbcCursorItemReader(#Value("#{jobParameters['table']}") String table) {
return new JdbcCursorItemReaderBuilder<DTO>()
.sql("select * from " + table)
// set other properties
.build();
}
You can find more details in the late binding section of the reference documentation.
I'm attempting to pass a value from application.properties into a custom ItemProcessor. However, using the #Value annotation always returns null, which isn't entirely unexpected. However, I'm at a loss for how to pass the necessary value in without #Value.
#Service
class FinancialRecordItemProcessor implements ItemProcessor<FinancialTransactionRecord, FinancialTransactionRecord> {
Logger log = LoggerFactory.getLogger(FinancialRecordItemProcessor)
// Start Configs
#Value('${base.url:<redacted URL>}')
String baseUrl
#Value('${access.token:null0token}')
String accessToken
// End Configs
#Override
FinancialTransactionRecord process(FinancialTransactionRecord financialRecord) throws IllegalAccessException{
// Test to ensure valid auth token
if (accessToken == null || accessToken == "null0token"){
throw new IllegalAccessException("You must provide an access token. " + accessToken + " is not a valid access token.")
}
}
Have you defined the context:property-placeholder ?!
For example if your configs are at classpath:/configs, you need :
<context:property-placeholder location="classpath:/configs/*.properties" />
You are on the right track. Have a look at #StepScope
From official docs:
Convenient annotation for step scoped beans that defaults the proxy
mode, so that it doesn't have to be specified explicitly on every bean
definition. Use this on any #Bean that needs to inject #Values from
the step context, and any bean that needs to share a lifecycle with a
step execution (e.g. an ItemStream). E.g.
Example:
#Bean
#StepScope
protected Callable<String> value(#Value("#{stepExecution.stepName}")
final String value) {
return new SimpleCallable(value);
}
So basically you can only inject values defined from your step contextbut also can inject values from your job context e.g.
#Value(value = "#{jobParameters['yourKey']}")
private String yourProperty;
The jobParameters can be set prior to job execution:
JobParameters jobParameters = new JobParametersBuilder()
.addLong("time", System.currentTimeMillis())
.addString("yourKey", "a value")
.toJobParameters();
final JobExecution jobExecution = jobLauncher.run(job, jobParameters);
Found a solution. Rather than using #Value on global variables as shown above, they need to be passed into a constructor, i.e.
FinancialRecordItemProcessor(String baseUrl, String accessToken){
super()
this.baseUrl = baseUrl
this.accessToken = accessToken
}
I try to configure Gson as my JSON mapper to accept "snake_case" query parameter, and translate them into standard Java "camelCase" parameters.
First of all, I know I could use the #SerializedName annotation to customise the serialized name of each field, but this will involve some manual work.
After doing some search, I believe the following approach should work (please correct me if I am wrong).
Use Gson as the default JSON mapper of Spring Boot
spring.http.converters.preferred-json-mapper=gson
Configuring Gson before GsonHttpMessageConverter is created as described here
Customising the Gson naming policy in step 2 according to GSON Field Naming Policy
private GsonHttpMessageConverter createGsonHttpMessageConverter() {
Gson gson = new GsonBuilder()
.setFieldNamingPolicy(FieldNamingPolicy.LOWER_CASE_WITH_UNDERSCORES)
.create();
GsonHttpMessageConverter gsonConverter = new GsonHttpMessageConverter();
gsonConverter.setGson(gson);
return gsonConverter;
}
Then I create a simple controller like this:
#RequestMapping(value = "/example/gson-naming-policy")
public Object testNamingPolicy(ExampleParam data) {
return data.getCamelCase();
}
With the following Param class:
import lombok.Data;
#Data
public class ExampleParam {
private String camelCase;
}
But when I call the controller with query parameter ?camel_case=hello, the data.camelCase could not been populated (and it's null). When I change the query parameters to ?camelCase=hello then it could be set, which mean my setting is not working as expected.
Any hint would be highly appreciated. Thanks in advance!
It's a nice question. If I understand how Spring MVC works behind the scenes, no HTTP converters are used for #ModelAttribute-driven. It can be inspected easily when throwing an exception from your ExampleParam constructor or the ExampleParam.setCamelCase method (de-Lombok first) -- Spring uses its bean utilities that use public (!) ExampleParam.setCamelCase to set the DTO value. Another proof is that no Gson.fromJson is never invoked regardless how your Gson converter is configured. So, your camelCase confuses you because the default Gson instance uses this strategy as well as Spring does -- so this is just a matter of confusion.
In order to make it work, you have to create a custom Gson-aware HandlerMethodArgumentResolver implementation. Let's assume we support POJO only (not lists, maps or primitives).
#Configuration
#EnableWebMvc
class WebMvcConfiguration
extends WebMvcConfigurerAdapter {
private static final Gson gson = new GsonBuilder()
.setFieldNamingPolicy(LOWER_CASE_WITH_UNDERSCORES)
.create();
#Override
public void addArgumentResolvers(final List<HandlerMethodArgumentResolver> argumentResolvers) {
argumentResolvers.add(new HandlerMethodArgumentResolver() {
#Override
public boolean supportsParameter(final MethodParameter parameter) {
// It must be never a primitive, array, string, boxed number, map or list -- and whatever you configure ;)
final Class<?> parameterType = parameter.getParameterType();
return !parameterType.isPrimitive()
&& !parameterType.isArray()
&& parameterType != String.class
&& !Number.class.isAssignableFrom(parameterType)
&& !Map.class.isAssignableFrom(parameterType)
&& !List.class.isAssignableFrom(parameterType);
}
#Override
public Object resolveArgument(final MethodParameter parameter, final ModelAndViewContainer mavContainer, final NativeWebRequest webRequest,
final WebDataBinderFactory binderFactory) {
// Now we're deconstructing the request parameters creating a JSON tree, because Gson can convert from JSON trees to POJOs transparently
// Also note parameter.getGenericParameterType() -- it's better that Class<?> that cannot hold generic types parameterization
return gson.fromJson(
parameterMapToJsonElement(webRequest.getParameterMap()),
parameter.getGenericParameterType()
);
}
});
}
...
private static JsonElement parameterMapToJsonElement(final Map<String, String[]> parameters) {
final JsonObject jsonObject = new JsonObject();
for ( final Entry<String, String[]> e : parameters.entrySet() ) {
final String key = e.getKey();
final String[] value = e.getValue();
final JsonElement jsonValue;
switch ( value.length ) {
case 0:
// As far as I understand, this must never happen, but I'm not sure
jsonValue = JsonNull.INSTANCE;
break;
case 1:
// If there's a single value only, let's convert it to a string literal
// Gson is good at "weak typing": strings can be parsed automatically to numbers and booleans
jsonValue = new JsonPrimitive(value[0]);
break;
default:
// If there are more than 1 element -- make it an array
final JsonArray jsonArray = new JsonArray();
for ( int i = 0; i < value.length; i++ ) {
jsonArray.add(value[i]);
}
jsonValue = jsonArray;
break;
}
jsonObject.add(key, jsonValue);
}
return jsonObject;
}
}
So, here are the results:
http://localhost:8080/?camelCase=hello => (empty)
http://localhost:8080/?camel_case=hello => "hello"
I created a driver which reads a config file, builds a list of objects (based on the config) and passes that list to MapReduce (MapReduce has a static attribute which holds a reference to that list of object).
It works but only locally. As soon as I run the job on a cluster config I will get all sort of errors suggesting that the list hasn't been built. It makes me think that I'm doing it wrong and on a cluster setup MapReduce is being run independently from the driver.
My question is how to correctly initialise a Mapper.
(I'm using Hadoop 2.4.1)
This is related to the problem of side data distribution.
There are two approaches for side data distribution.
1) Distributed Caches
2) Configuration
As you have the objects to be shared, we can use the Configuration class.
This discussion will depend on the Configuration class to make available an Object across the cluster, accessible to all Mappers and(or) Reducers. The approach here is quite simple. The setString(String, String) setter of the Configuration classed is harnessed to achieve this task. The Object that has to be shared across is serialized into a java string at the driver end and is de-serialized back to the object at the Mapper or Reducer.
In the example code below, I have used com.google.gson.Gson class for the easy serialization and deserialization. You can use Java Serialization as well.
Class that Represents the Object You need to Share
public class TestBean {
String string1;
String string2;
public TestBean(String test1, String test2) {
super();
this.string1 = test1;
this.string2 = test2;
}
public TestBean() {
this("", "");
}
public String getString1() {
return string1;
}
public void setString1(String test1) {
this.string1 = test1;
}
public String getString2() {
return string2;
}
public void setString2(String test2) {
this.string2 = test2;
}
}
The Main Class from where you can set the Configurations
public class GSONTestDriver {
public static void main(String[] args) throws Exception {
System.out.println("In Main");
Configuration conf = new Configuration();
TestBean testB1 = new TestBean("Hello1","Gson1");
TestBean testB2 = new TestBean("Hello2","Gson2");
Gson gson = new Gson();
String testSerialization1 = gson.toJson(testB1);
String testSerialization2 = gson.toJson(testB2);
conf.set("instance1", testSerialization1);
conf.set("instance2", testSerialization2);
Job job = new Job(conf, " GSON Test");
job.setJarByClass(GSONTestDriver.class);
job.setMapperClass(GSONTestMapper.class);
job.setNumReduceTasks(0);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(NullWritable.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
}
}
The mapper class from where you can retrieve the object
public class GSONTestMapper extends
Mapper<LongWritable, Text, Text, NullWritable> {
Configuration conf;
String inst1;
String inst2;
public void setup(Context context) {
conf = context.getConfiguration();
inst1 = conf.get("instance1");
inst2 = conf.get("instance2");
Gson gson = new Gson();
TestBean tb1 = gson.fromJson(inst1, TestBean.class);
System.out.println(tb1.getString1());
System.out.println(tb1.getString2());
TestBean tb2 = gson.fromJson(inst2, TestBean.class);
System.out.println(tb2.getString1());
System.out.println(tb2.getString2());
}
public void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
context.write(value,NullWritable.get());
}
}
The bean is converted to a serialized Json String using the toJson(Object src) method of the class com.google.gson.Gson. Then the serialised Json string is passed as value through the configuration instance and accessed by name from the Mapper. The string is deserialized there using the fromJson(String json, Class classOfT) method of the same Gson class. Instead of my test bean, you could place your objects.