I have two collections with the following payloads:
Collection-1 payload:-
{
name:"bucky",
email:"bucky#gmail.com"
}
collection-2 payload:-
{
score:"97",
subject:"Science"
}
I want to pass payload by merging collection-1 and 2,
Expected payload:-
{
"collection-1":{
name:"bucky",
email:"bucky#gmail.com"
},
"collection-2":{
score:"97",
subject:"Science"
}
}
How I can merge these two collections and generate the above-Expected payload?
Both collections have their respected DAO classes.
I was Doing Something like this :-
public Map<String,Object> getAllData{
Map<String, Object> result=new HashMap<>();
Map<String, Collection1> obj1=new HashMap<>();
Map<String, Collection2> obj2=new HashMap<>();
obj1=getData1();
obj2=getData2();
result.put("object-1",obj1);
result.put("object-2",obj1);
return result;
}
Related
I want to add Caching to my Spring Boot Backend. Saving the entries to the Cache seems to work since I can see the json list in Redis after my first request but once I send my second request (which would read the Cache) to the backend Spring throws an internal error and the request fails:
WARN 25224 --- [nio-8080-exec-2] .w.s.m.s.DefaultHandlerExceptionResolver :
Resolved [org.springframework.http.converter.HttpMessageNotWritableException:
Could not write JSON: java.lang.ClassCastException#291f1fc4;
nested exception is com.fasterxml.jackson.databind.JsonMappingException:
java.lang.ClassCastException#291f1fc4
(through reference chain: java.util.ArrayList[0]->java.util.LinkedHashMap["id"])]
My backend looks as it follows:
Config:
#Configuration
class RedisConfig {
#Bean
fun jedisConnectionFactory(): JedisConnectionFactory {
val jedisConnectionFactory = JedisConnectionFactory()
return jedisConnectionFactory
}
#Bean
fun redisTemplate(): RedisTemplate<String, Any> {
val myRedisTemplate = RedisTemplate<String, Any>()
myRedisTemplate.setConnectionFactory(jedisConnectionFactory());
return myRedisTemplate;
}
#Bean
fun cacheManager(): RedisCacheManager {
return RedisCacheManager.RedisCacheManagerBuilder.fromConnectionFactory(jedisConnectionFactory()).cacheDefaults(
RedisCacheConfiguration.defaultCacheConfig().disableCachingNullValues()
.serializeKeysWith(RedisSerializationContext.SerializationPair.fromSerializer(RedisSerializer.string()))
.serializeValuesWith(RedisSerializationContext.SerializationPair.fromSerializer(GenericJackson2JsonRedisSerializer(redisMapper())))
).build()
}
private fun redisMapper(): ObjectMapper {
return ObjectMapper() //.enableDefaultTyping(DefaultTyping.NON_FINAL, As.PROPERTY)
.setSerializationInclusion(JsonInclude.Include.NON_EMPTY)
}
}
Controller:
fun getPrivateRecipes(#RequestParam(required = false) langCode: String?): List<PrivateRecipeData> {
val lang = langCode ?: "en"
val userId = getCurrentUser().userRecord.uid
return privateRecipeCacheService.getPrivateRecipesCached(lang, userId)
}
Caching-Service
#Cacheable("privateRecipes")
fun getPrivateRecipesCached(lang: String, userId: String): List<PrivateRecipeData> {
return privateRecipeService.getPrivateRecipes(lang, userId)
}
I played around with the Cachable annotation, added keys, but it does not change the problem. The import and export of the list seems to be done with different classes. How to solve this?
In your ObjectMapper tell Jackson to use an ArrayList to hold collections of PrivateRecipeData instances like:
objectMapper.getTypeFactory().constructCollectionType(ArrayList.class, PrivateRecipeData.class);
One possible way to set it up:
One possible way is, in your cacheManager config, pass it a RedisTemplate configure with the right object mapper. Off the top of my head:
public RedisTemplate<String, Object> redisTemplate(...) {
Jackson2JsonRedisSerializer serializer = new ...
ObjectMapper objectMapper = new ObjectMapper();
//...
serializer.setObjectMapper(objectMapper);
RedisTemplate<String, Object> redisTemplate = new RedisTemplate<>();
// ...
redisTemplate.setValueSerializer(serializer);
redisTemplate.afterPropertiesSet();
return redisTemplate;
}
What I did in the end was just using the default ObjectMapper (providing no arguments to GenericJackson2JsonRedisSerializer):
#Bean
fun cacheManager(): RedisCacheManager {
return RedisCacheManager.RedisCacheManagerBuilder.fromConnectionFactory(jedisConnectionFactory()).cacheDefaults(
RedisCacheConfiguration.defaultCacheConfig().disableCachingNullValues()
.serializeKeysWith(RedisSerializationContext.SerializationPair.fromSerializer(RedisSerializer.string()))
.serializeValuesWith(
RedisSerializationContext.SerializationPair.fromSerializer(
GenericJackson2JsonRedisSerializer()
)
)
).build()
}
The serialization in Json now looks a bit different (containing class names as well) but thats totally fine since it is still human readable :)
[
"java.util.ArrayList",
[
{
"#class": "com.my.package.data.PrivateRecipeData",
"id": 1,
"img_src": "user/xxxxxx/recipeImgs/32758c1c-35cf-4f92-9e8c-0057f4447d6c.jpg",
"name": "VeggieBurger",
"instructions": [
"java.util.ArrayList",
[
{
"id": 6,
"recipeId": 1,
}
]
],
...
}
],
...
I have configured a job as follow, which is to read from db and write into files but by partitioning data on basis of sequence.
//Job Config
#Bean
public Job job(JobBuilderFactory jobBuilderFactory) throws Exception {
Flow masterFlow1 = (Flow) new FlowBuilder<Object>("masterFlow1").start(masterStep()).build();
return (jobBuilderFactory.get("Partition-Job")
.incrementer(new RunIdIncrementer())
.start(masterFlow1)
.build()).build();
}
#Bean
public Step masterStep() throws Exception
{
return stepBuilderFactory.get(MASTERPPREPAREDATA)
//.listener(customSEL)
.partitioner(STEPPREPAREDATA,new DBPartitioner())
.step(prepareDataForS1())
.gridSize(gridSize)
.taskExecutor(new SimpleAsyncTaskExecutor("Thread"))
.build();
}
#Bean
public Step prepareDataForS1() throws Exception
{
return stepBuilderFactory.get(STEPPREPAREDATA)
//.listener(customSEL)
.<InputData,InputData>chunk(chunkSize)
.reader(JDBCItemReader(0,0))
.writer(writer(null))
.build();
}
#Bean(destroyMethod="")
#StepScope
public JdbcCursorItemReader<InputData> JDBCItemReader(#Value("#{stepExecutionContext[startingIndex]}") int startingIndex,
#Value("#{stepExecutionContext[endingIndex]}") int endingIndex)
{
JdbcCursorItemReader<InputData> ir = new JdbcCursorItemReader<>();
ir.setDataSource(batchDataSource);
ir.setMaxItemCount(DBPartitioner.partitionSize);
ir.setSaveState(false);
ir.setRowMapper(new InputDataRowMapper());
ir.setSql("SELECT * FROM FIF_INPUT fi WHERE fi.SEQ > ? AND fi.SEQ < ?");
ir.setPreparedStatementSetter(new PreparedStatementSetter() {
#Override
public void setValues(PreparedStatement ps) throws SQLException {
ps.setInt(1, startingIndex);
ps.setInt(2, endingIndex);
}
});
return ir;
}
#Bean
#StepScope
public FlatFileItemWriter<InputData> writer(#Value("#{stepExecutionContext[index]}") String index)
{
System.out.println("writer initialized!!!!!!!!!!!!!"+index);
//Create writer instance
FlatFileItemWriter<InputData> writer = new FlatFileItemWriter<>();
//Set output file location
writer.setResource(new FileSystemResource(batchDirectory+relativeInputDirectory+index+inputFileForS1));
//All job repetitions should "append" to same output file
writer.setAppendAllowed(false);
//Name field values sequence based on object properties
writer.setLineAggregator(customLineAggregator);
return writer;
}
Partitioner provided for partitioning db is written separately in other file so as follows
//PartitionDb.java
public class DBPartitioner implements Partitioner{
public static int partitionSize;
private static Log log = LogFactory.getLog(DBPartitioner.class);
#SuppressWarnings("unchecked")
#Override
public Map<String, ExecutionContext> partition(int gridSize) {
log.debug("START: Partition"+"grid size:"+gridSize);
#SuppressWarnings("rawtypes")
Map partitionMap = new HashMap<>();
int startingIndex = -1;
int endSize = partitionSize+1;
for(int i=0; i< gridSize; i++){
ExecutionContext ctxMap = new ExecutionContext();
ctxMap.putInt("startingIndex",startingIndex);
ctxMap.putInt("endingIndex", endSize);
ctxMap.put("index", i);
startingIndex = endSize-1;
endSize += partitionSize;
partitionMap.put("Thread:-"+i, ctxMap);
}
log.debug("END: Created Partitions of size: "+ partitionMap.size());
return partitionMap;
}
}
This one is executing properly but problem is even after partitioning on the basis of sequence i am getting same rows in multiple files which is not right as i am providing different set of data for each partition. Can anyone tell me whats wrong. I am using HikariCP for Db connection pooling and spring batch 4
This one is executing properly but problem is even after partitioning on the basis of sequence i am getting same rows in multiple files which is not right as i am providing different set of data for each partition.
I'm not sure your partitioner is working properly. A quick test shows that it is not providing different sets of data as you are claiming:
DBPartitioner dbPartitioner = new DBPartitioner();
Map<String, ExecutionContext> partition = dbPartitioner.partition(5);
for (String s : partition.keySet()) {
System.out.println(s + " : " + partition.get(s));
}
This prints:
Thread:-0 : {endingIndex=1, index=0, startingIndex=-1}
Thread:-1 : {endingIndex=1, index=1, startingIndex=0}
Thread:-2 : {endingIndex=1, index=2, startingIndex=0}
Thread:-3 : {endingIndex=1, index=3, startingIndex=0}
Thread:-4 : {endingIndex=1, index=4, startingIndex=0}
As you can see, almost all partitions will have the same startingIndex and endingIndex.
I recommend you unit test your partitioner before using it in a partitioned step.
I want to rebinding ConfigurationProperties data.Read user documentation.
post http://localhost:8080/env,It working .
But post http://localhost:8080/env/reset,Cannot refresh all configurations.
Can only refresh keys that have visited /env.
I want to refresh all the configuration what should I do?
http://projects.spring.io/spring-cloud/spring-cloud.html#_endpoints
Posting key-value pairs to /env will trigger rebinding.
Posting to /env/reset will trigger too on condition that manager propertysource not empty.
If you are not updating environment by posting /env, you can use endpoint /refresh.
#ManagedOperation
public Map<String, Object> reset() {
Map<String, Object> result = new LinkedHashMap<String, Object>(map);
if (!map.isEmpty()) {
map.clear();
publish(new EnvironmentChangeEvent(publisher, result.keySet()));
}
return result;
}
#ManagedOperation
public void setProperty(String name, String value) {
if (!environment.getPropertySources().contains(MANAGER_PROPERTY_SOURCE)) {
synchronized (map) {
if (!environment.getPropertySources().contains(MANAGER_PROPERTY_SOURCE)) {
MapPropertySource source = new MapPropertySource(
MANAGER_PROPERTY_SOURCE, map);
environment.getPropertySources().addFirst(source);
}
}
}
if (!value.equals(environment.getProperty(name))) {
map.put(name, value);
publish(new EnvironmentChangeEvent(publisher, Collections.singleton(name)));
}
}
In a spring-boot 2.0 rest controller, I have created the following code which works as desired:
#ResponseBody
#GetMapping("/test3")
Mono<List<String>> test3(){
List<String> l1 = Arrays.asList("one","two","three");
List<String> l2 = Arrays.asList("four","five","six");
return Flux
.concat(Flux.fromIterable(l1),Flux.fromIterable(l2))
.collectList();
}
My problem comes from trying to do the same thing from an external datasource. I have created the following test case:
#ResponseBody
#GetMapping("/test4")
Flux<Object> test4(){
List<String> indecies = Arrays.asList("1","2");
return Flux.concat(
Flux.fromIterable(indecies)
.flatMap(k -> Flux.just(myRepository.getList(k))
.subscribeOn(Schedulers.parallel()),2
)
).collectList();
}
Where myRepository is the following:
#Repository
public class MyRepository {
List<String> l1 = Arrays.asList("one","two","three");
List<String> l2 = Arrays.asList("four","five","six");
Map<String, List<String>> pm = new HashMap<String, List<String>>();
MyRepository(){
pm.put("1", l1);
pm.put("2", l2);
}
List<String> getList(String key){
List<String> list = pm.get(key);
return list;
}
}
My code labeled test4 gives me the code hint error:
Type mismatch: cannot convert from Flux< List < String >> to Publisher < ?
extends Publisher < ? extends Object >>
So a few questions:
I thought that a Flux was a publisher? So why the error?
What am I doing wrong in test 4 so that it will output the same result as in test3?
The expected output is: [["one","two","three","four","five","six"]]
Using M. Deinum's comment, here is what works:
#ResponseBody
#GetMapping("/test6")
Mono<List<String>> test6(){
List<String> indecies = Arrays.asList("1","2");
return Flux.fromIterable(indecies)
.flatMap(k -> Flux.fromIterable(myRepository.getList(k)).subscribeOn(Schedulers.parallel()),2)
.collectList();
}
I found https://github.com/FasterXML/jackson-dataformat-yaml to deserialize/serialize YAML files. However, I'm having a hard time to deserialize/serialize the following:
I want to define a prefix to the actual document to be parsed as a POJO. Similar to a subtree of the document.
I want to define the POJO that represents the simple object representation instead of creating multiple objects.
The Error "Unrecognized field "spring" (class ConfigServerProperties), not marked as ignorable (one known property: "repos"])" is shown. But I don't know how to represent the prefix "spring.cloud.config.server.git" to be the root element of the POJO.
Document
spring:
cloud:
config:
server:
git:
repos:
publisher:
uri: 'https://github.company.com/toos/spring-cloud-config-publisher-config'
cloneOnStart: true
username: myuser
password: password
pullOnRequest: false
differentProperty: My Value
config_test_server_config:
uri: 'https://github.company.com/mdesales/config-test-server-config'
cloneOnStart: true
username: 226b4bb85aa131cd6393acee9c484ec426111d16
password: ""
completelyDifferentProp: this is a different one
For this document, the requirements are as follows:
* I want to define the prefix as "spring.cloud.config.server.git".
* I want to create a POJO that represents the object.
POJO
I created the following POJOs to represent this.
ConfigServerProperties: represents the top pojo containing the list of repos.
ConfigServerOnboard: represents each of the elements of the document.
Each properties are stored in a map, so that we can add as many different properties as possible.
Each class is as follows:
public class ConfigServerProperties {
private Map<String, ConfigServerOnboard> repos;
public void setRepos(Map<String, ConfigServerOnboard> repos) {
this.repos = repos;
}
public Map<String, ConfigServerOnboard> getRepos() {
return this.repos;
}
}
The second class is as follows:
public class ConfigServerOnboard {
private Map<String, String> properties;
public Map<String, String> getProperties() {
return properties;
}
public void setProperties(Map<String, String> properties) {
this.properties = properties;
}
}
Deserialize
The deserialization strategy I tried is as follows:
public static ConfigServerProperties parseProperties(File filePath)
throws JsonParseException, JsonMappingException, IOException {
ObjectMapper mapper = new ObjectMapper(new YAMLFactory());
JsonNodeFactory jsonNodeFactory = new JsonNodeFactory(false);
jsonNodeFactory.textNode("spring.cloud.config");
// tried to use this attempting to get the prefix
mapper.setNodeFactory(jsonNodeFactory);
ConfigServerProperties user = mapper.readValue(filePath, ConfigServerProperties.class);
return user;
}
Error Returned
Exception in thread "main" com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "spring" (class com.company.platform.config.onboarding.files.config.model.ConfigServerProperties), not marked as ignorable (one known property: "repos"])
at [Source: /tmp/config-server-onboards.yml; line: 3, column: 3] (through reference chain: com.company.platform.config.onboarding.files.config.model.ConfigServerProperties["spring"])
at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:62)
at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:834)
at com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:1094)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1470)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1448)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:282)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:140)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3798)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2740)
at com.company.platform.config.onboarding.files.config.model.ConfigServerProperties.parseProperties(ConfigServerProperties.java:37)
at com.company.platform.config.onboarding.files.config.model.ConfigServerProperties.main(ConfigServerProperties.java:42)
Edit 1: Looking for a possible SpringBoot Solution
I'm open to solutions using SpringBoot's ConfigurationProperties("spring.cloud.config.server.git"). That way, we could have the following:
#ConfigurationProperties("spring.cloud.config.server.git")
public class Configuration {
private Map<String, Map<String, String>> repos = new LinkedHashMap<String, new HashMap<String, String>>();
// getter/setter
}
Questions
How to set the root element of the document?
Deserialization must read the document and produce instances of the POJOs.
Serialization must produce the same document with updated values.
I had to come up with the following:
Create 6 classes, each of them with the property required for the prefix "spring.cloud.config.server.git"
SpringCloudConfigSpring.java
SpringCloudConfigCloud.java
SpringCloudConfigConfig.java
SpringCloudConfigServer.java
SpringCloudConfigGit.java
The holder of all of them is SpringCloudConfigFile.java.
The holder and all the classes have a reference to the next property, which has a reference to the next, etc, with their own setter/getter methods as usual.
public class SpringCloudConfigSpring {
private SpringCloudConfigCloud cloud;
public SpringCloudConfigCloud getCloud() {
return cloud;
}
public void setCloud(SpringCloudConfigCloud cloud) {
this.cloud = cloud;
}
}
Implemented the representation of the map easily.
The last one I used the reference of a TreeMap to keep the keys sorted, another map to represent any property that may be added, without changing the representation.
public class SpringCloudConfigGit {
TreeMap<String, Map<String, Object>> repos;
public TreeMap<String, Map<String, Object>> getRepos() {
return repos;
}
public void setRepos(TreeMap<String, Map<String, Object>> repos) {
this.repos = repos;
}
}
Results
Creating the verification as follows:
public static void main(String[] args) throws JsonParseException, JsonMappingException, IOException {
File config = new File("/tmp/config-server-onboards.yml");
SpringCloudConfigFile props = ConfigServerProperties.parseProperties(config);
props.getSpring().getCloud().getConfig().getServer().getGit().getRepos().forEach((appName, properties) -> {
System.out.println("################## " + appName + " #######################3");
System.out.println(properties);
if (appName.equals("github_pages_reference")) {
properties.put("name", "Marcello");
properties.put("cloneOnStart", true);
}
System.out.println("");
});
saveProperties(new File(config.getAbsoluteFile().getParentFile(), "updated-config-onboards.yml"), props);
}
The output is as follows:
################## config_onboarding #######################3
{uri=https://github.company.com/servicesplatform-tools/spring-cloud-config-onboarding-config, cloneOnStart=true, username=226b4bb85aa131cd6393acee9c484ec426111d16, password=, pullOnRequest=false}
################## config_test_server_config #######################3
{uri=https://github.company.com/rlynch2/config-test-server-config, cloneOnStart=true, username=226b4bb85aa131cd6393acee9c484ec426111d16, password=, pullOnRequest=false}
################## github_pages_reference #######################3
{uri=https://github.company.com/servicesplatform-tools/spring-cloud-config-reference-service-config, cloneOnStart=true, username=226b4bb85aa131cd6393acee9c484ec426111d16, password=, pullOnRequest=false}
There are obvious improvements required:
I'd like to have a solution with a single class;
I'd like to have a an ObjetMapper method that specifies the "subtree" of the YAML object tree that I'd like to parse.
Maybe a more sophisticated SpringBoot-like #ConfigurationProperties("spring.cloud.config.server.git") would help.
Helper methods for loading and saving the state of these instances.
Load Method
public static SpringCloudConfigFile parseProperties(File filePath)
throws JsonParseException, JsonMappingException, IOException {
ObjectMapper mapper = new ObjectMapper(new YAMLFactory());
SpringCloudConfigFile file = mapper.readValue(filePath, SpringCloudConfigFile.class);
return file;
}
Save Properties
public static void saveProperties(File filePath, SpringCloudConfigFile file) throws JsonGenerationException, JsonMappingException, IOException {
ObjectMapper mapper = new ObjectMapper(new YAMLFactory());
mapper.writeValue(filePath, file);
}
File Saved
It maintained the sorted keys as implemented.