Is there an elegant way to specify entity's fields to ignore by Spring Data Elasticsearch's ObjectMapper while left them being serialized for REST? - spring-boot

In other words, the common Jackson markup is not enough for serializing the same entity for using as the REST request response to the Angular frontend and to pass the object to Elasticsearch via the Jest client. Say, I have an image in the Entity as a byte array, and I'd like it to be stored to DB and be passed to the frontend, but don't like it being indexed by Elasticsearch to reduce the costs or quotas.
Now I have to use Jackson's JsonView to markup the fields to use for the Spring Data Elasticsearch's ObjectMapper:
#Entity
#Table(name = "good")
#org.springframework.data.elasticsearch.annotations.Document(indexName = "good", shards = 1, replicas = 0, refreshInterval = "-1")
public class Good implements Serializable {
private static final long serialVersionUID = 1L;
#Id
#GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "sequenceGenerator")
#SequenceGenerator(name = "sequenceGenerator")
#org.springframework.data.elasticsearch.annotations.Field(type = FieldType.Keyword)
#JsonView(Views.Elasticsearch.class)
private Long id;
#NotNull
#Column(name = "short_name", nullable = false)
#JsonView(Views.Elasticsearch.class)
#Field(store = true, index = true, type=FieldType.Text)
private String shortName;
#NotNull
#Column(name = "description", nullable = false)
#JsonView(Views.Elasticsearch.class)
#Field(store = true, index = true, type=FieldType.Text)
private String description;
#Lob
#Column(name = "image", nullable = false)
#JsonView(Views.Rest.class)
private byte[] image;
#Column(name = "image_content_type", nullable = false)
#JsonView(Views.Rest.class)
private String imageContentType;
#NotNull
#Column(name = "price", nullable = false)
#JsonView(Views.Elasticsearch.class)
#Field(store = true, index = true, type=FieldType.Integer)
private Integer price;
...
I have a clss for Views:
public class Views {
public static class Rest{}
public static class Elasticsearch{}
}
And the ObjectMapper set up in the corresponding Bean:
#Override
public String mapToString(Object object) throws IOException {
log.trace("Object to convert to JSON : {}",object);
log.trace("Converting to json for elasticsearch >>> {}",objectMapper.writer().withView(Views.Elasticsearch.class).writeValueAsString(object));
//log.trace("Converting to json for elasticsearch >>> {}",objectMapper.writeValueAsString(object));
return objectMapper.writerWithView(Views.Elasticsearch.class).writeValueAsString(object);
//return objectMapper.writeValueAsString(object);
}
So, I have to markup all the fields except the ignored to Elasticsearch with #JsonView(Views.Elasticsearch.class) and this is the error prone. Also, this field still requires #Field usage if I like to pass some parameters there like store or value type. When I have #JsonView(Views.Elasticsearch.class), but don't have #Field on some, the fields are created in the index on a fly, that allows them to search, but not in desired way.
The latest is the reason why if I just leave #Field there and don't place it over fields I don't want to index into Elasticsearch, the initial index indeed ignores them, but later requests pass the undesired field when the entity is serialized exactly the same way as it is done for the REST. And the index property is created on a fly, making resources being spent for the large binary object indexing. So it looks like #Field is used for the initial index creation on the startup, but are not configured to be used with ObjectMapper of the Spring Data Elasticsearch.
So, I'd like to make this ObjectMapper take only fields with #Field above them into account, i.e serialize the fields marked with #Field only and use no #JsonView staff. How can I configure it?

These are known problems when using the Jackson Object Mapper in Spring Data Elasticsearch (which als is the default) and this is one of the reasons, why in Spring Data Elasticsearch as of version 3.2 (which currently is in RC2 and will be available as 3.2.0.GA in mid-september), there is a different mapper available, the ElasticsearchEntityMapper.
This mapper still has to be setup explicitly, the reference documentation of 3.2.0.RC2 shows how to do this. Using this mapper the Jackson annotations do not influence the data stored in and read from Elasticsearch. And you can use the org.springframework.data.annotation.Transient annotation on a field to not have it stored in Elasticsearch.
The #Field annotation is used to setup the initial Elasticsearch mapping, properties not having this annotation are automatically mapped by Elasticsearch when they are inserted.

Related

Elastic search with Spring Data for Reactive Repository - Deleting based on nested attributes

I am using Spring data for Elastic Search and am using the ReactiveCrudRepository for stuff like finding and deleting. I noticed that with attributes that are in root and are simple objects, the deletion works (deleteByAttributeName). However if I have nested objects then it does not work.
Here's my entities
Book
#Data
#TypeAlias("book")
#Document(indexName = "book")
public class EsBook{
#Field(type = FieldType.Long)
private Long id;
#Field(type = FieldType.Nested)
private EsStats stats;
#Field(type = FieldType.Date, format = DateFormat.date)
private LocalDate publishDate;
}
Stats
#Data
#Builder
#NoArgsConstructor
#AllArgsConstructor
#EqualsAndHashCode
public class EsStats{
#Field(type = FieldType.Double)
private Double averageRating;
#Field(type = FieldType.Integer)
private Double totalRatings;
#Field(type = FieldType.Keyword)
private String category; //this can be null
}
Here is what I have tried and is working and not working
I used ReactiveCrudRepository to delete documents in index. For all the regular fields on Book Level like id or with id and publishDate deletion works perfectly. As soon as I use embedded object like Stats, it stops working. I see the documents and the stats that I am sending match atleast visually but never finds or deletes them.
I tried to use EqualsAndHashcode in the Stats assuming maybe iternally somehow does not consider equal for some reason. I also tried changing double data type to int, because on looking at the elastic search document, I see that average review if whole number like 3 is save as 3 but when we send it from Java, i see in the debug 3 being shown as 3.0, so I was doubting if that is the case, but does not seem so. Even changing the datatype to int deletion does not work.
public interface ReactiveBookRepository extends ReactiveCrudRepository<EsBook, String> {
Mono<Void> deleteById(long id); //working
Mono<Void> deleteByIdAndPublishDate(long id, LocalDate publishDate); //Nor working
Mono<Void> deleteByIdAndStats(long id, LocalDate startDate);
}
Any help will be appreciated
Have you verified that your Elasticsearch index mapping matches your Spring Data annotations?
Verify that the index mapping defines the stats field as a nested field type.
If not, then try changing your Spring annotation to:
#Field(type = FieldType.Object)
private EsStats stats;

spring boot resource as links

I am new to spring boot, I have used it to implement a rest API. I have a self referencing table where each item has a parent -tree structure- . I have used ManyToOne to implement this and i get a json object which holds the parent. If i get all the items the speed is very slow since there are a huge network latency and processing because my tree can hold up to 10000 item.
How can I represent this using links, i.e the the json object contains a link to parent and array of links to children. I have read that DTO can be used to implement this but I did not find a full details.
part of my code
#Entity
#Table(name = "Item", schema = "dbo")
#Getter
#Setter
public class Item {
#Id
#GeneratedValue
#Column()
private Integer ItemID;
#Column()
private String Project;
#Column()
private String Name;
#Column()
private Integer Version;
#ManyToOne(fetch = FetchType.LAZY)
#JoinColumn(name = "ParentItemID", insertable = false, updatable = false)
private Item parentItem;
My controller is strait forward
#Autowired
ItemService ItemService;
#RequestMapping ("/items")
public Iterable<Item> items(#RequestParam(value = "page", defaultValue = "0") Integer page,
#RequestParam(value = "size", defaultValue = "20") Integer size, Authentication auth) {
return ItemService.findPaginated(page, size);
}
I would be glad if i get more explanation on how to use DTOs , or another design to get links instead of full objects.
Easiest thing you can do is, return parent and children ids instead of Item objects.
#Entity
#Table(name = "Item", schema = "dbo")
public class Item {
.
.
private Integer ItemID;
private Integer parentItem;
.
.
Have another endpoint in your controller like /items/{itemId}
#GetMapping("/items/{itemId}")
public Item item(#PathVariable itemId, Authentication auth) {
return ItemService.findById(itemId);
Let your consumer grab the parent/children Ids and make separate REST calls
Alternatively, you can use Spring-HATEOAS
The Spring HATEOAS project is a library of APIs that we can use to
easily create REST representations that follow the principle of
HATEOAS (Hypertext as the Engine of Application State).
Generally speaking, the principle implies that the API should guide
the client through the through the application by returning relevant
information about the next potential steps, along with each response.
More Information Here

Spring mongodb annotation for 2dsphere index for a geospatial field in java...?

#JsonSerialize
#Document(collection = "fence")
#CompoundIndexes({
#CompoundIndex(name = "loc_groupId_idx",
def = "{ 'loc': 2dsphere, 'groups.groupId': 1 }",
unique = false) })
public class GeofenceMongoVO {
public GeofenceMongoVO() {}
#Id
private String fenceId;
#Field
private Long customerId;
#Field
private String fenceName;
#Field
private Byte type;
This is how I tried to ensure a compound index on a geospatial field and a field of a child document(groupId). But this is not working unfortunately. Is there a way by which I can ensure 2dsphere index from java code via annotations?
As of Spring Data MongoDB 1.10.10.RELEASE, you can annotate any field, whether it be at the document root or in a subdocument with:
#GeoSpatialIndexed(type = GeoSpatialIndexType.GEO_2DSPHERE)
private GeoJsonPoint myGeometry;
I'm not sure if it can be done with annotations yet, but I found a blog post here where they do it with an ensureIndex. Something like th
#Autowired
MongoTemplate template;
public void setupIndex()
{
template.indexOps(Location.class).ensureIndex( new GeospatialIndex("position") );
}

How can I include or exclude a record according to a boolean parameter using Spring Data JPA?

I am not so into Spring Data JPA and I have the following doubt about how to implement a simple query.
I have this AccomodationMedia entity class mapping the accomodation_media on my database:
#Entity
#Table(name = "accomodation_media")
public class AccomodationMedia {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
#Column(name = "id")
private Long id;
#Column(name = "id_accomodation")
private Long idAccomodation;
#Column(name = "is_master")
private boolean isMaster;
#Lob
#Column(name = "media")
private byte[] media;
private String description;
private Date time_stamp;
public AccomodationMedia() {
}
...............................................................
...............................................................
...............................................................
// GETTER AND SETTER METHODS
...............................................................
...............................................................
...............................................................
}
The instance of this class represents the photo associated to an accomodation (an hotel)
So as you can see in the prvious code snippet I have this field :
#Column(name = "id_accomodation")
private Long idAccomodation;
that contains the id of an accomodation (the id of an hotel on my database).
I also have this boolean field that specify if an image is the master image or not:
#Column(name = "is_master")
private boolean isMaster;
So, at this time, in my repository class I have this method that should return all the images associated to a specific hotel:
#Repository
public interface AccomodationMediaDAO extends JpaRepository<AccomodationMedia, Long> {
List<AccomodationMedia> findByIdAccomodation(Long accomodationId);
}
I want to modify this method passing also the boolean parameter that specify if have to be returned also the master image or only the images that are not master.
So I tryied doing in this way:
List<AccomodationMedia> findByIdAccomodationAndIsMaster(Long accomodationId, boolean isMaster);
but this is not correct because setting to true the isMaster parameter it will return only the master image (because it is first selecting all the Accomodation having a specific accomodation ID and then the one that have the isMaster field setted as true).
So, how can I correctly create this query that use the isMaster boolean parameter to include or exclude the AccomodationMedia instance that represent my master image?
I know that I can use also native SQL or HQL to do it but I prefer do it using the "query creation from method names"
I don't have how to test this, but essentially your final query should be:
id_accomodation = ?1 AND (is_master = ?2 OR is_master = false)
So I would try the following method signature:
findByIdAccomodationAndIsMasterOrIsMasterFalse(Long accomodationId, boolean isMaster);
I would go with two methods one for isMaster true, while second for false value like this:
List<AccomodationMedia> findByIdAccomodationAndIsMasterFalse(Long accomodationId);
List<AccomodationMedia> findByIdAccomodationAndIsMasterTrue(Long accomodationId);
Change your acommodation id as accomodationId instead of idAccomodation. When you write findByIdAccomodationAndIsMaster spring confusing findByIdAccomodationAndIsMaster
Try this this convention
#Column(name = "accomodation_id")
private Long accomodationId;

Spring data elasticSearch : Update entity using alias

I'm currently fighting with the spring-data-elasticsearch API. I need it to work on an alias with several indexes pointing on it.
Each indexes have the same types stored, but are juste day to day storage (1rst index are monday's resulsts, second are tuesday's resulsts....).
Some of the ElasticsearchRepository methods don't work because of the alias. I currently managed to do a search (findOne() equivalent) but I am not able to update an entity.
I don't know how to achieve that, I looked to the documentation and samples.. but I'm stuck.
My repository
public interface EsUserRepository extends ElasticsearchRepository<User, String>
{
#Query("{\"bool\" : {\"must\" : {\"term\" : {\"id_str\" : \"?0\"}}}}")
User findByIdStr(String idStr);
}
My Entity
#Document(indexName = "osintlab", type = "users")
public class User
{
// Elasticsearch internal id
#Id
private String id;
// Just a test to get the real object index (_index field), in order to save it
#Field(index = FieldIndex.analyzed, type = FieldType.String)
private String indexName;
// Real id, saved under the "id_str" field
#Field(type = FieldType.String)
private String id_str;
#Field(type = FieldType.String)
private List<String> tag_user;
}
What I tested
final IndexQuery indexQuery = new IndexQuery();
indexQuery.setId(user.getId());
indexQuery.setObject(user);
esTemplate.index(indexQuery);
userRepository.index(user));
userRepository.save(user))

Resources