Spring Data Elastic Search with dynamic document replicas and shards - elasticsearch

I'm using spring boot 2.4 and spring-data-elasticsearch 4.1. I have document like this
#Document(indexName = "test", replicas = 3, shards = 2)
public class TestDocument {
#Id
#Field(type = FieldType.Keyword)
private String Id;
#Field(type = FieldType.Object, enabled = false)
private String name;
...
getters
setters
}
And i want to override hardcoded values in replicas and shards in Document annotation from application.yml for index creation because this values can be differnet by environment of my service. Is there any way to do this?

You can disable auto-creation of an index by using
#Document(indexName = "test", createIndex = false)
and then create the index by using the IndexOperations.create(settings). The following code is taken from Spring Data Elasticsearch tests (https://github.com/spring-projects/spring-data-elasticsearch/blob/4.1.x/src/test/java/org/springframework/data/elasticsearch/core/ElasticsearchTemplateTests.java#L2361-L2384):
public void shouldCreateIndexWithGivenClassAndSettings() {
// given
String settings = "{\n" + " \"index\": {\n" + " \"number_of_shards\": \"1\",\n"
+ " \"number_of_replicas\": \"0\",\n" + " \"analysis\": {\n"
+ " \"analyzer\": {\n" + " \"emailAnalyzer\": {\n"
+ " \"type\": \"custom\",\n"
+ " \"tokenizer\": \"uax_url_email\"\n" + " }\n"
+ " }\n" + " }\n" + " }\n" + '}';
// when
indexOperations.delete();
indexOperations.create(parse(settings));
indexOperations.putMapping(SampleEntity.class);
indexOperations.refresh();
// then
Map<String, Object> map = indexOperations.getSettings();
assertThat(operations.indexOps(IndexCoordinates.of(INDEX_NAME_SAMPLE_ENTITY)).exists()).isTrue();
assertThat(map.containsKey("index.number_of_replicas")).isTrue();
assertThat(map.containsKey("index.number_of_shards")).isTrue();
assertThat((String) map.get("index.number_of_replicas")).isEqualTo("0");
assertThat((String) map.get("index.number_of_shards")).isEqualTo("1");
}
Instead of parsing the JSON for the settings you should create a Document in your code:
Document settings = Document.create();
settings.put("index.number_of_replicas", 42);
settings.put("index.number_of_shards", 42);

you can Add a #Setting annotion to you Entity, which is provided by spring-data-elasticsearch, in order to help you custom index info .
https://docs.spring.io/spring-data/elasticsearch/docs/current/reference/html/#elasticsearc.misc.index.settings

Related

How to map dynamic query to projection interface. not an entity class

#Entity
#Table(name = "delivery_status_summary", schema = "dsd")
#Data
public class DeliveryStatusSummaryV2 {
#Id
#GeneratedValue(strategy=GenerationType.SEQUENCE, generator="delivery_status_summary_id_seq")
#SequenceGenerator(name="delivery_status_summary_id_seq", sequenceName="delivery_status_summary_id_seq", allocationSize=1)
private Integer id;
#Column(name = "expected_delivery_date")
private LocalDateTime expectedDeliveryDate;
private Integer total;
#Column(name = "on_time")
private Integer onTime;
private Integer late;
private Integer pending;
#Column(name = "not_received")
private Integer notReceived;
}
Repository:
public interface DeliveryStatusSummaryRepository extends JpaRepository<DeliveryStatusSummary, Integer>, DeliveryStatusSummaryCustomRepository {
//language=SQL
String MANAGER_DELIVERY = "WITH dates AS (" +
" SELECT" +
" GENERATE_SERIES(" +
" CAST(:range_start AS TIMESTAMP)," +
" CAST(:range_end AS TIMESTAMP)," +
" INTERVAL '1 day'" +
" ) AS day" +
"), deliveries AS (" +
" SELECT *" +
" FROM dsd.delivery_status_summary AS dss" +
" WHERE dss.expected_delivery_date >= CAST(:range_start AS TIMESTAMP) AND" +
" dss.expected_delivery_date <= CAST(:range_end AS TIMESTAMP)" +
"), managers AS (" +
" SELECT *" +
" FROM dsd.teams as t" +
" WHERE t.manager_id IN (:manager_ids)" +
"), summary AS (" +
" SELECT t.manager_id," +
" dss.expected_delivery_date," +
" CAST(dss.expected_delivery_date AS DATE) AS day," +
" SUM(dss.total) AS total," +
" SUM(dss.on_time) AS on_time," +
" SUM(dss.late) AS late," +
" SUM(dss.pending) AS pending," +
" SUM(dss.not_received) AS not_received" +
" FROM deliveries AS dss" +
" INNER JOIN dsd.sla_datasets AS s ON dss.sla_id = s.sla_id" +
" INNER JOIN dsd.datasets AS ds ON ds.dataset_id = s.dataset_id" +
" INNER JOIN managers AS t on ds.team_id = t.ad_id" +
" INNER JOIN dsd.employee AS e on t.manager_id = e.id" +
" GROUP BY t.manager_id, dss.expected_delivery_date" +
")" +
"SELECT d.day AS expectedDeliveryDate, " +
"s.manager_id, " +
"COALESCE(s.total, 0) AS totalCount, " +
"COALESCE(s.on_time, 0) AS onTimeCount, " +
"COALESCE(s.late, 0) AS lateCount, " +
"COALESCE(s.pending, 0) AS pendingCount, " +
"COALESCE(s.not_received, 0) AS notReceivedCount " +
"FROM dates AS d " +
"LEFT JOIN summary AS s ON d.day = s.day " +
"ORDER BY s.manager_id, d.day";
#Query(value = MANAGER_DELIVERY, nativeQuery = true)
CompletableFuture<List<DeliveryStatusSummaryByManagerAndDate>> getDailyDeliveryStatusSummaryByManagers(
#Param("manager_ids") final Set<String> employeeIds,
#Param("range_start") ZonedDateTime rangeStart,
#Param("range_end") ZonedDateTime rangeEnd
);
}
Projection.
public interface DeliveryStatusSummaryByManagerAndDate {
String getManagerId();
LocalDate getExpectedDeliveryDate();
int getTotalCount();
int getOnTimeCount();
int getLateCount();
int getPendingCount();
int getNotReceivedCount();
}
getDailyDeliveryStatusSummaryByManagers works as expected.
But I need to have an opportunity to change group by section of this query on-the-fly, depending on user's input. So I decided to play with query as a plain string.
The idea is to put a query to string, and then depending on user's input, make query.replace('group placeholder', group by <needed list of fields>).
In order to archive this, I created a custom DeliveryStatusSummaryCustomRepository.
I decided not to make replace on a string so far, but to try to execute a simple query MANAGER_DELIVERY_QUERY.
public interface DeliveryStatusSummaryCustomRepository {
List<DeliveryStatusSummaryByManagerAndDate> getDailyDeliveryStatusSummaryByManagersV2();
}
#Repository
public class DeliveryStatusSummaryCustomRepositoryImpl implements DeliveryStatusSummaryCustomRepository {
private String MANAGER_DELIVERY_QUERY_TRY = "SELECT '1' AS managerId, " +
"CAST(dss.expected_delivery_date AS DATE) AS expectedDeliveryDate, " +
"0 AS totalCount, " +
"0 AS onTimeCount, " +
"0 AS lateCount, " +
"0 AS pendingCount, " +
"0 AS notReceivedCount " +
"FROM dsd.delivery_status_summary AS dss " +
"WHERE dss.expected_delivery_date >= CAST('01-01-2022' AS TIMESTAMP) AND dss.expected_delivery_date <= CAST('10-01-2022' AS TIMESTAMP)";
private final PrimaryDbConfig primaryDbConfig;
public DeliveryStatusSummaryCustomRepositoryImpl(PrimaryDbConfig primaryDbConfig) {
this.primaryDbConfig = primaryDbConfig;
}
#Override
public List<DeliveryStatusSummaryByManagerAndDate> getDailyDeliveryStatusSummaryByManagersV2() {
final LocalContainerEntityManagerFactoryBean em = primaryDbConfig.primaryEntityManager();
final EntityManager nativeEntityManager = em.createNativeEntityManager(new HashMap<>());
final Query query = nativeEntityManager.createQuery(MANAGER_DELIVERY_QUERY_TRY);
return query.getResultList();
}
}
But I got an error:
org.hibernate.hql.internal.ast.QuerySyntaxException: delivery_status_summary is not mapped
at org.hibernate.hql.internal.ast.util.SessionFactoryHelper.requireClassPersister(SessionFactoryHelper.java:169)
at org.hibernate.hql.internal.ast.tree.FromElementFactory.addFromElement(FromElementFactory.java:91)
at org.hibernate.hql.internal.ast.tree.FromClause.addFromElement(FromClause.java:77)
at org.hibernate.hql.internal.ast.HqlSqlWalker.createFromElement(HqlSqlWalker.java:333)
at org.hibernate.hql.internal.antlr.HqlSqlBaseWalker.fromElement(HqlSqlBaseWalker.java:3765)
at org.hibernate.hql.internal.antlr.HqlSqlBaseWalker.fromElementList(HqlSqlBaseWalker.java:3654)
at org.hibernate.hql.internal.antlr.HqlSqlBaseWalker.fromClause(HqlSqlBaseWalker.java:737)
at org.hibernate.hql.internal.antlr.HqlSqlBaseWalker.query(HqlSqlBaseWalker.java:593)
at org.hibernate.hql.internal.antlr.HqlSqlBaseWalker.selectStatement(HqlSqlBaseWalker.java:330)
at org.hibernate.hql.internal.antlr.HqlSqlBaseWalker.statement(HqlSqlBaseWalker.java:278)
at org.hibernate.hql.internal.ast.QueryTranslatorImpl.analyze(QueryTranslatorImpl.java:276)
at org.hibernate.hql.internal.ast.QueryTranslatorImpl.doCompile(QueryTranslatorImpl.java:192)
at org.hibernate.hql.internal.ast.QueryTranslatorImpl.compile(QueryTranslatorImpl.java:144)
at org.hibernate.engine.query.spi.HQLQueryPlan.<init>(HQLQueryPlan.java:113)
at org.hibernate.engine.query.spi.HQLQueryPlan.<init>(HQLQueryPlan.java:73)
at org.hibernate.engine.query.spi.QueryPlanCache.getHQLQueryPlan(QueryPlanCache.java:162)
at org.hibernate.internal.AbstractSharedSessionContract.getQueryPlan(AbstractSharedSessionContract.java:613)
at org.hibernate.internal.AbstractSharedSessionContract.createQuery(AbstractSharedSessionContract.java:725)
at org.hibernate.internal.AbstractSessionImpl.createQuery(AbstractSessionImpl.java:23)
at amp.ae.dataset.status.dashboard.library.repository.primary.DeliveryStatusSummaryCustomRepositoryImpl.getDailyDeliveryStatusSummaryByManagersV2(DeliveryStatusSummaryCustomRepositoryImpl.java:115)
at amp.ae.dataset.status.dashboard.library.repository.primary.DeliveryStatusSummaryCustomRepositoryImpl$$FastClassBySpringCGLIB$$f0397
I believe that the issue is related to DeliveryStatusSummaryByManagerAndDate is not entity, this is just projection.
If so, what approach to use for writing dynamic query taking into account that MANAGER_DELIVERY query is quite complicated to be re-written with Hibernate joins?
I heard about https://www.jooq.org.
Any advice appreciated.
You need to use EntityManager.createNativeQuery() method.
Also you can just autoware EntityManager
#Repository
public class DeliveryStatusSummaryCustomRepositoryImpl {
#PersistenceContext
private EntityManager entityManager;
}
To map query results to DTO
Replace the interface DeliveryStatusSummaryByManagerAndDate with POJO with getters and setters.
Use a result transformer from Hibernate
import org.hibernate.query.Query;
Query<DeliveryStatusSummaryByManagerAndDate> query = entityManager
.createNativeQuery(sql)
.unwrap(Query.class)
.setResultTransformer(
Transformers.aliasToBean(DeliveryStatusSummaryByManagerAndDate.class)
);
return query.getResultList();
Also check this mapping Hibernate query results to custom class?

JSoup, Extract specific text or image-link from website

This is in my HTML and I need to extract either the link of the image or the fileId to create a link.
{
"created": "2018-11-06T06:46:21.181Z",
"inRiverId": "58515",
"mediaInformation": {
"bildid": 67708,
"description": "ABC",
"excelImportField": "ABC",
"fileId": "41964",
"filename": "ABC",
"imageAccess": true,
"imageItemType": "ABC",
"imageStatus": "ABC",
"imageType": "ABC",
"itcl": "ABC",
"photographer": "ABC",
"projectName": "ABC",
"projectType": "Webb",
"type": "Bild",
"url": "https://static.john.com/images/products/41964.jpg"
},
The fileId is dynamic, therefor is the link also dynamic depending on which article I'm visiting on my website. The link will always start with "https://static.john.com/images/products/" and then it adds the fileId and ".jpeg" automatically.
The code above is of course just a piece of the code on the website, so there is more images on it, so it needs to extract it specifically.
The dream would be if Jsoup could get this output:
"https://static.john.com/images/products/" + "fileId" + ".jpeg"
I'm a beginner in Android Studio, and crazy-new to JSoup.
Bad news. Jsoup can't do this. Jsoup parses only HTML but this is a JSON fragment which will be used by Javascript after the page is loaded in the browser.
However my idea is to get this URL using regular expressions so try using this code:
String json = "{\n" +
" \"created\": \"2018-11-06T06:46:21.181Z\",\n" +
" \"inRiverId\": \"58515\",\n" +
" \"mediaInformation\": {\n" +
" \"bildid\": 67708,\n" +
" \"description\": \"ABC\",\n" +
" \"excelImportField\": \"ABC\",\n" +
" \"fileId\": \"41964\",\n" +
" \"filename\": \"ABC\",\n" +
" \"imageAccess\": true,\n" +
" \"imageItemType\": \"ABC\",\n" +
" \"imageStatus\": \"ABC\",\n" +
" \"imageType\": \"ABC\",\n" +
" \"itcl\": \"ABC\",\n" +
" \"photographer\": \"ABC\",\n" +
" \"projectName\": \"ABC\",\n" +
" \"projectType\": \"Webb\",\n" +
" \"type\": \"Bild\",\n" +
" \"url\": \"https://static.john.com/images/products/41964.jpg\"\n" +
" },";
Pattern p = Pattern.compile("\"url\": \"(https://static.john.com/images/products/\\d+.jp[e]*g)\"");
Matcher m = p.matcher(json);
if (m.find()) {
String imageUrl = m.group(1);
System.out.println(imageUrl);
}
and the output is: https://static.john.com/images/products/41964.jpg
In your case instead of json variable you should use document.html().
MainActivity.java
public class MainActivity extends AppCompatActivity {
public class JsoupGetImage {
public void main( String[] args ) throws IOException {
// Get the article number from TextView to complete link
String articlenumber = ((TextView) recyclerView.findViewHolderForAdapterPosition(0).itemView.findViewById(R.id.holderArticle)).getText().toString();
// My URL + article number
String url = "https://www.john.com/api/v1.0/articles/" + articlenumber;
// JSoup
Document doc = Jsoup.connect(url).get();
// Pattern - Find pattern using partial link
Pattern p = Pattern.compile("\"url\": \"(https://static.john.com/images/products/\\d+.jp[e]*g)\"");
// Matcher - Find match in my URL from Pattern
Matcher m = p.matcher(url);
if (m.find()) {
String imageUrl = m.group(1);
System.out.println(imageUrl);
}
}
}
}
Does this look good or am I missing something. How do I get my ImageView to show the image of the created link?
Kind regards,
John

Spring Data MongoDb - Criteria equivalent to a given query that uses $expr

I have a collection with documents like this:
{
"_id" : ObjectId("5a8ec4620cd3c2a4062548ec"),
"start" : 20,
"end" : 80
}
and I want to show the documents that overlap a given percentage (50%) with an interval (startInterval = 10, endInterval = 90).
I calculate the overlaping section with the following formula:
min(end , endInterval) - max(start, startInterval ) / (endInterval - startInterval)
In this example:
min(80,90) - max(20,10) / (90-10) = (80-20)/80 = 0.75 --> 75%
Then this document will be shown, as 75% is greater than 50%
I expressed this formula in mongo shell as:
db.getCollection('variants').find(
{
$expr: {
$gt: [
{
$divide: [
{
$subtract: [
{ $min: [ "$end", endInterval ] }
,
{ $max: [ "$start", startInterval ] }
]
}
,
{ $subtract: [ endInterval, startInterval ] }
]
}
,
overlap
]
}
}
)
where
overlap = 0.5, startInterval = 10 and endInterval= 90
It works fine in mongo shell.
I'm asking for an equivalent way to calculate this using Spring Data Criteria, since the $expr functionality I used in mongo shell is still to be implemented in Spring Data Mongo.
Currently I'm using Spring Boot 2.0.0, Spring Data MongoDb 2.0.5 and mongodb 3.6.
Thanks a lot for your time.
There is an open issue to add support for $expr : https://github.com/spring-projects/spring-data-mongodb/issues/2750
In the meantime, you can use an BasicQuery:
BasicQuery query = new BasicQuery("{ $expr: {'$gt': ['$results.cache.lastHit', '$results.cache.expiration']}}");
return ofNullable(mongoTemplate.findAndModify(query, updateDefinition, XXXX.class));
You can even concatenate your existing Criteria with BasicQuery, keeping it exclusive to the $expr:
Criteria criteria = Criteria.where("results.cache.cacheUpdateRetriesLeft").gt(4);
BasicQuery query = new BasicQuery("{ $expr: {'$gt': ['$results.cache.lastHit', '$results.cache.expiration']}}");
query.addCriteria(criteria);
return ofNullable(mongoTemplate.findAndModify(query, updateDefinition, XXXX.class));
Just in case it is helpful for somebody, I finally solved my problem using $redact.
String redact = "{\n" +
" \"$redact\": {\n" +
" \"$cond\": [\n" +
" {\n" +
" \"$gte\": [\n" +
" {\n" +
" \"$divide\": [\n" +
" {\n" +
" \"$subtract\": [\n" +
" {\n" +
" \"$min\": [\n" +
" \"$end\",\n" +
" " + endInterval + "\n" +
" ]\n" +
" },\n" +
" {\n" +
" \"$max\": [\n" +
" \"$start\",\n" +
" " + startInterval + "\n" +
" ]\n" +
" }\n" +
" ]\n" +
" },\n" +
" {\n" +
" \"$subtract\": [\n" +
" " + endInterval + "\n" +
" " + startInterval + "\n" +
" ]\n" +
" }\n" +
" ]\n" +
" },\n" +
" " + overlap + "\n" +
" ]\n" +
" },\n" +
" \"$$KEEP\",\n" +
" \"$$PRUNE\"\n" +
" ]\n" +
" }\n" +
" }";
RedactAggregationOperation redactOperation = new RedactAggregationOperation(
Document.parse(redact)
);
where RedactAggregationOperation is
public class RedactAggregationOperation implements AggregationOperation {
private Document operation;
public RedactAggregationOperation (Document operation) {
this.operation = operation;
}
#Override
public Document toDocument(AggregationOperationContext context) {
return context.getMappedObject(operation);
}
}
As you mentioned, Spring Data Mongo currently does not support $expr, so I have to use custom BSON document, and reflection of MongoTemplate.
public List<Variant> listTest() throws Exception {
double overlap = 0.5;
int startInterval = 10;
int endInterval= 90;
String jsonQuery = "{$expr:{$gt:[{$divide:[{$subtract:[{$min:[\"$end\","+endInterval+"]},{$max:[\"$start\","+startInterval+"]}]},{$subtract:["+endInterval+","+startInterval+"]}]},"+overlap+"]}}";
Document query = Document.parse(jsonQuery);
Method doFind = MongoTemplate.class.getDeclaredMethod("doFind", String.class, Document.class,Document.class,Class.class);
doFind.setAccessible(true);
return (List<Variant>) doFind.invoke(mongoTemplate, "variants", query, new Document(), Variant.class);
}
#NoArgsConstructor #Getter #Setter #ToString
public static class Variant{
int start;
int end;
}
As you may see, field mapping works OK.
Used Spring Data Mongo artifact is org.springframework:data.spring-data-mongodb:2.1.5.RELEASE
Support for $expr operator in spring-data-mongodb library is still non-existent. However there is a work around solution using MongoTemplate to solve this problem -
Aggregation.match() provides an overloaded method that accepts AggregationExpression as a parameter. This method can be used to create the query for $match aggregation pipeline with $expr operator as below -
Example usage of AggregationExpression for $match operator -
Aggregation aggregationQuery = Aggregation.newAggregation(Aggregation.match(AggregationExpression.from(MongoExpression.create("'$expr': { '$gte': [ '$foo', '$bar'] }"))));
mongoTemplate.aggregate(aggregationQuery, Entity.class);
Above code is the equivalent of query -
db.collection.aggregate([{"$match": {"$expr": {"$gte: ["$foo", "$bar"]}}}]);

Guava cache magic miss

Please help with this magic I observe here:
Map<String, RenditionMeta> map = cache.asMap();
System.out.println("Before iterating: " + map.containsKey(objectId));
for(String s : map.keySet()) {
if(s.equals(objectId)) {
System.out.println(s + " equals " + objectId + ":" + s.equals(objectId) + "-" + map.containsKey(objectId));
System.out.println(objectId.hashCode());
System.out.println(s.hashCode());
}
}
outputs:
Before iterating: false
09009e5d805f6b0b equals 09009e5d805f6b0b:true-false
1453886923
1453886923
Could someone explain how "true-false" is possible above?
cache is defined as
this.cache = CacheBuilder.newBuilder()
.concurrencyLevel(4)
.weakKeys()
.maximumSize(10000)
.expireAfterWrite(10, TimeUnit.MINUTES)
.build();
When you using weak keys option Guava cache uses identity equality for keys hence behaviour you observe.
CacheBuilder#weakKeys

Wicket Kendo DataTable filter issue

Are there any examples out there on how to property implement a filter using an IColumn under Wicket-Kendo DataTable component?
Note please.. I am using Wicket as the framework so I'd like to do this vi my framework..
I have the following:
columns.add(new PropertyColumn("Type", "publicationType") {
private static final long serialVersionUID = 1L;
#Override
public String getFilterable() {
return "{multi: true, dataSource: [{" +
"City: 'Seattle'," +
"},{" +
"City: 'Tacoma'," +
"},{" +
"City: 'Kirkland'," +
"},{" +
"City: 'Redmond'," +
"},{" +
"City: 'London'" +
"}]," +
"checkAll: false" +
"}}";
}
I have a datasource that I will use to add the actual options I need, but this does not render a checkable list for filtering on the column. Any advise??

Resources