Elasticsearch query time boosting produces result in inadequate order - elasticsearch

The ES search result for the given search keyword one two three seems to be wrong after applying boost feature per keyword. Please help me modifying my "faulty" query in order to accomplish "expected result" below as I described. I'm on ES 1.7.4 with LUCENE 4.10.4
Boosting criteria -three is regarded as the most important keyword:
word - boost
---- -----
one 1
two 2
three 3
ES index content - just showing MySQL dump to make the post shorter
mysql> SELECT id, title FROM post;
+----+-------------------+
| id | title |
+----+-------------------+
| 1 | one |
| 2 | two |
| 3 | three |
| 4 | one two |
| 5 | one three |
| 6 | one two three |
| 7 | two three |
| 8 | none |
| 9 | one abc |
| 10 | two abc |
| 11 | three abc |
| 12 | one two abc |
| 13 | one two three abc |
| 14 | two three abc |
+----+-------------------+
14 rows in set (0.00 sec)
Expected ES query result - The user is searching for one two three. I'm not fussed about the order of equally scored records. I mean if record 6 and 13 switches places, I don't mind.
+----+-------------------+
| id | title | my scores for demonstration purposes
+----+-------------------+
| 6 | one two three | (1+2+3 = 6)
| 13 | one two three abc | (1+2+3 = 6)
| 7 | two three | (2+3 = 5)
| 14 | two three abc | (2+3 = 5)
| 5 | one three | (1+3 = 4)
| 4 | one two | (1+2 = 3)
| 12 | one two abc | (1+2 = 3)
| 3 | three | (3 = 3)
| 11 | three abc | (3 = 3)
| 2 | two | (2 = 2)
| 10 | two abc | (2 = 2)
| 1 | one | (1 = 1)
| 9 | one abc | (1 = 1)
| 8 | none | <- This shouldn't appear
+----+-------------------+
14 rows in set (0.00 sec)
Unexpected ES query result - Unfortunately, This is what I get.
+----+-------------------+
| id | title | _score
+----+-------------------+
| 6 | one two three | 1.0013864
| 13 | one two three abc | 1.0013864
| 4 | one two | 0.57794875
| 3 | three | 0.5310148
| 7 | two three | 0.50929534
| 5 | one three | 0.503356
| 14 | two three abc | 0.4074363
| 11 | three abc | 0.36586377
| 12 | one two abc | 0.30806428
| 10 | two abc | 0.23231897
| 2 | two | 0.12812772
| 1 | one | 0.084527075
| 9 | one abc | 0.07408653
+----+-------------------+
ES query
curl -XPOST "http://127.0.0.1:9200/_search?post_dev" -d'
{
"query": {
"bool": {
"must": {
"match": {
"title": {
"query": "one two three"
}
}
},
"should": [
{
"match": {
"title": {
"query": "one",
"boost": 1
}
}
},
{
"match": {
"title": {
"query": "two",
"boost": 2
}
}
},
{
"match": {
"title": {
"query": "three",
"boost": 3
}
}
}
]
}
},
"sort": [
{
"_score": {
"order": "desc"
}
}
],
"from": "0",
"size": "100"
}'
Some more test queries:
This query doesn't produce any result.
This query doesn't order correctly as seem here.

# Index some test data
curl -XPUT "localhost:9200/test/doc/1" -d '{"title": "one"}'
curl -XPUT "localhost:9200/test/doc/2" -d '{"title": "two"}'
curl -XPUT "localhost:9200/test/doc/3" -d '{"title": "three"}'
curl -XPUT "localhost:9200/test/doc/4" -d '{"title": "one two"}'
curl -XPUT "localhost:9200/test/doc/5" -d '{"title": "one three"}'
curl -XPUT "localhost:9200/test/doc/6" -d '{"title": "one two three"}'
curl -XPUT "localhost:9200/test/doc/7" -d '{"title": "two three"}'
curl -XPUT "localhost:9200/test/doc/8" -d '{"title": "none"}'
curl -XPUT "localhost:9200/test/doc/9" -d '{"title": "one abc"}'
curl -XPUT "localhost:9200/test/doc/10" -d '{"title": "two abc"}'
curl -XPUT "localhost:9200/test/doc/11" -d '{"title": "three abc"}'
curl -XPUT "localhost:9200/test/doc/12" -d '{"title": "one two abc"}'
curl -XPUT "localhost:9200/test/doc/13" -d '{"title": "one two three abc"}'
curl -XPUT "localhost:9200/test/doc/14" -d '{"title": "two three abc"}'
# Make test data available for search
curl -XPOST "localhost:9200/test/_refresh?pretty"
# Search using function score
curl -XPOST "localhost:9200/test/doc/_search?pretty" -d'{
"query": {
"function_score": {
"query": {
"match": {
"title": "one two three"
}
},
"functions": [
{
"filter": {
"query": {
"match": {
"title": "one"
}
}
},
"weight": 1
},
{
"filter": {
"query": {
"match": {
"title": "two"
}
}
},
"weight": 2
},
{
"filter": {
"query": {
"match": {
"title": "three"
}
}
},
"weight": 3
}
],
"score_mode": "sum",
"boost_mode": "replace"
}
},
"sort": [
{
"_score": {
"order": "desc"
}
}
],
"from": "0",
"size": "100"
}'

Related

Grafana & Elastic - How to count sub array length

So I have a document that has two nested arrays i.e.
foo.bars[].baz[]
I am trying to figure out how I can use graphana to group by bars and give me a count of bar's for each bar. So it would look something like:
| bars.id| count|
| 1 | 10 |
| 2 | 15 |
| 3 | 20 |
What I have tried is the following:
Group by bars.id
Add a Sum metric for bars.baz.id
Override the script value to return 1
While this does give me the count of the bars, it does so for all bars in the document and not grouped by the bars.id i.e.
| bars.id| count|
| 1 | 45 |
| 2 | 45 |
| 3 | 45 |
Any help to achieve this would be very helpful.
Now if this can be done I have another more complex problem. I have another collection let's call it bobs that is a child of the root document. Now bobs isn't nested under the bars array but it has a bar.id field. I would also like to sum this based on that i.e.
{
bobs: [
{bar_id: 1},
{bar_id: 2},
],
bars: [
{id: 1, bazes: []},
{id: 2, bazes: []}
]
}
In this case I would also like in the table:
| bars.id| bobs.count|
| 1 | 1 |
| 2 | 1 |
| 3 | 0 |
Is this possible?

ElasticSearch Search for specified value within `FromX` and `ToY` fields

I want to query by specified value within range that made by value of two FromX and ToY fields, and search Title as text field by query_string query.
This example shows my goal:
Id | FromX | ToY | Title
-----------------------------
1 | 1 | 7 | Mohammad
2 | 2 | 3 | Ali
3 | 1 | 6 | MohammadAli
4 | 2 | 5 | MohammadReza
5 | 1 | 2 | AliReza
6 | 2 | 2 | Sayed Ali
example query:
value: 2 AND title: *Ali*
result for query:
Id | FromX | ToY | Title
-----------------------------
2 | 2 | 3 | Ali
3 | 1 | 6 | MohammadAli
5 | 1 | 2 | AliReza
6 | 2 | 2 | Sayed Ali
Update 1:
Add last record with Id=6 in the sample data and result.
The following query should give you what you expect:
{
"query": {
"bool": {
"filter": [
{
"range": {
"FromX": {
"lte": 2
}
}
},
{
"range": {
"ToY": {
"gte": 2
}
}
},
{
"query_string": {
"query": "*ali*"
}
}
]
}
}
}
However, not that prefix wildcards should be avoided at all cost as they will penalize the performance of your query. You should probably analyze your title field using ngrams and do normal match queries on the Title field, instead.

Parsing JSON file-jq [duplicate]

This question already has answers here:
jq not working on tag name with dashes and numbers
(2 answers)
Closed 4 years ago.
Whole file:https://1drv.ms/u/s!AizscpxS0QM4hJpEkp12VPHiKO_gBg
Using this command i get part bellow (get latest job)
jq '.|[ .executions[] | select(.job.name != null) | select(.job.name) ]
| sort_by(.id)
| reverse
| .[0] ' 1.json
{
"argstring": null,
"date-ended": {
"date": "2018-04-03T17:43:38Z",
"unixtime": 1522777418397
},
"date-started": {
"date": "2018-04-03T17:43:34Z",
"unixtime": 1522777414646
},
"description": "",
"executionType": "user",
"failedNodes": [
"172.30.61.88"
],
"href": "http://172.30.61.88:4440/api/21/execution/126",
"id": 126,
"job": {
"averageDuration": 4197,
"description": "",
"group": "",
"href": "http://172.30.61.88:4440/api/21/job/271cbcec-5042-4d52-b794-ede2056b2ab8",
"id": "271cbcec-5042-4d52-b794-ede2056b2ab8",
"name": "aa",
"permalink": "http://172.30.61.88:4440/project/demo/job/show/271cbcec-5042-4d52-b794-ede2056b2ab8",
"project": "demo"
},
"permalink": "http://172.30.61.88:4440/project/demo/execution/show/126",
"project": "demo",
"status": "failed",
"user": "administrator"
I managed to extract job name and status, now want to get date-ended.date ?
jq '.|[ .executions[] |select(.job.name != null) | select(.job.name) ]
| sort_by(.id)
| reverse
| .[0]
| "\(.status), \(.job.name)"' 1.json
With the "-r" command-line option, the following filter:
[.executions[] | select(.job.name != null)]
| sort_by(.id)
| reverse
| .[0]
| [.status, .job.name, ."date-ended".date]
| #csv
produces:
"failed","aa","2018-04-03T17:43:38Z"
An important point that you might have missed is that "-" is a "special" character in that it can signify negation or subtraction.
If your jq does not support the syntax ."date-ended".date, then you could fall back to the basic syntax: (.["date-ended"] | .date)
I guess you have troubles extracting .date-ended.date because the name contains a dash that is interpreted by jq as subtraction.
The solution is listed in the documentation:
If the key contains special characters, you need to surround it with double quotes like this: ."foo$", or else .["foo$"].
This means the last filter of your jq program should be:
"\(.status), \(.job.name), \(."date-ended".date)"

Configure Logstash to create an Elasticsearch document with nested arrays

I'm indexing my PostgreSQL data for Elasticsearch using the Logstash JDBC Input Plugin. I have two tables called REQUEST and ASSIGNMENT listed below.
How can I use Logstash to index the two tables into one Elasticsearch document of type REQUEST with a nested arrays for all child ASSIGNMENT records?
Table: REQUEST
REQUEST_ID | POC
---------- | ----------------
1234 | Jon Snow
1256 | Tyrion Lannister
Table: ASSIGNMENT
ASSIGN_ID | REQUEST_ID | STATUS | CREATED
--------- | ---------- | ------- | ----------
2345 | 1234 | New | 2017-01-06
2364 | 1234 | Working | 2017-03-12
2399 | 1234 | Working | 2017-05-20
5736 | 1256 | New | 2017-06-28
This is what I want my Elasticsearch document to look like. It is a sample of the _source value of the search result:
"_source": {
"request_id": "1234",
"poc": "Jon Snow",
"assignments": [
{
"assign_id": "2345",
"status": "New",
"created": "2017-01-06"
},
{
"assign_id": "2364",
"status": "Working",
"created": "2017-03-12"
},
{
"assign_id": "2399",
"status": "Working",
"created": "2017-05-20"
}
]
}

Elasticsearch index with jdbc driver

Sorry my english is bad
I am using elasticsearch and jdbc river. I have two table with many-to-many relations. For example:
product
+---+---------------+
| id| title |
+---+---------------+
| 1 | Product One |
| 2 | Product Two |
| 3 | Product Three |
| 4 | Product Four |
| 5 | Product Five |
+---+---------------+
product_category
+------------+-------------+
| product_id | category_id |
+------------+-------------+
| 1 | 1 |
| 1 | 2 |
| 1 | 3 |
| 2 | 4 |
| 2 | 5 |
+------------+-------------+
category
+---+---------------+
| id| name |
+---+---------------+
| 1 | Category One |
| 2 | Category Two |
| 3 | Category Three|
| 4 | Category Four |
| 5 | Category Five |
+---+---------------+
I want to use array type.
{
"id": 1,
"name": "Product one",
"categories": {"Category One", "Category Two", "Category Three"}
},
How should I write a sql?
Use elasticsearch-jdbc structured objects with sql, no need to group_concat:
SELECT
product.id AS _id,
product.id,
title,
name AS categories
FROM product
LEFT JOIN (
SELECT *
FROM product_category
LEFT JOIN category
ON product_category.category_id = category.id
) t
ON product.id = t.product_id
Since river has been deprecated since ES v1.5, maybe run a standalone importer is better.

Resources