I'm trying a simple terms aggregation but the result is not creating buckets. Here is a sample document:
"hits": {
"total": 27330,
"max_score": 0.8293952,
"hits": [
{
"_index": "policy",
"_type": "policy",
"_id": "W0051311PNWO",
"_score": 0.8293952,
"_source": {
"productname": "UK CARGO",
"alternateproductname": "ABC39393939",
"brokername": "Name***",
"agentname": "Name***",
"policyref": "ABC33333",
"client": "International Cargo Limited",
"addressline1": "",
"post/zipcode": "",
"telephone": null,
"bapolicyendorseid": 123334,
"prevcertnum": "",
"policystatus": "Endorsed",
"#version": "1",
"#timestamp": "2015-10-09T11:11:02.018Z"
}
},
Here is the aggregate search (in sense):
get policy/policy/_search
{
"aggs": {
"statuses": {
"terms": {
"field": "policystatus"
}
}
}
}
I'm trying to get the equivalent of:
select policystatus, count(*) from policy group by policystatus
The result is not showing buckets. It is showing regular document results:
{
"took": 6,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 227398,
"max_score": 1,
"hits": [
{
"_index": "policy",
"_type": "policy",
"_id": "04/QQQ/04UKI0018",
"_score": 1,
"_source": {
"productname": "2 RES 01/09/04",
"alternateproductname": "2 RES 01/09/04",
"brokername": "Blah LTD",
"agentname": "Insurance",
"policyref": "blah",
"client": "blah",
"addressline1": "blah",
"post/zipcode": "blah",
"telephone": null,
"bapolicyendorseid": 21427,
"prevcertnum": "04UKI0018",
"policystatus": "Pending",
"#version": "1",
"#timestamp": "2015-10-09T11:10:10.146Z"
}
},
Try this:
GET /policy/policy/_search?search_type=count
{
"aggs": {
"statuses": {
"terms": {
"field": "policystatus"
}
}
}
}
meaning capital letters GET and search_type=count to get only the buckets, not also the hits.
Related
I am reading the documentation of elastic search and using the fuzzy query but it won't work
I am following the below documentation
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-fuzzy-query.html
postman
GET https://localhost:0000/myindex/_search
I have a 3 records
{
"took": 3,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 4,
"relation": "eq"
},
"max_score": 1.0,
"hits": [
{
"_index": "myindex",
"_type": "default",
"_id": "ioSZw4UBBGNKkg-oBT13",
"_score": 1.0,
"_source": {
"name": "Rahul_Patil",
"contents": {
"topic": "Elasticsearch with postgresql configuration",
"details": "test index"
}
}
},
{
"_index": "myindex",
"_type": "default",
"_id": "jYRCxIUBBGNKkg-ojj36",
"_score": 1.0,
"_source": {
"name": "Rahul_Patilt_test_Index",
"contents": {
"topic": "Elasticsearch with postgresql configuration 2",
"details": "test index 3"
}
}
},
{
"_index": "myindex",
"_type": "default",
"_id": "j4RexIUBBGNKkg-owj3w",
"_score": 1.0,
"_source": {
"name": "Rahul_Patil_test_Index_updates_new",
"contents": {
"topic": "Elasticsearch with postgresql configuration update",
"details": "test index update"
}
}
},
{
"_index": "myindex",
"_type": "default",
"_id": "1",
"_score": 1.0,
"_source": {
"name": "abc_Index",
"contents": {
"topic": "abc 2",
"details": "abc 3"
}
}
}
]
}
}
now I change body->raw
GET https://localhost:0000/myindex/_search
body->raw
{
"query": {
"fuzzy": {
"_source.name": {
"value": "Ra"
}
}
}
}
the response is ok but I am not able to find whose name starts with Rahul_Patil
{
"took": 3,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 0,
"relation": "eq"
},
"max_score": null,
"hits": []
}
}
I want to display only the items that contain the word itself when "google" searches
How can I only search for items that have only the word "google"?
Request body
(Request created in postman)
{
"query": {
"bool": {
"must": [
{
"match": {
"body": "google"
}
}
]
}
}
}
Response body
(Request created in postman)
{
"took": 0,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 3,
"relation": "eq"
},
"max_score": 0.6587735,
"hits": [
{
"_index": "s_t",
"_type": "_doc",
"_id": "3",
"_score": 0.6587735,
"_source": {
"body": "google"
}
},
{
"_index": "s_t",
"_type": "_doc",
"_id": "4",
"_score": 0.5155619,
"_source": {
"body": "google map"
}
},
{
"_index": "s_t",
"_type": "_doc",
"_id": "5",
"_score": 0.5155619,
"_source": {
"body": "google-map"
}
}
]
}
}
I need this output
(Request created in postman)
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 3,
"relation": "eq"
},
"max_score": 0.69381464,
"hits": [
{
"_index": "s_t",
"_type": "_doc",
"_id": "3",
"_score": 0.69381464,
"_source": {
"body": "google"
}
}
]
}
}
In mysql with this query I reach my goal.
Similar query in mysql:
select * from s_t where body='google'
well i assume you automap or use a text in your mappings.
specify .keyword in your query. Note this is case sensitive.
{
"query": {
"bool": {
"must": [
{
"match": {
"body.keyword": "google"
}
}
]
}
}
}
If you only want to query your body field using exact match. You need to reindex it using keyword. Take a look at: Exact match in elastic search query
When we call the elasticsearch, say as follows:
POST https:////_search with body:
{
"from": 0,
"size": 1,
"query": {
"bool": {
"must": [
{
"range": {
"createdAt": {
"gt": "2019-11-11T10:00:00"
}
}
}
]
}
},
"sort": [
{
"createdAt" : {
"order" : "desc"
}
}
]
}
I see that I get only 1 result as pagination is set to 1 but total inside hits in response shows 2. This is the response I get:
{
"took": 4,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 2,
"relation": "eq"
},
"max_score": null,
"hits": [
{
"_index": “<index-name>”,
"_type": "_doc",
"_id": "5113c843-dff3-499f-a12e-44c7ac103bcf_0",
"_score": null,
"_source": {
"oId": "5113c843-dff3-499f-a12e-44c7ac103bcf",
"oItemId": 0,
"createdAt": "2019-11-13T11:00:00"
},
"sort": [
1573642800000
]
}
]
}
}
Doesn’t total doesn’t capture the pagination part? And it only cares about the query report? It should show the total count of items matching the query irrespective of the pagination set, right?
Yes, You are right that total doesn't capture the pagination part and just cares about the query report ie. whatever the total no of the document matches for a given query.
To be precise, it is as explained in official ES docs .
total (Object) Metadata about the number of returned documents.
Returned parameters include:
value: Total number of returned documents. relation: Indicates whether
the number of documents returned. Returned values are:
eq: Accurate gte: Lower bound, including returned documents
It means its the total no of returned documents, but as pagination is set to 1 in your example, inner hits have just 1 document.You can cross-check this understanding easily by creating a sample example as below:
Create a sample index with just 1 text field:
URL:- http://localhost:9200/{your-index-name}/ --> PUT method
{
"mappings": {
"properties": {
"name": {
"type": "text"
}
}
},
"settings": {
"index": {
"number_of_shards": "1",
"number_of_replicas": "1"
}
}
}
Once the above index is created index below 4 documents:
URL:- http://localhost:9200/{your-index-name}/_doc/{1,2,like..} --> POST method
{
"name": "foo 1"
}
{
"name": "foo bar"
}
{
"name": "foo"
}
{
"name": "foo 2"
}
Now when you hit below search query without pagination:
{
"query": {
"bool": {
"must": [
{
"match": {
"name": "foo"
}
}
]
}
}
}
It gives below response:
{
"took": 9,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 4, --> Note 4 here
"relation": "eq"
},
"max_score": 0.12199639,
"hits": [
{
"_index": "59638303",
"_type": "_doc",
"_id": "1",
"_score": 0.12199639,
"_source": {
"name": "foo"
}
},
{
"_index": "59638303",
"_type": "_doc",
"_id": "3",
"_score": 0.12199639,
"_source": {
"name": "foo"
}
},
{
"_index": "59638303",
"_type": "_doc",
"_id": "2",
"_score": 0.09271725,
"_source": {
"name": "foo bar"
}
},
{
"_index": "59638303",
"_type": "_doc",
"_id": "4",
"_score": 0.09271725,
"_source": {
"name": "foo 1"
}
}
]
}
}
But when you hit a search query with pagination:
{
"from": 0,
"size": 1,--> note size 1
"query": {
"bool": {
"must": [
{
"match": {
"name": "foo"
}
}
]
}
}
}
it gives below response
{
"took": 23,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 4, --> this is still 4
"relation": "eq"
},
"max_score": 0.12199639,
"hits": [
{
"_index": "59638303",
"_type": "_doc",
"_id": "1",
"_score": 0.12199639,
"_source": {
"name": "foo"
}
}
]
}
}
Now in the above query, you can change the size and check only inner-hits array gets change but the outer hits object which contains total always remains same as 4, this confirms your understanding is correct.
I'm looking for a solution to find duplicate(exact) Docs in ElasticSearch.
I've read https://qbox.io/blog/minimizing-document-duplication-in-elasticsearch and tried it but its results are not as I expected as example this is my sample simple query :
GET /last_month_ads/_search
{
"size": 0,
"fields": [
"title"
],
"aggs": {
"duplicateCount": {
"terms": {
"field": "title",
"size" : 3
},
"aggs": {
"duplicateDocuments": {
"top_hits": {}
}
}
}
}
}
and the result is
{
"took": 981,
"timed_out": false,
"_shards": {
"total": 2,
"successful": 2,
"failed": 0
},
"hits": {
"total": 482909,
"max_score": 0,
"hits": []
},
"aggregations": {
"duplicateCount": {
"doc_count_error_upper_bound": 11667,
"sum_other_doc_count": 1958146,
"buckets": [
{
"key": "CM",
"doc_count": 46867,
"duplicateDocuments": {
"hits": {
"total": 46867,
"max_score": 1,
"hits": [
{
"_index": "last_month_ads",
"_type": "ads",
"_id": "AV73EtoBQTqkjEa7YQG1",
"_score": 1,
"_source": {
"id": "20642316",
"cat_id": "43606",
"user_id": "1825875",
"title": "125 CM HOME",
"desc": "DESC"
}
},
{
"_index": "last_month_ads",
"_type": "ads",
"_id": "AV73EtpdQTqkjEa7YQHc",
"_score": 1,
"_source": {
"id": "20642379",
"cat_id": "43604",
"user_id": "4642299",
"title": "Home with Big CM",
"desc": "DESC"
}
},
{
"_index": "last_month_ads",
"_type": "ads",
"_id": "AV73Etp6QTqkjEa7YQHp",
"_score": 1,
"_source": {
"id": "20642409",
"cat_id": "43607",
"user_id": "4813303",
"title": "100 of live CM is here ",
"desc": "DESC"
}
}
]
}
}
},
}
]
}
}
}
I'm looking for Exact (or similar) titles not abundance words in titles, how can I get get Duplicate(similar) Docs in Elastic Search?
I 'am using Elastic search 5.1.1,how to get all data specified for these below fields (FeatureValue,FeatureName)
sample document
{
"_index": "rawdata",
"_type": "feed",
"_id": "591031",
"_score": 1,
"_source": {
"sourceproductname": "1-5-Size Relays",
"zmfrid": 4,
"sourcetitle": null,
"featurename": "Coil Magnetic System",
"localsourcepath": null,
"sourcingdate": "2017-01-08T22:00:00.000Z",
"migrationstatus": 3,
"featrueunit": null,
"inputkeyword": null,
"#version": "1",
"sourcetype": "DirectFeed",
"id": 591031,
"sourceid": 674,
"partdataid": null,
"partid": null,
"featurecondition": null,
"sourcingstatus": null,
"sourcetaxonomypath": "1-5-Size Relays",
"sourcename": "CrunchBase ",
"tags": [],
"sourceurl": "N/A",
"#timestamp": "2017-01-10T11:51:54.095Z",
"featurevalue": "Non-Polarized, Monostable",
"mfr": "feed",
"partdataattributeid": null,
"supplierfamily": "null",
"partnumber": "4-1617072-5"
}
}
I tried this
POST /rawdata/feed/_search?pretty=true
{
"_source": ["FeatureValue", "FeatureName"],
"query": {
"match_all":{}
}
}
sample result
{
"took": 4,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 386424,
"max_score": 1,
"hits": [
{
"_index": "rawdata",
"_type": "feed",
"_id": "591031",
"_score": 1,
"_source": {}
}
You simply need to lowercase the field names in the source filter since they are lowercased in your documents
POST /rawdata/feed/_search?pretty=true
{
"_source": ["featurevalue", "featurename"], <--- change this
"query": {
"match_all":{}
}
}