When I query my index with query_string, I am getting results
But when I query using term query, I dont get any results
{
"query": {
"bool": {
"must": [],
"must_not": [],
"should": [
{
"query_string": {
"default_field": "Printer.Name",
"query": "HL-2230"
}
}
]
}
},
"from": 0,
"size": 10,
"sort": [],
"aggs": {}
}
I know that term is not_analyzed and query_string is analyzed but Name is already as "HL-2230", why doesnt it match with term query? I tried also searching with "hl-2230", I still didnt get any result.
EDIT: mapping looks like as below. Printer is the child of Product. Not sure if this makes difference
{
"state": "open",
"settings": {
"index": {
"creation_date": "1453816191454",
"number_of_shards": "5",
"number_of_replicas": "1",
"version": {
"created": "1070199"
},
"uuid": "TfMJ4M0wQDedYSQuBz5BjQ"
}
},
"mappings": {
"Product": {
"properties": {
"index": "not_analyzed",
"store": true,
"type": "string"
},
"ProductName": {
"type": "nested",
"properties": {
"Name": {
"store": true,
"type": "string"
}
}
},
"ProductCode": {
"type": "string"
},
"Number": {
"index": "not_analyzed",
"store": true,
"type": "string"
},
"id": {
"index": "no",
"store": true,
"type": "integer"
},
"ShortDescription": {
"store": true,
"type": "string"
},
"Printer": {
"_routing": {
"required": true
},
"_parent": {
"type": "Product"
},
"properties": {
"properties": {
"RelativeUrl": {
"index": "no",
"store": true,
"type": "string"
}
}
},
"PrinterId": {
"index": "no",
"store": true,
"type": "integer"
},
"Name": {
"store": true,
"type": "string"
}
}
},
"aliases": []
}
}
As per mapping provided by you above
"Name": {
"store": true,
"type": "string"
}
Name is analysed. So HL-2230 will split into two tokens, HL and 2230. That's why term query is not working and query_string is working. When you use term query it will search for exact term HL-2230 which is not there.
Related
I have the following mapping in elasticsearch5.5:
"append": {
"type": "short"
},
"comment": {
"type": "text",
"index": "analyzed"
},
"create_time": {
"type": "date"
},
"desc_score": {
"type": "integer"
},
"goods_id": {
"type": "long"
},
"goods_name": {
"type": "keyword",
"index": "not_analyzed"
},
"handler": {
"type": "keyword",
"index": "not_analyzed"
},
"is_first_review": {
"type": "short"
},
"logistic_score": {
"type": "integer"
},
"mall_id": {
"type": "long"
},
"order_id": {
"type": "long"
},
"order_sn": {
"type": "keyword",
"index": "not_analyzed"
},
"parent_id": {
"type": "keyword",
"index": "not_analyzed"
},
"review_id": {
"type": "long"
},
"score": {
"type": "integer"
},
"service_score": {
"type": "integer"
},
"shipping_id": {
"type": "long"
},
"shipping_name": {
"type": "keyword",
"index": "not_analyzed"
},
"status": {
"type": "short"
},
"tracking_number": {
"type": "keyword",
"index": "not_analyzed"
},
"update_time": {
"type": "date"
},
"user_id": {
"type": "long"
}
When I profile in kibana with the following statement and got the result in the picture.
{
"from": 0,
"size": 10,
"query": {
"bool": {
"filter": [
{
"terms": {
"goods_id": [
"262628158"
],
"boost": 1.0
}
},
{
"terms": {
"status": [
"2",
"4"
],
"boost": 1.0
}
},
{
"range": {
"create_time": {
"from": "1514027649",
"to": "1514632449",
"include_lower": true,
"include_upper": true,
"boost": 1.0
}
}
}
],
"disable_coord": false,
"adjust_pure_negative": true,
"boost": 1.0
}
},
"sort": [
{
"create_time": {
"order": "desc"
}
}
]
}
I am confused why status filter cost so much time, and status is a field with only five value, this is perpahps the reason cause the problem but I do not know how to optimize my search clause. I have search a lot with google but got no answer yet. Any one can help me?
Finally I find the answer, it is because elasticsearch5.5 use kd-tree to store index data for numeric data type(short, long, etc.) for optimize range search which can be found in following article:
Better Query Planning For Range Queries
Tune For Search Speed
Elasticsearch Query Execution Order
Kd-tree is not suitable for exact term, and I changed the filed type to keyword.
working with Elasticsearch/Kibana and trying to search on field in a nested object. However it does not seem to work. Here's mapping that I use in a template:
{
"order": 0,
"template": "ss7_signaling*",
"settings": {
"index": {
"mapping.total_fields.limit": 3000,
"number_of_shards": "5",
"refresh_interval": "30s"
},
"mappings": {
"_default_": {
"dynamic_templates": [
{
"string_fields": {
"mapping": {
"fielddata": {
"format": "disabled"
},
"index": "no",
"type": "string"
},
"match_mapping_type": "string",
"match": "*"
}
}
],
"properties": {
"message": {
"index": "not_analyzed",
"type": "string"
},
"Protocol": {
"index": "not_analyzed",
"type": "string"
},
"IMSI": {
"index": "not_analyzed",
"type": "string"
},
"nested": {
"type": "nested",
"properties": {
"name": {
"type": "string",
"index": "not_analyzed"
}
}
},
"Timestamp": {
"format": "strict_date_optional_time||epoch_millis",
"type": "date"
},
"#timestamp": {
"type": "date"
},
"#version": {
"index": "not_analyzed",
"type": "string"
}
},
"_all": {
"norms": false,
"enabled": false
}
}
},
"aliases": {
"signaling": {}
}
}
When I do search kibana on single fields - everything works fine. Still though i cannot search on nested fields like 'nested.name'.
Example of my query in kibana: nested.name:hi
Thanks.
Kibana uses the query_string query underneath, and the latter does not support querying on nested fields.
It's still being worked on but in the meantime you need to proceed differently.
UPDATE:
As of ES 7.6, it is now possible to search on nested fields
Hello I have a problem with the combination of multiple queries within Elasticsearch.
The problem only occurs whenever I try to combine a multi_match query with the geo_distance query. The multi_match query works when the geo_distance query is not present and the geo_distance query works when the multi_match query is not present.
Whenever I execute the multi_match query without the geo_distance query I get the results that I expect. I also get the expected results when I try the geo_distance query without the multi_match query.
Boths results contain the dataset that I would expect to receive when both queries are executed together. But whenever I execute them together I receive 0 results.
When I combine the geo_distance query with a simple term query the search works. So I presume it is problem with the combination of queries.
I would appreciate any ideas.
My query is the following:
{
"query": {
"bool": {
"must": {
"bool": {
"should": {
"multi_match": {
"query": "CompanyName GmbH",
"fields": [
"originalName",
"legalName"
],
"type": "cross_fields",
"operator": "AND"
}
}
}
},
"filter": {
"bool": {
"should": {
"geo_distance": {
"location": [
9.87107,
51.69915
],
"distance": "30.0km",
"distance_type": "arc"
}
}
}
}
}
}
}
The mapping behind all of that is:
{
"customer": {
"aliases": {
},
"mappings": {
"customer-entity": {
"properties": {
"communication": {
"properties": {
"domain": {
"type": "string"
},
"email": {
"type": "string"
},
"landline": {
"type": "string"
},
"mobile": {
"type": "string"
}
}
},
"id": {
"type": "long"
},
"legalName": {
"type": "string",
"store": true
},
"location": {
"type": "geo_point"
},
"operatingModes": {
"type": "string"
},
"originalName": {
"type": "string",
"store": true
}
}
},
"homepage-entity": {
"_parent": {
"type": "customer-entity"
},
"_routing": {
"required": true
},
"properties": {
"customerId": {
"type": "string",
"store": true
},
"id": {
"type": "long"
},
"metas": {
"type": "string",
"store": true
}
}
},
"person-entity": {
"_parent": {
"type": "customer-entity"
},
"_routing": {
"required": true
},
"properties": {
"customerId": {
"type": "string",
"store": true
},
"firstName": {
"type": "string",
"store": true
},
"id": {
"type": "long"
},
"lastName": {
"type": "string",
"store": true
},
"personId": {
"type": "string",
"store": true
}
}
}
},
"settings": {
"index": {
"refresh_interval": "-1",
"number_of_shards": "1",
"creation_date": "1488920698118",
"store": {
"type": "fs"
},
"number_of_replicas": "0",
"uuid": "ZcLN5sxASXGUnKZMg8mBpw",
"version": {
"created": "2040499"
}
}
},
"warmers": {
}
}
}
I have this mapping on ES 1.7.3:
{
"customer": {
"aliases": {},
"mappings": {
"customer": {
"properties": {
"addresses": {
"type": "nested",
"include_in_parent": true,
"properties": {
"address1": {
"type": "string"
},
"address2": {
"type": "string"
},
"address3": {
"type": "string"
},
"country": {
"type": "string"
},
"latitude": {
"type": "double",
"index": "not_analyzed"
},
"longitude": {
"type": "double",
"index": "not_analyzed"
},
"postcode": {
"type": "string"
},
"state": {
"type": "string"
},
"town": {
"type": "string"
},
"unit": {
"type": "string"
}
}
},
"companyNumber": {
"type": "string"
},
"id": {
"type": "string",
"index": "not_analyzed"
},
"name": {
"type": "string"
},
"status": {
"type": "string"
},
"timeCreated": {
"type": "date",
"format": "dateOptionalTime"
},
"timeUpdated": {
"type": "date",
"format": "dateOptionalTime"
}
}
}
},
"settings": {
"index": {
"refresh_interval": "1s",
"number_of_shards": "5",
"creation_date": "1472372294516",
"store": {
"type": "fs"
},
"uuid": "RxJdXvPWSXGpKz8pdcF91Q",
"version": {
"created": "1050299"
},
"number_of_replicas": "1"
}
},
"warmers": {}
}
}
The spring application generates this query:
{
"query": {
"bool": {
"should": {
"query_string": {
"query": "(addresses.\\*:sample* AND NOT status:ARCHIVED)",
"fields": [
"type",
"name",
"companyNumber",
"status",
"addresses.unit",
"addresses.address1",
"addresses.address2",
"addresses.address3",
"addresses.town",
"addresses.state",
"addresses.postcode",
"addresses.country"
],
"default_operator": "or",
"analyze_wildcard": true
}
}
}
}
}
on which "addresses.*:sample*" is the only input.
"query": "(sample* AND NOT status:ARCHIVED)"
Code above works but searches all fields of the customer object.
Since I want to search only on address fields I used the "addresses.*"
Query works only if the fields of the address object are of String type and before I added longitude and latitude fields of double type on address object. Now the error occurs because of these two new fields.
Error:
Parse Failure [Failed to parse source [{
"query": {
"bool": {
"should": {
"query_string": {
"query": "(addresses.\\*:sample* AND NOT status:ARCHIVED)",
"fields": [
"type",
"name",
"companyNumber","country",
"state",
"status",
"addresses.unit",
"addresses.address1",
"addresses.address2",
"addresses.address3",
"addresses.town",
"addresses.state",
"addresses.postcode",
"addresses.country",
],
"default_operator": "or",
"analyze_wildcard": true
}
}
}
}
}
]]
NumberFormatException[For input string: "sample"
Is there a way to search "String" fields within a nested object using addresses.* only?
The solution was to add "lenient": true. As per the documentation: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html
lenient - If set to true will cause format based failures (like providing text to a numeric field) to be ignored.
if i have a JSON document indexed into Elasticsearch, like the following:
"_source": {
"pid_no": 19321,
"aggregator_id": null,
"inet_family": "ipv4-unicast",
"origin_code": "igp",
"extended_community": null,
"atomic_aggregate": null,
"adv_type": "announce",
"local_preference": 250,
"med_metric": 0,
"time_stamp": 1447534931,
"net_mask": "23",
"prefix4_": {
"last": 222,
"first": 111
},
"counter_no": 69668,
"confederation_path": "",
"as_set": null,
and i have tried successfully to filter all of the keys of the doc,
but, except the nested ones.
the query looks like:
GET /SNIP!/SNIP!/_search?routing=SNIP!
{
"query": {
"bool": {
"must": {
"query": {
"match_all": {}
}
},
"filter": {
"bool": {
"filter": [
{
"range": {
"local_preference": {
"gt": 150,
"lte": 250
}
}
},
>>> if i remove the filter below, matches the document.
>>> when i apply the filter, i get 0 hits
{
"and": [
{
"range": {
"prefix4_.first": {
"lte": 200
}
}
},
{
"range": {
"prefix4_.last": {
"gte": 200
}
}
}
]
}
]
}
}
}
}
}
it goes without saying that the mapping is done using integers in the corresponding fields (prefix4_.first,prefix4_.last)
could you please advise on why the filtering does not work ?
EDIT: the mapping looks like this
{
"mappings": {
"_default_": {
"_all": { "enabled": False },
"dynamic": True,
"_routing": { "required": True },
"properties": {
"pid_no": { "type": "string", "index": "not_analyzed", "store": "no" },
"counter_no": { "type": "long", "store": "no" },
"time_stamp": { "type": "date", "format": "epoch_second", "store": "no" },
"host_name": { "type": "string", "index": "not_analyzed", "store": "no" },
"local_ip": { "type": "ip", "store": "no" },
"peer_ip": { "type": "ip", "store": "no" },
"local_asn": { "type": "string", "index": "not_analyzed", "store": "no" },
"peer_asn": { "type": "string", "index": "not_analyzed", "store": "no" },
"inet_family": { "type": "string", "index": "not_analyzed", "store": "no" },
"next_hop": { "type": "ip", "store": "no" },
"net_block": { "type": "string", "index": "analyzed", "store": "no" },
"as_path": { "type": "string", "index": "analyzed", "store": "no" },
"cluster_list": { "type": "string", "index": "not_analyzed", "store": "no" },
"confederation_path": { "type": "string", "index": "not_analyzed", "store": "no" },
"local_preference": { "type": "integer", "store": "no" },
"originator_ip": { "type": "ip", "store": "no" },
"origin_code": { "type": "string", "index": "not_analyzed", "store": "no" },
"community_note": { "type": "string", "index": "analyzed", "store": "no" },
"med_metric": { "type": "long", "store": "no" },
"atomic_aggregate": { "type": "boolean", "store": "no" },
"aggregator_id": { "type": "string", "index": "analyzed", "store": "no" },
"as_set": { "type": "string", "index": "analyzed", "store": "no" },
"extended_community": { "type": "string", "index": "not_analyzed", "store": "no" },
"adv_type": { "type": "string", "index": "not_analyzed", "store": "no" },
"prefix_": { "type": "string", "index": "not_analyzed", "store": "no" },
"net_mask": { "type": "integer", "store": "no" },
"prefix4_": {
"type": "nested",
"properties": {
"first": { "type": "integer", "store": "no" },
"last": { "type": "integer", "store": "no" }
}
},
"prefix6_": {
"type": "nested",
"properties": {
"lofirst": { "type": "long", "store": "no" },
"lolast": { "type": "long", "store": "no" },
"hifirst": { "type": "long", "store": "no" },
"hilast": { "type": "long", "store": "no" }
}
}
}
}
},
"settings" : {
"number_of_shards": 1,
"number_of_replicas": 0,
"index": {
"store.throttle.type": "none",
"memory.index_buffer_size": "20%",
"refresh_interval": "1m",
"merge.async": True,
"merge.scheduler.type": "concurrent",
"merge.policy.type": "log_byte_size",
"merge.policy.merge_factor": 15,
"cache.query.enable": True,
"cache.filter.type": "node",
"fielddata.cache.type": "node",
"cache.field.type": "soft"
}
}
}
Elasticsearch provides multiple ways of mapping nested documents. You are using nested which indexes nested documents as separate documents behind the scenes and as such querying them requires the use of a nested query.
The simplest way of indexing nested JSON like you've shown is using the object type mapping. This would allow you to query the field the way you were expecting, however Elasticsearch flattens the hierarchy which may not be acceptable for you.
use nested filters to filter your documents on nested fields.
https://www.elastic.co/guide/en/elasticsearch/reference/1.4/query-dsl-nested-filter.html
{
"query": {
"filtered": {
"filter": {
"bool": {
"must": [
{
"term": {
"peer_ip": "pqr",
"_cache": true
}
},
{
"nested": {
"filter": {
"bool": {
"must": [
{
"terms": {
"first": [
"xyz"
],
"_cache": true
}
}
]
}
},
"path": "prefix4_",
"inner_hits": {}
}
},
{
"terms": {
"pid_no": [
"yyu"
],
"_cache": true
}
}
]
}
}
}
}
}