Related
when it comes to elasticsearch, i am new to it. That's why, my question is if i'd like to filter multiple fields, would aggregate might solve the problem? or what should i use?
{
"aggs": {
"filtered": {
"filter": {
"bool": {
"must": [
{ "term" : {"shop_slug" : "sharkys-dhammazedi.6326"}},
{ "term" :
{"slug" : "anchovy-essence"}}
]
}
}
}
}}
HERE Is The Mapping
"product_catalogue": {
"mappings": {
"products": {
"properties": {
"buy_online": {
"type": "long"
},
"category": {
"properties": {
"id": {
"type": "long"
},
"name": {
"type": "string"
},
"slug": {
"type": "string"
},
"tree": {
"properties": {
"2": {
"type": "string"
},
"3": {
"type": "string"
},
"5": {
"type": "string"
},
"10": {
"type": "string"
},
"11": {
"type": "string"
},
"12": {
"type": "string"
},
"13": {
"type": "string"
},
"14": {
"type": "string"
},
"15": {
"type": "string"
},
"16": {
"type": "string"
},
"17": {
"type": "string"
},
"18": {
"type": "string"
},
"19": {
"type": "string"
},
"20": {
"type": "string"
},
"21": {
"type": "string"
},
"22": {
"type": "string"
},
"23": {
"type": "string"
},
"24": {
"type": "string"
},
"25": {
"type": "string"
},
"26": {
"type": "string"
},
"27": {
"type": "string"
},
"28": {
"type": "string"
},
"29": {
"type": "string"
},
"30": {
"type": "string"
},
"31": {
"type": "string"
},
"32": {
"type": "string"
},
"33": {
"type": "string"
},
"34": {
"type": "string"
},
"35": {
"type": "string"
},
"36": {
"type": "string"
},
"37": {
"type": "string"
},
"38": {
"type": "string"
},
"39": {
"type": "string"
},
"40": {
"type": "string"
},
"41": {
"type": "string"
},
"42": {
"type": "string"
},
"43": {
"type": "string"
},
"46": {
"type": "string"
},
"62": {
"type": "string"
},
"72": {
"type": "string"
},
"73": {
"type": "string"
},
"74": {
"type": "string"
},
"75": {
"type": "string"
},
"77": {
"type": "string"
},
"137": {
"type": "string"
},
"139": {
"type": "string"
},
"140": {
"type": "string"
},
"141": {
"type": "string"
}
}
}
}
},
"click_collect": {
"type": "long"
},
"company_id": {
"type": "string"
},
"company_name": {
"type": "string"
},
"company_slug": {
"type": "string",
"index": "not_analyzed"
},
"condition": {
"type": "string"
},
"created_at": {
"type": "date",
"format": "dateOptionalTime"
},
"currency_iso": {
"type": "string"
},
"delivery_available": {
"type": "boolean"
},
"delivery_type": {
"properties": {
"id": {
"type": "long"
},
"name": {
"type": "string"
},
"pivot": {
"properties": {
"deliverytype_id": {
"type": "long"
},
"pricing": {
"type": "string"
},
"shop_id": {
"type": "long"
}
}
}
}
},
"description": {
"type": "string",
"analyzer": "snowball"
},
"group_id": {
"type": "string"
},
"image_url": {
"type": "string"
},
"listed_shop": {
"type": "integer"
},
"location": {
"type": "geo_point"
},
"mainPhoto": {
"type": "string"
},
"mm_description": {
"type": "string"
},
"mm_title": {
"type": "string"
},
"postcode": {
"type": "string"
},
"prices": {
"properties": {
"retail": {
"properties": {
"converted": {
"properties": {
"currencyIso": {
"type": "string"
},
"decimal": {
"type": "string"
},
"formatted": {
"type": "string"
}
}
},
"original": {
"properties": {
"currencyIso": {
"type": "string"
},
"decimal": {
"type": "string"
},
"formatted": {
"type": "string"
}
}
}
}
}
}
},
"shop_category": {
"type": "long"
},
"shop_id": {
"type": "long"
},
"shop_name": {
"type": "string"
},
"shop_slug": {
"type": "string",
"index": "not_analyzed"
},
"shop_stock_count": {
"type": "integer"
},
"shop_type": {
"properties": {
"id": {
"type": "long"
},
"name": {
"type": "string"
}
}
},
"slug": {
"type": "string"
},
"title": {
"type": "string",
"analyzer": "snowball"
},
"user_stock_count": {
"type": "integer"
},
"variant": {
"properties": {
"Colour": {
"type": "string"
},
"Disk Space": {
"type": "string"
},
"Length": {
"type": "string"
}
}
}
}
}
}
}
This query didn't seems to work out . Infact, it only filter one of the fields but can't resolve to filter both fields which is shop_slug and slug. Please kindly answer Thanks.
Having read all the tutorials on ES website, I cannot achieve my goal.
We are using ES 1.7.6 and I want to have only one single instance of documents matching my criteria. But what I get from ES is all the data matching the filter and the aggregation statistics.
GET _search
{
"size":1000,
"query":{
"bool":{
"should":[
{
"match":{
"isoMessage.fields.39":{
"query":"00"
}
}
}
]
}
},
"aggs":{
"group_by_CATI":{
"terms":{
"field":"isoMessage.fields.41"
}
}
}
}
Note that the index of isoMessage.fields.41 is set to not_analyzed.
Thanks for any help;
UPDATE: The mapping
{
"cm": {
"mappings": {
"Event": {
"properties": {
"deleted": {
"type": "boolean"
},
"id": {
"type": "string"
},
"isoMessage": {
"properties": {
"fields": {
"properties": {
"2": {
"type": "string"
},
"3": {
"type": "string"
},
"4": {
"type": "string"
},
"7": {
"type": "string"
},
"11": {
"type": "string"
},
"12": {
"type": "string"
},
"13": {
"type": "string"
},
"14": {
"type": "string"
},
"22": {
"type": "string"
},
"25": {
"type": "string"
},
"32": {
"type": "string"
},
"37": {
"type": "string"
},
"39": {
"type": "string"
},
"41": {
"type": "string"
},
"42": {
"type": "string"
},
"48": {
"type": "string"
},
"49": {
"type": "string"
},
"60": {
"type": "string"
},
"63": {
"type": "string"
},
"100": {
"type": "string"
},
"128": {
"type": "string"
}
}
},
"isReversal": {
"type": "boolean"
},
"isReversalDone": {
"type": "boolean"
},
"messageSpec": {
"type": "string"
},
"mti": {
"type": "string"
},
"request": {
"type": "boolean"
},
"response": {
"type": "boolean"
}
}
},
"msg": {
"type": "string"
},
"occurDate": {
"type": "long"
},
"receiver": {
"type": "string"
},
"rrn": {
"type": "string"
},
"sender": {
"type": "string",
"index": "not_analyzed"
},
"txId": {
"type": "string"
},
"version": {
"type": "long"
}
}
}
}
}
}
If I understand correctly, and you want to get only the aggregation result:
Change your size field from:
"size":1000
to
"size":0
in order to set the _search query displayed results limit. It won't affect the aggregation result, though.
I need to query elasticsearch & filter the result to be in a range of dates.
the thing is the date property is mapped as a string.
is it possible to do so ?
this is the search query i'm using:
{
"size": 1,
"from": 0,
"query": {
"bool": {
"must": [
{ "match": { "status": "active" }},
{ "match": { "last_action_state": "accepted" }}
],
"filter": [
{"missing" : { "field" : "store_id" }},
{ "range": { "list_time": { "gte": "2017/01/01 00:00:00", "lte": "2017/03/01 23:59:59", "format": "yyyy/MM/dd HH:mm:ss"}}}
]
}
}
}
the thing is i have no control over the mapping since it's created automatically by another program which index the documents, and i can't change the mapping once it's created.
ps: elasticsearch version: 2.3
UPDATE:
index info:
{
"avindex_v3": {
"aliases": {
"avindex": {}
},
"mappings": {
"ads": {
"properties": {
"account_id": {
"type": "long"
},
"ad_id": {
"type": "long"
},
"ad_params": {
"type": "string"
},
"body": {
"type": "string"
},
"category": {
"type": "long"
},
"city": {
"type": "long"
},
"company_ad": {
"type": "boolean"
},
"email": {
"type": "string"
},
"images": {
"type": "string"
},
"lang": {
"type": "string"
},
"last_action_state": {
"type": "string"
},
"list_date": {
"type": "long"
},
"list_id": {
"type": "long"
},
"list_time": {
"type": "string"
},
"modified_at": {
"type": "string"
},
"modified_ts": {
"type": "double"
},
"name": {
"type": "string"
},
"orig_date": {
"type": "long"
},
"orig_list_time": {
"type": "string"
},
"phone": {
"type": "string"
},
"phone_hidden": {
"type": "boolean"
},
"price": {
"type": "long"
},
"region": {
"type": "long"
},
"status": {
"type": "string"
},
"store_id": {
"type": "long"
},
"subject": {
"type": "string"
},
"type": {
"type": "string"
},
"user_id": {
"type": "long"
}
}
}
},
"settings": {
"index": {
"creation_date": "1493216710928",
"number_of_shards": "5",
"number_of_replicas": "1",
"uuid": "WEHGLF8iRyGk3Xgbmo7H8Q",
"version": {
"created": "2040499"
}
}
},
"warmers": {}
}
}
You can try to give it as a keyword like this :
{
"range": {
"list_time.keyword": {
"gte": "2020-08-12 22:24:55.56",
"lte": "2020-08-12 22:24:56.56"
}
}
}
I have a mapping of date field as :
"created": {
"type" : "date",
"format": "yyyy-MM-dd HH:mm:ss"
},
Now when i sort based on the above field:
"sort":[
{"created":{"order":"asc"}}
],
It takes only time part of the date while sorting and ignores the date part.
{
"_index": "somehting",
"_type": "UserActivity",
"_id": "81574",
"_score": null,
"_source": {
"created": "2016-03-29 00:00:07",
"appCode": "appcode",
"userId": "100008057363993"
},
"sort": [
"00:00:07"
]
},
How do sort based on the whole date?
Please note i cannot use scripting as its disabled on production server and i cannot re-index..
Adding the total mapping:
{
"someIndex": {
"mappings": {
"UserActivity": {
"_timestamp": {
"enabled": true,
"store": true,
"format": "yyyy-MM-dd HH:mm:ss"
},
"properties": {
"_table": {
"type": "string"
},
"_tableat": {
"type": "string"
},
"activity": {
"properties": {
"_table": {
"type": "string"
},
"_tableat": {
"type": "string"
},
"clientId": {
"type": "integer"
},
"code": {
"type": "string"
},
"created": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss"
},
"foreignName": {
"type": "string"
},
"frequency": {
"type": "integer"
},
"id": {
"type": "long"
},
"lastUpdated": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss"
},
"name": {
"type": "string"
},
"points": {
"type": "long"
},
"strategy": {
"type": "string"
}
}
},
"activityId": {
"type": "string"
},
"appCode": {
"type": "string"
},
"clientId": {
"type": "long"
},
"created": {
"type": "string"
},
"details": {
"type": "string"
},
"foreignName": {
"type": "string"
},
"id": {
"type": "long"
},
"lastUpdated": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss"
},
"notes": {
"type": "string"
},
"userId": {
"type": "long"
}
}
}
}
}
}
I have two indexes - one for Application model and the other for Databases model (many-to-many relational).
Each document is denormalized to contain attributes from the other model
Application
|_ vendor_name
|_ databases
|_ db_1
|_ db_2
Database
|_ database_applications
|_ app_1
|_vendor_name
|_ app_2
|_ vendor_name
Executing a multi-index search for a vendor name - it seems I'm getting the proper results from both Indexes.
The challenge is properly aggregating on the vendor_name field
using the following aggregation seems to work when the result is only from Database. I also tried field: '*vendor_name' but doesn't seem to work.
What am I missing? Should the model be changed?
aggregation:
vendor_name: {
terms: {
field: "database_applications.vendor_name"
}
}
UPDATE 1:
As per #Andrie-Stefan - Here's a more accurate representation of both indexes mappings (abbreviated for shortness):
Database
{
"company-company_databases": {
"aliases": {},
"mappings": {
"company_database": {
"properties": {
"company_applications": {
"properties": {
"application_id": {
"type": "long"
},
"application_name": {
"type": "string"
},
"business_owner": {
"type": "string"
},
"company_system_applications": {
"properties": {
"allow_add_request": {
"type": "string"
},
"allow_remove_request": {
"type": "string"
},
"asset_type": {
"type": "string"
},
"company_application_id": {
"type": "long"
},
"company_application_name": {
"type": "string"
},
"company_business_owner": {
"type": "string"
},
"company_division_id": {
"type": "long"
},
"company_it_app_steward": {
"type": "string"
},
"company_notes": {
"type": "string"
},
"company_system_id": {
"type": "long"
},
"company_vendor": {
"type": "string"
},
"id": {
"type": "long"
},
"it_app_steward": {
"type": "string"
},
"it_owner": {
"type": "string"
},
"last_modified": {
"type": "string"
},
"last_modified_by": {
"type": "string"
},
"media_location": {
"type": "string"
},
"media_source": {
"type": "string"
},
"name": {
"type": "string"
},
"owned_by": {
"type": "string"
},
"status": {
"type": "string"
},
"status_id": {
"type": "long"
},
"system_application": {
"properties": {
"division": {
"type": "string"
},
"id": {
"type": "long"
},
"name": {
"type": "string"
},
"owner_id": {
"type": "string"
},
"status": {
"type": "string"
},
"steward_id": {
"type": "string"
},
"vendor_name": {
"type": "string"
},
"vendor_url_web_site": {
"type": "string"
},
"version": {
"type": "string"
}
}
},
"vendor_name": {
"type": "string",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed"
}
}
},
"version": {
"type": "string"
}
}
},
"division_id": {
"type": "long"
},
"it_app_steward": {
"type": "string"
},
"notes": {
"type": "string"
},
"software_inventory_id": {
"type": "long"
},
"vendor": {
"type": "string"
}
}
},
"company_instances": {
"properties": {
"business_environment_id": {
"type": "long"
},
"cgi_service_id": {
"type": "long"
},
"char_set": {
"type": "string"
},
"confirmed_license_purchase_dt": {
"type": "string"
},
"company_server": {
"properties": {
"business_environment_id": {
"type": "long"
},
"division_id": {
"type": "long"
},
"domain": {
"type": "string"
},
"hw_platform_id": {
"type": "long"
},
"ip_address": {
"type": "string"
},
"location_id": {
"type": "long"
},
"no_of_cpu": {
"type": "long"
},
"notes": {
"type": "string"
},
"os_platform_id": {
"type": "long"
},
"os_version": {
"type": "string"
},
"server_id": {
"type": "long"
},
"server_name": {
"type": "string"
}
}
},
"description": {
"type": "string"
},
"division_id": {
"type": "long"
},
"edition_id": {
"type": "long"
},
"instance_id": {
"type": "long"
},
"instance_name": {
"type": "string"
},
"itap_have_access": {
"type": "string"
},
"listener_port": {
"type": "long"
},
"notes": {
"type": "string"
},
"patch_number": {
"type": "string"
},
"rdbms_type_id": {
"type": "long"
},
"server_id": {
"type": "long"
},
"service_level_id": {
"type": "long"
},
"version": {
"type": "string"
}
}
},
"db_security_model_id": {
"type": "long"
},
"schema_or_db": {
"type": "string"
},
"schema_or_db_id": {
"type": "long"
},
"schema_or_db_type_id": {
"type": "long"
}
}
}
},
"settings": {
"index": {
"creation_date": "1442976578465",
"uuid": "TxQZoNSpR5qa2Y2ERZzuYw",
"number_of_replicas": "1",
"number_of_shards": "5",
"version": {
"created": "1070299"
}
}
},
"warmers": {}
}
}
Application
{
"applications": {
"aliases": {},
"mappings": {
"application": {
"properties": {
"application_view": {
"properties": {
"app_name": {
"type": "string"
},
"app_status": {
"type": "string"
},
"app_steward_name": {
"type": "string"
},
"app_suite": {
"type": "string"
},
"app_vendor_name": {
"type": "string"
},
"app_version": {
"type": "string"
},
"assignment_group": {
"type": "string"
},
"business_domain_name": {
"type": "string"
},
"exception": {
"type": "string"
},
"id": {
"type": "long"
},
"it_owner_name": {
"type": "string"
},
"service_level": {
"type": "string"
}
}
},
"assignment_group": {
"type": "string"
},
"company_databases": {
"properties": {
"backup_history_info": {
"type": "string"
},
"company_applications": {
"properties": {
"alternate_name": {
"type": "string"
},
"application_id": {
"type": "long"
},
"application_name": {
"type": "string"
},
"business_owner": {
"type": "string"
},
"company_system_applications": {
"properties": {
"aka": {
"type": "string"
},
"allow_add_request": {
"type": "string"
},
"allow_remove_request": {
"type": "string"
},
"asset_type": {
"type": "string"
},
"contract_number": {
"type": "string"
},
"cost_level": {
"type": "string"
},
"company_alternate_name": {
"type": "string"
},
"company_application_id": {
"type": "long"
},
"company_application_name": {
"type": "string"
},
"company_business_owner": {
"type": "string"
},
"company_division_id": {
"type": "long"
},
"company_it_app_steward": {
"type": "string"
},
"company_notes": {
"type": "string"
},
"company_system_id": {
"type": "long"
},
"company_vendor": {
"type": "string"
},
"description": {
"type": "string"
},
"display_in_catalog": {
"type": "string"
},
"id": {
"type": "long"
},
"inform_of_removal": {
"type": "string"
},
"is_restricted": {
"type": "string"
},
"it_app_steward": {
"type": "string"
},
"it_owner": {
"type": "string"
},
"last_modified": {
"type": "string"
},
"last_modified_by": {
"type": "string"
},
"media_location": {
"type": "string"
},
"media_source": {
"type": "string"
},
"name": {
"type": "string"
},
"os_environment": {
"type": "string"
},
"owned_by": {
"type": "string"
},
"retirement_date": {
"type": "date",
"format": "dateOptionalTime"
},
"status": {
"type": "string"
},
"status_id": {
"type": "long"
},
"suite_name": {
"type": "string"
},
"system_application": {
"properties": {
"assignment_group": {
"type": "string"
},
"division": {
"type": "string"
},
"id": {
"type": "long"
},
"name": {
"type": "string"
},
"owner_id": {
"type": "string"
},
"status": {
"type": "string"
},
"steward_id": {
"type": "string"
},
"suite": {
"type": "string"
},
"vendor_name": {
"type": "string"
},
"vendor_url_web_site": {
"type": "string"
},
"version": {
"type": "string"
}
}
},
"vendor_name": {
"type": "string"
},
"version": {
"type": "string"
}
}
},
"division_id": {
"type": "long"
},
"it_app_steward": {
"type": "string"
},
"notes": {
"type": "string"
},
"software_inventory_id": {
"type": "long"
},
"vendor": {
"type": "string"
}
}
},
"company_instances": {
"properties": {
"business_environment_id": {
"type": "long"
},
"cgi_service_id": {
"type": "long"
},
"char_set": {
"type": "string"
},
"confirmed_license_purchase_dt": {
"type": "string"
},
"company_server": {
"properties": {
"business_environment_id": {
"type": "long"
},
"division_id": {
"type": "long"
},
"domain": {
"type": "string"
},
"hw_platform_id": {
"type": "long"
},
"ip_address": {
"type": "string"
},
"location_id": {
"type": "long"
},
"no_of_cpu": {
"type": "long"
},
"notes": {
"type": "string"
},
"os_platform_id": {
"type": "long"
},
"os_version": {
"type": "string"
},
"server_id": {
"type": "long"
},
"server_name": {
"type": "string"
}
}
},
"description": {
"type": "string"
},
"division_id": {
"type": "long"
},
"edition_id": {
"type": "long"
},
"instance_id": {
"type": "long"
},
"instance_name": {
"type": "string"
},
"itap_have_access": {
"type": "string"
},
"listener_port": {
"type": "long"
},
"location_id": {
"type": "long"
},
"notes": {
"type": "string"
},
"patch_number": {
"type": "string"
},
"rdbms_type_id": {
"type": "long"
},
"server_id": {
"type": "long"
},
"service_level_id": {
"type": "long"
},
"version": {
"type": "string"
}
}
},
"db_security_model_id": {
"type": "long"
},
"notes": {
"type": "string"
},
"schema_or_db": {
"type": "string"
},
"schema_or_db_id": {
"type": "long"
},
"schema_or_db_type_id": {
"type": "long"
}
}
},
"division": {
"type": "string",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed"
}
}
},
"id": {
"type": "long"
},
"name": {
"type": "string"
},
"owner": {
"properties": {
"email_address": {
"type": "string"
},
"id": {
"type": "long"
},
"name": {
"type": "string",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed"
}
}
},
"search_type": {
"type": "string"
},
"user_id": {
"type": "string"
}
}
},
"owner_id": {
"type": "string"
},
"status": {
"type": "string",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed"
}
}
},
"steward": {
"properties": {
"email_address": {
"type": "string"
},
"id": {
"type": "long"
},
"name": {
"type": "string",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed"
}
}
},
"search_type": {
"type": "string"
},
"user_id": {
"type": "string"
}
}
},
"steward_id": {
"type": "string"
},
"suite": {
"type": "string"
},
"vendor_name": {
"type": "string",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed"
}
}
},
"vendor_url_web_site": {
"type": "string"
},
"version": {
"type": "string"
}
}
}
},
"settings": {
"index": {
"creation_date": "1442970067540",
"uuid": "O7DTaCSESbqhjJpv62T0Wg",
"number_of_replicas": "1",
"number_of_shards": "5",
"version": {
"created": "1070299"
}
}
},
"warmers": {}
}
}
Fields can only be aggregated across indices if they are named alike. There is no wildcard syntax for aggregation fields.
Here is what your mapping currently defines:
INDEX: company-company_databases
TYPE: company_database
FIELD NAMES:
company_applications.company_system_applications.vendor_name
company_applications.company_system_applications.system_application.vendor_name
INDEX: applications
TYPE: application
FIELD NAMES:
company_databases.company_applications.company_system_applications.vendor_name
company_databases.company_applications.company_system_applications.system_application.vendor_name
As far as Elasticsearch is concerned, these fields have nothing in common (even though part of the path is vendor_name).
If your goal is to aggregate vendor_name across a query that spans the two indices, think about restructuring your indices/mappings to accomplish this.
Note that Elasticsearch doesn't model many-to-many relationships
If you can get away with duplicating Database info across applications, you might be able to re-formulate your relationships as a hierarchy, e.g.:
INDEX: applications
--
TYPE: application
FIELDS: vendor_name, etc...
--
TYPE: database_application
FIELDS: vendor_name, databases.<inner fields>, etc...
--
Then you'd be able to aggregate across types on the same field path vendor_name with the added bonus of querying a single applications index.