elasticsearch: unable to set geo_shape value using XContentBuilder - elasticsearch

I have following mapping in elastic search. I am able to PUT documents using Sense plugin but unable to do so using XContentBuilder to set the geo_shape field value. I am getting following error:
error:
[106]: index [streets], type [street], id [{dc872755-f307-4c5e-93f6-bba9c95791c7}], message [MapperParsingException[failed to parse [shape]]; nested: ElasticsearchParseException[shape must be an object consisting of type and coordinates];]
mapping:
PUT /streets
{
"mappings": {
"street": {
"properties": {
"id": {
"type": "string"
},
"shape": {
"type": "geo_shape",
"tree": "quadtree"
}
}
}
}
}
code:
val bulkRequest:BulkRequestBuilder = esClient.prepareBulk()
//inloop
xb = jsonBuilder().startObject()
xb.field("id", guid)
xb.field("shape", jsonString) // removing this line creates the index OK but without the geo_shape
xb.endObject()
bulkRequest.add(esClient.prepareIndex("streets", "street", guid).setSource(xb))
//end loop
val bulkResponse:BulkResponse = bulkRequest.execute().actionGet()
if(bulkResponse.hasFailures){
println(bulkResponse.buildFailureMessage())
}
jsonString:
{
"id": "{98b8fd8d-074c-4349-a83b-6e892bf2d0ef}",
"shape": {
"type": "LineString",
"coordinates": [
[-70.81866815832467, 43.12187109162505],
[-70.83054813653018, 43.15917412985851],
[-70.81320737213957, 43.23522269547419],
[-70.90108590067649, 43.28102004268419]
],
"crs": {
"type": "name",
"properties": {
"name": "EPSG:4326"
}
}
}
}
Appreciate any feedback?
Thanks

It might be a bit late for you, but this could help someone facing a similar issue even nowadays.
Following your index mapping for the document streets, we have these properties: id and shape.
In your error message, it's described that:
shape must be an object consisting of type and coordinates
So for your concrete case, the crs array is just not accepted (don't know exactly why you can't add extra parameters).
This is an example for how to add a document into the streets index using CURL:
curl -X POST "localhost:9200/streets/_doc?pretty" -H 'Content-Type: application/json' -d '
{
"id": 123,
"shape": {
"type": "Polygon",
"coordinates": [
[
[
32.85444259643555,
39.928694653732364
],
[
32.847232818603516,
39.9257985682691
],
[
32.837791442871094,
39.91947941109337
],
[
32.837276458740234,
39.91579296675271
],
[
32.85392761230469,
39.913423004886894
],
[
32.86937713623047,
39.91329133793421
],
[
32.88036346435547,
39.91539797880347
],
[
32.85444259643555,
39.928694653732364
]
]
]
}
}'
If you need to add a LineString, instead of a Polygon, just change the 'type' attribute from the 'shape'.
I hope this helps people having to add documents with shapes into an ElasticSearch database.

Related

Elasticsearch geospatial map, not able to render Linestring

the elasticsearch index contains json as below, only relevant element is show
"geoLocation": {
"coordinates": [ [ -90.66487121582031, 42.49201965332031 ], [ -90.66487884521484, 42.49202346801758 ], [ -90.6648941040039, 42.492034912109375 ], [ -90.66490936279297, 42.49203872680664 ], [ -90.66492462158203, 42.492042541503906 ], [ -90.6649398803711, 42.49204635620117 ], [ -90.66495513916016, 42.49205017089844 ], [ -90.66497039794922, 42.4920539855957 ], [ -90.66498565673828, 42.492061614990234 ], [ -90.66500854492188, 42.492061614990234 ], [ -90.66502380371094, 42.49207305908203 ], [ -90.6650390625, 42.4920654296875 ] ],
"type": "linestring"
},
The template for generating the mapping is as below
PUT _template/template_1?include_type_name=true
{
"index_patterns": ["metromind-its-alerts-day2-*"],
"settings": {
"number_of_shards": 2
},
"mappings": {
"logs": {
"properties": {
"geoLocation": {
"type": "geo_shape"
}
}
}
}
}
the mapping generated is shown below
mapping showing the geoLocation type
When Kibana Maps are used it detects the geo_shape
Kibana Map to render Linestring
Note However no Linestring is rendered, please suggest the resolution
The line string is there, in Dubuque IL, it's just that it's extra small at the scale of the earth.
Just click on the following icon and Elastic Map will focus on it and you'll see it:

graphQL filter array containing ALL

I am quite new to graphQL, and after searching the whole afternoon, i didn't found my answer to a relative quite simple problem.
I have two objects in my strapi backend :
"travels": [
{
"id": "1",
"title": "Bolivia: La Paz y Salar de Uyuni",
"travel_types": [
{
"name": "Culturales"
},
{
"name": "Aventura"
},
{
"name": "Ecoturismo"
}
]
},
{
"id": "2",
"title": "Europa clásica 2020",
"travel_types": [
{
"name": "Clasicas"
},
{
"name": "Culturales"
}
]
}
]
I am trying to get a filter where I search for travels containing ALL the user-selected travel_types.
I then wrote a query like that :
query($where: JSON){
travels (where:$where) {
id # Or _id if you are using MongoDB
title
travel_types {name}
}
And the parameter i try to input for testing :
{
"where":{
"travel_types.name_contains": ["Aventura"],
"travel_types.name_contains": ["Clasicas"]
}
}
This should return an empty array, because none of the travels have both Aventura and Clasicas travel-types.
But instead it returns the travel with id=2. It seems that only the second filter is taken.
I searched for a query which would be like Array.every() in javascript, but i wasn't able to find.
Does someone has an idea how to achieve this type of filtering ?
Thank you very much,

How to save geo data in elasticsearch

How do I index a document with below data in elasticsearch(geo datatype)?
<west>5.8663152683722</west>
<north>55.0583836008072</north>
<east>15.0418156516163</east>
<south>47.2701236047002</south>
I tried geo_point and its working for lon and lat's, not sure how to save this data. any help is highly appreciated.
You'll have to use the geo_shape datatype and convert your XML (I assume) semi-points into a line string or polygon before sync.
I'm gonna go with a polygon here.
Let's visualise the conventional cardinal directions:
North (+90)
|
(-180) West ——+—— East (+180)
|
South (-90)
geo_shape expects GeoJSON-like inputs so you'll need five coordinate points, the first and last of which are identical (according to the GeoJSON spec).
Therefore, borrowing from TurfJS and going from bottom left counter-clockwise,
const lowLeft = [west, south];
const topLeft = [west, north];
const topRight = [east, north];
const lowRight = [east, south];
return
[
[
lowLeft,
lowRight,
topRight,
topLeft,
lowLeft
]
]
Finally, let's create our index and plug your numbers in
PUT /example
{
"mappings": {
"properties": {
"location": {
"type": "geo_shape"
}
}
}
}
POST /example/_doc
{
"location":{
"type":"polygon",
"coordinates":[
[
[
5.8663152683722,
47.2701236047002
],
[
15.0418156516163,
47.2701236047002
],
[
15.0418156516163,
55.0583836008072
],
[
5.8663152683722,
55.0583836008072
],
[
5.8663152683722,
47.2701236047002
]
]
]
}
}
Then verify that the center of your square if indeed inside of your indexed polygon:
GET example/_search
{
"query": {
"geo_shape": {
"location": {
"shape": {
"type": "point",
"coordinates": [
10.45406545999425,
51.1642536027537
]
},
"relation": "intersects"
}
}
}
}

Query on number field matching string fields

I have an ELK stack setup. When I am performing a query on number fields then it is also matching against string fields. For example, I am sending Load Balancer logs to ELK and if I perform backend_processing_time:>5 on that then it is matching against backend_processing_time with value 0.001 too.
On kibana interface, it is showing that the query is matching string in the request message. I am not able to understand how a query against a number field is matching against a string.
In the dev tools section on kibana i tried to run the same query
GET _search
{
"query": {
"range" : {
"backend_processing_time" : {
"gte" : 50000000000
}
}
}
}
Even with so much backend_processing_time i am getting results. I am not able to understand why this is happening.
I searched on other fields also which are of number type and found that all the queries done on number field are getting matched with string type fields.
I am providing a sample search result which i get for backend_processing_time:>500000000 query. It can be seen in this result that backend_processing_time field is so small but still getting a hit.
{
"_index": "logstash-2017.05.10",
"_type": "prod-quizelb-logs",
"_id": "AVvzYRgL49GPTZAKoDer",
"_score": null,
"_source": {
"backendport": 80,
"received_bytes": 0,
"request": "http://en.meaww.com:80/locales/en.json",
"backend_response": 200,
"verb": "GET",
"message": "2017-05-10T17:19:52.881044Z Prod-ELB 172.68.144.71:34803 10.1.91.253:80 0.000075 0.000606 0.000019 200 200 0 1881 \"GET http://en.meaww.com:80/locales/en.json HTTP/1.1\" \"Mozilla/5.0 (Linux; Android 6.0.1; SM-C900F Build/MMB29M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/58.0.3029.83 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/122.0.0.17.71;]\" - -\n",
"type": "prod-quizelb-logs",
"clientport": 34803,
"request_processing_time": 0.000075,
"urihost": "en.meaww.com:80",
"response_processing_time": 0.000019,
"path": "/locales/en.json",
"#timestamp": "2017-05-10T17:21:18.280Z",
"port": "80",
"response": 200,
"bytes": 1881,
"clientip": "172.68.144.71",
"proto": "http",
"#version": "1",
"elb": "Prod-ELB",
"httpversion": "1.1",
"backendip": "10.1.91.253",
"backend_processing_time": 0.000606,
"timestamp": "2017-05-10T17:19:52.881044Z"
},
"fields": {
"#timestamp": [
1494436878280
],
"timestamp": [
1494436792881
]
},
"highlight": {
"backend_processing_time.keyword": [
"#kibana-highlighted-field#6.06E-4#/kibana-highlighted-field#"
],
"request": [
"#kibana-highlighted-field#http#/kibana-highlighted-field#://#kibana-highlighted-field#en.meaww.com#/kibana-highlighted-field#:#kibana-highlighted-field#80#/kibana-highlighted-field#/#kibana-highlighted-field#locales#/kibana-highlighted-field#/#kibana-highlighted-field#en.json#/kibana-highlighted-field#"
],
"elb.keyword": [
"#kibana-highlighted-field#Prod-ELB#/kibana-highlighted-field#"
],
"urihost.keyword": [
"#kibana-highlighted-field#en.meaww.com:80#/kibana-highlighted-field#"
],
"verb": [
"#kibana-highlighted-field#GET#/kibana-highlighted-field#"
],
"request.keyword": [
"#kibana-highlighted-field#http://en.meaww.com:80/locales/en.json#/kibana-highlighted-field#"
],
"type": [
"#kibana-highlighted-field#prod#/kibana-highlighted-field#-#kibana-highlighted-field#quizelb#/kibana-highlighted-field#-#kibana-highlighted-field#logs#/kibana-highlighted-field#"
],
"message": [
"2017-05-10T17:19:#kibana-highlighted-field#52.881044Z#/kibana-highlighted-field# #kibana-highlighted-field#Prod#/kibana-highlighted-field#-#kibana-highlighted-field#ELB#/kibana-highlighted-field# 172.68.144.71:34803 10.1.91.253:#kibana-highlighted-field#80#/kibana-highlighted-field# 0.000075 0.000606 0.000019 200 200 0 1881 \"#kibana-highlighted-field#GET#/kibana-highlighted-field# #kibana-highlighted-field#http#/kibana-highlighted-field#://#kibana-highlighted-field#en.meaww.com#/kibana-highlighted-field#:#kibana-highlighted-field#80#/kibana-highlighted-field#/#kibana-highlighted-field#locales#/kibana-highlighted-field#/#kibana-highlighted-field#en.json#/kibana-highlighted-field# #kibana-highlighted-field#HTTP#/kibana-highlighted-field#/1.1\" \"#kibana-highlighted-field#Mozilla#/kibana-highlighted-field#/5.0 (#kibana-highlighted-field#Linux#/kibana-highlighted-field#; #kibana-highlighted-field#Android#/kibana-highlighted-field# #kibana-highlighted-field#6.0.1#/kibana-highlighted-field#; #kibana-highlighted-field#SM#/kibana-highlighted-field#-#kibana-highlighted-field#C900F#/kibana-highlighted-field# #kibana-highlighted-field#Build#/kibana-highlighted-field#/#kibana-highlighted-field#MMB29M#/kibana-highlighted-field#; #kibana-highlighted-field#wv#/kibana-highlighted-field#) #kibana-highlighted-field#AppleWebKit#/kibana-highlighted-field#/#kibana-highlighted-field#537.36#/kibana-highlighted-field# (#kibana-highlighted-field#KHTML#/kibana-highlighted-field#, #kibana-highlighted-field#like#/kibana-highlighted-field# #kibana-highlighted-field#Gecko#/kibana-highlighted-field#) #kibana-highlighted-field#Version#/kibana-highlighted-field#/4.0 #kibana-highlighted-field#Chrome#/kibana-highlighted-field#/#kibana-highlighted-field#58.0.3029.83#/kibana-highlighted-field# #kibana-highlighted-field#Mobile#/kibana-highlighted-field# #kibana-highlighted-field#Safari#/kibana-highlighted-field#/#kibana-highlighted-field#537.36#/kibana-highlighted-field# [#kibana-highlighted-field#FB_IAB#/kibana-highlighted-field#/#kibana-highlighted-field#FB4A#/kibana-highlighted-field#;#kibana-highlighted-field#FBAV#/kibana-highlighted-field#/122.0.0.17.71;]\" - -\n"
],
"urihost": [
"#kibana-highlighted-field#en.meaww.com#/kibana-highlighted-field#:#kibana-highlighted-field#80#/kibana-highlighted-field#"
],
"path": [
"/#kibana-highlighted-field#locales#/kibana-highlighted-field#/#kibana-highlighted-field#en.json#/kibana-highlighted-field#"
],
"verb.keyword": [
"#kibana-highlighted-field#GET#/kibana-highlighted-field#"
],
"proto.keyword": [
"#kibana-highlighted-field#http#/kibana-highlighted-field#"
],
"port": [
"#kibana-highlighted-field#80#/kibana-highlighted-field#"
],
"type.keyword": [
"#kibana-highlighted-field#prod-quizelb-logs#/kibana-highlighted-field#"
],
"proto": [
"#kibana-highlighted-field#http#/kibana-highlighted-field#"
],
"elb": [
"#kibana-highlighted-field#Prod#/kibana-highlighted-field#-#kibana-highlighted-field#ELB#/kibana-highlighted-field#"
],
"backend_processing_time": [
"#kibana-highlighted-field#6.06E#/kibana-highlighted-field#-4"
],
"port.keyword": [
"#kibana-highlighted-field#80#/kibana-highlighted-field#"
]
},
"sort": [
1494436878280
]
}
EDIT
I got the mapping by running GET /logstash-2017.05.11/_mapping/prod-quizelb-logs query in kibana console.
The mapping which I am getting for backend_processing_time is showing this
"backend_processing_time": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword"
}
}
}
So it seems that this field is of text type thus causing this error to happen.
Now I have another confusion i.e. kibana is showing this as number but elasticsearch is showing this of type text. Also, this is getting mapped dynamically as i never created the mapping on my own. I think that they are getting created by logstash at the time grok filter is applied.
You need to take control of the mapping of those index(indices) so that your field will actually be a number. Otherwise, you will not be sure what kind of field type you'll have there. So, basically you need something like this, either in an index template, or a static mapping all the way:
"backend_processing_time": {
"type": "integer"
}
Remove space in your query_string. i.e Your query_string should look like this:
backend_processing_time:>0.5
Read more about query_string syntax here

How to index geojson file in elasticsearch?

I am trying to store spatial data in the form of geojson,csv files and shape files into elasticsearch USING PYTHON.I am new to elasticsearch and even after following the documentation i am not able to successfully index it. Any help would be appreciated.
sample geojson file :
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"properties": {
"ID_0": 105,
"ISO": "IND",
"NAME_0": "India",
"ID_1": 1288,
"NAME_1": "Telangana",
"ID_2": 15715,
"NAME_2": "Telangana",
"VARNAME_2": null,
"NL_NAME_2": null,
"HASC_2": "IN.TS.AD",
"CC_2": null,
"TYPE_2": "State",
"ENGTYPE_2": "State",
"VALIDFR_2": "Unknown",
"VALIDTO_2": "Present",
"REMARKS_2": null,
"Shape_Leng": 8.103535,
"Shape_Area": 127258717496
},
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
79.14429367552918,
19.500257885106404
],
[
79.14582245808431,
19.498859172536427
],
[
79.14600496956801,
19.498823981691853
],
[
79.14966523737327,
19.495821705263914
]
]
]
}
}
]
}
Code
import geojson
from datetime import datetime
from elasticsearch import Elasticsearch, helpers
def geojson_to_es(gj):
for feature in gj['features']:
date = datetime.strptime("-".join(feature["properties"]["event_date"].split('-')[0:2]) + "-" + feature["properties"]["year"], "%d-%b-%Y")
feature["properties"]["timestamp"] = int(date.timestamp())
feature["properties"]["event_date"] = date.strftime('%Y-%m-%d')
yield feature
with open("GeoObs.json") as f:
gj = geojson.load(f)
es = Elasticsearch(hosts=[{'host': 'localhost', 'port': 9200}])
k = ({
"_index": "YOUR_INDEX",
"_source": feature,
} for feature in geojson_to_es(gj))
helpers.bulk(es, k)
Explanation
with open("GeoObs.json") as f:
gj = geojson.load(f)
es = Elasticsearch(hosts=[{'host': 'localhost', 'port': 9200}])
This portion of the code loads an external geojson file, then connects to Elasticsearch.
k = ({
"_index": "conflict-data",
"_source": feature,
} for feature in geojson_to_es(gj))
helpers.bulk(es, k)
The ()s here creates a generator which we will feed to helpers.bulk(es, k). Remember _source is the original data as is in Elasticsearch speak - IE: our raw JSON. _index is just the index in which we want to put our data. You'll see other examples with _doc here. This is part of the mapping types and no longer exists in Elasticsearch 7.X+.
def geojson_to_es(gj):
for feature in gj['features']:
date = datetime.strptime("-".join(feature["properties"]["event_date"].split('-')[0:2]) + "-" + feature["properties"]["year"], "%d-%b-%Y")
feature["properties"]["timestamp"] = int(date.timestamp())
feature["properties"]["event_date"] = date.strftime('%Y-%m-%d')
yield feature
The function geojson uses a generator to produce events. A generator function will, instead of returning and finishingresume at the keywordyield` after each call. In this case, we are generating our GeoJSON features. In my code you also see:
date = datetime.strptime("-".join(feature["properties"]["event_date"].split('-')[0:2]) + "-" + feature["properties"]["year"], "%d-%b-%Y")
feature["properties"]["timestamp"] = int(date.timestamp())
feature["properties"]["event_date"] = date.strftime('%Y-%m-%d')
This is just an example of manipulating the data in the JSON before sending it out to Elasticsearch.
The key is in your mapping file you must have something tagged as geo_point or geo_shape. These data types are how Elasticsearch recognizes geo data. Example from my mapping file:
...
{
"properties": {
"geometry": {
"properties": {
"coordinates": {
"type": "geo_point"
},
"type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
...
That is to say, before uploading your GeoJSON data with Python, you need to create your index, and then apply a mapping file which includes either geo_shape or geo_point using something like:
curl -X PUT "localhost:9200/YOUR_INDEX?pretty"
curl -X PUT localhost:9200/YOUR_INDEX/_mapping?pretty -H "Content-Type: application/json" -d #mapping.json
You must separate the GeoJson features into (1) geometry and (2) properties/attributes parts. You cannot index GeoJson features and feature collections directly (see documentation), only the geometry part is supported as a field type.
So you final indexable document would look somewhat flattened:
{
"ID_0": 105,
"ISO": "IND",
"NAME_0": "India",
"ID_1": 1288,
"NAME_1": "Telangana",
"ID_2": 15715,
"NAME_2": "Telangana",
"VARNAME_2": null,
"NL_NAME_2": null,
"HASC_2": "IN.TS.AD",
"CC_2": null,
"TYPE_2": "State",
"ENGTYPE_2": "State",
"VALIDFR_2": "Unknown",
"VALIDTO_2": "Present",
"REMARKS_2": null,
"Shape_Leng": 8.103535,
"Shape_Area": 127258717496,
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
79.14429367552918,
19.500257885106404
],
[
79.14582245808431,
19.498859172536427
],
[
79.14600496956801,
19.498823981691853
],
[
79.14966523737327,
19.495821705263914
]
]
]
}
}

Resources