Elasticsearch ingest circle processor problem - elasticsearch

I'm trying to insert a location that is using a circle processor to generate the circle points. Location [30, 10] in the document is generated with the circle points properly.
docs: Circle processor | Elasticsearch Guide [8.5] | Elastic
PUT _ingest/pipeline/polygonize_circles
{
"description": "translate circle to polygon",
"processors": [
{
"circle": {
"field": "circle",
"error_distance": 1,
"shape_type": "geo_shape"
}
}
]
}
PUT circles
{
"mappings": {
"properties": {
"circle": {
"type": "geo_shape"
}
}
}
}
PUT circles/_doc/2?pipeline=polygonize_circles
{
"circle": {
"type": "circle",
"radius": "40m",
"coordinates": [35.539917, -78.472000]
}
}
GET circles/_doc/2
But if I use another location. The generated coordinate looks like an oval with the wrong radius.
my location [35.54171753710938, -78.472]
created coordinates:
"circle": {
"coordinates": [
[
[
35.54171753710938,
-78.472
],
[
35.54112406581135,
-78.47173430472324
],
[
35.540630197847186,
-78.47167003953963
],
[
35.540375564960186,
-78.47165140797998
],
[
35.54021828908823,
-78.47164529506406
],
[
35.54010640465818,
-78.47164274491611
],
[
35.54001650261309,
-78.47164162397662
],
[
35.53993651647515,
-78.47164003979641
],
[
35.539858062238046,
-78.47164049555624
],
[
35.53977439153409,
-78.47164207973975
],
[
35.5396750942597,
-78.47164321572573
],
[
35.5395458932956,
-78.47164846905713
],
[
35.539348254773515,
-78.47165779735127
],
[
35.53899878746994,
-78.47169061817682
],
[
35.53833938849573,
-78.47182842440924
],
[
35.53833938849573,
-78.47217157559075
],
[
35.53899878746994,
-78.47230938182317
],
[
35.539348254773515,
-78.47234220264872
],
[
35.5395458932956,
-78.47235153094286
],
[
35.5396750942597,
-78.47235678427425
],
[
35.53977439153409,
-78.47235792026024
],
[
35.539858062238046,
-78.47235950444374
],
[
35.53993651647515,
-78.47235996020358
],
[
35.54001650261309,
-78.47235837602337
],
[
35.54010640465818,
-78.47235725508388
],
[
35.54021828908823,
-78.47235470493592
],
[
35.540375564960186,
-78.47234859202001
],
[
35.540630197847186,
-78.47232996046036
],
[
35.54112406581135,
-78.47226569527675
],
[
35.54171753710938,
-78.472
]
]
],
"type": "Polygon"
}
coordinates mapping on google maps
Is it an issue or It's working as expected? Because the coordinates are not a circle so it's impacting the search result.

You need to specify your coordinate array using longitude first and then latitude, I think you did the opposite and your circle is in the middle of Antartica.
If you do it like this:
PUT circles/_doc/2?pipeline=polygonize_circles
{
"circle": {
"type": "circle",
"radius": "40m",
"coordinates": [-78.472000, 35.539917]
}
}
Then your circle doesn't look oval anymore:
From the official doc:
In GeoJSON and WKT, and therefore Elasticsearch, the correct coordinate order is longitude, latitude (X, Y) within coordinate arrays. This differs from many Geospatial APIs (e.g., Google Maps) that generally use the colloquial latitude, longitude (Y, X).

Related

Elasticsearch geospatial map, not able to render Linestring

the elasticsearch index contains json as below, only relevant element is show
"geoLocation": {
"coordinates": [ [ -90.66487121582031, 42.49201965332031 ], [ -90.66487884521484, 42.49202346801758 ], [ -90.6648941040039, 42.492034912109375 ], [ -90.66490936279297, 42.49203872680664 ], [ -90.66492462158203, 42.492042541503906 ], [ -90.6649398803711, 42.49204635620117 ], [ -90.66495513916016, 42.49205017089844 ], [ -90.66497039794922, 42.4920539855957 ], [ -90.66498565673828, 42.492061614990234 ], [ -90.66500854492188, 42.492061614990234 ], [ -90.66502380371094, 42.49207305908203 ], [ -90.6650390625, 42.4920654296875 ] ],
"type": "linestring"
},
The template for generating the mapping is as below
PUT _template/template_1?include_type_name=true
{
"index_patterns": ["metromind-its-alerts-day2-*"],
"settings": {
"number_of_shards": 2
},
"mappings": {
"logs": {
"properties": {
"geoLocation": {
"type": "geo_shape"
}
}
}
}
}
the mapping generated is shown below
mapping showing the geoLocation type
When Kibana Maps are used it detects the geo_shape
Kibana Map to render Linestring
Note However no Linestring is rendered, please suggest the resolution
The line string is there, in Dubuque IL, it's just that it's extra small at the scale of the earth.
Just click on the following icon and Elastic Map will focus on it and you'll see it:

How to save geo data in elasticsearch

How do I index a document with below data in elasticsearch(geo datatype)?
<west>5.8663152683722</west>
<north>55.0583836008072</north>
<east>15.0418156516163</east>
<south>47.2701236047002</south>
I tried geo_point and its working for lon and lat's, not sure how to save this data. any help is highly appreciated.
You'll have to use the geo_shape datatype and convert your XML (I assume) semi-points into a line string or polygon before sync.
I'm gonna go with a polygon here.
Let's visualise the conventional cardinal directions:
North (+90)
|
(-180) West ——+—— East (+180)
|
South (-90)
geo_shape expects GeoJSON-like inputs so you'll need five coordinate points, the first and last of which are identical (according to the GeoJSON spec).
Therefore, borrowing from TurfJS and going from bottom left counter-clockwise,
const lowLeft = [west, south];
const topLeft = [west, north];
const topRight = [east, north];
const lowRight = [east, south];
return
[
[
lowLeft,
lowRight,
topRight,
topLeft,
lowLeft
]
]
Finally, let's create our index and plug your numbers in
PUT /example
{
"mappings": {
"properties": {
"location": {
"type": "geo_shape"
}
}
}
}
POST /example/_doc
{
"location":{
"type":"polygon",
"coordinates":[
[
[
5.8663152683722,
47.2701236047002
],
[
15.0418156516163,
47.2701236047002
],
[
15.0418156516163,
55.0583836008072
],
[
5.8663152683722,
55.0583836008072
],
[
5.8663152683722,
47.2701236047002
]
]
]
}
}
Then verify that the center of your square if indeed inside of your indexed polygon:
GET example/_search
{
"query": {
"geo_shape": {
"location": {
"shape": {
"type": "point",
"coordinates": [
10.45406545999425,
51.1642536027537
]
},
"relation": "intersects"
}
}
}
}

what is the json mapping to insert geo data into elasticsearch?

what would be the json mapping to insert geo data into elasticsearch ??
if the sample json data as follows:
{ "type": "Feature", "properties": { "ID": 631861455.000000, "address": "1206 UPPER", "city": "la vegas", "state": "AL", "zip_code": "15656", "OGR_GEOMETRY": "POLYGON" }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -86.477551331, 32.490605650000099 ], [ -86.477637350999899, 32.4903921820001 ], [ -86.478257247, 32.490565591000099 ], [ -86.478250466, 32.490580239000103 ], [ -86.478243988, 32.490593680000096 ], [ -86.47823751, 32.490607122 ], [ -86.478231749, 32.490619100000096 ], [ -86.478224637, 32.490634065000101 ], [ -86.47821823699999, 32.490647540000097 ], [ -86.478211847999901, 32.490661035000095 ], [ -86.478205478999897, 32.490674526000099 ], [ -86.478202107999891, 32.490681666000093 ], [ -86.478199132, 32.4906880240001 ], [ -86.478192825999898, 32.490701523 ], [ -86.478186533, 32.490715047 ], [ -86.47818320899999, 32.490722209000097 ], [ -86.47818027999989, 32.490728569000098 ], [ -86.478174063, 32.490742125000097 ], [ -86.47816785099999, 32.490755654000097 ], [ -86.47816255799999, 32.490767236000096 ], [ -86.478159053999889, 32.490774513000105 ], [ -86.477551331, 32.490605650000099 ] ] ] } }
Look at Geo point mapping.
You need to define mapping.

How to insert large polar polygons in elasticsearch?

When inserting a large polygon near the south pole:
"polygon":{
"type":"polygon",
"coordinates":[
[
[
-134.97410583496094,
-61.81480026245117
],
[
-130.1757049560547,
-63.236000061035156
],
[
-125.17160034179688,
-64.40799713134766
],
[
-152.0446014404297,
-75.72830200195312
],
[
143.52340698242188,
-77.68319702148438
],
[
147.41830444335938,
-75.44519805908203
],
[
150.2816925048828,
-73.01909637451172
],
[
-162.17909240722656,
-71.5260009765625
],
[
-134.97410583496094,
-61.81480026245117
]
]
]
},
, the following error is returned.
{
"error" : "RemoteTransportException[[ISAAC][inet[/x.x.x.x:9300]][indices:data/write/index]]; nested: MapperParsingException[failed to parse [polygon]]; nested: InvalidShapeException[Self- intersection at or near point (-142.29442281263474, -71.62101996804898, NaN)]; ",
"status" : 400
}
The mapping of the type is:
curl -XPUT http://localhost:9200/files/_mapping/polar -d '
{
"polar" : {
"properties" : {
"startTimeRange" : { "type" : "date"},
"endTimeRange" : { "type" : "date"},
"productShortName" : {
"type": "string",
"index" : "not_analyzed"
},
"polygon" : {
"type" : "geo_shape",
"tree" : "quadtree",
"precision" : "1000m"
}
}
}
}
'
The intended shape is essential a rectangle crossing the dateline (anti-meridian).
It looks like the shape is being interpreted as a self-intersecting
polygon crossing the meridian (0 - Longitude).
What is the best way to represent the intended shape in elasticsearch?
Dateline and Pole Crossing is a known issue. Dateline crossing was fixed in ES 1.4.3 but the pole crossing patch will be released in a future version. For now (and this can be a serious PITA for ambiguous polygon's) you'll have to unwrap the Polygon into a MultiPolygon yourself (presumably at the application layer).
Here's an example using your data.
The original self-crossing poly (as seen at the following gist: https://gist.github.com/nknize/ea0e103a22bddae13dfb)
{
"type" : "Polygon",
"coordinates":[[
[
-134.97410583496094,
-61.81480026245117
],
[
-130.1757049560547,
-63.236000061035156
],
[
-125.17160034179688,
-64.40799713134766
],
[
-152.0446014404297,
-75.72830200195312
],
[
143.52340698242188,
-77.68319702148438
],
[
147.41830444335938,
-75.44519805908203
],
[
150.2816925048828,
-73.01909637451172
],
[
-162.17909240722656,
-71.5260009765625
],
[
-134.97410583496094,
-61.81480026245117
]
]]
}
Corrected version using polar "unwrapping" (seen at the following gist: https://gist.github.com/nknize/8e87ee88d3915498507e)
{
"type" : "MultiPolygon",
"coordinates":[[[
[
-180.0,
-72.092693931
],
[
-162.17909240722656,
-71.5260009765625
],
[
-134.97410583496094,
-61.81480026245117
],
[
-130.1757049560547,
-63.236000061035156
],
[
-125.17160034179688,
-64.40799713134766
],
[
-152.0446014404297,
-75.72830200195312
],
[ -173.3707512019,
-90.0
],
[
-180.0,
-90.0
],
[
-180.0,
-72.092693931
]
]],
[[
[
173.3707512019,
-90.0
],
[
143.52340698242188,
-77.68319702148438
],
[
147.41830444335938,
-75.44519805908203
],
[
150.2816925048828,
-73.01909637451172
],
[
180.0,
-72.092693931
],
[
180.0,
-90.0
],
[
173.3707512019,
-90.0
]
]]
]
}
Note that the above "corrected" MultiPolygon is a rough calculation (with floating point error) used just for this example.
Not an ideal answer, but I hope it helps!

Using d3.nest() with geojson files

How is d3.nest() used with geojson files?
My geojson data is formatted as follows:
"features": [
{ "type": "Feature", "properties": { "neighborhood": "Allerton", "boroughCode": "2", "borough": "Bronx", "#id": "http:\/\/nyc.pediacities.com\/Resource\/Neighborhood\/Allerton" }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -73.848597000000183, 40.871670000000115 ], [ -73.845822536836778, 40.870239076236174 ], [ -73.854559184633743, 40.859953835764252 ], [ -73.854665433068263, 40.859585694988056 ], [ -73.856388703358959, 40.857593635304482 ], [ -73.868881809153407, 40.857223150158326 ], [ -73.868317552728243, 40.857862062258313 ], [ -73.869553714672321, 40.857784095600181 ], [ -73.871024857620654, 40.857309948816905 ], [ -73.870480549987164, 40.865413584098484 ], [ -73.87055489856489, 40.869702798589863 ], [ -73.86721594442561, 40.869689663636713 ], [ -73.85745, 40.869533000000182 ], [ -73.855550000000108, 40.871813000000145 ], [ -73.853597967576576, 40.873288368674203 ], [ -73.848597000000183, 40.871670000000115 ] ] ] } }
But my nest command:
var nested_data = d3.nest()
.key(function(d, i) { console.log(d); return d.features.properties.neighborhood; })
.entries(map);
returns an empty array.
I want to nest my data to more easily filter it. Is this advised?
Assuming your geojson looks like the below
var map = {
type: "FeatureCollection",
"features": [
{
"type": "Feature",
"properties": {
"neighborhood": "Allerton",
"boroughCode": "2",
"borough": "Bronx",
"#id": "http:\/\/nyc.pediacities.com\/Resource\/Neighborhood\/Allerton"
},
"geometry": { /* various coordinates, etc */ }
]
}
So, what you want to do is:
d3.nest()
.key(function(d, i) {
return d.properties.neighborhood;
})
.entries(map.features);
You want to pass map.features since that's your array.

Resources