I'm mapping congressional districts in elasticsearch but my efforts are returning no results when trying to query by geoshape even though the query is clearly within the boundary of my WKT values. Here's some relevant data:
The mapping:
"mappings":{
"house":{
"properties":{
"state_name":{
"type":"text"
},
"district":{
"type":"integer"
},
"shape":{
"type":"geo_shape",
"strategy":"recursive"
}
}
}
An example of a WKT on the shape property: POLYGON((-122.612285 37.815224,-122.501272 37.821212,-122.499764 37.819724,-122.41871 37.852491,-122.418673 37.852505,-122.430183 37.918425,-122.432283 37.929824,-122.373982 37.883884,-122.373782 37.883725,-122.28246 37.709309,-122.28178 37.70823,-122.419802 37.708231,-122.420082 37.708231,-122.440893 37.716405,-122.440999 37.716488,-122.428203 37.731851,-122.428038 37.732016,-122.451147 37.731567,-122.453411 37.731568,-122.463399 37.752981,-122.463433 37.753028,-122.469667 37.738505,-122.470597 37.737665,-122.512732 37.735087,-122.578294 37.733988,-122.584728 37.779156,-122.588174 37.789362,-122.612285 37.815224))
My search query in kibana that is clearly inside that WKT:
GET /house/_search
{
"query":{
"bool": {
"must": {
"match_all": {}
},
"filter": {
"geo_shape": {
"shape": {
"shape": {
"type": "circle",
"coordinates" : [-122.421151, 37.758442],
"radius": "100m"
}
}
}
}
}
}
I've also tried geoshape type point with intersects and contains relation (which is not currently supported).
What else can I do to debug why these documents are not being found?
I realized that my mapping was incorrect. I was manually setting a geo_shape mapping on shape but when my indexes were created they were creating Shape (note capitalization change) properties with type text. Thus none of my indexes had any geo_shape data to search.
The most useful thing I used to debug and find this issue was a readout of the GET /house/mapping data which showed the two distinct properties that should have been a single property.
Related
I have created the simplest index in ES v7.10:
it maps mylocation field into geo_point (not geo_shape):
PUT /myindex
{
"mappings": {
"dynamic": "false",
"properties": {
"mylocation": {
"type": "geo_point"
}
}
}
}
then I pushed some data inside. I've omitted this to shorten the question...
When I query the below, all works fine:
{
"query": {
"bool": {
"must": {
"match_all": {}
},
"filter": {
"geo_shape": {
"mylocation": {
"shape": {
"type": "polygon",
"coordinates": [[ [ 13.0, 53.0 ], [ 0.0, 1.0 ], [ 0.0, 0.0 ], [ 13.0, 53.0 ] ]]
},
"relation": "intersects"
}
}
}
}
}
}
when I replace the intersects with disjoint, I get the error:
DISJOINT query relation not supported for Field [mylocation].
In elastic docs (for the relevant version, 7), it is mentioned that "Geoshape query filter documents indexed using the geo_shape or geo_point type" (and I am using the geo_point). Down the same page, it is written the "disjoint" is supported!
What am I missing? Why do I get the error?
No idea how I have missed it, but in v7.10 docs (link in the question) they write in the next paragraph:
When searching a field of type geo_point there is a single supported
spatial relation operator:
INTERSECTS - (default) Return all documents whose geo_point field
intersects the query geometry.
In v7.17 they do not have this limitation anymore...
Could you please help me on this? My Kibana Database within "Discover" contains a list of trades. I know want to find all trades within this DB that have been done in specific instruments (ISIN-Number). When I add a filter manually and switch to Elasticserach Query DSL, I find the following:
{
"query": {
"bool": {
"should": [
{
"match_phrase": {
"obdetails.isin": "CH0253592783"
}
},
{
"match_phrase": {
"obdetails.isin": "CH0315622966"
}
},
{
"match_phrase": {
"obdetails.isin": "CH0357659488"
}
}
],
"minimum_should_match": 1
}
}
}
Since I want to check the DB for more than 200 ISINS, this seems to be inefficient. Is there a way, in which I could just say "show me the trade if it contains one of the following 200 ISINs?".
I already googled and tried this, which did not work:
{
"query": {
"terms": {
"obdetails.isin": [ "CH0357659488", "CH0315622966"],
"boost": 1.0
}
}
}
The query works, but does not show any results.
To conclude. A field of type text is analyzed which basically converts the given data to a list of terms using given analyzers etc. rather than it being a single term.
Given behavior causes the terms query to not match these values.
Rather than changing the type of the field one may add an additional field of type keyword. That way a terms queries can be performed whilst still having the ability to match on the field.
{
"isin": {
"type" "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
The above example will add an extra field called obdetails.isin.keyword which can be used for terms. While still being able to use match queries on obdetails.isin
I would like to perform a spatial join as you would do in PostGIS. So far I have explored the various ways outlined with Geo-shape query, but they only were possible by using a single shape or by accessing a shape in a different index by id like here:
GET /gdelt/_search
{
"query": {
"bool": {
"filter": {
"geo_shape": {
"location": {
"indexed_shape": {
"index": "ne_countries",
"id": "0bnrXnMB0RPtPof-zD-F",
"path": "geometry"
}
}
}
}
}
}
}
Is there a way to spatially join an index with geo_point documents with another or the same index with geo_shape documents?
I'm cross-posting this from the elasticsearch forums (https://discuss.elastic.co/t/multiple-paths-in-nested-query/96851/1)
Below is an example, but first I’ll tell you about my use case, because I’m not sure if this is a good approach. I’m trying to automatically index a large collection of typed data. What this means is I’m trying to generate mappings and queries on those mappings all automatically based on information about my data. A lot of my data is relational, and I’m interested in being able to search accross the relations, thus I’m also interested in using Nested data types.
However, the issue is that many of these types have on the order of 10 relations, and I’ve got a feeling its not a good idea to pass 10 identical copies of a nested query to elasticsearch just to query 10 different nested paths the same way. Thus, I’m wondering if its possible to instead pass multiple paths into a single query? Better yet, if its possible to search over all fields in the current document and in all its nested documents and their fields in a single query. I’m aware of object fields, and they’re not a good fit because I want to retrive some data of matched nested documents.
In this example, I create an index with multiple nested types and some of its own types, upload a document, and attempt to query the document and all its nested documents, but fail. Is there some way to do this without duplicating the query for each nested document, or is that actually a performant way to do this? Thanks
PUT /my_index
{
"mappings": {
"type1" : {
"properties" : {
"obj1" : {
"type" : "nested",
"properties": {
"name": {
"type":"text"
},
"number": {
"type":"text"
}
}
},
"obj2" : {
"type" : "nested",
"properties": {
"color": {
"type":"text"
},
"food": {
"type":"text"
}
}
},
"lul":{
"type": "text"
},
"pucci":{
"type": "text"
}
}
}
}
}
PUT /my_index/type1/1
{
"obj1": [
{ "name":"liar", "number":"deer dog"},
{ "name":"one two three", "number":"you can call on me"},
{ "name":"ricky gervais", "number":"user 123"}
],
"obj2": [
{ "color":"red green blue", "food":"meatball and spaghetti"},
{ "color":"orange", "food":"pineapple, fish, goat"},
{ "color":"none", "food":"none"}
],
"lul": "lul its me user123",
"field": "one dog"
}
POST /my_index/_search
{
"query": {
"nested": {
"path": ["obj1", "obj2"],
"query": {
"query_string": {
"query": "ricky",
"all_fields": true
}
}
}
}
}
I'm afraid I don't know the terminology to succinctly describe what I'm trying to do, but I will explain what I'm currently doing and what I'd like to do. I'm trying to converge two search queries into a single query, taking geo point data from one index to use a search parameter for searching in a second index/doctype.
My current ES set up:
Indices and DocTypes:
|- 1) locations
|---- 1a) UK_postcode
|- 2) accounts
|---- 2a) client
Each of the Doctypes has a field names 'location' which is mapped to a GeoPoint type.
My Current Process:
1) Users search for clients based on keywords and distance from location (a UK postcode).
2) System takes the postcode and searches for the matching results to get the geo_point latitude and longitude data from the locations.UK_postcode.
3) System uses the provided keywords and latitude and longitude to search on the accounts.client index/doctype.
4) System returns nice looking results to the user, based on ES search results.
My Question:
Can steps 2 and 3 be rolled into a single search query? If yes how do I do this? I want to pass a postcode to the search query and for ES to find the geo_point data for fulfilling the requirements of a geo distance query on the client doctype.
Using pre-indexed shapes, you can definitely eliminate step 2. Note that this solution only works with pre-defined distances.
The main idea would be:
to store in your locations index a geo_shape of type circle for each postcode and each pre-defined distances.
to store in your accounts index a geo_shape of type Point for your client location
create a geo_shape query of type circle which would leverage the pre-indexed postcode shapes.
So as a quick example, you'd have this:
A. Create the postcode locations index:
PUT /locations
{
"mappings": {
"UK_postcode": {
"properties": {
"location": { "type" : "geo_shape" }
}
}
}
}
B. Create client locations index
PUT /accounts
{
"mappings": {
"client": {
"properties": {
"name": { "type": "string" }
"location": { "type" : "geo_shape" }
}
}
}
}
C. Create sample postcode circle of 1, 2, 3 mile radius for "M32 0JG"
PUT /locations/UK_postcode/M320JG-1
{
"location": {
"type" : "circle",
"coordinates" : [-2.30283674284007, 53.4556572899372],
"radius": "1mi"
}
}
PUT /locations/UK_postcode/M320JG-2
{
"location": {
"type" : "circle",
"coordinates" : [-2.30283674284007, 53.4556572899372],
"radius": "2mi"
}
}
# ... repeat until radius = 10
D. Create sample client very close to "M32 0JG"
PUT /accounts/client/1234
{
"name": "Big Corp"
"location": {
"type" : "point",
"coordinates" : [-2.30293674284007, 53.4557572899372]
}
}
E. Query all clients whose name matches "big" and who are in a 2-mile radius of the postcode "M32 0JG"
POST /accounts/client/_search
{
"bool": {
"must": [
{
"match": {
"name": "big" <--- free text name match
}
}
],
"filter": {
"geo_shape": {
"location": {
"indexed_shape": {
"id": "M320JG-2", <--- located within two miles of M32 0JG
"type": "UK_postcode",
"index": "locations",
"path": "location"
}
}
}
}
}
}