i have tried to find anything about my problem online but unfortunately i had no luck finding anything that would help me.
i have a custom javascript map built with mapbox-gl-js.
the map shows real estate objects which can be filtered by country, city etc. using a custom built filter bar. now on this filter bar there are dropdown fields and two input fields. the filters i built for the dropdown fields work just fine. in the code use equal-to-comparsions for this task.
the two input fields are there to filter the maximum price and a minimum square meters - here i use the less or equal than and greater or equal than comparsion filters.
for some reason these two filter's won't work as desired. for example if i filter for objects with max price 1000000 objects with greater price still show.
this is how my filter would look like in JSON:
["all",["<=","preis",1000000]]
this is how the features look like:
feature = {
"type": "Feature",
"geometry": {
"type": "Point",
"coordinates": [...]
},
"properties": {
[...],
"preis": 20000000,
[...]
}
}
i have also tried to reproduce this in a simple test map with simple objects - there the problem also exists.
does anyone have a clue why this is acting up on me or has anyone had or got an familiar issue?
thanks and br, John
Related
I have a product nested document containing a list of prices associated to different wholesalers.
Here is a document example :
{
"sku": "065879",
"name": "My product",
"price": [
{
"wholesaler": "1",
"location": "drm3btev3",
"price": "12.34"
},
{
"wholesaler": "2",
"location": "gbsuv7ztq",
"price": "45.67"
},
]
}
Given a customer's geo point, what is the correct query to get a list of documents sorted by price, using only the closest price for each document ?
Thanks by advance !
It's not a real answer but the global approach is to use a nested sort. Nested sort will allow you to filter the nested document on which you want to apply your sorting.
Then you should in the nested sort filter add a script query that will determine the closest wholesaler. The problem is that you cant work with geohash in painless. But if you convert your geohash to geopoint data type in, you will be able to use script distance features ( example here )
Then you could compute the minimal distance by iterating on all nested document and only match the one with the minimal distance.
But I have no idea of the performance impact and detailed implementation.
Good luck !
Edit: I found the answer, see below for Logstash <= 2.0 ===>
Plugin created for Logstash 2.0
Whomever is interested in this with Logstash 2.0 or above, I created a plugin that makes this dead simple:
The GEM is here:
https://rubygems.org/gems/logstash-filter-dateparts
Here is the documentation and source code:
https://github.com/mikebski/logstash-datepart-plugin
I've got a bunch of data in Logstash with a #Timestamp for a range of a couple of weeks. I have a duration field that is a number field, and I can do a date histogram. I would like to do a histogram over hour of day, rather than a linear histogram from x -> y dates. I would like the x axis to be 0 -> 23 instead of date x -> date y.
I think I can use the JSON Input advanced text input to add a field to the result set which is the hour of day of the #timestamp. The help text says:
Any JSON formatted properties you add here will be merged with the elasticsearch aggregation definition for this section. For example shard_size on a terms aggregation which leads me to believe it can be done but does not give any examples.
Edited to add:
I have tried setting up an entry in the scripted fields based on the link below, but it will not work like the examples on their blog with 4.1. The following script gives an error when trying to add a field with format number and name test_day_of_week: Integer.parseInt("1234")
The problem looks like the scripting is not very robust. Oddly enough, I want to do exactly what they are doing in the examples (add fields for day of month, day of week, etc...). I can get the field to work if the script is doc['#timestamp'], but I cannot manipulate the timestamp.
The docs say Lucene expressions are allowed and show some trig and GCD examples for GIS type stuff, but nothing for date...
There is this update to the BLOG:
UPDATE: As a security precaution, starting with version 4.0.0-RC1,
Kibana scripted fields default to Lucene Expressions, not Groovy, as
the scripting language. Since Lucene Expressions only support
operations on numerical fields, the example below dealing with date
math does not work in Kibana 4.0.0-RC1+ versions.
There is no suggestion for how to actually do this now. I guess I could go off and enable the Groovy plugin...
Any ideas?
EDIT - THE SOLUTION:
I added a filter using Ruby to do this, and it was pretty simple:
Basically, in a ruby script you can access event['field'] and you can create new ones. I use the Ruby time bits to create new fields based on the #timestamp for the event.
ruby {
code => "ts = event['#timestamp']; event['weekday'] = ts.wday; event['hour'] = ts.hour; event['minute'] = ts.min; event['second'] = ts.sec; event['mday'] = ts.day; event['yday'] = ts.yday; event['month'] = ts.month;"
}
This no longer appears to work in Logstash 1.5.4 - the Ruby date elements appear to be unavailable, and this then throws a "rubyexception" and does not add the fields to the logstash events.
I've spent some time searching for a way to recover the functionality we had in the Groovy scripted fields, which are unavailable for scripting dynamically, to provide me with fields such as "hourofday", "dayofweek", et cetera. What I've done is to add these as groovy script files directly on the Elasticsearch nodes themselves, like so:
/etc/elasticsearch/scripts/
hourofday.groovy
dayofweek.groovy
weekofyear.groovy
... and so on.
Those script files contain a single line of Groovy, like so:
Integer.parseInt(new Date(doc["#timestamp"].value).format("d")) (dayofmonth)
Integer.parseInt(new Date(doc["#timestamp"].value).format("u")) (dayofweek)
To reference these in Kibana, firstly create a new search and save it, or choose one of your existing saved searches (Please take a copy of the existing JSON before you change it, just in case) in the "Settings -> Saved Objects -> Searches" page. You then modify the query to add "Script Fields" in, so you get something like this:
{
"query" : {
...
},
"script_fields": {
"minuteofhour": {
"script_file": "minuteofhour"
},
"hourofday": {
"script_file": "hourofday"
},
"dayofweek": {
"script_file": "dayofweek"
},
"dayofmonth": {
"script_file": "dayofmonth"
},
"dayofyear": {
"script_file": "dayofyear"
},
"weekofmonth": {
"script_file": "weekofmonth"
},
"weekofyear": {
"script_file": "weekofyear"
},
"monthofyear": {
"script_file": "monthofyear"
}
}
}
As shown, the "script_fields" line should fall outside the "query" itself, or you will get an error. Also ensure the script files are available to all your Elasticsearch nodes.
I have an ElasticSearch index with 2 mappings (types).
In the app I need to display a paginated feed containing items of both types.
Currently the items are sorted just by creation date, but I also want to have control on how the items alternate with each other on the page.
For example, I want to set a rule for sequence "3 items of type A, 1 item of type B, and so on".
I need it to make sure items of both types are displayed on each page and equally distributed across the pages.
But as far as I see it's not possible to access another documents in custom score function script.
Of course it's easy to implement directly in the app logic, but it's not clear how to implement pagination using this way.
Any ideas on how to achieve that?
I don't think you can do this.
One approach (that doesn't work) is to keep a global variable in a script and to increment that once every document is being returned/processed. And then to take this number, divide it by 3 and get the modulo number. Based on this number, to sort the docs. But "global" variables are not possible in sripts.
The only two approaches that I can think of is to use a script to generate a random number and based on that to sort. In this way, you get some chances to have a "mixed list of types.
Or, if you want the smallest deterministic way of sorting the docs, still in a script take the ID of the document (you said is a number) modulo 3 it and use the value to sort.
For the random approach:
"sort": [
{
"date": {
"order": "desc"
}
},
{
"_script": {
"script": "Math.random()",
"type": "number",
"order": "asc"
}
}
]
Lets say I have data like this (lots of it)
{
"name" : "Coffee",
"quantity": 100,
"restock": 10
}
I want to use an odata $filter to show me ONLY items where the quantity is LESS than the restock number
Is it possible to do something like $filter=quantity lt restock
I know that specific example fails. Is there a way to do this? Or do I need to fetch everything and post process it?
That query should absolutely be possible in most (all?) versions of OData: see http://services.odata.org/V4/OData/OData.svc/Products?$filter=Rating lt Price for a working example.
I am newbie in Solr. I want to add a custom comparatorClass in Solr. I also need to use fields - term and count in my custom class which I have defined in my schema.xml.
Structure of indexing document :
"docs": [
{
"count": 98,
"term": "age",
},
{
"count": 6,
"term": "age assan",
},
{
"count": 5,
"term": "age but",
},
{
"count": 10,
"term": "age salman",
}]
I have stored ngrams with term and their count but solr gives frequency by own that I don't need. I want my count frequency which I have defined for each term. And that term and count, I need to use and want to sort with frequency(count) and then edit distance which I need to implement by creating own class in comparator class or there is something else which helps me. Please share..
How can I do this. Any help please.
Thanks.
You should be able to do this without implementing a custom similarity class. The first requirement is (from your description) a straight forward sort on the count value, while the latter can be implemented by sorting on the value from the strdist() function. You can also multiply or weight these values against each other in a single sort statement by using several functions.
If you really, really need to build your own scorer (which I don't think you need to do from your description) - these are usually written to explore other ranking algorithms than tf/idf, bm25 etc. for larger corpuses, a search on Google gives you many resources with pre-made, easy to adopt solutions. I particularly want to point out "This is the Nuclear Option" in Build Your Own Custom Lucene Query and Scorer:
Unless you just want the educational experience, building a custom Lucene Query should be the “nuclear option” for search relevancy. It’s very fiddly and there are many ins-and-outs. If you’re actually considering this to solve a real problem, you’ve already gone down the following paths [...]