I'm using olivere's v.5 elasticsearch library - https://godoc.org/github.com/olivere/elastic
Trying to check if a particular field exists in a document. According to ES docs, it's possible - https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-exists-query.html
But I can't find a Go query in the library to execute that. I found the NewExistsService but I don't see a method to search by field name. I need something like
elasticClient.Exists().Field("datefield")
which returns true or false. But 'Field' is not a method of 'NewExistsService'. Any help would be appreciated.
Related
I am trying to create one query in the Kibana search bar to retrieve some specific documents.
The goal is to get the documents that either have the field "myDate" before 2019-10-08 or "myDate" does not exist.
I have documents that meet one or the other condition.
I started by creating this query :
myDate:<=2019-10-08 OR NOT _exists_:myDate
But no documents were returned.
Since it did not work, I tried some other ways i found online :
myDate:<=2019-10-08 OR NOT (_exists_:myDate)
myDate:<=2019-10-08 OR !(_exists_:myDate)
myDate:<=2019-10-08 OR NOT (myDate:*)
But still, no results.
When I use either "part" of the "OR" condition, it works perfectly : I get either the documents who have myDate<=2019-10-08 or the ones that do not have a "myDate" field filled.
But when I try with both conditions, I get no document.
I have to use only the search bar to find these documents, neither an elasticsearch rest query nor by using kibana filters.
Thank you for your help :)
Below query works. Use Inspect button in kibana to see what query is actually being fired and make sure you are using correct index pattern as well.
(myDate:<=2019-12-31) OR (NOT _exists_:myDate)
Take a look at Query DSL documentation for Boolean operators for more better understanding with different use cases
I am using olivere/elastic library for elasticsearch in my go app . I have list of values for a particular field (say fieldA) of elasticsearch document. I want to update a particular field of all document by searching on field fieldA .
This : Updating a record in ElasticSearch using olivere/elastic in google go
explains the update part. But in my case in don't have Id of documents to be updated . So, either i can make search call to retrieve document ids and then update them , or is there another way am missing? Thanks in Advance.
If you need to update a list of documents, you can use the Update By Query API. The unit tests give you a hint about how the syntax looks like. However, if you have individual values for individual documents, I guess there's no other way than updating them one by one. The fastest way to achieve that is by using the Bulk API.
I have a use case which is a bit similar to the ES example of dynamic_template where I want certain strings to be analyzed and certain not.
My document fields don't have such a convention and the decision is made based on an external schema. So currently my flow is:
I grab the inputs document from the DB
I grab the approrpiate schema (same database, currently using logstash for import)
I adjust the name in the document accordingly (using logstash's ruby mutator):
if not analyzed I don't change the name
if analyzed I change it to ORIGINALNAME_analyzed
This will handle the analyzed/not_analyzed problem thanks to dynamic_template I set but now the user doesn't know which fields are analyzed so there's no easy way for him to write queries because he doesn't know what's the name of the field.
I wanted to use field name aliases but apparently ES doesn't support them. Are there any other mechanisms I'm missing I could use here like field rename after indexation or something else?
For example this ancient thread mentions that field.sub.name can be queried as just name but I'm guessing this has changed when they disallowed . in the name some time ago since I cannot get it to work?
Let the user only create queries with the original name. I believe you have some code that converts this user query to Elasticsearch query. When converting to Elasticsearch query, instead of using the field name provided by the user alone use both the field names ORIGINALNAME as well as ORIGINALNAME_analyzed. If you are using a match query, convert it to multi_match. If you are using a term query, convert it to a bool should query. I guess you get where I am going with this.
Elasticsearch won't mind if a field does not exists. This can be a problem if there is already a field with _analyzed appended in its original name. But with some tricks that can be fixed too.
I am using ELK to create dashboards from my log files. I have a log file with entries that contain an id value and a "success"/"failure" value, displaying whether an operation with a given id succeeded or failed. Each operation/id can fail an unlimited number of times and succeed at most once. In my Kibana dashboard I want to display the count of log entries with a "failure" value for each operation id, but I want to filter out cases where a "success" log entry for the id exists. i.e. I am only interested in operations that never succeeded. Any hints for tricks that would achieve this?
This is easy in Kibana 5 search bar. Just add a filter
!(_exists_:"your_variable")
you can toggle the filter or write the inverse query as
_exists_:"your_variable"
In Kibana 4 and Kibana 3 you can use this query which is now deprecated
_missing_:"your_variable"
NOTE: In Elasticsearch 7.x, Kibana now has a pull down to select KQL or Lucene style queries in the search bar. Be mindful that syntax such as _exists_:FIELD is a Lucene syntax and you need to set the pulldown accordingly.
In newer ELK versions (I think after Elasticsearch 6) you should use field:* to check if the field exist and not field:* to check if it's missing.
elastic search reference:
https://www.elastic.co/guide/en/elasticsearch/reference/6.5/query-dsl-query-string-query.html#_wildcards
! (_exists_:NAME) is not working for me. I use suggestion from:
https://discuss.elastic.co/t/kibana-5-0-0--missing--is-not-working-anymore/64336
NOT _exists_:NAME
UPDATE The problem I faced is that ES syntax forbids spaces after negation operators. Use one of:
NOT _exists_:FIELD
!_exists_:FIELD
-_exists_:FIELD
Check tutorial: https://www.timroes.de/2016/05/29/elasticsearch-kibana-queries-in-depth-tutorial/
NOTE: In Elasticsearch 7.x, Kibana now has a pull down to select KQL or Lucene style queries in the search bar. Be mindful that syntax such as _exists_:FIELD is a Lucene syntax and you need to set the pulldown accordingly.
In newer versions of Kibana the default language is now KQL (Kibana Query Language) not Lucene anymore. So most answers here are outdated. The query if a field exists is the following:
your_variable:*
and to answer your question you can just negate that:
not your_variable:*
You can find more documation on here: https://www.elastic.co/guide/en/kibana/7.15/kuery-query.html
You can also toggle back to Lucene if you click on that button inside the search field but in my opinion the new language is way easier to use:
One option would be to create an own query for this criteria in Kibana. Then just have your panel that does the counting just to use this query.
value:failure
More information here:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#query-string-syntax
I am trying to integrate SOLR with Magento on my development machine. We are upgrading Magento and I want to test if SOLR is working as well.
I am able to feed SOLR, the stats say that it has documents. In SOLR admin, when I put in : as query string, I do get the list of documents. But when I search for "maria mosters" for example, no results are returned.
I have tried SOLR 1.4.1 (which we run in production) and 3.4.0.
My schema.xml: http://pastebin.com/3a2J99re
Thank you for your replies. I finally got my answer, for my case.
I found out by checking the query string that was being logged by SOLR. This was for example:
127.0.0.1 - - [28/09/2011:09:05:34 +0000] "GET /solr/select?sort=score+desc&fl=id&spellcheck=true&spellcheck.count=2&qt=magento_nl&spellcheck.collate=true&spellcheck.dictionary=magento_spell_nl&spellcheck.extendedResults=true&fq=visibility%3A4+AND+store_id%3A1&version=1.2&wt=json&json.nl=map&q=%28maria+mosterd%29&start=0&rows=1 HTTP/1.0" 400 1405
When I requested this query the first time, it said that the field visibility was unknown. Apparently this field was added by Magento in the upgraded release. I added the field to the config, and ran the query again. Now it said that the dictionairy magento_spell_nl did not exist.
What happened?
The new Magento has a option called "Enable Search Suggestions". In my previous Magento version, this option didn't exist, so this spellchecker thing was not passed to the query string.
When I turned this setting of, I was able to use my exact copy of the production server.
*:*
would work as its matching all on all fields.
Search for maria mosters is going to search in the default field, if you are using the standard request handler.
The default search field set in the schema is fulltext and I don't see any copyfields into it.
So are you sure the field is populated.
If you are using any custom request handler via the qt param, are the proper fields included in it ?
sharing you solrconfig and full query might help for others to help you further.
Looks like your issue is that in your schema, you have the fulltext field defined as the default search field, but you are not populating that field. I would recommend either setting the default field to another field that you are populating or when you execute your query, specify the field that you want to search against... Example text_en:"maria monsters"
Please also see the SolrQuerySyntax page on the Solr Wiki for more details.