I'm on Kibana v7.4.2
I have 2 fields setup: message and message.keyword
If I just go to the Discover tab and search message.keyword:* it shows everything message:* does.
I want to search for a specific substring such as "[Special Message]". As I understand it, I have to use the message.keyword because message:"[Special Message]" will always be fuzzy. It will return entries that have this is a special message and not exactly here is [Special Message] which is insufficient for me.
However message.keyword:"[Special Message]" always returns No results match your search criteria
This behavior is observed the same in both KQL and Lucene.
How do I search field.keyword for something?
Related
I am facing a strange issue with a specific search.
I would expect the below two queries to return the same result set since space between keywords interpreted as AND anyway:
1) inurl:taskcracker Angela restart crash
2) inurl:taskcracker Angela AND restart crash
First one returns 42 results where the highlighted keywords on the search result page includes 'start' in addition to 'restart'. Whereas the second query returns only 2 results with 'restart' only (no 'start') which what I was expecting from the first search as well.
Please note that it does not matter whether I put an AND in front of other keywords on the 2nd query. It only makes a difference when I put or don't put an AND in front of 'restart'
I initially thought that maybe restart is in the synonyms list under Search > Query Settings > Synonym Data > English in the GSA admin panel but it is not there.
So the issue is when I don't put an explicit AND in front of 'restart' GSA expands it to include 'start' as well.
Any ideas whether this comes from a configuration somewhere on the admin panel or likely to be a bug?
This is probably because query expansion doesn't work when you put AND before "restart".
Can you try adding &entqr=0 to the URL being sent to the GSA? It controls the the query expansion policy. You can also try &entqrm=0 too if first one doesn't work.
I have an ElasticSearch instance with Kibana, holding a lot of user-level app data that I've accumulated over a few years. One of the fields is the Java version the user is running.
I'd like to graph Java versions over time, so I can have an idea whether it's reasonable to transition to a newer version. Unfortunately I can't find a way to aggregate 1.6.0_31, 1.6.0_32, 1.6.0_37, 1.6.0_51 as a single 1.6 entry, so the graph is nearly unreadable right now.
Is there a way in Kibana to aggregate the data, like a 'scripted field' that I could write a regex for? E.g. simplified_java: osjv % '\d\.\d' which would defined simplified_java as the part of the osjv field that matches a digit followed by a dot followed by a digit.
Currently it looks like Kibana only supports numeric scripted fields, which makes this hard. I'm not using LogStash, as I'm not really using 'logs', but rather a custom event reporting framework in my desktop application that (opt-in) reports usage statistics, so unfortunately I can't use any of its features.
I can manually do it, but I've already imported 2G of event data, and I'd hate to have to do it again, adding a new field just for what should be computable... :(
Is there a way to create a field based on a substring or regex in Kibana, or (failing that) a way to tell ElasticSearch to transparently do the same thing?
You can definitely do scripted fields in Kibana against string data in Elasticsearch, provided it is mapped as a keyword type. See the scripted field documentation for a tiny bit of info, and the scripted field blog post for better examples.
In short, you could do what you're looking for by building a scripted field that returns a substring:
def version = doc['osjv'].value; return (version != null) ? v.substring(0, v.lastIndexOf(".")-1) : version;
Keep in mind that there are performance implications with scripted fields since they run each time you view them.
A better approach could be to create a new field in your documents with the simplified_java value. You won't need to re-ingest all your data from source, but can instead do an Update By Query. Your query is just match_all{} and then you can define a script which creates the new field. So yes, there is indexing happening, but happening "in place":
POST your-index/_update_by_query
{
"script": {
"source": "def version = ctx._source.osjv; ctx._source.simplified_java = (version != null) ? version.substring(0, version.lastIndexOf(".")-1) : version",
"lang": "painless"
},
"query": {
"match_all": {}
}
}
...haven't tested either of those scripts, but would look something like them!
I am using Elasticsearch Java API to create indexes and write queries for search.
The indexes are created on various fields. One of the field is numeric(integer) on which index is created.
Now the input we get is in form of string. We have to search all the fields for the input provided. To search on numeric field we are using
QueryBuilders.rangeQuery() method.
But when it encounters any non integer value in "to" or "from" field it throws
SearchPhaseExecutionException[Failed to execute phase [query].
nested: NumberFormatException[For input string: \"30y\"]
How can I avoid this? Its fine that we do not get any search results, but I want to avoid this Exception as there can be cases where we get non integer input.
Another option is to check all the input tokens, which I want to avoid because it will add another level of check which will impact performance.
Is there any way I can accomplish this with elasticsearch API?
Another option is to check all the input tokens, which I want to avoid because it will add another level of check which will impact performance.
Checking/validating the user input is something that you should always do, in any case, whatever your performance requirements are. If you don't, you unnecessarily expose your cluster to unknown future threats, but also known ones that have been causing some damages lately and which can have a much worse impact on your cluster and/or business than a few milliseconds spent cleaning up user input. Elasticsearch is flexible and can do wonders, but you have to play nice with it, too.
That being said, if you really want to avoid secure coding best practices, you can use the following query that won't bark if the input data is not compliant.
{
"query": {
"simple_query_string": {
"query": "numfield:[10y TO *]"
}
}
}
simple_query_string is the equivalent of query_string but is much more permissive with the input and will never throw an exception.
I have a view control and make the query string by the search param from the application layout control.Full text index is already enabled, the code in the search property like below:
var queryStr="";
if(param.option=="byName"){
queryStr="[name]="+param.search;
}
else{
queryStr="[title]="+param.search;
}
return queryStr;
i found that the queryStr can be made correct,like "[name]=Vincent",but the view shows all documents contain the word "Vincent",not only the spesific field "name". Just like I used "Vincent" for search.
I want to know how to get the correct result.thank you!
Should it be?
"[name] CONTAINS "+param.search;
similar to this:
[Projectname] CONTAINS top secret
Use the word CONTAINS rather then = ?
I'm not 100% sure but there was a recent blog post on seaching just the other day: http://lostinxpages.com/2014/05/15/exporting-to-excel-using-queries-in-xpages/
Finally I find the problem. The full-text syntax works fine, both "field/FIELD/[]" or "contains/CONTAINS/=" can work. But I used an application layout for search. The search button generate two parameters: "option"(if you choosed before) and "search" by default. The search parameter is used as the value of the view control's search property directly. When I customed this property by myself ,it would not be used(if the ssjs return a null) or generate an " not understandable" error(if the ssjs return a string,which is normal in most situation). The solution is give that two parameters custom parameter name .like fieldName for option, searchText for search. After that, you can use param.fieldName and param.searchText to build your full-text search string.I have tried ,and it works fine now.
For the most part David Leedys answer should work. Some pointers on diagnosing FTI issues though.
1. Your search phrase must be surrounded by quotes. Example:
queryStr='[name] = "' + param.search + '"';
2. Get the actual fully constructed search string and test it in the Notes FTI search bar. Do you get the same incorrect results? If so, then fix the search string.
3. If it is working in the Notes client then add the following debug to the Domino servers notes.ini file (alternatively: set config from console).
Debug_FTV_Search=1
When you run a search it should generate debug like this: Search [name] CONTAINS "String"
IN FTGSearch
[22E8:008A-1710] option = 0x400219
[22E8:008A-1710] Query: ( FIELD name CONTAINS "String")
[22E8:008A-1710] Engine Query: ("String"%STEM#F134)
[22E8:008A-1710] GTR query performed in 10 ms. 2 documents found
[22E8:008A-1710] 0 documents disualified by deletion
[22E8:008A-1710] 0 documents disqualified by ACL
[22E8:008A-1710] 0 documents disqualified by IDTable
[22E8:008A-1710] 0 documents disqualified by NIF
[22E8:008A-1710] Results marshalled in 8 ms. 2 documents left
[22E8:008A-1710] OUT FTGSearch error = 0
[22E8:008A-1710] FTGSearch: found=2, returned=2, start=0, count=0, limit=0
[22E8:008A-1710] Total search time 22 ms.
You want to check the Query and Engine Query from client search vs XPage to see what is generated. If they don't match, update your question with the results so we can see what's going on.
The disqualified section tells you if search results were dropped. For example, if your XPage was running under credentials that were not allowed to view the documents, then disqualified by ACL would have a value.
4. The Notes client has two search syntax methods. There is "Notes" and "Web" style. By default for R9 (and R8.x IIRC) is Web style (client). The server uses Notes style.
You can change the client behavior in the Basic Settings preferences.
The Web style does not understand Notes syntax unless the first word in the search is a reserved keyword and all in uppercase.
Example.
Will use "web" search: field name contains "string"
Will use "Notes" search: FIELD name contains "string"
I am not sure if that impacts XPiNC though (never tested it).
When I run the query (tags:("a")) in elasticsearch, I get 0 results. My query URL looks like:
http://127.0.0.1:9200/haystack/_search?q=(tags%3A(%22a%22))
That is to be expected, since no objects have a tag set to "a".
Now when I change the condition, and add an AND, (org:("1") AND tags:("a")), I get 3 results back! The query URL looks like:
http://127.0.0.1:9200/haystack/_search?q=(org%3A(%221%22)%20AND%20tags%3A(%22a%22))
Getting more results back does not make any sense to me. I would expect that kind of behavior with the OR operator, but AND? What is going on?
Edit: This is caused by the snowball analyzer. See here