Cannot Parse The "search_phase_execution_exception" Into Go Structure *elastic.Error And Get The Root Cause - elasticsearch

My department uses the olivere elastic v7.0.26 as ElasticSearch Client,sometimes our front-end system returns some errormsg such as "Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]" but doesn't have any other useful msg to help troubleshooting,so I've searched GitHub about parsing errors to *elastic.Error and codes are just like this:
if err != nil {
log.WithFields(ctx, log.Fields{}).WithError(err).Warn("list query es error")
if ex, ok := err.(*elastic.Error); ok {
log.WithFields(ctx, log.Fields{"query": query, "status": ex.Status, "detail": ex.Details}).WithError(ex).Warnf("list query es err")
}
return res, err
}
But the strange thing happened is "list query es error" was printed on our log system but the next log "list query es err" didn't,i've used deep-paging (large from+size) to check the log,it works and print the elastic error such as root cause so i can get the max_result_window tips..But it returns a unparsing error which is returned as the search_phase_execution_exception error,the company code is not allowed to paste to the open-sourcing website,and I just wanna know what error can make elasticsearch returns the search_phase_execution_exception and the errcode 400?Really appreciate your help!
same to above description,but the point I need to supple is that the problem is not always happened,so I can exclude the index mapping field type error such as "text/keyword",just wanna get the whole error/exceptions elasticsearch can return,but I cannot find the relative documentions on elastic guide,that's ok if u can provide the guide/doc about this

uhh it seems like a joke XD,I've never learned about some troubleshooting technique,but after I asked this question,I realized that I can get the ElasticSearch server log,so I asked our company's SRE and got the log,it's a error about terms query exceed maximum,and the reason caused is our RPC interface return a exceptional result about nil data redis set key,it will be a big-key problem,but we found this and fix it.

Related

Grafeas golang filter string and DeleteOccurrence() API errors

I'm running into 2 separate issues using the Grafeas golang v1beta1 API.
What I'm trying to do
Call ListOccurrencesRequest() with a Filter to get a list of occurrences for deletion
Call DeleteOccurrence() on each occurrence from above list to delete it
Issue #1
I'm trying to set the Filter field using this GCP reference grafeas golang code as a guide.
filterStr := fmt.Sprintf(`kind=%q`, grafeas_common_proto.NoteKind_BUILD.String())
listReq := &grafeas_proto.ListOccurrencesRequest{
Parent: BuildProject,
Filter: filterStr,
PageSize: 100,
}
listOccResp, err := r.grafeasCommon.ListOccurrences(ctx, listReq)
for {
if err != nil {
log.Error("failed to iterate over occurrences", zap.NamedError("error", err))
return nil, err
}
...
But it looks like my filterStr is invalid, here's the error:
filterStr {"filterStr": "kind=\"BUILD\""}
failed to iterate over occurrences {"error": "rpc error: code = Internal desc = error while parsing filter expression: 4 errors occurred:\n\t* error parsing filter\n\t* Syntax error: token recognition error at: '=\"' (1:4)\n\t* Syntax error: token recognition error at: '\"' (1:11)\n\t* Syntax error: extraneous input 'BUILD' expecting <EOF> (1:6)\n\n"}
It looks like the \ escape character is causing trouble but I've tried it without it and get another flavor of same type of error.
Issue #2
When I call DeleteOccurrence(), I can see that the occurrence is in fact deleted from Grafeas by checking:
curl http://localhost:8080/v1beta1/projects/broker_builds/occurrences
But DeleteOccurrence() always sets the err
Code:
for _, o := range occToDelete {
log.Info("occToDelete", zap.String("occurrence", o))
_, err := r.grafeasCommon.DeleteOccurrence(ctx, &grafeas_proto.DeleteOccurrenceRequest{
Name: o,
})
if err != nil {
log.Error("failed to delete occurrence", zap.String("occurrence", o), zap.NamedError("error", err))
}
}
Error:
failed to delete occurrence {"occurrence": "projects/broker_builds/occurrences/f61a4c57-a3d3-44a9-86ee-5d58cb6c6052", "error": "rpc error: code = Internal desc = grpc: error while marshaling: proto: Marshal called with nil"}
I don't understand what the error is referring to.
This question was cross-posted on Grafeas message board.
Appreciate any help. Thanks.
Can you shed some details around the storage engine used, and the filtering implementations details?
Issue 1. filtering is not implemented in any of the storage engines in gitHub.com/grafeas/grafeas.
Issue 2. it depends what store you use, memstore/embededstore do not seem to be producing any errors similar to what you mentioned... if using postgresql store, are you trying to delete an occurrence twice?
Solution for Issue #1
I'm using grafeas-elasticsearch as the storage backend. It uses a different filter string format than the examples I had looked at in my original post.
For example, instead of = -> ==, AND -> &&, etc.
More examples can be seen here:
https://github.com/rode/grafeas-elasticsearch/blob/main/test/v1beta1/occurrence_test.go#L226
Solution for Issue #2
Known issue with grafeas
https://github.com/grafeas/grafeas/pull/456
https://github.com/grafeas/grafeas/pull/468
Unfortunately the latest tagged release of grafeas v0.1.6 does not include these fixes yet. So will need to pick them up on the next release.
Thanks to #Ovidiu Ghinet, that was a good tip

Queries that involve more than 250 virtual entity lookup field values fail with "An unexpected error occurred."

I'm receiving this mysterious error message when using a custom virtual entity data provider:
{
"error": {
"code": "0x80040216",
"message": "An unexpected error occurred.",
"#Microsoft.PowerApps.CDS.ErrorDetails.ApiExceptionSourceKey": "Plugin/Microsoft.Crm.ObjectModel.InsertLookupLogicalNamePlugin",
"#Microsoft.PowerApps.CDS.ErrorDetails.ApiStepKey": "ccb4d064-785c-eb11-a812-002248163c60",
"#Microsoft.PowerApps.CDS.ErrorDetails.ApiDepthKey": "1",
"#Microsoft.PowerApps.CDS.ErrorDetails.ApiActivityIdKey": "aac514e1-53ec-4ed9-9e47-d2643f0e92b1",
"#Microsoft.PowerApps.CDS.ErrorDetails.ApiPluginSolutionNameKey": "System",
"#Microsoft.PowerApps.CDS.ErrorDetails.ApiStepSolutionNameKey": "System",
"#Microsoft.PowerApps.CDS.ErrorDetails.ApiExceptionCategory": "SystemFailure",
"#Microsoft.PowerApps.CDS.ErrorDetails.ApiExceptionMesageName": "UnExpected",
"#Microsoft.PowerApps.CDS.ErrorDetails.ApiExceptionHttpStatusCode": "400",
"#Microsoft.PowerApps.CDS.HelpLink": "http://go.microsoft.com/fwlink/?LinkID=398563&error=Microsoft.Crm.CrmException%3a80040216&client=platform",
"#Microsoft.PowerApps.CDS.TraceText": "\r\n[Microsoft.Crm.ObjectModel: Microsoft.Crm.ObjectModel.InsertLookupLogicalNamePlugin]\r\n[ccb4d064-785c-eb11-a812-002248163c60: External plug-in implementation]\r\n\r\n",
"#Microsoft.PowerApps.CDS.InnerError.Message": "An unexpected error occurred."
}
}
It seems to occur more often with larger page sizes.
Plugin trace logs indicate the data provider ran successfully/no exception.
After some spelunking I found this error message in the on-prem server:
Query with entity reference to virtual entity can not exceed 250 limit. Please modify your query to reduce the number.
The limit appears to be unique lookup field values across the whole query. So in this example, if the results were as below:
Record
Lookup Column to Virtual Entity
Lookup Column to Virtual Entity
1
Value A
Value B
2
Value C
3
Value A
Value C
Would count as 3 towards the limit for that query.

Individually update a large amount of documents with the Python DSL Elasticsearch UpdateByQuery

I'm trying to use the UpdateByQuery to update a property of a large amount of documents. But as each document will have a different value, I need to execute ir one by one. I'm traversing a big amount of documents, and for each document I call this funcion:
def update_references(self, query, script_source):
try:
ubq = UpdateByQuery(using=self.client, index=self.index).update_from_dict(query).script(source=script_source)
ubq.execute()
except Exception as err:
return False
return True
Some example values are:
query = {'query': {'match': {'_id': 'VpKI1msBNuDimFsyxxm4'}}}
script_source = 'ctx._source.refs = [\'python\', \'java\']'
The problem is that when I do that, I got an error: "Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting".
If I change the max_compilations_rate using Kibana, it has no effect:
PUT _cluster/settings
{
"transient": {
"script.max_compilations_rate": "1500/1m"
}
}
Anyway, it would be better to use a parametrized script. I tried:
def update_references(self, query, script_source, script_params):
try:
ubq = UpdateByQuery(using=self.client, index=self.index).update_from_dict(query).script(source=script_source, params=script_params)
ubq.execute()
except Exception as err:
return False
return True
So, this time:
script_source = 'ctx._source.refs = params.value'
script_params = {'value': [\'python\', \'java\']}
But as I have to update the query and the parameters each time, I need to create a new instance of the UpdateByQuery for each document in the large collection, and the result is the same error.
I also tried to traverse and update the large collection with:
es.update(
index=kwargs["index"],
doc_type="paper",
id=paper["_id"],
body={"doc": {
"refs": paper["refs"] # e.g. [\\'python\\', \\'java\\']
}}
)
But I'm getting the following error: "Failed to establish a new connection: [Errno 99] Cannot assign requested address juil. 10 18:07:14 bib gunicorn[20891]: POST http://localhost:9200/papers/paper/OZKI1msBNuDimFsy0SM9/_update [status:N/A request:0.005s"
So, please, if you have any idea on how to solve this it will be really appreciated.
Best,
You can try it like this.
PUT _cluster/settings
{
"persistent" : {
"script.max_compilations_rate" : "1500/1m"
}
}
The version update is causing these errors.

Client-side validation of Elasticsearch query string

I have an application that uses NEST (Elasticsearch .NET client) to communicate with an Elasticsearch cluster. The integration allows the user to specify input for the "query_string" portion of a query.
The user may input an invalid query. Say "AND", which is invalid because the predicate is incomplete. But the error message that comes back from Elasticsearch is exceedingly verbose, and contains terminology that isn't very user-friendly, like "all shards failed".
Is there a way I can offer the user a more meaningful error message (say - "bad predicate"). Ideally, the users search string would be validated without an Elasticsearch round-trip, but I'll settle for a simpler error message however I can get it.
The error message returned by Elasticsearch is verbose but for parsing errors like these, Elasticsearch throws a QueryParsingException. If you examine the error message closely, you'll find the string QueryParsingException towards the end of the entire error message. This is the exception (and its message) you are interested in. For example, when I spelt must as mus2t in a search request, I get a huge error message by Elasticsearch and below is the last part of the error message.
QueryParsingException[[<index name>] bool query does not support [mus2t]]; }]
I got this when I spelt must as mus2t. You can parse and extract out this error message.
You can use validation api.
For following query
var validateResponse = client.Validate<Document>(descriptor => descriptor
.Explain()
.Query(query => query
.QueryString(qs => qs
.OnFields(f => f.Name)
.Query("AND"))));
you will get
org.elasticsearch.index.query.QueryParsingException: [indexname]
Failed to parse query [AND];
org.apache.lucene.queryparser.classic.ParseException: Cannot parse
'AND': Encountered " <AND> "AND "" at line 1, column 0. Was expecting
one of:
<NOT> ...
"+" ...
"-" ...
<BAREOPER> ...
"(" ...
"*" ...
<QUOTED> ...
<TERM> ...
<PREFIXTERM> ...
<WILDTERM> ...
<REGEXPTERM> ...
"[" ...
"{" ...
<NUMBER> ...
<TERM> ...
"*" ...
Still not so perfect for end user and it requires round-trip to ES, but maybe it will be helpful.

How to solve distinct-values error

How to solve this error in XQuery. I want the data to be distinct with out duplication in XML result. I tried to add distinct-values in front of the doc in for statement, but this error was depicted.
Engine name: Saxon-PE XQuery 9.5.1.3
Severity: fatal
Description: XPTY0019: Required item type of first operand of '/' is node(); supplied value has item type xs:anyAtomicType
Start location: 23:0
URL: http://www.w3.org/TR/xpath20/#ERRXPTY0019
This is code :
for $sv1 in distinct-values(doc('tc.xml')//term/year)
let $sv2 := doc('tc.xml')//term[year= $sv1]
let $sv3 := doc('tc.xml')//student[idStudent= $sv1/idStudent](:HERE IS THE ERROR LINE:)
let $sv4 := doc('tc.xml')//program[idStudent= $sv3/idStudent]
return
<Statistics>
{$sv1 }
<Count_Student>{count($sv2)}</Count_Student>
<a50_60>{count(doc('tc.xml')/mydb//program[doc('tc.xml')/mydb//term/year =$sv1][avg>= 50 and avg < 60])}</a50_60>
</Statistics>
thank you in advance.
distinct-values() will atomize your input, this means that $sv1/idStudent won't work because $sv1 is not an element. Instead of using $sv1 on the line that give an error I think you should be using $sv2.

Resources