elasticsearch fails VerifyRepositoryRequest in C# Nest client - elasticsearch

After updating to elasticsearch 7.5.1 following code is now causing an exception
"Entry point not found"
//old code worked just fine
var verifyRepositoryResponse = this.elasticClient.VerifyRepository(new VerifyRepositoryRequest("C__MYFOLDER_Backup"));
//new code throws
var verifyRepositoryResponse = this.elasticClient.Snapshot.VerifyRepository(new VerifyRepositoryRequest("C__MYFOLDER_Backup"));
Note that making following request via fiddler
POST http://localhost:9200/_snapshot/C__MYFOLDER_Backup/_verify HTTP/1.1
User-Agent: Fiddler
Host: localhost:9200
Content-Type: application/json
Content-Length: 0
returns
{"nodes":{"1cWG9trDRi--6I-46lOlBw":{"name":"DESKTOP-5L01F6I"}}}
any ideas what causes the problem here?
Unfortunately there is zero documentation on that matter.
UPDATE:
Calling the code with a non existing directory as argument
var verifyRepositoryResponse = this.elasticClient.Snapshot.VerifyRepository(new VerifyRepositoryRequest("C__DOESNOTEXIST_Backup"));
I will get a 4xx response as expected in this scenario.
Before all tests, repositories and db have been cleaned.
Update:
Adavanced error information:
"# FailureReason: Unrecoverable/Unexpected BadResponse while attempting POST on http://localhost:9200/_snapshot/C__MYFOLDER_Backup/_verify\r\n - [1] BadResponse: Node: http://localhost:9200/ Exception: EntryPointNotFoundException Took: 00:00:00.1859931\r\n# Audit
exception in step 1 BadResponse:\r\nSystem.EntryPointNotFoundException: Entry point was not found.\r\n at Elasticsearch.Net.Utf8Json.IJsonFormatter`1.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)\r\n at Deserialize(Object[] ,
JsonReader& , IJsonFormatterResolver )\r\n at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Byte[] bytes, Int32 offset, IJsonFormatterResolver resolver)\r\n at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Stream stream,
IJsonFormatterResolver resolver)\r\n at Elasticsearch.Net.DiagnosticsSerializerProxy.Deserialize[T](Stream stream)\r\n at Elasticsearch.Net.ResponseBuilder.SetBody[TResponse](ApiCallDetails details, RequestData requestData, Stream responseStream, String
mimeType)\r\n at Elasticsearch.Net.ResponseBuilder.ToResponse[TResponse](RequestData requestData, Exception ex, Nullable`1 statusCode, IEnumerable`1 warnings, Stream responseStream, String mimeType)\r\n at
Elasticsearch.Net.HttpWebRequestConnection.Request[TResponse](RequestData requestData)\r\n at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData)\r\n at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method,
String path, PostData data, IRequestParameters requestParameters)\r\n# Inner Exception: Entry point was not found.\r\nSystem.EntryPointNotFoundException: Entry point was not found.\r\n at Elasticsearch.Net.Utf8Json.IJsonFormatter`1.Deserialize(JsonReader&
reader, IJsonFormatterResolver formatterResolver)\r\n at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )\r\n at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Byte[] bytes, Int32 offset, IJsonFormatterResolver resolver)\r\n at
Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Stream stream, IJsonFormatterResolver resolver)\r\n at Elasticsearch.Net.DiagnosticsSerializerProxy.Deserialize[T](Stream stream)\r\n at
Elasticsearch.Net.ResponseBuilder.SetBody[TResponse](ApiCallDetails details, RequestData requestData, Stream responseStream, String mimeType)\r\n at Elasticsearch.Net.ResponseBuilder.ToResponse[TResponse](RequestData requestData, Exception ex, Nullable`1
statusCode, IEnumerable`1 warnings, Stream responseStream, String mimeType)\r\n at Elasticsearch.Net.HttpWebRequestConnection.Request[TResponse](RequestData requestData)\r\n at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData
requestData)\r\n at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)\r\n# Exception:\r\nElasticsearch.Net.UnexpectedElasticsearchClientException: Entry point was not found. --->
System.EntryPointNotFoundException: Entry point was not found.\r\n at Elasticsearch.Net.Utf8Json.IJsonFormatter`1.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)\r\n at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver
)\r\n at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Byte[] bytes, Int32 offset, IJsonFormatterResolver resolver)\r\n at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Stream stream, IJsonFormatterResolver resolver)\r\n at
Elasticsearch.Net.DiagnosticsSerializerProxy.Deserialize[T](Stream stream)\r\n at Elasticsearch.Net.ResponseBuilder.SetBody[TResponse](ApiCallDetails details, RequestData requestData, Stream responseStream, String mimeType)\r\n at
Elasticsearch.Net.ResponseBuilder.ToResponse[TResponse](RequestData requestData, Exception ex, Nullable`1 statusCode, IEnumerable`1 warnings, Stream responseStream, String mimeType)\r\n at Elasticsearch.Net.HttpWebRequestConnection.Request[TResponse](RequestData
requestData)\r\n at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData)\r\n at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)\r\n ---
End of inner exception stack trace ---\r\n at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)\r\n at
Nest.Specification.SnapshotApi.SnapshotNamespace.VerifyRepository(IVerifyRepositoryRequest request)\r\n at InforsHT.Genesis.Infrastructure.Data.Repositories.ElasticBackupAndRestoreProxy.IsBackupLocationPresent(String backupLocation) in
C:\\work\\eve\\Source\\InforsHT.Genesis.Infrastructure.Data\\Repositories\\ElasticBackupAndRestoreProxy.cs:line 44\r\n"

This is a bug. VerifyRepositoryResponse's Node property is attributed with a JsonFormatterAttribute that expects to deserialize an IDictionary<string, CompactNodeInfo> when the property type is IReadOnlyDictionary<string, CompactNodeInfo>. This is problematic for the underlying JSON serializer which uses strongly typed formatters, IJsonFormatter<T>, where the type T must match exactly.
I've opened an issue to track and a pull request to fix the issue. For now, you'd be able to workaround the bug by using the low level client exposed on the high level client
var response = client.LowLevel.Snapshot.VerifyRepository<DynamicResponse>("C__MYFOLDER_Backup");
and traverse DynamicResponse. Alternatively, use StringResponse and parse using JObject or JsonDocument and traverse.

Related

Query string parameters is shown as null although value is entered

I'm developing a AWS Lambda function which will be invoked from a API gateway. The Lambda function and the API gateway are setup and the user is supposed to enter the search string in query string parameters like: http://apigatewayurl?name=some name. Then the application has the logic to return the respective id.
The class declaration is as follows:
public class PayRollIdExtraction implements RequestHandler<APIGatewayProxyRequestEvent, APIGatewayProxyResponseEvent> {}
The handleRequest method is here:
public APIGatewayProxyResponseEvent handleRequest( final APIGatewayProxyRequestEvent input, final Context context)
{
System.out.println("Enter handleRequest...");
Map<String, String> headers = new HashMap<>();
headers.put("Accept", "application/json");
headers.put("Content-Type", "application/x-www-form-urlencoded");
headers.put("Access-Control-Allow-Origin", "*");
headers.put("Access-Control-Allow-Methods", "GET, POST, PUT, DELETE, OPTIONS");
headers.put("Access-Control-Allow-Headers", "origin, content-type, accept, x-requested-with");
headers.put("Access-Control-Max-Age", "3600");
APIGatewayProxyResponseEvent response = new APIGatewayProxyResponseEvent().withHeaders(headers);
String bodyText="";
if(input!=null) {
bodyText+="The value of input is: "+input;
Map<String, String> params= input.getQueryStringParameters();
if(params!=null) {
String enteredName=params.get("name");
PayRollIdExtraction px=new PayRollIdExtraction();
int output=px.execute(enteredName);
return response.withStatusCode(200).
withBody("The person id of "+enteredName+ " is: "+Integer.toString(output));
}else {
return response.withStatusCode(500)
.withBody(bodyText+" And The name paramter is null");
}
}else {
return response.withStatusCode(500)
.withBody("The input can't be null");
}
}
When I'm entering correct names as query string parameter like below:
http://apigatewayurl/?name=Timothy Hull
I'm getting the following response:
"statusCode":500,"headers":{"Accept":"application/json","Access-Control-Allow-Origin":"*","Access-Control-Allow-Methods":"GET, POST, PUT, DELETE, OPTIONS","Access-Control-Max-Age":"3600","Access-Control-Allow-Headers":"origin, content-type, accept, x-requested-with","Content-Type":"application/x-www-form-urlencoded"},"body":"The value of input is: {} And The name paramter is null"}
I'm clueless as why the input is printed as empty {} as I have clearly entered the correct available name? Can anyone please guide me? Let me know if any information is missing?

Stream response to downstream clients in SpringBoot

I have a controller proxy api endpoint where it receives different request payloads which are intended to different services. This controller validates payload and adds few headers based on certain rules. In this current context, i do not want to parse the received response from upstream services. proxy method should simply stream response to downstream clients so that it can scale well without going into any memory issues when dealing with large response payloads.
I have implemented method like this:
suspend fun proxyRequest(
url: String,
request: ServerHttpRequest,
customHeaders: HttpHeaders = HttpHeaders.EMPTY,
): ResponseEntity<String>? {
val modifiedReqHeaders = getHeadersWithoutOrigin(request, customHeaders)
val uri = URI.create(url)
val webClient = proxyClient.method(request.method!!)
.uri(uri)
.body(request.body)
modifiedReqHeaders.forEach {
val list = it.value.iterator().asSequence().toList()
val ar: Array<String> = list.toTypedArray()
#Suppress("SpreadOperator")
webClient.header(it.key, *ar)
}
return webClient.exchangeToMono { res ->
res.bodyToMono(String::class.java).map { b -> ResponseEntity.status(res.statusCode()).body(b) }
}.awaitFirstOrNull()
}
But this doesn't seems to be streaming. When i try to download large file, it is complaining failed to hold large data buffer.
Can someone help me in writing reactive streamed approach?
This is what i have done finally.
suspend fun proxyRequest(
url: String,
request: ServerHttpRequest,
response: ServerHttpResponse,
customHeaders: HttpHeaders = HttpHeaders.EMPTY,
): Void? {
val modifiedReqHeaders = getHeadersWithoutOrigin(request, customHeaders)
val uri = URI.create(url)
val webClient = proxyClient.method(request.method!!)
.uri(uri)
.body(request.body)
modifiedReqHeaders.forEach {
val list = it.value.iterator().asSequence().toList()
val ar: Array<String> = list.toTypedArray()
#Suppress("SpreadOperator")
webClient.header(it.key, *ar)
}
val respEntity = webClient
.retrieve()
.toEntityFlux<DataBuffer>()
.awaitSingle()
response.apply {
headers.putAll(respEntity.headers)
statusCode = respEntity.statusCode
}
return response.writeWith(respEntity.body ?: Flux.empty()).awaitFirstOrNull()
}
Let me know if this is truly sending data downstream and flushing?
Your first code snippet fails with memory issues because it is buffering in memory the whole response body as a String and forwards it after. If the response is quite large, you might fill the entire available memory.
The second approach also fails because instead of returning the entire Flux<DataBuffer> (so the entire response as buffers), you're only returning the first one. This fails because the response is incomplete.
Even if you manage to fix this particular issue, there are many other things to pay attention to:
it seems you're not returning the original response headers, effectively changing the response content type
you should not forward all the incoming response headers, as some of them are really up to the server (like transfer encoding)
what happens with security-related request/response headers?
how are you handling tracing and metrics?
You could take a look at the Spring Cloud Gateway project, which handles a lot of those subtleties and let you manipulate requests/responses.

HttpMessageNotReadableException: Required request body is missing (Occasional) - SpringMVC Default Controller Request Mapping HTTP Method?

I have the following code which represents an Ajax POST request:
#RequestMapping("/participant/insertEvent")
public boolean insertEvent(Principal principal, #RequestBody String json, HttpServletRequest request) throws Exception {
//...
//return true or false
}
JS:
$.ajax({
type : "post",
dataType : "json",
url : '/app/participant/insertEvent',
data : JSON.stringify({'p1': p1, 'p2' : p2})
});
The app is deployed in Production and everything has been working with many users the whole week.
This morning I got the following production issue at this method (insertEvent):
org.springframework.http.converter.HttpMessageNotReadableException: Required request body is missing: public boolean
app.controller.participant.AjaxOperationsController.insertEvent(java.security.Principal,java.lang.String,javax.servlet.http.HttpServletRequest) throws java.lang.Exception at
org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.readWithMessageConverters(RequestResponseBodyMethodProcessor.java:161) at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.resolveArgument(RequestResponseBodyMethodProcessor.java:130) at org.springframework.web.method.support.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:124) at
org.springframework.web.method.support.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:161) at ...
I'm wondering, is it because I didn't explicitly specify this is a method={RequestMethod.POST} ?
But if I don't specify an HTTP method, what is the default?
However, if this is the issue, then why does the app work 99% of the time, and not for that one user?
FYI, the error was caused by including in the JSON an un-encoded % (percent) special char coming from a TextArea value.

Kestrel server slow on "bad request data"

I have an IOT device (black box, can't reprogram it) that sends http POST requests (136 bytes of JSON, a string) over wired ethernet to my .NET core 2.2 very simple server console application.
I just output the string to the console.
[HttpPost]
public void Post([FromBody] RootObject root)
{
string adv = root.prt;
Console.WriteLine(adv);
}
I get to display less than 1 line per second, sometimes 2.
Using Fiddler as a reverse proxy, instead I receive between 5 and 10 http req per second, that is the correct behavior of the device.
So I enabled Information logs, and get this, every second, with that "Invalid request line" error:
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
Request starting HTTP/1.1 POST http://192.168.0.92/api/values application/json 136
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
Executing endpoint 'BluepycWeb.Controllers.ValuesController.Post (BluepycWeb)'
info: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[1]
Route matched with {action = "Post", controller = "Values"}. Executing action BluepycWeb.Controllers.ValuesController.Post (BluepycWeb)
02003F002293831000010033FF0006EFAA256B6D1A001E010201061AFF4C000215476C6F62616C2D54616700000000000000000000CD0101CC0001FF001FCC
info: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[1]
Executing action method BluepycWeb.Controllers.ValuesController.Post (BluepycWeb) with arguments (BluepycWeb.Controllers.RootObject) - Validation state: Valid
info: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[2]
Executed action method BluepycWeb.Controllers.ValuesController.Post (BluepycWeb), returned result Microsoft.AspNetCore.Mvc.EmptyResult in 0.2158ms.
info: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[2]
Executed action BluepycWeb.Controllers.ValuesController.Post (BluepycWeb) in 9.4032ms
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
Executed endpoint 'BluepycWeb.Controllers.ValuesController.Post (BluepycWeb)'
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
Request finished in 24.5625ms 200
info: Microsoft.AspNetCore.Server.Kestrel[17]
Connection id "0HLMO99UNQBGM" bad request data: "Invalid request line: '\x0D\x0A'"
Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Invalid request line: '\x0D\x0A'
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpParser`1.RejectRequestLine(Byte* requestLine, Int32 length)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpParser`1.GetUnknownMethod(Byte* data, Int32 length, Int32& methodLength)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpParser`1.ParseRequestLine(TRequestHandler handler, Byte* data, Int32 length)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpParser`1.ParseRequestLine(TRequestHandler handler, ReadOnlySequence`1& buffer, SequencePosition& consumed, SequencePosition& examined)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpParser`1.Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.IHttpParser<TRequestHandler>.ParseRequestLine(TRequestHandler handler, ReadOnlySequence`1& buffer, SequencePosition& consumed, SequencePosition& examined)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection.TakeStartLine(ReadOnlySequence`1 buffer, SequencePosition& consumed, SequencePosition& examined)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection.ParseRequest(ReadOnlySequence`1 buffer, SequencePosition& consumed, SequencePosition& examined)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection.TryParseRequest(ReadResult result, Boolean& endConnection)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequestsAsync[TContext](IHttpApplication`1 application)
If I send the same JSON paylod through Postman continuosly, no errors, no delays.
That error, that I don't know how to avoid nor by what is originated (but it seems it doesn't distub Fiddler) seems to hang my server for a second.
Any suggestion on how can I solve this problem?
Discard/correct the error?
Keep the error but not slowing down the data receive?
UPDATE:
Hosting with IISExpress, no problem, no "Invalid request line: '\x0D\x0A'" error, very fast.
Problem is only using Kestrel.
Thanks!
Yesterday I had same issue with my ASP .NETCore service.
Then realized I was sending HTTPS requests to my Kestrel server while it isn't configured to listen at HTTPS protocol, that's why Kestrel complaining.
If you want to use HTTPS in your kestrel you have multiple options to configure it, for example :
With assigned server certificate :
var host = new WebHostBuilder()
.UseKestrel(options =>
options.Listen(IPAddress.Any, 443, listenOptions =>
listenOptions.UseHttps("MyCert.pfx")))
With multiple hostnames and certificates :
public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
WebHost.CreateDefaultBuilder(args)
.UseStartup<Startup>()
.ConfigureKestrel((context, options) =>
{
options.ListenAnyIP(5005, listenOptions =>
{
listenOptions.UseHttps(httpsOptions =>
{
var localhostCert = CertificateLoader.LoadFromStoreCert(
"localhost", "My", StoreLocation.CurrentUser,
allowInvalid: true);
var exampleCert = CertificateLoader.LoadFromStoreCert(
"example.com", "My", StoreLocation.CurrentUser,
allowInvalid: true);
var subExampleCert = CertificateLoader.LoadFromStoreCert(
"sub.example.com", "My", StoreLocation.CurrentUser,
allowInvalid: true);
var certs = new Dictionary<string, X509Certificate2>(
StringComparer.OrdinalIgnoreCase);
certs["localhost"] = localhostCert;
certs["example.com"] = exampleCert;
certs["sub.example.com"] = subExampleCert;
httpsOptions.ServerCertificateSelector = (connectionContext, name) =>
{
if (name != null && certs.TryGetValue(name, out var cert))
{
return cert;
}
return exampleCert;
};
});
});
});

WebRequest time out error

I tried to build an application in Mono using the setting "Mono 2.10.2", but the the following errors.
Unhandled Exception: System.Net.WebException: The request timed out
at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) [0x00046] in C:\cygwin\tmp\monobuild\build\BUILD\mono-2.6.7\mcs\class\System\System.Net\HttpWebRequest.cs:813
at System.Net.HttpWebRequest.GetResponse () [0x0000e] in C:\cygwin\tmp\monobuild\build\BUILD\mono-2.6.7\mcs\class\System\System.Net\HttpWebRequest.cs:825
The bit of code causing this issue is:
string url = "http://webFarm/config.prop";
WebRequest req = WebRequest.Create(url);
WebResponse resp = req.GetResponse();
It failed immediately. But if I switch the setting to "Default Runtime" instead of "Mono 2.10.2", everything is OK. Could anyone help me with this?
Regards,
David
I'm also getting this error using the code:
HttpWebRequest objRequest =
(HttpWebRequest)WebRequest.Create("https://api-3t.sandbox.paypal.com/nvp");
objRequest.Timeout =15000;
objRequest.Method = "POST";
objRequest.ContentLength = strPost.Length;
using (StreamWriter myWriter = new
StreamWriter(objRequest.GetRequestStream()))
myWriter.Write(strPost);
causes exception at GetRequestStream() :
System.Net.WebException: The request timed out
at System.Net.HttpWebRequest.GetRequestStream () [0x0005f] in
/usr/src/redhat/BUILD/mono-2.10.2/mcs/class/System/System.Net/HttpWebRequest.cs:752
at NVPAPICaller.HttpCall (System.String NvpRequest) [0x00000] in <filename
unknown>:0
--------------------------------------------------------------------------------
Version information: Mono Runtime Version: 2.10.2 (tarball Mon Apr 18
18:57:39 UTC 2011); ASP.NET Version: 2.0.50727.1433
You should revert back to older Mono 2.10 , 2.10.1 will probably work.

Resources