Getting the request body of search response with 2.X NEST client - elasticsearch

I'm using the new 2.X NEST client. That part is important, because there were a great many breaking changes which will effect potential answers here.
Previously, I used the Glimpse Elasticsearch plugin to see the underlying queries being generated by NEST. However, it would appear that that plugin is no longer compatible with 2.X NEST. As a result, I'm trying to find a workaround to see the JSON query. The problem here is that the old way of accessing response.RequestInformation to get at the request body is gone. It seems to have been replaced with a combination of ApiCall, CallDetails, and DebugInformation. The problem here is that in all of these the request byte array is null unless you add .DisableDirectStreaming() to the ConnectionSettings instance you pass into ElasticClient. The problem there is that I'm handling all that using dependency injection with Ninject, so in something like a controller action, I have no access to the ConnectionSettings instance to make such a change. I suppose I could just add .DisableDirectStreaming() globally, but I have no idea what the potential consequences of that is and the documentation on this is frustratingly sparse.
So, there's a few avenues for an answer here, any of which I'll accept. First, if anyone has manage to get the Glimpse plugin functioning with 2.X, I'd love to know what you did. However, based on the fact that the underlying API has changed dramatically, my assumption is that the plugin is fundamentally broken until someone branches it for 2.X or Elastic comes out with their own version (which is supposedly coming at some undetermined point in the future).
Second, if there's some way to get at the request body without disabling direct streaming, and I simply missed it. I'd appreciate guidance there.
Third, if anyone has any ideas for how I can disable direct streaming at the controller action level, without affecting my Ninject setup or applying it globally, feel free to chime in.
Finally, it would be great if someone from the Elastic team can enlighten me to the effects of disabling direct streaming and what potential problems can arise from that, so I can make a determination about whether it would be wise to apply it globally or not.

With .DisableDirectStreaming() set to true, the request bytes and response bytes are buffered in memory streams to enable them to be available on the response via response.RequestBodyInBytes and response.ResponseBodyInBytes, respectively.
By default, it is set to false so the request type e.g. SearchDescriptor<T>, SearchRequest<T>, etc. is serialized directly to the request stream of the http request and similarly, the response type is deserialized from the response stream. The overhead of setting it to true is therefore keeping the request and response bytes around in memory for the lifetime of the response (and GC kicking in).
With Connection Settings, it's best to have one instance for the lifetime of the application; Serialization settings are cached per connection settings as well as caches for field and property expressions. There's no way currently to keep the request and response bytes around on a per request basis i.e. ad-hoc introspection, but I think this would be a useful addition; I'll add an issue for it :)
I'm not personally overly familiar with the Glimpse integration but I would expect it would require updating to work with NEST 2.x because of some of the changes. Having just given it a brief look, the changes look pretty straightforward. Looks like this can be done without having to set .DisableDirectStreaming() to true, but only grabbing the request bytes before they're written to the request stream.

Related

Any way to handle Stream Json with RestTemplate?

Is there any way to handle application/stream+json content with the old fashioned RestTemplate the way webClient does?
As far as my attempts go, wrapping the results of something like restTemplate.getForEntity in Flux.just(<convert response entity to mono here>) would just return the first element and stop at that, while webClient handles it properly, populating that resulting json with new entries as they appear. Haven't tried working with inputStream yet, but at first glance it doesn't seem to be what i need, despite having a "stream" in its name.
Unfortunately, using webClient would be a rather costly option in this case (still waiting for https://github.com/spring-projects/spring-security/issues/4921). I'd rather implement things that would "soon" appear officially only if there's absolutely no other way.
RestTemplate is exposing an API which is not meant to be used to stream the HTTP response. The underlying HTTP response is read and closed after each call, whereas the "application/stream+json" media type is meant for streaming responses.
I don't see any way to properly handle this use case (reading streaming responses) with RestTemplate, by design (check out SPR-14882 for another example of that). So you'll probably have to wait for that issue to be resolved or use another HTTP client with such features.

Why is ServiceStack caching in Service, not FilterAttribute?

In MVC and most other service frameworks I tried, caching is done via attribute/filter, either on the controller/action or request, and can be controlled through caching profile in config file. It seems offer more flexibility and also leave the core service code cleaner.
But ServiceStack has it inside the service. Are there any reason why it's done this way?
Can I add a CacheFilterAttribute, but delegate to service instead?
ToOptimizedResultUsingCache(base.Cache,cacheKey,()=> {
// Delegate to Request/Service being decorated?
});
I searched around but couldn't find an answer. Granted, it probably won't make much difference because the ServiceStack caching via delegate method is quite clean. And you seldom change caching strategy on the fly in real world. So this is mostly out of curiosity. Thanks.
Because the caching pattern involves, checking first to see if it is cached, if not to then execute the service, populate the cache, then return the result.
A Request Filter doesn't allow you to execute the service and a Response Filter means that the Service will always execute (i.e. mitigating the usefulness of the Cache), so the alternative would require a Request + Response filter combination where the logic would be split into 2 disjointed parts. Having it inside the Service, lets you see and reason about how it works and what exactly is going on, it also allows full access to calculate the uniqueHashKey used and exactly what and when (or even if) to Cache, which is harder to control with a generic black-box caching solution.
Although we are open to 'baking-in' built-in generic caching solutions (either via an attribute or ServiceRunner / base class). Add a feature request if you'd like to see this, specifying the preferred functionality/use-case (e.g. cache based on Time / Validity / Cache against user-defined Aggregate root / etc).

IBM JAX-RS 1.1 handling Dojo Ajax OPTIONS request

I am currently experimenting with IBM JAX-RS capability through WAS 7 and Web 2.0 Feature Pack 1.1.0.0. The problem I am running into is the client Dojo ajax call poses CORS situation and I am basically trying to figure out how I could properly handle the OPTIONS request.
I followed the example in the referenced url (although it's for WAS 8, it should not affect the part I am playing with); however, it is reported that the system cannot find any method that supports OPTIONS.
Any help will be appreciated!
Reference URL: http://www14.software.ibm.com/webapp/wsbroker/redirect?version=matt&product=was-base-iseries&topic=twbs_jaxrs_wadl_serving
Although it's less than a day, since I did not see any response, I want to post out the temp solution I pieced so far.
First of all, check out below link from Mozilla for a better understanding of CORS and OPTIONS request (it did the trick for me).
Link: https://developer.mozilla.org/en-US/docs/HTTP_access_control
Next, one update on the problem of IBM JAX-RS handling OPTIONS. It appears OPTIONS request is handled no different than other GET/POST, it requires an exact match of #Path value. This is not really desired, and in my case, I tried wild card in #Path value, that did not do the trick of triggering the system to handle OPTIONS.
Now, for the solution. With no surprise, I end up using a filter to forward the OPTIONS request to a method inside the resource annotated with #OPTIONS and fixed #Path value, e.g. #Path("options").
In filter, I am handling Allow-Origin and Allow-Header, then I left the resource's #OPTION method to deal with Allow-Methods.
It is very simple and basic solution. Along with proper checking inside filter, the header does not need to be added excessively for every request.
One catch here, it works for Chrome (Version 23.0.1271.95 m) and Mozilla (9.0.1), but not IE. I am still trying to figure out how IE is responding to these CORS. Hope this helps.

ProtocolViolationException Load testing web service (GET action with content-body)

I created an ASP.NET MVC4 Web API service (REST) with a single GET action. The action currently needs 11 input values, so rather than passing all of those values in the URL, I opted to encapsulate those values into a single class type and have it passed as Content-Body. When I test in Fiddler, I specify the verb as GET, and enter the JSON text in the "Request Body" input box. This works great!
The problem is when I attempt to perform Load Testing in Visual Studio 2010 Ultimate. I am able to specify the GET action and the JSON Content-Body just fine. But when I run the Load test, VS reports exceptions of type ProtocolViolationException (Cannot send a content-body with this verb-type) in the test results. The test executes in 1ms so I suspect the exceptions are causing the test to immediately abort. What can I do to avoid those exceptions? I'd prefer to not change my API to use URL arguments just to work-around the test tooling. If I should change the API for other reasons, let me know. Thanks!
I found it easier to put this answer rather than carry on the discussions.
Sending content with GET is not defined in RFC 2616 yet it has not been prohibited. So as far as the spec is concerned we are in a territory that we have to make our judgement.
GET is canonically used to get a resource. So you are retrieving this resource using this verb with the parameters you are sending. Since GET is both safe and idempotent, it is ideal for caching. Caching usually takes place based on the resource URI - and sometimes based on various headers. The point is cache implementations - AFAIK - would not use the GET content (and to be honest I have not seen any GET with content in real world). And it would not make sense to include the content in the key generation since it reduces the scalability of the caches.
If you have parameters to send, they must be in the URI since this is part of what defines that URI. As such, I strongly believe sending content with GET is wrong.
Even when you look at implementations such as OData, they put the criteria in the URI. I cannot imagine your (or any) applications requirements is beyond OData query requirements.

How can I implement a content converter in Firefox for all page elements?

I'm attempting to port over an Internet Explorer plugin to Firefox, but I'm not sure where to look for what I need.
Basically I need to be able to filter all content that is received by the browser with a certain Content-Type header. I tried implementing a stream converter, and this works, but only for the top-level document in the page, frame, or iframe. I had the same problem with IE, and getting around it was really hacky, and since I would ideally like this to be cross platform I would really like to be able to do this in Firefox without resorting to vtable hacks.
The content is served compressed with a proprietary compression format. So I need to receive the data, decompress it, and change the Content-Type back to what the original uncompressed file should have.
If there is a way to just filter all data received, that would probably be acceptable, I could handle parsing the header myself.
Thanks
I think I may have found what I needed. I came across this link which is used for tracing HTTP calls: http://blues.ath.cx/firekeeper/resources/http_tracer.html
There seems to be some problems with the JavaScript implementation for some reason, and I'm not a JavaScript guru to figure it out, but I've implemented it in C++ and initial results suggest that I should be able to modify it for my needs.
Basically we're replacing the nsIHttpProtocolHandler service with our own implementation, which keeps a reference to the initial implementation. When a call is made to the service, we just proxy it over to the saved original implementation. Then we provide our own implementation of nsIHttpChannel and nsIStreamListener which we use as proxies too.
Again we proxy most of the calls back off to the original handlers. But in OnDataAvailable, instead of passing the data on to the underlying nsIStreamListener, we save it using nsIStorageStream. Then in OnStopRequest, after we've gotten all of the data, we can decompress it and then call OnDataAvailable on the original handler, followed by OnStopRequest.
It has worked on some small simple tests so far, but I'll have to put it through some more rigorous tests... I'll also have to figure out if I can do the same thing with HTTPS.
The biggest problem I see at the moment is that it relies on some unfrozen interfaces such as nsIHttpChannelInternal. Can't be helped though as far as I can tell, and my version compatibility requirements are pretty small, so I can live with it if I have to.
In the meantime, if anybody has any other suggestions, I'm all ears :D

Resources