I'd like to log request and response body from incoming traffic to each my microservice.
Is it possible in Istio (Envoy) out-of-the-box?
I don't see body attribute for mapping in Mixer's EntryLog.
Maybe it will be added in future version of Istio?
Of course I can achieve this by implementing my own filter in microservise, but maybe there is better solution to achieve this.
If I understand your question correctly, then you should check out this documentation of Lua filters.
https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/lua_filter
https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/lua_filter#script-examples
body = handle:body() should give you the request or response body, depending upon the handle.
handle:logInfo(message) should help you log it.
For me print statement also had worked.
e.g. print(headers["Cache-Control"]) was putting the header value in the log of my app on GCP project's kubernetes cluster.
You need to apply an EnvoyFilter in your kubernetes cluster and in the lua code, you can log the request body.
Also keep in mind that 'The filter should be configured with the name envoy.lua' ONLY
Related
I have deployed an AWS Lambda function, written in Python, and AWS API Gateway structure to cause POST requests to an API endpoint to be redirected to my function. I want to upload a PDF document to my function and have it store the document in a S3 bucket. The problem I have is that the payload of any POST request to my API is being UTF-8 encoded. I don't want that but can't figure out the magic mojo to disable encoding of the request payload.
I am testing using curl, with the following command line:
curl -XPOST https://xxxxxxxxxx.execute-api.us-west-1.amazonaws.com/test -H 'content-type: application/pdf' --data-binary #document.pdf
UPDATE: I just found the following article describing how API Gateway and Lambda support uploading binary data:
https://aws.amazon.com/blogs/compute/handling-binary-data-using-amazon-api-gateway-http-apis/
This article suggests that all of the complexities that I discussed in the initial formation of my question (still provided below) should not be necessary. All I should need to do to upload binary content to my Lambda function is insure that my request includes an appropriate Content-Type header. I was already doing that, but I massaged my Curl command a bit (modified above) to define my request in exactly the way that is done in this article. I still get UTF-8 encoded data and NOT base-64 encoded data. I tried uploading a jpeg file rather than a PDF so I was doing exactly what was done in the article. Still no love. I don't get it. This article demonstrates exactly what I'm doing. But I don't get the result it suggests I should. Ggggrrrr.
ORIGINAL POST:
I am using Terraform to define my deployment. I want to cause the PDF to not be encoded/mangled at all. This is my first time using API Gateway, and I'm obviously missing some bit of config. The one thing I'm doing specifically right now to say that I want incoming payloads to be treated as binary is via the binary_media_types argument to my API definition in Terraform:
resource aws_api_gateway_rest_api proxy {
...
binary_media_types = [
"application/pdf",
"application/octet-stream",
"*/*"
]
This sets the Binary Media Types configuration associated with the API I've defined. I've confirmed via the AWS Console that this setting is having the desired effect...I can see these types in the console. I should need just the first item in the list, but I've added the others while I try to figure out the problem here. By adding that wildcard item, I believe that it shouldn't matter what the incoming Content-Type is...all payloads should be being treated as binary.
The other bit of config that I know about that might be important is the "integration contentHandling property". Here is the key bit of AWS docs that seems to explain all this:
I think the case that applies to me here is the one I've highlighted, per what I say above. This says to me that I shouldn't need to do anything else, per the "unspecified" value in the table for "contentHandling. I've tried setting the "contentHandling" argument on the integration record of my Terraform config, like this:
resource aws_api_gateway_integration proxy {
...
passthrough_behavior = "WHEN_NO_MATCH"
content_handling = "CONVERT_TO_BINARY"
}
I first tried only specifying the content_handling value. I've also tried setting that value to "CONVERT_TO_TEXT", hoping to then get base64-encoded data. Neither of these has any effect. I've tried adding the passthrough_behavior value as shown. I've also tried replacing "WHEN_NO_MATCH" with "WHEN_NO_TEMPLATES". Nothing I do changes the behavior. I haven't been able to figure out where these settings would show up in the AWS console. If I knew they were necessary, I'd explore this further. But I don't think I need to set these.
What am I missing? How can I POST a PDF document to my AWS Lambda function through API Gateway and have the payload of the request not be converted in any way? TIA!
NOTE: I am aware of this Q/A: PDF Uploaded via AWS API Gateway getting corrupted. The answer there doesn't apply to me, as I need to avoid having to form-encode the upload. The client code that will eventually be doing the upload is set in stone and sends a POST request with a payload that is just the bytes of the PDF.
Good morning. I am new to logic apps and I am trying to figure out how I can trigger the execution based on a GET URL with three parameters. All the examples I've found on Google show the URL being generated once the JSON and relative path is entered, but that's not happening for me. Perhaps it's because I am creating the logic app in VS.
Here's what my "When a HTTP request is received" step looks like in the logic app.
I also tried removing the JSON and just using the parameters to pass the values to the function, as shown below. I'm just not sure the best way to do this.
All I really need to do is get the three parameters into the logic app so I can perform a function call with the parameters. Any suggestions would be greatly appreciated!
Why is my HTTP request URL not being generated for a Logic App?
You need to click save, and then the url will be automatically generated for the When a HTTP request is received trigger:
You can use this expression to accept values through GET parameters:
triggerOutputs()['queries']['parameter-name']
For example:
Noteļ¼
Queries need to pass parameters in the form of json.
I am trying to modify an API GW config similar to what is found here https://oozio.medium.com/serverless-discord-bot-55f95f26f743
What I need to do is pass to lambda a custom static HTTP Header. I have defined in APIGW -> /{resource} -> Integration Request -> HTTP Headers: a header like {"Name": "application", "Mapped from": "'discord'", "Caching": false} (with single ticks as noted in the documentation for a static value)
This header never appears in the list of headers from the mapping template. I've spent the better part of a day trying to solve this.
Simplified question:
How do you access a static HTTP Header defined in Integration Request, in the Mapping Template?
TIA
After working with AWS support, this is apparently not possible.
Many Thanks for your kind patience. After our call I started to look for a possible
solution to fetch the statically set HTTP header in the mapping template, but
unfortunately could not find any solution.
I then discussed this scenario with my colleagues and also reached out to the
back-end API Gateway Service Team. It seems that this is something that is not
possible currently.
We cannot access the static header value defined in the integration request, in
the request body mapping template. If your use case needs to send the static header
in body for non proxy integration then it is recommended that you may define it
statically in the body mapping template directly.
I understand as you explained to me how you need to access the header in the mapping
template but unfortunately this does not seem to have a solution. Although the
statically set header is seen to be passed in the API Gateway logs, there is no
mechanism in the lambda service to fetch these headers.
Currently, I can ask the Service Team to consider the same as a feature request but
I wont be having a ETA available and also would need a solid reasoning since the
same can be done in the mapping template.
The workaround at this time (March 2021) has to be expressing it directly in the mapping template itself, or using Proxy for Lambda.
The following was provided as justification for the feature request.
When creating APIGW via a templating process (Such as terraform), being able to
express 'API Wide' variables is a useful tool for sending contextual information
to a backend integration. For example, a Lambda that can process requests from
multiple sources, but it needs to know which source should be used. While the
value can be statically expressed in the mapping template, having it be a
variable as part of the API allows for better visibility and management as its
an interact-able component, where as the mapping template is effectively a
complex string. This also does not need to be a header, it could simply be
"integration variables" that allow you to define simple k:v pairs that are either
passed as part of context, or accessible in $context within the mapping template.
While statically expressing it in the mapping template is a workaround, it is
only that.
I have used AMQP Publisher to publish the message in RabbitMQ then I use AMQP Consumer as listener. In the View Results Tree the messages from the queue in shown in the request tab of AMQP Consumer. My question is how to extract data from that request. I tried following the Bean Shell Post Processor but it seems it will only work on Http request. I tried to use JSR223 Post Processor and XPath extractor but it doesn't work as well. Any help?
I wanted to extract the documentId from the request. Here is the Request pattern.
I have already tried following links:
Extracting value from jmeter post request
how to extract value from request in Jmeter
How to extract the values from request input xml in jmeter
The statement that you tried something without sharing the code doesn't make sense
Posting JSON data or code as image is not the best idea
Any reason to extract data from the request? Normally people know everything about the request hence don't require to extract anything from it. Even if they do - they should normally able to store the request data into a JMeter Variable and apply the relevant Post-Processor to it.
Whatever, just in case here is the solution:
Add JSR223 PostProcessor (if you really want to do this using the Post-Processor) as a child of the request
Put the following code into "Script" area:
vars.put('foo', com.jayway.jsonpath.JsonPath.read(sampler.getArguments().getArgument(0).value,'$..documentId')[0])
That's it, you should be able to access the extracted value as ${foo} where required.
References:
JsonPath: Getting Started
Apache Groovy - Why and How You Should Use It
I'm using the new 2.X NEST client. That part is important, because there were a great many breaking changes which will effect potential answers here.
Previously, I used the Glimpse Elasticsearch plugin to see the underlying queries being generated by NEST. However, it would appear that that plugin is no longer compatible with 2.X NEST. As a result, I'm trying to find a workaround to see the JSON query. The problem here is that the old way of accessing response.RequestInformation to get at the request body is gone. It seems to have been replaced with a combination of ApiCall, CallDetails, and DebugInformation. The problem here is that in all of these the request byte array is null unless you add .DisableDirectStreaming() to the ConnectionSettings instance you pass into ElasticClient. The problem there is that I'm handling all that using dependency injection with Ninject, so in something like a controller action, I have no access to the ConnectionSettings instance to make such a change. I suppose I could just add .DisableDirectStreaming() globally, but I have no idea what the potential consequences of that is and the documentation on this is frustratingly sparse.
So, there's a few avenues for an answer here, any of which I'll accept. First, if anyone has manage to get the Glimpse plugin functioning with 2.X, I'd love to know what you did. However, based on the fact that the underlying API has changed dramatically, my assumption is that the plugin is fundamentally broken until someone branches it for 2.X or Elastic comes out with their own version (which is supposedly coming at some undetermined point in the future).
Second, if there's some way to get at the request body without disabling direct streaming, and I simply missed it. I'd appreciate guidance there.
Third, if anyone has any ideas for how I can disable direct streaming at the controller action level, without affecting my Ninject setup or applying it globally, feel free to chime in.
Finally, it would be great if someone from the Elastic team can enlighten me to the effects of disabling direct streaming and what potential problems can arise from that, so I can make a determination about whether it would be wise to apply it globally or not.
With .DisableDirectStreaming() set to true, the request bytes and response bytes are buffered in memory streams to enable them to be available on the response via response.RequestBodyInBytes and response.ResponseBodyInBytes, respectively.
By default, it is set to false so the request type e.g. SearchDescriptor<T>, SearchRequest<T>, etc. is serialized directly to the request stream of the http request and similarly, the response type is deserialized from the response stream. The overhead of setting it to true is therefore keeping the request and response bytes around in memory for the lifetime of the response (and GC kicking in).
With Connection Settings, it's best to have one instance for the lifetime of the application; Serialization settings are cached per connection settings as well as caches for field and property expressions. There's no way currently to keep the request and response bytes around on a per request basis i.e. ad-hoc introspection, but I think this would be a useful addition; I'll add an issue for it :)
I'm not personally overly familiar with the Glimpse integration but I would expect it would require updating to work with NEST 2.x because of some of the changes. Having just given it a brief look, the changes look pretty straightforward. Looks like this can be done without having to set .DisableDirectStreaming() to true, but only grabbing the request bytes before they're written to the request stream.