Envoy external processing filter: issues with Content-Length Header when processing request body - go

I'm having a bit of a struggle with using the External Processing filter for Envoy (described HERE. I'm currently using a similar setup to THIS EXAMPLE , where I want to do some processing on both the request and response bodies being sent to an upstream service using a GRPC server in Go.
The issue I'm facing is that when I try to make changes to the request body, meaning using a BodyMutation to send a new body (basically the same JSON but with one field altered as an example) the Content-Length header is missing when the request is forwarded to the upstream service, which causes an error. Yet if I don't alter the body (so no BodyMutation) the header is there as it should be.
Basically, if I try to set the processing response for the request body like so:
bytesToSend := b.RequestBody.Body
resp = &pb.ProcessingResponse{
Response: &pb.ProcessingResponse_RequestBody{
RequestBody: &pb.BodyResponse{
Response: &pb.CommonResponse{
HeaderMutation: &pb.HeaderMutation{
SetHeaders: []*core.HeaderValueOption{
{
Header: &core.HeaderValue{
Key: "Content-Length",
Value: strconv.Itoa(len(bytesToSend)),
},
},
},
},
BodyMutation: &pb.BodyMutation{
Mutation: &pb.BodyMutation_Body{
Body: bytesToSend,
},
},
},
},
},
ModeOverride: &v3.ProcessingMode{
ResponseHeaderMode: v3.ProcessingMode_SEND,
ResponseBodyMode: v3.ProcessingMode_NONE,
},
}
The Content-Length is removed completely. But if I don't set the BodyMutation, like so:
bytesToSend := b.RequestBody.Body
resp = &pb.ProcessingResponse{
Response: &pb.ProcessingResponse_RequestBody{
RequestBody: &pb.BodyResponse{
Response: &pb.CommonResponse{
HeaderMutation: &pb.HeaderMutation{
SetHeaders: []*core.HeaderValueOption{
{
Header: &core.HeaderValue{
Key: "Content-Length",
Value: strconv.Itoa(len(bytesToSend)),
},
},
},
},
},
},
},
ModeOverride: &v3.ProcessingMode{
ResponseHeaderMode: v3.ProcessingMode_SEND,
ResponseBodyMode: v3.ProcessingMode_NONE,
},
}
Then the Content-Length is present, and the HeaderMutation does set it to whatever I set it to (in the example it's just the length of the body again).
I'm not entirely clear on the behaviour here. Is there something in particular I should be looking out for, either in this code or Envoy's configuration?
Any pointers would be greatly appreciated.
EDIT: Seems like the behaviour here is: when the body is mutated, it switches to chunked transfer encoding, therefore removing the content length. Makes sense, but then if the upstream service requires a content length it causes issues. The Header Mutation is ignored, so computing the content length manually and sending it doesn't seem to work.

Related

Get more than one header using the metadataHeaders[] query parameter GmailAPI

When I make a GET http request for the metadataHeaders that only requests one of them like so:
https://gmail.googleapis.com/gmail/v1/users/me/messages/${messageId}?$format=metadata&metadataHeaders=From
it works just fine. But my question is how do I go about sending the array of headers that I want [to, from, subject], in my request? So basically, how would I restructure my metadataHeaders query parameter... found here ->
https://gmail.googleapis.com/gmail/v1/users/me/messages/${messageId}?$format=metadata&metadataHeaders=From
to also contain From, To, & Subject
I have been trying to figure out how to get these headers for quite a while to no avail. I tried looking at the documentation ( https://developers.google.com/gmail/api/reference/rest/v1/users.messages/get ) but although I know its possible thanks to it, I can't seem to find out how to implement it in my http request. I also tried looking through the stack overflow responses to similar questions but many weren't really useful at all since many of the questions were different from mine, using the oauth library, or in a programming language. All I care for is how to make the http request.
Headers is just an array So you can just add it more then once
metadataHeaders=from&metadataHeaders=to&metadataHeaders=subject
Request:
GET https://gmail.googleapis.com/gmail/v1/users/me/messages/185cf8d12166fc7a?format=metadata&metadataHeaders=from&metadataHeaders=to&key=[YOUR_API_KEY] HTTP/1.1
Authorization: Bearer [YOUR_ACCESS_TOKEN]
Accept: application/json
Resonse:
"payload": {
"partId": "",
"headers": [
{
"name": "From",
"value": "[REDACTED]"
},
{
"name": "To",
"value": "[REDACTED]#gmail.com"
},
{
"name": "Subject",
"value": "Security alert"
},
]
},

extract http request response and proceed to next http request by using while controller

I want to use while controller to keep sending GET HTTP Request and only proceed to next POST HTTP Request when the GET response "model_name": "Model-Test-20220221-0001"
POST HTTP Request - paramter model_name = Model-Test-${variable}-${counter}
While Controller
GET HTTP Request. sample GET response at below
Json Extractor
POST HTTP Request only when While Controller found the exact model_name.
JSON Extractor:
Names of created variable: modelname
Json path expressions: I have tried this but failed
$..valid_model_list[?(#.model_name = ${Model-Test-${variable}-${counter}})]
This is sample HTTP Request response:
{
"message": "success",
"valid_model_list": [
{
"meta_data": {
"corpus_list": [
"test1"
],
"id": "0c36effa244b4f6596d10f9e675303e1",
"sample_rate": 16000,
"split_ratio": {
"test": 40,
"train": 60
},
"model_name": "Model-Test-20220221-0001",
"status": "ok"
},
{
"meta_data": {
"corpus_list": [
"test1"
],
"id": "0c36effa244b4f6596d10f9e675303e1",
"sample_rate": 16000,
"split_ratio": {
"test": 40,
"train": 60
},
"model_name": "Model-Test-20220221-0002",
"status": "ok"
}
}
]
}
}
We're "unsure" as well as we don't know what you're trying to achieve.
JSON Extractor can extract values from the response, it's not possible to use it to extract values from the request, moreover you should "know" what you're sending in the request and extracting values from the response is the essential part of the correlation process
If you're looking for a JSONPath expression extracting the model_name attribute value you're supposed to provide full JSON. There is a Deep Scan operator so you should be able to use something like:
$..model_name
but if there are more than 1 matches you might need to limit the output with Filter Operators
More information:
JsonPath Getting Started
API Testing With JMeter and the JSON Extractor

Azure Logic App throwing 302 Redirect Error having Server=BIG IP in Response Header for HTTP

I am getting Redirect 302 error for HTTP Request in Logic App. I am calling OneIdentityServer to get access token. Then I am calling Rest API passing access token as Header for key Authorization. I am getting 302 Redirect error in response with headers information like Server = BIG IP, Location= /my.policy
The same above request when triggered through Postman or SOAPUI is working fine, I am getting successful response. But the same is failing in Azure Logic App.
I have also implemented the above scenario in function app as well. It is working file when I run the function app code from visual studio using Postman. But when I test the same function app after publishing it to Azure portal, it is giving same error.
It seems like I have the same issue as you. I found one-way that did not work for me but maybe you could give it a shot if it fits your needs?
The solution that I found in a blogpost was first to add the action "Switch" to the logic app flow and then configure Switch to run after the HTTP is both successful and has failed.
Secondly, the Switch action should trigger on the output of the statuscode from the HTTP request.
If the statuscode equals 302 you should make another HTTP request but with URI being the output of the location header from the first HTTP request. This made my logic app result in statuscode 200 but the response for my logic app was that I needed to login to get access to the API.
But maybe it could be worth giving this solution a shot for your logic app?
Here's the link of the blogpost if you need deeper instructions: http://www.alessandromoura.com.br/2018/11/21/dealing-with-http-302-in-logic-apps/
Do you still have issues with this? Here is a screenshot of my http action: HTTP Action that is working.
I have put my URL in an variable since it changing for each pagination. I also found out that to use the authentication token from the first HTTP request I needed to parse the body to be able to access the token, here is the schema I used to parse the HTTP body from the request where you get access token:
{
"properties": {
"access_token": {
"type": "string"
},
"expires_in": {
"type": "string"
},
"expires_on": {
"type": "string"
},
"ext_expires_in": {
"type": "string"
},
"not_before": {
"type": "string"
},
"resource": {
"type": "string"
},
"token_type": {
"type": "string"
}
},
"type": "object"
}

Google Speech API: the requested URL was not found on this server

I am attempting some simple tests on the Google Speech API, and when my server makes a request to this url (below), I get the 404. that's an error response. Not sure why.
https://speech.googleapis.com/v1/speech:recognize?key=[MY_API_KEY]
The body of my request looks like this:
{
"config": {
"languageCode": "en-US",
"encoding": "LINEAR16",
"sampleRateHertz": 16000,
"enableWordTimeOffsets": true,
"speechContexts": [{
"phrases": ["Some", "Helpful", "Phrases"]
}]
},
"audio":{
"uri":"gs://mydomain.com/my_file.mp3"
}
}
And here is the response:
As you can see, that is a valid resource path, unless I'm totally mistaken about something (I'm sure I am): https://cloud.google.com/speech-to-text/docs/reference/rest/v1/speech/recognize
Update 1:, Whenever I try this with the Google API explorer tool, I get this quota exceeded message (even though I have not yet issued a successful request to the API).
{
"error": {
"code": 429,
"message": "Quota exceeded for quota metric 'speech.googleapis.com/default_requests' and limit 'DefaultRequestsPerMinutePerProject' of service 'speech.googleapis.com' for consumer '[MY_API_KEY]'.",
"status": "RESOURCE_EXHAUSTED",
"details": [
{
"#type": "type.googleapis.com/google.rpc.Help",
"links": [
{
"description": "Google developer console API key",
"url": "https://console.developers.google.com/project/[my_project_id]/apiui/credential"
}
]
}
]
}
}
Update 2: Interestingly, I was able to get some 200 ok's using the Restlet client, but even in those cases, the response body is empty (see screenshot below)
I have made a test by using the exact URL and Body content you added to the post, however, I was able to execute the API call correctly.
I noticed that if I add some extra character to the URL, it fails with the same 400 error since it doesn't exist. I would suggest you to verify that the URL of your request doesn't contain a typo and that the client you use is executing the API call correctly. Also, ensure that your calling code is not encoding the url, which could cause issues given the colon : that appears in the url.
I recommend you to perform this test by using the Try this API tool directly or Restlet client which are the ones that I used to replicate this scenario.

Wiremock Standalone with Dynamic response

I have a standalone instance of Wiremock server. The mappings are stored as json files under the mappings folder. I have a POST request that needs to return a dynamic ID(integer) in the response. Is there a way to configure this in the json file?
To make the above examples work, I had to run the standalone jar with the --global-response-templating. Then I saw, for example, {{now}} working which is what I wanted. Not sure if the documentation specifies this -- I tried the always-useful --help.
In WireMock there are a number of response template helper functions for generating random strings. In the below example I'm using the one for generating a UUID, but several other options exist.
Mapping file: dynamic_id.json
{
"request": {
"method": "POST",
"url": "/dynamic_id"
},
"response": {
"headers": {
"Content-Type": "application/json"
},
"status": 200,
"body": "{{randomValue type='UUID'}}",
"transformers": ["response-template"]
}
}
Using an empty POST http://wiremock/dynamic_id will return an id similar to: c2e6bf32-c9a3-45c0-b988-94fad04cc7a2.
Start WireMock:
java -jar wiremock-standalone-2.18.0.jar --port 8181 --verbose --local-response-templating
This seems like a perfect use-case for OpenTable's Wiremock Body Transformer.
It can be easily integrated with the Standalone Server like this:
java -cp "wiremock-body-transformer-1.1.6.jar:wiremock-2.3.1-standalone.jar" com.github.tomakehurst.wiremock.standalone.WireMockServerRunner --verbose --extensions com.opentable.extension.BodyTransformer
And allows you to easily specify a dynamic variable that you would want to match in the response.
Here is an example to get a random integer without having to specify anything in the request, however if you need to match a specific variable in the request to the response, then that is also very doable with this extension and numerous examples can be found in the readme.
{
"request": {
"method": "POST",
"urlPath": "/transform",
},
"response": {
"status": 200,
"body": "{\"randomInteger\": \"$(!RandomInteger)\"}",
"headers": {
"Content-Type": "application/json"
},
"transformers": ["body-transformer"]
}
}
As #Jeff mentions, If you are running it as a stand-alone process, you need to add this flag --global-response-templating. This will apply templating to each and every reponse. However, few of your responses may be jsut plain json with no templating required.
In that case use --local-response-templating. and add this field inside reponse json:
response:{
"transformers": ["response-template"]
}

Resources