Uploading PDF to AWS Lambda via API Gateway mangles the bits...why? - aws-lambda

I have deployed an AWS Lambda function, written in Python, and AWS API Gateway structure to cause POST requests to an API endpoint to be redirected to my function. I want to upload a PDF document to my function and have it store the document in a S3 bucket. The problem I have is that the payload of any POST request to my API is being UTF-8 encoded. I don't want that but can't figure out the magic mojo to disable encoding of the request payload.
I am testing using curl, with the following command line:
curl -XPOST https://xxxxxxxxxx.execute-api.us-west-1.amazonaws.com/test -H 'content-type: application/pdf' --data-binary #document.pdf
UPDATE: I just found the following article describing how API Gateway and Lambda support uploading binary data:
https://aws.amazon.com/blogs/compute/handling-binary-data-using-amazon-api-gateway-http-apis/
This article suggests that all of the complexities that I discussed in the initial formation of my question (still provided below) should not be necessary. All I should need to do to upload binary content to my Lambda function is insure that my request includes an appropriate Content-Type header. I was already doing that, but I massaged my Curl command a bit (modified above) to define my request in exactly the way that is done in this article. I still get UTF-8 encoded data and NOT base-64 encoded data. I tried uploading a jpeg file rather than a PDF so I was doing exactly what was done in the article. Still no love. I don't get it. This article demonstrates exactly what I'm doing. But I don't get the result it suggests I should. Ggggrrrr.
ORIGINAL POST:
I am using Terraform to define my deployment. I want to cause the PDF to not be encoded/mangled at all. This is my first time using API Gateway, and I'm obviously missing some bit of config. The one thing I'm doing specifically right now to say that I want incoming payloads to be treated as binary is via the binary_media_types argument to my API definition in Terraform:
resource aws_api_gateway_rest_api proxy {
...
binary_media_types = [
"application/pdf",
"application/octet-stream",
"*/*"
]
This sets the Binary Media Types configuration associated with the API I've defined. I've confirmed via the AWS Console that this setting is having the desired effect...I can see these types in the console. I should need just the first item in the list, but I've added the others while I try to figure out the problem here. By adding that wildcard item, I believe that it shouldn't matter what the incoming Content-Type is...all payloads should be being treated as binary.
The other bit of config that I know about that might be important is the "integration contentHandling property". Here is the key bit of AWS docs that seems to explain all this:
I think the case that applies to me here is the one I've highlighted, per what I say above. This says to me that I shouldn't need to do anything else, per the "unspecified" value in the table for "contentHandling. I've tried setting the "contentHandling" argument on the integration record of my Terraform config, like this:
resource aws_api_gateway_integration proxy {
...
passthrough_behavior = "WHEN_NO_MATCH"
content_handling = "CONVERT_TO_BINARY"
}
I first tried only specifying the content_handling value. I've also tried setting that value to "CONVERT_TO_TEXT", hoping to then get base64-encoded data. Neither of these has any effect. I've tried adding the passthrough_behavior value as shown. I've also tried replacing "WHEN_NO_MATCH" with "WHEN_NO_TEMPLATES". Nothing I do changes the behavior. I haven't been able to figure out where these settings would show up in the AWS console. If I knew they were necessary, I'd explore this further. But I don't think I need to set these.
What am I missing? How can I POST a PDF document to my AWS Lambda function through API Gateway and have the payload of the request not be converted in any way? TIA!
NOTE: I am aware of this Q/A: PDF Uploaded via AWS API Gateway getting corrupted. The answer there doesn't apply to me, as I need to avoid having to form-encode the upload. The client code that will eventually be doing the upload is set in stone and sends a POST request with a payload that is just the bytes of the PDF.

Related

Structure of amadeus api https

usualy when I use API's I paste the entire url in browser and it print it out json format using json pro extension in chrome. Like this is a lot easier to copy the path of some data and render it to the page.But my problem is I don't know structure of https. I am not sure where I have to insert the key and secret code. In command line I print all data but I cannot get the path of specific data without using json probextension. Help please. Thank you
I recommend to read the following article:
https://developers.amadeus.com/self-service/apis-docs/guides/authorization-262
and later, review the following repository, you could tinker with the curl request here:
https://github.com/amadeus4dev/amadeus-code-examples
https://github.com/amadeus4dev/amadeus-code-examples/blob/master/airline_code_lookup/v1/get/curl/airline_code_lookup.sh

How log request and response body in Istio

I'd like to log request and response body from incoming traffic to each my microservice.
Is it possible in Istio (Envoy) out-of-the-box?
I don't see body attribute for mapping in Mixer's EntryLog.
Maybe it will be added in future version of Istio?
Of course I can achieve this by implementing my own filter in microservise, but maybe there is better solution to achieve this.
If I understand your question correctly, then you should check out this documentation of Lua filters.
https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/lua_filter
https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/lua_filter#script-examples
body = handle:body() should give you the request or response body, depending upon the handle.
handle:logInfo(message) should help you log it.
For me print statement also had worked.
e.g. print(headers["Cache-Control"]) was putting the header value in the log of my app on GCP project's kubernetes cluster.
You need to apply an EnvoyFilter in your kubernetes cluster and in the lua code, you can log the request body.
Also keep in mind that 'The filter should be configured with the name envoy.lua' ONLY

SonarQube REST APIs : Read Metrics for individual projects

My question:
I am using SonarQube version 7.1 and trying to extract the metrics and quality gate related to individual projects.
What we have tried
We were using Python SonarQube API to extract these data before our company upgraded to version 7.1. "api/resources" web service Deprecated since sonarqube5.4, so we cannot use it anymore.
I have also tried using getting data using CURL command via Web API using
curl -i -H "Content-Type: application/json" -H "x-api-key:token" -X GET 'http://MY_HOST/api/measures/component?metricKeys=key&component=project_key'
We are able to get a json payload for individual metrics, but involves tedious task of creating the URL every single time.
But I wanted to know if there is a better/smarter way to access these "measures", be it any language or implementation.
You could do this:
Call the API api/metrics/search first to get a (json) list of all the metrics and then iterate over that list and create a comma separated string of all the metric keys.
For example something like this: ncloc,complexity,violations .. as mentioned in the parameters example value in the API documentation here.
Then you could just add this comma separated list to the url as a parameter something like: http://MY_HOST/api/measures/component?metricKeys=ncloc,complexity,violations&component=project_key
and call it once to get the response for all metrics.
Also, I haven't tried this, but as per the latest documentation, the parameter component is optional. So if you omit that, ideally you should get a response with metrics of all the projects.

Google Drive API v3 : there isn't any way to get a download url for a google document?

The Google Drive API v2 to v3 migration guide says:
The exportLinks field has been removed from files. To export Google Documents, use the files.export method instead.
I don't want to export (download) the file right away. "files.export" will actually download the file. I want a link to download the file, later. This was possible in v2 by means of the exportLinks.
How can I in v3 accomplish the same? If it is not possible, why was this useful feature removed?
Besides, (similar problem to above) downloadUrl was also removed, and the suggested alternative ("files.get with ?alt=media") downloads the file instead of providing a download link. This means there is no way in v3 to get a public short lived URL for a file?
EDIT:
there is no way in v3 to get a public short lived URL for a file?
For regular files, apparently yes.
This seems to work fine (a public short lived link to the file with its right name and contents):
https://www.googleapis.com/drive/v3/files/ID?alt=media&access_token=TOKEN
For google apps files, no (not even private, as v2 exportLinks used to be).
https://www.googleapis.com/drive/v3/files/ID/exportmimeType=TYPEv&access_token=TOKEN
Similar to regular files, this URL is a short lived link to the file contents, but lacking of its right name.
BTW, I see the API is not behaving consistently: /drive/v3/files/FILEID delivers the right file name, but /drive/v3/files/FILEID/export does not.
I think the API itself should be setting the right Content-Disposition, as it is apparently doing when issuing a /drive/v3/files/FILEID call.
This file naming problem invalidates the workaround to the lack of ExportLinks in v3.
The v2 ExportLinks allowed me to link a file (which is not the same as getting its content right away). Anyone logged in and with the proper permissions was able to access it, and the link didn't needed any access_token, and it wasn't short lived. It was good and useful.
Building a link with a raw API call like /drive/v3/files/FILEID/export (with mandatory access_token) would be an close enough workaround (it is temporary and public, not the same as it was, anyway). However, the naming problem invalidates it.
In v2, regular files have a WebContentLink and google apps files have exportLinks. In v3 exportLinks are gone, and I don't see any suitable alternative to them.
Once you query for your file by id you can use the function getWebContentLink() to get the download link of the file (eg. $file->getWebContentLink() ).
I think you're placing too much emphasis on the word "method".
There is still a link to export a file, it's https://www.googleapis.com/drive/v3/files/fileIdxxxxx/export&mimeType=xxxxx/xxxxx. Make sure you URL encode the mime type.
Eg
https://www.googleapis.com/drive/v3/files/1fGBQ81haNU_nEiC5GITZD3bxT0ppL2LHg-C0ubD4Q_s/export?mimeType=text/csv&access_token=ya29.Gmo0BMvO-pVEPKsiD9j4D-NZVGE91MChRvwOcBSg3cTHt5uAClf-jFxcovQScbO2QQhwHS95eSGW1eQQcK5G1UQ6oI4BFEJJkntEBkgriZ14GbHuvpDL7LT2pKA--WiPuNoDDIuZMm5lWtlr
These links form part of the API, so the expectation is that you've written a client that sends authenticated requests, and deals with the response data. This explains why, if you simply paste the link into a browser without an access_token, it will fail. It also explains why the filename is export, ie. it isn't intended that your client would ever use a filename, but rather it should receive the data as a stream. This SO answer discusses the situation in more detail How to set name of file downloaded from browser?

Tyring to update a URL using Curl on Linux

I am trying to update an ticket and I am trying to do that through Linux. Not sure whether it can be done or not, in the way of searching, I found one blogger sharing, where he/she was also trying to update something
He/She has used the below commands to update the URL.
curl -i -u "script-user:password" -X PUT -d "short_description=Update+me" https:/0005000972
I suspect he/she is trying to update the URL with "Update me".
Is that right? or else Can some one explain what that blogger tried to do??
This is a RESTful request. That is a request using standard HTTP methods (GET, POST, PUT, DELETE) to query a server.
-X PUT specifies the method used
-d "...." specifies the data send to the server
https://... (ill formed in your example) is of course the URL of the target server
Usually, the PUT method is used to replace an existing attribute/value on the server. As the concrete parameters and/or methods available are service dependent, I can only guess here that the intend is to update some attribute named short_description to store the value Update Me (URL encoded -- or more formally x-www-form-urlencoded)
Maybe you should first read a little bit more about those topics, and then, if necessary post an other question describing more in details both the target server and the goal you're trying to achieve on it.

Resources