I would like to apply a reverse proxy on top of S3 to make content serving decisions based on the incoming request.
The S3 bucket is set to website mode and hosted publically.
I'll obviously have more logic to determine where I am getting the files from, but hopefully this will illustrate my desire.
This is using JavaScript, happy to use Go as well.
The following code does not work, but I'm not sure how best to get it working. Can I just send an arrayBuffer through?
module.exports.handler = async (event, context) => {
const data = await fetch(S3WebsiteURL + event.path)
const buffer = await data.arrayBuffer()
return {
headers: data.headers,
body: buffer,
statusCode: 200
}
}
I would use https://www.npmjs.com/package/request-promise
var rp = require('request-promise');
const htmlString = await rp(S3WebsiteURL + event.path);
return {
body: htmlString,
statusCode: 200
}
Try without headers and if it works, add header support.
I've found it difficult to use Lambda to proxy data - using API gateway, at least, it expects binary data in base-64 format at various points depending on how you set it up. They've improved things since I tried to do it that way last, so hopefully someone else can answer based on more recent experience.
If your content serving decisions are limited to access control (you don't need to transform the data you're serving), then you can use your lambda as a URL provider instead of a content provider - switch public sharing of the S3 bucket off, access the items using the S3 API, and have your lambda call S3.getSignedUrl() to get a link to the actual content. This means that only the callers of the lambda will have a valid URL to the content you want to protect, and depending on your application you can set the timeout on the pre-signed URL to be short enough you don't have to worry about it being shared.
The advantage here is that since the content itself doesn't get proxied through the lambda, your lambda runtime and memory costs can be lower and performance should be better.
Related
I've set up an Azure Functions proxy (using proxies.json). This should just pick the value given in the original request's url query string parameter and use that as a value for backendUri. So the goal is that the response of the call to the proxy contains the response of calling the URL that's in the url query string parameter directly. I need this because of CORS.
Here's my proxies.json
{
"$schema": "http://json.schemastore.org/proxies",
"proxies": {
"proxy1": {
"debug": true,
"matchCondition": {
"methods": ["GET"],
"route": "/proxy/"
},
"backendUri": "{request.querystring.url}"
}
}
}
When I call the proxy using https://not-an-actual-url.azurewebsites.net/proxy/?url=https://stackoverflow.com I'm getting back a 404. Same if I encode the value of the url parameter. If I set the backendUri in proxies.json to a static URL instead of trying to use the query string, it works, however.
To summarize, I want the value of backendUri to depend on the URL of the original request. As stated in the docs this should be possible. Quote from the docs:
Set the backend URL to another endpoint. This endpoint could be a function in another function app, or it could be any other API. The value does not need to be static, and it can reference application settings and parameters from the original client request.
When I call the proxy using
https://not-an-actual-url.azurewebsites.net/proxy/?url=https://stackoverflow.com
I'm getting back a 404. Same if I encode the value of the url
parameter. If I set the backendUri in proxies.json to a static URL
instead of trying to use the query string, it works, however.
Judging from your problem description, you don't seem to have a real HttpTrigger. You want to use function app as a server to forward requests to an address, right?
I think it is unrealistic that you want to dynamically get the url from the request and apply it to proxies.json. Because this file is already loaded when the function app is started, you cannot let the requested information enter, it will read your value as a normal string, if it is not a direct url, it cannot be read.
For CORS, you can find some free and public servers for forwarding, or build a server for forwarding by yourself. The proxies.json of function app may not realize your idea.
I have 2 Laravel(5.8) apps. One is a user application, the other is more of an API.
The Api has pdfs stored in the storage directory, I need to be able to allow a user to download the pdfs in the other application.
Really got no clue how to send the file over from app to the other.
The user app makes an api to the api with relevant file ids and things (fine), just can't work out how to send the file back, and then download it on the other end.
Tried things like return response()->stream($pdf) on the api and return response()->download($responeFromApiCall) and loads of other things but really getting nowhere fast.
Any advice welcome.
The laravel code you posted is basically correct, you can use one of stream(), download() or file() helper to serve a file.
file()
Serve a file using a binary response, the most straightforward solution.
// optional headers
$headers = [];
return response()->file(storage_path('myfile.zip'), $optionalHeaders);
You can also use your own absolute file path instead of the storage_path helper.
download()
The download helper is similar to file(), but it also sets a Content-Disposition header which will result in a download-popup if you retrieve the file using your browser.
$location = storage_path('myfiles/invoice_document.pdf');
// Optional: serve the file under a different filename:
$filename = 'mycustomfilename.pdf';
// optional headers
$headers = [];
return response()->download($location, $filename, $headers);
stream()
The stream() helper is a bit more complex and results in reading and serving a file in chunks. This can be used for situations where the filesize is unknown (like passing through another incoming stream from an API). It results in a Transfer-Encoding: chunked header which indicates that the data stream is divided into a series of non-overlapping chunks:
// optional status code
$status = 200;
// optional headers
$headers = [];
// The stream helper requires a callback:
return response()->stream(function() {
// Load a file from storage.
$stream = Storage::readStream('somebigfile.zip');
fpassthru($stream);
if(is_resource($stream)) {
fclose($stream);
}
}, $status, $headers);
Note: Storage::readStream takes the default storage disk as defined in config/filesystems.php. You can change disks using Storage::disk('aws')->readStream(...).
Retrieving your served file
Say your file is served under GET example.com/file, then another application can retrieve it with curl (assuming PHP). A popular wrapper for this would be Guzzle:
$client = new \GuzzleHttp\Client();
$file_path = __DIR__ . '/received_file.pdf';
$response = $client->get('http://example.com/file', ['sink' => $file_path]);
You can derive the filename and extension from the request itself by the way.
If your frontend is javascript, then you can retrieve the file as well but this another component which I dont have one simple example for.
So if I understand correctly you have two systems: user app and api. You want to serve a pdf from user app to the actual user. The user app does a call to api, which works fine. Your issue is converting the response of the api to a response which you can serve from your user app, correct?
In that case I'd say you want to proxy the response from the api via the user app to the user. I don't know what the connection between the user app and the api is, but if you use Guzzle you could look at this answer I posted before.
Here are the steps you should follow to get the PDF:
Make an API call using AJAX ( you are probably already doing it ) from your public site ( User site ) to the API server, with file ID.
API server fetches the PDF, copy it to the public/users/ directory, generate the URL of that PDF file.
Send the URL as a response back to the User site. Using JS add a button/ link in the DOM, to that PDF file.
Example:
jQuery.post(ajaxurl, data, function(response) {
var responseData = JSON.parse(response);
var result = responseData.result;
var pdf_path = responseData.pdf_path;
$(".nametag-loader").remove();
$(".pdf-download-wrapper").append("<a class='download-link' href='"+ pdf_path +"' target='_blank'>Download</a>");
});
So i'm using koa2,
if this request is multipart type, believe i need to process
ctx.request.body.fields
otherwise i process
ctx.request.body
So what's the best way to distinguish and handle these 2 cases ?
I guess my point of view is, if you're needing to check for a form submission as well as other activity, then perhaps your endpoint is doing a bit much? It very much depends on your use case of course, but keeping form data and other content types separate may be a clearer API.
Having said that, I don't see any reason to get more complicated than checking for the presence of fields. koa-body is a great way to go about this.
One thing you might like to consider is using the middleware stack to your advantage. For example, only do stuff if there's a form submission but allow other actions to take place:
router.post('/', koaBody({ multipart: true }),
async (ctx, next) => {
if (ctx.request.body.fields) {
// Handle form if present
}
await next() // Pass control down the stack
}
))
I've been attempting to inject a custom header for a error response status (and failing).
I have a very simple lambda being used
exports.handler = (event, context, callback) => {
// TODO implement
//callback(null, 'Hello from Lambda');
var error = {
name:"error",
message:"I am a failure",
statusCode: 400
};
error["x-test"] = 'foo';
callback(JSON.stringify(error), null);
};
In the api gateway, I've done the following:
set up CORS to include x-test
responsetemplate = "$input.path('$.errorMessage')"
responseparameter to include:
method.response.header.x-test = integration.response.body.x-test
Also, I have a statusCode mapped using '.*statusCode.*?400.*'
This has turned out empty.
so I decided to take a step back and see what happens if I do:
method.response.header.x-test = integration.response.body
I found that I get the stringified response of errorMessage.
{"x-test":"{\"errorMessage\":\"{\\\"name\\\":\\\"error\\\",\\\"message\\\":\\\"I am a failure\\\",\\\"statusCode\\\":400,\\\"x-test\\\":\\\"foo\\\"}\"}"}
So I decided to change the responsetemplate to force it to json by doing the following:
responsetemplate = "$util.parseJson($input.path('$.errorMessage'))"
and I still get the stringified response:
{"x-test":"{\"errorMessage\":\"{\\\"name\\\":\\\"error\\\",\\\"message\\\":\\\"I am a failure\\\",\\\"statusCode\\\":400,\\\"x-test\\\":\\\"foo\\\"}\"}"}
My guess is that it doesn't transform as expected, but only for the final output.
So how would you take a value and shove it into a header?
Thanks,
Kelly
I think this is more of a design choice regarding the limitation imposed by both Lambda and APIGateway. I will try my best to walk through my thoughts.
First of all, in Lambda, callback(error, result) function can either take an error string as first argument, or an object as result response. If you want to pass along a simple error message, you could definitely just do that. However, in your case, as you tried to pass along an entire error object, choosing the second option is clearly a better solution (in contrast to stringifying an object and parse it into object again). As a result, the final line of your Lambda function should be:
callback(null, error);
Yes, in this case, if you test your function in Lambda, the output result will no longer be red and flag it as an error, but this won't matter as you can format your headers and response in APIGateway.
Now you need to set things up in APIGateway, in which you need to make use of the object passed by Lambda.
It's actually rather easy to use method execution interface to configure headers.
In Method Response, you need to add the headers you want to include in the response for a specific status code, which in your case is x-test. (If you want the API to return different status codes, you can also configure that in this panel.)
Then go to Integration Response, under the same status code, you will see the added header available. According to this documentation from AWS, you can use integration.response.body.JSONPath_EXPRESSION to assign the header value (this is another reason that you should return object rather than string in Lambda, as there is no formal API to parse object from string at this stage). This time, as your Lambda is passing an object, the value of x-test header is:
integration.response.body['x-test']
Now your API should have the proper header included.
NOTE: In order to set up different status code in APIGateway, you should leave some distinguishable data fields (your statusCode: 400 should work perfectly) in you response body, so you can use RegEx to match those fields to a specific status code.
So... above doesn't work with success message. I found this blog though talking about error handling design pattern. Apparently what they suggest is only mapping status code when there is an error, in which case no body should be passed (only the errorMessage), as browser won't care about response body for a status code other than 200 anyway.
I guess after all, it is impossible to customize both status code and header at the same time with Lambda passing an object to APIGateway?
This is due to the fact that you are stringifying the error object coming from your Lambda function. API Gateway attempts to resolve the JSON-Path expression and can't evaluate "x-test" in a string. If you return an object instead of a string, this should work.
You may want to consider using proxy integrations which allow you to control the headers and status directly from your Lambda function.
Update: I've written a blog post on this topic with sample code # https://rpgreen.wordpress.com/2017/01/25/how-to-send-response-headers-for-aws-lambda-function-exceptions-in-api-gateway/
I am rather new to NodeJS so hopefully I am able to articulate my question(s) properly. My goal is to create a NodeJS application that will use the node-rest-client to GET data and asynchronously display it in HTML on client side.
I have several node-rest-client methods created and currently I am calling my GET data operation when a user navigates to the /getdata page. The response is successfully logged to the console but I'm stumbling on the best method to dynamically populate this data in an HTML table on the /getdata page itself. I'd like to follow Node best practices, ensure durability under high user load and ultimately make sure I'm not coding a piece of junk.
How can I bind data returned from my Express routes to the HTML front end?
Should I use separate "router.get" routes for each node-rest-method?
How can I bind a GET request to a button and have it GET new data when clicked?
Should I consider using socket.io, angularjs and ajax to pipe data from the server side to client side?
-Thank you for reading.
This is an example of the route that is currently rendering the getdata page as well as calling my getDomains node-rest-client method. The page is rendering correct and the data returned by getDomains is successfully printed to the console, however I'm having trouble getting the data piped to the /getdata page.
router.get('/getdata', function(req, res) {
res.render('getdata', {title: 'This is the get data page'});
console.log("Rendering:: Starting post requirement");
args = {
headers:{"Cookie":req.session.qcsession,"Accept":"application/xml"},
};
qcclient.methods.getDomains(args, function(data, response){
var theProjectsSTRING = JSON.stringify(data);
var theProjectsJSON = JSON.parse(theProjectsSTRING);
console.log('Processing JSON.Stringify on DATA');
console.log(theProjectsSTRING);
console.log('Processing JSON.Parse on theProjectsSTRING');
console.log('');
console.log('Parsing the array ' + theProjectsJSON.Domains.Domain[0].$.Name );
});
});
I've started to experiment with creating several routes for my different node-rest-client methods that will use res.send to return the data and the perhaps I could bind an AJAX call or use angularjs to parse the data and display it to the user.
router.get('/domaindata', function(req, res){
var theProjectsSTRING;
var theProjectsJSON;
args = {
headers:{"Cookie": req.session.qcsession,"Accept":"application/xml"},
};
qcclient.methods.getDomains(args, function(data, response){
//console.log(data);
theProjectsSTRING = JSON.stringify(data);
theProjectsJSON = JSON.parse(theProjectsSTRING);
console.log('Processing JSON.Stringify on DATA');
console.log(theProjectsSTRING);
console.log('Processing JSON.Parse on theProjectsSTRING');
console.log('');
console.log('Parsing the array ' + theProjectsJSON.Domains.Domain[0].$.Name );
res.send(theProjectsSTRING);
});
});
I looked into your code. You are using res.render(..) and res.send(..). First of all you should understand the basic request-response cycle. Request object gives us the values passed from routes, and response will return values back after doing some kind of processing on request values. More particularly in express you will be using req.params and req.body if values are passed through the body of html.
So all response related statements(res.send(..),res.json(..), res.jsonp(..), res.render(..)) should be at the end of your function(req,res){...} where you have no other processing left to be done, otherwise you will get errors.
As per the modern web application development practices in javascript, frameworks such as Ruby on Rails, ExpressJS, Django, Play etc they all work as REST engine and front end routing logic is written in javascript. If you are using AngularJS then ngRoute and open source ui-router makes work really easy. If you look closely into some of the popular MEAN seed projects such as mean.io, mean.js even they use the ExpressJS as REST engine and AngularJS does the rest of heavyweight job in front end.
Very often you will be sending JSON data from backend so for that you can use res.json(..). To consume the data from your endpoints you can use angularjs ngResource service.
Let's take a simplest case, you have a GET /domaindata end point :
router.get('/domaindata',function(req,res){
..
..
res.json({somekey:'somevalue'});
});
In the front end you can access this using angularJS ngResource service :
var MyResource = $resource('/domaindata');
MyResource.query(function(results){
$scope.myValue = results;
//myValue variable is now bonded to the view.
});
I would suggest you to have a look into the ui-router for ui front end routing.
If you are looking for sample implementation then you can look into this project which i wrote sometime back, it can also give you an overview of implementing login, session management using JSON Web Token.
There are lot of things to understand, let me know if you need help in anything.