We are using Fine Uploader in S3 mode against a non-traditional, though S3-compliant, service. It uses an unusual hostname and port. The /signature/ endpoint gets called with a payload like this:
{"headers":"POST\n\nimage/jpeg\n\nx-amz-acl:private\nx-amz-date:Tue, 05 Aug 2014 20:28:11 GMT\nx-amz-meta-qqfilename:Pomegranate.jpg\n/service-fqdn.com/37b5165cf4fd.jpg?uploads"}
In other words, Fine is (quite reasonably) treating this like a CNAME style virtual hosted bucket and generating the StringToSign accordingly. In fact, service-fqdn.com is a service endpoint to itself, and the canonical resource named in the StringToSign should be simply:
/bucketname/37b5165cf4fd.jpg?uploads
In this configuration, bucketname is provided on the path, not as part of the domain name.
Manually patching the incoming StringToSign inside the /signature/ endpoint results in a successful upload.
So what is the proper way to configure Fine Uploader to understand that service-fqdn.com is a service endpoint? [For comparison, in the common s3curl.pl utility, one needs to modify the source code with a list of known endpoints.]
Thanks,
Fine Uploader S3 currently expects that all endpoints are actual S3 buckets. There is a plan to update the S3 code to allow uploads to work seamlessly with any service that "acts like" an S3 endpoint. Part of that effort involves allowing uploads to a CloudFront distribution. Some internal discussion suggests that we will address the CF distro feature in such a way that uploads to any S3-like endpoint will work as a result.
Until then, as you have stated, the logic used to determine the bucket name will need to be tweaked.
As a side note, unless I am misremembering, I believe we have spoken about your specific use case via email. If I am correct, I would encourage you to bring this up in the email thread started back in July.
Related
I have a Spring Boot REST Microservice which expects an input file and converts this file to other formats using ffmpeg like so
ffmpeg -i <INPUT_FILE> -vf 'scale=320:-2' <OUTPUT_FILE>
I am calling this command at the moment in a Java ProcessBuilder and referencing the container location of the input and output files after pushing the microservice to PCF.
The ffmpeg binary and the input file are packaged in the jarfile
I understand I need to use cloud storage like NFS or S3 to specify the locations, but that is a secondary matter now.
My idea is to make the Microservice a PCF User-provided Service, so that bound Apps will supply the location of the input and converted files.
Also since there are different conversion functions, I have corresponding endpoints for each conversion function.
All examples I have seen with respect to Microservices have to do with Databases whereby you specify information like URL and credentials to access the external database,
which does not shed any light into what I have in mind.
So I have the following questions:
1) Is it possible to simply convert the running Conversion Microservice into a PCF User-provided Service? I know of the CUPS command, but I am not sure what to supply as Parameters, also since I have several endpoints
2) How could bound Apps call the endpoints of this Service and provide the locations of the input and output files.
I will appreciate code snippets if possbile
Thanks
What you're trying to do should be possible, but not quite in the way you describe it.
For starters, you can use a user provided service to pass details about a microservice between apps. Thus if you create a conversion microservice, you can pass information about the microservice to consuming apps via a user provided service.
Where I think you need to change your plans a bit is with the consuming apps. You can't pass the input/output files via a user provided service, at least not without sharing storage between the two and you don't want to go down that road. It's limiting as all apps have to share the storage and it will have scaling limits.
Instead what you'd want to do is to put the location and credentials required by a consuming party in the user provided service (this is what you're seeing with databases). For example, you'd probably need the hostname or route to your microservice and likely a user token/secret or username and password (that way you know who's making requests to your service).
Your consuming app could then read the information out of the bound user provided service and use that to contact your conversion microservice. At that point, it would be up to the consuming app to understand the API that you expose from your microservice so that the consuming app can send the file to be converted to the microservice and retrieve the processed file. You'd likely send/retrieve files over HTTP, and if I may make a recommendation you probably want to look at using the claim check pattern on your service since doing conversions can be time consuming.
Hope that helps!
I have used protobufs to define my models and gRPC service, and generated protos for Go (.pb.go). When I make gRPC call to a service, I want to find out in server the name of the client/application calling the server API.
Can the context parameter in the API be of any help here?
You can use the WithUserAgent dial option to set a custom user agent string on the client.
For retrieving the user agent on the server side, see this question. It's not fully answered, but you can just print the metadata map and see how the user agent gets stored in the metadata. And then post your findings there :)
As an option, you can use GRPC Metadata. It's not strong typed (because it's just map[string][]string), but may fit your needs good. Set custom metadata field on client side and check this field in server handler.
I got notified that Googles JSON-RPC and Global HTTP Batch Endpoints are deprecated. The affected api is "storage#v1" and "Global Batch Endpoints" in my case.
I tried to find, where the depricated api call comes from. But I'm using 24 buckets with several tools accessing it. So is there a way to log depricated calls? I enabled logging for buckets. I could not find any difference in the access log when doing batch request and when doing single requests.
Yes "Batching across multiple APIs in a single request" is being discontinued Discontinuing support for JSON-RPC and Global HTTP Batch Endpoints
But what its hard to understand is exactly what is being discontinued.
There are two batching endpoints. The Global one www.googleapis.com/batch
And the API specific one www.googleapis.com/batch/<api>/<version>.
So whats changing?
The global batching endpoing is going down. This means you wont be able to make calls to www.googleapis.com/batch anymore. What does that really mean in the worse case if you were making batch requests mixing two apis at the same time for example Drive and Gmail you wont be able to do that anymore.
In the future you are going to have to split batch requests up by API.
Will this effect you?
Code wise this depends on which client library you are currrently using. Some of them have already been updated to use the single api endpoint (JavaScript and .net) there are a few which have not been updated yet (php and java last i checked)
Now as for your buckets if i understand them correctly they all insert into the same place so your using the same api this probably inst going to effect you. You are also using Googles SDK and they are going to keep that updated.
Note
The blog post is very confusing and there is some internal emails going around Google right now in attempt to get things cleared up as far as what this means for developers.
You have to find where you are doing heterogeneous batch requests either directly or through libraries in your code. In any case batch requests are not reflected in your bucket logs because no API or API method per se was deprecated just a way to call send them.
In detail
You can bundle many requests to different APIs into one batch request. This batch will be sent to a one magical Google server that splits the batch and routes all the API requests in it into their respective service.
This Google server is going to be removed so everything has to be sent directly to the service.
What should you do?
I looks like you are making heterogeneous batch requests because only one service is mentioned, Storage. Probably you should do one of these options.
if you are using Cloud Libraries -> update them.
find if you are accessing the URL below
www.googleapis.com/batch
and replace it with the appropriate homogeneous batch API, which in your case is
www.googleapis.com/batch/storage/v1
in case you use batchPath, this seems to be a relevant article
Otherwise, if you make heterogeneous calls with gapi, which doesn't seem to be your case, split something like this:
request1 = gapi.client.urlshortener(...)
request2 = gapi.client.storage.buckets.update(...)
request3 = gapi.client.storage.buckets.update(...)
heterogeneousBatchRequest = gapi.client.newBatch();
heterogeneousBatchRequest.add(request1);
heterogeneousBatchRequest.add(request2);
heterogeneousBatchRequest.add(request3);
into something like this
request1 = gapi.client.urlshortener(...)
urlshortnerbatch = gapi.client.newBatch();
urlshortnerbatch.add(request1);
request2 = gapi.client.storage.buckets.update(...)
request3 = gapi.client.storage.buckets.update(...)
storagebatch.add(request2);
storagebatch.add(request3);
Official Documentation
Here it's described how to make batch request specifically with Storage API.
I'm new to parse and i've just setup my server and dashboard on my local machine.
For my use, i just not need the simple API from parse, i need to write a server (with NodeJS + Express) to handle users request.
I've just see how to integrate an Express application with parse, so my application instead of the server directly will use my server that will serve:
The standard Parse API (/classes etc)
All my others route, that could not to depend on Parse API
This is correct ?
Reading online i've see that Parse Cloud need to extend Parse functionality with additional "routing" (if i have understand well).
So, in my application i will have
The standard API (ad described up here)
All other routers (that could not depend on Parse)
Other routers (that come from Cloud) and use Parse API
So, Parse Cloud is just a "simple" way to write additional Routing ? (i've see that exists the job function too, but right now i've not studied it).
My question is just because i'm a little confused about the real needed, just would like to have more info on "when to use it"
Thanks
EDIT
I provide here an example (that in part come from Parse Docs).
I have a Video class with an director name field.
In my Application (iOs, Android etc) i setup a view that need to know all the Video provided from a particular director.
I will have three ways:
Get all Videos (/classes/videos) and then filter it directly in APP
Write an NodeJS + Express router endpoint (http://blabla.com/videos/XXX) where XXX is the director and then get the result with Parse JS API and send back it to the app
Write an Clound function (that if i have understand respond to /functions/) that do the same as the router one.
This is just a little example, but is this the usage of Parse Cloud ? (or at least, one on them :))
I have a bunch of markdown documents in Riak, which I'm exposing via a small Sinatra API with basic search functionality etc.
Each document has an associated image, also stored in Riak (in a different bucket). I'd like to have a client app display the documents alongside their associated images - so I need some way to make the images available, but as I'm only ever going to be requesting them by key it seems wasteful to serve them via a Sinatra app as I'm doing with the documents.
However I'm uneasy with serving them directly from Riak, because a) even using nginx to limit the acceptable requests, I worry about exposing more functionality than we want to and b) Riak throws a 403 for any request where the referrer is set, so by default using a direct-to-Riak url as the src of an img tag doesn't work.
So my question is - what's a good approach to take for serving the images? Add another endpoint to the Sinatra app? Direct from Riak using some Nginx wizardry that is currently beyond me? Or some other approach I haven't considered yet? This would ideally use Ruby as that's what the team I'm working with are more comfortable with.
Not sure if this question might be better suited to Server Fault - if so I'll move it over.
You're right to be concerned about exposing Riak to any direct connectivity. Until 2.0 arrives early next year, there is no security in the system (although the 403 for requests with a referrer is a security mechanism to protect against XSS), and even with security exposing any database directly to the Internet invites disaster.
I've not done anything with nginx, but all you'd really need to use it properly, I'd think, would be two features:
Ability to restrict requests to GET
Ability to restrict (or rewrite) requests to the proper bucket
Ability to strip out all HTTP headers that Riak includes in its result (which, since nginx is a proxy server and not a straight load balancer, seems like it should be straightforward)
Assuming that your images are the only content in that bucket, nginx feels like a reasonable choice here.