I have set up direct uploads to S3 with Shrine. This works great. Among others, I have the following plugins enabled:
Shrine.plugin :backgrounding
Shrine.plugin :store_dimensions
Shrine.plugin :restore_cached_data
Correct me if I'm wrong but image dimensions extraction appears to be done synchronously. If I let the user bulk upload images via Uppy and then persist them all, this seems to be taking a long time.
What I'd like to do is perform image dimensions extraction asynchronously - I don't need the dimensions available for the cached file. If possible, I'd like to do that in the background when the file gets promoted to the store. Is there a way to do it?
The way I got this to work is by making use of :refresh_metadata plugin, instead of :restore_cached_data which I used originally. Thanks to Janko for pointing me in the right direction.
Reading into the source code provided some useful insights. :store_dimensions plugin by itself doesn't extract dimensions - it adds width and height to the metadata hash so that when Shrine's base class requests metadata, they get extracted too.
By using :restore_cached_data, this was being done on every assignment. :restore_cached_data uses :refresh_metadata internally so we can use that knowledge to only call it when the file is promoted to the store.
I have :backgrounding and :store_dimensions set up in the initializer so the final uploader can be simplified to this:
class ImageUploader < Shrine
plugin :refresh_metadata
plugin :processing
process(:store) do |io, context|
io.refresh_metadata!(context)
io
end
end
This way persisting data we get from Uppy is super fast and we let the background job extract dimensions when the file is promoted to the store, so they can be used later.
Finally, should you have questions related to Shrine, I highly recommend its dedicated Google Group. Kudos to Janko for not only creating an amazing piece of software (seriously, go read the source), but also for his dedication to supporting the community.
Related
I am creating a web application in Go.
I have modified my working code so that it can read and write files on both a local filesystem and a bucket of Google Cloud Storage based on a flag.
Basically I included a small package in the middle, and I implemented my-own-pkg.readFile or my-own-pkg.WriteFile and so on...
I have replaced all calls in my code where I read or save files from the local filesystem with calls to my methods.
Finally these methods include a simple switch case that runs the standard code to read/write locally or the code to read/wrote from/to a gcp bucket.
My current problem
In some parts I need to perform a ReadDir to get the list of DirEntries and then cycle though them. I do not want to change my code except for replacing os.readDir with my-own-pkg.ReadDir.
So far I understand that there is not a native function in the gcp module. So I suppose (but here I need your help because I am just guessing) that I would need an implementation of fs.FS for the gcp. It being a new feature of go 1.6 I guess it's too early to find one.
So I am trying to create simply a my-own-pkg.ReadDir(folderpath) function that does the following:
case "local": { }
case "gcp": {
<Use gcp code sample to list objects in my bucket with Query.Prefix = folderpath and
Query.Delimiter="/"
Then create a slice of my-own-pkg.DirEntry (because fs.DkrEntry is just an interface and so it needs to be implemented... :-( ) and return them.
In order to do so I need to implement also the interface fs.DirEntry (which requires the implementation of interface for FileInfo and maybe something else...)
Question 1) is this the right path to follow to solve my issue or is there a better way?
Question 2) (only) if so, does the gcp method that lists object with a prefix and a delimiter return just files? I can't see a method that returns also the list of prefixes found
(If I have prefix/file1.txt and prefix/a/file2.txt I would like to get both "file1.txt" and "a" as files and prefixes...)
I hope I was enough clear... This time I can't include code because it's incomplete... But in case it helps I can paste what I can.
NOTE: by the way go 1.6 allowed me to solve elegantly a similar issue when dealing with assets either embedded or on the filesystem thanks to the existing implementation of fs.FS and the related ReadDirFS. So good if I could follow the same route 🙂
By the way I am going on studying and experimenting so in case I am successful I will contribute as well :-)
I think your abstraction layer is good but you need to know something on Cloud Storage: The directory doesn't exist.
In fact, all the object are put at the root of the bucket / and the fully qualified name of the object is /path/to/object.file. You can filter on a prefix, that return all the object (i.e. file because directory doesn't exist) with the same path prefix.
It's not a full answer to your question but I'm sure that you can think and redesign the rest of your code with this particularity in mind.
Consider the FHIR Patient data at http://spark.furore.com/fhir/Patient/f201.
How can I get the photo object referenced therein at URL "binary/#f006"??
I would have thought an HTTP GET on http://spark.furore.com/fhir/binary/#f006 would have done it, but alas...
the data there is wrong. Your conversion to the get was correct, but you ended up with a wrong URL because the reference is wrong in the first place.
It should say: url="Binary/f006" which would equate to a get of http://spark.furore.com/fhir/Binary/f006. That doesn't work either, which is another error in the way things are defined.
See http://gforge.hl7.org/gf/project/fhir/tracker/?action=TrackerItemEdit&tracker_item_id=6107 for follow ups
Yes, this reference is outdated, and we are not distributing Binaries currently as part of the examples in the FHIR specification. Our server Spark loads the examples from the specification when we initialize the database, hence the images are not there.
For now, I have uploaded the correct image to Binary/f006 and have updated the link in Patient/f201, so things should work now. When we re-initialize the database (we don't do this often), these changes will be reversed, but a simple PUT to Binary/f006 and an update of Patient/f201 will fix this of course.
My understanding of this was that perhaps CGPDFContext is to be used for editing PDF document data and CGPDFDocument is used for storing it, since the documentation doesn't list any ways to alter the content of a CGPDFDocument.
I'm also not quite sure what CGDataConsumer/Provider does. From reading the documentation I got the impression that the consumer/provider abstracts the relationship between the CG object and the CFData it writes to; so I don't have to do that myself. So I figured the following code would create a two page blank PDFdocument:
//Don't know exactly how large a PDF is so I gave it 1 MB for now
self->pdfData = CFDataCreateMutable(kCFAllocatorDefault, 1024);
self->consumerRef = CGDataConsumerCreateWithCFData(self->pdfData);
self.pdfRef = CGPDFContextCreate(self->consumerRef, NULL, NULL);
CGPDFContextBeginPage(self.pdfRef, NULL); //Creates a blank page?
CGPDFContextEndPage(self.pdfRef);
CGPDFContextBeginPage(self.pdfRef, NULL); //Creates a second blank page?
CGPDFContextEndPage(self.pdfRef);
//Copies the data from pdfRef's consumer into docRef's provider?
self.docRef = CGPDFDocumentCreateWithProvider(
CGDataProviderCreateWithCFData(
CFDataCreateCopy(kCFAllocatorDefault, self->pdfData)
));
It didn't work though, and NSLogging the first two pages of docRef returns NULL. I'm rather new at this, the C-Layer stuff in particular. Can someone explain to me the relationship between CGPDFContext, CGPDFDocument, CGDataConsumer & CGDataProvider and how I'd use them to create a blank PDF?
Your basic understanding is correct as far as I can see:
A CGPDFContext is a drawing context that "translates" everything that is drawn onto it to PDF instructions (typically for storage in a PDF file).
A CGPDFDocument is used to open an existing PDF file and get information from it.
When you want to create your own PDF file, you have two ways to do it as described here: https://developer.apple.com/library/mac/documentation/graphicsimaging/reference/CGPDFContext/Reference/reference.html
Use "CGPDFContextCreate" which you pass a data consumer. The data consumer gets the data and can do with it as it pleases (you could create a data consumer that passes the PDF onto the clipboard for example).
Use "CGPDFContextCreateWithURL" which you pass a URL. In that case your data will be written to a PDF file at that URL.
If you want to use these functions, have a look at this page https://developer.apple.com/library/mac/documentation/graphicsimaging/Conceptual/drawingwithquartz2d/dq_pdf/dq_pdf.html#//apple_ref/doc/uid/TP30001066-CH214-TPXREF101 which explains in detail how to create PDF files with a data provider and without (simply to a PDF).
To figure out what is happening I would start by trying to write a simple PDF file to disk before writing one to a data provider and then using that data provider immediately to read it again. Without trying your code however, let me point out that you didn't use "CGPDFContextClose" which is described in the document as closing the PDF document and flushing all information to output. You could actually having a situation where stuff is cached and not written to your data provider yet, simply because you haven't forced that.
This is really more of a "using what method" than a "how-to" question. I am creating a site in NodeJS with Express. So, each user has the ability to upload a profile picture, and my concern is how to route requests for these images. A couple of questions I have are:
Can I use express.static() to serve a default file if a valid one isn't specified? If not, am I going to have to specify a GET route for /img/profileand handle the FS querying there?
How can I find the correct image if multiple file extensions are allowed? (I originally just removed the file extension and they appeared in img tags anyway, is that okay?)
I am currently naming all pictures after their user's name. Looking ahead into the future (for apps I may have to scale), what are normal naming conventions for static user content? Are most stored with a UUID referencing the content in the database?
What else should I take into consideration that I may not have accounted for yet?
First question:
At present, I'd recommend storing your images in a predictable location that can be inferred from some combination of columns or fields in your database entries. One of those fields or columns would be a filename (accounts for different extensions). Then, if they haven't uploaded an image, you just lay down in your view code a reference to the generic "has not set an image" file.
express.static obviously can server static files, but it currently doesn't have a way to serve some other file if the one you wanted isn't there. Because this sounded like fun, I made some modifications to static to support what you request and submitted a couple of pull requests (the feature touched 2 different modules):
In send: https://github.com/visionmedia/send/pull/33
In connect: https://github.com/senchalabs/connect/pull/999
I don't know if those will get included in the project, but if you want to see my forks that have these changes, they are here:
https://github.com/bigohstudios/send
https://github.com/bigohstudios/connect
If this went through, you would mount a static at the root of your profile image files:
app.use(static('<profile image files root>', { fallback: 'anon.jpg'}))
Second question
I generally store the extension in my DB, then when I load the image, I have the correct extension to put into the src attribute on the img tag.
Third question
If you can guarantee unique names, then using those would work. I prefer using the DB id or a UUID as you suggest. It's less intuitive when scanning all the image uploads, but I rarely find myself doing that. I'm usually hunting for a specific image, and I can look up the identifier for that as needed.
Fourth question
You may end up storing the static content outside your app server (e.g. on S3). If that happens, then of course static won't help you.
I am porting a JAVA game to WP7. We have lots of images in our game and for loading them in JAVA use a function something like this "Resources.getImage(IMG_BULLETS);" where IMG_BULLETS is ID(an int) of the image.
But in WP7 we have to pass the path(a string) of image in order to load that.
Now my question is :
How to achieve a int-String mapping? so that I dont have to manually change the Id into path.
One possible solution comes in my mind is to have a .txt file having image path and its Id and parse that. But I am sure their is much better solution for this.
Note : we also have a multi-level folder structure for images and other files.
If you really want to refer to resources by int, you are going to have to map them some how to their path. As far as I know, there is no way to crawl your assets, and even if you could, i some how doubt you will get the same index order and you would have to redo your enum.
There are going to be a few hurdles you hit with trying to port and code changes. Another one, (one you will still have even with a txt file work around) will be that the contentManager.Load takes a type, eg, contentManager.Load<Texture2D>('path') returning a Texture2D.
One option would be to create your own singleton 'Resources' class that has get methods that take int to get the appropriate asset, however you will still need a mapping via xml or txt file of somekind. However this will require all your assets being loaded from the start which are more problems of their own (90mb memory limit and really long load time).
My advice would be look at the game development section of the App Hub and have a look at the game state example to help you see how you might be able to structure your game to work well with XNA.