I have an application where you can upload and view images.
For the view I need the images in different resolutions.
There are two ways how to proceed:
Scale the images dynamicly with assetics
Persist the image in different resolutions when uploaded
For me the 1. would be more comfortable but I am not sure if the performance is ok.
Does anyone have some experience about dynamic resizing with assetics?
Regards
I use Avalanche Imagine Bundle for this task. It allows you to configure filters, and then apply them in your templates.
As per your concern about performance - it's not generating the image for every request - only at first request, and it stores them in cache. You have to handle re-generating (i.e. deleting the cache) manually on entity level though (afaik)
Related
I have a website that is going to host thousands of images. Not only that, but each image has different size depending on where you are in the webpage - on the list page, the image is shown as 350x200 rectangle, on the sidebar pictures are 100x100 etc.
So when a user uploads an image to the website, I keep the original and make 4 resized copies for each size. So if 100 users upload an image, the result will be 500 images. I can't event think of what will happen with the different sizes of the different mobile devices...
I started using CloudFront to optimize speed. And when a user uploads an image I upload the original and the resized copies to Amazon S3 bucket.
But if tomorrow I decide to add another size, or change an existing size I have to run a script that deletes the old sized images and to uplaod the newly sized images, which means "get the original from S3, resize it, upload the new size". That is not practical at all. Imagine when I do a complete redesign of the website and the image sizes change completely, I would have to run a script that resizes each image to the new requirements and to delete the old images.
Is there a more practical way to acheive that?
I wanted to acheive the following scenario:
When the user uploads an image, I resize it and upload it to S3
CloudFront will take the images which are directly resized
When I add another size, I want CloudFront to see that this size is missing on S3 and pull it from an origin from my website.
I cannot think of a way to implement this. Any help or sharing best practices will be appretiated.
Rather than doing all this yourself, you could use an image resizing service such as:
Cloudinary
Imgix
They can resize images on-the-fly so you do not have to create and store them yourself. They can also manipulate them (rotate, watermark, colorize) images on-the-fly. Videos too!
If you choose not to use such a service, you could create your own virtual resizing service. The major choice is whether to:
Resize on-the-fly (using CloudFront for caching but not requiring any storage of the resized images), or
Resize upon request and store the result for future access (less processing cost, but involves storage cost)
It is not possible to have CloudFront "see that this size is missing on S3 and pull it from an origin from my website". (You might be able to do fancy stuff with 404 pages, but it wouldn't be worth the effort.)
For anyone doing this now. It's not for everyone, but due cost-saving efforts we just switched from Imgix to using AWS's Serverless Image Handler. Works great if your images are on S3 and maybe a good alternative solution if it lacks the features of Imgix.
I’ve been thinking a while about the best solution and as much as I read I get more and more confused. There are a lot of different libraries and helpers (most of them are outdated or for CI 1.x) and I really need your help.
I have a custom CMS based on CodeIgniter 2.1.3, news site that has about 40-50 images on the home page, but 80% of them are really small thumbnails in 3 different sizes and the other 20% of the images on the home page are in 2 sizes + for the inner pages when I list the news from a category there is 1 size of thumbnails. So in total I will need the original image for the news story, + 5-6 thumbnails sizes for the home page.
What’s the smartest way to deal with this? There will be let’s say 10-50 new news per day.
Is it still better to create 5-6 thumbnails per image during the upload?
What about the method “on the fly”? I’m more into this method, as I read, only the first visitor will call the library/helper to generate the thumbnails, and for the others the thumbnails will be already created so it won’t waste CPU. What about this method? Is it good practice?
What caching techniques I should use for these what I need?
Also I forgot to ask, how the other CMS system deal with generating the thumbnails? I mean about Wordpress, Drupal, Joomla, etc.
Do they store predefined sizes or generate them on the fly?
I guess their logic should be the best, or maybe not, but I want to implement something smart in my CodeIgniter CMS.
I didn’t mention, but I think it’s not important to this, I use Grocery CRUD for the admin panel.
Any help is appreciated.
Your best bet is to create images on the fly + use CDN like Amazon Cloudfront to cache the resized versions of your source image.
I’ve been using CodeIgniter for a number of years to build websites where lots of different sizes of images are used throughout the website. At the beginning I used to create every size needed out of the original image during the upload process (could easily end up with more than 5 thumbnails). This proved to be delivering the best performance – whenever you need an image of the certain size you just include it with no additional PHP processing. However I noticed that I end up with a huge number of images on the server, where the older ones may not even be used that often (e.g.: articles older than a year). Plus developing this way takes longer.
Then I started creating images on the fly, firstly using 3rd party libraries and later developed my own interface for CodeIgniter. This saves a lot of time, because during the upload process you save an original version of the image not worrying about resized versions. When displaying an image in the front end, all you normally need to do is to pass certain dimensions of the image required. Doing this way, not only you can get 5-6 versions of the image, but as many as you need. Also that’s a solution for the future when you redesign your website where the different sized images might be needed! What would you do when none of your 5 thumbnail options are no longer valid and you need different sizes?
You’re right, resizing an image on the fly can really be CPU consuming operation (especially when the large images are involved), therefore caching is a must. You can cache images right on your server or get CDN on top of that.
To keep the server tidy I normally run a cron job to delete on-the-fly images older than let's say a week. That saves space + doesn't cause harm - whenever image is needed to display, it'll just get recreated.
Check out timthumb, it's a script that resizes images on the fly and stores them in a cache. It's a simple as including an image tag with parameters in the URL.
ALso check this link which looks promising http://www.jenssegers.be/blog/31/Codeigniter-resizing-and-cropping-images-on-the-fly-continued
I love the way Drupal manage this. In Drupal 6 there was a module called imagecache (now is in core in Drupal 7, but functionality is very similar), which basically stores presets for images (image sizes, transformations, effects...) and when the visitor ask for an image the module generate different images based on presets and serve this images. This way you upload an image but have different images for different purposes.
The module has a really useful feature, if you want to change one preset, you can "flush" all the images related with that preset, so the visitors can see the changes.
Of course there are many other modules in Drupal related to imagecache or image styles, that add other effects like watermarks...
More information:
http://drupal.org/node/949222
http://drupal.org/node/163561
I have an app that allows users to take pictures and store them in the IsolatedStorage. These images are then retrieved to display them in a list. The app suffers from performance issues when there are more than 30+ images in the list. It is taking a long time to retrieve these images from the storage and bind them to the list. A lot of times, the app is just closed by OS because it takes too long.
Also, I think as better resolution devices are coming in the market, the images are getting created with bigger sizes and affecting performance even more.
I need some guidance on how to resolve this. Couple of things I have though about:
1) how can I asynchronously retrieve every image to be retrieved from IS so the page can load but the images are being retrieved in async fashion. Any sample code is appreciated.
2) the other approach will be where I create a thumbnail photo of the actual photo (smaller size and resolution) while storing the original photo and use this thumbnail photo in binding to the list later. If this should be done, then how should I go about processing the image on the fly to create a smaller size thumbnail version of the original photo.
Any other ideas are appreciated as well.
Thanks for your time.
If you are able to, I would recommend using the Async CTP and making the code that loads images from isolated storage async. The methods that generate the thumbnails can be async as well.
If you cannot do this (add these dependencies to your project), use a BackgroundWorker. Perhaps this article can help you get started.
This will allow your UI to be responsive while processing continues on threads in the background.
Do not pretend the Isolated Storage as a SQL Server. There will be great performance difference. If you want to process too much data, send them to server.
However, there is a method for getting a thumbnail. You can use it:
http://msdn.microsoft.com/en-us/library/system.drawing.image.getthumbnailimage.aspx
Also, please check this answer:
Create thumbnail image
my company has recently started to get problems with the image handling for our websites.
We have several websites (adult entertainment) that display images like dvd covers, snapshots and similar. We have about 100'000 movies and for each movie we have an average of 30 snapshots + covers. Almost every image has an additional version with blurring and overlay for non-members, this results in about 50 images per movie or a total of 5 million base images. Each of the images is available in several versions, depending on where it's placed on the page (thumbnail, original, small preview, not-so-small preview, small image in the top-list, etc.) which results in more images than i cared to count.
Now i had the idea to use a server to generate the images on-the-fly since it became quite clumsy to generate all the different images for all the different pages (as different pages sometimes even need different image sizes for basically the same task).
Does anyone know of an image processing server that can scale down images on-the-fly so we only need to provide the original images and the web guys can just request whatever size they need?
Requirements:
Very High performance (Several thousand users per day)
On-the-fly blurring and overlay creation
On-the-fly resize (with and without keeping aspect ratio)
Can handle millions of images
Must be able to read JPG, GIF, PNG and BMP and convert between them
Security is not that much of a concern as i.e. the unblurred images can already be reached by URL manipulation and more security would be nice but it's not required and frankly i stopped caring (After failing to get into my coworkers heads why (for our small reseller page) it's a bad idea to use http://example.com/view_image.php?filename=/data/images/01020304.jpg to display the images).
We tried PHP scripts to do this but the performance was too slow for this many users.
Thanks in advance for any suggestions you have.
I suggest you set up a dedicated web server to handle image resize and serve the final result. I have done something similar, although on a much smaller scale. It basically eliminates the process of checking for the cache.
It works like this:
you request the image appending the required size to the filename like http://imageserver/someimage.150x120.jpg
if the image exists, it will be returned with no other processing (this is the main point, the cache check is implicit)
if the image does not exist, handle the 404 not found via .htaccess and reroute the request to the script that generates the image of the required size
in the script specify the list of allowed sizes to avoid attacks like scripts requesting every possible size to shut your server down
keep this on a cookieless domain to minimize unnecessary traffic
EDIT: I don't think that PHP itself would slow the process much, as PHP scripting in this case is reduced to a minimum: the image scaling is done by a builtin library written in C. Whatever you do you'll have to use a library like this (GD or libmagick or so) so that's unavoidable. With my system at least you totally skip the overhead of checking the cache, thus further reducing PHP interaction. You can implement this on your existing server, so I guess it's a solution well suited for your budget.
Based on
We tried PHP scripts to do this but the performance was too slow for this many users.
I'm going to assume you weren't caching the results. I'd recommend caching the resulting images for a day or two (i.e. have your script check to see if the thumbnail has already been generated, if so use it, if it hasn't generate it on the fly).
This would improve performance dramatically as I'd imagine the main/start page probably has a lot more hits than random video X, thus when viewing the main page no images have to be created as they're cached. When User Y views Movie X, they won't notice the delay as much since it just has to generate that one page.
For the "On-the-fly resize" aspect - how important is bandwidth to you? I'd want to assume you're going through so much with movies that a few extra kb in images per request wouldn't do too much harm. If that's the case, you could just use larger images and set the width and height and let the browser do the scaling for you.
The ImageCache and Image Exact Sizes solutions from the Drupal community might do this, and like most solutions OSS use the libraries from ImageMagik
There are some AMI images for Amazons EC2 service to do image scaling. It used Amazon S3 for image storage, original and scales, and could feed them through to Amazons CDN service (Cloud Front). Check on EC2 site for what's available
Another option is Google. Google docs now supports all file types, so you can load the images up to a Google docs folder, and share the folder for public access. The URL's are kind of long e.g.
http://lh6.ggpht.com/VMLEHAa3kSHEoRr7AchhQ6HEzHVTn1b7Mf-whpxmPlpdrRfPW216UhYdQy3pzIe4f8Q7PKXN79AD4eRqu1obC7I
Add the =s paramter to scale the image, cool! e.g. for 200 pixels wide
http://lh6.ggpht.com/VMLEHAa3kSHEoRr7AchhQ6HEzHVTn1b7Mf-whpxmPlpdrRfPW216UhYdQy3pzIe4f8Q7PKXN79AD4eRqu1obC7I=s200
Google only charge USD5/year for 20GB. There is a full API for uploading docs etc
Other answers on SO
How best to resize images off-server
Ok first problem is that resizing an image with any language takes a little processing time. So how do you support thousands of clients? We'll you cache it so you only have to generate the image once. The next time someone asks for that image, check to see if it has already been generated, if it has just return that. If you have multiple app servers then you'll want to cache to a central file-system to increase your cache-hit ratio and reduce the amount of space you will need.
In order to cache properly you need to use a predictable naming convention that takes into account all the different ways that you want your image displayed, i.e. use something like myimage_blurred_320x200.jpg to save a jpeg that has been blurred and resized to 300 width and 200 height, etc.
Another approach is to sit your image server behind a proxy server that way all the caching logic is done automatically for you and your images are served by a fast, native web server.
Your not going to be able to serve millions of resized images any other way. That's how Google and Bing maps do it, they pre-generate all the images they need for the world at different pre-set extents so they can provide adequate performance and be able to return pre-generated static images.
If php is too slow you should consider using the 2D graphic libraries from Java or .NET as they are very rich and can support all your requirements. To get a flavour of the Graphics API here is a method in .NET that will resize any image to the new width or height specified. If you omit a height or width, it will resize maintaining the right aspect ratio. Note Image can be a created from a JPG, GIF, PNG or BMP:
// Creates a re-sized image from the SourceFile provided that retails the same aspect ratio of the SourceImage.
// - If either the width or height dimensions is not provided then the resized image will use the
// proportion of the provided dimension to calculate the missing one.
// - If both the width and height are provided then the resized image will have the dimensions provided
// with the sides of the excess portions clipped from the center of the image.
public static Image ResizeImage(Image sourceImage, int? newWidth, int? newHeight)
{
bool doNotScale = newWidth == null || newHeight == null; ;
if (newWidth == null)
{
newWidth = (int)(sourceImage.Width * ((float)newHeight / sourceImage.Height));
}
else if (newHeight == null)
{
newHeight = (int)(sourceImage.Height * ((float)newWidth) / sourceImage.Width);
}
var targetImage = new Bitmap(newWidth.Value, newHeight.Value);
Rectangle srcRect;
var desRect = new Rectangle(0, 0, newWidth.Value, newHeight.Value);
if (doNotScale)
{
srcRect = new Rectangle(0, 0, sourceImage.Width, sourceImage.Height);
}
else
{
if (sourceImage.Height > sourceImage.Width)
{
// clip the height
int delta = sourceImage.Height - sourceImage.Width;
srcRect = new Rectangle(0, delta / 2, sourceImage.Width, sourceImage.Width);
}
else
{
// clip the width
int delta = sourceImage.Width - sourceImage.Height;
srcRect = new Rectangle(delta / 2, 0, sourceImage.Height, sourceImage.Height);
}
}
using (var g = Graphics.FromImage(targetImage))
{
g.SmoothingMode = SmoothingMode.HighQuality;
g.InterpolationMode = InterpolationMode.HighQualityBicubic;
g.DrawImage(sourceImage, desRect, srcRect, GraphicsUnit.Pixel);
}
return targetImage;
}
In the time that this question has been asked, a few companies have sprung up to deal with this exact issue. It is not an issue that's isolated to you or your company. Many companies reach the point where they need to look for a more permanent solution for their image processing needs.
Services like imgix serve as a proxy and CDN for image operations like resizing and applying overlays. By manipulating the URL, you can apply different transformations to each image. imgix serves billions of requests per day.
You can also stand up services on your own and put them behind a CDN. Open source projects like imageproxy are good for this. This puts the burden of maintenance on your operations team.
(Disclaimer: I work for imgix.)
What you are looking for is best matched by Thumbor http://thumbor.readthedocs.org/en/latest/index.html , which is open source, backed by a huge company (means it will not disappear tomorrow), and ships with a lot of nice features like detecting what is important on an image when cropping.
For low-cost plus CDN I'd suggest to combine it with Cloudfront and AWS storage, or a comparable solution with a free CDN like Cloudflare. These might not be the best performing CDN providers, but at least still perform better than one server and also offload your image server on the cheap. Plus, it will save you a TON of bandwidth cost.
If each different image is uniquely identifiable by a single URL then I'd simply use a CDN such as AKAMAI. Let your PHP script do the job and let AKAMAI handle the load.
Since this kind of business doesn't usually have budget problems, that'd be the only place I'd look at.
Edit: that works only if you do find a CDN that will serve this kind of content for you.
This exact same problem is now being solved by image resize services dedicated to this task. They provide following features:
In built CDN - you need not worry about image distribution
Image resize on the fly - any size needed is available
No storage needed - you just store base image and all variants are handled by service
Ecosystem libraries - you can just include javascript and your job is done for all devices and all browsers.
One such service is Gumlet. You can also try some open source alternative like nginx plugin which can also resize image on the fly.
(I work for Gumlet.)
We use large background images (hi-res photos, up to 700 KB) for our page design.
It's part of the experience of the site that as you browse around, you see different images.
At the moment a different (random) image is loaded on each page request, from a pool of ~15 images, which could grow over time.
I'm looking for a sane way to optimize this:
To avoid the user having to download a big image file on every page view
To reduce load on the server (is this an issue, will the server keep the images in memory?)
The ideas I have so far include:
A timer which loads a different image at set intervals
Progressively loading other images in the background with ajax
Associating images with specific content (pages, tags)
The question is, how to keep it feeling somewhat random, while minimizing page load times and server hit?
I usually avoid sites with huge images, I am very impatient. I would rethink your design.
As a first step you should make sure, that the images can be properly cached:
use sane urls (no session id's etc)
set appropriate http headers ETag
Firstly, hearing that the background-images alone are 700kb astounds me. In addition to the content ON screen...that is a pretty heavy site.
For starters, I would try to use image compression tools. Two tools come to mind Imagemagick and PNGCrush. PNGCrush is excellent in reducing all the extraneous metadata attached to photos, without compromising photo quality.
I only recommend this as compressing the images will assist you in enabling the user to download a smaller quantity of content, which means quicker load times, which...at the end of the day...is what users want.
I would also cache the images, such that when a user re-visits the site, the image is already cached on their end. This minimises the HTTP requests that are made each time a user visits your site.
An example of where this technique is used on a commercial site is www.reactive.com. If you look the /js/headerImages.js file, they make use of image caching. Funnily enough, you will find the same src code at: http://javascript.internet.com/miscellaneous/random-image.html
Considering that you have mentioned that images are randomly loaded, I am assuming you are using a Javascript library such as jQuery to create the effect.
If you are, you can minimize page load times by using a CDN as opposed to referencing to a local copy of the jQuery lib which is stored on your server. I have performed performance testing on a site I made for a client, and over an average of 20 hits, saved 1.6 seconds through this technique!
Hope that helps for now :)