Resampling HTMLImageElement for animation - html5-canvas

An HTMLVideoElement can be resampled in order to get different frames into a texture over time.
For example, as shown at https://developer.mozilla.org/en-US/docs/Web/API/WebGL_API/Tutorial/Animating_textures_in_WebGL
However- when loading an animated gif into a HTMLImageElement, resampling does not show the updated texture. This is true even if the image is mounted on the dom and the different frames show on that copy.
Is there a standard way to display an animated gif in webgl, or must it be somehow rewritten into a spritesheet (or series of textures) at runtime?


GIFs aren't automatically animated with WebGL (or regular canvas for that matter) and there is no standard way of doing this.
Contrary to video elements GIF images will only draw the first frame via drawImage() while drawing video elements will draw current frame. This is in part because we don't really have access to any of the image's frames via API (this also applies to animated PNG files, aka APNG) and animated images will be handled only as an internal process conducted at the discretion of the browser and only when in DOM.
With video elements though we do have access to "frames", that is, time via currentTime so it's sort of implies that we want to deal with what we see or what exist at the current time.
You have to manually animate a GIF image though. This means you have to extract each frames as separate images/buffers first then show them at the rate you chose. The browser won't help you a bit here, but you can do this by parsing the file format manually.
Of course, this can be a bit tedious, but luckily there are people out there that has done all the lifting and hard work. For example gifuct (I have not tested it myself but there are others out there as well) will allow you to extract each frame from a GIF
Then render each frame you got from that into the frame buffer and upload to the GPU at the frame rate you choose.
Or:
pre-process the GIF into a spritesheet as you mention
or load it as an image sequence instead
or convert the GIF to a video (this may even reduce the total size)
And as a shameless plug if you should consider APNG instead: I have made apng-parser which does the same for APNG files.
My recommendation though is to convert the GIF/APNG to a video file which gives the animation capabilities for free, potentially smaller files, can be buffered and streamed for long animations, less code to include and typically a single file to deal with (you may have to provide different video formats for older browsers). Free software such as FFMpeg can help you with the conversion.

Related

animated gif vs video vs canvas - for speed & file size

Assuming a simple product demo e.g. the one found on http://www.sublimetext.com/
i.e. something this isn't traditional high res video and could be reasonable accomplished with:
animated gif
video (can be embedded youtube, custom html5 player, whatever is most competitive)
canvas
The question is, which performs better for the user? Both in terms of:
The size of the files the user must be downloaded to view the 'product demo'
The requirements in terms of processing power to display the 'product demo'
If you feel that there's a superior technology to accomplish this or another metric to judge its usefulness, let me know and I'll adjust accordingly.

I know it's already answered, but as you specifically referred to the Sublime Text animation I assume you're wanting to create something similar?
If that's the case then here is a post explaining how it was created by the Sublime Text author, himself:
http://www.sublimetext.com/~jps/animated_gifs_the_hard_way.html
The interesting part of the article is how he reduces the file size - which I believe is your question.

With a simple animation such as the one at the link you're referring to, with a very low frame rate, a simple animated-PNG of animated GIF will probably be the best solution.
However, you need to consider band-width factor in this. If the final size of the GIF or the PNG is large then probably a buffered video is probably better.
This is because the whole gif/png file needs to be downloaded before it shows (I am not sure how interleaved PNGs works when they contain animation though).
A video may be larger in file size, but as it is typically buffered you will be able to show the animation almost right away.
Using external hosts such as YouTube or others can be beneficial to your site as well as the band-width is drawn from those site and not from your server (in case you use a provider that limits or charge for this in various ways).
For more information on animated PNGs or APNG (as this is not so well-known):
https://en.wikipedia.org/wiki/APNG
The canvas in this is only a displaying device and not really necessary (an image container does the same job and can also animate the GIF/PNG whereas a canvas cannot).
If you use a lot of vectors then canvas can be considered.
CSS3 animation is also an option for things such as presentation slides.

Are there any benefits to using bitmaps?

I'm porting some CF 2.0 VB.Net apps to a newer version of a handset that has twice the screen resolution. So I have to double the dimensions of everything otherwise it all gets squished up into the top LH corner of the screen.
One screen had a bitmap which was 250K in size, and after I doubled the dimensions naturally it blew out to one MB. This isn't real good on a handheld, so I fired up irfanview and converted it to a .GIF. The .GIF was only 60KB in size, with no discernible change in the quality of the image.
To me, it seems a no-brainer : Convert all Bitmaps to Gif (or JPG) and get the same results for a fraction of the disk space (and probably quicker form loading times).
But does anyone know of a situation where you would use a bitmap in preference to a GIF/JPEG? I cannot find any.

I really can't think of any realistic example where you would prefer an bitmap to a GIF. Since GIF is a lossless format you loose no information when storing images. So after reading the file in your app you will have the same image data as if you have read a bitmap. And like you said: The file will be smaller and thus will probably will be read faster from disk.
JPEG is different because it's a lossy format, meaning you will lose information when storing images in it. You will need to decide if the loss of information is meaningful in your app.
Bitmaps would be preferable if and only if reading files from disk where faster than decompressing the file in memory.
And to be precise you would prefer bitmaps when storing images in main memory, so you can work easily on the data in your code. Which is actually what you most likely already have when you have loaded a file using an image library.

To cut a long story shorts, a BMP is stored as a series of pixels along with their colour. This is useful if you want to do such things as pattern recognition, movement detection and such like.
Bitmaps are typically used for their convenience - you can knock one up in paint without having specialist graphics software.

Detect frames that have a given image/logo with FFmpeg

I'm trying to split a video by detecting the presence of a marker (an image) in the frames. I've gone over the documentation and I see removelogo but not detectlogo.
Does anyone know how this could be achieved? I know what the logo is and the region it will be on.
I'm thinking I can extract all frames to png's and then analyse them one by one (or n by n) but it might be a lengthy process...
Any pointers?

ffmpeg doesn't have any such ability natively. The delogo filter simply works by taking a rectangular region in its parameters and interpolating that region based on its surroundings. It doesn't care what the region contained previously; it'll fill in the region regardless of what it previously contained.
If you need to detect the presence of a logo, that's a totally different task. You'll need to create it yourself; if you're serious about this, I'd recommend that you start familiarizing yourself with the ffmpeg filter API and get ready to get your hands dirty. If the logo has a distinctive color, that might be a good way to detect it.
Since what you're after is probably going to just be outputting information on which frames contain (or don't contain) the logo, one filter to look at as a model will be the blackframe filter (which searches for all-black frames).

You can write a detect-logo module, Decode the video(YUV 420P FORMAT), feed the raw frame to this module, Do a SAD(Sum of Absolute Difference) on the region where you expect a logo,if SAD is negligible its a match, record the frame number. You can split the videos at these frames.
SAD is done only on Y(luma) frames. To save processing you can scale the video to a lower resolution before decoding it.

I have successfully detect logo using a rpi and coral ai accelerator in conjunction with ffmeg to to extract the jpegs. Crop the image to just the logo then apply to your trained model. Even then you will need to sample a minute or so of video to determine the actual logos identity.

Image format to put inside PDF's to have fast rendering

I would like to know which image format inside PDF's is rendered fastest. I tested mupdf code and I figured out that image decoding takes an important part in rendering time. So I would like to know if there are image formats that would not impact very much on cpu load.

I dont think this is really a question of what is best simply within PDFs, however:
As a general rule, I have always found that pre-rendering the image's size to the actual size you wish to present on screen is the best way to get both size and rendering speed to what you want them to be. Simply dragging an image into a document doesnt bring the pixel count (thus size) down as most document types simply put a display size tag around the full image. This causes the display program to have to real-time resize the image for display. The less the display program has to real-time resize the image the faster it will display.
As for file types:
Bitmaps are generally considered the fastest to display as they (for the most part) are copy and paste the color for each pixel onto the screen pixel. They are generally considered the biggest file. Depending on your images, if they aren't noisy (have a lot of solid runs of the same color) then they can be RLE encoded. I have seen many RLE encoded images that are indeed even smaller than JPEG images, but it is very situational.
JPEGs tend to be the smallest for transfer and also generally display decently quick. As an opinion they are also the lowest quality images (look close, if you started with a perfectly clean image, JPEG compression will add noise to it unless using lossless compression)
PNGs tend to be my favorite. They can be lossless compressed, can be fairly small if using flattened PNGs (i.e. NOT ADOBE FIREWORKS PNGs) and do produce crisp images that render fairly quickly.
So to sum up: I would probably recommend flattened PNGs that have been pre-sized and saved to the size you wish to display on screen.

What are the different usecases of PNG vs. GIF vs. JPEG vs. SVG? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 1 year ago.
Improve this question
When should certain image file types be used when building websites or interfaces, etc?
What are their points of strength and weakness?
I know that PNG & GIF are lossless, while JPEG is lossy.
But what is the main difference between PNG & GIF?
Why should I prefer one over the other?
What is SVG and when should I use it?
If you don't care about each and every pixel, should you always use JPEG since it's the "lightest" one?

You should be aware of a few key factors...
First, there are two types of compression: Lossless and Lossy.
Lossless means that the image is made smaller, but at no detriment to the quality.
Lossy means the image is made (even) smaller, but at a detriment to the quality. If you saved an image in a Lossy format over and over, the image quality would get progressively worse and worse.
There are also different colour depths (palettes): Indexed color and Direct color.
Indexed means that the image can only store a limited number of colours (usually 256), controlled by the author, in something called a Color Map
Direct means that you can store many thousands of colours that have not been directly chosen by the author
BMP - Lossless / Indexed and Direct
This is an old format. It is Lossless (no image data is lost on save) but there's also little to no compression at all, meaning saving as BMP results in VERY large file sizes. It can have palettes of both Indexed and Direct, but that's a small consolation. The file sizes are so unnecessarily large that nobody ever really uses this format.
Good for: Nothing really. There isn't anything BMP excels at, or isn't done better by other formats.
GIF - Lossless / Indexed only
GIF uses lossless compression, meaning that you can save the image over and over and never lose any data. The file sizes are much smaller than BMP, because good compression is actually used, but it can only store an Indexed palette. This means that for most use cases, there can only be a maximum of 256 different colours in the file. That sounds like quite a small amount, and it is.
GIF images can also be animated and have transparency.
Good for: Logos, line drawings, and other simple images that need to be small. Only really used for websites.
JPEG - Lossy / Direct
JPEGs images were designed to make detailed photographic images as small as possible by removing information that the human eye won't notice. As a result it's a Lossy format, and saving the same file over and over will result in more data being lost over time. It has a palette of thousands of colours and so is great for photographs, but the lossy compression means it's bad for logos and line drawings: Not only will they look fuzzy, but such images will also have a larger file-size compared to GIFs!
Good for: Photographs. Also, gradients.
PNG-8 - Lossless / Indexed
PNG is a newer format, and PNG-8 (the indexed version of PNG) is really a good replacement for GIFs. Sadly, however, it has a few drawbacks: Firstly it cannot support animation like GIF can (well it can, but only Firefox seems to support it, unlike GIF animation which is supported by every browser). Secondly it has some support issues with older browsers like IE6. Thirdly, important software like Photoshop have very poor implementation of the format. (Damn you, Adobe!) PNG-8 can only store 256 colours, like GIFs.
Good for: The main thing that PNG-8 does better than GIFs is having support for Alpha Transparency.
PNG-24 - Lossless / Direct
PNG-24 is a great format that combines Lossless encoding with Direct color (thousands of colours, just like JPEG). It's very much like BMP in that regard, except that PNG actually compresses images, so it results in much smaller files. Unfortunately PNG-24 files will still be bigger than JPEGs (for photos), and GIFs/PNG-8s (for logos and graphics), so you still need to consider if you really want to use one.
Even though PNG-24s allow thousands of colours while having compression, they are not intended to replace JPEG images. A photograph saved as a PNG-24 will likely be at least 5 times larger than a equivalent JPEG image, with very little improvement in visible quality. (Of course, this may be a desirable outcome if you're not concerned about filesize, and want to get the best quality image you can.)
Just like PNG-8, PNG-24 supports alpha-transparency, too.
SVG - Lossless / Vector
A filetype that is currently growing in popularity is SVG, which is different than all the above in that it's a vector file format (the above are all raster). This means that it's actually comprised of lines and curves instead of pixels. When you zoom in on a vector image, you still see a curve or a line. When you zoom in on a raster image, you will see pixels.
For example:
This means SVG is perfect for logos and icons you wish to retain sharpness on Retina screens or at different sizes. It also means a small SVG logo can be used at a much larger (bigger) size without degradation in image quality -- something that would require a separate larger (in terms of filesize) file with raster formats.
SVG file sizes are often tiny, even if they're visually very large, which is great. It's worth bearing in mind, however, that it does depend on the complexity of the shapes used. SVGs require more computing power than raster images because mathematical calculations are involved in drawing the curves and lines. If your logo is especially complicated it could slow down a user's computer, and even have a very large file size. It's important that you simplify your vector shapes as much as possible.
Additionally, SVG files are written in XML, and so can be opened and edited in a text editor(!). This means its values can be manipulated on the fly. For example, you could use JavaScript to change the colour of an SVG icon on a website, much like you would some text (ie. no need for a second image), or even animate them.
In all, they are best for simple flat shapes like logos or graphs.

JPEG is not the lightest for all kinds of images(or even most). Corners and straight lines and plain "fills"(blocks of solid color) will appear blurry or have artifacts in them depending on the compression level. It is a lossy format, and works best for photographs where you can't see artifacts clearly. Straight lines(such as in drawings and comics and such) compress very nicely in PNG and it's lossless. GIF should only be used when you want transparency to work in IE6 or you want animation. GIF only supports a 256 color pallete but is also lossless.
So basically here is a way to decide the image format:
GIF if needs animation or transparency that works on IE6(note, PNG transparency works after IE6)
JPEG if the image is a photograph.
PNG if straight lines as in a comic or other drawing or if a wide color range is needed with transparency(and IE6 is not a factor)
And as commented, if you are unsure of what would qualify, try each format with different compression ratios and weigh the quality and size of the picture and choose which one you think is best. I am only giving rules of thumb.

I usually go with PNG, as it seems to have a few advantages over GIF. There used to be patent restrictions on GIF, but those have expired.
GIFs are suitable for sharp-edged line art (such as logos) with a limited number of colors. This takes advantage of the format's lossless compression, which favors flat areas of uniform color with well defined edges (in contrast to JPEG, which favors smooth gradients and softer images).
GIFs can be used for small animations and low-resolution film clips.
In view of the general limitation on the GIF image palette to 256 colors, it is not usually used as a format for digital photography. Digital photographers use image file formats capable of reproducing a greater range of colors, such as TIFF, RAW or the lossy JPEG, which is more suitable for compressing photographs.
The PNG format is a popular alternative to GIF images since it uses better compression techniques and does not have a limit of 256 colors, but PNGs do not support animations. The MNG and APNG formats, both derived from PNG, support animations, but are not widely used.

JPEG will have poor quality around sharp edges etc. and for this reason it is unsuitable for most web graphics. It excels at photographs.
Compared to GIF, PNG offers better compression, larger pallette and more features, including transparency. And it is lossless.

GIF is limited to 256 colors and do not support real transparency. You should use PNG instead of GIF because it offers better compression and features. PNG is great for small and simple images like logos, icons, etc.
JPEG has better compression with complex images like photos.

As of 2018, we have several new formats, better support for previous formats and some clever hacks of using videos instead of images.
For photographs
jpg - still the most widely supported image format.
webp - New format from google. Good potential, though browser support is not great.
For Icons and graphics
svg - whenever possible. It scales well in retina screens, editable in text editors and customisable via JS/CSS if loaded in DOM.
png - if it involves raster graphics (ie when created in photoshop). Supports transparency which is very essential in this use-case.
For Animations
svg - plus css animations for vector graphics. All advantages of svg + power of css animations.
gif - still the most widely supported animated image format.
mp4 - if animated images are actually short video clips. Twitter / Whatsapp converts gifs to mp4.
apng - decent browser support (i.e. no IE, Edge), but creating it is not as straightforward as gifs.
webp - close to using mp4. Poor support
This is a nice comparison of various animated image formats.
Finally, whichever be the format, make sure to optimize it - There are tools for each format (eg SVGO, Guetzli, OptiPNG etc) and can save considerable bandwidth.

There is a hack that can be done to use GIF images to show true color. One can prepare a GIF animation with 256 color paletted frames with 0 frame delay and set the animation to be shown only once. So, all frames could be shown at the same time. At the end, a true colored GIF image is rendered.
Many software is capable of preparing such GIF images. However, the output file size is larger than a PNG file. It must be used if it is really necessary.
Edit: As #mwfarnley mentioned, there might be hiccups. Still, there are still possible workarounds. One may see a working example here. The final rendered image looks like that:
full-color-gif-image

png has a wider color pallete than gif and gif is properitary while png is not. gif can do animations, what normal-png cannot. png-transparency is only supported by browser roughly more recent than IE6, but there is a Javascript fix for that problem. Both support alpha transparency.
In general I would say that you should use png for most webgraphics while using jpeg for photos, screenshots, or similiar because png compression does not work too good on thoose.

GIF based on a palette of 256 colours per image (at least in its basic incarnation). PNG can do "TrueColour", i.e. 16.7 Million colours out of the box. Lossless PNG compresses better than lossless GIFs. GIF can do "binary" transparency (0% opacity or 100% opacity). PNG can handle alpha transparencies.
All in all, if you don't need to use Alpha-transparent images and support IE6, PNG is probably the better choice when you need pixel-perfect images for vector illustrations and such. JPG is unbeatable for photographs.

Here's an updated answer that includes WebP format:
JPEG:
The JPEG file format was created to optimize photos and other images
that use complex color ranges.
When saving a JPEG (e.g. in Photoshop) you can set the optimization level you want to achieve from lossless meaning no detail is lost to extremely lossy.
In most cases for web applications, you can set the compression to
75% without much losing details.
When to use JPEG? Anytime you have a photo or a graphic with complex color gradients and you can't use webP.
PNG
PNG is primarily a lossless bitmap image format for HQ computer generated images.
Unlike a JPEG, it can have a transparent layer. When you see a transparent image or graphic on the web it's usually a PNG.
When to use PNG? Anytime you have a computer generated graphic or an image with transparency. PNG is not recommended for regular photos as
the file size will generally be significantly larger than the
equivalent JPEG or webP.
GIF:
GIF is a 256 color graphic format supporting both images and animations.
Way back, GIFs were often used for simple graphics and then were slowly replaced by JPEG and PNG.
GIF Images: Low file size and low quality. They have almost no colored depth they only have 256 colors to work with. Replace them with SVGs.
Animated GIF: They can become very large very quickly and can potentially create huge performance issues. Replace them with videos. (Twitter for example, converts all animated GIFs that are uploaded into standard video files and then share those video files instead of the animated GIF.)
When to use GIF? For a web application, just don't! Replace GIF images with SVGs; Replace animated GIFs with videos.
SVG
SVG is a web native graphics format describing lines and curves and shapes and allowing the browser to draw the graphics in real time.
SVGs are scalable, meaning the graphic will look good at any size, all the way from tiny to extremely large.
Even CSS can be applied to them.
When to use SVG? Anytime you have a computer generated graphic
that may need to be scaled or needs to be responsive - like an icon, a logo or a graph.
WebP
A brand new lossless and lossy image format with transparency that was created specifically for the web.
It aims to be a replacement for JPEG as it provides good compression with the addition of transparency.
WebP is supported by all modern browsers but have zero support in older browsers so using them right now requires a fallback using an older image format.
When to use webP? For images and computer graphics if you know your
audience will be using newer browsers. If you need to support older
browsers provide fallbacks to either JPEG or PNG.

The main difference is GIF is patented and a bit more widely supported. PNG is an open specification and alpha transparency is not supported in IE6. Support was improved in IE7, but not completely fixed.
As far as file sizes go, GIF has a smaller default color pallet, so they tend to be smaller file sizes at first glance. PNG files have a larger default pallet, however you can shrink their color pallet so that, when you do, they result in a smaller file size than GIF. The issue again is that this feature isn't as supported in Internet Explorer.
Also, because PNGs can support alpha transparency, they're the only option if you want a variation of transparency other than binary transparency.

If you opt for JPEG, and you're dealing with images for a website, you may want to consider the Google Guetzli perceptual encoder, which is freely available. In my experience, for a fixed quality Guetzli produces smaller files than standard JPEG encoding libraries, while maintaining full compatibility with the JPEG standard (so your images will have the same compatibility as common JPEG images).
The only drawback is that Guetzli takes lot of time to encode.. but this is done only once, when you prepare the image for the website, while the benefits remains forever! Smaller images will take less time to download, so your website speed will increase in the everyday use.

GIF has 8 bit (256 color) palette where PNG as upto 24 bit color palette. So, PNG can support more color and of course the algorithm support compression

As pointed out by #aarjithn, that WebP is a codec for storing photographs.
This is also a codec to store animations (animated image sequence). As of 2020, most mainstream browsers has out of the box support for it (compatibility table).
Note for WIC a plugin is available.
It has advantages over GIF because it is based on a video codec VP8 and has a broader color range than GIF, where GIF limits to 256 colors it expands it to 224 = 16777216 colors, still saving significant amount of space.

Resources