I have the following problem with my own approach. I would like some input if there is a better way to do what I did.
In my app I get images from a source where I can not change much. There is e.g. a list of users with some info about them and an avatar. I get the data from a Rest API where the avatar is send as a bas64 string. Since base64 encoded image strings can get quite long my idea was the first time I get an image I will decode it and save it in a temporary directory using the path_provider plugin.
It looks like this right now:
Future<void> getMediaFile(int id) async {
Directory temp = await getTemporaryDirectory();
File imageFile = File('${temp.path}/images/${id.toString()}.jpg');
var response = await _httpClient.getRequest(ApiPath.mediaFileById(id.toString()));
var mediaFile = MediaFile.fromJson(response["mediafile"]);
Uint8List decodedImage = base64Decode(mediaFile.fileContent);
await imageFile.create(recursive: true);
await imageFile.writeAsBytes(decodedImage);
}
And whenever I need a mediafile I will check if there already is a mediafile locally with the specific id.
Is there a better way to do this?
Is this approach not good because of any other reason?
I found a "cached_network_image" package but this can only be used if I would get the images via an url, which I can't do, because I can't change the backend in this case.
As with many questions like this, "it depends".
For a small number of small (<100kb), infrequently accessed files, this is a reasonable approach. Filesystem access is usually fast enough to avoid any noticeable latency issues.
If it's something larger that you're going to display on every page, it might be worthwhile loading the image into memory when your application starts (after saving to the local filesystem the first time).
If it's just a 128x128 PNG avatar, I doubt it will make a difference (and all things considered, I like to keep as little in memory as possible, so in that case I'd just load from the filesystem).
cached_network_image is only for caching images retrieved from the web, with some extra niceties to display placeholders, so that's not really relevant.
Related
I have an app to create reports with some data and images (min 1 img, max 6). This reports keeps saved on my app, until user sent it to API (which can be done at the same day that he registered a report, or a week later).
But my question is: What's the proper way to store this images (I'm using Realm), is it saving the path (uri) or a base64 string? My current version keeps the base64 for this images (500 ~~ 800 kb img size), and then after my users send his reports to API, I deleted this base64 hash.
I was developing a way to save the path to the image, and then I display it. But image-picker uri returned is temporary. So to do this, I need to copy this file to another place, then save the path. But doing it, I got (for kind of 2 or 3 days) 2x images stored on phone (using memory).
So before I develop all this stuff, I was wondering, will it (copy image to another path then save path) be more performant that save base64 hash (to store at phone), or it shouldn't make much difference?
I try to avoid text only answers; including code is best practice but the question about storing images comes up frequently and it's not really covered in the documentation so I thought it should be addressed at a high level.
Generally speaking, Realm is not a solution for storing blob type data - images, pdf's etc. There are a number of technical reasons for that but most importantly, an image can go well beyond the capacity of a Realm field. Additionally it can significantly impact performance (especially in a sync'ing use case)
If this is a local only app, storing the images on disk in the device and keep a reference to where they are (their path) stored in Realm. That will enable the app to be fast and responsive with a minimal footprint.
If this is a sync'd solution where you want to share images across devices or with other users, there are several cloud based solutions to accommodate image storage and then store a URL to the image in Realm.
One option is part of the MongoDB family of products (which also includes MongoDB Realm) called GridFS. Another option is a solid product we've leveraged for years is called Firebase Cloud Storage.
Now that I've made those statements, I'll backtrack just a bit and refer you to this article Realm Data and Partitioning Strategy Behind the WildAid O-FISH Mobile Apps which is a fantastic article about implementing Realm in a real-world use application and in particular how to deal with images.
In that article, note they do store the images in Realm for a short time. However, one thing they left out of that (which was revealed in a forum post) is that the images are compressed to ensure they don't go above the Realm field size limit.
I am not totally on board with general use of that technique but it works for that specific use case.
One more note: the image sizes mentioned in the question are pretty small (500 ~~ 800 kb img size) and that's a tiny amount of data which would really not have an impact, so storing them in realm as a data object would work fine. The caveat to that is future expansion; if you decide to later store larger images, it would require a complete re-write of the code; so why not plan for that up front.
In one of my pages, i display multiple images. I get the images through a route, which is connected to a controller method. Like so:
In view:
(inside foreach)
<img width="100" src="viewBook/get/...." >
In controller method:
$img = Image::make('images/test.jpg');
return $img->response('jpg');
But this is such an slow approach. When i display let's say 13 images, it almost takes 10 seconds. And when i inspect through Chrome DevTools, most of the time is consumed as "Idle". And of course they get loaded one by one.
How can i make it faster ? Maybe parallel responding, or increasing ngingx bandwidth ?
Thanks in advance.
I suppose you are using the Intervention Image package. However, there is no need to put all images through this package. If you do, the package will process the image and this takes a lot of time.
If you just want to return the image, you can do it with a built-in file response.
return response()->file('images/test.jpg');
This will load the file directly from the file system and will be much faster.
It's very difficult thing to improve response time. It depends on various reasons such as server, internet speed, location, size of your images etc.
For such instances, services like imagix(https://www.imgix.com/) can be used, and it worked well for us. It's way cheaper than tuning other parameters to improve response time.
I have searched all over the web, but not able to find the answer.
We have two methods in our application:
a) First one return HttpResponseMessage with 1 file inside. It uses StreamContent.
response.Content = new StreamContent(memStream);
response.Content.Headers.ContentLength = memStream.Length;
b) Second one return HttpResponseMessage including zipp-ed files (multiple files that are zipped). it uses ByteArrayContent.
response.Content = new ByteArrayContent(memStream.ToArray());
response.Content.Headers.ContentLength = memStream.ToArray().Length;
I just wanted to understand why in our application StreamContent is used when returning just one file and ByteArrayContent is used when returning zip-ed file. Is there some logic there or not and I can change to use the same way in both situations?
Without anything to back up my assumption other than hearsay, streams are supposed to be more efficient than byte arrays (they basically work with smaller buffers of bytes at a time).
In the case of a web app, I believe streaming becomes even more efficient as it allows the consumer to actually download the page in pieces as it becomes available, rather than wait for all of the content to become ready in memory.
But it looks like your app is using a MemoryStream in both cases, and so practically speaking it might not make much of a difference (because the memory stream is a wrapper around a byte array...in memory). It is however calling memStream.ToArray() twice, which is less efficient as it copies its internal buffer to a new array a second time just to get its length (which you can call directly with memStream.Length.
Of course, without knowing what the rest of the app is doing, maybe there's a reason for it to marshal all of the zipped data before providing it.
How can I check if an image file is changed on a website, from another website and then store it and the old version?
I'm using this to log the images on the server.
This is just a quick sketch of the simplest approach. If you want more detail on some part, just ask in the comments.
Sketch of solution
Download the image, compute a hash for it and store the image in file system and image ID + hash + file system path (and possibly other info such as time of request) in database.
When checking for update, get last available info for the same ID from the database and if hashes are not the same, the image was not updated. If you use cryptographic hash like MD5 or SHA1 and the hash changed, it is almost sure that the image changed too.
Setup a cronjob to run the script periodically.
To download the image, you could use $img = file_get_contents($url);. MD5 can be computed via $hash = md5($img);, SHA1 via $hash = sha1($img);. For storing use file_put_contents($path, $img);.
Optimization
There are several ways to optimize the job.
To cut on memory consumption, download the file directly to file system using file_put_contents($path, $url); and only after that compute the hash using $hash = md5_file($path); or $sha1_file($path);. This is better for larger images. The bad thing is that you have to read the data from file system again to compute the hash, so I think it would be slower.
Side note: Never optimize anything before you know that it really makes the code better. Always measure before, after and compare.
Another optimization could be done for saving on data transfers from remote server if the server sends reliable headers for caching. ETag is the best one because it should be based on the contents of the file. If it does not change, file should be the same. If you want just to check the headers, use $headers = get_headers($url, 1);. To fetch really just the headers, you should issue just HTTP request via HEAD method instead of GET. Check get_headers() manual for more info. To check the headers while getting response body, use file_get_contents() along with $http_response_header special variable.
Issuing requests indicating that you cached the image on last visit (via If-Modified-Since et al.) could serve the same purpose.
Etiquette and legal aspects
I told you how. Now I’ll tell you when (not).
Do not abuse the remote server. Remember that its owner has expenses to keep it up and running and definitely does not want to let it be occupied by your scripts for more than a negligible amount of time, transferring not much data. Adapt your polling period to type of target server and size of image. Adapting it to estimated frequency of change is not a bad idea too.
Be sure to have consent of image rights holder when storing its copy. Licensing can be a messy thing. Be careful, otherwise you can get into trouble.
If you plan to somehow crawl for the images, robots.txt standard might be of your interest. This file could tell you that you are not welcome and you should respect it.
Related questions
Some are related more, some less. People want to watch mainly HTML pages. That has other specifics, which is also why I did not flag this question as duplicate of one of these.
https://stackoverflow.com/q/11336431/2157640
https://stackoverflow.com/q/11182825/2157640
https://stackoverflow.com/q/13398512/2157640
https://stackoverflow.com/q/15207145/2157640
https://stackoverflow.com/q/1494488/2157640
webapp2_extras' sessions are pretty useful, and I've been using them to keep users logged in and such. But there's not much documentation about it. For instance: which is the memory limit for an entry in the session dictionary?
So far I've stored strings and numbers. But I need to store a whole image, uploaded from a HTML form (the idea is that this image is temporary and it may be discarded, so I don't want to store it in the datastore until I'm sure I have to do it). It doesn't seem to work, and I guess I'm hitting a memory problem:
self.session['photo_image'] = self.request.get("photo_image")
Is that so? I guess there are memory limits, but I can't find them. How else could I store a temporary image in GAE?
You can store it in 'instance memory', e.g. create a global list when your script starts up and append incoming images to that, then remove it once done. Of course, you'll soon run of of memory there also if you have lots of users/large files. And you'll lose it all when the instance shuts down and you'll (might) have problems if more then one instance is running.
So Memcache sounds perfect for this.
using Memcache
def get_data():
data = memcache.get('key')
if data is not None:
return data
else:
data = self.query_for_data()
memcache.add('key', data, 60)
return data
Yes, it's not 100% reliable as I'm sure you've heard but if you are not using 100's of files that are huge and keeping them round for ages you probably won't have issues. As it's first in first out (IIRC) if you process them more or less in order that'll be even less likely to lose data. I think memcache is going to be your best option here, try it.