I have searched all over the web, but not able to find the answer.
We have two methods in our application:
a) First one return HttpResponseMessage with 1 file inside. It uses StreamContent.
response.Content = new StreamContent(memStream);
response.Content.Headers.ContentLength = memStream.Length;
b) Second one return HttpResponseMessage including zipp-ed files (multiple files that are zipped). it uses ByteArrayContent.
response.Content = new ByteArrayContent(memStream.ToArray());
response.Content.Headers.ContentLength = memStream.ToArray().Length;
I just wanted to understand why in our application StreamContent is used when returning just one file and ByteArrayContent is used when returning zip-ed file. Is there some logic there or not and I can change to use the same way in both situations?
Without anything to back up my assumption other than hearsay, streams are supposed to be more efficient than byte arrays (they basically work with smaller buffers of bytes at a time).
In the case of a web app, I believe streaming becomes even more efficient as it allows the consumer to actually download the page in pieces as it becomes available, rather than wait for all of the content to become ready in memory.
But it looks like your app is using a MemoryStream in both cases, and so practically speaking it might not make much of a difference (because the memory stream is a wrapper around a byte array...in memory). It is however calling memStream.ToArray() twice, which is less efficient as it copies its internal buffer to a new array a second time just to get its length (which you can call directly with memStream.Length.
Of course, without knowing what the rest of the app is doing, maybe there's a reason for it to marshal all of the zipped data before providing it.
Related
I have the following problem with my own approach. I would like some input if there is a better way to do what I did.
In my app I get images from a source where I can not change much. There is e.g. a list of users with some info about them and an avatar. I get the data from a Rest API where the avatar is send as a bas64 string. Since base64 encoded image strings can get quite long my idea was the first time I get an image I will decode it and save it in a temporary directory using the path_provider plugin.
It looks like this right now:
Future<void> getMediaFile(int id) async {
Directory temp = await getTemporaryDirectory();
File imageFile = File('${temp.path}/images/${id.toString()}.jpg');
var response = await _httpClient.getRequest(ApiPath.mediaFileById(id.toString()));
var mediaFile = MediaFile.fromJson(response["mediafile"]);
Uint8List decodedImage = base64Decode(mediaFile.fileContent);
await imageFile.create(recursive: true);
await imageFile.writeAsBytes(decodedImage);
}
And whenever I need a mediafile I will check if there already is a mediafile locally with the specific id.
Is there a better way to do this?
Is this approach not good because of any other reason?
I found a "cached_network_image" package but this can only be used if I would get the images via an url, which I can't do, because I can't change the backend in this case.
As with many questions like this, "it depends".
For a small number of small (<100kb), infrequently accessed files, this is a reasonable approach. Filesystem access is usually fast enough to avoid any noticeable latency issues.
If it's something larger that you're going to display on every page, it might be worthwhile loading the image into memory when your application starts (after saving to the local filesystem the first time).
If it's just a 128x128 PNG avatar, I doubt it will make a difference (and all things considered, I like to keep as little in memory as possible, so in that case I'd just load from the filesystem).
cached_network_image is only for caching images retrieved from the web, with some extra niceties to display placeholders, so that's not really relevant.
I've implemented Scala Akka application that streams 4 different types of data from biomodule sensor (ECG, EEG, Breath and general data). These data (timestamp and value) are typically stored in 4 different CSV files. However, sometimes I have to store each sample in two different files with different timestamps, so application is writing in 8 different CSV files at the same time.
Initially I've implemented one Akka actor that is responsible for persisting data, which receive path to the file in which to write data, timestamp and value. However, this was a bottleneck, since a number of samples that I need to store is large (e.g. one ECG sample is received each 4ms). As a result, this actor had finished recording in very short experiment 1-2 minutes after experiment is over.
I've also tried with 4 actors for 4 different message types, with the idea to distribute work. I didn't notice significant improvement in performances.
I'm wondering if someone has an idea how to improve the performance. Is it better to use one actor for storing files, few actors or it is most efficient if I have one actor for each file? Or maybe, it doesn't make any difference? Could I improve my code for storing data?
This is my method responsible for storing data:
def processValue(sample: WaveformValue): Unit ={
val csvfilewriter=new PrintWriter(new BufferedWriter(new FileWriter(sample.filepath,true)))
csvfilewriter.append(sample.timestamp.toString)
csvfilewriter.append(",")
csvfilewriter.append(sample.value.toString)
csvfilewriter.append("\r\n")
csvfilewriter.flush()
csvfilewriter.close()
}
It seems to me that your bottleneck is I/O -- disk access. It looks like you are opening, writing to, and closing a file for each sample, which is very expensive. I would suggest:
Open each file just once, and close it at the end of all processing. You might need to store the file in a member variable, or if you have have an arbitrary collection of files then store them in a map in a member variable.
Don't flush after every sample write.
Use buffered writes for each file writer. This avoids flushing data to the filesystem with every write, which involves a system call and waiting for the data to be written to disk. I see that you're already doing this, but the benefit is lost since you are flushing/closing the file after each sample anyway.
I'm confused on this situation:
I've a Producer which produces an undetermined number of items from an underlining iterator, possibly a large number of them.
Each item must be mapped to a different interface (eg, wrapper, JavaBean from JSON structure).
So, I'm thinking that it would be good for Producer to return a stream, it's easier to write code that convert Iterator to Stream (using Spliterators and StreamSupport.stream()), then apply Stream.map() and return the final stream.
The problem is I have an invoker that does nothing with the resulting stream, eg, a unit test, yet I still want the mapping code to be invoked for every item. At the moment I'm simply calling Stream.count() from the invoker to force that.
Questions are:
Am I doing it wrong? Should I use different interfaces? Note that I think implementing next()/hasNext() for Iterator is cumbersome, mainly because it forces you to create a new class (even if it can be anonymous) and keep a pointer and check it. Same for collection views, returning a collection that is created and not a dynamic view over the underlining iterator is out of question (the input data set might be very large). The only alternative I like so far is a Java implementation of yield(). Neither do I want the stream to be consumed inside Producer (ie, forEach()), since some other invoker might want it to perform some real operation.
Is there a better best practice to force the stream processing?
I am developing an application that will stream multimedia files over torrents.
The backend needs to serve new pieces to the frontend as they arrive.
I need a mechanism to get notified when new pieces have arrived and been verified. From what I can tell, I could do this using block_finished_alerts. I would keep track of which blocks have arrived for a given piece, and read the piece when all blocks have arrived.
This solution seems kind of roundabout and I was wondering if there was a better way.
What you're asking for is called piece_finished_alert. It's posted every time a new piece completes downloading and passes the hash-check. To read a piece from disk, you may use torrent_handle::read_piece() (and get the result in read_piece_alert).
However, if you want to stream media, you probably want to use torrent_handle::set_piece_deadline() and set the flag to send read_piece_alerts as pieces come in. This will invoke the built-in streaming feature of libtorrent.
I fetch images with open-uri from a remote website and persist them on my local server within my Ruby on Rails application. Most of the images were shown without a problem, but some images just didn't show up.
After a very long debugging-session I finally found out (thanks to this blogpost) that the reason for this is that the class Buffer in the open-uri-libary treats files with less than 10kb in size as IO-objects instead of tempfiles.
I managed to get around this problem by following the answer from Micah Winkelspecht to this StackOverflow question, where I put the following code within a file in my initializers:
require 'open-uri'
# Don't allow downloaded files to be created as StringIO. Force a tempfile to be created.
OpenURI::Buffer.send :remove_const, 'StringMax' if OpenURI::Buffer.const_defined?('StringMax')
OpenURI::Buffer.const_set 'StringMax', 0
This works as expected so far, but I keep wondering, why they put this code into the library in the first place? Does anybody know a specific reason, why files under 10kb in size get treated as StringIO ?
Since the above code practically resets this behaviour globally for my entire application, I just want to make sure that I am not breaking anything else.
When one does network programming, you allocate a buffer of a reasonably large size and send and read units of data which will fit in the buffer. However, when dealing with files (or sometimes things called BLOBs) you cannot assume that the data will fit into your buffer. So, you need special handling for these large streams of data.
(Sometimes the units of data which fit into the buffer are called packets. However, packets are really a layer 4 thing, like frames are at layer 2. Since this is happening a layer 7, they might better be called messages.)
For replies larger than 10K, the open-uri library is setting up the extra overhead to write to a stream objects. When under the StringMax size, it just includes the string in the message, since it knows it can fit in the buffer.