I'm storing a List with around 3,000 objects in Isolatedstorage using Xml serialize.
It takes too long to deserialize this and I was wondering if you have any recommendations to speed it up.
The time is tolerable to deserialize up to 500 objects, but takes forever to deserialize 3000.
Does it take longer just on the emulator and will be faster on the phone?
I did a whole bunch of searching and some article said to use a binary stream reader, but I can't find it. Whether I store in binary or xml doesn't matter, I just want to persist the List.
I don't want to look at asynchronous loading just yet...
Firstly, some good info here already, so +1 there.
I would recommend reviewing these articles, giving you some good perspective on what performance you can expect using a variety of out of the box serialisation techniques.
Windows Phone 7 Serialization: Comparison | eugenedotnet blog
WP7 Serialization Comparison
You might also consider using multiple files if you don't need to load and write everything in one hit all the time.
I would reiterate Jeff's advice that it would really be a good idea to get any substantial work you find remaining after this onto a background thread so as not to degrade the user interaction experience.
It's fairly straight forward. Here is a walkthrough I often recommend and people find concise and helpful.
Phạm Tiểu Giao - Threads in WP7
And also this, recently by Shawn Wildermuth which looks quite good too.
Shawn Wildermuth - Architecting WP7 - Part 9 of 10: Threading
Check out the binary serializer that is a part of sharpSerializer:
http://www.sharpserializer.com/en/index.html
It's very easy and works quite well.
Here's a blog that talks about using it in WP7:
http://www.eugenedotnet.com/2010/12/windows-phone-7-serialization-sharpserializer/
I am using it like (consider this psuedocode, and using the functions listed on eugenedotnet)
in App.xaml.cs:
Application_Dectivated()
{
IsolatedStorageFileStream stream = store.OpenFile(fileName, FileMode.OpenOrCreate);
Serialize(stream,(obj)MyHugeList<myObject>);
}
Application_Activated()
{
IsolatedStorageFileStream stream = store.OpenFile(fileName, FileMode.Open);
Deserialize(stream,(obj)MyHugeList<myObject>);
}
For that many items, you need to build your optimized serialization story. I see many people using simple CSV and text formats to do this.
The built-in serializers just aren't going to be fast enough.
You should really consider doing this all on a background thread for a lot of reasons, though yes you have indicated you do not want to do this.
Related
I know basic SQL, and SQL is all I know when it comes to storing and retrieving data. I want to create 1 .exe and it should contain all ~100,000 key-value pairs (i have the data in .txt files) and maybe an extra attribute for description (this I would add myself - like a note to myself).
I also would like to write it in a new language I don't know yet; like python or C# (I have made desktop apps written in Java & VB.net all with SQL databases). So language will not be an issue and I would appreciate suggestions.
These key-value pairs might not need to be updated and I'm willing to re-compile/repackage the code to make 1 change in the data. The key is 6 letters long and 2 numbers at the end like hxnaaa01. Each of these letters represent or describe something about itself so I would also need to search for a specific letter on a specific position to get exactly what I need.
I know that regex would work well with what I need but all I mentioned is all I know. I don't know enough and I don't know what keywords to google.
I have read about XML and CSV. I don't really know what they are and I'm not sure how all of this would fit in 1 executable.
To summarize, I need:
1 executable (Windows Desktop App)
Search function ~100k KVP+1more attribute (using regex?)
no database
with GUI
ability to add a "note" to each KVP
should be fast and lightweight
1 executable (Windows Desktop App), no database
Data persistence will require either additional files, or a database. It's pretty much unavoidable, you can store data in memory, but it's only persisted for as long as it resides there.
You have another requirement: "fast and lightweight".
To achieve this requirement, you'll need to really think about your solution, what technology you use and how you can improve it in future.
Although searching through data is pretty trivial, an efficient solution is not. It requires upfront research into algorithms, data structures and general practices. (which is a rabbit hole itself).
In the case of JSON [1], you'll need to create an additional file to contain all your key/value pairs, you can use C# to create the extra file (on first launch, for example).
JSON promises to be lightweight, I tend to agree, some may not. When dealing with the filesystem, I think it can be agreed is often far from lightweight solution.
JSON is very readable though:
{
"key": "value",
"comment": "oh this is cool"
}
There's a lot of factors that play into something being fast and lightweight, so there's a need for some research on your part.
Honestly, depending on your experience, I wouldn't focus so much on the fast, I'd focus more on it working, then refactor that into something that's fast if it's too slow. [2]
And again, depending on your experience, I'd stick to opening the file, using a for/loop to find my key and do something with the data found, plus reward myself for having something that works.
TL;DR: you need either a file, or database for truly persistent storage, JSON or a remotely hosted MySQL would work. Try not to focus too much on fast before you have something that works.
https://www.json.org/json-en.html [1]
https://stackoverflow.com/a/5581595/2932298
https://stackify.com/premature-optimization-evil/ [2]
I am working on custom SAPUI5 app using ODataModel and for which I have to do the formatting for some of the fields which I will be displaying in List control.
I need to know which approach is good mentioned below is good w.r.t. performance of app.
1) Is it a good idea to use Formatter.js file and write each method for each field for formatting?
Example -
There are 2 fields which should be formatted before showing in UI and hence 2 formatter function.
2) Before binding Model to List - Do the formatting using Loop at each row.
Example -
Loop at OData.
--do formatting here for both the fields
move data to model.
Endloop.
Bind new model to UI
Is there any other way by which we can improve performance - apart from code minification or using grunt.
Appreciate your help.
Thanks,
Rahul
Replacing Formatters with other solutions is definitely NOT the point to start when optimizing performance not to mention that you will loose a lot of the convenience the ODataModel comes with when manually manipulating the data in it.
Formatter Performance
Anyways using a formatter is of course less performant then pre-formatting your data once after they were loaded. A formatter will be executed on every rerendering of your control. So you might not want to do heavy calculations or excessive looping in a formatter that is executed frequently. But given a normal usage using formatters is absolutely nothing you should worry about or that does noticeably affect end-user experience. Keep enjoying the convenience of formatters (and take a look at the cool Expression Binding).
General Performance Considerations
To improve performance it is first of all very important to identify the real bottle neck. In many cases this is simply the backend, there is usually much more to win with much less effort. Always keep that in mind. UI Code optimization is ridiculous as long as the main backend call runs for say 3s.
Things to improve your UI performance might be:
serve SAPUI5 from a CDN
use a Component-preload, can be generated with grunt-openui5 or gulp-ui5-preload (I think it does not minify XML yet so you could do that additionaly before creating Component-preload)
try to reduce the number of SAPUI5 libraries you are using
be aware of which SAPUI5 libraries you are NOT using and very consequently remove those (don't forget the dependencies section in Component metadata resp. manifest.json)
be aware that sap.ui.layout is a separate independent library (not registering it as such will result in a lot of extra requests)
if you use an ODataModel make sure you set useBatch to true (default in v2.ODataModel)
intelligently design your OData service (if you can influence it)
intelligently use $expands: sometimes it can make sense to preload $expand data on a parent binding that does not actually use it e.g. if you most probably need the data later on
think about bundling your app as native app and benefit from improved caching (Kapsel)
Check Performance: Speed Up Your App and Performance Issues
squeeze out some more bytes and save some requests by minifying/combining custom css or other resources if you have some
If you are generally interested in Web Performance I can recommend Steve Souders books.
I'm totally open for more ideas on SAPUI5 performance improvements! Anyone?
BR
Chris
the best practice is to do it this way. The formatter allows you to receive an input and return output. The formatter function will be called in runtime and will be called for each of the rows which are displayed in your list. The reason that it will be called for each of the rows is because that you cannot grantee that the input will be the same for all of the rows in the list.
The concept of binding is to loop on your data model and update the UI accordingly. It is much better to use binding because a lot of reasons like: maintainability, performance, separate the data layer from the presentation layer, core optimizations and more.
I have a question about the IsolatedStorageSettings in WP7.
normally i parsed all my Lists etc with json and then saved it to a IsolatedStorageFile. but now i'm asking myself why i'm doing this. isn't is easyer simply to save the lists etc to the IsolatedStorageSettings?
does the IsolatedStorageSettings have any limitations about the size? these are no lists with 100000 entries.
or is there any argument against these?
IsolatedStorageSettings was not developed to hold large quantities of application data like you propose. While you may currently be able to store data there, there is no guarantee that future updates won't fundamentally change the way this area works, thereby exposing yourself to risks about your app's function.
It is probably advisable, therefore, to stick with using the "normal" IsolatedStorageFile techniques you said you already use.
Nope, none at all. As long as your phone has > 10% of storage space you can do whatever you want. Who cares about everyone else's applications.
If the user says O.K. to the warning message then you can save more, but it could cause the application to crash
Howdy,
I was wondering if I was able to alter a distinct element in an XML file saved on a Windows Phone 7 device without actually having to serialize the whole file all over again.
As mentioned in previous answers, you can't save a fragment of XML on its own without saving the whole file. If file size was a big issue, then you could split the data into separate files (perhaps alphabetically; depends on the data), so that you're only saving a smaller dataset if you make a change.
As Matt mentioned, XML serialization does not provide the best performance on WP7 devices. Kevin Marshall has a great blog post detailing different serialization approaches and the performance of each. The fastest method is binary serialization, though there's nothing stopping you serializing the XML using the binary serialization approach.
In general, no - and this has little to do with Windows Phone 7 (although I don't know whether IsolatedStorageFileStream on WP7 even supports seeking).
I don't know of any mainstream filesystems with high level abstractions (such as those used by Java and C#) which allow you to delete or insert data in the middle of a file.
I suppose theoretically if you were happy to pad with whitespace, or never change the length of the data you're using, you could just overwrite the relevant bytes - but I don't think it would be a good idea at all. Very brittle and hard to work with.
Just go for overwriting the whole file.
I've read about systems which use the Flickr database of photos to fill in gaps in photos (http://blogs.zdnet.com/emergingtech/?p=629).
How feasible is a system like this? I was toying with the idea (not just a way of killing time but as a good addition to something I am coding) of using Flickr to get photos of a certain entity (in this case, race tracks) and reconstruct a model. My biggest concern is that there aren't enough photos of a particular track and even then, it would be difficult to tell if two photos are of the same part of the racetrack, in which case one of them may be irrelevant.
How feasible is something like this? Is it worth attempting by a sole developer?
Sounds like you're wanting to build a Photosynth style system - check out Blaise Aguera y Arcas' demo at TED back in 2007. There's a section about 4 minutes in where he builds a model of the Sagrada Família from photographs.
I say +1 for photosynth answer, its a great tool. Not sure how well you could incorporate it into your own app though.
Its definately feasable. Anything is possible. And yes, doable for a single developer, just depends how much free time you have. It would be great to see something like this integrated into Virtual Earth or Google Maps Street View. Someone who could nail some software like this could help 3D model the entire world based purely on photographs. That would be a great product and make any single developer rich and famous.
So get coding. :)
I have plenty of free time, as I am in between jobs.
One way to do it is to get an overhead view of the track layout, make a blueprint based on this model, and then get one photo of the track and mimic the track's road colour. That would be a start.
LINQ to Flickr on codeplex has a great API and would be helpful for your task.