Pretty much we've all done an installer here and there - and all of us did an installation of some behemoth of a program. Why do some installations take so much time? Case in point: Adobe CS suite (with newer versions you can take a vacation) or Visual Studio.
I know there are files to copy - most of the time unpack even. There are some registry keys to set (if under Windows), maybe a service or couple to start. Some installations probably even check hardware/software combination. All of this does not justify sllloooow installation time in some of the programs.
How can I speed it up?
It obviously depends what you're installing As Colin Pickard pointed out, you'll be shifting huge quantities of data onto the disk (+optional virus check etc.).
For installations I've built recently, we have to request the shut down of some Windows services, wait for that, and check that they really have shut down before continuing. That takes time.
I confess that in the above, that's not parallelised, whereas it could be. I suspect that installations are not necessarily optimised. They may well be the last thing that the team put together prior to release, and they may well figure that you're only going to do it once (and forget the pain upon completion). Obviously not an ideal state of affairs!
Visual Studio on my machine is 3.03GB - 16,842 files in 1,979 folders. Passing 3GB through virus scan and auditing software and onto the filesystem is too much for my (dualcore,2GB,sata2) system - it's CPU or IO bound the whole way through the process. That's why it takes so long.
Most installers not only pack, but also compress their contents, so at installation time all of these files must be decompressed. All of the data that is decompressed must be written to disk after it is decompressed as well.
Look at the time a zip operation takes on several files. It's also slow.
Many installers maintain a log that is flushed to the disk after each primitive operation so that even if installation encounters a fatal failure the log is preserved and can be sent to the software vendor. Such flushing sums up and significantly contributes to overall time.
Related
For my Wix project I am harvesting 4 directories, via the pre-build-event of visual studio, which will result in about 160mb of data, and about 220 files, but the build process tooks very long.
How can i speed that process up? I have one embedded media.cab file which will hold all the files. Is it the size or the amount of files that will slow the process down? Or is it the harvesting with the heat tool in the pre-build-event? Would it be faster with the HeatDirectory element?
Anyone made some experience with speeding this up?
For us, the vast majority of the time was spent invoking light (for the linking phase).
light is very slow at compressing cabinets. Changing the DefaultCompressionLevel in the .wixproj from high to mszip (or low or none) helps a lot. However, our build was still too slow.
It turns out that light handles cabinets independently, and automatically links them on multiple threads by default. To take advantage of this parallelism, you need to generate multiple cabinets. In our .wxs Product element, we had the following that placed everything in a single cabinet:
<MediaTemplate EmbedCab="yes" />
It turns out you can use the MaximumUncompressedMediaSize attribute to declare the threshold (in MB) at which you want files to be automatically sharded into different .cab files:
<MediaTemplate EmbedCab="yes" MaximumUncompressedMediaSize="2" />
Now light was much faster, even with high compression, but still not fast enough for incremental builds (where only a few files change).
Back in the .wixproj, we can use the following to set up a cabinet cache, which is ideal for incremental builds where few files change and most cabinets don't need to be regenerated:
<CabinetCachePath>$(OutputPath)cabcache\</CabinetCachePath>
<ReuseCabinetCache>True</ReuseCabinetCache>
Suppressing validation also gives a nice speedup (light.exe spends about a third of its time validating the .msi by default). We activate this for debug builds:
<SuppressValidation>True</SuppressValidation>
With these changes, our final (incremental) build went from over a minute to a few seconds for a 32 MB .msi output, and a full rebuild stayed well under a minute, even with the high compression level.
WiX Help File: How To: Optimize build speed. In other words: 1) Cabinet reuse and 2) multi-threaded cab creation are built-in mechanisms in WiX to speed up builds.
Hardware: The inevitable "throw hardware at it". New SSD and NVMe disks are so much faster than older IDE drives that you might want to try them as another way to improve build speed and installation speed. Obvious yes, but very important. It can really improve the speed of development. See this answer.
Challenges with NVMe drives?: 1) They run hot, 2) they usually have limited capacity (size), 3) they might be more vulnerable than older 2.5" drives when used in laptops (I am not sure - keep in mind that some NVMe drives are soldered solid to the motherboard on laptops), 4) data rescue can be a bit challenging if you don't have good quality external enclosures (form factor etc...), 5) NVMe drives are said to burn out over time, 6) They are still somewhat pricey - especially the larger capacity ones, and there are further challenges for sure - but overall: these drives are awesome.
Compression: You can try to compile your setup with a different compression level (for example none for debug builds). No compression makes builds faster. Here are illustrations for doing the opposite, setting higher compression (just use none instead of high for your purpose):
CompressionLevel: Msi two times larger than msm
MediaTemplate: How can I reduce the size of a 1GB MSI file using Orca?
A related answer on compression: What is the compression method used by MSI files?
Separate Setup: If you still go compressed, you could put prerequisites and merge modules in a separate setup to avoid compressing them for every build (or use release flags if you are in Installshield, or check the Preprocessor features in Wix).
External Source Files: I suppose you could use external source files if that's acceptable - then you don't have a lengthy compression operation taking place during the build, just a file copy (which keeps getting faster - especially with flash drives).
Shim: Another technique is to shim all the files you install to be 1 KB if what you are testing is the setup itself and its GUI and custom actions. It is then just a "shell" of a setup - which is a great way to test new custom actions to your setup. Many have written tools for this, but I don't have a link for you. There is always github.com to search.
Release Flags: Another way to save time is to use special release flags (Installshield only) to compile smaller versions of the setup you are working on at the moment (leaving out many features). WiX has similar possibilities via its preprocessor. More on WiX preprocessor practical use.
Debug Build: I usually use combinations of these techniques to make a debug build.
I normally use external source files when I experiment and add new features and keep rebuilding and installing the setup all the time.
Release flags to compile only part of the setup, cabinet reuse and release flags combined can save a lot of time depending on the size of your setup, the number of files and your hardware configuration.
Perhaps the most effective is a separate setup in my opinion (provided it is stable and not changing that often). Beware though: Wix to Install multiple Applications (the problems involved when it comes to splitting setups).
My take on it: go for a prerequisites-only separate setup. This is good also for Large Scale Deployment scenarios where corporate users want to use their own, standardized prerequisites and are annoyed with lots of embedded "junk" in a huge setup. A lot of package preparation time in large companies is spent taking out outdated runtimes and prerequisites. You can also deliver updates to these prerequisites without rebuilding your entire setup. Good de-coupling.
Links:
How can I speed up MSI package install and uninstall?
Simply put, don't harvest files. Please see my blog article: Dealing with very large number of files
The third downside is that your build will take A LOT longer to
perform since it's not only creating your package but that it's also
authoring and validating your component definitions.
I maintain an open source project on CodePlex called IsWiX. It contains project templates (scaffolding) and graphical designers to assist you in setting up and maintaining your WiX source. That said, it was designed around merge modules which slows the build down a bit as the .MSM has to be built and then merged into the .MSI. Pure fragments would be faster if you are really concerned about pure speed. That said I have many installers around 160mb and it doesn't take long at all.
And of course don't forget about having a fast build machine. CPU, RAM and SSD disk I/O all contribute to fast generation of MSIs. For my consulting, I use Microsoft Visual Studio Online (VSO). I have a Core i7-2600k Hyper-V server with 32GB of ram and a Samsung 850evo SSD. My build server (VM) runs a TFS proxy server for local SCC caching.
For fun, on the above machine, I took a 220 files from my system32 folder totaling 160MB. It took 30 seconds to build the MSM and 30 seconds to build the MSI for a total of 60 seconds. This is 'fast enough' for me. I would expect an MSI authored using only fragments to take 30 seconds.
In Windows 7(i.e. MSI 5.0), there is a property called MSIFASTINSTALL which will improve the performance of your installer. Or else, you can turn off the rollback option by setting property DISABLEROLLBACK. This property is available in earlier version of MSI 5.0 too.
Please share your knowledge to improve the install experience. Also, I cannot find the right way to improve the performance of Uninstall. We use huge set of files/folders (more than 70,000) and components like 35000. It hangs in the file costing process and do not know how to avoid this delay. Sometime it hangs for more than 2 or 3 hours to uninstall in XP or Vista machines
Edit:
I did some hack in my install by zipping the folders which has huge file sets and reduced the components size like Christopher said. It improved the performance drastically. yes ofcourse, I lost the MSI installer pattern by doing this concept and it is not recommended approach. However, it is trade off when we want to achieve this and our user really do not want to have file version details when we uninstall/upgrade the patches.
I had a similar situation, though the number of files was a bit less, about 25k. Most of those files were icons, which were never changed from one release to another. Only a major release (once per 2 years) might bring some changes to this area. A "quick & dirty" solution was zipping those icons and include this single file into the installation (not a component, just a file side by side with the MSI). During the installation this ZIP was extracted in the background thread, and RemoveFile table was used to delete icons on uninstall. It was faster than installing those 20k icons as separate components, even as components with many files. A good and correct solution was to convince the main application developers to put all those 20k icons into the 20 zip archives. Now these 20 zip files are installed as regular MSI components, and the application knows how to extract an icon on demand and cache it.
I would not recommend you to disable rollback. Though you'll save quite some installation time, you lose a standard guaranteed rollback option.
Uninstall takes more time than install because of rollback feature again. The way I understand it, when you uninstall, the MSI firstly creates a copy of every single file, then uninstalls every single file, and in case of success, drops every single backed up file. Hence, the uninstall time is about three times as much as the install time. I experienced the same problem when I took a default option to have 1 file per component. Though it is recommended, you should make a trade-off if you deal with an outstanding case.
Hope this clears up the things for you a bit.
The best option for improving the performance of your app is to reduce the number of files and components. While there may be a couple tweaks you can do to your MSI to improve the performance, the excessive number of files/components is the core issue and will be the gating factor on any performance improvements you make. Why do you need to install 70k files?
What's up people.
Something's been bothering me for a while now... and I was wondering if any of you might know of a workaround for this.
The C# solution im working on is a huge solution that contains about 20 projects and almost the same amount of unit test projects. Each projects contains hundreds of files. So opening and closing the solution takes a while... but once it's opened, everything is fine.
But, if I leave my computer up for the night (with my solution still opened in VS) and come back the next morning, everything I'll do in VS will be very slow for the next half hour or so.
I know why this happens... it's because Windows seems to remove idle processes from memory (RAM). And when I do something in VS, it takes the data from the pagefile and puts it back in the memory which slows everything single operations I do till the process' memory has been fully restored in RAM.
So my question is, is there a way to tell Windows that VS is a high priority process/application and to leave that process' memory in RAM?
Thanks in advance,
-Oli
I don't think this is possible. OTOH, you could put your computer in suspend-to-disk mode. That would pretty much freeze its state as it is when you leave (that is: VS in RAM) and restore it to the same when you start working. As an additional bonus, you would help to conserve energy and thus might save the earth.
You could alter your VS shortcut according to this article to boost the priority, but I don't know whether it would do what you describe for the process' memory.
Also solely for performance sake you could consider getting an SSD drive to replace your hard drive, if you haven't already. A friend of mine showed me his new laptop with an SSD onboard and it booted into Windows under a minute, and opened VS in less than 5 seconds.
Granted that was opening VS straight from the start menu, opening that huge a project hopefully would at least be significantly faster.
AFAIK, changing the process priority won't solve the problem, as the bottleneck seems to be I/O rather than CPU time. If the problem hurts your productivity, it would be well worth it to just buy a few more Gs of RAM (how much depends on your OS and budget). If you can get about 3-4GB of RAM, you can even eliminate the swap file (or close to eliminate it). This will prevent VS from sinking when idle.
Another option would be to create a tool that will walk VS's heap, forcing it into the main memory. This can be done by writing an add-in or by code injection. Have it run before you get to work, and you'll have VS up and about once you get to it. It will, however, require some work, and you might get more than you actually need in memory (some of VS's memory is in the swap file even when you work as usual, as with every other process).
We use SourceSafe 6.0d and have a DB that is about 1.6GB. We haven't had any problems yet, and there is no plan to change source control programs right now, but how big can the SourceSafe database be before it becomes an issue?
Thanks
I've had VSS problems start as low as 1.5-2.0 gigs.
The meta-answer is, don't use it. VSS is far inferior to a half-dozen alternatives that you have at your fingertips. Part of source control is supposed to be ensuring the integrity of your repository. If one of the fundamental assumptions of your source control tool is that you never know when it will start degrading data integrity, then you have a tool that invalidates its own purpose.
I have not seen a professional software house using VSS in almost a decade.
1 byte!
:-)
Sorry, dude you set me up.
Do you run the built-in ssarchive utility to make backups? If so, 2GB is the maximum size that can be restored. (http://social.msdn.microsoft.com/Forums/en-US/vssourcecontrol/thread/6e01e116-06fe-4621-abd9-ceb8e349f884/)
NOTE: the ssarchive program won't tell you this; it's just that if you try to restore a DB over 2GB, it will fail. Beware! All these guys who are telling you that they are running fine with larger DB are either using another archive program, or they haven't tested the restore feature.
I've actually run a vss db that was around 40 gig. I don't recommend it, but it is possible. Really the larger you let it go, the more you're playing with fire. I've heard instances where the db gets corrupted, and the items in source control were unrecoverable. I would definately back it up on a daily basis and start looking to change source control systems. Having been in the position of the guy who they call when it fails, I can tell you that it will really start to get stressful when you realize that it could just go down and never come back.
Considering the amount of problems SourceSafe can generate on its own, I would say the size has to be in the category "Present on disk" for it to develop problems.
I've administered a VSS DB over twice that size. As long as your are vigilant about running Analyze, you should be OK.
Sourcesafe recommends 3-5G with a "don't ever go over 13G".
In practice, however, ours is over 20G and seems to be running fine.
The larger you get, Analyze will find more and more problems including lost files, etc.
EDIT: Here is the official word: http://msdn.microsoft.com/en-us/library/bb509342(VS.80).aspx
I have found that Analyze/Fix starts getting annoyingly slow at around 2G on a reasonably powerful server. We run Analyze once per month on databases that are used by 20 or so developers. The utility finds occasional fixes to perform, but actual use has been basically problem free for years at my workplace.
The main thing according to Microsoft is make sure you never run out of disk space, whatever the size of the database.
http://msdn.microsoft.com/en-us/library/bb509342(VS.80).aspx
quote:
Do not allow Visual SourceSafe or the Analyze tool to run out of disk space while running. Running out of disk space in the middle of a complex operation can create serious database corruption
Our application takes significantly more time to launch after a reboot (cold start) than if it was already opened once (warm start).
Most (if not all) the difference seems to come from loading DLLs, when the DLLs' are in cached memory pages they load much faster. We tried using ClearMem to simulate rebooting (since its much less time consuming than actually rebooting) and got mixed results, on some machines it seemed to simulate a reboot very consistently and in some not.
To sum up my questions are:
Have you experienced differences in launch time between cold and warm starts?
How have you delt with such differences?
Do you know of a way to dependably simulate a reboot?
Edit:
Clarifications for comments:
The application is mostly native C++ with some .NET (the first .NET assembly that's loaded pays for the CLR).
We're looking to improve load time, obviously we did our share of profiling and improved the hotspots in our code.
Something I forgot to mention was that we got some improvement by re-basing all our binaries so the loader doesn't have to do it at load time.
As for simulating reboots, have you considered running your app from a virtual PC? Using virtualization you can conveniently replicate a set of conditions over and over again.
I would also consider some type of profiling app to spot the bit of code causing the time lag, and then making the judgement call about how much of that code is really necessary, or if it could be achieved in a different way.
It would be hard to truly simulate a reboot in software. When you reboot, all devices in your machine get their reset bit asserted, which should cause all memory system-wide to be lost.
In a modern machine you've got memory and caches everywhere: there's the VM subsystem which is storing pages of memory for the program, then you've got the OS caching the contents of files in memory, then you've got the on-disk buffer of sectors on the harddrive itself. You can probably get the OS caches to be reset, but the on-disk buffer on the drive? I don't know of a way.
How did you profile your code? Not all profiling methods are equal and some find hotspots better than others. Are you loading lots of files? If so, disk fragmentation and seek time might come into play.
Maybe even sticking basic timing information into the code, writing out to a log file and examining the files on cold/warm start will help identify where the app is spending time.
Without more information, I would lean towards filesystem/disk cache as the likely difference between the two environments. If that's the case, then you either need to spend less time loading files upfront, or find faster ways to load files.
Example: if you are loading lots of binary data files, speed up loading by combining them into a single file, then do a slerp of the whole file into memory in one read and parse their contents. Less disk seeks and time spend reading off of disk. Again, maybe that doesn't apply.
I don't know offhand of any tools to clear the disk/filesystem cache, but you could write a quick application to read a bunch of unrelated files off of disk to cause the filesystem/disk cache to be loaded with different info.
#Morten Christiansen said:
One way to make apps start cold-start faster (sort of) is used by e.g. Adobe reader, by loading some of the files on startup, thereby hiding the cold start from the users. This is only usable if the program is not supposed to start up immediately.
That makes the customer pay for initializing our app at every boot even when it isn't used, I really don't like that option (neither does Raymond).
One succesful way to speed up application startup is to switch DLLs to delay-load. This is a low-cost change (some fiddling with project settings) but can make startup significantly faster. Afterwards, run depends.exe in profiling mode to figure out which DLLs load during startup anyway, and revert the delay-load on them. Remember that you may also delay-load most Windows DLLs you need.
A very effective technique for improving application cold launch time is optimizing function link ordering.
The Visual Studio linker lets you pass in a file lists all the functions in the module being linked (or just some of them - it doesn't have to be all of them), and the linker will place those functions next to each other in memory.
When your application is starting up, there are typically calls to init functions throughout your application. Many of these calls will be to a page that isn't in memory yet, resulting in a page fault and a disk seek. That's where slow startup comes from.
Optimizing your application so all these functions are together can be a big win.
Check out Profile Guided Optimization in Visual Studio 2005 or later. One of the thing sthat PGO does for you is function link ordering.
It's a bit difficult to work into a build process, because with PGO you need to link, run your application, and then re-link with the output from the profile run. This means your build process needs to have a runtime environment and deal cleaning up after bad builds and all that, but the payoff is typically 10+ or more faster cold launch with no code changes.
There's some more info on PGO here:
http://msdn.microsoft.com/en-us/library/e7k32f4k.aspx
As an alternative to function order list, just group the code that will be called within the same sections:
#pragma code_seg(".startUp")
//...
#pragma code_seg
#pragma data_seg(".startUp")
//...
#pragma data_seg
It should be easy to maintain as your code changes, but has the same benefit as the function order list.
I am not sure whether function order list can specify global variables as well, but use this #pragma data_seg would simply work.
One way to make apps start cold-start faster (sort of) is used by e.g. Adobe reader, by loading some of the files on startup, thereby hiding the cold start from the users. This is only usable if the program is not supposed to start up immediately.
Another note, is that .NET 3.5SP1 supposedly has much improved cold-start speed, though how much, I cannot say.
It could be the NICs (LAN Cards) and that your app depends on certain other
services that require the network to come up. So profiling your application alone may not quite tell you this, but you should examine the dependencies for your application.
If your application is not very complicated, you can just copy all the executables to another directory, it should be similar to a reboot. (Cut and Paste seems not work, Windows is smart enough to know the files move to another folder is cached in the memory)