I install the VS2017 on Windows 7. After some time I receive the error:
MSI: C:\ProgramData\Microsoft\VisualStudio\Packages\Microsoft.VisualStudio.MinShell.Msi,version=15.6.27421.1\Microsoft.VisualStudio.MinShell.Msi.msi, Properties: REBOOT=ReallySuppress ARPSYSTEMCOMPONENT=1 MSIFASTINSTALL="7" VSEXTUI="1" VS7.3643236F_FC70_11D3_A536_0090278A1BB8="G:\Program Files (x86)\Microsoft Visual Studio\2017\Community"
Return code: 1632
Return code details: The Temp folder is on a drive that is full or is inaccessible. Free up space on the drive or verify that you have write permission on the Temp folder.
Log
G:\TEMP\dd_setup_20180318121545_006_Microsoft.VisualStudio.MinShell.Msi.log
I have checked the G: where the TEMP located. It has 200 GB free.
BUT one strange thing: this folder and all other folders are Read-Only. I uncheck it in the Properties, then close Properties dialog, open it again: it is Read-Only.
I can modify it, even MSI installer could: it created the log file there. But in the middle of installation the error occurs.
What is it and how I can solve this problem?
I run with log:
Machine policy value 'DisableUserInstalls' is 0
SRSetRestorePoint skipped for this transaction.
Note: 1: 1336 2: 3 3: C:\Windows\Installer\
MainEngineThread is returning 1632
No System Restore sequence number for this installation.
User policy value 'DisableRollback' is 0
Machine policy value 'DisableRollback' is 0
Incrementing counter to disable shutdown. Counter after increment: 0
Note: 1: 1402 2: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Installer\Rollback\Scripts 3: 2
Note: 1: 1402 2: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Installer\InProgress 3: 2
Decrementing counter to disable shutdown. If counter >= 0, shutdown will be denied. Counter after decrement: -1
Restoring environment variables
Decrementing counter to disable shutdown. If counter >= 0, shutdown will be denied. Counter after decrement: -1
MainEngineThread is returning 1632
Disc Space Reclaiming - Quick Wins?: Too much to read? The essential options (arguably).
Final Summary
This issue turned out to be a redirected TEMP and C:\Windows\Installer cache folder - with the latter being on an unavailable drive.
Please be careful redirecting system folders, in particular C:\Windows\Installer. It is a super-hidden system folder and side-effects are very common.
You must make sure that the relocated folder has the correct ACL permissions that the original folder had. This is crucially important for security reasons. For one thing the whole folder could be deleted by someone who do not understand what it is for - making all packages un-uninstallable and un-maintainable. There are also other security reasons.
Also: putting this folder on the network is not technically sound in my opinion - problems will result. A local drive is also problematic if drive letters change. Which brings me to the next point:
Lacking Space for your System SSD Drive?
If your real issue is lacking disk space on your system SSD drive, please consider some alternatives listed below. Proceed with care and at your own risk with every option. Most of them should be harmless.
Disc Space Visualizing: I have an ancient tool called SpaceMonger.exe which shows me a visual representation of whatever is taking up my disc space. Very useful. It seems this tool is no longer supported. Maybe check https://en.wikipedia.org/wiki/WinDirStat for a similar tool (untested by me - run it by virustotal.com).
DriverStore: And a word to the resident hacker in all computer guys: no, no - don't try to redirect %SystemRoot%\System32\DriverStore (!). "Seductive The Dark Side Is". "Run Forrest, Run!". "Careful With That Axe Eugene". Etc... You get the picture. Leaving out Monty Python allusions for now. Seriously: I do not know what low-level stuff could be involved in the boot process. One would have to ask Raymond Chen, but don't. He has important things to do. However: pnputil.exe, DriverStore Explorer - your own risk. Don't do it :-).
Overall Suggestions
UPDATE: For laptops I like to use a high capacity, low-profile USB flash drive and / or a high capacity SD-card permanently sitting in a port to
hold my downloads and installers, VS Help files, maybe even source code (riskier). An obvious, but somewhat "clunky" option.
One can combine this drive with the Library feature in Windows Explorer
to show the flash drive under whatever library you want (Downloads, Videos, Pictures, Source, etc...).
My preferred desktop disc cleanup options below would be: 7, 19, 2, 18, 1, 6, 11, 12 (in that order).
Preferred options for laptops: 7, 19, 2, 18, 6, 10 (reduce max cache sizes), 15, 17, 3 (in that order).
The real-world approach for me is a slightly different order: 2 (purge obsolete Windows Updates - this may also trim WinSxS - but I am not positive), 19 (uninstall unneccessary software - can be relatively quick), then I run SpaceMonger.exe to find space hogs and move them - this often involves zapping the Downloads folder (7) and purging, moving or clouding media files (Pictures, Videos, Music), then 6 for developer PCs (jogging Visual Studio and uninstall useless SDKs and help files), and 9 (eliminate hibernation - not great for laptops), 18 (enable compression - can take forever), and finally I might zap the recovery partitions (laptops) and create a new partition in its place to allow data files to be stored there (freeing up system partition space). This zapping is a high-risk operation - obviously. Very error-prone (especially if inexperienced users use the diskpart command-line tool or a Linux Live Boot tool - described below). And obviously verify that you have installation media AND a valid license key before wiping out recovery partitions - it has to be mentioned. Data files I move are usually: source code repository, downloads folder, outlook PST file, images and videos, etc... This procedure should reclaim many gigabytes of disc space. Don't do it for fun though - though risk should be acceptable for most of these options (barring the recovery partition zapping - it is relatively simple to do, but error prone).
Cleanup Options
Apply healthy skepticism to these options. They are not all terribly useful in many cases - just attempting to mention all kinds of tweaks. Potential easy, big wins without much configuration and fiddling could be 2, 6, 7, 9, 18. Options 2 and 18 are almost always time consuming, but very effective. Maybe hours for option 2 (especially on Windows 7 & 8 - do not abort when it is running) and even longer for option 18 on a large computer or a slow disk (but the operation can be cancelled).
Option 0, Cloud Storage is an implied overall option in this day and age. OneDrive Filer, GDisk, Dropbox, etc... Download data files on demand.
My Documents: It is generally much better to move user data folders to a network location or another, local drive (best) than to redirect system folders! Few system-entanglements.
I wouldn't move the desktop or other folders found here: HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\User Shell Folders, I would move "My Documents". Just right-click it in Windows Explorer, go to properties and there is a tab there with features to help you move it. Careful whilst doing this - a backup is in order first.
Pictures and Video might also be OK to move, but not the desktop or the other special folders - they may be involved in the boot or logon process (erroneous packages could cause that even for My Documents - nothing is without risk).
Streaming and media files from apps such as iTunes or similar can obviously totally hog a disc with limited capacity. I use SpaceMonger.exe to get an overview and then move the files somewhere else.
For computers with multiple users there will obviously be multiple "My Documents" folders to redirect.
Microsoft's Disk Cleanup Tool: Run cleanmgr.exe, select Clean up system files as described here: https://serverfault.com/q/573208/20599 (top).
UPDATE Oct.2018: "Downloads" folder is now a cleanup option! DO NOT ENABLE! It deletes the whole downloads folder without question. This issue appears corrected by now Oct.2021.
You can now zap the uninstalls for applied Windows Updates - this can give you back several gigabytes on your system drive. In the picture below I can zap 5.36 GB. For Windows 7 I have seen dozens of gigabytes being purgeable.
This tool might also slim down and shrink the WinSxS directory (the Win32 side-by-side assembly folder). I am not 100% positive.
Obviously you can remove unnecessary packages in Add / Remove Programs and remove system restore point (use the second tab in the image below to access these features):
Third Party Cleanup Applications: Third party tools such as CCleaner may be able to clean out even more space by wiping out cache files and temporary files for all kinds of applications and tools. This particular tool suffered a malware attack recently. Use at your own risk.
Personal opinion / suggestion: use only for test boxes or non-critical machines. The cleanup is quite awesome, but it also involves some risks (lost login passwords, lost system logs, etc...). Self-evident, but it should probably be mentioned.
My 2 cents: not a corporate solution, but may be fine for advanced home users who like to experiment and to keep their machines tuned.
Administrative Installations: For large MSI files, performing an administrative install will prevent the caching of the whole MSI file in C:\Windows\Installer. You must install from a proper network share so files are available for repair operations.
An administrative installation essentially extracts embedded CAB files from the MSI and allows the creation of a network installation point where all computers can pull files from instead of caching all files locally.
The generic method for running and administrative installation is: msiexec /a File.msi. More details in links below.
Extract MSI from EXE
What is the purpose of administrative installation initiated using msiexec /a?
How can I eliminate the huge, cached MSI files in C:\Windows\Installer?
There is a whole lot of installer caching going on - it is a little out of hand if you ask me.
Mounted Drives: Some guys dabble with mounting external drives as folders on their system drive. In other words another drive shows up as a regular folder on your system drive and functions as such (sample).
This I have no experience with, and I have doubts about its reliability over time. For all I know it might actually be better than several other options if you do it right (and never take out the physical drive).
I would do data file folders only (not settings folders, or core OS folders such as the desktop). Maybe for source control folders. If the link breaks, the data should still be safe and the system can still boot (and the link re-established).
UPDATE: Windows Explorer's "Include in library" is an alternative? (do have a peek) I like to create a "Source Code Library" with included folders from here and there.
Visual Studio: And the obvious cleanup options for Visual Studio (for completeness):
If you have downloaded MSDN help locally (Help => Add and Remove Help Content, remove items as appropriate and rely on online help instead or change the Local store path towards the bottom to use another drive for content).
Or you have several versions of the SDKs you do not need or you have Visual Studio features you do not need, get rid of them (In Visual Studio: Tools => Get Tools and Features... - get rid of unnecessary features - I often use the Individual Components view).
Downloads Folder: I am sure I have forgotten many viable options to get some more workspace without wrecking your box. One would be to clean out your Downloads folder and move all installers to a network location - this might be the biggest save of all for some people.
This also works great for laptops - it is just about the first thing I would do for a laptop with little disc space. If you will not have access to your network share of installers - for example whilst traveling - then just use a thumb drive or external hard drive to hold your installers and ISO files.
For computers with multiple users there will obviously be multiple download folders potentially full of stuff. Use a disk space visualizer to see (see link on top of list).
Page File: Some people move the system page file (pagefile.sys) from the system drive to another drive. Back in the day this caused me an unbootable system, but perhaps things are better now. Not the first thing I would do though - this is very core OS-stuff.
Obviously impossible for a laptop with only one drive (unless you erase the recovery partition and create a real, visible partition in its place).
I find this option risky, maybe I should have put it in the "dis-honerable mentions" part below.
Be careful. Maybe the "last-known good"-feature or system restore can help you if you get problems?
Hibernation File: the hibernation file on Windows systems will live on the system drive, and I am not aware of any way to move it anywhere else for very fundamental technical reasons. However, you can disable hibernation to get rid of the whole file. This will free up a few gigabytes on a modern computer.
You obviously lose the ability to put your machine into hibernation (memory dumped to disk), but sleep mode (low-power use mode / standby) should still be available.
Hibernation mode may be more desirable to keep on for laptops (if battery runs out whilst traveling the laptop can not auto-hibernate and you could lose data).
Application Temp & Cache Folders: The above mentioned CCleaner can wipe out a lot of temporary files for various applications (though I don't really recommend this for use - I use cleanmgr.exe instead - and CCleaner for test boxes).
Web Browsers (Firefox, Opera, Vivaldi, Chrome, IE, Edge, Safari, etc...) can also spam the disk with a lot of cache files and downloaded junk. It is possible to redirect all these folders, though I prefer to reduce them to a certain acceptable maximum size.
Plenty of other applications, of all kinds, leave trash on the system over time. Some of which can be cleaned with CCleaner mentioned above (or another such tool). Again not a tool recommendation. Use the cleanup features inside the application itself if available.
For computers with multiple users there will obviously be multiple cache folders folders to restrict and clean.
Special Data-Heavy Applications' Storage Folders: Some applications can potentially store enormous data files on your system drive (and outside "My Documents") that can be moved to other drives.
The biggest suspect is probably Outlook (in older versions at least) - or other email software (Thunderbird, Lotus Notes, etc...). For Outlook there is a single *.PST file storing all email and attachments, or a similar sync file if connected to Exchange. This file can be moved to different drive with relative ease. Some even resort to using the Web-interface only for their email and eliminate the local PST file (good for laptops).
Without going overboard, MS-SQL databases could be another type of massive data file that could be moved to a different drive with relative ease.
And this list could be made very big, but diminishing returns to add any more (web server folders, virtual machine images, media / video files (mentioned above), virtualized applications maybe, etc...).
For computers with multiple users there will obviously be multiple storage locations to redirect.
Source Control Working Folder & Repository: for a developer this is 100% self-evident - and almost embarrassing to list, but I just want to have it mentioned. It is also related to the previous point, but I add it as its own bullet point. You move both your working folder and your source code repository (if different, and if local) to a different drive than the system drive. For example GIT, Mercurial, Perforce, StarTeam, etc...
Build Process Junk: Beyond moving source control folders to other drives, it is also possible that certain processes generate huge log files that spam the system in unexpected locations at times. I hear MSBuild tends to enthusiastically create log files sprinkled across the system and I am not sure if normal Microsoft cleanup tools detect them (for example cleanmgr.exe mentioned above). And your source code could have lots of object files you can zap.
Visual Studio Code: silly option, but for ad-hoc developer laptops or traveling tech-workers, one could potentially rely on the smaller and multi-platform Visual Studio Code instead of Visual Studio to do small development testing / work. Significantly smaller install. Personal note: a bit odd the whole tool :-). Also browser version now?
Visual Studio Code (cross-platform).
What are the differences between Visual Studio Code and Visual Studio?
https://code.visualstudio.com/docs/supporting/faq
Download: https://www.visualstudio.com/
Windows Store Apps & Per User Installations: if there are multiple users on the box, several Store apps could be installed multiple times, once per user. Some cleanup could be done here if need be. I suppose some games could be quite big. And in the day and age of side-by-side installation features, we are now to deploy everything per-user? Odd.
Tweak Each Package Installation: almost every package you install can be modified slightly during installation to add less files to the system partition.
Redirect Application Installation Folder: this is an option I personally dislike, but it is used a lot. For every installation you redirect the installation folder to a different drive and folder hierarchy than the regular ProgramFilesFolder. This is done on a per-package basis, and not all packages support this. Typically you go to a "Custom" installation dialog where you perform "feature selection" (what setup features to install).
Leave Out Optional Features: most packages you install will have optional components that you can leave out or even run-from-source in the case of some MSI packages. Certain developer tools can often be tweaked quite a bit without too many side-effects. Large games are often installed to a regular non-SSD hard drive which is not the system drive.
Uninstall Windows Components: a few components can be added / removed from Windows. Click Turn Windows Features On or Off from the old-style Add / Remove Control Panel Applet. You can turn off / remove certain .NET versions, IE, IIS, Windows Media Player, Message Queue Server, Print to PDF, PowerShell and various other components. Maybe not that much to gain from this (some security benefits perhaps by removing some components - for example support for SMB 1.0 / CIFS file sharing or IIS).
Enable Compression For System Drive: you can enable compression on the whole system drive - with some performance penalties - provided the file system is NTFS. Simply Right-click the system drive => Properties => Compress drive to save disc space. This can take quite some time (old HD, SSDs are faster). You can also compress individual folders. I like to enable the "Show compressed or encrypted NTFS files in color" option in Windows Explorer. File Menu => Options => Show => Show compressed or encrypted NTFS files in color.
Uninstall Unnecessary Software: the forgotten obvious option mentioned in item 2 above, you should obviously uninstall any software that is not needed anymore. Common disk hogs: games, weird SDKs and development tools installed for testing, expired trial versions for various software, etc... To uninstall: Windows key + R, type appwiz.cpl and hit Enter.
User Data Cleanup: for certain uninstalled applications a lot of junk could be left in the %UserProfile% and in the %AllUsersProfile%. Cleanup is as usual risky, use caution, but there can be lots of junk here - sometimes gigabytes.
Great care must be taken during such cleanup. Zip up the folder first. "Big wins only" - why nitpick with tiny text files? Diminishing returns for real if you get bogged down in these folders. Use disc-space visualization tools to see the hogs.
%AllUsersProfile% - shared data
%UserProfile% and %UserProfile%\AppData - user specific data, remember to clean for all users (if multiple).
Stray Package Caches: as mentioned above a lot of caching goes on for MSI packages (and other installer packages). It is likely that a lot of these packages can be left behind after uninstall (this was the case with Installshield cached setups back in the day at least).
The most commonly known caching locations are described here: Cache locations for (MSI) packages. Clean at your own risk, obviously - I repeat it, and I mean it. Some gigabytes are commonly stored here.
Paths inline (just a selection, there can be many others):
WiX: %ProgramData%\Package Cache
Installshield: %SystemRoot%\Downloaded Installations (older IS setups) and %LocalAppData%\Downloaded Installations (newer IS setups)
Advanced Installer: [AppDataFolder][|Manufacturer]\[|ProductName] [|ProductVersion]\install
Visual Studio: %AllUsersProfile%\Microsoft\VisualStudio\Packages. See important tip in comment below (disable cache).
Package Distribution Cache Folders: SCCM and other package distribution systems have cache folders that get really big. For example ccmcache. These folders can usually be cleaned or re-configured to take less space.
There are no doubt numerous other little tricks, but please don't redirect system folders!
Alternative Approaches
(Dis)-Honorable Mentions: The below are not recommendations, but some alternative approaches. They are higher risk than the options above (which should be good enough), and best if you are setting up a new laptop fresh or reinstalling it, and want to get rid of pesky vendor recovery-partitions that you can do without.
Let's state the obvious with conviction: A lot of data is lost every year using these tools. So coffee or caffeine first. Glasses on. Look around. Adjust any pony tails and beards (ladies too). Speak to yourself in the third person. Assume a demonstrably insane posture and shout out "I do!" to really commit to the imminent disaster! Good luck! Fire in the hole! "Fire for effect". SNAFU. FUBAR. OK, enough already... I have had bad experiences - but no huge disasters (knock on wood) - with all these tools. Enough said - be careful, your data is important. Wife's baby pictures, your uncommitted code, etc...
diskmgmt.msc or diskpart.exe (Windows): open partition manager (diskmgmt.msc) and wipe out any recovery partitions or hidden partitions that you can live without and then expand your system disk to fill the whole physical disk or create a new visible partition.
Factory reset no longer possible (could be outdated anyway). You need installation media to reinstall (downloadable?).
Careful what you wipe out! Unrecoverable. Partitions are often protected and untouchable. They are also unmovable and un-expandable in many cases.
Maybe create a new, visible partition replacing the recovery partition and move data files and your downloads folder there to make more room on your system partition?
If the partitions are protected, you can use diskpart to delete them instead, or see next bullet point for gparted. Very easy to mess things up using diskpart though (command line).
gparted (Linux): you may be prevented from deleting a recovery partition from diskmgmt.msc (protected partitions). If you are adamant and insist, you can boot into a Linux Live Disc / System (booted from removable media) and delete using gparted for example.
I have done this to get rid of obsolete and useless recovery partitions and / or malware, and it worked just fine. But frankly I trust this gparted app as far as I can toss it. No offence to gparted, but playing well with Windows is challenging. Backup is crucial and mandatory for such risky endeavors - obviously.
Though risky (a Linux tool is updating the partition tables where your Windows partitions are declared) this may work for laptops where there is nowhere to redirect data folders since there is only one physical disk and you want the full disk for your system partition.
I think gparted even allows you to try to resize existing partitions at this point. I have never tried it. Good luck if you try. "Fire in the hole!".
Cloning: some use imaging tools or disk cloning features (hardware) to clone the old disk onto a bigger one. Backups essential obviously. Far from my comfort zone - just mentioning it. Not really relevant for this list (which was supposed to be about simple and effective measures to gain more disc space).
I believe there are features for this in gparted as well. Never tested.
Various hardware solutions. I gave them up years ago.
Why I am skeptical? Malware. Disk errors. Encryption. NTFS complexity? AD-problems (old & new drive in use post-clone)? Etc...
Several hard drive vendors seem to deliver proprietary solutions for this - these may be better tested than generic approaches?
File System Allocation Size: the file system used and its allocation size affects available space. Never bothered to look much at this, but a lot of space can be wasted by allocation size issues: Would SSD drives benefit from a non-default allocation unit size?
Allocation size cannot be easily / safely changed for a disk in use. There may be tools that can do it, but the benefits are
uncertain.
Modern Windows versions require NTFS as system partition file system. Other file systems such as FAT32 or exFAT have lower overhead (especially for smaller partitions - there will be more space available), and they are potentially faster but have more limitations. For FAT32 the biggest limitation is probably the 4GB max file size - not viable today.
The rest of this answer (below) was written during debugging - I will leave it in. It contains generic and general-purpose debugging options.
VC+ Runtimes
As seen in the link towards the bottom, other people have seen the same deployment error. Before getting into too much debugging, let's try the simplest approach possible. Please try to install the VC++ runtimes for 2017 (and 2015 perhaps) from here:
The latest supported Visual C++ downloads.
Potential General Fixes
This seems to be the better discussion online for this problem. I would first try the suggestion to run this tool: Microsoft Install and Uninstall Troubleshooter.
You can try this list of fixes as well. Crucially I would also try a reboot before trying again to release any potential locked files. Just to wipe the slate clean. The system's event log might have further information on the error seen (sometimes even beyond what is in an msiexec.exe log).
ACLs
What is the ACL (Access Control List) for your TEMP folder on that G: drive?
UPDATE: Also make sure the hidden folder C:\Windows\Installer exists and have the correct permission settings. You need to show protected operating system files in Windows Explorer to see this folder.
Verbose Logging
Try to create a proper, verbose log for the MSI install in question (much more informative than the log you refer to). This gives you something to start with to figure out what is happening. You can find some information on how to do logging here.
I would enable logging for all MSI installations for debugging purposes. See installsite.org on logging (section "Globally for all setups on a machine") for how to do this.
I prefer this default logging switched on for dev and test boxes. Typically you suddenly see an MSI error and you wish you had a log - now you can, always ready in %tmp%.
Quick Testing
In your case, I would go to C:\ProgramData\Microsoft\VisualStudio\Packages\Microsoft.VisualStudio.MinShell.Msi,version=15.6.27421.1\ to see if the MSI package is present on disk, and then I would launch it with logging enabled:
msiexec.exe /I "Microsoft.VisualStudio.MinShell.Msi.msi" /QN /L*V "C:\msilog.log"
Alternatively I would just double click the MSI file and see if I get a better, interactive error message. You will most likely need the verbose log to get any info.
See link in comment below (concrete error).
Just check c:\windows\temp and c:\windows\installer
do they exist and are they writable?
In my case i deleted c:\windows\installer previously and forgot about it, so i must recreate it.
The same error happens if UAC is disabled. Visual Studio installer can't write anything to TEMP if User Account Control is off. Solution - enable UAC.
It was VS 2019 and Windows Server 2012 R2 in my case.
We are upgrading our Progress application on 9.1D to 11.3. Is there any sample document which we should look for our migration.
Currently we have built a new server where we are installing OpenEdge Enterprise RDBMS 11.3.
Can we backup the current database and dump it to new version.
Any suggestions/documents ?
Generally Progress is very "kind" when upgrading but you have to take in mind that moving from 9.1d to 11.3 (11.4 is soon out by the way) is moving from 2002 to 2013. A lot has changed since then.
If you have program logic that relies on disc layout, os utilities (for example using UNIX, DOS or OS-COMMAND) they might be changed as well. So an upgrade might break even if the files compile without errors. You need to test everything!
You cannot directly backup and restore from 9.1D to 11.3, you need to dump & load.
What you need to do:
Back everything up! Don't miss this and make sure you save a copy of the backup. Back up database, scripts, program files (.p, .i, .r, .cls, etc). Everything! This is vital! Make sure you always have an untouched version left of the backup so you can restart if things go bad. Progress has built in utilities for backing up the database. OS utilities can also be used. Be aware that OS utilities can not be used to create online backups. The backed up database will most likely be corrupt. Shut down the database before backing up when using OS utilities.
Dump you current database. Data as well as schema. Don't forget to check for sequences etc.
Rebuild a new database on your new server with schema from old db.
If possible - move to Type 2 Storage Areas when doing an upgrade like this. It will increase perfomance. Check documentation and knowledgebase around required settings for this.
Load dumped data
Copy program files from old server to new
Recompile
Create startup scripts etc for starting databases as well as clients. Old parameters might not fit your new server, you most likely have more memory, faster CPU, larger discs etc.
All steps have several substeps. I suggest you dive into the documentation found at community.progress.com. You can also search the KnowledgeBase (knowledgebase.progress.com)
Also if you run into problems you can ask more specific questions here (but tag accordingly for example with openedge).
11.3 Documentation
9.1D Documentation
KnowledgeBase
I want to put some sort of "hook" into windows (only has to work on Windows Server 2008 R2 and above) which when I ask for a file on disk and it's not there it then requests it from a web server and caches it locally.
The files are immutable and have unique file names.
The application which is trying to open these files is written in C and just opens a file using the operating system in the normal way. Say it calls OpenFile asking for c:\scripts\1234.12.script, and that is there then it will just open it normally. If then it asks for c:\scripts\1234.13.script and it isn't then my hook in the operating system will then go and ask my web service for the file, download it and then return that file as it it were there all the time.
I'd prefer to write this as a usermode process (I've never written a windows driver), it should only fire when files are not found in a specific folder, and I'd prefer if possible to write it in a managed language (C# would be perfect). The files are small (< 50kB) and the web service is fast and the internet connection blinding so I'm not expecting it to take more than a second to download the file.
My question is - where do I start looking for information about this kind of thing? And if anyone has done anything similar - do you know what options I have (eg can it be done in C#?)?
You would need to create a kernel-mode filesystem filter driver which would intercept requests for opening such files and would "fake" those files. I should say that this is a very complicated task even for driver development. Our CallbackFilter product would be able to solve your problem however mechanism for "faking" files is not yet ready (we plan this feature for CallbackFilter 3). Until then I don't know any user-mode solutions (frankly speaking, no kernel-mode solutions as well) that would solve your problem.
If you can change the folder the application is accessing, then you can create a virtual file system and map it to the drive letter or a folder on NTFS drive. From the virtual file system you can direct most requests to/from real disk and if the file doesn't exist, you can download the file and cache it. Our other product, Callback File System, lets you do what I described in user-mode. If you have a one-time task you need to accomplish, and don't have a budget for it, please contact us anyway and maybe we can find some solution. There also exists an open-source solution with similar (but not so comprehensive) functionality named Dokan, yet I will refrain from commenting on its quality.
You can also try Dokan , it open source and you can check its discussion group for question and guides.
We're building a Windows-based application that traverses a directory structure recursively, looking for files that meet certain criteria and then doing some processing on them. In order to decide whether or not to process a particular file, we have to open that file and read some of its contents.
This approach seems great in principle, but some customers testing an early version of the application have reported that it's changing the last-accessed time of large numbers of their files (not surprisingly, as it is in fact accessing the files). This is a problem for these customers because they have archive policies based on the last-accessed times of files (e.g. they archive files that have not been accessed in the past 12 months). Because our application is scheduled to run more frequently than the archive "window", we're effectively preventing any of these files from ever being archived.
We tried adding some code to save each file's last-accessed time before reading it, then write it back afterwards (hideous, I know) but that caused problems for another customer who was doing incremental backups based on a file system transaction log. Our explicit setting of the last-accessed time on files was causing those files to be included in every incremental backup, even though they hadn't actually changed.
So here's the question: is there any way whatsoever in a Windows environment that we can read a file without the last-accessed time being updated?
Thanks in advance!
EDIT: Despite the "ntfs" tag, we actually can't rely on the filesystem being NTFS. Many of our customers run our application over a network, so it could be just about anything on the other end.
The documentation indicates you can do this, though I've never tried it myself.
To preserve the existing last access time for a file even after accessing a file, call SetFileTime immediately after opening the file handle with this parameter's FILETIME structure members initialized to 0xFFFFFFFF.
From Vista onwards NTFS does not update the last access time by default. To enable this see http://technet.microsoft.com/en-us/library/cc959914.aspx
Starting NTFS transaction and rolling back is very bad, and the performance will be terrible.
You can also do
FSUTIL behavior set disablelastaccess 0
I don't know what your client minimum requirements are, but have you tried NTFS Transactions? On the desktop the first OS to support it was Vista and on the server it was Windows Server 2008. But, it may be worth a look at.
Start an NTFS transaction, read your file, rollback the transaction. Simple! :-). I actually don't know if it will rollback the Last Access Date though. You will have to test it for yourself.
Here is a link to a MSDN Magazine article on NTFS transactions which includes other links. http://msdn.microsoft.com/en-us/magazine/cc163388.aspx
Hope it helps.
I have written a backup tool that is able to backup files and images of volumes for Windows. To detect which files have changed I use the Windows Change Journal. I already use the shadow copy functionality to do a consistent copy of both the files and the volume images.
To detect which blocks have changed I use hashes at the moment. This means the whole volume has to be read once (because to see which block has changed hashes of all blocks have to be calculated).
The backup integrated into Windows 7 is able to create incremental volume images without checking all blocks. I wasn't able to find an API for a kind of block level change journal.
Does anybody know how to access this information?
(I'm willing to dive deep into NTFS internals - even reading and parsing special files)
I don't think block level change info is available anywhere. Most probably what the Windows 7 integrated backup does is it installs a File System Filter Driver like some backup products does and anti-virus software. A filter driver can intercept all file system calls and in this way know which blocks changed. If you do this you can basically build your own change journal that works block level but only for the files that you are interested in.
I would really like to know a better answer myself here.
When you say Windows Change Journal I take it you are referring to the NTFS USN? It looks very much like the Windows 7 backup uses a combination of VSC and NTFS USN to detect changes and create incremental images much like you are already doing.