I am investigating a strange problem with my application, where the behaviour is different on 2 versions of Windows:
Windows XP (32-bit)
Windows Server 2008 (64-bit)
My findings are as follows.
Windows XP (32-bit)
When running my test scenario, the XML parser fails at a certain point during the parsing of a very large configuration file (see this question for more information).
At the time of failure, the process size is approximately 2.3GB. Note that a registry key has been set to allow the process to exceed the default maximum process size of 2GB (on 32-bit operating systems).
The system of the failure is a call to IXMLDOMDocument::load() failing, as described in the question linked above.
Windows Server 2008 (64-bit)
I run exactly the same test scenario in Windows Server 2008 -- the only variable is the operating system. When I look at my process under Task Manager, it has a * 32 next to it, which I am assuming means it is running in 32-bit compatibility mode.
What I am noticing is that at the point where the XML parsing fails on Windows XP, the process size on Windows Server 2008 is only about 1GB (IOW, approximately half the process size as on Windows XP).
The XML parsing does not fail on Windows Server 2008, it all works as it should.
My questions are:
Why would a 32-bit application (running in 32-bit mode) consume half the amount of memory on a 64-bit operating system? Is it really using half the memory, it is usual virtual memory differently, or is it something else?
Acknowledging that my application (seems) to be using half the amount of memory on Windows Server 2008, does anyone have any ideas as to why the XML parsing would be failing on Windows XP? Every time I run the test case, the error accessed via IXMLDOMParseError (see this answer) is different. Because this appears to be non-deterministic, it suggests to me that I am running into a memory usage problem rather than dealing with malformed XML.
You didn't say how you observed the process. I'll assume you used Taskmgr.exe. Beware that it's default view gives very misleading values in the Memory column. It shows Working set size, the amount of RAM that's being used by the process. That has nothing to do with the source of your problem, running out of virtual memory space. There is not much reason to assume that Windows 2008 would show the same value as XP, it has a significantly different memory manager.
You can see the virtual memory size as well, use View + Columns.
The reason your program doesn't bomb on a 64-bit operating system is because 32-bit processes have close to 4 gigabytes of addressable virtual memory. On a 32-bit operating system, it needs to share the address space with the operating system and gets only 2 gigabytes. More if you use the /3GB boot option.
Use the SAX parser to avoid consuming so much memory.
Not only are there differences in available memory between 32 bit and 64 bit (as discussed in previous answers), but its the availability of contiguous memory that may be killing your app on 32 bit.
On 32 bit machine your app's DLLs will be littering the memory landscape in the first 2GB of memory (app at 0x00400000, OS DLLs up at 0x7xxx0000, other DLLs elsewhere). Most likely the largest contiguous block you have available is about 1.1GB.
On a 64 bit machine (which gives you the 4GB address space with /LARGEADDRESSAWARE) you'll have a least one block in that 4GB space that is 2GB or more in size.
So there is your difference. If your XML parser is relying on a large blob of memory rather than many small blobs it may be that your XML parser is running out of contiguous usable space on 32 bit but is not running out of contiguous usable space on 64 bit.
If you want to visualize this on the 32 bit OS, grab a copy of VMValidator (free) and look at the Virtual view for a visualization of your memory and the Pages and Paragraphs views to see the data for each memory page/paragraph.
Related
In my computer its running 32bit OS on the 64bit processor. So all the application installed are 32bit and my computer having a 2GB RAM. If I install a 64bit OS while keeping the same 32bit applications, will this improve the performance?? Or is there any disadvantage?
Talking about 32-bit and 64-bit software means that such software was developed in order to take advantage of CPUs with 32-bit or 64-bit registers size.
Registers are tiny portions of memory (NOT the RAM) used directly by a processing unit to temporarily save operations results (kind of variables), e.g. when summing two numbers the ALU reads from two registers, performs the operation and writes the answer in another register. The bigger those registers, the bigger values it can handle. A 32-bit processor has registers than can handle values up to 32 bits of size, as well as 64-bit processors can deal with numbers of a double size than 32-bit-based processors, which
results in 2^32 more values.
A 32-bit OS is an OS than works performing CPU operations on a machine with 32-bit CPU registers; a 64-bit OS works fine on 64-bit processors, but not on 32-bit ones because their registers are too small for such OS.
As a program runs on an OS, 32-bit apps will run (well, most times they do) on a 64-bit OS albeit they will use only half of any CPU register they need, simply because the program was designed not to deal with values bigger than 2^32 bit: this means that a 32-bit app running on a 64-bit OS will run without getting any particular performance improvement. As well, a 64-bit app will never run on a 32-bit OS because it needs more resources than those actually available.
Hope I was clear and correct enough, I studied processor architectures years ago...
This applies for a 64-bit app on x86/x86_64:
The 64-bit system will consume more RAM and most importantly cache because the larger pointers. This may be a performance bottleneck in case there are lots of pointers.
The 64-bit system will give you twice as many registers, twice as large. This makes it possible to store more variables in registers, which will be a performance gain.
So the choice may depend on which application you decide to benchmark.
The point of 64 bit is that apps can use more than 2gb of virtual memory. If you have more than 4gb of ram and your apps need to use more than 2gb of memory (sql server!) then potentially there will be a performance memory (because it can store more pages in memory rather than reading them from disk)
If this isn't the case and you have the same amount of memory then you won't get a performance increase and possibly a slight decrease as (assuming you are on windows here) to handle 32 bit apps working on a 64 bit o/s there is a complex system of redirections that happen (app asks for system32 and is sent to syswow64) - some of these are simple but some like the com registration lookups can cause many reg opens under the hood (do a regopen for a clsid that doesn't exists and see how many kernel32 makes on your behalf using procmon!).
So unless you have a very specific use case, which it sounds like you don't, it won't be faster
I hope someone with a bit of knowledge can clear this up. There's many discussions about the reasons to run a 64-bit OS (e.g. Windows 7 x64), but many people seem to think that their old x86 apps will be able to take advantage of any RAM greater than 3.5GB.
As I understand it, though, x86 apps cannot address memory that high... unless they've been specifically programmed to (which very few will have).
Can someone knowledgeable clear this up for me, once and for all? Can 32-bit apps take advantage of a system running 8GB of RAM?
E.g. If a user decided (for whatever reason) to run several x86 apps at once, filling the RAM as much as possible, would the extra addressable memory available in Windows 7 x64 be used?
Thanks!
On a 64 bit system, 32 bit applications are able to use the full 4GB virtual address space, minus about 64K. A default 32 bit windows system will only allow a 32 bit process to use 2 GB of virtual address space. By specially configuring the OS it's possible to push that limit up to 3 GB, but it's still not as good as what you would get on a 64 bit version of windows.
If you have 8GB of ram, that 8 GB can be divided up between multiple 32 bit processes, and the entire 8 GB will be utilized if necessary. However, no single 32 bit process will be allocated more than 4 GB of memory.
Although i don't have sources to cite, but from my knowledge: 32bit app will not be able to address more than 4GB of memory itself, unless it uses some tricks(that is very unlikely), but if you have some 32bit apps running at the same time, they can all have 4GB each, and thus two 32bit apps should be able to use all 8GB of memory. Though I'm not 100% sure.
Yes. x86 apps cannot use more than 2GB of memory at once without special tricks, but they can use any memory available.
Adding to the other (correct) answers:
Instead of the term "application" the word "process" should be used. Applications often consist of multiple processes whereas the limits discussed here apply to single processes.
Thus applications benefit from x64 that either are linked with the LARGEADDRESSAWARE flag (they can use 4 GB instead of 2 GB) or that share the load between multiple processes.
32-bit processes can work with more than 4 GB RAM even on 32-bit systems by using AWE. But a 32-bit process can only ever use 2 GB at once (4 GB with LARGEADDRESSAWARE on 64 bit respectively). AWE is primarily used by databases where it is essential for performance that the entire database fit into RAM. It works by providing a 2 GB window into a larger chunk of memory.
Here are some articles for further reading:
Windows x64 – All the Same Yet Very Different, Part 1: Virtual Memory
Windows x64 – All the Same Yet Very Different, Part 2: Kernel Memory, /3GB, PTEs, (Non-) Paged Pool
x64? My Terminal Servers Run Just Fine With 32 Bits and 8/12/16 GB RAM!
E.g. If a user decided (for whatever
reason) to run several x86 apps at
once, filling the RAM as much as
possible, would the extra addressable
memory available in Windows 7 x64 be
used?
The answer is yes. That's one of the benefits a virtual address space gives us--the ability for each process to appear (to the process) as though it's executing in a linear address space that starts at 0 and goes up from there.
As far as each of the 32-bit applications is concerned, it has its own address space from 0 to 2 gigabytes (without special tricks). The operating system handles the virtual-to-physical address translation.
Imagine the same hardware running Windows XP 32bit, or Windows XP 64bit..
(being that it's a 64bit processor currently running XP 32bit)
2gigs of ram...
Will the performance of Visual Studio benefit from going to the 64bit OS?
The hardware and ram is currently out of my control... If I could throw more hardware or ram at it I would.
For 2 GB of RAM, most likely not. The biggest advantage of 64-bit is the additional address space. With 2GB of physical RAM, 32-bit OS's can map all of physical memory.
Above 3GB 64-bit OS does give some benefit, since some of your physical memory may be unusable on 32-bit OS's due to devices mapping physical addresses for their own use.
And obviously if you want to use above 4 GB of memory you should absolutely go for a 64-bit OS.
Yes, if you have sufficient physical memory, you will gain some benefits from running Visual Studio in 64-bit Windows as described in Visual Studio: Why is there no 64 bit version? (yet):
Doesn’t being a 64 bit application save you all kinds of page faults and
so forth?
A 64 bit address space for the process
isn’t going to help you with page
faults except in maybe indirect ways,
and it will definitely hurt you in
direct ways because your data is
bigger. In contrast a 64 bit
operating system could help you a lot!
If you’re running as a 32 bit app on a
64 bit OS then you get all of the 4G
address space and all of that could be
backed by physical memory (if you have
the RAM) even without you using 64 bit
pointers yourself. You’ll see
potentially huge improvements related
to the size of the disk cache (not in
your address space) and the fact that
your working set won’t need to be
eroded in favor of other processes as
much. Transient components and data
(like C++ compilers and their big .pch
files) stay cached in physical
memory, but not in your address space.
32 bit processes accrue all these benefits just as surely as 64 bit
ones. (my emphasis added)
not unless you add more RAM.
In fact 64 bits might even slow you down because 64-bits Windows sometimes have to load both 32-bits and 64-bits versions of a same DLL in RAM.
Add more RAM is the way to go because if you are using Windows 7 or Vista they aggressively caches disk files in RAM.
On Windows, under normal circumstances a 32 bit process can only access 2GB of RAM (or 3GB with a special switch in the boot.ini file). When running a 32 bit process on a 64 bit operating system, how much memory is available? Are there any special switches or settings that can change this?
2 GB by default. If the application is large address space aware (linked with /LARGEADDRESSAWARE), it gets 4 GB (not 3 GB, see http://msdn.microsoft.com/en-us/library/aa366778.aspx)
They're still limited to 2 GB since many application depends on the top bit of pointers to be zero.
4 GB minus what is in use by the system if you link with /LARGEADDRESSAWARE.
Of course, you should be even more careful with pointer arithmetic if you set that flag.
Nobody seems to touch upon the fact that if you have many different 32-bit applications, the wow64 subsystem can map them anywhere in memory above 4G, so on a 64-bit windows with sufficient memory, you can run many more 32-bit applications than on a native 32-bit system.
A 32-bit process is still limited to the same constraints in a 64-bit OS. The issue is that memory pointers are only 32-bits wide, so the program can't assign/resolve any memory address larger than 32 bits.
An single 32-bit process under a 64-bit OS is limited to 2Gb. But if it is compiled to an EXE file with IMAGE_FILE_LARGE_ADDRESS_AWARE bit set, it then has a limit of 4 GB, not 2Gb - see https://msdn.microsoft.com/en-us/library/aa366778(VS.85).aspx
The things you hear about special boot flags, 3 GB, /3GB switches, or /userva are all about 32-bit operating systems and do not apply on 64-bit Windows.
See https://msdn.microsoft.com/en-us/library/aa366778(v=vs.85).aspx for more details.
As about the 32-bit operating systems, contrary to the belief, there is no physical limit of 4GB for 32-bit operating systems. For example, 32-bit Server Operating Systems like Microsoft Windows Server 2008 32-bit can access up to 64 GB (Windows Server 2008 Enterprise and Datacenter editions) – by means of Physical Address Extension (PAE), which was first introduced by Intel in the Pentium Pro, and later by AMD in the Athlon processor - it defines a page table hierarchy of three levels, with table entries of 64 bits each instead of 32, allowing these CPUs to directly access a physical address space larger than 4 gigabytes – so theoretically, a 32-bit OS can access 2^64 bytes theoretically, or 17,179,869,184 gigabytes, but the segment is limited by 4GB. However, due to marketing reasons, Microsoft have limited maximum accessible memory on non-server operating systems to just 4GB, or, even, 3GB effectively. Thus, a single process can access more than 4GB on a 32-bit OS - and Microsoft SQL server is an example.
32-bit processes under 64-bit Windows do not have any disadvantage comparing to 64-bit processes in using shared kernel's virtual address space (also called system space). All processes, be it 64-bit or 32-bit, under 64-bit Windows share the same 64-bit system space.
Given the fact that the system space is shared across all processes, on 32-bit Windows, processes that create large amount of handles (like threads, semaphores, files, etc.) consume system space by kernel objects and can run out of memory even if you have lot of memory available in total. In contrast, on 64-bit Windows, the kernel space is 64-bit and is not limited by 4 GB. All system calls made by 32-bit applications are converted to native 64-bit calls in the user mode.
You've got the same basic restriction when running a 32bit process under Win64. Your app runs in a 32 but subsystem which does its best to look like Win32, and this will include the memory restrictions for your process (lower 2GB for you, upper 2GB for the OS)
The limit is not 2g or 3gb its 4gb for 32bit.
The reason people think its 3gb is that the OS shows 3gb free when they really have 4gb of system ram.
Its total RAM of 4gb. So if you have a 1 gb video card that counts as part of the total ram viewed by the 32bit OS.
4Gig not 3 not 2 got it?
From what I understand, a 32-bit process can only access 2 GB of memory on 32-bit Windows without the /3GB switch, and that some of that memory is taken up by the OS for its own diabolical reasons. This seems to mesh with my experience as we have an app that crashes when it reaches around 1.2 - 1.5 GB of RAM without memory exceptions, even though there is still plenty of memory available.
Would moving this 32-bit application to 64-bit Windows allowing it accesses more than 1.5 GB it can now? Would the application itself have to be upgraded to 64-bit?
Newer versions of Visual Studio have a new flag which make 32-bit apps "big address space aware". Basically what it does is say that if it's loaded on a 64-bit version of windows, then it will get 4GB (the limit of 32-bit pointers). This is certainly better than the 2 or 3 GB you get on 32-bit versions of windows. See http://msdn.microsoft.com/en-us/library/aa366778.aspx:
Most notably it says:
Limits on memory and address space
vary by platform, operating system,
and by whether the
IMAGE_FILE_LARGE_ADDRESS_AWARE value
of the LOADED_IMAGE structure and
4-gigabyte tuning (4GT) are in use.
IMAGE_FILE_LARGE_ADDRESS_AWARE is set
or cleared by using the
/LARGEADDRESSAWARE linker option.
Also see: http://msdn.microsoft.com/en-us/library/wz223b1z.aspx
Yes, under the right circumstances, a 32-bit process on Windows can access a full 4GB of memory, rather than the 2Gb it's normally limited to.
For this to work, you need the following:
The app must be running on a 64-bit OS
The app must be compiled with the /LARGEADDRESSAWARE flag.
The app should be tested to make sure it actually works properly in this case. ;) (specifically, code that relies on all pointers pointing to addresses below the 2GB boundary will obviously not work here)
Your app will be limited by the pointer size, in your example 32 bits.
If your app was to access more memory then you would need some sort of segmented memory architecture like we had in the 16 bit days where apps used 16bit pointers and offsets to access the full 32bit memory space.
WOW64 allows using 32-bit Windows application on 64-bit Windows, translating 32-bit pointers to real 64-bit pointers. And actually 32-bit addressing should allow accessing 4GB of memory.