I have an application that uses a fair number (don't ask!) of Thread Local Storage variables. It has been operable without issue for almost a decade ... until a recent surprise.
A customer has a laptop running Windows7 (64 bit)... and on his system, the TLS allocation fails. (My problem isn't exactly this, but close enough). He claims his system is pretty standard; he has a ZoneAlarm but claims everything else is stock; take that with a grain of salt). How do I find out who or what is allocating other TLS slots (and running me effectively out of slots)?
The application is a console standalone application and fairly vanilla: open/read/write files, some multithreading, practically nothing else, so I expect only the basic Windows DLLs to be actively involved. I know that some DLLs (even the Windows ones) may service thread-based activities and so will allocate their own TLS slots, and that's OK. But I don't expect that lots and lots are allocated by anything sane. Is there something else that can insert itself into my process space as part of my application startup (maybe a firewall)?
How many TLS slots should be allocated in a typical process when it starts up? Where is the demand coming from? Can I trace TLS calls to see who is allocating them somehow?
EDIT: Jan 1, 2013: Having some experience now, and having my tool report the TLS demand when it starts, I see something like the following:
Win XP 64 8 cores 16 Gb RAM: 3 TLS variables busy
Windows Vista 32 4 Gb RAM: 4 TLS variables busy
Win 7 Pro: 4 cores 8 Gb RAM: 7 TLS variables busy
Win 7 (French version): 2 cores, 4 GB: minimum of 25 TLS variables busy, more in some configurations
We're checking on Windows 8, but its TLS demand appears to be higher. But what on earth accounts for the huge difference between the two Windows 7 systems? Any why is there any need for TLS variables (presumably for DLLs to store thread state) before my application starts running, especially considering there was no need for in earlier version of Windows?
If it helps, we are running a 32 bit application.
Related
I run the same application on Windows 10 and Windows 7 (both 64-bit) and used Process Explorer to monitor the Virtual Size. I noticed that the Virtual Size of the application running on Windows 10 is much larger than running on Windows 7.
By using VMMap, in Private Data:
Windows 10 - 1 address with 4.2GB is reserved and not committed.
Windows 7 - 2 address with 15MB are reserved and not committed.
So my questions are:
Why there is a large memory has been reserved? Is it just a different memory management done by Win10 and Win7?
Or, there is something to do with the application that I ran? (The application has no problem to run on Win10 & Win7)
Is there any suitable indicator to track memory leak? (Eg: 'Committed' value in VMMap) It seems like so much discussion/argument on this :P
My original question is that, is this technically rational to check the required heap-size of my Java program on Windows 7, via VisualVM, and come to this conclusion that the program will require the same amount of heap on Linux(RedHat) as well?
I don't know how the system(OS or even CPU and RAM), affect memory management of JVM.
well, the windows is my development system with 4GB of RAM and a Core 2 Due CPU, however the
Linux is the production system with 32GB of RAM and multiple powerfull processors,
Actually, my concern is that the program on Linux might need more memory. less is ok.
I hope someone with a bit of knowledge can clear this up. There's many discussions about the reasons to run a 64-bit OS (e.g. Windows 7 x64), but many people seem to think that their old x86 apps will be able to take advantage of any RAM greater than 3.5GB.
As I understand it, though, x86 apps cannot address memory that high... unless they've been specifically programmed to (which very few will have).
Can someone knowledgeable clear this up for me, once and for all? Can 32-bit apps take advantage of a system running 8GB of RAM?
E.g. If a user decided (for whatever reason) to run several x86 apps at once, filling the RAM as much as possible, would the extra addressable memory available in Windows 7 x64 be used?
Thanks!
On a 64 bit system, 32 bit applications are able to use the full 4GB virtual address space, minus about 64K. A default 32 bit windows system will only allow a 32 bit process to use 2 GB of virtual address space. By specially configuring the OS it's possible to push that limit up to 3 GB, but it's still not as good as what you would get on a 64 bit version of windows.
If you have 8GB of ram, that 8 GB can be divided up between multiple 32 bit processes, and the entire 8 GB will be utilized if necessary. However, no single 32 bit process will be allocated more than 4 GB of memory.
Although i don't have sources to cite, but from my knowledge: 32bit app will not be able to address more than 4GB of memory itself, unless it uses some tricks(that is very unlikely), but if you have some 32bit apps running at the same time, they can all have 4GB each, and thus two 32bit apps should be able to use all 8GB of memory. Though I'm not 100% sure.
Yes. x86 apps cannot use more than 2GB of memory at once without special tricks, but they can use any memory available.
Adding to the other (correct) answers:
Instead of the term "application" the word "process" should be used. Applications often consist of multiple processes whereas the limits discussed here apply to single processes.
Thus applications benefit from x64 that either are linked with the LARGEADDRESSAWARE flag (they can use 4 GB instead of 2 GB) or that share the load between multiple processes.
32-bit processes can work with more than 4 GB RAM even on 32-bit systems by using AWE. But a 32-bit process can only ever use 2 GB at once (4 GB with LARGEADDRESSAWARE on 64 bit respectively). AWE is primarily used by databases where it is essential for performance that the entire database fit into RAM. It works by providing a 2 GB window into a larger chunk of memory.
Here are some articles for further reading:
Windows x64 – All the Same Yet Very Different, Part 1: Virtual Memory
Windows x64 – All the Same Yet Very Different, Part 2: Kernel Memory, /3GB, PTEs, (Non-) Paged Pool
x64? My Terminal Servers Run Just Fine With 32 Bits and 8/12/16 GB RAM!
E.g. If a user decided (for whatever
reason) to run several x86 apps at
once, filling the RAM as much as
possible, would the extra addressable
memory available in Windows 7 x64 be
used?
The answer is yes. That's one of the benefits a virtual address space gives us--the ability for each process to appear (to the process) as though it's executing in a linear address space that starts at 0 and goes up from there.
As far as each of the 32-bit applications is concerned, it has its own address space from 0 to 2 gigabytes (without special tricks). The operating system handles the virtual-to-physical address translation.
I am investigating a strange problem with my application, where the behaviour is different on 2 versions of Windows:
Windows XP (32-bit)
Windows Server 2008 (64-bit)
My findings are as follows.
Windows XP (32-bit)
When running my test scenario, the XML parser fails at a certain point during the parsing of a very large configuration file (see this question for more information).
At the time of failure, the process size is approximately 2.3GB. Note that a registry key has been set to allow the process to exceed the default maximum process size of 2GB (on 32-bit operating systems).
The system of the failure is a call to IXMLDOMDocument::load() failing, as described in the question linked above.
Windows Server 2008 (64-bit)
I run exactly the same test scenario in Windows Server 2008 -- the only variable is the operating system. When I look at my process under Task Manager, it has a * 32 next to it, which I am assuming means it is running in 32-bit compatibility mode.
What I am noticing is that at the point where the XML parsing fails on Windows XP, the process size on Windows Server 2008 is only about 1GB (IOW, approximately half the process size as on Windows XP).
The XML parsing does not fail on Windows Server 2008, it all works as it should.
My questions are:
Why would a 32-bit application (running in 32-bit mode) consume half the amount of memory on a 64-bit operating system? Is it really using half the memory, it is usual virtual memory differently, or is it something else?
Acknowledging that my application (seems) to be using half the amount of memory on Windows Server 2008, does anyone have any ideas as to why the XML parsing would be failing on Windows XP? Every time I run the test case, the error accessed via IXMLDOMParseError (see this answer) is different. Because this appears to be non-deterministic, it suggests to me that I am running into a memory usage problem rather than dealing with malformed XML.
You didn't say how you observed the process. I'll assume you used Taskmgr.exe. Beware that it's default view gives very misleading values in the Memory column. It shows Working set size, the amount of RAM that's being used by the process. That has nothing to do with the source of your problem, running out of virtual memory space. There is not much reason to assume that Windows 2008 would show the same value as XP, it has a significantly different memory manager.
You can see the virtual memory size as well, use View + Columns.
The reason your program doesn't bomb on a 64-bit operating system is because 32-bit processes have close to 4 gigabytes of addressable virtual memory. On a 32-bit operating system, it needs to share the address space with the operating system and gets only 2 gigabytes. More if you use the /3GB boot option.
Use the SAX parser to avoid consuming so much memory.
Not only are there differences in available memory between 32 bit and 64 bit (as discussed in previous answers), but its the availability of contiguous memory that may be killing your app on 32 bit.
On 32 bit machine your app's DLLs will be littering the memory landscape in the first 2GB of memory (app at 0x00400000, OS DLLs up at 0x7xxx0000, other DLLs elsewhere). Most likely the largest contiguous block you have available is about 1.1GB.
On a 64 bit machine (which gives you the 4GB address space with /LARGEADDRESSAWARE) you'll have a least one block in that 4GB space that is 2GB or more in size.
So there is your difference. If your XML parser is relying on a large blob of memory rather than many small blobs it may be that your XML parser is running out of contiguous usable space on 32 bit but is not running out of contiguous usable space on 64 bit.
If you want to visualize this on the 32 bit OS, grab a copy of VMValidator (free) and look at the Virtual view for a visualization of your memory and the Pages and Paragraphs views to see the data for each memory page/paragraph.
Obviously, that's 64-bit windows.
Also, what's the maximum amount of memory a single 64-bit process can use?
I was kind of counting on using it all...
(Yes, I know what I'm doing, please don't tell me that if I need that much RAM i must be doing something wrong)
Also, is this the same for a .Net 2.0 process? Or is there a lower limit for .Net?
What version of windows? it differs from XP to vista and from home to business versions of vista, and I would guess again for server.
see here for more info on maximum ram for diffrent windows versions
for Windows Server 2008 Datacenter MS quote 2 TB of physical memory.
Link
From http://technet.microsoft.com/en-us/library/cc758523.aspx
- Windows Server 2003, 64 bit Datacenter Edition supports physical memory up to 512GB
A single process should be able to use most of it, some will be used by the OS.
The answer from Re0sless is better then mine. The limit is now 2TB, in Datacenter SP2, and 2008.
We run Windows boxes with 16 gigs of memory, but that is because we are running multiple VM Ware instances, I presume you mean in a single instance. On Vista it depends upon the edition. It breaks out like this:
Vista Basic: 8 GB
Vista Home Premium: 16 GB
Vista Business/Enterprise/Ultimate: 128+ GB
Something we found out recently: with MySQL running on Win32, you can only use up to 2GB per process. On Win64, the memory is not managed as well and a single MySQL instance will run your memory into the ground. Ours used up all 16GB we have. So regarding how much memory 1 64-bit process can use: the answer is however much the OS allows.
According to wikipedia you can have 128 GB of physical RAM in a 64-bit Windows XP computer.
This is a Windows Server machine.
As for which edition (Datacenter, Enterprise, etc)... Whatever it takes to give my little .Net Process as much memory as it can.
Switch to Linux. You will not have any of these issues and you will get better performance.