We run multiple copies of our application at the same time on a multi core server with 2 NUMA nodes. We already set the affinity of the app to cores on either NUMA zero or one to increase memory access performance. However it occurred to me that different pages of dll's and even the exe are probably being loaded into different NUMAs, depending on the most local NUMA of the process that accessed a page first.
Is this happening? Is there anyway to configure windows to load a page from a dll or exe more than once if there are more than one NUMA? I'm guessing hell no.
hmmm. Would putting a copy of our exe in two directories work? Would be enough for windows to think the dll was different. If not I could rename each dll _0 or _1. Maybe there is something in the PE header I have to change to make each dll seem different.
We use windows server 2008 R2 and are upgrading to 2012 soon.
Related
Am I right in thinking there is no point in rebasing our dlls during our build if we use ASLR as the dlls will be rebased again anyway when the kernel comes to load them?
I am concerned that our application is often used on Terminal Services machines. So, if rebasing occurs at load time, we could end up with dlls being rebased for each process they are loaded into (there would be one process per session). And this would result in more memory usage and paging than we want to pay for. Do I need to be concerned?
I've found the following blog post that says the rebasing only happens once and it is system wide: Matt Evans - Enabling ASLR for memory savings?. I haven't seen any other references about this, so just wanted to be sure if I use ASLR and don't rebase during our build I won't cause memory problems on a Terminal Services box?
So based on my reading you should not have a problem. ASLR causes the the dll's to be loaded to semi random memory address and should not just start rebasing for every process. If you want to check memory use of dll's there is a free tool called MassiveRebase that lets you dynamically load two dll's and view info about their memory use. The was designed to view changes that aslr may have on memory.
The tool and more about it can be found here: http://www.tmurgent.com/appv/index.php/en/resources/tools/137-massive-rebase
Hope this helps.
Rebasing is still helpful. When the operating system loads, it applies a fixed random value to the DLL base.
The result is that the location a DLL is loaded to, is typical for a single boot, but different between machines and boots.
This means that a given DLL in lots of processes can be shared between processes, as all its code data is shared with the same value.
When a DLL is moved because it's address space is taken, it has to modify the fixups, and less of the DLL is shared, increasing system load.
If your DLL is not shared, then it does not affect resources.
The cost of fixing up a DLL used to be cheaper if it was loaded to the correct place, not sure if that is true for ASLR, but may still save resource loading time.
In Windows if there are two processes each using the same DLL then apparently each process separately loads the DLL into its address space while in Linux a shared-object is loaded once and mapped into different processes. Can someone explain to me the pros and cons of the Windows approach ?
I'm not sure the difference is so stark. Windows shares everything except the data segment between all users of a DLL by loading the DLL once and mapping the shared parts into each process. However, any global data in the DLL is loaded separately for each process so that processes don't unintentionally share data. I'd be surprised if linux wasn't very similar, otherwise shared libraries could pose significant security risks, not to mention potential reliability issues as well. Here are a couple references:
From stackoverflow:
Are .dll files loaded once for every program or once for all programs?
From wikipedia:
http://en.wikipedia.org/wiki/Dynamic-link_library
From a Microsoft article:
Address Space Layout Randomization (ASLR)
ASLR moves executable images into random locations when a system
boots, making it harder for exploit
code to operate predictably. For a
component to support ASLR, all
components that it loads must also
support ASLR. For example, if A.exe
consumes B.dll and C.dll, all three
must support ASLR. By default, Windows
Vista and later will randomize system
DLLs and EXEs, but DLLs and EXEs
created by ISVs must opt in to support
ASLR using the /DYNAMICBASE linker
option.
I don't quite get it. Take the base system DLLs loaded by every process on WIndows: NtDll.dll and kernel32.dll.
If a have a non-aware executable, will these system DLLs use ASLR? That is, will they load at a different base address after every system reboot on Win 7 for this executable or will they always load at the same base address after system reboot like they do on Win XP?
To make it more clear what I mean: My typical dummy program's startup stack will look like this:
write_cons.exe!wmain() Line 8 C++
write_cons.exe!__tmainCRTStartup() Line 583 + 0x19 bytes C
write_cons.exe!wmainCRTStartup() Line 403 C
> kernel32.dll!_BaseProcessStart#4() + 0x23 bytes
Looking at the asm of BaseProcessStart, I see on my XP box here:
_BaseProcessStart#4:
7C817054 push 0Ch
7C817056 push 7C817080h
7C81705B call __SEH_prolog (7C8024D6h)
7C817060 and dword ptr [ebp-4],0
...
Now what interests me is the following:
On Windows XP, the address will always be 0x7C817054, regardless of how many times I reboot this machine. If I were on Win7 with ASLR, will this address change between reboots if the executable that loads kernel32.dll is not enabled for ASLR?
(Note: For me, atm., there is only one minor use-case this address would be useful for: In Visual Studio, I can only set a "Data Breakpoint" for assembly level functions, that is a breakpoint # 0x7... - If I want to break in a specific ntdll.dll or kernel32.dll function, in Windows XP I do not have to adjust my breakpoints between reboots. With ASLR kicking in (the scope of this question) I would have to change the Data Breakpoints between reboots.)
Technically whether the system dlls get relocated or not, it shouldn't matter, as the linker will bind to symbols, not addresses. These symbols are resolved by the runtime loader into to addresses for the instanced system dlls, thus your binary should be none the wiser. From what i've seen however, windows 7 will reset the base randomization every reboot, including system dlls(note: this is from debuging WOW64 apps on widows server 2008 R2). You can also do a system wide disabling of ASLR via some registery edits, but thats not really relevant...
Update:
the section on ASLR in this article explains what gets relocated and when.
it doesn't mention if the base will reset every reboot, but for system dlls, its never going to be guaranteed to load at the same address twice, reboot or no reboot.
the important thing is according to article, everything needs to opt-in to ASLR for system dll's to be relocated.
Your program will resolve calls into system DLLs wherever they happen to be loaded. But, unless your executable is linked with /DYNAMICBASE, it will not be given a randomized base address. In other words, your exe will always load at the same base address.
If you want your exe to load at a randomized address, then you have to link it with /DYNAMICBASE, and every DLL that it references must also have been linked with /DYANMICBASE. The system DLLs (starting in Vista) are all linked with /DYNAMICBASE.
I always thought .dll is working the same way as .so in linux,
but some article says not.
Does windows ensure that memory will contain only one copy the same dll as in linux?
I don't have a clue to check it myself in windows,so I can only ask here.
UPDATE
Anyone knows how do verify this manually?
Here's a short description: DLL Hell, basically in modern Windows it use a technique called Memory Mapping where the DLL is loaded once, if both processes try to load the DLL from the same directory. So to answer your question, it is working the same was as in Linux.
If the DLL can be loaded at the same base virtual address in two processes then there will only be one copy of the DLL in physical memory.
Since Windows does not use position independent code, if a DLL cannot load at its preferred base address it will be rebased and thus not be able to share physical memory with other instances.
I watched a ASLRed dll images's based address for 32bit Process.
It's not a fully randomization. It just randomizated 1/2 probability.
For example, once I load a dll then the image is loaded on 0x12345678.
And I load the image again, the image is loaded on 0x23456789.(Base address is changed!)
But I load the image again
0x12345678
0x23456789
0x12345678
0x23456789
...
Why they did implement like this?
Is it for a crash report's frequency?(For getting same crash addresses of re-deployed dlls)
This is by design. Normally, Windows selects a preferred base address for an ASLR DLL when the DLL is first loaded, and then it keeps using that address until the system is rebooted. That way the DLL will be mapped at the same address in every process that loads it, allowing code pages to be shared.
However, if a DLL has been unloaded from every process, the system may sometimes select a different base address the next time the DLL is loaded. It does this to reduce virtual address space fragmentation, not for security reasons. This is what seems to be happening in your case.
It's documented as being at one of 1 of 256 possible starting addresses.
But i didn't think it even applied to a process, but to shared DLL's.
ASLR: is not on by default for process images. It's an opt-in thing, for compatiblity.(3)
Address Space Layout Randomization
(ASLR)
ASLR moves executable images into
random locations when a system boots,
making it harder for exploit code to
operate predictably. For a component
to support ASLR, all components that
it loads must also support ASLR. For
example, if A.exe consumes B.dll and
C.dll, all three must support ASLR. By
default, Windows Vista and later will
randomize system DLLs and EXEs, but
DLLs and EXEs created by ISVs must opt
in to support ASLR using the
/DYNAMICBASE linker option.
ASLR also randomizes heap and stack
memory:
When an application creates a heap in
Windows Vista and later, the heap
manager will create that heap at a
random location to help reduce the
chance that an attempt to exploit a
heap-based buffer overrun succeeds.
Heap randomization is enabled by
default for all applications running
on Windows Vista and later.
When a
thread starts in a process linked with
/DYNAMICBASE, Windows Vista and later
moves the thread's stack to a random
location to help reduce the chance
that a stack-based buffer overrun
exploit will succeed.
Had installed new Win8 RC x64 yesterday.
Watch out!
Kernel32.dll (64-bit version) have different base address in different processes (in single session, of course). Only ntdll.dll base address remains constant. I had to change the code, you can no longer rely on the permanent address Loadlibrary.