On Linux x86_64 systems (at least RHEL 6.4 running 2.6.32), the first syscall argument reported by /proc/pid/syscall for 32-bit processes seems to be bogus ... I'd like to determine why.
For example, if a process is blocked doing openat(AT_FDCWD, ...), I see the following in /proc/pid/syscall ...
64-bit:
257 0xffffffffffffff9c ...
32-bit:
295 0xffffffff810495c0 ...
AT_FD_CWD == -100 == 0xffffffffffffff9c, so the 64-bit case looks right, but the value for 32-bit seems to have been clobbered (the other arguments look right).
I know that I can find a struct pt_regs at the top of the stack, and I can find it via ...
#define task_pt_regs(tsk) ((struct pt_regs *)(tsk)->thread.sp0 - 1)
I see the bogus value there.
However ... strace seems to be able to find the correct value of the first argument even in the 32-bit case. It does ptrace(PTRACE_PEEKUSER), which AFAIK just looks at the same struct pt_regs.
I must be missing something ... what magic happens for strace that allows it to see valid register values via PTRACE_PEEKUSER? What happens to clobber the first argument as seen via /proc/pid/syscall?
Related
Can anyone tell me what the meaning of the value for st_ino is when running os.stat() on Windows (Python 3.5.3)?
In earlier Python releases, this contained dummy values, but this has recently changed and I couldn't find how it is calculated/generated. I suspect that it differs based on the file system (NTFS, FAT, ...)
Example
import os
stat = os.stat(r'C:\temp\dummy.pdf')
for attr in dir(stat):
if attr.startswith('st_'):
print('{}: {}'.format(attr,
stat.__getattribute__(attr)))
Result
st_atime: 1495113452.7421005
st_atime_ns: 1495113452742100400
st_ctime: 1495113452.7421005
st_ctime_ns: 1495113452742100400
st_dev: 2387022088
st_file_attributes: 33
st_gid: 0
st_ino: 10414574138828642
st_mode: 33060
st_mtime: 1494487966.9528062
st_mtime_ns: 1494487966952806300
st_nlink: 1
st_size: 34538
st_uid: 0
Background
I used the shutil.copyfile() function an ran into an SameFileError. After looking through the code (and despite what it says in the comments of shutil.py) the shutil._samefile() function does not compare pathnames in Windows. Instead, it uses os.path.samefile() which compares the st_ino and st_dev values.
Both source and target file resided on the same device (volume), which would explain why the value for st_dev was the identical. But I'm still puzzled as to why st_ino had the same value for both files.
Remark: both files were on a Sharepoint volume mounted using webDAV, so their st_ino value may have been 0 (dummy), which would explain why they were equal. I'm still curious though ;-)
Update
As I suspected, the value for st_ino returned for the files residing on the Sharepoint volume (WebDAV), was 0, as was the value for st_dev. This is the reason for (incorrect) SameFileError. Example output:
\\sharepoint#SSL\AUT.pdf os.stat_result(st_mode=33206, st_ino=0, st_dev=0, st_nlink=1, st_uid=0, st_gid=0, st_size=4717, st_atime=1495031011, st_mtime=1495031011, st_ctime=1495031570)
\\sharepoint#SSL\ING.pdf os.stat_result(st_mode=33206, st_ino=0, st_dev=0, st_nlink=1, st_uid=0, st_gid=0, st_size=4722, st_atime=1495031203, st_mtime=1495031203, st_ctime=1495031733)
\\sharepoint#SSL\WAG.pdf os.stat_result(st_mode=33206, st_ino=0, st_dev=0, st_nlink=1, st_uid=0, st_gid=0, st_size=4710, st_atime=1495031511, st_mtime=1495031511, st_ctime=1495031912)
On OS X El Capitan, my log file system.log feels with hundreds of the following lines at times
03/07/2016 11:52:17.000 kernel[0]: hfs_clonefile: cluster_read failed - 34
but there is no indication of the process where this happens. Apart from that, Disk Utility could not find any fault with the file system. But I would still like to know what is going on and it seems to me that dtrace should be perfectly suited to find out that faulty process but I am stuck. I know of the function return probe but it seems to require the PID, e.g.
dtrace -n 'pidXXXX::hfs_clonefile:return { printf("ret: %d", arg1); }'
Is there a way to tell dtrace to probe all processes? And then how would I print the process name?
You can try something like this (I don't have access to an OS X machine to test it)
#!/usr/sbin/dtrace -s
# pragma D option quiet
fbt::hfs_clonefile:return
/ args[ 1 ] != 0 /
{
printf( "\n========\nprocess: %s, pid: %d, ret value: %d\n", execname, pid, args[ 1 ] );
/* get kernel and user-space stacks */
stack( 20 );
ustack( 20 );
}
For the fbt probes, args[ 1 ] is the value returned by the function.
The dTrace script will print out the process name, pid, and return value from hfs_clonefile() whenever the return value is not zero. It also adds the kernel and user space stack traces. That should be more than enough data for you to find the source of the errors.
Assuming it works on OS X, anyway.
You can use the syscall provider rather than the pid provider to do this sort of thing. Something like:
sudo dtrace -n 'syscall::hfs_clonefile*:return /errno != 0/ { printf("ret: %d\n", errno); }'
The above command is a minor variant of what's used within the built-in DTrace-based errinfo utility. You can view /usr/bin/errinfo in any editor to see how it works.
However, there's no hfs_clonefile syscall, as least as far as DTrace is concerned, on my El Capitan (10.11.5) system:
$ sudo dtrace -l -n 'syscall::hfs*:'
ID PROVIDER MODULE FUNCTION NAME
dtrace: failed to match syscall::hfs*:: No probe matches description
Also, unfortunately the syscall provider is prevented from tracing system processes by the System Integrity Protection feature introduced with El Capitan (macOS 10.11). So, you will have to disable SIP which makes your system less secure.
I have a virtual Windows 7 x64 machine on a Windows 10 host, and I kernel debug it with windbg 10.0.10586.567. I'm running my own application on it, which I have full source and private symbols for. Whenever I break in and ask for stack traces of the app's threads, the backtrace always stops when one of my application's binaries are "hit."
So for instance, if I break in, switch to the process, and request a stacktrace with !thread [thread address] 1f, I get something like this (note the "early" zero return address at the last line):
fffff880`0534e870 fffff800`026d6992 nt!KiSwapContext+0x7a
fffff880`0534e9b0 fffff800`026d81a2 nt!KiCommitThreadWait+0x1d2
fffff880`0534ea40 fffff800`029c7a2e nt!KeDelayExecutionThread+0x186
fffff880`0534eab0 fffff800`026d08d3 nt!NtDelayExecution+0x59
fffff880`0534eae0 00000000`76e7165a nt!KiSystemServiceCopyEnd+0x13 (TrapFrame # fffff880`0534eae0)
00000000`00276708 000007fe`fcf91203 ntdll!NtDelayExecution+0xa
00000000`00276710 00000001`410e7dd9 KERNELBASE!SleepEx+0xab
00000000`002767b0 00000000`00000000 MyApp!MainMessageLoop+0x4b1 [d:\whatever\path\myapplication.cpp # 3024]
This looks very similar to when you you are missing a binary while debugging a user-mode dump (lack of unwind data) of an x64 process, except in that case the stack trace usually does not stop "this sudden", rather it goes astray at that point, and shows bogus values.
Some extra info/things I tried:
I have the correct symbol paths set up (both the Microsoft symbol server, and a local folder on the host with matching PDBs, even though the latter is not needed for just the stack trace)
I have a binary path set up (.exepath) containing matching binaries on the host (I've made absolutely sure of this; copied the binaries directly from the guest to the host machine)
If I put a breakpoint in one of the app's exported DLL functions, then when the debugger breaks in, I get a one-liner stack trace like this: 0000000000274b40 0000000000000000 MyAppDLL!SomeExportedFunction+0x32 [d:\whatever\path\myapplicationDLL.cpp # 232]
I've tried virtually every combination of commands to get a stacktrace (.process /i, .process /r /p, !process -1 7, .reloads, .reload /users, .reload /f MyApp.exe, !thread [address] 1f, etc.) with no success
Tried with an older version of windbg (6.11.0001.404) as well, same result
Also tried on Windows 8.1 as a guest with the very same binaries, same result
!sym noisy output (irrelevant lines omitted):
0: kd>.process /i [address]
0: kd>g
0: kd>.reload /user
0: kd> !process -1 2
0: kd> !thread [address] 1f
[...]
DBGHELP: d:\symbolcache\MyApp.pdb\76931C5A6C284779AD2F916CA324617E1\MyApp.pdb already cached
DBGHELP: MyApp - private symbols & lines
[...]
lmvm MyApp output:
[...]
Loaded symbol image file: MyApp.exe
Image path: C:\MyApp\MyApp.exe
[...]
Any ideas?
I accidentally stumbled into a linker switch that solves this problem: /DEBUGTYPE with the PDATA argument. If you link your binaries with this switch, unwind information will be copied into your PDBs.
I recompiled/relinked the application in question with /DEBUGTYPE:CV,PDATA (/DEBUGTYPE:CV is the default if /DEBUG is specified, see the documentation), now everything works like a charm, I always get full call stacks.
One strange aspect of this: windbg happily uses unwind data found in the PDBs, but ignores the very same data in the mapped binaries (both on the host machine).
This is not a perfect solution to the problem (or any solution at all, one might say), but I'm providing this provisional answer with a workaround.
You should be able to get the information you want, albeit not so well-formatted using something like dps #rsp L10.
In x86-64 you don't have a parallel of the x86 ebp-chain, but the return addresses are still on the stack. Those will give you the functions in the stack, and the values between them will be the arguments passed to the functions (and saved registers on the stack, etc.). A random example from Google (as I'm not on my Windows machine right now):
0:017> dps #rsp
00000000`1bb0fbb8 00000000`00000020
00000000`1bb0fbc0 00000000`00000000
00000000`1bb0fbc8 00000000`008bc6c6 Dolphin!ReadDataFromFifoOnCPU+0xb6 [d:\sources\comex\source\core\videocommon\fifo.cpp # 245]
00000000`1bb0fbd0 00000000`1ba0ffeb
00000000`1bb0fbd8 00000000`00000020
00000000`1bb0fbe0 00000000`00000020
00000000`1bb0fbe8 00000000`00000800
00000000`1bb0fbf0 00000000`1ba0ffeb
00000000`1bb0fbf8 00000000`008c2ff5 Dolphin!InterpretDisplayListPreprocess+0x45 [d:\sources\comex\source\core\videocommon\opcodedecoding.cpp # 87]
00000000`1bb0fc00 00000000`00000000
00000000`1bb0fc08 00000000`008bc041 Dolphin!RunGpu+0x81 [d:\sources\comex\source\core\videocommon\fifo.cpp # 389]
00000000`1bb0fc10 00000000`8064cbc0
00000000`1bb0fc18 00000000`1bb0fcc0
00000000`1bb0fc20 00000000`00000000
00000000`1bb0fc28 00000000`008c2dda Dolphin!OpcodeDecoder_Preprocess+0x14a [d:\sources\comex\source\core\videocommon\opcodedecoding.cpp # 326]
00000000`1bb0fc30 00000000`8064cbe0
Given that you have symbols, the return addresses are easily distinguishable.
The unwind data is lazy loaded for user mode modules, so it's not going to be mapped unless someone needs it. Unfortunately the kernel debugger doesn't force the information to be present for user images, so sometimes you get this behavior. You can see if the data is mapped or not by dumping the PE header (!dh) and checking the state of the Exception Directory (!pte imagename+offset).
Given that you own the app, try forcing the information to be resident by doing a stack walk NOP somewhere in your app:
PVOID stack[2];
(VOID)CaptureStackBackTrace(0, 2, (PVOID*)&stack, NULL);
That doesn't guarantee the entire directory will be present, but usually good enough.
I couldn't find a precise answer so I've decided to ask.
I've been reading the "Inside Windows Debugging" and in the sample it tells me to set a breakpoint on the kernel32!CreateProcessW.
But before that it uses the .symfix debugger command to set the debugger symbols search path to point to the Microsoft online symbols server. When I try to set the breakpoint I get an error that it cannot resolve the function (or something like that). It looks like this.
0:000> bp kernel32!CreateProcessW
Couldn't resolve error at 'kernel32!CreateProcessW'
It's probably because there's no "kernel32!CreateProcessW" in the list below.
0:000> x kernel32!CreateProcess*
76b90cb9 KERNEL32!CreateProcessWithTokenW (void)
76b90d84 KERNEL32!CreateProcessAsUserW (void)
76b90d84 KERNEL32!CreateProcessWithLogonW (void)
76b4e225 KERNEL32!CreateProcessWStub = <no type information>
76b72e04 KERNEL32!CreateProcessInternalAStub = <no type information>
76b72e15 KERNEL32!CreateProcessInternalWStub = <no type information>
76b72de2 KERNEL32!CreateProcessAStub = <no type information>
76b72df3 KERNEL32!CreateProcessAsUserWStub = <no type information>
Everything goes fine if I set the breakpoint to kernel32!CreateProcessWStub but I wondered why I couldn't find and set the breakpoint to the kernel32!CreateProcessW.
This book probably focuses on a reader who's using Windows 7. I'm using Windows 8.1 and thought maybe that kernel32!CreateProcessW got deprecated...
I'm extremely new to this field and apologize if this is a completely stupid question. But thanks for reading it anyway.
CreateProcessW is definitely NOT deprecated. Furthermore, the only documented entry point is still in kernel32.dll, so for all intents and purposes, you should continue calling CreateProcessW through kernel32.dll, and not through kernelbase.dll.
Here is some more details to help understand what you are observing. Windows team often moves code around, and for the last few releases they had strong habit of breaking larger DLL's into smaller ones, which includes kernel32, ole32, user32, gdi32 to name a few. It is not new, Raymond Chen wrote about this in 2006. However the mechanism Raymond describes is based on forwarders, while what you see here with kernel32!CreateProcessW is a stub, i.e. the function that calls kernelbase!CreateProcessW and then returns:
0:014> u kernel32!CreateProcessWStub l14
KERNEL32!CreateProcessWStub:
00007ffd`83cf58a8 4c8bdc mov r11,rsp
00007ffd`83cf58ab 4883ec58 sub rsp,58h
00007ffd`83cf58af 488b8424a8000000 mov rax,qword ptr [rsp+0A8h]
00007ffd`83cf58b7 498943f0 mov qword ptr [r11-10h],rax
... skip ...
00007ffd`83cf58f5 ff1555871100 call qword ptr [KERNEL32!_imp_CreateProcessW (00007ffd`83e0e050)]
00007ffd`83cf58fb 4883c458 add rsp,58h
00007ffd`83cf58ff c3 ret
The function called as you can see is kernelbase!CreateProcessW
0:014> ln poi kernel32!_imp_CreateProcessW
(00007ffd`82f92604) KERNELBASE!CreateProcessW | (00007ffd`82f926d0) KERNELBASE!MakeLocHashNode
Exact matches:
KERNELBASE!CreateProcessW (no parameter info)
I this case I don't know why Windows folks decided to use a stub rather than a forwarder, to me it seems it would be more efficient to simply forward a call, like most of the other refactoring, was done.
Kernel32.dll in Windows 8.1 still contains export symbol CreateProcessW. Command link /dump /exports prints out all export symbols:
c:\>link /dump /exports c:\Windows\System32\kernel32.dll | findstr CreateProcessW
220 DB 000058A8 CreateProcessW = CreateProcessWStub
You can use the same command to determine where you should set a breakpoint to. Similarly for forwarded exports:
c:\>link /dump /exports c:\Windows\System32\kernel32.dll | findstr EnterCriticalSection
298 129 EnterCriticalSection (forwarded to NTDLL.RtlEnterCriticalSection)
1418 589 TryEnterCriticalSection (forwarded to NTDLL.RtlTryEnterCriticalSection)
The reason why WinDbg cannot resolve symbol kernel32!CreateProcess, is probably just a bug in WinDbg. In this case, the symbols is not part of .PDB file, but contained in a special section in PE image, and apparently WinDbg does not handle that. What is interesting is that if .PDB file is not available, WinDbg is happy to use export table of PE image:
0:014> .sympath .
0:014> .reload
Reloading current modules ....
0:014> x kernel32!CreateProcessW
00007ffd`83cf58a8 <b>KERNEL32!CreateProcessW</b> (no parameter info)
Apparently, WinDbg decides to use either PDB symbols or export symbols, but not both.
In the book "Rootkit Arsenal" page 84 (Chapter 3) mentions:
..., we can view the contents of the
target machine's descriptor registers
using the command with the 0x100 mask:
kd> rM 0x100
and a paragraph below:
Note that the same task can be
accomplished by specifying the GDTR
components explicitly: kd> r gdtr ....
I run Windbg on my Win XP (inside VMWare) and choose the Kernel Debug -> Local.
My problem is in case of first command, windbg errors with:
lkd> rM 0x100
^ Operation not supported in current debug session 'rM 0x100'
and in the second command:
lkd> r gdtr
^ Bad register error in 'r gdtr'
Can anyone guide me ?
Right, you can't look at registers in a local kernel debug session. LiveKD works and you can also get the address indirectly through the PCR (!pcr).
-scott
I think I've found the solution:
Use two computers for kernel debugging instead of Local Kernel Debug.
(I used VMWare and am debugging through the COM port/named pipe)
I am thinking why this facility/feature (Local Kernel Debugging) is there if it's not complete ?