EDIT: Just a confirmation whether you can reproduce this or not would be useful. Only a single computer is needed to try this (no remote connection necessary).
Update It seems other can't reproduce this on Mac or Win7, so it's either WinXP-specific or specific to my machine. At this point I'm giving up.
It would be good to have a tutorial on how the Front End and the Kernel communicate, so we can debug remote kernel issues. Any such general answers (or links to tutorials elsewhere) are most welcome. I am already aware of Remote Kernel Strategies by Sascha Kratky.
Now the practical question: When I connect to a remote machine in any way (e.g. using remote kernel strategies), the front end will sooner or later freeze up. How can I prevent this?
Here's how to reproduce the problem on a single machine (no remote connection needed):
Create a new kernel configuration (Evaluation menu -> Kernel Configuration Options...)
Select Advanced Options.
Arguments to MLOpen:
-LinkName "8000#YOUR-IP-HERE,8001#YOUR-IP-HERE" -LinkMode Listen
-LinkProtocol TCPIP -LinkHost "YOUR-IP-HERE" -LinkOptions MLDontInteract
(Replace YOUR-IP-HERE with your computer's IP address. On Windows you can get this by typing ipconfig in a command window.)
Shell command to launch kernel: leave empty (we will do it manually)
Open a new notebook, set the notebook's kernel to the one you just set up, and evaluate something (1+1).
Now we need to launch the kernel manually, before the connection times out. So start a new kernel (math) in command line mode, and evaluate the following:
link=LinkConnect["8000,8001",LinkProtocol->"TCPIP"] (* connect to front end link *)
$ParentLink = link; (* set the front end link as parent link *)
Now the connection is established, and everything appears to work correctly (save for the one message in the front end's pink message window)
Evaluate Manipulate[n, {n, 0, 100, 1}]. First, this appears to work too. Now play with the slider. Sooner or later the front end will freeze and needs to be killed. For me, dragging the slider up and down for ~10 seconds without releasing it always suffices.
Why does the Front End freeze? How can I create a TCPIP-mode connection between the front end and the kernel in a way that everything will be working correctly?
Some notes:
I'm on Windows XP.
The problem never appears if I use LinkProtocol -> "SharedMemory".
Aborting calculations using Evaluation -> Abort Evaluation does work correctly.
I have verified using Links[] that MathLink`$PreemptiveLink and MathLink`$ServiceLink are created and LinkConnectedQ[] returns True on them.
Again, note that the Front End will freeze only after dragging the Manipualate slider continuously for a few seconds without releasing it.
Link to same question on MathGroup.
Related: Firewall settings for remote kernel to work (Mathematica)
(not an answer, but hard to write this as comment)
fyi, there is a little bit on information about how the frontend talks to the kernel here
http://reference.wolfram.com/mathematica/tutorial/AdvancedDynamicFunctionality.html
about half way down, it says
The front end and kernel communicate with each other through several
MathLink connections, known as the main link, the preemptive link,
and the service link. The main and preemptive links are pathways by
which the front end can send evaluation requests to the kernel, and
the kernel can respond with results. The service link works in reverse,
with the kernel sending requests to the front end.
and more there. May be that will help. Also, same information is in the PDF file on page 31:
http://www.wolfram.com/learningcenter/tutorialcollection/DynamicInteractivity/
The actual messaging protocol between the kernel and front end must be very advanced.
Related
Working on virtual machine environment, I'm using DbgView to obtain my printout log messages using debug_wsting
However, Working on large number of processes that produces those messages seem to consume lot to cpu and eventually halt the VM, So I'm looking of a way to offload this activity to the machine that runs this VM.
Perhaps there's a way to output the logs to remote windbg ?
Is there anything to configure on Client/Server ?
thanks
I never tried it on my own, but it's described in Windows Sysinternals Administrator's reference (Amazon). I thought I knew the Sysinternals tools quite well, but it turned aout I learned a lot. Totally worth reading.
To perform remote monitoring, DebugView runs in agent mode on the remote system [...]
To do so, run DbgView /a. Add /k if you need kernel messages as well and /gfor session 0.
[...] sending debug output it captures to a central DebugView viewer that displays the output.
and
To begin remote monitoring, press Ctrl+R or choose Connect from the Computer menu [...]
Be careful not to connect multiple viewers to a single computer because the debug output will be split between those viewers.
So, the output is there only once and after whatever machine fetches it, it's gone. Sounds normal.
I am new to kernel mode windows driver development and having issues getting debug information out of the WFP callout samples. I am looking for either more options or tips of what I am doing wrong with the options I have been trying.
To start with I downloaded the WFP sampler and followed the instructions in it's description. I am using a Windows 8.1 x64 host and a Windows 7 x64 target connected via a simulated serial cable. They are both VirtualBox VMs.
The primary command I am debugging with is
WFPSampler.Exe -s PROXY -l FWPM_LAYER_ALE_BIND_REDIRECT_V4 -aaid "C:\Program Files (x86)\Internet Explorer\iexplore.exe" -pla 10.0.2.15 -v
Where 10.0.2.15 is the IP address of a diffrent network interface than the routing table is endign traffic to. I also used the following inspect command as recommended by the instructions:
WFPSampler.exe -s BASIC_PACKET_EXAMINATION -l FWPM_LAYER_INBOUND_IPPACKET_V4 -v
"netsh wfp show state" shows the callout and filter associated with the expected layer
However I never got any messages from traceview as per the last steps of those instructions. Tracelog had similar lack of output. This was true with or without WPP tracing enabled on the driver project. I also tried higher verbosity, all to no effect.
Additionally the remote debugger in visual studio either stopped at "Driver Post Install Actions (x64) (possible reboot): Pass" or "Waiting to reconnect..." rebooting the target did not cause more output in the Debugger Immediate Window (the Output window never got any output). Rebooting the target VM did not get any unusual prompts (some things I read implied it should). The reconnect checkbox when setting up the debugger sometimes caused it to get past the "Waiting to reconnect..." prompt, other times it was not needed. I set a number of breakpoints that should have been hit including one at the top of every classify function and none were ever hit.
I tried to debug with WinDbg, but well, frankly I can't find the documentation about how to use this tool. I start it on the target machine and chose kernel debug, local. then I get a prompt that tells me the symbol search path is invalid and no clue what I am supposed to set it to. Any documentation on how to use this tool as opposed to just installing it may be helpful if it lets me debug these callout drivers.
Finally of course I tried just debugging it based on symptoms and I find that the examination callout does nothing as far as I can tell, while the proxy callout just eats all traffic from the targeted application, with one caveat. That caveat is that when I target firefox or chrome with the proxy callout and try to launch that app it fails to launch a UI and the partially started process cannot be killed from task manager.
I assumed that behavior may be the result of the sample failing to check FwpsQueryPacketInjectionState but modifying as best I can figure out that should be used does not seem to change the behavior.
So in short I am stuck and need direction please?
I started experimenting with the WFPSampler and also discovered that tracing didn't work. Oddly enough, there was a lot of additional code to make WPP tracing work on multiple OS's, but the sample doesn’t capture any trace events. So I debugged it and found the call to the nt!WmiTraceMessage always passed 2 as the TRACE_HANDLE. This seemed bad. Safe it to say, this wasn't exactly straight forward to unscramble WPP macros and then finally find the origin in the .vcxprog file. The definition of WPP_COMPID_LEVEL_LOGGER(COMPID,LEVEL)=2 is incorrect. While I was there, I also converted the DbgPrintEx Levels to match Tace_Level by adding 2 so Error/Warning match the model.
Complete the following steps for SysLib/WFPSampler and Sys/WFPSamplerCalloutDriver
Open the Project
Right Click on WFPSamplerCalloutDriver
Unload Project
Right Cick on WFPSamplerCalloutDriver
Edit WFPSamplerCalloutDriver.vcxproj
Type: WppPreprpocessorDefinitions
Change this definition in both places in the file:
To:
<WppTraceFunction>DbgPrintEx(COMPID,LEVEL,MSG,...)</WppTraceFunction
<WppPreprocessorDefinitions>WPP_COMPID_LEVEL_LOGGER(COMPID,lvl)=(WPP_CONTROL(0).Logger),;WPP_COMPID_LEVEL_ENABLED(COMPID,lvl)=(WPP_CONTROL(0).Level >= lvl+2)</WppPreprocessorDefinitions>
Save the File
Right Cick on WFPSamplerCalloutDriver
Click Reload Project
Rebuild
I believe if you adopt this model of using DbgPrintEx as your trace function, you can switch to WPP without editing the entire project. However, I still think it's better to just convert in your program.
Cheers,
John
Not exactly an answer to this question, I never got a debugger working, but an explanation of why the wfpsampler sample didn't work for proxying by ALE layers. The end goal of this project.
In the Proxy callout there were these lines (i think before my refactor they were in the PerformProxySocketRedirection method):
if (ipProtocol == IPPROTO_TCP)
pBindRequest1->portReservationToken = (pRedirectData)->pProxyData->tcpPortReservationToken;
else if (ipProtocol == IPPROTO_UDP)
pBindRequest1->portReservationToken = (pRedirectData)->pProxyData->udpPortReservationToken;
udpPortReservationToken or tcpPortReservationToken should have been initialized by a call to CreatePersistentUdpPortReservation or CreatePersistentTcpPortReservation and this is handled in a helper function named HlprWinSockCreatePortReservation, but said helper function is never called, from anywhere in the entire solution. Those fields are never set, anywhere, ever.
Additionaly, that value only needed set if the local port was being changed, but that block of code executed regardless.
remove these lines, and so long as you don't change the local port, the ale redirect sample works as you would expect including changing the outgoing interface if you change the local IP. If you do need to change the local port, you will have to use HlprWinSockCreatePortReservation to initialize the appropriate port reservation token variable in the PROXY_DATA. This must be done in user mode, so you need to do it in the service when setting up the filter.
Disclaimer: I think that is the only important change, but if you are here for a solution to this problem know that while tracking this down without a working debugger i changed a ridiculously large amount of code so there may have been more to it I forgot over the course of the last few weeks. But I did go back and verify that that block was always called and those fields were never set in the original.
I want to make a windows application whose GUI will be streamed to another device (allowing remote control). The point is that I'm not willing to rely on creating Windows Sessions to isolate the GUI I/O's (1)
To achieve this, I started observing some existing solutions that are able to enable remote access using this premise to see if I could get a clue about where to start.
One of these solution is Winflector (BTW: it is free up to 2 connections).
I got interested in this solution because it seems (I'm guessing) it detects only the repainted regions. What I took from my observations are that:
While the streamed application is "invisibly" running locally in the same session I'm logged in (it shows the application process in my task manager), the application window seems not to be created anyhow - at least Inspect can't get any window information/handle of the application process - It looks like sort of a "GUI StdOut Redirection".
Apparently, no additional Desktop is created;
Also apparently, no Mirror Driver is installed;
Using Process Explorer, I found out Winflector adds some thread's to the original application process. I suspect it is about the GUI redirection (by the thread's names);
The application is started by the Winflector server - that is, it has control about the CreateProcess arguments.
What is the most likely technique to be used in this case?
Windows Hook?
Windows Messages interception?
Special Display Driver?
Sort of Memory Device Context?
Where should I start researching to get a similar approach? Any open source project regarding this subject would also be very welcomed.
PS: By my programming experience, this is sort of a whole "new world" - sorry if my questions are redundant/obvious/non-sense.
(1) That is, this application could be spawned, streamed and interact
with the remote client using the same session which a local user is
already logged in, without conflicting the IO (like a regular VNC
would do, for example). PS: At this moment, I'm focusing only at the
output.
Just curious here: is it possible to invoke a Windows Blue Screen of Death using .net managed code under Windows XP/Vista? And if it is possible, what could the example code be?
Just for the record, this is not for any malicious purpose, I am just wondering what kind of code it would take to actually kill the operating system as specified.
The keyboard thing is probably a good option, but if you need to do it by code, continue reading...
You don't really need anything to barf, per se, all you need to do is find the KeBugCheck(Ex) function and invoke that.
http://msdn.microsoft.com/en-us/library/ms801640.aspx
http://msdn.microsoft.com/en-us/library/ms801645.aspx
For manually initiated crashes, you want to used 0xE2 (MANUALLY_INITIATED_CRASH) or 0xDEADDEAD (MANUALLY_INITIATED_CRASH1) as the bug check code. They are reserved explicitly for that use.
However, finding the function may prove to be a bit tricky. The Windows DDK may help (check Ntddk.h) - I don't have it available at the moment, and I can't seem to find decisive info right now - I think it's in ntoskrnl.exe or ntkrnlpa.exe, but I'm not sure, and don't currently have the tools to verify it.
You might find it easier to just write a simple C++ app or something that calls the function, and then just running that.
Mind you, I'm assuming that Windows doesn't block you from accessing the function from user-space (.NET might have some special provisions). I have not tested it myself.
I do not know if it really works and I am sure you need Admin rights, but you could set the CrashOnCtrlScroll Registry Key and then use a SendKeys to send CTRL+Scroll Lock+Scroll Lock.
But I believe that this HAS to come from the Keyboard Driver, so I guess a simple SendKeys is not good enough and you would either need to somehow hook into the Keyboard Driver (sounds really messy) or check of that CrashDump has an API that can be called with P/Invoke.
http://support.microsoft.com/kb/244139
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\i8042prt\Parameters
Name: CrashOnCtrlScroll
Data Type: REG_DWORD
Value: 1
Restart
I would have to say no. You'd have to p/invoke and interact with a driver or other code that lives in kernel space. .NET code lives far removed from this area, although there has been some talk about managed drivers in future versions of Windows. Just wait a few more years and you can crash away just like our unmanaged friends.
As far as I know a real BSOD requires failure in kernel mode code. Vista still has BSOD's but they're less frequent because the new driver model has less drivers in kernel mode. Any user-mode failures will just result in your application being killed.
You can't run managed code in kernel mode. So if you want to BSOD you need to use PInvoke. But even this is quite difficult. You need to do some really fancy PInvokes to get something in kernel mode to barf.
But among the thousands of SO users there is probably someone who has done this :-)
You could use OSR Online's tool that triggers a kernel crash. I've never tried it myself but I imagine you could just run it via the standard .net Process class:
http://www.osronline.com/article.cfm?article=153
I once managed to generate a BSOD on Windows XP using System.Net.Sockets in .NET 1.1 irresponsibly. I could repeat it fairly regularly, but unfortunately that was a couple of years ago and I don't remember exactly how I triggered it, or have the source code around anymore.
Try live videoinput using directshow in directx8 or directx9, most of the calls go to kernel mode video drivers. I succeded in lots of blue screens when running a callback procedure from live videocaptureing source, particulary if your callback takes a long time, can halt the entire Kernel driver.
It's possible for managed code to cause a bugcheck when it has access to faulty kernel drivers. However, it would be the kernel driver that directly causes the BSOD (for example, uffe's DirectShow BSODs, Terence Lewis's socket BSODs, or BSODs seen when using BitTorrent with certain network adapters).
Direct user-mode access to privileged low-level resources may cause a bugcheck (for example, scribbling on Device\PhysicalMemory, if it doesn't corrupt your hard disk first; Vista doesn't allow user-mode access to physical memory).
If you just want a dump file, Mendelt's suggestion of using WinDbg is a much better idea than exploiting a bug in a kernel driver. Unfortunately, the .dump command is not supported for local kernel debugging, so you would need a second PC connected over serial or 1394, or a VM connected over a virtual serial port. LiveKd may be a single-PC option, if you don't need the state of the memory dump to be completely self-consistent.
This one doesn't need any kernel-mode drivers, just a SeDebugPrivilege. You can set your process critical by NtSetInformationProcess, or RtlSetProcessIsCritical and just kill your process. You will see same bugcheck code as you kill csrss.exe, because you set same "critical" flag on your process.
Unfortunately, I know how to do this as a .NET service on our server was causing a blue screen. (Note: Windows Server 2008 R2, not XP/Vista).
I could hardly believe a .NET program was the culprit, but it was. Furthermore, I've just replicated the BSOD in a virtual machine.
The offending code, causes a 0x00000f4:
string name = string.Empty; // This is the cause of the problem, should check for IsNullOrWhiteSpace
foreach (Process process in Process.GetProcesses().Where(p => p.ProcessName.StartsWith(name, StringComparison.OrdinalIgnoreCase)))
{
Check.Logging.Write("FindAndKillProcess THIS SHOULD BLUE SCREEN " + process.ProcessName);
process.Kill();
r = true;
}
If anyone's wondering why I'd want to replicate the blue screen, it's nothing malicious. I've modified our logging class to take an argument telling it to write direct to disk as the actions prior to the BSOD weren't appearing in the log despite .Flush() being called. I replicated the server crash to test the logging change. The VM duly crashed but the logging worked.
EDIT: Killing csrss.exe appears to be what causes the blue screen. As per comments, this is likely happening in kernel code.
I found that if you run taskkill /F /IM svchost.exe as an Administrator, it tries to kill just about every service host at once.
What is your favorite technique for launching a windbg user-mode remote debugging session?
Why is do you prefer this technique over other techniques? (pros/cons)
There are at least four different ways to do user-mode remote debug using windbg, as documented in the "Remote Debugging" section of the debugging tools for windows help file.
run app on target then attach to it from the host windbg
have the host windbg use remote.exe to launch the app on the target
have the "smart client" host windbg launch the app on the target via a process server that is running on the target
run a windbg instance on the target machine using the option "-server" to automatically start a server, then connect to the server from a 2nd machine.
Option 1 is my favourite because it is the simplest. I get to launch the app in the normal way without worry about getting WinDbg to set the right working directory, pass any command line arguments, etc.
Fortunately I've not run into any cases where this hasn't worked!
There is no "the best" solution. Each of the possibilities has advantages and disadvantages and it's good to understand all of them. It depends on several factors like:
where are the symbols located
which PC has access to the Internet to download the OS symbols
what amount of data may you copy to the server (clients often accept better if it's just a single Exe)
what's the bandwidth between client and server
do you need other commands that just CDB/WinDbg, e.g. access to CMD, then consider remote.exe
who's available on the server side, a debugging expert whom you can easily tell a lot of cryptic commands or a normal user who barely knows how to start a command prompt
are both sides in a private network, so you need a "man in the middle" server to be able to access each other (or port forwarding as an alternative, which IT guys don't want and it may take days to get it set up)
From those 4 options, don't forget that clients often want to see exactly what you do, so they require an RDP session, Teamviewer or similar. That's something they understand.
I tend to use option 4 (-server) because it is the only one that doesn't "pop" when you break into the kernel debugger long enough for the TCP connection to timeout. But this is more complex and not fully satisfying. So I'm looking for "best practices".