Currently I am in a stage that can learn Windows kernel programming, but I am wondering after that which types of jobs I can do?
For Linux kernel, seems till now only a bunch of driver works related jobs there.
There are quite a few jobs for people who can write Windows drivers, but most of them are not at traditional software companies -- rather, they're at companies that produce hardware that needs to work with Windows.
There is also the possibility of writing Native API code that bypasses Win32, and talks to the Windows NT kernel directly. For the most part, MS seems to officially frown on people doing this -- but when they do it well (e.g. the former ntinernals.com) MS buys them up and apparently puts them to work for MS (assuming somebody else doesn't first, of course).
The bottom line is that it is a useful skill set, but not particularly likely to be a life-changing event, or anything on that order. Also keep in mind that writing kernel code is writing kernel code -- doing it well goes way beyond the usual definition of "meticulous". A good kernel code is a guy who routinely has to sit back down and physically relax, because he's so anal retentive that the first time he stood up, the chair stuck to his butt!
You could write low level code that interacted with it (optimization purposes I assume).
Also drivers like the Linux world as well.
Related
I'm a first year grad student trying to write an Operating System from scratch as a side project. I've read the Linux Programming Interface, Modern Operating Systems 4th edition, a bunch of articles on OSdev wiki's, and anything I can find on Google, but I'm having a tough time finding what I should be doing next after writing a simple bootloader, and a kernel that can take user input and display it back onto the screen.
I have a feeling that I need to create some drivers that interact with the file system and memory, but I'm not entirely sure. I'm trying to work my way up with just physical memory and one process running "kernel" for right now. I'll worry about virtual memory (pagging) and multi-processes later. If anyone can give me some kind of direction or better understanding of what happens when the kernel is finally put into memory, that would be great.
Thanks.
I would like to point a resource that will be of great help for you to understand this stuff in real details. One of the great and evolving resource that is being maintained on git.
https://github.com/0xAX/linux-insides/tree/master/Booting
As far as I'm aware, Windows hasn't been decompiled by anyone yet. Obviously it's complicated, but surely it should've been done by now to some degree?
My thinking behind this is that if the end-user has access to the software, and the computer is able to run it, then even an obfuscated version of it must be obtainable?
I'm obviously missing something, I'm just not sure what.
There's nothing preventing Windows from being decompiled (apart from EULA and similar legal bindings, of course). As you noted, the code must run on the CPU at some time, and the CPU must read the code from memory, and you can read from memory too. Some parts of the system can be a bit trickier, since to run the OS you need to give the OS some exclusive priviledges (that's how most modern protected OSes work), but it's nothing that can't be worked around. In any case, there's not a lot of effort to prevent Windows decompilation - that would have barely any benefit, while making debugging, error reporting and similar harder. Microsoft even goes so far to provide a special debug version of Windows that's even more tailored for software development.
The main point is that there's little reason to decompile Windows. What practical use would such a massive effort have? And if you're a corporation that needs access to Windows source codes (for example, when developing embedded solutions), you can get them. Just because Windows isn't open source doesn't mean the sources aren't available.
If you're not someone who needs their own version of Windows (common in the times of Windows CE), there's even less of a reason to decompile Windows. You need to stick to the defined public APIs anyway - that's a good practice regardless of whether the software is open source or not. APIs are contracts - implementation details you'd get through decompiling aren't. They might very well change with the next security hotfix or such. This is especially important given how serious Windows is about compatibility - it's quite rare for an update (or even a new major release) to break compatibility with old software.
So, if you want to decompile Windows, there's nothing technical that's really preventing you from doing so. But you're looking at tens of millions of lines of source code that was compiled by very smart compilers, with bits of handwritten optimised assembly thrown around, tons of compatibility workarounds that might as well be outright obfuscation (remember, you don't get the comments - just the actually compiled code). Are you willing to spend a few hundred thousand hours to satisfy your idle curiosity? :P
I don't mean for this question to be a flame bait but I'll be using Microsoft and their win32 API as a example of a legacy API.
Now what I am wondering here is Microsoft is spending a lots of their money and energy in maintaining their legacy API, including all of the "glitches/bugs/workaround" that are needed to keep the API functioning the same. Now I'm aware that in Windows 7 they are providing a way for the customer to run their application in a "Windows XP" VM which would be one such way for them to start cleaning up their win32 API because they could then push all of the application into the "Windows XP" VM.
So now what I am wondering is, is it possible to virtualization a legacy API in such way that an customer/program can still access and use it, yet at the same time be able to take advantage of the newer version/API? Because as far as I understand it, if the application is ran in the "Windows XP" VM, it won't be able to access any of the newer API/feature of Windows 7.
The thing that puzzles me about this question when it comes up is that Windows has been doing this since NT came out in the mid nineties. This is how NT runs DOS and Win16 programs, and how it always has. The NTVDM virtualization layer runs 16-bit apps under Win32 with very little special support from the core OS. This is just one example - another is WINE, which as I understand it does a pretty reasonabl job of running windows apps on top of an API set which is very different from that of windows. So it is definitely possible.
The more pertinent question would be why Microsoft would consider it. In order for you to think it is necessary you have to think two things. 1) There is something better to replace the win32 API with and 2) Maintaining the Win32 API is a burden.
Both of these are questionable. In the case of kernel duties, such as accessing hardware and synchronizing and doing threads and processes and memory the Win32 API does a pretty good job, and is ultimately quite close to what the kernel really does. If you think there is a better API then that must mean there is also a better kernel. I personally don't think that NT needs replacing right now. For graphics and windowing, admitedly gdi32 is a bit long in the tooth. But Microsoft solved that problem by building WPF right alongside it. This then brings in the burden question. Well, sure there are two APIs to maintain, but if you virtualized GDI on top of WPF you'd still have to maintain both anyway so there is no benefit there. The advantage of running both in parallel is that GDI already exists and is already tested. All you have to do is to fix the occasional bug, whereas a new virtualization layer would have to be written and tested all over again, which takes time away from making WPF better.
In terms of maintaining back compat, that isn't as much of a burden as it sounds. It is mainly a test question - you have to test that the API behaviour doesn't change, but again - those tests have already been written, so it isn't really any extra work.
So, to answer a question with a question, why would they bother?
This is an interesting question, at least to me, here are some of my thoughts.
Your understanding is correct, an application running in the XP VM only has access to the Win32 APIs provided by XP in the VM. One of the many ways that I have seen Microsoft's approach to enhancing specific APIs is to create new functions with the enhanced/fixed functionality and name the new function by append Ex and even ExEx to the original name, for example
GetVersion
GetVersionEx
For functions that accept pointers to structures, the structures are 'versioned' by using the size of the structure to determine the functionality required, so older code would be passing a previous size of the structure while newer code would be passing in the newer larger strucure and the API functions accordingly.
I guess, the problem has become that it is no longer just differences in how an API works, but more integral to the functioning of the operating system and the internal structures which have changes significantly enough that arguably badly written code is effectively broken.
As to your actual question, I guess it would be quite tough. Even if one thought to let the OS adjust how it executes code based on a target OS version in the PE header of the executable, what would happen if a newer DLL was loaded into the process that targeted the latest OS, now how should the OS handle this when the code is executing? IMHO, I think this would be very challenging, one frought with pitfalls that would ultimately fail.
Of course that is just my raw thoughts on the topic so I might be 100% wrong and there is some simple approach that just did not come to mind.
I couldn't find a suitable title for this. I'm going to express my query with examples.
Consider following softwares:
Process explorer from sysinternals (an advanced task manager)
Resource Manager : resmon.exe (lists each and every fine detail about resource usage about each process).
For me these softwares seems like miracles. I wonder how these are even made. C'mon how a user process can know such fine details about other processes? Who tells this software, what processes are running and what all resources are utilized? Which dlls are used? etc..
Does windows operating system give these software that information? I mean though (obviously the most lower level api) WIN32API. Are there some functions,which on calling return these values
abstractly say:
GetAllRunningProcesses()
GetMemoryUsedByProcess(Process* proc)
etc..
Other similar applications are
network Packet Capture software. How does it get information about all those packets? It clearly sits just infront of the NIC card. How is it possible?
Anti-virus: It scans memory for viruses. Intercepts other processes. Acts like a sandbox for the user application space. How? How??
If its WIN32API. I swear, I'm going to master it.
I don't want to create a multi-threaded application. I want to get information about other multithreaded applications.
I don't want to create a program which communicates using sockets. I want to learn how to learn how to capture all communication packets.
I actually want to work at the lower level. But I don't know, what should I learn. Please guide me in proper direction.
This is really a pretty open-ended question. For things like a list of running processes, look up "PSAPI" or "Toolhelp32". For memory information about a particular process, you can use VirtualQuery.
Capturing network packets is normally done by installing a device driver. If you look, you should be able to find a fair amount about how to write device drivers, though don't expect to create wonders overnight, and do expect to crash your machine a few times in the process (device drivers run in kernel mode, so it's easy for a mistake to crash the machine hard).
I can't say as much with any certainty about anti-virus, because I've never tried to write one. My immediate guess would be that their primary technique is API hooking. There's probably more to it than that, but offhand I've never spent enough time looking at them to know what.
Mark Russinovich's classic, Windows Internals, is the go-to book if you want to get deep in this kind of stuff. I notice that the just-released 5th edition includes Vista. Here's a sample chapter to peek at.
If you like Process Explorer, this is the guy who wrote that, and there are lots of examples using it in the book.
Plus, at 1232 hardcover pages, you can use it to press your clothes.
The Free MS Windows replacement operating system ReactOS has just released a new version. They have a large and active development team.
Have you tried your software with it yet?
if so what is your recommendation?
Is it time to start investigating it as a serious Windows replacement?
Targeting ReactOS specifically is a bit too narrow IMO -- perhaps a better focus is to target compatibility with WINE. Because ReactOS shares so many of its usermode DLLs with WINE, targeting WINE should result in the app running just fine on ReactOS.
Of course, there will always be things that WINE can't emulate well (hence the need for ReactOS). In this way, it seems that if something runs in WINE, it will run in ReactOS, whereas the fact that something runs in ReactOS doesn't mean that it will necessarily run in WINE.
Targeting WINE is well documented, perhaps easier to test, and by definition, should make your app compatible with ReactOS as a matter of course. In this way, you're not only gathering the rather large user base of current WINE users, but you're future-proofing yourself for whenever anyone wants to use your app with ReactOS.
In their homepage, at the Tour you can see a partial list of office, tools and games that already run OK (or more or less) at ReactOS. If you subscribe to the newsletter, you'll receive info about much more - for instance, I was quite surprised when I read most SQL Server 2000 tools actually work on ReactOS!! Query Analyzer, OSQL and Books Online work fine, Enterprise Manager and Profiler are buggy and the DBMS won't work at all.
At a former workplace (an all MS shop) we investigated seriously into it as a way to reduce our expenditure in licenses whilst keeping our in-house developed apps. Since it couldn't run MSDE fine, we had to abandon the project - hope in the future this will be solved and my ex-coworkers can push it again.
These announcements might as well be also on their homepage - I couldn't find them after 5 mins. of searching, though. Probably the easiest way to know all these compatibility issues is to join the newsletter, or look for its archives.
I have been tracking this OS' progress for quite some time. I believe it has all the potential to really bring an OSS operating system to the masses for it breaks the "chicken and egg" problem: it has applications and drivers from the very beginning (since it aims to have full ABI compatibility with MS Windows).
Just wait for their first beta, I won't be surprised if they surpass Linux in popularity really soon after that...
Post Edit: Found it! Look at section Support Database, it's the web place to go look for whether a particular piece of hardware of some program works on ReactOS.
ReactOS has been under development for a long long time.
They were in some hot water earlier because some of their code appeared to be line by line dissasembly of some NT kernel code, I think they have replaced all of it.
I wouldn't bother with cross platform testing until they hit the same market penetration as Linux, which I would wager is never.
Until ReactOS doesn't randomly crash just sitting there within 5 minutes of booting, I won't worry about testing my code on it. Don't get me wrong, I like ReactOS, but it's just not stable enough for any meaningful testing yet!
No, I do not think it is time to start thinking of it as a Windows replacement.
As the site states, it's still in the Alpha stages. More importantly, whos Windows replacement? Yours? Your users? The former is one thing, the latter is categorically a no-go.
As an aside, I'm not really sure who this OS is targetting. It has to be people who rely on Windows software but don't want to pay, because people who simply don't want Windows can use MacOS / Linux, and the support (community or otherwise) for these choices is good.
Moreover, if you use Linux you already have some amounts of Windows software support via Wine.
Back to people who rely on Windows software but don't want to pay. If they are home users they can just simply pirate it, if they are large business users they already have support contracts and trained people etc. It's hard enough for large businesses to be OK to update to new versions of Windows, let alone an open source replacement.
So I suppose that leaves small businesses who don't want to obtain illegal copies of MS software, can't afford the OS licences and rely on software that only runs on Windows and has bad of non-existent Wine compatibility.
It is a useful replacement for Windows when it runs 'your' software without crashing. At the moment it is not a general purpose os as it is too unstable (being only alpha) but people have used ReactOS successfully in anger for specific tasks already. As a windows replacement it has multiple potential uses, sandbox systems, test and development systems, multiple virtual instances, embedded devices, even packaging/bundling legacy apps with their own compatible o/s. Driver and application compatibility, freed from Microsoft's policy of planned obsolescence and regular GUI renewal, what's not to like?