I am looking for recommendations in each of the categories
free
charged
for easy to use debuggers fit for Portable Executable (native/non-.NET) malware analysis.
In the free category, Olly is the King. If you're using Win XP or older, SoftICE is fantastic as well, though finding a copy of it can be difficult these days.
In the paid category it is IDA Pro, hands down. Chris Eagle wrote an excellent book that shows you how to do advanced reverse engineering with IDA.
Also to clarify Marco, UPX can only be used to unpack exectuables that were packed with UPX in the first place. Many malware binaries exceeding modest sophistication use custom packing techniques, which cannot be reversed using UPX.
There is a windows free debuger (msdn.microsoft.com/en-us/windows/hardware/gg463009)
But you can unpack .exe files with upx (upx.sourceforge.net) -> opensource
And use a free debuger widely used (http://www.ollydbg.de/) --> my recommendation
One charged tool to work with Portable Executable is PE Explorer (www.heaventools.com/overview.htm)
Another great tool is IDA (https://www.hex-rays.com/products/ida/support/tutorials/unpack_pe/manual.shtml) --> my recommendation
If you cant get away with the GUI experience with OllyDbg, you can use the X64Dbg for 32 and 64 bit. There are also useful plugins. I usually use the IDAPro for static analysis.
As a developer, I am interested in the question of what technology/framework is best for building applications. The ultimate answer is to see what the best (in my opinion) applications are using. Yet, none of them reveal this in their About page. It's a travesty. Congress should pass a law requiring it. (No, seriously, it'd help the economy.)
A brief Google search didn't turn up a utility to do this. Does anyone have any tips?
Things I want to know: the platform (Native, .NET, Java, AIR), the GUI toolkit (WPF, Silverlight, Swing, JavaFX, various native ones), fundamental libraries (DI containers, persistence, etc.), etc., etc.
E.g., I am guessing Rosetta Stone is using Adobe AIR. I just started using Ebay's Turbo Lister today and am both impressed by its sophistication and baffled by its unusual style. What does it use?
Before continuing, I'd like to call out your "Congress should pass a law requiring it" as insane :)
Here are some tips on how you can find out such information on Windows:
Process Explorer and Process Hacker can tell you a ton about a running process.
Dependency Walker (or command line tools such as VC's depends.exe GCC's nm, etc.) can tell you about a executable's dependencies which are very revealing.
NTCore's Explorer Suite can show you a lot about a ton about a executable including it's resources such as it's manifest information.
Using the above and similar tools you can find out what technologies, libraries, etc. an application is using pretty quickly.
I have the following directive in my sources:
#import <scrrun.dll>
Can I rely on this DLL being pre-installed with Windows?
How relying on this limits Windows versions, which can be used?
You are asking us to predict what a very large company like Microsoft is going to do in the, what, next 10 years. Impossible to answer accurately of course.
Nevertheless, the scripting runtime is available in the latest version of Windows, got ported to 64-bits, has no obscure or desperately ancient or deprecated dependencies. It is used in many programs. Microsoft has a very strong commitment to app-compat.
Odds are very good that you are not going to have to worry about it for a long time.
How to learn advanced debugging techniques?
I am a C/Unix programmer, and as such rely on a good C debugger. I know Sun^WOracle mdb on Solaris, GNU gdb on Linux. I feel comfortable setting breakpoints, examining memory structures and such.
Yet, I know that those tools are way more powerful than that. Macros, custom walkers and I don't know what else. The learning curve on the other hand is very steep, as those tools sometimes seem to be arcane magic.
Any good texts? Practice problems? Other tips?
The GDB Pocket Reference is worth it's weight in diamonds, rubies, or something way lighter than gold. I use it all the time.
The online docs are also useful, but I actually like having a book on my desk.
Depends on the platform http://advancedwindowsdebugging.com/ is great but I assume from the gdb you aren't on windows
I have a desktop program I downloaded and installed. It runs from an .exe file.
Is there some way from the .exe file to tell what programming language was used to write the program?
Are there any tools are available to help with this?
What languages can be determined and which ones cannot?
Okay here are two of the sort of things I'm looking for:
Tips to Determine Whether an App is Written in Delphi or Not
This "IsDelphi" program by Bruce McGee will find all applications built with Delphi, Delphi for .Net or C++ Builder that are on your hard drive.
I use WinDowse (a small freeware utility written in Delphi) to spy the windows of the program.. for example if you look at the "Class" TabSheet you can discover the "Class" Name of the control..
For example:
TFormXX, TEditYY, TPanelZZZ for delphi apps
WindowsForms10.XXXX.yyy, for .NET apps
wxWindowsXXX for wxWindows apps
AfxWndXX for MFC/VC++ apps (I think)
I think this is the fastest way (although not the most accurate) to find information about apps..
I understand your curiosity.
You can identify Delphi and C++ Builder apps and their SKU by looking for a couple of specific resources that the linker adds. Specifically RC Data\DVCLAL and RC DATA\PACKAGEINFO. The XN Resource Editor makes this a lot easier, but it might choke on compressed EXEs.
EXE compressors complicate things a little. They can hide or scramble the contents of the resources. Programs compressed with UPX are easy to identify with a HEX editor because the first 2 sections in the PE header are named UPX0 and UPX1. You can use the app to decompress these.
Applications compiled with .Net aren't difficult to detect. Recent versions of Delphi even include an IsAssembly function, or you could do a little spelunking in the PE header. Check out the IsManaged function in IsDelphi.
Telling which .Net language was used is trickier. By default, VB.Net includes a reference to Microsoft.VisualBasic, and VCL.Net apps included Borland specific references. However, VCL.Net is defunct in favour of Delphi Prism, and you can add a reference to the VB assembly to any managed language.
I haven't looked at some of the apps that use signatures to identify the the compiler, so I don't know how well they work.
I hope this helps.
First, look to see what run time libraries it loads. A C program won't normally load Visual Basic's library.
Also, examine the executable for telltale strings. In most executables, this is near the end. If the program uses string constants, there might be a clue in how they are stored.
A good disassembler, plus of course an excellent understanding of the underlying CPU architecture, can often help you identify the runtime libraries that are in play. Unless the exe has been carefully "stripped" of symbols and/or otherwise masked, the names of symbols seen in runtime libraries will often provide you with programming-language hints, because different languages' standards specify different names, and vendors of compilers and accompanying runtime libraries usually respect those standards pretty closely.
Of course, you won't get there without knowledge of the various possible languages and their library standards -- and if the code's author was intent to mask the information, that's not too hard for them to do, either.
If you have available a large set of samples from known compilers, I should think this would be an excellent application for machine learning. I believe so-called "supervised learning" is relevant here. Unfortunately I know next to nothing about the topic—only that I have heard some impressive results presented at conferences.
You might dig through the proceedings of the Working Conference on Reverse Engineering to see if anyone else is interested in this problem.
Assuming this is an application for Windows...
Does Reflector recognize it as a .NET assembly? Then it's MSIL, 99% either VB or C#, but you'll likely never know which, nor does it matter.
Does it need an intrepreter (like Java?)? Then it's Java (or whatever the interpreter is.)
Check what runtime DLLs it requires.
Does it require the VB runtime dlls? Congratulations, VB from VisualStudio 6.0 or earlier.
Does it require the Delphi dlls? Congratulations, Delphi.
Did you make it this far? C/C++. Assume C++ unless it requires msys or cygwin dlls, in which case C has maybe a 25% chance.
Congratulations, this should come out correct for the vast majority of Windows software. This probably doesn't actually help you though, as a lot of the same things can be done in all of these languages.
IDA Pro Free (http://www.hex-rays.com/idapro/idadownfreeware.htm) may be helpful. Even if you don't understand assembly language, if you load the EXE into IDA Pro then its initial progress output might (if there are any telltale signs) include its best guess as to which compiler was used.
Start with various options to dumpbin. The symbol names, if not carefully erased, will give you all kinds of hints as to whether it is C, C++, CLR, or something else.
Other tools use signatures to identify the compiler used to create the executable, like PEiD, CFF Explorer and others.
They normally scan the entry point of the executable vs the signature.
Signature Explorer from CFF Explorer can give you an understanding of how one signature is constructed.
It looks like the VC++ linker from V6 up adds a signature to the PE header which youcan parse.
i suggest PEiD (freeware, closed source). Has all of Delphi for Win32 signatures, also can tell you which was packer used (if any).