What is the best way to unpack PE files? I've seen some tools from 7 years ago, like Quick Unpack. Is there anything more recent? Or is it better to run different tools for different packers since individual unpackers are likely more up-to-date?
There is no one size fits all solution.
The latest "all in one" unpackers are Quick Unpack and GUnpacker but they are both getting rather old.
There is Pe-sieve & mal_unpack from hasherezade, mal_unpack can work in an automated fashion. Mal_unpack is basically just an automated version of Pe-sieve. Pe-bear is another tool from hasherezade which helps you re-align sections after doing a dump, which requires that you know how to unpack manually but it make the process much easier.
There is a new online tool for unpacking malware called unpac.me from the OpenAnalysis team which can unpack just about anything.
Successful malware distributors are not using public crypters as often, heuristics for detecting public packers is too easy, which is why we have seen a reduction in AIO unpacker tools. In addition, Themida and VMProtect are the standard now and as they continue to add more features, they're becoming more difficult to unpack everyday. With the new virtualization features, automated unpacking is becoming almost impossible.
Even though Quick Unpack is old, I would not underestimate its power in present days. As long as you find the best setup, this tool will produce launchable dumps of Exes/Dlls packed by dozens of known packers + even unknown ones!
Make sure you are running an old OS (WinXP - 7) really or virtually, and don't forget that Quick Unpack is a dynamic unpacker (i.e. can hardly be used for malware analysis). Old does not mean bad :)
This concerns packers in their classical definition.
If you are looking for some "generic unpacker" for modern protectors (such as VMProtect, Themida, Enigma, Obsidium, etc.), then I do not think they will ever be made. There are some specific tools (both private and public) which can help you to automate "unpacking" partially, but the majority of work still needs to be done by hands to remove these kinds of protectors. But again, it depends on what you want to see in the end (analysable code, de-obfuscated dump, fully launchable reconstruct, ...).
Related
As far as I'm aware, Windows hasn't been decompiled by anyone yet. Obviously it's complicated, but surely it should've been done by now to some degree?
My thinking behind this is that if the end-user has access to the software, and the computer is able to run it, then even an obfuscated version of it must be obtainable?
I'm obviously missing something, I'm just not sure what.
There's nothing preventing Windows from being decompiled (apart from EULA and similar legal bindings, of course). As you noted, the code must run on the CPU at some time, and the CPU must read the code from memory, and you can read from memory too. Some parts of the system can be a bit trickier, since to run the OS you need to give the OS some exclusive priviledges (that's how most modern protected OSes work), but it's nothing that can't be worked around. In any case, there's not a lot of effort to prevent Windows decompilation - that would have barely any benefit, while making debugging, error reporting and similar harder. Microsoft even goes so far to provide a special debug version of Windows that's even more tailored for software development.
The main point is that there's little reason to decompile Windows. What practical use would such a massive effort have? And if you're a corporation that needs access to Windows source codes (for example, when developing embedded solutions), you can get them. Just because Windows isn't open source doesn't mean the sources aren't available.
If you're not someone who needs their own version of Windows (common in the times of Windows CE), there's even less of a reason to decompile Windows. You need to stick to the defined public APIs anyway - that's a good practice regardless of whether the software is open source or not. APIs are contracts - implementation details you'd get through decompiling aren't. They might very well change with the next security hotfix or such. This is especially important given how serious Windows is about compatibility - it's quite rare for an update (or even a new major release) to break compatibility with old software.
So, if you want to decompile Windows, there's nothing technical that's really preventing you from doing so. But you're looking at tens of millions of lines of source code that was compiled by very smart compilers, with bits of handwritten optimised assembly thrown around, tons of compatibility workarounds that might as well be outright obfuscation (remember, you don't get the comments - just the actually compiled code). Are you willing to spend a few hundred thousand hours to satisfy your idle curiosity? :P
I have a WIX 3.0 installer that is building 88 slightly different builds (cross product of 32 and 64-bit, 11 locales, four editions (Beta, Retail, Evaluation, Different Evaluation).
Each build has slightly different contents in addition to localized UI, so I can't just build one configuration with multiple locales.
The resulting MSI is about 120MB. I'm already using the CabCache.
The installer takes about 3-5 minutes per release to build, resulting in a pretty lengthy overall build time.
The install appears to be heavily disk bound during linking (light.exe).
Clearly making the disks faster could help. Does anybody have advice on how to set up a machine that could crank through these installers faster? (or advice on reconfiguring my WIX project to build more efficiently?)
Get an SSD. Like one of those with internal RAID architecture from e.g. OCZ. SSD is every developer's upgrade of the decade. Plus more RAM if swapping is an issue.
If you have common parts (that are not localized) you can create a merge module with the common parts and then just add the differencing stuff to each build.
I am not sure if you have any say or communication with the developers of the application that you are installing, but if you have to create that many MSI's mainly because of languages, have you considered just offering one Language MSI that delivers all the language specific files to a resources directory and then the user can choose which language they would like to use (but only install this if they need something other than the default language). Also it might be worth looking into having the product made in such a way that the user can pick from within which language is best, then having all the languages installed from the start.
As for your question about speeding up the build, that is a tricky one. Using Merge Modules I would rule out right away, as I don't see any actual gain coming out of that. Of course updating the hardware (as you said) will give some results, but again, I am not sure how much of a jump you would be making so it is hard to tell what kind of gain that would give. I think it might be best to go over your WXS with a fine tooth comb and see what is really going on in there. You can sometimes find things that are left over from the developement of the package, or from a previous tool that are really slowing you down. One example would be that my company recently switched to WiX from a more automated setup creation utility (leaving the name out on purpose cause I am listing the problems with it :P ) and it automatically created every folder under Windows that might possibly be needed in the running of a windows application, as well as the common files folder, the current user profile, and many many more. I think I ended up erasing in all over 100 empty directories that this old technology was nice enough to add for me. That is just one example of optimization that was done. It is amazing what can be found when you take the time to REALLY review what is going on under the hood.
In your wixproj setup file add this just before the end of file in <PropertyGroup> tag
<IncrementalGet>true</IncrementalGet>
This will tell WIX to compile only those files which are changed after the previous build.
What forces are at work keeping crufty old Make (with or without makefile generator tools) prominent as a build tool? Is it deficiencies in alternatives that keep them from being widely adopted, or insufficient publicity, or does something about Make keep it in place?
Despite Make's many weaknesses and difficulties dealing with large projects
(e.g. see http://freshmeat.net/articles/what-is-wrong-with-make) it appears to still be more widely used than newer, improved alternatives such as Scons, Jam, Rake, Cook, and others.
Are there measurable benefits to the alternatives, or are the "market shares" due mostly to opinion and experience of team leaders?
Ubiquity: I like Make because I can trust it will be available where I need it i.e. installed or easily installable on the target machine.
It's widely available, well documented, concise and powerful + best of all - no XML!.
I've been using it for close to 15 years and still haven't found something better. The coolest thing I've done with it is to have a master makefile generate makefiles for sub projects on-the-fly.
Regarding your question, which forces are keeping make alive ...its the force of habit.
simplicity - easy to do simple things
ubiquity - some version is on your system
speed - fast enough for most things
expressive - pretty good match to the job
nonobvious complexity - mainly large projects expose problems
It's availability on a large number of platforms probably helps. If writing a product for multiple platforms, knowing it will always be there is a plus point. It's a pain to have to port your build tool to a new platform before you can build your own project.
Hm, I never used make as a build system.
Other than that, it's a unique dataflow-programming language, where you can describe set of nodes, each serving specific purpose, describe their behavior, and let the manager handle and control the data flow between them.
We used scons on a relatively large project to replace make, and found that it was a reasonably flexible system, that allowed us to do some very necessary (but very unfortunate) hacking to get things to build the way we needed them to. Also, make is -strange-.
i think what would have to occur to see a big shift to another tool, is 1st the tool would have to be created.... that is significantly better. and to affect change, either one of the linux distros or one of the major packages would have to switch to it and probably keep the old one arround for compatibility. i would envision that the new build tool would be capable of generating the legacy makefiles. linux already demonstrated how well he can solve the source code control system with git. i have a pretty good hunch he could come up with something pretty cool and tie in with git.
At work I use ClearCase and SourceSafe, but have found some time to do some time to code for myself enroute thanks to a disposable laptop.
However, I wish I had a lightweight VCS on my system using which I would be able to make changes to my code during the commute and then push/grab them from my Linux systems.
I use git on my home system, but I can't really get it working on Windows. I don't want all that cygwin hack.
If it does not run natively on Windows, it just won't do.
What have you guys tried on your Windows system? Something that YOU use.
The big player at the moment seems to be Mercurial?
What would be best for a one (or maybe two) man team?
I just need to maintain :
Versioned copies of source code.
Checking in and out should be as less obtrusive as possible.
I am looking forward to a multiple Undo kind of feature (like that in an EMacs buffer) but persistent.
I really like the way git keeps track of lines moving between files in a source code set
I should be able to move part(s)/sub tree(s) of the source tree (each sub tree implies a module/plugin to my the main software I am building) to an archival system either completly or partially and restore them back from the archive as and when required and the system should track any changes to this tree as well.
I actually want to experiment with my code as much as possible without me manually keeping track of what I modified and what I need to undo once I try out some idea, so that I am back to where I want to continue from.
Notes : A similar topic came up a year ago : DVCS Choices - What's good for Windows?
I hope things have changed, and I really want people to share their own, real life experiences. Not something they recommend without using it or they think will work.
Bazaar and Mercurial both work very well on Windows. I posted in the question you linked, and since then, both have improved their Windows support even more. Using them is easy and flawless, and they even have GUIs if you swing that way.
I for one have switched from bazaar to git, and I've been pleased.
If you've a Clearcase background, why don't you take a look at Plastic SCM? Check this link, it will show you how it works on a distributed setup (and of course all the basic operations) http://codicesoftware.blogspot.com/2010/03/distributed-development-for-windows.html.
You won't miss any of the "good" clearcase features but all the shortcomings are simply gone (faster, installs on 45seconds, no cumbersome setup to use on a mixed Win/Linux scenario, built-in ACLs, excellent branching and merging, much better common ancestor algorithm, visualizations, better GUI, and you still have "selectors" in case you miss config_specs, but not being mandatory)
I have a build process for a large enterprise system comprising several dozen separate EXEs and DLLs. These use multiple languages, C, C++, Fortran, Python, Awk and a couple more. The build scripts are 4DOS batch processes which evolved over 4 decades. They are large and unwieldy and need constant care and feeding.
I must keep the Visual Studio solution and project files as the basic compile/link entities. What's the best tool for wrapping these disparate languages all together. 4DOS is very old and cumbersome.
EDIT:
Thanks gang. I think I'll try SCONS first because it's Python. We have plenty of people well versed in Python to be able to update and maintain it. I'm 61 now and it's not going to be me supporting this in the long term. I don't like anything requiring JAVA or XML because those are not languages already in our product mix and we have enough in play.
Those blog posts were good. He concluded that SCONS was best but simply too slow for his purposes. I'm not looking for speed in nightly builds. It's got until 7 AM. I want readability and maintainability.
For example Apache Ant
Ant is a good choice. I would also be tempted to try Rake.
I think the best choice is NAnt and MSBulid
Scons perhaps?
These may be a little a outdated - the build systems might have evolved quite a bit, but this should at least give you a better idea on what to expect:
The Quest for the Perfect Build System
The Quest for the Perfect Build System (Part 2)
Personally, I never needed anything special, that couldn't be achieved with VS project/solution files, makefile's and BATCH'es, so I won't be recommending anything in particular.
Scons definitely. It plays with fortran and C naturally, and it is python based so it shouldn't have any problem with that one either (never used it for py though, so can't tell from experience). Also, much more readable than the majority of them out there.
I know Maven isn't known to focus on anything but Java, but perhaps it might at least be worth mentioning. There have been some work towards enabling at least C/C++. When comparing to Ant, it's pluggable in a similar fashion, but it's declarative rather than imperative, with standardized dependency management, and a build result repository which may even be distributed.
ANT + terp for the C++ portions. terp plays nicely with VisualStudio as well as with many other C++ compilers on many platforms. ANT requires Java though, if only as the hosting technology. I don't know whether that is a no-no with your requirements or whether you just don't want to start writing Java code.