So I am trying to make a small programming language as a project. But the thing is I have no clue how to begin. I know how a compiler works and all but don't know how the code would be written for it. I searched everywhere for example for the code of any compiler but couldn't find anything. Where would I find this, or could someone explain to me the format and method I should use for creating the compiler? Also if you were wondering I am thinking of using c++ for writing the compiler.
I recommend taking a free online course in Compiler Construction. Guided material and thoughtfully designed coursework is the best way to ease into this sort of thing, in my opinion.
LLVM.org; open source modern C, C++, Objective-C and more
EDIT:
I found that doxygen can generate call graphs for classes, but I could not find any options or examples where the call(er) graph is generated for public/private members of the class(es) such as fields, methods, etc.? See the example that I provided below.
Is it possible to find links/relationships between 2 variables/objects in the code using some IDE tools and code editors, i.e. in Visual Studio, Sublime, etc.
e.g.
a=func(b,c);
w=func(a,c);
Here w and b are indirectly related to each other.
In convoluted code it is very difficult to manually find such relationships.
I understand that reflection and dynamic nature of some languages can limit such analysis.
You need to provide the language you are looking to use. If I take a guess and say C/C++ you can use CCTree and Cscope in general for this functionality. Most open source developers use Cscope extensively for this purpose.
Eclipse CDT also has call graph's. It is a bit of a pain to work outside of VisualStudio for this purpose I know. But cost is part of the reason to use open source instead.
Your best bet to cover all languages for the purpose of browsing is Exuberant Ctags. This works with a fair amount of editors and all the languages you listed. With that large a list of languages and use cases its probably worth your time to learn either vim or emacs and the integrations supported here.
For Python you can also take a look at pyscope with cscope. Another excellent alternative for Python is Rope. Rope supports finding definitions and usages as part of its standard set of tools.
Most developers do not need CCTree as browsing code bases with cscope is relatively straightforward. I have used exuberant ctags + emacs on a huge variety of language for years. It takes a touch of time to learn, but the upsides are
it's free, portable, and powerful. Another alternative to CCTree is codegraph for some of your target languages.
Found a list of tools and comparison:
https://github.com/OpenGrok/OpenGrok/wiki/Comparison-with-Similar-Tools
EDIT
possible in doxygen, but only for classes and their relationships
I found it, this is code map in VS Ultimate:
http://blogs.msdn.com/b/visualstudioalm/archive/2014/11/12/announcing-visual-studio-2015-preview-availability.aspx
I'm looking to write a tool that aims to convert debug symbols of one format to another format that's compatible for use under GDB. This seems like a tedious and potentially complex project so I'm not exactly sure how to tackling it.
Intially I'm aiming to convert the Turbo Debug Symbol table(TDS) emitted from borland compilers into something like stabs or dwarf format(seems like dwarf is prefer from my research). But ideally I want to design my tool to be easy enough to extend so it could convert other formats too later on. e.g. codeview4 or maybe even pdb.
My primary motivation for creating this are:
Interoperability. If I can convert a foreign debug format into a form gdb can work with then source-level debugging would be possible on binaries compiled from another compiler other than gcc. This means any frontend debugging interface that uses gdb as a backend will work as well.
No other tools exist. I did a google searching around for similar tools and the closest I've found is tds2dbg. But it doesn't quite do what I'm looking for.
What I have to work with at the moment:
I already have a debug hook API that can understand the TDS debug format. I can use that to help me get at the needed information from the source format I'm converting from.
For the scope of this project, I'm mainly interested in getting this to work under the win32 environment. Other platforms and tools I'm not really concerned about.
The target dwarf debug format I'm converting to. This one I'm really not familiar with at all. I have used gcc ported compilers like MinGW before and debugged them with gdb with the dwarf format. But I don't have any idea how this format is implemented on windows.
The last point is the one I'm concerned about. I'm reading through the dwarf spec documentation but I find I'm having trouble really understanding and comprehending how it works. There's so much detail in there but at the same time it doesn't have any details about how dwarf gets implemented on object files and image files on a platform that doesn't use ELF natively -- namely the PE-COFF format that windows uses. The documentation is also a very dry read, long sentences make it hard to understand and diagrams and illustrations are sparse. I came across an API called libDwarf that should take most of the parsing work out of interpreting dwarf. The problem is I'm still trying to get it to build and I don't know yet how it will work out.
I haven't written any code yet since I don't fully understand what it is I need to build. I have a feeling the biggest hurtle will be figuring out how to work with dwarf due to it's complexity. Googling for information on how dwarf works under windows hasn't turned up anything helpful either. Like for example, there's no information about the 'glue' code that's needed to contain dwarf within a PE executable image file. How are the dwarf sections exactly layed out? Are there any header information for each section? GDB clearly doesn't just take a 'raw' dwarf debug file and use it as is. So what kind of format does gdb expect the debug file to be in for it to be able to work with it?
My question is, how can I start on such a project? More importantly, where can I turn to for help when I inevitably get stuck on a problem?
Affinic Assembler for Windows
Affinic Assembler is an x86/x86-64 assembler for Windows that takes GAS-syntax assembly source with DWARF debug information and generates corresponding CodeView format sections in object file in order to make the linked program debuggable in Visual Studio. This program is good for Cygwin and MinGW users to port Linux code to Windows.
http://www.affinic.com/?page_id=48
You are asking several questions here :-)
I think you are heading in the right direction, using libdwarf.
BUT, have you taken a look at objcopy to see if this tool can do some of the work for you? It probably doesn't support borland, pdb or codeview4, but it might be worth looking into. (Another approach may be to extend objcopy to support the formats you are trying to convert between.)
I have used the dwarf-discuss mailing list sometimes when I have become stuck.
http://lists.dwarfstd.org/listinfo.cgi/dwarf-discuss-dwarfstd.org
As for the questions on dwarf, split them into separate questions and I will do my best to
answer them. :-)
I'm writing a compiler that converts source code (written in a small imperative programming language) to Intel AT&T 32-bit assembler.
I tend to spend a lot of time debugging, because of nasty offset-mistakes etc. in the generated code, and I would like to know if anyone knows of a tool to "walk through" the generated assembler code step-by-step, visualizing what's on the stack etc.
I use Ubuntu Linux as my development platform, and I'm comfortable with the terminal -- a GUI-program would be nice though. Does it exist? Or is there a good reason it doesn't (maybe the problem isn't so straightforward..?)
If you have good ideas for approaching debugging tasks in assembly code, I'll be glad to hear from you!
I like EDB (Evan's Debugger) on Linux. It has a nice, easy-to-use, QT4-based GUI. Its developer's goal is to make it similar to OllyDbg. And it's being actively maintained:
EDB on FreshMeat
I'm pretty sure it's installable through Synaptic on Ubuntu as well. Enjoy!
Is the end result of the compile process something that you can actually execute, and therefore examine in a debugger? If so the Data Display Debugger (ddd) might be useful.
My experience with debuggers such as Olly and EDB is quite sparse, so I wasn't able to solve my problem with those. I ended up
scattering calls around to a Debug function in the source code, nailing down bad register values
letting the compiler output HTML-formatted code with useful metadata for different iterations in the liveness analysis etc.
I have a desktop program I downloaded and installed. It runs from an .exe file.
Is there some way from the .exe file to tell what programming language was used to write the program?
Are there any tools are available to help with this?
What languages can be determined and which ones cannot?
Okay here are two of the sort of things I'm looking for:
Tips to Determine Whether an App is Written in Delphi or Not
This "IsDelphi" program by Bruce McGee will find all applications built with Delphi, Delphi for .Net or C++ Builder that are on your hard drive.
I use WinDowse (a small freeware utility written in Delphi) to spy the windows of the program.. for example if you look at the "Class" TabSheet you can discover the "Class" Name of the control..
For example:
TFormXX, TEditYY, TPanelZZZ for delphi apps
WindowsForms10.XXXX.yyy, for .NET apps
wxWindowsXXX for wxWindows apps
AfxWndXX for MFC/VC++ apps (I think)
I think this is the fastest way (although not the most accurate) to find information about apps..
I understand your curiosity.
You can identify Delphi and C++ Builder apps and their SKU by looking for a couple of specific resources that the linker adds. Specifically RC Data\DVCLAL and RC DATA\PACKAGEINFO. The XN Resource Editor makes this a lot easier, but it might choke on compressed EXEs.
EXE compressors complicate things a little. They can hide or scramble the contents of the resources. Programs compressed with UPX are easy to identify with a HEX editor because the first 2 sections in the PE header are named UPX0 and UPX1. You can use the app to decompress these.
Applications compiled with .Net aren't difficult to detect. Recent versions of Delphi even include an IsAssembly function, or you could do a little spelunking in the PE header. Check out the IsManaged function in IsDelphi.
Telling which .Net language was used is trickier. By default, VB.Net includes a reference to Microsoft.VisualBasic, and VCL.Net apps included Borland specific references. However, VCL.Net is defunct in favour of Delphi Prism, and you can add a reference to the VB assembly to any managed language.
I haven't looked at some of the apps that use signatures to identify the the compiler, so I don't know how well they work.
I hope this helps.
First, look to see what run time libraries it loads. A C program won't normally load Visual Basic's library.
Also, examine the executable for telltale strings. In most executables, this is near the end. If the program uses string constants, there might be a clue in how they are stored.
A good disassembler, plus of course an excellent understanding of the underlying CPU architecture, can often help you identify the runtime libraries that are in play. Unless the exe has been carefully "stripped" of symbols and/or otherwise masked, the names of symbols seen in runtime libraries will often provide you with programming-language hints, because different languages' standards specify different names, and vendors of compilers and accompanying runtime libraries usually respect those standards pretty closely.
Of course, you won't get there without knowledge of the various possible languages and their library standards -- and if the code's author was intent to mask the information, that's not too hard for them to do, either.
If you have available a large set of samples from known compilers, I should think this would be an excellent application for machine learning. I believe so-called "supervised learning" is relevant here. Unfortunately I know next to nothing about the topic—only that I have heard some impressive results presented at conferences.
You might dig through the proceedings of the Working Conference on Reverse Engineering to see if anyone else is interested in this problem.
Assuming this is an application for Windows...
Does Reflector recognize it as a .NET assembly? Then it's MSIL, 99% either VB or C#, but you'll likely never know which, nor does it matter.
Does it need an intrepreter (like Java?)? Then it's Java (or whatever the interpreter is.)
Check what runtime DLLs it requires.
Does it require the VB runtime dlls? Congratulations, VB from VisualStudio 6.0 or earlier.
Does it require the Delphi dlls? Congratulations, Delphi.
Did you make it this far? C/C++. Assume C++ unless it requires msys or cygwin dlls, in which case C has maybe a 25% chance.
Congratulations, this should come out correct for the vast majority of Windows software. This probably doesn't actually help you though, as a lot of the same things can be done in all of these languages.
IDA Pro Free (http://www.hex-rays.com/idapro/idadownfreeware.htm) may be helpful. Even if you don't understand assembly language, if you load the EXE into IDA Pro then its initial progress output might (if there are any telltale signs) include its best guess as to which compiler was used.
Start with various options to dumpbin. The symbol names, if not carefully erased, will give you all kinds of hints as to whether it is C, C++, CLR, or something else.
Other tools use signatures to identify the compiler used to create the executable, like PEiD, CFF Explorer and others.
They normally scan the entry point of the executable vs the signature.
Signature Explorer from CFF Explorer can give you an understanding of how one signature is constructed.
It looks like the VC++ linker from V6 up adds a signature to the PE header which youcan parse.
i suggest PEiD (freeware, closed source). Has all of Delphi for Win32 signatures, also can tell you which was packer used (if any).