Related
I have compiled GCC from source but I can't seem to fully understand the utility of gcc compiling itself three times.
What benefit does this serve ?
This answer says:
Build new version of GCC with existing C compiler
re-build new version of GCC with the one you just built
(optional) repeat step 2 for verification purposes.
Now my question is that once the first step is complete and the compiler is built why waste time rebuilding it ?
Is it just for verification ? If so, it seems pretty wasteful.
Things get more complicated over here,
The build for this is more complex than for prior packages, because
you’re sending more information into the configure script and the make
targets aren’t standard.
I mean the whole compiler is written in C right, so why not just do everything in one pass ?
What is the use of the 3-phase bootstrap ?
Thanks in advance.
Stage 2. and 3. are a good test for the compiler itself: If it can compile itself (and usually also some libraries like libgcc and libstdc++-v3) then it can chew non-trivial projects.
In stage 2. and 3., you can generate the compiler with different options, for example without optimization (-O0) or with optimization turned on (-O2). As the output / side effects of a program should not depend on the optimization level used, either version of the compiler must produce the same binary for the same source file, even though the two compilers are binary very different. This is yet another (run-time test) for the compiler.
If you prefer non-bootstrap for some reason, configure --disable-bootstrap.
Considering the question from an information theory perspective, the first stage in a three stage compilation of a compiler does not produce a compiler. It produces a hypothesis that requires experimental verification. The sign of a good compiler distribution package is that it will produce, out of the box and without further work for the system administrator or compiler developer, a working compiler of the distribution's version and with the desired features of that version of that brand of compiler.
Making that happen is not simple. Consider the variables in the target environment.
Target operating system brand
Operating system version
Operating system settings
Shell environment variables
Availability of headers for inclusion
Availability of libraries for linking
Settings passed to the build process
Architecture of the target processing unit
Number of processing units
Bus architecture
Other characteristics of the execution model
Mistakes the developers of the compiler might make
Mistakes the person building the compiler might make
In the GNU compiler tool set, and in many tarball distributions, the program "configure" attempts to produce a build configuration that adapts to as many of the permutations of these as is reasonably possible. The completion without error or warning from configure is not a guarantee that the compiler will function. Furthermore, and more importantly for this question, the completion of the build is no guarantee either.
The newly built compiler may function for HelloWorld.c but not for a collection of a thousand source files in a multi-project, multi-repository collection of software called, "Intelligent Interplanetary Control and Acquisition System."
Stage two and three are reasonable attempts at checking at least some of the compiler capabilities, since the compiler source itself is handy and demands quite a bit out of the hypothetically working compiler just built.
It is important to understand that the result of stage one and the result of stage two will not match. Their executables and other built artifacts are results from two different compilers. The stage one result is compiled with whatever the build system found in one of the directories listed in the "PATH" variable to compile C and C++ source code. The stage two result is compiled with the hypothetically working new compiler. The interesting probabilistic consideration is this:
If the result of using stage one's result to compile the compiler again equals exactly the result of using stage two's result to compile the compiler a third time, then both are likely correct for at least the features that the compiler's source code requires.
That last sentence may need to be reread a dozen times. Its actually a simple idea, but the redundancy of the verb compile and noun compiler can tie a knot that takes a few minutes to untie and be able to retie. The source, the target, and the action executed have the same linguistic root, not just once but three times.
The build instructions for the compiler, as of May 25th, 2020, states the converse, which is easier to understand but merely anecdotal, not getting at the cruz of the reason three stages are important.
If the comparison of stage2 and stage3 fails, this normally indicates that the stage2 compiler has compiled GCC incorrectly, and is therefore a potentially serious bug which you should investigate and report.
If we consider C/C++ development from a reliability assessment, test-first, eXtreme Programming, 6-Sigma, or Total Quality Management perspective, what component in a C/C++ development environment has to be more reliable than the compiler? Not many. And even the three stage bootstrapping of a compiler that the GNU compiler package has been using since early days is a reasonable but not an exhaustive test. That's why there are additional tests in the package.
From a continuous integration point of view, the entire body of software under development by those that are about to use the new compiler should be tested before and after a new compiler is compiled and deployed. That's the most convenient way to ensure new compiler didn't break the build.
Between the three reliability check points, most people are satisfied.
Ensuring the compiler compiles itself consistently
Other tests the compiler developers have put into their distribution
The developer or system administrators source code domain is not broken by the upgrade
On a mathematics side note, it is actually impossible to exhaustively test a compiler with the silicon and carbon available on planet earth. The bounds of recursion in C++ language abstractions (among other things) are infinite, so the silicon or time required places testing every permutation of source code cannot realistically exist. On the carbon side, no group of people can free up the requisite time to study the source sufficiently to guaranteed that some finite limit is not imposed in some way by the compiler source."
The three levels of checks, only one of which is the three stage bootstrap process, will likely suffice for most of us.
A further benefit of the three stage compile is that the new compiler is compiled with the new compiler which is presumably better either in terms of speed or resource consumption and possibly both.
I'm looking into how the v8 compiler works. I read an article which states source code is tokenized, parsed, an AST is constructed, then bytecode is generated (https://medium.com/dailyjs/understanding-v8s-bytecode-317d46c94775)
Is this bytecode an intermediate representation?
Short answer: No. Usually people use the terms "bytecode" and "intermediate representation" to mean two different things.
Long answer: It depends a bit on your definition (but for most definitions, "no" is still the right answer).
"Bytecode" in virtual machines like V8 refers to a representation that is used as input for an interpreter. The article you linked to gives a good description.
"Intermediate representation" or IR usually refers to data that a compiler uses internally, as an intermediate step (hence the name) between its input (usually the AST = abstract syntax tree, i.e. parsed version of the source text) and its output (usually machine code or byte code, but it could be anything, as in a source-to-source compiler).
So in a traditional setup, you have:
source --(parser)--> AST --(compiler front-end)--> IR --(compiler back-end)--> machine code
where the IR is usually modified several times as the compiler performs various optimizations on it, before finally generating machine code from it. There can also be several different IRs; for example V8's earlier optimizing compiler ("Crankshaft") had two: high-level IR "Hydrogen" and low-level IR "Lithium", whereas V8's current optimizing compiler ("Turbofan") even has three: "JavaScript-level nodes", "Simplified nodes", and "Machine-level nodes".
Now if you wanted to draw the boxes in your whiteboard diagram of the system a little differently, then instead of having a "parser" and a "compiler" you could treat everything between source and machine code as one big "compiler" (which as a first step parses the source). In that case, the AST would be a form of intermediate representation. But, as stated above, usually when people use the term IR they mean "compiler IR", not the AST.
In a virtual machine like V8, the overall execution pipeline is more complicated than described above. It starts with:
source --(parser)--> AST --(bytecode generator)--> bytecode
This bytecode is primarily used as input for V8's interpreter.
As an optimization, when V8 decides to run a function through the optimizing compiler, it does not start with the source code and a parser again, but instead the optimizing compiler uses the bytecode as its input. In diagram form:
bytecode --(interpreter)--> program execution
bytecode --(compiler front-end)--> IR --(compiler back-end)--> machine code --(CPU)--> program execution
Now here's the part where your perspective comes in: since the bytecode in V8 is not only used as input for the interpreter, but also as input for the optimizing compiler and in that sense as a step on the way from source text to machine code, if you wanted to call it a special form of intermediate representation, you wouldn't technically be wrong. It would be an unusual definition of the term though. When a compiler theory textbook talks about "intermediate representation", it does not mean "bytecode".
Is there a way to get information during or after compilation about what parts of the code have been optimized out, but without looking at the assembly or executing the code.
It'd be nice to know immediately if a big code chunk gets optimized away.
Sorry, but your expectations do not match what compilers actually do. Whether you're trying to find dead code or to find bugs that cause code that should run to be skipped, it is not information that a compiler can provide in an easy-to-read form.
With a compiler that translates each line of source code into a sequence of machine instructions, the compiler could easily tell you that it didn't include anything corresponding to a particular line. Of course it couldn't tell you if a line was translated to machine instructions but those machine instructions in fact won't ever be executed — code reachability is undecidable — but I don't think that's what you're after anyway.
The problem is that modern optimizing compilers are a lot more complex than that. A piece of code is often copied around and compiled multiple times under different assumptions (specialization, partial evaluation, loop unrolling, …). Or, conversely, pieces of code can be merged together (function inlining, …). There isn't a simple correspondence between source code and machine code. (That's why debuggers sometimes have trouble reporting the exact source code location of a binary instruction.)
If a big chunk of code gets optimized away, that may simply because it's one of many specialized copies and that particular specialization never happens (e.g. there's separate code for x==0 and x!=0, and separate code for y==0 and y!=0, and x and y are never 0 together so the x==0 && y==0 branch is eventually dropped). It may be something generated by a compile-time conditional instruction, such as a C macro that the compiler optimizes; this happens so often in C code that if compilers reported all such instances, that would create a lot of false positives.
Getting useful reports of potentially unused code or suspicious-looking program code that could indicate a bug requires a rather different kind of static analysis than what compilers do. There are tools that can do that, but they're typically not the same tools that convert source code to optimized machine code. Making static analysis tools that both detect potential problems often enough to be useful and don't produce so many false positives that they're practically unusable is not easy.
gcc -S
will output the assembly code that would have been passed to the assembler (and eventually been linked into the executable). If you squint the right way (and are patient), you can work backwards from that to confirm whether a given bit of code has actually been included in the executable, or was optimized away.
Obviously not something you'd do unless you have a suspicion that something was going on, given the time and effort required...
I just realized that binary compilers convert source code to the binary of the destination platform. Kind of obvious... but if a compiler works such way, then how can the same compiler be used for different systems like x86, ARM, MIPS, etc?
Shouldn't they be supposed to "know" the machine-language of the hardware platform to be able to know how to build the binary? Does a compiler(like gcc) knows the machine language of every single platform that is supported?
How is that system possible, and how can a compiler be optimized for that many platforms at the same time?
Yes, they have to "know" the machine language for every single platform they support. This is a required to generate machine code. However, compilation is a multi-step process. Usually, the first steps of the compilation are common to most architectures.
Taken from wikipedia
Structure of a compiler
Compilers bridge source programs in high-level
languages with the underlying hardware.
A compiler requires
determining the correctness of the syntax of programs,
generating correct and efficient object code,
run-time organization, and
formatting output according to assembler and/or linker conventions.
A
compiler consists of three main parts: the frontend, the middle-end,
and the backend.
The front end
checks whether the program is correctly
written in terms of the programming language syntax and semantics.
Here legal and illegal programs are recognized. Errors are reported,
if any, in a useful way. Type checking is also performed by collecting
type information. The frontend then generates an intermediate
representation or IR of the source code for processing by the
middle-end.
The middle end
is where optimization takes place. Typical
transformations for optimization are removal of useless or unreachable
code, discovery and propagation of constant values, relocation of
computation to a less frequently executed place (e.g., out of a loop),
or specialization of computation based on the context. The middle-end
generates another IR for the following backend. Most optimization
efforts are focused on this part.
The back end
is responsible for translating the IR from the middle-end into assembly code. The target
instruction(s) are chosen for each IR instruction. Register allocation
assigns processor registers for the program variables where possible.
The backend utilizes the hardware by figuring out how to keep parallel
execution units busy, filling delay slots, and so on. Although most
algorithms for optimization are in NP, heuristic techniques are
well-developed.
More this article which describes the structure of a compiler and on this one which deals with Cross compilers.
The http://llvm.org/ project will answer all of your questions in this regard :)
In a nutshell, cross HW compilers emit "intermediate representation" of the code , which is HW agnostic and then its being customized via the native tool chain
Yes it is possible, it's called Cross Compiler. Compilers usually first they generate the object code which is not understanable by the current machine but it can be migrated to the destiny machine with another compiler. Next, object code is "compiled" again and linked with external libraries of the target machines.
TL;DR: Yes, the compilers knows the target code, but you can compile in another hardware.
I recommend you to read attached links for information.
Every platform has its own toolchain, toolchain includes gcc,gdb,ld,nm etc.
Let's take specific example of gcc as of now. GCC source code has many layers including architecture dependent and independent part. Architecture dependent part contains procedures to handle architecture specific things like their stack, function calls, floating point operations. We need to cross compile the gcc source code for a specific architecture like for ARM. You can see its steps here for reference:- http://www.ailis.de/~k/archives/19-arm-cross-compiling-howto.html#toolchain.
This architecture dependent part is responsible for handling machine language operations.
I've Googled and poked around the Go website, but I can't find an explanation for Go's extraordinary build times. Are they products of the language features (or lack thereof), a highly optimized compiler, or something else? I'm not trying to promote Go; I'm just curious.
Dependency analysis.
The Go FAQ used to contain the following sentence:
Go provides a model for software
construction that makes dependency
analysis easy and avoids much of the
overhead of C-style include files and
libraries.
While the phrase is not in the FAQ anymore, this topic is elaborated upon in the talk Go at Google, which compares the dependency analysis approach of C/C++ and Go.
That is the main reason for fast compilation. And this is by design.
I think it's not that Go compilers are fast, it's that other compilers are slow.
C and C++ compilers have to parse enormous amounts of headers - for example, compiling C++ "hello world" requires compiling 18k lines of code, which is almost half a megabyte of sources!
$ cpp hello.cpp | wc
18364 40513 433334
Java and C# compilers run in a VM, which means that before they can compile anything, the operating system has to load the whole VM, then they have to be JIT-compiled from bytecode to native code, all of which takes some time.
Speed of compilation depends on several factors.
Some languages are designed to be compiled fast. For example, Pascal was designed to be compiled using a single-pass compiler.
Compilers itself can be optimized too. For example, the Turbo Pascal compiler was written in hand-optimized assembler, which, combined with the language design, resulted in a really fast compiler working on 286-class hardware. I think that even now, modern Pascal compilers (e.g. FreePascal) are faster than Go compilers.
There are multiple reasons why the Go compiler is much faster than most C/C++ compilers:
Top reason: Most C/C++ compilers exhibit exceptionally bad designs (from compilation speed perspective). Also, from compilation speed perspective, some parts of the C/C++ ecosystem (such as editors in which programmers are writing their code) aren't designed with speed-of-compilation in mind.
Top reason: Fast compilation speed was a conscious choice in the Go compiler and also in the Go language
The Go compiler has a simpler optimizer than C/C++ compilers
Unlike C++, Go has no templates and no inline functions. This means that Go doesn't need to perform any template or function instantiation.
The Go compiler generates low-level assembly code sooner and the optimizer works on the assembly code, while in a typical C/C++ compiler the optimization passes work on an internal representation of the original source code. The extra overhead in the C/C++ compiler comes from the fact that the internal representation needs to be generated.
Final linking (5l/6l/8l) of a Go program can be slower than linking a C/C++ program, because the Go compiler is going through all of the used assembly code and maybe it is also doing other extra actions that C/C++ linkers aren't doing
Some C/C++ compilers (GCC) generate instructions in text form (to be passed to the assembler), while the Go compiler generates instructions in binary form. Extra work (but not much) needs to be done in order to transform the text into binary.
The Go compiler targets only a small number of CPU architectures, while the GCC compiler targets a large number of CPUs
Compilers which were designed with the goal of high compilation speed, such as Jikes, are fast. On a 2GHz CPU, Jikes can compile 20000+ lines of Java code per second (and the incremental mode of compilation is even more efficient).
Compilation efficiency was a major design goal:
Finally, it is intended to be fast: it should take at most a few seconds to build a large executable on a single computer. To meet these goals required addressing a number of linguistic issues: an expressive but lightweight type system; concurrency and garbage collection; rigid dependency specification; and so on. FAQ
The language FAQ is pretty interesting in regards to specific language features relating to parsing:
Second, the language has been designed to be easy to analyze and can be parsed without a symbol table.
While most of the above is true, there is one very important point that was not really mentionend: Dependency management.
Go only needs to include the packages that you are importing directly (as those already imported what they need). This is in stark contrast to C/C++, where every single file starts including x headers, which include y headers etc. Bottom line: Go's compiling takes linear time w.r.t to the number of imported packages, where C/C++ take exponential time.
A good test for the translation efficiency of a compiler is self-compilation: how long does it take a given compiler to compile itself? For C++ it takes a very long time (hours?). By comparison, a Pascal/Modula-2/Oberon compiler would compile itself in less than one second on a modern machine [1].
Go has been inspired by these languages, but some of the main reasons for this efficiency include:
A clearly defined syntax that is mathematically sound, for efficient scanning and parsing.
A type-safe and statically-compiled language that uses separate compilation with dependency and type checking across module boundaries, to avoid unnecessary re-reading of header files and re-compiling of other modules - as opposed to independent compilation like in C/C++ where no such cross-module checks are performed by the compiler (hence the need to re-read all those header files over and over again, even for a simple one-line "hello world" program).
An efficient compiler implementation (e.g. single-pass, recursive-descent top-down parsing) - which of course is greatly helped by points 1 and 2 above.
These principles have already been known and fully implemented in the 1970s and 1980s in languages like Mesa, Ada, Modula-2/Oberon and several others, and are only now (in the 2010s) finding their way into modern languages like Go (Google), Swift (Apple), C# (Microsoft) and several others.
Let's hope that this will soon be the norm and not the exception. To get there, two things need to happen:
First, software platform providers such as Google, Microsoft and Apple should start by encouraging application developers to use the new compilation methodology, while enabling them to re-use their existing code base. This is what Apple is now trying to do with the Swift programming language, which can co-exist with Objective-C (since it uses the same runtime environment).
Second, the underlying software platforms themselves should eventually be re-written over time using these principles, while simultaneously redesigning the module hierarchy in the process to make them less monolithic. This is of course a mammoth task and may well take the better part of a decade (if they are courageous enough to actually do it - which I am not at all sure in the case of Google).
In any case, it's the platform that drives language adoption, and not the other way around.
References:
[1] http://www.inf.ethz.ch/personal/wirth/ProjectOberon/PO.System.pdf, page 6: "The compiler compiles itself in about 3 seconds". This quote is for a low cost Xilinx Spartan-3 FPGA development board running at a clock frequency of 25 MHz and featuring 1 MByte of main memory. From this one can easily extrapolate to "less than 1 second" for a modern processor running at a clock frequency well above 1 GHz and several GBytes of main memory (i.e. several orders of magnitude more powerful than the Xilinx Spartan-3 FPGA board), even when taking I/O speeds into account. Already back in 1990 when Oberon was run on a 25MHz NS32X32 processor with 2-4 MBytes of main memory, the compiler compiled itself in just a few seconds. The notion of actually waiting for the compiler to finish a compilation cycle was completely unknown to Oberon programmers even back then. For typical programs, it always took more time to remove the finger from the mouse button that triggered the compile command than to wait for the compiler to complete the compilation just triggered. It was truly instant gratification, with near-zero wait times. And the quality of the produced code, even though not always completely on par with the best compilers available back then, was remarkably good for most tasks and quite acceptable in general.
Go was designed to be fast, and it shows.
Dependency Management: no header file, you just need to look at the packages that are directly imported (no need to worry about what they import) thus you have linear dependencies.
Grammar: the grammar of the language is simple, thus easily parsed. Although the number of features is reduced, thus the compiler code itself is tight (few paths).
No overload allowed: you see a symbol, you know which method it refers to.
It's trivially possible to compile Go in parallel because each package can be compiled independently.
Note that Go isn't the only language with such features (modules are the norm in modern languages), but they did it well.
Quoting from the book "The Go Programming Language" by Alan Donovan and Brian Kernighan:
Go compilation is notably faster than most other compiled languages, even when building from scratch. There are three main reasons for the compiler’s speed. First, all imports must be explicitly listed at the beginning of each source file, so the compiler does not have to read and process an entire file to determine its dependencies. Second, the dependencies of a package form a directed acyclic graph, and because there are no cycles, packages can be compiled separately and perhaps in parallel. Finally, the object file for a compiled Go package records export information not just for the package itself, but for its dependencies too. When compiling a package, the compiler must read one object file for each import but need not look beyond these files.
The basic idea of compilation is actually very simple. A recursive-descent parser, in principle, can run at I/O bound speed. Code generation is basically a very simple process. A symbol table and basic type system is not something that requires a lot of computation.
However, it is not hard to slow down a compiler.
If there is a preprocessor phase, with multi-level include directives, macro definitions, and conditional compilation, as useful as those things are, it is not hard to load it down. (For one example, I'm thinking of the Windows and MFC header files.) That is why precompiled headers are necessary.
In terms of optimizing the generated code, there is no limit to how much processing can be added to that phase.
Simply ( in my own words ), because the syntax is very easy ( to analyze and to parse )
For instance, no type inheritance means, not problematic analysis to find out if the new type follows the rules imposed by the base type.
For instance in this code example: "interfaces" the compiler doesn't go and check if the intended type implement the given interface while analyzing that type. Only until it's used ( and IF it is used ) the check is performed.
Other example, the compiler tells you if you're declaring a variable and not using it ( or if you are supposed to hold a return value and you're not )
The following doesn't compile:
package main
func main() {
var a int
a = 0
}
notused.go:3: a declared and not used
This kinds of enforcements and principles make the resulting code safer, and the compiler doesn't have to perform extra validations that the programmer can do.
At large all these details make a language easier to parse which result in fast compilations.
Again, in my own words.
Go imports dependencies once for all files, so the import time doesn't increase exponentially with project size.
Simpler linguistics means interpreting them takes less computing.
What else?