To speed up the compilation of a large source file does it make more sense to prune back the sheer number of headers used in a translation unit, or does the cost of compiling code far outweigh the time it takes to process-out an include-guarded header?
If the latter is true an engineering effort would be better spent creating more, lightweight headers instead of less.
So how long does it take for a modern compiler to handle a header that is effectively include-guarded out? At what point would the inclusion of such headers become a hit on compilation performance?
(related to this question)
I read an FAQ about this the other day... first off, write the correct headers, i.e. include all headers that you use and don't depend on undocumented dependencies (which may and will change).
Second, compilers usually recognize include guards these days, so they're fairly efficient. However, you still need to open a lot of files, which may become a burden in large projects. One suggestion was to do this:
Header file:
// file.hpp
#ifndef H_FILE
#define H_FILE
/* ... */
#endif
Now to use the header in your source file, add an extra #ifndef:
// source.cpp
#ifndef H_FILE
# include <file.hpp>
#endif
It'll be noisier in the source file, and you require predictable include guard names, but you could potentially avoid a lot of include-directives like that.
Assuming C/C++, simple recompilation of header files scales non-linearly for a large system (hundreds of files), so if compilation performance is an issue, it is very likely down to that. At least unless you are trying to compile a million line source file on a 1980s era PC...
Pre-compiled headers are available for most compilers, but generally take specific configuration and management to work on non system-headers, which not every project does.
See for example:
http://www.cygnus-software.com/papers/precompiledheaders.html
'Build time on my project is now 15% of what it was before!'
Beyond that, you need to look at the techniques in:
http://www.amazon.com/exec/obidos/ASIN/0201633620/qid%3D990165010/002-0139320-7720029
Or split the system into multiple parts with clean, non-header-based interfaces between them, say .NET components.
The answer:
It can be very expensive!
I found an article where someone had done some testing of the very issue addressed here, and am astonished to find you can increase your compilation time under MSVC by at least an order of magnitude if you write your include guards properly:
http://www.bobarcher.org/software/include/index.html
The most astonishing line from the results is that a test file compiled under MSVC 2008 goes from 5.48s to 0.13s given the right include guard methodology.
Related
In a large C++ project Coverity analysis reports issues in files that we won't be fixing e.g. Boost libraries, STL headers, some 3rd party libraries etc.
Ideally there would be a mechanism to completely ignore these and not to increment the total count for such issues.
In Coverity Connect (v8.1) we've set up Components with file path regexp and that nicely filters the files in question when browsing but the total number of issues does not drop down. Two questions related to this:
is there a way to drop the number of total issues for files we don't care about? e.g. after such an issue has already been captured
if new code we introduce includes one of the offending boost/STL/etc headers, will this clock up the total issue counter? (clearly, that would be less than desirable)
Mandatory disclaimer first: Your customers won’t care that bugs in your code came from a third party. That said, the main answer at the link Yannis mentioned is generally the correct one: “use a component filter.” If it’s not working correctly for you, double-check your configuration. I found it quite robust, even with a negative look-ahead regex with over a hundred disjuncts.
Once such issue is found, you can mark it as false positive or ignore it all the way. You have to do this only once. In future analyze, when this issue is found again, it will keep this status. And no, if you use this include to other files too, the total issue counter won't go higher as long as the issue is in the same file.
Check this:
Can Synopsys Static Analysis (Coverity) automatically ignore issues in third-party or noncritical code?
We have the situation where we have a large number of inter-related DLLs that form our product. This is a very "old" product (as in it has been developed over 20 years) and has suffered in the past from different defaults for structure packing over several versions of Visual Studio.
So, in the many cases where #pragma pack has not been used in the DLL header files, but the structure alignment has been set instead in the project properties, we can have the situation where a project that imports the DLL (via its lib and header) has a different structure alignment and potentially results in mismatched structure sizes.
This is complicated by the fact that structs can be sized correctly by "accident" - e.g. if all members of the struct are unsigned int then pack(4) in the DLL and pack(2) in the importing project can work ok. Until, of course, someone amends a struct to add a bool for example.
I would like to remove all of this potential confusion by adding #pragma pack statements to the header files of all exporting modules but would first like to assess whether we have any such exposures in our current code (thinking about hard-to-track runtime bugs here). Also, it may be useful to introduce some automated checking into our build process to make sure we never hit these situations, even with third-party DLLs or LIBs.
So, my question:
Is it possible, from a compiled DLL, or its associated LIB, to determine what structure alignment was in force at the time the DLL was compiled? Similarly, is it possible to discover this for an EXE?
What am I wondering is if there is anything in the PE format or LIB (is that COFF?) that can be used to find this information?
UPDATE
Well, no good came from examining libs and dlls with dumpbin, so I'm going to try to get some info from the PDB files we generate from our Release builds. I found this as a starting point...
I would say that is not possible. C++ doesn't have type-information applied to it (unless enabled with RTTI, but won't be of much help for this problem). Structure is nothing but a sequence of bytes, for the programmer. Compiler will replace the variable.member with appropriate byte-alignment to access that data.
I doubt you have correct debugging information (i.e. PDB file) for the DLL to lookup the symbols. Even with that, it is not possible to find "packing" of a structure.
I have faced problem with structure sizes in different EXE/DLLs (having full source code), where sizeof is only tool we can use to find the difference (and go nested to find the root of problem). Even with this technique, it it not possible which packing is enabled for a particular structure.
I have literally hundreds of C++ source files used by many projects. They were written quite a while back, and they are all wrapped in packing pragmas:
#pragma pack(push, 1)
/* Code here ... */
#pragma pack(pop)
I have been put in charge of porting to x64. Amongst the many changes that need to be made, one is the requirement for a 16-byte aligned stack for Windows API calls. After some analysis of our system, we've determined that 1-byte structure alignment is not necessary and won't have any adverse affects on the system. I need to get rid of the 1-byte packing.
I know I can do a quick find/replace on all the files and just strip them out. This is an OK solution; I'm perfectly happy to do this if it's the only way. However, if I can avoid having to check in a revision which involves changes to literally hundreds of source files, and all the conflicts that might go with it, then that would be preferable.
Is there a way to get the Microsoft Compiler to ignore the #pragma pack?
As far as I know, there is no way to disable #pragmas when using MSVC.
What I would probably do is write a script in a language that is well suited to text processing (I'd probably use Ruby, but Python, Perl, sed should all do the job) and simply use that to comment out or remove the #pragma pack lines. This should be comparatively easy as the #pragmas are going to be the only statement on a give line of code, and the script languages do usually include functionality to iterate through a set of directories.
I have worked with OpenCL on a couple of projects, but have always written the kernel as one (sometimes rather large) function. Now I am working on a more complex project and would like to share functions across several kernels.
But the examples I can find all show the kernel as a single file (very few even call secondary functions). It seems like it should be possible to use multiple files - clCreateProgramWithSource() accepts multiple strings (and combines them, I assume) - although pyopencl's Program() takes only a single source.
So I would like to hear from anyone with experience doing this:
Are there any problems associated with multiple source files?
Is the best workaround for pyopencl to simply concatenate files?
Is there any way to compile a library of functions (instead of passing in the library source with each kernel, even if not all are used)?
If it's necessary to pass in the library source every time, are unused functions discarded (no overhead)?
Any other best practices/suggestions?
Thanks.
I don't think OpenCL has a concept of multiple source files in a program - a program is one compilation unit. You can, however, use #include and pull in headers or other .cl files at compile time.
You can have multiple kernels in an OpenCL program - so, after one compilation, you can invoke any of the set of kernels compiled.
Any code not used - functions, or anything statically known to be unreachable - can be assumed to be eliminated during compilation, at some minor cost to compile time.
In OpenCL 1.2 you link different object files together.
I am using Visual Studio, and it seems that getting rid of unused references and using statements speeds up my build time on larger projects. Are there other known ways of speeding up build time. What about for other languages and build environments?
What is typically the bottleneck during build/compile? Disk, CPU, Memory?
What is a list of/are good references for distributed builds?
The biggest improvement we made for our large C++ project was from distributing our builds. A couple of years ago, a full build would take about half an hour, while it's now about three minutes, of which one third is link time.
We're using a proprietary build system, but IncrediBuild is working fine for a lot of people (we couldn't get it to work reliably).
Fixing your compiler warnings should help quite a bit.
Buy a faster computer
At my previous job we had big problems with compilation time and one of the strategies we used was called the Envelope pattern see here.
Basically it attempts to minimize the amount of code copied in headers by the pre-processor by minimizing header size. It did this by moving anything that wasn't public to a private friend class, here's an example.
foo.h:
class FooPrivate;
class Foo
{
public:
Foo();
virtual ~Foo();
void bar();
private:
friend class FooPrivate;
FooPrivate *foo;
};
foo.cpp:
Foo::Foo()
{
foo = new FooPrivate();
}
class FooPrivate
{
int privData;
char *morePrivData;
};
The more include files you do this with the more it adds up. It really does help your compilation time.
It does make things difficult to debug in VC6 though as I learned the hard way. There's a reason it's a previous job.
If you're using a lot of files and a lot of templated code (STL / BOOST / etc.), then Bulk or Unity builds should cut down on build and link times.
The idea of Bulk Builds to break your project down into subsections and include all the CPP files in that subsection into a single file. Unity builds take this further by having a Single CPP file that is compiled that inludes all other CPP files.
The reason this is often faster is:
1) Templates are only evaluated once per Bulk File
2) Include files are opened / processed only once per Bulk File (assuming there is a proper #ifndef FILE__FILENAME__H / #define FILE__FILENAME__H / #endif wrapper in the include file). Reducing total I/O is a good thing for compile times.
3) The linker has much less data to work with (Single Unity OBJ file or several Bulk OBJ files) and is less likely to page to virtual memory.
EDIT Adding a couple of links here on stack overflow about Unity Builds.
Be wary of broad-sweeping "consider this directory and all subdirectories for header inclusion" type settings in your project. This will cause the compiler to have to iterate every directory until the header file requested is found, and can be a very expensive operation for however many headers you include in your project.
Please read this book. It's pretty good on the topic of physical structure your project into different files with minimal rebuilds.
Unforunately it was written before templates became that important. The templates are the real time killer when it comes to C++ compilation. Especialls if you make the mistake and use smart pointers everywhere. In this case you can only constanly upgrade to the latest CPU and recent SSD drives. MSVC is already the fastests existing C++ compiler if you use precompiled headers.
Visual Studio supports parallel builds, which can help, but the true bottleneck is Disk IO.
In C for instance - if you generate LST files your compile will take ages.
Don't compile with debug turned on.
For C++ the mayor bottleneck is the disk I/O. Many headers include other headers back and forth, which causes a lot of files to be opened and read through for each compilation unit.
You can reach significant improvement if you move the sources into a RAM-disk. Even more if you ensure that your source files read through exactly once.
So for new projects I began to include everything into a single file I call _.cpp. It's structure is like this:
/* Standard headers */
#include <vector>
#include <cstdio>
//...
/* My global macros*/
#define MY_ARRAY_SIZE(X) (sizeof(X)/sizeof(X[0]))
// My headers
#include "foo.h"
#include "bar.h"
//...
// My modules
#include "foo.cpp"
#include "bar.cpp"
And I only compile this single file.
My headers and source files does not include anything, and use namespaces to avoid clashes with other modules.
Whenever my program misses something, I add its header and source into this module only.
This way each source file and header is read exactly once, and builds very quickly. Compile times increase only linearly as you add more files, but not quadratically. My hobby project is about 40000 loc and 500 modules but still compiles about 10-20 seconds. If I move all sources and headers into a RAM-disk compile time reduces to 3 seconds.
Disadvantage of this, that existing codebases are quite difficult to refactor to use this scheme.
For C# - Using fixed versions for your assemblies instead of auto incremental ones greatly speeds up subsequent local builds.
assemblyinfo.cs
// takes longer to update version number changes on all references to this assembly
[assembly: AssemblyVersion("1.0.*")]
// this way already built assemblies do not need to be recompiled
// to update version number changes.
[assembly: AssemblyVersion("1.0.0.0")]
Compilation Time and brittle base class problem : I have written a blog on a way to improve compilation time in C++. Link.