I am writing a web server, mainly for practice, and decided to get to learn about make and makefiles. My aim is to automate builds to my vps, using Clang/LLVM to build the software, and save the diagnostics to a text file, for a script to submit to my email address. What I can not seem to achieve is to save the diagnostics to my file.
While my Clang builds successfuly, and generates the diagnostics, and my makefile seems to work, I have failed to redirect the diagnostics, both from inside the makefile, and the command line.
My makefile (working correctly, but slightly modified to accomodate the need to save results):
# Makefile to build Ironman HTTP Server
# We will use the clang frontend from the llvm compiler infrastructure
# for building
# --- targets
Ironman :
clang -o Ironman src/Ironman.c > report
# --- remove binary and executable files
clean:
rm -f Ironman
rm -f report
I suspect that ( I may be terribly wrong here) this happens because clang doesn't really return the diagnostics, it just prints them. I do not know if this is the case, and the Clang user guide doesn't suggest anything like that.
[EDIT]: I played with Clang a bit and saw that on a successful compilation it returns 0. The method that I tested it with is:
$ clang <source_file.c>
$ echo $?
0
This suggests that my theory may be correct, which complicates things :-\
Could somebody point me in the right direction?
Clang, like any other program, outputs diagnostics to stderr. You can redirect stderr to stdout like so:
Ironman :
clang -o Ironman src/Ironman.c > report 2>&1
Related
Here is a sample output of my cmake:
2017/10/27 07:51:46 Platform overridden to 'RHEL5_64'
-- cmake version: 3.2.3
-- Configuring done
-- Generating done
-- Build files have been written to: /local/home/etc
[3/3] Linking CXX shared library libsample_z.so
The last line actually shows progress (as indicated by [3/3]) and thus overwritten in-place; so I cannot see all the logs (i.e the messages correspond to [1/3] and [2/3]). I want cmake to print all logs to stay on its own line, like:
Linking CXX shared library libsample_x.so
Linking CXX shared library libsample_y.so
Linking CXX shared library libsample_z.so
What can be done in cmake to log like this?
The "problem" with ninja is, that it automatically detects if you are running from a shell where it can replace the progress output in line. And there are - as of October 2017 - no command line switches or environment variables to change this behavior.
Since it checks for the console's output buffer, I found that piping the output on my Windows console somewhere else does show multi-line outputs again. So i used the following pipe command:
cmake -G "Ninja" ..
cmake --build . > CON
NOTE: That will only work if you don't have this call inside a script that needs the stdout output itself again for piping it e.g. into a log file. Meaning the output is no longer on stdout after this pipe command.
Similar to the accept answer, another option is to tick ninja into not seeing the terminal by piping to cat.
ninja | cat -
This will get you multi line output, but you will loose any console coloring as well.
From man ninja:
-v show all command lines while building
How can I determine what will be the arguments of the ld command in the build process from inside a script running as "run script build phase"?
I was looking at xcodebuild -dry-run as an option, but then I need to understand what should be the arguments I supply it.
Any idea for a robust solution?
EDIT:
It seems that xcodebuild doesn't support LD and LDPLUSPLUS when the project includes swift source code. So the solution that #fpotter offered doesn't work on project with swift.
Any thoughts?
Xcode doesn't expose a nice way to do this. In a Run Script build phase, all you have to work with are the Xcode build settings provided to you in the environment.
If you really need the entire argument list to ld, there's a hack you can use. With the LD and LDPLUSPLUS build settings, you can make Xcode call a script of your own instead of the real ld. From that script, you could capture the args, call through to the real linker, and then do whatever post processing you like there rather than in a Run Script build phase.
Here's how you could do that:
Create an .xcconfig for your target.
It should look like this:
LD = $(SRCROOT)/ld-wrapper/clang
LDPLUSPLUS = $(SRCROOT)/ld-wrapper/clang++
SRCROOT points to your project's directory. The LDPLUSPLUS line is only required if your app has C++ or ObjC++ code. If you don't want to create an xcconfig, you can also add these as User-Defined build settings via the Xcode UI.
Create wrapper scripts for Xcode to call.
Install a script like this at <your project root>/ld-wrapper/wrapper.sh:
#!/bin/bash
set -o errexit
# Choose which clang to run (clang or clang++) depending on how we're invoked.
# If we're invoked via the 'clang' link, we'll run 'clang'. If we're invoked
# via the 'clang++' link, we'll run 'clang++'.
CLANG_PATH="$DEVELOPER_DIR"/Toolchains/XcodeDefault.xctoolchain/usr/bin/$(basename "$0")
"$CLANG_PATH" "$#"
echo "clang args: $#"
echo "do any post processing here."
Create symlinks for the wrapper script for clang and clang++:
cd <project root>/ld-wrapper
ln -s wrapper.sh clang
ln -s wrapper.sh clang++
That's it. It's ugly, but it works.
In my Makefile, I want to link to a library only if it is installed on the machine. So, for example if the library is hwloc, I want to do the following:
xfoo : foo.o
if (hwloc installed)
gcc foo.o -o $# -lhwloc
else
gcc foo.o -o $#
Is there anyway to do something like this? i.e. Is it possible to check if a specific library is installed and use that as a condition in a Makefile?
Here's the wrong answer:
xfoo : foo.o
if (hwloc installed); then gcc foo.o -o $# -lhwloc; else gcc foo.o -o $#; fi
Commands executed from a Makefile do not have to be just simple, single commands. Anything that a shell can execute, can be invoked from a Makefile. Including an entire script, sandwiched into one line.
Here's the right answer:
However, the above approach is the wrong one. You will find that many free software packages do this kind of thing all the time: conditionally link in a library, if it's available.
But the way that it's done is by running a separate configure script, before running make. Go grab the source tarball to a random free software package, of your choosing, and read the installation instructions. They will all tell you to run the configure script first, before running make.
A crushing majority of free software packages use the GNU toolchain to create their build system -- the configure script, and the Makefile. The GNU toolchain consists of autoconf and automake tools (also libtool in many cases). Go Google these, for more information.
There are also a few other, less popular toolchains, but the GNU toolchain is the most frequently one used, for this sort of a thing. So, to do something along the lines of what you're trying to do, the way this gets typically done is:
In the configure.ac file:
AC_CHECK_LIB(hwloc,some_function_in_the_hwloc_library,[LINK_HWLOC=-lhwloc])
AC_SUBST(LINK_HWLOC)
In the Makefile.am file:
hwloc_LDADD=#LINK_HWLOC#
That's it. That's the way this is done the countless number of times most free software packages need to do this exact same thing. autoconf and automake will take care of writing the shell script and the makefile, that implements this.
I don't have access to a Linux machine at the moment so pardon me my answer will be untested.
I will respectfully disagree with both of my predecessors.
First, using autotools to amend an existing Makefile is a bad idea. Autotools are made to avoid worrying about creating a good Makefile in a simple use case. It's as if OP asked "How to change + to - in my Python script" and the answer was "write a shell script to modify the script, save it in temporary file and execute the file"
Second answer, why do something manually when it can be painlessly done automatically?
So, IMHO the correct answer is, this is the exact use case for $(wildcard):
xfoo: foo.o $(wildcard libhwloc.a)
gcc $(patsubst lib%.a, -l%, $^) -o $#
Note: the library is installed or not ahead of time but not to be made during the build.
If you don't want to get involved with the autotools/etc. for this (which while a reasonable solution is also reasonable to want to avoid for something this simple) and you don't want to have to play guessing games about where people may or may not have this hwloc library installed then the best you can do is to let people turn the feature on manually.
Use three (or four) make variables. USE_HWLOC, HWLOC_LDLIBS, HWLOC_CFLAGS and possibly HWLOC_LDFLAGS.
Then when USE_HWLOC is defined you link against the library and use the other three variables in case they have also been set.
ifdef USE_HWLOC
HWLOC:=-lhwloc
else
HWLOC:=
HWLOC_LDLIBS:=
HWLOC_LDFLAGS:=
HWLOC_CFLAGS:=
endif
xfoo : foo.o
gcc foo.o -o $# $(HWLOC_LDLIBS) $(HWLOC)
gcc (or other compilers) often generate huge text output and it's very difficult to see where the error is or miss warnings. I've done some search but havn't found a clean simple solution to color code the compiler output (so for instance warnings are yellow, errors are red, etc...)
Gcc 4.9 seems to have added this feature via the -fdiagnostics-color flag:
here's an alternative if you are looking for something very simple:
#!/bin/bash -e
make ${#} 2>&1 | perl -wln -M'Term::ANSIColor' -e '
m/Building|gcc|g++|\bCC\b|\bcc\b/ and print "\e[1;32m", "$_", "\e[0m"
or
m/Error/i and print "\e[1;91m", "$_", "\e[0m"
or
m/Warning/i and print "\e[1;93m", "$_", "\e[0m"
or
m/Linking|\.a\b/ and print "\e[1;36m", "$_", "\e[0m"
or
print; '
Just alias your make to this script and make sure it's executable...
Debian and Ubuntu gives the colorgcc package for that purpose.
And I usually run gcc (and make) thru emacs with M-x compile then the messages are colorized.
addenda
GCC 4.9 has a native colorization facility and GCC 6 - released end of April 2016 - (and probably GCC 5 too) is enabling it by default (when stdout is a terminal).
Ok, I'll just leave a notice about my own (python based) tool also :)
It is called Pluggable Output Processor and designed not only to colorize output of one particular program. Here is sample GCC output before:
After:
See colorgcc, a perl script that coulours the gcc output.
How to install and use colorgcc to colorize your gcc compiler output:
At least 3 answers here so far mention colorgcc, but NONE OF THEM EXPLAIN HOW TO INSTALL IT! (And it's not obvious). So, here's how to install the latest version in Ubuntu!
Go here and click "Clone or download" --> "Download Zip". I saved it into "~/Downloads/Install_Files"
Navigate to it in your file browser and right click it and go to "Extract Here." I now have a directory called "~/Downloads/Install_Files/colorgcc-master".
Copy the "colorgcc.pl" script to "/usr/bin/colorgcc" to "install" it (be sure to use the correct directory according to where you extracted it above): sudo cp ~/Downloads/Install_Files/colorgcc-master/colorgcc.pl /usr/bin/colorgcc
Make it executable: sudo chmod +x /usr/bin/colorgcc
Make the "~/bin" directory if it does not yet exist: mkdir ~/bin
*Make symbolic links that point to "/usr/bin/colorgcc" so that whenever you call gcc or g++ it automatically calls colorgcc instead:
ln -s /usr/bin/colorgcc ~/bin/g++
ln -s /usr/bin/colorgcc ~/bin/gcc
(if you ever want to uninstall colorgcc for some reason just delete these symbolic links "~/bin/g++" and "~/bin/gcc", and the Perl script: "/usr/bin/colorgcc" and you're done)
Done!
Here is a sample g++ output now when I call g++ -Wall -std=c++11 time_until_overflow_2.cpp -o time_until_overflow_2:
*Note: making these symbolic links in "~/bin" only works if "~/bin" is in your PATH variable in a location before the folder where the actual gcc and g++ executables are located. To ensure you have "~/bin" in your path you can view the PATH variable contents with: echo $PATH. If you don't see "/home/YOUR_USERNAME/bin" at the beginning of your path, add it with: export PATH=~/bin:$PATH.
References:
See here for more info. and for where I originally learned most of these steps: https://imranfanaswala.wordpress.com/2009/02/02/setting-up-colorgcc/. Thanks Imran Fanaswala!
~GS
you can use GilCC which is a Ruby tool that will convert GCC output to color in real-time. Right now you have two choices: Perl script (colorGCC) or GilCC and if you already work with Ruby you will like GilCC.
Unique to GilCC; GilCC has warning and errors counters and also shows compile time, very handy when you are trying to improve things. Because it is in Ruby it is cross platform. It is flexible and you can add more gems to customize it anyway you want.
The link to the download page is here.
https://github.com/gilmotta/GilCC
Although GCC 4.9 has -fdiagnostics-color option to enable colored outputs to terminals, I have created a tiny tool called 'crror' to get colorized compiler output.
It supports outputs from make as well. I can add colorize patterns for other tools if anyone requires.
So there seems to be this problem with GNU Make's $(wildcard) function keeping a directory open on Windows. See (unasnwered) post "make is holding a directory open". Google does not provide much information on the topic.
In short: the Makefile uses the $(wildcard) function at some point, and keeps a directory open, which typically prevents the "make clean" rule to do its work correctly. Re-running "make clean" a second time usually solves it.
I'm using GNU Make version 3.81 under a standard DOS-Box. The author of the post linked to above is using Cygwin.
Has anyone found a fix for this?
Sounds like a file descriptor leak, all right -- harmless for very-short-lived processes (like make) on UNIX, but a right PITA on Windows.
As this is allegedly a bug in make, as opposed to a problem with its usage, it should be addressed first by validating that it still exists when built from source on the newest upstream version, and then by filing a bug report with the GNU make project (or with any distributor with whom you have an appropriate support contract), or diving into the source and attempting to fix it yourself.
It wouldn't hurt to try to reproduce on Linux -- checking for file descriptor leaks are much easier here, as one can just look at /proc/self/fd (or, for a child of make, /proc/$PPID/fd) for things that don't belong.
I did find a workaround for the problem, which at least lets me work in peace.
The problem was that the $(wildcard) function was used to collect the sources files. My clean rule, however, only deletes a directory - no need for the collecting to take please. So I basically put the part of the Makefile that needs to collect the sources files in a conditional statement:
# The clean rule is always parsed
clean:
rm -rf $(OUTPUT_DIRECTORY)
# The compile rule is only interpreted if we did not invoke 'make clean'. We
# can test the value of $(MAKECMDGOALS) for that:
ifeq ($(filter $(MAKECMDGOALS),clean),)
SOURCE_FILES := $(wildcard ...)
compile:
g++ $(SOURCE_FILES) ...
endif