caffe cmake error by ccache - gcc

Question

When I successfully 'cmake ..' caffe build, I preceed with 'make all' command but came up with an error below. I don't know if there is something wrong with NVCC or gcc.
[ 1%] Built target proto
[ 1%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_split_layer.cu.o
/usr/bin/ccache: invalid option -- 'E'
Usage:
ccache [options]
ccache compiler [compiler options]
compiler [compiler options] (via symbolic link)
Options:
-c, --cleanup delete old files and recalculate size counters
(normally not needed as this is done automatically)
-C, --clear clear the cache completely
-F, --max-files=N set maximum number of files in cache to N (use 0 for
no limit)
-M, --max-size=SIZE set maximum size of cache to SIZE (use 0 for no
limit; available suffixes: G, M and K; default
suffix: G)
-s, --show-stats show statistics summary
-z, --zero-stats zero statistics counters
-h, --help print this help text
-V, --version print version and copyright information
See also <http://ccache.samba.org>.
CMake Error at cuda_compile_generated_split_layer.cu.o.cmake:206 (message):
Error generating
/home/gpuusr/lpq/caffe-332/build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_split_layer.cu.o
make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_split_layer.cu.o] Error 1
make[1]: *** [src/caffe/CMakeFiles/caffe.dir/all] Error 2
make: *** [all] Error 2

Answer 1

I had a similar issue trying to build OpenCV with CUDA support. There's some odd interaction between CMake, CUDA, and ccache that I don't understand, it's trying to call ccache with invalid arguments. You can bypass this by specifying your system compiler for CUDA instead:
cmake -DCUDA_HOST_COMPILER=/usr/bin/g++ ..
(Or whatever compiler you're using.)

Answer 2

This used to be a bug in CMake. It was fixed with CMake release 3.4.0.
Proof
The code from here can be compiled with the following CMakeLists.txt.
cmake_minimum_required (VERSION 2.8.9 FATAL_ERROR)
project(CudaTest)
find_package(CUDA REQUIRED)
cuda_add_executable(hello_cuda hello.cu)
With /usr/lib/ccache on the PATH.
Building once with CMake 3.2 and once with CMake 3.7.1.
CMake 3.2:
$ make
...
/path/to/build-3.2
[100%] Building NVCC (Device) object CMakeFiles/hello_cuda.dir/hello_cuda_generated_hello.cu.o
/usr/bin/ccache: invalid option -- 'E'
...
CMake 3.7.1:
$ make
[ 50%] Building NVCC (Device) object CMakeFiles/hello_cuda.dir/hello_cuda_generated_hello.cu.o
[100%] Linking CXX executable hello_cuda
[100%] Built target hello_cuda
For the record
A small git bisect run over CMake 3.2.2 -- 3.7.1 shows that this is the commit that fixed it:
commit b405f01daaeaeda98d448e2f0d71ea685757a5bd
Author: Bill Hoffman <bill.hoffman#kitware.com>
Date: Fri Jun 12 14:16:09 2015 -0400
FindCUDA: Resolve a host compiler symlink only if it is Apple cc - clang
Otherwise using a "cc -> ccache" or similar symlink as the compiler
causes FindCUDA to select ccache as the host compiler. Update the logic
added by commit v3.1.0-rc1~354^2 (FindCUDA: Fix OSX Clang & no C
language enabled, 2014-06-12) to apply only in the specific case it is
needed.
This commit went into CMake 3.4.0. Compare git rev-list v3.4.0 | grep b405f01 with git rev-list v3.3.2 | grep b405f01.