I'm trying to install mxnet by cloning a git repository as follows:
git clone --recursive https://github.com/dmlc/mxnet
However, when I try to build it by dropping into the mxnet directory and running make as follows:
make -j $(nproc) USE_OPENCV=1 USE_BLAS=openblas USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1
I get errors indicating that cblas.h cannot be found - e.g.
/home/me/mxnet/3rdparty/mshadow/mshadow/./base.h:162:14: fatal error: cblas.h: No such file or directory
#include <cblas.h>
The output of make is a series of nvcc compilation commands that seem to be as follows:
/usr/local/cuda/bin/nvcc -std=c++11 -Xcompiler -D_FORCE_INLINES -O3 -ccbin /home/me/anaconda2/envs/deepnets/bin/x86_64-conda_cos6-linux-gnu-c++ -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=[sm_75,compute_75] --fatbin-options -compress-all -Xcompiler "-DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -O3 -DNDEBUG=1 -I/home/me/mxnet/3rdparty/mshadow/ -I/home/me/mxnet/3rdparty/dmlc-core/include -fPIC -I/home/me/mxnet/3rdparty/tvm/nnvm/include -I/home/me/mxnet/3rdparty/dlpack/include -I/home/me/mxnet/3rdparty/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -mf16c -I/usr/local/cuda/include -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -I/home/me/mxnet/3rdparty/mkldnn/build/install/include -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_MKLDNN=1 -DUSE_MKL=1 -I/home/me/mxnet/src/operator/nn/mkldnn/ -I/home/me/mxnet/3rdparty/mkldnn/build/install/include -DMXNET_USE_OPENCV=1 -I/usr/local/include/opencv -I/usr/local/include -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK -DMSHADOW_USE_CUDNN=1 -I/home/me/mxnet/3rdparty/cub -DMXNET_ENABLE_CUDA_RTC=1 -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0" --generate-dependencies -MT build/src/operator/tensor/elemwise_unary_op_trig_gpu.o src/operator/tensor/elemwise_unary_op_trig.cu >build/src/operator/tensor/elemwise_unary_op_trig_gpu.d
If I break out (what I believe) to be all the include directories, I can pick out these directories:
-I/home/me/mxnet/3rdparty/mshadow/
-I/home/me/mxnet/3rdparty/dmlc-core/include -fPIC
-I/home/me/mxnet/3rdparty/tvm/nnvm/include
-I/home/me/mxnet/3rdparty/dlpack/include
-I/home/me/mxnet/3rdparty/tvm/include
-Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -mf16c
-I/usr/local/cuda/include -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0
-I/home/me/mxnet/3rdparty/mkldnn/build/install/include -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_MKLDNN=1 -DUSE_MKL=1
-I/home/me/mxnet/src/operator/nn/mkldnn/
-I/home/me/mxnet/3rdparty/mkldnn/build/install/include -DMXNET_USE_OPENCV=1
-I/usr/local/include/opencv
-I/usr/local/include -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK -DMSHADOW_USE_CUDNN=1
-I/home/me/mxnet/3rdparty/cub
When I look for cblas.h, I get see these copies:
(deepnets) me#Chanticleer:~/mxnet$ sudo find /home -name 'cblas.h'
[sudo] password for me:
/home/me/anaconda2/pkgs/openblas-0.3.3-h9ac9557_1001/include/cblas.h
/home/me/anaconda2/pkgs/lapack-3.6.1-ha44fe06_2/include/cblas.h
/home/me/anaconda2/pkgs/openblas-0.3.5-h9ac9557_1001/include/cblas.h
/home/me/anaconda2/envs/deepnets/include/cblas.h
/home/me/mxnet/julia/deps/cblas.h
(deepnets) me#Chanticleer:~/mxnet$ sudo find /usr -name 'cblas.h'
/usr/include/atlas/cblas.h
/usr/include/openblas/cblas.h
/usr/include/cblas.h
Clearly none of these seem to be where the Makefile has been directed to look. Do I need to change the Makefile in some way (how?), or do I need to put a copy of cblas.h in some other directory, or at least a soft link to it. If so, which directory and how?
If you're compiling with nvcc, you can just use the last include file, since it's in your include directory already, like this:
nvcc -I/usr/include <other stuff here...>
You don't have to do anything as drastic as manually copying; in fact I would really recommend against it because you're going to have to remember to re-do this when there's an update.
To answer your question about the makefile, if you were going to add the declaration I would say it's pretty common to see it being added to the $(CPPFLAGS) variable, which stands for C/C++ Preprocessor Flags. However, I would do this instead:
INCLUDE_DIRS = -I/usr/include
COMPILE.cpp = $(CXX) $(CXXFLAGS) $(CPPFLAGS) $(INCLUDE_DIRS) $(TARGET_ARCH) -c
OUTPUT_OPTION = -o $#
%.o: %.cpp
$(COMPILE.cpp) $(OUTPUT_OPTION) $^
The (proper) convention is for your users to be able to customize the $(CXX), $(CXXFLAGS), and $(CPPFLAGS) variables. Essentially, you want them to pick and customize their compiler, flags, and build options. Since you obviously need this include file, however (or you wouldn't have gotten this error), I would recommend setting it completely separately to avoid any confusion.
Anyways, good luck, man. I hope this helps.
Related
The title might be confusing, but that's what happened...
Here's my Makefile.
all:mystdio libmystdio.a main.o myscanf.o myprintf.o
mystdio: main.o libmystdio.a
clang++ -o mystdio main.o -L. -lmystdio -Wall -g -std=c++17
libmystdio.a: myscanf.o myprintf.o
ar cr libmystdio.a myprintf.o myscanf.o
main.o: main.cpp mystdio.hpp
clang++ -c main.cpp -Wall -g -std=c++17
myscanf.o: myscanf.cpp mystdio.hpp utilities.hpp
clang++ -c myscanf.cpp -Wall -g -std=c++17
myprintf.o: myprintf.cpp mystdio.hpp utilities.hpp
clang++ -c myprintf.cpp -Wall -g -std=c++17
clean:
rm -rf mystdio *.o *.a *.a
When I made changes in myscanf.cpp or myprintf.cpp and then make file, the output showed that it had been compiled and mystdio is updated. But actually nothing happened.
For example, let's assume I add printf(":)\n"); at the beginning of a function in myscanf.cpp. Then I call the function in main.cpp. After making file, output in the terminal tells me that myscanf.o and related files were recompiled. But when the program runs, no :) is outputed.
However, when I add printf(":)\n"); in the main function and make file, it was outputed. This did confuse me. I wonder how can I solve it.
command used:
make && ./mystdio
When I changed something in myprintf.cpp and make again, here's output from the terminal.
clang++ -c myprintf.cpp -Wall -g -std=c++17
ar cr libmystdio.a myprintf.o myscanf.o
clang++ -o mystdio main.o -L. -lmystdio -Wall -g -std=c++17
By the way, if I use make clean before make && ./mystdio, it'll work correctly, but of course that's not what "make" is designed for...
Thanks for your help in advance!
First, I should admit makefiles are something that I'm very inexperienced at, so I apologize if this is an error that I should have been able to solve myself, but I have spent several hours on this, including reading the various answers on this site, and have been unable to discover a solution.
With that said, I have created the following makefile to compile my code on a Linux machine; it completes the sub compilations just fine, but when it comes to making the output itself, xPlatST, it throws an error.
g++ -std=c+=11 -g -Wall -pthread -c -o xPlatST.o xPlatST.cpp
g++ -std=c+=11 -g -Wall -pthread -c -o stdafx.o stdafx.cpp
g++ -std=c+=11 -g -Wall -pthread -c xPlatST xPlatST.o stdafx.o -L../hwloc
g++ error: xPlatST: No such file or directory
make: *** [xPlatST] Error 1
I believe it seems to think that the xPlatST is one of it's compilation files and thus can't find it, but for the life of me I can't work out why.
hwloc is a third party library, and should be unrelated to this issue. The code compiles just fine when compiled from the command line directly.
My files are xPlatST.cpp, xPlatST.h, stdafx.cpp, stdafh.h
Code is as follows:
CXX = g++ -std=c++11
INCLUDES =
LIBS = -L../hwloc
CXXFLAGS = -Wall -g -pthread
OBJS = xPlatST.o stdafx.o
xPlatST: ${OBJS}
${CXX} ${CXXFLAGS} ${INCLUDES} -c $# ${OBJS} ${LIBS}
clean:
-rm xPlatST *.o
Any help would be greatly appreciated; thank you in advance.
Your assumption is correct. Your recipe is trying to use xPlatST as a source. Change the -c into a -o in your rule:
${CXX} ${CXXFLAGS} ${INCLUDES} -o $# ${OBJS} ${LIBS}
The -c flag tells the compiler to take all files, compile, and assemble them into an object file (.o). The -o flag specifies the destination file.
I want to automate the compilation of a toy library using CUDA and C++. Then I write a Makefile as follows
CC=g++
NVCC=nvcc
CXXFLAGS= -fopenmp -O3 -Wextra -std=c++11
CUDAFLAGS= -std=c++11 -c -arch=sm_20
LIBS= -lopenblas -lpthread -lcudart -lcublas
LIBDIRS=-L/usr/local/cuda-7.5/lib64
INCDIRS=-I/usr/local/cuda-7.5/include
matrix_cuda.o: marix_cuda.cu
$(NVCC) $(CUDAFLAGS) matrix_cuda.cu
all: matrix_cuda.o
$(CC) -o test matrix_blas.cpp alg.cpp test.cpp matrix_cuda.o $(LIBDIRS) $(INCDIRS) $(LIBS) $(CXXFLAGS)
clean:
rm -rf test *.o
Typing make I get
make: *** No rule to make target `marix_cuda.cu', needed by `matrix_cuda.o'. Stop.
I never wrote a Makefile before. Where did I go wrong?
I think you have a typo in the CUDA file name
matrix_cuda.o: marix_cuda.cu
$(NVCC) $(CUDAFLAGS) matrix_cuda.cu
IMHO it should be
matrix_cuda.o: matrix_cuda.cu
$(NVCC) $(CUDAFLAGS) matrix_cuda.cu
This may take a couple of iterations.
1) First try this:
nvcc -std=c++11 -c -arch=sm_20 matrix_cuda.cu
If that works (and produces matrix_cuda.o, I presume), remove matrix_cuda.o and
2) try this makefile:
matrix_cuda.o: matrix_cuda.cu
nvcc -std=c++11 -c -arch=sm_20 matrix_cuda.cu
If that works,
3) try this:
g++ -o test matrix_blas.cpp alg.cpp test.cpp matrix_cuda.o -L/usr/local/cuda-7.5/lib64 -I/usr/local/cuda-7.5/include -lopenblas -lpthread -lcudart -lcublas -fopenmp -O3 -Wextra -std=c++11
If that works, remove test and
4) try this makefile:
test: matrix_cuda.o
g++ -o test matrix_blas.cpp alg.cpp test.cpp matrix_cuda.o -L/usr/local/cuda-7.5/lib64 -I/usr/local/cuda-7.5/include -lopenblas -lpthread -lcudart -lcublas -fopenmp -O3 -Wextra -std=c++11
matrix_cuda.o: matrix_cuda.cu
nvcc -std=c++11 -c -arch=sm_20 matrix_cuda.cu
If that works, remove test and matrix_cuda.o and
5) try that makefile again.
If that works, there are further refinements we can make.
Also your first make rule corresponds to compiling the object file matrix_cuda.o. The all make rule should come first since the first rule in the make file is the one that gets updated first when invoking the command make. GNU Make has great documentation explaining how to make simple to complex makefiles. You can check it out here:
https://www.gnu.org/software/make/manual/make.html.
Also another issue you are going to run into is that in your make recipe for the all rule, you are supposed to be linking together only object files to create the final executable. However you are trying to include matrix_blas.cpp alg.cpp test.cpp in this linking step. Instead of .cpp versions of these files they need to be .o versions (the compiled objects). Make can generate these object files for you. You just need to have a make rule and recipe for each one. For example:
matrix_blas.o: matrix_blas.cpp
$(CC) $(CXXFLAGS) -c matrix_blas.cpp -o matrix_blas.o
I want to add some commentary on this Makefile for future reference and better automation :
1. NVCC=nvcc environment variable is superfluous. There is only one compiler for NVIDIA GPUs and you would nonetheless have to change a lot of flags to compile for other architectures like AMD.
-arch=native is better suited if you want to deploy your code on multiple machines with GPUs having different architectures
-dc is the flag for separate compilation. The architecture must be specified before -dc see (with a generic Makefile): https://developer.nvidia.com/blog/separate-compilation-linking-cuda-device-code/
You should add automatic rule to avoid these typos errors :
%.o: %.cu
nvcc -dc $(CUDAFLAGS) $< -o $#
The Makefile will look for all dependencies ending by .o. If it needs to build one, it looks if it has the corresponding .cu file. I would actually remove the -dc from flags and be explicit here that I want to build object files.
$# is the name of the rule target. $< is the first prerequisite. Only one file at a time can be passed to nvcc with the -dc flag, so here $< is better than $^.
You may add the name of the executable in a variable so that you delete the same executable that you generated (and facilitate name change).
I added a .PHONY rule. This only removes the confusion that happens whenever someone writes a file named clean in the directory.
The Makefile can not differenciate between the file and the make clean rule.
CC=g++
CXXFLAGS= -fopenmp -O3 -Wextra -std=c++11
CUDAFLAGS= -std=c++11 -arch=sm_20
LIBS= -lopenblas -lpthread -lcudart -lcublas
LIBDIRS=-L/usr/local/cuda-7.5/lib64
INCDIRS=-I/usr/local/cuda-7.5/include
PROGRAM= test
%.o: %.cu
nvcc -dc $(CUDAFLAGS) $< -o $#
all: matrix_cuda.o
$(CC) -o $(PROGRAM) matrix_blas.cpp alg.cpp test.cpp matrix_cuda.o $(LIBDIRS) $(INCDIRS) $(LIBS) $(CXXFLAGS)
.PHONY: clean
clean:
rm -rf $(PROGRAM) *.o
I'm following Zed Shaw's tutorial "Learn C the Hard Way" and trying to teach myself c programming language.
On my ubuntu desktop, I encountered the linking problem he mentioned in the note of this post.
That is, when linking a static library with gcc, using a command like this:
gcc -Wall -g -DNDEBUG -lmylib ex29.c -o ex29
The linker fails to find the functions in the lib. To link correctly, I have to change the order of source file and lib to this:
gcc -Wall -g -DNDEBUG ex29.c -lmylib -o ex29
And I'm trying to use the makefile offered by Zed to automate unit test. The makefile looks like this:
TEST_SRC=$(wildcard tests/*_tests.c)
TESTS=$(patsubst %.c,%,$(TEST_SRC))
TARGET=build/libYOUR_LIBRARY.a
tests: CFLAGS += $(TARGET)
tests: $(TESTS)
sh ./tests/runtests.sh
The rest part of the makefile that isn't listed here can build the $(TARGET) lib flawlessly.
The problem is Zed append the lib to the $(CFLAGS) and use the implicit rule to compile the test files which leads to a command like this:
gcc -g -O2 -Wall -Wextra -Isrc -rdynamic -DNDEBUG tests/hashmap_tests.c build/mylib.a -o tests/list_tests
The command fails because of the link problem mentioned before as expected.
The solution I came up with was to write the compilation command explicitly like this so I can change the order:
$(TESTS): $(TARGET)
$(CC) $(CFLAGS) $^ $(TARGET) -o $#
This works fine if there is only one main source file. Unfortunately, I have several out there under the ./tests directory, and a command like this is a total disaster.
My question is, how should I change my makefile to make it work or is there any other way I can do the same work as elegant as expected?
CFLAGS holds compiler flags, like -g -O2. You should not add linker flags to it. CPPFLAGS holds preprocessor flags like -Isrc -DNDEBUG. LDFLAGS holds linker flags, which would include things like -L (capital L) if you need it to find libraries, and -rdynamic. And the LDLIBS variable holds libraries, so you should do this:
CPPFLAGS = -Isrc -DNDEBUG
CFLAGS = -g -O2 -Wall -Wextra
LDFLAGS = -rdynamic
LDLIBS = -lmylib
Now you can use the built-in rules for GNU make to build your program. You can see a list of the build-in rules by running make -p -f/dev/null.
Of course the above are just the default variables make defines and uses with its default rules. You don't have to use them, but in general it's better to follow conventions rather than flaunt them.
I get this error when I do the make:
relocation R_X86_64_32 against `vtable for Torch::MemoryDataSet' can not be used
when making a shared object; recompile with -fPIC
It says that I should recompile with the -fPIC option. I did that, adding
the -fPIC option to CFLAGS and CXXFLAGS, but I still get the same error. Is there any way to solve this? I have seen that this problem is related with the use of a 64-bit machine, and it is true that I am using one.
I had this problem quite a while back and if I remember correctly, the fix was moving the placement of -fPIC just after gcc in the command line. Made absolutely no sense, and less so now, but as I remember, that fixed it.
I encountered the same problem, but it had an extra twist. The answer by #clintm solved it, but I thought I would describe my variation of the problem here for future reference...
Makefile on 32-bit machine:
CXX=g++
CXXFLAGS= -O3 -Wall
...
...
%.o: %.c
$(CXX) $(CXXFLAGS) -fpic -c $<
libmylibrary.so: $(OBJECTS)
$(CXX) -shared -Wl,-soname,$# -o $# $(OBJECTS)
This compiled correctly. But the same Makefile failed when I tried it on a 64-bit machine. I changed "-fpic" to "-fPIC" and it still failed. I changed the object rule to:
%.o: %.c
$(CXX) -fPIC $(CXXFLAGS) -c $<
and it still failed.
Finally, I placed "-fPIC" in the actual compiler variable (so that now "-fPIC" appears in the rule for each object and the rule for the shared library):
CXX=g++ -fPIC
CXXFLAGS= -g -O3 -Wall
...
%.o: %.c
$(CXX) $(CXXFLAGS) -c -o $# $<
libalglib.so: $(OBJECTS)
$(CXX) -shared -Wl,-soname,$# -o $# $(OBJECTS)
And it worked!
Let's say you have some makefile like:
CFLAGS = -g -Wall
SOURCES = $(wildcard *.c)
OBJECTS = ...
TARGET = libmyawesomelib.a
all: $(TARGET) main
just add the -fPIC flag like so:
$(TARGET): CFLAGS += -fPIC
$(TARGET): $(OBJECTS)
.
.
.
so on so forth with the rest of the makefile.
I ran into this problem cross-compiling with the android-ndk toolchain. I ended up having to use
CC="$CROSS/bin/arm-linux-androideabi-gcc -pie --sysroot=$SYSROOT"
Neither -fPIC nor -fPIE worked for me in this situation.
I was cross compiling shadowsocks-libev on a CentOS 7 machine, the same problem happened to me, it works perfectly on my laptop with
CC=mipsel-unknown-linux-uclibc-gcc CXX=mipsel-unknown-linux-uclibc-g++ AR=mipsel-unknown-linux-uclibc-ar RANLIB=mipsel-unknown-linux-uclibc-ranlib make SHARED=1 CFLAGS=-fPIC
but on travis ci, it did not work, I have to add -fPIC to CC and CXX in order to get it to work
CC="mipsel-unknown-linux-uclibc-gcc -fPIC" CXX="mipsel-unknown-linux-uclibc-g++ -fPIC" AR=mipsel-unknown-linux-uclibc-ar RANLIB=mipsel-unknown-linux-uclibc-ranlib make SHARED=1
I had this issue after I upgraded gcc. I had one .o file (sqlite) that hadn't been cleaned by the Makefile and as a result I had this issue (assuming because it was compiled with an older version of gcc). After removing that file and rebuilding this error went away.
if the project you'd like to compile has a correct configure script use like this:
$ ./configure 'CFLAGS=-g -O2 -fPIC ....' --enable-some-thing
so the flag will be all the Makefile's rule ...
few days before i've need an elder ver. of VLC to compile on an x64 machine, it has a nice configure script ;-)