How does CMake detect changed files - makefile

Question

I have a "C"/C++ CMake project which works fine. However, I'm sometimes (re)building on a remote cluster where the time is slightly different. This machine runs Linux and I'm building using make. I'm wondering if there is some make/CMake way to change how the changes to the files are detected, e.g. to MD5 or diff rather than using timestamps. Otherwise I guess I'd either have to endure the constant make clean / make -j cycle or have to change my local time every time I'm working with that particular server.
I was poking CMake documentation to see if there is a flag which would change these settings but found none. How would this work on platforms which have no RTC (e.g. Raspberry)?

Answer 1

Right, so knowing that CMake / make does not do what I want and I don't want the hassle of synchronizing the time of my machine to the target, I came up with the following:
#!/bin/bash
touch src_hash.md5
echo -n make "$#" > mymake.sh
find `pwd`/../src `pwd`/../include -print0 |
while IFS= read -r -d $'\0' f; do
if [[ ! -d "$f" ]]; then
MD5=`md5sum "$f" | awk -v fn="$f" '{ print "\"" fn "\" " $1; }'`
echo $MD5 >> src_hash.md5.new
OLDMD5=`grep -e "^\"$f\"" src_hash.md5`
if [[ "$OLDMD5" == "" ]]; then
echo "$MD5 -- [a new file]"
continue # a new file, make can handle that well on its own
fi
HASH=`echo $MD5 | awk '{ print $2; }'`
OLDHASH=`echo $OLDMD5 | awk '{ print $2; }'`
if [[ "$HASH" != "$OLDHASH" ]]; then
echo "$MD5 -- changed from $OLDHASH"
echo -n " \"--what-if=${f}\"" >> mymake.sh
# this is running elsewhere, can't pass stuff via variables
fi
fi
done
touch src_hash.md5.new
mv src_hash.md5.new src_hash.md5
echo using: `cat mymake.sh`
echo >> mymake.sh # add a newline
chmod +x mymake.sh
./mymake.sh
rm -f mymake.sh
This keeps a list of source file hashes in src_hash.md5 and at each time it runs it compares the current files to those hashes (and updates the list accordingly).
At the end, it calls make, passing any arguments you give to the script (such as -j). It makes use of the --what-if= switch which tells make to act like the given file changed - that way the dependences of build targets on sources / headers are handled elegantly.
You might want to also pass the path to source / include files as arguments so that those wouldn't be hardcoded inside.
Or one more iteration on the said script, using touch to change and restore the file timestamps for situations when make is extra stubborn about not rebuilding anything:
#!/bin/bash
if [[ ! -d ../src ]]; then
>&2 echo "error: ../src is not a directory or does not exist"
exit -1
fi
if [[ ! -d ../include ]]; then
>&2 echo "error: ../include is not a directory or does not exist"
exit -1
fi
echo "Scanning for changed files in ../src and ../include"
touch src_hash.md5 # in case this runs for the first time
rm -f mymaketouch.sh
rm -f mymakerestore.sh
touch mymaketouch.sh
touch mymakerestore.sh
echo -n make "$#" > mymake.sh
CWD="`pwd`"
find ../src ../include -print0 |
while IFS= read -r -d $'\0' f; do
if [[ ! -d "$f" ]]; then
fl=`readlink -f "$CWD/$f"`
MD5=`md5sum "$fl" | awk -v fn="$fl" '{ print "\"" fn "\" " $1; }'`
HASH=`echo $MD5 | awk '{ print $2; }'`
echo $MD5 >> src_hash.md5.new
OLDMD5=`grep -e "^\"$fl\"" src_hash.md5`
OLDHASH=`echo $OLDMD5 | awk '{ print $2; }'`
if [[ "$OLDMD5" == "" ]]; then
echo "$f $HASH -- [a new file]"
continue # a new file, make can handle that well on its own
fi
if [[ "$HASH" != "$OLDHASH" ]]; then
echo "$f $HASH -- changed from $OLDHASH"
echo "touch -m \"$fl\"" >> mymaketouch.sh # will touch it and change modification time
stat "$fl" -c "touch -m -d \"%y\" \"%n\"" >> mymakerestore.sh # will restore it later on so that we do not run into problems when copying newer from a different system
echo -n " \"--what-if=$fl\"" >> mymake.sh
# this is running elsewhere, can't pass stuff via variables
fi
fi
done
echo using: `cat mymake.sh`
echo >> mymake.sh # add a newline
echo 'exit $?' >> mymake.sh
chmod +x mymaketouch.sh
chmod +x mymakerestore.sh
chmod +x mymake.sh
control_c() # run if user hits control-c
{
echo -en "\nrestoring modification times\n"
./mymakerestore.sh
rm -f mymaketouch.sh
rm -f mymakerestore.sh
rm -f mymake.sh
rm -f src_hash.md5.new
exit -1
}
trap control_c SIGINT
./mymaketouch.sh
./mymake.sh
RETVAL=$?
./mymakerestore.sh
rm -f mymaketouch.sh
rm -f mymakerestore.sh
rm -f mymake.sh
touch src_hash.md5.new # in case there was nothing new
mv src_hash.md5.new src_hash.md5
# do it now in case someone hits ctrl+c mid-build and not all files are built
exit $RETVAL
Or even run hashing in parallel in case you are building a large project:
#!/bin/bash
if [[ ! -d ../src ]]; then
>&2 echo "error: ../src is not a directory or does not exist"
exit -1
fi
if [[ ! -d ../include ]]; then
>&2 echo "error: ../include is not a directory or does not exist"
exit -1
fi
echo "Scanning for changed files in ../src and ../include"
touch src_hash.md5 # in case this runs for the first time
rm -f mymaketouch.sh
rm -f mymakerestore.sh
touch mymaketouch.sh
touch mymakerestore.sh
echo -n make "$#" > mymake.sh
CWD="`pwd`"
rm -f src_hash.md5.new # will use ">>", make sure to remove the file
find ../src ../include -print0 |
while IFS= read -r -d $'\0' f; do
if [[ ! -d "$f" ]]; then
fl="$CWD/$f"
(echo `md5sum "$f" | awk -v fn="$fl" '{ print "\"" fn "\" " $1; }'` ) & # parallel, echo is atomic (http://stackoverflow.com/questions/9926616/is-echo-atomic-when-writing-single-lines)
# run in parallel (remove the ampersand if you run into trouble)
fi
done >> src_hash.md5.new # >> is atomic but > wouldn't be
# this is fast
cat src_hash.md5 > src_hash.md5.diff
echo separator >> src_hash.md5.diff
cat src_hash.md5.new >> src_hash.md5.diff
# make a compound file for awk (could also read the other file in awk but this seems simpler right now)
cat src_hash.md5.diff | awk 'BEGIN { FS="\""; had_sep = 0; }
{
if(!had_sep && $1 == "separator")
had_sep = 1;
else {
sub(/[[:space:]]/, "", $3);
if(!had_sep)
old_hashes[$2] = $3;
else {
f = $2;
if((idx = index(f, "../")) != 0)
f = substr(f, idx, length(f) - idx + 1);
if($2 in old_hashes) {
if(old_hashes[$2] != $3)
print "\"" f "\" " $3 " -- changed from " old_hashes[$2];
} else
print "\"" f "\" -- a new file " $3;
}
}
}'
# print verbose for the user only
cat src_hash.md5.diff | awk 'BEGIN { FS="\""; had_sep = 0; }
{
if(!had_sep && $1 == "separator")
had_sep = 1;
else {
sub(/[[:space:]]/, "", $3);
if(!had_sep)
old_hashes[$2] = $3;
else {
if($2 in old_hashes) {
if(old_hashes[$2] != $3)
printf($2 "\0"); /* use \0 as a line separator for the below loop */
}
}
}
}' |
while IFS= read -r -d $'\0' fl; do
echo "touch -m \"$fl\"" >> mymaketouch.sh # will touch it and change modification time
stat "$fl" -c "touch -m -d \"%y\" \"%n\"" >> mymakerestore.sh # will restore it later on so that we do not run into problems when copying newer from a different system
echo -n " \"--what-if=$fl\"" >> mymake.sh
# this is running elsewhere, can't pass stuff via variables
done
# run again, handle files that require change
rm -f src_hash.md5.diff
echo using: `cat mymake.sh`
echo >> mymake.sh # add a newline
echo 'exit $?' >> mymake.sh
chmod +x mymaketouch.sh
chmod +x mymakerestore.sh
chmod +x mymake.sh
control_c() # run if user hits control-c
{
echo -en "\nrestoring modification times\n"
./mymakerestore.sh
rm -f mymaketouch.sh
rm -f mymakerestore.sh
rm -f mymake.sh
rm -f src_hash.md5.new
exit -1
}
trap control_c SIGINT
./mymaketouch.sh
./mymake.sh
RETVAL=$?
./mymakerestore.sh
rm -f mymaketouch.sh
rm -f mymakerestore.sh
rm -f mymake.sh
touch src_hash.md5.new # in case there was nothing new
mv src_hash.md5.new src_hash.md5
# do it now in case someone hits ctrl+c mid-build and not all files are built
exit $RETVAL