I have the following Makefile (simplified):
# how to specify an install target that
# (a) installs the requirements if (and only if) any .txt file has changed but
# (b) without triggering recompilation if any *.in file has changed?
#install: ???
# ???
# this may trigger recompilation
install1: requirements/*.txt
for reqs in requirements/*.txt; do pip install -r $$reqs; done
pip freeze > install1
# this always triggers installation even if requirements/*txt have not been modified
install2:
touch requirements/*.txt
make install1
# compile requirements.in to requirements.txt
compile: requirements/*.txt
requirements/%.txt: requirements/%.in
pip-compile --upgrade --no-annotate $<
make compile (re)compiles any modified .in file into the respective .txt.
make install should (re)install the requirements from the .txt file(s) whenever any of them changes, but should ignore changes to .in files that would trigger a recompilation. Neither install1 nor install2 targets above satisfy both these criteria. Is this possible somehow?
Related
I'm using make, I believe GNU Make on WSL. I'm using it for data science, building on cookiecutter-datascience.
In my mind before I started using make, the point of it was to keep track of which parts of a pipeline have changed, and only rerun the stages in the pipeline after it.
Here's a snippet from the makefile:
## Install Python Dependencies
requirements: test_environment
pip install -U pip setuptools wheel
pip install -r requirements.txt
## Make Dataset
data: requirements
$(PYTHON_INTERPRETER) src/data/make_dataset.py
When I run make data, it doesn't just rerun the data part of the pipeline and the succeeding stages, but it also reruns make requirements and make test_environment. But this is the opposite of what I want. Those stages come before, not after. If I have an expensive pipeline, I obviously don't want to rerun it over and over again.
In my case, I want it so that: If one of the raw (un-preprocessed) data files changes, I want it to rerun the data preprocessing. This should not include things like tracking whether the libraries have changed, because those steps logically preceed the data preprocessing.
You can try this:
## Install Python Dependencies
requirements.done: test_environment.done
pip install -U pip setuptools wheel
pip install -r requirements.txt
touch requirements.done
## Make Dataset
data: requirements.done
$(PYTHON_INTERPRETER) src/data/make_dataset.py
Make compares date of last modification of files. Your requirements, test_environment... are not files, they are what is called "phony" targets. As they don't exist, make tries to build them as soon as they are needed. If you want make to discover that something is up to date and does not need to be rebuilt, you must use files. The proposed solution uses empty, dummy, files (the *.done files), instead of your phony targets. These files are used only to store the date of actions in their last modification times.
Of course, you can use files named requirements, test_environment... if you prefer. The .done extension is just a way to identify these files as dummy markers.
I currently developing an interface for a library as a GNU Octave package. Normally packages in GNU Octave are installed via
pkg install tarball_of_the_package.tar.gz
But compressing the package for each test is more or less time consuming. Now my question is if it is possible to call somehow the pkg install mechanism from a directory which has a valid package structure like in the tarball? Running
pkg install .
from inside this directory yields an error:
unpack: FILETYPE must be "gunzip" for a directory
Even specifying the whole path as
pkg install /path/to/the/package/source
results in the same problem.
At the moment I am using GNU Octave 4.0.0 for my developments.
What most packages have is a Makefile at the root of the package with targets such as install that will handle that for you. See for example the Makefile for the statistics package which allow you to do:
$ hg clone http://hg.code.sf.net/p/octave/statistics
destination directory: statistics
requesting all changes
adding changesets
adding manifests
adding file changes
added 401 changesets with 996 changes to 172 files
updating to branch default
133 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd statistics/
$ make install
Creating package version 1.2.4 release ...
rm -rf "statistics-1.2.4"
hg archive --exclude ".hg*" --exclude "Makefile" --type files "statistics-1.2.4"
chmod -R a+rX,u+w,go-w "statistics-1.2.4"
tar cf - --posix "statistics-1.2.4" | gzip -9n > "statistics-1.2.4.tar.gz"
Installing package locally ...
octave --silent --eval 'pkg ("install", "statistics-1.2.4.tar.gz")'
For information about changes from previous versions of the statistics package, run 'news statistics'.
And of course, there's nothing stopping you from calling make install from the Octave session itself. The statistics package example is nicer because it only has m files. If your package also has code to be compiled, the image package has a more complex, but not by much, Makefile for that.
I have a CMake project with the following directory tree:
build/
assets/
dest/
<other files>
dest is a directory where all installed files should go:
The executable, which goes to dest/ with a simple make, this is controlled with CMAKE_RUNTIME_OUTPUT_DIRECTORY
The assets, located on assets/, which go to dest/ after a make install.
But I don't want to issue make install to copy all files do the dest/ dir: I want a simple make to do this.
In this sense, how do I add the install target to the default one (all)? Or, is there a better way to solve this?
Using the following wont cause recursion. Requires CMake >= 3.15.
add_custom_command(
TARGET ${MY_TARGET} POST_BUILD
COMMAND ${CMAKE_COMMAND} --install ${CMAKE_BINARY_DIR} --config $<CONFIG>
)
Extra : You may want to provide a default (local) install path so this doesn't fail on Windows.
if(CMAKE_INSTALL_PREFIX_INITIALIZED_TO_DEFAULT)
set(CMAKE_INSTALL_PREFIX "install" CACHE PATH "Default install path." FORCE)
endif()
This will execute the install target after building <target_name> (which could be all):
add_custom_command(
TARGET <target_name>
POST_BUILD
COMMAND ${CMAKE_COMMAND} --build ${CMAKE_BINARY_DIR} --target install
)
I've solved this problem by using the following macro to generate data dependency rules for the specified target. These rules require only a make and not a make install.
The macro maps an arbitrary file from your source tree into a "staging" tree, where each tree structure may be different, and you may optionally rename the file. The staged data is kept up-to-date, so if you change it in your source tree then it'll be updated the next time you make.
# Macro used to create dependencies for staging data files (e.g. config files, assets) and keeping them up-to-date.
# The given "source" file is copied (and possibly renamed) to "staged" for the given "target".
#
# It works by creating a rule that creates "staged" by copying "source", then creating a target that depends upon "staged",
# then making the given "target" depend upon the new target. Or in makefile speak:
#
# staged: source
# cp source staged
#
# targetData1: staged
#
# target: <existing dependencies...> targetData1
# <existing commands...>
#
# The intermediate rule is used for parallel build robustness. For details, see:
# http://www.cmake.org/cmake/help/v2.8.12/cmake.html#command:add_custom_command
#
# Example:
# target = myExeTarget
# source = "${CMAKE_CURRENT_SOURCE_DIR}/../../data/images/bush1.png"
# staged = "${STAGING_DATA_DIR}/images/bush1.png"
macro(add_data_dependency target source staged)
add_custom_command(
OUTPUT "${staged}"
COMMAND ${CMAKE_COMMAND} -E copy_if_different "${source}" "${staged}" # Dir(s) will be created if needed.
DEPENDS "${source}"
)
if (NOT DEFINED add_data_dependency_counter)
#message(status "Setting tmp counter......")
set(add_data_dependency_counter "0")
endif()
math(EXPR add_data_dependency_counter "${add_data_dependency_counter} + 1")
#message(status "tmp counter is ${add_data_dependency_counter}......")
set(customTarget "${target}Data${add_data_dependency_counter}")
add_custom_target(${customTarget} DEPENDS "${staged}")
add_dependencies(${target} ${customTarget})
endmacro()
In your case, usage would be something like:
add_data_dependency(myTarget "${CMAKE_CURRENT_SOURCE_DIR}/assets/file1.ext" "${CMAKE_CURRENT_SOURCE_DIR}/dest/file1.ext")
add_data_dependency(myTarget "${CMAKE_CURRENT_SOURCE_DIR}/assets/file2.ext" "${CMAKE_CURRENT_SOURCE_DIR}/dest/file2.ext")
add_data_dependency(myTarget "${CMAKE_CURRENT_SOURCE_DIR}/assets/rename_me.ext" "${CMAKE_CURRENT_SOURCE_DIR}/dest/renamed.ext")
:
:
Then whenever you make, the files will be copied if missing or out-of-date (assuming that myTarget is part of all).
The normal way to use CMake is to create a build directory outside your project and all compiled binaries are put there. When you are finished developing and want to install some binaries into your system then you call make install. In this way you keep your project source folder free from all compiler generated stuff.
Example file directory structure:
my_project/
my_project_build/
from my_project_build you call cmake ../my_project to generate build files. Call make to build it and all binaries will be in my_project_build.
I'm configuring a CMake project from source and build and install it with make, i.e.,
$ ls
./build/ ./source/
$ cd build/
$ cmake ../source/
[...]
$ make
[...]
$ make install
[...]
I noticed now that the project is badly configured in that it contains numerous files with the same name that get overridden in the installation directory, e.g, /usr/local/include/mystring.h gets written to more than once by make install.
Is there a systematic way to detect if files are overridden within the same project?
Well, you can wrap install() calls into your own function, which would store all installed files in cache list variable and check for existing item before actually call install().
You may also check ${PROJECT_BINARY_DIR}/install_manifest.txt file - maybe it would contain duplicates in your case.
I have a library that is built and linked to as part of my project. I want to provide the facility to OPTIONALLY have the library installed system-wide (or wherever ${CMAKE_INSTALL_PREFIX} is set). Otherwise, by default, the project's final build products will be linked statically to the library, and the former get installed, but the library binaries stay in the build directory.
In other words:
$ make
$ make install
will build and install, the programs, but only something like
$ make install.foo
will install the library to ${CMAKE_INSTALL_PREFIX}, building it first if needed.
I have something like this so far (simplified from the actual script, so there might be errors):
INCLUDE_DIRECTORIES( "${CMAKE_CURRENT_LIST_DIR}")
SET (FOO_LIBRARY "foo")
# Following builds library and makes it available to
# be linked other targets within project by:
# TARGET_LINK_LIBRARIES(${progname} ${FOO_LIBRARY})
ADD_LIBRARY(${FOO_LIBRARY}
foo/foo.cpp # and other sources ...
)
###########################################################
# Approach #1
# -----------
# Optionally allow users to install it by invoking:
#
# cmake .. -DINSTALL_FOO="yes"
#
# This works, but it means that users will have to run
# cmake again to switch back and forth between the libary
# installation and non-library installation.
#
OPTION(INSTALL_FOO "Install foo" OFF)
IF (INSTALL_FOO)
INSTALL(TARGETS ${FOO_LIBRARY} DESTINATION lib/foo)
SET(FOO_HEADERS foo/foo.h)
INSTALL(FILES ${FOO_HEADERS} DESTINATION include/foo)
UNSET(INSTALL_FOO CACHE)
ENDIF()
###########################################################
###########################################################
# Approach #2
# -----------
# Optionally allow users to install it by invoking:
#
# make install.foo
#
# Unfortunately, this gets installed by "make install",
# which I want to avoid
SET(FOO_INSTALL "install.foo")
ADD_CUSTOM_TARGET(${FOO_INSTALL}
COMMAND ${CMAKE_COMMAND}
-D COMPONENT=foo
-P cmake_install.cmake)
ADD_DEPENDENCIES(${FOO_INSTALL} ${FOO_LIBRARY})
INSTALL(TARGETS ${FOO_LIBRRARY}
DESTINATION lib/foo COMPONENT foo)
SET(FOO_HEADERS foo/foo.h)
INSTALL(FILES ${FOO_HEADERS}
DESTINATION include/foo COMPONENT foo)
###########################################################
As can be seen, approach #1 sort of works, but the required steps to install the library are:
$ cmake .. -DINSTALL_FOO="yes"
$ make && make install
And then, to go back to "normal" builds, the user has to remember to run cmake again without the "-DINSTALL_FOO" option, otherwise the library will be installed on the next "make install".
The second approach works when I run "make install.foo", but it also install the library if I run "make install". I would like to avoid the latter.
Does anyone have any suggestions on how to go about achieving this?
You are on the right track with approach #2. You can trick CMake into avoiding the installation of the FOO related files by using the OPTIONAL switch of the install command.
For the FOO library target, the commands have to modified in the following way:
SET (FOO_LIBRARY "foo")
ADD_LIBRARY(${FOO_LIBRARY} EXCLUDE_FROM_ALL
foo/foo.cpp
)
INSTALL(TARGETS ${FOO_LIBRARY}
DESTINATION lib/foo COMPONENT foo OPTIONAL)
The EXCLUDE_FROM_ALL is added to ADD_LIBRARY to prevent the library from building when you run a plain make. The installation of FOO_LIBRARY is made optional by adding the OPTIONAL switch.
Making the installation of the FOO_HEADERS optional requires the following changes:
SET(FOO_HEADERS foo/foo.h)
SET(BINARY_FOO_HEADERS "")
FOREACH (FOO_HEADER ${FOO_HEADERS})
ADD_CUSTOM_COMMAND(
OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${FOO_HEADER}
COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_CURRENT_SOURCE_DIR}/${FOO_HEADER} ${CMAKE_CURRENT_BINARY_DIR}/${FOO_HEADER}
DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/${FOO_HEADER})
LIST (APPEND BINARY_FOO_HEADERS ${CMAKE_CURRENT_BINARY_DIR}/${FOO_HEADER})
ENDFOREACH()
INSTALL(FILES ${BINARY_FOO_HEADERS}
DESTINATION include/foo COMPONENT foo OPTIONAL)
The FOREACH loop sets up custom commands which copy the FOO_HEADERS verbatim to the corresponding binary dir. Instead of directly using the headers in the current source dir the INSTALL(FILES ... command picks up the copied headers from the binary dir. The paths of the headers in the binary dir are collected in the variable BINARY_FOO_HEADERS.
Finally the FOO_INSTALL target has to be set up in the following way:
SET(FOO_INSTALL "install.foo")
ADD_CUSTOM_TARGET(${FOO_INSTALL}
COMMAND ${CMAKE_COMMAND}
-D COMPONENT=foo
-P cmake_install.cmake
DEPENDS ${BINARY_FOO_HEADERS})
ADD_DEPENDENCIES(${FOO_INSTALL} ${FOO_LIBRARY})
The custom FOO_INSTALL is added with a dependency on BINARY_FOO_HEADERS to trigger the copying of the header files. A target level dependency on FOO_LIBRARY triggers the building of the library when you run make install.foo.
The solution however has the following drawbacks:
Upon configuring CMake will issues a warning Target "foo" has EXCLUDE_FROM_ALL set and will not be built by default but an install rule has been provided for it. CMake will however do the right thing anyway.
Once you have run make install.foo a subsequent make install command will also install all FOO related files, because the built FOO library and the headers exist in the binary directory.
Check if this solution works for you.
You would have to use different commands to those you said you would like to use, but I think it addresses the problem fairly well.