I want to run Terraform fmt command only if my .tf/.tpl files in the current directory or subdirectories have changed. Here is my makefile, not sure if this is even possible with makefile or not:
TF_VERSION=0.10.7
.PHONY: format
README.md: ./variables.tf
terraform-docs md . > README.md
format: $(wildcard *.tf) $(wildcard */*.tf) $(wildcard */*.tpl)
docker run -ti -v $(shell pwd):/tmp hashicorp/terraform:${TF_VERSION} fmt /tmp
help:
#grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'
.DEFAULT_GOAL := format
The problem is that make runs format every time (even without any changes), and if I remove the last line, I have to run make format otherwise it just runs the first rule.
Use an empty file, whose modification time records the last time the rule was run:
format_timestamp: $(wildcard *.tf) ...
docker run -ti -v ...
touch $#
Related
I'm using GNU Make to build a dynamic web site but I need to build two versions. As a net result currently I run my makefile using two command line incantations. This is inefficient and can result in errors (I don't always remember to run both versions or make a typing error and they fail) thus I want to make it one build.
The Command Line incantations are:
sudo make release build=test
sudo make release build=release
The two incantations activate ifeq blocks that set the path and modify some files.
Part of the much simplified (to help readability) top level makefile:
subs = today tomorrow
files = index.php foot.php
ifeq ($(build),test)
export path = /var/www/html/cgi-test
$(shell cp -f head-test.php head.php)
$(shell sed -i '/"DB"/c\ define("DB", "db_test");' subs.php)
else ifeq ($(build),release)
export path = /var/www/html/cgi-new
$(shell cp -f head-release.php head.php)
$(shell sed -i '/"DB"/c\ define("DB", "db_new");' subs.php)
endif
.PHONY: all go
all:
$(MAKE) go
#for ALL in $(subs);\
do $(MAKE) -C $$ALL all || exit $$?;\
done;
go:
cp $(files) $(path)/.
The sub-makefiles have a very similar structure but don't need the ifeq blocks because the files and paths have already been setup.
I think I can simply move the shell commands into the .PHONY rules but I can't do that with the exports because I get errors "export: : bad variable name".
I could do it with a batch file and call the makefile twice but that sidesteps the problem rather than cures it and I wish to learn from the process.
So can anybody show me the way to do this in a makefile?
Thanks to Tripleee here is the answer that finally worked back ported to match my starting post. The one major change is that I have gone back to 'all' as the rule I expect to start the build habits die hard! - Thanks
.PHONY: all both rel-test rel-release
cgi-test := cgi-test
db-test := db_test
cgi-release := cgi-new
db-release := db_new
subs = today tomorrow
files = index.php foot.php
all: both
both: rel-test rel-release
rel-test rel-release: rel-%:
cp -f head-$*.php head.php
sed -i '/"DB"/c\ define("DB", "$(db-$*)");' subs.php
$(MAKE) go path=/var/www/html/strutts/$(cgi-$*)
#for ALL in $(subs);\
do $(MAKE) build=$* path=/var/www/html/strutts/$(cgi-$*) -C $$ALL all || exit $$?;\
done;
Something like this?
.PHONY: both rel-test rel-release
both: rel-test rel-release
cgi-test := cgi-test
db-test := db_test
cgi-release := cgi-new
db-release := db_new
rel-%:
cp -f head-$*.php head.php
sed -i '/"DB"/c\ define("DB", "$(db-$*)")' subs.php
$(MAKE) release build=$* path=/var/www/html/$(cgi-$*)
The reason the export can't be moved into a recipe is that you are using the export command of make itself, not the shell's command with the same name.
You absolutely should not use sudo unless you specifically require the output files to be owned and only writable by root. Even then, running as much as possible as a regular user would be proper hygiene; maybe add a sudo command inside the Makefile to copy the files to their final location.
I am trying to write a simple Makefile to build .expected files and compare them but I am failing.
APSSCHED=../../bin/apssched
BASE=.:../../base:../../examples
FLAGS=-DCOT
EXAMPLES=../../examples/
CASES=simple-binding1 simple-binding2
# skipping lines doesn't work ...
# run command and skip the first line
%.aps:
${APSSCHED} ${FLAGS} -p ${BASE} ${EXAMPLES}/$* | tail -n +2
# get all cases as an array to pipe it to different make targets
# maybe overcomplicating
cases:
echo ${CASES} | \
awk '{split($$0,numbers," ")} END {for(n in numbers){ print numbers[n] }}'
# create all .expected files from ${CASES}
build.expected:
$(MAKE) cases | xargs -n1 -I file /bin/bash -c '$(MAKE) file.build.expected'
# create single .expected file
%.build.expected:
$(MAKE) $*.aps > $*.expected
# compare result with
%.compare:
$(MAKE) $*.aps | diff $*.expected -
# run command for all cases and diff the result with corresponding expected
all:
$(MAKE) cases | xargs -n1 -I file /bin/bash -c '$(MAKE) file.compare'
clean.expected:
rm *.expected
Running make without any target and nothing happens.
echo simple-binding1 simple-binding2 | \
awk '{split($0,numbers," ")} END {for(n in numbers){ print numbers[n] }}'
simple-binding1
simple-binding2
I think the issue is with my cases target. I am not sure if I am on the right track.
I appreciate any help or hint.
I would avoid re-running make just to call a different target - it's a performance hit and may be unreliable (depending on rest of the Makefile) since separate calls may not be able to track dependencies correctly.
Moreover, I would avoid using | - every time a command is concatenated with pipe, exit code of piped command would be exit code of the last command. So a call like command | tail would return the exit code of tail (which would almost always succeed). Even if the command has failed, it would be covered with exit code 0 from tail and make will not detect the error and will not stop.
Thus said, I tried to rewrite your approach by just creating dependencies between the targets, like so:
$ cat Makefile
APSSCHED=../../bin/apssched
EXAMPLES=../../examples
BASE=.:../../base:$(EXAMPLES)
FLAGS=-DCOT
CASES=simple-binding1 simple-binding2
# Just for reproducing
$(EXAMPLES)/%.aps: ;
# Generate output and store it in a file
%.output: $(EXAMPLES)/%.aps
# echo is only for reproducing
echo $(APSSCHED) $(FLAGS) -p $(BASE) $< > $#
# Copy actual output as expected
%.expected: %.output
cp -f $< $#
# Compare actual output with expected
.PHONY: %.compare
%.compare: %.output | %.expected
diff $| $<
# Generate and verify all outputs
.PHONY: all
all: $(addsuffix .compare,$(CASES))
# Regenerate expected output
.PHONY: build.expected
build.expected: $(addsuffix .expected,$(CASES))
.PHONY: clean.expected
clean.expected:
-rm -f *.expected
Now the make build.expected will create expected output files, while make all or make will check the actual output against expected:
$ make build.expected
echo ../../bin/apssched -DCOT -p .:../../base:../../examples ../../examples/simple-binding1.aps > simple-binding1.output
cp -f simple-binding1.output simple-binding1.expected
echo ../../bin/apssched -DCOT -p .:../../base:../../examples ../../examples/simple-binding2.aps > simple-binding2.output
cp -f simple-binding2.output simple-binding2.expected
rm simple-binding1.output simple-binding2.output
$ make
echo ../../bin/apssched -DCOT -p .:../../base:../../examples ../../examples/simple-binding1.aps > simple-binding1.output
diff simple-binding1.expected simple-binding1.output
echo ../../bin/apssched -DCOT -p .:../../base:../../examples ../../examples/simple-binding2.aps > simple-binding2.output
diff simple-binding2.expected simple-binding2.output
rm simple-binding1.output simple-binding2.output
I have targets identified by a version number (7.0 7.1 7.2)
I have some templates that have a placeholder that needs to be replaced for each version such that I end up with a directory for each version with all the files in each directory having the appropriate replacements.
So, if the template directory is :
template\Dockerfile
I want to end up with :
7.0\Dockerfile
7.1\Dockerfile
7.2\Dockerfile
with the the tag in the template\Dockerfile being replaced with the version number.
This is a simplified example. There are actually 4 files in the template folder (and 2 are in a subdirectory).
A pipeline will run make build-targets and then the appropriate docker commands to build the containers and push them to the repository - that's the goal.
Whilst it would be nice to have an answer to all of this, I would like to learn, but I can't find anything about how to deal with sets of sources and sets of targets.
Any advice would be appreciated.
More details in response to comments
The versions are currently just 1 line in the Makefile: VERSIONS := 7.0 7.1 7.2 latest
The code to run will be a series of sed commands to take a template and replace a #version# tag in the file with the version number (except for latest in which case it will just remove the tag.
The template sources are all in the templates directory. As part of the sed command, the resulting filename will have the part templates replaced with the version (including latest).
A start ... (no idea if this is good or not - still learning)
VERSIONS := 7.0 7.1 7.2 latest
SOURCEDIR := ./templates
DESTDIR := ./
TEMPLATES := $(shell find $(SOURCEDIR) -type f)
# For each file in $(TEMPLATES) for each version in $(VERSIONS), create a corresponding file in $(DEST_DIR)/$(VERSION)
# Replace `#version#` with $(VERSION) in each file, except for the `latest` target where the `#version#` removed.
Based upon solutions provide below, my final code is:
VERSIONS := 7.0 7.1 7.2 latest
SOURCE_DIR := ./templates
TEMPLATES := $(shell find $(SOURCE_DIR) -type f)
TEMPLATES := $(patsubst $(SOURCE_DIR)/%,%,$(TEMPLATES))
DEST_DIR := ./
.DEFAULT_GOAL := all
.PHONY: all
# $(1): version
# $(2): template file (without the $(SOURCE_DIR)/ stem)
define generate_target_file
$(DEST_DIR)/$(1)/$(2): $(SOURCE_DIR)/$(2)
#echo "Making $$#"
#mkdir -p $$(dir $$#)
#if [ "$(1)" == "latest" ]; then \
sed 's/#version#//g' $$< > $$#; \
else \
sed 's/#version#/$(1)-/g' $$< > $$#; \
fi
all: $(DEST_DIR)/$(1)/$(2)
endef
$(foreach version,$(VERSIONS),$(foreach template,$(TEMPLATES),$(eval $(call generate_target_file,$(version),$(template)))))
list:
#echo 'The following targets are available :'
#$(MAKE) -pRrq -f $(lastword $(MAKEFILE_LIST)) : 2>/dev/null | \
awk -v RS= -F: '/^# File/,/^# Finished Make data base/ {if ($$1 !~ "^[#.]") {print $$1}}' | \
sort | \
egrep -v -e '^[^[:alnum:]]' -e '^$#$$' | \
xargs -0
If you are using GNU-make, you can easily create loops with make's foreach function. And you can nest them. The eval function can be used to programmatically instantiate make constructs and the call function allows you to create kind of macros:
V := 7.0 7.1 7.2 latest
S := ./templates
T := $(shell find $(S) -type f)
T := $(patsubst $(S)/%,%,$(T))
D := ./destdir
.DEFAULT_GOAL := all
.PHONY: all
# $(1): version
# $(2): template file (without the $(S)/ stem)
# $(3): replacement string
define MY_rule
$(D)/$(1)/$(2): $(S)/$(2)
mkdir -p $$(dir $$#)
sed 's/#version#/$(3)/g' $$< > $$#
all: $(D)/$(1)/$(2)
endef
$(foreach v,$(V),\
$(foreach t,$(T),\
$(eval $(call MY_rule,$(v),$(t),$(patsubst latest,,$(v))))))
The GNU make manual explains how this works. Beware the double expansion of the MY_rule macro, reason why some $ signs must be doubled.
I want make to remove all files except the source files and the make rule file (i.e. the file named makefile), so I added a phony rule at the end of my makefile:
.PHONY:clean
clean:
$(shell ls | grep -v "[.][ch]" | grep -v makefile | xargs rm)
This does what I intend. But make always complains
make: Nothing to be done for 'clean'.
After I run make clean. Why does this message appear? And how can I make it disappear?
The use of $(shell ...) is unnecessary. It runs the command, then the output is used as if it was part of the Makefile. There is no output, so the resulting rule is:
clean:
i.e. the actual list of commands to update the clean target is empty.
I need help: the following command line is not working
TESTS := $(shell cat test_cases_file | egrep -v ^\s*(#|$) )
all: $(TESTS)
when I launch:
make all
I get an error something like "call of shell command is not finished."
You're missing quotes in egrep parameter. If I were make I'd too believe that # is the beginning of comment.
Apparently simply adding quotes won't help, you'll need to escape the # too.
edit:
Actually $ has to be escaped too and in case of make it is done using $$
Try this:
TESTS := $(shell egrep -v '^\s*(\#|$$)' test_cases_file)
all: $(TESTS)
Finally, you don't really need to discard blank lines - that won't hurt:
TESTS := $(shell egrep -v '^\s*\#' test_cases_file)
all: $(TESTS)
What you've stumbled upon is probably the fact, that parentheses are not escapeable in GNU Make.
You can work around this problem by using helper script:
TESTS := $(shell ./grep.sh test_cases_file )
all: $(TESTS)
Where grep.sh is:
cat $1 | egrep -v ^\s*(#|$)