Terraform and Github Actions - makefile

I have a Makefile to compile Terraform code. When I am running Makefile up locally, everything is working correctly.
Makefile
BACKEND_CONFIG = -backend-config=resource_group_name=${AZURE_TERRAFORM_RESOURCE_GROUP_NAME} \
-backend-config=storage_account_name=${AZURE_TERRAFORM_STORAGE_ACCOUNT_NAME} \
-backend-config=container_name=${AZURE_TERRAFORM_STORAGE_CONTAINER_NAME} \
-backend-config=key=${AZURE_TERRAFORM_RESOURCE_GROUP_NAME}.tfstate \
VARIABLES = -var=path=${PWD}/config/resource-groups \
all:
up: init plan apply
down: init destroy
init:
terraform init -reconfigure ${BACKEND_CONFIG} src
plan:
terraform plan ${VARIABLES} -out="plan.out" src
apply:
terraform apply plan.out
destroy:
terraform destroy ${VARIABLES} src
.PHONY = all init plan apply destroy up down import az_up az_down
What I am trying to do now is to run Makefile up whenever I am making a push request. I added all of my secrets in Github secrets and created a Github actions.
terraform.yml
on:
push:
branches:
- dev
pull_request:
jobs:
terraform:
name: 'Terraform'
runs-on: ubuntu-latest
environment: production
defaults:
run:
shell: bash
steps:
- uses: actions/checkout#v2
- name: build application
run: make up
working-directory: ./infra_as_code
Makefile (new)
BACKEND_CONFIG = -backend-config=resource_group_name=${{ secrets.AZURE_TERRAFORM_RESOURCE_GROUP_NAME }} \
-backend-config=storage_account_name=${{ secrets.AZURE_TERRAFORM_STORAGE_ACCOUNT_NAME }} \
-backend-config=container_name=${{ secrets.AZURE_TERRAFORM_STORAGE_CONTAINER_NAME }} \
-backend-config=key=${{ secrets.AZURE_TERRAFORM_RESOURCE_GROUP_NAME }}.tfstate \
VARIABLES = -var=path=${PWD}/config/resource-groups \
all:
up: init plan apply
down: init destroy
init:
terraform init -reconfigure ${BACKEND_CONFIG} src
plan:
terraform plan ${VARIABLES} -out="plan.out" src
apply:
terraform apply plan.out
destroy:
terraform destroy ${VARIABLES} src
.PHONY = all init plan apply destroy up down import az_up az_down
error I am getting in Github Actions:
Run make up
make up
shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
terraform init -reconfigure -backend-config=resource_group_name=} -backend-config=storage_account_name=} -backend-config=container_name=} -backend-config=key=}.tfstate src
Too many command line arguments. Did you mean to use -chdir?
make: *** [Makefile:15: init] Error 1
Error: Process completed with exit code 2.

This is definitely wrong:
BACKEND_CONFIG = -backend-config=resource_group_name=${{ secrets.AZURE_TERRAFORM_RESOURCE_GROUP_NAME }} \
-backend-config=storage_account_name=${{ secrets.AZURE_TERRAFORM_STORAGE_ACCOUNT_NAME }} \
-backend-config=container_name=${{ secrets.AZURE_TERRAFORM_STORAGE_CONTAINER_NAME }} \
-backend-config=key=${{ secrets.AZURE_TERRAFORM_RESOURCE_GROUP_NAME }}.tfstate \
First, if you want to put explicit $ characters in a makefile recipe you need to escape them with $$. That's why you're getting this error: make is trying to expand these as make variables.
Second, if you want to pass these through the shell, you have to quote them for the shell. I'm assuming that you want the literal string ${{ ... }} to appear, in which case you want single quotes:
BACKEND_CONFIG = -backend-config=resource_group_name='$${{ secrets.AZURE_TERRAFORM_RESOURCE_GROUP_NAME }}' \
-backend-config=storage_account_name='$${{ secrets.AZURE_TERRAFORM_STORAGE_ACCOUNT_NAME }}' \
-backend-config=container_name='$${{ secrets.AZURE_TERRAFORM_STORAGE_CONTAINER_NAME }}' \
-backend-config=key='$${{ secrets.AZURE_TERRAFORM_RESOURCE_GROUP_NAME }}.tfstate'
(are you sure it's OK to add this whitespace after {{ and before }}?)
I know nothing about terraform or github actions so I have no idea if this syntax is what it wants, but it will pass through both make and the shell correctly.

${{ }} are replaced by github actions before the actual execution. Therefore that will not work within the Makefile.
Best option would be to use Make variables in the makefile and set them when calling Make:
BACKEND_CONFIG = -backend-config=resource_group_name=$(AZURE_TERRAFORM_RESOURCE_GROUP_NAME) \
-backend-config=storage_account_name=$(AZURE_TERRAFORM_STORAGE_ACCOUNT_NAME) \
-backend-config=container_name=$(AZURE_TERRAFORM_STORAGE_CONTAINER_NAME) \
-backend-config=key=$(AZURE_TERRAFORM_RESOURCE_GROUP_NAME).tfstate \
Then in github actions:
- name: build application
run: make AZURE_TERRAFORM_RESOURCE_GROUP_NAME=${{ secrets.AZURE_TERRAFORM_RESOURCE_GROUP_NAME }} \
AZURE_TERRAFORM_STORAGE_ACCOUNT_NAME=${{ secrets.AZURE_TERRAFORM_STORAGE_ACCOUNT_NAME }} \
AZURE_TERRAFORM_STORAGE_CONTAINER_NAME=${{ secrets.AZURE_TERRAFORM_STORAGE_CONTAINER_NAME }} \
AZURE_TERRAFORM_RESOURCE_GROUP_NAME=${{ secrets.AZURE_TERRAFORM_RESOURCE_GROUP_NAME }} \
up
working-directory: ./infra_as_code
I'm not a real Makefile expert, but as far as I know variables are used with $(VAR), not with '{' in make. But maybe ${} is also working.

Related

Is there a Buildroot setting for making kernel source available to compile against?

I've been given a .zip file containing source for a proprietary kernel module. Once unzip'd, there is an install script that needs to be run. The install script untar's the actual source and builds the kernel module. It requires kernel headers to compile against.
Here is my Buildroot .mk file:
FOOCO_VERSION = 1.0
FOOCO_SOURCE = cust_kernel_drvr.zip
FOOCO_SITE = /mnt/third-party/fooco
FOOCO_SITE_METHOD = local
define FOOCO_CONFIGURE_CMDS
unzip $(#D)/$(FOOCO_SOURCE) -d $(#D)
endef
define FOOCO_BUILD_CMDS
chmod +x $(#D)/TOOLS/Linux_x64/DRIVER/install
cd $(#D)/TOOLS/Linux_x64/DRIVER; $(SHELL) ./install
rm -rf $(#D)
endef
$(eval $(generic-package))
This results in the following log output and error:
(Note: I enabled debugging that shows the start and end of each step.)
DEBUG: start | rsync | fooco
>>> fooco 1.0 Syncing from source dir /mnt/third-party/fooco
rsync -au --chmod=u=rwX,go=rX --exclude .svn --exclude .git --exclude .hg --exclude .bzr --exclude CVS /mnt/third-party/fooco/ /root/buildroot-2022.02.1/output/build/fooco-1.0
DEBUG: end | rsync | fooco
DEBUG: start | configure | fooco
>>> fooco 1.0 Configuring
unzip /root/buildroot-2022.02.1/output/build/fooco-1.0/cust_kernel_drvr.zip -d /root/buildroot-2022.02.1/output/build/fooco-1.0
Archive: /root/buildroot-2022.02.1/output/build/foofo-1.0/cust_kernel_drvr.zip
[snip]
creating: /root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/
creating: /root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER/
inflating: /root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER/install
inflating: /root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER/cust_kernel_drvr-1.2.0.15-0.noarch.rpm
inflating: /root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER/cust_kernel_drvr.tar.gz
inflating: /root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER/license_gpl.txt
[snip]
DEBUG: end | configure | fooco
DEBUG: start | build | fooco
>>> fooco 1.0 Building
chmod +x /root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER/install
cd /root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER; /bin/bash ./install
Extracting archive..OK!
Compiling the driver...Error: make[1]: Entering directory '/root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER/fooco_cust/src/linux/driver'
common.mk:82: *** Kernel header files not in any of the expected locations.
common.mk:83: *** Install the appropriate kernel development package, e.g.
common.mk:84: *** kernel-devel, for building kernel modules and try again. Stop.
make[1]: Leaving directory '/root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER/fooco_cust/src/linux/driver'
Error: unable to find driver file (fooco_cust.ko) in /root/buildroot-2022.02.1/output/build/fooco-1.0/TOOLS/Linux_x64/DRIVER/fooco_cust/src/linux/driver
rm -rf /root/buildroot-2022.02.1/output/build/fooco-1.0
DEBUG: end | build | fooco
touch: cannot touch '/root/buildroot-2022.02.1/output/build/fooco-1.0/.stamp_built': No such file or directory
make: *** [/root/buildroot-2022.02.1/output/build/fooco-1.0/.stamp_built] Error 1
package/pkg-generic.mk:289: recipe for target
'/root/buildroot-2022.02.1/output/build/fooco-1.0/.stamp_built' failed
I found that the make files that came with the kernel module are looking in several places for the kernel headers:
/lib/modules/${BUILD_KERNEL}/source \
/lib/modules/${BUILD_KERNEL}/build \
/usr/src/linux-${BUILD_KERNEL} \
/usr/src/linux-$(${BUILD_KERNEL} | sed 's/-.*//') \
/usr/src/kernel-headers-${BUILD_KERNEL} \
/usr/src/kernel-source-${BUILD_KERNEL} \
/usr/src/linux-$(${BUILD_KERNEL} | sed 's/\([0-9]*\.[0-9]*\)\..*/\1/') \
/usr/src/linux \
/usr/src/kernels/${BUILD_KERNEL} \
/usr/src/kernels
Why is the kernel source not visible to this build? I thought that, since Buildroot is building the kernel as part of the overall process, the header files would be available for subsequent kernel module compiles. Am I missing a setting? I feel that I'm not understanding the Buildroot process in a basic way, even after referring to the manual many times.
I'm using Buildroot 2022.02.1 and kernel 5.15.33.
Your download/extract logic is very convoluted. You should really use something like this:
FOO_SITE = /mnt/third-party/fooco
FOO_SOURCE = cust_kernel_drvr.zip
FOO_SITE_METHOD = file
define FOO_EXTRACT_CMDS
unzip $(FOO_DLDIR)/$(FOOCO_SOURCE) -d $(#D)
endef
Regarding the build issue: it is impossible to help without studying the specific build system of this kernel module. Very likely you will need to pass some environment variables to tell the build system where your kernel source code is located, and possibly other things. But without looking at the specific details, it's impossible to help you.
You can have a look at how standard out of tree kernel modules are handled by looking at the package/pkg-kernel-module.mk code. However, that will not be directly useful to a package like yours that uses a custom installation script.
The magic was the LINUX_DIR variable which, according to Buildroot user manual:
contains the path to where the Linux kernel has been extracted and built.
I was able to patch the install script to send this variable to the make file that was looking for the kernel.

ldflags is ignored when building go project

I want to have multiple binaries in one repository, but also set the version via ldflags option.
With just one binary in a repository I have no problem, it works, but with the new structure for multiple binaries it doesn't seem to work.
I have set up a simple project on github .
The structure is simple
cmd/
- server/main.go
- service/main.go
libcommon/
- version.go
- ...
go.mod
Makefile
version.go
package libcommon
var (
Version = "dev"
Build = "now"
)
Makefile
BUILDDIR = bin
VERSION := $(shell git describe --tags --always --dirty)
BUILD := $(shell date +%Y-%m-%d\ %H:%M)
LDFLAGS=-ldflags="-w -s -X 'libcommon.Version=${VERSION}' -X 'libcommon.Build=${BUILD}'"
go build ${LDFLAGS} -o $(BUILDDIR)/ ./...
I call make install and the binary is put into bin/ directory, but when I run it it just prints out the default values, not the ones I'd assume to be in there.
Any idea on how I can get to set the version with the ldflags in this layout?
Thanks in advance.
To correctly set the variable with -ldflags you have to qualify the variable name with the full package import path:
In Makefile:
LDFLAGS=-ldflags="-w -s \
-X 'mymodule.com/path/to/libcommon.Version=${VERSION}' \
-X 'mymodule.com/path/to/libcommon.Build=${BUILD}'"
build:
go build ${LDFLAGS} -o $(BUILDDIR)/ ./...```

Multi-dependency makefile target

The problem I'm experiencing is an all target has dependencies on others that set a variable, then run matching dependencies.
Outcome - It will run the first dependency then stop.
Expected - Run both dependencies, setting the variable properly between each run
Is make smart enough to see pull and build were already ran and the dependency target itself has no execution, therefore it sees all dependencies as complete? Or I'm just abusing make in ways it should not be used?
Said makefile:
repo=rippio
image=default
pull:
#docker pull $(repo)/$(image):latest
build: pull
#sed -e 's/{{repo}}/$(repo)/' -e 's/{{image}}/$(image)/' Dockerfile.in > Dockerfile && \
docker build -t=$(repo)/$(image):custom .
#rm -f Dockerfile
node: image=node
node: | pull build
jdk8: image=jdk8
jdk8: | pull build
all: | node jdk8
TLDR
It is used to:
Pull the latest docker image
Run a generically designed Dockerfile against it to customize it
Tag it as :custom for internal use
Pretty handy for customizing images in a generic manner without managing many Dockerfiles.
Dockerfile template (Dockerfile.in), incase interested:
FROM {{repo}}/{{image}}:latest
... super secret sauce
UPDATE (ANSWER)
Thanks to #G.M., ended up with:
IMAGE_NAMES := node jdk8
TARGETS := $(patsubst %,build-%,$(IMAGE_NAMES))
repo=rippio
all: $(TARGETS)
build-%: pull-%
#$sed -e 's/{{repo}}/$(repo)/' -e 's/{{image}}/$*/' Dockerfile.in > Dockerfile-$* && \
$docker build -f=Dockerfile-$* -t=$(repo)/$*:custom .
#rm -f Dockerfile-$*
pull-%:
#$docker pull $(repo)/$*:latest
Which allows for:
Easy upkeep of 'all' targets, which constantly grows
Running parallel via make -j (note the Dockerfile-$* file pattern)
Much more beautiful than before
If you draw your dependency graph out long-hand you'll see that there are multiple paths from all to both pull and build -- one via each of node and jdk8. But make having reached/updated pull and build via one path will then assume that they are both up to date and, hence, not bother to update them further -- regardless of any change to target specific variables.
I think what you're trying to do (assuming I've understood correctly) might be more easily achieved using pattern rules.
IMAGE_NAMES := node jdk8
TARGETS := $(patsubst %,build-%,$(IMAGE_NAMES))
repo=rippio
all: $(TARGETS)
build-%: pull-%
#$sed -e 's/{{repo}}/$(repo)/' -e 's/{{image}}/$*/' Dockerfile.in > Dockerfile && \
$docker build -t=$(repo)/$*:custom .
#rm -f Dockerfile
pull-%:
#$docker pull $(repo)/$*:latest
Note: You currently have all build recipes using the same input/output file DockerFile. That will cause problems if you ever want to use parallel builds -- make -j etc. It might be wise to use the stem from the pattern rule match to uniquely identify the output file if that's possible.
Normally, if you invoke make with:
make all
and if none of pull, build, node, jdk8 are existing files, make should build pull and build. If you see only pull being made, it can be because you invoke make without specifying a goal. In this case make builds the first target it finds in the Makefile (pull in your case).
Anyway, there are several strange aspects in your Makefile: you use order-only prerequisites on what looks like phony targets and these phony targets are not declared as such.
I am not sure I fully understand what you are trying to do but maybe something like this would be a good starting point:
repo=rippio
image=default
.PHONY: all build node jdk8
all: node jdk8
node: image = node
jdk8: image = jdk8
build node jdk8:
#docker pull $(repo)/$(image):latest && \
sed -e 's/{{repo}}/$(repo)/' -e 's/{{image}}/$(image)/' Dockerfile.in > Dockerfile && \
docker build -t=$(repo)/$(image):custom . && \
rm -f Dockerfile
Note: if, instead of build you name the default target default you could even simplify further with:
repo=rippio
.PHONY: all default node jdk8
all: node jdk8
default node jdk8:
#docker pull $(repo)/$#:latest && \
sed -e 's/{{repo}}/$(repo)/' -e 's/{{image}}/$#/' Dockerfile.in > Dockerfile && \
docker build -t=$(repo)/$#:custom . && \
rm -f Dockerfile

How to automatically generate a Makefile help command

I recently found this article online, that explains how to setup a make help target that will automatically parse the Makefile for comments and display a nicely formatted help command.
It parses:
install: ## Install npm dependencies for the api, admin, and frontend apps
#echo "Installing Node dependencies"
#npm install
install-dev: install ## Install dependencies and prepared development configuration
#./node_modules/.bin/selenium-standalone install
#cp -n ./config/development.js-dist ./config/development.js | true
run-frontend-dev: webpack.PID ## Run the frontend and admin apps in dev (using webpack-dev-server)
Into:
install Install npm dependencies for the api, admin, and frontend apps
install-dev Install dependencies and prepared development configuration
run-frontend-dev Run the frontend and admin apps in dev (using webpack-dev-server)
But for some reason I can't get it working (on OSX at least). With this target:
help: ## Show the help prompt.
#grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'
When I run:
make help
I just get:
$ make help
make: Nothing to be done for `help'.
I've tried all of the different solutions from the comments in the article too, but nothing seems to work. What am I doing wrong?
Also, what would a variation on the script be that would allow comments to be placed on a line before the target, instead of after it. Like so:
# Build the source files with Babel.
build: $(shell find lib)
#rm -rf build
#mkdir -p build
#$(babel) lib $(babel_options)
# Seed the database with fake data.
db-fake:
#$(run) node bin/run-sql fake
Here is the full Makefile in question:
# Options.
DEBUG ?=
SOURCEMAPS ?= true
WATCH ?=
# Variables.
bin = ./node_modules/.bin
tests = $(shell find test/routes -depth 1)
shorthand-tests = $(patsubst test/routes/%.js,test-%,$(tests))
# Binaries.
babel = $(bin)/babel
mocha = $(bin)/mocha
node = node
nodemon = $(bin)/nodemon
run = heroku local:run
# Babel options.
babel_options = \
--out-dir build \
--copy-files
# Sourcemaps?
ifneq ($(SOURCEMAPS),false)
babel_options += --source-maps
endif
# Watch?
ifdef WATCH
babel_options += --watch
endif
# Development?
ifneq ($(NODE_ENV),production)
node = $(nodemon)
endif
# Debug?
ifdef DEBUG
node += debug
mocha += debug
endif
# Default.
default: help ## Default.
# Build the source files with Babel.
build: $(shell find lib) ## Build the source files with Babel.
#rm -rf build
#mkdir -p build
#$(babel) lib $(babel_options)
# Seed the database with fake data.
db-fake: ## Seed the database with fake data.
#$(run) node bin/run-sql fake
# Reset the database, dropping everything and the creating again.
db-reset: ## Reset the database, dropping everything and the creating again.
#$(run) node bin/run-sql drop
#$(run) node bin/run-sql create
# Seed the database with test data.
db-seed: ## Seed the database with test data.
#$(run) node bin/run-sql seed
# Show the help prompt.
help: ## Show the help prompt.
#awk '/^#/{c=substr($0,3);next}c&&/^[[:alpha:]][[:alnum:]_-]+:/{print substr($1,1,index($1,":")),c}1{c=0}' mm.mk | column -s: -t
# Open the PSQL interface.
psql: ## Open the PSQL interface.
#psql contentshq
# Run the development server.
server: ## Run the development server.
#$(run) $(node) bin/api
# Start the local postgres server.
start: ## Start the local postgres server.
#pg_ctl -D /usr/local/var/postgres -l /usr/local/var/postgres/server.log start
# Stop the local postgres server.
stop: ## Stop the local postgres server.
#pg_ctl -D /usr/local/var/postgres stop -s -m fast
# Run all of the tests.
test: ## Run all of the tests.
#$(run) $(mocha) $(tests)
# Run specific route tests.
$(shorthand-tests): ## Run specific route tests.
#$(run) $(mocha) test/routes/$(subst test-,,$#).js
# Watch the source files with Babel.
watch: ## Watch the source files with Babel.
#WATCH=true $(MAKE) build
# Phony targets.
.PHONY: help
.PHONY: test
.PHONY: watch
Here's a little awk script which will handle the case where the comment comes before the target name. I used column to put the text into columns, but you could do it in awk using a printf (and, if you wanted, adding the console codes to colorize the output); the advantage of using column is that it works out the column width automatically.
You'd insert it into the Makefile in the same way:
.PHONY: help
# Show this help.
help:
#awk '/^#/{c=substr($$0,3);next}c&&/^[[:alpha:]][[:alnum:]_-]+:/{print substr($$1,1,index($$1,":")),c}1{c=0}' $(MAKEFILE_LIST) | column -s: -t
You need a help rule that will actually extract and print the messages. Assuming the article you're talking about is http://marmelab.com/blog/2016/02/29/auto-documented-makefile.html, this is what you're looking for.
.PHONY: help
help:
#grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'
Your docs before the command approach might be doable but will require some more work. Most UNIX commands work line by line so it's easier to do with the comments as specified in the article.

Execute object files from different directories using Makefile

I have the following Makefile entry:
TEST_DIRS = abcd pqr xyz
test_lib :
for dir in $(TEST_DIRS); do \
$(MAKE) -C $$dir; \
done
run :
./abcd/test/abcd_test.o --log_level=message
./pqr/test/pqr_test.o --log_level=message
./xyz/test/xyz_parser_test.o --log_level=message
test : test_lib run
I don't want to write 3 separate commands for run target instead make it generic. So that everytime a new test file gets added, I dont want add a new command under run target. Can somebody help me?
TESTS := $(addprefix TEST_, $(TEST_DIRS))
run: $(TESTS)
TEST_%:
./$*/test/$*_test.o --log_level=message
Assuming the naming scheme is consistent (and those are just oddly named binaries and not actually object files) then something like this should work:
Using a shell loop:
run :
for name in $(TEST_DIRS); do \
./$$name/test/$${name}_test.o --log_level=message; \
done
Using make foreach:
run :
$(foreach name,$(TEST_DIRS),./$(name)/test/$(name)_test.o --log_level=message;)

Resources