Passing target variables from pattern match to static - makefile

Question

I'm writing a Makefile to wrap the deployment of an elastic beanstalk app to multiple environments. This is my first time writing an "advanced" makefile and I'm having a bit of trouble.
My goals are as follows:
make deploy by itself should deploy with ENVIRONMENT=$(USER) and ENV_TYPE=staging.
The environment to deploy to can be set via an environment variable. If ENVIRONMENT=production then ENV_TYPE=production, otherwise ENV_TYPE=staging.
As a shorthand for setting the environment variable, one can suffix the deploy target with a - and the name of the environment. For example: make deploy-production.
It's number 3 that is giving me the most trouble. Since any environment not named production is of type staging, I tried to use pattern matching to define the target for deploying to staging environments. Here's what I have currently:
ENVIRONMENT = $(USER)
ifeq ($ENVIRONMENT, production)
ENV_TYPE=production
else
ENV_TYPE=staging
endif
DOCKER_TAG ?= $(USER)
CONTAINER_PORT ?= 8000
ES_HOST = logging-ingest.$(ENV_TYPE).internal:80
.PHONY: deploy
deploy:
-pip install --upgrade awsebcli
sed "s/<<ES_HOST>>/$(ES_HOST)/" < 01-filebeat.template > .ebextensions/01-filebeat.config
sed "s/<<DOCKER_TAG>>/$(DOCKER_TAG)/" < Dockerrun.template | sed "s/<<CONTAINER_PORT>>/$(CONTAINER_PORT)/" > Dockerrun.aws.json
eb labs cleanup-versions --num-to-leave 10 --older-than 5 --force -v --region us-west-2
eb deploy -v coolapp-$(ENVIRONMENT)
.PHONY: deploy-%
deploy-%: ENVIRONMENT=$*
deploy-%: deploy
#echo # Noop required
.PHONY: deploy-production
deploy-production: ENVIRONMENT=production
deploy-production: ENV_TYPE=production
deploy-production: deploy
#echo # Noop required
The problem is in the last step of the deploy target. Namely, $(ENVIRONMENT) appears to be unset.
Example Output:
18:42 $ make -n deploy-staging
pip install --upgrade awsebcli
sed "s/<<ES_HOST>>/logging-ingest.staging.internal:80/" < 01-filebeat.template > .ebextensions/01-filebeat.config
sed "s/<<DOCKER_TAG>>/schultjo/" < Dockerrun.template | sed "s/<<CONTAINER_PORT>>/8000/" > Dockerrun.aws.json
eb labs cleanup-versions --num-to-leave 10 --older-than 5 --force -v --region us-west-2
eb deploy -v coolapp-
echo # Noop required
Desired Output:
18:42 $ make -n deploy-staging
pip install --upgrade awsebcli
sed "s/<<ES_HOST>>/logging-ingest.staging.internal:80/" < 01-filebeat.template > .ebextensions/01-filebeat.config
sed "s/<<DOCKER_TAG>>/schultjo/" < Dockerrun.template | sed "s/<<CONTAINER_PORT>>/8000/" > Dockerrun.aws.json
eb labs cleanup-versions --num-to-leave 10 --older-than 5 --force -v --region us-west-2
eb deploy -v coolapp-staging
echo # Noop required
So I have tried to implement Renaud's recursive Make solution, but have run into a minor hiccup. Here's a simplified Makefile that I'm using:
ENVIRONMENT ?= $(USER)
ifeq ($ENVIRONMENT, production)
ENV_TYPE = production
else
ENV_TYPE = staging
endif
ES_HOST = logging-ingest.$(ENV_TYPE).internal:80
.PHONY: deploy
deploy:
#echo $(ENVIRONMENT)
#echo $(ENV_TYPE)
#echo $(ES_HOST)
.PHONY: deploy-%
deploy-%:
#$(MAKE) --no-print-directory ENVIRONMENT=$* deploy
When I run make production, it looks like the if statements surrounding the ENV_TYPE definition are not run again. Here's my actual output:
12:50 $ make -n deploy-production
/Applications/Xcode.app/Contents/Developer/usr/bin/make --no-print-directory ENVIRONMENT=production deploy
echo production
echo staging
echo logging-ingest.staging.internal:80
The last two lines should say production rather than staging, implying there is something wrong with my conditional, but I haven't edited that conditional from earlier versions when it worked, so I'm a but confused. The same error happens if I invoke make with ENVIRONMENT set manually (e.g. ENVIRONMENT=production make deploy).

Answer 1

Your problem comes from the way target-specific variable values are inherited by target pre-requisites. What you are trying to do works for explicit target-specific variables:
$ cat Makefile
ENVIRONMENT = default
deploy:
#echo '$#: ENVIRONMENT = $(ENVIRONMENT)'
deploy-foo: ENVIRONMENT = foo
deploy-foo: deploy
#echo '$#: ENVIRONMENT = $(ENVIRONMENT)'
$ make deploy
deploy: ENVIRONMENT = default
$ make deploy-foo
deploy: ENVIRONMENT = foo
deploy-foo: ENVIRONMENT = foo
because the deploy-foo-specific ENVIRONMENT variable is assigned value foo during the first expansion phase (the one during which target-specific variable assignments, target lists and pre-requisite lists are expanded). So, deploy inherits this value.
But it does not work with your pattern-specific variables that use the $* automatic variable:
$ cat Makefile
ENVIRONMENT = default
deploy:
#echo '$#: ENVIRONMENT = $(ENVIRONMENT)'
deploy-%: ENVIRONMENT = $*
deploy-%: deploy
#echo '$#: ENVIRONMENT = $(ENVIRONMENT)'
$ make deploy
$ deploy: ENVIRONMENT = default
$ make deploy-foo
deploy: ENVIRONMENT =
deploy-foo: ENVIRONMENT = foo
The reason is that the deploy-foo-specific ENVIRONMENT variable is what is called a recursively expanded variable in make dialect (because you use the = assignment operator). It is expanded, recursively, but only when make needs its value, not when it is assigned. So, in the context of deploy-foo, it is assigned $*, not the pattern stem. ENVIRONMENT is passed as is to the context of the deploy pre-requisite and, in this context, $(ENVIRONMENT) is recursively expanded to $* and then to the empty string because there is no pattern in the deploy rule. You could try the simply expanded (non-recursive) variable flavour:
deploy-%: ENVIRONMENT := $*
that is immediately expanded, but the result would be the same because $* expands as the empty string during the first expansion. It is set only during the second expansion and can thus be used only in recipes (that make expands in a second phase).
A simple (but not super-efficient) solution consists in invoking make again:
deploy-%:
#$(MAKE) ENVIRONMENT=$* deploy
Example:
$ cat Makefile
ENVIRONMENT = default
deploy:
#echo '$(ENVIRONMENT)'
deploy-%:
#$(MAKE) --no-print-directory ENVIRONMENT=$* deploy
$ make
default
$ make deploy
default
$ make deploy-foo
foo
Note: GNU make supports a secondary expansion and one could think that it can be used to solve this problem. Unfortunately not: the secondary expansion expands only the pre-requisites, not the target-specific variable definitions.
As mentioned above, this recursive make is not very efficient. If efficiency is critical one single make invocation is preferable. And this can be done if you move all your variables processing in the recipe of a single pattern rule. Example if the shell that make uses is bash:
$ cat Makefile
deplo%:
#{ [[ "$*" == "y" ]] && ENVIRONMENT=$(USER); } || \
{ [[ "$*" =~ y-(.*) ]] && ENVIRONMENT=$${BASH_REMATCH[1]}; } || \
{ echo "$#: unknown target" && exit 1; }; \
echo "$#: ENVIRONMENT = $$ENVIRONMENT" && \
<do-whatever-else-is-needed>
$ USER=bar make deploy
deploy: ENVIRONMENT = bar
$ make deploy-foo
deploy-foo: ENVIRONMENT = foo
$ make deplorable
deplorable: unknown target
make: *** [Makefile:2: deplorable] Error 1
Do not forget to escape the recipe expansion by make ($$ENVIRONMENT).

Answer 2

Due to conversations w/#RenaudPacalet, I have learned that my approach mainly works because the variables defined in the deploy-% rules aren't used anywhere but the recipe...where they expand late, just before being passed to the shell. This lets me use $* in the variable definition because the variable definition wont be expanded until the second phase, when $* actually has a value.
The method for setting ENV_TYPE uses a trick with patsubst to produce the condition for an if by stripping the word "production" from the content of $ENVIRONMENT; in this context, an empty string selects the else case. So if $ENVIRONMENT is exactly equal to "production" then patsubst makes an empty string and the if evaluates to production, otherwise it evaluates to staging.
There's an explicit rule at the bottom for deploy- because that target would otherwise invoke some crazy implicit pattern rules that tried to compile deploy-.o
Finding that made me also consider the other error cases that could arise, so the first few lines define a function to ensure that, if a user specifies both ENVIRONMENT=X and uses a suffix Y, that there is an appropriate error message (rather than just having the suffix win). You can see the call to that function as the first line of the deploy-% recipe. There is another potential issue if $ENVIRONMENT is defined to have multiple words; the second deploy: line implements a test that will error out in this case---it tests that the word-count in $ENVIRONMENT is exactly 1 using the same patsubst/if trick as above.
It should also be noted that this makefile assumes the real work will be implemented in the recipe under deploy-%.
# define a function to detect mismatch between ENV and suffix
ORIG_ENV := $(ENVIRONMENT)
ENV_CHECK = $(if $(ORIG_ENV),$(if $(subst $(ORIG_ENV),,$(ENVIRONMENT)),\
$(error $$ENVIRONMENT differs from deploy-<suffix>.),),)
ENVIRONMENT ?= $(USER)
.PHONY: deploy
deploy: deploy-$(ENVIRONMENT)
deploy: $(if $(patsubst 1%,%,$(words $(ENVIRONMENT))),$(error Bad $$ENVIRONMENT: "$(ENVIRONMENT)"),)
.PHONY: deploy-%
deploy-%: ENVIRONMENT=$*
deploy-%: ENV_TYPE=$(if $(patsubst production%,%,$(ENVIRONMENT)),staging,production)
deploy-%:
$(call ENV_CHECK)
#echo ENVIRONMENT: $(ENVIRONMENT)
#echo ENV_TYPE: $(ENV_TYPE)
# keep it from going haywire if user specifies:
# ENVIRONMENT= make deploy
# or
# make deploy-
.PHONY: deploy-
deploy-:
$(error Cannot build with empty $$ENVIRONMENT)
Gives
$ USER=anonymous make deploy
ENVIRONMENT: anonymous
ENV_TYPE: staging
$ ENVIRONMENT=production make deploy
ENVIRONMENT: production
ENV_TYPE: production
$ ENVIRONMENT=test make deploy
ENVIRONMENT: test
ENV_TYPE: staging
$ make deploy-foo
ENVIRONMENT: foo
ENV_TYPE: staging
$ make deploy-production
ENVIRONMENT: production
ENV_TYPE: production
$ ENVIRONMENT=foo make deploy-production
Makefile:14: *** $ENVIRONMENT differs from deploy-<suffix>.. Stop.
$ ENVIRONMENT= make deploy
Makefile:24: *** Bad $ENVIRONMENT: "". Stop.
$ make deploy-
Makefile:24: *** Cannot build with empty $ENVIRONMENT. Stop.
$ ENVIRONMENT="the more the merrier" make deploy
Makefile:10: *** Bad $ENVIRONMENT: "the more the merrier". Stop.
Reflecting on how this works, it's not simple at all. There are various interpretations of $ENVIRONMENT...for example in the line deploy: deploy-$(ENVIRONMENT), that sense of $ENVIRONMENT gets the one that comes in from the shell's environment (possibly being set to $(USER) if absent). There's another sense in the recipe line #echo ENVIRONMENT: $(ENVIRONMENT) which will be the one assigned in deploy-%: ENVIRONMENT=$* just above, but after expansion. I am struck by the analogy with scoping or shadowing of variables in programming.