I have a makefile that deploys x project in aws
The goal is to execute the make like this:
make deploy-production
And see output like this
Please select the project:
1) api
2) web
3) somethingelse
You press then the number and make continues, by assigning the choice to a variable named project
I have already "autogenerating" targets for supported envs, just want to build the multichoice for projects.
Sample makefile of mine with removed nonrelevant stuff:
ENVIRONMENTS := development staging production
TARGETS := $(foreach X,$(ENVIRONMENTS),deploy-$X)
$(TARGETS):
$(eval ENV:=$(subst deploy-,,$(#)))
# here want to ask for project, env i have already as resolved from above line
Well, make can run any shell script. So if you write a shell script that will ask the user for input and accept an answer you can run it from make.
You could do something like this:
deploy-production:
#echo Please select the project:; \
echo '1) api'; \
echo '2) web'; \
echo '3) somethingelse'; \
read -p 'Enter value: ' result && $(MAKE) CHOICE=$$result got-choice
There is a LOT left out here: handling invalid values, converting the CHOICE value into a target that would then be a prerequisite of the got-choice target, etc. etc. I don't really think this is a good way forward but you could make it work.
Related
QUESTION
Using Make how do I run a command for every directory that contains a file matching *.csproj but does not including a file matching *.Test.csproj using pure make.
SCENARIO
I have previously used Fake and Rake extensively but this is my first time using Make to do anything over and above the simple use of dumb targets.
I have a simple makefile that compiles a .net core solution, runs some tests and then packages up a nuget package. Here is a simplified example.
build:
dotnet build ./src \
...
test:
dotnet test ./src/TestProject \
...
package:
dotnet pack ./src/PackageProject \
...
I now want to introduce additional projects which are also packaged but I do not want to specify each project to package individually. I want the make file to automagically pick up each project that can be packaged. This method has been tried in tested in various Fake builds.
I have been able to implement the following matching on the projects csproj extension and works fine but I have not been able to filter out the test project which also gets packaged.
package: ./src/**/%.csproj
%.csproj:
dotnet pack $(#D) \
...
I have been trying to understand the Make pattern rules and how to apply the filter-out function but have sadly failed. $(filter-out src/**/*.Test.csproj, src/**/%.csproj)
Would appreciate any help on figuring this one out.
EDIT
Based on the question from MadScientist if I run the following using make package using this dumbed down example:
package: ./src/*/%.csproj
%.csproj :
echo $(#)
I get the following output:
echo src/Namespace.Project1/%.csproj
src/Namespace.Project1/%.csproj
echo src/Namespace.Project2/%.csproj
src/Namespace.Project2/%.csproj
echo src/Namespace.Test/%.csproj
src/Namespace.Test/%.csproj
Additionally based on MadScientist's comments I have also been able to create a list of the directories I want to call the dotnet pack command against but I am now stuck on how to call the target for each match.
Note: I am trying to keep this pure Make and avoid using any bash specific syntax
projects := $(filter-out $(dir $(wildcard ./src/Bombora.Namespace*Test/.) ), $(dir $(wildcard ./src/Namespace.*/.) ) )
package:
echo $(projects)
Results in:
echo ./src/Namespace.Project1/ ./src/Namespace.Project2/
./src/Namespace.Project1/ ./src/Namespace.Project2/
EDIT
I have been able to make this work but I do not know if I have gone about it the correct way or if I am abusing something which will come back to bite me later.
This is what I am now doing which is working as expected:
PACKAGE_PROJECTS := $(filter-out $(wildcard ./src/Namespace*Test/*.csproj), $(wildcard ./src/Namespace*/*.csproj) )
package: $(PACKAGE_PROJECTS)
$(PACKAGE_PROJECTS): .
dotnet pack $(#D) \
...
Make implements standard globbing as defined by POSIX. It doesn't provide advanced globbing as implemented in some advanced shells like zsh (or bash if you enable it).
So, ** is identical *; there's no globbing character that means "search in all subdirectories". If you want to do that you need to use find.
Also, in make a pattern is a template that can match some target that you specifically want to build. It's not a way to find targets. And pattern rules only are pattern rules if the target contains the pattern character %; putting a % in a prerequisite of an explicit target doesn't do anything, make treats it as if it were just a % character.
so:
package: ./src/**/%.csproj
is identical to writing:
package: ./src/*/%.csproj
where it finds files matching the literal string %.csproj of which you probably don't have any.
I don't see how this package target does anything at all.
I don't understand what exactly you want to do: you need to make your question more explicit. Make works on targets and prerequisites. So, what is the target you want to build and what are the prerequisites used to create that target? What is the make command line you are invoking, what is the output you got, and what is the intended output you want?
ETA
You asked:
At the end of the day the question is about how do I run a command for every directory that contains a file matching *.csproj but not including *.Test.csproj using pure make.
This will get you that list:
TEST_PROJECTS := $(dir $(wildcard src/*/*.Test.csproj))
PROJECTS := $(filter-out $(TEST_PROJECTS),$(dir $(wildcard src/*/*.csproj)))
projects: $(PROJECTS)
$(PROJECTS):
...run commands...
.PHONY: $(PROJECTS)
You combine setting PROJECTS into one line if you prefer.
Please see below changes which will help you understand applying a filter with make targets and getting ahead.
Usage of filter and filter-out:
filter : Select words in text that match one of the pattern words.
Syntax : $(filter pattern...,text)
filter-out: Select words in text that do not match any of the pattern words.
Syntax : $(filter-out pattern...,text)
In the below example I will use filter to match pattern and if the conditions evaluates to true , then you can use your target with whatever execution you would like to do.
# List the project extensions that you would like to support
MYPROJECTS=.csproj .xyzproj
# MY_FILES would contain all files eg: abc.csproj def.csproj abc.xyzproj
# Below GETPATTERN will extract the suffix: eg. it would produce result .csproj .csproj .xyzproj
GETPATTERN:= $(suffix $(MY_FILES))
ifeq ($(filter $(GETPATTERN),$(MYPROJECTS)),)
package:
dotnet pack ./src/PackageProject
else
package: <default or any other dependency you want>
dotnet pack ./src/PackageDefaultProject
endif
I have a number of makefiles that build and run tests. I would like to create a script that makes each one and notes whether the tests passed or failed. Though I can determine test status within each make file, I am having trouble finding a way to communicate that status to the caller of the make command.
My first thought is to somehow affect the return value of the make command, though this does not seem possible. Can I do this? Is there some other form of communication I can use to express the test status to the bash script that will be calling make? Perhaps by using environment variables?
Thanks
Edit: It seems that I cannot set the return code for make, so for the time being I will have to make the tests, run them in the calling script instead of the makefile, note the results, and then manually run a make clean. I appreciate everyone's assistance.
Make will only return one of the following according to the source
#define MAKE_SUCCESS 0
#define MAKE_TROUBLE 1
#define MAKE_FAILURE 2
MAKE_SUCCESS and MAKE_FAILURE should be self-explanatory; MAKE_TROUBLE is only returned when running make with the -q option.
That's pretty much all you get from make, there doesn't seem to be any way to set the return code.
The default behavior of make is to return failure and abandon any remaining targets if something failed.
for directory in */; do
if ( cd "$directory" && make ); then
echo "$0: Make in $directory succeeded" >&2
else
echo "$0: Make in $directory failed" >&2
fi
done
Simply ensure each test leaves its result in a file unique to that test. Least friction will be to create test.pass if thes test passes, otherwise create test.fail. At the end of the test run gather up all the files and generate a report.
This scheme has two advantages that I can see:
You can run the tests in parallel (You do us the -jn flag, don't you? (hint: it's the whole point of make))
You can use the result files to record whether the test needs to be re-run (standard culling of work (hint: this is nearly the whole point of make))
Assuming the tests are called test-blah where blah is any string, and that you have a list of tests in ${tests} (after all, you have just built them, so it's not an unreasonable assumption).
A sketch:
fail = ${#:%.pass=%.fail}
test-passes := $(addsuffix .pass,${tests})
${test-passes}: test-%.pass: test-%
rm -f ${fail}
touch $#
$* || mv $# ${fail}
.PHONY: all
all: ${test-passes}
all:
# Count the .pass files, and the .fail files
echo '$(words $(wildcard *.pass)) passes'
echo '$(words $(wildcard *.fail)) failures'
In more detail:
test-passes := $(addsuffix .pass,${tests})
If ${tests} contains test-1 test-2 (say), then ${test-passes} will be test-1.pass test-2.pass
${test-passes}: test-%.pass: test-%
You've just gotta love static pattern rules.
This says that the file test-1.pass depends on the file test-1. Similarly for test-2.pass.
If test-1.pass does not exist, or is older than the executable test-1, then make will run the recipe.
rm -f ${fail}
${fail} expands to the target with pass replaced by fail, or test-1.fail in this case. The -f ensures the rm returns no error in the case that the file does not exist.
touch $# — create the .pass file
$< || mv $# ${fail}
Here we run the executable
If it returns success, our work is finished
If it fails, the output file is deleted, and test-1.fail is put in its place
Either way, make sees no error
.PHONY: all — The all target is symbolic and is not a file
all: ${test-passes}
Before we run the recipe for all, we build and run all the tests
echo '$(words $(wildcard *.pass)) passes'
Before passing the text to the shell, make expands $(wildcard) into a list of pass files, and then counts the files with $(words). The shell gets the command echo 4 passes (say)
You run this with
$ make -j9 all
Make will keep 9 jobs running at once — lovely if you have 8 CPUs.
I have a Makefile that defines docker-compose project.
It essentially assembles me a command:
COMMAND := docker-compose --project-name=$(PREFIX) --file=$(FILE_PATH)
up:
$(COMMAND) up -d
I would like to add a target named dc to which I would be able to pass any arguments I want.
I know there is one solution:
target:
$(COMMAND) $(ARGS)
And then call it with make target ARGS="--help" for example.
But isn't there an easier way like in bash $# ? I would like to skip the ARGS=... part and send everything to the command after target name.
Not really. The make program interprets all arguments (that don't contain =) as target names to be built and there's no way you can override that. So even though you can obtain the list of arguments given on the command line (via the GNU make-specific $(MAKECMDGOALS) variable) you can't prevent those arguments from being considered targets.
You could do something like this, which is incredibly hacky:
KNOWN_TARGETS = target
ARGS := $(filter-out $(KNOWN_TARGETS),$(MAKECMDGOALS))
.DEFAULT: ;: do nothing
.SUFFIXES:
target:
$(COMMAND) $(ARGS)
(untested). The problem here is you have to keep KNOWN_TARGETS up to date with all the "real" targets so you can remove them from the list of targets given on the command line. Then add the .DEFAULT target which will be run for any target make doesn't know how to build, which does nothing. Reset the .SUFFIXES meta-target to remove built-in rules.
I suspect this still will have weird edge-cases where it doesn't work.
Also note you can't just add options like --help to the make command line, because make will interpret them itself. You'll have to prefix them with -- to force make to ignore them:
make target -- --help
Another option would be to add a target like this:
target%:
$(COMMAND) $*
Then you can run this:
make "target --help"
But you have to include the quotes.
In general I just recommend you reconsider what you want to do.
You could write a bash wrapper script to do what you'd like:
#/bin/bash
make target ARGS=\"$#\"
The reason you don't want to do it in make, is that make parses the command line parameters before it parse the makefile itself, so by the time you read the makefile, the targets, variables, etc have already been set. This means that make will have already interpreted the extra parameters as new targets, variables etc.
A target that re-run make containerized
.PHONY: all containerized
ifeq ($(filter containerized,$(MAKECMDGOALS)),containerized)
.NOTPARALLEL: containerized
MAKEOVERRIDES ?=
containerized: ## Build inside a container
#docker run image_with_make make $(MAKEOVERRIDES) $(filter-out containerized,$(MAKECMDGOALS))
else
# other targets here
all: xxxx
endif
Executing
make containerized all runs make all in container
The first answer is correct, no passthru of args. However, here is a plausible path for experimentation, use of branch by include selection:
# Makefile:
COMMAND := $(PYTHON) this_shit_got_real.py
LOCAL_MK ?= local.mk
# '-' important, absence of LOCAL_MK is not cause for error, just run with no overrides
- include $(LOCAL_MK)
target:
$(COMMAND) $(ARGS)
Now see how you add branching with env:
echo "ARGS=--help">>local.mk
# make target
And the other cli controlled branch
echo "ARGS=--doit">>runner.mk
# LOCAL_MK=runner.mk make target
A = "demo"
%.o:%.cpp
$(CC) -c $^ $(A) -o $#
default:$(all_objs)
game:A = $(shell read -p 'Enter game version: ' gv && echo $$gv)
game:$(all_objs)
Just a snippet makefile above. If I make game, main problem is each compilation of sources will expand $(A) and it will request user to input game version over and over. $(A) has default content "demo" only if user doesn't make game target.
So, is there any way to set $(A) to be expanded && ?
game:A:=$(shell read -p 'Enter game version: ' gv && echo $$gv)
Notice ':='
Update:
Reading user input in real build targets is not a good idea. Make is widely used and whoever invokes it - they wouldn't expect this.
I see two ways to do what you wanted:
make version=some_version. It will override version variable (name may be almost anything). If you have assigned its default value in makefile - it will be changed to some_version (unless variable declared with override flag, which disables this behaviour)
Create make config rule, which would perform required configuration actions and save them into, say, config.mk. In makefile you could have -include config.mk ('-' sign says make that this file may be missing. If it is required (e.g. contains some mandatory configuration options), remove minus sign).
Is it possible to include Makefiles dynamically? For example depending on some environment variable? I have the following Makefiles:
makefile
app1.1.mak
app1.2.mak
And there is an environment variable APP_VER which could be set to 1.1.0.1, 1.1.0.2, 1.2.0.1, 1.2.0.2.
But there will be only two different makefiles for 1.1 and 1.2 lines.
I have tried to write the following Makefile:
MAK_VER=$$(echo $(APP_VER) | sed -e 's/^\([0-9]*\.[0-9]*\).*$$/\1/')
include makefile$(MAK_VER).mak
all: PROD
echo MAK_VER=$(MAK_VER)
But it does not work:
$ make all
"makefile$(echo", line 0: make: Cannot open makefile$(echo
make: Fatal errors encountered -- cannot continue.
UPDATE:
As far as I understand make includes files before it calculates macros.
That's why it tries to execute the following statement
include makefile.mak
instead of
include makefile1.1.mak
You have two problems: your method of obtaining the version is too complicated, and your include line has a flaw. Try this:
include app$(APP_VER).mak
If APP_VER is an environmental variable, then this will work. If you also want to include the makefile called makefile (that is, if makefile is not the one we're writing), then try this:
include makefile app$(APP_VER).mak
Please note that this is considered a bad idea. If the makefile depends on environmental variables, it will work for some users and not others, which is considered bad behavior.
EDIT:
This should do it:
MAK_VER := $(subst ., ,$(APP_VER))
MAK_VER := $(word 1, $(MAK_VER)).$(word 2, $(MAK_VER))
include makefile app$(MAK_VER).mak
Try this:
MAK_VER=$(shell echo $(APP_VER) | sed -e 's/^\([0-9]*\.[0-9]*\).*$$/\1/')
MAK_FILE=makefile$(MAK_VER).mak
include $(MAK_FILE)
all:
echo $(MAK_VER)
echo $(MAK_FILE)
Modifying the outline solution
Have four makefiles:
makefile
app1.1.mak
app1.2.mak
appdummy.mak
The app.dummy.mak makefile can be empty - a symlink to /dev/null if you like. Both app.1.1.mak and app.1.2.mak are unchanged from their current content.
The main makefile changes a little:
MAK_VER = dummy
include makefile$(MAK_VER).mak
dummy:
${MAKE} MAK_VER=$$(echo $(APP_VER) | sed -e 's/^\([0-9]*\.[0-9]*\).*$$/\1/') all
all: PROD
...as now...
If you type make, it will read the (empty) dummy makefile, and then try to build the dummy target because it appears first. To build the dummy target, it will run make again, with APP_VER=1.1 or APP_VER=1.2 on the command line:
make APP_VER=1.1 all
Macros set on the command line cannot be changed within the makefile, so this overrides the line in the makefile. The second invocation of make, therefore, will read the correct version-specific makefile, and then build all.
This technique has limitations, most noticeably that it is fiddly to arrange for each and every target to be treated like this. There are ways around it, but usually not worth it.
Project organization
More seriously, I think you need to review what you're doing altogether. You are, presumably, using a version control system (VCS) to manage the source code. Also, presumably, there are some (significant) differences between the version 1.1 and 1.2 source code. So, to be able to do a build for version 1.1, you have to switch from the version 1.1 maintenance branch to the version 1.2 development branch, or something along those lines. So, why isn't the makefile just versioned for 1.1 or 1.2? If you switch between versions, you need to clean out all the derived files (object files, libraries, executables, etc) that may have been built with the wrong source. You have to change the source code over. So why not change the makefile too?
A build script to invoke make
I also observe that since you have the environment variable APP_VER driving your process, that you can finesse the problem by requiring a standardized 'make invoker' that sorts out the APP_VER value and invokes make correctly. Imagine that the script is called build:
#!/bin/sh
: ${APP_VER:=1.2.0.1} # Latest version is default
case $APP_VER in
[0-9].[0-9].*)
MAK_VER=`echo $APP_VER | sed -e 's/^\(...\).*/\1/'`
;;
*) echo "`basename $0 .sh`: APP_VER ($APP_VER) should start with two digits followed by dots" 1>&2;
exit 1;;
esac
exec make MAK_VER=$MAK_VER "$#"
This script validates that APP_VER is set, giving an appropriate default if it is not. It then processes that value to derive the MAK_VER (or errors out if it is incorrect). You'd need to modify that test after you reach version 10, of course, since you are planning to be so successful that you will reach double-digit version numbers in due course.
Given the correct version information, you can now invoke your makefile with any command line arguments.
The makefile can be quite simple:
MAK_VER = dummy
include app$(MAK_VER).mak
all: PROD
...as now...
The appdummy.mak file now contains a rule:
error:
echo "You must invoke this makefile via the build script" 1>&2
exit 1
It simply points out the correct way to do the build.
Note that you can avoid the APP_VER environment variable if you keep the product version number under the VCS in a file, and the script then reads the version number from the file. And there could be all sorts of other work done by the script, ensuring that correct tools are installed, other environment variables are set, and so on.