I am doing some data validation. Basically, I have a list of files and I want to make sure they are consistent. For a file to be consistent, they need to match the number of occurences.
grep -c ValidId file.????.txt
wc -l < output.????.txt
If they both match, I would like to create a done file so I don't have to recheck again. This will be a routine in my Makefile for data validation.
How would I go about doing this?
The pattern match % can be used in multiple dependencies in the rule. One can use that to keep track of the consistent files, so that running make and correcting errors would be checking only the not-yet-consistent files:
FILES=$(wildcard file.????.txt)
RES_FILES=$(FILES:file.%=.result.%)
all: $(RES_FILES)
.result.%.txt: file.%.txt output.%.txt
if [ `grep -c ValidId < file.$*.txt` -eq \
`wc -l < output.$*.txt` ]; then \
touch $# ; \
fi
clean:
#rm -f $(RES_FILES)
Related
I have a command which sometimes updates a target file, sometimes it leaves it unchanged, but I cannot tell which one of those will happen. I would like to create a dependency that should run ONLY if the first file is updated. I could not do this. Here is a simple Makefile that reproduces this problem:
all: file2.txt
file1.txt:
#if [ ! -f file1.txt -o $(shell bash -c "expr \$$RANDOM \% 2") = 1 ]; \
then \
echo "update" >> file1.txt; \
echo "file1.txt overwritten"; \
else \
echo "file1.txt is unchanged"; \
fi
.PHONY: file1.txt
file2.txt: file1.txt
cp file1.txt file2.txt
The test in the rule for file1.txt will generate file1.txt if it does not exists or if a random number modulo 2 is 1. I would like to see cp file1.txt file2.txt executed if and only if file1.txt is overwritten.
By adding the .PHONY setting you are specifically telling make that no matter what that target is always considered out of date, so you're directly contradicting what you want to do.
You can use another method to handle this via a "force target", like this:
FORCE: ;
file1.txt: FORCE
...
and remove the .PHONY setting.
I am trying to write a simple Makefile to build .expected files and compare them but I am failing.
APSSCHED=../../bin/apssched
BASE=.:../../base:../../examples
FLAGS=-DCOT
EXAMPLES=../../examples/
CASES=simple-binding1 simple-binding2
# skipping lines doesn't work ...
# run command and skip the first line
%.aps:
${APSSCHED} ${FLAGS} -p ${BASE} ${EXAMPLES}/$* | tail -n +2
# get all cases as an array to pipe it to different make targets
# maybe overcomplicating
cases:
echo ${CASES} | \
awk '{split($$0,numbers," ")} END {for(n in numbers){ print numbers[n] }}'
# create all .expected files from ${CASES}
build.expected:
$(MAKE) cases | xargs -n1 -I file /bin/bash -c '$(MAKE) file.build.expected'
# create single .expected file
%.build.expected:
$(MAKE) $*.aps > $*.expected
# compare result with
%.compare:
$(MAKE) $*.aps | diff $*.expected -
# run command for all cases and diff the result with corresponding expected
all:
$(MAKE) cases | xargs -n1 -I file /bin/bash -c '$(MAKE) file.compare'
clean.expected:
rm *.expected
Running make without any target and nothing happens.
echo simple-binding1 simple-binding2 | \
awk '{split($0,numbers," ")} END {for(n in numbers){ print numbers[n] }}'
simple-binding1
simple-binding2
I think the issue is with my cases target. I am not sure if I am on the right track.
I appreciate any help or hint.
I would avoid re-running make just to call a different target - it's a performance hit and may be unreliable (depending on rest of the Makefile) since separate calls may not be able to track dependencies correctly.
Moreover, I would avoid using | - every time a command is concatenated with pipe, exit code of piped command would be exit code of the last command. So a call like command | tail would return the exit code of tail (which would almost always succeed). Even if the command has failed, it would be covered with exit code 0 from tail and make will not detect the error and will not stop.
Thus said, I tried to rewrite your approach by just creating dependencies between the targets, like so:
$ cat Makefile
APSSCHED=../../bin/apssched
EXAMPLES=../../examples
BASE=.:../../base:$(EXAMPLES)
FLAGS=-DCOT
CASES=simple-binding1 simple-binding2
# Just for reproducing
$(EXAMPLES)/%.aps: ;
# Generate output and store it in a file
%.output: $(EXAMPLES)/%.aps
# echo is only for reproducing
echo $(APSSCHED) $(FLAGS) -p $(BASE) $< > $#
# Copy actual output as expected
%.expected: %.output
cp -f $< $#
# Compare actual output with expected
.PHONY: %.compare
%.compare: %.output | %.expected
diff $| $<
# Generate and verify all outputs
.PHONY: all
all: $(addsuffix .compare,$(CASES))
# Regenerate expected output
.PHONY: build.expected
build.expected: $(addsuffix .expected,$(CASES))
.PHONY: clean.expected
clean.expected:
-rm -f *.expected
Now the make build.expected will create expected output files, while make all or make will check the actual output against expected:
$ make build.expected
echo ../../bin/apssched -DCOT -p .:../../base:../../examples ../../examples/simple-binding1.aps > simple-binding1.output
cp -f simple-binding1.output simple-binding1.expected
echo ../../bin/apssched -DCOT -p .:../../base:../../examples ../../examples/simple-binding2.aps > simple-binding2.output
cp -f simple-binding2.output simple-binding2.expected
rm simple-binding1.output simple-binding2.output
$ make
echo ../../bin/apssched -DCOT -p .:../../base:../../examples ../../examples/simple-binding1.aps > simple-binding1.output
diff simple-binding1.expected simple-binding1.output
echo ../../bin/apssched -DCOT -p .:../../base:../../examples ../../examples/simple-binding2.aps > simple-binding2.output
diff simple-binding2.expected simple-binding2.output
rm simple-binding1.output simple-binding2.output
I'm new to Makefiles and I want to modify the extension of a set of files. The following command works on the shell:
for file in path/*.ext1; do j=`echo $file | cut -d . -f 1`;j=$j".ext2";echo mv $file $j; done
However, I'm not sure how to run this in a Makefile. I tried running
$(shell for file in path/*.ext1; do j=`echo $file | cut -d . -f 1`;j=$j".ext2";echo mv $file $j; done)
But this never did what I needed it to do. What do I need to do to make this work on the Makefile? How do I call it in a section?
The immediate answer to your question is that the $ character is special to make: it introduces a make variable. If you want to pass a $ to the shell, you'll have to write two of them: $$.
So, your shell function invocation would have to be written as:
$(shell for file in path/*.ext1; do j=`echo $$file | cut -d . -f 1`;j=$$j".ext2";echo mv $$file $$j; done)
However, this is almost certainly not a good way to do what you want. You don't really describe clearly what you want to do, however. If you just want to have a target in a makefile that can be invoked to make this change, you can use:
fixext:
for file in path/*.ext1; do \
j=`echo $$file | cut -d . -f 1`; \
j=$$j".ext2"; \
echo mv $$file $$j; \
done
Or, taking advantage of some useful shell shortcuts, you could just run:
fixext:
for file in path/*.ext1; do \
echo mv $$file $${file%.*}.ext2; \
done
Now if you run make fixext it will perform those steps.
But, a much more make-like way to do it would be to write a single rule that knows how to rename one file, then use prerequisites to have them all renamed:
TARGETS = $(patsubst %.ext1,%.ext2,$(wildcard path/*.ext1))
fixext: $(TARGETS)
%.ext2 : %.ext1
mv $< $#
Now you can even run make -j5 and do 5 of the move commands in parallel...
you can also add rename blocks at the top of your file eg to change a suffix
output := $(input:.mov=.mp4)
but this won't work inside a make command as far as I can see
check:
output := $(input:.mov=.mp4)
gives
$ input=walkthrough.mov make check
output := walkthrough.mp4
make: output: No such file or directory
make: *** [check] Error 1
I'm trying to create a makefile (GNU make) that does the following:
A script generates a bunch of files--filenames not known in advance.
Each one of these files is converted to a different file.
After all are converted, all of these files are combined into a single output file.
How do I create a makefile with a "bellcurve"-patterned dependency graph, where the intermediate source and target files are not known in advance?
Conceptually I'm doing the following:
combined.pdf: $(filter combined.pdf, $(wildcard *.pdf))
cat *.pdf > combined.pdf
%.pdf: %.svg
cp $^ $#
$(wildcard *.svg):
# recipe is for simple example
# actually the *.svg files are not known in advance
echo a > a.svg
echo b > b.svg
echo c > c.svg
.PHONY: clean
clean:
${RM} *.svg *.pdf *.d
Of course this doesn't work: Make evaluates the targets and sources before it runs the target that actually creates the svg. Also, there's no way to make sure all svgs are converted before they are combined.
I realized I could create dependencies and include them into the makefile, but I had trouble getting this to work too:
.PHONY: clean
include deps.d
combined.pdf: deps.d
cat *.pdf > combined.pdf
%.pdf: %.svg
cp $^ $#
deps.d:
## recipe is for simple example
## actually the *.svg files are not known in advance
echo a > a.svg
echo b > b.svg
echo c > c.svg
## we know what files exist now, so we can establish dependencies
## "a.pdf : a.svg"
echo *.svg : > deps.d
## combined.pdf: a.pdf b.pdf c.pdf
ls *.svg \
| awk '{targetfn=$$0; sub(/\.svg$$/, ".pdf", targetfn); print targetfn, ":", $$0;}' \
>> deps.d
## combined.pdf: a.pdf b.pdf c.pdf
echo combined.pdf : $$(echo *.svg | sed -e 's/\.svg/\.pdf/g') >> deps.d
clean:
${RM} *.pdf *.svg *.d
However this still isn't connecting the dependency graph properly. When I run this, make quits as follows:
Makefile:3: deps.d: No such file or directory
echo a > a.svg
echo b > b.svg
echo c > c.svg
echo *.svg : > deps.d
ls *.svg \
| awk '{targetfn=$0; sub(/\.svg$/, ".pdf", targetfn); print targetfn, ":", $0;}' \
>> deps.d
echo combined.pdf : $(echo *.svg | sed -e 's/\.svg/\.pdf/g') >> deps.d
make: Nothing to be done for `a.svg'.
I still seem to have the problem that the make doesn't know about the rules in deps.d.
Also, this still doesn't solve the problem of building all the dependencies. I thought of using a marker file like this:
%.pdf: %.svg
cp $^ $#
## if all svgs are converted, touch a target allpdfs
if [ $(ls -1 *.svg | wc -l) -eq $(ls -1 *.pdf | grep -v combined\.pdf | wc -l) ]; touch allpdfs; fi
But there's no way to inform make that "allpdfs" may be created by this rule.
I'm surprised that moving the include directive makes a difference (what version of Make are you using?), but there is a simpler way. Your use of deps.d is in effect a recursive use of Make -- Make is arranging to execute itself a second time -- so we might as well make it official:
combined.pdf: ALL_SVGS
$(MAKE) ALL_PDFS
rm -f $# # just in case it exists already
cat *.pdf > $#
.PHONY: ALL_SVGS
ALL_SVGS:
# recipe is for simple example
# actually the *.svg files are not known in advance
echo a > a.svg
echo b > b.svg
echo c > c.svg
# These variables will be empty in the first execution of Make
SVGS = $(wildcard *.svg)
PDFS = $(patsubst %.svg,%.pdf,$(SVGS))
.PHONY: ALL_PDFS
ALL_PDFS: $(PDFS))
%.pdf: %.svg
cp $^ $#
This isn't an answer exactly, because I don't know why this works, but I discovered that if I move the include directive after the target that creates the included file, everything works.
I.e. do this:
deps.d:
....
include deps.d
Because my deps.d includes enough dependency information, there's no need to have an intermediate target allpdfs file. Everything Just Works, even with make -j.
However, I don't know why this works. The include documentation isn't enlightening me.
UPDATE
I noticed the following note at the very bottom of the make manual discussing Automatic Prerequisites:
Note that the ‘.d’ files contain target definitions; you should be sure to place the include directive after the first, default goal in your makefiles or run the risk of having a random object file become the default goal. See How Make Works.
So what happened is that the first rule inside the generated deps.d became the default target, causing the mysterious premature completion of the build. So the solution is just to make sure include directives are not before your intended default target.
I was just working on this exact problem in a slightly different setting. Here is a clean solution - no need for recursion and such (and you can tweak the sed if you like):
include deps.d
combined.pdf:
cat *.pdf > combined.pdf
%.pdf: %.svg
cp $^ $#
deps.d:
echo a > a.svg
echo b > b.svg
echo c > c.svg
echo 'combined.pdf:' *.svg | sed 's/\.svg/\.pdf/g' > deps.d
Enjoy!
I'm making a Makefile that moves an output file (foo.o) to a different directory (baz).
The output file moves as desired to the directory. However since make won't recompile the output file if I type make again, mv gets an error when it tries to move the non-existent empty file to the directory baz.
So this is what I have defined in my rule make all after all compilation:
-test -e "foo.o" || mv -f foo.o ../baz
Unfortunately, I'm still getting errors.
Errors in Recipes (from TFM)
To ignore errors in a recipe line, write a - at the beginning of the
line's text (after the initial tab).
So the target would be something like:
moveit:
-mv foo.o ../baz
I notice nobody has actually answered the original question itself yet, specifically how to ignore errors (all the answers are currently concerned with only calling the command if it won't cause an error).
To actually ignore errors, you can simply do:
mv -f foo.o ../baz 2>/dev/null; true
This will redirect stderr output to null, and follow the command with true (which always returns 0, causing make to believe the command succeeded regardless of what actually happened), allowing program flow to continue.
+#[ -d $(dir $#) ] || mkdir -p $(dir $#)
is what I use to silently create a folder if it does not exist. For your problem something like this should work
-#[ -e "foo.o" ] && mv -f foo.o ../baz
-test -e "foo.o" || if [ -f foo.o ]; then mv -f foo.o ../baz; fi;
That should work
Something like
test -e "foo.o" && mv -f foo.o ../baz
should work: the operator should be && instead of ||.
You can experiment with this by trying these commands:
test -e testfile && echo "going to move the file"
test -e testfile || echo "going to move the file"
I faced the same problem and as I am generating files, they always have different time. Workaround is set the same time to the files: touch -d '1 June 2018 11:02' file. In that case, gzip generates the same output and same md5sum. In my scenario, I don't need the time for the files.