I want to read a file called metafile inside a makefile.
metafile looks something like this:
file1
file2
file3
I need to read this metafile inside my makefile line by line and check if the files mentioned inside exists or not and only print names of files which exists.
I've tried a few things without success. like:
FILE=`cat metafile`
for line in $(FILE); if [ -e $${line} ]; then echo $${line} fi; done;
You can put an arbitrary piece of shell script in a target. Keeping the file's contents in a Makefile variable does not make any sense to me, unless you also need the data in other targets for other reasons. (If so, you cannot use backticks, anyway.)
target:
#while read -r file; do \
test -e "$$file" && echo "$$file"; \
done <metafile
For what it's worth, the while loop is a safer and more idiomatic way to loop over a file's lines in a shell script than the for loop with backticks, even though you see that a lot.
The # prevents Make from echoing the shell script commands; take that out if for some reason you need to see them. In fact, I recommend against using this, especially while you are debugging -- use make -s to have make run silently once you are confident your recipe works correctly.
A more idiomatic way to do this in a Makefile is to have a target depend on these files, and use Make's own logic:
target: file1 file2 file3
#echo $(filter-out $?,$^)
This is GNU Make syntax; it might get more complex if you want to be portable to other Make flavors (to the point where maybe the shell script is preferable after all). It will echo everything on one line, but if you need separate lines, that should be a trivial fix.
I would simply build a small auxiliary Makefile snippet and include the dependencies:
target: target.d
target.d: metafile
sed 's/^/target: /' $< >$#
include target.d
This builds a small list of dependencies so you don't need to list them in the target: dependencies explicitly; so instead of file1 file2 file3 in the recipe above, the dependencies would live in the generated target.d which would contain
target: file1
target: file2
target: file3
You need to filter out the dependency on target.d (or leave it undeclared; I believe GNU Make should cope).
Related
Considering that every command is run in its own shell, what is the best way to run a multi-line bash command in a makefile? For example, like this:
for i in `find`
do
all="$all $i"
done
gcc $all
You can use backslash for line continuation. However note that the shell receives the whole command concatenated into a single line, so you also need to terminate some of the lines with a semicolon:
foo:
for i in `find`; \
do \
all="$$all $$i"; \
done; \
gcc $$all
But if you just want to take the whole list returned by the find invocation and pass it to gcc, you actually don't necessarily need a multiline command:
foo:
gcc `find`
Or, using a more shell-conventional $(command) approach (notice the $ escaping though):
foo:
gcc $$(find)
As indicated in the question, every sub-command is run in its own shell. This makes writing non-trivial shell scripts a little bit messy -- but it is possible! The solution is to consolidate your script into what make will consider a single sub-command (a single line).
Tips for writing shell scripts within makefiles:
Escape the script's use of $ by replacing with $$
Convert the script to work as a single line by inserting ; between commands
If you want to write the script on multiple lines, escape end-of-line with \
Optionally start with set -e to match make's provision to abort on sub-command failure
This is totally optional, but you could bracket the script with () or {} to emphasize the cohesiveness of a multiple line sequence -- that this is not a typical makefile command sequence
Here's an example inspired by the OP:
mytarget:
{ \
set -e ;\
msg="header:" ;\
for i in $$(seq 1 3) ; do msg="$$msg pre_$${i}_post" ; done ;\
msg="$$msg :footer" ;\
echo msg=$$msg ;\
}
The ONESHELL directive allows to write multiple line recipes to be executed in the same shell invocation.
all: foo
SOURCE_FILES = $(shell find . -name '*.c')
.ONESHELL:
foo: ${SOURCE_FILES}
FILES=()
for F in $^; do
FILES+=($${F})
done
gcc "$${FILES[#]}" -o $#
There is a drawback though : special prefix characters (‘#’, ‘-’, and ‘+’) are interpreted differently.
https://www.gnu.org/software/make/manual/html_node/One-Shell.html
Of course, the proper way to write a Makefile is to actually document which targets depend on which sources. In the trivial case, the proposed solution will make foo depend on itself, but of course, make is smart enough to drop a circular dependency. But if you add a temporary file to your directory, it will "magically" become part of the dependency chain. Better to create an explicit list of dependencies once and for all, perhaps via a script.
GNU make knows how to run gcc to produce an executable out of a set of .c and .h files, so maybe all you really need amounts to
foo: $(wildcard *.h) $(wildcard *.c)
What's wrong with just invoking the commands?
foo:
echo line1
echo line2
....
And for your second question, you need to escape the $ by using $$ instead, i.e. bash -c '... echo $$a ...'.
EDIT: Your example could be rewritten to a single line script like this:
gcc $(for i in `find`; do echo $i; done)
I have a shell script and i read all .s files in the specified folder first and then compile them to object file with a loop and after that link them to executable file.
this:
FILES=PTscalar_1.0/mibenchforpt/security/sha/*.s
for sfile in $FILES
do
echo "------------------------------------------------"
echo $sfile
objectFile="${sfile%.s}.o"
exefile="${objectFile%.o}.ex"
simplescalar/bin/sslittle-na-sstrix-as -o $objectFile $sfile
done
but I have a problem: in sha mibench program we have 2 files that each of them is in this flow:
.c -> .s -> .o
but at the last stage two .o files should be linked into one executable file.
how I can get two file names at the same time and create a command to link them.
main code is this:
simplescalar/bin/sslittle-na-sstrix-ld -o __sha.ex _sha.o _sha_driver.o
is there any way to see inside of FILES like this:
OFILES=PTscalar_1.0/mibenchforpt/security/sha/*.o
simplescalar/bin/sslittle-na-sstrix-ld -o $exefile OFILES[0] OFILES[1]
and after that doing that in a loop for all files with this pattern
first file is like *.o or *_main.o
second is: *_driver.o
Thanks
Obviously this is possible in shell. However many people find that the make utility is better for building software than shell scripts simply because of these dependencies. take a look at GNU Make. Its documentation contains numerous examples of what you're trying to do.
Caveat: Your tags "linux shell" do not specify a specific shell. POSIX sh, the standard specifying minimum required behavior for /bin/sh, does not support arrays; you should use a specific shell, such as bash or ksh, which does. To do this, you need to start your script with an appropriate shebang (such as #!/bin/bash instead of #!/bin/sh), and do any manual invocations with the correct shell (so bash -x myscript if you would otherwise use sh -x myscript... though if you've set the shebang correctly and have +x permissions, you can always just ./myscript)
# this is broken
FILES=PTscalar_1.0/mibenchforpt/security/sha/*.s
...does not create an array.
# this works in bash, ksh, and zsh
files=( PTscalar_1.0/mibenchforpt/security/sha/*.s )
does create an array, which can be expanded as "${files[#]}". So:
# this works in bash and ksh, and probably zsh
for file in "${files[#]}"; do
...
done
However, in this particular case, you don't have a reason to use an array at all:
# this works with absolutely any POSIX-compatible shell
for file in PTscalar_1.0/mibenchforpt/security/sha/*.s; do
echo "$sfile"
objectFile=${sfile%.s}.o
exefile=${objectFile%.o}.ex
simplescalar/bin/sslittle-na-sstrix-as -o "$objectFile" "$sfile"
done
Note a few corrections made in the above:
The right-hand-side of assignments in with no literal whitespace in their syntax do not need to be quoted.
All expansions (such as $objectFile) do need to be quoted, so, "$objectFile".
...yes, this does include echo; to test this, run s='*' and compare the output of echo $s to echo "$s".
To address the follow-up question you edited in:
ofiles=( PTscalar_1.0/mibenchforpt/security/sha/*.o )
simplescalar/bin/sslittle-na-sstrix-ld -o "$exefile" "${ofiles[0]}" "${ofiles[1]}"
...is a literal answer, but this would need to be edited if you had two or more outputs. Much better to do it this way instead:
ofiles=( PTscalar_1.0/mibenchforpt/security/sha/*.o )
simplescalar/bin/sslittle-na-sstrix-ld -o "$exefile" "${ofiles[#]}"
I created this file and it worked:
#!/bin/bash
#compile to assembly:
FILES=*_driver.s
for sdriverfile in $FILES
do
echo "------------------------------------------------"
# s file
echo $sdriverfile
sfile="${sdriverfile%_driver.s}.s"
echo $sfile
# object files
obj="${sfile%.s}.o"
obj_driver="${sdriverfile%.s}.o"
#exe file
exefile="${sfile%.s}_as.ex"
echo $exefile
#compile
/home/mahdi/programs/simplescalar/bin/sslittle-na-sstrix-as -o $obj $sfile
/home/mahdi/programs/simplescalar/bin/sslittle-na-sstrix-as -o $obj_driver $sdriverfile
#link
/home/mahdi/programs/simplescalar/bin/sslittle-na-sstrix-ld -o $exefile $obj $obj_driver -L /home/mahdi/programs/simplescalar/sslittle-na-sstrix/lib -lc -L /home/mahdi/programs/simplescalar/lib/gcc-lib/sslittle-na-sstrix/2.7.2.3/ -lgcc
done
thanks for answers.
I have a variable containing list of files separated with string _NEWLINE_. I need to output that variable into a file so that each file is in a separate line. The trick is that it needs to works on FreeBSD and Solaris.
This is what I am trying now:
echo "lib/alarms-1.2/priv/snmp_conf/agent.conf: lib/alarms/priv/snmp_conf/agent.conf_NEWLINE_lib/alarms-1.2/priv/snmp_conf/agent.conf.src: lib/alarms/priv/snmp_conf/agent.conf.src_NEWLINE_lib/alarms-1.2/priv/snmp_conf/community.conf: lib/alarms/priv/snmp_conf/community.conf" | sed 's|_NEWLINE_|\'$'\n|g'
This works on FreeBSD and in shell on Solaris. But when run in GNUmakefile on Solaris I am getting this (notice $ at the end of each line):
lib/alarms-1.2/priv/snmp_conf/agent.conf: lib/alarms/priv/snmp_conf/agent.conf$
lib/alarms-1.2/priv/snmp_conf/agent.conf.src: lib/alarms/priv/snmp_conf/agent.conf.src$
lib/alarms-1.2/priv/snmp_conf/community.conf: lib/alarms/priv/snmp_conf/community.conf$
If I remove \'$' from sed then it works on Solaris but doesn't on FreeBSD. Maybe there is a way of telling which version to use depending on which system the makefile is executed?
EDIT:
Thanks to the solution proposed by bobbogo I created an exemplary makefile that provides the desired outcome and seems to be working on both FreeBSD and Solaris:
one-line := lib/alarms-1.2/priv/snmp_conf/agent.conf: lib/alarms/priv/snmp_conf/agent.conf_NEWLINE_lib/alarms-1.2/\
priv/snmp_conf/agent.conf.src: lib/alarms/priv/snmp_conf/agent.conf.src_NEWLINE_lib/alarms-1.2/priv/snmp_conf/comm\
unity.conf: lib/alarms/priv/snmp_conf/community.conf
many-lines := { echo '$(subst _NEWLINE_,' && echo ',${one-line})'; }
.PHONY: all
all:
$(shell $(many-lines) > test.txt)
If this is GNU make, then do it all in make.
one-line := lib/alarms-1.2/priv/snmp_conf/agent.conf: lib/alarms/priv/snmp_conf/agent.conf_NEWLINE_lib/alarms-1.2/priv/snmp_conf/agent.conf.src: lib/alarms/priv/snmp_conf/agent.conf.src_NEWLINE_lib/alarms-1.2/priv/snmp_conf/community.conf: lib/alarms/priv/snmp_conf/community.conf
define \n
endef
many-lines := $(subst _NEWLINE_,${\n},${one-line})
Now ${many-lines} has just what you want. Annoyingly, it's quite hard to use in shell lines. If you do this:
tgt:
echo '${many-lines}'
make will invoke a separate shell for each line. The first shell invocation will get an un-paired ' and exit with an error.
.ONESHELL:
tgt:
echo '${many-lines}'
will work in an invasive sort of way. The proper fix is to ensure each line of ${many-lines} has valid sh syntax. Some mouthfull like:
echolines = $(subst ${\n},'${\n}echo ',echo '${many-lines}')
.PHONY: aa
aa:
$(call echolines,${many-lines})
Sheesh.
Tried many different solutions, including defining \n as mentioned in this answer: Add a newline in Makefile 'foreach' loop
The real problem is inconsistent implementation of the echo command across platforms, and the fact that by default make invokes shell commands using sh, which itself is quite inflexible.
I found a better way thanks to this answer: "echo -e" when called directly and when called via a shell-script
The better way is to use printf instead of echo
Construct the string with \n instead of _NEWLINE_ to separate parts that go into separate lines in the output file:
some_string = lib/alarms-1.2/priv/snmp_conf/target_params.conf: lib/alarms/priv/snmp_conf/target_params.conf\nlib/alarms-1.2/priv/snmp_conf/community.conf: lib/alarms/priv/snmp_conf/community.conf\n
and then in the makefile print it simply as this:
#printf "$(some_string)" >> $(some_file)
Works on both, FreeBSD and Solaris.
Disclaimer: I have no experience with Solaris or FreeBSD... here goes anyway.
In make, you can use $(patsubst pattern,replacement,text) to substitute a pattern.
try this...
FILENAMES := "lib/alarms-1.2/priv/snmp_conf/agent.conf: lib/alarms/priv/snmp_conf/agent.conf_NEWLINE_lib/alarms-1.2/priv/snmp_conf/agent.conf.src: lib/alarms/priv/snmp_conf/agent.conf.src_NEWLINE_lib/alarms-1.2/priv/snmp_conf/community.conf: lib/alarms/priv/snmp_conf/community.conf"
.PHONY: all
all:
#echo $(patsubst _NEWLINE_,${\n},$(FILENAMES))
As an alternative, I think your first approach will work, if you just double the $ to "escape" it:
sed 's|_NEWLINE_|\'$$'\n|g'
Considering that every command is run in its own shell, what is the best way to run a multi-line bash command in a makefile? For example, like this:
for i in `find`
do
all="$all $i"
done
gcc $all
You can use backslash for line continuation. However note that the shell receives the whole command concatenated into a single line, so you also need to terminate some of the lines with a semicolon:
foo:
for i in `find`; \
do \
all="$$all $$i"; \
done; \
gcc $$all
But if you just want to take the whole list returned by the find invocation and pass it to gcc, you actually don't necessarily need a multiline command:
foo:
gcc `find`
Or, using a more shell-conventional $(command) approach (notice the $ escaping though):
foo:
gcc $$(find)
As indicated in the question, every sub-command is run in its own shell. This makes writing non-trivial shell scripts a little bit messy -- but it is possible! The solution is to consolidate your script into what make will consider a single sub-command (a single line).
Tips for writing shell scripts within makefiles:
Escape the script's use of $ by replacing with $$
Convert the script to work as a single line by inserting ; between commands
If you want to write the script on multiple lines, escape end-of-line with \
Optionally start with set -e to match make's provision to abort on sub-command failure
This is totally optional, but you could bracket the script with () or {} to emphasize the cohesiveness of a multiple line sequence -- that this is not a typical makefile command sequence
Here's an example inspired by the OP:
mytarget:
{ \
set -e ;\
msg="header:" ;\
for i in $$(seq 1 3) ; do msg="$$msg pre_$${i}_post" ; done ;\
msg="$$msg :footer" ;\
echo msg=$$msg ;\
}
The ONESHELL directive allows to write multiple line recipes to be executed in the same shell invocation.
all: foo
SOURCE_FILES = $(shell find . -name '*.c')
.ONESHELL:
foo: ${SOURCE_FILES}
FILES=()
for F in $^; do
FILES+=($${F})
done
gcc "$${FILES[#]}" -o $#
There is a drawback though : special prefix characters (‘#’, ‘-’, and ‘+’) are interpreted differently.
https://www.gnu.org/software/make/manual/html_node/One-Shell.html
Of course, the proper way to write a Makefile is to actually document which targets depend on which sources. In the trivial case, the proposed solution will make foo depend on itself, but of course, make is smart enough to drop a circular dependency. But if you add a temporary file to your directory, it will "magically" become part of the dependency chain. Better to create an explicit list of dependencies once and for all, perhaps via a script.
GNU make knows how to run gcc to produce an executable out of a set of .c and .h files, so maybe all you really need amounts to
foo: $(wildcard *.h) $(wildcard *.c)
What's wrong with just invoking the commands?
foo:
echo line1
echo line2
....
And for your second question, you need to escape the $ by using $$ instead, i.e. bash -c '... echo $$a ...'.
EDIT: Your example could be rewritten to a single line script like this:
gcc $(for i in `find`; do echo $i; done)
This question already has answers here:
How to use shell commands in Makefile
(2 answers)
Closed 9 months ago.
I would like to use a loop to find some files and rename them:
for i in `find $# -name *_cu.*`;do mv $i "$(echo $i|sed s/_cu//)"
done
This works in the shell. But how can I do this in a makefile recipe?
There are two main things you need to know when putting non-trivial shell fragments into make recipes:
Commands in the recipe are (of course!) executed one at a time, where command means "tab-prefixed line in the recipe", possibly spread over several makefile lines with backslashes.
So your shell fragment has to be written all on one (possibly backslashed) line. Moreover it's effectively presented to the shell as a single line (the backslashed-newlines are not plain newlines so are not used as command terminators by the shell), so must be syntactically correct as such.
Both shell variables and make variables are introduced by dollar signs ($#, $i), so you need to hide your shell variables from make by writing them as $$i. (More precisely, any dollar sign you want to be seen by the shell must be escaped from make by writing it as $$.)
Normally in a shell script you would write separate commands on separate lines, but here you effectively only get a single line so must separate the individual shell commands with semicolons instead. Putting all this together for your example produces:
foo: bar
for i in `find $# -name *_cu.*`; do mv $$i "$$(echo $$i|sed s/_cu//)"; done
or equivalently:
foo: bar
for i in `find $# -name *_cu.*`; do \
mv $$i "$$(echo $$i|sed s/_cu//)"; \
done
Notice that the latter, even though it's laid out readably on several lines, requires the same careful use of semicolons to keep the shell happy.
I found this useful, trying to use for loops to build multiple files:
PROGRAMS = foo bar other
.PHONY all
all: $(PROGRAMS)
$(PROGRAMS):
gcc -o $# $#.c
It will compile foo.c, bar.c, other.c into foor bar other executables
I spend good time on this and finally had it working. I had an easy solution using the global variable in makefile available for all targets, however I don`t want that so this is how I did it.
target:
$(eval test_cont=$(shell sh -c "docker ps | grep test" | awk '{print $$1}'))
for container in $(test_cont);do \
docker cp ssh/id_rsa.pub $${container}:/root/.ssh/authorized_keys; \
docker exec -it $${container} chown root.root /root/.ssh/authorized_keys; \
done