Multi-line bash commands in makefile - bash

Considering that every command is run in its own shell, what is the best way to run a multi-line bash command in a makefile? For example, like this:
for i in `find`
do
all="$all $i"
done
gcc $all

You can use backslash for line continuation. However note that the shell receives the whole command concatenated into a single line, so you also need to terminate some of the lines with a semicolon:
foo:
for i in `find`; \
do \
all="$$all $$i"; \
done; \
gcc $$all
But if you just want to take the whole list returned by the find invocation and pass it to gcc, you actually don't necessarily need a multiline command:
foo:
gcc `find`
Or, using a more shell-conventional $(command) approach (notice the $ escaping though):
foo:
gcc $$(find)

As indicated in the question, every sub-command is run in its own shell. This makes writing non-trivial shell scripts a little bit messy -- but it is possible! The solution is to consolidate your script into what make will consider a single sub-command (a single line).
Tips for writing shell scripts within makefiles:
Escape the script's use of $ by replacing with $$
Convert the script to work as a single line by inserting ; between commands
If you want to write the script on multiple lines, escape end-of-line with \
Optionally start with set -e to match make's provision to abort on sub-command failure
This is totally optional, but you could bracket the script with () or {} to emphasize the cohesiveness of a multiple line sequence -- that this is not a typical makefile command sequence
Here's an example inspired by the OP:
mytarget:
{ \
set -e ;\
msg="header:" ;\
for i in $$(seq 1 3) ; do msg="$$msg pre_$${i}_post" ; done ;\
msg="$$msg :footer" ;\
echo msg=$$msg ;\
}

The ONESHELL directive allows to write multiple line recipes to be executed in the same shell invocation.
all: foo
SOURCE_FILES = $(shell find . -name '*.c')
.ONESHELL:
foo: ${SOURCE_FILES}
FILES=()
for F in $^; do
FILES+=($${F})
done
gcc "$${FILES[#]}" -o $#
There is a drawback though : special prefix characters (‘#’, ‘-’, and ‘+’) are interpreted differently.
https://www.gnu.org/software/make/manual/html_node/One-Shell.html

Of course, the proper way to write a Makefile is to actually document which targets depend on which sources. In the trivial case, the proposed solution will make foo depend on itself, but of course, make is smart enough to drop a circular dependency. But if you add a temporary file to your directory, it will "magically" become part of the dependency chain. Better to create an explicit list of dependencies once and for all, perhaps via a script.
GNU make knows how to run gcc to produce an executable out of a set of .c and .h files, so maybe all you really need amounts to
foo: $(wildcard *.h) $(wildcard *.c)

What's wrong with just invoking the commands?
foo:
echo line1
echo line2
....
And for your second question, you need to escape the $ by using $$ instead, i.e. bash -c '... echo $$a ...'.
EDIT: Your example could be rewritten to a single line script like this:
gcc $(for i in `find`; do echo $i; done)

Related

/bin/sh: -c: line 1: syntax error: unexpected end of file in bash [duplicate]

Considering that every command is run in its own shell, what is the best way to run a multi-line bash command in a makefile? For example, like this:
for i in `find`
do
all="$all $i"
done
gcc $all
You can use backslash for line continuation. However note that the shell receives the whole command concatenated into a single line, so you also need to terminate some of the lines with a semicolon:
foo:
for i in `find`; \
do \
all="$$all $$i"; \
done; \
gcc $$all
But if you just want to take the whole list returned by the find invocation and pass it to gcc, you actually don't necessarily need a multiline command:
foo:
gcc `find`
Or, using a more shell-conventional $(command) approach (notice the $ escaping though):
foo:
gcc $$(find)
As indicated in the question, every sub-command is run in its own shell. This makes writing non-trivial shell scripts a little bit messy -- but it is possible! The solution is to consolidate your script into what make will consider a single sub-command (a single line).
Tips for writing shell scripts within makefiles:
Escape the script's use of $ by replacing with $$
Convert the script to work as a single line by inserting ; between commands
If you want to write the script on multiple lines, escape end-of-line with \
Optionally start with set -e to match make's provision to abort on sub-command failure
This is totally optional, but you could bracket the script with () or {} to emphasize the cohesiveness of a multiple line sequence -- that this is not a typical makefile command sequence
Here's an example inspired by the OP:
mytarget:
{ \
set -e ;\
msg="header:" ;\
for i in $$(seq 1 3) ; do msg="$$msg pre_$${i}_post" ; done ;\
msg="$$msg :footer" ;\
echo msg=$$msg ;\
}
The ONESHELL directive allows to write multiple line recipes to be executed in the same shell invocation.
all: foo
SOURCE_FILES = $(shell find . -name '*.c')
.ONESHELL:
foo: ${SOURCE_FILES}
FILES=()
for F in $^; do
FILES+=($${F})
done
gcc "$${FILES[#]}" -o $#
There is a drawback though : special prefix characters (‘#’, ‘-’, and ‘+’) are interpreted differently.
https://www.gnu.org/software/make/manual/html_node/One-Shell.html
Of course, the proper way to write a Makefile is to actually document which targets depend on which sources. In the trivial case, the proposed solution will make foo depend on itself, but of course, make is smart enough to drop a circular dependency. But if you add a temporary file to your directory, it will "magically" become part of the dependency chain. Better to create an explicit list of dependencies once and for all, perhaps via a script.
GNU make knows how to run gcc to produce an executable out of a set of .c and .h files, so maybe all you really need amounts to
foo: $(wildcard *.h) $(wildcard *.c)
What's wrong with just invoking the commands?
foo:
echo line1
echo line2
....
And for your second question, you need to escape the $ by using $$ instead, i.e. bash -c '... echo $$a ...'.
EDIT: Your example could be rewritten to a single line script like this:
gcc $(for i in `find`; do echo $i; done)

Declare variable in makefile using the bash for

I want to declare a variable using a for from bash in a makefile. That's what I tried:
SRCS="path/to/foo.c path/to/boo.c path/to/bar.c"
OBJS=$(for file in $SRCS; do TEMP="$TEMP $(basename $file .c).o"; done; echo $TEMP)
This command:
for file in $SRCS; do TEMP="$TEMP $(basename $file .c).o"; done
works on bash when echoing TEMP. But OBJS in makefile is empty. What am I missing here?
Desired output:
foo.o boo.o bar.o
First of all there's better ways to do this without using bash's for. You could use the make constructs to generate these lists. See the documentation on $(notdir), $(addprefix...) and $(addsuffix ...)
However, to answer your question on this particular example:
First, you are putting quotes around sources. Make does not interpret quotes as special characters, and thus SRCS will expand to "path/to/foo.c path/to/boo.c path/to/bar.c" (including the quotes). This will mess up your for loop later on.
The next thing is the reference to $SRCS -- make will interpret this as $S followed by the literal RCS (which is not what you want). You have to use braces around multi0letter variables in bash as so -- $(SRCS)
Next, TEMP=$TEMP.... When make sees $TEMP, it will immediately attempt to expand it. Because TEMP is not set to anything in the make context, it will expand to a null string -- this is before it invokes the bash shell... What you wanted to do in this case is use $$TEMP, which make will expand to $TEMP.
The following does what you want: note the $(info) lines are just for debugging.
SRCS:=path/to/foo.c path/to/boo.c path/to/bar.c
OBJS:=$(shell for file in $(SRCS); do TEMP="$$TEMP $$(basename $$file .c).o"; done; echo $$TEMP)
$(info for file in $(SRCS); do TEMP="$$TEMP $$(basename $$file .c).o"; done; echo $$TEMP)
$(info OBJS=$(OBJS))
If you wanted to do this in makefile without bash or sh, you might try:
OBJS:=$(notdir $(SRCS:.c=.o))
Note: this is gnu-make specific syntax and may not work on other makes.

how to work at the same time with multiple files inside file name array in linux shell script?

I have a shell script and i read all .s files in the specified folder first and then compile them to object file with a loop and after that link them to executable file.
this:
FILES=PTscalar_1.0/mibenchforpt/security/sha/*.s
for sfile in $FILES
do
echo "------------------------------------------------"
echo $sfile
objectFile="${sfile%.s}.o"
exefile="${objectFile%.o}.ex"
simplescalar/bin/sslittle-na-sstrix-as -o $objectFile $sfile
done
but I have a problem: in sha mibench program we have 2 files that each of them is in this flow:
.c -> .s -> .o
but at the last stage two .o files should be linked into one executable file.
how I can get two file names at the same time and create a command to link them.
main code is this:
simplescalar/bin/sslittle-na-sstrix-ld -o __sha.ex _sha.o _sha_driver.o
is there any way to see inside of FILES like this:
OFILES=PTscalar_1.0/mibenchforpt/security/sha/*.o
simplescalar/bin/sslittle-na-sstrix-ld -o $exefile OFILES[0] OFILES[1]
and after that doing that in a loop for all files with this pattern
first file is like *.o or *_main.o
second is: *_driver.o
Thanks
Obviously this is possible in shell. However many people find that the make utility is better for building software than shell scripts simply because of these dependencies. take a look at GNU Make. Its documentation contains numerous examples of what you're trying to do.
Caveat: Your tags "linux shell" do not specify a specific shell. POSIX sh, the standard specifying minimum required behavior for /bin/sh, does not support arrays; you should use a specific shell, such as bash or ksh, which does. To do this, you need to start your script with an appropriate shebang (such as #!/bin/bash instead of #!/bin/sh), and do any manual invocations with the correct shell (so bash -x myscript if you would otherwise use sh -x myscript... though if you've set the shebang correctly and have +x permissions, you can always just ./myscript)
# this is broken
FILES=PTscalar_1.0/mibenchforpt/security/sha/*.s
...does not create an array.
# this works in bash, ksh, and zsh
files=( PTscalar_1.0/mibenchforpt/security/sha/*.s )
does create an array, which can be expanded as "${files[#]}". So:
# this works in bash and ksh, and probably zsh
for file in "${files[#]}"; do
...
done
However, in this particular case, you don't have a reason to use an array at all:
# this works with absolutely any POSIX-compatible shell
for file in PTscalar_1.0/mibenchforpt/security/sha/*.s; do
echo "$sfile"
objectFile=${sfile%.s}.o
exefile=${objectFile%.o}.ex
simplescalar/bin/sslittle-na-sstrix-as -o "$objectFile" "$sfile"
done
Note a few corrections made in the above:
The right-hand-side of assignments in with no literal whitespace in their syntax do not need to be quoted.
All expansions (such as $objectFile) do need to be quoted, so, "$objectFile".
...yes, this does include echo; to test this, run s='*' and compare the output of echo $s to echo "$s".
To address the follow-up question you edited in:
ofiles=( PTscalar_1.0/mibenchforpt/security/sha/*.o )
simplescalar/bin/sslittle-na-sstrix-ld -o "$exefile" "${ofiles[0]}" "${ofiles[1]}"
...is a literal answer, but this would need to be edited if you had two or more outputs. Much better to do it this way instead:
ofiles=( PTscalar_1.0/mibenchforpt/security/sha/*.o )
simplescalar/bin/sslittle-na-sstrix-ld -o "$exefile" "${ofiles[#]}"
I created this file and it worked:
#!/bin/bash
#compile to assembly:
FILES=*_driver.s
for sdriverfile in $FILES
do
echo "------------------------------------------------"
# s file
echo $sdriverfile
sfile="${sdriverfile%_driver.s}.s"
echo $sfile
# object files
obj="${sfile%.s}.o"
obj_driver="${sdriverfile%.s}.o"
#exe file
exefile="${sfile%.s}_as.ex"
echo $exefile
#compile
/home/mahdi/programs/simplescalar/bin/sslittle-na-sstrix-as -o $obj $sfile
/home/mahdi/programs/simplescalar/bin/sslittle-na-sstrix-as -o $obj_driver $sdriverfile
#link
/home/mahdi/programs/simplescalar/bin/sslittle-na-sstrix-ld -o $exefile $obj $obj_driver -L /home/mahdi/programs/simplescalar/sslittle-na-sstrix/lib -lc -L /home/mahdi/programs/simplescalar/lib/gcc-lib/sslittle-na-sstrix/2.7.2.3/ -lgcc
done
thanks for answers.

How to split in GNU makefile list of files into separate lines

I have a variable containing list of files separated with string _NEWLINE_. I need to output that variable into a file so that each file is in a separate line. The trick is that it needs to works on FreeBSD and Solaris.
This is what I am trying now:
echo "lib/alarms-1.2/priv/snmp_conf/agent.conf: lib/alarms/priv/snmp_conf/agent.conf_NEWLINE_lib/alarms-1.2/priv/snmp_conf/agent.conf.src: lib/alarms/priv/snmp_conf/agent.conf.src_NEWLINE_lib/alarms-1.2/priv/snmp_conf/community.conf: lib/alarms/priv/snmp_conf/community.conf" | sed 's|_NEWLINE_|\'$'\n|g'
This works on FreeBSD and in shell on Solaris. But when run in GNUmakefile on Solaris I am getting this (notice $ at the end of each line):
lib/alarms-1.2/priv/snmp_conf/agent.conf: lib/alarms/priv/snmp_conf/agent.conf$
lib/alarms-1.2/priv/snmp_conf/agent.conf.src: lib/alarms/priv/snmp_conf/agent.conf.src$
lib/alarms-1.2/priv/snmp_conf/community.conf: lib/alarms/priv/snmp_conf/community.conf$
If I remove \'$' from sed then it works on Solaris but doesn't on FreeBSD. Maybe there is a way of telling which version to use depending on which system the makefile is executed?
EDIT:
Thanks to the solution proposed by bobbogo I created an exemplary makefile that provides the desired outcome and seems to be working on both FreeBSD and Solaris:
one-line := lib/alarms-1.2/priv/snmp_conf/agent.conf: lib/alarms/priv/snmp_conf/agent.conf_NEWLINE_lib/alarms-1.2/\
priv/snmp_conf/agent.conf.src: lib/alarms/priv/snmp_conf/agent.conf.src_NEWLINE_lib/alarms-1.2/priv/snmp_conf/comm\
unity.conf: lib/alarms/priv/snmp_conf/community.conf
many-lines := { echo '$(subst _NEWLINE_,' && echo ',${one-line})'; }
.PHONY: all
all:
$(shell $(many-lines) > test.txt)
If this is GNU make, then do it all in make.
one-line := lib/alarms-1.2/priv/snmp_conf/agent.conf: lib/alarms/priv/snmp_conf/agent.conf_NEWLINE_lib/alarms-1.2/priv/snmp_conf/agent.conf.src: lib/alarms/priv/snmp_conf/agent.conf.src_NEWLINE_lib/alarms-1.2/priv/snmp_conf/community.conf: lib/alarms/priv/snmp_conf/community.conf
define \n
endef
many-lines := $(subst _NEWLINE_,${\n},${one-line})
Now ${many-lines} has just what you want. Annoyingly, it's quite hard to use in shell lines. If you do this:
tgt:
echo '${many-lines}'
make will invoke a separate shell for each line. The first shell invocation will get an un-paired ' and exit with an error.
.ONESHELL:
tgt:
echo '${many-lines}'
will work in an invasive sort of way. The proper fix is to ensure each line of ${many-lines} has valid sh syntax. Some mouthfull like:
echolines = $(subst ${\n},'${\n}echo ',echo '${many-lines}')
.PHONY: aa
aa:
$(call echolines,${many-lines})
Sheesh.
Tried many different solutions, including defining \n as mentioned in this answer: Add a newline in Makefile 'foreach' loop
The real problem is inconsistent implementation of the echo command across platforms, and the fact that by default make invokes shell commands using sh, which itself is quite inflexible.
I found a better way thanks to this answer: "echo -e" when called directly and when called via a shell-script
The better way is to use printf instead of echo
Construct the string with \n instead of _NEWLINE_ to separate parts that go into separate lines in the output file:
some_string = lib/alarms-1.2/priv/snmp_conf/target_params.conf: lib/alarms/priv/snmp_conf/target_params.conf\nlib/alarms-1.2/priv/snmp_conf/community.conf: lib/alarms/priv/snmp_conf/community.conf\n
and then in the makefile print it simply as this:
#printf "$(some_string)" >> $(some_file)
Works on both, FreeBSD and Solaris.
Disclaimer: I have no experience with Solaris or FreeBSD... here goes anyway.
In make, you can use $(patsubst pattern,replacement,text) to substitute a pattern.
try this...
FILENAMES := "lib/alarms-1.2/priv/snmp_conf/agent.conf: lib/alarms/priv/snmp_conf/agent.conf_NEWLINE_lib/alarms-1.2/priv/snmp_conf/agent.conf.src: lib/alarms/priv/snmp_conf/agent.conf.src_NEWLINE_lib/alarms-1.2/priv/snmp_conf/community.conf: lib/alarms/priv/snmp_conf/community.conf"
.PHONY: all
all:
#echo $(patsubst _NEWLINE_,${\n},$(FILENAMES))
As an alternative, I think your first approach will work, if you just double the $ to "escape" it:
sed 's|_NEWLINE_|\'$$'\n|g'

How to use a for loop in make recipe [duplicate]

This question already has answers here:
How to use shell commands in Makefile
(2 answers)
Closed 9 months ago.
I would like to use a loop to find some files and rename them:
for i in `find $# -name *_cu.*`;do mv $i "$(echo $i|sed s/_cu//)"
done
This works in the shell. But how can I do this in a makefile recipe?
There are two main things you need to know when putting non-trivial shell fragments into make recipes:
Commands in the recipe are (of course!) executed one at a time, where command means "tab-prefixed line in the recipe", possibly spread over several makefile lines with backslashes.
So your shell fragment has to be written all on one (possibly backslashed) line. Moreover it's effectively presented to the shell as a single line (the backslashed-newlines are not plain newlines so are not used as command terminators by the shell), so must be syntactically correct as such.
Both shell variables and make variables are introduced by dollar signs ($#, $i), so you need to hide your shell variables from make by writing them as $$i. (More precisely, any dollar sign you want to be seen by the shell must be escaped from make by writing it as $$.)
Normally in a shell script you would write separate commands on separate lines, but here you effectively only get a single line so must separate the individual shell commands with semicolons instead. Putting all this together for your example produces:
foo: bar
for i in `find $# -name *_cu.*`; do mv $$i "$$(echo $$i|sed s/_cu//)"; done
or equivalently:
foo: bar
for i in `find $# -name *_cu.*`; do \
mv $$i "$$(echo $$i|sed s/_cu//)"; \
done
Notice that the latter, even though it's laid out readably on several lines, requires the same careful use of semicolons to keep the shell happy.
I found this useful, trying to use for loops to build multiple files:
PROGRAMS = foo bar other
.PHONY all
all: $(PROGRAMS)
$(PROGRAMS):
gcc -o $# $#.c
It will compile foo.c, bar.c, other.c into foor bar other executables
I spend good time on this and finally had it working. I had an easy solution using the global variable in makefile available for all targets, however I don`t want that so this is how I did it.
target:
$(eval test_cont=$(shell sh -c "docker ps | grep test" | awk '{print $$1}'))
for container in $(test_cont);do \
docker cp ssh/id_rsa.pub $${container}:/root/.ssh/authorized_keys; \
docker exec -it $${container} chown root.root /root/.ssh/authorized_keys; \
done

Resources