What does sed 's expression do in a Makefile - makefile

I have Makefile from an older project which I have to modify for a new system. It has the following global definitions for its source and object files:
SRCFILES = $(wildcard *.c) $(wildcard tasks/*.c)
SRCDIR = .
OBJDIR = objdir
OBJFILES = $(SRCFILES:%.c=${OBJDIR}/%.o)
The Makefile was a mess and I had to modify it heavily. I manged to compile the code and generate the .o files from the .c source code. The expected .elf file is also compiled correctly.
the problem is that I can do not understand what the code is doing in the following lines I searched so many websites but no luck. any ideas?
sed 's,^\(.*\)\.o[ :]*,$(#D)/\1.o $(#D)/\1.d : ,g' < $#.$$$$ > $#; \
rm -f $#.$$$$
The makefile is as follows
lwip.elf: build_echo $(OBJFILES)
#set -e; echo Linking the object files...; \
sparc-elf-gcc $(OBJFILES) -o $#;
#echo Successfully linked the files
#echo lwip.elf file ready to load...
build_echo: httpserver_raw/fsdata.c
#echo Building the source files first...
#echo The Modified or New source code to be build are :
clean:
- rm -f lwip.elf
$(OBJDIR)/%.o: $(SRCDIR)/%.c
#set -e; echo ' $<'; \
sparc-elf-gcc-4.4.2 -Wall -O2 -g -msoft-float -mcpu=v8 -c -o $# $(CFLAGS) -I. -I lwip/include -I lwip/include/ipv4 -I lwip/leon3 $<
$(OBJDIR)/%.d: $(SRCDIR)/%.c
#set -e; rm -f $#; mkdir -p $(#D); \
sparc-elf-gcc -MM -I. -I lwip/include -I lwip/include/ipv4 -I lwip/leon3 $< > $#.$$$$; \
sed 's,^\(.*\)\.o[ :]*,$(#D)/\1.o $(#D)/\1.d : ,g' < $#.$$$$ > $#; \
rm -f $#.$$$$
-include $(patsubst %.c,$(OBJDIR)/%.d,$(SRCFILES))
Note: The sparc-elf-gcc is a special cross-compiler used for the sparcV8 architecture.

There's a lot going on in there:
sed 's,^\(.*\)\.o[ :]*,$(#D)/\1.o $(#D)/\1.d : ,g' < $#.$$$$ > $#; \
rm -f $#.$$$$
Let's break it down.
There's the sed command itself: sed 's,^\(.*\)\.o[ :]*,$(#D)/\1.o $(#D)/\1.d : ,g',
whose input is redirected (<) from a file designated by #.$$$$, and
whose output is redirected (>) to a file designated by $#, then
there's a second command, rm -f $#.$$$$.
Furthermore, that's in the context of a larger list of commands, which set it up with
#set -e; rm -f $#; mkdir -p $(#D); \
sparc-elf-gcc -MM -I. -I lwip/include -I lwip/include/ipv4 -I lwip/leon3 $< > $#.$$$$;
In interpreting that, you have to first recognize several make implicit variables:
$#, which represents the name of the target of the rule
$(#D), which represents the directory of the target name (i.e. the target's dirname)
$<, which represents the first prerequisite
Next, you need to recognize that the dollar sign has special meaning to make, so if you want a literal $ you need to double it. The $$$$ simply repeat this, so that what gets passed to the shell is just two dollar signs ($$). The shell, in turn, replaces that with its own process ID -- a relatively common (but nevertheless insecure) idiom for generating unique file names.
So, the part of the command that you omitted from your callout ensures that the target's destination directory exists, then runs a (cross-?) gcc with flags that cause it to emit information about header dependencies. These are captured in a file, something like objdir/src.d.12345, then in the part you called out, that file is redirected into sed, with the output going to the final target, something like objdir/src.d, after which the intermediate file is removed.
The sed command itself is performing a substitution (s) on all lines, replacing every occurrence (g) of the pattern ^\(.*\)\.o[ :]* with a replacement that I'll come back to in a moment. First the pattern. The ^ anchors the pattern to the beginning of a line. The \(\) are metacharacters delimiting a capturing group, and the .* inside matches any number of any characters. Following that must be a literal decimal point, \., an o, and zero or more occurrences or the space and colon characters ([ :]*). For example, that would match this, up to but not including the header name:
objdir/tasks/t1.o : my_header.h
The replacement uses the aforementioned $(#D) -- which is expanded by make, not the shell or sed -- and \1, which sed expands to the text matched by the first capturing group. Thus, that would transform the above example line to this:
objdir/tasks/t1.o objdir/tasks/t1.d : my_header.h
In short, then, all that to say that it munges the computed dependency list to make the dependency file that is the target of the rule list itself as depending on all the same files that the corresponding object file does.

Related

problems with kernel not being loaded

I'm creating an OS and when I compile the code nothing happened, simply nothing(No errors, warnings or anything) I"m thinking that the make file has some issues.
Makefile:
build_kernel:
echo "Building kernel..."
${ASM} ./src/kernel/kernel_entry.asm -f elf64 -o ${BUILD_DIR}/kernel_entry.o
${C_COMPILER} -c ./src/kernel/kernel.c -o ${BUILD_DIR}/kernel_start.o
${C_COMPILER} -c ./src/kernel/drivers/printutils.c -o ${BUILD_DIR}/kernel_printutils.o
${C_COMPILER} -c ./src/kernel/drivers/port.c -o ${BUILD_DIR}/kernel_ports.o
echo "kernel build complete."
link:
echo "Linking..."
${LINKER} -o ${BUILD_DIR}/kernel.bin ${BUILD_DIR}/kernel_ports.o \
${BUILD_DIR}/kernel_printutils.o ${BUILD_DIR}/kernel_start.o \
${BUILD_DIR}/kernel_entry.o -Ttext 0x1000 --oformat binary
echo "Linking complete"
run:
echo "Running qemu..."
qemu-system-x86_64 -fda ${BUILD_DIR}/os.bin
merge_binary:
echo "Merging binary..."
cat ${BUILD_DIR}/boot.bin ${BUILD_DIR}/kernel.bin > ${BUILD_DIR}/os.bin
echo "Binary merged."
post_build:
rm -f ${BUILD_DIR}/boot.bin
rm -f ${BUILD_DIR}/kernel.bin
rm -f ${BUILD_DIR}/kernel.o
rm -f ${BUILD_DIR}/kernel_entry.o
rm -f ${BUILD_DIR}/kernel_ports.o
rm -f ${BUILD_DIR}/kernel_printutils.o
rm -f ${BUILD_DIR}/kernel_start.o
I'm wondering what is happening with the makefile and is this the correct way to compile all the code into object files and link them.
Any help will be appreciated.
I'm just commenting on your make style -- this does not answer why you are outputting nothing (and it is unclear what you mean by that -- if you run make, it should output at least echo "Building kernel..."...). As far as the makefile style goes, this seems to be built using a scripting mentality rather than a make mentality. Consider your first part:
build_kernel:
echo "Building kernel..."
${ASM} ./src/kernel/kernel_entry.asm -f elf64 -o ${BUILD_DIR}/kernel_entry.o
${C_COMPILER} -c ./src/kernel/kernel.c -o ${BUILD_DIR}/kernel_start.o
${C_COMPILER} -c ./src/kernel/drivers/printutils.c -o ${BUILD_DIR}/kernel_printutils.o
${C_COMPILER} -c ./src/kernel/drivers/port.c -o ${BUILD_DIR}/kernel_ports.o
echo "kernel build complete."
This has several issues. First is the name -- this looks to build a bunch of artifacts rather than building the kernel. Also, the recipe never produces a file named build_kernel, thus this should have been a phony target. Next, this is actually a script, which builds four separate things. These could be separated out into four separate rules, each which builds one thing, and then the main target would be dependent on this. Thus, it might look like:
.PHONY: build_kernel_objs
build_kernel_objs: ${C_OBJS} ${ASM_OBJS}
#echo done building $#
${BUILD_DIR}/kernel_start.o : ./src/kernel/kernel.c
${C_COMPILER} -c $< -o $#
${BUILD_DIR}/kernel_printutils.o : ./src/kernel/kernel_printutils.c
${C_COMPILER} -c $< -o $#
${BUILD_DIR}/kernel_ports.o : ./src/kernel/kernel_ports.c
${C_COMPILER} -c $< -o $#
Note that the above is repetitive, and if you have hundreds of files, will bolat very quickly. This can also be done using static pattern rules:
C_FILES := \
./src/kernel/kernel_start.c
./src/kernel/kernel_printutils.c
./src/kernel/kernel_ports.c
ASM_FILES := \
./src/kernel/kernel_entry.asm
C_OBJS := ${C_FILES :./src/kernel/%.c=${BUILD_DIR}/%.o}
ASM_OBJS := ${ASM_FILES :./src/kernel/%.asm=${BUILD_DIR}/%.o}
${C_OBJS} : ${BUILD_DIR}/%.o : ./src/kernel/%.c
${C_COMPILER} -c $< $#
.PHONY: build_kernel_objs
build_kernel_objs: ${C_OBJS} ${ASM_OBJS}
#echo "done building $#"
These have several advantages over what you've done -- first, make will only ever build the objects that are out of date, so it doesn't do needless work. It can also build the files in parallel if a -j option is specified on the make command line. Next, it's more maintainable -- if you have to add extra files, you can do it in one place, and everything works out. Also, the .PHONY prevents the make from failing if you happen to have a file named build_kernel_objs in your make directory. Lastly, the # in front of the echo lines prevents the actual echo command from being echoed, which will look nicer.
On caveat to this is that it does not handle modification of header files (as written, if a header file is updated, c files that depend on it would not be rebuilt. See here for some notes about getting around that.
The next section, link, the makefile recipe should reflect the target.
.PHONY: link
link : ${BUILD_DIR}/kernel.bin
${BUILD_DIR}/kernel.bin: ${C_OBJS} ${ASM_OBJS}
${LINKER} -o $# $^ -Ttext 0x1000 --oformat binary
This creates a phony target link, so you can type make link. It will only do the link if any of the C objects or ASM objects have been updated. The same concept applies to your merge_binary target
For run, this seems to be somewhat contentious, but a common rule of thumb is that a make should be used to make an executable, not to run it. A separate shell script is better suited if you want to invoke your built target with specific parameters.
Lastly, your post_build rule should likely be renamed to CLEAN, and declared as a phony.

error in using tab in makefile how to fix it

when I try to do make for the makefile it shows me (makefile error missing separator. stop)
my make file is
PROGS = $(patsubst %.c,%,$(SRCS))
all: $(PROGS)
%: %.c
arm-linux-gnueabihf-gcc --static $< -o $#
clean:
rm -f $(PROGS)
I try to use the following command but I could not solve my problem
cat -e -t -v Makefile
SRCS^I=^I$(wildcard^I*.c)^I$
PROGS^I=^I$(patsubst^I%.c,%,$(SRCS))$
all:^I$(PROGS)^I$
%:^I%.c^I$
arm-linux-gnueabihf-gcc^I--static^I$<^I-o^I$#^I$
clean:^I$
rm^I-f^I$(PROGS)^I$ ```
Crazy. You have TAB characters in all the places you don't need them, and you don't have TAB characters in any of the (two) places you must have them. I don't know how you managed that.
In your makefile above the places you MUST have TAB chars are the recipe lines. In other words, your makefile should look like this from cat -e -t -v Makefile:
SRCS = $(wildcard *.c)$
PROGS = $(patsubst %.c,%,$(SRCS))$
all: $(PROGS)$
%: %.c$
^Iarm-linux-gnueabihf-gcc --static $< -o $#$
clean:$
^Irm -f $(PROGS)$
As for configuring your text editor, there are a ton of text editors out there, even on Ubuntu, and we have no idea which one you are using.
I recommend you use a programmer's editor, which has a special mode that understands makefile syntax.

Generate a directory if a target name has a directory part

I have this makefile code:
$(DIRS):
#echo " MKDIR build/tmp/base/socket/$#"
$(Q)mkdir -p $#/
%.a.s:
#echo " CC build/tmp/base/socket/$#"
$(Q)$(CC) $(CFLAGS_A) -S $< -o $#
%.so.s:
#echo " CC build/tmp/base/socket/$#"
$(Q)$(CC) $(CFLAGS_SO) -S $< -o $#
%.o: %.s
#echo " AS build/tmp/base/socket/$#"
$(Q)$(AS) $< -o $#
tcp/client.a.s: $(TCP_CLIENT_SRC) $(TCP_CLIENT_INC)
tcp/client.so.s: $(TCP_CLIENT_SRC) $(TCP_CLIENT_INC)
tcp/server.a.s: $(TCP_SERVER_SRC) $(TCP_SERVER_INC)
tcp/server.so.s: $(TCP_SERVER_SRC) $(TCP_SERVER_INC)
I'd like to add a | tcp in the prerequisites of all targets that have a leading tcp/ in their name, but I'd like to be able to do it in one line. I wouldn't want to append that manually to every line that needs it.
I thought of adding this new rule:
tcp/%.s: | tcp
but it isn't doing anything.
I also thought of the more generic, which would be nicer, but same results:
%.s: | $(#D)
How should I write that?
A workaround would be to call mkdir every time (include it in both %.a.s and %.so.s rules), but that would add unnecessary calls to mkdir, wouldn't it?
I am not a fan of pattern rules. They are too arbitrary for my tastes.
(What actually happens depends on what files you may have lying around on your hard disk.)
You cannot just add a prerequisite to a given set of targets using a pattern rule
Well you can if you use static pattern rules. This is a much nicer idiom. Here we prefix a pattern rule with the actual list of sources you want the pattern rule to apply to. This is good where you can describe dependencies using make's noddy pattern matching.
A sketch:
%.a: ; date >$# # Pattern rule
tcp: ; mkdir -p $# # Explicit rule
tcp/a.a: tcp/%.a: | tcp ; # Static pattern rule!
.PHONY: all
all: tcp/a.a
all: c.a
all: dir/b.a
all: ; : $# Success
And we have:
$ make all
mkdir -p tcp
date >tcp/a.a
date >c.a
date >dir/b.a
/bin/sh: dir/b.a: No such file or directory
make: *** [Makefile:3: dir/b.a] Error 1
Here we have told make that before it builds (i.e., "runs the recipe for") tcp/a.a, it must first build tcp. That works. We didn't tell make about the directory for dir/b.a, so that failed. Note that the .a file's recipe is still in an ordinary pattern rule. This is just by way of exposition. I would definitely change this.
Yeah, in this case the pattern rule for tcp/ is over-the-top. Consider though that before you create tcp/a.a, you might first need to create a auto/a.src (say).
tcp/a.a: tcp/%.a: auto/%.src | tcp ; # Static pattern rule!
Easily extensible.
targets := tcp/a.a tcp/b.a tcp/c.a
${targets}: tcp/%.a: auto/%.src | tcp ; # Static pattern rule!
[By the way, in your original makefile, your archive and shared object should depend on the .o files, not the source (???)]
You cannot just add a prerequisite to a given set of targets using a pattern rule. That's not how pattern rules work: they are rules: they must have a recipe associated with them. A pattern rule without a recipe actually deletes that pattern rule (see Canceling Implicit Rules).
You can either create two sets of pattern rules, one for all targets and a second one for just targets that start with tcp/ that have an extra prerequisite, but you have to write the entire pattern rule twice, including a recipe, not just the pattern line.
Or just put the mkdir into the recipe. A mkdir on a directory that already exists won't even be noticeable.
This answer is only to show a working Makefile implementing #bobbogo 's answer.
This Makefile is a leaf of a Makefile tree, so all variables not defined here are exported by upper Makefiles.
Makefile:
#! /usr/bin/make -f
################################################################################
# *AUTHOR*
# FULL NAME "Alejandro Colomar Andrés"
################################################################################
################################################################################
DIRS = \
$(CURDIR)/tcp
OBJ = \
$(CURDIR)/tcp/client.o \
$(CURDIR)/tcp/server.o \
$(CURDIR)/foo.o
SRC = \
$(SRC_DIR)/base/socket/tcp/client.c \
$(SRC_DIR)/base/socket/tcp/server.c \
$(SRC_DIR)/base/socket/foo.c
DEP = $(OBJ:.o=.d)
BOTH_OBJ = $(subst .a.o,.a.o ,$(join $(OBJ:.o=.a.o),$(OBJ:.o=.so.o)))
BOTH_ASM = $(subst .a.s,.a.s ,$(join $(OBJ:.o=.a.s),$(OBJ:.o=.so.s)))
NEEDDIR = $(DEP) $(BOTH_ASM)
################################################################################
PHONY := all
all: $(BOTH_OBJ)
#:
$(DIRS): $(CURDIR)/%:
#echo " MKDIR build/tmp/base/socket/$*"
$(Q)mkdir -p $#
$(NEEDDIR): | $(DIRS)
$(CURDIR)/%.d: $(SRC_DIR)/base/socket/%.c
#echo " CC -M build/tmp/base/socket/$*.d"
$(Q)$(CC) $(CFLAGS_A) -MG -MT"$#" \
-MT"$(CURDIR)/$*.a.s" -MT"$(CURDIR)/$*.so.s" \
-M $< -MF $#
$(CURDIR)/%.a.s: $(SRC_DIR)/base/socket/%.c $(CURDIR)/%.d
#echo " CC build/tmp/base/socket/$*.a.s"
$(Q)$(CC) $(CFLAGS_A) -S $< -o $#
$(CURDIR)/%.so.s: $(SRC_DIR)/base/socket/%.c $(CURDIR)/%.d
#echo " CC build/tmp/base/socket/$*.so.s"
$(Q)$(CC) $(CFLAGS_SO) -S $< -o $#
$(CURDIR)/%.o: $(CURDIR)/%.s
#echo " AS build/tmp/base/socket/$*.o"
$(Q)$(AS) $< -o $#
include $(DEP)
PHONY += clean
clean:
$(Q)rm -rf *.o *.s *.d
################################################################################
# Declare the contents of the PHONY variable as phony.
.PHONY: $(PHONY)
################################################################################
######## End of file ###########################################################
################################################################################
output:
MKDIR build/tmp/base/socket/tcp
CC -M build/tmp/base/socket/foo.d
CC -M build/tmp/base/socket/tcp/server.d
CC -M build/tmp/base/socket/tcp/client.d
CC build/tmp/base/socket/tcp/client.a.s
AS build/tmp/base/socket/tcp/client.a.o
CC build/tmp/base/socket/tcp/client.so.s
AS build/tmp/base/socket/tcp/client.so.o
CC build/tmp/base/socket/tcp/server.a.s
AS build/tmp/base/socket/tcp/server.a.o
CC build/tmp/base/socket/tcp/server.so.s
AS build/tmp/base/socket/tcp/server.so.o
CC build/tmp/base/socket/foo.a.s
AS build/tmp/base/socket/foo.a.o
CC build/tmp/base/socket/foo.so.s
AS build/tmp/base/socket/foo.so.o

Makefile: what does $#~ mean?

I am working on an old makefile that contains the following snippet to generate a shared library:
lib$(LIBNAME).so.$(SOLIBREV): $(OBJS)
$(RM) $#~
#SONAME=`echo $# | sed 's/\.[^\.]*$$//'`; set -x; \
$(CC) -o ./$#~ -shared -Wl,-soname,$$SONAME $(OBJS) $(SOEXTRALIBS) -lc;
$(MV) $#~ $#
$(MV) $# lib$(LIBNAME).so
Now I need to modify that. I know that $# specifies the target but what meaning does the tilde in "$#~" have?
By the way SOLIBREV stands for so-library-revision.
It doesn't mean anything special. It's just $# followed by a literal ~. A ~ suffix on filenames is often used for temporary files, so this recipe is using a temporary file named after the target but with the extra ~ suffix.

Using sed in a makefile; how to escape variables?

I am writing a generic makefile to build static libraries. It seems to work well so far, except for the line calling sed:
# Generic makefile to build a static library
ARCH = linux
CFLAGS = -O3 -Wall
SOURCES = src
BUILD_DIR = build/$(ARCH)
TARGET = $(BUILD_DIR)/libz.a
CFILES = $(foreach dir,$(SOURCES),$(wildcard $(dir)/*.c))
OBJECTS = $(addprefix $(BUILD_DIR)/,$(CFILES:.c=.o))
# Pull in the dependencies if they exist
# http://scottmcpeak.com/autodepend/autodepend.html
-include $(OBJECTS:.o=.dep)
default: create-dirs $(TARGET)
$(TARGET): $(OBJECTS)
$(AR) -rc $(TARGET) $^
$(BUILD_DIR)/%.o: %.c
$(CC) $(CFLAGS) -c $< -o $#
$(CC) -M $(CFLAGS) $*.c > $(BUILD_DIR)/$*.tmp
sed s/.*:/$(BUILD_DIR)\\/$*.o:/ $(BUILD_DIR)/$*.tmp > $(BUILD_DIR)/$*.dep
#rm $(BUILD_DIR)/$*.tmp
.PHONY: create-dirs
create-dirs:
#for p in $(SOURCES); do mkdir -p $(BUILD_DIR)/$$p; done
.PHONY: clean
clean:
rm -fr $(BUILD_DIR)
sed is used to replace the path/name of the object file with the full path of where the object actually is. e.g. 'src/foo.o:' is replaced with 'build/linux/src/foo.o:' in this example.
$(BUILD_DIR) and $* in the replacement string both contain forward slashes when expanded - how do I pass them to sed?
Note: This might have been answered here before, but I am so far unable to apply those answers to my specific problem!
You can use anything else than forward slashes as separator in sed. E.g. sed s~foo~bar~g
You can use double quotes " (at least in the shell), and variables will still be expanded: echo "Hello $PLANET"
If you want the path to expand into a file you use
sed -i "s/TO_REPLACE/$(subst, /,\/,${PATH_VARIABLE})/g" filename.txt
If your variable PATH_VARIABLE looked like /opt/path/user/home/etc
it will now inflate to:
\ /opt\ /path\ /user\ /home\ /etc
This should allow sed to insert the ' / ' correctly.
-Matt

Resources