using sed in Makefile inside docker container - bash

I am using a debian-based docker container to build a LaTeX project. The following rule succeeds when run on the host (not inside docker):
.PHONY : timetracking
timetracking:
$(eval TODAY := $(if $(PAGE),$(PAGE),$(shell TZ=$(TIMEZ) date +%Y-%m-%d)))
touch $(PAGES)/$(WEEKLY)/$(TODAY).tex
cat template/page-header-footer/head.tex > $(PAGES)/$(WEEKLY)/$(TODAY).tex;
cat template/page-header-footer/pagestart.tex >> $(PAGES)/$(WEEKLY)/$(TODAY).tex;
echo {Week of $(TODAY)} >> $(PAGES)/$(WEEKLY)/$(TODAY).tex;
cat template/page-header-footer/timetracking.tex >> $(PAGES)/$(WEEKLY)/$(TODAY).tex;
cat template/page-header-footer/tail.tex >> $(PAGES)/$(WEEKLY)/$(TODAY).tex;
cat $(PAGES)/$(WEEKLY)/$(TODAY).tex \
| sed 's/1 January/'"$$(TZ=$(TIMEZ) date +'%d %B')/g" \
| sed 's/Jan 1/'"$$(TZ=$(TIMEZ) date +'%b %d')/g" \
| sed 's/Jan 2/'"$$(TZ=$(TIMEZ) date +'%b %d' -d '+1 days')/g" \
| sed 's/Jan 3/'"$$(TZ=$(TIMEZ) date +'%b %d' -d '+2 days')/g" \
| sed 's/Jan 4/'"$$(TZ=$(TIMEZ) date +'%b %d' -d '+3 days')/g" \
| sed 's/Jan 5/'"$$(TZ=$(TIMEZ) date +'%b %d' -d '+4 days')/g" \
> $(PAGES)/$(WEEKLY)/$(TODAY).tex;
but when the same rule is run within the docker container, it has variable behavior:
Succeeds (file generated as expected)
Creates a blank file (unexpected)
Creates a file filled with NUL characters (unexpected)
This behavior is a result of the modifications made with sed. The template files have some text containing "January 1" and "Jan 1", "Jan 2", "Jan 3", etc. which are to be replaced.
I would like help understanding:
why does this rule behave erratically inside docker
how can I rewrite the rule to behave reliably with docker
At the moment I can run this rule (and others like it) on the host, so long as I have basic tools like Make and sed installed. But it would be ideal if I could dockerize the entire workflow.
By request, the Dockerfile contents are below. Most of the installation instructions are irrelevant since this question is around make and sed. The tools directory contains a deb file for pandoc, and is also irrelevant to this question.
FROM debian:buster
RUN apt -y update
RUN apt -y install vim
RUN apt -y install make
RUN apt -y install texlive-full
RUN apt -y install biber
RUN apt -y install bibutils
RUN apt -y install python-pygments
RUN apt -y install cysignals-tools
RUN apt -y install sagemath
RUN apt -y install python-sagetex
RUN apt -y install sagetex
COPY tools /tools
RUN dpkg -i /tools/*deb
WORKDIR /results
ENTRYPOINT ["/usr/bin/make"]

There's a race condition in your shell syntax. When you run
cat file.tex \
| sed ... \
> file.tex
first the shell opens the output file for writing (processing the > file.tex), then it creates the various subprocesses and starts them, and then at the end of this cat(1) opens the output file for reading. It's possible, but not guaranteed, that the "open for write" step will truncate the file before the "open for read" step gets any content from it.
The easiest way to get around this is to have sed(1) edit the file in place using its -i option. This isn't a POSIX sed option, but both GNU sed (Debian/Ubuntu images) and BusyBox (Alpine images) support it. sed(1) supports multiple -e options to run multiple expressions, so you can use a single sed command to do this.
# (Bourne shell syntax, not escaped for Make)
sed \
-e 's/1 January/'"$(TZ=$(TIMEZ) date +'%d %B')/g" \
-e 's/Jan 1/'"$(TZ=$(TIMEZ) date +'%b %d')/g" \
-e 's/Jan 2/'"$(TZ=$(TIMEZ) date +'%b %d' -d '+1 days')/g" \
-e 's/Jan 3/'"$(TZ=$(TIMEZ) date +'%b %d' -d '+2 days')/g" \
-e 's/Jan 4/'"$(TZ=$(TIMEZ) date +'%b %d' -d '+3 days')/g" \
-e sed 's/Jan 5/'"$(TZ=$(TIMEZ) date +'%b %d' -d '+4 days')/g" \
-i \
$(PAGES)/$(WEEKLY)/$(TODAY).tex
Be careful with this option, though. In GNU sed, -i optionally takes an extension parameter to keep a backup copy of the file, and the optional parameter can have confusing syntax. In BusyBox sed, -i does not take a parameter. In BSD sed (MacOS hosts) the parameter is required.
If you have to deal with this ambiguity, you can work around it by separately creating and renaming the file.
sed e 's/.../.../g' -e 's/.../.../g' ... \
$(PAGES)/$(WEEKLY)/$(TODAY).tex \
> $(PAGES)/$(WEEKLY)/$(TODAY).tex.new
mv $(PAGES)/$(WEEKLY)/$(TODAY).tex.new $(PAGES)/$(WEEKLY)/$(TODAY).tex
In a Make context you might just treat these as separate files.
# lots GNU Make extensions
export TZ=$(TIMEZ)
TODAY := $(if $(PAGE),$(PAGE),$(shell date +%Y-%m-%d))
BASENAME := $(PAGES)/$(WEEKLY)/$(TODAY)
.PHONY: timestamps
timestamps: $(BASENAME).pdf
$(BASENAME).pdf: $(BASENAME).tex
pdflatex $<
$(BASENAME).tex: $(BASENAME)-original.tex
sed \
-e "s/1 January/$$(date +'%d %B')/g" \
...
$< > $#
$(BASENAME)-original.tex: \
template/page-header-footer/head.tex \
template/page-header-footer/pagestart.tex \
template/page-header-footer/timetracking.tex \
template/page-header-footer/tail.tex
cat template/page-header-footer/head.tex > $#
cat template/page-header-footer/pagestart.tex >> $#
echo {Week of $(TODAY)} >> $#
cat template/page-header-footer/timetracking.tex >> $#
cat template/page-header-footer/tail.tex >> $#
I've taken advantage of Make's automatic variables to reduce repetition here: $# is the current target (on the left-hand side of the rule name, the file we're building) and $< is its first dependency (the first thing after the colon).
You also may consider whether some of this can be done in TeX itself. For example, there are packages to format date stamps and built-in macros to include files. If you can put all of this in the .tex file itself then you don't need the complex Make syntax.

Related

Bash script - Some commands don't work in sh file

I have some troubles with my bash script. The end of my file doesn't work but every commands work outside the file. I have two strings as argument $1 and $2. $acl_line and $usebackend_line are numbers and they are good.
Here is my end file :
sed -i "$((acl_line+1))i \ \tacl\t\t is_$2_$1\t\thdr_com(host)\t-i $2.$1" /my_doc/haproxy/haproxy.cfg
sed -i "$((usebackend_line+1))i \ \tuse_backend\t$2_$1\tif is_$2_$1" /my_doc/haproxy/haproxy.cfg
echo -en "\nbackend $2_$1\n\tserver $2_$1 163.172.167.52:$3 maxconn 1024" >> /my_doc/haproxy/haproxy.cfg
cp -r "./model/*" "./script/lp_domains/$1/$2/"
sed -i 's/lp_ports/$ports/g' "./script/lp_domains/$1/$2/my_doc.yml"
sed -i 's/lp_name/$2-$1/g' "./script/lp_domains/$1/$2/my_doc.yml"
Thanks for your anwsers :)
If $1 and $2 should be interpolated, you cannot use single quotes.
Moreover, copying a file and then running sed -i on it is wasteful and error-prone. Just run sed and perform your substitutions at the same time.
sed -i -e "$((acl_line+1))i \ \tacl\t\t is_$2_$1\t\thdr_com(host)\t-i $2.$1" \
-e "$((usebackend_line+1))i \ \tuse_backend\t$2_$1\tif is_$2_$1" /my_doc/haproxy/haproxy.cfg \
-e "\$a\
backend $2_$1\n\tserver $2_$1 163.172.167.52:$3 maxconn 1024" /my_doc/haproxy/haproxy.cfg
# remove ./model/my_doc.yml; instead have a template ./my_doc.yml.in
cp -r "./model/*" "./script/lp_domains/$1/$2/"
sed -e "s/lp_ports/$ports/g" -e "s/lp_name/$2-$1/g" \
my_doc.yml.in >"./script/lp_domains/$1/$2/my_doc.yml"
(You should probably do something similar with haproxy.cfg.in actually.)
I have fixed my errors. It was just permissions errors, Sed create some temporary files so i add permissions to my user. Thanks for your help !

How to execute SED command from Makefile [duplicate]

This question already has answers here:
Write Dollar sign in sed command pattern inside makefile
(2 answers)
Closed last month.
So I am attempting to run a sed command from a Makefile on RaspberryPi running Raspbian. The commands I am using works perfectly when I type them directly into the terminal, but when I attempt to execute them from a Makefile I get the following feedback:
sed: -e expression #1, char 14: extra characters after command
#
# Quick and dirty Makefile for logsnag
#
CC = gcc
INCLUDE = -I.
CFILES = myThing.c
OBJS = myThing.o
CFLAGS = ${INCLUDE}
all: myThing
myThing: ${OBJS}
${CC} ${CFLAGS} ${OBJS} -o myThing
myThing.o: ${CFILES}
${CC} ${CFLAGS} myThing.c -c
install: myThing
sudo cp -f myThing/usr/local/bin
sudo cp -f ../bin/startlogging.sh /usr/local/bin
sudo cp -f ../cfg/rotateThing.cfg /etc
if [ ! -d /var/log/thingLog ]; then\
sudo mkdir /var/log/thingLog;\
fi;
sudo sed -i -e '$i touch /var/log/thingLog/thing.log /var/log/thingLog/myThing \n' /etc/rc.local;
sudo sed -i -e '$i logrotate -f /etc/rotateThing.cfg \n' /etc/rc.local;
sudo sed -i -e '$i touch /var/log/thingLog/thing.log /var/log/thingLog/myThing \n' /etc/rc.local;
sudo sed -i -e '$i /usr/local/bin/startlogging.sh > /var/log/thingLog/myThing 2>&1 & \n' /etc/rc.local;
clean:
rm -f myThing *.o
Your problem is that Makefile variable expansion looks a lot like shell variable expansion. That is, if you have a single-letter variable in a Makefile:
X=Some string
Then you refer to this variable like:
$X
So when you have a command like this in one of your build stanzas:
sed -i -e '$i /usr/local/bin/startlogging.sh > /var/log/thingLog/myThing 2>&1 & \n'
The $i gets replaced by make (with an empty string), resulting in
an invalid sed command. You can fix this by escaping the $ by
doubling it:
sed -i -e '$$i /usr/local/bin/startlogging.sh > /var/log/thingLog/myThing 2>&1 & \n'
This is discussed in the Make
documentation.

How to define subroutines in a Makefile

I am working on a Makefile which has a¹ receipt producing some file using M4. It uses some complex shell constructions to compute macro values which have to be passed to M4. How can I organize code to avoid redundant declarations displayed in the following example?
M4TOOL= m4
M4TOOL+= -D PACKAGE=$$(cd ${PROJECTBASEDIR} && ${MAKE} -V PACKAGE)
M4TOOL+= -D VERSION=$$(cd ${PROJECTBASEDIR} && ${MAKE} -V VERSION)
M4TOOL+= -D AUTHOR=$$(cd ${PROJECTBASEDIR} && ${MAKE} -V AUTHOR)
M4TOOL+= -D RDC960=$$(openssl rdc960 ${DISTFILE} | cut -d ' ' -f 2)
M4TOOL+= -D SHA256=$$(openssl sha256 ${DISTFILE} | cut -d ' ' -f 2)
Portfile: Portfile.m4
${M4TOOL} ${.ALLSRC} > ${.TARGET}
¹ Actually a lot!
You should define pseudo-commands using the -c option of the shell, like this:
PROJECTVARIABLE=sh -c 'cd ${PROJECTBASEDIR} && ${MAKE} -V $$1' PROJECTVARIABLE
OPENSSLHASH=sh -c 'openssl $$1 $$2 | cut -d " " -f 2' OPENSSLHASH
Note the use of $ or $$ to use bsdmake variable expansion or shell variable expansion. With these defintions you can reorganise your code like this:
M4TOOLS+= -D PACKAGE=$$(${PROJECTVARIABLE} PACKAGE)
M4TOOLS+= -D VERSION=$$(${PROJECTVARIABLE} VERSION)
M4TOOLS+= -D AUTHOR=$$(${PROJECTVARIABLE} AUTHOR)
M4TOOLS+= -D RMD160=$$(${OPENSSLHASH} rmd160 ${DISTFILE})
M4TOOLS+= -D SHA256=$$(${OPENSSLHASH} sha256 ${DISTFILE})
The result is arguably easier to read and maintain. When you write such scripts, remember to use error codes and stderr to report errors.
PS: You can take a look at the COPYTREE_SHARE macro in /usr/ports/Mk/bsd.port.mk on a FreeBSD system. It illustrates well the technique.

How to get the file size on Unix in a Makefile?

I would like to implement this as a Makefile task:
# step 1:
curl -u username:password -X POST \
-d '{"name": "new_file.jpg","size": 114034,"description": "Latest release","content_type": "text/plain"}' \
https://api.github.com/repos/:user/:repo/downloads
# step 2:
curl -u username:password \
-F "key=downloads/octocat/Hello-World/new_file.jpg" \
-F "acl=public-read" \
-F "success_action_status=201" \
-F "Filename=new_file.jpg" \
-F "AWSAccessKeyId=1ABCDEF..." \
-F "Policy=ewogIC..." \
-F "Signature=mwnF..." \
-F "Content-Type=image/jpeg" \
-F "file=#new_file.jpg" \
https://github.s3.amazonaws.com/
In the first part however, I need to get the file size (and content type if it's easy, not required though), so some variable:
{"name": "new_file.jpg","size": $(FILE_SIZE),"description": "Latest release","content_type": "text/plain"}
I tried this but it doesn't work (Mac 10.6.7):
$(shell du path/to/file.js | awk '{print $1}')
Any ideas how to accomplish this?
If you have GNU coreutils:
FILE_SIZE=$(stat -L -c %s $filename)
The -L tells it to follow symlinks; without it, if $filename is a symlink it will give you the size of the symlink rather than the size of the target file.
The MacOS stat equivalent appears to be:
FILE_SIZE=$(stat -L -f %z)
but I haven't been able to try it. (I've written this as a shell command, not a make command.) You may also find the -s option useful:
Display information in "shell output", suitable for initializing variables.
For reference, an alternative method is using du with -b bytes output and -s for summary only. Then cut to only keep the first element of the return string
FILE_SIZE=$(du -sb $filename | cut -f1)
This should return the same result in bytes as #Keith Thompson answer, but will also work for full directory sizes.
Extra: I usually use a macro for this.
define sizeof
$$(du -sb \
$(1) \
| cut -f1 )
endef
Which can then be called like,
$(call sizeof,$filename_or_dirname)
I think this is a case where parsing the output of ls is legitimate:
% FILE_SIZE=`ls -l $filename | awk '{print $5}'`
(no it's not: use stat, as noted by Keith Thompson)
For the type, you can use
% FILE_TYPE=`file --mime-type --brief $filename`

Wget page title

Is it possible to Wget a page's title from the command line?
input:
$ wget http://bit.ly/rQyhG5 <<code>>
output:
If it’s broke, fix it right - Keeping it Real Estate. Home
This script would give you what you need:
wget --quiet -O - http://bit.ly/rQyhG5 \
| sed -n -e 's!.*<title>\(.*\)</title>.*!\1!p'
But there are lots of situations where it breaks, including if there is a <title>...</title> in the body of the page, or if the title is on more than one line.
This might be a little better:
wget --quiet -O - http://bit.ly/rQyhG5 \
| paste -s -d " " \
| sed -e 's!.*<head>\(.*\)</head>.*!\1!' \
| sed -e 's!.*<title>\(.*\)</title>.*!\1!'
but it does not fit your case as your page contains the following head opening:
<head profile="http://gmpg.org/xfn/11">
Again, this might be better:
wget --quiet -O - http://bit.ly/rQyhG5 \
| paste -s -d " " \
| sed -e 's!.*<head[^>]*>\(.*\)</head>.*!\1!' \
| sed -e 's!.*<title>\(.*\)</title>.*!\1!'
but there is still ways to break it, including no head/title in the page.
Again, a better solution might be:
wget --quiet -O - http://bit.ly/rQyhG5 \
| paste -s -d " " \
| sed -n -e 's!.*<head[^>]*>\(.*\)</head>.*!\1!p' \
| sed -n -e 's!.*<title>\(.*\)</title>.*!\1!p'
but I am sure we can find a way to break it. This is why a true xml parser is the right solution, but as your question is tagged shell, the above it the best I can come with.
The paste and the 2 sed can be merged in a single sed, but is less readable. However, this version has the advantage of working on multi-line titles:
wget --quiet -O - http://bit.ly/rQyhG5 \
| sed -n -e 'H;${x;s!.*<head[^>]*>\(.*\)</head>.*!\1!;T;s!.*<title>\(.*\)</title>.*!\1!p}'
Update:
As explain in the comments, the last sed above uses the T command which is a GNU extension. If you do not have a compatible version, you can use:
wget --quiet -O - http://bit.ly/rQyhG5 \
| sed -n -e 'H;${x;s!.*<head[^>]*>\(.*\)</head>.*!\1!;tnext;b;:next;s!.*<title>\(.*\)</title>.*!\1!p}'
Update 2:
As above still not working on Mac, try:
wget --quiet -O - http://bit.ly/rQyhG5 \
| sed -n -e 'H;${x;s!.*<head[^>]*>\(.*\)</head>.*!\1!;tnext};b;:next;s!.*<title>\(.*\)</title>.*!\1!p'
and/or
cat << EOF > script
H
\$x
\$s!.*<head[^>]*>\(.*\)</head>.*!\1!
\$tnext
b
:next
s!.*<title>\(.*\)</title>.*!\1!p
EOF
wget --quiet -O - http://bit.ly/rQyhG5 \
| sed -n -f script
(Note the \ before the $ to avoid variable expansion.)
It seams that the :next does not like to be prefixed by a $, which could be a problem in some sed version.
The following will pull whatever lynx thinks the title of the page is, saving you from all of the regex nonsense. Assuming the page you are retrieving is standards compliant enough for lynx, this should not break.
lynx -dump example.com | sed '2q;d'

Resources