Parameterised makefile function - makefile

I'm using a makefile to run docker, where I first collect some modules to download, so that they can be cached and then run docker. I wanted to parameterize this, but I don't think I'm doing this in the best way. Pointers to make this more concise would be really appreciated.
franz:
$(eval REPO_VERSION := $(shell grep franz requirements/github.txt | cut -d'#' -f3 | cut -d'#' -f1))
if [ -d docker/franz ]; then \
echo "Updating franz to [$(REPO_VERSION)]"; \
cd docker/franz && git fetch && git checkout $(REPO_VERSION); \
else \
echo "Cloning franz to [$(REPO_VERSION)]"; \
git clone --branch $(REPO_VERSION) git#github.com:dubizzle/franz.git docker/franz 2> /dev/null; \
fi \
lilith:
$(eval REPO_VERSION := $(shell grep lilith requirements/github.txt | cut -d'#' -f3 | cut -d'#' -f1))
if [ -d docker/lilith ]; then \
echo "Updating lilith to [$(REPO_VERSION)]"; \
cd docker/lilith && git fetch && git checkout $(REPO_VERSION); \
else \
echo "Cloning lilith to [$(REPO_VERSION)]"; \
git clone --branch $(REPO_VERSION) git#github.com:dubizzle/lilith.git docker/lilith 2> /dev/null; \
fi \
dependencies: franz lilith
git archive --format tar.gz --output docker/archive.tar.gz $(GIT_REF)
Basically, this first updates requirements that are on github, downloads them, checks what version is needed, and then updates to that version. If this could be made a function, a parameterised version would be:
$(eval REPO_VERSION := $(shell grep <repo-name> requirements/github.txt | cut -d'#' -f3 | cut -d'#' -f1))
if [ -d docker/<repo-name> ]; then \
echo "Updating <repo-name> to [$(REPO_VERSION)]"; \
cd docker/<repo-name> && git fetch && git checkout $(REPO_VERSION); \
else \
echo "Cloning <repo-name> to [$(REPO_VERSION)]"; \
git clone --branch $(REPO_VERSION) git#github.com:dubizzle/<repo-name>.git docker/<repo-name> 2> /dev/null; \
fi \
I've seen some examples using define, and call, and eval, but, I can't figure out the right combination to make it work.
Any help with this would be much appreciated.

From the tutorial mentioned above, to pull the information to SO.
This is GNU make, mind.
Defining a template:
define RULES_template
$(1)/obj/%.o: $(1)/src/%.c
$$(CC) $$(CFLAGS) $$(CFLAGS_global) $$(CFLAGS_$(1)) -c $$< -o $$#
endef
This uses one parameter ($(1)), which gets substituted as appropriate. The number of parameters is not declared, you just add $(1), $(2) etc. to the template. Note the duplication of $$ everywhere else.
$(foreach module,$(MODULES),$(eval $(call RULES_template,$(module))))
This calls the template mentioned above for each token in $(MODULES).
call RULES_template,foo instantiates the template with one parameter, foo. eval then parses the output as Makefile syntax (as opposed to, for example, putting it into some variable).
This has been ages ago, and I never used that code in a productive environment, so I am a bit fuzzy on the details. I hope it helps, anyway.
CMake not only is cross-platform, but also has much better primitives to handle sophisticated build mechanics. I can recommend it.

Related

using sed in Makefile inside docker container

I am using a debian-based docker container to build a LaTeX project. The following rule succeeds when run on the host (not inside docker):
.PHONY : timetracking
timetracking:
$(eval TODAY := $(if $(PAGE),$(PAGE),$(shell TZ=$(TIMEZ) date +%Y-%m-%d)))
touch $(PAGES)/$(WEEKLY)/$(TODAY).tex
cat template/page-header-footer/head.tex > $(PAGES)/$(WEEKLY)/$(TODAY).tex;
cat template/page-header-footer/pagestart.tex >> $(PAGES)/$(WEEKLY)/$(TODAY).tex;
echo {Week of $(TODAY)} >> $(PAGES)/$(WEEKLY)/$(TODAY).tex;
cat template/page-header-footer/timetracking.tex >> $(PAGES)/$(WEEKLY)/$(TODAY).tex;
cat template/page-header-footer/tail.tex >> $(PAGES)/$(WEEKLY)/$(TODAY).tex;
cat $(PAGES)/$(WEEKLY)/$(TODAY).tex \
| sed 's/1 January/'"$$(TZ=$(TIMEZ) date +'%d %B')/g" \
| sed 's/Jan 1/'"$$(TZ=$(TIMEZ) date +'%b %d')/g" \
| sed 's/Jan 2/'"$$(TZ=$(TIMEZ) date +'%b %d' -d '+1 days')/g" \
| sed 's/Jan 3/'"$$(TZ=$(TIMEZ) date +'%b %d' -d '+2 days')/g" \
| sed 's/Jan 4/'"$$(TZ=$(TIMEZ) date +'%b %d' -d '+3 days')/g" \
| sed 's/Jan 5/'"$$(TZ=$(TIMEZ) date +'%b %d' -d '+4 days')/g" \
> $(PAGES)/$(WEEKLY)/$(TODAY).tex;
but when the same rule is run within the docker container, it has variable behavior:
Succeeds (file generated as expected)
Creates a blank file (unexpected)
Creates a file filled with NUL characters (unexpected)
This behavior is a result of the modifications made with sed. The template files have some text containing "January 1" and "Jan 1", "Jan 2", "Jan 3", etc. which are to be replaced.
I would like help understanding:
why does this rule behave erratically inside docker
how can I rewrite the rule to behave reliably with docker
At the moment I can run this rule (and others like it) on the host, so long as I have basic tools like Make and sed installed. But it would be ideal if I could dockerize the entire workflow.
By request, the Dockerfile contents are below. Most of the installation instructions are irrelevant since this question is around make and sed. The tools directory contains a deb file for pandoc, and is also irrelevant to this question.
FROM debian:buster
RUN apt -y update
RUN apt -y install vim
RUN apt -y install make
RUN apt -y install texlive-full
RUN apt -y install biber
RUN apt -y install bibutils
RUN apt -y install python-pygments
RUN apt -y install cysignals-tools
RUN apt -y install sagemath
RUN apt -y install python-sagetex
RUN apt -y install sagetex
COPY tools /tools
RUN dpkg -i /tools/*deb
WORKDIR /results
ENTRYPOINT ["/usr/bin/make"]
There's a race condition in your shell syntax. When you run
cat file.tex \
| sed ... \
> file.tex
first the shell opens the output file for writing (processing the > file.tex), then it creates the various subprocesses and starts them, and then at the end of this cat(1) opens the output file for reading. It's possible, but not guaranteed, that the "open for write" step will truncate the file before the "open for read" step gets any content from it.
The easiest way to get around this is to have sed(1) edit the file in place using its -i option. This isn't a POSIX sed option, but both GNU sed (Debian/Ubuntu images) and BusyBox (Alpine images) support it. sed(1) supports multiple -e options to run multiple expressions, so you can use a single sed command to do this.
# (Bourne shell syntax, not escaped for Make)
sed \
-e 's/1 January/'"$(TZ=$(TIMEZ) date +'%d %B')/g" \
-e 's/Jan 1/'"$(TZ=$(TIMEZ) date +'%b %d')/g" \
-e 's/Jan 2/'"$(TZ=$(TIMEZ) date +'%b %d' -d '+1 days')/g" \
-e 's/Jan 3/'"$(TZ=$(TIMEZ) date +'%b %d' -d '+2 days')/g" \
-e 's/Jan 4/'"$(TZ=$(TIMEZ) date +'%b %d' -d '+3 days')/g" \
-e sed 's/Jan 5/'"$(TZ=$(TIMEZ) date +'%b %d' -d '+4 days')/g" \
-i \
$(PAGES)/$(WEEKLY)/$(TODAY).tex
Be careful with this option, though. In GNU sed, -i optionally takes an extension parameter to keep a backup copy of the file, and the optional parameter can have confusing syntax. In BusyBox sed, -i does not take a parameter. In BSD sed (MacOS hosts) the parameter is required.
If you have to deal with this ambiguity, you can work around it by separately creating and renaming the file.
sed e 's/.../.../g' -e 's/.../.../g' ... \
$(PAGES)/$(WEEKLY)/$(TODAY).tex \
> $(PAGES)/$(WEEKLY)/$(TODAY).tex.new
mv $(PAGES)/$(WEEKLY)/$(TODAY).tex.new $(PAGES)/$(WEEKLY)/$(TODAY).tex
In a Make context you might just treat these as separate files.
# lots GNU Make extensions
export TZ=$(TIMEZ)
TODAY := $(if $(PAGE),$(PAGE),$(shell date +%Y-%m-%d))
BASENAME := $(PAGES)/$(WEEKLY)/$(TODAY)
.PHONY: timestamps
timestamps: $(BASENAME).pdf
$(BASENAME).pdf: $(BASENAME).tex
pdflatex $<
$(BASENAME).tex: $(BASENAME)-original.tex
sed \
-e "s/1 January/$$(date +'%d %B')/g" \
...
$< > $#
$(BASENAME)-original.tex: \
template/page-header-footer/head.tex \
template/page-header-footer/pagestart.tex \
template/page-header-footer/timetracking.tex \
template/page-header-footer/tail.tex
cat template/page-header-footer/head.tex > $#
cat template/page-header-footer/pagestart.tex >> $#
echo {Week of $(TODAY)} >> $#
cat template/page-header-footer/timetracking.tex >> $#
cat template/page-header-footer/tail.tex >> $#
I've taken advantage of Make's automatic variables to reduce repetition here: $# is the current target (on the left-hand side of the rule name, the file we're building) and $< is its first dependency (the first thing after the colon).
You also may consider whether some of this can be done in TeX itself. For example, there are packages to format date stamps and built-in macros to include files. If you can put all of this in the .tex file itself then you don't need the complex Make syntax.

Using wget in shell trouble with variable that has \

I'm trying to run a script for pulling finance history from yahoo. Boris's answer from this thread
wget can't download yahoo finance data any more
works for me ~2 out of 3 times, but fails if the crumb returned from the cookie has a "\" character in it.
Code that sometimes works looks like this
#!usr/bin/sh
symbol=$1
today=$(date +%Y%m%d)
tomorrow=$(date --date='1 days' +%Y%m%d)
first_date=$(date -d "$2" '+%s')
last_date=$(date -d "$today" '+%s')
wget --no-check-certificate --save-cookies=cookie.txt https://finance.yahoo.com/quote/$symbol/?p=$symbol -O C:/trip/stocks/stocknamelist/crumb.store
crumb=$(grep 'root.*App' crumb.store | sed 's/,/\n/g' | grep CrumbStore | sed 's/"CrumbStore":{"crumb":"\(.*\)"}/\1/')
echo $crumb
fileloc=$"https://query1.finance.yahoo.com/v7/finance/download/$symbol?period1=$first_date&period2=$last_date&interval=1d&events=history&crumb=$crumb"
echo $fileloc
wget --no-check-certificate --load-cookies=cookie.txt $fileloc -O c:/trip/stocks/temphistory/hs$symbol.csv
rm cookie.txt crumb.store
But that doesn't seem to process in wget the way I intend either, as it seems to be interpreting as described here:
https://askubuntu.com/questions/758080/getting-scheme-missing-error-with-wget
Any suggestions on how to pass the $crumb variable into wget so that wget doesn't error out if $crumb has a "\" character in it?
Edited to show the full script. To clarify I've got cygwin installed with wget package. I call the script from cmd prompt as (example where the script above is named "stocknamedownload.sh, the stock symbol I'm downloading is "A" from the startdate 19800101)
c:\trip\stocks\StockNameList>bash stocknamedownload.sh A 19800101
This script seems to work fine - unless the crumb returned contains a "\" character in it.
The following implementation appears to work 100% of the time -- I'm unable to reproduce the claimed sporadic failures:
#!/usr/bin/env bash
set -o pipefail
symbol=$1
today=$(date +%Y%m%d)
tomorrow=$(date --date='1 days' +%Y%m%d)
first_date=$(date -d "$2" '+%s')
last_date=$(date -d "$today" '+%s')
# store complete webpage text in a variable
page_text=$(curl --fail --cookie-jar cookies \
"https://finance.yahoo.com/quote/$symbol/?p=$symbol") || exit
# extract the JSON used by JavaScript in the page
app_json=$(grep -e 'root.App.main = ' <<<"$page_text" \
| sed -e 's#^root.App.main = ##' \
-e 's#[;]$##') || exit
# use jq to extract the crumb from that JSON
crumb=$(jq -r \
'.context.dispatcher.stores.CrumbStore.crumb' \
<<<"$app_json" | tr -d '\r') || exit
# Perform our actual download
fileloc="https://query1.finance.yahoo.com/v7/finance/download/$symbol?period1=$first_date&period2=$last_date&interval=1d&events=history&crumb=$crumb"
curl --fail --cookie cookies "$fileloc" >"hs$symbol.csv"
Note that the tr -d '\r' is only necessary when using a native-Windows jq mixed with an otherwise native-Cygwin set of tools.
You are adding quotes to the value of the variable instead of quoting the expansion. You are also trying to use tools that don't know what JSON is to process JSON; use jq.
wget --no-check-certificate \
--save-cookies=cookie.txt \
"https://finance.yahoo.com/quote/$symbol/?p=$symbol" \
-O C:/trip/stocks/stocknamelist/crumb.store
# Something like thist; it's hard to reverse engineer the structure
# of crumb.store from your pipeline.
crumb=$(jq 'CrumbStore.crumb' crumb.store)
echo "$crumb"
fileloc="https://query1.finance.yahoo.com/v7/finance/download/$symbol?period1=$first_date&period2=$last_date&interval=1d&events=history&crumb=$crumb"
echo "$fileloc"
wget --no-check-certificate \
--load-cookies=cookie.txt "$fileloc" \
-O c:/trip/stocks/temphistory/hs$symbol.csv

How to define subroutines in a Makefile

I am working on a Makefile which has a¹ receipt producing some file using M4. It uses some complex shell constructions to compute macro values which have to be passed to M4. How can I organize code to avoid redundant declarations displayed in the following example?
M4TOOL= m4
M4TOOL+= -D PACKAGE=$$(cd ${PROJECTBASEDIR} && ${MAKE} -V PACKAGE)
M4TOOL+= -D VERSION=$$(cd ${PROJECTBASEDIR} && ${MAKE} -V VERSION)
M4TOOL+= -D AUTHOR=$$(cd ${PROJECTBASEDIR} && ${MAKE} -V AUTHOR)
M4TOOL+= -D RDC960=$$(openssl rdc960 ${DISTFILE} | cut -d ' ' -f 2)
M4TOOL+= -D SHA256=$$(openssl sha256 ${DISTFILE} | cut -d ' ' -f 2)
Portfile: Portfile.m4
${M4TOOL} ${.ALLSRC} > ${.TARGET}
¹ Actually a lot!
You should define pseudo-commands using the -c option of the shell, like this:
PROJECTVARIABLE=sh -c 'cd ${PROJECTBASEDIR} && ${MAKE} -V $$1' PROJECTVARIABLE
OPENSSLHASH=sh -c 'openssl $$1 $$2 | cut -d " " -f 2' OPENSSLHASH
Note the use of $ or $$ to use bsdmake variable expansion or shell variable expansion. With these defintions you can reorganise your code like this:
M4TOOLS+= -D PACKAGE=$$(${PROJECTVARIABLE} PACKAGE)
M4TOOLS+= -D VERSION=$$(${PROJECTVARIABLE} VERSION)
M4TOOLS+= -D AUTHOR=$$(${PROJECTVARIABLE} AUTHOR)
M4TOOLS+= -D RMD160=$$(${OPENSSLHASH} rmd160 ${DISTFILE})
M4TOOLS+= -D SHA256=$$(${OPENSSLHASH} sha256 ${DISTFILE})
The result is arguably easier to read and maintain. When you write such scripts, remember to use error codes and stderr to report errors.
PS: You can take a look at the COPYTREE_SHARE macro in /usr/ports/Mk/bsd.port.mk on a FreeBSD system. It illustrates well the technique.

BASH WHILE loop not hitting CASE switch

Hey guys title pretty much says it, but I'm echoing two variables into a BASH loop to kick it off, and (supposedly) using a case to be able to identify where they are and run a similar but separate (missing -k flag on second go around) wget statement. I hit my git checkout but it doesn't seem like I'm entering my cases. How do I fix this, or is there a better way to do it since I'm just dropping a -k flag?
#!/bin/bash
echo -e "render\nstorage" | while read x; do
git checkout "$x"
case $x in
$1)
wget "${WGDOMAIN}" -r -l INF -k -p \
--no-check-certificate \
--strict-comments \
--warc-header="Operator: Web Archiver" \
--warc-file="$WGDOMAIN" \
--warc-dedup="${WGDOMAIN}.cdx" \
--warc-cdx=on 2> session.log
;;
$2)
wget "${WGDOMAIN}" -r -l INF -p \
--no-check-certificate \
--strict-comments \
--warc-header="Operator: Web Archiver" \
--warc-file="$WGDOMAIN" \
--warc-dedup="${WGDOMAIN}.cdx" \
--warc-cdx=on 2> session.log
;;
$1|$2)
git add . && git ci -m"Archived: ${DATE}"
git push origin "$x"
;;
esac
done
Called with no positional parameters your script will not do anything inside the case statement as $1 and $2 are empty. Besides that, the last case option $1|$2 will never be reached as the prior ones will match. May be you should get that commands out of the case.

How to get the file size on Unix in a Makefile?

I would like to implement this as a Makefile task:
# step 1:
curl -u username:password -X POST \
-d '{"name": "new_file.jpg","size": 114034,"description": "Latest release","content_type": "text/plain"}' \
https://api.github.com/repos/:user/:repo/downloads
# step 2:
curl -u username:password \
-F "key=downloads/octocat/Hello-World/new_file.jpg" \
-F "acl=public-read" \
-F "success_action_status=201" \
-F "Filename=new_file.jpg" \
-F "AWSAccessKeyId=1ABCDEF..." \
-F "Policy=ewogIC..." \
-F "Signature=mwnF..." \
-F "Content-Type=image/jpeg" \
-F "file=#new_file.jpg" \
https://github.s3.amazonaws.com/
In the first part however, I need to get the file size (and content type if it's easy, not required though), so some variable:
{"name": "new_file.jpg","size": $(FILE_SIZE),"description": "Latest release","content_type": "text/plain"}
I tried this but it doesn't work (Mac 10.6.7):
$(shell du path/to/file.js | awk '{print $1}')
Any ideas how to accomplish this?
If you have GNU coreutils:
FILE_SIZE=$(stat -L -c %s $filename)
The -L tells it to follow symlinks; without it, if $filename is a symlink it will give you the size of the symlink rather than the size of the target file.
The MacOS stat equivalent appears to be:
FILE_SIZE=$(stat -L -f %z)
but I haven't been able to try it. (I've written this as a shell command, not a make command.) You may also find the -s option useful:
Display information in "shell output", suitable for initializing variables.
For reference, an alternative method is using du with -b bytes output and -s for summary only. Then cut to only keep the first element of the return string
FILE_SIZE=$(du -sb $filename | cut -f1)
This should return the same result in bytes as #Keith Thompson answer, but will also work for full directory sizes.
Extra: I usually use a macro for this.
define sizeof
$$(du -sb \
$(1) \
| cut -f1 )
endef
Which can then be called like,
$(call sizeof,$filename_or_dirname)
I think this is a case where parsing the output of ls is legitimate:
% FILE_SIZE=`ls -l $filename | awk '{print $5}'`
(no it's not: use stat, as noted by Keith Thompson)
For the type, you can use
% FILE_TYPE=`file --mime-type --brief $filename`

Resources