How to execute SED command from Makefile [duplicate] - bash

This question already has answers here:
Write Dollar sign in sed command pattern inside makefile
(2 answers)
Closed last month.
So I am attempting to run a sed command from a Makefile on RaspberryPi running Raspbian. The commands I am using works perfectly when I type them directly into the terminal, but when I attempt to execute them from a Makefile I get the following feedback:
sed: -e expression #1, char 14: extra characters after command
#
# Quick and dirty Makefile for logsnag
#
CC = gcc
INCLUDE = -I.
CFILES = myThing.c
OBJS = myThing.o
CFLAGS = ${INCLUDE}
all: myThing
myThing: ${OBJS}
${CC} ${CFLAGS} ${OBJS} -o myThing
myThing.o: ${CFILES}
${CC} ${CFLAGS} myThing.c -c
install: myThing
sudo cp -f myThing/usr/local/bin
sudo cp -f ../bin/startlogging.sh /usr/local/bin
sudo cp -f ../cfg/rotateThing.cfg /etc
if [ ! -d /var/log/thingLog ]; then\
sudo mkdir /var/log/thingLog;\
fi;
sudo sed -i -e '$i touch /var/log/thingLog/thing.log /var/log/thingLog/myThing \n' /etc/rc.local;
sudo sed -i -e '$i logrotate -f /etc/rotateThing.cfg \n' /etc/rc.local;
sudo sed -i -e '$i touch /var/log/thingLog/thing.log /var/log/thingLog/myThing \n' /etc/rc.local;
sudo sed -i -e '$i /usr/local/bin/startlogging.sh > /var/log/thingLog/myThing 2>&1 & \n' /etc/rc.local;
clean:
rm -f myThing *.o

Your problem is that Makefile variable expansion looks a lot like shell variable expansion. That is, if you have a single-letter variable in a Makefile:
X=Some string
Then you refer to this variable like:
$X
So when you have a command like this in one of your build stanzas:
sed -i -e '$i /usr/local/bin/startlogging.sh > /var/log/thingLog/myThing 2>&1 & \n'
The $i gets replaced by make (with an empty string), resulting in
an invalid sed command. You can fix this by escaping the $ by
doubling it:
sed -i -e '$$i /usr/local/bin/startlogging.sh > /var/log/thingLog/myThing 2>&1 & \n'
This is discussed in the Make
documentation.

Related

using sed in Makefile inside docker container

I am using a debian-based docker container to build a LaTeX project. The following rule succeeds when run on the host (not inside docker):
.PHONY : timetracking
timetracking:
$(eval TODAY := $(if $(PAGE),$(PAGE),$(shell TZ=$(TIMEZ) date +%Y-%m-%d)))
touch $(PAGES)/$(WEEKLY)/$(TODAY).tex
cat template/page-header-footer/head.tex > $(PAGES)/$(WEEKLY)/$(TODAY).tex;
cat template/page-header-footer/pagestart.tex >> $(PAGES)/$(WEEKLY)/$(TODAY).tex;
echo {Week of $(TODAY)} >> $(PAGES)/$(WEEKLY)/$(TODAY).tex;
cat template/page-header-footer/timetracking.tex >> $(PAGES)/$(WEEKLY)/$(TODAY).tex;
cat template/page-header-footer/tail.tex >> $(PAGES)/$(WEEKLY)/$(TODAY).tex;
cat $(PAGES)/$(WEEKLY)/$(TODAY).tex \
| sed 's/1 January/'"$$(TZ=$(TIMEZ) date +'%d %B')/g" \
| sed 's/Jan 1/'"$$(TZ=$(TIMEZ) date +'%b %d')/g" \
| sed 's/Jan 2/'"$$(TZ=$(TIMEZ) date +'%b %d' -d '+1 days')/g" \
| sed 's/Jan 3/'"$$(TZ=$(TIMEZ) date +'%b %d' -d '+2 days')/g" \
| sed 's/Jan 4/'"$$(TZ=$(TIMEZ) date +'%b %d' -d '+3 days')/g" \
| sed 's/Jan 5/'"$$(TZ=$(TIMEZ) date +'%b %d' -d '+4 days')/g" \
> $(PAGES)/$(WEEKLY)/$(TODAY).tex;
but when the same rule is run within the docker container, it has variable behavior:
Succeeds (file generated as expected)
Creates a blank file (unexpected)
Creates a file filled with NUL characters (unexpected)
This behavior is a result of the modifications made with sed. The template files have some text containing "January 1" and "Jan 1", "Jan 2", "Jan 3", etc. which are to be replaced.
I would like help understanding:
why does this rule behave erratically inside docker
how can I rewrite the rule to behave reliably with docker
At the moment I can run this rule (and others like it) on the host, so long as I have basic tools like Make and sed installed. But it would be ideal if I could dockerize the entire workflow.
By request, the Dockerfile contents are below. Most of the installation instructions are irrelevant since this question is around make and sed. The tools directory contains a deb file for pandoc, and is also irrelevant to this question.
FROM debian:buster
RUN apt -y update
RUN apt -y install vim
RUN apt -y install make
RUN apt -y install texlive-full
RUN apt -y install biber
RUN apt -y install bibutils
RUN apt -y install python-pygments
RUN apt -y install cysignals-tools
RUN apt -y install sagemath
RUN apt -y install python-sagetex
RUN apt -y install sagetex
COPY tools /tools
RUN dpkg -i /tools/*deb
WORKDIR /results
ENTRYPOINT ["/usr/bin/make"]
There's a race condition in your shell syntax. When you run
cat file.tex \
| sed ... \
> file.tex
first the shell opens the output file for writing (processing the > file.tex), then it creates the various subprocesses and starts them, and then at the end of this cat(1) opens the output file for reading. It's possible, but not guaranteed, that the "open for write" step will truncate the file before the "open for read" step gets any content from it.
The easiest way to get around this is to have sed(1) edit the file in place using its -i option. This isn't a POSIX sed option, but both GNU sed (Debian/Ubuntu images) and BusyBox (Alpine images) support it. sed(1) supports multiple -e options to run multiple expressions, so you can use a single sed command to do this.
# (Bourne shell syntax, not escaped for Make)
sed \
-e 's/1 January/'"$(TZ=$(TIMEZ) date +'%d %B')/g" \
-e 's/Jan 1/'"$(TZ=$(TIMEZ) date +'%b %d')/g" \
-e 's/Jan 2/'"$(TZ=$(TIMEZ) date +'%b %d' -d '+1 days')/g" \
-e 's/Jan 3/'"$(TZ=$(TIMEZ) date +'%b %d' -d '+2 days')/g" \
-e 's/Jan 4/'"$(TZ=$(TIMEZ) date +'%b %d' -d '+3 days')/g" \
-e sed 's/Jan 5/'"$(TZ=$(TIMEZ) date +'%b %d' -d '+4 days')/g" \
-i \
$(PAGES)/$(WEEKLY)/$(TODAY).tex
Be careful with this option, though. In GNU sed, -i optionally takes an extension parameter to keep a backup copy of the file, and the optional parameter can have confusing syntax. In BusyBox sed, -i does not take a parameter. In BSD sed (MacOS hosts) the parameter is required.
If you have to deal with this ambiguity, you can work around it by separately creating and renaming the file.
sed e 's/.../.../g' -e 's/.../.../g' ... \
$(PAGES)/$(WEEKLY)/$(TODAY).tex \
> $(PAGES)/$(WEEKLY)/$(TODAY).tex.new
mv $(PAGES)/$(WEEKLY)/$(TODAY).tex.new $(PAGES)/$(WEEKLY)/$(TODAY).tex
In a Make context you might just treat these as separate files.
# lots GNU Make extensions
export TZ=$(TIMEZ)
TODAY := $(if $(PAGE),$(PAGE),$(shell date +%Y-%m-%d))
BASENAME := $(PAGES)/$(WEEKLY)/$(TODAY)
.PHONY: timestamps
timestamps: $(BASENAME).pdf
$(BASENAME).pdf: $(BASENAME).tex
pdflatex $<
$(BASENAME).tex: $(BASENAME)-original.tex
sed \
-e "s/1 January/$$(date +'%d %B')/g" \
...
$< > $#
$(BASENAME)-original.tex: \
template/page-header-footer/head.tex \
template/page-header-footer/pagestart.tex \
template/page-header-footer/timetracking.tex \
template/page-header-footer/tail.tex
cat template/page-header-footer/head.tex > $#
cat template/page-header-footer/pagestart.tex >> $#
echo {Week of $(TODAY)} >> $#
cat template/page-header-footer/timetracking.tex >> $#
cat template/page-header-footer/tail.tex >> $#
I've taken advantage of Make's automatic variables to reduce repetition here: $# is the current target (on the left-hand side of the rule name, the file we're building) and $< is its first dependency (the first thing after the colon).
You also may consider whether some of this can be done in TeX itself. For example, there are packages to format date stamps and built-in macros to include files. If you can put all of this in the .tex file itself then you don't need the complex Make syntax.

Bash: Parse Urls from file, process them and then remove them from the file

I am trying to automate a procedure where the system will fetch the contents of a file (1 Url per line), use wget to grab the files from the site (https folder) and then remove the line from the file.
I have made several tries but the sed part (at the end) cannot understand the string (I tried escaping characters) and remove it from that file!
cat File
https://something.net/xxx/data/Folder1/
https://something.net/xxx/data/Folder2/
https://something.net/xxx/data/Folder3/
My line of code is:
cat File | xargs -n1 -I # bash -c 'wget -r -nd -l 1 -c -A rar,zip,7z,txt,jpg,iso,sfv,md5,pdf --no-parent --restrict-file-names=nocontrol --user=test --password=pass --no-check-certificate "#" -P /mnt/USB/ && sed -e 's|#||g' File'
It works up until the sed -e 's|#||g' File part..
Thanks in advance!
Dont use cat if it's posible. It's bad practice and can be problem with big files... You can change
cat File | xargs -n1 -I # bash -c
to
for siteUrl in $( < "File" ); do
It's be more correct and be simpler to use sed with double quotes... My variant:
scriptDir=$( dirname -- "$0" )
for siteUrl in $( < "$scriptDir/File.txt" )
do
if [[ -z "$siteUrl" ]]; then break; fi # break line if him empty
wget -r -nd -l 1 -c -A rar,zip,7z,txt,jpg,iso,sfv,md5,pdf --no-parent --restrict-file-names=nocontrol --user=test --password=pass --no-check-certificate "$siteUrl" -P /mnt/USB/ && sed -i "s|$siteUrl||g" "$scriptDir/File.txt"
done
#beliy answers looks good!
If you want a one-liner, you can do:
while read -r line; do \
wget -r -nd -l 1 -c -A rar,zip,7z,txt,jpg,iso,sfv,md5,pdf \
--no-parent --restrict-file-names=nocontrol --user=test \
--password=pass --no-check-certificate "$line" -P /mnt/USB/ \
&& sed -i -e '\|'"$line"'|d' "File.txt"; \
done < File.txt
EDIT:
You need to add a \ in front of the first pipe
I believe you just need to use double quotes after sed -e. Instead of:
'...&& sed -e 's|#||g' File'
you would need
'...&& sed -e '"'s|#||g'"' File'
I see what you trying to do, but I dont understand the sed command including pipes. Maybe some fancy format that I dont understand.
Anyway, I think the sed command should look like this...
sed -e 's/#//g'
This command will remove all # from the stream.
I hope this helps!

Bash script - grep output

I need an output for multiple grep commands.
patterns: ([^"#]+)
wget -q -O - http://www.site1.com | grep -o -E -m 1 'site1content = "([^"#]+)"'
wget -q -O - http://www.site2.com | grep -o -E -m 1 'site2content"([^"#]+)"
.........
Output file:
http://www.site1.com***pattern
http://www.site2.com***pattern
Just redirect the output of your commands to a file.
wget -q -O - http://www.site1.com | grep -o -E -m 1 'site1content = "([^"#]+)"' > output.txt
wget -q -O - http://www.site2.com | grep -o -E -m 1 'site2content"([^"#]+)"' >> output.txt
> overwrites old content and >> appends to the end of the file.
Edit:
Not pretty but a quick and dirty solution might be
echo 'http://www.site1.com***'`wget -q -O - http://www.site1.com | grep -o -E -m 1 'site1content = "([^"#]+)"'` > output.txt
(untested)
As is the output you got from the above commends consists only of the pattern found because of the -o parameter:
http://explainshell.com/explain?cmd=grep+-o
I suggest using the above site for an explanation.

echo quotes in bash script

I'm creating an automatic network configuration script and in it i have
#!/bin/bash
sudo rm /etc/default/ifplugd
sudo echo "INTERFACES=""
HOTPLUG_INTERFACES="wlan0 eth0"
ARGS="-q -f -u0 -d10 -w -I"
SUSPEND_ACTION="stop"" > /etc/default/ifplugd
however on viewing /etc/default/ifplugd some of the quotes are missing
INTERFACES=
HOTPLUG_INTERFACES=wlan0 eth0
ARGS=-q -f -u0 -d10 -w -I
SUSPEND_ACTION=stop
How do I configure the script so it includes the quotes between the first and last echo ones?
How about:
sudo sh -c 'cat <<END >/etc/default/ifplugd
INTERFACES=""
HOTPLUG_INTERFACES="wlan0 eth0"
ARGS="-q -f -u0 -d10 -w -I"
SUSPEND_ACTION="stop"
END
'
You don't need to explicitly rm, the > redirection will truncate the file before writing the new content.
You need to escape the " marks with a \ prefix, like this:
#!/bin/bash
sudo rm /etc/default/ifplugd
sudo echo "INTERFACES=\"\"
HOTPLUG_INTERFACES=\"wlan0 eth0\"
ARGS=\"-q -f -u0 -d10 -w -I\"
SUSPEND_ACTION=\"stop\"" > /etc/default/ifplugd
A heredoc provides an elegant solution:
sudo tee << EOF /etc/default/ifplugd
INTERFACES=""
HOTPLUG_INTERFACES="wlan0 eth0"
ARGS="-q -f -u0 -d10 -w -I"
SUSPEND_ACTION="stop"
EOF
This way, you don't have to manually quote each and every "" around, and you are not removing the ifplugd file, so you won't need to reset permissions after creating it.

how do I pass ' [single quote character] as argument in linux [bash shell]?

I've abc.py file which accepts argument -p [password] & -c [command].
Now I can run this file as follows :
./abc.py -p 'a!s!d!f' -c 'ifconfig'
a!s!d!f is my password. As password contains ! character, so I have to send it as argument in ' '. I tried to send it in " " but didn't work.
Now I want to run this code as follows :
./abc.py -p 'a!s!d!f' -c './abc.py -p 'a!s!d!f' -c 'ifconfig''
I'm giving ./abc.py -p 'a!s!d!f' -c 'ifconfig' as a argument to abc.py
The problem is, I'm unable to send ' characher as an argument to abc.py
I need this ' character to be sent as input.
I tried using \ escape character as:
./abc.py -p 'a!s!d!f' -c './abc.py -p \'a!s!d!f\' -c \'ifconfig\''
But not working. How do I do this? Any help would be greatly appreciated.
You need to quote both ' and !:
./abc.py -p 'a!s!d!f' -c "./abc.py -p 'a!s!d!f' -c 'ifconfig'"
$ cat p.py
import sys
print sys.argv
In Korn shell:
$ python p.py -p 'a!s!d!f' -c "./abc.py -p 'a!s!d!f' -c 'ifconfig'"
['p.py', '-p', 'a!s!d!f', '-c', "./abc.py -p 'a!s!d!f' -c 'ifconfig'"]
In bash ! is not treated specially only if enclosed in single quotes, so it can be done like this:
$ python p.py -p 'a!s!d!f' -c './abc.py -p '"'"'a!s!d!f'"'"' -c config'
['p.py', '-p', 'a!s!d!f', '-c', "./abc.py -p 'a!s!d!f' -c config"]
Notice that the result is different then when you quote the whole string with double quotes:
$ python p.py -c "./abcy.py -p 'a\!s\!d\!f' -c 'ifconfig'"
['p.py', '-c', "./abcy.py -p 'a\\!s\\!d\\!f' -c 'ifconfig'"]
In Bash (which follows the POSIX shell standard), single quotes preserve every character literally, which means there is no way to escape contents within single quotes. Your choices are:
Concatenate differently-quoted strings by placing them next to each other:
./abc.py -c "./abc.py -p '"'a!s!d!f'"' -c 'ifconfig'"
Use double-quotes and escape the ! characters:
./abc.py -c "./abcy.py -p 'a\!s\!d\!f' -c 'ifconfig'"

Resources