Makefile Variable Assignment Executes Early - makefile

I have a Makefile rule that requires storing the results from shell commands into variables for later use. For some reason, it seems that the $(shell) call executes as soon as my rule is match, as opposed to when it is encountered during execution.
The file is as follows:
TMPDEV=/tmp/disk.img
$(TMPDEV):
fallocate -l 806354944 $(TMPDEV)
sudo parted --script $(TMPDEV) unit s mklabel msdos \
mkpart primary fat16 2048 526335 \
mkpart primary fat32 526336 1050623 \
mkpart primary NTFS 1050624 1574911 \
quit
$(eval TMPDISK := $(shell sudo partx --verbose -a $(TMPDEV) | tail -1 | cut -d':' -f1))
echo $(TMPDISK)
sudo mkfs.fat -F 16 -n FAT16 $(TMPDISK)p1
It is impossible to know what the value of TMPDISK will be until at least after the fallocate call; that is why the $(eval) statement is delayed until after the disk image is partitioned.
The output I receive is:
$ make
partx: stat failed /tmp/disk.img: No such file or directory
fallocate -l 806354944 /tmp/disk.img || dd if=/dev/zero of=/tmp/disk.img bs=1b count=1574912
sudo parted --script /tmp/disk.img unit s mklabel msdos \
mkpart primary fat16 2048 526335 \
mkpart primary fat32 526336 1050623 \
mkpart primary NTFS 1050624 1574911 \
quit
echo
The fact that partx errors out (and thus TMPDISK is set to empty) before any of the other commands execute makes me think that $(shell) is called earlier than intended. Is there anyway to delay the shell call and the assignment to TMPDISK until the appropriate line?

It is not possible to delay expansion like this. Make always expands all the variables in the entire recipe first, before it sends any part of the recipe to the shell. There is no way to "defer" that until later.
Generally, it's not typical to use $(shell ...) in a recipe, since the recipe is already running in a shell. And setting make variables inside a recipe via $(eval ...) is also highly unusual.
I would recommend that you rewrite this recipe to use shell variables, not make variables; it will be much more understandable:
TMPDEV=/tmp/disk.img
$(TMPDEV):
fallocate -l 806354944 $(TMPDEV)
sudo parted --script $(TMPDEV) unit s mklabel msdos \
mkpart primary fat16 2048 526335 \
mkpart primary fat32 526336 1050623 \
mkpart primary NTFS 1050624 1574911 \
quit
TMPDISK=$$(sudo partx --verbose -a $(TMPDEV) | tail -1 | cut -d':' -f1); \
echo $$TMPDISK; \
sudo mkfs.fat -F 16 -n FAT16 $${TMPDISK}p1

I had the same problem. The eval command expands even before the target command runs. It was answered here below for my question by Simon Gibbons.
makefile variable assignment under a target
The solution is to add another target for the "fallocate" command and add that as a dependency to $(TMPDEV) target.

Related

How to write numeric expressions in Makefile?

I want to write statements about Podman in Makefile. A UID mapping is used here. But I found that I was always unable to do numerical calculations.
Below is my Makefile. But here ${uid}+1 and similar operations will become empty strings. How should I solve this problem?
Thanks!
HOST_GEM5 := /mnt/disk/cuiyujie/workspace/workGem5/gem5
SIM := ${HOST_GEM5}/X86/gem5.opt
SHELL := /bin/bash
DOCKER_GEM5 := /usr/local/src/gem5
subuidSize=$(shell $(( $(podman info --format "{{ range .Host.IDMappings.UIDMap }}+{{.Size }}{{end }}" ) - 1 )))
subgidSize=$(shell $(( $(podman info --format "{{ range .Host.IDMappings.GIDMap }}+{{.Size }}{{end }}" ) - 1 )))
uid := $(shell id -u)
gid := $(shell id -g)
DOCKER_DIRS_MAP := \
-v ${HOST_GEM5}/runScripts:${DOCKER_GEM5}/runScripts
MAP_CMD := \
--user ${uid}:${gid} \
--uidmap ${uid}:0:1 \
--uidmap 0:1:${uid} \
--uidmap $(($uid+1)):$(($uid+1)):$(($subuidSize-$uid)) \
--gidmap ${gid}:0:1 \
--gidmap 0:1:${gid} \
--gidmap $(($gid+1)):$(($gid+1)):$(($subgidSize-$gid))
.PHONY: default clean run build
default: build
build:
podman run -it --rm ${DOCKER_DIRS_MAP} --security-opt seccomp=unconfined \
${MAP_CMD} \
gerrie/gem5:v1 "/bin/bash"
clean:
rm -rf ${HOST_GEM5}/build/*
Make always uses /bin/sh as the shell it invokes, both for recipes and for $(shell ...) functions. /bin/sh is a POSIX-conforming shell. The syntax you're using is not POSIX shell syntax: it's special enhanced syntax that is only available in the bash shell.
You can either rewrite your scripting to work in POSIX shells (probably by using the expr program to do the math), or add this to your makefile to tell it you want to use bash instead of /bin/sh:
SHELL := /bin/bash
Note, of course, that your makefile will now no longer work on any system that doesn't have a /bin/bash shell.

Using GNU parallel command with gfind to gain in runtime for gupdatedb tool

I make follow to the previous post combine parallel and gfind
I would like to build the gupdatedb database, containing all from main root / excepted the PRUNEPATHSlisted more below. I am working on MacOS 10.15 Catalina.
So, I tried to modify the gupdatedb script on MacOS 10.15 to benefit from parallel command like this (notice the # : A2 part) :
# : A2
cat | parallel -j32 $find {} $SEARCHPATHS $FINDOPTIONS \
\( $prunefs_exp -type d -regex "$PRUNEREGEX" \) \
-prune -o $print_option * :::
If I don't use cat |, I have the following warning message :
parallel: Warning: Input is read from the terminal. You are either an expert
parallel: Warning: (in which case: YOU ARE AWESOME!) or maybe you forgot
parallel: Warning: ::: or :::: or -a or to pipe data into parallel. If so
parallel: Warning: consider going through the tutorial: man parallel_tutorial
parallel: Warning: Press CTRL-D to exit.
and the process seems to hang.
Unfortunately, multiple threads of $find = gfind don't seem to run in the same time :
I have launched the script like this : sudo time gupdatedb
and below the result of : ps aux | grep find :
root 84865 0.0 0.0 4459044 15828 s002 S+ 1:43PM 0:00.10 perl /usr/local/bin/parallel -j32 /usr/local/Cellar/findutils/4.7.0/bin/gfind {} / ( -fstype 9P -o -fstype NFS -o -fstype afs -o -fstype autofs -o -fstype cifs -o -fstype coda -o -fstype devfs -o -fstype devpts -o -fstype ftpfs -o -fstype iso9660 -o -fstype mfs -o -fstype ncpfs -o -fstype nfs -o -fstype nfs4 -o -fstype proc -o -fstype shfs -o -fstype smbfs -o -fstype sysfs -o -type d -regex \(^/afs$\)\|\(^/amd$\)\|\(^/proc$\)\|\(^/sfs$\)\|\(^/tmp$\)\|\(^/usr/tmp$\)\|\(^/var/tmp$\)\|\(^/Volumes$\) ) -prune -o -print0 Applications Library System Users Volumes bin cores dev etc home opt private sbin tmp usr var :::
root 84863 0.0 0.0 4268280 796 s002 S+ 1:43PM 0:00.00 /usr/local/Cellar/findutils/4.7.0/libexec/gfrcode -0
root 84861 0.0 0.0 4282172 708 s002 S+ 1:43PM 0:00.00 /bin/sh /usr/local/Cellar/findutils/4.7.0/libexec/bin/gupdatedb
root 84853 0.0 0.0 4273980 1164 s002 S+ 1:43PM 0:00.01 /bin/sh /usr/local/Cellar/findutils/4.7.0/libexec/bin/gupdatedb
root 84850 0.0 0.0 5396228 10288 s008 S+ 1:43PM 0:00.27 vim /usr/local/Cellar/findutils/4.7.0/libexec/bin/gupdatedb
root 84849 0.0 0.0 4788896 6740 s008 S+ 1:43PM 0:00.03 sudo vim /usr/local/Cellar/findutils/4.7.0/libexec/bin/gupdatedb
Finally, the database may not be built, I am checking the size of : /usr/local/var/locate/locatedb.n and /usr/local/var/locate/locatedb but nothing is changing.
What's wrong in the syntax I used with parallel ? (especially, I don't know how to handle the ... ::: options part of command)
PS : I have set in gupdatedb :
# Directories to not put in the database, which would otherwise be.
: ${PRUNEPATHS="
/afs
/amd
/proc
/sfs
/tmp
/usr/tmp
/var/tmp
/Volumes
"}
and
# You can set these in the environment, or use command-line options,
# to override their defaults:
# Any global options for find?
: ${FINDOPTIONS=}
# What shell shoud we use? We should use a POSIX-ish sh.
: ${SHELL="/bin/sh"}
# Non-network directories to put in the database.
: ${SEARCHPATHS="/"}
Update 1
To be more accurate, here a post where I ask for a potential optimization (parallelization) with the couple parallel/find :
example of a potential parallelization with coupled parallel/find
I would like to do the same optimization but for script gupdatedb.
Update 2
I followed the advice of :
the defaut command into gupdatedb concerning my issue is :
$find $SEARCHPATHS $FINDOPTIONS \
\( $prunefs_exp \
-type d -regex "$PRUNEREGEX" \) -prune -o $print_option
So, I have just modified like this :
parallel -j32 $find {} $SEARCHPATHS $FINDOPTIONS \
\( $prunefs_exp \
-type d -regex "$PRUNEREGEX" \) -prune -o $print_option ::: /
and I get the following error :
/bin/sh: -c: line 0: syntax error near unexpected token `('
/bin/sh: -c: line 0: `/usr/local/Cellar/findutils/4.7.0/bin/gfind / / ( -fstype 9P -o -fstype NFS -o -fstype afs -o -fstype autofs -o -fstype cifs -o -fstype coda -o -fstype devfs -o -fstype devpts -o -fstype ftpfs -o -fstype iso9660 -o -fstype mfs -o -fstype ncpfs -o -fstype nfs -o -fstype nfs4 -o -fstype proc -o -fstype shfs -o -fstype smbfs -o -fstype sysfs -o -type d -regex \(^/private/tmp$\)\|\(^/private/var/folders$\)\|\(^/private/var/tmp$\)\|\(^*/Backups.backupdb$\)\|\(^/System$\)\|\(^/Volumes$\) ) -prune -o -print0'
What might be wrong here?
Update 3
here the script gupdatedb where you can see from line 300 my different tries :
#! /bin/sh
# updatedb -- build a locate pathname database
# Copyright (C) 1994-2019 Free Software Foundation, Inc.
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.
# csh original by James Woods; sh conversion by David MacKenzie.
#exec 2> /tmp/updatedb-trace.txt
#set -x
version='
updatedb (GNU findutils) 4.7.0
Copyright (C) 1994-2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by Eric B. Decker, James Youngman, and Kevin Dalley.
'
# File path names are not actually text, anyway (since there is no
# mechanism to enforce any constraint that the basename of a
# subdirectory has the same character encoding as the basename of its
# parent). The practical effect is that, depending on the way a
# particular system is configured and the content of its filesystem,
# passing all the file names in the system through "sort" may generate
# character encoding errors in text-based tools like "sort". To avoid
# this, we set LC_ALL=C. This will, presumably, not work perfectly on
# systems where LC_ALL is not the way to do locale configuration or
# some other seting can override this.
LC_ALL=C
export LC_ALL
# We can't use substitution on PACKAGE_URL below because it
# (correctly) points to https://www.gnu.org/software/findutils/ instead
# of the bug reporting page.
usage="\
Usage: $0 [--findoptions='-option1 -option2...']
[--localpaths='dir1 dir2...'] [--netpaths='dir1 dir2...']
[--prunepaths='dir1 dir2...'] [--prunefs='fs1 fs2...']
[--output=dbfile] [--netuser=user] [--localuser=user]
[--dbformat] [--version] [--help]
Please see also the documentation at http://www.gnu.org/software/findutils/.
Report (and track progress on fixing) bugs in the updatedb
program via the GNU findutils bug-reporting page at
https://savannah.gnu.org/bugs/?group=findutils or, if
you have no web access, by sending email to <bug-findutils#gnu.org>.
"
changeto=/
for arg
do
# If we are unable to fork, the back-tick operator will
# fail (and the shell will emit an error message). When
# this happens, we exit with error value 71 (EX_OSERR).
# Alternative candidate - 75, EX_TEMPFAIL.
opt=`echo $arg|sed 's/^\([^=]*\).*/\1/'` || exit 71
val=`echo $arg|sed 's/^[^=]*=\(.*\)/\1/'` || exit 71
case "$opt" in
--findoptions) FINDOPTIONS="$val" ;;
--localpaths) SEARCHPATHS="$val" ;;
--netpaths) NETPATHS="$val" ;;
--prunepaths) PRUNEPATHS="$val" ;;
--prunefs) PRUNEFS="$val" ;;
--output) LOCATE_DB="$val" ;;
--netuser) NETUSER="$val" ;;
--localuser) LOCALUSER="$val" ;;
--changecwd) changeto="$val" ;;
--dbformat) dbformat="$val" ;;
--version) fail=0; echo "$version" || fail=1; exit $fail ;;
--help) fail=0; echo "$usage" || fail=1; exit $fail ;;
*) echo "updatedb: invalid option $opt
Try '$0 --help' for more information." >&2
exit 1 ;;
esac
done
frcode_options=""
case "$dbformat" in
"")
# Default, use LOCATE02
;;
LOCATE02)
;;
slocate)
frcode_options="$frcode_options -S 1"
;;
*)
# The "old" database format is no longer supported.
echo "Unsupported locate database format ${dbformat}: Supported formats are:" >&2
echo "LOCATE02, slocate" >&2
exit 1
esac
if true
then
sort="/usr/bin/sort -z"
print_option="-print0"
frcode_options="$frcode_options -0"
else
sort="/usr/bin/sort"
print_option="-print"
fi
getuid() {
# format of "id" output is ...
# uid=1(daemon) gid=1(other)
# for `id's that don't understand -u
id | cut -d'(' -f 1 | cut -d'=' -f2
}
# figure out if su supports the -s option
select_shell() {
if su "$1" -s $SHELL -c false < /dev/null ; then
# No.
echo ""
else
if su "$1" -s $SHELL -c true < /dev/null ; then
# Yes.
echo "-s $SHELL"
else
# su is unconditionally failing. We won't be able to
# figure out what is wrong, so be conservative.
echo ""
fi
fi
}
# You can set these in the environment, or use command-line options,
# to override their defaults:
# Any global options for find?
: ${FINDOPTIONS="-mindepth 1 -maxdepth 1"}
#: ${FINDOPTIONS=""}
# What shell shoud we use? We should use a POSIX-ish sh.
: ${SHELL="/bin/sh"}
# Non-network directories to put in the database.
: ${SEARCHPATHS="/"}
# Network (NFS, AFS, RFS, etc.) directories to put in the database.
: ${NETPATHS=}
# Directories to not put in the database, which would otherwise be.
: ${PRUNEPATHS="
/afs
/amd
/proc
/sfs
/tmp
/usr/tmp
/var/tmp
"}
# Trailing slashes result in regex items that are never matched, which
# is not what the user will expect. Therefore we now reject such
# constructs.
for p in $PRUNEPATHS; do
case "$p" in
/*/) echo "$0: $p: pruned paths should not contain trailing slashes" >&2
exit 1
esac
done
# The same, in the form of a regex that find can use.
test -z "$PRUNEREGEX" &&
PRUNEREGEX=`echo $PRUNEPATHS|sed -e 's,^,\\\(^,' -e 's, ,$\\\)\\\|\\\(^,g' -e 's,$,$\\\),'`
# The database file to build.
: ${LOCATE_DB=/usr/local/var/locate/locatedb}
# Directory to hold intermediate files.
if test -z "$TMPDIR"; then
if test -d /var/tmp; then
: ${TMPDIR=/var/tmp}
elif test -d /usr/tmp; then
: ${TMPDIR=/usr/tmp}
else
: ${TMPDIR=/tmp}
fi
fi
export TMPDIR
# The user to search network directories as.
: ${NETUSER=daemon}
# The directory containing the subprograms.
if test -n "$LIBEXECDIR" ; then
: LIBEXECDIR already set, do nothing
else
: ${LIBEXECDIR=/usr/local/Cellar/findutils/4.7.0/libexec}
fi
# The directory containing find.
if test -n "$BINDIR" ; then
: BINDIR already set, do nothing
else
: ${BINDIR=/usr/local/Cellar/findutils/4.7.0/bin}
fi
# The names of the utilities to run to build the database.
: ${find:=${BINDIR}/gfind}
: ${frcode:=${LIBEXECDIR}/gfrcode}
make_tempdir () {
# This implementation is adapted from the GNU Autoconf manual.
{
tmp=`
(umask 077 && mktemp -d "$TMPDIR/updatedbXXXXXX") 2>/dev/null
` &&
test -n "$tmp" && test -d "$tmp"
} || {
# This method is less secure than mktemp -d, but it's a fallback.
#
# We use $$ as well as $RANDOM since $RANDOM may not be available.
# We also add a time-dependent suffix. This is actually somewhat
# predictable, but then so is $$. POSIX does not require date to
# support +%N.
ts=`date +%N%S || date +%S 2>/dev/null`
tmp="$TMPDIR"/updatedb"$$"-"${RANDOM:-}${ts}"
(umask 077 && mkdir "$tmp")
}
echo "$tmp"
}
checkbinary () {
if test -x "$1" ; then
: ok
else
eval echo "updatedb needs to be able to execute $1, but cannot." >&2
exit 1
fi
}
for binary in $find $frcode
do
checkbinary $binary
done
: ${PRUNEFS="
9P
NFS
afs
autofs
cifs
coda
devfs
devpts
ftpfs
iso9660
mfs
ncpfs
nfs
nfs4
proc
shfs
smbfs
sysfs
"}
if test -n "$PRUNEFS"; then
prunefs_exp=`echo $PRUNEFS |sed -e 's/\([^ ][^ ]*\)/-o -fstype \1/g' \
-e 's/-o //' -e 's/$/ -o/'`
else
prunefs_exp=''
fi
# Make and code the file list.
# Sort case insensitively for users' convenience.
rm -f $LOCATE_DB.n
trap 'rm -f $LOCATE_DB.n; exit' HUP TERM
if {
cd "$changeto"
if test -n "$SEARCHPATHS"; then
if [ "$LOCALUSER" != "" ]; then
# : A1
su $LOCALUSER `select_shell $LOCALUSER` -c \
"$find $SEARCHPATHS $FINDOPTIONS \
\\( $prunefs_exp \
-type d -regex '$PRUNEREGEX' \\) -prune -o $print_option"
else
# : A2
# ORIGINAL VERSION : sequential find
#$find $SEARCHPATHS $FINDOPTIONS \
# \( $prunefs_exp \
# -type d -regex "$PRUNEREGEX" \) -prune -o $print_option ::: /
# Parallel version 1
#parallel -j 32 $find $SEARCHPATHS $FINDOPTIONS \
# \( $prunefs_exp \
# -type d -regex "$PRUNEREGEX" \) -prune -o $print_option ::: /
# Parallel version 2
parallel -j 32 $find {} $FINDOPTIONS \
$prunefs_exp -type d -regex $PRUNEREGEX -prune -o $print_option ::: */*
fi
fi
if test -n "$NETPATHS"; then
myuid=`getuid`
if [ "$myuid" = 0 ]; then
# : A3
su $NETUSER `select_shell $NETUSER` -c \
"$find $NETPATHS $FINDOPTIONS \\( -type d -regex '$PRUNEREGEX' -prune \\) -o $print_option" ||
exit $?
else
# : A4
$find $NETPATHS $FINDOPTIONS \( -type d -regex "$PRUNEREGEX" -prune \) -o $print_option ||
exit $?
fi
fi
} | $sort | $frcode $frcode_options > $LOCATE_DB.n
then
: OK so far
true
else
rv=$?
echo "Failed to generate $LOCATE_DB.n" >&2
rm -f $LOCATE_DB.n
exit $rv
fi
# To avoid breaking locate while this script is running, put the
# results in a temp file, then rename it atomically.
if test -s $LOCATE_DB.n; then
chmod 644 ${LOCATE_DB}.n
mv ${LOCATE_DB}.n $LOCATE_DB
else
echo "updatedb: new database would be empty" >&2
rm -f $LOCATE_DB.n
fi
exit 0
I launch the gupdatedb command like this :
sudo gupdatedb --prunepaths='/private/tmp /private/var/folders /private/var/tmp */Backups.backupdb /System /Volumes' --localpaths='/' --output=$HOME/locatedb_gupdatedb_PARALLEL
Update 4
My bounty expires tomorrow. Using default gupdatedb, all the indexing takes about 30 minutes. If I could manage to use correctly parallel with the core of gupdatedb script, i.e when this latter indexes with gfind command, which gain factor can I expect ?
and last request : how to fix the error :
/bin/sh: -c: line 0: syntax error near unexpected token `('
/bin/sh: -c: line 0: `/usr/local/Cellar/findutils/4.7.0/bin/gfind / / ( -fstype 9P -o -fstype NFS -o -fstype afs -o -fstype autofs -o -fstype cifs -o -fstype coda -o -fstype devfs -o -fstype devpts -o -fstype ftpfs -o -fstype iso9660 -o -fstype mfs -o -fstype ncpfs -o -fstype nfs -o -fstype nfs4 -o -fstype proc -o -fstype shfs -o -fstype smbfs -o -fstype sysfs -o -type d -regex \(^/private/tmp$\)\|\(^/private/var/folders$\)\|\(^/private/var/tmp$\)\|\(^*/Backups.backupdb$\)\|\(^/System$\)\|\(^/Volumes$\) ) -prune -o -print0'
with the command :
parallel -j32 $find {} $FINDOPTIONS \
\( $prunefs_exp \
-type d -regex "$PRUNEREGEX" \) -prune -o $print_option ::: /
?
You don't need ::: if there's nothing after it, and {} is pointless too if you don't have any sources. Without more information about what exactly you would want to parallelize, we can't really tell you what you should use instead.
But for example, if you want to run one find in each of /etc, /usr, /bin, and /opt, that would look like
parallel find {} -options ::: /etc /usr /bin /opt
This could equivalently be expressed without ::::
printf '%s\n' /etc /usr /bin /opt |
parallel find {} -options
So the purpose of ::: is basically to say "I want to specify the things to parallelize over on the command line instead of receiving them on standard input"; but if you don't provide this information, either way, parallel doesn't know what to replace {} with.
I'm not saying this particular use makes sense for your use case, just hopefully clarifying the documentation (again).
To get any meaningful speedup from using parallel, you need to make sure that you have resources to make the process faster. There are two challenges here:
The updatedb process is IO bound. Usually, you use parallel to take advantages of multi-core system, and spread CPU bound process over multiple cores.
The updatedb process require exclusive access to the database (usually in /var/lib/mlcoate/mlocate.db). Even if you get any benefits from splitting the updatedb over multiple cores, you will have to place the output into multiple databases. This approach will require passing all database names (separate with ':' to locate with '-d')
Unless you system has multiple disk drives (or you are accessing network drives), you will gain very little from running parallel find.
If you system has multiple disk drives (and/or network drives) you can run each file system in parallel, using a script like
Assuming you have 2 additional disks mounted on /mnt/disk1, /mnt/disk2
# Index root
updatedb --output=/var/lib/mlocate/local.db -E '/mnt/disk1 /mnt/disk2' &
# Index 1st extra disk (or network drive)
updatedb --output=/var/lib/mlocate/disk1.db -U /mnt/disk1 &
# Index 2nd extra disk (or network drive)
updatedb --output=/var/lib/mlocate/disk2.db -U /mnt/disk2 &
wait
You should set the environment variable LOCATE_PATH to point to all the databases
export
LOCATE_PATH=/var/lib/mlocate/local.db:/var/lib/mlocate/disk1.db:/var/lib/mlocate/disk2.db
locate ...

Why isn't my Arch Linux fully automated installer working?

I've been working on a simple fully-automated installer for ArchLinux recently.
I started with a basic three-partition system; that made a bootable system.
I added LVM and several more logical partitions; that worked also.
I'm now trying to add LUKS encryption; that isn't working.
I want to have LUKS inside of LVM for more flexibility. For a BIOS system with a single disk, that should look something like this:
Raw Partitions
+-----------+------+-------+-----------+
| Partition | Name | Size | Flags |
+-----------+------+-------+-----------+
| /dev/sda1 | grub | 2MB | bios_grub |
| /dev/sda2 | boot | 200MB | boot |
| /dev/sda3 | lvm | | lvm |
+-----------+------+-------+-----------+
LVM Partitions
+-------------+--------------+-------+------+
| LVM Device | LUKS Device | Name | Size |
+-------------+--------------+-------+------+
| LvmDvc-root | LuksDvc-root | root | 2GB |
| LvmDvc-home | LuksDvc-home | home | 2GB |
| LvmDvc-var | LuksDvc-var | var | 1G |
| LvmDvc-usr | LuksDvc-usr | usr | 1G |
| LvmDvc-swap | LuksDvc-swap | swap | 4G |
+-------------+--------------+-------+------+
LvmDvc-root is decrypted to LuksDvc-root using a passphrase.
All other LVM devices are decrypted using keys stored in /etc/ctyptkeys.
Partitions are mounted as:
/dev/mapper/LuksDvc-root -> /
/dev/sda2 -> /boot
/dev/mapper/LuksDvc-home -> /home
/dev/mapper/LuksDvc-var -> /var
/dev/mapper/LuksDvc-usr -> /usr
From what I can tell, disk partitioning and system install work just fine. I receive a slew of errors about lvmetad not being loaded during grub configuration, but the documentation in Arch's wiki indicates this is a non-issue (https://wiki.archlinux.org/index.php/GRUB#UEFI_systems_2). I also receive these same errors in the previous version of my script (using LVM, but not LUKS), and it produces a bootable system. So I don't think this error message indicates a problem.
When I boot the system, I get through GRUB just fine. I am presented with a dialogue to decrypt the root partition:
A password is required to access the LuksDvc-root volume:
Enter passphrase for /dev/mapper/LvmDvc-root:
I enter the passphrase used during installation, and receive this message:
No key available with this passphrase.
I'm using a very simple passphrase for testing (asdfasdf), so I doubt I'm messing it up. I can decrypt and mount the whole system from the live installer without incident, I just can't make it happen at boot.
I'm not sure what information would be most helpful for solving this. Here is the script I use to install the system:
#!/usr/bin/env bash
set -ex -o pipefail -o nounset
# Raw Partitioning
parted --script --align optimal -- /dev/sda mklabel gpt
parted --script --align optimal -- /dev/sda mkpart primary 2 4
parted --script --align optimal -- /dev/sda name 1 bios_grub
parted --script --align optimal -- /dev/sda set 1 bios_grub on
parted --script --align optimal -- /dev/sda mkpart primary 4 204
parted --script --align optimal -- /dev/sda name 2 boot
parted --script --align optimal -- /dev/sda set 2 boot on
parted --script --align optimal -- /dev/sda mkpart primary 204 -1
parted --script --align optimal -- /dev/sda name 3 lvm
parted --script --align optimal -- /dev/sda set 3 lvm on
# LVM Partitioning
pvcreate -ff --yes /dev/sda3
vgcreate LvmDvc /dev/sda3
lvcreate --zero y --wipesignatures y --name root --size 2G LvmDvc
lvcreate --zero y --wipesignatures y --name home --size 2G LvmDvc
lvcreate --zero y --wipesignatures y --name var --size 1G LvmDvc
lvcreate --zero y --wipesignatures y --name usr --size 1G LvmDvc
lvcreate --zero y --wipesignatures y --name swap --size 4G LvmDvc
# Root Partition
echo asdfasdf | cryptsetup -q --key-file - luksFormat /dev/mapper/LvmDvc-root
echo asdfasdf | cryptsetup -q --key-file - luksOpen /dev/mapper/LvmDvc-root LuksDvc-root
mkfs.ext4 -q /dev/mapper/LuksDvc-root
mkdir -p /mnt/archbox
mount /dev/mapper/LuksDvc-root /mnt/archbox
# Boot Partition
mkfs.ext4 -q /dev/sda2
# Encrypted Partitions
mkdir -p /mnt/archbox/etc/cryptkeys
chmod 400 /mnt/archbox/etc/cryptkeys
dd if=/dev/random of=/mnt/archbox/etc/cryptkeys/home bs=512 count=4 iflag=fullblock
chmod 400 /mnt/archbox/etc/cryptkeys/home
cryptsetup -q --key-file /mnt/archbox/etc/cryptkeys/home luksFormat /dev/mapper/LvmDvc-home
cryptsetup -q --key-file /mnt/archbox/etc/cryptkeys/home luksOpen /dev/mapper/LvmDvc-home LuksDvc-home
mkfs.ext4 -q /dev/mapper/LuksDvc-home
dd if=/dev/random of=/mnt/archbox/etc/cryptkeys/var bs=512 count=4 iflag=fullblock
chmod 400 /mnt/archbox/etc/cryptkeys/var
cryptsetup -q --key-file /mnt/archbox/etc/cryptkeys/var luksFormat /dev/mapper/LvmDvc-var
cryptsetup -q --key-file /mnt/archbox/etc/cryptkeys/var luksOpen /dev/mapper/LvmDvc-var LuksDvc-var
mkfs.ext4 -q /dev/mapper/LuksDvc-var
dd if=/dev/random of=/mnt/archbox/etc/cryptkeys/usr bs=512 count=4 iflag=fullblock
chmod 400 /mnt/archbox/etc/cryptkeys/usr
cryptsetup -q --key-file /mnt/archbox/etc/cryptkeys/usr luksFormat /dev/mapper/LvmDvc-usr
cryptsetup -q --key-file /mnt/archbox/etc/cryptkeys/usr luksOpen /dev/mapper/LvmDvc-usr LuksDvc-usr
mkfs.ext4 -q /dev/mapper/LuksDvc-usr
dd if=/dev/random of=/mnt/archbox/etc/cryptkeys/swap bs=512 count=4 iflag=fullblock
chmod 400 /mnt/archbox/etc/cryptkeys/swap
cryptsetup -q --key-file /mnt/archbox/etc/cryptkeys/swap luksFormat /dev/mapper/LvmDvc-swap
cryptsetup -q --key-file /mnt/archbox/etc/cryptkeys/swap luksOpen /dev/mapper/LvmDvc-swap LuksDvc-swap
mkswap /dev/mapper/LuksDvc-swap
# Mount
mkdir -p /mnt/archbox/boot
mount /dev/sda2 /mnt/archbox/boot
mkdir -p /mnt/archbox/home
mount /dev/mapper/LuksDvc-home /mnt/archbox/home
mkdir -p /mnt/archbox/var
mount /dev/mapper/LuksDvc-var /mnt/archbox/var
mkdir -p /mnt/archbox/usr
mount /dev/mapper/LuksDvc-usr /mnt/archbox/usr
swapon /dev/mapper/LuksDvc-swap
# Packages
mkdir -p ./cache-dir
rm -f /mnt/archbox/var/lib/pacman/db.lck
pacstrap /mnt/archbox --cachedir ./cache-dir base grub
# Root password
echo "root:asdfasdf" | chpasswd --root /mnt/archbox
# FSTab
genfstab -U -p /mnt/archbox >> /mnt/archbox/etc/fstab
# CryptTab
echo "" > /mnt/archbox/etc/crypttab
echo "home /dev/mapper/LvmDvc-home /mnt/archbox/etc/cryptkeys/home" >> /mnt/archbox/etc/crypttab
echo "usr /dev/mapper/LvmDvc-usr /mnt/archbox/etc/cryptkeys/usr" >> /mnt/archbox/etc/crypttab
echo "var /dev/mapper/LvmDvc-var /mnt/archbox/etc/cryptkeys/var" >> /mnt/archbox/etc/crypttab
echo "swap /dev/mapper/LvmDvc-swap /mnt/archbox/etc/cryptkeys/swap" >> /mnt/archbox/etc/crypttab
# Ramdisk
file=/mnt/archbox/etc/mkinitcpio.conf
search="^\s*MODULES=.*$"
replace="MODULES=\\\"virtio virtio_blk virtio_pci virtio_net\\\""
grep -q "$search" "$file" && sed -i "s#$search#$replace#" "$file" || echo "$replace" >> "$file"
search="^\s*HOOKS=.*$"
replace="HOOKS=\\\"base udev autodetect modconf block keymap encrypt lvm2 filesystems keyboard shutdown fsck usr\\\""
grep -q "$search" "$file" && sed -i "s#$search#$replace#" "$file" || echo "$replace" >> "$file"
arch-chroot /mnt/archbox mkinitcpio -p linux
# Bootloader
arch-chroot /mnt/archbox grub-install --target=i386-pc --recheck /dev/sda
file=/mnt/archbox/etc/default/grub
search="^\s*GRUB_CMDLINE_LINUX=.*$"
replace="GRUB_CMDLINE_LINUX=\\\"init=/usr/lib/systemd/systemd cryptdevice=/dev/mapper/LvmDvc-root:LuksDvc-root root=/dev/mapper/LuksDvc-root quiet\\\""
grep -q "$search" "$file" && sed -i "s#$search#$replace#" "$file" || echo "$replace" >> "$file"
search="^\s*GRUB_DISABLE_LINUX_UUID=.*$"
replace="GRUB_DISABLE_LINUX_UUID=true"
grep -q "$search" "$file" && sed -i "s#$search#$replace#" "$file" || echo "$replace" >> "$file"
arch-chroot /mnt/archbox grub-mkconfig -o /boot/grub/grub.cfg
Does anything stick out as blatantly wrong? What should I be doing differently? Can I provide additional/specific information?
tl;dr - Install script seems to work, but I can't decrypt the system at boot. Halp!
I found the main problem: formatting the root LUKS volume should be done with a password, not a key file. Taking out the option --key-file - on the formatting and opening for the root volume fixes the problem I was seeing.
There are some other issues, in the script, but I'll come back after I've tried and failed to fix them.
appreciate any updates of your script; I'm currently testing it. Note, in the above script, the isolated '-' is a syntax error:
echo asdfasdf | cryptsetup -q --key-file - luksFormat /dev/mapper/LvmDvc-root
echo asdfasdf | cryptsetup -q --key-file - luksOpen /dev/mapper/LvmDvc-root LuksDvc-root
after taking out '--key-file' as you suggested, the two lines should be:
echo asdfasdf | cryptsetup -q luksFormat /dev/mapper/LvmDvc-root
echo asdfasdf | cryptsetup -q luksOpen /dev/mapper/LvmDvc-root LuksDvc-root
after this changing the script did run through with the warnings you mentioned. when booting the disk generated by the script, root volume password and then
dev/mapper/Lucks-Dvc-root: clean, ...
ERROR: device '' not found. Skipping fsck.
mount: wrong fs type, bad option, bad superblock ,
missing code page or helper program, ... try dmesg | tail
ERROR: root device mounted success..., but /usr/lib/systemd/systemd does not exist.
Bailing out...
sh: cant access tty: job control turned off
[rootfs /]#
when searching for this error, found:
"Looks like that script doesn't install systemd-sysvcompat, which provides the /sbin/init symlink to /usr/lib/systemd/systemd. You might want to file a bug report and/or use pacstrap in the future."
-- https://bbs.archlinux.org/viewtopic.php?id=146712
(post by WorMzy 2013-02-06 15:11:34)
What about using LVM on top of LUKS instead of the other way around used in your script?
I used the following instructions on a number of machines and it worked just fine: http://is.gd/OoDx1d

Makefile Target Dependency on Whether Target Already Exists

I am trying to write a Makefile whose targets depend on the existence of a disk file. The disk file itself merely needs to be created; it does not depend on any other actions that may update it. If I do not give it any dependencies, the file is re-created every time I run make on one of the targets.
Is there a way to have a target depend on whether it exists?
This is part of the Makefile I have. The $(TMPDEV) file only needs to be created if it doesn't exist, otherwise it should be considered up-to-date.
TMPDEV="/tmp/disk.img"
$(TMPDEV):
fallocate -l 806354944 $(TMPDEV) || dd if=/dev/zero of=$(TMPDEV) bs=1b count=1574912
sudo parted --script $(TMPDEV) unit s mklabel msdos \
mkpart primary fat16 2048 526335 \
mkpart primary fat32 526336 1050623 \
mkpart primary NTFS 1050624 1574911 \
quit
$(eval TMPDISK := $(shell sudo partx --verbose -a $(TMPDEV) | tail -1 | cut -d':' -f1))
sudo mkfs.fat -F 16 -n FAT16 $(TMPDISK)p1
sudo mkfs.fat -F 32 -n FAT32 $(TMPDISK)p2
sudo mkfs.ntfs -L NTFS $(TMPDISK)p3
sudo partx -d $(TMPDISK)
sudo losetup -d $(TMPDISK)
testresults: $(TMPDEV)
touch testresults
analytics: $(TMPDEV)
touch analytics
Remove the quotes:
TMPDEV="/tmp/disk.img"
Make doesn't use/need quotes. You are saying that the target here:
$(TMPDEV):
is, literally, this file including the quotes:
"/tmp/disk.img":
that file never exists, so the rule is always re-run.

EC2 - Create AMI - Cannot connect to new instance

I am experiencing difficulty trying to launch a AMI from an EBS volume. I am basically trying to launch another instance of a Linux (i386) based AMI that I have already configured the way I want. I have followed many guides for the past week. So far, I am able to create the custom private AMI but I am unable to connect to it after launching the new instance. I suspect that the AMI I have created is miss-configured in some way (maybe files didnt get fully copied over).
Anyhow here are the basic steps I'm going through to try to create the AMI:
ec2-create-volume -K pk-xxxxxx.pem -C
cert-xxxxxx.pem --size 10
--availability-zone us-east-1a
ec2-attach-volume -K pk-xxxxxx.pem -C
cert-xxxxxx.pem vol-xxxxxx --instance
xxxxxx --device /dev/sdh
yes | mkfs -t ext3 /dev/sdh
mkdir/mnt/ebsimage
echo '/dev/sdh /mnt/ebsimage ext3
defaults,noatime 0 0' >> /etc/fstab
mount /mnt/ebsimage
umount /mnt/ebsimage
ec2-detach-volume -K pk-xxxxxx.pem -C
cert-xxxxxx.pem vol-xxxxxx --instance
xxxxxx
ec2-create-snapshot -K pk-xxxxxx.pem
-C cert-xxxxxx.pem vol-xxxxxx
ec2reg -K pk-xxxxxx.pem -C
cert-xxxxxx.pem -s snap-xxxxx -a i386
-d -n --kernel aki-xxxxx --ramdisk ari-xxxxxx
I'm pretty sure either my commands around mount are messed up or my commands around ec2reg are messed up. Any suggestions?
I have also tried replacing
yes | mkfs -t ext3 /dev/sdh
mkdir/mnt/ebsimage
echo '/dev/sdh
/mnt/ebsimage ext3 defaults,noatime 0
0' >> /etc/fstab
mount /mnt/ebsimage
with a script designed to use rsync and add some other details but again the new instance of the ami launched cannot be connected to. Here is a copy of the script.
#!/bin/sh
vol=/dev/sdh
ebsmnt=/mnt/ebsimage
mkdir ${ebsmnt}
mkfs.ext3 -F ${vol}
sync
echo "mount $vol $ebsmnt"
mount $vol $ebsmnt
mkdir ${ebsmnt}/mnt
mkdir ${ebsmnt}/proc
mkdir ${ebsmnt}/sys
devdir=${ebsmnt}/dev
echo "mkdir ${devdir}"
mkdir ${devdir}
mknod ${devdir}/null c 1 3
mknod ${devdir}/zero c 1 5
mknod ${devdir}/tty c 5 0
mknod ${devdir}/console c 5 1
ln -s null ${devdir}/X0R
rsync -rlpgoD -t -r -S -l -vh \
--exclude /sys --exclude /proc \
--exclude /dev \
--exclude /media --exclude /mnt \
--exclude /sys --exclude /ebs --exclude /mnt \
-x /* ${ebsmnt}
df -h
Because I have the same results as the first example, I'm not sure if I'm closer to solving this issue or further away. Any help would be appreciated.
To create your EBS AMI from an S3 based AMI, you can use my blog post:
http://www.capsunlock.net/2009/12/create-ebs-boot-ami.html
I don't know which distribution you are trying to run, but if you want to run debian, there is a script which manages the entire bootstrapping process including ami creation (EBS boot).
You can find it on my github account:
https://github.com/andsens/ec2debian-build-ami
The script has been thoroughly tested and allows you to include other scripts in order to customize your ami. If you want to modify the script itself, just fork it, at least you then have a base to work from, where you know everything works.
I would not recommend the process you outlined though, it seems quite 'messy'.

Resources