I've been working on a simple fully-automated installer for ArchLinux recently.
I started with a basic three-partition system; that made a bootable system.
I added LVM and several more logical partitions; that worked also.
I'm now trying to add LUKS encryption; that isn't working.
I want to have LUKS inside of LVM for more flexibility. For a BIOS system with a single disk, that should look something like this:
Raw Partitions
+-----------+------+-------+-----------+
| Partition | Name | Size | Flags |
+-----------+------+-------+-----------+
| /dev/sda1 | grub | 2MB | bios_grub |
| /dev/sda2 | boot | 200MB | boot |
| /dev/sda3 | lvm | | lvm |
+-----------+------+-------+-----------+
LVM Partitions
+-------------+--------------+-------+------+
| LVM Device | LUKS Device | Name | Size |
+-------------+--------------+-------+------+
| LvmDvc-root | LuksDvc-root | root | 2GB |
| LvmDvc-home | LuksDvc-home | home | 2GB |
| LvmDvc-var | LuksDvc-var | var | 1G |
| LvmDvc-usr | LuksDvc-usr | usr | 1G |
| LvmDvc-swap | LuksDvc-swap | swap | 4G |
+-------------+--------------+-------+------+
LvmDvc-root is decrypted to LuksDvc-root using a passphrase.
All other LVM devices are decrypted using keys stored in /etc/ctyptkeys.
Partitions are mounted as:
/dev/mapper/LuksDvc-root -> /
/dev/sda2 -> /boot
/dev/mapper/LuksDvc-home -> /home
/dev/mapper/LuksDvc-var -> /var
/dev/mapper/LuksDvc-usr -> /usr
From what I can tell, disk partitioning and system install work just fine. I receive a slew of errors about lvmetad not being loaded during grub configuration, but the documentation in Arch's wiki indicates this is a non-issue (https://wiki.archlinux.org/index.php/GRUB#UEFI_systems_2). I also receive these same errors in the previous version of my script (using LVM, but not LUKS), and it produces a bootable system. So I don't think this error message indicates a problem.
When I boot the system, I get through GRUB just fine. I am presented with a dialogue to decrypt the root partition:
A password is required to access the LuksDvc-root volume:
Enter passphrase for /dev/mapper/LvmDvc-root:
I enter the passphrase used during installation, and receive this message:
No key available with this passphrase.
I'm using a very simple passphrase for testing (asdfasdf), so I doubt I'm messing it up. I can decrypt and mount the whole system from the live installer without incident, I just can't make it happen at boot.
I'm not sure what information would be most helpful for solving this. Here is the script I use to install the system:
#!/usr/bin/env bash
set -ex -o pipefail -o nounset
# Raw Partitioning
parted --script --align optimal -- /dev/sda mklabel gpt
parted --script --align optimal -- /dev/sda mkpart primary 2 4
parted --script --align optimal -- /dev/sda name 1 bios_grub
parted --script --align optimal -- /dev/sda set 1 bios_grub on
parted --script --align optimal -- /dev/sda mkpart primary 4 204
parted --script --align optimal -- /dev/sda name 2 boot
parted --script --align optimal -- /dev/sda set 2 boot on
parted --script --align optimal -- /dev/sda mkpart primary 204 -1
parted --script --align optimal -- /dev/sda name 3 lvm
parted --script --align optimal -- /dev/sda set 3 lvm on
# LVM Partitioning
pvcreate -ff --yes /dev/sda3
vgcreate LvmDvc /dev/sda3
lvcreate --zero y --wipesignatures y --name root --size 2G LvmDvc
lvcreate --zero y --wipesignatures y --name home --size 2G LvmDvc
lvcreate --zero y --wipesignatures y --name var --size 1G LvmDvc
lvcreate --zero y --wipesignatures y --name usr --size 1G LvmDvc
lvcreate --zero y --wipesignatures y --name swap --size 4G LvmDvc
# Root Partition
echo asdfasdf | cryptsetup -q --key-file - luksFormat /dev/mapper/LvmDvc-root
echo asdfasdf | cryptsetup -q --key-file - luksOpen /dev/mapper/LvmDvc-root LuksDvc-root
mkfs.ext4 -q /dev/mapper/LuksDvc-root
mkdir -p /mnt/archbox
mount /dev/mapper/LuksDvc-root /mnt/archbox
# Boot Partition
mkfs.ext4 -q /dev/sda2
# Encrypted Partitions
mkdir -p /mnt/archbox/etc/cryptkeys
chmod 400 /mnt/archbox/etc/cryptkeys
dd if=/dev/random of=/mnt/archbox/etc/cryptkeys/home bs=512 count=4 iflag=fullblock
chmod 400 /mnt/archbox/etc/cryptkeys/home
cryptsetup -q --key-file /mnt/archbox/etc/cryptkeys/home luksFormat /dev/mapper/LvmDvc-home
cryptsetup -q --key-file /mnt/archbox/etc/cryptkeys/home luksOpen /dev/mapper/LvmDvc-home LuksDvc-home
mkfs.ext4 -q /dev/mapper/LuksDvc-home
dd if=/dev/random of=/mnt/archbox/etc/cryptkeys/var bs=512 count=4 iflag=fullblock
chmod 400 /mnt/archbox/etc/cryptkeys/var
cryptsetup -q --key-file /mnt/archbox/etc/cryptkeys/var luksFormat /dev/mapper/LvmDvc-var
cryptsetup -q --key-file /mnt/archbox/etc/cryptkeys/var luksOpen /dev/mapper/LvmDvc-var LuksDvc-var
mkfs.ext4 -q /dev/mapper/LuksDvc-var
dd if=/dev/random of=/mnt/archbox/etc/cryptkeys/usr bs=512 count=4 iflag=fullblock
chmod 400 /mnt/archbox/etc/cryptkeys/usr
cryptsetup -q --key-file /mnt/archbox/etc/cryptkeys/usr luksFormat /dev/mapper/LvmDvc-usr
cryptsetup -q --key-file /mnt/archbox/etc/cryptkeys/usr luksOpen /dev/mapper/LvmDvc-usr LuksDvc-usr
mkfs.ext4 -q /dev/mapper/LuksDvc-usr
dd if=/dev/random of=/mnt/archbox/etc/cryptkeys/swap bs=512 count=4 iflag=fullblock
chmod 400 /mnt/archbox/etc/cryptkeys/swap
cryptsetup -q --key-file /mnt/archbox/etc/cryptkeys/swap luksFormat /dev/mapper/LvmDvc-swap
cryptsetup -q --key-file /mnt/archbox/etc/cryptkeys/swap luksOpen /dev/mapper/LvmDvc-swap LuksDvc-swap
mkswap /dev/mapper/LuksDvc-swap
# Mount
mkdir -p /mnt/archbox/boot
mount /dev/sda2 /mnt/archbox/boot
mkdir -p /mnt/archbox/home
mount /dev/mapper/LuksDvc-home /mnt/archbox/home
mkdir -p /mnt/archbox/var
mount /dev/mapper/LuksDvc-var /mnt/archbox/var
mkdir -p /mnt/archbox/usr
mount /dev/mapper/LuksDvc-usr /mnt/archbox/usr
swapon /dev/mapper/LuksDvc-swap
# Packages
mkdir -p ./cache-dir
rm -f /mnt/archbox/var/lib/pacman/db.lck
pacstrap /mnt/archbox --cachedir ./cache-dir base grub
# Root password
echo "root:asdfasdf" | chpasswd --root /mnt/archbox
# FSTab
genfstab -U -p /mnt/archbox >> /mnt/archbox/etc/fstab
# CryptTab
echo "" > /mnt/archbox/etc/crypttab
echo "home /dev/mapper/LvmDvc-home /mnt/archbox/etc/cryptkeys/home" >> /mnt/archbox/etc/crypttab
echo "usr /dev/mapper/LvmDvc-usr /mnt/archbox/etc/cryptkeys/usr" >> /mnt/archbox/etc/crypttab
echo "var /dev/mapper/LvmDvc-var /mnt/archbox/etc/cryptkeys/var" >> /mnt/archbox/etc/crypttab
echo "swap /dev/mapper/LvmDvc-swap /mnt/archbox/etc/cryptkeys/swap" >> /mnt/archbox/etc/crypttab
# Ramdisk
file=/mnt/archbox/etc/mkinitcpio.conf
search="^\s*MODULES=.*$"
replace="MODULES=\\\"virtio virtio_blk virtio_pci virtio_net\\\""
grep -q "$search" "$file" && sed -i "s#$search#$replace#" "$file" || echo "$replace" >> "$file"
search="^\s*HOOKS=.*$"
replace="HOOKS=\\\"base udev autodetect modconf block keymap encrypt lvm2 filesystems keyboard shutdown fsck usr\\\""
grep -q "$search" "$file" && sed -i "s#$search#$replace#" "$file" || echo "$replace" >> "$file"
arch-chroot /mnt/archbox mkinitcpio -p linux
# Bootloader
arch-chroot /mnt/archbox grub-install --target=i386-pc --recheck /dev/sda
file=/mnt/archbox/etc/default/grub
search="^\s*GRUB_CMDLINE_LINUX=.*$"
replace="GRUB_CMDLINE_LINUX=\\\"init=/usr/lib/systemd/systemd cryptdevice=/dev/mapper/LvmDvc-root:LuksDvc-root root=/dev/mapper/LuksDvc-root quiet\\\""
grep -q "$search" "$file" && sed -i "s#$search#$replace#" "$file" || echo "$replace" >> "$file"
search="^\s*GRUB_DISABLE_LINUX_UUID=.*$"
replace="GRUB_DISABLE_LINUX_UUID=true"
grep -q "$search" "$file" && sed -i "s#$search#$replace#" "$file" || echo "$replace" >> "$file"
arch-chroot /mnt/archbox grub-mkconfig -o /boot/grub/grub.cfg
Does anything stick out as blatantly wrong? What should I be doing differently? Can I provide additional/specific information?
tl;dr - Install script seems to work, but I can't decrypt the system at boot. Halp!
I found the main problem: formatting the root LUKS volume should be done with a password, not a key file. Taking out the option --key-file - on the formatting and opening for the root volume fixes the problem I was seeing.
There are some other issues, in the script, but I'll come back after I've tried and failed to fix them.
appreciate any updates of your script; I'm currently testing it. Note, in the above script, the isolated '-' is a syntax error:
echo asdfasdf | cryptsetup -q --key-file - luksFormat /dev/mapper/LvmDvc-root
echo asdfasdf | cryptsetup -q --key-file - luksOpen /dev/mapper/LvmDvc-root LuksDvc-root
after taking out '--key-file' as you suggested, the two lines should be:
echo asdfasdf | cryptsetup -q luksFormat /dev/mapper/LvmDvc-root
echo asdfasdf | cryptsetup -q luksOpen /dev/mapper/LvmDvc-root LuksDvc-root
after this changing the script did run through with the warnings you mentioned. when booting the disk generated by the script, root volume password and then
dev/mapper/Lucks-Dvc-root: clean, ...
ERROR: device '' not found. Skipping fsck.
mount: wrong fs type, bad option, bad superblock ,
missing code page or helper program, ... try dmesg | tail
ERROR: root device mounted success..., but /usr/lib/systemd/systemd does not exist.
Bailing out...
sh: cant access tty: job control turned off
[rootfs /]#
when searching for this error, found:
"Looks like that script doesn't install systemd-sysvcompat, which provides the /sbin/init symlink to /usr/lib/systemd/systemd. You might want to file a bug report and/or use pacstrap in the future."
-- https://bbs.archlinux.org/viewtopic.php?id=146712
(post by WorMzy 2013-02-06 15:11:34)
What about using LVM on top of LUKS instead of the other way around used in your script?
I used the following instructions on a number of machines and it worked just fine: http://is.gd/OoDx1d
Related
I am trying to create partition using the below script. It works, but sometimes I need to re-run the same script due to some automation stuff during the time I am getting error like already in use.
parted /dev/sdc --script mklabel gpt mkpart xfspart xfs 0% 100%
mkfs.xfs /dev/sdc1
mkdir -p /opt/app
lsblk
echo "/dev/sdc1 /opt/app xfs defaults 0 1" >>/etc/fstab
mount -a
df -Th
Error log when I executing the same script again.
+ parted /dev/sdc --script mklabel gpt mkpart xfspart xfs 0% 100%
Error: Partition(s) on /dev/sdc are being used.
+ mkfs.xfs /dev/sdc1
mkfs.xfs: /dev/sdc1 contains a mounted filesystem
How can I skip if the file system from /dev/sdc is already mounted?
You can check if the mount is already there, and skip the script if it's found;
if grep -qs '/dev/sdc' /proc/mounts; then
echo "Skipping mount since /dev/sdc already exists"
else
parted /dev/sdc --script mklabel gpt mkpart xfspart xfs 0% 100%
mkfs.xfs /dev/sdc1
mkdir -p /opt/app
lsblk
echo "/dev/sdc1 /opt/app xfs defaults 0 1" >>/etc/fstab
mount -a
df -Th
fi
im trying to filter newly added drives like da0,da1...(depends on hard disk type)etc. this should be happen through dynamically like if i run ls /dev/ | grep "da\|ada\|vtbd\|nvd" i should get only newly added drive or unmounted drives. so i need to delete mounted drive and should filter unmounted drives. how to filter this using sed or awk command?
root#ostest:~ # lsblk
DEVICE MAJ:MIN SIZE TYPE LABEL MOUNT
da0 0:93 5.0G zfs - -
da1 0:104 5.0G zfs - -
da2 0:105 5.0G zfs - -
da3 0:107 5.0G zfs - -
vtbd0 0:87 50G GPT - -
vtbd0p1 0:89 256K freebsd-boot gptid/0599c6ae-5f2f-11eb-a6b0-1707445d9baa -
vtbd0p2 0:90 2.0G freebsd-swap gpt/swap0 -
vtbd0p3 0:91 48G freebsd-zfs gpt/disk0 <ZFS>
vtbd1 0:88 482K cd9660 iso9660/config-2 /var/lib/cloud/seed/config_drive
You can run the following command for listing unmounted volumes:
lsblk --noheadings --raw | awk '{print substr($0,0,4)}' | uniq -c | grep 1 | awk '{print "/dev/"$2}' | grep -E "(/dev/.*)[[:digit:]]"
With this command we are listing all the partitions ignoring the disks by using grep -E "(/dev/.*)[[:digit:]].
Below is a snippet of my automation script, ownership gets changed for the directory (or)mount point - /deploy/umbro/$Client to ind$Client:ind as expected, but on the other hand, ownership for the directory (or)mount point, under the case statements are not getting changed.
Still remains as root:root
Not exactly sure where I have gone wrong.
#!/bin/bash
Client=$1
Region=$2
sudo mkfs -t xfs /dev/nvme1n1
sudo mkfs -t xfs /dev/nvme2n1
#Mount point creation for nvme2n1
mkdir -p /deploy/umbro/$Client
mount -t xfs /dev/nvme2n1 /deploy/umbro/$Client
sudo echo UUID=$(sudo blkid | grep /dev/nvme2n1 | grep -Eo [\"].*[\"] | awk '{print $1}'| tr -d '"') /deploy/umbro/$Client xfs defaults,nofail 0 2 >> /etc/fstab
perm=ind$Client:ind
chown -R $perm /deploy/umbro/$Client
#Mount point creation for nvme1n1, based on region
case $Region in
AUS)
mkdir -p /deploy/umbro/$Client/checkpoint/default/logs
chown -R ind$Client:ind /deploy/umbro/$Client/checkpoint/default/logs
mount -t xfs /dev/nvme1n1 /deploy/umbro/$Client/checkpoint/default/logs
sudo echo UUID=$(sudo blkid | grep /dev/nvme1n1 | grep -Eo [\"].*[\"] | awk '{print $1}'| tr -d '"') /deploy/umbro/$Client/checkpoint/default/logs xfs defaults,nofail 0 2 >> /etc/fstab
;;
EUR)
mkdir -p /deploy/umbro/$Client/checkpoint/arm/logs
chown -R ind$Client:ind /deploy/umbro/$Client/checkpoint/arm/logs
mount -t xfs /dev/nvme1n1 /deploy/umbro/$Client/checkpoint/arm/logs
sudo echo UUID=$(sudo blkid | grep /dev/nvme1n1 | grep -Eo [\"].*[\"] | awk '{print $1}'| tr -d '"') /deploy/umbro/$Client/checkpoint/arm/logs xfs defaults,nofail 0 2 >> /etc/fstab
;;
......
......
esac#!/bin/bash
Client=$1
Region=$2
sudo mkfs -t xfs /dev/nvme1n1
sudo mkfs -t xfs /dev/nvme2n1
#Mount point creation for nvme2n1
mkdir -p /deploy/umbro/$Client
mount -t xfs /dev/nvme2n1 /deploy/umbro/$Client
sudo echo UUID=$(sudo blkid | grep /dev/nvme2n1 | grep -Eo [\"].*[\"] | awk '{print $1}'| tr -d '"') /deploy/umbro/$Client xfs defaults,nofail 0 2 >> /etc/fstab
perm=ind$Client:ind
chown -R $perm /deploy/umbro/$Client
#Mount point creation for nvme1n1, based on region
case $Region in
AUS)
mkdir -p /deploy/umbro/$Client/checkpoint/default/logs
chown -R ind$Client:ind /deploy/umbro/$Client/checkpoint/default/logs
mount -t xfs /dev/nvme1n1 /deploy/umbro/$Client/checkpoint/default/logs
sudo echo UUID=$(sudo blkid | grep /dev/nvme1n1 | grep -Eo [\"].*[\"] | awk '{print $1}'| tr -d '"') /deploy/umbro/$Client/checkpoint/default/logs xfs defaults,nofail 0 2 >> /etc/fstab
;;
EUR)
mkdir -p /deploy/umbro/$Client/checkpoint/arm/logs
chown -R ind$Client:ind /deploy/umbro/$Client/checkpoint/arm/logs
mount -t xfs /dev/nvme1n1 /deploy/umbro/$Client/checkpoint/arm/logs
sudo echo UUID=$(sudo blkid | grep /dev/nvme1n1 | grep -Eo [\"].*[\"] | awk '{print $1}'| tr -d '"') /deploy/umbro/$Client/checkpoint/arm/logs xfs defaults,nofail 0 2 >> /etc/fstab
;;
......
......
esac
AWS EC2 - Red Hat Enterprise Linux Server release 7.7,
user - root
Strange observation is that, if I manually do the below steps, ownership gets changed recursievly till logs folder.
cd /deploy/umbro/$Client
chown -R ind$Client:ind checkpoint/
chown after mounting worked. thank you #thatotherguy for the nice explanation.
I am trying to write a Makefile whose targets depend on the existence of a disk file. The disk file itself merely needs to be created; it does not depend on any other actions that may update it. If I do not give it any dependencies, the file is re-created every time I run make on one of the targets.
Is there a way to have a target depend on whether it exists?
This is part of the Makefile I have. The $(TMPDEV) file only needs to be created if it doesn't exist, otherwise it should be considered up-to-date.
TMPDEV="/tmp/disk.img"
$(TMPDEV):
fallocate -l 806354944 $(TMPDEV) || dd if=/dev/zero of=$(TMPDEV) bs=1b count=1574912
sudo parted --script $(TMPDEV) unit s mklabel msdos \
mkpart primary fat16 2048 526335 \
mkpart primary fat32 526336 1050623 \
mkpart primary NTFS 1050624 1574911 \
quit
$(eval TMPDISK := $(shell sudo partx --verbose -a $(TMPDEV) | tail -1 | cut -d':' -f1))
sudo mkfs.fat -F 16 -n FAT16 $(TMPDISK)p1
sudo mkfs.fat -F 32 -n FAT32 $(TMPDISK)p2
sudo mkfs.ntfs -L NTFS $(TMPDISK)p3
sudo partx -d $(TMPDISK)
sudo losetup -d $(TMPDISK)
testresults: $(TMPDEV)
touch testresults
analytics: $(TMPDEV)
touch analytics
Remove the quotes:
TMPDEV="/tmp/disk.img"
Make doesn't use/need quotes. You are saying that the target here:
$(TMPDEV):
is, literally, this file including the quotes:
"/tmp/disk.img":
that file never exists, so the rule is always re-run.
I am experiencing difficulty trying to launch a AMI from an EBS volume. I am basically trying to launch another instance of a Linux (i386) based AMI that I have already configured the way I want. I have followed many guides for the past week. So far, I am able to create the custom private AMI but I am unable to connect to it after launching the new instance. I suspect that the AMI I have created is miss-configured in some way (maybe files didnt get fully copied over).
Anyhow here are the basic steps I'm going through to try to create the AMI:
ec2-create-volume -K pk-xxxxxx.pem -C
cert-xxxxxx.pem --size 10
--availability-zone us-east-1a
ec2-attach-volume -K pk-xxxxxx.pem -C
cert-xxxxxx.pem vol-xxxxxx --instance
xxxxxx --device /dev/sdh
yes | mkfs -t ext3 /dev/sdh
mkdir/mnt/ebsimage
echo '/dev/sdh /mnt/ebsimage ext3
defaults,noatime 0 0' >> /etc/fstab
mount /mnt/ebsimage
umount /mnt/ebsimage
ec2-detach-volume -K pk-xxxxxx.pem -C
cert-xxxxxx.pem vol-xxxxxx --instance
xxxxxx
ec2-create-snapshot -K pk-xxxxxx.pem
-C cert-xxxxxx.pem vol-xxxxxx
ec2reg -K pk-xxxxxx.pem -C
cert-xxxxxx.pem -s snap-xxxxx -a i386
-d -n --kernel aki-xxxxx --ramdisk ari-xxxxxx
I'm pretty sure either my commands around mount are messed up or my commands around ec2reg are messed up. Any suggestions?
I have also tried replacing
yes | mkfs -t ext3 /dev/sdh
mkdir/mnt/ebsimage
echo '/dev/sdh
/mnt/ebsimage ext3 defaults,noatime 0
0' >> /etc/fstab
mount /mnt/ebsimage
with a script designed to use rsync and add some other details but again the new instance of the ami launched cannot be connected to. Here is a copy of the script.
#!/bin/sh
vol=/dev/sdh
ebsmnt=/mnt/ebsimage
mkdir ${ebsmnt}
mkfs.ext3 -F ${vol}
sync
echo "mount $vol $ebsmnt"
mount $vol $ebsmnt
mkdir ${ebsmnt}/mnt
mkdir ${ebsmnt}/proc
mkdir ${ebsmnt}/sys
devdir=${ebsmnt}/dev
echo "mkdir ${devdir}"
mkdir ${devdir}
mknod ${devdir}/null c 1 3
mknod ${devdir}/zero c 1 5
mknod ${devdir}/tty c 5 0
mknod ${devdir}/console c 5 1
ln -s null ${devdir}/X0R
rsync -rlpgoD -t -r -S -l -vh \
--exclude /sys --exclude /proc \
--exclude /dev \
--exclude /media --exclude /mnt \
--exclude /sys --exclude /ebs --exclude /mnt \
-x /* ${ebsmnt}
df -h
Because I have the same results as the first example, I'm not sure if I'm closer to solving this issue or further away. Any help would be appreciated.
To create your EBS AMI from an S3 based AMI, you can use my blog post:
http://www.capsunlock.net/2009/12/create-ebs-boot-ami.html
I don't know which distribution you are trying to run, but if you want to run debian, there is a script which manages the entire bootstrapping process including ami creation (EBS boot).
You can find it on my github account:
https://github.com/andsens/ec2debian-build-ami
The script has been thoroughly tested and allows you to include other scripts in order to customize your ami. If you want to modify the script itself, just fork it, at least you then have a base to work from, where you know everything works.
I would not recommend the process you outlined though, it seems quite 'messy'.