Intermittent failure executing on USB insert with auto-mount on Raspberry Pi - usb-drive

I've set up a raspberry pi to execute a command if a USB stick is inserted, and the command calls an executable on the stick.
This works about 80% of the time, but intermittently fails - seemingly at random. Because of the unpredictability I assume this is a race condition, however I'm not too familiar with where the risks are as I've pieced together the approach from information online. Most of the information comes from here.
The USB stick is auto-mounted with the following entry in /etc/fstab. I'm aware of the risk of /dev/sda1 changing but that does not appear to be the problem here:
/dev/sda1 /media/usb vfat defaults,rw,nofail,user,umask=000 0 0
A service waits for the USB to mount with the following configuration
[Unit]
Description=USB Mount Trigger
Requires=media-usb.mount
After=media-usb.mount
[Service]
ExecStart=/script.sh
[Install]
WantedBy=media-usb.mount
media-usb.mount comes from systemctl list-units -t mount, and /script.sh calls the USB stick's executable.
In failure cases, where the USB's executable is not called, I see the following from systemctl status service_name:
Nov 15 22:49:14 raspberrypi systemd[1]: Dependency failed for USB Mount Trigger.
Nov 15 22:49:14 raspberrypi systemd[1]: service_name.service: Job service_name.service/start failed with result 'dependency'.
In these cases if I execute systemctl list-units -t mount I do not see media-usb.mount and my USB stick is not mounted to /media/usb.
I think that an error / race condition in service_name.service causing the USB mount to die, because (I believe) a successful mount is required to trigger the service. If the USB is never inserted, systemctl status service_name simply reports Active: inactive (dead), so something is triggering the service to try to execute.

Related

How to start systemd user service when device is removed, and stop it when device is inserted

Systemd allows starting/stopping service from udev rule using SYSTEMD_USER_WANTS environment and StopWhenUnneeded option. But, service will be started when device is inserted and stopped when device is removed. What I need is vice versa:
start service when device removed
stop service when device inserted
Since it is user service, running 'systemctl start/stop ...' from udev rule fails.
udev rule for this question is
..., ACTION=="add", RUN+="/usr/bin/su USER -c 'systemctl --user stop my-service'"
..., ACTION=="remove", RUN+="/usr/bin/su USER -c 'systemctl --user stop my-service'"
The important points are:
Instead of SYSTEMD_WANTS/SYSTEMD_USER_WANTS, service should be start/stop using systemctl since start/stop does not match device add/remove.
To start/stop service from other user su + systemctl --user is used.
Program pass to RUN udev key should be either from /usr/lib/udev or absolute pass must be used (man udev).

analyze systemd journal of a crashed / dead system

i resently upgraded a system. after reboot i was not able to login again. all users have been rejected with Login incorrect. systemd with journaling was running and writing error messages to file in /var/log/journal as usual.
i so booted a system from a revovery usb stick (same distribution) mounted the root device of the failed system /mnt and tried to analyze the logs with journalctl --root=/mnt/var/log/journal -xe. journalctl did not find journal files.
question: how can i read systemd journal content of a dead system using a recovery system?
have fun
I may be a bit late, but I stumbled upon this question and here is what I found:
journalctl logs are located in /var/log/journal/*
journalctl app can read foreign journal files with the following switches:
--file= followed by the *.journal file of your choice. This option may be used multiples times
--root= followed by the root directory of you choice, probably a mounted partition
--image= followed by a disk image,
files as argument, with the option --file

How systemd (v229) feeds hw watchdog

I enabled hw watchdog in systemd (RuntimeWatchdogSec=10) and believe that it was actually enabled:
# dmesg | grep watchdog
[ 3.328676] systemd[1]: Hardware watchdog 'OMAP Watchdog', version 0
[ 3.336760] systemd[1]: Set hardware watchdog to 10s.
Then I wanted to test out if the system really restarts when systemd stops working. As I could not find systemd documentation regarding which systemd daemon feeds the hw watchdog, I tried to kill wildly all systemd daemons one after another (systemd-udevd, systemd-timesyncd, systemd-networkd, systemd-journald, etc) until the console became non function after I killed the systemd daemon (which has a symlink /sbin/init in my system). Until this point, my system still did not restart. Any hints?
I've found out that systemd daemon actually opened /dev/watchdog:
# lsof /dev/watchdog
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd 1 root 4w CHR 10,130 0t0 6871 /dev/watchdog
And in my kernel CONFIG_WATCHDOG_NOWAYOUT is not set --> When I killed systemd, no process opened /dev/watchdog and the hardware watchdog was just disabled. When I enabled CONFIG_WATCHDOG_NOWAYOUT, board resets when systemd stops.

Docker container can't see a serial port device

I'm trying to run a Docker container with access to a serial port on the host.
Here is what I did:
I used a Mac
Installed drivers on the host
(http://www.prolific.com.tw/US/ShowProduct.aspx?p_id=229&pcid=41)
Plugged in the device
Ran ls /dev/t* that returned
/dev/tty.usbserial - so it worked
Ran the container, docker run -it --privileged -v /dev:/dev
node:4.4.0 /bin/bash, and then ls /dev/t* inside the container which didn't return the /dev/tty.usbserial device...
I played a lot with different variations of parameters, but I haven't found the working one :)
Also the --device flag is not suitable for me since the device might be reconnected and the name could differ from /dev/tty.usbserial.
You can check if the script described in "Notification of new USB devices in docker container" (from Guido Diepen -- gdiepen) can help.
He too runs his container with the --privileged argument to allow it to access the devices. And he mounts the host directory /dev/bus/usb to the /dev/bus/usb directory within the container with the argument -v /dev/bus/usb:/dev/bus/usb when starting said container.
The script uses both inotifywait and lsusb to wait for devices to be (un)plugged and check if it was the device we are interested in.
The inotifywait will keep on listening to inodes create/delete events under the dev/bus/usb directory and will execute commands whenever an inode corresponding to a relevant device has been just created.
See also, once you have detected an plugged USB device, How to get Bus and Device relationship for a /dev/ttyUSB (not related to Docker, but still relevant).
As pointed by #pgayvallet on GitHub:
As the daemon runs inside a VM in Docker Desktop, it is not possible to actually share a mac host device with the container inside the VM, and this will most definitely never be possible.

EC2 small to micro instance downgrade problems

Have a few applications where EC2 small instances are, well, too large. So the announcement of micro instances is just what the doctor ordered.
I'd like to take a small instance's EBS volume, detach it, and pair it up with a micro instance. At some point it might be great to go the other way and upsize a micro instance to a small or beyond.
For this failed experiment I tried:
Creating a new small instance with the Alestic Ubuntu 10.04 32 bit AMI (ami-1234de7b). Boots like a charm.
Power down my freshly minted micro instance, detach the volume that was created for me in the previous step.
Attach the small instance's volume to the micro instance.
Power up.
Nada.
What's odd is there is no console log output until I power down. Then I see it all.
[ 0.000000] Reserving virtual address space above 0xf5800000
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
...
[ 1.221261] VFS: Mounted root (ext3 filesystem) readonly on device 8:1.
[ 1.221261] VFS: Mounted root (ext3 filesystem) readonly on device 8:1.
[ 1.222164] devtmpfs: mounted
[ 1.222202] Freeing unused kernel memory: 216k freed
[ 1.223409] Write protecting the kernel text: 4328k
[ 1.223760] Write protecting the kernel read-only data: 1336k
init: console-setup main process (63) terminated with status 1
%Ginit: plymouth main process (45) killed by SEGV signal
init: plymouth-splash main process (196) terminated with status 2
cloud-init running: Thu, 09 Sep 2010 17:37:54 +0000. up 2.61 seconds
mountall: Disconnected from Plymouth
init: hwclock-save main process (291) terminated with status 1
Checking for running unattended-upgrades: * Asking all remaining processes to terminate...
[80G
[74G[ OK ]
* All processes ended within 1 seconds....
[80G
[74G[ OK ]
* Deconfiguring network interfaces...
[80G
[74G[ OK ]
* Deactivating swap...
[80G
[74G[ OK ]
* Unmounting local filesystems...
[80G
[74G[ OK ]
* Will now halt
[ 185.599636] System halted.
This method of swapping has worked well between same sized instanced in the past and it's my first attempt at doing this between different sizes. Is this just not possible or am I missing something fundamental in my EC2 knowledge?
Even though this will probably be migrated to Server Fault, I ran into the exact same problem with this instance earlier today.
It appears that this image assumes that there will be ephemeral storage present, when there is none on the micro instances. To work around this, comment out the following line in /etc/fstab:
/dev/sda2 /mnt auto defaults,comment=cloudconfig 0 0
This should prevent the instance from hanging on startup, or at least it did for me with ami-1234de7b.
I created a new micro instance using alestic ami's (ami-2c354b7e). I was able to login to the system normally the first time. But once I reboot the system, I was not able to login again.
commenting the line indicated above worked for me. "/dev/sda2 /mnt auto defaults,comment=cloudconfig 0 0"
Commenting the line out doesn't fix it fully. If you reboot, it will write the same line back in. You need to:
$ l="deb http://archive.ubuntu.com/ubuntu lucid-proposed main"
$ echo "$l" | sudo tee -a /etc/apt/sources.list
$ sudo apt-get update && sudo apt-get install cloud-init
$ dpkg-query --show cloud-init
I'm assuming this will be fixed in the official Ubuntu release soon and you won't have to do this, but for now...
Source: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/634102
Also, we have a couple images based off the official Ubuntu AMI's that work on Micro's: http://blog.simpledeployr.com/2010/09/new-ruby-amis-with-latest-ubuntu-lucid.html
I don't see a problem on your side. This could be a problem in Amazon's infrastructure.

Resources