I need to provide a single update file to a customer in order to update an embedded system via USB. The system is built using Yocto. I'm curious if the plan I have to implement USB updating is viable, or if I'm missing something that should be obvious.
opkg exists on the system, but in order to use opkg update it needs to have a repo to pull from. Since I have no network capabilities I will need to put the entire repo on a USB drive. Since I need to provide a single file to the customer the repo will then need to be a tar file.
Procedure
Plug USB drive in
The udev rule calls a script and pushes it to the background since this will be a long process (see this)
Un-tar the repo update file
opkg update
Notify user they may remove USB drive
At least from a high-level point of view does this sound like it is a good way to update an embedded system via USB? What pitfalls might exist?
Your use case is covered by SWUpdate - it is maybe worth to take a look at my project (github.com/sbabic/swupdate). Upgrading from USB with a single image file is one of the use case, and you can use meta-swupdate (listed on openembedded) to generate the single image with all artifacts.
Well, regarding what pitfalls might exist, one of the biggest pitfall would likely be a power outage in the middle of the process. How do you recover in that case? (The answer might depend on what type of embedded device your making). (Personally, I'm in favour of complete image based upgrades and not package based upgrades).
Regarding your scenario with the tarball, do you have enough space on your device to unpack out? It might be wiser to instead of distributing a tarball with your opkg repository, to distribute eg a ext2 or squashfs image of the repository. That would allow you to mount it from the USB drive using the loopback device.
Apart from that, as long as you have a good way to communicate work the user, your approach should work. The main issue is what do you do in case of an interrupted upgrade process. That is something you need to think about in advance.
When deciding on the update format and design there are several things to consider; including what you want to update (e.g. kernel, applications, bootloader); there is a paper on the most popular designs here: https://mender.io/user/pages/04.resources/_white-papers/Software%20Updates.pdf
In your case (no network, lots of storage available on USB) the easiest approach is probably a full rootfs update. I am involved in an open source project Mender.io which does a dual A/B rootfs update and integrates with the Yocto Project to make it easy and fast to enable updates on devices without custom low-level coding.
Related
I wrote a C++ program with multiple classes and divided it into multiple files, which is intended to run on an embedded device (raspi 2 to be specific) that has no internet access. Therefore building the source and installing the dependencies on every device would be very laborious.
Is there a way to compile the program on one of the devices (as an exception to the others, this one has internet access), so that I can just transfer the build files, e.g. via USB, to the other devices? This should also include the various libraries I used so that I don't have to install them on every device. These are mainly std, but also a self-cloned and build library and a with apt installed library (I linked the libraries used as an example, but they shouldn't affect the process, I guess).
I'm using CMake. Is there an option, to make CMake compile a program into a (set of) files that run independently of the system-installed libraries with other words: they run without the need to have the required libraries installed on the system, but shipped with the build files.
Edit:
My main problem is, that I cannot get a certain dependency on the target devices due to a lack of internet access. Can I build the package and also include the library in that build, without me having to install it?
I'm not sure I fully understand why you need internet for your deployment but I can give you several methods and you'll choose the one you seem the best.
Method A: Cloning the SD card image
During your development phase, you ended to successfully have a RPi device working and you want to replicate this. You can use some tools to duplicate your image on another SD card, N times and eventually, this could be sufficient to make it work.
Pros: Very quick method
Cons: Usually, your development phase involve adjustments, trying different tools, different versions, etc.. your original RPi image is not clean and so you'll replicate this. Definitely not valid for an industrial project for could be sufficient for a personal one.
Method B: Create deployments scripts
You can create a deployment script on your computer to copy, configure, install what you need. Assuming you start with a certain version of Raspberry pi OS, you flash it, then you boot your PI that is connected via Ethernet for example. You can start a script on your computer that will:
Copy needed sources / packages / binaries
(Optional) compile sources (if you have a compiler that suits your need on RPi OS)
Miscellaneous configuration
To do all these, a script like this can do the job:
PI_USERNAME="pi"
PI_PASSWORD="raspberry"
PI_IPADDRESS="192.168.0.3"
# example on how to execute a command remotely
sshpass -p ${PI_PASSWORD} ssh ${PI_USERNAME}#${PI_IPADDRESS} sudo apt update
# example on how to copy a local file on the RPI
sshpass -p ${PI_PASSWORD} scp local_dir/nlohmann/json.hpp ${PI_USERNAME}#${PI_IPADDRESS}:/home/pi/sources
Importante note:
Hard coded credentials is not recommended.
This script is assuming you are using linux but you'll find equivalent tool under Windows.
This assume your RPI has a fixed IP but you can still improve this script to automatically find the RPI on the network. (lots of possibilities)
Pros: While you create this deployment script, you'll force yourself to start from a clean image and no dirty environment is duplicated.
Cons: Take a bit longer than method A
Method C: Create your own Raspberry PI image using Yocto
Yocto is a tool to create your own images, suitable for Raspberry PI. You can customize absolutely everything and produce an sdcard image you can just flash your RPis SD cards.
Pros: Very complete tool, industrial process
Cons: Quite complicated to deal with, not suitable for beginners, time cost
Saying in the comments it's only for 10 devices and that you were a bit scared to cross compile, I would not promote the Yocto method for you. I would not recommend the method A as well mostly because of the dirty environment duplication (but up to you in the end). The method B with the deployment script may be the best to go.
I have a big collection of folders for projects I'm working on. I've been trying to find a better way to sort them all for a long time and I want to write an app that creates groups based on whatever criteria I say, such as "folders from 2011" or "folders containing a x type of file" etc.
This is fairly straightforward, and wouldn't present much of a problem to code using its own UI in winForms or WPF or something. But I think it would be far better if I could make these folders appear to be part of the filesystem, so other apps (like existing file explorers) can see them.
Is this possible? Would it cause problems I haven't considered? How do I go about doing it if it is possible?
One way I thought of doing it would be to have the app monitor the filesystem and create folder shortcuts every time there's a change, but I'm curious about whether its possible to actually present a fake filesystem to explorer through a 'gateway' folder
EDIT: Ok it's obviously possible since http://www.virtualfolder.net/ can do it, and now that I think of it so can TrueCrypt, although it would be nice if it didn't have to appear as a separate drive. So the question becomes, how do I implement it?
You can create a Shell Namespace Extension that gathers the file information you want and displays it within Windows Explorer any way you wish. You can choose where your extension is located, whether as its own top-level node, a child of another system virtual folder/extension, or as a child of a file system folder.
Writing a SNE is not trivial, but it is a lot easier then writing a lower-level file system driver, and it does not require special driver-oriented compilers. Any compiler that supports developing COM objects will work.
This is accomplished using filesystem drivers or filesystem filter drivers. First let you create a virtual filesystem and mount it to a drive letter and also to a folder on NTFS drive (folder must exist but its contents are "replaced" with a virtual filesystem directory tree). Filesystem filter drivers let you introduce virtual files and folders in existing folders without replacing them.
VirtualFolder uses filesystem driver as it creates a drive letter.
Both types of drivers are written in C and work in kernel-mode. Writing them requires deep knowledge of Windows internals and experience with driver development (since filesystem drivers are one of the most complicated driver types).
We offer several products related to virtual storage. One of them, Callback File System, is a filesystem driver. It calls your user-mode code to perform actual filesystem functions. Another product, CallbackFilter, is an FS filter driver (and it also calls your user-mode code). However, current version of CallbackFilter doesn't let you introduce virtual files and folders (this would be implemented in the next release).
There's also Pismo File Mount product available, they use filter driver techniques. You can check with them if what you need can be accomplished.
From what I gather you are looking for a way to present the results of predefined file queries to appear as though they are located at a specific location in the file system. If that is correct you may want to look into Hard Links and Junctions. There are limits on what you can do with these file system services. However it is really straight forward to implement.
I'm hoping to find a C/C++ library that can read a number of files off an ext formatted volume from within an application in Windows. I do not need to mount this volume in a traditional way, all I need is API access to the files. Read only is fine. My one application is the only application who needs access to the volume.
In short, instead of an installable filesystem for Windows, I would prefer a library such that drivers do not need to be installed. I'm able to detect when the disk arrives, the volumes location etc.
Most important to me is the ability to read the files off the volume reliably and without the need for an installed filesystem.
I do not need write support
The ext2read project is able to read ext2, ext3 and ext4 filesystems. It achieves this entirely from user space and does not rely on a kernel driver.
It's free software (GPL) and the source code is available on GitHub. Some of the more technical aspects are also discussed at length in the project blog.
(Disclaimer: I'm not affiliated with this project in any way, but I think it can solve your problem.)
For a variety of reasons, revolving around cost of copy and the travails of the Windows filesystem, I need to mount a stream encoder as a drive, so that incoming data can simply be blindly directed at this "drive", aggregated, and encoded, without the source program being any the wiser. This would be basically trivial on Linux, but seems to be an uphill struggle on windows.
Specifically, I'd like to be able to "mount" a tar builder, which I know sounds odd, but there's a compelling reason for doing this. Is there a utility or library that deals with this? Perhaps an obscure part of the Windows API?
This looks promising... but it seems intended to mount folders or similar, rather than "devices." I do have control of where the data is written, so I can specify an arbitrary path.
Having experience with virtual drives (see our Virtual Storage product line) I can say that your task needs some redefining. As said in comments, drives (or, better say, filesystems) in Windows are expected to be namely filesystems (unlike Unix world), and as such they must support certain reading and enumeration operations, which is not something you'd expect.
Probably the closest you can do is a virtual drive in memory whose contents are then passed to your application in some way. The user will drag the data to your drive and on unmounting (or on other command) drive contents are passed to other program.
Several of our products can be used for your task (see CallbackDisk, Callback File System and SolFS OS Edition on Virtual Storage page), yet they are all commercial products. If you have a one-time or short-term task, you can build something for your use with a trial key.
There exist free approaches to your task, namely Pismo File Mount and Dokan, but I don't know how well they fit.
I have written a backup tool that is able to backup files and images of volumes for Windows. To detect which files have changed I use the Windows Change Journal. I already use the shadow copy functionality to do a consistent copy of both the files and the volume images.
To detect which blocks have changed I use hashes at the moment. This means the whole volume has to be read once (because to see which block has changed hashes of all blocks have to be calculated).
The backup integrated into Windows 7 is able to create incremental volume images without checking all blocks. I wasn't able to find an API for a kind of block level change journal.
Does anybody know how to access this information?
(I'm willing to dive deep into NTFS internals - even reading and parsing special files)
I don't think block level change info is available anywhere. Most probably what the Windows 7 integrated backup does is it installs a File System Filter Driver like some backup products does and anti-virus software. A filter driver can intercept all file system calls and in this way know which blocks changed. If you do this you can basically build your own change journal that works block level but only for the files that you are interested in.
I would really like to know a better answer myself here.
When you say Windows Change Journal I take it you are referring to the NTFS USN? It looks very much like the Windows 7 backup uses a combination of VSC and NTFS USN to detect changes and create incremental images much like you are already doing.