I'm hoping to find a C/C++ library that can read a number of files off an ext formatted volume from within an application in Windows. I do not need to mount this volume in a traditional way, all I need is API access to the files. Read only is fine. My one application is the only application who needs access to the volume.
In short, instead of an installable filesystem for Windows, I would prefer a library such that drivers do not need to be installed. I'm able to detect when the disk arrives, the volumes location etc.
Most important to me is the ability to read the files off the volume reliably and without the need for an installed filesystem.
I do not need write support
The ext2read project is able to read ext2, ext3 and ext4 filesystems. It achieves this entirely from user space and does not rely on a kernel driver.
It's free software (GPL) and the source code is available on GitHub. Some of the more technical aspects are also discussed at length in the project blog.
(Disclaimer: I'm not affiliated with this project in any way, but I think it can solve your problem.)
Related
I want to study the source files of some of the device drivers that are installed and loaded on either a raspberry pi(raspian), beaglebone(debian) or a my laptop(ubuntu).
My aim is to learn how to properly implement my own modules by studying the source files of some drivers that actually works.
I am particularly interested in drivers that communicates with actual hardware (USB, I2C, SPI, UART etc).
Can someone tell me how to find these sources? are they available in some particular folder i.e something like /usr/src/**** or do I have to download all of the kernel source files from a particular kernel release?
All advice's, opinions and recommendations are most appreciated.
p.s I have read "Linux Kernel Development 3rd edition" but please tell me if
you know any other free/open-source books on the subject.
Best regards
Henrik
Linux Source directory and description :
arch/ -
The arch sub-directory contains all of the architecture specific kernel code.
Example :
1. 'arch/arm/' will have your board related configuration file.
like 'arch/arm/mach-omap/' will have omap specific source code.
2. 'arch/arm/config' Generates a new kernel configuration with the
default answer being used for all options. The default values
are taken from a file located in the arch/$ARCH/defconfig
file,where $ARCH refers to the specific architecture for which
the kernel is being built.
3. arch/arm/boot have kernel zImage, dtb image after compilation.
block/ -
This folder holds code for block-device drivers. Block devices are devices that accept and send data in blocks. Data blocks are chunks of data instead of a continual stream.
crypto/ -
This folder contains the source code for many encryption algorithms.
example, “sha1_generic.c” is the file that contains the code for
the sha1 encryption algorithm.
Documentation/ -
It has kernel related information in text format.
drivers/ - All of the system's device drivers live in this directory. They are further sub-divided into classes of device driver.
Example,
1. drivers/video/backlight/ has blacklight driver source which
will control display brightness.
2. drivers/video/display/ has display driver source.
3. drivers/input/ has input driver source code. like touch,
keyboard and mouse driver.
4. drivers/char/ has charter driver source code.
5. drivers/i2c/ has i2c subsystem and driver source code.
6. drivers/pci/ has pci subsytem and driver related source code.
7. drivers/bluetooth has Bluetooth driver file.
8. drivers/power has power and battery driver.
firmware/ -
The firmware folder contains code that allows the computer to read and understand signals from devices. For illustration, a webcam manages its own hardware, but the computer must understand the signals that the webcam is sending the computer.
fs/ -
All of the file system code. This is further sub-divided into directories, one per supported file system, for example vfat and ext2.
kernel/ -
The code in this folder controls the kernel itself. For instance, if a debugger needed to trace an issue, the kernel would use code that originated from source files in this folder to inform the debugger of all of the actions that the kernel performs. There is also code here for keeping track of time. In the kernel folder is a directory titled "power". Some code in this folder provide the abilities for the computer to restart, power-off, and suspend.
net/ -
net
The kernel's networking code.
lib
This directory contains the kernel's library code. The architecture specific library code can be found in arch/*/lib/.
scripts
This directory contains the scripts (for example awk and tk scripts) that are used when the kernel is configured.
lib/ -
This directory contains the kernel's library code. The architecture specific library code can be found in arch/*/lib/.
scripts/ -
This directory contains the scripts (for example awk and tk scripts) that are used when the kernel is configured.
mm/ -
This directory contains all of the memory management code. The architecture specific memory management code lives down in arch/*/mm/, for example arch/i386/mm/fault.c.
ipc/ -
This directory contains the kernels inter-process communications code.
**init/ -**The init folder has code that deals with the startup of the kernel (INITiation). The main.c file is the core of the kernel. This is the main source code file the connects all of the other files.
sound/ - This is where all of the sound card drivers are.
There are few more directory certs, crypto, security, include, virt and usr etc....
There are a few different methods that I use for viewing kernel related source, and I'm sure there are a few other good methods out there as well. You will find that the methods are largely the same.
Head on over to kernel.org and download the kernel of your choice. You will find driver related source under /<path to your downloaded kernel>/drivers. For example, I have downloaded and extracted kernel 4.5.5 to /usr/src/linux-4.5.5, and access the source for my drivers via /usr/src/linux-4.5.5/drivers.
Use a linux cross-reference website. Personally, I use the one hosted on free-electrons. These websites are nice for their free-text or identifier searches.
Browse Linus Torvalds' linux repo hosted on github.
Never mind, I found the source files under
~/linux/drivers/
example:
nano ~/linux/drivers/spi/spi-bitbang.c
Sorry, for any inconvenience!
I need to provide a single update file to a customer in order to update an embedded system via USB. The system is built using Yocto. I'm curious if the plan I have to implement USB updating is viable, or if I'm missing something that should be obvious.
opkg exists on the system, but in order to use opkg update it needs to have a repo to pull from. Since I have no network capabilities I will need to put the entire repo on a USB drive. Since I need to provide a single file to the customer the repo will then need to be a tar file.
Procedure
Plug USB drive in
The udev rule calls a script and pushes it to the background since this will be a long process (see this)
Un-tar the repo update file
opkg update
Notify user they may remove USB drive
At least from a high-level point of view does this sound like it is a good way to update an embedded system via USB? What pitfalls might exist?
Your use case is covered by SWUpdate - it is maybe worth to take a look at my project (github.com/sbabic/swupdate). Upgrading from USB with a single image file is one of the use case, and you can use meta-swupdate (listed on openembedded) to generate the single image with all artifacts.
Well, regarding what pitfalls might exist, one of the biggest pitfall would likely be a power outage in the middle of the process. How do you recover in that case? (The answer might depend on what type of embedded device your making). (Personally, I'm in favour of complete image based upgrades and not package based upgrades).
Regarding your scenario with the tarball, do you have enough space on your device to unpack out? It might be wiser to instead of distributing a tarball with your opkg repository, to distribute eg a ext2 or squashfs image of the repository. That would allow you to mount it from the USB drive using the loopback device.
Apart from that, as long as you have a good way to communicate work the user, your approach should work. The main issue is what do you do in case of an interrupted upgrade process. That is something you need to think about in advance.
When deciding on the update format and design there are several things to consider; including what you want to update (e.g. kernel, applications, bootloader); there is a paper on the most popular designs here: https://mender.io/user/pages/04.resources/_white-papers/Software%20Updates.pdf
In your case (no network, lots of storage available on USB) the easiest approach is probably a full rootfs update. I am involved in an open source project Mender.io which does a dual A/B rootfs update and integrates with the Yocto Project to make it easy and fast to enable updates on devices without custom low-level coding.
I have a big collection of folders for projects I'm working on. I've been trying to find a better way to sort them all for a long time and I want to write an app that creates groups based on whatever criteria I say, such as "folders from 2011" or "folders containing a x type of file" etc.
This is fairly straightforward, and wouldn't present much of a problem to code using its own UI in winForms or WPF or something. But I think it would be far better if I could make these folders appear to be part of the filesystem, so other apps (like existing file explorers) can see them.
Is this possible? Would it cause problems I haven't considered? How do I go about doing it if it is possible?
One way I thought of doing it would be to have the app monitor the filesystem and create folder shortcuts every time there's a change, but I'm curious about whether its possible to actually present a fake filesystem to explorer through a 'gateway' folder
EDIT: Ok it's obviously possible since http://www.virtualfolder.net/ can do it, and now that I think of it so can TrueCrypt, although it would be nice if it didn't have to appear as a separate drive. So the question becomes, how do I implement it?
You can create a Shell Namespace Extension that gathers the file information you want and displays it within Windows Explorer any way you wish. You can choose where your extension is located, whether as its own top-level node, a child of another system virtual folder/extension, or as a child of a file system folder.
Writing a SNE is not trivial, but it is a lot easier then writing a lower-level file system driver, and it does not require special driver-oriented compilers. Any compiler that supports developing COM objects will work.
This is accomplished using filesystem drivers or filesystem filter drivers. First let you create a virtual filesystem and mount it to a drive letter and also to a folder on NTFS drive (folder must exist but its contents are "replaced" with a virtual filesystem directory tree). Filesystem filter drivers let you introduce virtual files and folders in existing folders without replacing them.
VirtualFolder uses filesystem driver as it creates a drive letter.
Both types of drivers are written in C and work in kernel-mode. Writing them requires deep knowledge of Windows internals and experience with driver development (since filesystem drivers are one of the most complicated driver types).
We offer several products related to virtual storage. One of them, Callback File System, is a filesystem driver. It calls your user-mode code to perform actual filesystem functions. Another product, CallbackFilter, is an FS filter driver (and it also calls your user-mode code). However, current version of CallbackFilter doesn't let you introduce virtual files and folders (this would be implemented in the next release).
There's also Pismo File Mount product available, they use filter driver techniques. You can check with them if what you need can be accomplished.
From what I gather you are looking for a way to present the results of predefined file queries to appear as though they are located at a specific location in the file system. If that is correct you may want to look into Hard Links and Junctions. There are limits on what you can do with these file system services. However it is really straight forward to implement.
For a variety of reasons, revolving around cost of copy and the travails of the Windows filesystem, I need to mount a stream encoder as a drive, so that incoming data can simply be blindly directed at this "drive", aggregated, and encoded, without the source program being any the wiser. This would be basically trivial on Linux, but seems to be an uphill struggle on windows.
Specifically, I'd like to be able to "mount" a tar builder, which I know sounds odd, but there's a compelling reason for doing this. Is there a utility or library that deals with this? Perhaps an obscure part of the Windows API?
This looks promising... but it seems intended to mount folders or similar, rather than "devices." I do have control of where the data is written, so I can specify an arbitrary path.
Having experience with virtual drives (see our Virtual Storage product line) I can say that your task needs some redefining. As said in comments, drives (or, better say, filesystems) in Windows are expected to be namely filesystems (unlike Unix world), and as such they must support certain reading and enumeration operations, which is not something you'd expect.
Probably the closest you can do is a virtual drive in memory whose contents are then passed to your application in some way. The user will drag the data to your drive and on unmounting (or on other command) drive contents are passed to other program.
Several of our products can be used for your task (see CallbackDisk, Callback File System and SolFS OS Edition on Virtual Storage page), yet they are all commercial products. If you have a one-time or short-term task, you can build something for your use with a trial key.
There exist free approaches to your task, namely Pismo File Mount and Dokan, but I don't know how well they fit.
When I boot up Linux Mint from a Live CD, I am able to save files to the "File System". But where are these files being saved to? Can't be the disc, since it's a CDR. I don't think it's stored in the RAM, because it can only hold so much data and isn't really intended to be used as a "hard drive". The only other option is the hard drive... but it's certainly not saving to any partition on the hard drive I know about, since none of them are mounted. Then where are my files being saved to??
Believe it or not, it's a ramdisk :)
All live distros mount a temporary hard disk in RAM memory. The process is completely user-transparent and is all because of the magic of Linux kernel.
The OS, in fact, first allocates an area of your RAM memory into a virtual device, then mounts it as a regular hard drive in your file system.
Once you reboot, you lose all your data from that ramdrive.
Ramdrive is needed by almost all software running on Live CDs. In fact, almost all programs, in particular desktop managers, are designed in order to write files, even temporary, during their execution.
As an example, there are two ways to run KDE on a Live CD: either modify its code deeply in order to disallow you to change wallpaper etc. (the desktop settings are stored inside ~/.kde) or redeploy it onto a writable file system such as a ramdrive in order to avoid write fails on read-only file systems.
Obviously, you can mount your real HDD or any USB drive into your virtual file system and make all writes to them permanent, but by default no live distro mounts your drives into the root file system, instead they usually mount into specific subdirectories like /mnt, /media, /windows
Hope to have been of help.
It does indeed emulate a disk using RAM; from Wikipedia:
It is able to run without permanent
installation by placing the files that
typically would be stored on a hard
drive into RAM, typically in a RAM
disk, though this does cut down on the
RAM available to applications.
RAM. In Linux, and indeed most unix systems, any kind of device is seen as a file system.
For example, to get memory info on linux you use cat /proc/meminfo, where cat is used to read files. Then, there's all sorts of strange stuff like /dev/random (to read random crap) and /dev/null (to throw away crap). ;-)
To make it persistent - use a USB device - properly formatted and with a special name. See here:
https://help.ubuntu.com/community/LiveCD/Persistence