I have been closely following the Open ZFS development scene for OS X for the last few years. Things have progressed significantly over the last several months since the sad problems that occurred with Greenbytes, etc., but I have been pleased to see that we're finally close to getting real Spotlight support in place. I noticed this email passing by the other day from Jorgen Lundman (who has put a great deal of personal time into getting this going and contributing to the community) and thought perhaps others here might be interested in chiming in on this, his, topic regarding implementing Spotlight support for ZFS on OS X:
To summarize, I think the crux of this question boils down to this:
So then, what does "mds" use to iterate the mounted file systems? I do not
think the sources for "Spotlight-800.28" was ever released so we can't just
go look and learn, like we did for xnu, and IOkit.
It doesn't use the BSD getfsstat(), more likely it asks IOKit, and for some
reason rejects the lower mounts.
And the body of the email for convenience:
Hey guys,
So one of our long-term issues in OpenZFSonOSX is to play nice with Spotlight.
We have reached the point where everything sometimes pretends to work.
For example;
# mdfind helloworld4
/Volumes/hfs1/helloworld4.jpg
/Volumes/hfs2/helloworld4.jpg
/Volumes/zfs1/helloworld4.jpg
/Volumes/zfs2/helloworld4.jpg
Great, picks it up in our regular (control group) HFS mounted filesystems,
as well as the 2 ZFS mounts.
Mounted as:
/dev/disk2 on /Volumes/zfs1 (zfs, local, journaled)
/dev/disk2s1 on /Volumes/zfs2 (zfs, local, journaled)
# diskutil list
/dev/disk1
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *42.9 GB disk1
1: ZFS 42.9 GB disk1s1
2: 6A945A3B-1DD2-11B2-99A6-080020736631 8.4 MB disk1s9
/dev/disk2
#: TYPE NAME SIZE IDENTIFIER
0: zfs_pool_proxy FEST *64.5 MB disk2
1: zfs_filesystem_proxy ssss 64.5 MB disk2s1
So you can see, the actual pool disk is /dev/disk1, and the fake nodes we
create for mounting as /dev/disk2*, as it appears to be required by
Spotlight to work at all. We internally also let the volumes auto-mount,
from issuing "diskutil mount -mountPoint %s %s".
We are not a VOLFS, so there is no ".vol/" directory, nor will mdutil -t
work. But these two points are true for MS-DOS as well, and that does work
with Spotlight.
We correctly reply to zfs.fsbundle's zfs.util for "-p" (volume name) and
"-k" (get uuid), done pre-flight to mounting by DA.
Using FSMegaInfo tool, we can confirm that stat, statfs, readdir, and
similar tests appear to match that of HFS.
So then, the problem.
The problem comes from mounting zfs inside zfs. Ie,
When we mount
/Volumes/hfs1/
/Volumes/hfs1/hfs2/
/Volumes/zfs1/
/Volumes/zfs1/zfs2/
# mdfind helloworld4
/Volumes/hfs1/helloworld4.jpg
/Volumes/hfs1/hfs2/helloworld4.jpg
/Volumes/zfs1/helloworld4.jpg
Absent is of course, "/Volumes/zfs1/zfs2/helloworld4.jpg".
Interestingly, this works
# mdfind -onlyin /Volumes/zfs1/zfs2/ helloworld4
/Volumes/zfs1/zfs2/helloworld4.jpg
And additionally, mounting in reverse:
/Volumes/hfs2/
/Volumes/hfs2/hfs1/
/Volumes/zfs2/
/Volumes/zfs2/zfs1/
# mdfind helloworld4
/Volumes/hfs2/helloworld4.jpg
/Volumes/hfs2/hfs1/helloworld4.jpg
/Volumes/zfs2/helloworld4.jpg
So whichever ZFS filesystem was mounted first, works, but not the second.
So the individual ZFS filesystems are both equal. It is as if it doesn't
realise the lower mount is its own device.
So then, what does "mds" use to iterate the mounted fileystems? I do not
think the sources for "Spotlight-800.28" was ever released so we can't just
go look and learn, like we did for xnu, and IOkit.
It doesn't use the BSD getfsstat(), more likely it asks IOKit, and for some
reason rejects the lower mounts.
Some observations:
# /System/Library/Filesystems/zfs.fs/zfs.util -k disk2
87F06909-B1F6-742F-7355-F0D597849138
# /System/Library/Filesystems/zfs.fs/zfs.util -k disk2s1
8F60C810-2D29-FCD5-2516-2D02EED4566B
# grep uu /Volumes/zfs1/.Spotlight-V100/VolumeConfiguration.plist
<key>uuid.87f06909-b1f6-742f-7355-f0d597849138</key>
# grep uu /Volumes/zfs1/zfs2/.Spotlight-V100/VolumeConfiguration.plist
<key>uuid.8f60c810-2d29-fcd5-2516-2d02eed4566b</key>
Any assistance is appreciated, the main issue tracking Spotlight is;
https://github.com/openzfsonosx/zfs/issues/116
The branch for it;
https://github.com/openzfsonosx/zfs/tree/issue116
vfs_getattr;
https://github.com/openzfsonosx/zfs/blob/issue116/module/zfs/zfs_vfsops.c#L2307
It appears to be down to some undocumented expectations in the vfs_vget method, to lookup entries based entirely on the inode number. Ie, stat /.vol/16777222/1102011
It is expected that vfs_vget sets the vnode_name correctly here, using a call like vnode_update_identity() or similar.
Related
So, I have a raspberry pie 3 B. I wanted to know how much space is used so I plugged the micro sd card in my mac and opened the terminal.
When I ran the command:
df -h /dev/disk2
I got:
df: /dev/disk2: Raw devices not supported
What should I do now?
PS: I don't want to plug the RPI in.
You'd indeed need to mount the disk first, like Ferrybig said.
Note: It could be, since it's for the RPI, that you SD card is formatted with one of the EXT file system variants. To access those under MacOS, you'll need something like FUSE (free, but I have never used) or Paragon's ExtFS (commercial, I've used it quite often). Some SD cards for the RPI are FAT32 formatted - those should work just fine under MacOS.
Easiest way to mount a volume, if you don't want to mess too much with the commandline parameters, is by opening Disk Utlity. Find the disk/partition you'd like to mount, right click it and select "Mount". This works for "known" file system types.
Now after the disk has been mounted, type "mount" in terminal to see where it's mounted. It will show several lines, one of them could be (as an example):
/dev/disk2s1 on /Volumes/Untitled (ntfs, local, nodev, nosuid, read-only, noowners)
(there may be more that start with "/dev/disk2sX")
df- h /Volumes/Untitled will now show the disk space info, for example:
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/disk2s1 15Gi 57Mi 15Gi 1% 19 15239289 0% /Volumes/Untitled
If your disk2 has multiple partitions, then you'd need to repeat the steps for all disks that start with "/dev/disk2sX" (where X is a number).
I had a btrfs partition on a 6 disk array without raid (metadata in raid10, but data in single), and one of the disks just died.
So I lost some of my data, ok, I knew that.
But two questions:
Is it possible to know (using metadata I suppose) what data I have lost?
Is it possible to do some kind of a "btrfs delete missing" on this kind of setup, in order to recover access in rw to my other data, or I must copy all my data on a new partition?
Edit : just to be clear, I can mount it in read only with mount -o recovery,ro,degraded
And btrfs fi df /Data
Data, single: total=6.65TiB, used=6.65TiB
System, RAID1: total=32.00MiB, used=768.00KiB
Metadata, RAID1: total=13.00GiB, used=10.99GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
I'm a very very lucky guy, and I think I fixed my problem (thanks to the help of btrfs mailing list).
In my situation "btrfs-debug-tree -t 3 /dev/sda6" does not mention the missing disk anywhere (data or metadata). So there was nothing at all in the missing device.
Thus, patching the kernel with this patch allow me to mount the array in rw in degraded and a simple btrfs device remove missing did the trick.
So my array is fixed and my data seems fine (scrub in progress)
One thing I learned though is that the single mode should never ever be used.
I think FS stands for filesystem, but I don't know what BLK stands for. Not only that, but what are the meanings behind the pci hierarchy parameters. i.e. When I see HD(1,MBR,0x0003B) what does "1","MBR", and what looks to be an address, stand for?
Here's the mapping table I'm looking at in UEFI shell:
Mapping table
FS0: Alias(s):HD21a0e0b:;BLK1:
PciRoot(0x0)/Pci(0x1D,0x0)/USB(0x0,0x0)/USB(0x4,0x0)/HD(1,MBR,0x0003B)
FS1: Alias(s):HD23a0a1:;BLK4:
PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x0,0x0,0x0)/HD(1,MBR,0x00000000,0x3F)
BLK3: Alias(s):
PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x0,0x0,0x0)
BLK0: Alias(s):
PciRoot(0x0)/Pci(0x1D,0x0)/USB(0x0,0x0)/USB(0x4,0x0)
BLK2: Alias(s):
PciRoot(0x0)/Pci(0x1D,0x0)/USB(0x0,0x0)/USB(0x4,0x0)/HD(2,MBR,0x0003B)
I'm guessing BLK's are available ports and FS's are physical things that are plugged into those ports. It looks like once somethign is plugged into a BLK, it becomes an FS, but still retains its BLK value. i.g. FS0=BLK1
According to archwiki:
fsX means filesystem
blkX means block device or data storage device
MBR should mean Master Boot Record
HD should mean Hard Drive
1 might mean Primary, 2 Secondary Partition
That hex number after MBR could be the device signature or disk identifier. Or maybe an offset of that device to important information.
Links that might help further:
RHEL 5 Installation Guide EFI Shell Guide
Red Hat 7.1 Itanium EFI Shell Guide
HP Knowledge Base: "UEFI Shell 'fs' devices gone after restore from image backup"
OpenVMS: Firmware upgrades from a USB stick (on UEFI)
SourceForge EFI Shell Development Documentation
I am interested in iterating all mounted file systems on OSX (currently running 10.9 Mavericks). I am looking for something similar to getmntent() or the output of the mount shell command (although I want to do it from objective C, so parsing the output of a shell command is obviously not optimal).
I have been looking a bit at the disk arbitration framework, and it appears that I could be notified about mount and unmount events using this framework. I may be missing something there, but it isn't clear to me if there is a way to iterate existing mounted file systems using Disk Arbitration.
I have explored using getfsent() which seemed like it would provide a solution, but after testing I discovered that I am not getting more than one entry from iterating getfsent(). See the following code:
struct fstab* fsentry;
setfsent();
fsentry = getfsent();
while(fsentry)
{
//do something with fsentry
fsentry = getfsent();
}
endfsent();
The only entry I am getting here is for the / file system. The second time I call getfsent() it returns NULL, as if there are no more entries. The mount command shows me several others including a mounted cifs/smb file system:
/dev/disk0s2 on / (hfs, local, journaled)
devfs on /dev (devfs, local, nobrowse)
map -hosts on /net (autofs, nosuid, automounted, nobrowse)
map auto_home on /home (autofs, automounted, nobrowse)
//user#<ip address>/public on /Volumes/public (smbfs, nodev, nosuid, mounted by user)
So it seems like getfsent() starts doing what I expect, but for some reason stops?
My question in summary is: What is the best way to iterate file systems on OSX?
If anyone has an answer to why I am only getting one result from getfsent() I would also be interested in that.
There are a couple of different ways to enumerate mounted volumes on OS X, each a using different set of APIs. At the highest (and easiest) level, you can use NSFileManager's mountedVolumeURLsIncludingResourceValuesForKeys:options:. Here's an abbreviated example:
NSArray *urls = [[NSFileManager defaultManager] mountedVolumeURLsIncludingResourceValuesForKeys:#[NSURLVolumeNameKey] options:0];
for (NSURL *url in urls) {
NSLog(#"Volume mounted at: %#", [url path]);
}
The next option takes us back to C territory - and you were so close with your original approach. On OS X (and BSD), there isn't getmntent(); instead, there is getmntinfo(), which is strikingly similar. To list mounted volumes via getmntinfo(), you can do the following:
struct statfs* mounts;
int num_mounts = getmntinfo(&mounts, MNT_WAIT);
if (num_mounts < 0) {
// do something with the error
}
for (int i = 0; i < num_mounts; i++) {
NSLog(#"Disk type '%s' mounted at: %s", mounts[i].f_fstypename, mounts[i].f_mntonname);
}
I've used both of these APIs side-by-side since the release of 10.6. getmntinfo() is always more complete than [NSFileManager mountedVolumeURLsIncludingResourceValuesForKeys:options:]: the latter will filter the /dev and other filesystems that you may or may not want to know about. It is generally reliable, however, for the disks you plug into your system.
The purpose behind the DiskArbitration framework is different, as you noticed. DiskArbitration is about monitoring and managing disk assets. With DA, you can get called whenever a new disk is mounted or unmounted. You can also manage those disks by renaming, mounting, unmounting, or ejecting them, as well as inserting yourself in the mount/unmount process - and potentially suspending requests to do the same. But, as you pointed out, it does not provide an interface for listing existing disks. Once you do get your list of mounted volumes, DA is an excellent next stop (depending, of course, on your reason for getting that list!).
I'm trying to figure out the available disk space programmatically in windows. For this, I need to first get a list of the available drives, then check which of those are local drives and then query the available bytes on each local drive.
I'm a bit stuck on the first part, where the API presents two functions:
GetLogicalDrives (http://msdn.microsoft.com/en-us/library/aa364972(VS.85).aspx) which gives you a DWORD with the bits set (bit 0 if drive A is present, bit 1 if drive B etc)
GetLogicalDriveStrings (http://msdn.microsoft.com/en-us/library/aa364975(VS.85).aspx) which gives you the actual strings.
Now, although I'll be using strings later on, I'd prefer using the first option for querying. However, on my system a DWORD is typedef-ed to "unsigned long", which is 4 bytes, whereas drive letters only range A-Z (26 - i think - characters). Obviously, one can define more than 26 drives on their system (however unlikely they are to do so) - so I was wondering if there was any convention for those drives. Can someone point me to a resource on this?
Thanks.
DWORD is always 4 bytes, regardless of the system (it's a Win32 type).
The maximum for drive letters in Windows is 26. Because English alphabet has only 26 letters :). However, Windows allows two ways to mount a volume:
to a drive letter
to a directory (on an NTFS volume).
You can mount one volume to multiple locations (but no more than one drive letter, IIRC). A GUI for this task is presented by Control Panel -> Administrative Tools -> Computer Management -> Disk Management.
If you want to have more than 26 drives with the additional drives being redirects to already active drives and are okay with them not working properly in most programs, then you can assign more with the following method (be warned they won't even show up in the file explorer):
subst ♪: C:\Temp\
cd /D ♪:\
and to delete them (also they aren't preserved through restarts):
subst /D ♪:
You can enumerate all volumes and their mount points as described in this article.
You could use WMI. The following WMI query should list all drives:
SELECT * FROM Win32_DiskDrive
It it not sufficient to enumerate MS-DOS drives (there can be at most 26 of them, by the way, although each can be bound twice, once globally and once locally in your session), a volume can, for example, be mounted to a directory. What you want is probably to enumerate all volumes in the system, using FindFirstVolume et al. Take a look at the associated MSDN example.