Related
As explained by Dropbox, Smart Sync is a feature "that helps you save space on your hard drive. Access every file and folder in your Dropbox account from your computer, using virtually no hard drive space. ... With Smart Sync, content on your computer is available as either online-only, local, or in mixed state folders."
Last night and this morning, I moved a large quantity of files from an external disk into Dropbox folders on my MacBook (MacOS Mojave Version 10.14.4), then selected those Dropbox folders to be "online-only". The files rather quickly synched with Dropbox on the cloud -- I saw them appear in the local folders of a desktop computer that shares the dropbox -- but the grey icons (for "online only") took a long time to display in Finder. (More than twenty hours later, two larger folders still show the blue icon, for "synching", even though their contents have long appeared on the other computer.)
With growing alarm, I watched as each new directory added to Dropbox ratcheted up the amount of space used on the MacBook to dangerous levels (93%) even as large directories marked as "online only" continued to sync to the Dropbox cloud. I could only restore available space by moving some content back to an external disk.
Confusingly, information about how much space really remained was inconsistent. df showed 58 GB available:
Filesystem 1G-blocks Used Available Capacity Mounted on
/dev/disk1s1 465 403 58 88% /
while About this Mac => Storage showed 232 GB available.
According to one source, "the Storage tab in About This Mac ... can be useful as it is the only guide to what types of data are taking up storage space, but when you want to know how much space is used or free on any volume or disk, use Disk Utility: it’s much more likely to be accurate." Confusingly, however, my Disk Utility displayed both results:
433.68 GB used, 3.95 GB on other volumes, 62.45 GB free
Capacity 500.07 GB, Available: 232 GB (169.55 GB purgeable), Used: 433 GB
As explained by Dropbox, "setting files to be online only will free up space on your hard drive within minutes (as long as your computer is online and able to sync to Dropbox). However: ... macOS 10.13 (High Sierra) uses ... APFS. With APFS, the operating system takes snapshots of the file system and available hard drive space. These snapshots may not update after you've used Smart Sync to set Dropbox files as online only. This means that hard drive space you freed up with Smart Sync may not be immediately reflected or available if this snapshot hasn't updated. This hard drive space should eventually be freed up by the OS, but the amount of time this will take can vary. This isn't a behavior specific to Dropbox, but instead the designed behavior of macOS." On APFS, the placeholders for "online-only files use a small amount of space on your hard drive to store information about the file, such as its name and size. This uses less space than the full file." Indeed, files marked as "online-only" continue to show their non-zero (online) sizes (e.g., with ls and os.path.getsize()) as if they were still available locally.
I gather this is a MacOS (i.e., APFS) issue, not specific to Dropbox.
My question: If Disk Utility shows 232 GB "available" but only 62.45 GB "free", what are the consequences? Would bad things happen if I were to add another 100 GB of files to the disk?
I am of course reluctant to add more content than space free just "as an experiment" but see how this could happen unintentionally.
THIS HELPED ME: https://www.cbackup.com/articles/dropbox-taking-up-space-on-mac-6688.hmtl.html#A1
Solution 4. Clear the Dropbox cache folder
Generally, there is a hidden folder that containing Dropbox cache stored in your Dropbox root folder, named ".dropbox.cache". Only when the function of viewing hidden files and folders is enabled in the operating system, you can see the folder.
If you delete a large number of files from Dropbox, but the hard drive of your computer does not reflect these deletions, the deleted files may be saved in the cache folder. So, you can manually clear the cache to clear some space on the hard drive by following the steps below:
Open the Finder and select Go to folder... from the Go menu.
A dialog box should appear. Now copy and paste the following line into the box and press the return key:
~/Dropbox/.dropbox.cache
This will take you directly to the Dropbox cache folder. Delete the files in your cache by dragging them out of the Dropbox cache folder and into your Trash.
I have a large (100gb+) database that I'm trying to run with the official postgres image.
I can't store the data in a docker volume because the ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2 file in Docker for Mac has a size limitation of about 60gb.
I'm hesitant to mount a host directory as a volume because file access in mounted host directory volumes is much slower than regular volumes.
These are some useful links that go into more detail on these issues:
This discusses the size limitation of the Docker.qcow2 file
This also discusses the size limitation of the Docker.qcow2 file
This discusses the mounted host directory volume speed issue
This gives a nice description of how to replace the Docker.qcow2 file with a file that can grow larger
This discusses how the Docker.qcow2 file doesn't shrink as its contents are removed (this isn't directly related, but can further complicate the problem)
Do you all just eat the speed loss and mount a host directory? Do you manually create a qcow2 file that can grow larger with qemu (if you do this, do you need to maintain this file between upgrades)? Do you do something else to handle this issue?
The following script delete and re-create a new shrinked Docker.qcow2 file preserving the images passed as arguments.
https://blog.mrtrustor.net/post/clean-docker-for-mac/
Hope this helps.
For a situation like this, I would definitely recommend creating a qcow image that can grow larger.
It is a relatively straightforward process, and you get the performance benefits that are generally quite necessary when running a large database.
Docker for Mac 18.06 switches from the qcow2 file format to the raw file format, which improves speed and disk usage. The core issue still persists, in that Docker has a limited amount of space that it can use for all of its data. However, you can now set what that limit is within Preferences -> Disk -> Disk image size.
Docker for Mac version 17.12 was the first version to introduce the raw file format, but there were some bugs in the initial release that caused them to remove it as a feature temporarily. You can search that page for 'raw' to look back through the feature's history.
There's a great note here about how Docker for Mac reports its disk usage...
This will display the logical size:
ls -alh ~/Library/Containers/com.docker.docker/Data/vms/0/Docker.raw
While this will display the physical size:
du -h ~/Library/Containers/com.docker.docker/Data/vms/0/Docker.raw
When I boot up Linux Mint from a Live CD, I am able to save files to the "File System". But where are these files being saved to? Can't be the disc, since it's a CDR. I don't think it's stored in the RAM, because it can only hold so much data and isn't really intended to be used as a "hard drive". The only other option is the hard drive... but it's certainly not saving to any partition on the hard drive I know about, since none of them are mounted. Then where are my files being saved to??
Believe it or not, it's a ramdisk :)
All live distros mount a temporary hard disk in RAM memory. The process is completely user-transparent and is all because of the magic of Linux kernel.
The OS, in fact, first allocates an area of your RAM memory into a virtual device, then mounts it as a regular hard drive in your file system.
Once you reboot, you lose all your data from that ramdrive.
Ramdrive is needed by almost all software running on Live CDs. In fact, almost all programs, in particular desktop managers, are designed in order to write files, even temporary, during their execution.
As an example, there are two ways to run KDE on a Live CD: either modify its code deeply in order to disallow you to change wallpaper etc. (the desktop settings are stored inside ~/.kde) or redeploy it onto a writable file system such as a ramdrive in order to avoid write fails on read-only file systems.
Obviously, you can mount your real HDD or any USB drive into your virtual file system and make all writes to them permanent, but by default no live distro mounts your drives into the root file system, instead they usually mount into specific subdirectories like /mnt, /media, /windows
Hope to have been of help.
It does indeed emulate a disk using RAM; from Wikipedia:
It is able to run without permanent
installation by placing the files that
typically would be stored on a hard
drive into RAM, typically in a RAM
disk, though this does cut down on the
RAM available to applications.
RAM. In Linux, and indeed most unix systems, any kind of device is seen as a file system.
For example, to get memory info on linux you use cat /proc/meminfo, where cat is used to read files. Then, there's all sorts of strange stuff like /dev/random (to read random crap) and /dev/null (to throw away crap). ;-)
To make it persistent - use a USB device - properly formatted and with a special name. See here:
https://help.ubuntu.com/community/LiveCD/Persistence
I have to write a bat script for a test scenario where the software that we are testing fails to write to file due to a disk full error. The test script must be automated, so that we can run it on overnight tests, for example. The test script must also work at different computers, so installing a software like a virtual machine wouldn't be the best solution in this case.
How can I simulate that error in a Windows environment?
You could try writing to a full floppy disk.
Edit:
In light of your edited question, you could set up a network share with no disk space quota and write to that. The error will then be produced regardless of the logged on user or machine.
For Windows XP or later:
This command can get the amount of free space for the c:\ drive:
for /f "usebackq tokens=1-5" %%A in (`dir c:\ ^| find "bytes free"`) do (
set FREE_SPACE=%%C
)
Replace c:\ with your drive, as needed.
You can then take some space away from this value so you have a little room to work with:
set /a FREE_SPACE=FREE_SPACE-1024
or however much space you want to keep free.
You can use the fsutil command to create a file to fill up the free space on the disk:
fsutil file createnew c:\spacehog.dat %FREE_SPACE%
Run your test, writing to the drive. After you write 1024 bytes or so you should run out of space.
Download and install TrueCrypt. You can then create a virtual partition of whatever size you want (a couple of megabytes), mount it and then fill it with a couple of documents.
Best Option: Microsoft's consume program
Reasons:
It tests the system disk (vs a separate drive)
It's fast - run the program to fill the disk instantly, stop when no longer needed
It's easy - No creating and deleting files. No extra test partition hanging around. Installation is required, but you can use a simple command afterward.
It's scriptable
Steps:
Install the Windows Server 2003 Resource Kit Tools (Works fine on Windows 7)
cd "%ProgramFiles(x86)%\Windows Resource Kits\Tools" (or whereever it's installed)
consume.exe -disk-space
Command output:
C:\Program Files (x86)\Windows Resource Kits\Tools>consume.exe
Universal Resource Consumer - Just an innocent stress program, v 0.1.0
Copyright (c) 1998, 1999, Microsoft Corporation
consume RESOURCE [-time SECONDS]
RESOURCE can be one of the following:
-physical-memory
-page-file
-disk-space
-cpu-time
-kernel-pool
C:\Program Files (x86)\Windows Resource Kits\Tools>consume.exe -disk-space
Consume: Message: Total disk space: 96049 Mb
Consume: Message: Free disk space: 14705 Mb
Consume: Message: Free per user space: 14705 Mb
Consume: Message: Attempting to use: 14705 Mb
Consume: Message: Reattempting to use: 14705 Mb
Consume: Message: Sleeping ...
Other Options:
Windows 7 has a virtual hard drive feature. Basically do the following: Computer Management > Disk Management > Action Menu > Create VHD > Right click disk and Initialize > Right click
Generate large files (should be instant) until your disk is full using a shell command or Dummy File Generator program. Another prorgram: SpaceHog.
It might seem like a bit much, but one thing I can think of is to use a virtual machine, and set its virtual disk to just big enough to fit the OS on. Fill it with some garbage files to tip it over the edge, then run your program.
Create a secondary partition, fill it with junk and then run your program there.
You could setup a small ramdisk and write to that. See this page for some free ramdisk products.
Create a new user accout, set a quota for it, and use runas to run your app as that user. (not exactly the same as disk full, but should have similar consequences.)
The operating system will respond differently to it's system drive filling than to other drives filling and as such your application will do so too, surely? Simply filling a drive irrespective of what the physical media is used isn't going to be a accurate test.
Can't you mock the file system event for a full disk? Why would you want to wait until the disk is full? Wouldn't you want to monitor disk space periodically and warn the user when the disk is with a percentage margin of filling? Rather than wait until the disk space is terminal simply prevent your application from working until the issue is resolved, not doing so could effect any data IO and be unrecoverable!
If the test has to be a hard integration test then automating a virtual machine, deploying the application and then fill the remaining space with a recursive script is feasible.
Best thing that works on every computer (as testing is not neceessarily done on a dedicated machine) would be a ramdrive/ramdisk that could be set up on the fly.
Only found a Virtual Disk SDK so far see here that maybe could be included in your buzild process.
Different idea: maybe your testing computers could be set up to write to a shared network folder (that is full) mount as a drive?
I have made a modification to the above script to make it compatiable with Windows 7... Essentially adding the switch "/-c" to the for statement. This removes the thousands seperator as fsutil does not like it in the statement.
for /f "usebackq tokens=1-5" %%A in (`dir /-c d:\ ^| find "bytes free"`) do (set FREE_SPACE=%%C)
fsutil file createnew d:\largefile.txt %FREE_SPACE%
use a very small iscsi target
I'm need to find a method to programmatically determine which disk drive Windows is using to boot. In other words, I need a way from Windows to determine which drive the BIOS is using to boot the whole system.
Does Windows expose an interface to discover this? With how big the Windows API is, I'm hoping there is something buried in there that might do the trick.
Terry
p.s. Just reading the first sectors of the hard disk isn't reveling anything. On my dev box I have two hard disks, and when I look at the contents of the first couple of sectors on either of the hard disks I have a standard boiler plate MBR.
Edit to clarify a few things.
The way I want to identify the device is with a string which will identify a physical disk drive (as opposed to a logical disk drive). Physical disk drives are of the form "\\.\PHYSICALDRIVEx" where x is a number. On the other hand, a logical drive is identified by a string of the form, "\\.\x" where x is a drive letter.
Edit to discuss a few of the ideas that were thrown out.
Knowing which logical volume Windows used to boot doesn't help me here. Here is the reason. Assume that C: is using a mirrored RAID setup. Now, that means we have at least two physical drives. Now, I get the mapping from Logical Drive to Physical Drive and I discover that there are two physical drives used by that volume. Which one did Windows use to boot? Of course, this is assuming that the physical drive Windows used to boot is the same physical drive that contains the MBR.
Go into Control Panel
System and Security
Administrative Tools
Launch the System Configuration tool
If you have multiple copies of Windows installed, the one you are booted with will be named such as:
Windows 7 (F:\Windows)
Windows 7 (C:\Windows) : Current OS, Default OS
Unless C: is not the drive that windows booted from.Parse the %SystemRoot% variable, it contains the location of the windows folder (i.e. c:\windows).
You can use WMI to figure this out. The Win32_BootConfiguration class will tell you both the logical drive and the physical device from which Windows boots. Specifically, the Caption property will tell you which device you're booting from.
For example, in powershell, just type gwmi Win32_BootConfiguration to get your answer.
That depends on your definition of which disk drive Windows used to boot. I can think of 3 different answers on a standard BIOS system (who knows what an EFI system does):
The drive that contains the active MBR
The active partition, with NTLDR (the system partition)
The partition with Windows on it (the boot partition)
2 and 3 should be easy to find - I'm not so sure about 1. Though you can raw disk read to find an MBR, that doesn't mean it's the BIOS boot device this time or even next time (you could have multiple disks with MBRs).
You really can't even be sure that the PC was started from a hard drive - it's perfectly possible to boot Windows from a floppy. In that case, both 1 and 2 would technically be a floppy disk, though 3 would remain C:\Windows.
You might need to be a bit more specific in your requirements or goals.
You type diskpart, list disk and check disks for boot.
Ex:
dispart
list disk
select disk 0
detail disk
The disk with Boot volume is disk with windows installed:
There is no boot.ini on a machine with just Vista installed.
How do you want to identify the drive/partition: by the windows drive letter it is mapped to (eg. c:\, d:) or by how its hardware signature (which bus, etc).
For the simple case check out GetSystemDirectory
Try HKEY_LOCAL_MACHINE\SYSTEM\Setup\SystemPartition
You can try use simple command line. bcdedit is what you need, just run cmd as administrator and type bcdedit or bcdedit \v, this doesn't work on XP, but hope it is not an issue.
Anyway for XP you can take a look into boot.ini file.
a simpler way
search downloads in the start menu and click on downloads in the search results to see where it will take you the drive will be highlighted in the explorer.
On Windows 10.
Open "Computer Management"
Look for "Storage" in list "left top side of page"
select "Disk Management"
On section of page showing the list of disks and the partitions find the disk that has the partition assigned as drive C:
On that disk containing C: partition
Use the right mouse button to select the Square section containing The Disk Number, Type of drive and size in GB . When menu opens select the Properties.
A window will open showing what drive hardware was used.