Is the Anaconda installer available as a zip file? The links usually used have sizes similar to or +/- 10 MB as compared to the originals. I have a hard daily usage limit and a single download will exhaust it.
Related
I'm trying to extract the .tar imagenet archive on Windows. This has 150 gb of files, and is estimated to take 95 hours to extract using an SSD and the 7zip file manager. Is there a reasonable way to speed this up, such as by using a Python library or command line options instead?
If you are using an external SSD then it is slowing you down. Copy the tar to a hard drive (or internal SSD) and try again. I bet it goes 10x faster.
My operating system is Windows 10 and I have a problem with the free space dropping for no reason.
A couple of days ago I ran a python code in jupyter notebook, and in the middle of the execution my C drive ran out of space (there was ~50 GB free space), and since then the C drive free space changes significantly (even shrinks to few MBs) without no obvious reason.
Since then I found some huge files in a pycharm temporary directory, and I freed 47GB of space, but after a short time, it runs out of space again ( I am not even running any code anymore)!
When I restart, the free space gradually starts to increase, and again after a some time, it shrinks to a few GB or even MBs.
PS. I installed WinDirStat to show me the stat of the disk space, and it shows 93 GB under this path: C:\ProgramData\Microsoft\Search\Data\Applications\Windows\Files\Windows.edb, but I can't open Data folder in the file explorer, and it shows 0 bytes when I open the folder properties.
Windows.edb is an index database of the Windows Search function. It provides data to speed up searching in the file system due to indexing of files. There are several guides in the internet about reducing it's size. The radical way would be deleting it but I do not recomment this. You had to turn Windows Search off to do so:
net stop "Windows Search"
del %PROGRAMDATA%\Microsoft\Search\Data\Applications\Windows\Windows.edb
net start "Windows Search"
You wrote in your question that the file suddenly grew while your program was running. Maybe files will be created there. These files should be set to not be indexed. You should do that for the folder where the files are created. If this all fails, you could finally turn indexing off which slows down Windows Search.
I do a lot of C++ development using Visual Studio Code on Windows 10. My development computer has limited hard disk capacity and Low disk space warnings are frequent.
After analysis with the excellent WinDirStat utility, %APPDATA%\Code\User\workspaceStorage was found to contain hundreds of MBs. There are dozens of random 32-character folders (eg. "9731bc4ee103e04b5a91aff76967e74b"), and each of these containing .\ms-vscode.cpptools\.BROWSE.VC.DB files which are typically >30 MB.
Can these files, or better the entire %APPDATA%\Code\User\workspaceStorage folder, be safely deleted with a batch script on Windows startup?
You can change the storage path of the database files. After that, I think the database files can be deleted safely.
Add this to the global settings.json file:
"C_Cpp.default.browse.databaseFilename": "${workspaceFolder}/.vscode/.BROWSE.VC.DB"
See more details here
BTW, you can also set "C_Cpp.intelliSenseCachePath" to change the IntelliSense cache path.
Learn more from this issue and official docs
As explained by Dropbox, Smart Sync is a feature "that helps you save space on your hard drive. Access every file and folder in your Dropbox account from your computer, using virtually no hard drive space. ... With Smart Sync, content on your computer is available as either online-only, local, or in mixed state folders."
Last night and this morning, I moved a large quantity of files from an external disk into Dropbox folders on my MacBook (MacOS Mojave Version 10.14.4), then selected those Dropbox folders to be "online-only". The files rather quickly synched with Dropbox on the cloud -- I saw them appear in the local folders of a desktop computer that shares the dropbox -- but the grey icons (for "online only") took a long time to display in Finder. (More than twenty hours later, two larger folders still show the blue icon, for "synching", even though their contents have long appeared on the other computer.)
With growing alarm, I watched as each new directory added to Dropbox ratcheted up the amount of space used on the MacBook to dangerous levels (93%) even as large directories marked as "online only" continued to sync to the Dropbox cloud. I could only restore available space by moving some content back to an external disk.
Confusingly, information about how much space really remained was inconsistent. df showed 58 GB available:
Filesystem 1G-blocks Used Available Capacity Mounted on
/dev/disk1s1 465 403 58 88% /
while About this Mac => Storage showed 232 GB available.
According to one source, "the Storage tab in About This Mac ... can be useful as it is the only guide to what types of data are taking up storage space, but when you want to know how much space is used or free on any volume or disk, use Disk Utility: it’s much more likely to be accurate." Confusingly, however, my Disk Utility displayed both results:
433.68 GB used, 3.95 GB on other volumes, 62.45 GB free
Capacity 500.07 GB, Available: 232 GB (169.55 GB purgeable), Used: 433 GB
As explained by Dropbox, "setting files to be online only will free up space on your hard drive within minutes (as long as your computer is online and able to sync to Dropbox). However: ... macOS 10.13 (High Sierra) uses ... APFS. With APFS, the operating system takes snapshots of the file system and available hard drive space. These snapshots may not update after you've used Smart Sync to set Dropbox files as online only. This means that hard drive space you freed up with Smart Sync may not be immediately reflected or available if this snapshot hasn't updated. This hard drive space should eventually be freed up by the OS, but the amount of time this will take can vary. This isn't a behavior specific to Dropbox, but instead the designed behavior of macOS." On APFS, the placeholders for "online-only files use a small amount of space on your hard drive to store information about the file, such as its name and size. This uses less space than the full file." Indeed, files marked as "online-only" continue to show their non-zero (online) sizes (e.g., with ls and os.path.getsize()) as if they were still available locally.
I gather this is a MacOS (i.e., APFS) issue, not specific to Dropbox.
My question: If Disk Utility shows 232 GB "available" but only 62.45 GB "free", what are the consequences? Would bad things happen if I were to add another 100 GB of files to the disk?
I am of course reluctant to add more content than space free just "as an experiment" but see how this could happen unintentionally.
THIS HELPED ME: https://www.cbackup.com/articles/dropbox-taking-up-space-on-mac-6688.hmtl.html#A1
Solution 4. Clear the Dropbox cache folder
Generally, there is a hidden folder that containing Dropbox cache stored in your Dropbox root folder, named ".dropbox.cache". Only when the function of viewing hidden files and folders is enabled in the operating system, you can see the folder.
If you delete a large number of files from Dropbox, but the hard drive of your computer does not reflect these deletions, the deleted files may be saved in the cache folder. So, you can manually clear the cache to clear some space on the hard drive by following the steps below:
Open the Finder and select Go to folder... from the Go menu.
A dialog box should appear. Now copy and paste the following line into the box and press the return key:
~/Dropbox/.dropbox.cache
This will take you directly to the Dropbox cache folder. Delete the files in your cache by dragging them out of the Dropbox cache folder and into your Trash.
I have created a dataset of millions (>15M, so far) of images for a machine-learning project, taking up over 500GB of storage. I created them on my Macbook Pro but want to get them to our DGX1 (GPU cluster) somehow. I thought it would be faster to copy to a fast external SSD (2x nvme in raid0) and then plug that drive directly into local terminal and copy it to the network scratch disk. I'm not so sure anymore, as I've been cp-ing to the external drive for over 24 hrs now.
I tried using the finder gui to copy at first (bad idea!). For a smaller dataset (2M images), I used 7zip to create a few archives. I'm now using the terminal in MacOS to copy the files using cp.
I tried "cp /path/to/dataset /path/to/external-ssd"
Finder was definitely not the best approach as it took forever at the "preparing" to copy stage.
Using 7zip to archive the dataset increased the "file" transfer speed, but it took over 4 days(!) to extract the files, and that for a dataset an order of magnitude smaller.
Using the command line cp, started off quickly but seems to have slowed down. Activity monitor says I'm getting 6-8k IO's on the disk. It's been 24 hours and it isn't quite halfway done.
Is there a better way to do this?
rsync is the preferred tool for this kind of workload. It is used for both local and network copies.
Main benefits are (excerpt from manpage):
delta-transfer algorithm, which reduces the amount of data sent
if it is interrupted for any reason, then you can restart it easily with very little cost. It can even restart part way through a large file
options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied.
Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.
Regarding command usage and syntax, for local transfers is almost the same as cp:
rsync -az /path/to/dataset /path/to/external-ssd