Is there a tmpfs kind of solution where I can write into Ruby's memory which only persists until that ruby instance is complete.
File.write('/ruby_tmpfs/path/to/file', 'Some glorious content')
It get consumed in same script like this:
read_file_function_i_cannot_change_which_expects_file_path('/ruby_tmpfs/path/to/file')
Is there a tmpfs kind of solution where I can write into Ruby's memory which only persists until that ruby instance is complete.
tmpfs is a temporary file storage paradigm implemented in many Unix-like operating systems. It is intended to appear as a mounted file system, but data is stored in volatile memory instead of a persistent storage device.
https://en.wikipedia.org/wiki/Tmpfs
I've never heard of such a feature in Ruby or its stdlib.
Searching for "ruby in-memory file" revealed memfs, which I have never heard of before today, but sounds relevant.
MemFs is an in-memory filesystem .. intended for tests but you can use it for any other scenario needing in memory file system.
Using only the stdlib, mktmpdir is probably the best alternative. It will use non-volatile storage, but the OS will eventually delete it.
Related
I am writing a disk tool that needs to be able to read and possibly modify a live volume.
If the volume is in use, e.g. if it's the boot volume, one cannot simply unmount the volume to perform these operations.
Disk Utility (or the underlying diskutil tool) is able to lock a mounted volume for this purpose when it runs First Aid or when it resizes the startup volume through the "Partitions" dialog.
What API can I use in order to accomplish the same? I.e. what calls do I have to make to have the volume locked (and pending writes flushed), and, optionally, how do I best lock the UI so that the user can't accidentally use other apps that attempt to modify the volume in the meantime?
I am okay with private frameworks and other undocumented ways. This is for a repair program, after all. It won't appear in the Mac App Store ;)
It appears that fcntl (fd, F_FREEZE_FS) is the solution.
I found this call by looking into the code of the command fsck_apfs. It actually offers the "-l" option for "live fsck (lock down for verify-only)".
And now that I know what to look for, I realize that it's also documented in Amit Singh's Mac OS X Internals book.
I notice that in many disk storage systems, such as SQLite, IStream(Created on File). When they are opened exclusively, they will get better performance.
For SQLite, it is at "PRAGMA LOCKING_MODE" section in https://blog.devart.com/increasing-sqlite-performance.html
For IStream, based on document for SHCreateStreamOnFileEx at https://learn.microsoft.com/zh-cn/windows/win32/stg/stgm-constants, it said "In transacted mode, sharing of STGM_SHARE_DENY_WRITE or STGM_SHARE_EXCLUSIVE can significantly improve performance because they do not require snapshots."
Therefore, I just wonder in Windows, whether the genereal disk file will get better performance if I open it as read mode, together with share exclusively mode? In the past, when opening a file for read purpose, I only set it share mode to deny write instead of share exclusively, though there are no other processes or threads that will try to read the file at the same time.
When I boot up Linux Mint from a Live CD, I am able to save files to the "File System". But where are these files being saved to? Can't be the disc, since it's a CDR. I don't think it's stored in the RAM, because it can only hold so much data and isn't really intended to be used as a "hard drive". The only other option is the hard drive... but it's certainly not saving to any partition on the hard drive I know about, since none of them are mounted. Then where are my files being saved to??
Believe it or not, it's a ramdisk :)
All live distros mount a temporary hard disk in RAM memory. The process is completely user-transparent and is all because of the magic of Linux kernel.
The OS, in fact, first allocates an area of your RAM memory into a virtual device, then mounts it as a regular hard drive in your file system.
Once you reboot, you lose all your data from that ramdrive.
Ramdrive is needed by almost all software running on Live CDs. In fact, almost all programs, in particular desktop managers, are designed in order to write files, even temporary, during their execution.
As an example, there are two ways to run KDE on a Live CD: either modify its code deeply in order to disallow you to change wallpaper etc. (the desktop settings are stored inside ~/.kde) or redeploy it onto a writable file system such as a ramdrive in order to avoid write fails on read-only file systems.
Obviously, you can mount your real HDD or any USB drive into your virtual file system and make all writes to them permanent, but by default no live distro mounts your drives into the root file system, instead they usually mount into specific subdirectories like /mnt, /media, /windows
Hope to have been of help.
It does indeed emulate a disk using RAM; from Wikipedia:
It is able to run without permanent
installation by placing the files that
typically would be stored on a hard
drive into RAM, typically in a RAM
disk, though this does cut down on the
RAM available to applications.
RAM. In Linux, and indeed most unix systems, any kind of device is seen as a file system.
For example, to get memory info on linux you use cat /proc/meminfo, where cat is used to read files. Then, there's all sorts of strange stuff like /dev/random (to read random crap) and /dev/null (to throw away crap). ;-)
To make it persistent - use a USB device - properly formatted and with a special name. See here:
https://help.ubuntu.com/community/LiveCD/Persistence
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
How to invalidate the file system cache?
I'm writing a disk intensive win32 program. The first time it runs, it runs a lot slower while it scans the user's folders using FindFirstFile()/FindNextFile().
How can I repeat this first time performance without rebooting? Is there any way to force the system to discard everything in its disk cache?
I know that if I were reading a single file, I can disable caching by passing the FILE_FLAG_NO_BUFFERING flag to a call to CreateFile(). But it doesn't seem possible to do this when searching for files.
Have you thought about doing it on a different volume, and dismounting / remounting the volume? That will cause the vast majority of everything to be re-read from disk (though the cache down there won't care).
You need to create enough memory pressure to cause the memory manager and cache manager to discard the previously caches results. For the cache manager, you could try to open a large (I.e. Bigger than physical ram) file with caching enabled and then read it backwards (to avoid any sequential I/o optimizations). The interactions between vm and cache manager are a little more complex and much more dependent on os version.
There are also caches on the controller (possibly, but unlikely) and on the disk drive itself (likely). There are specific IoCtls to flush this cache, but in my experience, disk firmware is untested in this arena.
Check out the Clear function of CacheSet by SysInternals.
You could avoid a physical reboot by using a virtual machine.
I tried all the methods in the answers, including CacheSet, but they would not work for FindFirstFile/FindNextfile(). Here is what worked:
Scanning files over the network. When scanning a shared drive, it seems that windows does not cache the folders, so it is slow every time.
The simplest way to make any algorithm slower is to insert calls to Sleep(). This can reveal lots of problems in multi-threaded code, and that is what I was really trying to do.
I was wondering how can I make the IO faster by writing and reading temporary files to main memory. I've used to write to /dev/shm in Linux.
But now I'm in Windows 7. Anyone knows the answer ?
Thanks
If I understood it correctly (based on this post) what you are looking for is Memory Mapped Files.
You can use CreateFile() with FILE_ATTRIBUTE_TEMPORARY, and Windows should try to keep it in cache as much as possible.