Filestore has a limit of 128 File Locks.
Are There Ways to monitor how many file lock? What happens if you reach this limit?
I checked on stack driver, but all metric haven't any link with file locks
Thank you very much for any suggestions.
Giorgio
(https://cloud.google.com/filestore/docs/limits)
A Cloud Filestore instance can have up to 128 file locks across all clients of the instance.
You could use the lslocks from the util-linux package as mentioned on this other forum, the limit would not be shown on Stackdriver as it would need to monitor the files on your Filestore instance which Stackdriver doesn't do for privacy reasons.
If you reach the limit, it would mean that you would not be able to create another lock on the file (you would get an error when the 129th user or process tries to open it). You could ask for an increase to this limit to the GCP Support team, however, there is no guarantee that they will increase the limit.
Hope you find this helpful!
You can not monitor the lock file in Stackdriver.
These are the metrics available for monitoring Filestore instance:
Bytes written: Number of bytes written.
Bytes read: Number of bytes read.
Disk read operation count: Number of disk read operations. If the
Filestore instance caches the data, some read operations will not be
reflected as disk reads.
Disk write operation count: Number of disk write operations. If the
Filestore instance caches the data, some write operations will not be
reflected as disk writes.
Free disk bytes: Number of free disk bytes.
Free disk bytes percent: Percentage of free disk bytes.
Procedure call count: Returns the same information as the nfsstat -s
command.
Time (in milliseconds) spent on disk read operations: Time spent on
disk reads.
Time (in milliseconds) spent on disk write operations: Time spent on
disk writes.
Used disk bytes: Number of used disk bytes.
Used disk bytes percent: Percentage of used disk bytes.
The Filestore Engineer Team is considering including lock file as a new metric, but there is no ETA for it.
You can only open a ticket to Google Cloud Platform Support and ask for this metric link.
In case you reach the limit, the time to access the data will increase.
Please keep in mind that you can request additional quota using the Cloud Console. However limits cannot generally be increased unless specifically noted.
Related
My question is about disks in Google Cloud.
I can’t understand why do we need to increase the disk capacity in order to increase performance (transfer/read/write). If the disks are not local, then data is transferred over the network between the VM and the disk, that I understand.
Can someone explain in simple clear words, why do we need to increase the disk capacity from 500GB to 1TB? How does this affect the transfer / read / write speed?
If it is not difficult, could you exmplain some simple example?
Thanks you very much.
This is how GCP is designed.
You can have a look at how the IOPS changes with capacity increase and machine type (like N1, N2, number of CPU's).
Example:
For example, consider a 1,000 GB SSD persistent disk attached to an
instance with an N2 machine type and 4 vCPUs. The read limit based
solely on the size of the disk is 30,000 IOPS. However, because the
instance has 4 vCPUs, the read limit is restricted to 15,000 IOPS.
Also have in mind that:
Performance also depends on the number of vCPUs on your VM instance due to network egress caps on write throughput.
Example:
In a situation where persistent disk is competing with IP traffic for
network egress bandwidth, 60% of the maximum write bandwidth goes to
persistent disk traffic, leaving 40% for IP traffic. Click below to
see an example of how to calculate the maximum persistent disk write
traffic that a VM instance can issue.
To optimize your disk performance you can do the following:
- change disk size (thus increasing IOPS)
- change machine type (to the one with higher network cap limit)
Here you can read how VM type affects GCP network caps.
Is it possible to track the total amount of bytes read and written to the file pagefile.sys? The idea is to graph the rate of paging activity over time and identify memory bottleneck situations.
I've looked at FileSystemWatcher but it seems to notify only of changed events, but not bytes read/written.
I think there must be an API that allows it, since Resource Monitor can do it (see first image).
EDIT
As state in the link above, "The performance counters \Memory\Page/sec, \Memory\Page Reads/sec, and \Memory\Page Inputs/sec measure hard page faults (faults that must be resolved by disk) which may or may not be related to a page file or a low physical memory condition. Hard page faults are a normal function of the operating system and happen when reading the portions of image files (DLLs and EXEs) as they are needed, when reading memory mapped files, or when reading from a page file".
Therefore, performance counters are not enough for this task. I need to detect real page file activity and not "normal" hard faults.
Yes there is a way to query the information you are looking for. It is called performance counter in windows and there is a rich set of APIs for both consumer and producer of performance counters. Since you want to consume the performance counter, you should be looking at the consumer APIs. The following links will help you in getting the information that you need.
Performance Counters
About Performance Coutners
Consuming Counter Data
I have an application which needs to create many small files in maximum performance (less than 1% of them may be read later), and I want to avoid using asynchronous file API to keep the simplicity of my code. The size of total files written cannot be pre-determined, so I figured that to achieve maximum performance, I would need:
1.Windows to utilize all unused RAM for cache (especially for file writes), with no regard of relibility or other issues. If I have more than 1GB of unused RAM and I create one million of 1KB files, I expect Windows to report "DONE" immediately, even if it has written nothing to disk yet.
OR
2.A memory-based file system backed by real on-disk file system. Whenever I write files, it should first write everything in memory only, and then update on-disk file system in background. No delay in synchronous calls unless there isn't enough free memory. Note it's different from tmpfs or RAM disk implementations on Windows, since they require fixed amount of memory and utilize page file when there isn't enough RAM.
For option 1, I have tested VHD files - while it does offer as high as 50% increase of total performance under some configurations, it still does flushing-to-disk and cause writing to wait unnecessarily. And it appears there is no way I can disable journaling or further tweak the write caching behavior.....
For option 2, I have yet found anything similar..... Do such things exist?
There is a situation that whenever a write occurs, fsync is used with that write. Then, how to minimise the disk access? How the kernel does this?
fsync() transfers ("flushes") all modified in-core data of (i.e., modified buffer cache pages for) the file referred to by the file descriptor fd to the disk device (or other permanent storage device).
I think Kernel can transfer the data of all the modified buffers to the hard disk periodically after some time. So that it can minimise the disk access.
Please give some suggestions/hints.
In general, try to avoid overthinking it. Don't call fsync, just let the kernel decide when to do the physical write.
Here are kernel options for ext4, which you can use to tune the kernel's behavior to your needs - but this would be a server tuning exercise rather than something you could implement from your application:
http://kernel.org/doc/Documentation/filesystems/ext4.txt
This might be an interesting one:
"
max_batch_time=usec Maximum amount of time ext4 should wait for
additional filesystem operations to be batch
together with a synchronous write operation.
"
I am looking to optimize my disk IO, and am looking around to try to find out what the disk cache size is. system_profiler is not telling me, where else can I look?
edit: my program is processing entire volumes: I'm doing a secure-wipe, so I loop through all of the blocks on the volume, reading, randomizing the data, writing... if I read/write 4k blocks per IO operation the entire job is significantly faster than r/w a single block per operation. so my question stems from my search to find the ideal size of a r/w operation (ideal in terms of performance:speed). please do not point out that for a wipe-program I don't need the read operation, just assume that I do. thx.
Mac OS X uses a Unified Buffer Cache. What that means is that in the kernel VM objects and files are them at some level, same thing, and the size of the available memory for caching is entirely dependent on the VM pressure in the rest of the system. It also means the read and write caching is unified, if an item in the read cache is written to it just gets marked dirty and then will be written to disk when changes are committed.
So the disk cache may be very small or gigabytes large, and dynamically changes as the system is used. Because of this trying to determine the cache size and optimize based on it is a losing fight. You are much better off looking at doing things that inform the cache how to operate better, like checking with the underlying device's optimal IO size is, or identifying data that should not be cached and using F_NOCACHE.