Transactional log file system(/var/mqm/log) become full and i am getting MQRC 2102 resource problem with Queue Manager while attempting client connection to this queue manager. What course of action we can do to resolve this?
LogPrimaryFiles=2
LogSecondaryFiles=8
LogFilePages=16384
LogType=CIRCULAR
LogBufferPages=0
LogPath=/var/mqm/log/QMGRA/
LogWriteIntegrity=TripleWrite
Is adding additional disk space to /var/mqm/log is the only solution?
I have few queues that were full,but queue storage file system were only 60% used.
Please give me some ideas on this.
Log file pages are 4096 bytes each so a setting of LogFilePages=16384 results in log files extents of 64MB each. With a setting of LogPrimaryFiles=2 and LogSecondaryFiles=8 there can be up to 10 log files for a total of 640MB. If the file system that the circular logs resides on is less than this amount, it may fill up.
The optimum solution here is to increase the size of the log file disk allocation to something a little larger than the log file extents require. If that is not possible or you need a temporary fix then it is necessary to change the size of the log file requirement by reducing the number of extents and restarting the QMgr. Note that you can adjust the number of log extents but not the size of the extents. If it becomes necessary to change the LogFilePages=16384 parameter then it is necessary to rebuild the QMgr.
The number and size of of extents represents the total amount of data that can be under syncpoint at once but 640MB is generous in most cases. In terms of time, it also limits the longest possible duration of a unit of work on an active QMgr. This is because an outstanding transaction will be rolled back if it happens that the head pointer in the log file ever overtakes the tail pointer. For example, suppose a channel goes into retry. This holds a batch of messages under syncpoint and holds that log extent active. As applications and other channels perform their normal operations, additional transactions drive the head pointer forward. Eventually all extents will be used, and although there may be very few outstanding transactions the oldest one will be rolled back to free up that extent and advance the tail pointer forward. If the error log shows many transactions are rolled back to free log space then you really would need to allocate more space to the log file partition and bump the number of extents.
Related
I'm testing nifi SplitRecord with a small file of only 11 records
However, SplitRecord hangs for a long time. I don't get a clue what it is doing.
Processor Hung
SPlitRecord Properties:
more properties
Is Records Per Split controlling
the maximum, or the minimum, or exact number of records per split?
if the total number of records is less than records per split, what's the behavior of SplitRecords? does it wait until a time-out and then put all on-hold records in to a single split?
After about 10 minutes or random number of start/stop/terminate/restart
it may trigger the processor to split the data sooner.
Records Per Split controls the maximum, see "SplitRecord.java" for the code. If there are fewer records than the RECORDS_PER_SPLIT value, it will immediately push them all out.
However, it does look like it is creating a new FlowFile, even if the total record count is less than the RECORDS_PER_SPLIT value, meaning it's doing disk writing regardless of whether a split really occured.
I would probably investigate two things:
Host memory - how much memory does the host have? How much is configured as NiFi max heap? How much total system memory is in use/free? NiFi performs best when plenty of system memory is left for file cache.
Host's disks, specifically the disk that has the Content Repository on it. Capacity? IO? Is it shared with other services? FlowFile content is written to the Content Repository, if the disk is shared with the OS, or other busy services (or other NiFi repos) it can really slow content modification down.
Note: your NiFi version over 3 years old, please consider upgrading.
The Elasticsearch documentation mentions that a scroll query or point in time operation can put greater pressure on shard's disk, memory, or os via open file handles as older segments cannot be merged.
Is the amount of data retained due to an open search context that would otherwise be deleted proportional to the size of the segment the updated data happened to be on and not proportional to the amount of data that's updated as perceived by the client?
For example, if a client updated a 5KB document and internally the data for this document was on a 10MB segment which ends up getting merged, the entire 10MB segment would be retained when it otherwise would've been deleted. So in essence, the memory/disk impact of this context staying open is 10MB rather than 5KB. Is this correct?
If this is the case, is there any bound on how large a retained segment can be? Would a faster rate of indexing or larger document being indexed result in seeing more memory consumption? Is there anyway to do some back of the envelope calculation based on an applications access patterns to determine what kind of worst case you might expect? - or will there always be some probability that an unlucky update causes a merging operation and a large segment to be retained that can potentially cause resource exhaustion?
I am working on project that contains a fixed-size buffer of type (FIFO): First input First Output, where clients send their requests to that buffer, and the system handles them.When the buffer is full, I have to apply one of the following overloading policies (Drop Policies): DRPH : Drop one Request from the Head of buffer. DRPT: Drop one Request from Tail of buffer.DRPR: Drop 25% of elements in the buffer randomly. BLCK: block new connections until space is available in buffer.
I made a simulation to measure the performance using Httperf by sending many requests per second and measuring the response time, but I have got unstable values for response time especially when the requests number is large. so by simulation I can not get the best drop policy. I repeated the simulation many times, each time I have got different values.
The question is :
theoretically, what is the best buffer management drop policy among the mentioned policies? .
It definitely depends on your data and in which order it is needed. But usually, with a FIFO, the data at the end of the buffer is the oldest and so the one with the least likelhood to be required again. So DRPR is probably the best solution. But only if you can afford losing data (e.g. because it can be re-inserted later). If that is not the case you have to block connections until buffer space is available again.
Another thing: I would strive for a dynamic buffer. Start with a reasonable default size and see how quick it fills up. Above a certain rate increase the buffer size (and below a certain threshold you can lower it again) up to a certain maximum.
We come across a scenario where disk space was occupied for empty queues in linux environment.
Our queue manager ended unexpectedly as the file system become full and we need to empty the q file to bring back the queue manager.
But actually we dont have any messages at all in queue. This is showing a particularl queue.
Why the disk space is held here? what is the root cause?
WMQ does not shrink the queue files in real time. For example, you have 100 messages on a queue and you consume the first one. WMQ does not then shrink the file and move all the messages up by one position. If it tried to do that for each message, you'd never be able to get the throughput that you currently see in the product.
What does occur is that WMQ will shrink the queue files at certain points in the processing lifecycle. There is some latency between a queue becoming empty and the file under it shrinking it but this latency is normally so small as to be unnoticeable.
The event you are describing could in theory occur under some very specific conditions however it would be an extremely rare. In fact in the 15 years I've been working with WMQ I've only ever seen a couple of instances where the latency in shrinking a queue file was even noticeable. I would guess that what is actually going on here is that one of your assumptions or observations is faulty. For example:
Was the queue actually empty?
The queue was most definitely empty after you blew away the file. How do you know it was empty before you blew away the file?
If there were non-persistent messages on any queue, the queue will be empty after the QMgr restarts. This is another case where the queue can appear to be empty after the QMgr is restarted but was not at the time of failure.
If a message is retrieved from a queue under syncpoint, the queue depth decrements but the message is still active in the queue file. If a queue is emptied in a single transaction it retains it's full depth until the COMMIT occurs. This can make it look like the queue is empty when it is not.
Was it actually the queue file that filled up the file system?
Log extents can fill the file system, even with circular logs. For example, with a large value for secondary extents log files can expand significantly and then disappear just as quickly.
FDC files can fill up the file system, depending on how the allocations were made.
Was it even MQ?
If the QMgr shares filesystem space with other users or apps, transient files can fill up the space.
One of the issues that we see very frequently is that an application will try to put more than 5,000 messages on a queue and receive a QFULL error. The very first thing most people then do is set MAXDEPTH(999999999) to make sure this NEVER happens again. The problem with this is that QFULL is a soft error from which an application can recover but filling up the filesystem is a hard error which can bring down the entire QMgr. Setting MAXDEPTH(999999999) trades a manageable soft error for a fatal error. It is the responsibility of the MQ administrator to make sure that MAXDEPTH and MAXMSGL on the queues are set such that the underlying filesystem does not fill. In most shops additional monitoring is in place on all the filesystems to raise alerts well before they fill.
So to sum up, WMQ does a very good job of shrinking queue files in most cases. In particular, when a queue empties this is a natural point of synchronization at which the file can be shrunk and this usually occurs within seconds of the queue emptying. You have either hit a rare race condition in which the file was not shrunk fast enough or there is something else going on here that is not readily apparent in your initial analysis. In any case, manage MAXDEPTH and MAXMSGL such that no queue can fill up the filesystem and write the code to handle QFULL conditions.
If I have large dataset and do random updates then I think updates are mostly disk bounded (in case append only databases there is not about seeks but about bandwidth I think). When I update record slightly one data page must be updated, so if my disk can write 10MB/s of data and page size is 16KB then i can have max 640 random updates per second. In append only databases apout 320 per second bacause one update can take two pages - index and data. In other databases bacause of ranom seeks to update page in place can be even worse like 100 updates per second.
I assume that one page in cache has only one update before write (random updates). Going forward the same will by for random inserts around all data pages (for examle not time ordered UUID) or even worst.
I refer to the situation when dirty pages (after update) must be flushed to disk and synced (can't longer stay in cache). So updates per second count is in this situation disk bandwidth bounded? Are my calculations like 320 updates per second likely? Maybe I am missing something?
"It depends."
To be complete, there are other things to consider.
First, the only thing distinguishing a random update from an append is the head seek involved. A random update will have the head dancing all over the platter, whereas an append will ideally just track like record player. This also assumes that each disk write is the full write and completely independent of all other writes.
Of course, that's in a perfect world.
With most modern databases, each update will typically involve, at a minimum, 2 writes. One for the actual data, the other for the log.
In a typical scenario, if you update a row, the database will make the change in memory. If you commit that row, the database will acknowledge that by making a note in the log, while keeping the actual dirty page in memory. Later, when the database checkpoints it will right the dirty pages to the disk. But when it does this, it will sort the blocks and write them as sequentially as it can. Then it will write a checkpoint to the log.
During recovery when the DB crashed and could not checkpoint, the database reads the log up to the last checkpoint, "rolls it forward" and applies those changes to actual disk page, marks the final checkpoint, then makes the system available for service.
The log write is sequential, the data writes are mostly sequential.
Now, if the log is part of a normal file (typical today) then you write the log record, which appends to the disk file. The FILE SYSTEM will then (likely) append to ITS log that change you just made so that it can update it's local file system structures. Later, the file system will, also, commit its dirty pages and make it's meta data changes permanent.
So, you can see that even a simple append can invoke multiple writes to the disk.
Now consider an "append only" design like CouchDB. What Couch will do, is when you make a simple write, it does not have a log. The file is its own log. Couch DB files grow without end, and need compaction during maintenance. But when it does the write, it writes not just the data page, but any indexes affected. And when indexes are affected, then Couch will rewrite the entire BRANCH of the index change from root to leaf. So, a simple write in this case can be more expensive than you would first think.
Now, of course, you throw in all of the random reads to disrupt your random writes and it all get quite complicated quite quickly. What I've learned though is that while streaming bandwidth is an important aspect of IO operations, overall operations per second are even more important. You can have 2 disks with the same bandwidth, but the one with the slower platter and/or head speed will have fewer ops/sec, just from head travel time and platter seek time.
Ideally, your DB uses dedicated raw storage vs a file system for storage, but most do not do that today. The advantages of file systems based stores operationally typically outweigh the performance benefits.
If you're on a file system, then preallocated, sequential files are a benefit so that your "append only" isn't simply skipping around other files on the file system, thus becoming similar to random updates. Also, by using preallocated files, your updates are simply updating DB data structures during writes rather than DB AND file system data structures as the file expands.
Putting logs, indexes, and data on separate disks allow multiple drives to work simultaneously with less interference. Your log can truly be append only for example compared to fighting with the random data reads or index updates.
So, all of those things factor in to throughput on DBs.