I am working with a SuSE machine (cat /etc/issue: SUSE Linux Enterprise Server 11 SP1 (i586)) running Postgresql 8.1.3 and the Slony-I replication system (slon version 1.1.5). We have a working replication setup going between two databases on this server, which is generating log shipping files to be sent to the remote machines we are tasked to maintain. As of this morning, we ran into a problem with this.
For a while now, we've had strange memory problems on this machine - the oom-killer seems to be striking even when there is plenty of free memory left. That has set the stage for our current issue to occur - we ran a massive update on our system last night, while replication was turned off. Now, as things currently stand, we cannot replicate the changes out - slony is attempting to compile all the changes into a single massive log file, and after about half an hour or so of running, it trips over the oom-killer issue, which appears to restart the replication package. Since it is constantly trying to rebuild that same package, it never gets anywhere.
My first question is this: Is there a way to cap the size of Slony log shipping files, so that it writes out no more than 'X' bytes (or K, or Meg, etc.) and after going over that size, closes the current log shipping file and starts a new one? We've been able to hit about four megs in size before the oom-killer hits with fair regularity, so if I could cap it there, I could at least start generating the smaller files and hopefully eventually get through this.
My second question, I guess, is this: Does anyone have a better solution for this issue than the one I'm asking about? It's quite possible I'm getting tunnel vision looking at the problem, and all I really need is -a- solution, not necessarily -my- solution.
Related
I've been having issues with my parity instance freezing up while syncing. Given enough time it'll generally pick up again but sometimes it will stay where it is for hours. Restarting the instance temporarily solves the problem and it will immediately start syncing like normal.
For example, it might sync to the latest block at 15:00:00 and then not pick up anything until 15:30:00 (or a few hours later) and then start syncing until it catches back up to the top.
Peers are generally 20+ and absolutely nothing new will show up in the console until parity decides to resume (or I restart)
Running off of a windows server instance. Off of an HDD (Not ideal I know but except for these pauses it has been sufficient and the fact that restarting parity fixes the issue makes me believe it is unrelated to the hard drive performance)
Running parity 1.10.8 with the cmd:
parity.exe --jsonrpc-apis all --cache-size 1024 --db-compaction hdd --tracing off --pruning fast
I have a large number of wallets that I query regularly with the RPC APIs to check for changes.
Did anyone encounter anything like this?
That is a known bug and fixed a while ago, if you keep experiencing this, please ugprade to the latest Parity Ethereum 2.x releases!
https://github.com/paritytech/parity-ethereum/releases/latest
Note: I work for Parity Technologies.
I'm running PALallax, which is a custom version of Kibana / ElasticSearch for Palo Alto firewalls. I have it installed on CentOS 7 with more than enough resources (4 processors, 16GB of RAM). It works fine - however, almost every single day, half way through, Kibana will stop showing results and end up with the dreaded "no results found". I know it works, though. The log file continues to grow (which is big, by the way - about 11GB half way through the day). No matter what I do, I can't get any information to display until I delete the log and indices files on the server and reboot - then it starts working again.
I've looked through logs all around the system and can't figure out what is going on. I'm not an Linux expert, so unfortunately I've run out of ideas and have nothing else to try. I've spent countless days googling different things and haven't been able to isolate any specific problem in the logs.
Any suggestions on where to look? Are my logs too big? I can see that I'm not running out of RAM while this is happening. I always have it set for 'last hour' worth of data, set to auto-refresh every 5 minutes.
Monitor the free disk space and set up automatic deletion of old indices to avoid running out of disk space.
I am running Cloudera Hadoop on my laptop and Oracle VirtualBox VM.
I have given 5.6 GB out of mine 8 and six from eight cores as well.
And still I am not able to keep it up and running.
Even without load services would not stay up and running and when I try a query at least Hive will be down within 20 minutes. And sometimes they go down like dominoes: one after another.
More memory seemed to help some: with 3GB and all services, Hue was blinking with red colors when the Hue itself managed to get up. And after rebooting it would takes 30 - 60 minutes before I manage to get the system up enough to even try running anything on it.
There has been two sensible notes (that I have managed to find):
- Warning of swapping.
- Crashing note when the system used 26 GB of virtual memory which was not enough.
My dataset is less than one megabyte, so it is hard to understand why the system would go up to dozens of gigabytes, but for whatever was reason for that has passed: now the system is running more steadily around the 5.6 GB that I have given to it after closing down a few services: see my answer to myself.
And still it is just more stable. Right after I got a warning of swapping and the Hive went down again. What could be reason for more-or-less all Hadoop services going down if the VM starts to swap?
I don't have enough reputation to post the picture to here, but when Hive went down again it was swapping 13 pages / second and utilizing 5.9 GB / 5.6 GB. So basically my system starts crashing more-or-less right after it start to swap. "428 pages were swapped to disk in the previous 15 minute(s)"
I have used default installation options as far as hard drive is concerned.
Only addition is a shared folder between Windows and VM. That works somewhat strangely locking files all the time, so I used it just like FTP and only for passing files from one system to another. Thus I can go days without using it, but systems still crash, so that is not the cause either.
Now that the system is mostly up, services crash still about twice a day: Service Monitor and Hive are quite even with their crashing frequency. After those come Activity Monitor and Event Server, which appear to crash always together. I believe Yarn crashes as well, but it gets up on its own. Last time Hive crashed first, and then it got followed by Service Monitor, Hive (second time), Activity Monitor and Event Server all.
As swap is disk, perhaps the problem is with disk:
# cat /etc/fstab
# swapoff -a
# badblocks -v /dev/VolGroup/lv_swap
Checking blocks 0 to 8388607
Checking for bad blocks (read-only test): done
Pass completed, 0 bad blocks found.
# badblocks -vw /dev/VolGroup/lv_swap
Checking for bad blocks in read-write mode
From block 0 to 8388607
Testing with pattern 0xaa: done
Reading and comparing: done
Testing with pattern 0x55: done
Reading and comparing: done
Testing with pattern 0xff: done
Reading and comparing: done
Testing with pattern 0x00: done
Reading and comparing: done
Pass completed, 0 bad blocks found.
So nothing wrong with swap disk and I have not noticed any disk error anywhere else either.
Note that you could check file system from Windows side also. But I expect that if you make Windows to fix your Linux file system, you have good chances of destroying your Linux with that, so I did my checks somewhat pessimistically, because AFAIK these commands are safe to execute.
About half of the services kept going down, so giving more specifics would be a long story.
I succeeded to get the system more stable by closing down flume, hbase, impala, ks_indexer, oozie, spark and sqoop. And by increasing more memory to some remaining services that complained they had not been given enough memory.
Also I fixed couple of thing on the Windows side, I am not sure which one of these helped:
- MsMpEng.exe kept my hard drive busy. I didn't have permissions to kill it, but I decreased its priority to lowest possible.
- CcmExec.exe got to loop on my DVD and kept reading it for forever. This I solved by taking the DVD out from the drive. Then later on I killed the process tree to keep it from bothering for a while.
I found these using Windows resource manager.
The VM requires 4GB: http://www.cloudera.com/content/cloudera-content/cloudera-docs/DemoVMs/Cloudera-QuickStart-VM/cloudera_quickstart_vm.html You should use that.
I am not clear whether you are using the QuickStart VM though. It's set up to run just the essential services and tuned to conserve memory rather than exploit lots of memory.
It sounds like you are running your own installation, on one virtual machine, on your Windows machine. You may be running an entire cluster's worth of services on one desktop machine. Each of these services has master, worker processes, monitoring processes, etc. You don't need most of them.
You also probably have left memory settings at default suitable for a server-class machine of 16+ GB RAM. Remember these services usually run across many machines, not all on one.
Finally, you're clearly swapping, and that makes things incredibly slow. Remember this is all through a VM too!
Bottom line, use the QuickStart VM if you really want a 1-machine cluster tuned correctly. If you want a real cluster or more services, you need more hardware.
Also consider: cloudera.com/live contains a full CDH 5.1 cluster + sample data, running on demand on AWS. Of course, the advantage of the VM is that you can BYOD, but if you're simply looking for a hands-on Hadoop experience, Live is a great option.
How do I improve Subversion client update performance? It appears to be disk bound on the client.
Details:
CollabNet Windows client version 1.6.2 (r37639)
Windows XP SP2
3 GB RAM with PF Usage around 1 GB and System Cache of 1.1 GB.
Disk has write caching enabled
Update takes 7-15 minutes (when very little to update).
Checkout has 36,083 directories/files (from svn list)
Repository has 58,750 revisions.
Checkout takes about 2.7 GB
Perf monitor shows % Disk Write time stays near 90% during update.
Max Disk Read Bytes/sec got up to 12.8M and write got up to 5.2M
CPU, paging file usage, and network usage are all low.
Watching the server performance seems to show that it isn't a bottleneck.
I'm especially interested in answers besides getting a faster disk (especially configuration changes).
Updates from some of the suggestions:
I need the whole thing so sparse directories won't work.
Another client (TortoiseSVN) takes 7 minutes also
TortoiseSVN icon overlays have be configured so they don't cause the problem.
Anti-virus is configured to to skip that directory is it isn't causing the problem.
I experience exatly the same thing. Recently replaced Perforce with svn, but if we cannot overcome the performance problems on Windows me must consider another tool.
Using svn 1.6.6, Win XP and Vista clients. RedHat server.
My observations matches yours:
Huge disk-write activity.
Antivirus not a bottleneck.
No matter witch svn-clients are used.
No server or network bottleneck.
Complementary info
More than 3 times faster operations on:
Linux (Ubuntu).
Linux (Ubuntu) running on VirtualBox at Win Vista host.
Win XP running on VMWare at RedHat host.
Do you need every bit of the repository on your working copy? If you truly only care about particular portions of the tree, look into Subversion's Sparse Directories (a.k.a. "Sparse Checkouts") feature. It allows you to manipulate your working copy so it only contains those directories of interest.
Just as an example, you might use this to prune documentation, installer-related files, etc. Depending on what you truly need on your local machine, embracing this approach could make a serious dent in your wait times.
Try svn client version 1.5.. It helped me on my Vista laptop. Versions 1.6. are extremely slow.
This is more likely to be your network and the amount of data moved as well as your client. Are you using Tortoise? I find it to be a bit slow myself when moving that much data!
Are you using TortoiseSVN? If so, the Icon Overlays do slow down operations. If you go to TortoiseSVN Settings/Icon Overlays there are several settings you can tweak to control the level to which you want to use the Overlays, including turning them off completely. See if that affects your performance.
Do you run a virus checker that uses on-access scanning? That can really make it crawl. If so, turn it off and see if that helps. Most scanners will have a way to exclude specific directories if that helps.
Nobody seems to be pointing out the one reason that I often consider a design flaw. Subversion creates a second "pristine" copy of the checkout for offline operations. If you're checking out 4G of files, it's actually writing 8G to disk.
Compare a checkout to an export. That will show you the massive difference when writing those second copies.
There's nothing you can do about that.
Upgrade to svn 1.7
From Discussion of Slow Performance of SVN Update:
The update process in svn 1.6 goes something like this:
search the entire working copy, to see what's there at the moment, and locking it so no one changes the answer during the next steps
tell that to the server
receive from the server whatever new stuff you need, applying the changes to the files as you go
recurse over the entire working copy again, unlocking it
If there are many directories and files, steps 1 and 4 can take up a
lot of time. This would be consistent with your observation of long
delays with no network traffic.
Working copy format was changed in svn 1.7. Now all meta information is stored in SQLite database in root folder of working copy and there is no need to perform steps 1 and 4 any more which consumed most of the time durring svn update.
I'm experiencing painfully slow operations with one of our SVN repositories/projects.
For example, it's taking 5-10 minutes to revert the changes in one small file (10 KB). Or about 40-60 minutes to check out the project of 100 MB.
There are about 30 other projects on the same server, some vastly bigger than this one, and none of them preform like this.
One thing to note is that this project is a Magento project. It's not very large in terms of disk space, but I have 23k Files and 11k folders, and I have read SVN preforms badly when there are lots of little files; is this true? And is there anything I can do to speed things up?
The Subversion working copy performs quite badly when there's a huge number of directories, like in your case. For write operations (even only locally) to the working copy, the working copy has to be locked, which means that a lock file is created in every directory (that's 11k file creates), then the action executes, and the those 11k files are deleted again.
Subversion 1.7 is moving to a different working copy format, that should resolve these problems. Until then there's a few tricks you might try to speed things up, like excluding the working copy from your virus scanner, disabling file monitors on the directory (like TortoiseSvnCache), and trying to reduce the total number of directories. (Perhaps by checking out a few separate working copies)
There is a known issue with the use of the recycle bin with revert which causes slow reverting. Emptying your recycle bin and setting TortoiseSVN not to use it during revert operations both speed up this operation (see http://www.nabble.com/Revert-is-too-slow-td18222196.html).
This has definitely sped up my revert operations.
I experienced extreme slowness with Subversion on Windows after changing my password. I had to delete all directories and files from %APPDATA%\Subversion\auth.
Now SVN is fast as a hare. My slowness occurred via both TortoiseSVN and the command line.
SVN is slow if you use NFS (Network File System) for the working copy. This could be your problem.
We have face similar issue, the problem was TortoiseSvn (Version 1.9.7). For example, the repo browser took about 10 minutes to initial.
We have turned of the Show Locks feature and every thing fixed!
Right click on a folder and select Tortoise\Settings then General\Dialog 3 then deselect Show Locks
Also some good hints can be found at http://tigris-scm.10930.n7.nabble.com/Workaround-for-slow-RepositoryBrowser-on-large-repositories-td92324.html
Reverting changes in SVN is a local operation which shouldn't go to the server at all. So it sounds as though the problem is in your working copy of the project.
Try running 'svn cleanup' in the working copy; you may also want to check if you have problems with the hard drive or filesystem.
Our SVN was running painfully slow through TortoiseSVN, Eclipse and command line. Commits and exports were slow. Our Zend Framework-based PHP projects would take an age to update and popping in a small commit of about three files would take 5-10 minutes.
Our SVN virtual machine (CentOS) only had 700 MB of RAM which seemed reasonable for a Linux CLI only running Subversion via Apache and has been running fine for about one year. We've only got about 20 projects and only three developers.
I've upped it to 1.5 GB of RAM and things are running much faster now, back to our old speeds.
I also suffered a large slowdown after upgrading to TortoiseSVN 1.7.3.
Then I discovered I had a separate install of SVN 1.6.5. I uninstalled both and reinstalled TortoiseSVN and now things are much better. First update of the day in TortoiseSVN is still slow (1-2 minutes), but fast after that.
I have some projects which use the Eclipse IDE. If you capture the Eclipse project directories you get hundreds and hundreds of tiny files which has the same effect for my project as you're suffering on yours.
I think that when you check files out SVN does so one at a time which means that projects with huge numbers of files are always going to be slow and there's not much you can do about it (aside from avoiding frequent whole-repository operations).
Making changes to a single file shouldn't be slow though.
You may try the suggestions in another post on Stack Overflow about slow SVN. It could also be due to using a BDB database.