I'm running PALallax, which is a custom version of Kibana / ElasticSearch for Palo Alto firewalls. I have it installed on CentOS 7 with more than enough resources (4 processors, 16GB of RAM). It works fine - however, almost every single day, half way through, Kibana will stop showing results and end up with the dreaded "no results found". I know it works, though. The log file continues to grow (which is big, by the way - about 11GB half way through the day). No matter what I do, I can't get any information to display until I delete the log and indices files on the server and reboot - then it starts working again.
I've looked through logs all around the system and can't figure out what is going on. I'm not an Linux expert, so unfortunately I've run out of ideas and have nothing else to try. I've spent countless days googling different things and haven't been able to isolate any specific problem in the logs.
Any suggestions on where to look? Are my logs too big? I can see that I'm not running out of RAM while this is happening. I always have it set for 'last hour' worth of data, set to auto-refresh every 5 minutes.
Monitor the free disk space and set up automatic deletion of old indices to avoid running out of disk space.
Related
I have a 9Mb PBIX containing small tables and one table with 250k rows. Data imported from various xlsx & JSON sources. Machine is Windows 10 Pro, 2.6GHz, 64 bit, 16GB RAM.
On the Power BI service online the performance is ok, but on desktop it's practically unworkable. With task manager I can see that it is using 7Mb of memory, but almost 100% CPU, half an hour after opening - while on a blank tab with no visualisations.
I don't understand what it is doing in the background and how I can improve the situation.
There is the 'Allow data preview to download in the background' setting, but I think this is only relevant to the query editor? Would clearing the cache or changing cache settings help?
I am aware of performance analyzer and the query diagnostics tools, but neither seem relevant since the queries are not refreshing and there are no visualisations loading.
Am at a bit of a loss - any help greatly appreciated.
Thanks
Update: Having disabled parallel load and background refresh in Data load settings I noticed that finally the issue seemed to go away (though not immediately). Eventually, when reopening the pbix, mashup containers did not appear and CPU and memory was not being killed. Then at some point Power BI got stuck and had to close and the problem reappeared even though the data load settings were still disabled. Restarting the machine seemed to clear the problem once again.
It seems then, that some zombie processes can persist through application close and re-open. Has anyone else noticed this, can confirm or refute it, suggest what is going on or any steps on how to avoid/prevent? It's very annoying!
Thanks
I have also noticed the same issue, for opening 5 mb pbix file, power bi eating 12 GB of memory, and 90%+ CPU utilization, Power BI Desktop is poorly managed product by Microsoft.
I have just noticed that my C drive is getting full, whereas it still had 30 GB free space 3 days ago.
Given last days activity I can't find any reason for this.
Now I realize that my C free space is getting lower and lower even though there's no current activity on my PC (except that it's turned on).
Every 2 minutes, I lose approximately 100 MB of free space, even though I don't download anything.
I launched my antivirus and I have closed my internet connection in order to see if the free space would stop decreasing, but it continued decreasing at the same pace.
I checked the task manager and notice there was a software running which I think was named "One Drive setup.exe" (during the past weeks, I had many pop up windows saying I had to update onedrive, but there was a problem with the auto update etc... but I didn't car because I don't even know what OneDrive is and I don't think I use it). So I killed this running task.
I thought it had stopped the loss of free space (I even gained 100 MB), but the decrease started again.
Now I connected to Internet again.
I got 300 MB free space back and now it seems constant since 4 minutes. Maybe these little ups and downs can be due to the current antivirus scanning.
But what can explain the loss of 30 GB during the past 2 or 3 days?
Could it be windows update? How can i check this with windows 10?
Could it be a virus or something bad?
Please, answer quickly, i only have 17GB left :-(
Thanks
Which version of Windows OS are you using?
Turn off/ disable System Restore point, this way you will able to recover some space. Other than use CCleaner (https://www.piriform.com/ccleaner/download) to clean your system.
They release patch I believe. But u also can use built in disk cleanup tool(https://support.microsoft.com/en-us/help/4026616/windows-disk-cleanup-in-windows-10).
Also uninstalled OneDrive/Google Drive unless u actively use this service. OneDrive sync with cloud files so that u can use those off line.
I am running Cloudera Hadoop on my laptop and Oracle VirtualBox VM.
I have given 5.6 GB out of mine 8 and six from eight cores as well.
And still I am not able to keep it up and running.
Even without load services would not stay up and running and when I try a query at least Hive will be down within 20 minutes. And sometimes they go down like dominoes: one after another.
More memory seemed to help some: with 3GB and all services, Hue was blinking with red colors when the Hue itself managed to get up. And after rebooting it would takes 30 - 60 minutes before I manage to get the system up enough to even try running anything on it.
There has been two sensible notes (that I have managed to find):
- Warning of swapping.
- Crashing note when the system used 26 GB of virtual memory which was not enough.
My dataset is less than one megabyte, so it is hard to understand why the system would go up to dozens of gigabytes, but for whatever was reason for that has passed: now the system is running more steadily around the 5.6 GB that I have given to it after closing down a few services: see my answer to myself.
And still it is just more stable. Right after I got a warning of swapping and the Hive went down again. What could be reason for more-or-less all Hadoop services going down if the VM starts to swap?
I don't have enough reputation to post the picture to here, but when Hive went down again it was swapping 13 pages / second and utilizing 5.9 GB / 5.6 GB. So basically my system starts crashing more-or-less right after it start to swap. "428 pages were swapped to disk in the previous 15 minute(s)"
I have used default installation options as far as hard drive is concerned.
Only addition is a shared folder between Windows and VM. That works somewhat strangely locking files all the time, so I used it just like FTP and only for passing files from one system to another. Thus I can go days without using it, but systems still crash, so that is not the cause either.
Now that the system is mostly up, services crash still about twice a day: Service Monitor and Hive are quite even with their crashing frequency. After those come Activity Monitor and Event Server, which appear to crash always together. I believe Yarn crashes as well, but it gets up on its own. Last time Hive crashed first, and then it got followed by Service Monitor, Hive (second time), Activity Monitor and Event Server all.
As swap is disk, perhaps the problem is with disk:
# cat /etc/fstab
# swapoff -a
# badblocks -v /dev/VolGroup/lv_swap
Checking blocks 0 to 8388607
Checking for bad blocks (read-only test): done
Pass completed, 0 bad blocks found.
# badblocks -vw /dev/VolGroup/lv_swap
Checking for bad blocks in read-write mode
From block 0 to 8388607
Testing with pattern 0xaa: done
Reading and comparing: done
Testing with pattern 0x55: done
Reading and comparing: done
Testing with pattern 0xff: done
Reading and comparing: done
Testing with pattern 0x00: done
Reading and comparing: done
Pass completed, 0 bad blocks found.
So nothing wrong with swap disk and I have not noticed any disk error anywhere else either.
Note that you could check file system from Windows side also. But I expect that if you make Windows to fix your Linux file system, you have good chances of destroying your Linux with that, so I did my checks somewhat pessimistically, because AFAIK these commands are safe to execute.
About half of the services kept going down, so giving more specifics would be a long story.
I succeeded to get the system more stable by closing down flume, hbase, impala, ks_indexer, oozie, spark and sqoop. And by increasing more memory to some remaining services that complained they had not been given enough memory.
Also I fixed couple of thing on the Windows side, I am not sure which one of these helped:
- MsMpEng.exe kept my hard drive busy. I didn't have permissions to kill it, but I decreased its priority to lowest possible.
- CcmExec.exe got to loop on my DVD and kept reading it for forever. This I solved by taking the DVD out from the drive. Then later on I killed the process tree to keep it from bothering for a while.
I found these using Windows resource manager.
The VM requires 4GB: http://www.cloudera.com/content/cloudera-content/cloudera-docs/DemoVMs/Cloudera-QuickStart-VM/cloudera_quickstart_vm.html You should use that.
I am not clear whether you are using the QuickStart VM though. It's set up to run just the essential services and tuned to conserve memory rather than exploit lots of memory.
It sounds like you are running your own installation, on one virtual machine, on your Windows machine. You may be running an entire cluster's worth of services on one desktop machine. Each of these services has master, worker processes, monitoring processes, etc. You don't need most of them.
You also probably have left memory settings at default suitable for a server-class machine of 16+ GB RAM. Remember these services usually run across many machines, not all on one.
Finally, you're clearly swapping, and that makes things incredibly slow. Remember this is all through a VM too!
Bottom line, use the QuickStart VM if you really want a 1-machine cluster tuned correctly. If you want a real cluster or more services, you need more hardware.
Also consider: cloudera.com/live contains a full CDH 5.1 cluster + sample data, running on demand on AWS. Of course, the advantage of the VM is that you can BYOD, but if you're simply looking for a hands-on Hadoop experience, Live is a great option.
I am using Dell Latitude E6420.
2 days back while using it Windows crashed 4-5 times, displaying Blue Screen error with error codes IRQL_NOT_LESS_OR_EQUAL, PAGE_FAULT_IN_NONPAGED_AREA.
after that it showed Hard Disk problems. Since then, it takes almost 1.5-2 hours to completely boot the Windows.
Also after the boot process is completed, none of the application is working fine. Everything hangs for unlimited time period.
If anyone knows any solution for this, please guide me. (facing great trouble these days :( )
Check your ram.
Open your PC. and take one of the ram out(if it's possible. ) turn it on. check if problem still exists. then put it back in. take another one out.. check. rinse and repeat :D
Check your HDD.
Take your HDD out. put it in another pc as data.. check for HDD errors. do it for the other one.. rinse and repeat.
Check your CPU.
Well.. you COULD try switching it with another one.. but if in my case if your problem didn't get resolved with 1,2 i'd get another PC.. or .. try reinstalling windows on a DIFFERENT HDD
I am working with a SuSE machine (cat /etc/issue: SUSE Linux Enterprise Server 11 SP1 (i586)) running Postgresql 8.1.3 and the Slony-I replication system (slon version 1.1.5). We have a working replication setup going between two databases on this server, which is generating log shipping files to be sent to the remote machines we are tasked to maintain. As of this morning, we ran into a problem with this.
For a while now, we've had strange memory problems on this machine - the oom-killer seems to be striking even when there is plenty of free memory left. That has set the stage for our current issue to occur - we ran a massive update on our system last night, while replication was turned off. Now, as things currently stand, we cannot replicate the changes out - slony is attempting to compile all the changes into a single massive log file, and after about half an hour or so of running, it trips over the oom-killer issue, which appears to restart the replication package. Since it is constantly trying to rebuild that same package, it never gets anywhere.
My first question is this: Is there a way to cap the size of Slony log shipping files, so that it writes out no more than 'X' bytes (or K, or Meg, etc.) and after going over that size, closes the current log shipping file and starts a new one? We've been able to hit about four megs in size before the oom-killer hits with fair regularity, so if I could cap it there, I could at least start generating the smaller files and hopefully eventually get through this.
My second question, I guess, is this: Does anyone have a better solution for this issue than the one I'm asking about? It's quite possible I'm getting tunnel vision looking at the problem, and all I really need is -a- solution, not necessarily -my- solution.