Does anyone know how well tested LevelDB is and what is its status for use in production? It's a relatively new library and when I checked the source code it didn't appear to be handling errors too well. Does anyone use LevelDB in production and can comment on my question?
We use LevelDB in our website, but wrapped in SSDB(https://github.com/ideawu/ssdb), the LevelDB network server, with hash/zset data types support. Our SSDB instance serves 100 million queries per day.
LevelDB has a lot of high-visibility problems https://github.com/bitcoin/bitcoin/issues/2770 and the code is so poorly written that a bounty was needed to find a fix https://bitcointalk.org/index.php?topic=337294.0;all And the leveldb discussion group is predominantly bug reports about very fundamental database functionality that fails to work as advertised. https://groups.google.com/forum/#!forum/leveldb (e.g., "snapshots" aren't actually snapshots, and can be tainted by subsequent writes https://groups.google.com/forum/#!topic/leveldb/IAKJaL2zqZM etc...)
On the date that this question was asked, LevelDB was certainly NOT production ready and anyone who thought so was delusional. The code quality is abysmal, as confirmed by independent developers https://twitter.com/rescrv/status/406106256890286080
One place it is used in a production environment is the Bitcoin project. Within bitcoin, it's usage is critical for the security of the platform. See the release notes for Bitcoin QT 0.8.0
How do you qualify "relatively new" as it was out in 2011?
Can you please give more detail on "not handling errors too well"?
LevelDB is used as a backend in Riak and Hyperdex, which have both customised it to improve throughput under huge loads. There was a great video from Ricon East 2013 explaining the Riak changes made by Basho. (taken down at some point prior to 2019-03).
Note that RocksDB is another major fork, by Facebook, which is recommended for serverside. History of it forking from LevelDB is on WikiPedia. You can read about how RocksDB handles errors on this page:
Currently in RocksDB, any error during a write operation (write to
WAL, Memtable Flush, background compaction etc) causes the database
instance to go into read-only mode by default and further user writes
are not accepted....
Call DB::Resume() to manually resume the DB and put it in read-write
mode. This function will clear the error, purge any obsolete files,
and restart background flush and compaction operations. At present, it
only supports resuming from background errors that happen during
compaction. In the future, we will add more cases.
Related
We work with a lot of data and have a high throughput of files going through our Nifi instances. We have recently been losing providence records and don't seem to understand what the cause is.
Below are some details, if relevant:
We have our Providence database on its own drive in the cloud, and are not seeing any high IO usage or resource contention.
We have added additional threads to this, as well as 999k file handles.
If it means anything, providence data is kept for 2 weeks in our configuration.
We are on Nifi version 1.15.3, but are planning on an upgrade in the near future.
Any ideas on what the cause may be and how to remediate this? Thanks!
I recently came upon a SO question where the op asked in which scenarios redis frees up memory. It seems they were recommended a hard start is a potential way, however this is untested in the case of redis. Can anyone let me know for sure whether this works?
I have a live environment, I don't want to have to restart redis-server, but its memory foot print is debilitating now and I'm on the verge of a server migration. So it's important for me to remove as much bloat as possible (and there's a ton of bloat).
I'm not sure what you mean by "bloat", but attaching your server's INFO ALL output may be helpful.
By default, Redis uses jemalloc as a memory allocator. The allocator is in charge of actually freeing RAM for the OS to reclaim, after Redis frees it. Redis v4 and above include the ability to force the allocator to purge the freed RAM (MEMORY PURGE, see https://github.com/antirez/redis-doc/pull/851).
Regardless of purge, there's also the matter of memory fragmentation. While v4 has the experimental active defrag feature, a restart is the way to "fix" that in prior versions.
To mitigate a restart and the downtime involved, use Redis' replication to create a slave and failover your apps to it before restarting the original master.
A Twisted app I have was constantly getting killed due to memory problems. The program grew in size, consuming all of the system's memory before being shut down by the os. Restart and repeat.
This is on a virtual server, so I doubled the memory, and the issue resolved - the daemon stabilized at around 1.25GB of memory
Does anyone have advice on how I can best profile this to tell what/where all the memory is getting sucked up into ?
If info on the app helps, I'm using the twisted reactor and internet.timer.TimerService to poll a database for items to update through three 'services'. the items to process are pushed into a twisted.internet.defer.DeferredList , and their processing occurs in a deferToThread block. In the deferred process there are a handful of blocking operations ( fetching web pages, etc ) and a lot of HTML parsing ( beautiful soup and other libraries ). I've suggested the reactor.threadpool size to be 10 and each 'service' defers to thread using a SemaphoreService that has 10 tokens. I really expected this daemon to max out at around 400MB of memory, not 3x that.
This is more of a generic share of thoughts how I debug memory leak/usage problems in my twisted applications.
Twisted has a ssh server support, and is something which I add in to almost all of my projects in development.
The ssh provides a interactive python interpreter access to the method which has python garbage collector available and a number of helper functions which allow me to a) inspect count of the instances from a same class, b) start and stop inspection of changes of that count over time and c) to get all references of that class. The nice thing with the interactive interpreter is that it allows ad-hoc introspection of offending instances, their relation to other objects and the state of process they are in. This so far has always proven a valuable instrument to pinpoint exact location where I have forgot / unforseen the ref release problems in my projects.
What is the best way to check memory usage in an ASP.NET MVC3 application?
I have been told by my hosting provider to recyle the IIS application pool every so often to improve the speed of the site. Is this what is 'recommended practice'? Surely I shouldn't need to restart my application every so often? I'd much rather find out if it is an issue with memory usage in my application and correct it. So any tips & best practices you use would be quite helpful too.
The application is based on ASP.NET MVC3, C# and EF Code First. Any guidance, links appreciated.
EDIT:
I found this page after I posted, which is quite useful. But I'd still like to hear any other views.
ASP.NET MVC and EF Code First Memory Usage
Thank you
I have a site that never recycles (until the machine is rebooted weekly)
Your application generally should keep performing fine. If it doesn't, there is some leak.
This can occur because
Cache never expires
Cache never expires
Session storage keeps growing and never times out
ObjectContexts are never disposed and kept in the session, etc
Objects that should be disposed aren't
Objects that are created via a dependency injection container aren't setup to release after each request, and thus potentially have internal collections that keep growing.
There are more causes - but these are a few main ones.
So the question really is 'there is no best practice - it depends on your app'
If you are worried about current sessions during a restart, keep in mind a restart can be quick and current requests are allowed to finish (sometimes) and forms authentication tokens will survive the restart, however sessions will not unless you configure an out of process state server.
If your memory usage keeps growing, then setup a restart schedule, otherwise do once a week or never - or setup once memory goes to XYZ then reset. ASP.NET will restart automatically once a certain threshold is reached as well based on what the hoster has setup on memoryLimit:
http://msdn.microsoft.com/en-us/library/7w2sway1.aspx
By default IIS recycles the application pool automatically at an interval (I think is 29 hours or so) but that is surely set by the host, no matter how little or how much memory you're the process is using. THe recycling trigger can be a time interval or when the process hits a certain memory usage limit. I'm sure any shared host has both of them set.
About memory usage, you can use the GC.GetTotalMemory method which will give you an approximate usage. Even when using Perfmon the readings aren't very accurate but it gives you an idea.
//global.asax.cs
void Application_EndRequest(object o,EventArgs a)
{
var ctype=Context.Response.Headers["Content-Type"];
if (ctype == null || !ctype.Contains("text/html")) return;
Context.Response.Write(string.format("<p>Memory usage: {0}</p>",GC.GetTotalMemory(false)));
}
Be aware that you'll see the usage increasing increasing until the GC kicks in and the usage will drop to a more 'realistic' value.
If you have the money I recommend a specialized tool such as the Memory profiler
Other things you can do to at least be ready if the application has memory or performance problems:
Proper layering of the application, means you can refactor the more inefficient parts without affecting the others.
The Repository pattern will be very helpful, because you can start using EF , find out that EF uses to much memory (like in the link you've found), but then you could switch the repository implementation to use PetaPoco or Dapper.net.
In general an OR\M is more of a heavy library, if the application doesn't need ORM features but just a quick way to work with a db, use from the beginning a mico-Orm like those mentioned above.
Always dispose objects implementing IDisposable.
When dealing with large db records, use pagination. It's good for both server resources usage and user experience
Apply the YAGNI (You Aint Gonna Need It) principle as much as possible, this somehow implies a bit of TDD :)
We have a JPA -> Hibernate -> Oracle setup, where we are only able to crank up to 22 transactions per seconds (two reads and one write per transaction). The CPU and disk and network are not bottlenecking.
Is there something I am missing? I wonder if there could be some sort of oracle imposed limit that the DBA's have applied?
Network is not the problem, as when I do raw reads on the table, i can do 2000 reads per second. The problem is clearly writes.
CPU is not the problem on the app server, the CPU is basically idling.
Disk is not the problem on the app server, the data is completely loaded into memory before the processing starts
Might be worth comparing performance with a different client technology (or even just a simple test using SQL*Plus) to see if you can beat this performance anyway - it may simply be an under-resourced or misconfigured database.
I'd also compare the results for SQLPlus running directly on the d/b server, to it running locally on whatever machine your Java code is running on (where it is communicating over SQLNet). This would confirm if the problem is below your Java tier.
To be honest there are so many layers between your JPA code and the database itself, diagnosing the cause is going to be fun . . . I recall one mysterious d/b performance problem resolved itself as a misconfigured network card - the DBAs were rightly insistent that the database wasn't showing any bottlenecks.
It sounds like the application is doing a transaction in a bit less than 0.05 seconds. If the SELECT and UPDATE statements are extracted from the app and run them by themselves, using SQL*Plus or some other tool, how long do they take, and if you add up the times for the statements do they come pretty near to 0.05? Where does the data come from that is used in the queries, and which eventually gets used in the UPDATE? It's entirely possible that the slowdown is not the database but somewhere else in the app, such a the data acquisition phase. Perhaps something like a profiler could be used to find out where the app is spending its time.
Share and enjoy.