our game server use LevelDB as data-cache.
Recently, We found a serious problem:
When server had a deadlock, we kill is use kill -9, some data in LevelDB are broken.
So, I have two doubt:
Why data broken? I am sure my data had write back to LevelDB?
Someway to repair broken LevelDB ?
About first questions:
Use WriteBatch-es (aka atomic operations)
Use Synchronous writes
Read this article (sections Atomic Updates and Synchronous Writes). Seems like you don't do it.
About second questions: use backups :D ALWAYS USE BACKUPS!
Related
Can anyone tell me if there is a way to prioritize reading versus writing in the tarantool slave? There is a task which is more important to read than to write, but during writing to the master, the changed records are blocked for some time on the slave. In general, I understand that this is the correct behavior for databases. But maybe there is an opportunity to prioritize reading versus writing on a slave?
all transactions (both read and write) are served in a single thread, there is not prioritization between them.
You can read about it in more details here: https://www.tarantool.io/en/doc/1.10/book/box/atomic/
I understand that with MSI, if we have a piece of memory in shared state, even if no one else uses it, we would have to broadcast that we are moving to modified. This is a problem that MESI fixes.
However, when we do use MESI, when moving from invalid to exclusive, we need to broadcast that we want to read this, and wait if there are not HIT reponses. How is this any better?
I need to read it (from memory) and then we do the things I said. But it seems like I don't save myself any broadcasting, I just do it when getting my stuff instead of doing it when I move from shared to modified.
Consider the case where you load first, then store. With MSI you'd read into Shared, then need to go off-core again to get exclusive ownership before committing a store.
With MESI you read into Exclusive state for the pure load, and then flipping to Modified is local; no off-core communication.
Turns out this is the example Wikipedia gives in https://en.wikipedia.org/wiki/MESI_protocol#Advantages_of_MESI_over_MSI
I'm currently using MySql to store my sessions. It works great, but it is a bit slow.
I've been asked to use Redis, but I'm wondering if it is a good idea because I've heard that Redis delays write operations. I'm a bit afraid because sessions need to be real-time.
Has anyone experienced such problems?
Redis is perfect for storing sessions. All operations are performed in memory, and so reads and writes will be fast.
The second aspect is persistence of session state. Redis gives you a lot of flexibility in how you want to persist session state to your hard-disk. You can go through http://redis.io/topics/persistence to learn more, but at a high level, here are your options -
If you cannot afford losing any sessions, set appendfsync always in your configuration file. With this, Redis guarantees that any write operations are saved to the disk. The disadvantage is that write operations will be slower.
If you are okay with losing about 1s worth of data, use appendfsync everysec. This will give great performance with reasonable data guarantees
This question is really about real-time sessions, and seems to have arisen partly due to a misunderstanding of the phrase 'delayed write operations' While the details were eventually teased out in the comments, I just wanted to make it super-duper clear...
You will have no problems implementing real-time sessions.
Redis is an in-memory key-value store with optional persistence to disk. 'Delayed write operations' refers to writes to disk, not the database in general, which exists in memory. If you SET a key/value pair, you can GET it immediately (i.e in real-time). The policy you select with regards to persistence (how much you delay the writes) will determine the upper-bound for how much data could be lost in a crash.
Basically there are two main types available: async snapsnots and fsync(). They're called RDB and AOF respectively. More on persistence modes on the official page.
The signal handling of the daemonized process syncs to disk when it receives a SIGTERM for instance, so the data will still be there after a reboot. I think the daemon or the OS has to crash before you'll see an integrity corruption, even with the default settings (RDB snapshots).
The AOF setting uses an Append Only File that logs the commands the server receives, and recreates the DB from scratch on cold start, from the saved file. The default disk-sync policy is to flush once every second (IIRC) but can be set to lock and write on every command.
Using both the snapshots and the incremental log seems to offer both a long term don't-mind-if-I-miss-a-few-seconds-of-data approach with a more secure, but costly incremental log. Redis supports clustering out of the box, so replication can be done too it seems.
I'm using the default RDB setting myself and saving the snapshots to remote FTP. I haven't seen a failure that's caused a data loss yet. Acute hardware failure or power outages would most likely, but I'm hosted on a VPS. Slim chance of that happening :)
Our app was originally built with NHibernate and its limitations of batch processing in mind. However, over time it has transformed into a data cruncher and we are observing a significant performance decay.
The session ends up having to maintain about 1000 objects or more and our profiling has revealed that auto flushing and dirty checking are the biggest offenders here. We tried shutting auto flush and managing it ourselves on Save/Update operations but that led to disastrous performance for a batch save/update.
We're now looking at the option of evicting unrequired objects from the session.
I came across 2nd level-cache eviction method (sessionFactory.Evict(typeof(Cat));) which lets us evict by type but we do not use a 2nd level cache. Can I still use this method to evict objects from the 1st level cache?
I also read about one pattern of fetching objects, evicting them from session, and then reassociating them, if needed, with session by calling Update() on them. Is this a recommended and accepted pattern cause I also read that NH3 has put up a wall to this? (We can still use it as we have not upgraded to NH3)
While we realize that we are not using NHibernate in the best way, we are just looking to improve the current situation somehow. Answers to the above questions and any other suggestions/recommendations are greatly appreciated. Thanks.
Update
After looking at NH documentation and code, I realize that 1 is probably not possible. I'm still looking at some pointers or tips on using Evict(). I was able to drastically reduce the number of objects in a session. But still do not know if there is a price to pay while updating or deleting evicted objects. Thanks for your help in advance.
It's hard to say without knowing more about your requirements but maybe you could use IStatelessSession. It doesn't have a 1st level cache to worry about.
Ayende has a good post on using it for bulk operations
here
Why not use more sessions, instead of one large one? That, in conjunction with turning off autoflush has helped me in the past. Also, you should really think about using HQL for bulk updates if possible.
I know that this is old, but I just came across this while looking for something else -- having just solved this. I did solve as Trent mentioned, by using more than one session. I would create one session to fetch all of the objects I wanted, then closed that session. The case I had, was iterating through the list and operating on each object and trying to commit on each iteration. I would then create the foreach over my list, creating and disposing of a new session inside the loop, reattaching my object from the list to the new session. That took a process that was taking about 2.5 hours down to 2 minutes 40 seconds!
See this article for the inspiration to how I solved it -- although not exactly as I have unit of work wrappers around NHibernate:
http://weblogs.asp.net/ricardoperes/archive/2013/03/21/attaching-disconnected-entities-in-nhibernate-without-going-to-the-database.aspx
1.How to understand asynchronous io in Windows??
2.If I write/read something to the file using asynchronous io :
WriteFile();
ReadFile();
WriteFile();
How many threads does the OS generate to accomplish these task?
Do the 3 task run simultaneously and in multi-threading way
or run one after another just with different order?
3.Can I use multithreading and in each thread using a asynchronous io
to read or write the same file？
1.How to understand asynchronous io in Windows??
Read the Win32 documentation. Search on the web. Don't expect an answer to such a large, broad question here in SO.
2.If I write/read something to the file using asynchronous io :
WriteFile();
ReadFile();
WriteFile();
How many threads does the OS generate to accomplish these task?
I don't think it does. It will re-use existing thread contexts to execute kernel function calls. Basically the OS schedules the work and borrows a thread to do it - which is fine, since the kernel context is always the same.
3.Can I use multithreading and in each thread using a asynchronous io to read or write
the same file？
I believe so, yes. I don't know that the order of execution is guaranteed to match the order of submission, in which case you will obtain unpredictable results if you issue concurrent reads/writes on the same byte ranges.
To your questions:
How many threads does the OS generate
to accomplish these task?
Depends if you are using the windows pools, iocp, etc. Generally you decide.
Do the 3 task run simultaneously and
in multi-threading way or run one
after another just with different
order?
This depends on your architecture. On a single-cored machine, the 3 tasks would run one after another and the order would be os decided. On a multi-cored machine these might run together, depending on how the OS scheduled the threads.
3.Can I use multithreading and in each thread using a asynchronous io to read
or write the same file？
That is out of my knowledge so someone else would need to answer that one.
I suggest getting a copy of Windows via C/C++ as that has a very large chapter on Asynchronous IO.
I guess it depends which operating system you are using. But you shouldnt have to worry about this anyhow, it is transparent and should not affect how you write your code.
If you use the standard read and write in windows, you don't have to care that the system may not write it immediately, unless you are writing on the command-line and are waiting for the user to type some input. The OS is responsible for ensuring that what you write will eventually be written to the hard drive, and will do a much better job that you can do anyway.
If you are working on some weird asynchronous io, then please reformat your question.
I suggest looking for Jeffery Richter's books on Win32 programming. They are very well-written guides for just this sort of thing.
I think he has a newer book(s?) on C#, so watch out that you don't buy the wrong one.