When writing to Chronicle Queue, the default write doesn't flush to disk, so I believe anything that is in the linux kernel dirty page cache is lost. What's the best approach to get guaranteed recovery in the event of power failure? Would a battery backed raid array along with enforced flush on write be a good approach? Or is it better to use replication with an ack from the second machine before assuming the write is safely recorded? Which of these approaches would have the best performance? Theoretically the power failure could affect both machines if on the same power grid....
anything that is in the linux kernel dirty page cache is lost.
Yes
What's the best approach to get guaranteed recovery in the event of power failure?
Replicate the data to a second or third machine. That way even if the whole machine/data centre can't be recovered you can continue operation without data loss.
Would a battery backed raid array along with enforced flush on write be a good approach?
You have to trust the reliability of the hardware, something Chronicle can't guarantee and many of our clients have been burnt on before.
Or is it better to use replication with an ack from the second machine before assuming the write is safely recorded?
It depends on your requirements. This is best practice in our opinion, though many clients don't feel they need this option.
Another approach is to replicate the data to a secondary machine and have the secondary process the data. This can halve network latency introduced.
Which of these approaches would have the best performance?
The best performance is to assume a manual process will used in the event of a failure and be willing to accept a small loss. In this case, you process everything as soon as possible.
Note: There are some alternatives.
You can wait for ack for only the critical messages, other message types could be processed immediately.
You can allow a window where you process messages if no more than N haven't been acknowledged.
Theoretically the power failure could affect both machines if on the same power grid....
This is where 2+1 replication might be an option. One backup server nearby to recover normal operation in the event of the failure of a rack or part of one. AN a second backup off site, which is slower to replicate but has fair less chance of also failing.
Related
How do I keep the in-memory data structures in sync between two processes. Both processes are the same process(server) - one is active and the other one is a stand-by. The stand-by needs to take over in case of crash/or similar of the active. For the standby to take over the active, the in-memory data structures need to be kept in-sync. Can I use Virtual Synchrony? Will it help? If it would is there any library that I can use? I am coding on C++ on Windows(Visual Studio).
If that is not a solution what is a good solution I can refer to?
TIA
The easiest solution to implement is to store the state in a separate database, so that when you fail over, the standby will just continue using the same database. If you are worried about the database crashing, pay the money and complexity required to have main and standby databases, with the database also failing over. This is an attempt to push the complexity of handling state across failovers onto the database. Of course you may find that the overhead of database transactions becomes a bottleneck. It might be tempting to go NoSQL for this, but remember that you are probably relying on the ACID guarantees you get with a traditional database. If you ditch these, typically getting eventual consistency in return, you will have to think about what this means on failover. Will you lose a small amount of recent information on failover? do you care?
Virtual synchrony looks interesting. I have searched for similar things and found academic pages like http://www.cs.cornell.edu/ken/, some of which, like this, have links to open source software produced by research groups. I have never used them. I seem to remember reports that they worked pretty well for small number of machines with very good connectivity, but hit performance problems with scale, which I presume won't be a problem for you.
Once upon a time people built multiprocess systems on Unix machines by having the processes communicate via shared memory, or memory mapped files. For very simple data structures, this can be made to work. One problem you have is if one of the processes crashes halfway through modifying the shared data - will this mess up the other processes? You can solve these problems, but you are in danger of discovering that you have implemented everything inside the database that I described in my first paragraph.
You can go for in memory database like memcached or redis.
My web application maintains in memory cache of domain entities which are read/written at high frequency. To make application clustered, i need to synchronize / externalize this cache.
Which will be better option amongst memcached and infinispan considering following application facts-
cache will be read/written at high frequency per second
if infinispan, data need to replicated across nodes near- real time
high concurrent write should not create conflicts issue if replication is slow.
I feel memcached will solve this purpose well since it's centralized and does not need replication delay like infinispan. Can experts provide opinion on this?
Unfortunately I'm not a Memcached expert but let me tell you more about some fundamental concepts so that you could pick the best option for your use case...
First, centralized vs decentralized - if you have only one node in your system, it will be faster (as you said there is no replication). However what will happen if the node is down? Or another scenario - what will happen if the node gets full (as you said you will perform a lot of read/writes per second)? One solution for that is to use master/slave replication where writes are propagated to the slave node asynchronously. This solution will save you in case the node is down but won't do any good if the node is full (if master node is full, slave will get full a couple of minutes later).
Data consistency - if you have more than 1 node in your system, your data might get out of sync. Imagine asynchronous replication between 2 nodes and a client connected to each of them. Both clients perform a write to the same key at the same exact moment. It might seems unlikely but believe me, with highly concurrent reads and writes it will happen. The only way to solve this problem is to use synchronous replication with majority of nodes up and running (or with so called consensus).
Back to your scenario - if a broken node is not a problem for you (for example, you can switch to some other data source automatically) and your data won't grow - go ahead for 1 node solution or master/slave replication. If your data need to be strongly consistent - make sure you're doing sync replication (and possibly with transactions but you need to refer to the user manual for guidance). Otherwise I would recommend picking a more versatile solution which will allow you to add/remove nodes without taking down whole system and will have an option for sync/async replication.
From my experience, people care too much about data consistency whereas should care much more about scalability. And a final piece of advice - please define your performance criteria before evaluating any solution (something like, my writes need to take no longer than X and reads no longer than Y. Define also confidence level for your criteria (I need 99.5% of all reads to be less than X).
I have been reading into Hadoop and one thing I have read a few times is that Hadoop is good for offline computing. What does that mean "offline computing"?
The terms "online" and "offline" are sometimes used synonymously with the terms "real-time" and "batch" respectively. For example, "online machine learning" refers to learning that occurs continuously as new training data becomes available. By contrast, "offline machine learning" refers to learning from a batch of training data.
Hadoop is good at processing large amounts of data in batches (typically executed on a relatively infrequent basis, e.g. hourly), as opposed to processing data in real-time as it becomes available.
In on-line computing, user will wait for response to his query and expects the response in real-time or near real-time.
In offline computing, user will trigger/schedule an operation at a specific time and does not expect the response in real-time. Once the operation completes, he will check the result after sometime.
offline-computing will be done on huge sets of data, which may take hours to complete job and hence user is not interested in real time response. He will fire & forget the computing operation. He will check the results at his own time.
AFAIK, "offline computing" is exactly what it says. It's a system disconnected from the web (usually for security reasons). But it's also related to dedicated servers that can access local databases, so they don't really need to connect to the web and neither they want to waste any resource.
So I think the document you've read was meaning that Hadoop is great for dedicated servers that need to process long running dedicated calculations, where CPU is a must. In this case, "offline" is usually related to performance, more than security I guess.
I am a software developer but wannabe architect new to the server scalability world.
In the context of multiple services working with the same data set, aiming to scale for redundancies and load balancing.
The question is: In a idealistic system, should services try to optimize their internal processing to reduce the amount of queries done to the remote server cache for better performance and less bandwidth at the cost of some local memory and code base or is it better to just go all-in and query the remote cache as the single transaction point every time any transaction need processing done on the data?
When I read about Redis and even general database usage online, the later seems to be the common option. Every nodes of the scaled application have no memory and read and write directly to the remote cache on every transactions.
But as a developer, I ask if this isn't a tremendous waste of resources? Whether you are designing at electronic chips level, at inter-thread, inter-process or inter-machine, I do believe it's the responsibility of each sub-system to do whatever it can to optimize its processing without depending on the external world if it can and hence reduce overall operation time.
I mean, if the same data is read over hundreds or time from the same service without changes (write), isn't it just more logical to keep a local cache and wait for notifications of changes (pub/sub) and only read only these changes to update the cache instead reading the bigger portion of data every time a transaction require it? On the other hand, I understand that this method implies that the same data will be duplicated at multiple place (more ram usage) and require some sort of expiration system not to keep the cache from filling up.
I know Redis is built to be fast. But however fast it is, in my opinion there's still a massive difference between reading directly from local memory versus querying an external service, transfer data over network, allocating memory, deserialize into proper objects and garbage collect it when you are finished with it. Anyone have benchmark numbers between in-process dictionaries query versus a Redis query on the localhost? Is it a negligible time in the bigger scheme of things or is it an important factor?
Now, I believe the real answer to my question until now is "it depends on your usage scenario", so let's elaborate:
Some of our services trigger actions on conditions of data change, others periodically crunch data, others periodically read new data from external network source and finally others are responsible to present data to users and let them trigger some actions and bring in new data. So it's a bit more complex than a single web pages deserving service. We already have a cache system codebase in most services, and we have a message broker system to notify data changes and trigger actions. Currently only one service of each type exist (not scaled). They transfer small volatile data over messages and bigger more persistent (changing less often) data over SQL. We are in process of moving pretty much all data to Redis to ease scalability and performances. Now some colleagues are having a heated discussion about whether we should abandon the cache system altogether and use Redis as the common global cache, or keep our notification/refresh system. We were wondering what the external world think about it. Thanks
(damn that's a lot of text)
I would favor utilizing in-process memory as much as possible. Any remote query introduces latency. You can use a hybrid approach and utilize in-process cache for speed (and it is MUCH faster) but put a significantly shorter TTL on it, and then once expired, reach further back to Redis.
I'm trying to work out in my head the best way to structure a Cocoa app that's essentially a concurrent download manager. There's a server the app talks to, the user makes a big list of things to pull down, and the app processes that list. (It's not using HTTP or FTP, so I can't use the URL-loading system; I'll be talking across socket connections.)
This is basically the classic producer-consumer pattern. The trick is that the number of consumers is fixed, and they're persistent. The server sets a strict limit on the number of simultaneous connections that can be open (though usually at least two), and opening new connections is expensive, so in an ideal world, the same N connections are open for the lifetime of the app.
One way to approach this might be to create N threads, each of which would "own" a connection, and wait on the request queue, blocking if it's empty. Since the number of connections will never be huge, this is not unreasonable in terms of actual system overhead. But conceptually, it seems like Cocoa must offer a more elegant solution.
It seems like I could use an NSOperationQueue, and call setMaxConcurrentOperationCount: with the number of connections. Then I just toss the download requests into that queue. But I'm not sure, in that case, how to manage the connections themselves. (Just put them on a stack, and rely on the queue to ensure I don't over/under-run? Throw in a dispatch semaphore along with the stack?)
Now that we're in the brave new world of Grand Central Dispatch, does that open up any other ways of tackling this? At first blush, it doesn't seem like it, since GCD's flagship ability to dynamically scale concurrency (and mentioned in Apple's recommendations on Changing Producer-Consumer Implementations) doesn't actually help me. But I've just scratched the surface of reading about it.
EDIT:
In case it matters: yes, I am planning on using the asynchronous/non-blocking socket APIs to do the actual communication with the server. So the I/O itself does not have to be on its own thread(s). I'm just concerned with the mechanics of queuing up the work, and (safely) doling it out to the connections, as they become available.
If you're using CFSocket's non-blocking calls for I/O, I agree, that should all happen on the main thread, letting the OS handle the concurrency issues, since you're just copying data and not really doing any computation.
Beyond that, it sounds like the only other work your app needs to do is maintain a queue of items to be downloaded. When any one of the transfers is complete, the CFSocket call back can initiate the transfer of the next item on the queue. (If the queue is empty, decrement your connection count, and if something is added to an empty queue, start a new transfer.) I don't see why you need multiple threads for that.
Maybe you've left out something important, but based on your description the app is I/O bound, not CPU bound, so all of the concurrency stuff is just going to make more complicated code with minimal impact on performance.
Do it all on the main thread.
For posterity's sake, after some discussion elsewhere, the solution I think I'd adopt for this is basically:
Have a queue of pending download operations, initially empty.
Have a set containing all open connections, initially empty.
Have a mutable array (queue, really) of idle open connections, initially empty.
When the user adds a download request:
If the array of idle connections is not empty, remove one and assign the download to it.
If there are no idle connections, but the number of total connections has not reached its limit, open a new connection, add it to the set, and assign the download to it.
Otherwise, enqueue the download for later.
When a download completes: if there are queued requests, dequeue one
and give it to the connection; otherwise, add the connection to the idle list.
All of that work would take place on the main thread. The work of decoding the results of each download would be offloaded to GCD, so it can handle throttling the concurrency, and it doesn't clog the main thread.
Opening a new connection might take a while, so the process of creating a new one might be a tad more complicated in actual practice (say, enqueue the download, initiate the connection process, and then dequeue it when the connection is fully established). But I still think my perception of the possibility of race conditions was overstated.