i'm wanting to generate a random salt value and put it into the Application state.
Now, i'm using a web farm, so the Application state will be different per machine. I don't want to purchase distributed state farm apps, either.
So .. what are some solutions for this? I thought i could hard-code it in the code OR the web.config file .. but that means the same salt for eva .. not very safe.
Anyone have any suggestions?
Remember - i'm after a unique key (eg a guid) that is the same across all machines. Maybe a config file is the only way?
If I understand correctly, you want the machines to share a value, and you don't want the value to be the same forever. Ideally you'd prefer not to store it.
So, have the "first" machine generate a random value at startup, (using whatever entropy it can such as /dev/random. If you don't need a secure value, and don't have enough entropy at startup to create one anyway, use the time or whatever), and communicate it to all the others. As new machines join the cluster, they need to be able to find the value from one machine already in the cluster. Machines dropping out make no difference.
Which machine is the "first"? Well, if you can always boot one machine before any others, and give it time to get to the point of generating a value, then you can use the trivial algorithm:
1) Look for other machines. If you find one, ask it the value.
2) If you don't find one, generate the value yourself.
If multiple machines are starting up at once then they need to decide amongst themselves which is the "leader". You could do this by choosing one yourself (e.g. a machine declares itself "leader" as soon as it receives a particular connection via the admin interface: on startup each machine waits until it either gets this connection, or hears from another machine that the other machine is the leader). It's trivial to do automatically on a token ring: the machine with the least MAC address or whatever is leader. But nobody uses token ring any more...
At the opposite extreme of an unreliable network I'm not sure it's even possible, unless all the machines know how many there will be in total (in which case it's just like the token ring, except that they all talk to each other until they've figured out who's the leader). With reliable broadcast, which is what you can assume within reasonable bounds on ethernet, I'm sure there's an optimal algorithm published somewhere, but I forget what it is (if I ever knew). I'd guess that everyone broadcasts who they think the leader is at regular intervals (including their own claim if they've not yet seen a better one). Once you've been listening to that for long enough (approx one interval), you'll know who the leader is, and you can start using the seed.
If the value is a secret, then obviously communication within the cluster must be secure. You might get that for free, depending on the network architecture.
In a web farm scenario the machine key stored in machine.config might be what you are looking for. It is a randomly generated hash that must be the same across machines in the web farm.
Here's a fun/interesting way:
At the start of each day, i.e., after 00:00, lookup twitter or digg or friendfeed or anything that has frequently changing data for a detail just before/at 00:00. All the machines would get this same data AND it's guaranteed to be changing. In order to make it a secret, combine this detail with some secret key or the MAC of the DNS server or some such thing.
I think this is simple enough and a fun thing. Of course, this would mean your website would depend on this external website for a trivial thing and might be difficult if the site is down. But in such cases you can retain and continue with previous day's data.
Just wanted to write about it as a fun/interesting way :)
Related
I'm just wondering, usually huge web apps have a few server instances. When is a new server instance better than just upgrading the current one's memory, cpu and etc.?
I understand location reasons, but wondering about any other reasons.
Here are a few reasons off the top of my head why scaling horizontally can be better than scaling vertically:
Fault tolerance. A server can do down (or be taken down by you for
maintenance/updates) and the site can stay online.
Red/black deployments. Let's say you have a new version of your software you want to roll out, but you're not 100% confident in how it works. If you have two or more servers, you can update one (and segment a % of your users to only use that server) and see how it goes. This way if there are egregious problems, you can roll back and only a few users will have seen problems.
You can treat your hardware as cattle and not pets. If you try to scale up one server to as large as it will go, you will need to be very careful with it. It will become a delicate flower that you don't allow anyone to touch. However if you have a bunch of cheaper, commoditized servers you can lose a few, update some, replace them at will. This will save you time and money in the long run.
Lastly, at some point you will not be able to scale up your one precious snowflake server. At that point what do you do? You now have to scale out while also dealing with a massive amount of traffic this one machine is handling. Better to do it earlier when things are simpler!
Which is better in terms of performance, 2 medium role instances or 4 small role instances?
What are the pro's and cons of each configuration?
The only real way to know if you gain advantage of using larger instances is trying and measuring, not asking here. This article has a table that says that a medium instance has everything twice as large as a small one. However in real life your mileage may vary and how this affects your application only you can measure.
Smaller roles have one important advantage - if instances fail separately you get smaller performance degradation. Supposing you know about "guaranteed uptime" requirement of having at least two instances, you have to choose between two medium and four small instances. If one small instance fails you lose 1/4 of your performance, but if one medium instance fails you lose half of performance.
Instances will fail if for example you have an unhandled exception inside Run() of your role entry point descendant and sometimes something just goes wrong big time and your code can't handle this and it'd better just restart. Not that you should deliberately target for such failures but you should expect them and take measures to minimize impact to your application.
So the bottom line is - it's impossible to say which gets better performance, but uptime implications are just as important and they are clearly in favor of smaller instances.
Good points by #sharptooth. One more thing to consider: When scaling in, the fewest number of instances is one, not zero. So, say you have a worker role that does some nightly task for an hour, and it requires either 2 Medium or 4 Small instances to get the job done in that timeframe. When the work is done, you may want to save costs by scaling to one instance and let it run as one instance for 23 hours until the next nightly job. With a single Small instance, you'll burn 23 core-hours, and with a single Medium instance, you'll burn 46 core-hours. This thinking also applies to your Web role, but probably more-so since you will probably have minimum two instances to make sure you have uptime SLA (it may not be as important for you to have SLA on your worker if, say, your end user never interacts with it and it's just for utility purposes).
My general rule of thumb when sizing: Pick the smallest VM size that can properly do the work, and then scale out/in as needed. Your choice will primarily be driven by CPU, RAM, and network bandwidth needs (and don't forget you need network when moving data between Compute and Storage).
For a start, you won't get the guaranteed uptime of 99% unless you have at least 2 roles role instances, this allows one to die and be restarted while the other one takes the burden. Otherwise, it is a case of how much you want to pay and what specs you get on each. It has not caused me any hassle having more than one role role instance, Azure hides the difficult stuff.
One other point maybe worth a mention if you use four small roles you would be able to run two in one datacenter and two in another datacenter and use traffic manager to route people at least which is closer. This might give you some performance gains.
Two mediums will give you more options to store stuff in cache at compute level and thus more in cache rather than coming off SQL Azure it is going to be faster.
Ideally you have to follow #sharptooth and measure and test. This is all very subjective and I second David also you want to start as small as possible and scale outwards. We run this way, you really want to think about designing your app around a more sharding aspect as this fits azure model better than working in traditional sense of just getting a bigger box to run everything on, at some point you run out into limits thinking in the bigger box process, ie.Like SQL Azure Connection limits.
Using technologies like Jmeter is your friend here and should give you some tools to test your app.
http://jmeter.apache.org/
Could someone give me a list of real use cases of Paxos. That is real problems that require consensus as part of a bigger problem.
Is the following a use case of Paxos?
Suppose there are two clients playing poker against each other on a poker server. The poker server is replicated. My understanding of Paxos is that it could be used to maintain consistency of the inmemory data structures that represent the current hand of poker. That is, ensure that all replicas have the exact same inmemory state of the hand.
But why is Paxos necessary? Suppose a new card needs to be dealt. Each replica running the same code will generate the same card if everything went correct. Why can't the clients just request the latest state from all the replicated servers and choose the card that appears the most. So if one server had an error the client will still get the correct state from just choosing the majority.
You assume all the servers are in sync with each other (i.e., have the same state), so when a server needs to select the next card, each of the servers will select the exact same card (assuming your code is deterministic).
However, your servers' state also depends on the the user's actions. For example, if a user decided to raise by 50$ - your server needs to store that info somewhere. Now, suppose that your server replied "ok" to the web-client (I'm assuming a web-based poker game), and then the server crashed. Your other servers might not have the information regarding the 50$ raise, and your system will be inconsistent (in the sense that the client thinks that the 50$ raise was made, while the surviving servers are oblivious of it).
Notice that majority won't help here, since the data is lost. Moreover, suppose that instead of the main server crashing, the main server plus another one got the 50$ raise data. In this case, using majority could even be worse: if you get a response from the two servers with the data, you'll think the 50$ raise was performed. But if one of them fails, then you won't have majority, and you'll think that the raise wasn't performed.
In general, Paxos can be used to replicate a state machine, in a fault tolerant manner. Where "state machine" can be thought of as an algorithm that has some initial state, and it advances the state deterministically according to messages received from the outside (i.e., the web-client).
More properly, Paxos should be considered as a distributed log, you can read more about it here: Understanding Paxos – Part 1
Update 2018:
Mysql High Availability uses paxos: https://mysqlhighavailability.com/the-king-is-dead-long-live-the-king-our-homegrown-paxos-based-consensus/
Real world example:
Cassandra uses Paxos to ensure that clients connected to different cluster nodes can safely perform write operations by adding "IF NOT EXISTS" to write operations. Cassandra has no master node so two conflicting operations can to be issued concurrently at multiple nodes. When using the if-not-exists syntax the paxos algorithm is used order operations between machines to ensure only one succeeds. This can then be used by clients to store authoritative data with an expiration lease. As long as a majority of Cassandra nodes are up it will work. So if you define the replication factor of your keyspace to be 3 then 1 node can fail, of 5 then 2 can fail, etc.
For normal writes Caassandra allows multiple conflicting writes to be accepted by different nodes which may be temporary unable to communicate. In that case doesn't use Paxos so can loose data when two Writes occur at the same time for the same key. There are special data structures built into Cassandra that won't loose data which are insert-only.
Poker and Paxos:
As other answers note poker is turn based and has rules. If you allow one master and many replicas then the master arbitrates the next action. Let's say a user first clicks the "check" button then changes their mind and clicks "fold". Those are conflicting commands only the first should be accepted. The browser should not let them press the second button it will disable it when they pressed the first button. Since money is involved the master server should also enforce the rules and only allow one action per player per turn. The problem comes when the master crashes during the game. Which replica can become master and how do you enforce that only one replica becomes master?
One way to handle choosing a new master is to use an external strong consistently service. We can use Cassandra to create a lease for the master node. The replicas can timeout on the master and attempt to take the lease. As Cassandra is using Paxos it is fault tolerant; you can still read or update the lease even if Cassandra nodes crash.
In the above example the poker master and replicas are eventually consistent. The master can send heartbeats so the replicas know that they are still connected to the master. That is fast as messages flow in one direction. When the master crashes there may be race conditions in replicas trying to be the master. Using Paxos at that point gives you strong consistently on the outcome of which node is now the master. This requires additional messages between nodes to ensure a consensus outcome of a single master.
Real life use cases:
The Chubby lock service for loosely-coupled distributed systems
Apache ZooKeeper
Paxos is used for WAN-based replication of Subversion repositories and high availability of the Hadoop NameNode by the company I work for (WANdisco plc.)
In the case you describe, you're right, Paxos isn't really necessary: A single central authority can generate a permutation for the deck and distribute it to everyone at the beginning of the hand. In fact, for a poker game in general, where there's a strict turn order and a single active player as in poker, I can't see a sensible situation in which you might need to use Paxos, except perhaps to elect the central authority that shuffles decks.
A better example might be a game with simultaneous moves, such as Jeopardy. Paxos in this situation would allow all the servers to decide together what sequence a series of closely timed events (such as buzzer presses) occurred in, such that all the servers come to the same conclusion.
I am implementing my first syncing code. In my case I will have 2 types of iOS clients per user that will sync records to a server using a lastSyncTimestamp, a 64 bit integer representing the Unix epoch in milliseconds of the last sync. Records can be created on the server or the clients at any time and the records are exchanged as JSON over HTTP.
I am not worried about conflicts as there are few updates and always from the same user. However, I am wondering if there are common things that I need to be aware of that can go wrong with a timestamp based approach such as syncing during daylight savings time, syncs conflicting with another, or other gotchas.
I know that git and some other version control system eschew syncing with timestamps for a content based negotiation syncing approach. I could imagine such an approach for my apps too, where using the uuid or hash of the objects, both peers announce which objects they own, and then exchange them until both peers have the same sets.
If anybody knows any advantages or disadvantages of content-based syncing versus timestamp-based syncing in general that would be helpful as well.
Edit - Here are some of the advantages/disadvantages that I have come up with for timestamp and content based syncing. Please challenge/correct.
Note - I am defining content-based syncing as simple negotiation of 2 sets of objects such as how 2 kids would exchange cards if you gave them each parts of a jumbled up pile of 2 identical sets of baseball cards and told them that as they look through them to announce and hand over any duplicates they found to the other until they both have identical sets.
Johnny - "I got this card."
Davey - "I got this bunch of cards. Give me that card."
Johnny - "Here is your card. Gimme that bunch of cards."
Davey - "Here are your bunch of cards."
....
Both - "We are done"
Advantages of timestamp-based syncing
Easy to implement
Single property used for syncing.
Disadvantages of timestamp-based syncing
Time is a relative concept to the observer and different machine's clocks can be out of sync. There are a couple ways to solve this. Generate timestamp on a single machine, which doesn't scale well and represents a single point of failure. Or use logical clocks such as vector clocks. For the average developer building their own system, vector clocks might be too complex to implement.
Timestamp based syncing works for client to master syncing but doesn't work as well for peer to peer syncing or where syncing can occur with 2 masters.
Single point of failure, whatever generates the timestamp.
Time is not really related to the content of what is being synced.
Advantages of content-based syncing
No per peer timestamp needs to be maintained. 2 peers can start a sync session and start syncing based on the content.
Well defined endpoint to sync - when both parties have identical sets.
Allows a peer to peer architecture, where any peer can act as client or server, providing they can host an HTTP server.
Sync works with the content of the sets, not with an abstract concept time.
Since sync is built around content, sync can be used to do content verification if desired. E.g. a SHA-1 hash can be computed on the content and used as the uuid. It can be compared to what is sent during syncing.
Even further, SHA-1 hashes can be based on previous hashes to maintain a consistent history of content.
Disadvantages of content-based syncing
Extra properties on your objects may be needed to implement.
More logic on both sides compared to timestamp based syncing.
Slightly more chatty protocol (this could be tuned by syncing content in clusters).
Part of the problem is that time is not an absolute concept. Whether something happens before or after something else is a matter of perspective, not of compliance with a wall clock.
Read up a bit on relativity of simultaneity to understand why people have stopped trying to use wall time for figuring these things out and have moved to constructs that represent actual causality using vector clocks (or at least Lamport clocks).
If you want to use a clock for synchronization, a logical clock will likely suit you best. You will avoid all of your clock sync issues and stuff.
I don't know if it applies in your environment, but you might consider whose time is "right", the client or the server (or if it even matters)? If all clients and all servers are not sync'd to the same time source there could be the possibility, however slight, of a client getting an unexpected result when syncing to (or from) the server using the client's "now" time.
Our development organization actually ran into some issues with this several years ago. Developer machines were not all sync'd to the same time source as the server where the SCM resided (and might not have been sync'd to any time source, thus the developer machine time could drift). A developer machine could be several minutes off after a few months. I don't recall all of the issues, but it seems like the build process tried to get all files modified since a certain time (the last build). Files could have been checked in, since the last build, that had modification times (from the client) that occurred BEFORE the last build.
It could be that our SCM procedures were just not very good, or that our SCM system or build process were unduly susceptible to this problem. Even today, all of our development machines are supposed to sync time with the server that has our SCM system on it.
Again, this was several years ago and I can't recall the details, but I wanted to mention it on the chance that it is significant in your case.
You could have a look at unison. It's file-based but you might find some of the ideas interesting.
I'm not even sure the following is possible, but it never hurts to ask:
I have two nodes running the same application. Each machine needs a sequence generator that will give it a number between 0 and 1e6. If a node has used a number, the other node must not use it. The generator should reset every night at midnight. No number should be used twice in the same day, even if a machine restarts. We'd like to avoid any solution involving databases, distributed caches or filesystems. Let's assume we will never need more than 1e6 numbers per day. The numbers do not have to be used in sequence.
So far we have thought of the following:
1) Machine A uses odd numbers, machine B uses even numbers.
Pros: no shared state.
Cons: a machine might run out of numbers when there are plenty left. If a machine restarts, it will reuse previously used numbers.
2) Machine A countr from 0 to 1e6, machine B from 1e6 to 0.
Pros: no shared state. Garantees that all available numbers will be consumed before running into problems.
Cons: doesn't scale to more than two machines. Same problem when a machine restarts.
What do you think? Is there a magic algorithm that will fulfill our requirements without needing to write anything to disk?
No number should be used twice in the same day, even if a machine restarts.
Since you don't want to use any persistent state, this suggests to me that the number must depend on the time somehow. That is the only way in which the algorithm can tell two distinct startups apart. Can't you just use a combination (node, timestamp) for sufficiently fine timestamps, instead of your numbers?
Why not just have a small serivce that hands out IDs upon request? This scales to more than one machine, and doesn't require a change to the client if you need to change the ID allocation algorithm. This is rather simple to implement and quite easy to maintain going forward.
I really think the best way would be to have some machine that hands out numbers on requests (maybe even number ranges if you want to avoid too many queries) that wrote things out to disk.
If you're really against it, you could be really clever with method 1 if you can gaurantee the rate at which numbers are consumed. For example the machine could use the current time to determine where in its range to begin. I.E. if it's noon, begin at the middle of my range. This could be tweaked if you can put an upper limit on the amount of numbers generated per second (or generic time interval). This still has the problem of wasted tags and is pretty convoluted just to avoid writing a single number to disk.