I need to programmatically (using Python) validate whether the signature generated by using HSM RnG is actually random.
Didn't found much content online, hence any help would be appreciated !!
How do I validate the randomness of ECDSA signature generated using HSM?
As Squeamish indicated in the comments, there is not really any way of making sure that the outcome of a random number generator is really random. For instance, it could just be the output of Pi at one particular location. Generally, you'll have to trust the FIPS or Common Criteria certification of the platform, but note that neither one is airtight.
Furthermore, you can also rely on the fact that HSM companies want to make money. One way certain to lose (all) marketshare is to show that you cannot be trusted. Multiple companies, including RSA Labs, already found this out to their detriment. If you don't fully trust a HSM company, then don't buy anything off of them; but note that there is no such thing as perfect security in a practical sense.
It may be possible to do some simple experiments. If you ask for signature generation using the same data then you should never get the same signature (r and s values for ECDSA). This shows that the output of the RNG at least doesn't repeat. Testing this - once - with several connections / sessions might be a good idea.
You could also verify and then concatenate all the resulting x values (used for the final comparison) and then run a randomness test (such as the diehard suite) over it.
Finally, if you don't trust the entropy source of the HSM's RNG, note that you can add additional seeds to a HSM. Please check the documentation of your HSM how it reacts to adding / mixing seeds.
This article gives an overview of methods for checking the randomness of some given data:
There are many practical measures of randomness for a binary sequence.
These include measures based on statistical tests, transforms, and
complexity or a mixture of these. A well-known and widely-used
collection of tests was the Diehard Battery of Tests, introduced by
Marsaglia; this was extended to the TestU01 suite by L'Ecuyer and
Simard. The use of Hadamard transform to measure randomness was
proposed by S. Kak and developed further by Phillips, Yuen, Hopkins,
Beth and Dai, Mund, and Marsaglia and Zaman.
A standardized test suite is defined in the NIST Special Publication 800-22.
Also check out this discussion.
Related
We have a script to stand up a new web server at my job. The script involves creating a number of accounts to run services, app pools etc.
We needed to create a password for each of these users -- i.e. generate a 32-or-so-character ASCII string to be used as a logon password.
We had a disagreement as to whether one ought to use a cryptographically-secure PRNG for this job, or whether using a non-cryptographically secure PRNG (with a time-dependent seed) would suffice (we work in .NET, so the concrete decision was between generating strings with System.Random and using a RNGCryptoServiceProvider -- however, this is not a language-specific issue).
Must one use cryptographically-secure randomness for generating passwords, or is a sensibly-seeded plain PRNG sufficient?
In many cases, an attacker can easily recover the state of a (non-cryptographic) random number generator from a few output values – without knowing anything about the seed. After that, it's trivial to predict all future and all previous random numbers.
How many outputs are required for this depends on the algorithm. In the case of a linear congruential generator, such as Java's java.util.Random, the state can be recovered from two outputs. For Mersenne Twister, used in PHP and Python among others, you need to obtain 624 outputs. I'm not familiar with .NET, but I'd think it's a similar story.
There is no complex math involved at all. See for yourself:
Linear congruential generator, part 1
Linear congruential generator, part 2
Mersenne Twister, part 1
Mersenne Twister, part 2
Conclusion: Use a cryptographically secure random number generator for anything that has to do with security.
Theoretically, if an attacker knows your exact algorithm for producing random passwords (say, he got his hands on your code), and he knows that you seeded your PRNG with system time, then he could reproduce the passwords generated at each instant in time to whatever resolution your system timer has, reducing his brute-force password search by orders of magnitude.
System timer has essentially zero entropy. If you seed a RNG with something like /dev/random (on Linux/ OSX) or CryptGenRandom (Windows), then the fact that the PRNG itself is not CS probably won't matter, because an attacker would have to get more than one password's worth of data to be able to crack it. But then if you're already using a CSPRNG to seed a PRNG, you might as well just use it to create the password in the first place.
Non-CS PRNGs are fast, and great for things like game simulations and Monte Carlo integration, but security passwords, nonces, keys, and such should really use secure algorithms.
That said, security, as always, is not a "yes/no" question--it's always a matter of cost/benefit. The right choices always depend on the value of what you're protecting, the cost of your efforts to protect it, your likely attackers, the cost of failure, and so on, so there's no single right choice for every situation.
So here is my stupid quesion:
PGP/GPGP can used to sign text, others use public key to verify them.
Let's say, asymmetric cryptographic algorithm deal with space.
Are there any algorithm that can deal with time?
E.g. Sign a document at 2011-10-10 10:10:10, others can verify this document is really, truely signed at 2011-10-10 10-10-10.
I know there are maybe solutions based on a common trusted time source, but is it possible that a pure algorithm solution exists?
Or perhaps some amazing dedicated piece of hardware can do this? (E.g. half-life of certain well-known material, etc.)
This is out of the scope of a cryptographic algorithm, because there's no such thing as absolute time (and if there were, there's no way to measure it from inside a computer). What you're asking for requires a trusted third party - the wikipedia article on Trusted Timestamping describes this. Another approach is one similar to that used in bitcoin, where distributed agreement is created on a sequence of events; the order of events is decided by the order in which they became part of the consensus. This provides a relative measure of time as a sequence of events, not a wallclock time.
OpenPGP signs data (not text), as does any other signing mechanism. Data has no special meaning besides the one humans put into it. "3.1415926" can be treated as value of Pi expressed in decimals or as a randomizer salt or as a password to one's home computer, you choose.
Consequently, time is just some large number, relative to starting point agreed by people (eg. 30 Dec 1899 in OLE time). You can sign time as you would sign any other data. The question is where you get the value. As Nick Johnson pointed, you need a trusted time source. This can be some NTP server (then you need to sign the value) or some TSP server (it produces signed values if you trust it).
In general, you can read RFC 3161 about Timestamping protocol which relies on trusted third parties. While there exist other schemes and they are great, TSP protocol described in RFC 3161 became the most widely used approach nowadays.
I was wondering about new trends in cryptography. Which algorithms are new ? Which are improved and which died beacuse of the time that past ?
For example EEC ( Elliptic Curve Cryptography ) is quite new approach, but definitly not the only one. Could you name some of them ?
ECC actually originates from the 80's; it is not exactly new.
In the context of asymmetric encryption and digital signatures, there has been in the last few years much research on pairings. Pairings open up the next level. Conceptually: symmetric cryptography is for problems with one entity (all entities share a secret key, thus they are the "same" entity), asymmetric cryptography is for problems with two entities (the signer and the verifier), and pairings are a tool for protocols with three entities (for instance, electronic cash, where there are the bank, the merchant and the buyer). The only really practical pairings found so far use elliptic curves, but with a much higher dose of mathematics.
As for more classical asymmetric encryption and signatures, there has been some work on many other algorithms, such as HFE, which seems especially good with regards to signature sizes, or lattice-based cryptography. This is still quite new. It takes some time (say a dozen years or so) before a newly created algorithm becomes mature enough to be standardized.
Following work by Bellovin and Merritt in 1992, some Password Authenticated Key Exchange protocols have been described. These protocols are meant to allow for password-based mutual authentication immune to offline dictionary attacks (i.e. the protocols imply that an attacker, even if actively masquerading as one of the parties, cannot obtain enough information to test passwords at his leisure; each guess from the attacker must go through an interaction with one of the entities who knows the password). IEEE group P1363 is working on writing standards on that subject.
In the area of symmetric encryption, the AES has been a bit "final". A few stream ciphers have been designed afterwards (stream ciphers are supposed to provide better performance, at the cost of less generic applicability); some were analyzed by the eSTREAM project. There has been quite some work on encryption modes, which try to combine symmetric encryption and integrity checks in one efficient system (see for instance GCM and CWC).
Hash functions have been a hot subject lately. A bunch of old hash functions, including the famous MD5, were broken in 2004. There is an ongoing competition to determine the next American standard hash function, codenamed SHA-3.
An awful lot of work has been done on some implementation issues, in particular side-channel leaks (how secret data leaks through power consumption, timing, residual electro-magnetic emissions...) and how to block them.
The main problem of contemporary cryptography is not finding algorithms but whole concepts and approaches for different situations (but of course the algorithms are continually improved too).
We have today
Symmetric algorithms (AES)
Asymmetric algorithms (RSA, ECC)
Key exchange (Diffie-Hellman-Key-Exchange, Shamir's no key protocol)
Secret sharing (intersection of n-dimensional planes)
Cryptographic hash functions (SHA)
Some have proven insecure and were improved
DES due to a much to small key-space
MD5
and some are broken
Merke/Hellman knapsack cryptosystem
Monoalphabetic subsitution
Naive Vigenère
Which particular algorithm is chosen is often a question of available resources (elliptic curves need smaller keys that RSA algorithm for comparable safety) or just of standardization (as tanascius pointed out, there are competitions for such algorithms). Totally new trends usually start when a whole class of cryptosystems has been shown vulnerable against a specific attack (man-in-the-middle, side-channel) or scientific progress is made (quantum cryptography).
Of course, there is also steganography which doesn't attempt so conceal the content but the existence of a secret message by hiding it in other documents.
Currently there is the NIST hash function competition running with the goal to find a replacement for the older SHA-1 and SHA-2 functions. So this is about a cryptographic hash function.
You can have a look at the list of the accepted algorithms for round two, and you can get whitepapers to all of the algorithms taking part there.
I am not up-to-date, but I doubt that there are any completely new approaches for the algorithms.
EDIT: Well, one of the candidates was the Elliptic curve only hash, but it is listed under "Entrants with substantial weaknesses" ^^
People have covered most other things; I'll talk about design:
Block ciphers: Traditional block cipher (e.g. DES) uses a Feistel structure. There's a move to a more general S-P network (Rijndael, Serpent) which are easier to parallelize, and cipher modes which support parallelization (CS, GCM) and efficient authenticated encryption (CS, GCM, IGE/BIGE).
Hashes: Traditional hashes use the obvious Merkle–Damgård construction. This has some undesirable properties:
Length extension is trivial (to some extent this is mitigated by proper finalization)
Several attacks on the collision resistance (multicollisions, Joux 2004; "expandable message" attacks, Kelsey 2005; herding attacks, Kelsey 2006).
In the popular Davies–Meyer mode used in MD{4,5} and SHA-{0,1,2} — hi=hi−1⊕E(mi, hi−1)) — every message block has a fixed-point D(mi,0). This makes the expandable message attack much easier.
The SHA-3 contest notice (070911510–7512–01) also suggests randomized hashing and resistance against length extension (currently achieved with HMAC for MD5/SHA-1/SHA-2) and parallelization (a few hashes specify a tree hashing mode).
There's a general trend towards avoiding potential table lookups (e.g. Threefish, XTEA) to mitigate cache timing attacks.
Closed. This question needs to be more focused. It is not currently accepting answers.
Closed 6 years ago.
Locked. This question and its answers are locked because the question is off-topic but has historical significance. It is not currently accepting new answers or interactions.
Given the extremely high requirements for unpredictability to prevent casinos from going bankrupt, what random number generation algorithm and seeding scheme is typically used in devices like slot machines, video poker machines, etc.?
EDIT: Related questions:
Do stateless random number generators exist?
True random number generator
Alternative Entropy Sources
There are many things that the gaming sites have to consider when choosing/implementing an RNG. Without due dilligence, it can go spectacularly wrong.
To get a licence to operate a gaming site in a particular jurisdiction usually requires that the RNG has been certified by an independent third-party. The third-party testers will analyse the source code and run statistical tests (e.g. Diehard) to ensure that the RNG behaves randomly. Reputable poker sites will usually include details of the certification that their RNG has undergone (for example: PokerStars's RNG page).
I've been involved in a few gaming projects, and for one of them I had to design and implement the RNG part, so I had to investigate all of these issues. Most poker sites will use some hardware device for entropy, but they won't rely on just hardware. Usually it will be used in conjunction with a pseudo-RNG (PRNG). There are two main reasons for this. Firstly, the hardware is slow, it can only extract a certain number of bits of entropy in a given time period from whatever physical process it is monitoring. Secondly, hardware fails in unpredictable ways that software PRNGs do not.
Fortuna is the state of the art in terms of cryptographically strong PRNGs. It can be fed entropy from one or more external sources (e.g. a hardware RNG) and is resilient in the face of attempted exploits or RNG hardware failure. It's a decent choice for gaming sites, though some might argue it is overkill.
Pokerroom.com used to just use Java's SecureRandom (they probably still do, but I couldn't find details on their site). This is mostly good enough, but it does suffer from the degrees of freedom problem.
Most stock RNG implementations (e.g. Mersenne Twister) do not have sufficient degrees of freedom to be able to generate every possible shuffle of a 52-card deck from a given initial state (this is something I tried to explain in a previous blog post).
EDIT: I've answered mostly in relation to online poker rooms and casinos, but the same considerations apply to physical video poker and video slots machines in real world casinos.
For a casino gaming applications, I think the seeding of the algorithm is the most important part to make sure all games "booted" up don't run through the same sequence or some small set of predictable sequences. That is, the source of entropy leading to the seed for the starting position is the critical thing. Beyond that, any good quality random number generator where each bit position as has a ~50/50 probability of being 1/0 and the period is relatively long would be sufficient. For example, something like the Mersenne twister PRNG has such properties.
Using cryptographically secure random generators only becomes important when the actual output of the random generator can be viewed directly. For example, if you were monitoring each number actually generated by the number generator - after viewing many numbers in the sequence - with a non-cryptographic generator information about that sequence can lead to establishing information about all the internal state of the generator. At this point, if you know what the algorithm looks like, you would be able to predict future numbers and that would be bad. The cryptographic generator prevents that reverse engineering back to the internal state so that predicting future numbers becomes "impossible".
However, in the case of a casino game, you would (or should) have no visibility to the actual numbers being generated under the hood. Each time a random number is generated - say a 32-bit number - that number will be used then, for example, mod 52 for a deck shuffling algorithm....no where in that process do you have any idea what numbers were being generated by the algorithm to shuffle that deck. That is, most of the bits of "randomness" is just being thrown out and even the ones being used you have no visibility to. Therefore, no way to reverse engineer the state.
Getting back to a true source of entropy to seed the whole process, that is the hard part. See the Wikipedia entry on entropy for some starting points on techniques.
As an aside, if you did want cryptographically sequence random numbers from a "regular" algorithm, a simple approach is to take a few random numbers in sequence, concatenate them together and then run something like MD5 or SHA-1 on them and the result is just as random and also cryptographically secure. That is, you just made your own "secure" random number generator.
We've been using the Protego R210-USB TRNG (and the non-usb version before that) as random seed generators in casino applications, with java.security.SecureRandom
on top. We had The Swedish National Laboratory of Forensic Science perform a separate audit of the R210, and it passed without a flaw.
You probably need a cryptographically secure pseudo-random generator. There are a lot of variants. Google "Blum-Blum-Shub", for example.
The security properties of these pseudo-random generators will generally be that, even when the attacker can observe polynomially many outputs from such generators, it won't be feasible to guess the next output with a probability much better than random guessing. Also, it is not feasible to distinguish the output of such generators from truly random bits. The security holds even when all the algorithms and parameters are known by the attacker (except for the secret seed).
The security of the generators is often measured with respect to a security parameter. In the case of BBS, it is the size of the modulus. This is no different from other crypto stuff. For example, RSA is secure only when the key is long enough.
Note that, the output of such generators may not be uniform (in fact, can be far away from uniform in statistical sense). But since no one can distinguish the two distributions without infinite computing power, these generators will suffice in most applications that require truly random bits.
Bear in mind, however, that these cryptographically secure pseudo-random generators are usually slow. So if speed is indeed a concern, less rigorous approaches may be more relevant, such as using hash functions, as suggested by Jeff.
Casino slot machines generate random numbers continuously at very high speed and use the most recent result(s) when the user pulls the lever (or hits the button) to spin the reels.
Even a simplistic generator can be used. Even if you knew the algorithm used, you cannot observe where in the sequence it is because nearly all the results are discarded. If somehow you did know where it was in the sequence, you'd have to have millisecond or better timing to take advantage of it.
Modern "mechanical reel" machines use PRNGs and drive the reels with stepper motors to simulate the old style spin-and-brake.
http://en.wikipedia.org/wiki/Slot_machine#Random_number_generators
http://entertainment.howstuffworks.com/slot-machine3.htm
Casinos shouldn't be using Pseudo-random number generators, they should be using hardware ones: http://en.wikipedia.org/wiki/Hardware_random_number_generator
I suppose anything goes for apps and offshore gambling nowadays, but all these other answers are incomplete, at least for Nevada Gaming Control Board licensed machines, which I believe the question is originally about.
The technical specifications for RNGs licensed in Nevada for gaming purposes are laid out in Regulation 14.040(2).
As of May 24, 2012, here is a summary of the rules a RNG must follow:
Static seeds cannot be used. You have to seed the RNG using a millisecond time source or other true entropy source which has no external readout anywhere on the machine. (This helps to reduce the incidence of "magic number" attacks)
The RNG must continue generating numbers in its sequence at least 100 times per second when the game is not being played. (This helps to avoid timing attacks)
RNG outputs cannot be reused; they must be used exactly once if at all and then thrown away.
Multi-system cabinets must use a separate RNG and separate seed for each game.
Games that use RNGs for helping to choose numbers on behalf of the player (such as Lotto Quick Pick) must use a separate RNG for that process.
Games must not roll RNGs until they are actually needed in a game. (i.e. you need to wait until the player chooses to deal or spin before generating RNGs)
The RNG must pass a 95% confidence chi-squared test based on 10,000 trials as apart of a system test. It must display a warning if this test fails, and it must disable play if it fails twice in a row.
It must remember, and be able to report on, the last 10 test results as described in 7.
Every possible game outcome must be generatable by the RNG. As a pessimal example, linear congruential generators don't actually generate every possible output in their range, so they're not very useful for gaming.
Additionally, your machine design has to be submitted to the gaming commission and it has to be approved, which is expensive and takes lots of time. There are a few third-party companies that specialize in auditing your new RNG to make sure it's random. Gaming Laboratories publishes an even stricter set of standards than Nevada does. They go into much greater detail about the limitations of hardware RNGs, and Nevada in particular likes to see core RNGs that it's previously approved. This can all get very expensive, which is why many developers prefer to license an existing previously-approved RNG for new game projects.
Here is a fun list of random number generator attacks to keep you up late at night.
For super-nerds only: the source for most USB hardware RNGs is typically an avalanche diode. However, the thermal noise produced by this type of diode is not quantum-random, and it is possible to influence the randomness of an avalanche diode by significantly lowering the temperature.
As a final note, someone above recommended just using a Mersenne Twister for random number generation. This is a Bad Idea unless you are taking additional entropy from some other source. The plain vanilla Mersenne Twister is highly inappropriate for gaming and cryptographic applications, as described by its creator.
If you want to do it properly you have to get physical - ERNIE the UK national savings number picker uses a shot noise in Neon tubes.
I for sure have seen a german gambling machine that was not allowed to be ran commercially after a given date, so I suppose it was a PNRG with a looong one time pad seed list.
Most poker sites use hardware random number generators. They will also modify the output to remove any scaling bias and often use 'pots' of numbers which can be 'stirred' using entropic events (user activity, serer i/o events etc). Quite often the resultant numbers just index pre-generated decks (starting off as a sorted list of cards).
I've seen it mentioned in many places that randomness is important for generating keys for symmetric and asymmetric cryptography and when using the keys to encrypt messages.
Can someone provide an explanation of how security could be compromised if there isn't enough randomness?
Randomness means unguessable input. If the input is guessable, then the output can be easily calculated. That is bad.
For example, Debian had a long standing bug in its SSL implementation that failed to gather enough randomness when creating a key. This resulted in the software generating one of only 32k possible keys. It is thus easily possible to decrypt anything encrypted with such a key by trying all 32k possibilities by trying them out, which is very fast given today's processor speeds.
The important feature of most cryptographic operations is that they are easy to perform if you have the right information (e.g. a key) and infeasible to perform if you don't have that information.
For example, symmetric cryptography: if you have the key, encrypting and decrypting is easy. If you don't have the key (and don't know anything about its construction) then you must embark on something expensive like an exhaustive search of the key space, or a more-efficient cryptanalysis of the cipher which will nonetheless require some extremely large number of samples.
On the other hand, if you have any information on likely values of the key, your exhaustive search of the keyspace is much easier (or the number of samples you need for your cryptanalysis is much lower). For example, it is (currently) infeasible to perform 2^128 trial decryptions to discover what a 128-bit key actually is. If you know the key material came out of a time value that you know within a billion ticks, then your search just became 340282366920938463463374607431 times easier.
To decrypt a message, you need to know the right key.
The more possibly keys you have to try, the harder it is to decrypt the message.
Taking an extreme example, let's say there's no randomness at all. When I generate a key to use in encrypting my messages, I'll always end up with the exact same key. No matter where or when I run the keygen program, it'll always give me the same key.
That means anyone who have access to the program I used to generate the key, can trivially decrypt my messages. After all, they just have to ask it to generate a key too, and they get one identical to the one I used.
So we need some randomness to make it unpredictable which key you end up using. As David Schmitt mentions, Debian had a bug which made it generate only a small number of unique keys, which means that to decrypt a message encrypted by the default OpenSSL implementation on Debian, I just have to try this smaller number of possible keys. I can ignore the vast number of other valid keys, because Debian's SSL implementation will never generate those.
On the other hand, if there was enough randomness in the key generation, it's impossible to guess anything about the key. You have to try every possible bit pattern. (and for a 128-bit key, that's a lot of combinations.)
It has to do with some of the basic reasons for cryptography:
Make sure a message isn't altered in transit (Immutable)
Make sure a message isn't read in transit (Secure)
Make sure the message is from who it says it's from (Authentic)
Make sure the message isn't the same as one previously sent (No Replay)
etc
There's a few things you need to include, then, to make sure that the above is true. One of the important things is a random value.
For instance, if I encrypt "Too many secrets" with a key, it might come out with "dWua3hTOeVzO2d9w"
There are two problems with this - an attacker might be able to break the encryption more easily since I'm using a very limited set of characters. Further, if I send the same message again, it's going to come out exactly the same. Lastly, and attacker could record it, and send the message again and the recipient wouldn't know that I didn't send it, even if the attacker didn't break it.
If I add some random garbage to the string each time I encrypt it, then not only does it make it harder to crack, but the encrypted message is different each time.
The other features of cryptography in the bullets above are fixed using means other than randomness (seed values, two way authentication, etc) but the randomness takes care of a few problems, and helps out on other problems.
A bad source of randomness limits the character set again, so it's easier to break, and if it's easy to guess, or otherwise limited, then the attacker has fewer paths to try when doing a brute force attack.
-Adam
A common pattern in cryptography is the following (sending text from alice to bob):
Take plaintext p
Generate random k
Encrypt p with k using symmetric encryption, producing crypttext c
Encrypt k with bob's private key, using asymmetric encryption, producing x
Send c+x to bob
Bob reverses the processes, decrypting x using his private key to obtain k
The reason for this pattern is that symmetric encryption is much faster than asymmetric encryption. Of course, it depends on a good random number generator to produce k, otherwise the bad guys can just guess it.
Here's a "card game" analogy: Suppose we play several rounds of a game with the same deck of cards. The shuffling of the deck between rounds is the primary source of randomness. If we didn't shuffle properly, you could beat the game by predicting cards.
When you use a poor source of randomness to generate an encryption key, you significantly reduce the entropy (or uncertainty) of the key value. This could compromise the encryption because it makes a brute-force search over the key space much easier.
Work out this problem from Project Euler, and it will really drive home what "lots of randomness" will do for you. When I saw this question, that was the first thing that popped into my mind.
Using the method he talks about there, you can easily see what "more randomness" would gain you.
A pretty good paper that outlines why not being careful with randomness can lead to insecurity:
http://www.cs.berkeley.edu/~daw/papers/ddj-netscape.html
This describes how back in 1995 the Netscape browser's key SSL implementation was vulnerable to guessing the SSL keys because of a problem seeding the PRNG.