If you had a table that had 100,000,000 email addresses (example) and you want to store them securely but you don't want to take a huge hit with performance when you retrieve them, how would you go about storing them?
Encryption algorithms are designed to be fast. If you have decided that encrypting your data is important (and personal information such as email address is important as #Marc B pointed out) then the question becomes "which algorithm has the right performance for my use case?"
I'd suggest you look at how much throughput you are expecting and test it. Use an industry standard algorithm and see if it meets your needs. Whatever you do, don't roll your own encryption. Crypto is hard to get right so trust the experts.
I'd suggest you start with AES/CBC/PKCS5. Store the ciphertext as a blob in the database. Key storage is always tricky and has been discussed on StackOverflow a fair bit so with any luck you can get advice on that aspect fairly easily. If you get sufficient throughput with that algorithm then you're done. If not, look around for a faster algorithm. I recommend http://bouncycastle.org as a good crypto library. It has a bunch of algorithms implemented so you can experiment and find one that meets your needs.
Did I mention that you shouldn't roll your own encryption? Good.
Related
I am planning on generating a set of public/private keys from a deterministic identifying piece of information from a person and was planning on using fingerprints.
My question, therefore, is: what is the output of a fingerprint scanner? Is there any deterministic output I could use, or is it always going to be a matter of "confidence level"? i.e. Do I always get a "number" which, if matched exactly to the database, will allow access, or do I rather get a number which, if "close enough" to the stored value on the database, allows access, based on a high degree of confidence, rather than an exact match?
I am quite sure the second option is the answer but just wanted to double-check. Is there any way to get some sort of deterministic output? My hope was to re-generate keys every time rather than actually storing fingerprint data. That way a wrong fingerprint would simply generate a new and useless key.
Any suggestions?
Thanks in advance.
I would advise against it for several reasons.
The fingerprints are not entirely deterministic. As suggested in #ImSimplyAnna answer, you might 'round' the results in order to have more chances to obtain a deterministic result. But that would significantly reduce the number of possible/plausible fingerprints, and thus not meet the search space size requirement for a cryptographic algorithm. On top of it, I suspect the entropy of such result to be somehow low, compared to the requirements of modern algorithm which are always based on high quality random numbers.
Fingerprints are not secret, we expose them to everyone all the time, and they can be revealed to an attacker at any time, and stored in a picture using a simple camera. A key must be a secret, and the only place we know we can store secrets without exposing them is our brain (which is why we use passwords).
An important feature for cryptographic keys is the possibility to generate new one if there is a reason to believe the current ones might be compromised. This is not possible with fingerprints.
That is why I would advise against it. Globally, I discourage anyone (myself included) to write his/her own cryptographic algorithm, because it is so easy to screw them up. It might be the easiest thing to screw up, out of all the things you could write, because attacker are so vicicous!
The only good approach, if you're not a skilled specialist, is to use libraries that are used all around, because they've been written by experts on the matter, and they've been subject to many attacks and attempts to break them, so the ones still standing will offer much better levels of protection that anything a non specialist could write (or basically anything a single human could write).
You can also have a look at this question, on the crypto stack exchange. They also discourage the OP in using anything else than a battle hardened algorithm, or protocol.
Edit:
I am planning on generating a set of public/private keys from a
deterministic identifying piece of information
Actually, It did not strike me at first (it should have), but keys MUST NOT be generated from anything which is not random. NEVER.
You have to generate them randomly. If you don't, you already give more information to the attacker than he/she wants. Being a programmer does not make you a cryptographer. Your user's informations are at stake, do not take any chance (and if you're not a cryptographer, you actually don't stand any).
A fingerprint scanner looks for features where the lines on the fingerprint either split or end. It then calculates the distances and angles between such features in an attempt to find a match.
Here's some more reading on the subject:
https://www.explainthatstuff.com/fingerprintscanners.html
in the section "How fingerprints are stored and compared".
The source is the best explanation I can find, but looking around some more it seems that all fingerprint scanners use some variety of that algorithm to generate data that can be matched.
Storing raw fingerprints would not only take up way more space on a database but also be a pretty significant security risk if that information was ever leaked, so it's not really done unless absolutely necessary.
Judging by that algorithm, I would assume that there is always some "confidence level". The angles and distances will never be 100% equal between scans, so there has to be some leeway to make sure a match is still found even if the finger is pressed against the scanner a bit harder or the finger is at a slightly different angle.
Based on this, I'd assume that generating a key pair based on a fingerprint would be possible, if you can figure out a way to make similar scans result in the same information. Simply rounding the angles and distances may work, but may introduce cases where two different people generate the same key pairs, or cases where different scans of the same fingerprint have a high chance of generating several different keys.
I was researching about how MD5 is known to have collisions, So its not secure enough. I am looking for some hashing algorithm that even super computers will take time to break.So can you tell me what hashing algorithm will keep my passwords safe for like next coming 20 years of super computing advancement.
Use a key derivation function with a variable number of rounds, such as bcrypt.
The passwords you encrypt today, with a hashing difficulty that your own system can handle without slowing down, will always be vulnerable to the faster systems of 20 years in the future. But by increasing the number of rounds gradually over time you can increase the amount of work it takes to check a password in proportion with the increasing power of supercomputers. And you can apply more rounds to existing stored passwords without having to go back to the original password.
Will it hold up for another 20 years? Difficult to say: who knows what crazy quantum crypto and password-replacement schemes we might have by then? But it certainly worked for the last 10.
Note also that entities owning supercomputers and targeting particular accounts are easily going to have enough power to throw at it that you can never protect all of your passwords. The aim of password hashing is to mitigate the damage from a database leak, by limiting the speed at which normal attackers can recover passwords, so that as few accounts as possible have already been compromised by the time you've spotted the leak and issued a notice telling everyone to change their passwords. But there is no 100% solution.
As someone else said, what you're asking is practically impossible to answer. Who knows what breakthroughs will be made in processing power over the next twenty years? Or mathematics?
In addition you aren't telling us many other important factors, including against which threat models you aim to protect. For example, are you trying to defend against an attacker getting a hold of a hashed password database and doing offline brute-forcing? An attacker with custom ASICs trying to crack one specific password? Etc.
With that said, there are things you can do to be as secure and future-proof as possible.
First of all, don't just use vanilla cryptographic hash algorithms; they aren't designed with your application in mind. Indeed they are designed for other applications with different requirements. For one thing, they are fast because speed is an important criterion for a hash function. And that works against you in this case.
Additionally some of the algorithms you mention, like MD5 or SHA1 have weaknesses (some theoretical, some practical) and should not be used.
Prefer something like bcrypt, an algorithm designed to resist brute force attacks by being much slower than a general purpose cryptographic hash whose security can be “tuned” as necessary.
Alternatively, use something like PBKDF2 which is. Designed to run a password through a function of your choice a configurable number of times along with a salt, which also makes brute forcing much more difficult.
Adjust the iteration count depending on your usage model, keeping in mind that the slower it is, the more security against brute-force you have.
In selecting a cryptographic hash function for PBKDF, prefer SHA-3 or, if you can't use that, prefer one of the long variants of SHA-2: SHA-384 or SHA-512. I'd steer clear of SHA-256 although I don't think there's an issue with it in this scenario.
In any case, use the largest possible and best salt you can; I'd suggest that you use a good cryptographically secure PRNG and never use a salt less than 64 bits (note: that I am talking about the length of the salt generated, not the value returned).
Will these recommendations help 20 years down the road? Who knows - I'd err on the side of caution and say "no". But if you need security for that long a timeframe, you should consider using something other than passwords.
Anyways, I hope this helps.
Here are two pedantic answers to this question:
If P = NP, there is provably no such hash function (and vice versa, incidentally). Since it has not been proven that P != NP at the time of this writing, we cannot make any strong guarantees of that nature.
That being said, I think it's safe to say that supercomputers developed within the next 20 years will take "time" to break your hash, regardless of what it is. Even if it is in plaintext some time is required for I/O.
Thus, the answer to your question is both yes and no :)
We have some data that we are trying to synchronize between N machines and a centralized server, and I'm looking for a way to do this that is relatively efficient and robust.
Looking around, it appears that this is called a "set reconciliation problem". It's good to have a label for it, but searching on that turns up a lot of fairly academic work, which is at times a bit difficult to gauge in terms of its usefulness for our data, which is best described as contact lists in terms of its properties: objects (people) with multiple fields that do get updated, but not that often.
Our system involves a central server and machines connected to it. The central server, ideally, is the 'good' copy. A feature that's nice to have also, is the ability to force the machines to resend by tweaking something on the server.
So far, my thinking is along the lines of a UUID for each object and something like a version or timestamp (per object and or per collection of objects?) to use to tell which data to attempt to synchronize... but my thinking is still a bit fuzzy, and I thought asking would probably lead to a better solution than trying to invent this on my own.
It is not easy, and the perfect solution is academical. So you are on the good track.
You can craft a sync algorithm for your own problem, relaxing some of the requirements of the general solution.
I delivered a presentation on these topics at the last JsDay in Italy.
Here are my slides: http://www.slideshare.net/matteocollina/operational-transformation-12962149
Let me know if they help you, or if you need some assistance.
Is there an (easy) way to encrypt data so that it takes a certain amount of cpu hours to decrypt it? Maybe a series of encryptions with short key lengths, a variable one-way function or anything?
It's probably not of great use, but how would this encryption scheme be called and are there tools for it?
edit:
To get no varying results for the brute force break time, shouldn't I use many rounds with an xor-feedback?
I just came up with this algo (for a symmetric block cipher with equal value and key length)... maybe it's non-sense
round 1
create a zero-block
create a random-block-1
encipher value:zero-block with key:random-block1 => gives lock-output-1
round 2
create a zero-block
create a random-block-2
encipher value:zero-block with key:random-block2 => gives temp
xor temp with random-block-1 => gives lock-output-2
and so on
The xor operation with random-block-1 would be there so that the unlock routine will have to find random-block-1 before it can start brute forcing on lock-output-2.
lock-output-1 + lock-output-2 .. lock-output-N would be the complete lock-output. When the unlock routine has found N key-blocks that each give zero on all lock-output blocks, it can use the N key-blocks as a whole to decipher the actual data.
Then I'd also need a formula to calculate how many rounds would give a maximum variation of e.g. 10% for the wanted amount of CPU hours.
I guess there must exist a simmilar algorithm out there.
The concept is called timed commitment, as defined by Boneh and Naor. The data you want to encrypt is said to be committed by one party (which I call the sender), such that another party (the receiver) may, at some tunable cost, recover the data.
The method described by Boneh and Naor is quite more advanced than what you suggest. Their timed commitment scheme has the three following properties:
Verifiable recovery: the sender is able to convince the receiver that he really did commit a proper value which the receiver will be able to recover by applying a substantial but feasible amount of CPU muscle to it.
Recovery with proof: once the recovery has been done, it is verifiable efficiently: a third party wishing to verify that the recovered value is indeed the one which was committed, can do so efficiently (without applying hours of CPU to it).
Immunity against parallel attacks: the recovery process cannot benefit from having access to a thousand PC: one cannot go much faster than what can be done with a single CPU.
With these properties, a timed commitment becomes a worthwhile tool in some situations; Boneh and Naor mainly discuss contract signing, but also honesty preserving auctions and a few other applications.
I am not aware of any actual implementation or even a defined protocol for timed commitments, beyond the mathematical description by Boneh and Naor.
You can encrypt it normally, and release just enough information about the key such that a brute force attack will take X CPU hours.
No, you can't do that reliably, because
the attacker could rent a powerful computer (a computing cloud for example) and use it for highly parsllel much faster attack
so far computers become faster and faster as time passes - what took a day yesterday might take one minute in two years
Well, to know the amount of CPU hours for any kind of decryption, it does not really matter how the encryption takes place. Instead you would have to make sure
what decryption algorithm the decrypter will use (perhaps a non-so-far invented one?)
which implementation of that algorithm he will use
which CPU/hardware he will use.
Each of these 3 parameters can make a difference in speed of at least a factor 1000 or more.
A cryption algorithm is considered as cracked when someone found a way to get the password faster than a brute force attack (in average).
It's the case for some algorithms like MD5 so make sure you pick one algorithm that isn't cracked (yet)
For other algorithms, even if they are not cracked, they are still vulnerable to brute force attacks... it might take a while but everything that is crypted might be decrypted... it's only a question of time and resources.
If a guy have a huge zombie computer farm working for him around the world, it might take few hours to crack something that would take years for a guy with a single laptop.
If you want a maximum of security, you can mix a couple of existing cryption algoritm with custom algorithm of your own. Someone can still try to crack your data, but most likely, unless you are dealing with national top secret data, it will probably never append.
It is relative, a computer is going to decrypt fast depending on its computing power, and the selected algorithm to encrypt depends on the data you want to protect, so with a normal computer a good encryption algorithm an average computer takes its time to decrypt cause there always is a price for good things, but i recommend you Elliptic curve cryptography cause it has power to encrypt and its time to be decrypted is very good, you can take a look on it.
that is what i can say about it.-
As many will know, one-way encryption is a handy way to encrypt user passwords in databases. That way, even the administrator of the database cannot know a user's password, but will have to take a password guess, encrypt that with the same algorithm and then compare the result with the encrypted password in the database. This means that the process of figuring out the password requires massive amounts of guesses and a lot of processing power.
Seeing that computers just keep getting faster and that mathematicians are still developing these algorithms, I'm wondering which one is the most secure considering modern computing power and encryption techniques.
I've been using MD5 almost exclusively for years now, and I'm wondering if there's something more I should be doing. Should I be contemplating a different algorithm?
Another related question: How long should a field typically be for such an encrypted password? I must admit that I know virtually nothing about encryption, but I'm assuming that an MD5 hash (as an example) can be longer and would presumably take more processing power to crack. Or does the length of the field not matter at all, provided that the encrypted password fits in it in the first place?
Warning: Since this post was written in 2010, GPUs have been widely deployed to brute-force password hashes. Moderately-priced GPUs
can run ten billion MD5s per second. This means that even a
completely-random 8-character alphanumeric password (62 possible
characters) can be brute forced in 6 hours. SHA-1 is only slightly
slower, it'd take one day. Your user's passwords are much weaker, and
(even with salting) will fall at a rate of thousands of passwords per
second. Hash functions are designed to be fast. You don't want this
for passwords. Use scrypt, bcrypt, or PBKDF-2.
MD5 was found to be weak back in 1996, and should not be used anymore for cryptographic purposes. SHA-1 is a commonly used replacement, but has similar problems. The SHA-2 family of hash functions are the current replacement of SHA-1. The members of SHA-2 are individually referred to as SHA-224, SHA-256, SHA-384, and SHA-512.
At the moment, several hash functions are competing to become SHA-3, the next standardised cryptographic hashing algorithm. A winner will be chosen in 2012. None of these should be used yet!
For password hashing, you may also consider using something like bcrypt. It is designed to be slow enough to make large scale brute force attacks infeasible. You can tune the slowness yourself, so it can be made slower when computers are becoming faster.
Warning: bcrypt is based on an older two-way encryption algorithm, Blowfish, for which better alternatives exist today. I do not think that the cryptographic hashing properties of bcrypt are completely understood. Someone correct me if I'm wrong; I have never found a reliable source that discusses bcrypt's properties (other than its slowness) from a cryptographic perspective.
It may be somewhat reassuring that the risk of collisions matters less for password hashing than it does for public-key cryptography or digital signatures. Using MD5 today is a terrible idea for SSL, but not equally disastrous for password hashing. But if you have the choice, simply pick a stronger one.
Using a good hash function is not enough to secure your passwords. You should hash the passwords together with salts that are long and cryptographically random. You should also help your users pick stronger passwords or pass phrases if possible. Longer always is better.
Great question! This page is a good read. In particular, the author claims that MD5 is not appropriate for hashing passwords:
The problem is that MD5 is fast. So are its modern competitors, like SHA1 and SHA256. Speed is a design goal of a modern secure hash, because hashes are a building block of almost every cryptosystem, and usually get demand-executed on a per-packet or per-message basis.
Speed is exactly what you don’t want in a password hash function.
The article then goes on to explain some alternatives, and recommends Bcrypt as the "correct choice" (his words, not mine).
Disclaimer: I have not tried Bcrypt at all. Consider this a friendly recommendation but not something I can back up with my own technical experience.
To increase password strength you should use a wider variety of symbols. If you have 8-10 characters in the password it becomes pretty hard to crack. Although making it longer will make it more secure, only if you use numeric/alphabetic/other characters.
SHA1 is another hashing (one way encryption) algorithm, it is slower, but is has a longer digest. (encoded messsage) (160 bit) where MD5 only has 128 bit.
Then SHA2 is even more secure, but it used less.
salting the password is always an extra level of defense
$salt = 'asfasdfasdf0a8sdflkjasdfapsdufp';
$hashed = md5( $userPassword . $salt );
Seeing that computers just keep getting faster and that mathematicians are still developing these algorithms
RSA encryption is secure in that it relies on a really big number being hard to factor. Eventually, computers will get fast enough to factor the number in a reasonable amount of time. To stay ahead of the curve, you use a bigger number.
However, for most web sites, the purpose of hashing passwords is to make it inconvenient for someone with access to the database to read the password, not to provide security. For that purpose, MD5 is fine1.
The implication here is that if a malicious user gains access to your entire database, they don't need the password. (The lock on the front door won't stop me from coming in the window.)
1 Just because MD5 is "broken" doesn't mean you can just reverse it whenever you want.
Besides being a cryptographically secure one-way function, a good hash function for password protection should be hard to brute force - i.e. slow by design. scrypt is one of the best in that area. From the homepage:
We estimate that on modern (2009) hardware, if 5 seconds are spent computing a derived key, the cost of a hardware brute-force attack against scrypt is roughly 4000 times greater than the cost of a similar attack against bcrypt (to find the same password), and 20000 times greater than a similar attack against PBKDF2.
That said, from commonly available hash functions, doing a few thousand of iterations of anything from the SHA family is pretty reasonable protection for non-critical passwords.
Also, always add a salt to make it impossible to share effort for brute forcing many hashes at a time.
NIST is currently running a contest to select a new hashing algorith, just as they did to select the AES encryption algorithm. So the answer to this question will likely be different in a couple of years.
You can look up the submissions and study them for yourself to see if there's one that you'd like to use.