I need to deterministically extend public and private keys (ethereum). How can I do this, who can suggest an algorithm?
The blog post linked in the question comments mentions risks related to generating vanity addresses from a deterministic source of randomness.
A vanity address is an address that contains a predefined set of bytes, for example starts with 4 zeros. These vanity tools use a loop to generate a random private key, its corresponding address, and validate if the address matches the criteria. If the address doesn't match the criteria, it runs another iteration of the loop.
Simplified example, looking for address starting with 0x1
Generate private key 0x123, its corresponding address is 0x456. Doesn't match the criteria, trying again.
Generate private key 0xd3f, its corresponding address is 0xb7c. Doesn't match the criteria, trying again.
Generate private key 0x837, its corresponding address ix 0x154. Success.
Some of these vanity tools generate private keys by using a deterministic seed generating algorithm, so it is possible (with some brute force guessing) to recreate the specific private keys that were used and reclaim access to these addresses.
A solution that is considered safe is to let each user generate their own pseudo-random seed. A great example is tracking the movements of their mouse cursor as you can see on this page.
Or generally, accept the seed from multiple sources - e.g. /dev/random from multiple physical machines.
Edit: Generating multiple addresses from an xpub:
const HDNode = ethers.utils.HDNode.fromExtendedKey("xpub661MyMwAqRbcGXSTUwJkaLaZFy6xdVk8XuRuSEFThgCPFEjrh9WAyREtMhuHA7XMMDXXTm41GfjKDCsLJ2fGfVp1ACyuDKwsShNcGWenHET");
const address0 = HDNode.derivePath("0").address;
const address1 = HDNode.derivePath("1").address;
Related
I'm writing some code in C# that matches a pattern from the subject and then ingests the email. To initialize my datastore, I go through the current Microsoft.Office.Interop.Outlook.Table.
while (!table.EndOfTable)
{
Row row = table.GetNextRow();
string entryId = row["EntryID"].ToString();
this.SaveInXML(entryId, row);
}
It seems pretty simple. Well, I also have an event (Application.ItemLoad) that I'm watching, too. I notice that in the event the MailItem's EntryID is completely different than the Table's EntryID. In fact, the string lengths are not even the same (See example below). Why is this? Shouldn't they be the same? The item has not moved folders, so I'd assume it's the same. Thank you, all.
Example code:
NameSpace ns = this.Folder.Application.GetNamespace("MAPI");
var mi = ns.GetItemFromID("EF0000003E65593F1D361C44AFBFA24E6F365D6E04782F00") as MailItem;
string entryId = mi.EntryID;
System.Diagnostics.Debug.WriteLine("EF0000003E65593F1D361C44AFBFA24E6F365D6E04782F00");
System.Diagnostics.Debug.WriteLine(entryId);
// Output Produced:
// EF0000003E65593F1D361C44AFBFA24E6F365D6E04782F00
// 000000003E65593F1D361C44AFBFA24E6F365D6E0700CC348F1AD97A224B9898503750437E4700000000010C0000CC348F1AD97A224B9898503750437E470000F59160590000
//
// Notice that the second WriteLine isn't even remotely close to the EntryID that I requested.
Entry identifiers come in two types: short-term and long-term.
Short-term entry identifiers are faster to construct, but their uniqueness is guaranteed only over the life of the current session on the current workstation.
Long-term entry identifiers have a more prolonged lifespan. Short-term entry identifiers are used primarily for rows in tables and entries in dialog boxes, whereas long-term entry identifiers are used for many objects such as messages, folders, and distribution lists.
Use the MailItem.EntryID property if you need to get a long-term entry identifiers.
Entry identifiers cannot be compared directly because one object can be represented by two different binary values. Use the NameSpace.CompareEntryIDs method to determine whether two entry identifiers represent the same object.
As Eugene noted, you have two kinds of entry ids - long term and short term. Even for long-term entry ids, they can be different depending on how the item was opened. Long term entry ids always start with "00000000". Short term entry ids can only be used in the current MAPI session and therefore should not be persisted to be used across different sessions.
You must treat entry id as black boxes and never compare them directly - always use Namespace.CompareEntryIDs.
I have two go services, let's call them A and B. B holds an RSA key pair, while A only knows the public key. I want them to know if they agree on some value V.
I want to do this by having B encrypt encrypt V using the public key and have A do a comparison, but all the crytpo/rsa functions take an RNG which adds entropy and makes each hash of V different. That means I can't compare the hashes.
Is there a function in the go standard library that will deterministicly hash V?
Note: I can achieve this by using a fresh RNG seeded with the same value everytime I hash V, but I want to be able to compute this hash from other languages and that would tie me to Go's RNG.
I want to do this by having B encrypt encrypt V using the public key and have A do a comparison…
You're using the wrong primitive.
If you want the owner of a private key to prove that they have some data, have them Sign that data. The recipient can Verify that signature using the public key.
Use the SignPSS and VerifyPSS methods to do this. The signature will not be deterministic, but it doesn't need to be -- the recipient will still be able to verify it.
Take a look at the docs for EncryptOAEP:
The random parameter is used as a source of entropy to ensure that encrypting the same message twice doesn't result in the same ciphertext.
So the random data does not affect the reader's ability to decrypt the message with only the public key. The cipher text bytes will be different each time you encrypt the same value, which is a good thing.
Take a look at the examples on Encrypt/Decrypt OAEP in those docs. It should be sufficient to get you moving the right direction.
I want to implement an algorithm / function (a kind of license algorithm), let's call it F1, which returns a fixed key for a full range of input values, say mac addresses.
And then implement the inverse function (say F2), which should return true if the 2 inputs are:
- The previously generated key from F1
- A valid input mac address that belongs to the address range provided previously to F1
To be more specific about why I want to do:
I'm implementing a SW code for BT (Bluetooth) platforms (chips / modules) that should work only on a specific range of BT mac addresses.
The user should provide us with a range of BT addresses (i.e. start address - end address), then using F1 we should provide him with a license key (with a fixed value for all his BT addresses).
Then during MP (mass production) of the BT modules, the module manufacturer shall store this key in the EEPROM on the module.
At module power-up, the function F2 should read the value of the key in EEPROM (as input 1), & read the BT mac address of the module (as input 2), then should return true if this address belongs to the BT address range used for generating the key with F1.
Any ideas?
Where should I start reading? Any keywords?
Hmm, let's say you have two different overlapping ranges, R1 and R2, with two different keys provided by your F1 function, K1 and K2. Then, F2 should return true if it reads either K1 or K2 for a MAC address in the overlap, right? In that case, you need F1 as a reverse-encryptable algorithm that takes start&end or start&mask (if approppriate) and turns it into a "passphrase" by a certain key, then F2 will use either the same key or private key to decrypt the stored value, parse the output vs MAC and return if it's inside the range. So, I think one of the solutions is encryption/decryption via asymmethric keys, one you use as private and for encrypting ranges, the other you provide as public (as it'll be stored explicitly somewhere to be used in F2) to your manufacturer to decrypt your encrypted range.
I was reading a tutorial on how to salt a key to make your encryption secure, but couldn't make much of it. I don't know a lot about cryptography, and need some help. I am using commoncrypto to encrypt files, and am done, except for the fact that it isn't secure...
This is what I have:
- (NSData *)AES256EncryptWithKey:(NSString *)key
{
// 'key' should be 32 bytes for AES256, will be null-padded otherwise
char keyPtr[kCCKeySizeAES256 + 1]; // room for terminator (unused)
bzero( keyPtr, sizeof( keyPtr ) ); // fill with zeroes (for padding)
NSLog(#"You are encrypting something...");
// fetch key data
[key getCString:keyPtr maxLength:sizeof( keyPtr ) encoding:NSUTF8StringEncoding];
NSUInteger dataLength = [self length];
//See the doc: For block ciphers, the output size will always be less than or
//equal to the input size plus the size of one block.
//That's why we need to add the size of one block here
size_t bufferSize = dataLength + kCCBlockSizeAES128;
void *buffer = malloc( bufferSize );
size_t numBytesEncrypted = 0;
CCCryptorStatus cryptStatus = CCCrypt( kCCEncrypt, kCCAlgorithmAES128, kCCOptionPKCS7Padding,
keyPtr, kCCKeySizeAES256,
NULL /* initialization vector (optional) */,
[self bytes], dataLength, /* input */
buffer, bufferSize, /* output */
&numBytesEncrypted );
if( cryptStatus == kCCSuccess )
{
//the returned NSData takes ownership of the buffer and will free it on deallocation
return [NSData dataWithBytesNoCopy:buffer length:numBytesEncrypted];
}
free( buffer ); //free the buffer
return nil;
}
If someone can help me out, and show me exactly how I would implement salt, that would be great! Thanks again!
dYhG9pQ1qyJfIxfs2guVoU7jr9oniR2GF8MbC9mi
Enciphering text
AKA jumbling it around, to try and make it indecipherable. This is the game you play in cryptography. To do this, you use deterministic functions.
Encrypting involves using a function which takes two parameters: usually a short, fixed length one, and an arbitrary length one. It produces output the same size as the second parameter.
We call the first parameter the key; the second, the plaintext; and the output, the ciphertext.
This will have an inverse function (which is sometimes the same one), which has the same signature, but given instead ciphertext will return the plaintext (when using the same key).
Obviously the property of a good encryption function is that the plaintext is not easily determinable from the ciphertext, without knowing the key. An even better one will produce ciphertext that is indistinguishable from random noise.
Hashing involves a function which takes one parameter, of arbitrary size, and returns an output of fixed size. Here, the goal is that given a particular output, it should be hard to find any input that will produce it. It is a one-way function, so it has no inverse. Again, it's awesome if the output looks completely random.
The problem with determinism
The above is all very well and good, but we have a problem with our ultimate goals of indecipherability when designing implementations of these functions: they're deterministic! That's no good for producing random output.
While we can design functions that still produce very random-looking output, thanks to confusion and diffusion, they're still going to give the same output given the same input. We both need this, and don't like it. We would never be able to decipher anything with a non-deterministic crypto system, but we don't like repeatable results! Repeatable means analysable... determinable (huh.). We don't want the enemy to see the same two ciphertexts and know that they came from the same input, that would be giving them information (and useful techniques for breaking crypto-systems, like rainbow tables). How do we solve this problem?
Enter: some random stuff inserted at the start.
That's how we defeat it! We prepend (or sometimes better, append), some unique random input with our actual input, everytime we use our functions. This makes our deterministic functions give different output even when we give the same input. We send the unique random input (when hashing, called a salt; when encrypting, called an Initialisation Vector, or IV) along with the ciphertext. It's not important whether the enemy sees this random input; our real input is already protected by our key (or the one-way hash). All that we were actually worried about is that our output is different all the time, so that it's non-analysable; and we've achieved this.
How do I apply this knowledge?
OK. So everybody has their app, and within it their cryptosystem protecting parts of the app.
Now we don't want to go reinventing the wheel with cryptosystems (Worst. Idea. Ever.), so some really knowledgable people have already come up with good components that can build any system (i.e, AES, RSA, SHA2, HMAC, PBKDF2). But if everyone is using the same components, then that still introduces some repeatability! Fortunately, if everyone uses different keys, and unique initial random inputs, in their own cryptosytem, they should be fine.
Enough already! Talk about implementation!
Let's talk about your example. You're wanting to do some simple encryption. What do we want for that? Well, A) we want a good random key, and B) we want a good random IV. This will make our ciphertext as secure as it can get. I can see you haven't supplied a random IV - it's better practice to do so. Get some bytes from a [secure/crypto]-random source, and chuck it in. You store/send those bytes along with the ciphertext. Yes, this does mean that the ciphertext is a constant length bigger than the plaintext, but it's a small price to pay.
Now what about that key? Sometimes we want a remember-able key (like.. a password), rather than a nice random one that computers like (if you have the option to just use a random key - do that instead). Can we get a compromise? Yes! Should we translate ASCII character passwords into bytes to make the key? HELL NO!
ASCII characters aren't very random at all (heck, they generally only use about 6-7 bits out of 8). If anything, what we want to do is make our key at least look random. How do we do this? Well, hashing happens to be good for this. What if we want to reuse our key? We'll get the same hash... repeatability again!
Luckily, we use the other form of unique random input - a salt. Make a unique random salt, and append that to your key. Then hash it. Then use the bytes to encrypt your data. Add the salt AND the IV along with your ciphertext when you send it, and you should be able to decrypt on the end.
Almost done? NO! You see the hashing solution I described in the paragraph above? Real cryptographers would call it amateurish. Would you trust a system which is amateurish? No! Am I going to discuss why it's amateurish? No, 'cus you don't need to know. Basically, it's just not REALLY-SUPER-SCRAMBLED enough for their liking.
What you need to know is that they've already devised a better system for this very problem. It's called PBKDF2. Find an implementation of it, and [learn to] use that instead.
Now all your data is secure.
Salting just involves adding a random string to the end of the input key.
So generate a random string of some length:
Generate a random alphanumeric string in cocoa
And then just append it to the key using:
NSString *saltedKey = [key stringByAppendingString:salt];
Unless salt is being used in a different way in the article you read this should be correct.
How a random salt is normally used:
#Ca1icoJack is completely correct in saying that all you have to do is generate some random data and append it to the end. The data is usually binary as opposed to alphanumeric though. The salt is then stored unencrypted alongside each hashed password, and gets concatenated with the user's plaintext password in order to check the hash every time the password gets entered.
What the is the point of a SALT if it's stored unencrypted next to the hashed password?
Suppose someone gets access to your hashed passwords. Human chosen passwords are fairly vulnerable to being discovered via rainbow tables. Adding a hash means the rainbow table needs to not only include the values having any possible combination of alphanumeric characters a person might use, but also the random binary salt, which is fairly impractical at this point in time. So, basically adding a salt means that a brute force attacker who has access to both the hashed password and the salt needs to both figure how how the salt was concatenated to the password (before or after, normally) and brute force each password individually, since readily available rainbow tables don't include any random binary data.
Edit: But I said encrypted, not hashed:
Okay, I didn't read very carefully, ignore me. Someone is going to have to brute-force the key whether it's salted or not with encryption. The only discernable benefit I can see would be as that article says to avoid having the same key (from the user's perspective) used to encrypt the same data produce the same result. That is useful in encryption for different reasons (encrypted messages tend to have repeating parts which could be used to help break the encryption more easily) but the commenters are correct in noting that it is normally not called an salt in this instance.
Regardless, the trick is to concatenate the salt, and store it alongside each bit of encrypted data.
My problem:
I'm looking for a way to represent a person's name and address as an encoded id. The id should contain only alpha-numeric characters, be collision-proof, and be represented in a smallest number of characters possible. My first thought was to simply use a cryptographic hash function like MD5 or SHA1, but this seems like overkill (security isn't important - doesn't need to be one-way) and I'd prefer to find something that would produce a shorter id. Does anyone know of an existing algorithm that fits this problem?
In other words, what is the best way to implement the following function so that the return value is the same consistently for the same input, collisions are unlikely, and ids are less than 20 characters?
>>> make_fake_id(fname = 'Oscar', lname = 'Grouch', stnum = '1', stname = 'Sesame', zip = '12345')
N1743123734
Application Context (for those that are interested):
This will be used for a record linkage app. Given an input name and address we search a very large database for the best match and return the database id and other data (how we do this is not important here). If there isn't a match I need to generate this psuedo/generated/derived id from the search input (entity's name and address data). Every search record should result in an output record with either a real (the actual database id resulting from a match/link) or this generated psuedo/generated/derived id. The psuedo id will be prefixed with a character (e.g. N) to differentiate it from a real id.
I know you said no to MD5 and SHA1, but I think you should consider them anyway. As well as being well studied hashing algorithms, the length gives you more protection against possible collisions. No hash is collision-proof, but the cryptographic ones generally are less collision-prone than something you couuld come up with yourself.
Use a cryptographic hash for its collision resistance, not its other qualities
Use as many bytes from the hash as you want (truncate)
convert to alpha-numeric characters
You can also truncate the alpha-numeric string instead of the hash
An easy way to do this: hash the data, encode in base64, remove all non-alpha-numeric characters, truncate.
N_HASH_CHARS = 11
import hashlib, re
def digest(name, address):
hash = hashlib.md5(name + "|" + address).digest().encode("base64")
alnum_hash = re.sub(r'[^a-zA-Z0-9]', "", hash)
return alnum_hash[:N_HASH_CHARS]
How many alpha-numeric characters should you keep? Each character gives you around 5.95 bits of entropy (log(62,2)). 11 characters give you 65.5 bits of entropy, which should be enough to avoid a collision for the first 2**32.7 users (about 7 billion).
A good solution is somewhat dependent on your application. Do you know how many users and what the set of all users is? If you provide more details you would get better help.
I agree with the other poster suggesting serial numbers. OTOH, if you really, really really want to do something else:
Create a SHA1 hash from the data, and store it in a table with a serial number field.
Then, when you get the data, calculate the hash, look it up on the table, get the serial, and that's your id. If it's not on the table, insert it.
I wonder whether you intend to "assign" these ids to the users? If so, I would expect your users to hate anything that you propose; who would want a user id of "AAAAA01"?
So, if these ids are visible to the user, then you should just let them pick what they like and check them for uniqueness (easy). If they are not visible to the user (e.g., internal primary key), then just generate them sequentially using an appropriate technique such as an Oracle Sequence or SQL Server AutoNumber (also easy).
If these ids are an attempt to detect a user that is registering more than once, then I would agree that you should consider a cryptographic hash followed by a full comparison of the registration data (name, address, etc.). However, to be usable, you will need to translate the data into a canonical form (standardized letter case, whitespace, canonical street address, etc.) before computing the hash or making the comparison. Otherwise, you will mismatch based on trivial differences.
EDIT: Now that I understand the problem space better based on your edits, I think that it is highly unlikely that your algorithm (so far) will catch most matches. Beyond my suggestion to canonicalize the inputs, I recommend that you consider an approach that results in a ranked list of a handful of possible matches (to be resolved by a human if possible) rather than an all-or-nothing attempt at a single match. In other words, I recommend a search approach rather than a lookup approach.
Is that feasible in your situation?
Well, if there's more than one person at the same address with the same name, you're toast here, (w/o adding code to detect this and add a discriminator of some kind).
but assuming that issue is not, then the street address and zip code portion of the full addresss is sufficient to guaranteee uniqueness there, so adding enough data from the name should take care of the issue...
Do you have access to a database, or other persistence mechanism, where you could generate and maintain key values for each address? Then keep the address and individual entities in two keyed dictionary structures, where the key is autogenerated for each new distinct address, person encountered... and then use the autogenerated alpha-numeric key...
You could use AAAAA01 for first person at first address,
AAAAA02 for second person at first address,
AAAAB07 for the seventh resident at the second adresss, etc.
If you donlt have any way to generate and maintain these entity-Key mappings then you need to use the full street address/Zip and fullNAme, or a hash value of the same, although the Hash value approach has a smnall chance of generating duplicates...