One hard working day I noticed that GUIDs I've been generating with usual .NET's Guid.NewGuid() method had the same number 4 in the beginning of the third block:
efeafa5f-fe21-4ab4-ba82-b9eefd5fa225
480b64d0-6762-4afe-8496-ac7cf3292898
397579c2-a4f4-4611-9fda-16e9c1e52d6a
...
There were ten of them appearing on the screen once a second or so. I've kept my eye on this pattern right after the fifth GUID. Finally, the last one had the same four bits inside and I've decided that I'm a lucky guy. I went home and felt that the whole world is opened for such an exceptional person as me. Next week I found a new work, cleaned my room and made a call to my parents.
But today I've faced the same pattern again. Thousand times. And I don't feel the Chosen One anymore.
I've googled it and now I know about UUID and a canonical format with 4 reserved bits for version and 2 for variant.
Here's a snippet to experiment with:
static void Main(string[] args)
{
while (true)
{
var g = Guid.NewGuid();
Console.WriteLine(BitConverter.ToString(g.ToByteArray()));
Console.WriteLine(g.ToString());
Console.ReadLine();
}
}
But still there is one thing I don't understand (except how to go on living). Why do we need these reserved bits? I see how it can harm - exposing internal implementation details, more collisions (still nothing to worry about, but one day...), more suicides - but I don't see any benefit. Can you help me to find any?
It is so that if you update the algorithm you can change that number. Otherwise 2 different algorithms could produce the exact same UUID for different reasons, leading to a collision. It is a version identifier.
For example, consider a contrived simplistic UUID format:
00000000-00000000
time - ip
now suppose we change that format for some reason to:
00000000-00000000
ip - time
This could generate a collision when a machine with IP 12.34.56.78 generates a UUID using the first method at time 01234567, and later a second machine with IP 01.23.45.67 generates a UUID at time 12345678 using the newer method. But if we reserve some bits for a version identifier, this cannot possibly cause a collision.
The value 4 specifically refers to a randomly generated UUID (therefore it relies on the miniscule chance of collisions given so many bits) rather than other methods which could use combinations of the time, mac address, pid, or other sorts of time & space identifiers to guarantee uniqueness.
See here for the relevant spec: https://www.rfc-editor.org/rfc/rfc4122#section-4.1.3
Related
I am looking for alternative to GUIDs for key generation in a distributed app. For example supposed I have Bob, James, and Jack all running a bug tracking application on their desktop where they can do thing like create bug tickets ala JIRA, or Bugzilla ... etc. When a ticket is created it is assigned a number such as T-1, T-2, T-3, T-4 ... etc. Tickets need to have a stable ID and should be creatable without having to consult a central server.
I understand that this is what GUID's are really good for but it in my case displaying a GUID in a UI is ugly people can't just copy and paste it and discuss it on a phone call, I really want integers or some sort of short string that is easy to talk about read in one glance .. etc.
Is there a way to use the bitcoin block chain as some sort of counter?
You may evaluate the approach taken by git. They use sha1 hash of commit information. And then abbreviate IDs are allowed which are much shorter and easier to read\transfer manually.
Having the number of bugs in your tracker is not going to reach millions that should be sufficient. Once it is you'll just need a longer abbreviation.
There seem to be plenty info around on how git calculates hash IDs and abbreviates them.
If I recall correctly how UUIDv1 works - it's "just" putting together the mac address and a very exact timestamp + maybe some additional integer. As your mac address should be unique (unless you've fiddled with it) and there are only so many UUIDs one computer can generate within a nano second, the resulting ID will be unique.
This is a very general and uninformed way to create IDs. If you'd implement a version of it yourself for your specific use case you could get much smaller IDs.
Assuming you can identify each node with a bug tracking system with a simple and unique string - for instance "Bob", "James", "Jack" - and you can create unique continuous integers within each node, you could combine those two and have IDs like "Bob-1", "James-12", ...
As you can see, actually there has to be again one central point, which will assign the unique strings, however depending on the number of nodes and how long they stay within the system, this could be as well done just by a human being.
The additional disadvantage (or advantage, depends how you look at it) of this approach (as well as of UUIDv1) would be, that you'd know where the ticket has been created as well as order of the tickets within one system.
I'm looking for a good way to anonymize data in my database while retaining the capability of aggregating / summarizing statistical information.
As an example, let's say I want to track clicks by IP address per hour but I don't actually want to store the IP address.
My first thought is to store only a hash (e.g. SHA-256) of the IP. However, I'm not sure this provides sufficient security. If an attacker got ahold of our database and was determined to reverse our anonymization they could generate a rainbow table of IP's and get back the real IP info fairly easily.
My next thought was to add a static prefix to the IP before hashing (e.g. 192.168.1.10 becomes MY_SECRET_STRING-192.168.1.10). Of course, if the attacker finds the static prefix then it is essentially useless.
I've been searching for sound solutions to this problem and I haven't found anything I really like so far. Are there any well known methods for anonymizing data like this?
If someone have access to your salt and database I would say it's almost impossible (if not impossible) to keep them from creating some sort of collision table and "cracking" your hashes. The only option you have is to make their job hard/expensive.
Using a static salt is a bad idea though, this since the whole point of a salt is to prevent an attacker from generating a rainbow table for all your records. The uniqueness is what makes a salt a good salt , this since the purpose of the salt is to make each hash unique regardless if the original content was the same as another record (thus obligating an attacker to brute-force each row to figure out its content).
Also something that is worth noticing is that salts don't need to be secret, so you can just store your salt in an additional column.
There is this nice article about salting and hashing if you have any doubt about the topic.
The problem with the described approach is that in the end, just like an attacker, you won't be able to tell which of the rows are the same IPs.
One potential solution I can see if you really really need to implement this is having a table where you store the IPs + click count, and then every 1 hour have a process to anonymize the data by simply replacing all the IPs/hash from the last hour with a good RANDOM value. This in the end means that you will only be able to group the clicks per hour without knowing the actual IP, but, please notice two things:
Although an attacker will never be able to figure out the past data, you will have 1 hour worth of data that is not anonymized at any given time. Meaning that an attacker could "spy" on you and store this information over time which could become a much bigger problem than "we just leaked 1 hour worth of data".
You won't be able to tell the same IP apart between each hour. For example: if IP 127.0.0.1 did 3 click from 17:00 to 18:00 and the same IP did 6 clicks from 18:00 to 19:00 you wouldn't able to tell that 127.0.0.1 did 9 clicks from 17:00 to 19:00.
Also to make the hourly non-anonymized IP a bit more hard to crack you could have a function that takes an IP and generates unique salt and then caches that unique salt for that IP till the next hour, meaning that each IP would have its own unique salt every hour. This way the attacker would have to calculate a new rainbow table for each row every hour and you could still figure out what IP row to increment|create.
Why, yes, there are. The most well-known is called "salting". Basically, instead of adding a static string to all of the plain texts, you add a unique string to each one. This string is randomly or algorithmically generated and stored separately. It doesn't make a single hash any harder to crack, but it prevents use of tables to crack multiple hashes. See the wikipedia article on Salt(crytography).
That being said, I think that a one-way hash of the IP is sufficient. An attacker would have to crack each IP address. No matter what method you use, once an IP is cracked then all of the records for that IP will be exposed. But cracking one IP doesn't help with any of the others.
Let's look at the following Go code:
package main
import "fmt"
type Vertex struct {
Lat, Long float64
}
var m map[string]Vertex
func main() {
m = make(map[string]Vertex)
m["Bell Labs"] = Vertex{
40.68433, 74.39967,
}
m["test"] = Vertex{
12.0, 100,
}
fmt.Println(m["Bell Labs"])
fmt.Println(m)
}
It outputs this:
{40.68433 74.39967}
map[Bell Labs:{40.68433 74.39967} test:{12 100}]
However, if I Change one minor part of the test vertex declaration, by moving the right "}" 4 spaces, like so:
m["test"] = Vertex{
12.0, 100,
}
.. then the output changes to this:
{40.68433 74.39967}
map[test:{12 100} Bell Labs:{40.68433 74.39967}]
Why the heck does that little modification affect the order of my map?
Map "order" depends on the hash function used. The hash function is randomized to prevent denial of service attacks that use hash collisions. See the issue tracker for details:
http://code.google.com/p/go/issues/detail?id=2630
Map order is not guaranteed according to the specification. Although not done in current go implementations, a future implementation could do some compacting during GC or other operation that changes the order of a map without the map being modified by your code. It is unwise to assume a property not defined in the specification.
A map is an unordered group of elements of one type, called the element type, indexed by a set of unique keys of another type, called the key type.
A map shouldn't always print its key-element in any fixed order:
See "Go: what determines the iteration order for map keys?"
However, in the newest Go weekly release (and in Go1 which may be expected to be released this month), the iteration order is randomized (it starts at a pseudo-randomly chosen key, and the hashcode computation is seeded with a pseudo-random number).
If you compile your program with the weekly release (and with Go1), the iteration order will be different each time you run your program.
It isn't exactly spelled out like that in the spec though (Ref Map Type):
A map is an unordered group of elements of one type, called the element type, indexed by a set of unique keys of another type, called the key type.
Actually, the specs do spell it out, but in the For statement section:
The iteration order over maps is not specified and is not guaranteed to be the same from one iteration to the next.
If map entries that have not yet been reached are deleted during iteration, the corresponding iteration values will not be produced.
If map entries are inserted during iteration, the behavior is implementation-dependent, but the iteration values for each entry will be produced at most once.
If the map is nil, the number of iterations is 0.
This has been introduced by code review 5285042 in October 2011:
runtime: random offset for map iteration
The go-nuts thread points out:
The reason that "it's there to stop people doing bad stuff" seemed particularly weak.
Avoiding malicious hash collisions makes a lot more sense.
Further the pointer to the code makes it possible in development to revert that behaviour in the case that there is an intermittent bug that's difficult to work through.
To which Patrick Mylund Nielsen replies:
Dan's note was actually the main argument why Python devs were reluctant to adopt hash IV randomization--it broke their unit tests! PHP ultimately chose not to do it at all, and instead limited the size of the http.Request header, and Oracle and others didn't think it was a language problem at all.
Perl saw the problem and applied a fix similar to Go's which was included in Perl 5.8.1 in 2003.
I might be wrong, but I think they were the only ones to actually care then when this paper was presented: "Denial of Service via Algorithmic Complexity Attacks", attacking hash tables.
(Worst-case hash table collisions)
For others, this, which became very popular about a year ago, was a good motivator:
"28c3: Effective Denial of Service attacks against web application platforms (YouTube video, December 2011)", which shows how a common flaw in the implementation of most of the popular web
programming languages and platforms (including PHP, ASP.NET, Java, etc.) can be (ab)used to force web application servers to use 99% of CPU for several minutes to hours for a single HTTP request.
This attack is mostly independent of the underlying web application and just
relies on a common fact of how web application servers typically work..
If you've used GoToMeeting, that's the type of ID I want. I'd like it to be random so that it obfuscates the number of items being tracked and short, so that it's easy to reference manually; UUIDs are way too long. I'd like to avoid hitting persistent storage merely for performance reasons, but I can't think of any other way to avoid collisions. Is 9 digits enough to do something time-based?
In response to questions:
I'm building a ticket-tracking application. This ID would be used as the primary key for a table, but it would be needed before the record is persisted which would result in an extra database call that I'd like to avoid if possible.
I'd like to keep it at a 9 digit int. I consider a UUID to be too long because people are going to have to reference the ID manually (via email, phone, etc.).
I'm thinking of using the time of generation somehow. Since time is always ticking on forward, it would continually limit the set of potential IDs, excluding those that had already been generated.
One way is to take a unique number or string (like a random UUID) then calculate a fixed-length digest (such as MD5 or SHA-1) and/or encode it in a higher base (like base64) to shorten it further.
Git does something similar where it generates a sha numbers for commits (and other events) and then the user can references the numbers manually in order to lookup those commits. The trick they used is that the user doesn't have to enter the whole string in order to find the correct event, they simply have to enter a long enough string that it doesn't collide with any other commit currently in the repository. In general this only require 5 or so hex digits for relatively large repositories.
I am trying to create a unique CD-KEY to put in our product's box, just like a normal CD-KEY found in standard software boxes that users use to register the product.
However we are not selling software, we are selling DNA collection kit for criminal and medical purposes. Users will receive a saliva collection kit by mail with the CD-KEY on it and they will use that CD-KEY to create an account on our website and get their results. The results from the test will be linked to the CD-KEY. This is the only way that we will have to link the results to the patients. It is therefore important that it does not fail :)
One of the requirements would be that the list of CD-KEYs must be sufficiently "spread" apart so that there is no possibility of someone entering an incorrect CD-KEY and still having it approved for someone else kit, thereby mixing up two kits. That could cost us thousands of dollars in liability.
For example, it cannot be a incremental sequence of numbers such as
00001
00002
00003
...
The reason is that if someone receives the kit 00002, but registers it as 000003 by accident, then his results will be matched to someone else. So it must be like credit card numbers... Unless a valid sequence is entered, your chances of randomly hitting a valid number is 1 in a million...
Also, we are selling over 50,000 kits annually to various providers (who will generate their own CD-KEYS using our algorithm) so we cannot maintain a list of all previously issued CD-KEYS to check for duplicate. The algorithm must generate unique CD-KEYs.
We also require the ability to verify that the CD-KEY is valid using a quick check algorithm, so that we can inform the user if the code he enters is invalid. This leaves out many hashing or MD5 algorithms I believe. And it cannot be a 128 bit because, who would take that time to type it out on the computer screen?
So far this is what I was thinking the final CD-KEY structure would look like
(4 char product code) - (4 char reseller code) - (12 char unique, verifiable CD-KEY)
Ex. 384A - GTLD - {4565 - FR54 - EDF3}
To insure the uniqueness of the KEYS, I could include the current date (20090521) as part of the source. We wont generate unique keys more than once a week, so this value changes often enough for the purpose of unique initial value.
What possible algorithm can I use to generate the unique keys?
Create the strings <providername>000001, <providername>000002, etc. or whatever and encrypt them with a public key, and that's your "CD-KEY" that the user enters. Decrypt the CD-KEY with the private key and validate that when decrypted you get a valid string with a valid provider name.
Credit Card numbers use the Luhn algorithm you might want to look at something similar to that.
I use SeriousBit Ellipter link for software protection but I don't see any reason you could generate a group of unique keys each week and us the library to verify the key validity when entered into your web site. You can also encode optional services into the key allow you to control how the sample is processed from the key (that's if you have different service levels).
As it uses an encrypted method of key generation in the first place and it's relatively cheap, it's certainly worth a look I would say.
I finally settled for a cd-key of this form
<TIMESTAMP>-<incremented number>-<8 char MD5 hash>-<checksumdigit>
I used the mod 11 ISBN checksum digit algorithm.
Generate GUID and catenate a random number to it. GUID is guaranteed to be unique and random number will make it improbable to hit a code accidentally. Just don't modify the GUID in any way or you might compromise the uniqueness.
http://msdn.microsoft.com/en-us/library/aa475087.aspx