Suppose array index according to hashing function for string "temp" is 155 and location 155 is pre-occupied then location 156 is tried. Suppose location 156 is available, so this entry is saved in location 156 instead of 155. Later I find another string "another_temp", which maps to location 156. Again this is saved in next available location 157.
The question is : Later if I want to find out location of "another_temp", how I would know that it is 157 not 156, even though hash function returned 156?
Thanks.
You need to compare the key, not just the hash code. In a hash table, you anyway need to store the key ("temp" and "another_temp" in your case). It's not enough to just store and compare the hash value, because hash values are not unique.
There are a few problems with open addressing. One is: what to do after deleting an entry? Usually you store a special "deleted" marker. Another problem is: if there is a collision, should you increment the hash code? You will find more details in Wikipedia.
Related
I understood the process to shorten the URL with base 62 at How do I create a URL shortener?.
Steps given are
Think of an alphabet we want to use. In your case, that's [a-zA-Z0-9]. It contains 62 letters.
Take an auto-generated, unique numerical key (the auto-incremented id of a MySQL table for example).
For this example, I will use 12510 (125 with a base of 10).
Now you have to convert 12510 to X62 (base 62)
My question is why not just create unique numerical key and return it ? What is the advantage of concerting numerical key > Base 62 > then Finally some alphanumeric number ?
Is it because final alphanumeric number will be much smaller than unique numerical key ?
Yes. The idea is to make it short and usable in a URL. A number in base 62 will use fewer characters than the same number in base 10. Notice also that URL shorteners use short hosts, such as g.co.
I can see you understand that, yes, a number written in base 62 takes less characters than a number in base 10 just like a number in base 10 takes less characters than a number in base 2 (e.g. 0101 is 3 characters longer than just '5').
So, I'll answer specifically "Why".
Sometimes a link is shortened to be more visually pleasing. A company worried about their public perception likely doesn't want their links to look like an error code due to how long they are so they resort to shortening. That's why some url shortening services allow you to add your own "vanity url" which customizes the domain name, so that a link can be shortened and branded.
Other times a link is shortened to minimize character count when working with constraints, like Twitter. For example, at my company we shortened the links in our automated Twilio messages because SMS messages that contain more than 160 characters are technically 2 concatenated messages so it is more expensive to send.
And finally if the link is being shared through a medium that cannot be directly clicked on (e.g. verbally, on paper), making it shorter makes it much easier to type into an address bar manually. (Imagine trying to type the url to this SO question when someone is reading it to you.) I assume this is also at least partially why the base used for these links usually stop at around 62. If you start including other arbitrary characters to higher the base and consequentially make the link marginally shorter, it'll become harder to communicate, read and type. ("domain.name/5omeC0d3" vs "domian.name/🈲}♠"
While investigating a ConflictError (see this previous question) I saw a lot of persistent.mapping.PersistentMapping conflicts.
Looking at a specific one it turned out to be a PersistentMapping for plone.scale.
Turns out that a random object with just one image has 562 keys on it, no wonder why it gets a conflict error...
Some context on the object that holds this plone.scale annotation:
- dexterity content type
- one of its behaviors has an image field (plone.namedfile.field.NamedBlobImage)
The code to see it is as following:
Start a debugging instance: ./bin/instance debug
from ZODB.utils import p64
OID = 0x568428 # got from zeo client logs
mapping = app._p_jar[p64(OID)]
len(mapping) # that returns 562
The mysterious part is that only 4 keys on that persistent mapping are tuples, while the other 558 are just hashes.
A brief look at plone.scale.storage.AnnotationStorage.scale method seems to imply that there should be only one to one relation from tuples and hashes keys on the persistent mapping.
Further investigating the elements reveals that, indeed, if you look at the width and height properties from all elements there are only 4 different combinations (the ones from the tuples itself).
As a new scale is generated whenever the modified time is bigger (see the scale method pointed above) and plone.namedfield.scaling.ImageScaling.modified uses context as the source for modified, that means that at every single update of the object a new scale will be generated?
So two questions arise from the previous:
my assumption of only 4 scales are really used and the other 558 are old and useless is true?
provided a yes on the previous, shouldn't they be cleaned up then?
You may be right, but surely the correct place to report this is https://dev.plone.org/newticket
I have a design issue regarding HASH function.
In my program I am using a hash table of size 2^13, where the slot is calculated based on the value of the node(the hash key) which I want to insert.
Now, say my each node has two value |A|B| however I am inserting value into hash table using A.
Later on, I want to search a particular node which B not A.
Is it possible to that way? Is yes, could you highlight some design approaches?
The constraint is that I have to use A as the hash key.
Sorry, I can't share the code. Small example:
Value[] = {Part1, Part2, Part3};
insert(value)
check_for_index(value.part1)
value.part1 to be used to calculate the index of the slot.
Once slot is found then insert the "value"
Later on,
search_in_hash(part2)
check_for_index("But here I need the value.part1 to check for slot index")
So, how can I relate the part1, part2 & part3 such that I later on I can find the slot by either part2 or part3
If the problem statement is vague kindly let me know.
Unless you intend to do a search element-by-element (in which case you don't need a hash, just a plain list), then what you basically ask is - can I have a hash such that hash(X) == hash(Y), but X!=Y, so that you could map to a location using part1 and then map to the same one using part2 or 3. That completely goes against what hashing stands for.
What you should do is (as viraptor also suggested), create 3 structures, each hashed using a different part of the value, and push the full value to all 3. Then when you need to search use the proper hash by the part you want to search by.
for e.g.:
value[] = {part1, part2, part3};
hash1.insert(part1, value)
hash2.insert(part2, value)
hash3.insert(part3, value)
then
hash2.search_in_hash(part2)
or
hash3.search_in_hash(part3)
The above 2 should produce the exact same values.
Also make sure that all data manipulations (removing values, changing them), is done on all 3 structures simultaneously. For e.g. -
value = hash2.search_in_hash(part2)
hash1.remove(value.part1)
hash2.remove(part2) // you can assert that part2 == value.part2
hash3.remove(value.part3)
My problem:
I'm looking for a way to represent a person's name and address as an encoded id. The id should contain only alpha-numeric characters, be collision-proof, and be represented in a smallest number of characters possible. My first thought was to simply use a cryptographic hash function like MD5 or SHA1, but this seems like overkill (security isn't important - doesn't need to be one-way) and I'd prefer to find something that would produce a shorter id. Does anyone know of an existing algorithm that fits this problem?
In other words, what is the best way to implement the following function so that the return value is the same consistently for the same input, collisions are unlikely, and ids are less than 20 characters?
>>> make_fake_id(fname = 'Oscar', lname = 'Grouch', stnum = '1', stname = 'Sesame', zip = '12345')
N1743123734
Application Context (for those that are interested):
This will be used for a record linkage app. Given an input name and address we search a very large database for the best match and return the database id and other data (how we do this is not important here). If there isn't a match I need to generate this psuedo/generated/derived id from the search input (entity's name and address data). Every search record should result in an output record with either a real (the actual database id resulting from a match/link) or this generated psuedo/generated/derived id. The psuedo id will be prefixed with a character (e.g. N) to differentiate it from a real id.
I know you said no to MD5 and SHA1, but I think you should consider them anyway. As well as being well studied hashing algorithms, the length gives you more protection against possible collisions. No hash is collision-proof, but the cryptographic ones generally are less collision-prone than something you couuld come up with yourself.
Use a cryptographic hash for its collision resistance, not its other qualities
Use as many bytes from the hash as you want (truncate)
convert to alpha-numeric characters
You can also truncate the alpha-numeric string instead of the hash
An easy way to do this: hash the data, encode in base64, remove all non-alpha-numeric characters, truncate.
N_HASH_CHARS = 11
import hashlib, re
def digest(name, address):
hash = hashlib.md5(name + "|" + address).digest().encode("base64")
alnum_hash = re.sub(r'[^a-zA-Z0-9]', "", hash)
return alnum_hash[:N_HASH_CHARS]
How many alpha-numeric characters should you keep? Each character gives you around 5.95 bits of entropy (log(62,2)). 11 characters give you 65.5 bits of entropy, which should be enough to avoid a collision for the first 2**32.7 users (about 7 billion).
A good solution is somewhat dependent on your application. Do you know how many users and what the set of all users is? If you provide more details you would get better help.
I agree with the other poster suggesting serial numbers. OTOH, if you really, really really want to do something else:
Create a SHA1 hash from the data, and store it in a table with a serial number field.
Then, when you get the data, calculate the hash, look it up on the table, get the serial, and that's your id. If it's not on the table, insert it.
I wonder whether you intend to "assign" these ids to the users? If so, I would expect your users to hate anything that you propose; who would want a user id of "AAAAA01"?
So, if these ids are visible to the user, then you should just let them pick what they like and check them for uniqueness (easy). If they are not visible to the user (e.g., internal primary key), then just generate them sequentially using an appropriate technique such as an Oracle Sequence or SQL Server AutoNumber (also easy).
If these ids are an attempt to detect a user that is registering more than once, then I would agree that you should consider a cryptographic hash followed by a full comparison of the registration data (name, address, etc.). However, to be usable, you will need to translate the data into a canonical form (standardized letter case, whitespace, canonical street address, etc.) before computing the hash or making the comparison. Otherwise, you will mismatch based on trivial differences.
EDIT: Now that I understand the problem space better based on your edits, I think that it is highly unlikely that your algorithm (so far) will catch most matches. Beyond my suggestion to canonicalize the inputs, I recommend that you consider an approach that results in a ranked list of a handful of possible matches (to be resolved by a human if possible) rather than an all-or-nothing attempt at a single match. In other words, I recommend a search approach rather than a lookup approach.
Is that feasible in your situation?
Well, if there's more than one person at the same address with the same name, you're toast here, (w/o adding code to detect this and add a discriminator of some kind).
but assuming that issue is not, then the street address and zip code portion of the full addresss is sufficient to guaranteee uniqueness there, so adding enough data from the name should take care of the issue...
Do you have access to a database, or other persistence mechanism, where you could generate and maintain key values for each address? Then keep the address and individual entities in two keyed dictionary structures, where the key is autogenerated for each new distinct address, person encountered... and then use the autogenerated alpha-numeric key...
You could use AAAAA01 for first person at first address,
AAAAA02 for second person at first address,
AAAAB07 for the seventh resident at the second adresss, etc.
If you donlt have any way to generate and maintain these entity-Key mappings then you need to use the full street address/Zip and fullNAme, or a hash value of the same, although the Hash value approach has a smnall chance of generating duplicates...
I'm looking to create a simple short-lived reservation system, and I'd like to generate confirmation numbers that are
unique
random-looking
alphanumeric
short-ish, at least much shorter than 32 character-long strings returned by sha1
I'm only looking to have ~500 reservations, so I don't imagine high likelyhood of collissions.
One idea I had is generate an sha1 hash based on a date-time stamp and username, then truncating it to its first 10 characters. Would something like that be reliably unique enough for the purposes of processing ~500 reservations?
There should be no difference in the randomness of any given bit of a SHA-1 hash, so that's possible. Another way would be to fold the hash into itself using XOR until you have 60 bits worth of data, then encode it using Base 64 to get a mostly alpha-numeric result.
This is only necessary if you want to be able to generate the same Id repeatedly for the same input data. Otherwise, if a random id that you generate once, and hold onto after that, use Anders' suggestion. If you get a conflict, just generate another one.
You can use whatever, even a plain random number generator; however, you should check that the reservation code isn't already present. If this is the case, add characters ('x') to the string (date+user) until you get a new random/sha1/etc.
I'm only looking to have ~500 reservations, so I don't imagine high likelyhood of collissions.
Another stupid idea: generate 1000 or 2000 unique random numbers with the desired properties, store them somewhere, and assign them to the users as they register :)
Here's one way to do it in Perl:
sub get_random_name()
{
my #chars=('a'..'z','A'..'Z');
my $random_string;
foreach (1..22)
{
# rand #chars will generate a random
# number between 0 and scalar #chars
$random_string .= $chars[rand #chars];
}
return $random_string . "-" . time();
}
I don't remember how long the time() part is, so you may have to adjust the numbers to fit your length. You can also remove that part if you don't need it.
If it's really just 500, then pre-generate 20,000 of them, into a table, then get the "next unused one" when you need it.
Some good tips on this question: How do I create a random alpha-numeric string in C++?
I'd avoid including characters like "1", "l", and "O", "0" and "5", "S", and "Z", "2" in your string, to make it easier for customers when they need to read your reservation code over the phone. The algorithm presented at that link should help you do this.
use a guid? 16 characters, though if you really don't care about collision, you could just choose the first n characters.
In C# you can use http://www.dotnetfunda.com/forums/thread1357-how-do-generate-unique-alpha-numeric-random-number-in-aspnet.aspx (the super easy way, they say)