I have some unique codes that are generated from strings (ex: website host names) in various independent components of my application.
These codes are meant to be used by machines only so i would like to keep them as short as possible.
The below algorithm would be applied to every word in the string. The output words would be concatenated with a dash to generate the unique code.
The current algorithm I have used:
- Skip word if length is less than 6
- Leave first character as is
- Remove every wowel in the word from the second character onwards
architectural digest eu => archtctrl-dgst-eu
arizona foothills magazine => arzn-fthlls-mgzn
Is there a better way to shorten an English word leaving it as recognisable as possible to a human reader?
The output should be deterministic and produce the same shortened version whenever it is run on the same input.
A good algorithm should also minimise the number of clashes for similarly spelt words.
I have some unique codes that are generated from strings
I am afraid that is not true. There are many English words that will reduce to the same 'code word' when stripped of their vowels. For example, 'leaving' -> 'living' Given, this is fairly rare, it could still cause issues.
How important is it that these 'code words' remain human-readable if as you say, they are meant to be used by machines only? If its not that important, I'd suggest looking into some simpler compression algorithms like Huffman Coding or LZW Compression. Then if the user needs to see the translation of the code word, just uncompress it.
If you must keep it human-readable, I'm not sure that there is much more you can do to shorten it. You could take a look at specific latin + greek roots, and determine if you can shorten those any more by hand, and then just substitute those out automatically.
Alternatively, you could turn to a phonetic approach. Automatically search the pronunciation of the word, and then see if that is any shorter (or itself can be compressed, taking 'cee' to 'C', or 'kay' to 'K'). This would be much more time and CPU intensive, but its still an option if you really, really need short but yet readable codes.
What you're generating sounds like what's called a "slug". There are many libraries to handle this for blogs or site generators that should suit your purposes. Here's a usage example from a Python library called slugify:
txt = "___This is a test ---"
r = slugify(txt)
self.assertEqual(r, "this-is-a-test")
Slug libraries generally work like this:
replacing non-ascii linguistic characters via a mapping (ex: 影師嗎 -> ying-shi-ma)
replace accented latin letters with ascii equivalents via a mapping (ex: C'est déjà l'été. -> c-est-deja-l-ete)
remove beginning and trailing spaces/punctuation
convert remaining spaces and punctuation to dashes, collapsing multiple dashes in a row to a single dash
If you want to make slugs shorter you could remove vowels or, more simply, use a maximum length.
I would like to know how do compression algorithms actually write their notes. Let's say "4x4x" means 4 times "4x".
What if algorithm labels repeated characters like this:
23*("text") and text that repeats is 34*("something")
How would program know this is not a label for repeated text, but an actual text. I don't know how to explain this better.
After compressing this string:
"Compression programs label how many times string is repeated like this: 4x("text"), this is repeated repeated repeated ."
You would get:
"Compression programs label how many times string is repeated like this: 4x("text"), this is 3x("repeated") ."
How will algorithm know witch one is compressed one?
Roughly, there are 3 approaches:
Everything is a special symbol. So, the text would be encoded as 1x"Compression", 1x"programs", 1x"label" and so on.
Escaping. This refers to using a special character to denote a symbol. So, 4x("text") would be represented as $4x("text"), and the presence of $ means that what follows is a special compression sequence. Of course, for this to work, a trick is required to allow us to include $ in normal text. The trick is very simple: $1x("$").
Dictionaries. The compression algorithms you have in mind are variations of the extremely simplistic "Run Length Encoding" algorithm. (Look it up.) This algorithm is practically no good for nothing and very rarely used nowadays. Modern compression algorithms like LZW are much more sophisticated, and they make use of dictionaries, where every input combination maps to a string of bytes to output. A full explanation would be too long to include in an answer, but feel free to look up LZW.
The usual way this is handled is to recognize (during compression) that the original text contains a character sequence that will cause problems for decompression and to "escape" it somehow. There are many ways to do this, but the simplest for the example you posted might be to compress the text to this:
"Compression programs label how many times string is repeated like this: 1x("4x")("text"), this is 3x("repeated").
That way, the "4x" (or anything else that might look like a repeat count) is not taken for a repeat count because it has been captured as text to be repeated (albeit only once). And the text ("text") is not prefaced by a repeat count, so it will go through the decompression phase unchanged.
Note that this particular encoding scheme has other problems (such as repeated text that itself contains a double quote mark). But all these problems can be addressed with proper escape processing.
Tesseract OCR engine sometimes outputs text that has no meaning, i want to design an algorithm that neglects any text or word that has no meaning, below is some sort of output text that i want to neglect,my simple solution is to count the words in the recognized text that's separated by " " and the text which has too many words will be garbage(Hint: i'm scanning images which at most will contains 40 words) any idea will be helpful,thanks.
wo:>"|axnoA1wvw\
ldflfig
°J!9O‘ !P99W M9N 6 13!-|15!Cl ‘I-/Vl
978 89l9 Z0 3+ 3 'l9.l.
97 999 VLL lLOZ+ 3 9l!q°lN
wo0'|axno/(#|au1e>1e: new;
1=96r2a1ey\1 1uauud0|e/\e(]
|8UJB){ p8UJL|\7'
Divide the output text into words. Divide the words into triples. Count the triple frequencies, and compare to triple frequencies from text of a known-good text corpus (EG all the articles from some mailing list discussing what you intend to OCR, minus the header lines).
When I say "triples", I mean:
whe, hen, i, say, tri, rip, ipl, ple, les, i, mea, ean
...so "i" has a frequency of 2 in this short example, while the others are all frequency 1.
If you do a frequency count of each of these triples for a large document in your intended language, it should become possible to be reasonably accurate in guessing whether a string is in the same language.
Granted, it's heuristic.
I've used a similar approach for detecting English passwords in a password changing program. It worked pretty well, though there's no such thing as a perfect "obvious password rejecter".
Check the words against a dictionary?
Of course, this will have false-positives for things like foreign-phrases or code. The problem in general is intractable (ex. is this code or gibberish? :) ). The only (nearly) perfect method would be to use this as a heuristic to flag certain sections for human review.
Background
While at the Gym the other day, I was working with my combination lock, and realized something that would be useful to me as a programmer. To wit, my combination is three seperate sets of numbers that either sound alike, or have some other relation that makes them easy to remember. For instance, 5-15-25, 7-17-2, 6-24-5. These examples seem easy to remember.
Question
How would I implement something similar for passwords? Yes, they ought to be hard to crack, but they also should be easy for the end user to remember. Combination Locks do that with a mix of numbers that have similar sounds, and with numbers that have similar properties (7-17-23: All Prime, 17 rolls right off the tongue after 7, and 23 is another prime, and is (out of that set), the 'hard' one to remember).
Criteria
The Password should be easy to remember. Dog!Wolf is easy to remember, but once an attacker knows that your website gives out that combination, it makes it infinitely easier to check.
The words or letters should mostly follow the same sounds (for the most part).
At least 8 letters
Not use !##$%^&*();'{}_+<>?,./ These punctuation marks, while appropriate for 'hard' passwords, do not have an 'easy to remember' sound.
Resources
This question is language-agnostic, but if there's a specific implementation for C#, I'd be glad to hear of it.
Update
A few users have said that 'this is bad password security'. Don't assume that this is for a website. This could just be for me to make an application for myself that generates passwords according to these rules. Here's an example.
The letters
A-C-C-L-I-M-O-P 'flow', and they happen to be two
regular words put together
(Acclimate and Mop). Further,
when a user says these letters, or
says them as a word, it's an actual
word for them. Easy to remember, but
hard to crack (dictionary attack,
obviously).
This question has a two-part goal:
Construct Passwords from letters that sound similar (using alliteration) or
Construct Passwords that mesh common words similarly to produce a third set of letters that is not in a dictionary.
You might want to look at:
The pronouncable password generation algorithm used by apg and explained in FIPS-181
Koremutake
First of all make sure the password is long. Consider using a "pass-phrase" instead of a single "pass-word". Breaking pass-phrases like "Dogs and wolves hate each other." is very hard yet they are quite easy to remember.
Some sites may also give you an advice which may be helpful, like Strong passwords: How to create and use them (linked from Password checker, which is a useful tool on its own).
Also, instead of trying to create easy to remember password, in some cases a much better alternative is to avoid remembering the password at all by using (and educating your users to use) a good password management utility (see What is your favourite password storage tool?) - when doing this, the only part left is to create a hard to crack password, which is easy (any long enough random sentence will do).
I am surprised no one has mentioned the Multics algorithm described at http://www.multicians.org/thvv/gpw.html , which is similar to the FIPS algorithm but based on trigraphs rather than digraphs. It produces output such as
ahmouryleg
thasylecta
tronicatic
terstabble
I have ported the code to python as well: http://pastebin.com/f6a10de7b
You could use Markov Chains to generate words that sounds like English(or any other language you want) but they are not actual words.
The question of easy to remember is really subjective, so I don't think you can write an algorithm like this that will be good for everyone.
And why use short passwords on web sites/computer applications instead of pass phrases? They are easy to remember but hard to crack.
After many years, I have decided to use the first letter of words in a passphrase. It's impossible to crack, versatile for length and restrictions like "you must have a digit", and hard to make errors.
This works by creating a phrase. A crazy fun vivid topic is useful!
"Stack Overflow aliens landed without using rockets or wheels".
Take the first letter, your password is "soalwurow"
You can type this quickly and accurately since you're not remembering letter by letter, you're just speaking a sentence inside your head.
I also like having words alternate from the left and right side of the keyboard, it gives you a fractionally faster typing speed and more pleasing rhythm. Notice in my example, your hands alternate left-right-left-right.
I have a few times used a following algorithm:
Put all lowercase vowels (from a-z) into an array Vowels
Put all lowercase consonants (from a-z) into another array Consonants
Create a third array Pairs of two letters in such a way, that you create all possible pairs of letters between Vowels and Consonants ("ab", "ba", "ac", etc...)
Randomly pick 3-5 elements from Pairs and concatenate them together as string Password
Randomly pick true or false
If true, remove the last letter from Password
If false, don't do anything
Substitute 2-4 randomly chosen characters in Password with its uppercase equivalent
Substitute 2-4 randomly chosen characters in Password with a randomly chosen integer 0-9
Voilá - now you should have a password of length between 5 and 10 characters, with upper and lower case alphanumeric characters. Having vowels and consonants take turns frequently make them semi-pronounceable and thus easier to remember.
FWIW I quite like jumbling word syllables for an easy but essentially random password. Take "Bongo" for example as a random word. Swap the syllables you get "Gobong". Swap the o's for zeros on top (or some other common substitution) and you've got an essentially random character sequence with some trail that helps you remember it.
Now how you pick out syllables programmatically - that's a whole other question!
When you generate a password for the user and send it by email, the first thing you should do when they first login if force them to change their password. Passwords created by the system do not need to be easy to remember because they should only be needed once.
Having easy to remember, hard to guess passwords is a useful concept for your users but is not one that the system should in some manner enforce. Suppose you send a password to your user's gmail account and the user doesn't change the password after logging in. If the password to the gmail account is compromised, then the password to your system is compromised.
So generating easy to remember passwords for your users is not helpful if they have to change the password immediately. And if they aren't changing it immediately, you have other problems.
I prefer giving users a "hard" password, requiring them to change it on the first use, and giving them guidance on how to construct a good, long pass phrase. I would also couple this with reasonable password complexity requirements (8+ characters, upper/lower case mix, and punctuation or digits). My rationale for this is that people are much more likely to remember something that they choose themselves and less likely to write it down somewhere if they can remember it.
A spin on the 'passphrase' idea is to take a phrase and write the first letters of each word in the phrase. E.g.
"A specter is haunting Europe - the specter of communism."
Becomes
asihe-tsoc
If the phrase happens to have punctation, such as !, ?, etc - might as well shove it in there. Same goes for numbers, or just substitute letters, or add relevant numbers to the end. E.g. Karl Marx (who said this quote) died in 1883, so why not 'asihe-tsoc83'?
I'm sure a creative brute-force attack could capitalise on the statistical properties of such a password, but it's still orders of magnitude more secure than a dictionary attack.
Another great approach is just to make up ridiculous words, e.g. 'Barangamop'. After using it a few times you will commit it to memory, but it's hard to brute-force. Append some numbers or punctuation for added security, e.g. '386Barangamop!'
Here's part 2 of your idea prototyped in a shell script. It takes 4, 5 and 6 letter words (roughly 50,000) from the Unix dictionary file on your computer, and concatenate those words on the first character.
#! /bin/bash
RANDOM=$$
WORDSFILE=./simple-words
DICTFILE=/usr/share/dict/words
grep -ve '[^a-z]' ${DICTFILE} | grep -Ee '^.{4,6}$' > ${WORDSFILE}
N_WORDS=$(wc -l < ${WORDSFILE})
for i in $(seq 1 20); do
password=""
while [ ! "${#password}" -ge 8 ] || grep -qe"^${password}$" ${DICTFILE}; do
while [ -z "${password}" ]; do
password="$(sed -ne "$(( (150 * $RANDOM) % $N_WORDS + 1))p" ${WORDSFILE})"
builtfrom="${password}"
done
word="$(sort -R ${WORDSFILE} | grep -m 1 -e "^..*${password:0:1}")"
builtfrom="${word} ${builtfrom}"
password="${word%${password:0:1}*}${password}"
done
echo "${password} (${builtfrom})"
done
Like most password generators, I cheat by outputting them in sets of twenties. This is often defended in terms of "security" (someone looking over your shoulder), but really its just a hack to let the user just pick the friendliest password.
I found the 4-to-6 letter words from the dictionary file still containing obscure words.
A better source for words would be a written document. I copied all the words on this page and pasted them into a text document, and then ran the following set of commands to get the actual english words.
perl -pe 's/[^a-z]+/\n/gi' ./624425.txt | tr A-Z a-z | sort -u > ./words
ispell -l ./words | grep -Fvf - ./words > ./simple-words
Then I used these 500 or so very simple words from this page to generate the following passwords with the shell script -- the script parenthetically shows the words that make up a password.
backgroundied (background died)
soundecrazy (sounding decided crazy)
aboupper (about upper)
commusers (community users)
reprogrammer (replacing programmer)
alliterafter (alliteration after)
actualetter (actual letter)
statisticrhythm (statistical crazy rhythm)
othereplacing (other replacing)
enjumbling (enjoying jumbling)
feedbacombination (feedback combination)
rinstead (right instead)
unbelievabut (unbelievably but)
createdogso (created dogs so)
apphours (applications phrase hours)
chainsoftwas (chains software was)
compupper (computer upper)
withomepage (without homepage)
welcomputer (welcome computer)
choosome (choose some)
Some of the results in there are winners.
The prototype shows it can probably be done, but the intelligence you require about alliteration or syllable information requires a better data source than just words. You'd need pronunciation information. Also, I've shown you probably want a database of good simple words to choose from, and not all words, to better satisfy your memorable-password requirement.
Generating a single password the first time and every time -- something you need for the Web -- will take both a better data source and more sophistication. Using a better programming language than Bash with text files and using a database could get this to work instantaneously. Using a database system you could use the SOUNDEX algorithm, or some such.
Neat idea. Good luck.
I'm completely with rjh. The advantage of just using the starting letters of a pass-phrase is that it looks random, which makes it damn hard to remember if you don't know the phrase behind it, in case Eve looks over your shoulder as you type the password.
OTOH, if she sees you type about 8 characters, among which 's' twice, and then 'o' and 'r' she may guess it correctly the first time.
Forcing the use of at least one digit doesn't really help; you simply know that it will be "pa55word" or "passw0rd".
Song lyrics are an inexhaustible source of pass-phrases.
"But I should have known this right from the start"
becomes "bishktrfts". 10 letters, even only lowercase gives you 10^15 combinations, which is a lot, especially since there's no shortcut for cracking it. (At 1 million combinations a second it takes 30 years to test all 10^15 combinations.)
As an extra (in case Eve knows you're a Police fan), you could swap e.g. the 2nd and 3rd letter, or take the second letter of the third word. Endless possibilities.
System generated passwords are a bad idea for anything other than internal service accounts or temporary resets (etc).
You should always use your own "passphrases" that are easy for you to remember but that are almost impossible to guess or brute force. For example the password for my old university account was.
Here to study again!
That is 20 characters using upper and lower case with punctuation. This is an unbelievably strong password and there is no piece of software that could generate a more secure one that is easier to remember for me.
Take look at the gpw tool. The package is also available in Debian/Ubuntu repositories.
One way to generate passwords that 'sound like' words would be to use a markov chain. An n-degree markov chain is basically a large set of n-tuples that appear in your input corpus, along with their frequency. For example, "aardvark", with a 2nd-degree markov chain, would generate the tuples (a, a, 1), (a, r, 2), (r, d, 1), (d, v, 1), (v, a, 1), (r, k, 1). Optionally, you can also include 'virtual' start-word and end-word tokens.
In order to create a useful markov chain for your purposes, you would feed in a large corpus of english language data - there are many available, including, for example, Project Gutenburg - to generate a set of records as outlined above. For generating natural language words or sentences that at least mostly follow rules of grammar or composition, a 3rd degree markov chain is usually sufficient.
Then, to generate a password, you pick a random 'starting' tuple from the set, weighted by its frequency, and output the first letter. Then, repeatedly select at random (again weighted by frequency) a 'next' tuple - that is, one that starts with the same letters that your current one ends with, and has only one letter different. Using the example above, suppose I start at (a, a, 1), and output 'a'. My only next choice is (a, r, 2), so I output another 'a'. Now, I can choose either (r, d, 1) or (r, k, 1), so I pick one at random based on their frequency of occurrence. Suppose I pick (r, k, 1) - I output 'r'. This process continues until you reach an end-of-word marker, or decide to stop independently (since most markov chains form a cyclic graph, you can potentially never finish generating if you don't apply an artificial length limitation).
At a word level (eg, each element of the tuple is a word), this technique is used by some 'conversation bots' to generate sensible-seeming nonsense sentences. It's also used by spammers to try and evade spam filters. At a letter level, as outlined above, it can be used to generate nonsense words, in this case for passwords.
One drawback: If your input corpus doesn't contain anything other than letters, nor will your output phrases, so they won't pass most 'secure' password requirements. You may want to apply some post-processing to substitute some characters for numbers or symbols.
edit: After answering, I realized that this is in no way phonetically memorable. Leaving the answer anyway b/c I find it interesting. /edit
Old thread, I know... but it's worth a shot.
1) I'd probably build the largest dictionary you can ammass. Arrange them into buckets by part of speech.
2)Then, build a grammar that can make several types of sentences. "Type" of sentence is determined by permutations of parts of speech.
3)Randomly (or as close to random as possible), pick a type of sentence. What is returned is a pattern with placeholders for parts of speech (n-v-n would be noun-verb-noun)
3)Pick words at random in each part of speech bucket to stand in for the placeholders. Fill them in. (The example above might become something like car-ate-bicycle.)
4)randomly scan each character deciding whether or not you want to replace it with either a similar-sounding character (or set of characters), or a look-alike. This is the hardest step of the problem.
5) resultant password would be something like kaR#tebyCICle
6) laugh at humorous results like the above that look like "karate bicycle"
I would really love to see someone implement passwords with control characters like "<Ctrl>+N" or even combo characters like "A+C" at the same time. Converting this to some binary equivalent would, IMHO, make password requirements much easier to remember, faster to type, and harder to crack (MANY more combinations to check).