Seems memcache client doesn't support UTF-8 string as its key. But I have to use i18n. Anyway to fix it?
java.lang.IllegalArgumentException: Key contains invalid characters: ``HK:00:A Kung Wan''
at net.spy.memcached.MemcachedClient.validateKey(MemcachedClient.java:232)
at net.spy.memcached.MemcachedClient.addOp(MemcachedClient.java:254)
The issue here isn't UTF encoding. It's the fact that your key contains a space. Keys cannot have spaces, new lines, carriage returns, or null characters.
The line of code that produces the exception is below
if (b == ' ' || b == '\n' || b == '\r' || b == 0) {
throw new IllegalArgumentException
("Key contains invalid characters: ``" + key + "''");
}
Base64 Encode your key just before passing them to memcached client's set() and get() methods.
A general solution to handle all memcached keys with special characters, control characters, new lines, spaces, unicode characters, etc. is to base64 encode the key just before you pass it to the set() and get() methods of memcached.
// pseudo code for set
memcachedClient.set(Base64.encode(key), value);
// pseudo code for get
memcachedClient.get(Base64.encode(key));
This converts them into characters memcached is guaranteed to understand.
In addition, base64 encoding has no performance penalty (unless you are a nano performance optimization guy), base64 is reliable and takes only about 30% extra length.
Works like a charm!
Related
I have an Encoder (using openssl) that can encrypt and decrypt strings like so:
new_addresses
=> ["fasfds", "someaddress", "123- this is also a valid address"]
[8] pry(#<Sinatra::Application>)> Encoder.encrypt(new_addresses.join(' '))
=> "55C2FB253468204EA9D3F5CE6D58DC4088BD52731B90B9C0C8EB5FE7FA1CD4E7B41F0A84DC46C69E09A10DC1931C6A976A58E29C"
[9] pry(#<Sinatra::Application>)> enc=_
=> "55C2FB253468204EA9D3F5CE6D58DC4088BD52731B90B9C0C8EB5FE7FA1CD4E7B41F0A84DC46C69E09A10DC1931C6A976A58E29C"
[10] pry(#<Sinatra::Application>)> Encoder.decrypt(enc)
=> "fasfds someaddress 123- this is also a valid address"
The issue I have here is that I have no idea which were the original 3 addresses. The new_addresses which are merely params that come in from a form are an array separated by commas. But when I join them together and encode it, I lose the comma delimiter and the array structure when I decrypt it so I have no idea what were the original 3 addresses. Any ideas on what I can do so that after I decrypt the string, I still can detect on what the original 3 addresses are.
These are valid characters in an address:
' '
-
_
^
%
$
#
...
really any characters.
It looks like your encryption algorithm uses only the characters 0-9 and A-Z. In that case, you can use any character that is not one of those characters to join() your encrypted strings together, for instance "-":
encrypted_str = "55C2FB253-3F5CE6D58DC4-B5FE7FA1CD4E7"
encyrpted_pieces = encrypted_str.split '-'
decrypted_pieces = encrypted_pieces.map do |piece|
Encoder.decrypt piece
end
On the other hand, if you want to join your strings together first, then encrypt the combined string, you can use the non printing ascii character named NUL to glue the pieces together. NUL's ascii code is 0, which can be represented by the hex escape \x00 inside a String:
decrypted_str = "fasfds\x00someaddress\x00123- this is also a valid address"
puts decrypted_str
pieces = decrypted_str.split "\x00"
p pieces
--output:--
fasfdssomeaddress123- this is also a valid address
["fasfds", "someaddress", "123- this is also a valid address"]
Magic.
Of course, the separator character should be a character that won't appear in the input. If the input can be binary data, e.g. an image, then you can't use \x00 as the separator.
These are valid characters in an address:
' '
-
_
^
%
$
#
...
Note that you didn't list a comma, which would be an obvious choice for the separator.
I've gotten lost in an edge case of sorts. I'm working on a conversion of some old plaintext documentation to reST/Sphinx format, with the intent of outputting to a few formats (including HTML and text) from there. Some of the documented functions are for dealing with bitstrings, and a common case within these is a sentence like the following: Starting character is the blank " " which has the value 0.
I tried writing this as an inline literal the following ways: Starting character is the blank `` `` which has the value 0. or Starting character is the blank :literal:` ` which has the value 0. but there are a few problems with how these end up working:
reST syntax objects to a whitespace immediately inside of the literal, and it doesn't get recognized.
The above can be "fixed"--it looks correct in the HTML () and plaintext (" ") output--with a non-breaking space character inside the literal, but technically this is a lie in our case, and if a user copied this character, they wouldn't be copying what they expect.
The space can be wrapped in regular quotes, which allows the literal to be properly recognized, and while the output in HTML is probably fine (" "), in plaintext it ends up double-quoted as "" "".
In both 2/3 above, if the literal falls on the wrap boundary, the plaintext writer (which uses textwrap) will gladly wrap inside the literal and trim the space because it's at the start/end of the line.
I feel like I'm missing something; is there a good way to handle this?
Try using the unicode character codes. If I understand your question, this should work.
Here is a "|space|" and a non-breaking space (|nbspc|)
.. |space| unicode:: U+0020 .. space
.. |nbspc| unicode:: U+00A0 .. non-breaking space
You should see:
Here is a “ ” and a non-breaking space ( )
I was hoping to get out of this without needing custom code to handle it, but, alas, I haven't found a way to do so. I'll wait a few more days before I accept this answer in case someone has a better idea. The code below isn't complete, nor am I sure it's "done" (will sort out exactly what it should look like during our review process) but the basics are intact.
There are two main components to the approach:
introduce a char role which expects the unicode name of a character as its argument, and which produces an inline description of the character while wrapping the character itself in an inline literal node.
modify the text-wrapper Sphinx uses so that it won't break at the space.
Here's the code:
class TextWrapperDeux(TextWrapper):
_wordsep_re = re.compile(
r'((?<!`)\s+(?!`)|' # whitespace not between backticks
r'(?<=\s)(?::[a-z-]+:)`\S+|' # interpreted text start
r'[^\s\w]*\w+[a-zA-Z]-(?=\w+[a-zA-Z])|' # hyphenated words
r'(?<=[\w\!\"\'\&\.\,\?])-{2,}(?=\w))') # em-dash
#property
def wordsep_re(self):
return self._wordsep_re
def char_role(name, rawtext, text, lineno, inliner, options={}, content=[]):
"""Describe a character given by unicode name.
e.g., :char:`SPACE` -> "char:` `(U+00020 SPACE)"
"""
try:
character = nodes.unicodedata.lookup(text)
except KeyError:
msg = inliner.reporter.error(
':char: argument %s must be valid unicode name at line %d' % (text, lineno))
prb = inliner.problematic(rawtext, rawtext, msg)
return [prb], [msg]
app = inliner.document.settings.env.app
describe_char = "(U+%05X %s)" % (ord(character), text)
char = nodes.inline("char:", "char:", nodes.literal(character, character))
char += nodes.inline(describe_char, describe_char)
return [char], []
def setup(app):
app.add_role('char', char_role)
The code above lacks some glue to actually force the use of the new TextWrapper, imports, etc. When a full version settles out I may try to find a meaningful way to republish it; if so I'll link it here.
Markup: Starting character is the :char:`SPACE` which has the value 0.
It'll produce plaintext output like this: Starting character is the char:` `(U+00020 SPACE) which has the value 0.
And HTML output like: Starting character is the <span>char:<code class="docutils literal"> </code><span>(U+00020 SPACE)</span></span> which has the value 0.
The HTML output ends up looking roughly like: Starting character is the char:(U+00020 SPACE) which has the value 0.
What is the idiomatic way to remove non-ASCII characters from file contents in D?
I tried:
auto s = (cast(string) std.file.read(myPath)).filter!( a => a < 128 ).array;
which gave me:
std.utf.UTFException#C:\D\dmd2\windows\bin\..\..\src\phobos\std\utf.d(1109): Invalid UTF-8 sequence (at index 1)
and s is dstring ; and:
auto s = (cast(string) std.file.read(myPath)).tr("\0-~", "", "cd");
which gives me:
core.exception.UnicodeException#src\rt\util\utf.d(290): invalid UTF-8 sequence
at runtime.
I am trying to parse (with the almost deprecated std.xml module) xml files in a unsupported encoding, but I am ok with removing the offending characters.
If you do anything to consider it a string, D tries to treat it as UTF-8. Instead, treat it as a series of bytes, so replace your cast(string) with cast(ubyte[]) and do the filter.
After reading and filtering it, you can /then/ cast it back into a string. So this should do what you need:
auto s = cast(string) (cast(ubyte[])(std.file.read(myPath)).filter!( a => a < 128 ).array);
I'm hashing a password using SHA512. I'm using Entity Framework Code-First for my ORM.
Hashing Algorithm
public static string CreateSHA512Hash(string pwd, string salt)
{
string saltAndPwd = String.Concat(pwd, salt);
var ae = new ASCIIEncoding();
byte[] hashValue, messageBytes = ae.GetBytes(saltAndPwd);
var sHhash = new SHA512Managed();
hashValue = sHhash.ComputeHash(messageBytes);
sHhash.Dispose();
return ae.GetString(hashValue);
}
Code for generating salt:
//Generate a cryptographic random number.
var rng = new RNGCryptoServiceProvider();
var buff = new byte[size];
rng.GetBytes(buff);
rng.Dispose();
// Return a Base64 string representation of the random number.
return Convert.ToBase64String(buff);
Problem:
For some reason, it seems the hash function would randomly generate some characters, which the ones after those are not saved to the database. In this case (I'm not sure if there are other characters that does this), but it is \0.
For eg. Password: testuser. Salt: uvq5i4CfMcOMjKPkwhhqxw==
Hash generated: ????j???7?o\0?dE??????:???s?x??u?',Vj?mNB??c???4H???vF\bd?T? (copied during dubug mode in visual studio).
But EF actually saves ????j???7?o to the database. If I try to use the text visualizer in debug mode, it cuts it off also. If you noticed, it gets cut off right at the \0. All I could find about it is that its a null character.
Question
How can I save this null character in the database using Entity Framework Code-First? If this can't be saved, how can I prevent the SHA512 from generating these characters for me?
I recommend encoding the hash with Base64 before saving. On the other hand, encoding the salt with Base64 before adding to the password sounds strange.
A SHA-256 hash does not generate characters, it generates bytes. If you want to have a character string, as opposed to a byte array, you need to convert the bytes into a character format. As #wRAR has suggested, Base64 is one common way to do it or else you could just use a hex string.
What you should probably do:
Return the array of bytes for the SHA512 hash not a string.
Use a BINARY(64) database column to hold your hash value.
Why your method doesn't work:
These ASCII strings are NULL terminated
NULL is as you said \0
SHA512 creates a byte array and any byte can be NULL
To answer your specific question:
wRAR above was saying.
return Convert.ToBase64String(hashValue);
The CodeIgniter 2.0.2 requires to set an encryption key in the config file i.e. $config['encryption_key'] , if you want to use Session class. Can it be any string? Any example of secure encryption_key?
Thanks.
The key should be as random as possible and it must not be a regular
text string, nor the output of a hashing function, etc.
To save your key to your application/config/config.php, open the file and set:
$config['encryption_key'] = 'yourKeyHere'
Random Key Generator
It's important for you to know that the encoded messages the encryption function generates will be approximately 2.6 times longer than the original message. For example, if you encrypt the string "my super secret data", which is 21 characters in length, you'll end up with an encoded string that is roughly 55 characters (we say "roughly" because the encoded string length increments in 64 bit clusters, so it's not exactly linear). Keep this information in mind when selecting your data storage mechanism. Cookies, for example, can only hold 4K of information.
In addition to the answer by Chumillas, I personally use this Random Key Generator for my CodeIgniter encryption strings. Quick and easy.
Codeigniter 3.1.0
YOU MUST NOT USE REGULAR TEXT FOR 'encryption_key'
"The key should be as random as possible and it must not be a regular text string, nor the output of a hashing function, etc. In order to create a proper key, you must use the Encryption library’s create_key() method"
$this->load->library('encryption');
$key = $this->encryption->create_key(16);
// Get a hex-encoded representation of the key:
$key = bin2hex($this->encryption->create_key(16));
// Put the same value in your config with hex2bin(),
// so that it is still passed as binary to the library:
$config['encryption_key'] = hex2bin(<your hex-encoded key>);
Source: https://codeigniter.com/userguide3/libraries/encryption.html#setting-your-encryption-key
Type this into your terminal:
php -r 'echo bin2hex(random_bytes(16)), "\n";'
It'll output a string where you update your config.php
Just go to application/config
open config.php file
find out the word
$config['encryption_key'] = '';
replace this with
$config['encryption_key'] = 'your_encryption_key_here';
I am using the following code in my app's installer. It takes 128 bytes of random data (converted to a hex string), and takes two characters at a time, converting to decimal, and checking they're in an acceptable range (alphanumeric, with symbols, no whitespace or characters that won't play nice with your editor or config file - aka no ')
32 characters is 128 bits, so it works well with the block cipher.
function random_key_string() {
$source = bin2hex(openssl_random_pseudo_bytes(128));
$string = '';
$c = 0;
while(strlen($string) < 32) {
$dec = gmp_strval(gmp_init(substr($source, $c*2, 2), 16),10);
if($dec > 33 && $dec < 127 && $dec !== 39)
$string.=chr($dec);
$c++;
}
return $string;
}
To save your key to your application/config/config.php, open the file and set:
on line 227 $config['encryption_key'] = "YOUR KEY";