Is there a bash script available to generate a HMAC-SHA1 hash?
The equivalent of the following PHP code:
hash_hmac("sha1", "value", "key", TRUE);
Parameters
true : When set to TRUE, outputs raw binary data. FALSE outputs lowercase hexits.
Thanks.
see HMAC-SHA1 in bash
In bash itself, no, it can do a lot of stuff but it also knows when to rely on external tools.
For example, the Wikipedia page provides a Python implementation which bash can call to do the grunt work for HMAC_MD5, repeated below to make this answer self-contained:
#!/usr/bin/env python
from hashlib import md5
trans_5C = "".join(chr(x ^ 0x5c) for x in xrange(256))
trans_36 = "".join(chr(x ^ 0x36) for x in xrange(256))
blocksize = md5().block_size
def hmac_md5(key, msg):
if len(key) > blocksize:
key = md5(key).digest()
key += chr(0) * (blocksize - len(key))
o_key_pad = key.translate(trans_5C)
i_key_pad = key.translate(trans_36)
return md5(o_key_pad + md5(i_key_pad + msg).digest())
if __name__ == "__main__":
h = hmac_md5("key", "The quick brown fox jumps over the lazy dog")
print h.hexdigest() # 80070713463e7749b90c2dc24911e275
(keeping in mind that Python also contains SHA1 stuff as well, see here for details on how to use HMAC with the hashlib.sha1() constructor).
Or, if you want to run the exact same code as PHP does, you could try running it with phpsh, as detailed here.
Related
Can I use pexpect in a way that ignores ANSI escape codes (especially colors) in the output? I am trying to do this:
expect('foo 3 bar 5')
...but sometimes I get output with ANSI-colored numbers. The problem is I don't know which numbers will have ANSI colors and which won't.
Is there a way to use pexpect but have it ignore ANSI sequences in the response from the child process?
Here's a not entirely satisfying proposal, subclassing 2 routines of the pexpect classes pexpect.Expecter and pexpect.spawn so that incoming data can have the escape sequences removed before they get added to the buffer and tested for pattern match. It is a lazy implementation in that it assumes any escape sequence will always be read atomically, but coping with split reads is more difficult.
# https://stackoverflow.com/a/59413525/5008284
import re, pexpect
from pexpect.expect import searcher_re
# regex for vt100 from https://stackoverflow.com/a/14693789/5008284
class MyExpecter(pexpect.Expecter):
ansi_escape = re.compile(rb'\x1B[#-_][0-?]*[ -/]*[#-~]')
def new_data(self, data):
data = self.ansi_escape.sub(b'', data)
return pexpect.Expecter.new_data(self, data)
class Myspawn(pexpect.spawn):
def expect_list(self, pattern_list, timeout=-1, searchwindowsize=-1,
async=False):
if timeout == -1:
timeout = self.timeout
exp = MyExpecter(self, searcher_re(pattern_list), searchwindowsize)
return exp.expect_loop(timeout)
This assumes you use the expect() call with a list, and do
child = Myspawn("...")
rc = child.expect(['pat1'])
For some reason I had to use bytes rather than strings as I get the data before it is decoded, but that may just be because of a currently incorrect locale environment.
This workaround partially defeats the purpose of using pexpect but it satisfies my requirements.
The idea is:
expect anything at all (regex match .*) followed by the next prompt (which in my case is xsh $ - note the backslash in the "prompt" regex)
get the after property
trim off the prompt: [1:]
remove ANSI escape codes from that
compare the filtered text with my "expected" response regex
with pexpect.spawn(XINU_CMD, timeout=3, encoding='utf-8') as c:
# from https://stackoverflow.com/a/14693789/5008284
ansi_escape = re.compile(r"\x1B[#-_][0-?]*[ -/]*[#-~]")
system_prompt_wildcard = r".*xsh \$ " # backslash because prompt is "xsh $ "
# tests is {command:str, responses:[str]}
for test in tests:
c.sendline(test["cmd"])
response = c.expect([system_prompt_wildcard, pexpect.EOF, pexpect.TIMEOUT]) #=> (0|1|2)
if response != 0: # any error
continue
response_text = c.after.split('\n')[1:]
for expected, actual in zip(test['responses'], response_text):
norm_a = ansi_escape.sub('', norm_input.sub('', actual.strip()))
result = re.compile(norm_a).findall(expected)
if not len(result):
print('NO MATCH FOUND')
In an attempt to recreate the getenvironment(..) C function of _winapi.c (direct link) in plain python using ctypes, I'm wondering how the following C code could be translated:
buffer = PyMem_NEW(Py_UCS4, totalsize);
if (! buffer) {
PyErr_NoMemory();
goto error;
}
p = buffer;
end = buffer + totalsize;
for (i = 0; i < envsize; i++) {
PyObject* key = PyList_GET_ITEM(keys, i);
PyObject* value = PyList_GET_ITEM(values, i);
if (!PyUnicode_AsUCS4(key, p, end - p, 0))
goto error;
p += PyUnicode_GET_LENGTH(key);
*p++ = '=';
if (!PyUnicode_AsUCS4(value, p, end - p, 0))
goto error;
p += PyUnicode_GET_LENGTH(value);
*p++ = '\0';
}
/* add trailing null byte */
*p++ = '\0';
It seems that the function ctypes.create_unicode_buffer(..) (doc, code) is doing something quite close that I could reproduce if only I could have an access to Py_UCS4 C type or be sure of its link to any other type accessible to python through ctypes.
Would c_wchar be a good candidate ?, but it seems I can't make that assumption, as python 2.7 could be compiled in UCS-2 if I'm right (source), and I guess windows is really waiting fo UCS-4 there... even if it seems that ctypes.wintypes.LPWSTR is an alias to c_wchart_p in cPython 2.7 (code).
For this question, it is safe to make the assumption that the target platform is python 2.7 on Windows if that helps.
Context (if it has some importance):
I'm in the process of delving for the first time in ctypes to attempt a plain python fix at cPython 2.7's bug hitting windows subprocess.Popen(..) implementation. This bug is a won't fix. This bug prevents the usage of unicode in command line calls (as executable name or arguments). This is fixed in python 3, so I'm having a go at reverse implementing in plain python the actual cPython3 implementation of the required CreateProcess(..) in _winapi.c which calls in turn getenvironment(..).
This possible workaround was mentionned in the comments of this answer to a question related to subprocess.Popen(..) unicode issues.
This doesn't answer the part in the title about build specifically UCS4 buffer. But it gives a partial answer to the question in bold and manage to create a unicode buffer that seems to work on my current python 2.7 on windows: (so maybe UCS4 is not required).
So we are here taking the assumption that c_wchar is what windows require (if it is UCS4 or UCS2 is not so clear to me yet, and it might have no importance, but I recon having a very light confidence in my knowledge here).
So here is the python code that reproduces the C code as requested in the question:
## creation of buffer of size totalsize
wenv = (c_wchar * totalsize)()
wenv.value = (unicode("").join([
unicode("%s=%s\0") % (key, value)
for k, v in env.items()])) + "\0"
This wenv can then be fed to CreateProcessW and this seems to work.
I want to retrieve bibtex data (for building a bibliography) by sending a DOI (Digital Object Identifier) to http://www.crossref.org from within matlab.
The crossref API suggests something like this:
curl -LH "Accept: text/bibliography; style=bibtex" http://dx.doi.org/10.1038/nrd842
based on this source.
Another example from here suggests the following in ruby:
open("http://dx.doi.org/10.1038/nrd842","Accept" => "text/bibliography; style=bibtex"){|f| f.each {|line| print line}}
Although I've heard ruby rocks I want to do this in matlab and have no clue how to translate the ruby message or interpret the crossref command.
The following is what I have so far to send a doi to crossref and retrieve data in xml (in variable retdat), but not bibtex, format:
clear
clc
doi = '10.1038/nrd842';
URL_PATTERN = 'http://dx.doi.org/%s';
fetchurl = sprintf(URL_PATTERN,doi);
numinputs = 1;
www = java.net.URL(fetchurl);
is = www.openStream;
%Read stream of data
isr = java.io.InputStreamReader(is);
br = java.io.BufferedReader(isr);
%Parse return data
retdat = [];
next_line = toCharArray(br.readLine)'; %First line contains headings, determine length
%Loop through data
while ischar(next_line)
retdat = [retdat, 13, next_line];
tmp = br.readLine;
try
next_line = toCharArray(tmp)';
if strcmp(next_line,'M END')
next_line = [];
break
end
catch
break;
end
end
%Cleanup java objects
br.close;
isr.close;
is.close;
Help translating the ruby statement to something matlab can send using a script such as that posted to establish the communication with crossref would be greatly appreciated.
Edit:
Additional constraints include backward compatibility of the code (back at least to R14) :>(. Also, no use of ruby, since that solves the problem but is not a "matlab" solution, see here for how to invoke ruby from matlab via system('ruby script.rb').
You can easily edit urlread for what you need. I won't post my modified urlread function code due to copyright.
In urlread, (mine is at C:\Program Files\MATLAB\R2012a\toolbox\matlab\iofun\urlread.m), as the least elegant solution:
Right before "% Read the data from the connection." I added:
urlConnection.setRequestProperty('Accept','text/bibliography; style=bibtex');
The answer from user2034006 lays the path to a solution.
The following script works when urlread is modified:
URL_PATTERN = 'http://dx.doi.org/%s';
doi = '10.1038/nrd842';
fetchurl = sprintf(URL_PATTERN,doi);
method = 'post';
params= {};
[string,status] = urlread(fetchurl,method,params);
The modification in urlread is not identical to the suggestion of user2034006. Things worked when the line
urlConnection.setRequestProperty('Content-Type','application/x-www-form-urlencoded');
in urlread was replaced with
urlConnection.setRequestProperty('Accept','text/bibliography; style=bibtex');
I worked today in a simple script to checksum files in all available hashlib algorithms (md5, sha1.....) I wrote it and debug it with Python2, but when I decided to port it to Python 3 it just won't work. The funny thing is that it works for small files, but not for big files. I thought there was a problem with the way I was buffering the file, but the error message is what makes me think it is something related to the way I am doing the hexdigest (I think) Here is a copy of my entire script, so feel free to copy it, use it and help me figure out what the problem is with it. The error I get when checksuming a 250 MB file is
"'utf-8' codec can't decode byte 0xf3 in position 10: invalid continuation byte"
I google it, but can't find anything that fixes it. Also if you see better ways to optimize it, please let me know. My main goal is to make work 100% in Python 3. Thanks
#!/usr/local/bin/python33
import hashlib
import argparse
def hashFile(algorithm = "md5", filepaths=[], blockSize=4096):
algorithmType = getattr(hashlib, algorithm.lower())() #Default: hashlib.md5()
#Open file and extract data in chunks
for path in filepaths:
try:
with open(path) as f:
while True:
dataChunk = f.read(blockSize)
if not dataChunk:
break
algorithmType.update(dataChunk.encode())
yield algorithmType.hexdigest()
except Exception as e:
print (e)
def main():
#DEFINE ARGUMENTS
parser = argparse.ArgumentParser()
parser.add_argument('filepaths', nargs="+", help='Specified the path of the file(s) to hash')
parser.add_argument('-a', '--algorithm', action='store', dest='algorithm', default="md5",
help='Specifies what algorithm to use ("md5", "sha1", "sha224", "sha384", "sha512")')
arguments = parser.parse_args()
algo = arguments.algorithm
if algo.lower() in ("md5", "sha1", "sha224", "sha384", "sha512"):
Here is the code that works in Python 2, I will just put it in case you want to use it without having to modigy the one above.
#!/usr/bin/python
import hashlib
import argparse
def hashFile(algorithm = "md5", filepaths=[], blockSize=4096):
'''
Hashes a file. In oder to reduce the amount of memory used by the script, it hashes the file in chunks instead of putting
the whole file in memory
'''
algorithmType = hashlib.new(algorithm) #getattr(hashlib, algorithm.lower())() #Default: hashlib.md5()
#Open file and extract data in chunks
for path in filepaths:
try:
with open(path, mode = 'rb') as f:
while True:
dataChunk = f.read(blockSize)
if not dataChunk:
break
algorithmType.update(dataChunk)
yield algorithmType.hexdigest()
except Exception as e:
print e
def main():
#DEFINE ARGUMENTS
parser = argparse.ArgumentParser()
parser.add_argument('filepaths', nargs="+", help='Specified the path of the file(s) to hash')
parser.add_argument('-a', '--algorithm', action='store', dest='algorithm', default="md5",
help='Specifies what algorithm to use ("md5", "sha1", "sha224", "sha384", "sha512")')
arguments = parser.parse_args()
#Call generator function to yield hash value
algo = arguments.algorithm
if algo.lower() in ("md5", "sha1", "sha224", "sha384", "sha512"):
for hashValue in hashFile(algo, arguments.filepaths):
print hashValue
else:
print "Algorithm {0} is not available in this script".format(algorithm)
if __name__ == "__main__":
main()
I haven't tried it in Python 3, but I get the same error in Python 2.7.5 for binary files (the only difference is that mine is with the ascii codec). Instead of encoding the data chunks, open the file directly in binary mode:
with open(path, 'rb') as f:
while True:
dataChunk = f.read(blockSize)
if not dataChunk:
break
algorithmType.update(dataChunk)
yield algorithmType.hexdigest()
Apart from that, I'd use the method hashlib.new instead of getattr, and hashlib.algorithms_available to check if the argument is valid.
I need to mimic what MySQL does when encrypting and decrypting strings using built-in functions AES_ENCRYPT() and AES_DECRYPT().
I have read a couple of blog posts and apparently MySQL uses AES 128-bit encryption for those functions. On top of that, since this encryption requires a 16-bit key, MySQL pads the string with x0 chars (\0s) until it's 16-bit in size.
The algorithm in C from MySQL source code is spotted here.
Now I need to replicate what MySQL does in a Rails application, but every single thing I tried, doesn't work.
Here's a way to replicate the behavior I am getting:
1) Create a new Rails app
rails encryption-test
cd encryption-test
2) Create a new scaffolding
script/generate scaffold user name:string password:binary
3) Edit your config/database.yml and add a test MySQL database
development:
adapter: mysql
host: localhost
database: test
user: <<user>>
password: <<password>>
4) Run the migration
rake db:migrate
5) Enter console, create an user and update its password from MySQL query
script/console
Loading development environment (Rails 2.2.2)
>> User.create(:name => "John Doe")
>> key = "82pjd12398JKBSDIGUSisahdoahOUASDHsdapdjqwjeASIduAsdh078asdASD087asdADSsdjhA7809asdajhADSs"
>> ActiveRecord::Base.connection.execute("UPDATE users SET password = AES_ENCRYPT('password', '#{key}') WHERE name='John Doe'")
That's where I got stuck. If I attempt to decrypt it, using MySQL it works:
>> loaded_user = User.find_by_sql("SELECT AES_DECRYPT(password, '#{key}') AS password FROM users WHERE id=1").first
>> loaded_user['password']
=> "password"
However if I attempt to use OpenSSL library, there's no way I can make it work:
cipher = OpenSSL::Cipher::Cipher.new("AES-128-ECB")
cipher.padding = 0
cipher.key = key
cipher.decrypt
user = User.find(1)
cipher.update(user.password) << cipher.final #=> "########gf####\027\227"
I have tried padding the key:
desired_length = 16 * ((key.length / 16) + 1)
padded_key = key + "\0" * (desired_length - key.length)
cipher = OpenSSL::Cipher::Cipher.new("AES-128-ECB")
cipher.key = key
cipher.decrypt
user = User.find(1)
cipher.update(user.password) << cipher.final #=> ""|\e\261\205:\032s\273\242\030\261\272P##"
But it really doesn't work.
Does anyone have a clue on how can I mimic the MySQL AES_ENCRYPT() and AES_DECRYPT() functions behavior in Ruby?
Thanks!
For future reference:
According to the blog post I sent before, here's how MySQL works with
the key you provide AES_ENCRYPT / DECRYPT:
"The algorithm just creates a 16 byte
buffer set to all zero, then loops
through all the characters of the
string you provide and does an
assignment with bitwise OR between the
two values. If we iterate until we
hit the end of the 16 byte buffer, we
just start over from the beginning
doing ^=. For strings shorter than 16
characters, we stop at the end of the
string."
I don't know if you can read C, but here's the mentioned snippet:
http://pastie.org/425161
Specially this part:
bzero((char*) rkey,AES_KEY_LENGTH/8); /* Set initial key */
for (ptr= rkey, sptr= key; sptr < key_end; ptr++,sptr++)
{
if (ptr == rkey_end)
ptr= rkey; /* Just loop over tmp_key until we used all key */
*ptr^= (uint8) *sptr;
}
So I came up with this method (with a help from Rob Biedenharn, from ruby forum):
def mysql_key(key)
final_key = "\0" * 16
key.length.times do |i|
final_key[i%16] ^= key[i]
end
final_key
end
That, given a string returns the key MySQL uses when encrypting and decrypting. So all you need now is:
def aes(m,k,t)
(aes = OpenSSL::Cipher::AES128.new("ECB").send(m)).key = k
aes.update(t) << aes.final
end
def encrypt(key, text)
aes(:encrypt, key, text)
end
def decrypt(key, text)
aes(:decrypt, key, text)
end
To use openssl lib, built into ruby, and then you can make the two "final" methods:
def mysql_encrypt(s, key)
encrypt(mysql_key(key), s)
end
def mysql_decrypt(s, key)
decrypt(mysql_key(key), s)
end
And you're set! Also, complete code can be found in this Gist:
http://gist.github.com/84093
:-)
Generally you don't want to pad the key, you pad/unpad the data to be encrypted/decrypted. That could be another source of problems. I suggest using test data of a complete number of blocks to eliminate this possibility.
Also, I suspect the key for the OpenSSL API requires a "literal" key, not an ASCII representation of the key as you have in your code.
Given the paucity of the OpenSSL ruby docs and if you speak a little Java, you may want to prototype in JRuby with the BouncyCastle provider - this is something that I've done to good effect when working with TwoFish (not present in OpenSSL API).
EDIT: I re-read your comment about padding the key. You have some bits/bytes confusion in your question, and I'm not sure how this applies in any case since your posted key is 89 characters (712 bits) in length. Perhaps you should try with a 128 bit key/password to eliminate this padding phenomenon?
Incidentally, MySQL devs should be spanked for weak crypto, there are better ways to stretch passwords than by simply padding with zero bytes :(
If you don't mind using an openssl implementation attr_encrypted is a gem that will allow drop-in encryption on most classes, ActiveRecord or not. It unfortunately will not be compatible with MySQL's AES_EN/DECRYPT functions though.