I am trying to convert a Node.js script to Golang. However, I always get the error "error: illegal base64 data at input byte 40" while decoding Base64. May someone help me, already debug and read documentation for few hours... Help is needed!
// Node.js exampe (code copied from Facebook Instant game SDK example)
const CryptoJS = require('crypto-js');
var firstpart = signedRequest.split('.')[0];
firstpart = firstpart.replace(/-/g, '+').replace(/_/g, '/');
const signature = CryptoJS.enc.Base64.parse(firstpart).toString(); // <-- fail here
const dataHash = CryptoJS.HmacSHA256(signedRequest.split('.')[1], '<APP_SECRET>').toString();
var isValid = signature === dataHash;
const json = CryptoJS.enc.Base64.parse(signedRequest.split('.')[1]).toString(CryptoJS.enc.Utf8);
const data = JSON.parse(json);
Code above is the sample code from Facebook, and the following code (below) is written by me.
parts := strings.Split(signedRequest, ".")
firstPart := parts[0]
replaced := strings.Replace(firstPart, "-", "+", -1)
replaced = strings.Replace(replaced, "_", "/", -1)
signatureByte, err := base64.StdEncoding.DecodeString(replaced) // <-- ERROR here
if err != nil {
fmt.Println("error:", err)
return false, err
}
signature := string(signatureByte)
dataHash := createHmacSHA256(parts[1], "<APP_SECRET>") // TODO: not sure, to string or hex string?
isValid := signature == dataHash
if isValid {
return true, nil
}
The Go Playground is at here https://play.golang.org/p/ilSbqamFdV_-
First of all, get rid of the strings.Replaces and just use base64.URLEncoding instead of StdEncoding, since the URL alphabet is clearly what your data is in.
Also, standard base64 data is padded, but your data is not padded, so you need the "raw" encoding, i.e. base64.RawURLEncoding. This works:
firstPart := parts[0]
signatureByte, err := base64.RawURLEncoding.DecodeString(firstPart)
https://play.golang.org/p/Pj_LLfirU8M
See the base64 package docs and the base64 standard for more info.
Related
I have an ECDSA public key that that is returned to me from an HSM in ASN.1 DER format. I need to create a bitcoin compatible key 33 byte. When I print out key in hex.EncodeToString(pubkey) I get the following output:
3056301006072a8648ce3d020106052b8104000a034200049bb8e80670371f45508b5f8f59946a7c4dea4b3a23a036cf24c1f40993f4a1daad1716de8bd664ecb4596648d722a4685293de208c1d2da9361b9cba74c3d1ec
I use an online decoder here: https://holtstrom.com/michael/tools/asn1decoder.php
And it outputs:
0x049bb8e80670371f45508b5f8f59946a7c4dea4b3a23a036cf24c1f40993f4a1daad1716de8bd664ecb4596648d722a4685293de208c1d2da9361b9cba74c3d1ec
I can then take that and hex.DecodeString(str) which gives me the necessary format to input this into addrPubKey, err := btcutil.NewAddressPubKey(bs, &chaincfg.TestNet3Params).
How do I decode this in golang to get the 0x049... output?
Thanks
The first thing we need is to use the encoding/asn1 package from the standard library.
You only have to give go the right struct to decode into. From your link we can see that we have a SEQUENCE that contains another SEQUENCE with two OBJECTIDENTIFIER and a BITSTRING. In go this will be:
type Ids struct {
OBi1 asn1.ObjectIdentifier
OBi2 asn1.ObjectIdentifier
}
type PubKey struct {
Id Ids
Bs asn1.BitString
}
Now we only have to UnMarshall the data to this structure:
str := `3056301006072a8648ce3d020106052b8104000a034200049bb8e80670371f45508b5f8f59946a7c4dea4b3a23a036cf24c1f40993f4a1daad1716de8bd664ecb4596648d722a4685293de208c1d2da9361b9cba74c3d1ec`
bstring, err := hex.DecodeString(str)
if (err != nil) {
panic(err)
}
var decode PubKey
_, err = asn1.Unmarshal(bstring, &decode)
if (err != nil) {
panic(err)
}
fmt.Println(hex.EncodeToString(decode.Bs.Bytes))
Note that you don't have to encode the string to hex and back again, since Unmarshall accepts a byte array
This will print the expected result:
049bb8e80670371f45508b5f8f59946a7c4dea4b3a23a036cf24c1f40993f4a1daad1716de8bd664ecb4596648d722a4685293de208c1d2da9361b9cba74c3d1ec
Once again you probably don't need to encode to string.
i have the following JWT payload:
{"exp": 4724377561} (some date 100 years from now)
Encoding it in Go yields ewogICAiZXhwIjogNDcyNDM3NzU2MQp9
Using jwt.io it is however encoded to eyJleHAiOjQ3MjQzNzc1NjF9 which yields a different signature when signed. I use jwt.io's signatures in test fixtures.
I would like to not use 3rd party JWT tools, to keep my dependencies slim.
I am suspecting some sort of character encoding is the issue here.
To keep the tests readable, I am using plain-text JSON in the fixtures.
The way i use my test fixtures is the following (omitting other test cases):
import (
"encoding/base64"
"reflect"
"testing"
)
var testData = []struct {
name string
header string
payload string
signature string
pass bool
errorType reflect.Type
}{{
name: "Succeed if token not expired",
header: `{"typ":"JWT","alg":"HS256"}`,
payload: `{"exp": 4724377561}`,
signature: "JHtMKvPSMa5BD22BsbxiP1-ELRh1XkIKkarRSev0ZjU",
pass: true,
}}
func TestParseJwt(t *testing.T) {
HmacSecret = []byte("My super secret key")
for _, tst := range testData {
jwt64 :=
base64.RawURLEncoding.EncodeToString([]byte(tst.header)) + "." +
base64.RawURLEncoding.EncodeToString([]byte(tst.payload)) + "." +
tst.signature
_, err := ParseJwt(jwt64)
if tst.pass {
if err != nil {
t.Fatal(tst.name, err)
}
} else {
// If an error was expected to be thrown, assert that it is the correct one.
if reflect.TypeOf(err).String() != tst.errorType.String() {
t.Fatal(tst.name, err)
}
}
}
}
The difference comes simply from the library "compacting" the JSON before applying Base64 encoding.
See this example:
ss := []string{
`{"exp": 4724377561}`,
`{"exp":4724377561}`,
}
for _, s := range ss {
fmt.Println(base64.RawURLEncoding.EncodeToString([]byte(s)), s)
}
Output (try it on the Go Playground):
eyJleHAiOiA0NzI0Mzc3NTYxfQ {"exp": 4724377561}
eyJleHAiOjQ3MjQzNzc1NjF9 {"exp":4724377561}
The second output matches what you expect. To remove insignificant whitespace in Go use json.Compact.
converting back the encoded strings on https://www.base64decode.org/ tells you what's going on:
ewogICAiZXhwIjogNDcyNDM3NzU2MQp9
is decoded to
{
"exp": 4724377561
}
while the jwt.io string is decoded to
{"exp":4724377561}
so https://jwt.io trims all whitespace and linebreaks before encoding.
As an exercise, I'm trying to implement a mock SMTP server with CRAM-MD5 authentication in Go (without following RFC 2195, since it looks like it doesn't matter to the client what format the pre-hashed challenge is in; I also assume there is only one user "bob" with password "pass"). But I can't seem to get it right as the hash in response is always different from what I have on the server. I send the email using Go as such (running it as a separate package):
{...}
smtp.SendMail("localhost:25", smtp.CRAMMD5Auth("bob", "pass"),
"bob#localhost", []string{"alice#localhost"}, []byte("Hey Alice!\n"))
{...}
Here's what I do when I get the authentication acknowledgement from the client:
{...}
case strings.Contains(ms, "AUTH CRAM-MD5"):
rndbts = make([]byte, 16) // Declared at package level
b64b := make([]byte, base64.StdEncoding.EncodedLen(16))
rand.Read(rndbts)
base64.StdEncoding.Encode(b64b, rndbts)
_, err = conn.Write([]byte(fmt.Sprintf("334 %x\n", b64b)))
{...}
And this is what I do with the client's response:
{...}
{
ms = strings.TrimRight(ms, "\r\n") // The response to the challenge
ds, _ := base64.StdEncoding.DecodeString(ms)
s := strings.Split(string(ds), " ")
login := s[0] // I can get the login from the response.
h := hmac.New(md5.New, []byte("pass"))
h.Write(rndbts)
c := make([]byte, 0, ourHash.Size()) // From smtp/auth.go, not sure why we need this.
validPass := hmac.Equal(h.Sum(c), []byte(s[1]))
{...}
}
{...}
And the validPass is never true. I omitted error handling from the excerpts for brevity, but they're there in the actual code (though they're always nil). Why are the hashes different? I have looked at the source code for net/smtp, and it seems to me that I'm going in the right direction, but not quite.
I hope this helps! Runnable version is at https://play.golang.org/p/-8shx_IcLV. Also note that you'll need to fix your %x (should be %s) so you're sending the right challenge down to the client. Right now I think you're trying to hex-encode your base64 string.
Once you've fixed that, I believe this code should help you to construct the right response string on the server and compare it to what the client sent.
// Example values taken from http://susam.in/blog/auth-cram-md5/
challenge := []byte("<17893.1320679123#tesseract.susam.in>")
username := []byte("alice")
password := []byte("wonderland")
clientResponse := []byte("YWxpY2UgNjRiMmE0M2MxZjZlZDY4MDZhOTgwOTE0ZTIzZTc1ZjA=")
// hash the challenge with the user's password
h := hmac.New(md5.New, password)
h.Write(challenge)
hash := h.Sum(nil)
// encode the result in lowercase hexadecimal
hexEncoded := hex.EncodeToString(hash)
// prepend the username and a space
toEncode := []byte(string(username) + " " + hexEncoded)
// base64-encode the whole thing
b64Result := make([]byte, base64.StdEncoding.EncodedLen(len(toEncode)))
base64.StdEncoding.Encode(b64Result, toEncode)
// check that this is equal to what the client sent
if hmac.Equal(b64Result, clientResponse) {
fmt.Println("Matches!")
}
I am verifying the identity of the sender of a piece of data. I am provided the RSA public key in a PEM format and I know the data is passed through the SHA256 hashing function. The equivalent verification on the node.js platform:
Ticket.prototype.verify = function (ticket) {
if (!ticket) return null;
var pubkey = fs.readFileSync('/etc/SCAMP/auth/ticket_verify_public_key.pem');
var parts = ticket.split(',');
if (parts[0] != '1') return null;
var sig = new Buffer(parts.pop().replace(/-/g,'+').replace(/_/g,'/'), 'base64');
var valid = crypto.createVerify('sha256').update( new Buffer(parts.join(',')) ).verify( pubkey, sig )
Which can verify:
1,3063,21,1438783424,660,1+20+31+32+34+35+36+37+38+39+40+41+42+43+44+46+47+48+50+53+56+59+60+61+62+67+68+69+70+71+75+76+80+81+82+86+87+88+102+104+105+107+109+110+122+124,PcFNyWjoz_iiVMgEe8I3IBfzSlUcqUGtsuN7536PTiBW7KDovIqCaSi_8nZWcj-j1dfbQRA8mftwYUWMhhZ4DD78-BH8MovNVucbmTmf2Wzbx9bsI-dmUADY5Q2ol4qDXG4YQJeyZ6f6F9s_1uxHTH456QcsfNxFWh18ygo5_DVmQQSXCHN7EXM5M-u2DSol9MSROeBolYnHZyE093LgQ2veWQREbrwg5Fcp2VZ6VqIC7yu6f_xYHEvU0-ZsSSRMAMUmhLNhmFM4KDjl8blVgC134z7XfCTDDjCDiynSL6b-D-
by splitting on the last ,. The left side of the split is the ticket data I care about, the right side is the signature which I need to verify before I can use the ticket data.
I have tried to port the logic to go:
func TestSigVerification(t *testing.T) {
block, _ := pem.Decode(signingPubKey)
if block == nil {
t.Errorf("expected to block to be non-nil CERTIFICATE", block)
}
key, err := x509.ParsePKIXPublicKey(block.Bytes)
if err != nil {
t.Errorf("could not parse PKIXPublicKey: `%s`", key)
}
rsaPubKey, ok := key.(*rsa.PublicKey)
if !ok {
t.Errorf("couldn't cast to rsa.PublicKey!")
}
ticket,_ := ParseTicketBytes(fullTicketBytes)
h := sha256.New()
h.Write(ticketBytes)
digest := h.Sum(nil)
err = rsa.VerifyPKCS1v15(rsaPubKey, crypto.SHA256, digest, ticket.Signature)
if err != nil {
t.Errorf("could not verify ticket: `%s` (digest: `%v`)", err, digest )
}
}
But I'm pretty sure VerifyPKCS1v15 is not equivalent to node's crypto.createVerify and this test case fails. What should I be using? How can I use the public key to decrypt the signature and get the sha256? once I have the decrypted sha256 value I could just do a basic comparison with the sha256 I have generated.
Here's a runnable playground example: http://play.golang.org/p/COx2OG-AiA
Though I couldn't get it to work, I suspect the issue is that you'll need to convert the sig from base64 into bytes via the base64 encoding. See this example here:
http://play.golang.org/p/bzpD7Pa9mr (especially lines 23 to 28, where they have to encode the sig from bytes to base64 string to print it, then feed the byte version into the sig check, indicating that you have to use the byte version and not base64 string)
Which I stumbled across on this post:
Signing and decoding with RSA-SHA in GO
I've found that golang generally expects bytes everywhere in byte encoding. I tried to decode your sig string from base64 to bytes however, even after replacing the '-' with '+' and the '_' with '/' it still won't work, for reasons unknown to me:
http://play.golang.org/p/71IiV2z_t8
At the very least this seems to indicate that maybe your sig is bad? If it isn't valid base64? I think if you can find a way to solve this the rest should work!
I tried running your code and it doesn't look like your ticket is well formed. It's not long enough. Where did you get the value from?
Double check that ticket -- that may just be enough to get you going.
package main
import "fmt"
import "encoding/json"
type Track struct {
XmlRequest string `json:"xmlRequest"`
}
func main() {
message := new(Track)
message.XmlRequest = "<car><mirror>XML</mirror></car>"
fmt.Println("Before Marshal", message)
messageJSON, _ := json.Marshal(message)
fmt.Println("After marshal", string(messageJSON))
}
Is it possible to make json.Marshal not escape < and >? I currently get:
{"xmlRequest":"\u003ccar\u003e\u003cmirror\u003eXML\u003c/mirror\u003e\u003c/car\u003e"}
but I am looking for something like this:
{"xmlRequest":"<car><mirror>XML</mirror></car>"}
As of Go 1.7, you still cannot do this with json.Marshal(). The source code for json.Marshal shows:
> err := e.marshal(v, encOpts{escapeHTML: true})
The reason json.Marshal always does this is:
String values encode as JSON strings coerced to valid UTF-8,
replacing invalid bytes with the Unicode replacement rune.
The angle brackets "<" and ">" are escaped to "\u003c" and "\u003e"
to keep some browsers from misinterpreting JSON output as HTML.
Ampersand "&" is also escaped to "\u0026" for the same reason.
This means you cannot even do it by writing a custom func (t *Track) MarshalJSON(), you have to use something that does not satisfy the json.Marshaler interface.
So, the workaround, is to write your own function:
func (t *Track) JSON() ([]byte, error) {
buffer := &bytes.Buffer{}
encoder := json.NewEncoder(buffer)
encoder.SetEscapeHTML(false)
err := encoder.Encode(t)
return buffer.Bytes(), err
}
https://play.golang.org/p/FAH-XS-QMC
If you want a generic solution for any struct, you could do:
func JSONMarshal(t interface{}) ([]byte, error) {
buffer := &bytes.Buffer{}
encoder := json.NewEncoder(buffer)
encoder.SetEscapeHTML(false)
err := encoder.Encode(t)
return buffer.Bytes(), err
}
https://play.golang.org/p/bdqv3TUGr3
In Go1.7 the have added a new option to fix this:
encoding/json:
add Encoder.DisableHTMLEscaping This provides a way to disable the escaping of <, >, and & in JSON strings.
The relevant function is
func (*Encoder) SetEscapeHTML
That should be applied to a Encoder.
enc := json.NewEncoder(os.Stdout)
enc.SetEscapeHTML(false)
Simple example: https://play.golang.org/p/SJM3KLkYW-
This doesn't answer the question directly but it could be an answer if you're looking for a way how to deal with json.Marshal escaping < and >...
Another way to solve the problem is to replace those escaped characters in json.RawMessage into just valid UTF-8 characters, after the json.Marshal() call.
It will work as well for any letters other than < and >. (I used to do this to make non-English letters to be human readable in JSON :D)
func _UnescapeUnicodeCharactersInJSON(_jsonRaw json.RawMessage) (json.RawMessage, error) {
str, err := strconv.Unquote(strings.Replace(strconv.Quote(string(_jsonRaw)), `\\u`, `\u`, -1))
if err != nil {
return nil, err
}
return []byte(str), nil
}
func main() {
// Both are valid JSON.
var jsonRawEscaped json.RawMessage // json raw with escaped unicode chars
var jsonRawUnescaped json.RawMessage // json raw with unescaped unicode chars
// '\u263a' == '☺'
jsonRawEscaped = []byte(`{"HelloWorld": "\uC548\uB155, \uC138\uC0C1(\u4E16\u4E0A). \u263a"}`) // "\\u263a"
jsonRawUnescaped, _ = _UnescapeUnicodeCharactersInJSON(jsonRawEscaped) // "☺"
fmt.Println(string(jsonRawEscaped)) // {"HelloWorld": "\uC548\uB155, \uC138\uC0C1(\u4E16\u4E0A). \u263a"}
fmt.Println(string(jsonRawUnescaped)) // {"HelloWorld": "안녕, 세상(世上). ☺"}
}
https://play.golang.org/p/pUsrzrrcDG-
I hope this helps someone.
Here's my workaround:
// Marshal is a UTF-8 friendly marshaler. Go's json.Marshal is not UTF-8
// friendly because it replaces the valid UTF-8 and JSON characters "&". "<",
// ">" with the "slash u" unicode escaped forms (e.g. \u0026). It preemptively
// escapes for HTML friendliness. Where text may include any of these
// characters, json.Marshal should not be used. Playground of Go breaking a
// title: https://play.golang.org/p/o2hiX0c62oN
func Marshal(i interface{}) ([]byte, error) {
buffer := &bytes.Buffer{}
encoder := json.NewEncoder(buffer)
encoder.SetEscapeHTML(false)
err := encoder.Encode(i)
return bytes.TrimRight(buffer.Bytes(), "\n"), err
}
No, you can't.
A third-party json package might be the choice rather than the std json lib.
More detail:https://github.com/golang/go/issues/8592
I had a requirement to store xml inside json :puke:
At first I was having significant difficulty unmarshalling that xml after passing it via json, but my issue was actually due to trying to unmarshall the xml string as a json.RawMessage. I actually needed to unmarshall it as a string and then coerce it into []byte for the xml.Unmarshal.
type xmlInJson struct {
Data string `json:"data"`
}
var response xmlInJson
err := json.Unmarshall(xmlJsonData, &response)
var xmlData someOtherStructThatMatchesTheXmlFormat
err = xml.Unmarshall([]byte(response.Data), &xmlData)
Custom function is not kind of the best solution.
How about another library to solve this.
I use gabs
import
go get "github.com/Jeffail/gabs"
use
message := new(Track)
resultJson,_:=gabs.Consume(message)
fmt.Println(string(resultJson.EncodeJSON()))
I solve that problem like this.