I have an ECDSA public key that that is returned to me from an HSM in ASN.1 DER format. I need to create a bitcoin compatible key 33 byte. When I print out key in hex.EncodeToString(pubkey) I get the following output:
3056301006072a8648ce3d020106052b8104000a034200049bb8e80670371f45508b5f8f59946a7c4dea4b3a23a036cf24c1f40993f4a1daad1716de8bd664ecb4596648d722a4685293de208c1d2da9361b9cba74c3d1ec
I use an online decoder here: https://holtstrom.com/michael/tools/asn1decoder.php
And it outputs:
0x049bb8e80670371f45508b5f8f59946a7c4dea4b3a23a036cf24c1f40993f4a1daad1716de8bd664ecb4596648d722a4685293de208c1d2da9361b9cba74c3d1ec
I can then take that and hex.DecodeString(str) which gives me the necessary format to input this into addrPubKey, err := btcutil.NewAddressPubKey(bs, &chaincfg.TestNet3Params).
How do I decode this in golang to get the 0x049... output?
Thanks
The first thing we need is to use the encoding/asn1 package from the standard library.
You only have to give go the right struct to decode into. From your link we can see that we have a SEQUENCE that contains another SEQUENCE with two OBJECTIDENTIFIER and a BITSTRING. In go this will be:
type Ids struct {
OBi1 asn1.ObjectIdentifier
OBi2 asn1.ObjectIdentifier
}
type PubKey struct {
Id Ids
Bs asn1.BitString
}
Now we only have to UnMarshall the data to this structure:
str := `3056301006072a8648ce3d020106052b8104000a034200049bb8e80670371f45508b5f8f59946a7c4dea4b3a23a036cf24c1f40993f4a1daad1716de8bd664ecb4596648d722a4685293de208c1d2da9361b9cba74c3d1ec`
bstring, err := hex.DecodeString(str)
if (err != nil) {
panic(err)
}
var decode PubKey
_, err = asn1.Unmarshal(bstring, &decode)
if (err != nil) {
panic(err)
}
fmt.Println(hex.EncodeToString(decode.Bs.Bytes))
Note that you don't have to encode the string to hex and back again, since Unmarshall accepts a byte array
This will print the expected result:
049bb8e80670371f45508b5f8f59946a7c4dea4b3a23a036cf24c1f40993f4a1daad1716de8bd664ecb4596648d722a4685293de208c1d2da9361b9cba74c3d1ec
Once again you probably don't need to encode to string.
Related
I am trying to decode proto data. But the proto is not properly decoded.
This is how I am doing:
decodedStr, err := base64.StdEncoding.DecodeString(request.Body)
if err != nil {
panic("malformed input")
}
data := &tracepb.ExportTraceServiceRequest{}
if err := proto.Unmarshal(decodedStr, data); err != nil {
log.Fatalln("Failed to parse:", err)
}
log.Printf("Response - %v", data)
The output is like this:
Response - resource_spans:{resource:{attributes:{key:"service.name" value:{string_value:"node_app"}} attributes:{key:"telemetry.sdk.language" value:{string_value:"nodejs"}} attributes:{key:"telemetry.sdk.name" value:{string_value:"opentelemetry"}} attributes:{key:"telemetry.sdk.version" value:{string_value:"1.8.0"}} attributes:{key:"process.pid" value:{int_value:1}} attributes:{key:"process.executable.name" value:{string_value:"node"}} attributes:{key:"process.command" value:{string_value:"/usr/app/index.js"}} attributes:{key:"process.command_line" value:{string_value:"/usr/local/bin/node /usr/app/index.js"}} attributes:{key:"process.runtime.version" value:{string_value:"18.13.0"}} attributes:{key:"process.runtime.name" value:{string_value:"nodejs"}} attributes:{key:"process.runtime.description" value:{string_value:"Node.js"}}} scope_spans:{scope:{name:"#opentelemetry/instrumentation-express" version:"0.32.0"} spans:{trace_id:"\xb5\x81\x91\x8b\x02\x9a/\xf1\x08\x06\xaf~\xea\x9fQ\xc0" span_id:"T\x06\x89m\x1ex\xf9A" parent_span_id:"?\xbc\x18`O\xa5\xb8\xe1" name:"middleware - query" kind:SPAN_KIND_INTERNAL start_time_unix_nano:1673434036590614272 end_time_unix_nano:1673434036590671104 attributes:{key:"http.route" value:{string_value:"/"}} attributes:{key:"express.name" value:{string_value:"query"}} attributes:{key:"express.type" value:{string_value:"middleware"}} status:{}} spans:{trace_id:"\xb5\x81\x91\x8b\x02\x9a/\xf1\x08\x06\xaf~\xea\x9fQ\xc0" span_id:"\xd5c\xf7>\xf6Cxz" parent_span_id:"?\xbc\x18`O\xa5\xb8\xe1" name:"middleware - expressInit" kind:SPAN_KIND_INTERNAL start_time_unix_nano:1673434036590760704
Not sure why traceId is shown like this:
spans:{trace_id:"\xb5\x81\x91\x8b\x02\x9a/\xf1\x08\x06\xaf~\xea\x9fQ\xc0"
Am new to GoLang. Any help would be greatly appreciated
The trace_id field appears to contain the ID as binary data instead of hex. The generated proto String method will render the binary data as a string. Hence non-printable characters as displayed as ASCII escape sequences.
If you want to display the data in other format (eg, Hex) you will need to implement your own function to render the proto.
Used encoding/hex module's hex.EncodeToString() function to convert bytes to hex
i have the following JWT payload:
{"exp": 4724377561} (some date 100 years from now)
Encoding it in Go yields ewogICAiZXhwIjogNDcyNDM3NzU2MQp9
Using jwt.io it is however encoded to eyJleHAiOjQ3MjQzNzc1NjF9 which yields a different signature when signed. I use jwt.io's signatures in test fixtures.
I would like to not use 3rd party JWT tools, to keep my dependencies slim.
I am suspecting some sort of character encoding is the issue here.
To keep the tests readable, I am using plain-text JSON in the fixtures.
The way i use my test fixtures is the following (omitting other test cases):
import (
"encoding/base64"
"reflect"
"testing"
)
var testData = []struct {
name string
header string
payload string
signature string
pass bool
errorType reflect.Type
}{{
name: "Succeed if token not expired",
header: `{"typ":"JWT","alg":"HS256"}`,
payload: `{"exp": 4724377561}`,
signature: "JHtMKvPSMa5BD22BsbxiP1-ELRh1XkIKkarRSev0ZjU",
pass: true,
}}
func TestParseJwt(t *testing.T) {
HmacSecret = []byte("My super secret key")
for _, tst := range testData {
jwt64 :=
base64.RawURLEncoding.EncodeToString([]byte(tst.header)) + "." +
base64.RawURLEncoding.EncodeToString([]byte(tst.payload)) + "." +
tst.signature
_, err := ParseJwt(jwt64)
if tst.pass {
if err != nil {
t.Fatal(tst.name, err)
}
} else {
// If an error was expected to be thrown, assert that it is the correct one.
if reflect.TypeOf(err).String() != tst.errorType.String() {
t.Fatal(tst.name, err)
}
}
}
}
The difference comes simply from the library "compacting" the JSON before applying Base64 encoding.
See this example:
ss := []string{
`{"exp": 4724377561}`,
`{"exp":4724377561}`,
}
for _, s := range ss {
fmt.Println(base64.RawURLEncoding.EncodeToString([]byte(s)), s)
}
Output (try it on the Go Playground):
eyJleHAiOiA0NzI0Mzc3NTYxfQ {"exp": 4724377561}
eyJleHAiOjQ3MjQzNzc1NjF9 {"exp":4724377561}
The second output matches what you expect. To remove insignificant whitespace in Go use json.Compact.
converting back the encoded strings on https://www.base64decode.org/ tells you what's going on:
ewogICAiZXhwIjogNDcyNDM3NzU2MQp9
is decoded to
{
"exp": 4724377561
}
while the jwt.io string is decoded to
{"exp":4724377561}
so https://jwt.io trims all whitespace and linebreaks before encoding.
I need some help with unmarshaling. I have this example code:
package main
import (
"encoding/json"
"fmt"
)
type Obj struct {
Id string `json:"id"`
Data []byte `json:"data"`
}
func main() {
byt := []byte(`{"id":"someID","data":["str1","str2"]}`)
var obj Obj
if err := json.Unmarshal(byt, &obj); err != nil {
panic(err)
}
fmt.Println(obj)
}
What I try to do here - convert bytes to the struct, where type of one field is []byte. The error I get:
panic: json: cannot unmarshal string into Go struct field Obj.data of
type uint8
That's probably because parser already sees that "data" field is already a slice and tries to represent "str1" as some char bytecode (type uint8?).
How do I store the whole data value as one bytes array? Because I want to unmarshal the value to the slice of strings later. I don't include a slice of strings into struct because this type can change (array of strings, int, string, etc), I wish this to be universal.
My first recommendation would be for you to just use []string instead of []byte if you know the input type is going to be an array of strings.
If data is going to be a JSON array with various types, then your best option is to use []interface{} instead - Go will happily unmarshal the JSON for you and you can perform checks at runtime to cast those into more specific typed variables on an as-needed basis.
If []byte really is what you want, use json.RawMessage, which is of type []byte, but also implements the methods for JSON parsing. I believe this may be what you want, as it will accept whatever ends up in data. Of course, you then have to manually parse Data to figure out just what actually IS in there.
One possible bonus is that this skips any heavy parsing because it just copies the bytes over. When you want to use this data for something, you use a []interface{}, then use a type switch to use individual values.
https://play.golang.org/p/og88qb_qtpSGJ
package main
import (
"encoding/json"
"fmt"
)
type Obj struct {
Id string `json:"id"`
Data json.RawMessage `json:"data"`
}
func main() {
byt := []byte(`{"id":"someID","data":["str1","str2", 1337, {"my": "obj", "id": 42}]}`)
var obj Obj
if err := json.Unmarshal(byt, &obj); err != nil {
panic(err)
}
fmt.Printf("%+v\n", obj)
fmt.Printf("Data: %s\n", obj.Data)
// use it
var d []interface{}
if err := json.Unmarshal(obj.Data, &d); err != nil {
panic(err)
}
fmt.Printf("%+v\n", d)
for _, v := range d {
// you need a type switch to deterine the type and be able to use most of these
switch real := v.(type) {
case string:
fmt.Println("I'm a string!", real)
case float64:
fmt.Println("I'm a number!", real)
default:
fmt.Printf("Unaccounted for: %+v\n", v)
}
}
}
Your question is:
convert bytes array to struct with a field of type []byte
But you do not have a bytearray but a string array. Your question is not the same as your example. So let answer your question, there are more solutions possible depending in how far you want to diverge from your original requirements.
One string can be converted to one byte-slice, two strings need first to be transformed to one string. So that is problem one. The second problem are the square-brackets in your json-string
This works fine, it implicitly converts the string in the json-string to a byte-slice:
byt := []byte(`{"id":"someID","data":"str1str2"}`)
var obj Obj
if err := json.Unmarshal(byt, &obj); err != nil {
panic(err)
}
fmt.Println(obj)
I am verifying the identity of the sender of a piece of data. I am provided the RSA public key in a PEM format and I know the data is passed through the SHA256 hashing function. The equivalent verification on the node.js platform:
Ticket.prototype.verify = function (ticket) {
if (!ticket) return null;
var pubkey = fs.readFileSync('/etc/SCAMP/auth/ticket_verify_public_key.pem');
var parts = ticket.split(',');
if (parts[0] != '1') return null;
var sig = new Buffer(parts.pop().replace(/-/g,'+').replace(/_/g,'/'), 'base64');
var valid = crypto.createVerify('sha256').update( new Buffer(parts.join(',')) ).verify( pubkey, sig )
Which can verify:
1,3063,21,1438783424,660,1+20+31+32+34+35+36+37+38+39+40+41+42+43+44+46+47+48+50+53+56+59+60+61+62+67+68+69+70+71+75+76+80+81+82+86+87+88+102+104+105+107+109+110+122+124,PcFNyWjoz_iiVMgEe8I3IBfzSlUcqUGtsuN7536PTiBW7KDovIqCaSi_8nZWcj-j1dfbQRA8mftwYUWMhhZ4DD78-BH8MovNVucbmTmf2Wzbx9bsI-dmUADY5Q2ol4qDXG4YQJeyZ6f6F9s_1uxHTH456QcsfNxFWh18ygo5_DVmQQSXCHN7EXM5M-u2DSol9MSROeBolYnHZyE093LgQ2veWQREbrwg5Fcp2VZ6VqIC7yu6f_xYHEvU0-ZsSSRMAMUmhLNhmFM4KDjl8blVgC134z7XfCTDDjCDiynSL6b-D-
by splitting on the last ,. The left side of the split is the ticket data I care about, the right side is the signature which I need to verify before I can use the ticket data.
I have tried to port the logic to go:
func TestSigVerification(t *testing.T) {
block, _ := pem.Decode(signingPubKey)
if block == nil {
t.Errorf("expected to block to be non-nil CERTIFICATE", block)
}
key, err := x509.ParsePKIXPublicKey(block.Bytes)
if err != nil {
t.Errorf("could not parse PKIXPublicKey: `%s`", key)
}
rsaPubKey, ok := key.(*rsa.PublicKey)
if !ok {
t.Errorf("couldn't cast to rsa.PublicKey!")
}
ticket,_ := ParseTicketBytes(fullTicketBytes)
h := sha256.New()
h.Write(ticketBytes)
digest := h.Sum(nil)
err = rsa.VerifyPKCS1v15(rsaPubKey, crypto.SHA256, digest, ticket.Signature)
if err != nil {
t.Errorf("could not verify ticket: `%s` (digest: `%v`)", err, digest )
}
}
But I'm pretty sure VerifyPKCS1v15 is not equivalent to node's crypto.createVerify and this test case fails. What should I be using? How can I use the public key to decrypt the signature and get the sha256? once I have the decrypted sha256 value I could just do a basic comparison with the sha256 I have generated.
Here's a runnable playground example: http://play.golang.org/p/COx2OG-AiA
Though I couldn't get it to work, I suspect the issue is that you'll need to convert the sig from base64 into bytes via the base64 encoding. See this example here:
http://play.golang.org/p/bzpD7Pa9mr (especially lines 23 to 28, where they have to encode the sig from bytes to base64 string to print it, then feed the byte version into the sig check, indicating that you have to use the byte version and not base64 string)
Which I stumbled across on this post:
Signing and decoding with RSA-SHA in GO
I've found that golang generally expects bytes everywhere in byte encoding. I tried to decode your sig string from base64 to bytes however, even after replacing the '-' with '+' and the '_' with '/' it still won't work, for reasons unknown to me:
http://play.golang.org/p/71IiV2z_t8
At the very least this seems to indicate that maybe your sig is bad? If it isn't valid base64? I think if you can find a way to solve this the rest should work!
I tried running your code and it doesn't look like your ticket is well formed. It's not long enough. Where did you get the value from?
Double check that ticket -- that may just be enough to get you going.
package main
import "fmt"
import "encoding/json"
type Track struct {
XmlRequest string `json:"xmlRequest"`
}
func main() {
message := new(Track)
message.XmlRequest = "<car><mirror>XML</mirror></car>"
fmt.Println("Before Marshal", message)
messageJSON, _ := json.Marshal(message)
fmt.Println("After marshal", string(messageJSON))
}
Is it possible to make json.Marshal not escape < and >? I currently get:
{"xmlRequest":"\u003ccar\u003e\u003cmirror\u003eXML\u003c/mirror\u003e\u003c/car\u003e"}
but I am looking for something like this:
{"xmlRequest":"<car><mirror>XML</mirror></car>"}
As of Go 1.7, you still cannot do this with json.Marshal(). The source code for json.Marshal shows:
> err := e.marshal(v, encOpts{escapeHTML: true})
The reason json.Marshal always does this is:
String values encode as JSON strings coerced to valid UTF-8,
replacing invalid bytes with the Unicode replacement rune.
The angle brackets "<" and ">" are escaped to "\u003c" and "\u003e"
to keep some browsers from misinterpreting JSON output as HTML.
Ampersand "&" is also escaped to "\u0026" for the same reason.
This means you cannot even do it by writing a custom func (t *Track) MarshalJSON(), you have to use something that does not satisfy the json.Marshaler interface.
So, the workaround, is to write your own function:
func (t *Track) JSON() ([]byte, error) {
buffer := &bytes.Buffer{}
encoder := json.NewEncoder(buffer)
encoder.SetEscapeHTML(false)
err := encoder.Encode(t)
return buffer.Bytes(), err
}
https://play.golang.org/p/FAH-XS-QMC
If you want a generic solution for any struct, you could do:
func JSONMarshal(t interface{}) ([]byte, error) {
buffer := &bytes.Buffer{}
encoder := json.NewEncoder(buffer)
encoder.SetEscapeHTML(false)
err := encoder.Encode(t)
return buffer.Bytes(), err
}
https://play.golang.org/p/bdqv3TUGr3
In Go1.7 the have added a new option to fix this:
encoding/json:
add Encoder.DisableHTMLEscaping This provides a way to disable the escaping of <, >, and & in JSON strings.
The relevant function is
func (*Encoder) SetEscapeHTML
That should be applied to a Encoder.
enc := json.NewEncoder(os.Stdout)
enc.SetEscapeHTML(false)
Simple example: https://play.golang.org/p/SJM3KLkYW-
This doesn't answer the question directly but it could be an answer if you're looking for a way how to deal with json.Marshal escaping < and >...
Another way to solve the problem is to replace those escaped characters in json.RawMessage into just valid UTF-8 characters, after the json.Marshal() call.
It will work as well for any letters other than < and >. (I used to do this to make non-English letters to be human readable in JSON :D)
func _UnescapeUnicodeCharactersInJSON(_jsonRaw json.RawMessage) (json.RawMessage, error) {
str, err := strconv.Unquote(strings.Replace(strconv.Quote(string(_jsonRaw)), `\\u`, `\u`, -1))
if err != nil {
return nil, err
}
return []byte(str), nil
}
func main() {
// Both are valid JSON.
var jsonRawEscaped json.RawMessage // json raw with escaped unicode chars
var jsonRawUnescaped json.RawMessage // json raw with unescaped unicode chars
// '\u263a' == '☺'
jsonRawEscaped = []byte(`{"HelloWorld": "\uC548\uB155, \uC138\uC0C1(\u4E16\u4E0A). \u263a"}`) // "\\u263a"
jsonRawUnescaped, _ = _UnescapeUnicodeCharactersInJSON(jsonRawEscaped) // "☺"
fmt.Println(string(jsonRawEscaped)) // {"HelloWorld": "\uC548\uB155, \uC138\uC0C1(\u4E16\u4E0A). \u263a"}
fmt.Println(string(jsonRawUnescaped)) // {"HelloWorld": "안녕, 세상(世上). ☺"}
}
https://play.golang.org/p/pUsrzrrcDG-
I hope this helps someone.
Here's my workaround:
// Marshal is a UTF-8 friendly marshaler. Go's json.Marshal is not UTF-8
// friendly because it replaces the valid UTF-8 and JSON characters "&". "<",
// ">" with the "slash u" unicode escaped forms (e.g. \u0026). It preemptively
// escapes for HTML friendliness. Where text may include any of these
// characters, json.Marshal should not be used. Playground of Go breaking a
// title: https://play.golang.org/p/o2hiX0c62oN
func Marshal(i interface{}) ([]byte, error) {
buffer := &bytes.Buffer{}
encoder := json.NewEncoder(buffer)
encoder.SetEscapeHTML(false)
err := encoder.Encode(i)
return bytes.TrimRight(buffer.Bytes(), "\n"), err
}
No, you can't.
A third-party json package might be the choice rather than the std json lib.
More detail：https://github.com/golang/go/issues/8592
I had a requirement to store xml inside json :puke:
At first I was having significant difficulty unmarshalling that xml after passing it via json, but my issue was actually due to trying to unmarshall the xml string as a json.RawMessage. I actually needed to unmarshall it as a string and then coerce it into []byte for the xml.Unmarshal.
type xmlInJson struct {
Data string `json:"data"`
}
var response xmlInJson
err := json.Unmarshall(xmlJsonData, &response)
var xmlData someOtherStructThatMatchesTheXmlFormat
err = xml.Unmarshall([]byte(response.Data), &xmlData)
Custom function is not kind of the best solution.
How about another library to solve this.
I use gabs
import
go get "github.com/Jeffail/gabs"
use
message := new(Track)
resultJson,_:=gabs.Consume(message)
fmt.Println(string(resultJson.EncodeJSON()))
I solve that problem like this.