This question already has an answer here:
How to transfer hex strings to []byte directly in Go?
(1 answer)
Closed 12 months ago.
I have the following hex data created by converting 5 values(consists of name, numeric and date field) to TLV
0115426f627320426173656d656e74205265636f726473020f3130303032353930363730303030330314323032322d30342d32355431353a33303a30305a040a323130303130302e393905093331353031352e3135
This hex data needs to be further encoded to Base64. I wrote the below code for that
func TLVsToBase64(v string) string { // v - the TLV in hex format
encodedTLV := b64.StdEncoding.EncodeToString([]byte(v))
return encodedTLV
}
The output(which is wrong) of the aforementioned hex data is below:
MDExNTQyNmY2MjczMjA0MjYxNzM2NTZkNjU2ZTc0MjA1MjY1NjM2ZjcyNjQ3MzAyMGYzMTMwMzAzMDMyMzUzOTMwMzYzNzMwMzAzMDMwMzMwMzE0MzIzMDMyMzIyZDMwMzQyZDMyMzU1NDMxMzUzYTMzMzAzYTMwMzA1YTA0MGEzMjMxMzAzMDMxMzAzMDJlMzkzOTA1MDkzMzMxMzUzMDMxMzUyZTMxMzU=
The desired output is:
ARVCb2JzIEJhc2VtZW50IFJlY29yZHMCDzEwMDAyNTkwNjcwMDAwMwMUMjAyMi0wNC0yNVQxNTozMDowMFoECjIxMDAxMDAuOTkFCTMxNTAxNS4xNQ==
I am new to Go, so please help me to troubleshoot the issue. I might missed something
Your input is the hexadecimal representation of some data. And your expected output is not the Base64 encoding of the UTF-8 data of the hex representation, but rather the data (the bytes) the hex encoding represent, so first decode the bytes e.g. using hex.DecodeString():
func TLVsToBase64(v string) (string, error) { // v - the TLV in hex format
data, err := hex.DecodeString(v)
if err != nil {
return "", err
}
encodedTLV := base64.StdEncoding.EncodeToString(data)
return encodedTLV, nil
}
Testing it:
s := "0115426f627320426173656d656e74205265636f726473020f3130303032353930363730303030330314323032322d30342d32355431353a33303a30305a040a323130303130302e393905093331353031352e3135"
fmt.Println(TLVsToBase64(s))
Output is what you expect (try it on the Go Playground):
ARVCb2JzIEJhc2VtZW50IFJlY29yZHMCDzEwMDAyNTkwNjcwMDAwMwMUMjAyMi0wNC0yNVQxNTozMDowMFoECjIxMDAxMDAuOTkFCTMxNTAxNS4xNQ== <nil>
Related
Sorry if it's a stupid question or I didn't give enough information. I have a string which should represent an ID: "\x8f\x04.\x8b8\x8e\nP\xbd\xe3\vLf\xd6W*\x92vb\x8b2", and I'm confused on what it is? I try to decode it with utf-8, utf-16, and gbk but none of them works. I realized the \x means hexadecimal, but what is \v and \nP?
The text in the question looks like binary data encoded to a Go interpreted string literal. Use strconv.Unquote to convert the text back to binary data:
s, err := strconv.Unquote(`"\x8f\x04.\x8b8\x8e\nP\xbd\xe3\vLf\xd6W*\x92vb\x8b2"`)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%x\n", s) // prints 8f042e8b388e0a50bde30b4c66d6572a9276628b32
fmt.Printf("%q\n", s) // prints "\x8f\x04.\x8b8\x8e\nP\xbd\xe3\vLf\xd6W*\x92vb\x8b2"
The Go language specification defines the syntax. The \n represents a byte with the value 10. The \v represents a byte with the value 11. The \xXX is hexadecimal as noted in the question.
I have an ECDSA public key that that is returned to me from an HSM in ASN.1 DER format. I need to create a bitcoin compatible key 33 byte. When I print out key in hex.EncodeToString(pubkey) I get the following output:
3056301006072a8648ce3d020106052b8104000a034200049bb8e80670371f45508b5f8f59946a7c4dea4b3a23a036cf24c1f40993f4a1daad1716de8bd664ecb4596648d722a4685293de208c1d2da9361b9cba74c3d1ec
I use an online decoder here: https://holtstrom.com/michael/tools/asn1decoder.php
And it outputs:
0x049bb8e80670371f45508b5f8f59946a7c4dea4b3a23a036cf24c1f40993f4a1daad1716de8bd664ecb4596648d722a4685293de208c1d2da9361b9cba74c3d1ec
I can then take that and hex.DecodeString(str) which gives me the necessary format to input this into addrPubKey, err := btcutil.NewAddressPubKey(bs, &chaincfg.TestNet3Params).
How do I decode this in golang to get the 0x049... output?
Thanks
The first thing we need is to use the encoding/asn1 package from the standard library.
You only have to give go the right struct to decode into. From your link we can see that we have a SEQUENCE that contains another SEQUENCE with two OBJECTIDENTIFIER and a BITSTRING. In go this will be:
type Ids struct {
OBi1 asn1.ObjectIdentifier
OBi2 asn1.ObjectIdentifier
}
type PubKey struct {
Id Ids
Bs asn1.BitString
}
Now we only have to UnMarshall the data to this structure:
str := `3056301006072a8648ce3d020106052b8104000a034200049bb8e80670371f45508b5f8f59946a7c4dea4b3a23a036cf24c1f40993f4a1daad1716de8bd664ecb4596648d722a4685293de208c1d2da9361b9cba74c3d1ec`
bstring, err := hex.DecodeString(str)
if (err != nil) {
panic(err)
}
var decode PubKey
_, err = asn1.Unmarshal(bstring, &decode)
if (err != nil) {
panic(err)
}
fmt.Println(hex.EncodeToString(decode.Bs.Bytes))
Note that you don't have to encode the string to hex and back again, since Unmarshall accepts a byte array
This will print the expected result:
049bb8e80670371f45508b5f8f59946a7c4dea4b3a23a036cf24c1f40993f4a1daad1716de8bd664ecb4596648d722a4685293de208c1d2da9361b9cba74c3d1ec
Once again you probably don't need to encode to string.
Why does Golang base64 to Hex produce a different encoding value than the online converter?
Original String:
ARVIN
Base64 encoded:
QVJWSU4=
Golang (base64 to hex):
51564a575355343d
Online (base64 to hex):
415256494e
package main
import (
"encoding/hex"
"fmt"
)
func main() {
base64 := "QVJWSU4="
hx := hex.EncodeToString([]byte(base64))
fmt.Println("Original String: ARVIN")
fmt.Println()
fmt.Println(base64 + " ==> " + hx)
}
You can convert the un-encoded value directly to hex:
h := hex.EncodeToString([]byte("ARVIN"))
fmt.Println(h) // prints 415256494e
Given the program starts with the base64 encoding, the program must decode the base64 string to bytes and then encode the bytes to a hex string. This is what the online tool does.
The code in the question encodes the base64 string to a hex string. It's missing the decode step.
Here's how to re-encode base64 to hex in Go:
p, err := base64.StdEncoding.DecodeString("QVJWSU4=")
if err != nil {
// handle error
}
h := hex.EncodeToString(p)
fmt.Println(h) // prints 415256494e
Run it in the playground.
Example string:
"\u0410\u043b\u0435\u043a\u0441\u0430\u043d\u0434\u0440\u044b! \n\u0421\u043f\u0430\u0441\u0438\u0431\u043e \ud83d\udcf8 link.ru \u0437\u0430 \n#hashtag Русское слово, an English word"
Without this \ud83d\udcf8 my func works well:
func convertUnicode(text string) string {
s, err := strconv.Unquote(`"` + text + `"`)
if err != nil {
// Error.Printf("can't convert: %s | err: %s\n", text, err)
return text
}
return s
}
My question is how to detect that text contains this kind of entries? And how to convert it to emoji or how to remove from the text? Thanks
Well, probably not so simple as neither \ud83d nor \udcf8 are valid code points but together are a surrogate pair used in UTF-16 encoding to encode \U0001F4F8. Now strconv.Unquote will give you two surrogate halves which you have to combine yourself.
Use strconv.Unquote to unquote as you did.
Convert to []rune for convenience.
Find surrogate pairs with unicode/utf16.IsSurrogate.
Combine surrogate pairs with unicode/utf16.DecodeRune.
Convert back to string.
I tried to decode a valid (based on my understanding) base64 encoded string in Go with:
data, err := base64.StdEncoding.DecodeString(s)
if err != nil {
...
}
A full example is here. I have a string "eyJlbWFpbF9hZGRyZXNzIjoiIiwiZXhwIjoxNDQ3NzIzMzY4LCJmaXJzdG5hbWUiOiIiLCJpYXQiOjE0NDc0NjQxNjgsImlzcyI6Imh0dHA6Ly91ZGFjaXR5LmNvbSIsImtpZCI6ImE3ZTg5ZWQyMSIsImxhc3RuYW1lIjoiIiwidXNlcl9pZCI6IjEyMzQ1Njc4IiwidXNlcm5hbWUiOiJoYW5zb2xvQGhvdGguY29tIn0", which can be properly decoded for example here
or even in your browser's console with atob(that_string);, but for some reason go complains with:
illegal base64 data at input byte 236
Notice, that I can decode some other strings. So why can not I base64decode a valid encoded string in Go?
Your input does not have any padding. Therefore, you should use base64.RawStdEncoding over base64.StdEncoding:
data, err := base64.RawStdEncoding.DecodeString(s)
Example: https://play.golang.org/p/ZWfzYXQ5Ye