I tried to decode a valid (based on my understanding) base64 encoded string in Go with:
data, err := base64.StdEncoding.DecodeString(s)
if err != nil {
...
}
A full example is here. I have a string "eyJlbWFpbF9hZGRyZXNzIjoiIiwiZXhwIjoxNDQ3NzIzMzY4LCJmaXJzdG5hbWUiOiIiLCJpYXQiOjE0NDc0NjQxNjgsImlzcyI6Imh0dHA6Ly91ZGFjaXR5LmNvbSIsImtpZCI6ImE3ZTg5ZWQyMSIsImxhc3RuYW1lIjoiIiwidXNlcl9pZCI6IjEyMzQ1Njc4IiwidXNlcm5hbWUiOiJoYW5zb2xvQGhvdGguY29tIn0", which can be properly decoded for example here
or even in your browser's console with atob(that_string);, but for some reason go complains with:
illegal base64 data at input byte 236
Notice, that I can decode some other strings. So why can not I base64decode a valid encoded string in Go?
Your input does not have any padding. Therefore, you should use base64.RawStdEncoding over base64.StdEncoding:
data, err := base64.RawStdEncoding.DecodeString(s)
Example: https://play.golang.org/p/ZWfzYXQ5Ye
Related
I am trying to decode proto data. But the proto is not properly decoded.
This is how I am doing:
decodedStr, err := base64.StdEncoding.DecodeString(request.Body)
if err != nil {
panic("malformed input")
}
data := &tracepb.ExportTraceServiceRequest{}
if err := proto.Unmarshal(decodedStr, data); err != nil {
log.Fatalln("Failed to parse:", err)
}
log.Printf("Response - %v", data)
The output is like this:
Response - resource_spans:{resource:{attributes:{key:"service.name" value:{string_value:"node_app"}} attributes:{key:"telemetry.sdk.language" value:{string_value:"nodejs"}} attributes:{key:"telemetry.sdk.name" value:{string_value:"opentelemetry"}} attributes:{key:"telemetry.sdk.version" value:{string_value:"1.8.0"}} attributes:{key:"process.pid" value:{int_value:1}} attributes:{key:"process.executable.name" value:{string_value:"node"}} attributes:{key:"process.command" value:{string_value:"/usr/app/index.js"}} attributes:{key:"process.command_line" value:{string_value:"/usr/local/bin/node /usr/app/index.js"}} attributes:{key:"process.runtime.version" value:{string_value:"18.13.0"}} attributes:{key:"process.runtime.name" value:{string_value:"nodejs"}} attributes:{key:"process.runtime.description" value:{string_value:"Node.js"}}} scope_spans:{scope:{name:"#opentelemetry/instrumentation-express" version:"0.32.0"} spans:{trace_id:"\xb5\x81\x91\x8b\x02\x9a/\xf1\x08\x06\xaf~\xea\x9fQ\xc0" span_id:"T\x06\x89m\x1ex\xf9A" parent_span_id:"?\xbc\x18`O\xa5\xb8\xe1" name:"middleware - query" kind:SPAN_KIND_INTERNAL start_time_unix_nano:1673434036590614272 end_time_unix_nano:1673434036590671104 attributes:{key:"http.route" value:{string_value:"/"}} attributes:{key:"express.name" value:{string_value:"query"}} attributes:{key:"express.type" value:{string_value:"middleware"}} status:{}} spans:{trace_id:"\xb5\x81\x91\x8b\x02\x9a/\xf1\x08\x06\xaf~\xea\x9fQ\xc0" span_id:"\xd5c\xf7>\xf6Cxz" parent_span_id:"?\xbc\x18`O\xa5\xb8\xe1" name:"middleware - expressInit" kind:SPAN_KIND_INTERNAL start_time_unix_nano:1673434036590760704
Not sure why traceId is shown like this:
spans:{trace_id:"\xb5\x81\x91\x8b\x02\x9a/\xf1\x08\x06\xaf~\xea\x9fQ\xc0"
Am new to GoLang. Any help would be greatly appreciated
The trace_id field appears to contain the ID as binary data instead of hex. The generated proto String method will render the binary data as a string. Hence non-printable characters as displayed as ASCII escape sequences.
If you want to display the data in other format (eg, Hex) you will need to implement your own function to render the proto.
Used encoding/hex module's hex.EncodeToString() function to convert bytes to hex
This question already has an answer here:
How to transfer hex strings to []byte directly in Go?
(1 answer)
Closed 12 months ago.
I have the following hex data created by converting 5 values(consists of name, numeric and date field) to TLV
0115426f627320426173656d656e74205265636f726473020f3130303032353930363730303030330314323032322d30342d32355431353a33303a30305a040a323130303130302e393905093331353031352e3135
This hex data needs to be further encoded to Base64. I wrote the below code for that
func TLVsToBase64(v string) string { // v - the TLV in hex format
encodedTLV := b64.StdEncoding.EncodeToString([]byte(v))
return encodedTLV
}
The output(which is wrong) of the aforementioned hex data is below:
MDExNTQyNmY2MjczMjA0MjYxNzM2NTZkNjU2ZTc0MjA1MjY1NjM2ZjcyNjQ3MzAyMGYzMTMwMzAzMDMyMzUzOTMwMzYzNzMwMzAzMDMwMzMwMzE0MzIzMDMyMzIyZDMwMzQyZDMyMzU1NDMxMzUzYTMzMzAzYTMwMzA1YTA0MGEzMjMxMzAzMDMxMzAzMDJlMzkzOTA1MDkzMzMxMzUzMDMxMzUyZTMxMzU=
The desired output is:
ARVCb2JzIEJhc2VtZW50IFJlY29yZHMCDzEwMDAyNTkwNjcwMDAwMwMUMjAyMi0wNC0yNVQxNTozMDowMFoECjIxMDAxMDAuOTkFCTMxNTAxNS4xNQ==
I am new to Go, so please help me to troubleshoot the issue. I might missed something
Your input is the hexadecimal representation of some data. And your expected output is not the Base64 encoding of the UTF-8 data of the hex representation, but rather the data (the bytes) the hex encoding represent, so first decode the bytes e.g. using hex.DecodeString():
func TLVsToBase64(v string) (string, error) { // v - the TLV in hex format
data, err := hex.DecodeString(v)
if err != nil {
return "", err
}
encodedTLV := base64.StdEncoding.EncodeToString(data)
return encodedTLV, nil
}
Testing it:
s := "0115426f627320426173656d656e74205265636f726473020f3130303032353930363730303030330314323032322d30342d32355431353a33303a30305a040a323130303130302e393905093331353031352e3135"
fmt.Println(TLVsToBase64(s))
Output is what you expect (try it on the Go Playground):
ARVCb2JzIEJhc2VtZW50IFJlY29yZHMCDzEwMDAyNTkwNjcwMDAwMwMUMjAyMi0wNC0yNVQxNTozMDowMFoECjIxMDAxMDAuOTkFCTMxNTAxNS4xNQ== <nil>
I am verifying the identity of the sender of a piece of data. I am provided the RSA public key in a PEM format and I know the data is passed through the SHA256 hashing function. The equivalent verification on the node.js platform:
Ticket.prototype.verify = function (ticket) {
if (!ticket) return null;
var pubkey = fs.readFileSync('/etc/SCAMP/auth/ticket_verify_public_key.pem');
var parts = ticket.split(',');
if (parts[0] != '1') return null;
var sig = new Buffer(parts.pop().replace(/-/g,'+').replace(/_/g,'/'), 'base64');
var valid = crypto.createVerify('sha256').update( new Buffer(parts.join(',')) ).verify( pubkey, sig )
Which can verify:
1,3063,21,1438783424,660,1+20+31+32+34+35+36+37+38+39+40+41+42+43+44+46+47+48+50+53+56+59+60+61+62+67+68+69+70+71+75+76+80+81+82+86+87+88+102+104+105+107+109+110+122+124,PcFNyWjoz_iiVMgEe8I3IBfzSlUcqUGtsuN7536PTiBW7KDovIqCaSi_8nZWcj-j1dfbQRA8mftwYUWMhhZ4DD78-BH8MovNVucbmTmf2Wzbx9bsI-dmUADY5Q2ol4qDXG4YQJeyZ6f6F9s_1uxHTH456QcsfNxFWh18ygo5_DVmQQSXCHN7EXM5M-u2DSol9MSROeBolYnHZyE093LgQ2veWQREbrwg5Fcp2VZ6VqIC7yu6f_xYHEvU0-ZsSSRMAMUmhLNhmFM4KDjl8blVgC134z7XfCTDDjCDiynSL6b-D-
by splitting on the last ,. The left side of the split is the ticket data I care about, the right side is the signature which I need to verify before I can use the ticket data.
I have tried to port the logic to go:
func TestSigVerification(t *testing.T) {
block, _ := pem.Decode(signingPubKey)
if block == nil {
t.Errorf("expected to block to be non-nil CERTIFICATE", block)
}
key, err := x509.ParsePKIXPublicKey(block.Bytes)
if err != nil {
t.Errorf("could not parse PKIXPublicKey: `%s`", key)
}
rsaPubKey, ok := key.(*rsa.PublicKey)
if !ok {
t.Errorf("couldn't cast to rsa.PublicKey!")
}
ticket,_ := ParseTicketBytes(fullTicketBytes)
h := sha256.New()
h.Write(ticketBytes)
digest := h.Sum(nil)
err = rsa.VerifyPKCS1v15(rsaPubKey, crypto.SHA256, digest, ticket.Signature)
if err != nil {
t.Errorf("could not verify ticket: `%s` (digest: `%v`)", err, digest )
}
}
But I'm pretty sure VerifyPKCS1v15 is not equivalent to node's crypto.createVerify and this test case fails. What should I be using? How can I use the public key to decrypt the signature and get the sha256? once I have the decrypted sha256 value I could just do a basic comparison with the sha256 I have generated.
Here's a runnable playground example: http://play.golang.org/p/COx2OG-AiA
Though I couldn't get it to work, I suspect the issue is that you'll need to convert the sig from base64 into bytes via the base64 encoding. See this example here:
http://play.golang.org/p/bzpD7Pa9mr (especially lines 23 to 28, where they have to encode the sig from bytes to base64 string to print it, then feed the byte version into the sig check, indicating that you have to use the byte version and not base64 string)
Which I stumbled across on this post:
Signing and decoding with RSA-SHA in GO
I've found that golang generally expects bytes everywhere in byte encoding. I tried to decode your sig string from base64 to bytes however, even after replacing the '-' with '+' and the '_' with '/' it still won't work, for reasons unknown to me:
http://play.golang.org/p/71IiV2z_t8
At the very least this seems to indicate that maybe your sig is bad? If it isn't valid base64? I think if you can find a way to solve this the rest should work!
I tried running your code and it doesn't look like your ticket is well formed. It's not long enough. Where did you get the value from?
Double check that ticket -- that may just be enough to get you going.
I'm building an API that accepts JSON data POSTed to it.
I have the following user struct and recently I changed the password datatype to []byte from string so that it "plays nicely" with the bcrypt package.
type User struct {
Id string `json:"id,omitempty"`
Email string `json:"email,omitempty"`
Username string `json:"username,omitempty"`
Password []byte `json:"password,omitempty"`
Name string `json:"name,omitempty"`
}
However, I'm now getting an internal error returned in the JSON response illegal base64 data at input byte 4 when a user is POSTed with a password of 5 or more characters to the API. There are no issues if the password is 4 or less characters.
I've pinpointed the error to this block of code:
err := json.NewDecoder(req.Body).Decode(User)
if err != nil && err != io.EOF {
return err
}
Any ideas on a fix?
The problem lies in using []byte instead of string for password. This is because the encoding/json package will expect a base64 encoded string when decoding to []byte.
The documentation for encoding/json says:
Array and slice values encode as JSON arrays, except that []byte encodes as a base64-encoded string, and a nil slice encodes as the null JSON object.
So, just change it to string:
Password string `json:"password,omitempty"`
When you want to use it with bcrypt, then you just convert the string to []byte:
[]byte(user.Password)
If you want to extract a byte array directly from JSON, you can use the datatype *json.RawMessage, which is defined at encoding/json/stream.go:247 (in Go 1.9) as:
type RawMessage []byte
This special type instructs the unmarshaller to skip decoding and simply create a slice of the relevant bytes. Since this is still just a byte array though, you can use it directly with bcrypt functions:
type User struct {
...
Password *json.RawMessage `json:"password,omitempty"`
...
}
...
err := bcrypt.CompareHashAndPassword(*user.Password, storedPassword)
Note the pointer dereference (*) when reading the value.
I am getting: error: illegal base64 data at input byte 4
When passing in Base64Image into base64.StdEncoding.DecodeString(str):
data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAGQAAABkCAYA...
Let me know if you need the full base64, I have just pasted in the first part as it looks like the problem is within 4 byte?
data, errBase := base64.StdEncoding.DecodeString(Base64Image)
if errBase != nil {
fmt.Println("error:", errBase)
return false
}
Do you know why?
Not all of your input string you try to decode is Base64 encoded form.
What you have is a Data URI scheme, that provides a way to include data in-line in web pages as if they were external resources.
It has a format of:
data:[<MIME-type>][;charset=<encoding>][;base64],<data>
Where in your case image/png is the MIME-type, the optional charset is missing, and ";base64" is a constant string indicating that <data> is encoded using Base64 encoding.
To acquire the data (that is the Base64 encoded form), cut off the prefix up to the comma (comma included):
input := "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAGQAAABkCAYA"
b64data := input[strings.IndexByte(input, ',')+1:]
fmt.Println(b64data)
Output:
iVBORw0KGgoAAAANSUhEUgAAAGQAAABkCAYA
Of which you can now decode:
data, err := base64.StdEncoding.DecodeString(b64data)
if err != nil {
fmt.Println("error:", err)
}
fmt.Println(data)
Output:
[137 80 78 71 13 10 26 10 0 0 0 13 73 72 68 82 0 0 0 100 0 0 0 100 8 6 0]
Try it on the Go Playground.
Also this may happen if you encoded using StdEncoding and trying to decode using RawStdEncoding
Sometimes this happens if your base64 string is not properly padded with == at the end.
It's because your string isn't in base64 until after the comma data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAGQAAABkCAYA...
import "strings" and use split to get the half after the comma and then call decodestring with that.
import "strings"
data, errBase := base64.StdEncoding.DecodeString(strings.Split(Base64Image, "base64,")[1]))
if errBase != nil {
fmt.Println("error:", errBase)
return false
}
EDIT: made the split token base64, because it's more specific to your input.