I'm building an API that accepts JSON data POSTed to it.
I have the following user struct and recently I changed the password datatype to []byte from string so that it "plays nicely" with the bcrypt package.
type User struct {
Id string `json:"id,omitempty"`
Email string `json:"email,omitempty"`
Username string `json:"username,omitempty"`
Password []byte `json:"password,omitempty"`
Name string `json:"name,omitempty"`
}
However, I'm now getting an internal error returned in the JSON response illegal base64 data at input byte 4 when a user is POSTed with a password of 5 or more characters to the API. There are no issues if the password is 4 or less characters.
I've pinpointed the error to this block of code:
err := json.NewDecoder(req.Body).Decode(User)
if err != nil && err != io.EOF {
return err
}
Any ideas on a fix?
The problem lies in using []byte instead of string for password. This is because the encoding/json package will expect a base64 encoded string when decoding to []byte.
The documentation for encoding/json says:
Array and slice values encode as JSON arrays, except that []byte encodes as a base64-encoded string, and a nil slice encodes as the null JSON object.
So, just change it to string:
Password string `json:"password,omitempty"`
When you want to use it with bcrypt, then you just convert the string to []byte:
[]byte(user.Password)
If you want to extract a byte array directly from JSON, you can use the datatype *json.RawMessage, which is defined at encoding/json/stream.go:247 (in Go 1.9) as:
type RawMessage []byte
This special type instructs the unmarshaller to skip decoding and simply create a slice of the relevant bytes. Since this is still just a byte array though, you can use it directly with bcrypt functions:
type User struct {
...
Password *json.RawMessage `json:"password,omitempty"`
...
}
...
err := bcrypt.CompareHashAndPassword(*user.Password, storedPassword)
Note the pointer dereference (*) when reading the value.
Related
I've seen several articles with similar errors, but none seems to work for me. I've seen the marbles samples, as well as many others, and still can't pinpoint the error.
I'm in fabric 2.x. The chaincode below works just fine when saving data, but fails when reading with the following message:
ERROR] Error submitting transaction: No valid responses from any peers. Errors:
peer=org1peer-api.127-0-0-1.nip.io:8080, status=500, message=Error handling success response. Value did not match schema:
1. return: Additional property field1 is not allowed
2. return: Additional property field2 is not allowed
3. return: field1,omitempty is required
4. return: field2,omitempty is required
type Asset struct {
Field1 string `json:"field1,omitempty"`
Field2 string `json:"field2,omitempty"`
}
func (c *AssetContract) CreateAsset(ctx contractapi.TransactionContextInterface, assetID string, values string) (bool, error) {
// convert json input to byte array
txData := []byte(values)
// convert byte array to HlpAsset struct
asset := new(asset)
err = json.Unmarshal(txData, &asset)
// convert struct back to bytes
txBytes, err := json.Marshal(asset)
return true, ctx.GetStub().PutState(assetID, txBytes)
}
func (c *AssetContract) ReadAsset(ctx contractapi.TransactionContextInterface, assetID string) (*Asset, error) {
txBytes, _ := ctx.GetStub().GetState(assetID)
// convert byte array to HlpAsset struct
asset := new(Asset)
err = json.Unmarshal(txBytes, &asset)
return asset, nil
}
testing with the following input data:
assetID: "3",
values: "{\"field1\":\"123\",\"field2\":\"a05\"}"
In addition, I'm not exactly sure why I need to Unmarshal/Marshal. Couldn't I just convert the stringified JSON to byte and save that? I know that works, is it "only" for data validation purposes that this is required?
Anyway, thanks a bunch.
In omitempty fields, set also metadata:",optional" to pass validation. And you cannot set json:"omitempty" metadata:",optional" on all fields of your model. The default Fabric 2.X Go chaincode's transaction serializer does not like it for any reason. If you really need omitempty for those fields, you can add any other dumb field.
You can add a dumb boolean...
type Asset struct {
Dumb bool `json:"dumb"`
Field1 string `json:"field1,omitempty" metadata:",optional"`
Field2 string `json:"field2,omitempty" metadata:",optional"`
}
...or add the key/ID itself to the model...
type Asset struct {
ID string `json:"id"`
Field1 string `json:"field1,omitempty" metadata:",optional"`
Field2 string `json:"field2,omitempty" metadata:",optional"`
}
...or a document type...
type Asset struct {
DocType string `json:"docType"`
Field1 string `json:"field1,omitempty" metadata:",optional"`
Field2 string `json:"field2,omitempty" metadata:",optional"`
}
As an alternative, you can try to override default ContractChaincode's TransactionSerializer (https://pkg.go.dev/github.com/hyperledger/fabric-contract-api-go/contractapi#ContractChaincode). I've done it when migrating a chaincode that already had its own input validation from 1.X to 2.X, to avoid metadata checks and to return seralization errors in my own format; and may work for your case.
Or you can return a string instead of an *Asset from ReadAsset as a workaround to avoid the check that is causing your error, so that it is deserialized only in the client. In fact, I find not much coherent to receive an string in CreateAsset, but return an *Asset in ReadAsset. I would use the same format for both (*Asset, preferably, unless you are stuck with your issue).
I found this question where this bug is mentioned:
Supplying additional valid data in JSON tag of struct property causes schema to fail.
The bug has been closed for around a year, however, I still experience that behavior. Overriding the JSON property using medatada didn't work either.
Using the struct like this worked perfectly:
type Asset struct {
Field1 string `json:"field1"`
Field2 string `json:"field2"`
}
I have an ECDSA public key that that is returned to me from an HSM in ASN.1 DER format. I need to create a bitcoin compatible key 33 byte. When I print out key in hex.EncodeToString(pubkey) I get the following output:
3056301006072a8648ce3d020106052b8104000a034200049bb8e80670371f45508b5f8f59946a7c4dea4b3a23a036cf24c1f40993f4a1daad1716de8bd664ecb4596648d722a4685293de208c1d2da9361b9cba74c3d1ec
I use an online decoder here: https://holtstrom.com/michael/tools/asn1decoder.php
And it outputs:
0x049bb8e80670371f45508b5f8f59946a7c4dea4b3a23a036cf24c1f40993f4a1daad1716de8bd664ecb4596648d722a4685293de208c1d2da9361b9cba74c3d1ec
I can then take that and hex.DecodeString(str) which gives me the necessary format to input this into addrPubKey, err := btcutil.NewAddressPubKey(bs, &chaincfg.TestNet3Params).
How do I decode this in golang to get the 0x049... output?
Thanks
The first thing we need is to use the encoding/asn1 package from the standard library.
You only have to give go the right struct to decode into. From your link we can see that we have a SEQUENCE that contains another SEQUENCE with two OBJECTIDENTIFIER and a BITSTRING. In go this will be:
type Ids struct {
OBi1 asn1.ObjectIdentifier
OBi2 asn1.ObjectIdentifier
}
type PubKey struct {
Id Ids
Bs asn1.BitString
}
Now we only have to UnMarshall the data to this structure:
str := `3056301006072a8648ce3d020106052b8104000a034200049bb8e80670371f45508b5f8f59946a7c4dea4b3a23a036cf24c1f40993f4a1daad1716de8bd664ecb4596648d722a4685293de208c1d2da9361b9cba74c3d1ec`
bstring, err := hex.DecodeString(str)
if (err != nil) {
panic(err)
}
var decode PubKey
_, err = asn1.Unmarshal(bstring, &decode)
if (err != nil) {
panic(err)
}
fmt.Println(hex.EncodeToString(decode.Bs.Bytes))
Note that you don't have to encode the string to hex and back again, since Unmarshall accepts a byte array
This will print the expected result:
049bb8e80670371f45508b5f8f59946a7c4dea4b3a23a036cf24c1f40993f4a1daad1716de8bd664ecb4596648d722a4685293de208c1d2da9361b9cba74c3d1ec
Once again you probably don't need to encode to string.
I tried to using database/sql for query database row into a Go type, my codes snippet following:
type User struct {
user_id int64
user_name string
user_mobile string
password string
email interface{}
nickname string
level byte
locked bool
create_time string
comment string // convert <nil> to *string error
}
func TestQueryUser(t *testing.T) {
db := QueryUser(driverName, dataSourceName)
stmtResults, err := db.Prepare(queryAll)
defer stmtResults.Close()
var r *User = new(User)
arr := []interface{}{
&r.user_id, &r.user_name, &r.user_mobile, &r.password, &r.email,
&r.nickname, &r.level, &r.locked, &r.create_time, &r.comment,
}
err = stmtResults.QueryRow(username).Scan(arr...)
if err != nil {
t.Error(err.Error())
}
fmt.Println(r.email)
}
MySQL:
As your see, some fields that has NULL value, so I have to set interface{} type into User struct of Go, which convert NULL to nil.
--- FAIL: TestQueryUser (0.00s)
user_test.go:48: sql: Scan error on column index 9: unsupported Scan, storing driver.Value type <nil> into type *string
Somebody has a better way？ or I must change the MySQL field and set its DEFAULT ' '
First the short answer : There is some types in sql package for example sql.NullString (for nullable string in your table and guess Nullint64 and NullBool and ... usage :) ) and you should use them in your struct.
The long one : There is two interface for this available in go , first is Scanner and the other is Valuer for any special type in database, (for example,I use this mostly with JSONB in postgres) you need to create a type, and implement this two(or one of them) interface on that type.
the scanner is used when you call Scan function. the data from the database driver, normally in []byte is the input and you are responsible for handling it. the other one, is used when the value is used as input in query. the result "normally" is a slice of byte (and an error) if you need to only read data, Scanner is enough, and vice-versa, if you need to write parameter in query the Valuer is enough
for an example of implementation, I recommend to see the types in sql package.
Also there is an example of a type to use with JSONB/JSON type in postgresql
// GenericJSONField is used to handle generic json data in postgres
type GenericJSONField map[string]interface{}
// Scan convert the json field into our type
func (v *GenericJSONField) Scan(src interface{}) error {
var b []byte
switch src.(type) {
case []byte:
b = src.([]byte)
case string:
b = []byte(src.(string))
case nil:
b = make([]byte, 0)
default:
return errors.New("unsupported type")
}
return json.Unmarshal(b, v)
}
// Value try to get the string slice representation in database
func (v GenericJSONField) Value() (driver.Value, error) {
return json.Marshal(v)
}
driver value is often []byte but string and nil is acceptable. so this could handle null-able fields too.
I tried to decode a valid (based on my understanding) base64 encoded string in Go with:
data, err := base64.StdEncoding.DecodeString(s)
if err != nil {
...
}
A full example is here. I have a string "eyJlbWFpbF9hZGRyZXNzIjoiIiwiZXhwIjoxNDQ3NzIzMzY4LCJmaXJzdG5hbWUiOiIiLCJpYXQiOjE0NDc0NjQxNjgsImlzcyI6Imh0dHA6Ly91ZGFjaXR5LmNvbSIsImtpZCI6ImE3ZTg5ZWQyMSIsImxhc3RuYW1lIjoiIiwidXNlcl9pZCI6IjEyMzQ1Njc4IiwidXNlcm5hbWUiOiJoYW5zb2xvQGhvdGguY29tIn0", which can be properly decoded for example here
or even in your browser's console with atob(that_string);, but for some reason go complains with:
illegal base64 data at input byte 236
Notice, that I can decode some other strings. So why can not I base64decode a valid encoded string in Go?
Your input does not have any padding. Therefore, you should use base64.RawStdEncoding over base64.StdEncoding:
data, err := base64.RawStdEncoding.DecodeString(s)
Example: https://play.golang.org/p/ZWfzYXQ5Ye
type A struct {
Id int64
Email sql.NullString
Phone sql.NullString
}
Assume I have one record in the database
A{1, "x#x.com", "1112223333"}
Send an update request via PUT
curl -X PUT -d '{"Email": "y#y.com", "Phone": null}' http://localhost:3000/a/1
Here is the psuedo algorithm that would work with a full PUT request (i.e. update all fields of the record A - but it will cause difficulties with the PATCH request semantics - delta update)
-- Unmarshal json into empty record
a := A{}
json.Unmarshal([]byte(request.body), &a)
-- Load the record from the database
aFromDb = <assume we got record from db> //A{1, "x#x.com", "1112223333"}
-- Compare a and aFromDB
-- Notice the email change and set it on aFromDb - ok
-- Notice the phone number change -- but wait! Was it set to NULL in the JSON explicitly or was it not even included in the JSON? i.e. was the json request - {"Email": "y#y.com", "Phone": null} or was it {"Email": "y#y.com"}?
How can we tell by just looking at the unmarshaled json into the struct a?
Is there another method to do the update via rest (with patch semantics)? I am looking for a generic way to do it (not tied to a particular struct).
I created a separate datatype for this purpose. This example is for an int64 (actually string-encoded int64), but you can easily change it to a string as well. The idea behind it is, that the UnmarshalJSON method will only be called if the value is present in the JSON. The type will implement the Marshaler and the Unmarshaler.
// Used as a JSON type
//
// The Set flag indicates whether an unmarshaling actually happened on the type
type RequiredInt64 struct {
Set bool
Int64 int64
}
func (r RequiredInt64) MarshalJSON() ([]byte, error) {
lit := strconv.FormatInt(r.Int64, 10)
return json.Marshal(lit)
}
func (r *RequiredInt64) UnmarshalJSON(raw []byte) error {
var lit string
var err error
if err = json.Unmarshal(raw, &lit); err != nil {
return err
}
r.Int64, err = strconv.ParseInt(lit, 10, 64)
if err != nil {
return err
}
r.Set = true
return nil
}
So, if Set is false, you know that the value was not present in the JSON.
Try adding this tags to the struct:
type A struct {
Id int64 `json:"Id,omitempty"`
Email sql.NullString `json:"Email,omitempty"`
Phone sql.NullString `json:"Phone,omitempty"`
}
In this way if you are serializing and the field is empty then the json will not contain the field.
When deserializing though the field will have a either a value or it will have the default value for the type (Nil for the pointer or empty string for strings).
You could potentially write your own marshalling/uinmarshalling of your struct and react to the raw response within, although it might be non-obvious witgh inspection what that those functions are manipulating.
Alternatively, you could not omitempty within your fields and force null populated fields.
Or, maybe leveraging a different flavor of patching, perhaps http://jsonpatch.com/, which is more explicit in the nature of your modifications. This would require the client to be more understanding of the state of changes than say for a put.