Can I copy the value of pointer of interface with tags in golang with unknown type
I have a function called "CensorPayload" it will receive a pointer and Unmarshal and censor some field that have tag called "censored" but I don't want to update value in argument called "target" that mean I have to copy the value of target to new interface{}. I try many way such as reflect.New but the tag does not pass through Unmarshal or Do function.
type HelloWorld struct {
Hello string `json:"hello"`
Password string `json:"password" censored:"*****"`
}
a := HelloWorld{}
CensorPayload(&a, `{"hello": "world", "password": "12345"}`)
func CensorPayload(target interface{}, payload string) (string, error) {
err := json.Unmarshal([]byte(payload), target)
if err != nil {
return "", err
}
err = Do(target)
if err != nil {
return "", err
}
censoredPayload, err := json.Marshal(target)
if err != nil {
return "", err
}
return string(censoredPayload), nil
}
There are quite a few ways to improve and accomplish this. I'll start with some improvements, and work my way up to possible implementations of what you're looking for.
Improvements
First up, I'd avoid using target interface{} as an argument. There is absolutely nothing there to guarantee that the caller will pass in a pointer. At the very least, I'd consider using a trick that protobuf uses to ensure arguments of the interface proto.Message are pointer values. This is done by adding an empty method with pointer receiver to the types. In this case, that would be:
type JSONPtr interface {
JSONPtr()
}
// implemented:
func (*HelloWorld) JSONPtr() {} // empty
func CensorPayload(target JSONPtr, payload string) (string, error) {
// pretty much the same as you have here
}
Another option would be to use generics, depending on how many types like this you actually need to handle, this could be a valid use-case for them:
type CensorTypes interface {
*HelloWorld | *AnotherType | *YetAnother
}
func CensorPayload[T CensorTypes](target T, payload string) (string, error) {
// same as what you have now
}
Reducing duplication with embedding
If you have a number of types with a Password field, all of which need to be capable of being censored, you could just split out the Password field into its own type and embed it where needed:
type Password struct {
Password string `json:"password"`
}
type HelloWorld struct {
// regular fields here
Password // embed the password field
}
In case you need to use this password censoring a lot, I'd definitely take this approach. Towards the end, I'll include an example where I use embedded types to accomplish what you're trying to do with minimal code duplication...
Possible implementations
All this does, however, is ensure that a pointer is passed, which the interface approach already does anyway. You want to be able to censor the password, and have a specific call that does this in your code, so why not implement a method on the relevant types, and add that to an interface. Adding generics, we can get a compile-time check to ensure that the relevant method(s) were implemented correctly (ie with a pointer receiver):
type Censored interface {
*HelloWorld | *AnotherType // and so on
CensorPass()
}
func (h *HelloWorld) CensorPass() { // pointer receiver
h.Password = "******"
}
// in case `Password` is an embedded type, no need to implement it on `HelloWorld`
func (p *Password) CensorPass() {
p.Password = "******"
}
func CenssorPayload[T Censored](target T, payload string) (string, error) {
// same as before until:
// Do() is replaced with:
target.CensorPass()
// the rest is the same
}
Now, if your HelloWorld type implements CensorPass with a value receiver (func (h HelloWorld) CensorPass), or the caller fails to pass in a pointer value as the target argument, you'll get an error.
Option 2: custom marshallers
Another option here is to create a wrapper type that embeds all of the types you need to handle, and implement a custom marshaller interface on said type:
type Wrapper struct {
*HelloWorld
*AnotherType
*YestAnother
}
func (w Wrapper) MarshalJSON() ([]byte, error) {
if w.HelloWorld != nil {
w.HelloWorld.Password = "*****"
return json.Marshal(w.HelloWorld)
}
if w.AnotherType != nil {
w.AnotherType.Pass = "*****"
w.AnotherType.AnotherSensitiveField = "*******"
return json.Marshal(w.AnotherType)
}
func w.YetAnother != nil {
w.YetAnother.User = "<censored>"
return json.Marshal(w.YetAnother)
}
}
Populating this Wrapper type can all be handled by a single function and a type-switch. I'm using any (short for interface{} below, but I'd recommend you use an interface or generics for the reasons given above if you want to call this method from other packages, if you're calling it from CensorPayload alone, and you've already changed it to use interfaces/generics, the code below is fine:
func wrap(target any) (*Wrapper, error) {
switch tt := target.(type) {
case *HelloWorld:
return &Wrapper{
HelloWorld: tt,
}, nil
case *AnotherType:
return &Wrapper{
AnotherType: tt,
}, nil
case *YetAnother:
return &Wrapper{
YetAnother: tt,
}, nil
}
// if we reach this point, the type passed to this function is unknown/can't be censored
return nil, errors.New("type not supported by wrapper")
}
With this in place, you can simply change your CensorPayload function to something like this:
// CensorPayload using interface to ensure pointer argument
func CensorPayload(target JSONPtr, payload string) (string, error) {
if err := json.Unmarshal(target, []byte(payload)); err != nil {
return "", err
}
wrapped, err := wrap(target)
if err != nil {
// type not handled by wrapper, either return an error
return "", err // target was not supported
// or in case that means this data should not be censored, just return the marhsalled raw data:
raw, err := json.Marshal(target)
if err != nil {
return "", err
}
return string(raw), nil
}
raw, err := json.Marshal(wrapped) // this will call Wrapper.MarshalJSON and censor what needs to be censored
if err != nil {
return "", err
}
return string(raw), nil
}
What I'd go for
I'd probably use the Censored type in the CensorPayload function. It has the benefit of not needing the wrapper at all, shifts the responsibility of knowing what fields/values to censor on to the data-types, and ensures that both the CensorPass method is implemented correctly (pointer receiver), and the actual value being passed is a pointer. To reduce the total LoC to a minimum, I'd leave it to the caller to cast the []byte return value to a string, too. Same goes for the payload argument. Raw JSON data is represented as a byte slice, so the payload argument, IMO, should reflect this:
func CensorPayload[T Censored](target T, payload []byte) ([]byte, error) {
if err := json.Unmarshal(target, payload); err != nil {
return nil, err
}
target.CensorPass() // type knows what fields to censor
return json.Marshal(target) // return the data censored & marshalled
}
As mentioned above, I'd combine this approach with commonly censored fields being embedded types:
type Password struct {
Password `json:"password"`
}
type AnotherSensitiveField struct {
AnotherSensitiveField string `json:"example_db_ip"`
}
// Password implements CensorPass, is embedded so we don't need to do anything here
type HelloWorld struct {
// fields
Password
}
// this one has 2 fields that implement CensorPass
// we need to ensure both are called
type AnotherType struct {
// fields
Password
AnotherSensitiveField
}
// this is a one-off, so we need to implement CensorPass on the type
type YetAnother struct {
// all fields - no password
User string `json:"user"` // this one needs to be censored
}
func (a *AnotherType) CensorPass() {
a.Password.CensorPass()
a.AnotherSensitiveField.CensorPass()
}
func (y *YetAnother) CensorPass() {
y.User = "<censored>"
}
func (p *Password) CensorPass() {
p.Password = "******"
}
func (a *AnotherSensitiveField) CensorPass() {
a.AnotherSensitiveField = "******"
}
Any type that embeds one type like Password will automatically have a CensorPass method. The types are embedded, and no other embedded types have a name collision, so the top level type can resolve HelloWorld.CensorPass to HelloWorld.Password.CensorPass(). This doesn't work for AnotherType, as it has 2 embedded types that provide this method. To get around this, we simply implement the method at the top level, and pass the call on to the embedded types. The third example is one where we want to censor a specific field that isn't used anywhere else (yet). Creating a separate type to embed isn't required, so we can just implement the method directly on the YetAnother type. Should we have data where user credentials are present, and we wish to censor multiple fields in different places, we can easily create a type for this:
type Credentials struct {
User string `json:"user"`
Pass string `json:"pass"`
}
type FullName struct {
Name string `json:"name"`
First string `json:"first_name"`
}
func (c *Credentials) CensorPass() {
c.User = "<redacted>"
c.Pass = "******"
}
func (f *FullName) CensorPass() {
if len(f.First) > 0 {
f.First = string([]rune(f.First)[0])
}
if len(f.Last) == 0 {
return
}
last := make([]string, 0, 2)
for _, v := range strings.Split(f.Last) {
if len(v) > 0 {
last = append(last, string([]rune(v)[0]))
}
}
last = append(last, "") // for the final .
f.Last = strings.Join(last, ". ") // initials only
}
And all we have to do is embed this type wherever we need it. Once that's done, our main function still looks like this:
func CensorPayload[T Censored](target T, payload []byte) ([]byte, error) {
if err := json.Unmarshal(target, payload); err != nil {
return nil, err
}
target.CensorPass() // type knows what fields to censor
return json.Marshal(target) // return the data censored & marshalled
}
But now, we can add new types to this Censored constraint very easily:
type Login struct {
Credentials // username pass
FullName
}
type Person struct {
FullName
Status string `json:"status"`
}
func (n *NewType) CensorPass() {
n.Credentials.CensorPass()
n.FullName.CensorPass()
}
With these couple of lines of code, we now have 2 more types that can be passed in to the CensorPayload function (by updating the constraint, of course). Given a Login and Person payload like:
// login
{
"user": "myLogin",
"pass": "weakpass",
"name": "Pothering Smythe",
"first_name": "Emanual",
}
// person
{
"name": "Doe",
"first_name": "John",
"status": "missing",
}
We should get the output:
{
"user": "<redacted>",
"pass": "******",
"name": "P. S. "
"first_name": "E",
}
{
"name": "D. ",
"first_name": "J",
"status": "missing",
}
Note:
I've written all of the code above without testing it out first. There could by typo's and mistakes here and there, but it should contain enough useful information and examples to get you started. If I have some spare time, I'll probably give this a whirl and update the snippets where needed
I'm currently using gRPC for my API communication. Now I need the value of my request can accept any data type, be it a struct object or just a primitive data type like single integer, string, or boolean. I tried using google.protobuf.Any as below but it doesn't work when I want to marshal and unmarshal it.
message MyRequest {
google.protobuf.Any item = 1; // could be 9, "a string", true, false, {"key": value}, etc.
}
proto-compiled result:
type MyRequest struct {
Item *anypb.Any `protobuf:"bytes,1,opt,name=item,proto3" json:"item,omitempty"`
}
I will marshal the incoming request through gRPC and save it as a JSON string in the database:
bytes, err = json.Marshal(m.request.Item)
if err != nil {
return err
}
itemStr := string(bytes)
But when I want to unmarshal the string back into *anypb.Any by:
func getItem(itemStr string) (structpb.Value, error) {
var item structpb.Value
err := json.Unmarshal([]byte(itemStr), &item)
if err != nil {
return item, err
}
// also I've no idea on how to convert `structpb.Value` to `anypb.Any`
return item, nil
}
It returns an error which seems to be because the struct *anypb.Any doesn't contain any of the fields in my encoded item string. Is there a correct way to solve this problem?
Consider using anypb
In the documentation it shows an example of how to unmarshal it
m := new(foopb.MyMessage)
if err := any.UnmarshalTo(m); err != nil {
... // handle error
}
... // make use of m
I'm using GoLang Validator on a struct to validate its fields. Even though I don't add required tag, it still behaves as if it is required.
type Order struct {
// ... other fields
UserID string `json:"userId" validate:"uuid4"`
// ... other fields
}
if err = validator.New().Struct(i); err != nil {
return err
}
Output: User ID: unknown error
It is not required hence the value is zero-value but it still returns an error. Am I doing something wrong here?
You should add the omitempty validator to allow empty values. Try out the code below on Go playground.
type Order struct {
// ... other fields
UserID string `json:"omitempty,userId" validate:"omitempty,uuid4"`
// ... other fields
}
if err := validator.New().Struct(Order{}); err != nil {
return err
}
Note that to marshal the struct to JSON you also need to set the omitempty validator if you want to allow empty values...
func (t *ballot) initBallot(stub shim.ChaincodeStubInterface, args []string) peer.Response {
if len(args) != 2 {
return shim.Error("Incorrect number of arguments. Expecting 2")
}
// ==== Input sanitation ====
fmt.Println("- start init ballot")
if len(args[0]) == 0 {
return shim.Error("1st argument must be a non-empty string")
}
if len(args[1]) == 0 {
return shim.Error("2nd argument must be a non-empty string")
}
personFirstName := args[0]
personLastName := args[1]
hash := sha256.New()
hash.Write([]byte(personFirstName + personLastName)) // ballotID is created based on the person's name
ballotID := hex.EncodeToString(hash.Sum(nil))
voteInit := "VOTE INIT"
// ==== Create ballot object and marshal to JSON ====
Ballot := ballot{personFirstName, personLastName, ballotID, voteInit}
ballotJSONByte, err := json.Marshal(Ballot)
if err != nil {
return shim.Error(err.Error())
}
err = stub.PutState(string(ballotID), ballotJSONByte)
//FIXME:0-------------------------------------------------
ballotAsByte, err := stub.GetState(string(ballotID))
if err != nil {
return shim.Error(err.Error())
}
BBBallot := ballot{}
//umarshal the data to a new ballot struct
json.Unmarshal(ballotAsByte, &BBBallot)
//
fmt.Println(BBBallot)
fmt.Println(BBBallot.personFirstName)
return shim.Success([]byte(ballotID))
}
Above is the code and this is the test script i am running it against
func Test_Invoke_initBallot(t *testing.T) {
scc := new(ballot)
stub := shim.NewMockStub("voting", scc)
res := stub.MockInvoke("1", [][]byte{[]byte("initBallot"), []byte("John"), []byte("C")})
if res.Status != shim.OK {
t.Log("bad status received, expected: 200; received:" + strconv.FormatInt(int64(res.Status), 10))
t.Log("response: " + string(res.Message))
t.FailNow()
}
if res.Payload == nil {
t.Log("initBallot failed to create a ballot")
t.FailNow()
}
}
I am trying to read from the ledger after putting the transaction in. However, I have been getting empty responses from both of the Println statements.
// PutState puts the specified `key` and `value` into the transaction's
// writeset as a data-write proposal. PutState doesn't effect the ledger
// until the transaction is validated and successfully committed.
// Simple keys must not be an empty string and must not start with null
// character (0x00), in order to avoid range query collisions with
// composite keys, which internally get prefixed with 0x00 as composite
// key namespace.
PutState(key string, value []byte) error
It does say on the documentation that putState does not commit transactions to the ledger until its validated, but I am just trying to test my chaincode using the MockStub without setting up the fabric network. What is the fix to this problem?
P.S the problem has been solved, here is the right way to set up a struct
type ballot struct {
PersonFirstName string
PersonLastName string
BallotID string
VoteInit string
}
You haven't provided the code for the ballot struct yet. But from what you provided, I have a guess what might be going on. I think you probably haven't exported the fields and your struct looks like this:
type ballot struct {
personFirstName string
personLastName string
ballotID string
voteInit string
}
But when you tried to convert this object to JSON using json.Marshal(Ballot), none of the fields are added to the JSON object because they were not exported. All that you have to do in this case is exporting the necessary fields (using Uppercase letter at the beginning of field names). Your updated struct should look something like the following:
type ballot struct {
PersonFirstName string
PersonLastName string
BallotID string
VoteInit string
}
This is a very common mistake many newcomers make. Wish you all the best in your journey forward!!!
P.S. Please edit your question and add the code of you ballot struct here even if this solution solves your problem as that might help others in the future. Also, please add proper indentation to the code and add the last } symbol in the code block.
I thought I have asserted (as far as I've learnt Go), but I keep getting this error
cannot use readBack["SomePIN"] (type interface {}) as type string in argument to c.String: need type assertion
Here is my code (this snippet is from a Request Handler function and I'm using Echo Web framework and Tiedot NoSQL database)
// To get query result document, simply
// read it [as stated in the Tiedot readme.md]
for id := range queryResult {
readBack, err := aCollection.Read(id)
if err != nil {
panic(err)
}
if readBack["OtherID"] == otherID {
if _, ok := readBack["SomePIN"].(string); ok {
return c.String(http.StatusOK, readBack["SomePIN"])
}
}
}
You are asserting readBack["SomePIN"] as a string - in the if statement. That doesn't make any change to readBack["SomePIN"], however - it's still an interface{}. In Go, nothing ever changes type. Here's what will work:
for id := range queryResult {
readBack, err := aCollection.Read(id)
if err != nil {
panic(err)
}
if readBack["OtherID"] == otherID {
if somePIN, ok := readBack["SomePIN"].(string); ok {
return c.String(http.StatusOK, somePIN)
}
}
}
You were tossing the string value from your type assertion, but you want it. So keep it, as somePIN, and then use it.
Final note - using the value, ok = interfaceVal.(type) syntax is a good practice. If interfaceVal turns out to be a non-string, you'll get value = "" and ok = false. If you eliminate the ok value from the type assertion and interfaceVal is a non-string, the program will panic.
It looks like your converting to a concrete type and throwing away the conversion, I think this should work:
if somePinString, ok := readBack["SomePIN"].(string); ok {
return c.String(http.StatusOK, somePinString)
}