I am trying to use github.com/dullgiulio/pingo and send my custom struct
type LuaPlugin struct {
Name string
List []PluginTable
}
type PluginTable struct {
Name string
F lua.LGFunction
}
// LoadPlugins walks over the plugin directory loading all exported plugins
func LoadPlugins() {
//
p := pingo.NewPlugin("tcp", "plugins/test")
// Actually start the plugin
p.Start()
// Remember to stop the plugin when done using it
defer p.Stop()
gob.Register(&LuaPlugin{})
gob.Register(&PluginTable{})
var resp *LuaPlugin
// Call a function from the object we created previously
if err := p.Call("MyPlugin.SayHello", "Go developer", &resp); err != nil {
log.Print(err)
} else {
log.Print(resp.List[0])
}
}
However I am always getting nil for the F field of ym struct. This is what I am sending on the client
// Create an object to be exported
type MyPlugin struct{}
// Exported method, with a RPC signature
func (p *MyPlugin) SayHello(name string, msg *util.LuaPlugin) error {
//
//
*msg = util.LuaPlugin{
Name: "test",
List: []util.PluginTable{
{
Name: "hey",
F: func(L *lua.LState) int {
log.Println(L.ToString(2))
return 0
},
},
},
}
return nil
}
Is it not possible to send custom data types over RPC?
I'm not familiar with the library however, you could try converting the struct to a byte slice before transport. Late reply, might help others....
Simple conversion: returns a struct as bytes
func StructToBytes(s interface{}) (converted []byte, err error) {
var buff bytes.Buffer
encoder := gob.NewEncoder(&buff)
if err = encoder.Encode(s); err != nil {
return
}
converted = buff.Bytes()
return
}
Decoder: returns a wrapper to decode the bytes into
func Decoder(rawBytes []byte) (decoder *gob.Decoder) {
reader := bytes.NewReader(rawBytes)
decoder = gob.NewDecoder(reader)
return
}
Example:
type MyStruct struct {
Name string
}
toEncode := MyStruct{"John Doe"}
// convert the struct to bytes
structBytes, err := StructToBytes(toEncode)
if err != nil {
panic(err)
}
//-----------
// send over RPC and decode on other side
//-----------
// create a new struct to decode into
var DecodeInto MyStruct
// pass the bytes to the decoder
decoder := Decoder(structBytes)
// Decode into the struct
decoder.Decode(&DecodeInto)
fmt.Println(DecodeInto.Name) // John Doe
Due to the use of the gob package you can swap types and typecast to a certain extent.
For more see: https://golang.org/pkg/encoding/gob/
Related
I'm making an http request in golang to an external api. It gives a general response of {"error":[]string, "result":changing interface{}}. depending on the function that is making the request, the Result field changes. Since I know the structure of the Result field for each function I run, I want to be able to change the value of Result before unmarshalling to json. I've tried to do this with the following code:
func GetAssets(output *Resp, resultType interface{}) error {
return publicRequest("/Assets", output, resultType)
}
func publicRequest(endPoint string, output *Resp, resultType interface{}) error {
url := Rest_url + Pub_rest_url + endPoint //"https://api.kraken.com/0/public/Assets in this case
resp, err := http.Get(url)
if err != nil {
return err
}
defer resp.Body.Close()
output.Result = resultType
return json.NewDecoder(resp.Body).Decode(&output)
}
Here is how it's being ran in main
type Resp struct {
Error []string `json:"error"`
Result interface{} `json:"result"`
}
type AssetInfo struct {
Aclass string `json:"aclass"`
Altname string `json:"altname"`
Decimals int `json:"decimals"`
Display int `json:"display_decimals"`
}
func main() {
var result map[string]AssetInfo
jsonData := Resp{}
rest_api_client.GetAssets(&jsonData, result)
fmt.Println(jsonData)
}
The issue is that it doesn't unmarshal correctly. A map is created for each asset, but the data contained inside of each asset is also being stored inside of a map. I'm not sure if I explained this well, but here is the current response after unmarshalling to understand what I mean.
Here is the data type of Resp.Result: map[string]interface {}
{[] map[1INCH:map[aclass:currency altname:1INCH decimals:10 display_decimals:5] AAVE:map[aclass:currency altname:AAVE decimals:10 display_decimals:5] ACA:map[aclass:currency altname:ACA decimals:10 display_decimals:5] ADA:map[aclass:currency altname:ADA decimals:8 display_decimals:6]...}
The response type I'm looking for is map[string]AssetInfo. Hopefully it could be unmarshalled like this:
{[] map[1INCH:{currency 1INCH 10 5} AAVE:{currency AAVE 10 5} ACA:{currency ACA 10 5} ADA:{currency ADA 8 6} ADA.S:{currency ADA.S 8 6}...}
Any help? I'd rather keep the Resp struct as generic as possible and just change the value of the Result field (if this is even possible to do correctly) since I plan to have multiple functions that call different endpoints of the api, and they'll all have the same underlying response type of the Resp struct with different Result types
You can view a working example in the following repo:
https://github.com/alessiosavi/GoArbitrage/blob/e107af466852b1ed30c2413eb4401595f7412b4f/markets/kraken/kraken.go
Basically, I've defined the following structure:
type Tickers struct {
Error []interface{} `json:"error"`
Result map[string]Ticker `json:"result"`
}
type Ticker struct {
Aclass string `json:"aclass"`
Altname string `json:"altname"`
Decimals int `json:"decimals"`
DisplayDecimals int `json:"display_decimals"`
}
Than I execute the request in the following way:
const KRAKEN_TICKERS_URL string = `https://api.kraken.com/0/public/Assets`
type Kraken struct {
PairsNames []string `json:"pairs_name"`
Pairs map[string]datastructure.KrakenPair `json:"pairs"`
OrderBook map[string]datastructure.KrakenOrderBook `json:"orderbook"`
MakerFee float64 `json:"maker_fee"`
TakerFees float64 `json:"taker_fee"`
// FeePercent is delegated to save if the fee is in percent or in coin
FeePercent bool `json:"fee_percent"`
Tickers []string
}
// Init is delegated to initialize the maps for the kraken
func (k *Kraken) Init() {
k.Pairs = make(map[string]datastructure.KrakenPair)
k.OrderBook = make(map[string]datastructure.KrakenOrderBook)
k.SetFees()
}
// SetFees is delegated to initialize the fee type/amount for the given market
func (k *Kraken) SetFees() {
k.MakerFee = 0.16
k.TakerFees = 0.26
k.FeePercent = true
}
func (k *Kraken) GetTickers() error {
res := datastructure.Tickers{}
var err error
var request req.Request
var data []byte
var tickers []string
resp := request.SendRequest(KRAKEN_TICKERS_URL, "GET", nil, nil, false, 10*time.Second)
if resp.Error != nil {
zap.S().Debugw("Error during http request. Err: " + resp.Error.Error())
return resp.Error
}
if resp.StatusCode != 200 {
zap.S().Warnw("Received a non 200 status code: " + strconv.Itoa(resp.StatusCode))
return errors.New("NON_200_STATUS_CODE")
}
data = resp.Body
if err = json.Unmarshal(data, &res); err != nil {
zap.S().Warn("ERROR! :" + err.Error())
return err
}
zap.S().Infof("Data: %v", res.Result)
tickers = make([]string, len(res.Result))
i := 0
for key := range res.Result {
tickers[i] = res.Result[key].Altname
i++
}
k.Tickers = tickers
return nil
}
I am a little bit confused here and although I have searched a lot on this, something is clearly missing from my knowledge and I am asking your help.
I have created a Hyperledger Fabric Network and installed a chaincode in it. And I want to make a function that retrieves all the World State inputs about the Keys. I have done it already with the bytes.Buffer and it worked. But what I want to do is to do it with a struct.
So, I created the following struct that has only the key:
type WSKeys struct {
Key string `json: "key"`
Namespace string `json: "Namespace"`
}
And this is my code function:
func (s *SmartContract) getAllWsDataStruct(APIstub shim.ChaincodeStubInterface , args []string) sc.Response {
var keyArrayStr []WSKeys
resultsIterator, err := APIstub.GetQueryResult("{\"selector\":{\"_id\":{\"$ne\": null }} }")
if err != nil {
return shim.Error("Error occured when trying to fetch data: "+err.Error())
}
for resultsIterator.HasNext() {
// Get the next record
queryResponse, err := resultsIterator.Next()
if err != nil {
return shim.Error(err.Error())
}
fmt.Println(queryResponse)
var qry_key_json WSKeys
json.Unmarshal([]byte(queryResponse), &qry_key_json)
keyArray = append(keyArray, qry_key_json)
}
defer resultsIterator.Close()
all_bytes, _ := json.Marshal(keyArray)
fmt.Println(keyArray)
return shim.Success(all_bytes)
}
When executing the above I get the following error:
cannot convert queryResponse (type *queryresult.KV) to type []byte
I can get the results correctly if I, for example do this:
func (s *SmartContract) getAllWsDataStruct(APIstub shim.ChaincodeStubInterface , args []string) sc.Response {
var keyArray []string
resultsIterator, err := APIstub.GetQueryResult("{\"selector\":{\"_id\":{\"$ne\": null }} }")
if err != nil {
return shim.Error("Error occured when trying to fetch data: "+err.Error())
}
for resultsIterator.HasNext() {
// Get the next record
queryResponse, err := resultsIterator.Next()
if err != nil {
return shim.Error(err.Error())
}
fmt.Println(queryResponse)
keyArray = append(keyArray, queryResponse.Key)
}
defer resultsIterator.Close()
all_bytes, _ := json.Marshal(keyArray)
fmt.Println(keyArray)
return shim.Success(all_bytes)
}
But, why I get the above error when trying to add the queryResponse into a custom struct?
Do I need to add it to a struct that is only its type?
Please someone can explain what I am missing here?
The error statement is verbose enough to indicate, that your []byte conversion failed for the type queryResponse which, with a bit of lookup seems to be a struct type. In Go you cannot natively convert a struct instance to its constituent bytes without encoding using gob or other means.
Perhaps your intention was to use the Key record in the struct for un-marshalling
json.Unmarshal([]byte(queryResponse.Key), &qry_key_json)
Below is the prototype code for 3 layer API(with some design limitations):
// Sample program demonstrating interface composition.
package main
import (
"errors"
"fmt"
"io"
"math/rand"
"time"
)
func init() {
rand.Seed(time.Now().UnixNano())
}
// =============================================================================
// Data is the structure of the data we are copying.
type Data struct {
Line string
}
// =============================================================================
// Puller declares behavior for pulling data.
type Puller interface {
Pull(d *Data) error
}
// Storer declares behavior for storing data.
type Storer interface {
Store(d *Data) error
}
// PullStorer declares behavior for both pulling and storing.
type PullStorer interface {
Puller
Storer
}
// =============================================================================
// Xenia is a system we need to pull data from.
type Xenia struct {
Host string
Timeout time.Duration
}
// Pull knows how to pull data out of Xenia.
func (*Xenia) Pull(d *Data) error {
switch rand.Intn(10) {
case 1, 9:
return io.EOF
case 5:
return errors.New("Error reading data from Xenia")
default:
d.Line = "Data"
fmt.Println("In:", d.Line)
return nil
}
}
// Pillar is a system we need to store data into.
type Pillar struct {
Host string
Timeout time.Duration
}
// Store knows how to store data into Pillar.
func (*Pillar) Store(d *Data) error {
fmt.Println("Out:", d.Line)
return nil
}
// =============================================================================
// System wraps Xenia and Pillar together into a single system.
type System struct {
Xenia
Pillar
}
// =============================================================================
// pull knows how to pull bulks of data from any Puller.
func pull(p Puller, data []Data) (int, error) {
for i := range data {
if err := p.Pull(&data[i]); err != nil {
return i, err
}
}
return len(data), nil
}
// store knows how to store bulks of data from any Storer.
func store(s Storer, data []Data) (int, error) {
for i := range data {
if err := s.Store(&data[i]); err != nil {
return i, err
}
}
return len(data), nil
}
// Copy knows how to pull and store data from any System.
func Copy(ps PullStorer, batch int) error {
data := make([]Data, batch)
for {
i, err := pull(ps, data)
if i > 0 {
if _, err := store(ps, data[:i]); err != nil {
return err
}
}
if err != nil {
return err
}
}
}
// =============================================================================
func main() {
sys := System{
Xenia: Xenia{
Host: "localhost:8000",
Timeout: time.Second,
},
Pillar: Pillar{
Host: "localhost:9000",
Timeout: time.Second,
},
}
if err := Copy(&sys, 3); err != io.EOF {
fmt.Println(err)
}
}
My understanding is,
If you pass u, the interface contains type of u, and a pointer to a copy of u. If you pass &u, the interface contains the type of &u and the address of u.
So, below is my understanding for variables p, s & ps, from above code:
But, I would like to confirm,
1) On pull(ps, data), does first element of p contain type *System or Xenia type, as first element?
2) On store(ps, data[:i]), does first element of s contain type *System or Pillar type?
The type is *System.
You can confirm this by adding simple print statement, e.g. fmt.Printf("%T\n", p)
The program will receive many msg, msg has different struct "Data", so I define the Msg struct:
type Msg struct {
MsgType int
Data interface{}
}
type Data1 struct {
//msg type 1 Data struct
}
type Data2 struct {
//msg type 2 Data struct
}
func (msgStr string) {
msg := Msg{}
if err := json.Unmarshal([]byte(msgStr), &msg); err != nil {
//log err
}
switch msg.MsgType{
case 1:
//convert msg.Data to a type 1 struct
case 2:
//convert msg.Data to a type 2 struct
}
}
But print out the msg.Data, it is a map, not interface{}, so when I convert it to Data1 by msg.Data.(Data1), got an err.
So,
1. Why interface{} auto convert to map?
2. How to convert it to Data1 struct I want?
3. What is the best practices in this scenes.
1. Because it sees a JSON object, and, as documented, a JSON object becomes a map[string]interface{} when stored into an interface{} (this is the only type that can hold whatever is in a JSON object in generality).
2. Given your current situation, you could assign each field of the map to the appropriate field of a new Data1 or Data2.
3. The ideal way to handle this is to use json.RawMessage to defer the decoding of Data until you know what it is. This can be handled like so:
type Msg struct {
MsgType int
Data interface{}
}
func (m *Msg) UnmarshalJSON(b []byte) (err error) {
var tmp struct {
MsgType int
Data json.RawMessage
}
err = json.Unmarshal(b, &tmp)
if err != nil {
return
}
m.MsgType = tmp.MsgType
switch (tmp.MsgType) {
case 1:
data := Data1{}
err = json.Unmarshal(tmp.Data, &data)
if err != nil {
return
}
m.Data = data
case 2:
data := Data2{}
err = json.Unmarshal(tmp.Data, &data)
if err != nil {
return
}
m.Data = data
default:
return errors.New("invalid DataType")
}
return
}
And then you can call json.Unmarshal or json.Decode directly on a *Msg and its Data will be decoded as you want.
I am trying to implement my own decoding layer ontop of TCP, so far it only works when I create a packet without any Eth/IP/TCP header and set its layer to my custom layer manually. The data of the custom protocol is inside an ordinary TCP payload.
How do I decode only the payload of the TCP layer as another layer?
package main
import (
"fmt"
"github.com/google/gopacket"
"github.com/google/gopacket/pcap"
)
var (
pcapFile string = "capt.pcap"
handle *pcap.Handle
err error
)
type CustomLayer struct {
SomeByte byte
AnotherByte byte
restOfData []byte
}
var CustomLayerType = gopacket.RegisterLayerType(
2001,
gopacket.LayerTypeMetadata{
"CustomLayerType",
gopacket.DecodeFunc(decodeCustomLayer),
},
)
func (l CustomLayer) LayerType() gopacket.LayerType {
return CustomLayerType
}
func (l CustomLayer) LayerContents() []byte {
return []byte{l.SomeByte, l.AnotherByte}
}
func (l CustomLayer) LayerPayload() []byte {
return l.restOfData
}
func decodeCustomLayer(data []byte, p gopacket.PacketBuilder) error {
p.AddLayer(&CustomLayer{data[0], data[1], data[2:]})
// nil means this is the last layer. No more decoding
return nil
}
func main() {
handle, err = pcap.OpenOffline(pcapFile)
if err != nil {
log.Fatal(err)
}
defer handle.Close()
packetSource := gopacket.NewPacketSource(handle, handle.LinkType())
for packet := range packetSource.Packets() {
tcpLayer := packet.Layer(layers.LayerTypeTCP)
if tcpLayer != nil {
fmt.Println("TCP layer detected.")
tcp, _ := tcpLayer.(*layers.TCP)
fmt.Println("Sequence number: ", tcp.Seq)
customLayer := packet.Layer(CustomLayerType)
if customLayer != nil { // always nil
customLayerContent, _ := customLayer.(*CustomLayer)
// Now we can access the elements of the custom struct
fmt.Println("Payload: ", customLayerContent.LayerPayload())
fmt.Println("SomeByte element:", customLayerContent.SomeByte)
fmt.Println("AnotherByte element:", customLayerContent.AnotherByte)
}
}
fmt.Println()
}
}
Most of the code is from this great post by devdungeon.
As no one responded I am going to answer it myself now.
Basically we have 3 options to handle this:
Create an extended TCP layer that handles our additional bytes and override default one by setting layers.LinkTypeMetadata[layers.LinkTypeTCP] to our extended version. Have a look at this example.
Create a new packet from the TCP payload using gopacket.NewPacket setting firstLayerDecoder to CustomLayerType and decode it normally.
As you mostly don't need an actual layer but instead a filled CustomLayer struct simply write a DecodeBytesToCustomStruct function where you pass TCP payload. This way we can even return multiple structs from one packets payload which wouldn't be possible otherwise.
Omit all CustomLayer code from above.
type CustomStruct struct {
SomeByte byte
AnotherByte byte
restOfData []byte
}
func (customStruct *CustomStruct) DecodeStructFromBytes(data []byte) error {
customStruct.SomeByte = data[0]
customStruct.AnotherByte = data[1]
customStruct.restOfData = data[2:]
return nil
}
In your main.go
for packet := range packetSource.Packets() {
tcpLayer := packet.Layer(layers.LayerTypeTCP)
if tcpLayer != nil {
tcp, _ := tcpLayer.(*layers.TCP)
if tcp.Payload != nil && len(tcpLayer.Payload) > 0 {
customStruct := CustomStruct{}
customStruct.DecodeStructFromBytes(tcp.Payload)
fmt.Println("SomeByte element:", customStruct.SomeByte)
}
}
}
tcp.Payload is the same as packet.ApplicationLayer().Payload()