I want to write a file cache in Go. I am using gob encoding, and saving to a file, but my get function has some problem:
package main
import (
"encoding/gob"
"fmt"
"os"
)
var (
file = "tmp.txt"
)
type Data struct {
Expire int64
D interface{}
}
type User struct {
Id int
Name string
}
func main() {
user := User{
Id: 1,
Name: "lei",
}
err := set(file, user, 10)
if err != nil {
fmt.Println(err)
return
}
user = User{}
err = get(file, &user)
if err != nil {
fmt.Println(err)
return
}
//user not change.
fmt.Println(user)
}
func set(file string, v interface{}, expire int64) error {
f, err := os.OpenFile(file, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0600)
if err != nil {
return err
}
defer f.Close()
//wrapper data
//save v in data.D
data := Data{
Expire: expire,
D: v,
}
gob.Register(v)
enc := gob.NewEncoder(f)
err = enc.Encode(data)
if err != nil {
return err
}
return nil
}
func get(file string, v interface{}) error {
f, err := os.OpenFile(file, os.O_RDONLY, 0600)
if err != nil {
return err
}
defer f.Close()
var data Data
dec := gob.NewDecoder(f)
err = dec.Decode(&data)
if err != nil {
return err
}
//get v
v = data.D
fmt.Println(v)
return nil
}
The get function passes interface type and I want to change the value, but not change.
http://play.golang.org/p/wV7rBH028o
In order to insert an unknown value into v of type interface{}, you need to use reflection. This is somewhat involved, but if you want to support this in full, you can see how its done by walking through the decoding process in some of the encoding packages (json, gob).
To get you started, here's a basic version of your get function using reflection. This skips a number of checks, and will only decode something that was encoded as a pointer.
func get(file string, v interface{}) error {
f, err := os.OpenFile(file, os.O_RDONLY, 0600)
if err != nil {
return err
}
defer f.Close()
rv := reflect.ValueOf(v)
if rv.Kind() != reflect.Ptr || rv.IsNil() {
panic("need a non nil pointer")
}
var data Data
dec := gob.NewDecoder(f)
err = dec.Decode(&data)
if err != nil {
return err
}
dv := reflect.ValueOf(data.D)
if dv.Kind() != reflect.Ptr {
panic("didn't decode a pointer")
}
rv.Elem().Set(dv.Elem())
return nil
}
I would actually suggest an easier way to handle this in your own code, which is to have the Get function return an interface{}. Since you will know what the possible types are at that point, you can use a type switch to assert the correct value.
An alternative approach is to return directly the value from the file:
func get(file string) (interface{}, error) {
f, err := os.OpenFile(file, os.O_RDONLY, 0600)
if err != nil {
return nil, err
}
defer f.Close()
var data Data
dec := gob.NewDecoder(f)
err = dec.Decode(&data)
if err != nil {
return nil,err
}
fmt.Println(data.D)
return data.D,nil
}
full working example: http://play.golang.org/p/178U_LVC5y
Related
I want to achieve exactly opposite of the solution given here, zipping a slice of byte into another slice of byte -
Convert zipped []byte to unzip []byte golang code
Something like -
func ZipBytes(unippedBytes []byte) ([]byte, error) {
// ...
}
[I am going to upload that zipped file as multipart form data for a POST request]
You can compress directly into memory using a bytes.Buffer.
The following example uses compress/zlib since it is the opposite of the example given in the question. Depending on your use case you could easily change it to compress/gzip as well (very similar APIs).
package data_test
import (
"bytes"
"compress/zlib"
"io"
"testing"
)
func compress(buf []byte) ([]byte, error) {
var out bytes.Buffer
w := zlib.NewWriter(&out)
if _, err := w.Write(buf); err != nil {
return nil, err
}
if err := w.Close(); err != nil {
return nil, err
}
return out.Bytes(), nil
}
func decompress(buf []byte) (_ []byte, e error) {
r, err := zlib.NewReader(bytes.NewReader(buf))
if err != nil {
return nil, err
}
defer func() {
if err := r.Close(); e == nil {
e = err
}
}()
return io.ReadAll(r)
}
func TestRoundtrip(t *testing.T) {
want := []byte("test data")
zdata, err := compress(want)
if err != nil {
t.Fatalf("compress: %v", err)
}
got, err := decompress(zdata)
if err != nil {
t.Fatalf("decompress: %v", err)
}
if !bytes.Equal(want, got) {
t.Errorf("roundtrip: got = %q; want = %q", got, want)
}
}
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"strings"
)
func main() {
fmt.Println(fecthData())
}
func fecthData() (map[string]interface{}, error) {
body := strings.NewReader("dil_kodu=tr")
req, err := http.NewRequest("POST", "https://www.haremaltin.com/dashboard/ajax/doviz", body)
if err != nil {
// handle err
return nil, err
}
req.Header.Set("X-Requested-With", "XMLHttpRequest")
resp, err := http.DefaultClient.Do(req)
if err != nil {
// handle err
return nil, err
}
defer resp.Body.Close()
jsonData, err := ioutil.ReadAll(resp.Body)
if err != nil {
panic(err)
return nil, err
}
var data map[string]interface{}
err = json.Unmarshal(jsonData, &data)
if err != nil {
return nil, err
}
return data, nil
}
You can see full code above and I have a go response as below and it is nested map as you see and want to reach "data-ATA5_ESKI-satis" value which is 34319. Is there anybody to help me.
Thank you for your time
A part of response below:
map[data:map[AEDTRY:map[alis:4.6271 code:AEDTRY dir:map[alis_dir: satis_dir:] dusuk:4.7116 kapanis:4.6224 satis:4.7271 tarih:17-06-2022 19:41:45 yuksek:4.7276] AEDUSD:map[alis:0.2680 code:AEDUSD dir:map[alis_dir: satis_dir:] dusuk:0.27 kapanis:0.268 satis:0.2700 tarih:17-06-2022 19:30:02 yuksek:0.27]... ALTIN:map[alis:1024.790 code:ALTIN dir:map[alis_dir:down satis_dir:down] dusuk:1029.05 kapanis:1032.13 satis:1030.650 tarih:17-06-2022 19:41:58 yuksek:1040] ATA5_ESKI:map[alis:33869 code:ATA5_ESKI dir:map[alis_dir:down satis_dir:down] dusuk:34266 kapanis:34112 satis:34319 tarih:17-06-2022 19:41:58 yuksek:34630] XPTUSD:map[alis:933 code:XPTUSD dir:map[alis_dir: satis_dir:] dusuk:936 kapanis:953 satis:936 tarih:17-06-2022 19:41:58 yuksek:957]] meta:map[fiyat_guncelleme:2000 fiyat_yayini:web_socket time:1.655484118278e+12 time_formatted:]]
for _, v := range data { // we need value part of the map
m, ok := v.(map[string]interface{}) // we need the convert the map
// into interface for iteration
if !ok {
fmt.Printf("Error %T", v)
}
for k, l := range m {
if k == "ATA_ESKI"{ // the value we want is inside of this map
a, ok := l.(map[string]interface{}) // interface convert again
if !ok {
fmt.Printf("Error %T", v)
}
for b,c := range a{
if b == "satis"{ // the value we want
fmt.Println("Price is", c)
}
}
}
}
}
We can get the value adding this iteration before "return data, nil" at the end but I think there must be easier methods for this.
I am trying to send a raw transaction with eth.sendTransaction but I am getting an error that says {"code":-32000,"message":"unknown account"}. I am not sure what is causing this and I cant seem to find an answer on the internet. Can anyone help me figure it out? Here is my code:
func ExecuteSignedTransaction(rawTransaction string) {
var hash web3.Hash
data := make(map[string]interface{})
data["data"] = rawTransaction
err := Web3HTTPClient.Call("eth_sendTransaction", &hash, data)
if err != nil{
fmt.Println(err)
Os.Exit(1)
}
fmt.Println("Sent tx hash:", hash)
}
So, what I might do here:
import (
"strings"
"crypto/ecdsa"
"math/big"
"github.com/ethereum/go-ethereum/ethclient"
"github.com/ethereum/go-ethereum/crypto"
"github.com/ethereum/go-ethereum/accounts/abi/bind"
)
var chainId = big.NewInt(1) // chain id for the ethereum mainnet, change according to needs
func ecdsaPrivateKeyFromHex(privKeyHex string) *ecdsa.PrivateKey {
ecdsaKey, err := crypto.HexToECDSA(privKeyHex)
if err != nil { panic(err) }
return ecdsaKey
}
func newTransactOpts(privKey *ecdsa.PrivateKey) *bind.TransactOpts {
transactOpts, err := bind.NewKeyedTransactorWithChainID(privKey, chainId)
if err != nil { panic(err) }
return transactOpts
}
func newRpcClient() *ethclient.Client {
c, err := ethclient.Dial("insert rpc url here")
if err != nil { panic(err) }
return c
}
// note: constructing the *types.Transaction object left as
// an exercise to the reader
func ExecuteTransaction(rawTransaction *types.Transaction) {
privKeyHex := "0xblahblahblahblahblah" // use your own account's private key
transactOpts := newTransactOpts(ecdsaPrivateKeyFromHex(privKeyHex))
signedTxn, err := transactOpts.Signer(transactOpts.From, rawTransaction)
if err != nil { panic(err) }
rpcClient := newRpcClient()
if err := rpcClient.SendTransaction(context.Background(), signedTxn); err != nil { panic(err) }
// do whatever
}
I'm learning Go and I have a question:
I have an Info type that is defined like that:
type Info struct {
ID ID `json:"id,omitempty"`
DisplayName string `json:"display_name,omitempty"`
}
I made a custom UnmarshallJSON function to unmarshall this struct because as an input I have either:
An []interface{} with at position [0] an int and [1] a string
A boolean always equals to false meaning that the field is null
I want that when the input is false, the Info is nil.
Here's the UnmarshallJSON function
func (i *Info) UnmarshalJSON(data []byte) error {
var v []interface{}
if err := json.Unmarshal(data, &v); err != nil {
var v bool
if err = json.Unmarshal(data, &v); err != nil {
return err
}
return nil
}
i.ID = ID(v[0].(float64))
i.DisplayName = v[1].(string)
return nil
}
It's ugly, and I would like to know if there's a better option.
Thank you very much.
Fist you should be more defensive about unexpected types and length to avoid a panic. Then you can unmarshal into a []json.RawMessage to defer unmarshaling of the elements until you are ready. Finally you should guard against your invalid true.
Here is my best effort, please others feel free to edit (here is a playground):
func (i *Info) UnmarshalJSON(data []byte) error {
var raw interface{}
if err := json.Unmarshal(data, &raw); err != nil {
return err
}
switch r := raw.(type) {
case []interface{}:
case bool:
if r {
return errors.New("unexpected true, must be array or false")
}
return nil
default:
return fmt.Errorf("unexpected type %T, must be array or false", r)
}
var v []json.RawMessage
if err := json.Unmarshal(data, &v); err != nil {
return err
}
if len(v) != 2 {
return fmt.Errorf("unexpected length %d, must be 2", len(v))
}
if err := json.Unmarshal(v[0], &i.ID); err != nil {
return err
}
if err := json.Unmarshal(v[1], &i.DisplayName); err != nil {
return err
}
return nil
}
Imagine these functions needs to be used how can I make this calls generic so that I don't repeat almost the same code.
with "encoding/csv"
func getDataFromCSVFiles(files []string) (error, Data) {
data := Data{}
for _, file := range files {
f, err := os.Open(file)
if err != nil {
panic(err)
return err, data
}
defer f.Close()
r := charmap.ISO8859_1.NewDecoder().Reader(f)
reader := csv.NewReader(r)
for i := 1;;i++ {
rec, err := reader.Read()
if i == 1 {
//Skipping header
continue
}
if err != nil {
if err == io.EOF {
break
}
//TODO log error line and csv filename
log.Fatal(err)
}
addWorkbook(rec, &data)
}
}
return nil, data
}
and with
import fw "github.com/hduplooy/gofixedwidth" which is almost the same except calling fw.NewReader
func getDataFromPRNFiles(files []string) (error, Data) {
data := Data{}
for _, file := range files {
f, err := os.Open(file)
if err != nil {
panic(err)
return err, data
}
defer f.Close()
r := charmap.ISO8859_1.NewDecoder().Reader(f)
reader := fw.NewReader(r)
for i := 1;;i++ {
rec, err := reader.Read()
if i == 1 {
//Skipping header
continue
}
if err != nil {
if err == io.EOF {
break
}
//TODO log error line and csv filename
log.Fatal(err)
}
addWorkbook(rec, &data)
}
}
return nil, data
}
The only apparent difference is:
reader := csv.NewReader(r)
versus:
reader := fw.NewReader(r)
I'm not sure what fw is but presumably both readers implement a common interface:
type StringSliceReader interface {
Read() ([]string, error)
}
So you could pass the openers (csv.NewReader and fw.NewReader) as function arguments:
func getDataFromFiles(files []string, func(r io.Reader) StringArrayReader) (error, Data) {
//...
}
but you'd need to wrap them in little functions to get around the return types:
func newCSVReader(r io.Reader) StringSliceReader {
return csv.NewReader(r)
}
func newFWReader(r io.Reader) StringSliceReader {
return fw.NewReader(r)
}
Also, defer queues up things to execute when the function exits, not on the next iteration of a loop. So if you do this:
for _, file := range files {
f, err := os.Open(file)
if err != nil {
panic(err)
return err, data
}
defer f.Close()
//...
}
and files has a hundred entries then you'll have a hundred open files before any of them are closed. You probably want to move that loop body to a separate function so that you only have one file open at a time.
Furthermore, error is usually the last return value from a function so you should return data, err to be more idiomatic.
The result could look something like this:
type StringSliceReader interface {
Read() ([]string, error)
}
type NewReader func(r io.Reader) StringSliceReader
func newCSVReader(r io.Reader) StringSliceReader {
return csv.NewReader(r)
}
func newFWReader(r io.Reader) StringSliceReader {
return fw.NewReader(r)
}
func getDataFrom(file string, data *Data, newReader NewReader) error {
f, err := os.Open(file)
if err != nil {
return err
}
defer f.Close()
r := charmap.ISO8859_1.NewDecoder().Reader(f)
reader := newReader(r)
for i := 1; ; i++ {
rec, err := reader.Read()
if i == 1 {
continue
}
if err != nil {
if err == io.EOF {
break
}
log.Fatal(err)
}
addWorkbook(rec, data)
}
return nil
}
func getDataFromFiles(files []string, newReader NewReader) (Data, error) {
data := Data{}
for _, file := range files {
err := getDataFrom(file, newReader, &data)
if err != nil {
panic(err)
return data, err
}
}
return data, nil
}
and you could say getDataFromFiles(files, newCSVReader) to read CSVs or getDataFromFiles(files, newFWReader) to read FW files. If you want to read from something else, you'd just need a NewReader function and something that implements the StringSliceReader interface.
You might want to bury/hide the charmap.ISO8859_1.NewDecoder().Reader(f) stuff inside the NewReader functions to make it easier to read non-Latin-1 encoded files. You could also replace newReader NewReader with a map[string]NewReader in getDataFromFiles and choose the NewReader to use based on the file's extension or other format identifier.