How to get gRPC response limit - go

I have Go gRPC endpoint that returns an array of some items. I want to limit the number of items in response to make fit the maximum that gRPC allows to send.
My idea is to get the maximum in handler func, divide it by item size and voilà. But how can I get response max size?
I don't want to set max response size to make my class independent of grpc instantiation.
main.go
package main
import (
"log"
"main/api"
"main/router"
"net"
"google.golang.org/grpc"
)
func main() {
l, err := net.Listen("tcp", "0.0.0.0:138080")
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
grpcServer := grpc.NewServer()
r := router.New()
api.RegisterTestGenerateServiceServer(grpcServer, r)
err = grpcServer.Serve(l)
if err != nil {
log.Fatalf("failed to run server: %v", err)
}
}
router.go
package router
import (
"context"
"main/api"
"main/model"
"unsafe"
)
type router struct {
}
func New() *router {
return &router{}
}
func (r *router) TestCall(context.Context, *api.TestCallRequest) (*api.TestCallResponse, error) {
items := somewhere.GetItems()
apiItems := transform.ToAPIItems(items)
itemSize := unsafe.Sizeof(model.Item{})
responseSize := someWonderFuncGetGrpcResponseMaxSize()
NItems := responseSize / itemSize
return &api.TestCallResponse{
Items: apiItems[:NItems],
}, nil
}

Try updating grpc.MaxCallSendMsgSize(s int) on your client to enable your client send larger message.
This worked for me, as example:
func main() {
// create connection
opts := []grpc.DialOption{
// grpc.WithInsecure(),
grpc.WithDefaultCallOptions(grpc.MaxCallRecvMsgSize(math.MaxInt32)), // uncomment for large amount of data returned
}
cc, err := grpc.Dial("localhost:9089", opts...)
if err != nil {
fmt.Printf("failed to dial: %v\n", err)
return
}
defer cc.Close()
client := pb.NewGrpcServiceClient(cc)
data, err := client.GetDataFromGRPCService()

Related

How to subscribe to the event "incoming transaction successful"?

How can I subscribe to the event "incoming transaction successful". That is, I want to know that ether has come to my wallet. How to do it using subscription. I do not understand anything.
package main
import (
"context"
"github.com/ethereum/go-ethereum"
"github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/core/types"
"github.com/ethereum/go-ethereum/ethclient"
"log"
)
func main() {
client, err := ethclient.Dial("wss://mainnet.infura.io/ws/v3/XXXXXXXXXXXXXXXX")
if err != nil {
log.Fatal(err)
}
accs := map[string]string{
"0x92321477416e93Ea452f16015e2F2a13B3BDe8B7":"12e2cc06fb999fa29306f10db6b366e61a4946b9527286a0c56640c94cebd950",
}
keys := make([]common.Address, 0, len(accs))
for k := range accs {
keys = append(keys, common.HexToAddress(k))
}
var ch = make(chan types.Log)
sub, err := client.SubscribeFilterLogs(context.Background(), ethereum.FilterQuery{
BlockHash: nil,
FromBlock: nil,
ToBlock: nil,
Addresses: keys,
Topics: nil,
}, ch)
if err != nil {
log.Fatal(err)
}
defer sub.Unsubscribe()
for l := range ch {
// ???
}
}
Help me please. Where i can find example?
enter image description here
I'm looking through infura.io's API documentation and I'm not finding that they have an endpoint for payouts. The image that you linked above is their UI option for email notifications and has nothing to do with the API. In order to subscribe to an action, it would have to be initiated on their end. You would have to provide them with a callback to execute when that action occurs. Your callback would do the alerting, but they would call it when the trigger (a payout) occurred. Do they have a place to enter webhooks? If so, this would be your subscription.
While this option may not be a subscription, a possible workaround would be for you to poll the getBalance endpoint and compare the result to the previous result, and if there is an increase, alert you.
My solution
sub, err := client.SubscribeNewHead(context.Background(), ch)
get block
b, err := client.BlockByNumber(context.Background(), l.Number)
check all transactions
for _, tx := range b.Transactions() {
msg, err := tx.AsMessage(types.NewEIP155Signer(tx.ChainId()))
}
in msg.To() address )))
actually go-ethereum already provided a demo in their test script:
import (
"context"
"log"
"os"
"os/signal"
"syscall"
"github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/ethclient"
"github.com/ethereum/go-ethereum/ethclient/gethclient"
"github.com/ethereum/go-ethereum/rpc"
)
const (
url = "https://mainnet.infura.io/v3/xxxxxxxx"
wss = "wss://eth-mainnet.alchemyapi.io/v2/xxxxxxxxxxx"
// wss = "wss://mainnet.infura.io/ws/v3/xxxxxxxx"
)
func watch() {
backend, err := ethclient.Dial(url)
if err != nil {
log.Printf("failed to dial: %v", err)
return
}
rpcCli, err := rpc.Dial(wss)
if err != nil {
log.Printf("failed to dial: %v", err)
return
}
gcli := gethclient.New(rpcCli)
txch := make(chan common.Hash, 100)
_, err = gcli.SubscribePendingTransactions(context.Background(), txch)
if err != nil {
log.Printf("failed to SubscribePendingTransactions: %v", err)
return
}
for {
select {
case txhash := <-txch:
tx, _, err := backend.TransactionByHash(context.Background(), txhash)
if err != nil {
continue
}
data, _ := tx.MarshalJSON()
log.Printf("tx: %v", string(data))
}
}
}
func DoTest() {
go watch()
signalChan := make(chan os.Signal, 1)
signal.Notify(signalChan, syscall.SIGINT, syscall.SIGTERM)
<-signalChan
}

Go GRPC Bidirectional Stream Performance

We are developing a high frequency trading platform and in one of our components we have implemented the grpc with golang. And we needed to use bidirectional streaming in one of our usecases , we made a sample implementation as in below code , however when we test the performance of the code by checking the difference between timestamps of the logs in
Recv Time %v Index: %v Num: %v
Send Time %v, Index: %v, Num: %v
we found out that calling .Send method of the stream from client side and receiving the same data by calling .Recv on the server side tooks approximately 400-800 microseconds which is too low for us. We need maximum 10-50 microseconds performance , and when we read the guidelines we saw that grpc can go up to nanoseconds if both client and server is in the same computer (Which is exactly our case)
So I think we are missing some options or some performance tricks about it. Does anyone know what we can do to increase this performance problem
Client Code:
package main
import (
"context"
"log"
"math/rand"
pb "github.com/pahanini/go-grpc-bidirectional-streaming-example/src/proto"
"time"
"google.golang.org/grpc"
)
func main() {
rand.Seed(time.Now().Unix())
// dail server
conn, err := grpc.Dial(":50005", grpc.WithInsecure())
if err != nil {
log.Fatalf("can not connect with server %v", err)
}
// create stream
client := pb.NewMathClient(conn)
stream, err := client.Max(context.Background())
if err != nil {
log.Fatalf("openn stream error %v", err)
}
var max int32
ctx := stream.Context()
done := make(chan bool)
msgCount := 100
fromMsg := 0
// first goroutine sends random increasing numbers to stream
// and closes int after 10 iterations
go func() {
for i := 1; i <= msgCount; i++ {
// generate random nummber and send it to stream
rnd := int32(i)
req := pb.Request{Num: rnd}
if i-1 >= fromMsg {
sendTime := time.Now().UnixNano()
log.Printf("Send Time %v, Index: %v, Num: %v", sendTime,i-1,req.Num)
}
if err := stream.Send(&req); err != nil {
log.Fatalf("can not send %v", err)
}
//afterSendTime := time.Now().UnixNano()
//log.Printf("After Send Time %v", afterSendTime)
//log.Printf("---------------")
//log.Printf("%d sent", req.Num)
//time.Sleep(time.Millisecond * 200)
}
if err := stream.CloseSend(); err != nil {
log.Println(err)
}
}()
// third goroutine closes done channel
// if context is done
go func() {
<-ctx.Done()
if err := ctx.Err(); err != nil {
log.Println(err)
}
close(done)
}()
<-done
log.Printf("finished with max=%d", max)
}
Server Code:
package main
import (
"io"
"log"
"net"
"time"
pb "github.com/pahanini/go-grpc-bidirectional-streaming-example/src/proto"
"google.golang.org/grpc"
)
type server struct{}
func (s server) Max(srv pb.Math_MaxServer) error {
log.Println("start new server")
var max int32
ctx := srv.Context()
i := 0
fromMsg := 0
for {
// exit if context is done
// or continue
select {
case <-ctx.Done():
return ctx.Err()
default:
}
// receive data from stream
req, err := srv.Recv()
if err == io.EOF {
// return will close stream from server side
log.Println("exit")
return nil
}
if err != nil {
log.Printf("receive error %v", err)
continue
}
if i >= fromMsg {
recvTime := time.Now().UnixNano()
log.Printf("Recv Time %v Index: %v Num: %v", recvTime,i,req.Num)
}
i++
// continue if number reveived from stream
// less than max
if req.Num <= max {
continue
}
// update max and send it to stream
/*
max = req.Num
resp := pb.Response{Result: max}
if err := srv.Send(&resp); err != nil {
log.Printf("send error %v", err)
}
*/
//log.Printf("send new max=%d", max)
}
}
func main() {
// create listiner
lis, err := net.Listen("tcp", ":50005")
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
// create grpc server
s := grpc.NewServer()
pb.RegisterMathServer(s, server{})
// and start...
if err := s.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
}

Web server and listening nats at the same time

My code reads input from terminal and send those value to nats while it needs to have an http endpoint.
Separately it works but when I combine all of them it does not read from nats. If you could point me to a right direction I would appreciate.
package main
import (
"bufio"
"fmt"
nats "github.com/nats-io/nats.go"
"html/template"
"log"
"net/http"
"os"
)
func main() {
wd, err := os.Getwd()
if err != nil {
log.Fatal(err)
}
tmpl := template.Must(template.ParseFiles(wd + "/template/main.html"))
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
data := TodoPageData{
PageTitle: "Demo",
}
tmpl.Execute(w, data)
})
http.ListenAndServe(":8081", nil)
type message struct {
content string
}
var messages []message
nc, err := nats.Connect(
nats.DefaultURL,
)
if err != nil {
log.Fatal(err)
}
defer nc.Close()
// Subscribe
if _, err := nc.Subscribe("updates", func(m *nats.Msg) {
fmt.Printf("Received a message: %s\n", string(m.Data))
}); err != nil {
log.Fatal(err)
}
// io r/w
scanner := bufio.NewScanner(os.Stdin)
for scanner.Scan() {
if err := nc.Publish("updates", []byte(scanner.Text())); err != nil {
log.Fatal(err)
}
messages = append(messages, message{scanner.Text()})
for _, message := range messages {
fmt.Println(message.content)
}
}
if scanner.Err() != nil {
// handle error.
}
}
http.ListenAndServe is a blocking call. Start it on a new goroutine:
go http.ListenAndServe(":8081", nil)

Poll API, pass result to chan, pass from chan to Websocket. Panic

I'm writing a small package which does a GET request to an external API every 2 seconds. It takes the value from this request and passes it into a channel. I have made this channel available to a http.handler (chi router) which upgrades to a websocket where the front-end will grab the value in realtime. the panic error is a lot of lines but i guess the most important is this:
2018/11/14 16:47:55 http: response.WriteHeader on hijacked connection
2018/11/14 16:47:55 http: response.Write on hijacked connection
Aside from that I'm sure there is a better way of doing this. Any experienced Gophers out there have any pointers to help a noob such as myself improve this?
package currencyticker
import (
"bitbucket.org/special/api/config"
"encoding/json"
"fmt"
"github.com/go-chi/chi"
"github.com/go-chi/render"
"github.com/gorilla/websocket"
"github.com/leekchan/accounting"
"io/ioutil"
"log"
"math/big"
"net/http"
"time"
)
var (
ac = accounting.Accounting{Precision: 2}
from = "USD"
to = "EUR,SWK"
url = "https://min-api.currencyapi.com/data/price?fsym=" + from + "&tsyms=" + to
messages = make(chan float64)
)
var wsupgrader = websocket.Upgrader{
ReadBufferSize: 1024,
WriteBufferSize: 1024,
CheckOrigin: func(r *http.Request) bool {
return true // Disable CORS for testing
},
}
// Config - init
type Config struct {
*config.Config
}
type result map[string]float64
// New - init the configs
func New(configuration *config.Config) *Config {
return &Config{configuration}
}
// Routes - api urls
func (config *Config) Routes() *chi.Mux {
router := chi.NewRouter()
router.Use(
render.SetContentType(render.ContentTypeHTML), // Set content-Type headers as application/json
)
router.Get("/", config.GetPrice) // subscribe to new tweets
return router
}
func (config *Config) GetPrice(w http.ResponseWriter, r *http.Request) {
conn, err := wsupgrader.Upgrade(w, r, nil)
if err != nil {
fmt.Println(fmt.Printf("Failed to set websocket upgrade: %+v ", err))
return
}
for {
time.Sleep(1 * time.Second)
price := <-messages
w, err := conn.NextWriter(websocket.TextMessage)
if err != nil {
fmt.Println("ws error", err)
}
currVal := ac.FormatMoneyBigFloat(big.NewFloat(price))
if _, err := w.Write([]byte(currVal)); err != nil {
fmt.Printf("w.Write() returned %v", err)
}
w.Close()
}
}
// start getting the price of ether as soon as they ap starts
func init() {
go startPollingPriceAPI()
}
// Go Routine to start polling
func startPollingPriceAPI() {
for {
time.Sleep(2 * time.Second)
go getPriceFromAPI()
}
}
func getPriceFromAPI() {
w := http.Client{
// Timeout: time.Second * 3,
}
req, _ := http.NewRequest(http.MethodGet, url, nil)
res, err := w.Do(req)
if err != nil {
log.Println("err getting price [req]: ", err)
}
body, err := ioutil.ReadAll(res.Body)
if err != nil {
log.Println("err getting price [io-read]: ", err)
}
r := result{}
if jsonErr := json.Unmarshal(body, &r); jsonErr != nil {
log.Println("err getting price [json]: ", jsonErr)
}
fmt.Println("1 Dollar = €", r["EUR"])
messages <- r["EUR"]
}

go routine - why websocket reports the connection as closed?

I'm trying to create a client and a server using Go but for some reason the server reports the connection as "closed". As the code is trivial I can't think of anything wrong with my code. Any help is appreciated.
package main
import (
log "github.com/golang/glog"
"net/http"
"golang.org/x/net/websocket"
"time"
"flag"
)
type server struct {
payload chan string
}
// srv pushes the messages received via ws into srv.payload
func (srv *server) serve(ws *websocket.Conn) {
go func() {
var msg string
if err := websocket.Message.Receive(ws, &msg); err != nil {
log.Exit(err)
}
srv.payload <- msg
}()
return
}
// This example demonstrates a trivial client/ server.
func main() {
flag.Parse()
srv := server{payload: make(chan string, 10)}
http.Handle("/echo", websocket.Handler(srv.serve))
go func() {
err := http.ListenAndServe(":12345", nil)
if err != nil {
log.Errorf("ListenAndServe: " + err.Error())
}
}()
// give the server some time to start listening
time.Sleep(3 *time.Second)
//dial and test the response.
ws, err := websocket.Dial("ws://localhost:12345/echo", "", "http://localhost/?x=45")
if err != nil {
log.Exit(err)
}
ms := "test"
if err := websocket.Message.Send(ws, ms); err != nil {
log.Exit(err)
}
msg := <-srv.payload
if msg != ms{
log.Errorf("msg %v is not %v", ms)
}
}
Error
t.go:21] read tcp 127.0.0.1:12345->127.0.0.1:43135:
Edit:
After some try and error I've found that if I remove the go routine from the serve method it works but it doesn't make sense to me. Any idea why it doesn't work when websocket.Message.Receive is in a separate go routine?
package main
import (
log "github.com/golang/glog"
"net/http"
"golang.org/x/net/websocket"
"time"
"flag"
)
type server struct {
payload chan string
}
// srv pushes the messages received via ws into srv.payload
func (srv *server) serve(ws *websocket.Conn) {
var msg string
if err := websocket.Message.Receive(ws, &msg); err != nil {
log.Exit(err)
}
srv.payload <- msg
return
}
// This example demonstrates a trivial client/ server.
func main() {
flag.Parse()
srv := server{payload: make(chan string, 10)}
go func() {
http.Handle("/echo", websocket.Handler(srv.serve))
err := http.ListenAndServe(":12345", nil)
if err != nil {
log.Errorf("ListenAndServe: " + err.Error())
}
}()
// give the server some time to start listening
time.Sleep(3 *time.Second)
//dial and test the response.
ws, err := websocket.Dial("ws://localhost:12345/echo", "", "http://localhost/?x=45")
if err != nil {
log.Exit(err)
}
ms := "test"
if err := websocket.Message.Send(ws, ms); err != nil {
log.Exit(err)
}
msg := <-srv.payload
if msg != ms{
log.Errorf("msg %v is not %v", ms)
}
}
The websocket server closes the connection when the handler returns.
Removing the Go routine is the correct fix.

Resources