invalid memory address when creating grpc client request with repeated field - go

I have an issue with my grpc server and client, I'm trying to create a repeated field in the grpc and send it to the client. but it failed on error, what am I doing wrong on my grpc client/server? i tried send the RequestsCommand as value but this didn't work to.
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x165cf0e]
goroutine 40 [running]:
main.(*Server).RunCommands(0x172b960?, {0x18be540?, 0xc00031c030?}, 0x107a360?)
<autogenerated>:1 +0x2e
github.com/mygithub/some-package/proto._ExecuterService_RunCommands_Handler({0x1701a20?, 0xc0002430d0}, {0x18be540, 0xc00031c030}, 0xc000318070, 0x0)
..../executer_grpc.pb.go:93 +0x170
executer_grpc.pb.go:93 is :
in := new(CommandsRequest)
return srv.(ExecuterServiceServer).RunCommands(ctx, in)
on the server side I have the following:
grpc_server.go:
package main
import (
"fmt"
pb "github.com/mygithub/some-package/proto"
"google.golang.org/grpc"
)
type Server struct {
pb.ExecuterServiceServer
}
func startServer() {
addr := GetAgentAddress()
lis, err := net.Listen("tcp", addr)
if err != nil {
log.Fatalf("Faild to listen on %v\n", err)
}
s := grpc.NewServer()
pb.RegisterExecuterServiceServer(s, &Server{})
if err = s.Serve(lis); err != nil {
log.Fatalf("Failed to serve: %v\n", err)
}
}
grpc_client.go:
package main
import (
"context"
"encoding/json"
"fmt"
v1 "github.com/mygithub/some-package/api/v1"
pb "github.com/mygithub/some-package/proto"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
)
func sendCommandsWithClient() (string, error) {
command := v1.CommandSpec{
BashCommand: "ls",
Async: false,
}
addr := "localhost:4000"
conn, err := grpc.Dial(addr, grpc.WithTransportCredentials(insecure.NewCredentials()))
if err != nil {
return "", err
}
defer conn.Close()
client := pb.NewExecuterServiceClient(conn)
if err != nil {
return "", err
}
commandRequest := &pb.CommandRequest{
BashCommand: command.BashCommand,
Slug: "test",
Async: command.Async,
}
req := &pb.CommandsRequest{}
req.Commands = append(req.Commands, commandRequest)
aAsJsonString, _ := json.Marshal(req)
fmt.Println(string(aAsJsonString))
res, err := client.RunCommands(context.Background(), req)
if err != nil {
return "", err
}
return res.Content, nil
}
func main() {
_, err := sendCommandsWithClient()
if err != nil {
return
}
}
RunCommands:
func (s *Server) RunCommands(ctx context.Context, in *pb.CommandsRequest) (*pb.CommandsResponse, error) {
log.Println("test")
command := v1.CommandSpec{BashCommand: "ls", Slug: "test", Async: true}
CommandsQueue := append(CommandsQueue, command)
size := len(CommandsQueue)
return &pb.CommandsResponse{Status: 0, Content: strconv.Itoa(size)}, nil
}
go version: go version go1.18.2 darwin/amd64
protoc: proto3

you should go build or go run with all of your go files.
go run main.go model.go somefile.go

Related

GCP - get project NAT GW's

We have account on GCP which contain valid cloud Nat, now we want to get those values via
GCP sdk, I've tried the following and get empty response (maybe I use the wrong API and it not ListExternalVpnGatewaysRequest)
package main
import (
"context"
"fmt"
compute "cloud.google.com/go/compute/apiv1"
"google.golang.org/api/iterator"
computepb "google.golang.org/genproto/googleapis/cloud/compute/v1"
)
func main() {
ctx := context.Background()
c, err := compute.NewExternalVpnGatewaysRESTClient(ctx)
if err != nil {
fmt.Println(err)
}
defer c.Close()
proj := "dev-proj"
req := &computepb.ListExternalVpnGatewaysRequest{
//Filter: new(string),
//MaxResults: new(uint32),
//OrderBy: new(string),
//PageToken: new(string),
Project: proj,
//ReturnPartialSuccess: new(bool),
}
it := c.List(ctx, req)
for {
resp, err := it.Next()
if err == iterator.Done {
break
}
if err != nil {
fmt.Println(err)
}
// TODO: Use resp.
_ = resp
fmt.Println(resp)
}
}
I need to get the following values using GCP GO SDK
update
I tried the following as-is and I got error
package main
import (
"context"
"fmt"
"google.golang.org/api/compute/v1"
"log"
)
func main() {
project := "my-proj"
region := "my-region"
ctx := context.Background()
computeService, err := compute.New(ctx)
if err != nil {
log.Fatal(err)
}
req := computeService.Routers.List(project, region)
if err := req.Pages(ctx, func(page *compute.RouterList) error {
for _, router := range page.Items {
// process each `router` resource:
fmt.Printf("%#v\n", router)
// NAT Gateways are found in router.nats
}
return nil
}); err != nil {
log.Fatal(err)
}
}
Error is: ./main.go:16:36: cannot use ctx (type context.Context) as type *http.Client in argument to compute.New
A VPN Gateway is not the same as a NAT Gateway.
Use this code to list routers. Within the list of routers, is the NAT Gateways
import "google.golang.org/api/compute/v1"
// Replace with valid values for your project
project := "my-project"
region := "my-region"
ctx := context.Background()
c, err := google.DefaultClient(ctx, compute.CloudPlatformScope)
if err != nil {
log.Fatal(err)
}
computeService, err := compute.New(c)
if err != nil {
log.Fatal(err)
}
req := computeService.Routers.List(project, region)
if err := req.Pages(ctx, func(page *compute.RouterList) error {
for _, router := range page.Items {
// process each `router` resource:
fmt.Printf("%#v\n", router)
// NAT Gateways are found in router.nats
}
return nil
}); err != nil {
log.Fatal(err)
}
SDK Documentation

go function - panic: runtime error: invalid memory address or nil pointer dereference

Thanks in advance for everyone's help. I've marked lines 41, 42, and 58 where the problem occurs. I have not been able to track what is throwing the error. It appears all the variables are properly assigned. I use the command:
go run file.go 'command' 'ip address'
ex. - go run file.go uptime 8.8.8.8
The error is:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x54fb36]
goroutine 20 [running]:
golang.org/x/crypto/ssh.(*Client).NewSession(0x0, 0x3, 0xc42009e310, 0x10)
/home/user/go/src/golang.org/x/crypto/ssh/client.go:130 +0x26
main.executeCmd(0x7ffce5f83a51, 0x6, 0x7ffce5f83a58, 0xd, 0x5d3077, 0x2, 0xc42009a820, 0x0, 0x0)
/home/user/ssh.go:58 +0x16f
main.main.func1(0xc4200d2000, 0x7ffce5f83a51, 0x6, 0xc42009a820, 0x7ffce5f83a58, 0xd, 0x5d3077, 0x2)
/home/user/ssh.go:42 +0x7a
created by main.main
/home/user/ssh.go:41 +0x41b
package main
import (
"bytes"
"fmt"
"io/ioutil"
"log"
"os"
"time"
"golang.org/x/crypto/ssh"
)
func main() {
cmd := os.Args[1]
hosts := os.Args[2:]
results := make(chan string, 10)
timeout := time.After(10 * time.Second)
port := "22"
var hostKey ssh.PublicKey
key, err := ioutil.ReadFile("/home/user/file.pem")
if err != nil {
log.Fatalf("unable to read private key: %v", err)
}
signer, err := ssh.ParsePrivateKey(key)
if err != nil {
log.Fatalf("unable to parse private key: %v", err)
}
config := &ssh.ClientConfig{
User: "ec2-user",
Auth: []ssh.AuthMethod{
ssh.PublicKeys(signer),
},
HostKeyCallback: ssh.FixedHostKey(hostKey),
}
for _, hostname := range hosts {
go func(hostname string, port string) { // LINE 41
results <- executeCmd(cmd, hostname, port, config) // LINE 42
}(hostname, port)
}
for i := 0; i < len(hosts); i++ {
select {
case res := <-results:
fmt.Print(res)
case <-timeout:
fmt.Println("Timed out!")
return
}
}
}
func executeCmd(command, hostname string, port string, config *ssh.ClientConfig) (string, error) {
conn, err := ssh.Dial("tcp", fmt.Sprintf("%s:%s", hostname, port), config)
if err != nil {
return "", err
}
defer conn.Close()
session, err := conn.NewSession() //LINE 58
if err != nil {
return "", err
}
defer session.Close()
var stdoutBuf bytes.Buffer
session.Stdout = &stdoutBuf
if err := session.Run(command); err != nil {
return "", err
}
return fmt.Sprintf("%s -> %s", hostname, stdoutBuf.String()), nil
}
As #JoshuaKolden pointed out, you need to check the error coming out of ssh.Dial and then not continue if the value is not nil. Here's a version of your executeCmd function with error checking:
func executeCmd(command, hostname string, port string, config *ssh.ClientConfig) (string, error) {
conn, err := ssh.Dial("tcp", fmt.Sprintf("%s:%s", hostname, port), config)
if err != nil {
return "", err
}
defer conn.Close()
session, err := conn.NewSession() //LINE 58
if err != nil {
return "", err
}
defer session.Close()
var stdoutBuf bytes.Buffer
session.Stdout = &stdoutBuf
if err := session.Run(command); err != nil {
return "", err
}
return fmt.Sprintf("%s -> %s", hostname, stdoutBuf.String()), nil
}
You will of course need to adapt the goroutine that calls executeCmd for error handling as well. Maybe print the error message to os.Stderr to find out what the problem is.
The reason why you get a panic is because you have an error coming out of ssh.Dial, leaving the value of the variable conn as nil. If you look at client.go line 130 in the ssh package you can see that it does a method call on the pointer receiver c to OpenChannel. Normally this isn't a problem for struct methods that take a pointer receiver, but in this case OpenChannel is a method of the Conn interface that is embedded in the Client struct. Go panics when an interface method is invoked with a nil receiver.
Edit: A potential way to use this new version of the function in your goroutine:
go func(hostname string, port string) { // LINE 41
result, err := executeCmd(cmd, hostname, port, config) // LINE 42
if err != nil {
fmt.Fprintf(os.Stderr, "error running cmd on host %s port %s: %s", hostname, port, err)
return
}
results <- result
}(hostname, port)
Unfortunately, you cannot send to a channel and assign to another variable in the same statement. Also, you need to check for the error and do something if it's not nil, in this case I chose printing it to standard error.

tail -f several logs and send every new line to kafka using sarama

I'd like to tail -f several log files in /var/log (I thing one goroutine per log would be fine) and every goroutine will keep on "watching" forever on log and send every new line with kafka client sarama.
Here is my code (does not work) :
package main
import (
"flag"
"github.com/Shopify/sarama"
"log"
"os"
"fmt"
"strings"
"github.com/hpcloud/tail"
"github.com/spf13/viper"
//"io/ioutil"
"reflect"
)
func produce(producer sarama.SyncProducer, cfg *sarama.Config, brokers *string, topic string, logger *log.Logger, log string, t *tail.Tail){
logger.Printf("Entering produce")
logger.Println(strings.Split(*brokers, ","))
logger.Println(reflect.TypeOf(strings.Split(*brokers, ",")))
logger.Println(log)
/*t, err := tail.TailFile(log, tail.Config{Follow: true, ReOpen: true})
if err != nil {
fmt.Println(fmt.Errorf("Error with tail: %v\n", err.Error()))
}*/
for line := range t.Lines {
//logger.Println(line)
//logger.Println(line.Text)
msg := &sarama.ProducerMessage{Topic: topic, Value: sarama.StringEncoder(line.Text)}
_, _, err := producer.SendMessage(msg)
if err != nil {
logger.Printf("FAILED to send message: %s\n", err)
}
}
}
func main() {
//Getting config file params
viper.SetConfigName("config")
viper.AddConfigPath("/root/work/src/linux2kafka/")
err := viper.ReadInConfig()
if err != nil {
panic(err)
}
viper.WatchConfig()
logList := viper.Get("log_list")
//logListString, err := ioutil.ReadFile(logList.(string))
//fmt.Println(logList)
//fmt.Println(reflect.TypeOf(logList))
logsConfig := strings.Split(logList.(string),",")
// print logs to watch
/*for i := range logsConfig {
fmt.Println(logsConfig[i])
}*/
brokerList := viper.Get("brokerList")
brokers := flag.String("brokers", brokerList.(string), "Comma separated kafka brokers list") //must be set in config.toml
topic := flag.String("topic", "test0", "Kafka topic to send messages to")
flag.Parse()
logger := log.New(os.Stdout, "producer ", log.Lmicroseconds)
cfg := sarama.NewConfig()
//Wait for replication
cfg.Producer.RequiredAcks = -1
cfg.Producer.Flush.Frequency = 333
cfg.Producer.Flush.Messages = 1000
cfg.Producer.Flush.MaxMessages = 3000
producer, err := sarama.NewSyncProducer(strings.Split(*brokers, ","), cfg)
if err != nil {
logger.Fatalln(err)
}
defer func() {
if err := producer.Close(); err != nil {
logger.Fatalln(err)
}
}()
for i := range logsConfig {
fmt.Println("go")
t, err := tail.TailFile(logsConfig[i], tail.Config{Follow: true, ReOpen: true})
if err != nil {
fmt.Println(fmt.Errorf("Error with tail: %v\n", err.Error()))
}
go produce(producer, cfg, brokers, *topic, logger, logsConfig[i], t)
}
}
And here are my errors (nothing is received by the consumer):
root#home:~/work/src/linux2kafka# go run main.go
go
producer 15:54:44.297745 Entering produce
root#home:~/work/src/linux2kafka# go run main.go
go
root#home:~/work/src/linux2kafka# go run main.go
go
root#home:~/work/src/linux2kafka# go run main.go
go
producer 15:55:01.951155 Entering produce
producer 15:55:01.951193 [localhost:9092 localhost:9092]
producer 15:55:01.951205 []string
producer 15:55:01.951214 /root/work/src/linux2kafka/test/log
panic: send on closed channel
goroutine 56 [running]:
panic(0x756440, 0xc820164290)
/usr/local/go/src/runtime/panic.go:464 +0x3e6
github.com/Shopify/sarama.(*syncProducer).SendMessage(0xc8201742a0, 0xc820176300, 0x0, 0x0, 0x0, 0x0)
/root/work/src/github.com/Shopify/sarama/sync_producer.go:66 +0x156
main.produce(0x7f65528661b8, 0xc8201742a0, 0xc82008ea20, 0xc82000b230, 0x8855c0, 0x5, 0xc8200789b0, 0xc820011320, 0x23, 0xc82017e000)
/root/work/src/linux2kafka/main.go:31 +0x5d2
created by main.main
/root/work/src/linux2kafka/main.go:85 +0x9f7
exit status 2
I cannot understand what I'm doing wrong. thx
Here is the modified code, is that ok ? :
package main
import (
"flag"
"github.com/Shopify/sarama"
"log"
"os"
"fmt"
"strings"
"github.com/hpcloud/tail"
"github.com/spf13/viper"
//"io/ioutil"
//"reflect"
)
//func produce(producer sarama.SyncProducer, cfg *sarama.Config, brokers *string, topic string, logger *log.Logger, log string, t *tail.Tail){
func produce(cfg *sarama.Config, brokers *string, topic string, logger *log.Logger, log string, t *tail.Tail){
logger.Println("Entering produce")
/*logger.Println(strings.Split(*brokers, ","))
logger.Println(reflect.TypeOf(strings.Split(*brokers, ",")))
logger.Println(log)*/
logger.Printf("sarama.NewSyncProducer")
producer, err := sarama.NewSyncProducer(strings.Split(*brokers, ","), cfg)
if err != nil {
logger.Fatalln(err)
}
defer func() {
if err := producer.Close(); err != nil {
logger.Fatalln(err)
}
}()
/*t, err := tail.TailFile(log, tail.Config{Follow: true, ReOpen: true})
if err != nil {
fmt.Println(fmt.Errorf("Error with tail: %v\n", err.Error()))
}*/
for line := range t.Lines {
//logger.Println(line)
//logger.Println(line.Text)
logger.Printf("ProduceMessage")
msg := &sarama.ProducerMessage{Topic: topic, Value: sarama.StringEncoder(line.Text)}
_, _, err := producer.SendMessage(msg)
if err != nil {
logger.Printf("FAILED to send message: %s\n", err)
}
}
}
func main() {
//Getting config file params
viper.SetConfigName("config")
viper.AddConfigPath("/root/work/src/linux2kafka/")
err := viper.ReadInConfig()
if err != nil {
panic(err)
}
viper.WatchConfig()
logList := viper.Get("log_list")
//logListString, err := ioutil.ReadFile(logList.(string))
//fmt.Println(logList)
//fmt.Println(reflect.TypeOf(logList))
logsConfig := strings.Split(logList.(string),",")
// print logs to watch
/*for i := range logsConfig {
fmt.Println(logsConfig[i])
}*/
brokerList := viper.Get("brokerList")
brokers := flag.String("brokers", brokerList.(string), "Comma separated kafka brokers list") //must be set in config.toml
topic := flag.String("topic", "test0", "Kafka topic to send messages to")
flag.Parse()
logger := log.New(os.Stdout, "producer ", log.Lmicroseconds)
cfg := sarama.NewConfig()
//Wait for replication
cfg.Producer.RequiredAcks = -1
cfg.Producer.Flush.Frequency = 333
cfg.Producer.Flush.Messages = 1000
cfg.Producer.Flush.MaxMessages = 3000
for i := range logsConfig {
fmt.Println("go")
t, err := tail.TailFile(logsConfig[i], tail.Config{Follow: true, ReOpen: true})
if err != nil {
fmt.Println(fmt.Errorf("Error with tail: %v\n", err.Error()))
}
go produce(cfg, brokers, *topic, logger, logsConfig[i], t)
}
}
But still not working... it's not printing the first Println
root#home:~/work/src/linux2kafka# go run main.go
go
root#home:~/work/src/linux2kafka# go run main.go
go
root#home:~/work/src/linux2kafka# go run main.go
go

How to connect remote ssh server with socks proxy?

package main
import (
"errors"
"fmt"
"net"
"golang.org/x/crypto/ssh"
)
var (
socks string = "127.0.0.1:8000"
server string = "192.168.1.1:2222"
cmd string = ""
login string = "root"
password string = ""
)
func main() {
c, err := netConn(socks)
if err != nil {
fmt.Println(err)
}
conf := &ssh.ClientConfig{
User: login,
Auth: []ssh.AuthMethod{
ssh.Password(password),
},
}
client, err := Dialer(conf, c, server)
if err != nil {
fmt.Println(err)
}
session, err := client.NewSession()
defer session.Close()
if err != nil {
fmt.Println(err)
}
session.Run(cmd)
}
func netConn(addr string) (net.Conn, error) {
conn, err := net.Dial("tcp", addr)
if err != nil {
return nil, errors.New(fmt.Sprintf("Unable to connect to %v", err))
}
return conn, nil
}
func Dialer(conf *ssh.ClientConfig, c net.Conn, host string) (*ssh.Client, error) {
conn, chans, reqs, err := ssh.NewClientConn(c, host, conf)
if err != nil {
return nil, err
}
return ssh.NewClient(conn, chans, reqs), nil
}
Client panic
ssh: handshake failed: EOF
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x80f3e2d]
goroutine 1 [running]:
panic(0x8201b60, 0x18522030)
/usr/lib/go/src/runtime/panic.go:464 +0x326
golang.org/x/crypto/ssh.(*Client).NewSession(0x0, 0x1, 0x0, 0x0)
/home/user/go/src/golang.org/x/crypto/ssh/client.go:129 +0xcd
main.main()
/home/user/go/src/ss/i.go:34 +0x276
And socks server in log
[ERR] socks: Unsupported SOCKS version: [83]
How i can make it posible? Thanks.
You can use the x/net/proxy package as follows:
package main
import (
"log"
"golang.org/x/crypto/ssh"
"golang.org/x/net/proxy"
)
func main() {
sshConfig := &ssh.ClientConfig{
User: "user",
// Auth: .... fill out with keys etc as normal
}
client, err := proxiedSSHClient("127.0.0.1:8000", "127.0.0.1:22", sshConfig)
if err != nil {
log.Fatal(err)
}
// get a session etc...
}
func proxiedSSHClient(proxyAddress, sshServerAddress string, sshConfig *ssh.ClientConfig) (*ssh.Client, error) {
dialer, err := proxy.SOCKS5("tcp", proxyAddress, nil, proxy.Direct)
if err != nil {
return nil, err
}
conn, err := dialer.Dial("tcp", sshServerAddress)
if err != nil {
return nil, err
}
c, chans, reqs, err := ssh.NewClientConn(conn, sshServerAddress, sshConfig)
if err != nil {
return nil, err
}
return ssh.NewClient(c, chans, reqs), nil
}
As you can see from the ssh package API, the function
func Dial(network, addr string, config *ClientConfig) (*Client, error)
directly returns an object of type ssh.Client.
Therefore you can shorten a lot your code with:
func main() {
conf := &ssh.ClientConfig{
User: login,
Auth: []ssh.AuthMethod{
ssh.Password(password),
},
}
client, err := ssh.Dial("tcp", server, conf)
if err != nil {
fmt.Println(err)
}
defer client.Close()
session, err := client.NewSession()
defer session.Close()
if err != nil {
fmt.Println(err)
}
session.Run(cmd)
}
Hope this helps !

go routine - why websocket reports the connection as closed?

I'm trying to create a client and a server using Go but for some reason the server reports the connection as "closed". As the code is trivial I can't think of anything wrong with my code. Any help is appreciated.
package main
import (
log "github.com/golang/glog"
"net/http"
"golang.org/x/net/websocket"
"time"
"flag"
)
type server struct {
payload chan string
}
// srv pushes the messages received via ws into srv.payload
func (srv *server) serve(ws *websocket.Conn) {
go func() {
var msg string
if err := websocket.Message.Receive(ws, &msg); err != nil {
log.Exit(err)
}
srv.payload <- msg
}()
return
}
// This example demonstrates a trivial client/ server.
func main() {
flag.Parse()
srv := server{payload: make(chan string, 10)}
http.Handle("/echo", websocket.Handler(srv.serve))
go func() {
err := http.ListenAndServe(":12345", nil)
if err != nil {
log.Errorf("ListenAndServe: " + err.Error())
}
}()
// give the server some time to start listening
time.Sleep(3 *time.Second)
//dial and test the response.
ws, err := websocket.Dial("ws://localhost:12345/echo", "", "http://localhost/?x=45")
if err != nil {
log.Exit(err)
}
ms := "test"
if err := websocket.Message.Send(ws, ms); err != nil {
log.Exit(err)
}
msg := <-srv.payload
if msg != ms{
log.Errorf("msg %v is not %v", ms)
}
}
Error
t.go:21] read tcp 127.0.0.1:12345->127.0.0.1:43135:
Edit:
After some try and error I've found that if I remove the go routine from the serve method it works but it doesn't make sense to me. Any idea why it doesn't work when websocket.Message.Receive is in a separate go routine?
package main
import (
log "github.com/golang/glog"
"net/http"
"golang.org/x/net/websocket"
"time"
"flag"
)
type server struct {
payload chan string
}
// srv pushes the messages received via ws into srv.payload
func (srv *server) serve(ws *websocket.Conn) {
var msg string
if err := websocket.Message.Receive(ws, &msg); err != nil {
log.Exit(err)
}
srv.payload <- msg
return
}
// This example demonstrates a trivial client/ server.
func main() {
flag.Parse()
srv := server{payload: make(chan string, 10)}
go func() {
http.Handle("/echo", websocket.Handler(srv.serve))
err := http.ListenAndServe(":12345", nil)
if err != nil {
log.Errorf("ListenAndServe: " + err.Error())
}
}()
// give the server some time to start listening
time.Sleep(3 *time.Second)
//dial and test the response.
ws, err := websocket.Dial("ws://localhost:12345/echo", "", "http://localhost/?x=45")
if err != nil {
log.Exit(err)
}
ms := "test"
if err := websocket.Message.Send(ws, ms); err != nil {
log.Exit(err)
}
msg := <-srv.payload
if msg != ms{
log.Errorf("msg %v is not %v", ms)
}
}
The websocket server closes the connection when the handler returns.
Removing the Go routine is the correct fix.

Resources