I'm working on a giant dictionary of words -> language, the data for which I have, but what I need is to have one thread running a daemon, written in Go, which keeps all of this in memory (yes, I have that much memory too) and is "callable" by other Go apps.
I'm sure this is a standard type of thing to do, but to be honest I've never attempted such a thing before and am not familiar enough to know where to find information on how to do this.
Having it run as a daemon is easy. My problem is what is an efficient way of calling this app from another Go app, which will need to be done many millions of times.
I'm thinking something along the lines of:
connection, err := InitateConnectionToApp()
for _, someword := range mysliceofstrings {
languageofword := connection.FindIt(someword)
}
Then the daemon somehow receives this request, looks up the value in its map and delivers it back.
I hope that makes sense. I have tried looking on Google but there is nothing I can find specific to Go.
If anyone can tell me where to start that would be great.
You could use RPC Go's standard Remote Procedure Call package.
Just Expose your api and then create a client to invoke the method remotely.
Simple example copy pasted from the docs:
package server
type Args struct {
A, B int
}
type Quotient struct {
Quo, Rem int
}
type Arith int
func (t *Arith) Multiply(args *Args, reply *int) error {
*reply = args.A * args.B
return nil
}
func (t *Arith) Divide(args *Args, quo *Quotient) error {
if args.B == 0 {
return errors.New("divide by zero")
}
quo.Quo = args.A / args.B
quo.Rem = args.A % args.B
return nil
}
func main() {
arith := new(Arith)
rpc.Register(arith)
rpc.HandleHTTP()
l, e := net.Listen("tcp", ":1234")
if e != nil {
log.Fatal("listen error:", e)
}
go http.Serve(l, nil)
}
Related
I have recently looked into Go plugins instead of manually loading .so files myself.
Basically, I have a game server application, and I want to be able to load plugins (using plugins package) when the server starts. Then, in the plugin itself, I want to be able to call exported functions that are a part of the server.
Say I have this plugin, which is compiled to example_plugin.so using go build -buildmode=plugin:
package main
import "fmt"
func init() {
fmt.Println("Hello from plugin!")
}
Then say I have this server application, which loads the plugin (and ultimately calls the "init" function under the hood):
package main
import (
"fmt"
"plugin"
)
func main() {
fmt.Println("Server started")
if _, err := plugin.Open("example_plugin.so"); err != nil {
panic(err)
}
}
// some API function that loaded plugins can call
func GetPlayers() {}
The output is:
Server started
Hello from plugin!
This works as expected, however I want to be able to call that GetPlayers function (and any other exported functions in the server application, ideally) from the plugin (and any other plugins.) I was thinking about making some sort of library consisting of interfaces containing API functions that the server implements, however I have no idea where to start. I know I will probably need to use a .a file or something similar.
For clarification, I am developing this application for use on Linux, so I am fine with a solution that only works on Linux.
Apologies if this is poorly worded, first time posting on SO.
As mentioned in the comments, there is a Lookup function. In the documentation for the module they have the following example:
// A Symbol is a pointer to a variable or function.
// For example, a plugin defined as
//
// var V int
//
// func F() { fmt.Printf("Hello, number %d\n", V) }
//
// may be loaded with the Open function and then the exported package
// symbols V and F can be accessed
package main
import (
"fmt"
"plugin"
)
func main() {
p, err := plugin.Open("plugin_name.so")
if err != nil {
panic(err)
}
v, err := p.Lookup("V")
if err != nil {
panic(err)
}
f, err := p.Lookup("F")
if err != nil {
panic(err)
}
*v.(*int) = 7
f.(func())() // prints "Hello, number 7"
}
I think the most confusing lines here are
*v.(*int) = 7
f.(func())() // prints "Hello, number 7"
The first one of them performs a type assertion to *int to assert that v is indeed a pointer to int. That is needed since Lookup returns an interface{} and in order to do anything useful with a value, you should clarify its type.
The second line performs another type assertion, this time making sure that f is a function with no arguments and no return values, after which, immediately calls it. Since function F from the original module was referencing V (which we've replaced with 7), this call will display Hello, number 7.
I'm using github.com/sirupsen/logrus and github.com/pkg/errors. When I hand an error wrapped or created from pkg/errors, all I see in the log out is the error message. I want to see the stack trace.
From this issue, https://github.com/sirupsen/logrus/issues/506, I infer that logrus has some native method for working with pkg/errors.
How can I do this?
The comment on your Logrus issue is incorrect (and incidentally, appears to come from someone with no affiliation with Logrus, and who has made no contributions to Logrus, so not actually from "the Logrus team").
It is easy to extract the stack trace in a pkg/errors error, as documented:
type stackTracer interface {
StackTrace() errors.StackTrace
}
This means that the easiest way to log the stack trace with logrus would be simply:
if stackErr, ok := err.(stackTracer); ok {
log.WithField("stacktrace", fmt.Sprintf("%+v", stackErr.StackTrace()))
}
As of today, when my a pull request of mine was merged with pkg/errors, this is now even easier, if you're using JSON logging:
if stackErr, ok := err.(stackTracer); ok {
log.WithField("stacktrace", stackErr.StackTrace())
}
This will produce a log format similar to "%+v", but without newlines or tabs, with one log entry per string, for easy marshaling into a JSON array.
Of course, both of these options force you to use the format defined by pkg/errors, which isn't always ideal. So instead, you can iterate through the stack trace, and produce your own formatting, possibly producing a format easily marshalable to JSON.
if err, ok := err.(stackTracer); ok {
for _, f := range err.StackTrace() {
fmt.Printf("%+s:%d\n", f, f) // Or your own formatting
}
}
Rather than printing each frame, you can coerce it into any format you like.
The inference is wrong. Logrus does not actually know how to handle the error.
Update the Logrus team official said that this is NOT a supported feature, https://github.com/sirupsen/logrus/issues/895#issuecomment-457656556.
A Java-ish Response
In order to universally work with error handlers in this way, I composed a new version of Entry, which is from Logrus. As the example shows, create a new Entry with what ever common fields you want (below the example is a logger set in a handler that keeps track of the caller id. Pass PgkError through your layers as you work the Entry. When you need to log specific errors, like call variables experiencing the error, start with the PkgError.WithError(...) then add your details.
This is a starting point. If you want to use this generally, implement all of the Entity interface on PkgErrorEntry. Continue to delegate to the internal entry, but return a new PkgErrorEntry. Such a change would make the value true drop in replacement for Entry.
package main
import (
"fmt"
"github.com/sirupsen/logrus"
"strings"
unwrappedErrors "errors"
"github.com/pkg/errors"
)
// PkgErrorEntry enables stack frame extraction directly into the log fields.
type PkgErrorEntry struct {
*logrus.Entry
// Depth defines how much of the stacktrace you want.
Depth int
}
// This is dirty pkg/errors.
type stackTracer interface {
StackTrace() errors.StackTrace
}
func (e *PkgErrorEntry) WithError(err error) *logrus.Entry {
out := e.Entry
common := func(pError stackTracer) {
st := pError.StackTrace()
depth := 3
if e.Depth != 0 {
depth = e.Depth
}
valued := fmt.Sprintf("%+v", st[0:depth])
valued = strings.Replace(valued, "\t", "", -1)
stack := strings.Split(valued, "\n")
out = out.WithField("stack", stack[2:])
}
if err2, ok := err.(stackTracer); ok {
common(err2)
}
if err2, ok := errors.Cause(err).(stackTracer); ok {
common(err2)
}
return out.WithError(err)
}
func someWhereElse() error {
return unwrappedErrors.New("Ouch")
}
func level1() error {
return level2()
}
func level2() error {
return errors.WithStack(unwrappedErrors.New("All wrapped up"))
}
func main() {
baseLog := logrus.New()
baseLog.SetFormatter(&logrus.JSONFormatter{})
errorHandling := PkgErrorEntry{Entry: baseLog.WithField("callerid", "1000")}
errorHandling.Info("Hello")
err := errors.New("Hi")
errorHandling.WithError(err).Error("That should have a stack.")
err = someWhereElse()
errorHandling.WithError(err).Info("Less painful error")
err = level1()
errorHandling.WithError(err).Warn("Should have multiple layers of stack")
}
A Gopher-ish way
See https://www.reddit.com/r/golang/comments/ajby88/how_to_get_stack_traces_in_logrus/ for more detail.
Ben Johnson wrote about making errors part of your domain. An abbreviated version is that you should put tracer attributes onto a custom error. When code directly under your control errors or when an error from a 3rd party library occurs, the code immediately dealing with the error should put a unique value into the custom error. This value will print as part of the custom error's Error() string implementation.
When developers get the log file, they will be able to grep the code base for that unique value. Ben says "Finally, we need to be able to provide all this information plus a logical stack trace to our operator so they can debug issues. Go already provides a simple method, error.Error(), to print error information so we can utilize that."
Here's Ben's example
// attachRole inserts a role record for a user in the database
func (s *UserService) attachRole(ctx context.Context, id int, role string) error {
const op = "attachRole"
if _, err := s.db.Exec(`INSERT roles...`); err != nil {
return &myapp.Error{Op: op, Err: err}
}
return nil
}
An issue I have with the grep-able code is that it's easy for the value to diverge from the original context. For example, say the name of the function was changed from attachRole to something else and the function was longer. It possible that the op value can diverge from the function name. Regardless, this appears to satisfy the general need of tracing to a problem, while treating errors a first class citizens.
Go2 might throw a curve at this into more the Java-ish response. Stay tuned.
https://go.googlesource.com/proposal/+/refs/changes/97/159497/3/design/XXXXX-error-values.md
Use custom hook to extract stacktrace
import (
"fmt"
"github.com/pkg/errors"
"github.com/sirupsen/logrus"
)
type StacktraceHook struct {
}
func (h *StacktraceHook) Levels() []logrus.Level {
return logrus.AllLevels
}
func (h *StacktraceHook) Fire(e *logrus.Entry) error {
if v, found := e.Data[logrus.ErrorKey]; found {
if err, iserr := v.(error); iserr {
type stackTracer interface {
StackTrace() errors.StackTrace
}
if st, isst := err.(stackTracer); isst {
stack := fmt.Sprintf("%+v", st.StackTrace())
e.Data["stacktrace"] = stack
}
}
}
return nil
}
func main() {
logrus.SetFormatter(&logrus.TextFormatter{DisableQuote: true})
logrus.AddHook(&StacktraceHook{})
logrus.WithError(errors.New("Foo")).Error("Wrong")
}
Output
time=2009-11-10T23:00:00Z level=error msg=Wrong error=Foo stacktrace=
main.main
/tmp/sandbox1710078453/prog.go:36
runtime.main
/usr/local/go-faketime/src/runtime/proc.go:250
runtime.goexit
/usr/local/go-faketime/src/runtime/asm_amd64.s:1594
I'm coming at Golang from a Python background and I am trying to wrap my head around various new concepts.
One thing I have come across is this function in net.go:
func (c *conn) ok() bool { return c != nil && c.fd != nil }
This function is called by multiple net.go methods, e.g. conn.Read:
// Read implements the Conn Read method.
func (c *conn) Read(b []byte) (int, error) {
if !c.ok() {
return 0, syscall.EINVAL
}
I am trying to understand how the ok() method can be called on conn, despite the fact that ok() does not appear to be an interface of conn.
Certainly I do not seem to be able to call ok() from my client code:
func main() {
conn, err := net.Dial("tcp", "www.reddit.com:80")
if err != nil {
os.Exit(-1)
}
fmt.Println(&conn.ok())
}
Output:
./server.go:14:22: conn.ok undefined (type net.Conn has no field or method ok)
Any pointers appreciated...
From Go document :
An identifier may be exported to permit access to it from another
package. An identifier is exported if the first character of the
identifier's name is a Unicode upper case letter
So , ok function is not exported and you can't access it outside of net package.
Go does not use public/private keywords for visibility of an identifier. If the initial character is an upper case letter, the identifier is exported(public); otherwise it is not:
upper case initial letter: Name is visible to clients of package
otherwise: name (or _Name) is not visible to clients of package
There is no field or method like ok in net.Conn that what the error says and that is correct.
when you try to read and write into the conn , you would get err and number of bytes read or write it into the connection.
Consider the following example program which is primitive stack implementation in Go:
package main
import "fmt"
import "errors"
const MAX_SIZE = 10
var a [10]int
var top int = -1
func main() {
printStack()
push(1)
printStack()
push(23)
printStack()
pop()
push(2)
push(24)
push(56)
push(87)
push(97)
push(47)
push(37)
push(31)
push(69)
printStack()
push(75)
println("Top element is", getTop())
}
func push(x int) (int, error) {
if top >= (MAX_SIZE - 1) {
return 0, errors.New("Error: Prevented Stackoverflow. Stack full")
}
top += 1
a[top] = x
return 0, nil
}
func pop() {
top -= 1
}
func getTop() int {
return a[top]
}
func printStack() {
fmt.Println(top+1, "Stack: ", a, "Top", top)
}
Now, I read Error handling and Go & it seems the above way of returning multiple values is the way to handle errors in go.
But what I don't understand is that does do gophers check of errors on every statement? Coming from other languages this concept it bit hard for me to grok. So the question is
What is the idiomatic way of handling errors in above problem?
Is considered a bad practice if I don't check for errors? if yes, then I am supposed to check the output of push() everytime I call it?
Basically what I want to know if how would a experienced gopher do error handling in the above program?
Play URL: https://play.golang.org/p/_AnZLvY-28
[Update]
Added a real-world program where I fetch data from database & output to json. http://play.golang.org/p/dDApCYprjU
Yes, the idiomatic way to produce errors is to return multiple values. The idiomatic way to handle errors is this:
val, err := myFunc()
if err != nil {
// do something with err
}
// do something with val
At the end of the day it's a judgement call, but it's almost always good practice to handle errors. The code you're writing is also a bit unusual, you normally don't have 10 calls to the same function back-to-back, so the verbose error handling you'd have to do is more a result of the original code. For instance, you could use a loop:
for _, num := range []int{2, 24, 56, 87, 97, 47, 37, 31, 69} {
_, err := push(num)
if err != nil {
panic(err)
}
}
You have some other things that are more problematic than not handling the push errors though. One minor thing is there is no reason for push to always return 0, why not only have it return an error, instead of an int and an error? The bigger problem is that pop keeps decrementing top, and getTop just accesses a[top], so you can easily get a runtime panic if top becomes negative from having popped too much. You probably want some error handling or other safeguards in your pop and getTop functions.
This is a standard way of handling the errors in go-lang. There are two options that can follow after an error took place.
1.log the error and entirely quit the program
log the error and do some relevant task
Option one example :
func funcX() {
rtnval, err := funcY()
if err != nil {
fmt.Println("error: ", err)
os.Exit(1)
}
//rtnval
}
I'm working on learning Go as my first compiled language (coming from php/python). My first project was a small POST hook listener for Bitbucket, which fetches and then checks out a Git repository via os/exec. I'm now trying to replace the os/exec calls with git2go. I'm running into a snag with the authentication, though. I have the following code:
package main
import (
git "github.com/libgit2/git2go"
"log"
)
func main() {
_, Cred := git.NewCredSshKey("git","~/.ssh/id_rsa.pub","~/.ssh/id_rsa","")
log.Println(Cred.Type())
gitH,err := git.OpenRepository(".")
if (err != nil) {
log.Fatalln(err)
}
remotes,err := gitH.ListRemotes()
if (err != nil) {
log.Fatalln(err)
}
log.Println(remotes)
origin,err := gitH.LoadRemote("origin")
if (err != nil) {
log.Fatalln(err)
}
err = origin.Fetch(nil,"")
if (err != nil) {
log.Fatalln(err)
}
}
When I run this I get authentication required but no callback set.
Looking at the docs, it looks like I need to add a call to origin.SetCallbacks() which expects a RemoteCallbacks struct. RemoteCallbacks has the function CredentialsCallback which returns an int and a Cred pointer. Since NewCredSshKey returns the same values, I tried adding the following:
var cb git.RemoteCallbacks
cb.CredentialsCallback = git.NewCredSshKey("git","~/.ssh/id_rsa.pub","~/.ssh/id_rsa","")
origin.SetCallbacks(cb)
which gives the errors multiple-value git.NewCredSshKey() in single-value context and
cannot use cb (type git.RemoteCallbacks) as type *git.RemoteCallbacks in function argument.
I think I'm completely misunderstanding how this works, and I haven't been able to find any examples using this library. Tips or pointers to some examples would be much appreciated.
A Couple of things:
CredentialsCallback needs to be set to a function that matches it's signature, not the output of such a function. However, the signature for NewCredSshKey isn't correct in the first place, only its return values match. The correct signature is:
func(url string, username_from_url string, allowed_types CredType) (int, *Cred)
The second error cannot use cb (type git.RemoteCallbacks) as type *git.RemoteCallbacks is because you need a pointer to a RemoteCallbacks.
Either declare and initialize it as a pointer:
cb := &git.RemoteCallbacks{}
// or alternatively
// cb := new(git.RemoteCallbacks)
or take the address of when passing it as an argument:
origin.SetCallbacks(&cb)