Go: Call a function passing pointer to array - go

I have spent a bit of time attempting to do this, and I think I need a global array (not slice), and I want to pass it as a pointer, not by value. The function receiving the pointer needs to test for nil, and if nil, read the array from disk, using eg: "baForm, oOserr = ioutil.ReadFile(sFormName)". The calling function may pass either a global array or an array local to the calling function which I presume will be garbage collected.
The reason for doing this is that I want a standard function to read the forms from disk, and often-used forms are stored globally. Despite whether some may think there is a better way, I still want to know how to achieve this ie: a) have global or local array, b) not pass by value, c) The global arrays will be read only once from disk and the local arrays will be read each time the function is called. TIA.

From reading your description, I can't see why passing a pointer to an array is any way better than passing a slice -- but it's up to you.
You can pass a pointer just like you do in C -- attach an asterisk (*) to the declaration, and perpend an ampersand (&) to the value when you call the function.
Just remember that in Go, the array size is part of its type. This means that your function declaration will have the array size embedded into it, so you can't call the function using an array of any different size. This reason alone is generally enough to warrant using a slice instead of an array.

Here's a sample program that maintains a dynamic forms buffer based on use count. If the ReadForm function finds a form, it returns the address of the form and a nil error.
package main
import (
"fmt"
"io/ioutil"
"math"
"os"
"sync"
)
type Form struct {
Name string
useCount int64
Data []byte
}
// The capacity of the forms buffer.
const formsCap = 2
// The forms buffer.
var (
forms = make(map[string]*Form, formsCap)
formsLock sync.RWMutex
)
func ReadForm(name string) (form *Form, err os.Error) {
formsLock.RLock()
form, ok := forms[name]
formsLock.RUnlock()
if !ok {
form = &Form{name, 0, nil}
}
if form.Data == nil {
data, err := ioutil.ReadFile(name + ".form")
if err != nil {
return nil, err
}
form = &Form{name, 0, data}
formsLock.Lock()
if len(forms) >= formsCap {
minForm := &Form{useCount: math.MaxInt64}
for i, f := range forms {
if f.useCount < minForm.useCount {
minForm = f
}
}
minform.Data = nil
}
forms[name] = form
formsLock.Unlock()
}
form.useCount++
return form, nil
}
func main() {
// form files are named name.form e.g. form1.form
for _, name := range []string{"form1", "form2", "form3"} {
f, err := ReadForm(name)
if err != nil {
fmt.Println(err)
} else {
fmt.Println(string(f.Data))
}
}
fmt.Println(len(forms), forms)
}
EDIT: Map operations are not atomic. Revise the sample program to use a mutex for concurrent access to the forms map.

Related

How to use a custom type of Slice in a Receiver Method

I am new to Golang, and I know that a Receiver Method should be called by reference not by value. But the weird thing is that after naming a slice type, this is not working!!
I named a set of processes as Group, where Process is a struct of whatever content.
type Group []Process
A Group has a set of related methods in which I use them commonly through receiver methods. This Search function is just a simple sample.
func (G *Group) Search(addr string) (*Process, error) {
for _, ps := range *G {
if ps.Address == addr {
return &ps, nil
}
}
return nil, errors.New("Group: Address Not found")
}
Then I call it by
p, err := group.Search(address)
p can be changed normally. But the changes are affecting only p (which is not really pointing to a slice element of the Group), not the slice element in Group.
Question: How to use a custom type of Slice in a Receiver Method.
There's nothing wrong with how you're using a slice type as a receiver. There is something wrong with your loop and how you're returning a pointer. Change
for _, ps := range *G {
if ps.Address == addr {
return &ps, nil
}
}
into
for i, ps := range *G {
if ps.Address == addr {
return &(*G)[i], nil
}
}
and it should work — you'll be returning a pointer into the slice, instead of a pointer to a local variable.

Error trying to use an empty interface parameter for returning data

I'm trying to write a wrap around a function that uses an interface{} parameter to return data, by adding cache.
My problem is that once I have a valid interface{} I don't know how to assign it to be returned in the parameter. The wrapped call is (github.Client) .Do in github API client and the problem hit me when I tried to add caching with go-cache
This somewhat my function
func (c *cachedClient) requestAPI(url string, v interface{}) error {
x, found := c.cache.Get(url)
if found { // Found in cache code
log.Printf("cached: %v", x)
v = x // HERE: this do not work. x contains a valid copy of what I want but how do I store it in v?
return nil
}
req, _ := c.githubClient.NewRequest("GET", url, nil) // not found I cache, request it
res, err := c.githubClient.Do(*c.context, req, v)
if err != nil {
return err
}
if res.StatusCode != 200 {
return fmt.Errorf("Error Getting %v: %v", url, res.Status)
}
c.cache.Add(url, v, cache.DefaultExpiration) // store in cache
return nil // once here v works as expected and contain a valid item
}
It fails when has to return a cached value when I try to use it like this:
// Some init code c is a cachedClient
i := new(github.Issue)
c.requestAPI(anAPIValidURLforAnIssue, i)
log.Printf("%+v", i) // var i correctly contains an issue from the github api
o := new(github.Issue)
c.requestAPI(anAPIValidURLforAnIssue, o)
log.Printf("%+v", o) // var o should have been get from the cache but here is empty
So basically my problem is that when I correctly recover a cached item it is good but I can not store it in the parameter meant to be used to store it. I can not work with subclasses because the call I'm wrapping is using an interface{} already. And I can not move it to return values because you can't return a generic interface. How do I make the interface{} x be stored in v to have it available outside?
To archive what you want you need to use a bit of reflection magic.
Please try to replace v = x with next code snippet:
reflect.ValueOf(v).Elem().Set(reflect.ValueOf(x).Elem())
Note from OP: I had to add the last .Elem() to make this work.
NOTE: in the call of the requestAPI method you should use a pointer to the value:
let's say the cached value is of type int. Then you should call requestAPI like:
var dst int // destination of the cached value or a newly retrieved result
cc.requestAPI(url, &dst)
With certain assumptions like you are storing json data in your cache below is how I will try. Errors not handled.
package main
import (
"encoding/json"
"fmt"
)
type Data struct {
Name string
}
func main() {
var d Data
requestAPI(&d)
fmt.Println(d)
}
func requestAPI(v interface{}) {
var cache_res interface{} = []byte("{\"Name\":\"CCC\"}")
//assume you got cache_res from cache
x, _ := cache_res.([]byte)
_ = json.Unmarshal(x, &v)
}
Actually above is what githubClient.Do is also doing. It checks whether v satisfies io.Writer interface, if yes write data to v. If not then it unmarshals json into v as shown above. So same can be done from cache.
Check here:
https://github.com/google/go-github/blob/v32.1.0/github/github.go#L586
If the cache object is specific then below can be used. You don't deal with empty interface{} because you should be able to pass your specific type to c.githubClient.Do as v. Since it uses json package, it will detect the type information and accordingly fill the values into it.
Lets say you store type Data struct
In below code other details eliminated like condition checking whether to cache & error handling
package main
import (
"fmt"
)
type Data struct {
Name string
}
func main() {
var d Data
requestAPI(&d)
fmt.Println(d)
}
func requestAPI(v *Data) {
var cache_res interface{} = Data{"CCC"}
//assume you got cache_res from cache
x, _ := cache_res.(Data)
*v = x
//in case you did not find it in cache then githubClient.Do should unmarshal
//contents of response body into v *Data if Data fields match that of json
//res, err := c.githubClient.Do(*c.context, req, v)
}

How to range over slice of a custom type

I'm trying to write in Go custom cache for Google DataStore (more precisely - a wrapper around one of existing cache libraries). At cache initialisation, it should accept any custom type of struct (with appropriately-defined datastore fields), which then would be the basis for all items stored. The idea is that cache can be created/initialised for various types which reflect the structure of a particular DataStore entry (CustomEntry)
Approach 1 - store reflect.Type and use it. Problem encountered - can't iterate over a slice of a custom type
type CustomEntry struct {
Data struct {
name string `datastore:"name,noindex"`
address []string `datastore:"address,noindex"`
} `datastore:"data,noindex"`
}
func (cache *MyCache) CacheData(dataQuery string, dataType reflect.Type) {
slice := reflect.MakeSlice(reflect.SliceOf(dataType), 10, 10)
if keys, err := DataStoreClient.GetAll(cache.ctx, datastore.NewQuery(dataQuery), &slice); err != nil {
//handle error
} else {
for i, dataEntry:= range slice {
// ERROR: Cannot range over 'slice' (type Value)
cache.Set(keys[i].Name, dataEntry)
}
}
//usage: Cache.CacheData("Person", reflect.TypeOf(CustomEntry{})
Approach 2 - accept an array of interfaces as arguments. Problem encountered = []CustomEntry is not []interface{}
func (cache *MyCache) CacheData(dataQuery string, dataType []interface{}) {
if keys, err := DataStoreClient.GetAll(cache.ctx, datastore.NewQuery(dataQuery), &dataType); err != nil {
//handle error
} else {
for i, dataEntry:= range slice {
// this seems to work fine
cache.Set(keys[i].Name, dataEntry)
}
}
//usage:
var dataType []CustomEntry
Cache.CacheData("Person", data)
// ERROR: Cannot use 'data' (type []CustomEntry) as type []interface{}
Any suggestions would be highly appreciated.
I have found a solution and thought it might be worth sharing in case anyone else has a similar problem.
The easiest way is to initiate a slice of structs which the DataStore is expected to receive, and then to pass a pointer to it as an argument (interface{}) into the desired function. DataStore, similarly to a few unmarshaling functions (I have tried with JSON package) will be able to successfully append the data to it.
Trying to dynamically create the slice within the function, given a certain Type, which would be then accepted by a function (such as DataStore client) might be quite difficult (I have not managed to find a way to do it). Similarly, passing a slice of interfaces (to allow for easy iteration) only complicates things.
Secondly, in order to iterate over the data (e.g. to store it in cache), it is necessary to:
(1) retrieve the underlying value of the interface (i.e. the pointer itself) - this can be achieved using reflect.ValueOf(pointerInterface),
(2) dereference the pointer so that we obtain access to the underlying, iterable slice of structs - this can be done by invoking .Elem(),
(3) iterate over the underlying slice using .Index(i) method (range will not accept an interface, even if the underlying type is iterable).
Naturally, adding a number of switch-case statements might be appropriate to ensure that any errors are caught rather than cause a runtime panic.
Hence the following code provides a working solution to the above problem:
In main:
var data []customEntry
c.CacheData("Person",&data)
And the function itself:
func (cache *MyCache) CacheData(dataQuery string, data interface{}) error {
if keys, err := DataStoreClient.GetAll(cache.ctx, datastore.NewQuery(dataQuery), data); err != nil {
return err
} else {
s := reflect.ValueOf(data).Elem()
for i := 0; i < s.Len(); i++ {
cache.Set(keys[i].Name, s.Index(i), 1)
}
}
}

Reading ETW providers using go

I'm trying to access the EnumerateTraceGuids function from Advapi32.dll in go.
I'm at the very early stage and still trying to decipher what is it that I must do. I have the following code that keeps giving me Error: 87, meaning ERROR_INVALID_PARAMETER.
I've used this file as a starting point though it's only writing and not reading :
https://github.com/moby/moby/blob/master/daemon/logger/etwlogs/etwlogs_windows.go
Official documentation for the function I'm trying to call is here :
https://msdn.microsoft.com/en-us/library/windows/desktop/aa363713(v=vs.85).aspx
It requires GuidPropertiesArray [in, out] An array of pointers to TRACE_GUID_PROPERTIES structures. This structure is the following (https://msdn.microsoft.com/en-us/library/windows/desktop/aa364143(v=vs.85).aspx)
typedef struct _TRACE_GUID_PROPERTIES {
GUID Guid;
ULONG GuidType;
ULONG LoggerId;
ULONG EnableLevel;
ULONG EnableFlags;
BOOLEAN IsEnable;
} TRACE_GUID_PROPERTIES, *PTRACE_GUID_PROPERTIES;
I have the following code to try and do this :
package main
import (
"errors"
"fmt"
"syscall"
"unsafe"
"github.com/sirupsen/logrus"
"golang.org/x/sys/windows"
)
const (
win32CallSuccess = 0
MaxProv = 50
nbProviders = 50
)
var (
modAdvapi32 = windows.NewLazySystemDLL("Advapi32.dll")
procEnumerateTraceGuids = modAdvapi32.NewProc("EnumerateTraceGuids")
)
type ulong int32
type TRACE_GUID_PROPERTIES struct {
Guid syscall.GUID
GuidType ulong
LoggerId ulong
EnableLevel ulong
EnableFlags ulong
IsEnable bool
}
func callEnumerateTraceGuids() error {
GuidPropertiesArray:= make([]TRACE_GUID_PROPERTIES, 1)
ptr := &GuidPropertiesArray[0]
ret, _, _ := procEnumerateTraceGuids.Call(uintptr(unsafe.Pointer(&ptr)), MaxProv, nbProviders)
if ret != win32CallSuccess {
errorMessage := fmt.Sprintf("Failed to register ETW provider. Error: %d", ret)
logrus.Error(errorMessage)
return errors.New(errorMessage)
}
return nil
}
func main() {
callEnumerateTraceGuids()
}
At this point I'm not sure what is it that I must do. I've tried a lot of variation of initializing the array without success.
Hoping someone can point me in the right direction.
Thanks !
Edit : Changed code based on comments but still getting the same error.
PS : This is my first time posting to stackoverflow and I've already been told that I'm lazy less than 12 hours after posting my question (yay!) so not sure I'm asking this right...I am not too familiar with go and never called windows DLL from go before and since I keep hitting that ERROR_INVALID_PARAMETER I thought of reaching out to try and pass this first wall to be able to grasp some concepts at the same time. Hope this helps understands my request (ie. I come in peace).
OK, I had a bit of free time and an access to a Windows XP box,
so I've decided to dust off my Windows programming skills
and wrote a working solution:
package main
import (
"golang.org/x/sys/windows"
"log"
"syscall"
"unsafe"
)
var (
modAdvapi32 = windows.NewLazySystemDLL("advapi32")
procEnumerateTraceGuids = modAdvapi32.NewProc("EnumerateTraceGuids")
)
type traceGuidProperties struct {
guid syscall.GUID
guidType uint32
loggerId uint32
enableLevel uint32
enableFlags uint32
isEnable uint32
}
func enumerateTraceGuids(ptr **traceGuidProperties, count uint32, out *uint32) error {
rc, _, _ := procEnumerateTraceGuids.Call(uintptr(unsafe.Pointer(ptr)),
uintptr(count), uintptr(unsafe.Pointer(out)))
if rc != 0 {
return syscall.Errno(rc)
}
return nil
}
func enumTraceGuids() ([]*traceGuidProperties, error) {
var errMoreData = syscall.Errno(234)
var (
dummyProps traceGuidProperties
dummyPtr = &dummyProps
count uint32
)
err := enumerateTraceGuids(&dummyPtr, 0, &count)
if err != errMoreData {
return nil, err
}
items := make([]*traceGuidProperties, count)
for i := range items {
items[i] = new(traceGuidProperties)
}
for {
err = enumerateTraceGuids(&items[0], count, &count)
if err == nil {
break
}
if err != errMoreData {
return nil, err
}
for i := 0; i < int(count)-len(items); i++ {
items = append(items, new(traceGuidProperties))
}
}
return items[:count], nil
}
func main() {
log.SetFlags(0)
data, err := enumTraceGuids()
if err != nil {
log.Fatal(err)
}
log.Printf("len(data)=%d\n", len(data))
for i := range data {
log.Println(*(data[i]))
}
}
The key points:
I was wrong when I told you that
«you … should allocate an array of structs (not pointers)»—in fact
EnumerateTraceGuids indeed expects an array of pointers.
As hinted here,
there are two subtleties with how EnumerateTraceGuids works:
Contrary to what its documentation states,
it actually supports being called with its PropertyArrayCount
parameter set to 0, in which case it's expected to return ERROR_MORE_DATA
while having set GuidCount to the number of elements of the input
array required for the (next) call to complete successfully.
IOW, that way we know how many trace GUIDs the system currently
"knows about".
Still, even in this case, the function performs validity check
on the input array (see below).
As it turns out, the function expects an array of pointers to
TRACE_GUID_PROPERTIES blocks allocated by you.
In other words, if it says you it knows about 10 trace GUIDs,
you have to allocate 10 values of type TRACE_GUID_PROPERTIES,
then make an array of 10 pointers to those values and pass a pointer
to the 1st element of that array to the function.
Notice that there's an inherent race between changes occuring
in the system (those traces added or removed for any number of reasons)
and the calls to EnumerateTraceGuids.
This means if the first call to this function told you it "knows"
about 10 trace GUIDs, on the next call it may turn out
there's already 20 trace GUIDs, or 5 GUIDs
(or any other number of them FWIW).
So we account for both of these possibilities in the following way:
First we do a call with a pointer to a single (but valid)
TRACE_GUID_PROPERTIES value, allocated statically
(hence the function "sees" what looks like an array of a single element),
while telling the function the input "array" has zero elements.
We expect the function to fail with ERROR_MORE_DATA
and put the actual number of trace GUIDs it "knows" about into the variable
we've supplied it a pointer to.
We allocate that much TRACE_GUID_PROPERTIES memory blocks
the function indicated on the first call.
For this, we use the new() built-in function which behaves somewhat
like malloc() in the standard C library—it allocates the memory for
a value of the specified type and returns a pointer to the allocated
memory block.
We create an array of pointers to these allocated memory blocks
and call EnumerateTraceGuids again.
If it succeeds, we handle the possibility it returned less
elements than we've allocated, and reslice our slice.
If it fails with ERROR_MORE_DATA, we extend our slice with
whatever the number of elements is needed (allocating memory for their
TRACE_GUID_PROPERTIES blocks first), and try calling the function again.
The "magic number" 234 is the actual code for the ERROR_MORE_DATA value.
Sorry for the initial confusion.

Go: Assign multiple return value function to new and old variable

In go there are functions which return two values or more values, commonly one is an error. Suppose that I want to store the first return value into an already initialized variable, but I would like to initialize the variable to contain the error inline. Is there a way to do this?
For example, say I had this code
var a int
//This code doesn't compile because err doesn't exist
a, err = SomeFuncWithTwoReturnValues()
//This code doesn't compile either
a, err := SomeFuncWithTwoReturnValues()
I know you could do this, but I was hoping there was a way to do it all inline
var a int
var err error
a, err = SomeFuncWithTwoReturnValues()
or
a, err := SomeFuncWithTwoReturnValues()
EDIT: The code above actually compiles, so I looked back at my code to drill down more and have created a quick sample that actually replicates the problem (not just in my mind...).
package main
func myfunc() (int, int) {
return 1, 1
}
func main() {
a := make([]int, 1)
a[0], b := myfunc()
a[0] = b
}
Compiler says main.go|9| non-name a[0] on left side of :=. If I make it = instead of := though then b is never created. I get the feeling that there is not shorthand way to do it though.
As you've mentioned in the comments, you'll need to use the = operator in order to assign to a variable you've already declared. The := operator is used to simultaneously declare and assign a variable. The two are the same:
var x int
x = 5
//is the same as
x := 5
This solution will at least compile:
package main
func myfunc() (int, int) {
return 1, 1
}
func main() {
var b int
a := make([]int, 1)
a[0], b = myfunc()
a[0] = b
}
To answer your question, I don't think there is a way to simultaneously use an undeclared and a declared variable when returning multiple values. That would be trying to use two different operators simultaneously.
Edit: just saw your example from the code that compiles, so it appears you're already familiar with go's assignment operators. I'll leave the example up anyway.
Golang is not a very consistent language. This is a good example. At the beginning I was confused and it would be much simpler if they would always allow the := operator. The compiler is smart enough to detect already declared variables:
package main
import "fmt"
func testFunc() (int,error) {
return 42,fmt.Errorf("Test Error")
}
func main() {
number1,err := testFunc() // OK
number2,err := testFunc() // OK, even if err is already defined
number1,err = testFunc() // OK
// number1,err := testFunc() // ERROR: no new variables on left side of :=
fmt.Println(number1,number2,err)
}
Playground Link: https://play.golang.org/p/eZVB-kG6RtX
It's not consistent, because golang allows you to use := for already declared variables if you assign to them while also introducing a new variable. So the compiler can detect that variables already exists and skip their declaration. But the golang developers decided to allow that only if you introduce at least one new value. The last example shows that.
I ran into this situation like this:
package main
import "os"
func main() {
var cache struct { dir string }
// undefined: err
cache.dir, err = os.UserCacheDir()
// non-name cache.dir on left side of :=
cache.dir, err := os.UserCacheDir()
if err != nil {
panic(err)
}
println(cache.dir)
}
as you discovered, this issue does not have a clean solution. You can declare
an extra variable:
dir, err := os.UserCacheDir()
if err != nil {
panic(err)
}
cache := userCache{dir}
Or, while more verbose, you can declare the error beforehand. This can save
memory, as Go does not use a Rust ownership model:
var (
cache struct { dir string }
err error
)
cache.dir, err = os.UserCacheDir()
As mention in the spec, while using:=, if one of the variables is new, then the old one will just be assigned with the new data.
Unlike regular variable declarations, a short variable declaration may redeclare variables provided they were originally declared earlier in the same block (or the parameter lists if the block is the function body) with the same type, and at least one of the non-blank variables is new. As a consequence, redeclaration can only appear in a multi-variable short declaration. Redeclaration does not introduce a new variable; it just assigns a new value to the original.
field1, offset := nextField(str, 0)
field2, offset := nextField(str, offset) // redeclares offset
As mentioned by the other answers you cannot use assignment and declaration in the same return statement. You have to use either.
However I guess the main reason for your question is cleaning up the code so you don't have to declare an extra err variable above the method or function statement.
You can solve this in two ways:
Declare a global var err error variable and use it in the assignment:
var err error
func MyFunc(someInput string) {
var a int
a, err = someOtherFunction()
}
If your method or function returns an error you can use the declared return variable
func MyFunc(someInput string) (err error) {
var a int
a, err = someOtherFunction()
return
}
I mainly have the problem in methods when I want to assign something to a struct member, e.g.:
type MyStruct struct {
so string
}
func (m *MyStruct) SomeMethod() (err error) {
m.so, err = SomeFunction()
// handle error and continue or return it
return
}

Resources