Conversion between pointers should be made using unsafe.Pointer() and uintptr.
I am writing an interpreter using Go. This is very simple fragment using an EID struct to carry pairs (type,values) between different sections of native code. This code is surprising because the same print statement gets two different values (see the Foo() method). The object is "encapsulated" into an EID and transformed back to an object.
The code compiles but the result is deeply broken.
If you run this you get:
~/go% go run testBug.go
create object 0xc000068e28 with class 0xc00000c060
here is y:0xc000068e28, y.Isa: 0xc00000c060"
here is y:0xc000068e28, y.Isa: 0x2c
package main
import (
"fmt"
"unsafe"
)
type EID struct {
SORT *Class
VAL uintptr
}
// access to VAL
func OBJ(x EID) *Anything { return (*Anything)(unsafe.Pointer(x.VAL)) }
func INT(x EID) int { return (int)((uintptr)(unsafe.Pointer(x.VAL))) }
// useful utility get the pointer as a uintptr
func (x *Anything) Uip() uintptr { return uintptr(unsafe.Pointer(x)) }
type Anything struct {
Isa *Class
}
func (x *Anything) Id() *Anything { return x }
type Object struct {
Anything
name string
}
type Class struct {
Object
Super *Class
}
type Integer struct {
Anything
Value int
}
func MakeObject(c *Class) *Anything {
o := new(Object)
o.Isa = c
return o.Id()
}
// this is the surprising example - EID is passed but the content is damaged
func (c *Class) Foo() EID {
x := c.Bar()
y := OBJ(x)
z := y.Isa
fmt.Printf("here is y:%p, y.Isa: %p\n", y, z)
fmt.Printf("here is y:%p, y.Isa: %p\n", y, y.Isa) // this produces a different value !
return x
}
func (c *Class) Bar() EID {
UU := EID{c, MakeObject(c).Uip()}
fmt.Printf("create object %p with class %p\n", OBJ(UU), OBJ(UU).Isa)
return UU
}
var aClass *Class
var aInteger *Class
func main() {
aClass := new(Class)
aClass.Isa = aClass
aClass.Foo()
}
Clearly the uintptr to pointer has to be local and cannot happen in two different places (Foo() and Bar() here). I have found a workaround but I curious about this strange behavior.
When you store a pointer (of any concrete type or even of type unsafe.Pointer) into a uintptr, this hides the pointer-ness from Go's garbage collector. Go is therefore free to GC the underlying object if there is no other pointer to it.
When you convert a uintptr to unsafe.Pointer, the object, a pointer to which the value stored in the uintptr converts, needs to exist. If it's been GC'ed, it no longer exists. Hence the "safe" way to take some pointer value p of any type *T and store it in a uintptr is to store it instead in unsafe.Pointer. The unsafe.Pointer object is visible to Go's garbage collector, as a pointer, so this keeps the actual object alive.
You'll see this pattern in some of the Go internal software:
// need to keep the pointer alive while we make a syscall
p := unsafe.Pointer(foo)
ret := syscall.SyscallN(..., uintptr(foo), ...)
The apparently pointless creation of local variable p serves to protect the underlying object from being GC'ed while the OS system call reads its bytes. (Note that this is being overly chummy with the compiler since the assignment to p appears to be dead code here. Perhaps the internal software is fancier than this, and/or they're using //go:... comments as well.)
This same pattern works in the Go playground if I take your not-quite-minimal reproducible example and make the obvious minimal changes to it. Whether that's sufficient (and precisely how you'd like to use this same concept in your interpreter) is another question entirely, but see playground copy. Note: I had to add one closing brace to your program but after that it exhibited the same behavior you saw; here's that version. It draws two warnings from go vet about misuse of unsafe.Pointer, which my updated version doesn't.
I'm currently learning Go and am following a tutorial about how to use Go with Stripe. There is this example code:
package main
import (
"fmt"
"github.com/stripe/stripe-go"
"github.com/stripe-go/customer"
)
func main() {
sc := &client.API{}
sc.Init("somekey")
c, _ := sc.Customers.Get("customerid", nil)
// ...
}
What is/could be the reason that sc stores the pointer to the struct and not the struct itself?
[To supplement the comment you received]
While in this case with the small code sample it's hard to say, in most scenarios you'll see non-trivial types passed around by pointer to enable modification. As an anti-example, consider this code which uses a variable of a struct type by value:
type S struct {
ID int
}
func (s S) UpdateID(i int) {
s.ID = i
}
func main() {
s := S{}
s.UpdateID(99)
fmt.Println(s.ID)
}
What do you think this will print? It will print 0, because methods with value receivers cannot modify the underlying type.
There's much information about this in Go - read about pointers, and about how methods should be written. This is a good reference: https://golang.org/doc/faq#methods_on_values_or_pointers, and also https://golang.org/doc/effective_go#pointers_vs_values
Back to your example: typically non-trivial types such as those representing a "client" for some services will be using pointers because method calls on such types should be able to modify the types themselves.
I'm trying to call SHGetImageList in Go in order to extract icons from .EXE files. The problem is that I don't know how to create/pass an "IImageList2 interface" in Go, which is something SHGetImageList requires.
I've tried miscellaneous things for a few hours, but it all results in the same E_NOINTERFACE error. Basically they're all "shots in the dark" ([]byte array to see if I could receive any "data" at all, an actual interface{} in Go containing the same functions as the IImagelist2 interface defined by MSDN, etc). If it is of any relevance, I do have a working version of this in C# using something along the lines of http://www.pinvoke.net/default.aspx/shell32.shgetimagelist, but I simply have no real clue on how to "translate" that to Go. Any help would be much appreciated.
Example Go code below, with some info and links to MSDN in the comments.
package main
import (
"fmt"
"syscall"
"unsafe"
)
var (
shell32 = syscall.MustLoadDLL("shell32.dll")
// https://msdn.microsoft.com/en-us/library/windows/desktop/bb762179(v=vs.85).aspx
procSHGetFileInfo = shell32.MustFindProc("SHGetFileInfoW")
//https://msdn.microsoft.com/en-us/library/windows/desktop/bb762185(v=vs.85).aspx
procSHGetImageList = shell32.MustFindProc("SHGetImageList")
)
func main() {
someExeFile := `c:\windows\explorer.exe`
iconIndex := GetIconIndex(someExeFile)
// The problem:
HRESULT, _, _ := procSHGetImageList.Call(
uintptr(SHIL_JUMBO),
uintptr(unsafe.Pointer(&IID_IImageList2)),
// I don't know how pass/create an "IImageList interface" in Go,
// or if it's even possible without relying on CGO.
// IImageList interface:
// https://msdn.microsoft.com/en-us/library/windows/desktop/bb761419(v=vs.85).aspx
// Currently there's just a pointer to an empty []byte so that the code will compile.
// HRESULT naturally contains the error code E_NOINTERFACE (2147500034),
// which makes sense seeing as I'm not passing a valid interface.
uintptr(unsafe.Pointer(&[]byte{})),
)
fmt.Println(iconIndex, HRESULT)
}
const SHIL_JUMBO = 0x4
const shGetFileInfoLen = 3
const shGetFileInfoFlags = 16400 //(SysIconIndex|LargeIcon|UseFileAttributes)
// use SHGetFileInfo to get the icon index (only value we care about)
func GetIconIndex(fileName string) int {
buf := make([]uint16, shGetFileInfoLen)
ret, _, _ := procSHGetFileInfo.Call(
uintptr(unsafe.Pointer(syscall.StringToUTF16Ptr(fileName))),
0,
uintptr(unsafe.Pointer(&buf[0])),
shGetFileInfoLen,
shGetFileInfoFlags,
)
if ret != 0 && buf[2] > 0 {
return int(buf[2])
}
return 0
}
// From: "192B9D83-50FC-457B-90A0-2B82A8B5DAE1"
var IID_IImageList2 = &GUID{0x192b9d83, 0x50fc, 0x457b, [8]byte{0x90, 0xa0, 0x2b, 0x82, 0xa8, 0xb5, 0xda, 0xe1}}
// http://msdn.microsoft.com/en-us/library/windows/desktop/aa373931.aspx
type GUID struct {
Data1 uint32
Data2 uint16
Data3 uint16
Data4 [8]byte
}
UPDATE
The problem now is that it's exiting with an error(0xC0000005) when calling the Syscall to get the actual icon pointer.
package main
import (
"fmt"
"syscall"
"unsafe"
)
var (
shell32 = syscall.MustLoadDLL("shell32.dll")
// https://msdn.microsoft.com/en-us/library/windows/desktop/bb762179(v=vs.85).aspx
procSHGetFileInfo = shell32.MustFindProc("SHGetFileInfoW")
//https://msdn.microsoft.com/en-us/library/windows/desktop/bb762185(v=vs.85).aspx
procSHGetImageList = shell32.MustFindProc("SHGetImageList")
ole32 = syscall.MustLoadDLL("ole32.dll")
procCoInitialize = ole32.MustFindProc("CoInitialize")
)
func main() {
someExeFile := `c:\windows\explorer.exe`
procCoInitialize.Call()
iconIndex := GetIconIndex(someExeFile)
var imglist *IImageList
hr, _, _ := procSHGetImageList.Call(
uintptr(SHIL_JUMBO),
uintptr(unsafe.Pointer(&IID_IImageList)),
uintptr(unsafe.Pointer(&imglist)),
)
// These look OK
fmt.Println(iconIndex, hr, imglist.Vtbl.GetIcon)
var hIcon uintptr
// GetIcon: https://msdn.microsoft.com/en-us/library/windows/desktop/bb761463(v=vs.85).aspx
hr, _, _ = syscall.Syscall(imglist.Vtbl.GetIcon,
uintptr(unsafe.Pointer(imglist)),
uintptr(iconIndex),
getIconFlags,
uintptr(unsafe.Pointer(&hIcon)),
)
// Errors: "Process finished with exit code -1073741819 (0xC0000005)"
fmt.Println("hIcon:", hIcon) // Never reaches this
}
// ILD_TRANSPARENT | ILD_IMAGE
const getIconFlags = 0x00000001 | 0x00000020
const SHIL_JUMBO = 0x4
const shGetFileInfoLen = 3
const shGetFileInfoFlags = 16400 //(SysIconIndex|LargeIcon|UseFileAttributes)
// use SHGetFileInfo to get the icon index (only value we care about)
func GetIconIndex(fileName string) int {
buf := make([]uint16, shGetFileInfoLen)
ret, _, _ := procSHGetFileInfo.Call(
uintptr(unsafe.Pointer(syscall.StringToUTF16Ptr(fileName))),
0,
uintptr(unsafe.Pointer(&buf[0])),
shGetFileInfoLen,
shGetFileInfoFlags,
)
if ret != 0 && buf[2] > 0 {
return int(buf[2])
}
return 0
}
// From: "46EB5926-582E-4017-9FDF-E8998DAA0950"
var IID_IImageList = GUID{0x46eb5926, 0x582e, 0x4017, [8]byte{0x9f, 0xdf, 0xe8, 0x99, 0x8d, 0xaa, 0x09, 0x50}}
// http://msdn.microsoft.com/en-us/library/windows/desktop/aa373931.aspx
type GUID struct {
Data1 uint32
Data2 uint16
Data3 uint16
Data4 [8]byte
}
type IImageList struct {
Vtbl *IImageListVtbl
}
type IImageListVtbl struct {
Add uintptr
ReplaceIcon uintptr
SetOverlayImage uintptr
Replace uintptr
AddMasked uintptr
Draw uintptr
Remove uintptr
GetIcon uintptr
GetImageInfo uintptr
Copy uintptr
Merge uintptr
Clone uintptr
GetImageRect uintptr
GetIconSize uintptr
SetIconSize uintptr
GetImageCount uintptr
SetImageCount uintptr
SetBkColor uintptr
GetBkColor uintptr
BeginDrag uintptr
EndDrag uintptr
DragEnter uintptr
DragLeave uintptr
DragMove uintptr
SetDragCursorImage uintptr
DragShowNolock uintptr
GetDragImage uintptr
GetItemFlags uintptr
GetOverlayImage uintptr
}
Oh, I see the actual problem now.
uintptr(unsafe.Pointer(&IID_IImageList2)),
...
var IID_IImageList2 = &GUID{0x192b9d83, 0x50fc, 0x457b, [8]byte{0x90, 0xa0, 0x2b, 0x82, 0xa8, 0xb5, 0xda, 0xe1}}
Your IID_IImageList2 is already a pointer. In your call, you're taking a pointer to that pointer, which means the address is used as the GUID. You should either do
uintptr(unsafe.Pointer(&IID_IImageList2)),
...
var IID_IImageList2 = GUID{0x192b9d83, 0x50fc, 0x457b, [8]byte{0x90, 0xa0, 0x2b, 0x82, 0xa8, 0xb5, 0xda, 0xe1}}
or
uintptr(unsafe.Pointer(IID_IImageList2)),
...
var IID_IImageList2 = &GUID{0x192b9d83, 0x50fc, 0x457b, [8]byte{0x90, 0xa0, 0x2b, 0x82, 0xa8, 0xb5, 0xda, 0xe1}}
That way, the GUID itself is used as the GUID, not its location in memory.
Interfacing COM with Go is going to be painful.
A COM interface like IImageList2 is a list of function pointers, and you can't use those C function pointers from Go directly; you have to use syscall.Syscall() and its siblings (depending on the number of arguments the individual function takes) to call them.
At its core, a COM interface instance is a structure whose first and only field is a pointer to this list of methods. So it'd be something along the lines of
type IImageList2Vtbl struct {
QueryInterface uintptr
AddRef uintptr
Release uintptr
|
type IImageList2 struct {
Vtbl *IImageList2Vtbl
}
When you pass a pointer to a variable of type IImageList2 to the Windows API function you call to create that object, be it SHGetImageList(), CoCreateInstance(), D3D11CreateDeviceAndSwapChain(), or what have you, the system will fill in the Vtbl entry with a pointer in read-only system memory that contains the list of functions.
So the first thing you have to do is make sure the methods are in the right order, so the Go structure IImageList2Vtbl and the list that Windows will give you match. Unfortunately, MSDN isn't good at this; you'll have to spelunk through the header files. Here's what I get:
type IImageList2Vtbl struct {
QueryInterface uintptr
AddRef uintptr
Release uintptr
Add uintptr
ReplaceIcon uintptr
SetOverlayImage uintptr
Replace uintptr
AddMasked uintptr
Draw uintptr
Remove uintptr
GetIcon uintptr
GetImageInfo uintptr
Copy uintptr
Merge uintptr
Clone uintptr
GetImageRect uintptr
GetIconSize uintptr
SetIconSize uintptr
GetImageCount uintptr
SetImageCount uintptr
SetBkColor uintptr
GetBkColor uintptr
BeginDrag uintptr
EndDrag uintptr
DragEnter uintptr
DragLeave uintptr
DragMove uintptr
SetDragCursorImage uintptr
DragShowNolock uintptr
GetDragImage uintptr
GetItemFlags uintptr
GetOverlayImage uintptr
Resize uintptr
GetOriginalSize uintptr
SetOriginalSize uintptr
SetCallback uintptr
GetCallback uintptr
ForceImagePresent uintptr
DiscardImages uintptr
PreloadImages uintptr
GetStatistics uintptr
Initialize uintptr
Replace2 uintptr
ReplaceFromImageList uintptr
}
I'm sure I got this wrong; please report any errors. :)
This comes from commoncontrols.h. Go with the C-style, not C++-style, interface definitions (if there are more than one), as that will have all the methods, including those of interfaces that IImageList2 derives from (IUnknown, which all interfaces derive from, and IImageList).
Er wait, I lied: Windows doesn't expect you to give it the memory for the IImageList2. That's because COM interfaces are just like Go interfaces: they're a set of methods that any implementation can implement. So in reality you have to let Windows give you a pointer to an IImageList2, not an IImageList2Vtbl.
So what do we do? We store each instance as a pointer to the interface, and then pass a pointer to that to the creation functions.
So from that, we have
var imglist *IImageList2
hr, _, _ := procSHGetImageList.Call(
uintptr(SHIL_JUMBO),
uintptr(unsafe.Pointer(&IID_IImageList2)),
uintptr(unsafe.Pointer(&imglist)),
)
Notice that we pass a pointer to the IImageList2. I'll call this the instance, and the Vtbl member the vtable.
Now when you want to call a method, you have to add an extra first parameter: the instance itself. You can look on MSDN for the return types and other parameters:
// HRESULT EndDrag(void);
hr, _, _ = syscall.Syscall(instance.Vtbl.EndDrag,
uintptr(unsafe.Pointer(imglist)))
// HRESULT SetOverlayImage(int, int);
hr, _, _ = syscall.Syscall(instance.Vtbl.SetOverlayImage,
uintptr(unsafe.Pointer(imglist)),
4,
2)
Notice that we pass imglist this time, not &imglist.
Now because C and C++ don't support multiple returns like Go does, functions typically return a HRESULT (the COM equivalent of Go's error) and have you pass in pointers to the rest of the return values at the end of the argument list. For other COM interfaces, we follow the rules above. For other type, we refer to the Windows Data Types page to see what each named type represents, and remember that C int is always Go int32 on Windows, even on 64-bit systems.
// HRESULT AddMasked(HBITMAP, COLORREF, (OUT) int *);
// we'll use uintptr for HBITMAP and int32 for COLORREF
var index int32
hr, _, _ = syscall.Syscall(instance.Vtbl.AddMasked,
uintptr(unsafe.Pointer(instance)),
hbitmap,
mask,
uintptr(unsafe.Pointer(&index)))
Finally, COM requires us to call the Release() method when we're done with an object. This takes no extra parameters and its return value is irrelevant. You can stuff this in a defer if you want.
syscall.Syscall(instance.Vtbl.Release,
uintptr(unsafe.Pointer(instance)))
Note: I don't know if SHGetImageList() requires COM to be initialized. If it doesn't, ignore this part until you need some other COM interface.
Oh but that's not enough, because you also have to initialize COM. And COM can operate in a number of threading models. Only two are important: single-threaded apartment, which is used for anything related to GUI, and multithreaded apartment, which is used for special purposes.
The CoInitialize(), CoInitializeEx(), and CoUninitialize() functions all handle COM initialization and uninitialization. You already know how to call them; they're just normal DLL functions.
But beware! If you need a single-threaded apartment, you have to use runtime.LockOSThread() to ensure Go doesn't move the current goroutine to another OS thread from under you. If you don't do this, things will break in bizarre ways.
This is all a load of work, and you have to do it for every interface you're going to use. You might as well use a package that someone already made that does the heavy lifting for you. There are several packages, such as go-ole, that do the basic stuff. I don't see one that provides IImageList2, but you might be able to piggyback off an existing one. There seems to be only one reference for IImageList.
Good luck!
Is there an example or method of getting a Windows system's idle time using Go?
I've been looking at the documentation at the Golang site but I think I'm missing how to access (and use) the API to get system information including the idle time.
Go's website is hardcoded to show the documentation for the standard library packages on Linux. You will need to get godoc and run it yourself:
go get golang.org/x/tools/cmd/godoc
godoc --http=:6060
then open http://127.0.0.1:6060/ in your web browser.
Of note is package syscall, which provides facilities for accessing functions in DLLs, including UTF-16 helpers and callback generation functions.
Doing a quick recursive search of the Go tree says it doesn't have an API for GetLastInputInfo() in particular, so unless I'm missing something, you should be able to call that function from the DLL directly:
user32 := syscall.MustLoadDLL("user32.dll") // or NewLazyDLL() to defer loading
getLastInputInfo := user32.MustFindProc("GetLastInputInfo") // or NewProc() if you used NewLazyDLL()
// or you can handle the errors in the above if you want to provide some alternative
r1, _, err := getLastInputInfo.Call(uintptr(arg))
// err will always be non-nil; you need to check r1 (the return value)
if r1 == 0 { // in this case
panic("error getting last input info: " + err.Error())
}
Your case involves a structure. As far as I know, you can just recreate the structure flat (keeping fields in the same order), but you must convert any int fields in the original to int32, otherwise things will break on 64-bit Windows. Consult the Windows Data Types page on MSDN for the appropriate type equivalents. In your case, this would be
var lastInputInfo struct {
cbSize uint32
dwTime uint32
}
Because this (like so many structs in the Windows API) has a cbSize field that requires you to initialize it with the size of the struct, we must do so too:
lastInputInfo.cbSize = uint32(unsafe.Sizeof(lastInputInfo))
Now we just need to pass a pointer to that lastInputInfo variable to the function:
r1, _, err := getLastInputInfo.Call(
uintptr(unsafe.Pointer(&lastInputInfo)))
and just remember to import syscall and unsafe.
All args to DLL/LazyDLL.Call() are uintptr, as is the r1 return. The _ return is never used on Windows (it has to do with the ABI used).
Since I went over most of what you need to know to use the Windows API in Go that you can't gather from reading the syscall docs, I will also say (and this is irrelevant to the above question) that if a function has both ANSI and Unicode versions, you should use the Unicode versions (W suffix) and the UTF-16 conversion functions in package syscall for best results.
I think that's all the info you (or anyone, for that matter) will need to use the Windows API in Go programs.
Regarding for answer from andlabs. This is ready for use example:
import (
"time"
"unsafe"
"syscall"
"fmt"
)
var (
user32 = syscall.MustLoadDLL("user32.dll")
kernel32 = syscall.MustLoadDLL("kernel32.dll")
getLastInputInfo = user32.MustFindProc("GetLastInputInfo")
getTickCount = kernel32.MustFindProc("GetTickCount")
lastInputInfo struct {
cbSize uint32
dwTime uint32
}
)
func IdleTime() time.Duration {
lastInputInfo.cbSize = uint32(unsafe.Sizeof(lastInputInfo))
currentTickCount, _, _ := getTickCount.Call()
r1, _, err := getLastInputInfo.Call(uintptr(unsafe.Pointer(&lastInputInfo)))
if r1 == 0 {
panic("error getting last input info: " + err.Error())
}
return time.Duration((uint32(currentTickCount) - lastInputInfo.dwTime)) * time.Millisecond
}
func main() {
t := time.NewTicker(1 * time.Second)
for range t.C {
fmt.Println(IdleTime())
}
}
This is code print idle time every second. Try run and don't touch mouse/keyboard
How do I obtain the underlying syscall.Handle for a *net.UDPConn on Windows? I want this handle to set the IP_MULTICAST_TTL via syscall.SetsockoptInt. On Linux I do the following:
func setTTL(conn *net.UDPConn, ttl int) error {
f, err := conn.File()
if err != nil {
return err
}
defer f.Close()
fd := int(f.Fd())
return syscall.SetsockoptInt(fd, syscall.SOL_IP, syscall.IP_MULTICAST_TTL, ttl)
}
But on Windows, the implicit dup inside *net.UDPConn's File() fails with:
04:24:49 main.go:150: dup: not supported by windows
And in the source code is marked as a to-do. How can I get this handle? Is there some other way to set the TTL if not?
Update0
I've submitted the shortcomings to the Go issue tracker:
SetTTL for *net.UDPConn
Implement dup for netFD on Windows
The short answer is impossible. But since that isn't an answer you want to hear, I will give you the right way and wrong way to solve the problem.
The right way:
implement dup() for Windows.
submit to Go as a changeset
wait for it to be released to use it
Obviously the right way has some issues... but I highly recommend doing it. Go needs windows developers to fix up these types of serious problems. The only reason this can't be done in Windows is no one implemented the function
The wrong way:
Until the patch you write gets accepted and released, you can fake it through unsafe. The way the following code works by mirroring the exact structure of a net.UDPConn. This included copying over all structs from net that make up a UDPConn. Then unsafe is used to assert that the local UDPConn is the same as net's UDPConn. The compiler can not check this and takes your word for it. Were the internals of net to ever change, it would compile but god knows what it would do.
All code is untested.
package reallyunsafenet
import (
"net"
"sync"
"syscall"
"unsafe"
)
// copied from go/src/pkg/net/fd_windows.go
type ioResult struct {
qty uint32
err error
}
// copied from go/src/pkg/net/fd_windows.go
type netFD struct {
// locking/lifetime of sysfd
sysmu sync.Mutex
sysref int
closing bool
// immutable until Close
sysfd syscall.Handle
family int
sotype int
isConnected bool
net string
laddr net.Addr
raddr net.Addr
resultc [2]chan ioResult
errnoc [2]chan error
// owned by client
rdeadline int64
rio sync.Mutex
wdeadline int64
wio sync.Mutex
}
// copied from go/src/pkg/net/udpsock_posix.go
type UDPConn struct {
fd *netFD
}
// function to get fd
func GetFD(conn *net.UDPConn) syscall.Handle {
c := (*UDPConn)(unsafe.Pointer(conn))
return c.fd.sysfd
}