I'm writing an app that interacts with the Windows API from a Windows Service.
After loads of help from #chowey here, I sort of got the hang of things and started a basic library which I've put on GitHub here.
I've now moved on to "Services", with the requirement to list all Windows Services on a machine, start, stop, restart them. The start/stop/restart look pretty straight forward once you've got a service handle to work with, but I'm struggling with getting a list of installed services.
EnumServicesStatusEx in Advapi32.dll is the function I need to call, but it requires a pointer to pre-allocated memory for an array of ENUM_SERVICE_STATUS_PROCESS structs.
You can call the function with a null pointer and it will return the memory allocation size required, but I don't believe there is a way to directly allocate memory in Go.
At first I thought I could get the memory allocation requirement, divide it by the SizeOf the struct using the unsafe package, create a slice containing that number of elements, then pass a pointer to the first element to the function, but it says the memory needs to include space for the string data, which this wouldn't.
Does anyone know how this could be accomplished, pretty please? :).
After the suggestions from #alex, I've got the following example code working.
Looks like we create a byte slice of the right size then use the unsafe class to cast to our struct type.
_, _, _ = svcEnumServicesStatusEx.Call(
uintptr(handle),
uintptr(uint32(SVC_SC_ENUM_PROCESS_INFO)),
uintptr(uint32(SVC_SERVICE_WIN32)),
uintptr(uint32(SVC_SERVICE_STATE_ALL)),
uintptr(0),
0,
uintptr(unsafe.Pointer(&bytesReq)),
uintptr(unsafe.Pointer(&numReturned)),
uintptr(unsafe.Pointer(&resumeHandle)),
uintptr(0),
)
if bytesReq > 0 {
var buf []byte = make([]byte, bytesReq)
ret, _, _ := svcEnumServicesStatusEx.Call(
uintptr(handle),
uintptr(uint32(SVC_SC_ENUM_PROCESS_INFO)),
uintptr(uint32(SVC_SERVICE_WIN32)),
uintptr(uint32(SVC_SERVICE_STATE_ALL)),
uintptr(unsafe.Pointer(&buf[0])),
uintptr(bytesReq),
uintptr(unsafe.Pointer(&bytesReq)),
uintptr(unsafe.Pointer(&numReturned)),
uintptr(unsafe.Pointer(&resumeHandle)),
uintptr(0),
)
if ret > 0 {
var sizeTest ENUM_SERVICE_STATUS_PROCESS
iter := uintptr(unsafe.Pointer(&buf[0]))
for i := uint32(0); i < numReturned; i++ {
var data *ENUM_SERVICE_STATUS_PROCESS = (*ENUM_SERVICE_STATUS_PROCESS)(unsafe.Pointer(iter))
fmt.Printf("Service Name: %s - Display Name: %s - %#v\r\n", syscall.UTF16ToString((*[4096]uint16)(unsafe.Pointer(data.lpServiceName))[:]), syscall.UTF16ToString((*[4096]uint16)(unsafe.Pointer(data.lpDisplayName))[:]), data.ServiceStatusProcess)
iter = uintptr(unsafe.Pointer(iter + unsafe.Sizeof(sizeTest)))
}
} else {
return nil, fmt.Errorf("Failed to get Service List even with allocated memory.")
}
} else {
return nil, fmt.Errorf("Unable to get size of required memory allocation.")
}
Related
The Problem:
Right now, I'm logging my SQL query and the args that related to that query, but what will happen if my args weight a lot? say 100MB?
The Solution:
I want to iterate over the args and once they exceeded the 0.5MB I want to take the args up till this point and only log them (of course I'll use the entire args set in the actual SQL query).
Where am stuck:
I find it hard to find the size on the disk of an interface{}.
How can I print it? (there is a nicer way to do it than %v?)
The concern is mainly focused on the first section, how can I find the size, I need to know the type, if its an array, stack, heap, etc..
If code helps, here is my code structure (everything sits in dal pkg in util file):
package dal
import (
"fmt"
)
const limitedLogArgsSizeB = 100000 // ~ 0.1MB
func parsedArgs(args ...interface{}) string {
currentSize := 0
var res string
for i := 0; i < len(args); i++ {
currentEleSize := getSizeOfElement(args[i])
if !(currentSize+currentEleSize =< limitedLogArgsSizeB) {
break
}
currentSize += currentEleSize
res = fmt.Sprintf("%s, %v", res, args[i])
}
return "[" + res + "]"
}
func getSizeOfElement(interface{}) (sizeInBytes int) {
}
So as you can see I expect to get back from parsedArgs() a string that looks like:
"[4378233, 33, true]"
for completeness, the query that goes with it:
INSERT INTO Person (id,age,is_healthy) VALUES ($0,$1,$2)
so to demonstrate the point of all of this:
lets say the first two args are equal exactly to the threshold of the size limit that I want to log, I will only get back from the parsedArgs() the first two args as a string like this:
"[4378233, 33]"
I can provide further details upon request, Thanks :)
Getting the memory size of arbitrary values (arbitrary data structures) is not impossible but "hard" in Go. For details, see How to get memory size of variable in Go?
The easiest solution could be to produce the data to be logged in memory, and you can simply truncate it before logging (e.g. if it's a string or a byte slice, simply slice it). This is however not the gentlest solution (slower and requires more memory).
Instead I would achieve what you want differently. I would try to assemble the data to be logged, but I would use a special io.Writer as the target (which may be targeted at your disk or at an in-memory buffer) which keeps track of the bytes written to it, and once a limit is reached, it could discard further data (or report an error, whatever suits you).
You can see a counting io.Writer implementation here: Size in bits of object encoded to JSON?
type CounterWr struct {
io.Writer
Count int
}
func (cw *CounterWr) Write(p []byte) (n int, err error) {
n, err = cw.Writer.Write(p)
cw.Count += n
return
}
We can easily change it to become a functional limited-writer:
type LimitWriter struct {
io.Writer
Remaining int
}
func (lw *LimitWriter) Write(p []byte) (n int, err error) {
if lw.Remaining == 0 {
return 0, io.EOF
}
if lw.Remaining < len(p) {
p = p[:lw.Remaining]
}
n, err = lw.Writer.Write(p)
lw.Remaining -= n
return
}
And you can use the fmt.FprintXXX() functions to write into a value of this LimitWriter.
An example writing to an in-memory buffer:
buf := &bytes.Buffer{}
lw := &LimitWriter{
Writer: buf,
Remaining: 20,
}
args := []interface{}{1, 2, "Looooooooooooong"}
fmt.Fprint(lw, args)
fmt.Printf("%d %q", buf.Len(), buf)
This will output (try it on the Go Playground):
20 "[1 2 Looooooooooooon"
As you can see, our LimitWriter only allowed to write 20 bytes (LimitWriter.Remaining), and the rest were discarded.
Note that in this example I assembled the data in an in-memory buffer, but in your logging system you can write directly to your logging stream, just wrap it in LimitWriter (so you can completely omit the in-memory buffer).
Optimization tip: if you have the arguments as a slice, you may optimize the truncated rendering by using a loop, and stop printing arguments once the limit is reached.
An example doing this:
buf := &bytes.Buffer{}
lw := &LimitWriter{
Writer: buf,
Remaining: 20,
}
args := []interface{}{1, 2, "Loooooooooooooooong", 3, 4, 5}
io.WriteString(lw, "[")
for i, v := range args {
if _, err := fmt.Fprint(lw, v, " "); err != nil {
fmt.Printf("Breaking at argument %d, err: %v\n", i, err)
break
}
}
io.WriteString(lw, "]")
fmt.Printf("%d %q", buf.Len(), buf)
Output (try it on the Go Playground):
Breaking at argument 3, err: EOF
20 "[1 2 Loooooooooooooo"
The good thing about this is that once we reach the limit, we don't have to produce the string representation of the remaining arguments that would be discarded anyway, saving some CPU (and memory) resources.
A related questions is here https://stackoverflow.com/a/12965872/6421681.
In go, you can do:
func numsInFactorial(n int) (nums []int) {
// `nums := make([]int)` is not needed
for i := 1; i <= n; i++ {
nums = append(nums, i)
}
return
}
However,the following doesn't work:
func mapWithOneKeyAndValue(k int, v int) (m map[int]int) {
m[k] = v
return
}
An error is thrown:
panic: assignment to entry in nil map
Instead, you must:
func mapWithOneKeyAndValue(k int, v int) map[int]int {
m := make(map[int]int)
m[k] = v
return
}
I can't find the documentation for this behavior.
I have read through all of effective go, and there's no mention of it there either.
I know that named return values are defined (i.e. memory is allocated; close to what new does) but not initialized (so make behavior isn't replicated).
After some experimenting, I believe this behavior can be reduced into understanding the behavior of the following code:
func main() {
var s []int // len and cap are both 0
var m map[int]int
fmt.Println(s) // works... prints an empty slice
fmt.Println(m) // works... prints an empty map
s = append(s, 10) // returns a new slice, so underlying array gets allocated
fmt.Println(s) // works... prints [10]
m[10] = 10 // program crashes, with "assignment to entry in nil map"
fmt.Println(m)
}
The issue seems that append likely calls make and allocates a new slice detecting that the capacity of s is 0. However, map never gets an explicit initialization.
The reason for this SO question is two-pronged. First, I would like to document the behavior on SO. Second, why would the language allow non-initializing definitions of slice and map? With my experience with go so far, it seems to be a pragmatic language (i.e. unused variables lead to compilation failure, gofmt forces proper formatting), so it would make sense for it to prevent the code from compiling.
Try to assign in nil slice by index - you will get "panic: runtime error: index out of range" (example: https://play.golang.org/p/-XHh1jNyn5g)
The only reason why append function works with nil, is that append function can do reallocation for the given slice.
For example, if you trying to to append 6th element to slice of 5 elements with current capacity 5, it will create the new array with new capacity, copy all the info from old one, and swap the data array pointers in the given slice. In my understanding, it is just golang implementation of dynamic arrays.
So, the nil slice is just a special case of slice with not enough capacity, so it would be reallocated on any append operation.
More details on https://blog.golang.org/go-slices-usage-and-internals
From https://blog.golang.org/go-maps-in-action
A nil map behaves like an empty map when reading, but attempts to write to a nil map will cause a runtime panic; don't do that. To initialize a map, use the built in make function
It seems like a nil map is considered a valid empty map and that's the reason they don't allocate memory for it automatically.
I'm trying to access the EnumerateTraceGuids function from Advapi32.dll in go.
I'm at the very early stage and still trying to decipher what is it that I must do. I have the following code that keeps giving me Error: 87, meaning ERROR_INVALID_PARAMETER.
I've used this file as a starting point though it's only writing and not reading :
https://github.com/moby/moby/blob/master/daemon/logger/etwlogs/etwlogs_windows.go
Official documentation for the function I'm trying to call is here :
https://msdn.microsoft.com/en-us/library/windows/desktop/aa363713(v=vs.85).aspx
It requires GuidPropertiesArray [in, out] An array of pointers to TRACE_GUID_PROPERTIES structures. This structure is the following (https://msdn.microsoft.com/en-us/library/windows/desktop/aa364143(v=vs.85).aspx)
typedef struct _TRACE_GUID_PROPERTIES {
GUID Guid;
ULONG GuidType;
ULONG LoggerId;
ULONG EnableLevel;
ULONG EnableFlags;
BOOLEAN IsEnable;
} TRACE_GUID_PROPERTIES, *PTRACE_GUID_PROPERTIES;
I have the following code to try and do this :
package main
import (
"errors"
"fmt"
"syscall"
"unsafe"
"github.com/sirupsen/logrus"
"golang.org/x/sys/windows"
)
const (
win32CallSuccess = 0
MaxProv = 50
nbProviders = 50
)
var (
modAdvapi32 = windows.NewLazySystemDLL("Advapi32.dll")
procEnumerateTraceGuids = modAdvapi32.NewProc("EnumerateTraceGuids")
)
type ulong int32
type TRACE_GUID_PROPERTIES struct {
Guid syscall.GUID
GuidType ulong
LoggerId ulong
EnableLevel ulong
EnableFlags ulong
IsEnable bool
}
func callEnumerateTraceGuids() error {
GuidPropertiesArray:= make([]TRACE_GUID_PROPERTIES, 1)
ptr := &GuidPropertiesArray[0]
ret, _, _ := procEnumerateTraceGuids.Call(uintptr(unsafe.Pointer(&ptr)), MaxProv, nbProviders)
if ret != win32CallSuccess {
errorMessage := fmt.Sprintf("Failed to register ETW provider. Error: %d", ret)
logrus.Error(errorMessage)
return errors.New(errorMessage)
}
return nil
}
func main() {
callEnumerateTraceGuids()
}
At this point I'm not sure what is it that I must do. I've tried a lot of variation of initializing the array without success.
Hoping someone can point me in the right direction.
Thanks !
Edit : Changed code based on comments but still getting the same error.
PS : This is my first time posting to stackoverflow and I've already been told that I'm lazy less than 12 hours after posting my question (yay!) so not sure I'm asking this right...I am not too familiar with go and never called windows DLL from go before and since I keep hitting that ERROR_INVALID_PARAMETER I thought of reaching out to try and pass this first wall to be able to grasp some concepts at the same time. Hope this helps understands my request (ie. I come in peace).
OK, I had a bit of free time and an access to a Windows XP box,
so I've decided to dust off my Windows programming skills
and wrote a working solution:
package main
import (
"golang.org/x/sys/windows"
"log"
"syscall"
"unsafe"
)
var (
modAdvapi32 = windows.NewLazySystemDLL("advapi32")
procEnumerateTraceGuids = modAdvapi32.NewProc("EnumerateTraceGuids")
)
type traceGuidProperties struct {
guid syscall.GUID
guidType uint32
loggerId uint32
enableLevel uint32
enableFlags uint32
isEnable uint32
}
func enumerateTraceGuids(ptr **traceGuidProperties, count uint32, out *uint32) error {
rc, _, _ := procEnumerateTraceGuids.Call(uintptr(unsafe.Pointer(ptr)),
uintptr(count), uintptr(unsafe.Pointer(out)))
if rc != 0 {
return syscall.Errno(rc)
}
return nil
}
func enumTraceGuids() ([]*traceGuidProperties, error) {
var errMoreData = syscall.Errno(234)
var (
dummyProps traceGuidProperties
dummyPtr = &dummyProps
count uint32
)
err := enumerateTraceGuids(&dummyPtr, 0, &count)
if err != errMoreData {
return nil, err
}
items := make([]*traceGuidProperties, count)
for i := range items {
items[i] = new(traceGuidProperties)
}
for {
err = enumerateTraceGuids(&items[0], count, &count)
if err == nil {
break
}
if err != errMoreData {
return nil, err
}
for i := 0; i < int(count)-len(items); i++ {
items = append(items, new(traceGuidProperties))
}
}
return items[:count], nil
}
func main() {
log.SetFlags(0)
data, err := enumTraceGuids()
if err != nil {
log.Fatal(err)
}
log.Printf("len(data)=%d\n", len(data))
for i := range data {
log.Println(*(data[i]))
}
}
The key points:
I was wrong when I told you that
«you … should allocate an array of structs (not pointers)»—in fact
EnumerateTraceGuids indeed expects an array of pointers.
As hinted here,
there are two subtleties with how EnumerateTraceGuids works:
Contrary to what its documentation states,
it actually supports being called with its PropertyArrayCount
parameter set to 0, in which case it's expected to return ERROR_MORE_DATA
while having set GuidCount to the number of elements of the input
array required for the (next) call to complete successfully.
IOW, that way we know how many trace GUIDs the system currently
"knows about".
Still, even in this case, the function performs validity check
on the input array (see below).
As it turns out, the function expects an array of pointers to
TRACE_GUID_PROPERTIES blocks allocated by you.
In other words, if it says you it knows about 10 trace GUIDs,
you have to allocate 10 values of type TRACE_GUID_PROPERTIES,
then make an array of 10 pointers to those values and pass a pointer
to the 1st element of that array to the function.
Notice that there's an inherent race between changes occuring
in the system (those traces added or removed for any number of reasons)
and the calls to EnumerateTraceGuids.
This means if the first call to this function told you it "knows"
about 10 trace GUIDs, on the next call it may turn out
there's already 20 trace GUIDs, or 5 GUIDs
(or any other number of them FWIW).
So we account for both of these possibilities in the following way:
First we do a call with a pointer to a single (but valid)
TRACE_GUID_PROPERTIES value, allocated statically
(hence the function "sees" what looks like an array of a single element),
while telling the function the input "array" has zero elements.
We expect the function to fail with ERROR_MORE_DATA
and put the actual number of trace GUIDs it "knows" about into the variable
we've supplied it a pointer to.
We allocate that much TRACE_GUID_PROPERTIES memory blocks
the function indicated on the first call.
For this, we use the new() built-in function which behaves somewhat
like malloc() in the standard C library—it allocates the memory for
a value of the specified type and returns a pointer to the allocated
memory block.
We create an array of pointers to these allocated memory blocks
and call EnumerateTraceGuids again.
If it succeeds, we handle the possibility it returned less
elements than we've allocated, and reslice our slice.
If it fails with ERROR_MORE_DATA, we extend our slice with
whatever the number of elements is needed (allocating memory for their
TRACE_GUID_PROPERTIES blocks first), and try calling the function again.
The "magic number" 234 is the actual code for the ERROR_MORE_DATA value.
Sorry for the initial confusion.
I'm trying to use syscall with user32.dll to get the contents of the clipboard. I expect it to be image data from a Print Screen.
Right now I've got this:
if opened := openClipboard(0); !opened {
fmt.Println("Failed to open Clipboard")
}
handle := getClipboardData(CF_BITMAP)
// get buffer
img, _, err := Decode(buffer)
I need to get the data into a readable buffer using the handle.
I've had some inspiration from AllenDang/w32 and atotto/clipboard on github. The following would work for text, based on atotto's implementation:
text := syscall.UTF16ToString((*[1 << 20]uint16)(unsafe.Pointer(handle))[:])
But how can I get a buffer containing image data I can decode?
[Update]
Going by the solution #kostix provided, I hacked together a half working example:
image.RegisterFormat("bmp", "bmp", bmp.Decode, bmp.DecodeConfig)
if opened := w32.OpenClipboard(0); opened == false {
fmt.Println("Error: Failed to open Clipboard")
}
//fmt.Printf("Format: %d\n", w32.EnumClipboardFormats(w32.CF_BITMAP))
handle := w32.GetClipboardData(w32.CF_DIB)
size := globalSize(w32.HGLOBAL(handle))
if handle != 0 {
pData := w32.GlobalLock(w32.HGLOBAL(handle))
if pData != nil {
data := (*[1 << 25]byte)(pData)[:size]
// The data is either in DIB format and missing the BITMAPFILEHEADER
// or there are other issues since it can't be decoded at this point
buffer := bytes.NewBuffer(data)
img, _, err := image.Decode(buffer)
if err != nil {
fmt.Printf("Failed decoding: %s", err)
os.Exit(1)
}
fmt.Println(img.At(0, 0).RGBA())
}
w32.GlobalUnlock(w32.HGLOBAL(pData))
}
w32.CloseClipboard()
AllenDang/w32 contains most of what you'd need, but sometimes you need to implement something yourself, like globalSize():
var (
modkernel32 = syscall.NewLazyDLL("kernel32.dll")
procGlobalSize = modkernel32.NewProc("GlobalSize")
)
func globalSize(hMem w32.HGLOBAL) uint {
ret, _, _ := procGlobalSize.Call(uintptr(hMem))
if ret == 0 {
panic("GlobalSize failed")
}
return uint(ret)
}
Maybe someone will come up with a solution to get the BMP data. In the meantime I'll be taking a different route.
#JimB is correct: user32!GetClipboardData() returns a HGLOBAL, and a comment example over there suggests using kernel32!GlobalLock() to a) globally lock that handle, and b) yield a proper pointer to the memory referred to by it.
You will need to kernel32!GlobalUnlock() the handle after you're done with it.
As to converting pointers obtained from Win32 API functions to something readable by Go, the usual trick is casting the pointer to an insanely large slice. To cite the "Turning C arrays into Go slices" of "the Go wiki article on cgo":
To create a Go slice backed by a C array (without copying the original
data), one needs to acquire this length at runtime and use a type
conversion to a pointer to a very big array and then slice it to the
length that you want (also remember to set the cap if you're using Go 1.2 > or later), for example (see http://play.golang.org/p/XuC0xqtAIC for a
runnable example):
import "C"
import "unsafe"
...
var theCArray *C.YourType = C.getTheArray()
length := C.getTheArrayLength()
slice := (*[1 << 30]C.YourType)(unsafe.Pointer(theCArray))[:length:length]
It is important to keep in mind that the Go garbage collector will not
interact with this data, and that if it is freed from the C side of
things, the behavior of any Go code using the slice is nondeterministic.
In your case it will be simpler:
h := GlobalLock()
defer GlobalUnlock(h)
length := somehowGetLengthOfImageInTheClipboard()
slice := (*[1 << 30]byte)(unsafe.Pointer((uintptr(h)))[:length:length]
Then you need to actually read the bitmap.
This depends on the format of the Device-Independent Bitmap (DIB) available for export from the clipboard.
See this and this for a start.
As usually, definitions of BITMAPINFOHEADER etc are easily available online in the MSDN site.
I have a large number of allocated slices (a few million) which I have appended to. I'm sure a large number of them are over capacity. I want to try and reduce memory usage.
My first attempt is to iterate over all of them, allocate a new slice of len(oldSlice) and copy the values over. Unfortunately this appears to increase memory usage (up to double) and the garbage collection is slow to reclaim the memory.
Is there a good general way to slim down memory usage for a large number of over-capacity slices?
Choosing the right strategy to allocate your buffers is hard without knowing the exact problem.
In general you can try to reuse your buffers:
type buffer struct{}
var buffers = make(chan *buffer, 1024)
func newBuffer() *buffer {
select {
case b:= <-buffers:
return b
default:
return &buffer{}
}
}
func returnBuffer(b *buffer) {
select {
case buffers <- b:
default:
}
}
The heuristic used in append may not be suitable for all applications. It's designed for use when you don't know the final length of the data you'll be storing. Instead of iterating over them later, I'd try to minimize the amount of extra capacity you're allocating as early as possible. Here's a simple example of one strategy, which is to use a buffer only while the length is not known, and to reuse that buffer:
type buffer struct {
names []string
... // possibly other things
}
// assume this is called frequently and has lots and lots of names
func (b *buffer) readNames(lines bufio.Scanner) ([]string, error) {
// Start from zero, so we can re-use capacity
b.names = b.names[:0]
for lines.Scan() {
b.names = append(b.names, lines.Text())
}
// Figure out the error
err := lines.Err()
if err == io.EOF {
err = nil
}
// Allocate a minimal slice
out := make([]string, len(b.names))
copy(out, b.names)
return out, err
}
Of course, you'll need to modify this if you need something that's safe for concurrent use; for that I'd recommend using a buffered channel as a leaky bucket for storing your buffers.