In the docs it is said that
If stored on disk, the File's underlying concrete type will be an
*os.File.
In this case everything is clear. Great. But, what happens if not, if the file is stored in memory?
My actual problem is that I´m trying to get the size of the different files stored in memory that I got though an html form but I can not use os.Stat to do fileInfo.Size() because I don´t have the location of the file, just it´s name.
fhs := req.MultipartForm.File["files"]
for _, fileHeader := range fhs {
file, _ := fileHeader.Open()
log.Println(len(file)) // Gives an error because is of type multipart.File
fileInfo, err := os.Stat(fileHeader.Filename) // Gives an error because it´s just the name, not the complete path
// Here I would do things with the file
}
You can exploit the fact that multipart.File implements io.Seeker to find its size.
cur, err := file.Seek(0, 1)
size, err := file.Seek(0, 2)
_, err := file.Seek(cur, 0)
The first line finds the file's current offset. The second seeks to the end of the file and returns where it is in relation to the beginning of the file. This is the size of the file. The third seeks to the offset we were at before trying to find the size.
You can read more about the seek method here.
if you call parseMultipartForm(0) this will write the entire file to disk instead of storing anything in memory, followed by f, _ := FormFile("file") then you can stat the file with fi, _ := f.(*os.File).Stat()
Depending on what you want to do with the data, the best thing to do may be to read the file into a byte slice with ioutil.ReadAll. (You might want the data as a byte slice eventually, anyway.) Once you've done that, you an find the length with len.
Related
I'm rather new to Golang and not sure yet, how to use certain language constructs. Currently I have following code (with test debug outputs), which does not provide expected result:
json, _ := json.Marshal(struct)
fmt.Println(json)
f,_ := os.Create(fmt.Sprintf("/tmp/%s.json", "asd"))
i,_ := f.Write(json)
fmt.Println(i)
b, err := ioutil.ReadAll(f)
fmt.Print(b)
I expect the following behaviour:
translating the struct to a byte array
creating a new file
append the byte array to the file
However, the file is always empty when I run the code in my environment (AWS Lambda), as well as using it in the Golang Playground.
The output of above code looks like this:
[123 34 ... <hug array of bytes>]
1384
[]
which leads me to believe I'm using f.Write() not correctly, although I followed the package documentation. All other outputs indicate expected behavior, so what is my mistake? I'm somewhat restricted to using the File interface, otherwise I'd have gone with ioutil.WriteFile(). My assumption is a misunderstanding of pointer/values at some point, but the compiler prevented a usage of &f.
After f.Write(), your current position in the file is at the end of it, so ioutil.ReadAll() will read from that position and return nothing.
You need to call f.Sync() to make sure that the data is persistently saved to the disk, and then f.Seek(0, 0) to rewind to the beginning of the file first.
Update: from comments, it seems that you only need to serialize the JSON and pass it forward as io.Reader, for that you don't really need a file, thanks to bytes.Buffer:
data, _ := json.Marshal(s)
buf := bytes.NewBuffer(data)
b, _ := ioutil.ReadAll(buf)
fmt.Print(string(b))
I have a device, which continues to send data over a serial port.
Now I want to read this and process it.
The data send this delimiter "!" and
as soon as this delimiter appears I want to pause reading to processing the data thats already been received.
How can I do that? Is there any documentation or examples that I can read or follow.
For reading data from a serial port you can find a few packages on Github, e.g. tarm/serial.
You can use this package to read data from your serial port. In order to read until a specific delimiter is reached, you can use something like:
config := &serial.Config{Name: "/dev/ttyUSB", Baud: 9600}
s, err := serial.OpenPort(config)
if err != nil {
// stops execution
log.Fatal(err)
}
// golang reader interface
r := bufio.NewReader(s)
// reads until delimiter is reached
data, err := r.ReadBytes('\x21')
if err != nil {
// stops execution
log.Fatal(err)
}
// or use fmt.Printf() with the right verb
// https://golang.org/pkg/fmt/#hdr-Printing
fmt.Println(data)
See also: Reading from serial port with while-loop
bufio's reader unfortunately did not work for me - it kept crashing after a while. This was a no-go since I needed a stable solution for a low-performance system.
My solution was to implement this suggestion with a small tweak. As noted, if you don't use bufio, the buffer gets overwritten every time you call
n, err := s.Read(buf0)
To fix this, append the bytes from buf0 to a second buffer, buf1:
if n > 0 {
buf1 = append(buf1, buf0[:n]...)
}
Then parse the bytes stored in buf1. If you find a subset you're looking for, process it further.
make sure to clear the buffers in a suitable manner
make sure to limit the frequency the loop is running with (e.g. time.Sleep)
I have a list of delimited files to be read after I obtained their path. Instead of saving path as a string, I'm wondering can I simply store a list of *bufio.Scanner so those will be much easier to be read later (and code will be cleaner too)? Here is a quick example:
func main(){
scannerList := read(filenameList)
dowork(scannerList)
}
func read(filenameList []string) (scannerList []*bufio.Scanner){
for _, filename := range filenameList{
op, _ := os.Open(filename)
defer op.Close()
scanner := bufio.NewScanner(op)
scannerList = append(scannerList, scanner)
}
return
}
func dowork(scannerList []*bufio.Scanner){
for _, scanner := range scannerList{
for scanner.Scan(){
//read stuff
}
//do stuff
}
}
My code similar to above example compiles, but I don't know if this is recommended (or works). Any comments? Thanks!
A Scanner is a complicated structure, and one that embeds a buffer. The buffer can grow dynamically (depending on what the scan function requests) up to 64kB (MaxScanTokenSize).
So in general it is not a good idea to keep redundant Scanners around, as the buffers cannot be released until the Scanners are discarded. But perhaps a few extra kilobytes of memory don't matter much in your case.
I'm trying to use syscall with user32.dll to get the contents of the clipboard. I expect it to be image data from a Print Screen.
Right now I've got this:
if opened := openClipboard(0); !opened {
fmt.Println("Failed to open Clipboard")
}
handle := getClipboardData(CF_BITMAP)
// get buffer
img, _, err := Decode(buffer)
I need to get the data into a readable buffer using the handle.
I've had some inspiration from AllenDang/w32 and atotto/clipboard on github. The following would work for text, based on atotto's implementation:
text := syscall.UTF16ToString((*[1 << 20]uint16)(unsafe.Pointer(handle))[:])
But how can I get a buffer containing image data I can decode?
[Update]
Going by the solution #kostix provided, I hacked together a half working example:
image.RegisterFormat("bmp", "bmp", bmp.Decode, bmp.DecodeConfig)
if opened := w32.OpenClipboard(0); opened == false {
fmt.Println("Error: Failed to open Clipboard")
}
//fmt.Printf("Format: %d\n", w32.EnumClipboardFormats(w32.CF_BITMAP))
handle := w32.GetClipboardData(w32.CF_DIB)
size := globalSize(w32.HGLOBAL(handle))
if handle != 0 {
pData := w32.GlobalLock(w32.HGLOBAL(handle))
if pData != nil {
data := (*[1 << 25]byte)(pData)[:size]
// The data is either in DIB format and missing the BITMAPFILEHEADER
// or there are other issues since it can't be decoded at this point
buffer := bytes.NewBuffer(data)
img, _, err := image.Decode(buffer)
if err != nil {
fmt.Printf("Failed decoding: %s", err)
os.Exit(1)
}
fmt.Println(img.At(0, 0).RGBA())
}
w32.GlobalUnlock(w32.HGLOBAL(pData))
}
w32.CloseClipboard()
AllenDang/w32 contains most of what you'd need, but sometimes you need to implement something yourself, like globalSize():
var (
modkernel32 = syscall.NewLazyDLL("kernel32.dll")
procGlobalSize = modkernel32.NewProc("GlobalSize")
)
func globalSize(hMem w32.HGLOBAL) uint {
ret, _, _ := procGlobalSize.Call(uintptr(hMem))
if ret == 0 {
panic("GlobalSize failed")
}
return uint(ret)
}
Maybe someone will come up with a solution to get the BMP data. In the meantime I'll be taking a different route.
#JimB is correct: user32!GetClipboardData() returns a HGLOBAL, and a comment example over there suggests using kernel32!GlobalLock() to a) globally lock that handle, and b) yield a proper pointer to the memory referred to by it.
You will need to kernel32!GlobalUnlock() the handle after you're done with it.
As to converting pointers obtained from Win32 API functions to something readable by Go, the usual trick is casting the pointer to an insanely large slice. To cite the "Turning C arrays into Go slices" of "the Go wiki article on cgo":
To create a Go slice backed by a C array (without copying the original
data), one needs to acquire this length at runtime and use a type
conversion to a pointer to a very big array and then slice it to the
length that you want (also remember to set the cap if you're using Go 1.2 > or later), for example (see http://play.golang.org/p/XuC0xqtAIC for a
runnable example):
import "C"
import "unsafe"
...
var theCArray *C.YourType = C.getTheArray()
length := C.getTheArrayLength()
slice := (*[1 << 30]C.YourType)(unsafe.Pointer(theCArray))[:length:length]
It is important to keep in mind that the Go garbage collector will not
interact with this data, and that if it is freed from the C side of
things, the behavior of any Go code using the slice is nondeterministic.
In your case it will be simpler:
h := GlobalLock()
defer GlobalUnlock(h)
length := somehowGetLengthOfImageInTheClipboard()
slice := (*[1 << 30]byte)(unsafe.Pointer((uintptr(h)))[:length:length]
Then you need to actually read the bitmap.
This depends on the format of the Device-Independent Bitmap (DIB) available for export from the clipboard.
See this and this for a start.
As usually, definitions of BITMAPINFOHEADER etc are easily available online in the MSDN site.
I'm reading in and at the same time parsing (decoding) a file in a custom format, which is compressed with zlib. My question is how can I efficiently uncompress and then parse the uncompressed content without growing the slice? I would like to parse it whilst reading it into a reusable buffer.
This is for a speed-sensitive application and so I'd like to read it in as efficiently as possible. Normally I would just ioutil.ReadAll and then loop again through the data to parse it. This time I'd like to parse it as it's read, without having to grow the buffer into which it is read, for maximum efficiency.
Basically I'm thinking that if I can find a buffer of the perfect size then I can read into this, parse it, and then write over the buffer again, then parse that, etc. The issue here is that the zlib reader appears to read an arbitrary number of bytes each time Read(b) is called; it does not fill the slice. Because of this I don't know what the perfect buffer size would be. I'm concerned that it might break up some of the data that I wrote into two chunks, making it difficult to parse because one say uint64 could be split from into two reads and therefore not occur in the same buffer read - or perhaps that can never happen and it's always read out in chunks of the same size as were originally written?
What is the optimal buffer size, or is there a way to calculate this?
If I have written data into the zlib writer with f.Write(b []byte) is it possible that this same data could be split into two reads when reading back the compressed data (meaning I will have to have a history during parsing), or will it always come back in the same read?
You can wrap your zlib reader in a bufio reader, then implement a specialized reader on top that will rebuild your chunks of data by reading from the bufio reader until a full chunk is read. Be aware that bufio.Read calls Read at most once on the underlying Reader, so you need to call ReadByte in a loop. bufio will however take care of the unpredictable size of data returned by the zlib reader for you.
If you do not want to implement a specialized reader, you can just go with a bufio reader and read as many bytes as needed with ReadByte() to fill a given data type. The optimal buffer size is at least the size of your largest data structure, up to whatever you can shove into memory.
If you read directly from the zlib reader, there is no guarantee that your data won't be split between two reads.
Another, maybe cleaner, solution is to implement a writer for your data, then use io.Copy(your_writer, zlib_reader).
OK, so I figured this out in the end using my own implementation of a reader.
Basically the struct looks like this:
type reader struct {
at int
n int
f io.ReadCloser
buf []byte
}
This can be attached to the zlib reader:
// Open file for reading
fi, err := os.Open(filename)
if err != nil {
return nil, err
}
defer fi.Close()
// Attach zlib reader
r := new(reader)
r.buf = make([]byte, 2048)
r.f, err = zlib.NewReader(fi)
if err != nil {
return nil, err
}
defer r.f.Close()
Then x number of bytes can be read straight out of the zlib reader using a function like this:
mydata := r.readx(10)
func (r *reader) readx(x int) []byte {
for r.n < x {
copy(r.buf, r.buf[r.at:r.at+r.n])
r.at = 0
m, err := r.f.Read(r.buf[r.n:])
if err != nil {
panic(err)
}
r.n += m
}
tmp := make([]byte, x)
copy(tmp, r.buf[r.at:r.at+x]) // must be copied to avoid memory leak
r.at += x
r.n -= x
return tmp
}
Note that I have no need to check for EOF because I my parser should stop itself at the right place.