Using GoLang SDK for google cloud storage.
Cannot find how to download files in chunks.
The Google Cloud documentation says to download an object from Cloud Storage, you should use the following:
rc, err := client.Bucket(bucket).Object(object).NewReader(ctx)
if err != nil {
return nil, err
}
defer rc.Close()
data, err := ioutil.ReadAll(rc)
if err != nil {
return nil, err
}
return data, nil
Source: https://cloud.google.com/storage/docs/downloading-objects#storage-download-object-code_sample
Given their SDK returns an io.Reader, you don't need to worry about the underlying method being used to be able to reference the download in chunks (although, quickly looking through their source, it just implements http.NewRequest, which does what you want, using the same logic).
The reason it doesn't seem to "chunked" from their example is because of the usage of ioutil.ReadAll, which although great for simple use cases, extracts all of the Readers data into memory (meaning it also has to wait for the data to become available).
For a better understanding of how to deal with a Reader in steps, I recommend taking a look at https://tour.golang.org/methods/21 for a tour of io.Reader and how you can use it more efficiently.
Related
After a lot of tests, we cannot seem to match the speed of gsutil when using the GS Go client libraries.
Even a skeleton file with simplest io.Copy() take A LOT longer the the simplest gsutil.
ctx := context.Background()
client, err := storage.NewClient(ctx, option.WithCredentialsFile(*flags.credsFile))
bucket := client.Bucket("my_bucket")
File, _ := os.Open("path_to_file")
wc := bucket.Object("remoteFile").NewWriter(ctx)
_, _ = io.Copy(wc, File)
err = wc.Close()
Also tried with io.CopyBuffer() when buffer is 128x1024, better, but still slow.
Any way to speed up the upload while using go? we dont want to call any external utilities...
It sounds like the io.Copy implementation is not GCS aware, and instead is doing actual byte copies (reading from the source file and writing to the destination file). In contrast, gsutil is calling the GCS Rewrite API, which for the cases where the source and destination are in the same location and storage class, will do metadata-only copies (avoiding byte copying). Doing it that way is far faster, which would match what you're observing, performance-wise.
Can you use a GCS aware Go implementation -- i.e., one that will call Rewrite rather than reading/writing the underlying object bytes?
I am building a simple testing API in golang for uploading and download image files(PNG, JPEG, JPG):
/pic [POST] for uploading the image and save it to a folder; /pic [GET] for downloading the image to the client.
I have successfully built the /pic [POST] and the image is successfully uploaded to the server's file. And I can open the file in the storage folder. (In both windows localhost server and ubuntu server)
However, when I built the /pic [GET] for downloading the picture, I am able to download the file to the client (my computer), but the downloaded file is somehow corrupted since when I try to open it with different image viewer such as gallery or Photoshop, it says "It looks like we don't support this file format". So it seems like the download is not successful.
Postman result:
File opening in gallery:
Any ideas on why this happens and how should I fix it?
The golang code for downloading the pic is as follows(omitting the error handling):
func PicDownload(w http.ResponseWriter, r *http.Request){
request := make(map[string]string)
reqBody, _ := ioutil.ReadAll(r.Body)
err = json.Unmarshal(reqBody, &request)
// Error handling
file, err := os.OpenFile("./resources/pic/" + request["filename"], os.O_RDONLY, 0666)
// Error handling
buffer := make([]byte, 512)
_, err = file.Read(buffer)
// Error handling
contentType := http.DetectContentType(buffer)
fileStat, _ := file.Stat()
// Set header
w.Header().Set("Content-Disposition", "attachment; filename=" + request["filename"])
w.Header().Set("Content-Type", contentType)
w.Header().Set("Content-Length", strconv.FormatInt(fileStat.Size(), 10))
// Copying the file content to response body
io.Copy(w, file)
return
}
When you are reading the first 512 bytes from the file in order to determine the content type, the underlying file stream pointer moves forward by 512 bytes. When you later call io.Copy, reading continues from that position.
There are two ways to correct this.
The first is to call file.Seek(0, io.SeekStart) before the call to io.Copy(). This will place the pointer back to the start of the file. This solution requires the least amount of code, but means reading the same 512 bytes from the file twice which causes some overhead.
The second solution is to create a buffer that contains the entire file using buffer := make([]byte, fileStat.Size() and using that buffer for both the http.DetectContentType() call and to write the output (write it with w.Write(buffer) instead of using io.Copy(). This approach has the possible downside of loading the entire file into memory at once, which isn't ideal for very large files (io.Copy uses 32KB chunks instead of loading the whole file).
Note: As Peter mentioned in a comment, you must ensure users cannot traverse your filesystem by posting ../../ or something as a filename.
On any given computer (OSX, WIn, Lin, etc) there are any number of connected network adapters... whether it's Wi-Fi, BlueTooth, Ethernet or other... And depending on the routing there may be multiple active devices.
In my NARROW use case I want to know what the current CONNECTED default adapter type (Wi-Fi etc...) although once that's known there are easily some others as well as some details.
Here is an example shell script that mostly works and converting it to Go is easy enough... it just seems to me there must be a native GO way.
You can use Interfaces() from net package
Sample:
package main
import (
"fmt"
"net"
)
func main() {
l, err := net.Interfaces()
if err != nil {
panic(err)
}
for _, f := range l {
if f.Flags&net.FlagUp > 0 {
fmt.Printf("%s is up\n", f.Name)
}
}
}
On Linux, you'd need to use the RTNETLINK interface, and there appears to be
a number of packages
which implement wrappers around this layer.
Basically, you'd issue an RTM_GETLINK request and then look at the type of each interface (ifi_type field in that manual).
The available types are here — you'd need those which are 802.2 (Ethernet) and 802.11 (Wi-Fi).
You could also try to implement a "low-tech" approach by first using
net.Interfaces() and then querying their classes by trying to read
a file named "/sys/class/net/{iface_name}/type" (as explained here). That type is what the ifi_type field described above contains.
Note that this approach has some drawbacks:
Interfaces might came and go anytime, so there's an inherent race between
obtaining their list via net.Interfaces() and querying each interface
via that /sys virtual filesystem.
That /sys filesystem might not be mounted on a particular system.
That would be quite unusual for a typical desktop or a server system
but not so much for some "weird" installation (such as embedded etc).
Hi I am using the qml library for go to create UI's. I am trying to learn how to pass information from the UI (qml) to go to then "do something" with.
QML is working if it is just a UI. I can run that fine when I do:
func main() {
if len(os.Args) != 2 {
fmt.Fprintf(os.Stderr, "usage: %s <qml file>\n", os.Args[0])
os.Exit(1)
}
if err := qml.Run(run); err != nil {
fmt.Fprintf(os.Stderr, "error: %v\n", err)
os.Exit(1)
}
}
func run() error {
engine := qml.NewEngine()
engine.On("quit", func() { os.Exit(0) })
component, err := engine.LoadFile(os.Args[1])
if err != nil {
return err
}
window := component.CreateWindow(nil)
window.Show()
window.Wait()
return nil
}
However when I add some code, to try and "learn" something from the UI I get the run time error:
panic: runtime error: cgo argument has Go pointer to Go pointer
The code I am adding is:
window.On("visibleChanged", func(visible bool) {
if (visible) {
fmt.Println("Width:", window.Int("width"))
}
})
I am running "go version go1.6 darwin/amd64" on OSX El Capitan
Any ideas why? A google suggests this was an error in Go 1.6 Beta, but I am running the latest stable version (installed a couple of days ago).
If it's not a simple fix, can someone explain "why" this is occuring?
The problem is that when C code stores a Go pointer (in this case, a pointer to your callback function), the garbage collector cannot track that pointer in the C code, and may garbage collect the memory that the pointer is pointing to if no Go code is referencing it. This will cause the C code to crash when it attempts to access that memory. All the runtime knows is that the C code retained the pointer (that's why it can panic), but it doesn't know what the C code will do with it later and for how long it will keep it.
To avoid this, the trick used by most libraries was to hold on to a pointer in Go as well (e.g. in a global map), to ensure that the memory is protected from the garbage collector. go-qml uses this trick as well. This trick works, but the compiler and garbage collector have no idea that it does, they cannot verify that you're not making a mistake (e.g. deleting the Go pointer, while the C code still has its pointer).
With Go 1.6, the Go developers decided to be very strict about this, and they no longer allow C code to retain a Go pointer at all. However, if you disable this check, everything will still work in this case, because go-qml implements the trick correctly (it may break in the future however, e.g. if go implements a moving garbage collector).
Here's the issue about it: https://github.com/go-qml/qml/issues/170
Side note: In this specific case, what gets passed to C is a pointer to an interface{}, which itself contains a pointer to the function. That's why you get the error "cgo argument has Go pointer to Go pointer". The reason this isn't allowed is that it's more difficult to protect these pointers from the GC for the duration of the C call, and it's not worth it, so it's forbidden instead (https://github.com/golang/go/issues/12416#issuecomment-136473697).
However, even if this were allowed, the code would still be violating the rule about C code keeping a copy of the Go pointer.
This isn't actually a problem in Go 1.6, since it doesn't implement a moving garbage collector, but the rules were made so that it can be implemented later.
If you're just playing around, I suggest trying with go 1.5.3. Go 1.6 introduced a different set of constraints on pointers to memory when using cgo, a more restrictive set, and it's possible some go packages that were designed for the older version of go are now breaking a go rule or two.
If this is the case, getting the older package to work with go 1.6, where C is allowed to call go closures, could be harder to fix. But I don't have first hand experience with that yet.
Thanks for all the help here. I have written up a beginners tutorial on using QML with Go. It can be viewed here. I will continually update it if I run into any more errors and find fixes. Thanks everyone for your help. QML/GO is an awesome combination.
https://golog.co/blog/article/Using_Go_with_QML_part_1
I've been stucked with this issue for hours:
I have a main process served as a TCP server, the main process call Fork(), pass its net.Listener()'s FD to child process. Then child process can use net.Filelistener() to inherit this FD.
I have researched this issue through many open-sourced codes, also did some experiments. But unfortunately none of these solutions satisfy me for now since they are not portable, you also need many low-level jobs which are dangerous.
If there's any solution to pass net.Listener()'s FD to child process SAFELY, I'd be glad to know.
What I've tried for now:
Environment values, not portable, will cause chaos with many FDs, not safe since can be changed from outside.
Dup FD & Clear FD_CLOEXEC then exec/fork, portable but not supported by Go API, a syscall.NoCloseOnExec() change submitted to dev team was rejected since they want to keep syscall clean.
Set SO_REUSEADDR so child process can listen to port instantly, close parent's listener before that. Failed, not portable, not supported by Go API, also unsafe.
exec.Command.ExtraFiles(), have no idea how to get inherited FDs from child process, do I need a config file to save FD & names? This solution also have a bug, read exec's document for more detail.
All right guys, I've written a simple test case of this issue(with solution 4):
https://github.com/reckhou/go-fd-pass-test
Also include 2 executables on OS X & Linux. I tried Go 1.1 & Go 1.1.1 but this issue still remains.
The easiest way is to pass the listener in the ExtraFiles field of exec.Cmd.
Example of parent:
var l *net.TCPListener
cmd := exec.Command(...)
f, err := l.File()
cmd.ExtraFiles = []*os.File{f}
Example of child:
l, err := net.FileListener(os.NewFile(3, "listener"))
You may also want to generalize this and have the child accept PROGRAMNAME_LISTENER_FD as an environment variable. Then the parent would set the environment variable to 3 before starting the child.
I am looking for same to listen loopback adresss and read data and send to exec command in WINDOWS .
Any suggestion as windows don’t support File descriptars
IN UNIX WE DO AS FOLLOWS
service := ":1200"
tcpAddr, err := net.ResolveTCPAddr("tcp", service)
listener, err := net.ListenTCP("tcp",tcpAddr)
if err != nil {
return "", err
}
defer listener.Close()
file, err := listener.File() --> unix supports file descriptor / how about in windows ??
if err != nil {
return "", err
}
cmd := exec.Command( execname)
cmd.ExtraFiles = []*os.File{file} --> in unix it support / how about in windows to add data ???
cmd.Start()