I have a zip file stored on Google Drive (it is shared publicly). I want to know how to download it in Golang. This current code just creates a blank file named "file.zip":
package main
import (
"fmt"
"io"
"net/http"
"os"
)
func main() {
url := "https://docs.google.com/uc?export=download&id=0B2Q7X-dUtUBebElySVh1ZS1iaTQ"
fileName := "file.zip"
fmt.Println("Downloading file...")
output, err := os.Create(fileName)
defer output.Close()
response, err := http.Get(url)
if err != nil {
fmt.Println("Error while downloading", url, "-", eerrror)
return
}
defer response.Body.Close()
n, err := io.Copy(output, response.Body)
fmt.Println(n, "bytes downloaded")
}
This appears to be a bug, either with Google drive or with golang, I'm not sure which!
The problem is that the first URL you gave redirects to a second URL which looks something like this
https://doc-00-c8-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/8i67l6m6cdojptjuh883mu0qqmtptds1/1376330400000/06448503420061938118/*/0B2Q7X-dUtUBebElySVh1ZS1iaTQ?h=16653014193614665626&e=download
Note the * in the URL which is legal according to this stack overflow question. However it does have a special meaning as a delimeter.
Go fetches the URL with the * encoded as %2A like this
https://doc-00-c8-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/8i67l6m6cdojptjuh883mu0qqmtptds1/1376330400000/06448503420061938118/%2A/0B2Q7X-dUtUBebElySVh1ZS1iaTQ?h=16653014193614665626&e=download
Which Google replies "403 Forbidden" to.
Google doesn't seem to be resolving the %2A into a *.
According to this article on wikipedia reserved characters (of which * is one) used in a URI scheme: if it is necessary to use that character for some other purpose, then the character must be percent-encoded.
I'm not enough of an expert on this to say who is right, but since Google wrote both parts of the problem it is definitely their fault somewhere!
Here is the program I was using for testing
I found the solution.
Use: https://googledrive.com/host/ID
Instead of: https://docs.google.com/uc?export=download&id=ID
I'm still investigating on why this is happening, in the meanwhile you can use this workaround:
http://play.golang.org/p/SzGBAiZdGJ
CheckRedirect is called when a redirect happens and you can add an Opaque path to avoid having the URL url-encoded.
Francesc
Related
This seems like a really simple issue but for the life of me I cannot figure it out.
All I want to do is unzip and extract the contents of a tar.gz file. On godoc there seems to be a reference to a function that does exactly this. (https://godoc.org/golang.org/x/build/internal/untar).
There are no examples though and I can't seem to figure it out. Worse yet, though, I can't even figure out how to get access to the function.
Right now I have:
package main
import (
"io"
"os"
//???
)
func main() {
f, err := os.Open("foo.tar.gz")
if err != nil {
panic(err)
}
defer f.Close()
var freader io.ReadCloser = f
err = untar.Untar(freader, ".") // <- Compiler doesn't recognize
if err != nil {
panic(err)
}
freader.Close()
}
You can't. Packages with internal as part of their name can't be used by any package outside of the same repo. In other words, golang.org/x/build/internal/untar is for golang.org/x/build only, and the compiler won't allow you to reference it from anywhere else.
You can, if you really like, copy the source of that package into a new file, and use it from there, but you can't use it directly. This gives the developers of the internal untar package complete freedom to break their interface without having to worry about other users.
Short: How can I read the CGI var REMOTE_USER on golang using fastcgi?
Long:
I'm trying to write a program in go to work behind a httpd using fcgi over a socket. The httpd does the ssl termination and provides basic auth. I need to read $REMOTE_USER, but I cannot in golang, while I can in perl.
My code is based on this fcgi example. I try
func homeView(w http.ResponseWriter, r *http.Request) {
user, pass, authok := r.BasicAuth()
But authok is always false, user and pass remain empty, although I know for sure that the authorization (done by httpd) was OK. To eliminate other errors, I have done it in perl:
my $socket = FCGI::OpenSocket("/run/fcgi-check.sock", 5);
my $q = FCGI::Request(\*STDIN, \*STDOUT, \*STDERR, \%ENV, $socket);
while ($q->Accept() >= 0) {
my $c = CGI::Simple->new;
my $user_id = $c->remote_user();
and it works fine in perl.
To debug, I printed the output of r.Header and I got:
map[Authorization:[]
Am I right that the header that go sees does no hold any information about any authorization? But it does in perl.
Here is a full but minimal golang code example that demonstrates the problem (on OpenBSD 5.8 with go version go1.4.2 openbsd/amd64 and OpenBSDs httpd with 'authenticate "/" with restricted_users' in httpd.conf.
package main
import (
"github.com/gorilla/mux"
"io"
"log"
"fmt"
"net"
"net/http"
"net/http/fcgi"
)
func homeView(w http.ResponseWriter, r *http.Request) {
headers := w.Header()
headers.Add("Content-Type", "text/html")
headers.Add("Cache-Control", "no-cache, no-store, must-revalidate")
headers.Add("Pragma", "no-cache")
headers.Add("Expires", "0")
r.ParseForm()
user, pass, authok := r.BasicAuth()
if authok {
io.WriteString(w, fmt.Sprintln("Auth OK"))
io.WriteString(w, fmt.Sprintln("user is: "+user+", pass is: "+pass))
} else {
io.WriteString(w, fmt.Sprintln("Auth NOT OK"))
}
}
func main() {
r := mux.NewRouter()
r.HandleFunc("/check/", homeView)
var err error
listener, err := net.Listen("unix", "/run/fcgi-check.sock")
if err != nil {
log.Fatal(err)
}
defer listener.Close()
err = fcgi.Serve(listener, r)
if err != nil { log.Fatal(err)}
}
Help will be appreciated!
Thanks in advance!
T.
Go 1.9 will expose cgi environment variables. As seen in this closed ticket:
https://go-review.googlesource.com/c/40012
The simple answer (as of go version 1.4.2) is that go currently does not support the transfer of CGI variable REMOTE_USER.
While #JimB is correct on that you're wrong in your approach, I'll answer the question as stated.
The net/http/fcgi package uses the machinery of net/http/cgi to populate an instance of http.Request—which is passed to your handler—with "parameters" (key/value pairs) submitted by the webserver during the FastCGI session (call).
This is done here.
Now if you'll inspect the relevant bit of the net/http/cgi code, you'll see that the variables which are not mapped to specific dedicated fields of http.Request get converted to HTTP "headers".
This means, your code should be able to access the variable you need using something like
ruser := r.Header.Get("Remote-User")
Update 2015-12-02: the reseach performed by #JimB and the OP showed that there's apparently no way to read the REMOTE_USER variable under FastCGI. Sorry for the noise.
This core change to the fcgi package is in review and is close to being merged. If it's no longer relevant to you, hopefully it will be useful to others.
Hi I'm trying to wrap my head around what seems to be a problem with multibyte support in this open source library (https://github.com/carbocation/go-instagram/). I am using the code below to retrieve information about the tag blue in swedish. How ever I get an empty array when trying.
fmt.Println("Starting instagram download.")
client := instagram.NewClient(nil)
client.ClientID = "myid"
media, _, _ := client.Tags.RecentMedia("blå", nil)
fmt.Println(media)
I have tried using the api trough the browser and there are several pictures tagged with the tag. I have also tried using the code snippet with tags in English like blue and that returns the latest pictures as well. I would be glad if any one could explain why this might happen. Id like to update the lib so it supports multi-byte but I haven't got the go knowledge required. Is this a go problem or a problem with the library?
Thank you
The problem is in validTagName():
// Strip out things we know Instagram won't accept. For example, hyphens.
func validTagName(tagName string) (bool, error) {
//\W matches any non-word character
reg, err := regexp.Compile(`\W`)
if err != nil {
return false, err
}
if reg.MatchString(tagName) {
return false, nil
}
return true, nil
}
In Go, \W matches precisely [^0-9A-Za-z_]. This validation check is incorrect.
I'm using streadway/amqp to do a tie in from rabbitmq to our alert system. I need a method that can return a list of all the currently declared queues (exchanges would be nice too!) so that I can go through and get all the message counts.
I'm digging through the api documentation here...
http://godoc.org/github.com/streadway/amqp#Queue
...but I don't seem to be finding what I'm looking for. We're currently using a bash call to 'rabbitmqctl list_queues' but that's a kludge way to get this information, requires a custom sudo setting, and fires off hundreds of log entries a day to the secure log.
edit: method meaning, 'a way to get this piece of information' as opposed to an actual call, though a call would be great I don't believe it exists.
Answered my own question. There isn't a way! The amqp spec doesn't have a standard way of finding this out which seems like a glaring oversight to me. However, since my backend is rabbitmq with the management plugin, I can make a call to that to get this information.
from https://stackoverflow.com/a/21286370/5076297 (in python, I'll just have to translate this and probably also figure out the call to get vhosts):
import requests
def rest_queue_list(user='guest', password='guest', host='localhost', port=15672, virtual_host=None):
url = 'http://%s:%s/api/queues/%s' % (host, port, virtual_host or '')
response = requests.get(url, auth=(user, password))
queues = [q['name'] for q in response.json()]
return queues
edit: In golang (this was a headache to figure out as I haven't done anything with structures in years)
package main
import (
"fmt"
"net/http"
"encoding/json"
)
func main() {
type Queue struct {
Name string `json:name`
VHost string `json:vhost`
}
manager := "http://127.0.0.1:15672/api/queues/"
client := &http.Client{}
req, _ := http.NewRequest("GET", manager, nil)
req.SetBasicAuth("guest", "guest")
resp, _ := client.Do(req)
value := make([]Queue, 0)
json.NewDecoder(resp.Body).Decode(&value)
fmt.Println(value)
}
Output looks like this (I have two queues named hello and test)
[{hello /} {test /}]
I have a short Go program that runs the go list -json command for several packages, stores the output of each run of the command in a json.RawMessage, appends each json.RawMessage into a slice of json.RawMessages, and then returns the result to the server after concatenating each of the json.RawMessages together and compacting the json. However, there is an error message that gets produced when I run json.Compact that I can't locate the source of. Googling this error message reveals that most people who seem to encounter it--whether it's for an invalid , or some other character--have a hard time finding the source of it.
invalid character ',' looking for beginning of value
The code with comments is available to view here on play.golang.org (although it won't run there) and also below.
Question: can you explain the source of this error and how to prevent it?
(Note, some of the packages were included just for testing purposes)
package main
import (
"expvar"
"encoding/json"
"bytes"
"fmt"
"github.com/go-martini/martini"
"github.com/zenazn/goji"
"github.com/zenazn/goji/web"
"go/build"
"log"
"math/rand"
"net/http"
_ "net/http/pprof"
"os/exec"
)
type myType struct {
J []json.RawMessage
}
var pack map[string]string
type GoList struct {
Imports []string
}
type Import struct {
Dir string
ImportPath string
Name string
Target string
Standard bool
Root string
GoFiles []string
Imports []string
Deps []string
}
const contentTypeJSON = "application/json"
func main() {
http.HandleFunc("/importgraph", func(w http.ResponseWriter, r *http.Request) { importGraph(w, r) })
http.HandleFunc("/", handler)
http.ListenAndServe(":8080", nil)
}
func handler(w http.ResponseWriter, r *http.Request) {
fmt.Println("Inside handler")
fmt.Fprintf(w, "Hello world from my Go program!")
}
func importGraph(w http.ResponseWriter, r *http.Request) {
pack = make(map[string]string)
var t myType
cmd := exec.Command("go", "list", "-json")
stdout, err := cmd.Output()
if err != nil {
println(err.Error())
return
}
var list GoList
err = json.Unmarshal(stdout, &list)
for _, d := range list.Imports {
//get the imports for each of the packages listed by go list -json
t.imports(d)
}
var buff bytes.Buffer
//concatenate the separate json.RawMessages together into json
buff.WriteByte('[')
for i, j := range t.J {
if i != 0 {
buff.WriteByte(',')
}
buff.Write([]byte(j))
}
buff.WriteByte(']')
var buffer bytes.Buffer
if err := json.Compact(&buffer, buff.Bytes()); err != nil {
println(err.Error()) //error message: invalid character ',' looking for beginning of value
return
}
w.Header().Set("Content-Type", contentTypeJSON)
w.Write(buffer.Bytes())
}
func (myObj *myType) imports(pk string) error {
cmd := exec.Command("go", "list", "-json", pk)
stdout, _ := cmd.Output()
pack[pk] = pk
var deplist Import
json.Unmarshal(stdout, &deplist)
var newj json.RawMessage
json.Unmarshal(stdout, &newj)
myObj.J = append(myObj.J, newj)
for _, imp := range deplist.Imports {
if _, ok := pack[imp]; !ok {
myObj.imports(imp) //recursive call to get the imports of the imports etc
}
}
return nil
}
First, as has been commented, are you sure you can't use
the go/build package directly rather than running go list?
I Wouldn't use println (or fmt.Println) inside HTTP handlers. It's much better to use log.Println and/or get the error into the ResponseWriter. Also, it's a good idea to wrap your ListenAndServe call with log.Fatal.
When printing/logging error values you can just use err, no need to have err.Error().
Further, when you actually want to do something more detailed than just reporting/logging the error message you can look at it's type and other info. For example, log.Printf("verbose error info: %#v", err) gives:
&json.SyntaxError{msg:"invalid character ',' looking for beginning of value", Offset:0}
I tried this because I know the json package returns various error types with additional info and I was hoping the offset value would be of help. If it had been then something like this might have been helpful:
if err := json.Compact(…) {
if err != nil {
log.Println("json.Compact:", err)
if serr, ok := err.(*json.SyntaxError); ok {
log.Println("Occurred at offset:", serr.Offset)
// … something to show the data in buff around that offset …
}
}
}
But offset zero isn't helpful :(
So although this doesn't identify you problem hopefully
it can be of some help to your further investigation.
Edit:
So after adding:
log.Println("Write file:", ioutil.WriteFile("data.json", buff.Bytes(), 0600))
to the above error handling block I then ran a JSON validator on the resultant file and it identified this piece:
"XTestImports": [
"io",
"log",
"net"
]
},,{
"Dir": "/usr/local/go/src/mime",
"ImportPath": "mime",
"Name": "mime",
Note the double ,,.
That should tell you whete the error in your code is.
But if not, you need to skip empty entries, either when processing t.J or when you build it. The later is better and just involves:
if len(newj) > 0 {
myObj.J = append(myObj.J, newj)
}
(where btw you don't check for errors from json.Unmarshal so it's not clear if that is supposed to ever be empty or if it's empty due to a preceeding error. Never ignore error returns!)
I also encountered the same error message in a Go program, but the error message was within the HTTP response error, in HTML format when my HTTP response parser expected JSON.
For me, the solution was to change my request to include setting the Content-Type header to application/json. How you do this depends on which http client library you happen to be using; if you have access to the http.Header core type, you can set the header with .Set(...).
I realize the scope of this fix for me may not apply to the original question, but I came here first after googling and thought this would help others, since the message was not particularly obvious at first glance. The hint is that the invalid < character is the first HTML character in the error/response, which is likely the result of the request type not being set to application/json, thus the server responds with a non JSON response.
For me the issue was I was trying to parse the already parsed JSON.
I was also facing this error "invalid character 'N' looking for beginning of value".
This error was coming while "unmarshalling the non-json response into a json". I was expecting a json response, so wrote go code to unmarshal it into a json. But, due to URL change, the response that I was getting was a text ie. "404 Not found" error, which cannot be unmarshalled into a json.
"invalid character 'N' looking for beginning of value"
So, to summarise, this error appears when we are trying to unmarshal a non-json response (text/html/xml) into json.
Reason for this eerie error message is :
// When unmarshaling quoted strings, invalid UTF-8 or
// invalid UTF-16 surrogate pairs are not treated as an error.
// Instead, they are replaced by the Unicode replacement
// character U+FFFD.
https://golang.org/src/encoding/json/decode.go
In my case I saved my json as string then parsed it by :
stringData = JSON.parse(myJsonString)
I also had the same error another time using gin-context-ShouldBind() (https://godoc.org/github.com/gin-gonic/gin#Context.ShouldBind) and mapping my json to go object:
error was because it needs a json as string, so I used : JSON.stringify(jsonObject) when sending my request from front-end part.
And in case someone has the same problem as me, I needed to call JSON.stringify on my post data.
I encountered a similar problem with my error message being:
invalid character 'I' looking for beginning of value
In my case, i was trying to decode BSON using json.Unmarshal. Json doesn't recognize the ISODate type, which caused this error.
I had a similar issue. For me I omitted the first letter of my authorization token. So instead of
"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbWFpbCI6InJhcGhhZWxuZ0BlbWFpbC5jb20iLCJleHAiOjE2MTM5NTQzMjB9.yPGC937VNAF8Qg05Z1x3fZ3zu_MUs-cA_Iag5-4RcJE"
I used this
"yJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbWFpbCI6InJhcGhhZWxuZ0BlbWFpbC5jb20iLCJleHAiOjE2MTM5NTQzMjB9.yPGC937VNAF8Qg05Z1x3fZ3zu_MUs-cA_Iag5-4RcJE"