I'm writing a Go client to create backups via a REST-API. The REST-API Response with a multipart form data to a GET-Request. So the content of the response (type *http.Response) body looks like this:
--1ceb25134a5967272c26c9f3f543e7d26834a5967272c26c9f3f595caf08
Content-Disposition: form-data; name="configuration"; filename="test.gz"
Content-Type: application/x-gzip
...
--1ceb25134a5967272c26c9f3f543e7d26834a5967272c26c9f3f595caf08--
How can I extract the zip file from the response body?
I tried to use the builtin (net/http) methods but these requires an Request struct.
Use the mime/multipart package. Assuming that resp is the *http.Response, use the following code to iterate through the parts.
contentType := resp.Header.Get("Content-Type")
mediaType, params, err := mime.ParseMediaType(contentType)
if err != nil {
log.Fatal(err)
}
if strings.HasPrefix(mediaType, "multipart/") {
mr := multipart.NewReader(resp.Body, params["boundary"])
for {
p, err := mr.NextPart()
if err == io.EOF {
return
}
if err != nil {
log.Fatal(err)
}
// p.FormName() is the name of the element.
// p.FileName() is the name of the file (if it's a file)
// p is an io.Reader on the part
// The following code prints the part for demonstration purposes.
slurp, err := ioutil.ReadAll(p)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Part %q, %q: %q\n", p.FormName(), p.FileName(), slurp)
}
}
The code in the answer handles errors by calling log.Fata. Adjust the error handling to meet the needs of your application.
Related
I am trying to stream JSON text from a websocket. However after an initial read I noticed that the stream seems to break/disconnect. This is from a Pleroma server (think: Mastodon). I am using the default Golang websocket library.
package main
import (
"bufio"
"fmt"
"log"
"golang.org/x/net/websocket"
)
func main() {
origin := "https://poa.st/"
url := "wss://poa.st/api/v1/streaming/?stream=public"
ws, err := websocket.Dial(url, "", origin)
if err != nil {
log.Fatal(err)
}
s := bufio.NewScanner(ws)
for s.Scan() {
line := s.Text()
fmt.Println(line)
}
}
After the initial JSON text response, the for-loop breaks. I would expect it to send a new message every few seconds.
What might be causing this? I am willing to switch to the Gorilla websocket library if I can use it with bufio.
Thanks!
Although x/net/websocket connection has a Read method with the same signature as the Read method in io.Reader, the connection does not work like an io.Reader. The connection will not work as you expect when wrapped with a bufio.Scanner.
The poa.st endpoint sends a stream of messages where each message is a JSON document. Use the following code to read the messages using the Gorilla package:
url := "wss://poa.st/api/v1/streaming/?stream=public"
ws, _, err := websocket.DefaultDialer.Dial(url, nil)
if err != nil {
log.Fatal(err)
}
defer ws.Close()
for {
_, p, err := ws.ReadMessage()
if err != nil {
log.Fatal(err)
}
// p is a []byte containing the JSON document.
fmt.Printf("%s\n", p)
}
The Gorilla package has a helper method for decoding JSON messages. Here's an example of how to use that method.
url := "wss://poa.st/api/v1/streaming/?stream=public"
ws, _, err := websocket.DefaultDialer.Dial(url, nil)
if err != nil {
log.Fatal(err)
}
defer ws.Close()
for {
// The JSON documents are objects containing two fields,
// the event type and the payload. The payload is a JSON
// document itself.
var e struct {
Event string
Payload string
}
err := ws.ReadJSON(&e)
if err != nil {
log.Fatal(err)
}
// TODO: decode e.Payload based on e.Event
}
I've got an HTTP Post method, which successfully posts data to an external third party API and returns a response.
I then need data returned from this response to post to my database.
The response contains a few piece of data, but I only need the 'access_token' and 'refresh_token' from it.
As a result, what I'm attempting to do is convert the response from a string into individual components in a new data struct I've created - to then pass to my database.
However, the data is showing as blank, despite it successfully being written to my browser. I'm obviously doing something fundamentally wrong, but not sure what..
Here's my code:
type data struct {
Access_token string `json:"access_token"`
Refresh_token string `json:"refresh_token"`
}
func Fetch(w http.ResponseWriter, r *http.Request) {
client := &http.Client{}
q := url.Values{}
q.Add("grant_type", "authorization_code")
q.Add("client_id", os.Getenv("ID"))
q.Add("client_secret", os.Getenv("SECRET"))
q.Add("redirect_uri", "https://callback-url.com")
q.Add("query", r.URL.Query().Get("query"))
req, err := http.NewRequest("POST", "https://auth.truelayer-sandbox.com/connect/token", strings.NewReader(q.Encode()))
if err != nil {
log.Print(err)
fmt.Println("Error was not equal to nil at first stage.")
os.Exit(1)
}
req.Header.Add("Content-Type", "application/x-www-form-urlencoded")
resp, err := client.Do(req)
if err != nil {
fmt.Println("Error sending request to server")
os.Exit(1)
}
respBody, _ := ioutil.ReadAll(resp.Body)
d := data{}
err = json.NewDecoder(resp.Body).Decode(&d)
if err != nil {
fmt.Println(err)
}
fmt.Println(d.Access_token)
fmt.Println(d.Refresh_token)
w.WriteHeader(resp.StatusCode)
w.Write(respBody)
}
With ioutil.ReadAll you read the body, already. The second time you pass to NewDecoder(resp.Body) the stream was consumed.
You can use instead json.Unmarshal(respBody, &d).
One more advice, don't ignore the error on ioutil.ReadAll
I wrote a little web crawler and had known that the Response is a zip file.
In my limited experience with golang programing, I only know how to unzip a existing file.
Can I unzip the Response.Body in memory without saving it in hard disk in advance?
Updating answer for handling Zip file response body in-memory.
Note: Ensure you have sufficient memory for handling zip file.
package main
import (
"archive/zip"
"bytes"
"fmt"
"io/ioutil"
"log"
"net/http"
)
func main() {
resp, err := http.Get("zip file url")
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
zipReader, err := zip.NewReader(bytes.NewReader(body), int64(len(body)))
if err != nil {
log.Fatal(err)
}
// Read all the files from zip archive
for _, zipFile := range zipReader.File {
fmt.Println("Reading file:", zipFile.Name)
unzippedFileBytes, err := readZipFile(zipFile)
if err != nil {
log.Println(err)
continue
}
_ = unzippedFileBytes // this is unzipped file bytes
}
}
func readZipFile(zf *zip.File) ([]byte, error) {
f, err := zf.Open()
if err != nil {
return nil, err
}
defer f.Close()
return ioutil.ReadAll(f)
}
By default Go HTTP client handles Gzip response automatically. So do typical read and close of response body.
However there is a catch in it.
// Reference https://github.com/golang/go/blob/master/src/net/http/transport.go
//
// DisableCompression, if true, prevents the Transport from
// requesting compression with an "Accept-Encoding: gzip"
// request header when the Request contains no existing
// Accept-Encoding value. If the Transport requests gzip on
// its own and gets a gzipped response, it's transparently
// decoded in the Response.Body. However, if the user
// explicitly requested gzip it is not automatically
// uncompressed.
DisableCompression bool
What it means is; If you add a header Accept-Encoding: gzip manually in the request then you have to handle Gzip response body by yourself.
For Example -
reader, err := gzip.NewReader(resp.Body)
if err != nil {
log.Fatal(err)
}
defer reader.Close()
body, err := ioutil.ReadAll(reader)
if err != nil {
log.Fatal(err)
}
fmt.Println(string(body))
I am trying to upload photos to Twitter. I created a multipart writer and creating a file field using that named media but when I send my request to Twitter it keeps responding missing media field.
Am I missing something?
Here is my code
f, err := os.Open("/Users/nikos/Desktop/test.png")
errored:
if nil != err {
fmt.Println(err)
return
}
var img = new(bytes.Buffer)
enc := base64.NewEncoder(base64.StdEncoding, img)
_, err = io.Copy(enc, f)
if nil != err {
goto errored
}
body := new(bytes.Buffer)//Multipart body
writer := multipart.NewWriter(body)
cl, err := twitter.OauthClient.MakeHttpClient(&oauth.AccessToken{
Token: "xxx",
Secret: "yyy",
})
err = writer.WriteField("media_data", img.String())//base64 version of the image (i tried both binary and base64 versions neither will work)
if nil != err {
goto errored
}
part, err := writer.CreateFormFile("media", "test.png")//actual binary file multiparted and it is named media.
if nil != err {
goto errored
}
_, err = io.Copy(part, f)
if nil != err {
goto errored
}
req, err := http.NewRequest("POST",
"https://upload.twitter.com/1.1/media/upload.json",
body)
if nil != err {
goto errored
}
res, err := cl.Do(req)
if nil != err {
goto errored
}
//and twitter responds that there is no field attached named media
_, err = io.Copy(os.Stdout, res.Body)
fmt.Println(res)
if nil != err {
goto errored
}
Updates: Just referred Twitter API Upload parameter. As per your code snippet you're using both fields media and media_data. You have to use only one -
Upload using base64 -> field name is media_data
Upload using raw -> field name is media
And, you have to add Content-Type header.
req, err := http.NewRequest("POST",
"https://upload.twitter.com/1.1/media/upload.json",
body)
req.Header.Set("Content-Type", writer.FormDataContentType())
if err := writer.Close(); err != nil {
log.Println(err)
}
// Now fire the http request
PS: While composing an answer, in 30 secs gap, #cerise-limón added comment, also close the multipart writer as mentioned by #cerise-limón.
Asked in the comment:
Twitter accepts application/octet-stream, you may not need below approach.
Adding multi-part with user supplied Content-Type instead of application/octet-stream. Basically you have to do same implementation as convenience wrapper with your content-type.
writer := multipart.NewWriter(body)
h := make(textproto.MIMEHeader)
h.Set("Content-Disposition", fmt.Sprintf(`form-data; name="%s"; filename="%s"`,
escapeQuotes(fieldname), escapeQuotes(filename)))
h.Set("Content-Type", "image/png")
part, err := writer.CreatePart(h)
// use part same as before
Definition of escapeQuotes from multiple-part package.
var quoteEscaper = strings.NewReplacer("\\", "\\\\", `"`, "\\\"")
func escapeQuotes(s string) string {
return quoteEscaper.Replace(s)
}
I'm trying to build a crawler in Golang. I'm using net/http library to download the html file from url. I'm trying to save http.resp and http.Header into file.
How to convert these two file from their respective format into string so that, it could be written to a text file.
I also see a question asked earlier on parsing a stored html response file. Parse HTTP requests and responses from text file in Go . Is there any way to save the url response in this format.
Go has an httputil package with a response dump.
https://golang.org/pkg/net/http/httputil/#DumpResponse.
The second argument of response dump is a bool of whether or not to include the body. So if you want to save just the header to a file, set that to false.
An example function that would dump the response to a file could be:
import (
"io/ioutil"
"net/http"
"net/http/httputil"
)
func dumpResponse(resp *http.Response, filename string) error {
dump, err := httputil.DumpResponse(resp, true)
if err != nil {
return err
}
return ioutil.WriteFile(filename, dump, 0644)
}
Edit: Thanks to #JimB for pointing to the http.Response.Write method which makes this a lot easier than I proposed in the beginning:
resp, err := http.Get("http://google.com/")
if err != nil{
log.Panic(err)
}
f, err := os.Create("output.txt")
defer f.Close()
resp.Write(f)
This was my first Answer
You could do something like this:
resp, err := http.Get("http://google.com/")
body, err := ioutil.ReadAll(resp.Body)
// write whole the body
err = ioutil.WriteFile("body.txt", body, 0644)
if err != nil {
panic(err)
}
This was the edit to my first answer:
Thanks to #Hector Correa who added the header part. Here is a more comprehensive snippet, targeting your whole question. This writes header followed by the body of the request to output.txt
//get the response
resp, err := http.Get("http://google.com/")
//body
body, err := ioutil.ReadAll(resp.Body)
//header
var header string
for h, v := range resp.Header {
for _, v := range v {
header += fmt.Sprintf("%s %s \n", h, v)
}
}
//append all to one slice
var write []byte
write = append(write, []byte(header)...)
write = append(write, body...)
//write it to a file
err = ioutil.WriteFile("output.txt", write, 0644)
if err != nil {
panic(err)
}
Following on the answer by #Riscie you could also pick up the headers from the response with something like this:
for header, values := range resp.Header {
for _, value := range values {
log.Printf("\t\t %s %s", header, value)
}
}