Multipart http request in Go - go

I'm attempting to use Go to write a utility that authenticates and uploads a file by making a multipart http request to our server. Everything seems to go okay, except the file is not arriving on the server. Looking at it further it appears the multipart in the request is empty. Code and request output below. What am I missing in my Go code?
The Code: (I've changed the URL...)
package main
import (
"net/http"
"mime/multipart"
"strings"
"fmt"
"io/ioutil"
"io"
"os"
"bytes"
"flag"
"encoding/json"
)
var (
filename = flag.String("filename", "", "file to upload")
name = flag.String("name", "", "name to give file on server")
username = flag.String("username", "", "username for authentication")
password = flag.String("password", "", "password for authentication")
)
func main() {
flag.Parse()
// Create multipart
var b bytes.Buffer
w := multipart.NewWriter(&b)
f, _ := os.Open(*filename) //open file to send
defer f.Close()
fw, err := w.CreateFormFile("file", *name) //give file a name
if err != nil {
fmt.Println(err)
}
if _, err := io.Copy(fw, f); err != nil { //copy the file to the multipart buffer
fmt.Println(err)
}
w.Close()
// print the head of the multipart data
bs := b.Bytes()
fmt.Printf("%+v\n\n", string(bs[:1000]))
// Send authentication/login
r, e := http.Post("https://mysite/login", "application/json", strings.NewReader(fmt.Sprintf("{\"username\":\"%s\",\"password\":\"%s\"}", *username, *password)))
if e != nil {
fmt.Println(e)
} else {
// Get the token from the body
type Body struct {
Token string
}
// convert json to get the token
body, _ := ioutil.ReadAll(r.Body)
bd := bytes.NewBuffer(body)
dec := json.NewDecoder(bd)
var m Body
dec.Decode(&m)
// Upload file
req, err := http.NewRequest("POST", "https://mysite/api/apps", &b)
if err != nil {
fmt.Printf("%v\n", err)
}
req.Header.Set("Authentication", fmt.Sprintf("Bearer: %s", m.Token))
req.Header.Set("Content-Type", w.FormDataContentType())
client := &http.Client{}
res, err := client.Do(req)
if err != nil {
fmt.Printf("%v\n", err)
}
// print status and request body
fmt.Println(res.Status)
fmt.Printf("%+v\n", res.Request)
}
}
The first thing I print is the byte buffer, b, which contains the multipart data, everything looks good from here. (it was an xml file)
--83451b003d8e5cc38c0e8f60ad318e522cad4818cf293745c84ec36d26d5
Content-Disposition: form-data; name="file"; filename="snapshot-162224-820-99"
Content-Type: application/octet-stream
<manifest>
<projects>
<project name=........
Next I print the status of the request.
200 OK
Then I printed out the request structure, here is where I saw the MultipartForm was empty.
&{Method:GET URL:https://mysite/home/ Proto: ProtoMajor:0 ProtoMinor:0 Header:map[Authentication:[Bearer: DY0LCJL0g] Content-Type:[multipart/form-data; boundary=83451b003d8e5cc38c0e8f60ad318e522cad4818cf293745c84ec36d26d5] Referer:[http://mysite/home/]] Body:<nil> GetBody:<nil> ContentLength:0 TransferEncoding:[] Close:false Host: Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil> Response:0xc42018a360 ctx:<nil>}

I highly doubt if the server really received nothing. The behavior of the printed body having a nil body is expected and documented in http.Response
// Request is the request that was sent to obtain this Response.
// Request's Body is nil (having already been consumed).
// This is only populated for Client requests.
Request *Request
If you want to debug the request body sented, you should either use a mock server or a proxy.
On another note, your code's attempt to login is not going to work. It does not maintain the cookie of login info, so the later requests cannot utilize them.

Related

http.Get("example url") is giving resize output file failed error

I am writing a code in GoLang to get data from a url but when I make http.Get() request, the response I get is as per the following.
"compile_output": "# command-line-arguments\n/usr/local/go-1.13.5/pkg/tool/linux_amd64/link: resize output file failed: truncate $WORK/b001/exe/a.out: file too large\n",
Please suggest me how to tackle this.
Code is as per the following:
import (
"fmt"
"io/ioutil"
"net/http"
)
func main() {
// define the URL to get data from
url := "my_example_url"
// use http.Get() to make a GET request to the URL
resp, err := http.Get(url)
if err != nil {
fmt.Println("Error getting data:", err)
return
}
defer resp.Body.Close()
// read the response body
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println("Error reading response:", err)
return
}
// print the response body
fmt.Println(string(body))
}
I tried reading through GoLang URl Documentations, but didn't help.

Adding HTTP Headers to a GET request [duplicate]

This question already has answers here:
How to set headers in http get request?
(4 answers)
set headers for request using http.Client and http.Transport
(1 answer)
Set UserAgent in http request
(1 answer)
Closed last year.
I'm creating a reverse proxy in Golang and I'm having trouble trying to grab online examples on how to add HTTP headers to an API call.
Here is my API call:
package handlers
import (
"fmt"
"io/ioutil"
"log"
"net/http"
)
func (s *Server) getApiCall(w http.ResponseWriter, r *http.Request) {
resp, err := http.Get("https://url.com")
if err != nil {
log.Fatalln(err)
}
//We Read the response body on the line below.
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatalln(err)
}
//Convert the body to type string
sb := string(body)
log.Printf(sb)
fmt.Fprintf(w, sb)
}
Where in my function can I add an Authorization header with the a Bearer <TOKEN> as its value?
I'm new with Go, to my understanding this should be enough code, but if you need more detail about my backend just add a comment in what needs clarification.
You can't use http.Get, instead use http.NewRequest to create a request object then add the headers to it.
One example:
client := &http.Client{}
req, err := http.NewRequest("GET", "https://url.com", nil)
if err != nil {
// handle error
}
req.Header.Add("Authorization", "Bearer ...")
resp, err := client.Do(req)
if err != nil {
// handle error
}
For more details take a look at http package doc.

Golang bufio from websocket breaking after first read

I am trying to stream JSON text from a websocket. However after an initial read I noticed that the stream seems to break/disconnect. This is from a Pleroma server (think: Mastodon). I am using the default Golang websocket library.
package main
import (
"bufio"
"fmt"
"log"
"golang.org/x/net/websocket"
)
func main() {
origin := "https://poa.st/"
url := "wss://poa.st/api/v1/streaming/?stream=public"
ws, err := websocket.Dial(url, "", origin)
if err != nil {
log.Fatal(err)
}
s := bufio.NewScanner(ws)
for s.Scan() {
line := s.Text()
fmt.Println(line)
}
}
After the initial JSON text response, the for-loop breaks. I would expect it to send a new message every few seconds.
What might be causing this? I am willing to switch to the Gorilla websocket library if I can use it with bufio.
Thanks!
Although x/net/websocket connection has a Read method with the same signature as the Read method in io.Reader, the connection does not work like an io.Reader. The connection will not work as you expect when wrapped with a bufio.Scanner.
The poa.st endpoint sends a stream of messages where each message is a JSON document. Use the following code to read the messages using the Gorilla package:
url := "wss://poa.st/api/v1/streaming/?stream=public"
ws, _, err := websocket.DefaultDialer.Dial(url, nil)
if err != nil {
log.Fatal(err)
}
defer ws.Close()
for {
_, p, err := ws.ReadMessage()
if err != nil {
log.Fatal(err)
}
// p is a []byte containing the JSON document.
fmt.Printf("%s\n", p)
}
The Gorilla package has a helper method for decoding JSON messages. Here's an example of how to use that method.
url := "wss://poa.st/api/v1/streaming/?stream=public"
ws, _, err := websocket.DefaultDialer.Dial(url, nil)
if err != nil {
log.Fatal(err)
}
defer ws.Close()
for {
// The JSON documents are objects containing two fields,
// the event type and the payload. The payload is a JSON
// document itself.
var e struct {
Event string
Payload string
}
err := ws.ReadJSON(&e)
if err != nil {
log.Fatal(err)
}
// TODO: decode e.Payload based on e.Event
}

Named cookie not present

I am building a website that will rely on cookies for various things.
Then I decided to have a function that sets a cookie then read the same cookie in order to see if the browser allows cookies.
But this fails.
The template in ./views/index.html
{{define "index"}}template{{end}}
The main code:
package main
import (
"fmt"
"html/template"
"log"
"net/http"
"strconv"
"time"
"github.com/gorilla/handlers"
"github.com/gorilla/mux"
)
var tmpl *template.Template
func main(){
port :=":8088"
router := mux.NewRouter()
router.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
//Set test cookie
cookieName := strconv.FormatInt(time.Now().UnixNano(), 10)
cookieValue := strconv.FormatInt(time.Now().UnixNano(), 10)
fmt.Println("cookieName:" + cookieName)
fmt.Println("cookieValue:" + cookieValue)
cookie := http.Cookie{Name: cookieName, Value: cookieValue, Path: "/"}
http.SetCookie(w, &cookie)
//Get cookies
fmt.Println("Range over cookies")
for _, c := range r.Cookies() {
fmt.Println(c)
}
//Get test cookie by name
c, err := r.Cookie(cookieName)
if err != nil {
fmt.Println("Error: " + err.Error())
} else {
fmt.Println(c.Value)
}
err = tmpl.ExecuteTemplate(w, "index", "")
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
})
var err error
tmpl, err = template.ParseGlob("views/*")
if err != nil {
panic(err.Error())
}
router.PathPrefix("/").HandlerFunc(func(res http.ResponseWriter, req *http.Request) {
http.FileServer(http.Dir("./static/")).ServeHTTP(res, req)
})
fmt.Println("Server running on localhost" + port)
err = http.ListenAndServe(port, handlers.CompressHandler(router))
if err != nil {
log.Fatal(err)
}
}
This is terminal output:
Server running on localhost:8088
cookieName:1636243636497412077
cookieValue:1636243636497413613
Range over cookies
Error: http: named cookie not present
Any pointers to what my issue might be?
You are checking r.Cookies before you have sent the cookie to the client. You must send the cookie and then if you want to check their cookie, send a second request. It would be much easier to just open the browser and look to see if your cookie is there after you send your first response.
The method Request.Cookie gets a cookie from request Cookie headers.
The function http.SetCookie adds a Set-Cookie header to the response headers. You can observe the result of http.SetCookie using this code:
fmt.Println(w.Header()["Set-Cookie"])
The named cookie is not present in the current request because http.SetCookie does not modify the current request.
The flow of cookie values is this:
The server sets cookies in a response using the Set-Cookie header.
The client stores the cookies in a "cookie jar".
The client adds matching cookies from the jar to requests using the Cookie request header.
The server gets the cookies form the request headers.
Try this code. Load the page in the browser and refresh to observe the flow of cookie values.
const cookieName = "example"
cookieValue := strconv.FormatInt(time.Now().UnixNano(), 10)
fmt.Printf("Set cookie %s=%s\n", cookieName, cookieValue)
cookie := http.Cookie{Name: cookieName, Value: cookieValue, Path: "/"}
http.SetCookie(w, &cookie)
c, err := r.Cookie(cookieName)
if err != nil {
fmt.Printf("Get cookie %s error: %v\n", cookieName, err)
} else {
fmt.Printf("Get cookie %s=%s\n", cookieName, c.Value)
}

Is there a Go http method that converts an entire http response to a byte slice?

I have been wondering if there is already a method to write all of an http/Response into a []byte? I've found responses that note that the body can be converted easily into a []byte by doing ioutil.ReadAll(response.Body), but is there an already-built solution that writes all of the information (including status codes, headers, trailers, etc.)?
The reason I ask is because I wish to transmit this entire response through a socket to a client, and the Write method of the net library requires a byte array.
httputil.DumpResponse is what you need (Also suggested by Adrian). The following code should help:
package main
import (
"fmt"
"net/http"
"net/http/httptest"
"net/http/httputil"
"os"
)
func main() {
// Create a test server
server := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
// Set Header
w.Header().Set("HEADER_KEY", "HEADER_VALUE")
// Set Response Body
fmt.Fprintln(w, "DUMMY_BODY")
}))
defer server.Close()
// Request to the test server
resp, err := http.Get(server.URL)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
defer resp.Body.Close()
// DumpResponse takes two parameters: (resp *http.Response, body bool)
// where resp is the pointer to the response object. And body is boolean
// to dump body or not
dump, err := httputil.DumpResponse(resp, true)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
// Dump the response ([]byte)
fmt.Printf("%q", dump)
}
Output:
"HTTP/1.1 200 OK\r\nContent-Length: 11\r\nContent-Type: text/plain; charset=utf-8\r\n
Date: Wed, 18 Nov 2020 17:43:40 GMT\r\n
Header_key: HEADER_VALUE\r\n\r\n
DUMMY_BODY\n"

Resources