Include client multi part form file in new POST request - go

Think I might be missing something obvious here. I'm attempting to grab the file from a client request hitting my server and forwarding that to an external API for processing by creating a new multipart request and copying the file over. In this case, the API is looking for a FormFile under the "files" key. The receiving API keeps telling me the file has invalid mime type application/octet-stream
API Call Documentation
func forwardFile(r *http.Request) (string, error) {
file, fileHandler, err := r.FormFile("image")
if err != nil {
return "", err
}
defer file.Close()
body := &bytes.Buffer{}
writer := multipart.NewWriter(body)
part, err := writer.CreateFormFile("files", fileHandler.Filename)
if err != nil {
return "", err
}
if _, err := io.Copy(part, file); err != nil {
return "", err
}
writer.Close()
req, _ := http.NewRequest("POST", newUploadUrl, body)
req.Header.Add("Content-Type", writer.FormDataContentType())
client := &http.Client{}
response, err := client.Do(req)
}
Thank you for your time.

I guess you need to change your content type to multipart/form-data

Solved it by creating a MIMEHeader and populating the disposition and content type myself, see below:
partHeader := textproto.MIMEHeader{}
disposition := fmt.Sprintf("form-data; name=\"files\"; filename=\"%s\"", fileHandler.Filename)
partHeader.Add("Content-Disposition", disposition)
partHeader.Add("Content-Type", "image/png")
part, err := writer.CreatePart(partHeader)
if _, err := io.Copy(part, file); err != nil {
log.Print("Error copying")
return "", err
}

Related

Connection hangs when sending large file using io.Pipe to pCloud

I have a weird issue with this code:
func PrepareFileUpload(filePath, url string) (*http.Request, error) {
pr, pw := io.Pipe()
mpw := multipart.NewWriter(pw)
go func() {
defer pw.Close()
part, err := mpw.CreateFormFile("file", filepath.Base(filePath))
if err != nil {
return
}
file, err := os.Open(filePath)
if err != nil {
return
}
defer file.Close()
if _, err = io.Copy(part, file); err != nil {
return
}
err = mpw.Close()
if err != nil {
return
}
}()
req, err := http.NewRequest("POST", url, pr)
req.Header.Set("Content-Type", mpw.FormDataContentType())
return req, err
}
which I use like this:
filePath := "foo.bar"
s := []byte("Test file")
ioutil.WriteFile(filePath, s, 0644)
values := url.Values{}
values.Set("folderid", "123456")
values.Set("filename", filepath.Base(filePath))
values.Set("nopartial", "1")
u := url.URL{
Scheme: "https",
Host: "eapi.pcloud.com",
Path: "/uploadfile",
RawQuery: values.Encode(),
}
req, err := PrepareFileUpload(filePath, u.String())
if err != nil {
log.Fatal(err)
}
req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", "ACCESS_TOKEN"))
resp, err := http.DefaultClient.Do(req)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
retData, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
fmt.Println(string(retData))
For some reason, when used with the pCloud API, this hangs when running http.DefaultClient.Do(req). I have tried creating my own test server in Go, and there are no issues there, so I'm thinking it's some issues with the communication with the Go client and the pCloud server, but I can't figure out what it is (I've tried forcing HTTP/1.1, but no dice).
When uploading files without io.Pipe and with bytes.Buffer instead, everything is OK, but that doesn't work with large files (OOM).
The only warning I get when enabling verbose HTTP debugging is:
2022/04/21 10:43:29 http2: Transport failed to get client conn for eapi.pcloud.com:443: http2: no cached connection was available
This doesn't happen when I force HTTP/1.1, but the connection still hangs, so I'm not sure how relevant this error is.
Does anyone have any idea what could be the cause and how to fix it? Any help is appreciated.

How to return a body from another request

req, err := http.NewRequest("GET", "https://api.github.com/repos/octocat/Hello-World/pulls/1347", nil)
req.Header.Set("Accept", "application/vnd.github.v3.patch")
if err != nil {
check(err)
}
body, err := ioutil.ReadAll(req.Body)
ctxt.JSON(http.StatusOK, body)
Here I need to send api response from body of github api. But here I'm getting the following error:
"runtime error: invalid memory address or nil pointer dereference"
You're creating a new GET request with a nil body. See the function signature for http.NewRequest
func NewRequest(method, url string, body io.Reader) (*Request, error)
so when you access resp.Body, of course it's going to be nil.
Also, http.NewRequest just returns a request, it doesn't actually perform it.
To actually make the GET request with your request, you need to pass it to a http client's Do method. Like so:
response, err := http.DefaultClient.Do(resp)
EDIT: I would also add that naming your request as resp is confusing. I would recommend renaming the variable to req or request
this code will solve your issues.
client := &http.Client{}
apiURL := "https://api.github.com/repos/octocat/Hello-World/pulls/1347"
req, err := http.NewRequest("GET", apiURL, nil)
if err != nil {
check(err)
}
req.Header.Add("Accept", "application/vnd.github.v3.patch")
response, err := client.Do(req)
if err != nil {
check(err)
}
defer response.Body.Close()
contents, err := ioutil.ReadAll(response.Body)
if err != nil {
check(err)
}
ctxt.JSON(http.StatusOK, string(contents))

Stream file from one http endpoint to another

I am trying to stream a file from one http endpoint to another, and avoid storing large files on disk. I thought I had working with this code, but it is creating empty files:
// out, err := os.Create(key)
resp, err := http.Get("http://source_url.com/_content/" + key)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
// now stream the file straight to the endpoint using put
req, err := http.NewRequest("PUT", "http://dest_url.com/_content/"+key, resp.Body)
if err != nil {
log.Fatal(err)
}
req.Header.Set("Content-Type", "application/octet-stream")
client := &http.Client{}
res, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
fmt.Println(key, res.ContentLength, res.Status)

Multipart file field is unreadable

I am trying to upload photos to Twitter. I created a multipart writer and creating a file field using that named media but when I send my request to Twitter it keeps responding missing media field.
Am I missing something?
Here is my code
f, err := os.Open("/Users/nikos/Desktop/test.png")
errored:
if nil != err {
fmt.Println(err)
return
}
var img = new(bytes.Buffer)
enc := base64.NewEncoder(base64.StdEncoding, img)
_, err = io.Copy(enc, f)
if nil != err {
goto errored
}
body := new(bytes.Buffer)//Multipart body
writer := multipart.NewWriter(body)
cl, err := twitter.OauthClient.MakeHttpClient(&oauth.AccessToken{
Token: "xxx",
Secret: "yyy",
})
err = writer.WriteField("media_data", img.String())//base64 version of the image (i tried both binary and base64 versions neither will work)
if nil != err {
goto errored
}
part, err := writer.CreateFormFile("media", "test.png")//actual binary file multiparted and it is named media.
if nil != err {
goto errored
}
_, err = io.Copy(part, f)
if nil != err {
goto errored
}
req, err := http.NewRequest("POST",
"https://upload.twitter.com/1.1/media/upload.json",
body)
if nil != err {
goto errored
}
res, err := cl.Do(req)
if nil != err {
goto errored
}
//and twitter responds that there is no field attached named media
_, err = io.Copy(os.Stdout, res.Body)
fmt.Println(res)
if nil != err {
goto errored
}
Updates: Just referred Twitter API Upload parameter. As per your code snippet you're using both fields media and media_data. You have to use only one -
Upload using base64 -> field name is media_data
Upload using raw -> field name is media
And, you have to add Content-Type header.
req, err := http.NewRequest("POST",
"https://upload.twitter.com/1.1/media/upload.json",
body)
req.Header.Set("Content-Type", writer.FormDataContentType())
if err := writer.Close(); err != nil {
log.Println(err)
}
// Now fire the http request
PS: While composing an answer, in 30 secs gap, #cerise-limón added comment, also close the multipart writer as mentioned by #cerise-limón.
Asked in the comment:
Twitter accepts application/octet-stream, you may not need below approach.
Adding multi-part with user supplied Content-Type instead of application/octet-stream. Basically you have to do same implementation as convenience wrapper with your content-type.
writer := multipart.NewWriter(body)
h := make(textproto.MIMEHeader)
h.Set("Content-Disposition", fmt.Sprintf(`form-data; name="%s"; filename="%s"`,
escapeQuotes(fieldname), escapeQuotes(filename)))
h.Set("Content-Type", "image/png")
part, err := writer.CreatePart(h)
// use part same as before
Definition of escapeQuotes from multiple-part package.
var quoteEscaper = strings.NewReplacer("\\", "\\\\", `"`, "\\\"")
func escapeQuotes(s string) string {
return quoteEscaper.Replace(s)
}

http.Post data-binary, curl equivalent in golang

I'm trying to use net/http to post a json file to ElasticSearch. Normally in Curl I would do the following:
curl -XPOST localhost:9200/prod/aws -d #aws.json
In golang I've used an example but it has not worked. I can see it posting but something must be set incorrectly. I've tested the JSON file I am using and it's good to go.
Go code:
target_url := "http://localhost:9200/prod/aws"
body_buf := bytes.NewBufferString("")
body_writer := multipart.NewWriter(body_buf)
jsonfile := "aws.json"
file_writer, err := body_writer.CreateFormFile("upfile", jsonfile)
if err != nil {
fmt.Println("error writing to buffer")
return
}
fh, err := os.Open(jsonfile)
if err != nil {
fmt.Println("error opening file")
return
}
io.Copy(file_writer, fh)
body_writer.Close()
http.Post(target_url, "application/json", body_buf)
If you want to read json from file then use .
jsonStr,err := ioutil.ReadFile("filename.json")
if(err!=nil){
panic(err)
}
Simple way to post json in http post request.
req, err := http.NewRequest("POST", url, bytes.NewBuffer(jsonStr))
req.Header.Set("Content-Type", "application/json")
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
fmt.Println("response Status:", resp.Status)
body, _ := ioutil.ReadAll(resp.Body)
fmt.Println("response Body:", string(body))
This should work
Note that you can Post with an io.Reader as the body:
file, err := os.Open("./aws.json")
resp, err := http.Post(targetUrl, "application/json", file)
// TODO: handle errors
This might work better than reading the file contents into memory first, especially if the file is very large.

Resources