Multipart file field is unreadable - go

I am trying to upload photos to Twitter. I created a multipart writer and creating a file field using that named media but when I send my request to Twitter it keeps responding missing media field.
Am I missing something?
Here is my code
f, err := os.Open("/Users/nikos/Desktop/test.png")
errored:
if nil != err {
fmt.Println(err)
return
}
var img = new(bytes.Buffer)
enc := base64.NewEncoder(base64.StdEncoding, img)
_, err = io.Copy(enc, f)
if nil != err {
goto errored
}
body := new(bytes.Buffer)//Multipart body
writer := multipart.NewWriter(body)
cl, err := twitter.OauthClient.MakeHttpClient(&oauth.AccessToken{
Token: "xxx",
Secret: "yyy",
})
err = writer.WriteField("media_data", img.String())//base64 version of the image (i tried both binary and base64 versions neither will work)
if nil != err {
goto errored
}
part, err := writer.CreateFormFile("media", "test.png")//actual binary file multiparted and it is named media.
if nil != err {
goto errored
}
_, err = io.Copy(part, f)
if nil != err {
goto errored
}
req, err := http.NewRequest("POST",
"https://upload.twitter.com/1.1/media/upload.json",
body)
if nil != err {
goto errored
}
res, err := cl.Do(req)
if nil != err {
goto errored
}
//and twitter responds that there is no field attached named media
_, err = io.Copy(os.Stdout, res.Body)
fmt.Println(res)
if nil != err {
goto errored
}

Updates: Just referred Twitter API Upload parameter. As per your code snippet you're using both fields media and media_data. You have to use only one -
Upload using base64 -> field name is media_data
Upload using raw -> field name is media
And, you have to add Content-Type header.
req, err := http.NewRequest("POST",
"https://upload.twitter.com/1.1/media/upload.json",
body)
req.Header.Set("Content-Type", writer.FormDataContentType())
if err := writer.Close(); err != nil {
log.Println(err)
}
// Now fire the http request
PS: While composing an answer, in 30 secs gap, #cerise-limón added comment, also close the multipart writer as mentioned by #cerise-limón.
Asked in the comment:
Twitter accepts application/octet-stream, you may not need below approach.
Adding multi-part with user supplied Content-Type instead of application/octet-stream. Basically you have to do same implementation as convenience wrapper with your content-type.
writer := multipart.NewWriter(body)
h := make(textproto.MIMEHeader)
h.Set("Content-Disposition", fmt.Sprintf(`form-data; name="%s"; filename="%s"`,
escapeQuotes(fieldname), escapeQuotes(filename)))
h.Set("Content-Type", "image/png")
part, err := writer.CreatePart(h)
// use part same as before
Definition of escapeQuotes from multiple-part package.
var quoteEscaper = strings.NewReplacer("\\", "\\\\", `"`, "\\\"")
func escapeQuotes(s string) string {
return quoteEscaper.Replace(s)
}

Related

Include client multi part form file in new POST request

Think I might be missing something obvious here. I'm attempting to grab the file from a client request hitting my server and forwarding that to an external API for processing by creating a new multipart request and copying the file over. In this case, the API is looking for a FormFile under the "files" key. The receiving API keeps telling me the file has invalid mime type application/octet-stream
API Call Documentation
func forwardFile(r *http.Request) (string, error) {
file, fileHandler, err := r.FormFile("image")
if err != nil {
return "", err
}
defer file.Close()
body := &bytes.Buffer{}
writer := multipart.NewWriter(body)
part, err := writer.CreateFormFile("files", fileHandler.Filename)
if err != nil {
return "", err
}
if _, err := io.Copy(part, file); err != nil {
return "", err
}
writer.Close()
req, _ := http.NewRequest("POST", newUploadUrl, body)
req.Header.Add("Content-Type", writer.FormDataContentType())
client := &http.Client{}
response, err := client.Do(req)
}
Thank you for your time.
I guess you need to change your content type to multipart/form-data
Solved it by creating a MIMEHeader and populating the disposition and content type myself, see below:
partHeader := textproto.MIMEHeader{}
disposition := fmt.Sprintf("form-data; name=\"files\"; filename=\"%s\"", fileHandler.Filename)
partHeader.Add("Content-Disposition", disposition)
partHeader.Add("Content-Type", "image/png")
part, err := writer.CreatePart(partHeader)
if _, err := io.Copy(part, file); err != nil {
log.Print("Error copying")
return "", err
}

How can I resolve 400 status code by sending POST in Golang?

I have Python script, that works fine
def register():
url = prime_url + '/v2/mobile/user/register?lang=ru'
payload = {
'email' : 'test#test.test.mail.com',
'deviceId' : 'testId',
'password' : 'Test1234'
}
response = requests.post(url, data = payload)
print(response.text)
Response:
{"success":true,"data":"SUCCESS_FIRST_STAGE_REGISTER","params":"Two-factor authentication code sent to test#testtest.test.mail.com","code":200,"runTime":2.391624927520752}
I wrote code on Golang:
func postRequest(target string, params string) {
var jsonStr = []byte(`{"email":"testtestetetsees#mail.ru", "deviceId":"ftefst891", "password":"qwertyQwerty132"}`)
req, err := http.NewRequest("POST", target, bytes.NewBuffer(jsonStr))
if err != nil {
log.Fatalln(err)
}
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
log.Fatalln(err)
}
defer resp.Body.Close()
var result map[string]interface{}
json.NewDecoder(resp.Body).Decode(&result)
fmt.Println(result)
}
Response:
map[code:400 data:ERROR_VALIDATE params:map[deviceId:[Device Id cannot be blank.] email:[Email cannot be blank.] password:[Password cannot be blank.]] runTime:0.023465871810913086 success:false]
I see that problem is by sending JSON string. What should I do?
As mentioned by Peter in the comments you can use PostForm once you convert the data into url.Values.
If the server expects content in urlencoded form but the input you have is json you'll have to convert it first.
var data = []byte(`{"email":"testtestetetsees#mail.ru", "deviceId":"ftefst891", "password":"qwertyQwerty132"}`)
m := map[string]string{}
if err := json.Unmarshal(data, &m); err != nil {
panic(err)
}
v := url.Values{}
for key, val := range m {
v.Add(key, val)
}
resp, err := http.PostForm("https://example.com", v)
if err != nil {
panic(err)
}
// ...
If the input you have is already in urlencoded form but it needs escaping you can parse it using url.ParseQuery and let the result of that do the escaping.
var data = "email=testtestetetsees#mail.ru&deviceId=ftefst891&password=qwertyQwerty132"
v, err := url.ParseQuery(data)
if err != nil {
panic(err)
}
resp, err := http.PostForm("https://example.com", v)
if err != nil {
panic(err)
}
// ...
Mention content type as "application/json"
For simplicity you can use http.Post, use the below code as a example:
resp, err := http.Post(target, "application/json", bytes.NewBuffer([]byte("{\"email\":\"testtestetetsees#mail.ru\", \"deviceId\":\"ftefst891\", \"password\":\"qwertyQwerty132\"}")))

golang: can't execute t.execute

I'm trying to make an Handler to update one row each time getting data from a submitt button,
here is my code:
func RowHandler(res http.ResponseWriter, req *http.Request) {
if req.Method != "POST" {
http.ServeFile(res, req, "homepage.html")
return
}
Person_id := req.FormValue("Person_id")
stmt, err := db.Prepare("update Cityes set Status='right' where Person_id=?")
if err != nil {
log.Print("error ", err)
}
_, err = stmt.Exec(&Person_id)
t, err := template.ParseFiles("city_update.html") //hier i just want to show a text in html Page
if err != nil {
log.Fatal(err)
}
err = t.Execute(res, "/city_update")
}
Here instead of following
err = t.Execute(res, "/city_update")
pass data to be used to fill your template as send arguement to Execute not string. link to doc
For example .
err = t.Execute(res,struct{ID string}{Person_id})

Testing Go http.Request.FormFile?

How do I set the Request.FormFile when trying to test an endpoint?
Partial code:
func (a *EP) Endpoint(w http.ResponseWriter, r *http.Request) {
...
x, err := strconv.Atoi(r.FormValue("x"))
if err != nil {
a.ren.Text(w, http.StatusInternalServerError, err.Error())
return
}
f, fh, err := r.FormFile("y")
if err != nil {
a.ren.Text(w, http.StatusInternalServerError, err.Error())
return
}
defer f.Close()
...
}
How do I use the httptest lib to generate a post request that has value that I can get in FormFile?
You don't need to mock the complete FormFile struct as suggested by the other answer. The mime/multipart package implements a Writer type that lets you create a FormFile. From the docs
CreateFormFile is a convenience wrapper around CreatePart. It creates
a new form-data header with the provided field name and file name.
func (w *Writer) CreateFormFile(fieldname, filename string) (io.Writer, error)
Then, you can pass this io.Writer to httptest.NewRequest, which accepts a reader as an argument.
request := httptest.NewRequest("POST", "/", myReader)
To do this, you can either write the FormFile to an io.ReaderWriter buffer or use an io.Pipe. Here is a complete example that makes use of pipes:
func TestUploadImage(t *testing.T) {
// Set up a pipe to avoid buffering
pr, pw := io.Pipe()
// This writer is going to transform
// what we pass to it to multipart form data
// and write it to our io.Pipe
writer := multipart.NewWriter(pw)
go func() {
defer writer.Close()
// We create the form data field 'fileupload'
// which returns another writer to write the actual file
part, err := writer.CreateFormFile("fileupload", "someimg.png")
if err != nil {
t.Error(err)
}
// https://yourbasic.org/golang/create-image/
img := createImage()
// Encode() takes an io.Writer.
// We pass the multipart field
// 'fileupload' that we defined
// earlier which, in turn, writes
// to our io.Pipe
err = png.Encode(part, img)
if err != nil {
t.Error(err)
}
}()
// We read from the pipe which receives data
// from the multipart writer, which, in turn,
// receives data from png.Encode().
// We have 3 chained writers!
request := httptest.NewRequest("POST", "/", pr)
request.Header.Add("Content-Type", writer.FormDataContentType())
response := httptest.NewRecorder()
handler := UploadFileHandler()
handler.ServeHTTP(response, request)
t.Log("It should respond with an HTTP status code of 200")
if response.Code != 200 {
t.Errorf("Expected %s, received %d", 200, response.Code)
}
t.Log("It should create a file named 'someimg.png' in uploads folder")
if _, err := os.Stat("./uploads/someimg.png"); os.IsNotExist(err) {
t.Error("Expected file ./uploads/someimg.png' to exist")
}
}
This function makes use of the image package to generate a file dynamically taking advantage of the fact that you can pass an io.Writer to png.Encode. In the same vein, you could pass your multipart Writer to generate the bytes in a CSV format (NewWriter in package "encoding/csv"), generating a file on the fly, without needing to read anything from your filesystem.
If you have a look at the implementation of the FormFile function you'll see that it reads the exposed MultipartForm field.
https://golang.org/src/net/http/request.go?s=39022:39107#L1249
// FormFile returns the first file for the provided form key.
1258 // FormFile calls ParseMultipartForm and ParseForm if necessary.
1259 func (r *Request) FormFile(key string) (multipart.File, *multipart.FileHeader, error) {
1260 if r.MultipartForm == multipartByReader {
1261 return nil, nil, errors.New("http: multipart handled by MultipartReader")
1262 }
1263 if r.MultipartForm == nil {
1264 err := r.ParseMultipartForm(defaultMaxMemory)
1265 if err != nil {
1266 return nil, nil, err
1267 }
1268 }
1269 if r.MultipartForm != nil && r.MultipartForm.File != nil {
1270 if fhs := r.MultipartForm.File[key]; len(fhs) > 0 {
1271 f, err := fhs[0].Open()
1272 return f, fhs[0], err
1273 }
1274 }
1275 return nil, nil, ErrMissingFile
1276 }
In your test you should be able to create a test instance of multipart.Form and assign it to your request object - https://golang.org/pkg/mime/multipart/#Form
type Form struct {
Value map[string][]string
File map[string][]*FileHeader
}
Of course this will require that you use a real filepath which isn't great from a testing perspective. To get around this you could define an interface to read FormFile from a request object and pass a mock implementation into your EP struct.
Here is a good post with a few examples on how to do this: https://husobee.github.io/golang/testing/unit-test/2015/06/08/golang-unit-testing.html
I combined these and other answers into an Echo example without pipes or goroutines:
func Test_submitFile(t *testing.T) {
path := "testfile.txt"
body := new(bytes.Buffer)
writer := multipart.NewWriter(body)
part, err := writer.CreateFormFile("object", path)
assert.NoError(t, err)
sample, err := os.Open(path)
assert.NoError(t, err)
_, err = io.Copy(part, sample)
assert.NoError(t, err)
assert.NoError(t, writer.Close())
e := echo.New()
req := httptest.NewRequest(http.MethodPost, "/", body)
req.Header.Set(echo.HeaderContentType, writer.FormDataContentType())
rec := httptest.NewRecorder()
c := e.NewContext(req, rec)
c.SetPath("/submit")
if assert.NoError(t, submitFile(c)) {
assert.Equal(t, 200, rec.Code)
assert.Contains(t, rec.Body.String(), path)
fi, err := os.Stat(expectedPath)
if os.IsNotExist(err) {
t.Fatal("Upload file does not exist", expectedPath)
}
assert.Equal(t, wantSize, fi.Size())
}
}
By combining the previous answers, this worked for me:
filePath := "file.jpg"
fieldName := "file"
body := new(bytes.Buffer)
mw := multipart.NewWriter(body)
file, err := os.Open(filePath)
if err != nil {
t.Fatal(err)
}
w, err := mw.CreateFormFile(fieldName, filePath)
if err != nil {
t.Fatal(err)
}
if _, err := io.Copy(w, file); err != nil {
t.Fatal(err)
}
// close the writer before making the request
mw.Close()
req := httptest.NewRequest(http.MethodPost, "/upload", body)
req.Header.Add("Content-Type", mw.FormDataContentType())
res := httptest.NewRecorder()
// router is of type http.Handler
router.ServeHTTP(res, req)

How to efficiently store html response to a file in golang

I'm trying to build a crawler in Golang. I'm using net/http library to download the html file from url. I'm trying to save http.resp and http.Header into file.
How to convert these two file from their respective format into string so that, it could be written to a text file.
I also see a question asked earlier on parsing a stored html response file. Parse HTTP requests and responses from text file in Go . Is there any way to save the url response in this format.
Go has an httputil package with a response dump.
https://golang.org/pkg/net/http/httputil/#DumpResponse.
The second argument of response dump is a bool of whether or not to include the body. So if you want to save just the header to a file, set that to false.
An example function that would dump the response to a file could be:
import (
"io/ioutil"
"net/http"
"net/http/httputil"
)
func dumpResponse(resp *http.Response, filename string) error {
dump, err := httputil.DumpResponse(resp, true)
if err != nil {
return err
}
return ioutil.WriteFile(filename, dump, 0644)
}
Edit: Thanks to #JimB for pointing to the http.Response.Write method which makes this a lot easier than I proposed in the beginning:
resp, err := http.Get("http://google.com/")
if err != nil{
log.Panic(err)
}
f, err := os.Create("output.txt")
defer f.Close()
resp.Write(f)
This was my first Answer
You could do something like this:
resp, err := http.Get("http://google.com/")
body, err := ioutil.ReadAll(resp.Body)
// write whole the body
err = ioutil.WriteFile("body.txt", body, 0644)
if err != nil {
panic(err)
}
This was the edit to my first answer:
Thanks to #Hector Correa who added the header part. Here is a more comprehensive snippet, targeting your whole question. This writes header followed by the body of the request to output.txt
//get the response
resp, err := http.Get("http://google.com/")
//body
body, err := ioutil.ReadAll(resp.Body)
//header
var header string
for h, v := range resp.Header {
for _, v := range v {
header += fmt.Sprintf("%s %s \n", h, v)
}
}
//append all to one slice
var write []byte
write = append(write, []byte(header)...)
write = append(write, body...)
//write it to a file
err = ioutil.WriteFile("output.txt", write, 0644)
if err != nil {
panic(err)
}
Following on the answer by #Riscie you could also pick up the headers from the response with something like this:
for header, values := range resp.Header {
for _, value := range values {
log.Printf("\t\t %s %s", header, value)
}
}

Resources