http.Post data-binary, curl equivalent in golang - go

I'm trying to use net/http to post a json file to ElasticSearch. Normally in Curl I would do the following:
curl -XPOST localhost:9200/prod/aws -d #aws.json
In golang I've used an example but it has not worked. I can see it posting but something must be set incorrectly. I've tested the JSON file I am using and it's good to go.
Go code:
target_url := "http://localhost:9200/prod/aws"
body_buf := bytes.NewBufferString("")
body_writer := multipart.NewWriter(body_buf)
jsonfile := "aws.json"
file_writer, err := body_writer.CreateFormFile("upfile", jsonfile)
if err != nil {
fmt.Println("error writing to buffer")
return
}
fh, err := os.Open(jsonfile)
if err != nil {
fmt.Println("error opening file")
return
}
io.Copy(file_writer, fh)
body_writer.Close()
http.Post(target_url, "application/json", body_buf)

If you want to read json from file then use .
jsonStr,err := ioutil.ReadFile("filename.json")
if(err!=nil){
panic(err)
}
Simple way to post json in http post request.
req, err := http.NewRequest("POST", url, bytes.NewBuffer(jsonStr))
req.Header.Set("Content-Type", "application/json")
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
fmt.Println("response Status:", resp.Status)
body, _ := ioutil.ReadAll(resp.Body)
fmt.Println("response Body:", string(body))
This should work

Note that you can Post with an io.Reader as the body:
file, err := os.Open("./aws.json")
resp, err := http.Post(targetUrl, "application/json", file)
// TODO: handle errors
This might work better than reading the file contents into memory first, especially if the file is very large.

Related

Include client multi part form file in new POST request

Think I might be missing something obvious here. I'm attempting to grab the file from a client request hitting my server and forwarding that to an external API for processing by creating a new multipart request and copying the file over. In this case, the API is looking for a FormFile under the "files" key. The receiving API keeps telling me the file has invalid mime type application/octet-stream
API Call Documentation
func forwardFile(r *http.Request) (string, error) {
file, fileHandler, err := r.FormFile("image")
if err != nil {
return "", err
}
defer file.Close()
body := &bytes.Buffer{}
writer := multipart.NewWriter(body)
part, err := writer.CreateFormFile("files", fileHandler.Filename)
if err != nil {
return "", err
}
if _, err := io.Copy(part, file); err != nil {
return "", err
}
writer.Close()
req, _ := http.NewRequest("POST", newUploadUrl, body)
req.Header.Add("Content-Type", writer.FormDataContentType())
client := &http.Client{}
response, err := client.Do(req)
}
Thank you for your time.
I guess you need to change your content type to multipart/form-data
Solved it by creating a MIMEHeader and populating the disposition and content type myself, see below:
partHeader := textproto.MIMEHeader{}
disposition := fmt.Sprintf("form-data; name=\"files\"; filename=\"%s\"", fileHandler.Filename)
partHeader.Add("Content-Disposition", disposition)
partHeader.Add("Content-Type", "image/png")
part, err := writer.CreatePart(partHeader)
if _, err := io.Copy(part, file); err != nil {
log.Print("Error copying")
return "", err
}

Connection hangs when sending large file using io.Pipe to pCloud

I have a weird issue with this code:
func PrepareFileUpload(filePath, url string) (*http.Request, error) {
pr, pw := io.Pipe()
mpw := multipart.NewWriter(pw)
go func() {
defer pw.Close()
part, err := mpw.CreateFormFile("file", filepath.Base(filePath))
if err != nil {
return
}
file, err := os.Open(filePath)
if err != nil {
return
}
defer file.Close()
if _, err = io.Copy(part, file); err != nil {
return
}
err = mpw.Close()
if err != nil {
return
}
}()
req, err := http.NewRequest("POST", url, pr)
req.Header.Set("Content-Type", mpw.FormDataContentType())
return req, err
}
which I use like this:
filePath := "foo.bar"
s := []byte("Test file")
ioutil.WriteFile(filePath, s, 0644)
values := url.Values{}
values.Set("folderid", "123456")
values.Set("filename", filepath.Base(filePath))
values.Set("nopartial", "1")
u := url.URL{
Scheme: "https",
Host: "eapi.pcloud.com",
Path: "/uploadfile",
RawQuery: values.Encode(),
}
req, err := PrepareFileUpload(filePath, u.String())
if err != nil {
log.Fatal(err)
}
req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", "ACCESS_TOKEN"))
resp, err := http.DefaultClient.Do(req)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
retData, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
fmt.Println(string(retData))
For some reason, when used with the pCloud API, this hangs when running http.DefaultClient.Do(req). I have tried creating my own test server in Go, and there are no issues there, so I'm thinking it's some issues with the communication with the Go client and the pCloud server, but I can't figure out what it is (I've tried forcing HTTP/1.1, but no dice).
When uploading files without io.Pipe and with bytes.Buffer instead, everything is OK, but that doesn't work with large files (OOM).
The only warning I get when enabling verbose HTTP debugging is:
2022/04/21 10:43:29 http2: Transport failed to get client conn for eapi.pcloud.com:443: http2: no cached connection was available
This doesn't happen when I force HTTP/1.1, but the connection still hangs, so I'm not sure how relevant this error is.
Does anyone have any idea what could be the cause and how to fix it? Any help is appreciated.

Returning data back to client from Go HTTP request

I've written a simple Fetch Go function which calls an API, and generates a response.
When called, it successfully logs the data to the console which is pulled from the API.
What I want to do though is take the final 'respBody' variable generated from reading the response body, and then return it back to my frontend client - but I can't figure out how.
All the examples just use Println, and I've searched the docs but can't find anything.
Can anyone tell me how to change my code so I can return the respBody back to the client?
Here's my function:
func Fetch(w http.ResponseWriter, r *http.Request) {
client := &http.Client{}
req, err := http.NewRequest("GET", "https://pro-api.coinmarketcap.com/v1/cryptocurrency/listings/latest", nil)
if err != nil {
log.Print(err)
os.Exit(1)
}
resp, err := client.Do(req)
if err != nil {
fmt.Println("Error sending request to server")
os.Exit(1)
}
respBody, _ := ioutil.ReadAll(resp.Body)
fmt.Println(string(respBody)) // This is the final bit where I want to send this back to the client.
}
Your function is a HandlerFunc, which contains the ResponseWriter interface, in your case it's w.
So, you can write data using http.ResponseWriter:
func Fetch(w http.ResponseWriter, r *http.Request) {
client := &http.Client{}
req, err := http.NewRequest("GET", "https://pro-api.coinmarketcap.com/v1/cryptocurrency/listings/latest", nil)
if err != nil {
log.Print(err)
os.Exit(1)
}
resp, err := client.Do(req)
if err != nil {
fmt.Println("Error sending request to server")
os.Exit(1)
}
respBody, _ := ioutil.ReadAll(resp.Body)
// Here:
w.WriteHeader(resp.StatusCode)
w.Write(respBody)
}
You can use use io.Copy(w, resp.Body) instead, remember to close body using defer resp.Body.Close().
You can simply copy the contents of the response body to the response writer:
io.Copy(w,resp.Body)
Since you can only read the body once, the solution above will not allow you to get the body. If you also want to log it, or process it somehow, you can read it and then write it to the response writer.
respBody, _ := ioutil.ReadAll(resp.Body)
fmt.Println(string(respBody))
w.Write(respBody)

How to return a body from another request

req, err := http.NewRequest("GET", "https://api.github.com/repos/octocat/Hello-World/pulls/1347", nil)
req.Header.Set("Accept", "application/vnd.github.v3.patch")
if err != nil {
check(err)
}
body, err := ioutil.ReadAll(req.Body)
ctxt.JSON(http.StatusOK, body)
Here I need to send api response from body of github api. But here I'm getting the following error:
"runtime error: invalid memory address or nil pointer dereference"
You're creating a new GET request with a nil body. See the function signature for http.NewRequest
func NewRequest(method, url string, body io.Reader) (*Request, error)
so when you access resp.Body, of course it's going to be nil.
Also, http.NewRequest just returns a request, it doesn't actually perform it.
To actually make the GET request with your request, you need to pass it to a http client's Do method. Like so:
response, err := http.DefaultClient.Do(resp)
EDIT: I would also add that naming your request as resp is confusing. I would recommend renaming the variable to req or request
this code will solve your issues.
client := &http.Client{}
apiURL := "https://api.github.com/repos/octocat/Hello-World/pulls/1347"
req, err := http.NewRequest("GET", apiURL, nil)
if err != nil {
check(err)
}
req.Header.Add("Accept", "application/vnd.github.v3.patch")
response, err := client.Do(req)
if err != nil {
check(err)
}
defer response.Body.Close()
contents, err := ioutil.ReadAll(response.Body)
if err != nil {
check(err)
}
ctxt.JSON(http.StatusOK, string(contents))

How to duplicate a cURL command with file as body

is there anyone can help me to converting this cURL command to Go?
curl -X PUT -H 'Content-Type: image/jpg' \
-H "Content-Length: 132093" \
-T "/Users/ikmal/Downloads/catcute.jpg" \
"https://hootsuite-video.s3.amazonaws.com/production/18395606_a9245dd7-73d6-4392-af4a-1cd9bf359cfb.jpg?AWSAccessKeyId=AKIAIM7ASX2JTE3ZFAAA&Expires=1543304067&Signature=n9sZkQ%2BF1DGuiYHqixrrvmoxIXQ%3D"
I already try generator like:
1. https://mholt.github.io/curl-to-go/
it created Go code like this:
// Generated by curl-to-Go: https://mholt.github.io/curl-to-go
req, err := http.NewRequest("PUT", "https://hootsuite-video.s3.amazonaws.com/production/18395606_a9245dd7-73d6-4392-af4a-1cd9bf359cfb.jpg?AWSAccessKeyId=AKIAIM7ASX2JTE3ZFAAA&Expires=1543304067&Signature=n9sZkQ%2BF1DGuiYHqixrrvmoxIXQ%3D", nil)
if err != nil {
// handle err
}
req.Header.Set("Content-Type", "image/jpg")
req.Header.Set("Content-Length", "132093")
resp, err := http.DefaultClient.Do(req)
if err != nil {
// handle err
}
defer resp.Body.Close()
2. https://curl.trillworks.com/
it created Go code like this:
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
)
func main() {
client := &http.Client{}
req, err := http.NewRequest("PUT", "https://hootsuite-video.s3.amazonaws.com/production/18395606_a9245dd7-73d6-4392-af4a-1cd9bf359cfb.jpg?AWSAccessKeyId=AKIAIM7ASX2JTE3ZFAAA&Expires=1543304067&Signature=n9sZkQ%2BF1DGuiYHqixrrvmoxIXQ%3D", nil)
if err != nil {
log.Fatal(err)
}
req.Header.Set("Content-Type", "image/jpg")
req.Header.Set("Content-Length", "132093")
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
bodyText, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s\n", bodyText)
}
But neither gives me proper code because when I run those on project, it returning some error, because it doesn't generate this part:
-T "/Users/ikmal/Downloads/catcute.jpg"
and generate it as "nil" instead.
UPDATE :
What I have done for this problem is adding this code:
file, err := os.Open("/Users/ikmal/Downloads/catcute.jpg")
if err != nil {
panic(err)
}
defer file.Close()
so I put the variable "file" into body request, here is my final code:
func main(){
file, err := os.Open("/Users/ikmal/Downloads/catcute.jpg")
if err != nil {
panic(err)
}
defer file.Close()
client := &http.Client{}
req, err := http.NewRequest("PUT", "https://hootsuite-video.s3.amazonaws.com/production/18395606_a9245dd7-73d6-4392-af4a-1cd9bf359cfb.jpg?AWSAccessKeyId=AKIAIM7ASX2JTE3ZFAAA&Expires=1543304067&Signature=n9sZkQ%2BF1DGuiYHqixrrvmoxIXQ%3D", file)
if err != nil {
log.Fatal(err)
}
req.Header.Set("Content-Type", "image/jpg")
req.Header.Set("Content-Length", "132093")
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
bodyText, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s\n", bodyText)
}
then when I build and run this program, it returning response like this:
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>NotImplemented</Code>
<Message>A header you provided implies functionality that is not implemented</Message>
<Header>Transfer-Encoding</Header>
<RequestId>FBD2CEAF71EA4DEA</RequestId>
<HostId>K6hDrHIJr5YtoIBn2d64bfuLBgs6F17gKQV9jrTJ31X987A5gshhqtnKDs3lW2uSliBJwk1pri4=</HostId>
</Error>
It seems the error came from headers problem, but I'm sure I type the header correctly. Is there something wrong that I missed?
You must using multipart to upload content of image.
f, err := os.Open("./Users/ikmal/Downloads/catcute.jpg")
if err != nil{
log.Fatal(err)
}
defer f.Close()
var buf = new(bytes.Buffer)
writer := multipart.NewWriter(buf)
part, _ := writer.CreateFormFile("image", "dont care about name")
io.Copy(part, f)
writer.Close()
req, _ := http.NewRequest("POST", "https://hootsuite-video.s3.amazonaws.com/production/18395606_a9245dd7-73d6-4392-af4a-1cd9bf359cfb.jpg?AWSAccessKeyId=AKIAIM7ASX2JTE3ZFAAA&Expires=1543304067&Signature=n9sZkQ%2BF1DGuiYHqixrrvmoxIXQ%3D", buf)
req.Header.Set("Content-Type", writer.FormDataContentType())
client := &http.Client{}
res, _ := client.Do(req)
defer res.Body.Close()
b, _ := ioutil.ReadAll(res.Body)
fmt.Println(string(b))

Resources