HTTP Post containing binary data in golang [duplicate] - go

This question already has answers here:
POST data using the Content-Type multipart/form-data
(7 answers)
Closed 8 months ago.
I hope I can explain this right. I'm trying to make an HTTP post request that contains binary data (a file). This is for DeepStack image processing. In Python I have the following working:
image_data = open(file,"rb").read()
try:
response = requests.post("http://deepstack.local:82/v1/vision/detection",files={"image":image_data},timeout=15).json()
In Go, I started with the basic example from here: https://golangtutorial.dev/tips/http-post-json-go/
Modifying this a bit for my use, the relevant lines are:
data, err := ioutil.ReadFile(tempPath + file.Name())
if err != nil {
log.Print(err)
}
httpposturl := "http://deepstack.local:82/v1/vision/custom/combined"
fmt.Println("HTTP JSON POST URL:", httpposturl)
var jsonData = []byte(`{"image": ` + data + `}`)
request, error := http.NewRequest("POST", httpposturl, bytes.NewBuffer(jsonData))
request.Header.Set("Content-Type", "application/json; charset=UTF-8")
This results in an error:
invalid operation: `{"image": ` + data (mismatched types untyped string and []byte)`
the "data" variable at this point is []uint8 ([]byte). I realize, at a high level, what is wrong. I'm trying to join two data types that are not the same. That's about it though. I've tried a bunch of stuff that I'm pretty sure anyone familiar with Go would immediately realize was wrong (declaring jsonData as a byte, converting data to a string, using os.Open instead of ioutil.ReadFile, etc.). I'm just kind of stumbling around blind though. I can't find an example that doesn't use a plain string as the JSON data.
I would appreciate any thoughts.
--- ANSWER ---
I'm marking Dietrich Epp's answer as accepted, because he gave me what I asked for. However, RedBlue in the comments gave me what I actually needed. Thank you both. The code below is modified just a bit from this answer: https://stackoverflow.com/a/56696333/2707357
Change the url variable to your DeepStack server, and the file name to one that actually exists, and the response body should return the necessary information.
package main
import (
"bytes"
"fmt"
"io"
"io/ioutil"
"mime/multipart"
"net/http"
"os"
)
func createMultipartFormData(fieldName, fileName string) (bytes.Buffer, *multipart.Writer) {
var b bytes.Buffer
var err error
w := multipart.NewWriter(&b)
var fw io.Writer
file := mustOpen(fileName)
if fw, err = w.CreateFormFile(fieldName, file.Name()); err != nil {
fmt.Println("Error: ", err)
}
if _, err = io.Copy(fw, file); err != nil {
fmt.Println("Error: ", err)
}
w.Close()
return b, w
}
func mustOpen(f string) *os.File {
r, err := os.Open(f)
if err != nil {
pwd, _ := os.Getwd()
fmt.Println("PWD: ", pwd)
panic(err)
}
return r
}
func main() {
url := "http://deepstack.local:82/v1/vision/custom/combined"
b, w := createMultipartFormData("image", "C:\\go_sort\\temp\\person.jpg")
req, err := http.NewRequest("POST", url, &b)
if err != nil {
return
}
// Don't forget to set the content type, this will contain the boundary.
req.Header.Set("Content-Type", w.FormDataContentType())
client := &http.Client{}
response, error := client.Do(req)
if err != nil {
panic(error)
}
defer response.Body.Close()
fmt.Println("response Status:", response.Status)
fmt.Println("response Headers:", response.Header)
body, _ := ioutil.ReadAll(response.Body)
fmt.Println("response Body:", string(body))
}

It's really such a small error. This is all your question boils down to, as far as I can tell:
var data []byte // with some value
jsonData := []byte(`{"image": ` + data + `}`)
All you have to do is change this to use append() or something similar:
jsonData := append(
append([]byte(`{"image": `), data...),
'}')
The reason is that you can't use + to concatenate []byte in Go. You can use append(), though.

Related

Golang bufio from websocket breaking after first read

I am trying to stream JSON text from a websocket. However after an initial read I noticed that the stream seems to break/disconnect. This is from a Pleroma server (think: Mastodon). I am using the default Golang websocket library.
package main
import (
"bufio"
"fmt"
"log"
"golang.org/x/net/websocket"
)
func main() {
origin := "https://poa.st/"
url := "wss://poa.st/api/v1/streaming/?stream=public"
ws, err := websocket.Dial(url, "", origin)
if err != nil {
log.Fatal(err)
}
s := bufio.NewScanner(ws)
for s.Scan() {
line := s.Text()
fmt.Println(line)
}
}
After the initial JSON text response, the for-loop breaks. I would expect it to send a new message every few seconds.
What might be causing this? I am willing to switch to the Gorilla websocket library if I can use it with bufio.
Thanks!
Although x/net/websocket connection has a Read method with the same signature as the Read method in io.Reader, the connection does not work like an io.Reader. The connection will not work as you expect when wrapped with a bufio.Scanner.
The poa.st endpoint sends a stream of messages where each message is a JSON document. Use the following code to read the messages using the Gorilla package:
url := "wss://poa.st/api/v1/streaming/?stream=public"
ws, _, err := websocket.DefaultDialer.Dial(url, nil)
if err != nil {
log.Fatal(err)
}
defer ws.Close()
for {
_, p, err := ws.ReadMessage()
if err != nil {
log.Fatal(err)
}
// p is a []byte containing the JSON document.
fmt.Printf("%s\n", p)
}
The Gorilla package has a helper method for decoding JSON messages. Here's an example of how to use that method.
url := "wss://poa.st/api/v1/streaming/?stream=public"
ws, _, err := websocket.DefaultDialer.Dial(url, nil)
if err != nil {
log.Fatal(err)
}
defer ws.Close()
for {
// The JSON documents are objects containing two fields,
// the event type and the payload. The payload is a JSON
// document itself.
var e struct {
Event string
Payload string
}
err := ws.ReadJSON(&e)
if err != nil {
log.Fatal(err)
}
// TODO: decode e.Payload based on e.Event
}

Go-Gin read request body many times

I am trying to restore the context with it's data after performing validation on it's data.I need the data to keep moving as need it later on in the next function.
I am new to golang and the below code is as far I could go. any help and a better approach is much appreciated.
thanks in advance.
the validation middleware
func SignupValidator(c *gin.Context) {
// Read the Body content
// var bodyBytes []byte
// if c.Request.Body != nil {
// bodyBytes, _ = ioutil.ReadAll(c.Request.Body)
// }
var user entity.User
if err := c.ShouldBindJSON(&user); err != nil {
validate := validator.New()
if err := validate.Struct(&user); err != nil {
c.JSON(http.StatusBadRequest, gin.H{
"error": err.Error(),
})
c.Abort()
return
}
// c.Request.Body = ioutil.NopCloser(bytes.NewBuffer(bodyBytes))
}
// Read the Body content
var bodyBytes []byte
if c.Request.Body != nil {
bodyBytes, _ = ioutil.ReadAll(c.Request.Body)
}
fmt.Println(string(bodyBytes)) // this empty
c.Next()
}
route
auth.POST("login", gin.Logger(), validations.SignupValidator, func(ctx *gin.Context) {
ctx.JSON(200, videoController.Save(ctx))
})
You can try this.
ByteBody, _ := ioutil.ReadAll(c.Request.Body)
c.Request.Body = ioutil.NopCloser(bytes.NewBuffer(ByteBody))
You can then use ByteBody however you want without side-effects on c.Request.Body
Here is an example of reading body twice with ShouldBindBodyWith, check it:
package main
import (
"log"
"net/http"
"github.com/gin-gonic/gin"
"github.com/gin-gonic/gin/binding"
)
type ParamsOne struct {
Username string `json:"username"`
}
type ParamsTwo struct {
Username string `json:"username"`
}
func main() {
r := gin.New()
r.POST("/", func(c *gin.Context) {
var f ParamsOne
// Read ones
if err := c.ShouldBindBodyWith(&f, binding.JSON); err != nil {
log.Printf("%+v", err)
}
log.Printf("%+v", f)
var ff ParamsTwo
if err := c.ShouldBindBodyWith(&ff, binding.JSON); err != nil {
log.Printf("%+v", err)
}
log.Printf("%+v", ff)
c.IndentedJSON(http.StatusOK, f)
})
r.Run(":4000")
}
Output:
$example: ./example
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
- using env: export GIN_MODE=release
- using code: gin.SetMode(gin.ReleaseMode)
[GIN-debug] POST / --> main.main.func1 (1 handlers)
[GIN-debug] Listening and serving HTTP on :4000
2020/07/05 10:47:03 {Username:somename}
2020/07/05 10:47:03 {Username:somename}
As #Philidor has shown ShouldBindBodyWith should do the trick, in my case I decided to go with something similar to #spehlivan, because of two reasons:
ShouldBindBodyWith requires that the following binds are also ShouldBindBodyWith, it means I need to change all my previous code, which uses c.Bind
You need to explicitly tell to ShouldBindBodyWith the binding type you are trying to do, JSON, Form, ProtoBuf, etc, other binds like c.Bind detects it automatically.
This is what it looks like:
var input models.SomeInput
bodyCopy := new(bytes.Buffer)
// Read the whole body
_, err := io.Copy(bodyCopy, c.Request.Body)
if err != nil {
log.Println(err)
c.JSON(http.StatusBadRequest, gin.H{"error": "Error reading API token"})
c.Abort()
return
}
bodyData := bodyCopy.Bytes()
// Replace the body with a reader that reads from the buffer
c.Request.Body = ioutil.NopCloser(bytes.NewReader(bodyData))
err = c.Bind(&input)
// Some code here...
// Replace the body with a reader that reads from the buffer
c.Request.Body = ioutil.NopCloser(bytes.NewReader(bodyData))
Pay attention I replaced the c.Request.Body twice, for the bind in that code snippet and then at the end, for the next bind, in another place of my code (this snippet is from a middleware, the next bind is called from the controller).
In my case I needed to do this because the API Token is sent in the request body, which I don't recommend, it should be sent in the request header.

how to fix 'name' is undefined on object error? [closed]

Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 3 years ago.
Improve this question
I'm making an API post request to bigpanda using Go.
https://docs.bigpanda.io/reference#create-plan
I have below code and when I try to make API post getting name is undefined on object error.
2019/08/23 18:38:04 {
"status" : "failure",
"error" : "{\"obj\":[{\"msg\":[\"'name' is undefined on object: {\\\"maintenance_plan\\\":{\\\"name\\\":\\\"\\\\\\\"name\\\\\\\": \\\\\\\"scheduled host maintenance\\\\\\\",\\\",\\\"condition\\\":\\\"\\\\\\\"condition\\\\\\\": {\\\\\\\"=\\\\\\\": [\\\\\\\"host\\\\\\\", \\\\\\\"prod-api-1\\\\\\\"]},\\\",\\\"start\\\":\\\"\\\\\\\"start\\\\\\\": \\\\\\\"1566514810\\\\\\\",\\\",\\\"end\\\":\\\"\\\\\\\"end\\\\\\\": \\\\\\\"156651600\\\\\\\"\\\"}}\"],\"args\":[]}]}"
}
package main
import (
"bytes"
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
)
type Maintenace_Plan struct {
Name string `json:"name"`
//Condition map[string]map[string][]string `json:condition`
Condition string `json:"condition"`
Start string `json:"start"`
End string `json:"end"`
}
type Payload struct {
Maintenace_Plan Maintenace_Plan `json:"maintenance_plan"`
}
func main() {
name := `"name": "scheduled host maintenance",`
create_plan := `"condition": {"=": ["host", "prod-api-1"]},`
start_time := `"start": "1566514810",`
end_time := `"end": "156651600"`
data := Payload{
Maintenace_Plan: Maintenace_Plan{
Name: name,
Condition: create_plan,
Start: start_time,
End: end_time,
},
}
payloadBytes, err := json.Marshal(data)
if err != nil {
fmt.Println(err)
}
body := bytes.NewReader(payloadBytes)
req, err := http.NewRequest("POST", "https://api.bigpanda.io/resources/v2.0/maintenance-plans", body)
if err != nil {
log.Fatal(err)
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Authorization", "Bearer <token>")
resp, err := http.DefaultClient.Do(req)
defer resp.Body.Close()
body_1, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
log.Println(string(body_1))
}
Looks like body is incorrect.
Is there any way to fix the code.
That error message is probably being returned from the API Call you're making and not the JSON Marshal. I suspect it's due to the way you're marshalling your payload - you're writing JSON to the fields and then JSON marshalling it so you end up with a payload that looks like:
{"maintenance_plan":{"name":"\"name\": \"scheduled host maintenance\",","condition":"\"condition\": {\"=\": [\"host\", \"prod-api-1\"]},","start":"\"start\": \"1566514810\",","end":"\"end\": \"156651600\""}}
Notice the double "name: "\name\".
The way to fix it would be to do something like:
data := Payload{
MaintenancePlan: MaintenancePlan{
Name: "scheduled host maintenance",
Condition: map[string][]string{
"=": []string{"host", "prod-api-1"},
},
StartTime: "1566514810",
EndTime: "156651600",
},
}
var buf bytes.Buffer
err := json.NewEncoder(&buf).Encode(data)
if err != nil {
// Handle me
}
req, err := http.NewRequest(http.MethodPost, "https://foo/bar", &buf)
// continue
Example: https://play.golang.org/p/J6wrsLYkvwF

How to efficiently store html response to a file in golang

I'm trying to build a crawler in Golang. I'm using net/http library to download the html file from url. I'm trying to save http.resp and http.Header into file.
How to convert these two file from their respective format into string so that, it could be written to a text file.
I also see a question asked earlier on parsing a stored html response file. Parse HTTP requests and responses from text file in Go . Is there any way to save the url response in this format.
Go has an httputil package with a response dump.
https://golang.org/pkg/net/http/httputil/#DumpResponse.
The second argument of response dump is a bool of whether or not to include the body. So if you want to save just the header to a file, set that to false.
An example function that would dump the response to a file could be:
import (
"io/ioutil"
"net/http"
"net/http/httputil"
)
func dumpResponse(resp *http.Response, filename string) error {
dump, err := httputil.DumpResponse(resp, true)
if err != nil {
return err
}
return ioutil.WriteFile(filename, dump, 0644)
}
Edit: Thanks to #JimB for pointing to the http.Response.Write method which makes this a lot easier than I proposed in the beginning:
resp, err := http.Get("http://google.com/")
if err != nil{
log.Panic(err)
}
f, err := os.Create("output.txt")
defer f.Close()
resp.Write(f)
This was my first Answer
You could do something like this:
resp, err := http.Get("http://google.com/")
body, err := ioutil.ReadAll(resp.Body)
// write whole the body
err = ioutil.WriteFile("body.txt", body, 0644)
if err != nil {
panic(err)
}
This was the edit to my first answer:
Thanks to #Hector Correa who added the header part. Here is a more comprehensive snippet, targeting your whole question. This writes header followed by the body of the request to output.txt
//get the response
resp, err := http.Get("http://google.com/")
//body
body, err := ioutil.ReadAll(resp.Body)
//header
var header string
for h, v := range resp.Header {
for _, v := range v {
header += fmt.Sprintf("%s %s \n", h, v)
}
}
//append all to one slice
var write []byte
write = append(write, []byte(header)...)
write = append(write, body...)
//write it to a file
err = ioutil.WriteFile("output.txt", write, 0644)
if err != nil {
panic(err)
}
Following on the answer by #Riscie you could also pick up the headers from the response with something like this:
for header, values := range resp.Header {
for _, value := range values {
log.Printf("\t\t %s %s", header, value)
}
}

Go file downloader

I have the following code which is suppose to download file by splitting it into multiple parts. But right now it only works on images, when I try downloading other files like tar files the output is an invalid file.
UPDATED:
Used os.WriteAt instead of os.Write and removed os.O_APPEND file mode.
package main
import (
"errors"
"flag"
"fmt"
"io/ioutil"
"log"
"net/http"
"os"
"strconv"
)
var file_url string
var workers int
var filename string
func init() {
flag.StringVar(&file_url, "url", "", "URL of the file to download")
flag.StringVar(&filename, "filename", "", "Name of downloaded file")
flag.IntVar(&workers, "workers", 2, "Number of download workers")
}
func get_headers(url string) (map[string]string, error) {
headers := make(map[string]string)
resp, err := http.Head(url)
if err != nil {
return headers, err
}
if resp.StatusCode != 200 {
return headers, errors.New(resp.Status)
}
for key, val := range resp.Header {
headers[key] = val[0]
}
return headers, err
}
func download_chunk(url string, out string, start int, stop int) {
client := new(http.Client)
req, _ := http.NewRequest("GET", url, nil)
req.Header.Add("Range", fmt.Sprintf("bytes=%d-%d", start, stop))
resp, _ := client.Do(req)
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatalln(err)
return
}
file, err := os.OpenFile(out, os.O_WRONLY, 0600)
if err != nil {
if file, err = os.Create(out); err != nil {
log.Fatalln(err)
return
}
}
defer file.Close()
if _, err := file.WriteAt(body, int64(start)); err != nil {
log.Fatalln(err)
return
}
fmt.Println(fmt.Sprintf("Range %d-%d: %d", start, stop, resp.ContentLength))
}
func main() {
flag.Parse()
headers, err := get_headers(file_url)
if err != nil {
fmt.Println(err)
} else {
length, _ := strconv.Atoi(headers["Content-Length"])
bytes_chunk := length / workers
fmt.Println("file length: ", length)
for i := 0; i < workers; i++ {
start := i * bytes_chunk
stop := start + (bytes_chunk - 1)
go download_chunk(file_url, filename, start, stop)
}
var input string
fmt.Scanln(&input)
}
}
Basically, it just reads the length of the file, divides it with the number of workers then each file downloads using HTTP's Range header, after downloading it seeks to a position in the file where that chunk is written.
If you really ignore many errors like seen above then your code is not supposed to work reliably for any file type.
However, I guess I can see on problem in your code. I think that mixing O_APPEND and seek is probably a mistake (Seek should be ignored with this mode). I suggest to use (*os.File).WriteAt instead.
IIRC, O_APPEND forces any write to happen at the [current] end of file. However, your download_chunk function instances for file parts can be executing in unpredictable order, thus "reordering" the file parts. The result is then a corrupted file.
1.the sequence of the go routine is not sure。
eg. the execute result maybe as follows:
...
file length:20902
Range 10451-20901:10451
Range 0-10450:10451
...
so the chunks can't just append.
2.when write chunk datas must have a sys.Mutex
(my english is poor,please forget it)

Resources