Go lang RPC return EOF error - go

I'm using http to call RPC with code below
func (c *CallClient) Wallet(method string, req, rep interface{}) error {
client := &http.Client{}
data, _ := EncodeClientRequest(method, req)
reqest, _ := http.NewRequest("POST", c.endpoint, bytes.NewBuffer(data))
resp, err := client.Do(reqest)
if err != nil {
return err
}
defer resp.Body.Close()
io.Copy(ioutil.Discard, resp.Body)
return DecodeClientResponse(resp.Body, rep)
}
with EncodeClientRquest && DecodeClientResponse
// EncodeClientRequest encodes parameters for a JSON-RPC client request.
func EncodeClientRequest(method string, args interface{}) ([]byte, error) {
c := &clientRequest{
Version: "2.0",
Method: method,
Params: [1]interface{}{args},
Id: uint64(rand.Int63()),
}
return json.Marshal(c)
}
// DecodeClientResponse decodes the response body of a client request into
// the interface reply.
func DecodeClientResponse(r io.Reader, reply interface{}) error {
var c clientResponse
if err := json.NewDecoder(r).Decode(&c); err != nil {
return err
}
if c.Error != nil {
return fmt.Errorf("%v", c.Error)
}
if c.Result == nil {
return errors.New("result is null")
}
return json.Unmarshal(*c.Result, reply)
}
And I got error EOF.

This line:
io.Copy(ioutil.Discard, resp.Body)
reads the whole resp.Body, leaving the reader with no more bytes to be read. Therefore any successive calls to resp.Body.Read will return EOF and the json.Decoder.Decode method does use the io.Reader.Read method when decoding the given reader's content, so...
And since resp.Body is an io.ReadCloser, which is an interface that does not support "rewinding", and you want to read the body content more than once (ioutil.Discard and json.Decode), you'll have to read the body into a variable that you can re-read afterwards. It's up to you how you do that, a slice of bytes, or bytes.Reader, or something else.
Example using bytes.Reader:
func (c *CallClient) Wallet(method string, req, rep interface{}) error {
client := &http.Client{}
data, err := EncodeClientRequest(method, req)
if err != nil {
return err
}
reqest, err := http.NewRequest("POST", c.endpoint, bytes.NewBuffer(data))
if err != nil {
return err
}
resp, err := client.Do(reqest)
if err != nil {
return err
}
defer resp.Body.Close()
// get a reader that can be "rewound"
buf := bytes.NewBuffer(nil)
if _, err := io.Copy(buf, resp.Body); err != nil {
return err
}
br := bytes.NewReader(buf.Bytes())
if _, err := io.Copy(ioutil.Discard, br); err != nil {
return err
}
// rewind
if _, err := br.Seek(0, 0); err != nil {
return err
}
return DecodeClientResponse(br, rep)
}

Related

Post multi-part form data for audio file

I'm wrote a function that posts multipart form data to an endpoint but doesn't seem to be working properly. Here's the code
func Upload(filePath string) error {
client := &http.Client{
Timeout: time.Second * 10,
}
// New multipart writer.
body := &bytes.Buffer{}
file, err := os.Open(filePath)
if err != nil {
return err
}
writer := multipart.NewWriter(body)
fw, err := writer.CreateFormFile("file", filepath.Base(filePath))
if err != nil {
return err
}
_, err = io.Copy(fw, bufio.NewReader(file))
if err != nil {
return err
}
// Close multipart writer.
writer.Close()
req, err := http.NewRequest("POST", "http://localhost:5050/upload", bytes.NewReader(body.Bytes()))
if err != nil {
return err
}
req.Header.Set("Content-Type", writer.FormDataContentType())
rsp, _ := client.Do(req)
if rsp == nil {
return fmt.Errorf("failed to upload")
}
if rsp.StatusCode != http.StatusOK {
fmt.Printf("Request failed with response code: %d", rsp.StatusCode)
}
fmt.Println("rsp: ", rsp.StatusCode)
return nil
}
The handler on the API receiving the POST is this:
func (m *MetadataService) uploadHandler(res http.ResponseWriter, req *http.Request) {
file, handler, err := req.FormFile("file")
if err != nil {
panic(err) //dont do this
}
defer file.Close()
fmt.Println("req: ", req)
// Create a buffer to store the header of the file in
fileHeader := make([]byte, 512)
// Copy the headers into the FileHeader buffer
if _, err := file.Read(fileHeader); err != nil {
panic(err) //dont do this
}
// set position back to start.
if _, err := file.Seek(0, 0); err != nil {
panic(err) //dont do this
}
// copy example
f, err := os.OpenFile(handler.Filename, os.O_WRONLY|os.O_CREATE, 0666)
if err != nil {
panic(err) //please dont
}
defer f.Close()
buf := bytes.NewBuffer(nil)
if _, err := io.Copy(buf, file); err != nil {
panic(err)
}
fmt.Println("filename:", handler.Filename)
fmt.Println("file.(Sizer).Size():", file.(Sizer).Size())
fmt.Println("contentType:", http.DetectContentType(fileHeader))
io.WriteString(res, id)
}
The code panics when reading the fileHeader thought. The content type when sent is "application/octet-stream" when I'd expect it to be "audio/mpeg" since i'm uploading a song. Not sure what's wrong with the aforementioned Upload function. Need help! Thank you!

Zip a Directory and not Have the Result Saved in File System

I am able to zip a file using logic similar to the zip writer seen here.
This results in an array of bytes ([]byte) being created within the bytes.Buffer object that is returned. I would just like to know if there is there any way I can upload this 'zipped' array of bytes to an API endpoint that expects a 'multipart/form-data' request body (without having to save it locally).
Supplementary information:
I have code that utilizes this when compressing a folder. I am able to successfully execute an HTTP POST request with the zip file to the endpoint with this logic.
However, this unfortunately saves zipped files in a user's local file system. I would like to try to avoid this :)
You can create multipart writer and write []byte zipped data into field with field name you like and file name like below.
func addZipFileToReq(zipped []byte) (*http.Request, error){
body := bytes.NewBuffer(nil)
writer := multipart.NewWriter(body)
part, err := writer.CreateFormFile(`fileField`, `filename`)
if err != nil {
return nil, err
}
_, err = part.Write(zipped)
if err != nil {
return nil, err
}
err = writer.Close()
if err != nil {
return nil, err
}
r, err := http.NewRequest(http.MethodPost, "https://example.com", body)
if err != nil {
return nil, err
}
r.Header.Set("Content-Type", writer.FormDataContentType())
return r, nil
}
If you want to stream-upload the zip, you should be able to do so with io.Pipe. The following is an incomplete and untested example to demonstrate the general idea. To make it work you'll need to modify it and potentially fix whatever bugs you encounter.
func UploadReader(r io.Reader) error {
req, err := http.NewRequest("POST", "<UPLOAD_URL>", r)
if err != nil {
return err
}
// TODO set necessary headers (content type, auth, etc)
res, err := http.DefaultClient.Do(req)
if err != nil {
return err
} else if res.StatusCode != 200 {
return errors.New("not ok")
}
return nil
}
func ZipDir(dir string, w io.Writer) error {
zw := zip.NewWriter(w)
defer zw.Close()
return filepath.Walk(dir, func(path string, fi os.FileInfo, err error) error {
if err != nil {
return err
}
if !fi.Mode().IsRegular() {
return nil
}
header, err := zip.FileInfoHeader(fi)
if err != nil {
return err
}
header.Name = path
header.Method = zip.Deflate
w, err := zw.CreateHeader(header)
if err != nil {
return err
}
f, err := os.Open(path)
if err != nil {
return err
}
defer f.Close()
if _, err := io.Copy(w, f); err != nil {
return err
}
return nil
})
}
func UploadDir(dir string) error {
r, w := io.Pipe()
ch := make(chan error)
wg := sync.WaitGroup{}
wg.Add(1)
go func() {
defer wg.Done()
defer w.Close()
if err := ZipDir(dir, w); err != nil {
ch <- err
}
}()
wg.Add(1)
go func() {
defer wg.Done()
defer r.Close()
if err := UploadReader(r); err != nil {
ch <- err
}
}()
go func() {
wg.Wait()
close(ch)
}()
return <-ch
}

Golang API Post upload file

I'm new to golang and I'm trying to write a function that uploads a file with a post request to API server. I try Post API in Postman, it is OK but in my code I have some error like this image
This is my golang code:
func (c *Client) PostUploadFile(endpoint string, params map[string]string) []byte {
url := "/examples/image/text.txt"
body := &bytes.Buffer{}
writer := multipart.NewWriter(body)
// Open the file
file, err := os.Open(url)
if err != nil {
// return nil, err
}
// Close the file later
defer file.Close()
part, err := writer.CreateFormFile("file", filepath.Base(url))
_, err = io.Copy(part, file)
if err != nil {
fmt.Println(err)
// return nil, err
}
for key, val := range params {
_ = writer.WriteField(key, val)
}
err = writer.Close()
if err != nil {
// return nil, err
}
fmt.Println("Data request:")
fmt.Println(body)
fmt.Println("Endpoint:")
fmt.Println(c.BaseUrl + endpoint)
req, requestErr := http.NewRequest("POST", c.BaseUrl+endpoint, body)
if requestErr != nil {
log.Fatalln(requestErr)
}
req.Header.Add("auth_token", c.AuthToken)
req.Header.Add("accept", "application/json")
// req.Header.Add("Content-Type", "application/json")
req.Header.Add("Content-Type", writer.FormDataContentType())
client := &http.Client{}
fmt.Println("Response:")
resp, err := client.Do(req)
fmt.Println(resp)
if err != nil {
log.Println(err)
return []byte(``)
}
return c.parseBody(resp)
}
and this is a param formdata in body:
fmt.Printf("%+v\n", c.UploadImage(map[string]string{
"file": "/examples/image/text.txt",
"wfs_id": "30578",
"id": "59284",
"element_id": "119726",
}))

How to get response with JWT in Golang

I'm trying get the response from an API that uses JSON Web token, I need use the header : {
Authorization: "Bearer token"
}
But I would like to keep the timeout of the http.Client that I'm using. How could I do it?
var myClient = &http.Client{Timeout: 10 * time.Second}
func getJson(url string, target interface{}) error {
r, err := myClient.Get(url)
if err != nil {
return err
}
defer r.Body.Close()
return json.NewDecoder(r.Body).Decode(target)
}
net/http.Request has a Header field that you can directly edit, but this means you can't use the shortcut client.Get method. Something more like:
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
return err
}
req.Header = map[string][]string{
"Authorization": {fmt.Sprintf("Bearer %s", jwt)},
}
r, err := myClient.Do(req)
...
You could do something like this,
func getJson(url string, target interface{}) error {
req, err := http.NewRequest(http.MethodGet, url, nil)
if nil != err {
return err
}
r, err := myClient.Do(req)
if err != nil {
return err
}
defer r.Body.Close()
return json.NewDecoder(r.Body).Decode(target)
}
Alternatively , you could also use context to control the request timeout
func getJsonWithContext(url string, target interface{}) error {
req, err := http.NewRequest(http.MethodGet, url, nil)
if nil != err {
return err
}
ctx, cancel := context.WithTimeout(context.Background(), time.Second*1)
defer cancel()
reqWithContext := req.WithContext(ctx)
r, err := myClient.Do(reqWithContext)
if err != nil {
return err
}
defer r.Body.Close()
return json.NewDecoder(r.Body).Decode(target)
}

net/http: http: ContentLength=222 with Body length 0

I'm trying to retry a request if there is a connection/proxy error. For some reasons I keep getting this error which doesn't seem to recover regardless the attepts to retry the request:
Post https://m.somewebsite.co.uk/api/di/34433: http: ContentLength=222 with Body length 0
Am I doing something wrong? My first suspicion is that the http.Request is consumed somehow so on the next attempts it's no longer good. Should I manage a copy?
func Post(URL string, form url.Values, cl *http.Client) ([]byte, error) {
req, err := http.NewRequest("POST", URL, strings.NewReader(form.Encode()))
if err != nil {
log.Error(err)
return nil, err
}
req.Header.Set("User-Agent", ua)
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
rsp, err := do(cl, req)
if err != nil {
return nil, err
}
defer rsp.Body.Close()
b, err := ioutil.ReadAll(rsp.Body)
if err != nil {
log.Error(err)
return nil, err
}
return b, nil
}
func do(cl *http.Client, req *http.Request)(*http.Response, error){
rsp, err := cl.Do(req)
for i := 0; IsErrProxy(err); i++ {
log.Errorf("Proxy is slow or down ")
time.Sleep(6 * time.Second)
5t rsp, err = cl.Do(&ncp)
if err == nil{
return rsp, nil
}
if i > 10 {
return nil, fmt.Errorf("after %v tries error: %v", i, err)
}
}
return rsp, err
}
The problem is that the request body is read to the end on the first call to Do(). On subsequent calls to Do(), no data is read from the response body.
The fix is to move the creation of the body reader inside the for loop. This requires that the request also be created inside the for loop.
func Post(URL string, form url.Values, cl *http.Client) ([]byte, error) {
body := form.Encode()
for i := 0; i < 10; i++ {
req, err := http.NewRequest("POST", URL, strings.NewReader(body))
if err != nil {
log.Error(err)
return nil, err
}
req.Header.Set("User-Agent", ua)
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
rsp, err := cl.Do(req)
if err == nil {
defer rsp.Body.Close()
b, err := ioutil.ReadAll(rsp.Body)
if err != nil {
log.Error(err)
return nil, err
}
return b, nil
}
if !IsErrorProxy(err) {
return nil, err
}
log.Errorf("Proxy is slow or down ")
time.Sleep(6 * time.Second)
}
return nil, fmt.Errorf("after 10 tries error: %v", err)
}
Let's see what does http.NewRequest do
rc, ok := body.(io.ReadCloser)
if !ok && body != nil {
rc = ioutil.NopCloser(body)
}
// The host's colon:port should be normalized. See Issue 14836.
u.Host = removeEmptyPort(u.Host)
req := &Request{
Method: method,
URL: u,
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: make(Header),
Body: rc,
Host: u.Host,
}
body is type of io.Reader, and convert to io.ReaderCloser by ioutil.NopCloser. As #Cerise Limón said, the Request.Body had been read and the stream is closer, so when you Do() again, the Body Length is 0.
So, we could reset the Request.Body before invoke Do.
func Post(URL string, form url.Values, cl *http.Client) ([]byte, error) {
requestBodyString := form.Encode()
req, err := http.NewRequest("POST", URL, strings.NewReader(requestBodyString))
// ...
rsp, err := do(cl, req, requestBodyString)
//...
return b, nil
}
func do(cl *http.Client, req *http.Request, requestBodyString string)(*http.Response, error){
rsp, err := cl.Do(req)
for i := 0; IsErrProxy(err); i++ {
log.Errorf("Proxy is slow or down ")
time.Sleep(6 * time.Second)
// reset Request.Body
req.Body = ioutil.NopCloser(strings.NewReader(requestBodyString))
rsp, err = cl.Do(&req)
if err == nil{
return rsp, nil
}
if i > 10 {
return nil, fmt.Errorf("after %v tries error: %v", i, err)
}
}
return rsp, err
}
I found a way to do this without re-creating the request every time. here's a sample code, which is a very slight modification of #Cerise Limón and is similar to #Rambo's code in that it creates the request only once:
func Post(URL string, data *bytes.Buffer, cl *http.Client) ([]byte, error) {
var err error
req, err := http.NewRequest("POST", URL, ioutil.NopCloser(data))
if err != nil {
log.Errorf("Unable to create the request: %v", err)
return nil, err
}
req.Header.Set("User-Agent", ua)
for i := 0; i < 10; i++ {
rsp, err := cl.Do(req)
if err == nil && rsp.StatusCode == 200 {
defer rsp.Body.Close()
b, err := ioutil.ReadAll(rsp.Body)
if err != nil {
fmt.Printf("Error: %v", err)
return nil, err
}
return b, nil
}
log.Errorf("Proxy is slow or down %v", err)
time.Sleep(1 * time.Second)
}
return nil, fmt.Errorf("after 10 tries error: %v", err)
}
func main() {
client := http.Client{}
data := []byte{0, 1, 2, 3, 4}
Post("http://server/my/api/resource/", bytes.NewBuffer(data), &client)
}

Resources