I'm having a client-server code for http2, I want to send file from client to server. But I'm stuck on how to do that. Means how to break my file in small chucks and send it via that connection.
Any help and link will be appreciated.
Thanks
//http2 client
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
"golang.org/x/net/http2"
)
func main() {
client := http.Client{
// InsecureTLSDial is temporary and will likely be
// replaced by a different API later.
Transport: &http2.Transport{InsecureTLSDial: true},
}
// Perform the request
resp, err := client.Post("https://localhost:9191/hello/sayHello", "text/plain", bytes.NewBufferString("Hello Go!"))
if err != nil {
log.Fatalf("Failed get: %s", err)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatalf("Failed reading response body: %s", err)
}
fmt.Printf("Got response %d: %s %s", resp.StatusCode, resp.Proto, string(body))
}
Related
I am writing a code in GoLang to get data from a url but when I make http.Get() request, the response I get is as per the following.
"compile_output": "# command-line-arguments\n/usr/local/go-1.13.5/pkg/tool/linux_amd64/link: resize output file failed: truncate $WORK/b001/exe/a.out: file too large\n",
Please suggest me how to tackle this.
Code is as per the following:
import (
"fmt"
"io/ioutil"
"net/http"
)
func main() {
// define the URL to get data from
url := "my_example_url"
// use http.Get() to make a GET request to the URL
resp, err := http.Get(url)
if err != nil {
fmt.Println("Error getting data:", err)
return
}
defer resp.Body.Close()
// read the response body
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println("Error reading response:", err)
return
}
// print the response body
fmt.Println(string(body))
}
I tried reading through GoLang URl Documentations, but didn't help.
Edit: The main problem turned out to be the actual uploading process and not the deadlock that occured, which was simply caused by a misplaced wg.Wait()
I am trying to upload a file to an online file hosting service (https://anonfiles.com/) via their API. There is an upload file size limit of 20GB.
I can upload a simple text file that is around 2KB with the code below. However, if I try to do the same with a larger file, lets say, around 2MB, I get the following error from their API: No file chosen.
I thought this was because the code (below) was not waiting for the go routine to properly finish, so I added a wait group. I then got this error from Go: fatal error: all goroutines are asleep - deadlock!.
I have tried removing the WaitGroup below that seems to be causing the deadlock; but then the code below the go routine will run before the go routine is actually finished.
With the WaitGroup removed, I can still upload files that are KB in size, but files that are larger do not upload to the file hosting correctly, since I receive the No file chosen error from their API.
package main
import (
"fmt"
"io"
"log"
"math/rand"
"mime/multipart"
"net/http"
"os"
"sync"
"time"
)
func main() {
client := http.Client{}
// Upload a >2MB wallpaper.
file, err := os.Open("./wallpaper.jpg")
if err != nil {
log.Fatal(err)
}
defer file.Close()
reader, writer := io.Pipe()
multipart := multipart.NewWriter(writer)
/*
Added Waitgroup to make sure the routine properly finishes. Instead, causes deadlock.
wg := new(sync.WaitGroup)
wg.Add(1)
*/
go func() {
fmt.Println("Starting Upload...")
defer wg.Done()
defer writer.Close()
defer multipart.Close()
part, err := multipart.CreateFormFile("file", file.Name())
if err != nil {
log.Fatal(err)
}
fmt.Println("Copying...")
if _, err = io.Copy(part, file); err != nil {
log.Fatal(err)
}
}()
fmt.Println("The code below will run before the goroutine is finished; without the WaitGroup.")
req, err := http.NewRequest(http.MethodPost, "https://api.anonfiles.com/upload", reader)
if err != nil {
log.Fatal(err)
}
req.Header.Add("Content-Type", multipart.FormDataContentType())
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
wg.Wait()
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
fmt.Println(string(body))
}
I have researched several issues, but none seem to apply to my problem. What is causing this to lock up? What can be done differently? Perhaps this is some rookie mistake, any suggestions or help would be appreciated.
TL;DR
Set the Content-Length header of the request.
A working demo is attached to the end of this answer.
Debugging
I think the deadlock issue is not important here. Your purpose is to upload files to https://anonfiles.com/. So I will focus on debugging the uploading issue.
First, let's upload a file with curl:
curl -F "file=#test.txt" https://api.anonfiles.com/upload
It works.
Then let's upload the same file with your demo, it fails with the misleading response:
{
"status": false,
"error": {
"message": "No file chosen.",
"type": "ERROR_FILE_NOT_PROVIDED",
"code": 10
}
}
Now let's replace the target https://api.anonfiles.com/upload with https://httpbin.org/post so that we can compare the requets:
{
"args": {},
"data": "",
"files": {
"file": "aaaaaaaaaa\n"
},
"form": {},
"headers": {
- "Accept": "*/*",
- "Content-Length": "197",
- "Content-Type": "multipart/form-data; boundary=------------------------bd4a81e725230fa6",
+ "Accept-Encoding": "gzip",
+ "Content-Type": "multipart/form-data; boundary=2d4e7969789ed6ef6ff3e7b815db3aa040fd3994a34fbaedec85240dc5af",
"Host": "httpbin.org",
- "User-Agent": "curl/7.81.0",
- "X-Amzn-Trace-Id": "Root=1-63747739-2c1dab1b122b7e3a4db8ca79"
+ "Transfer-Encoding": "chunked",
+ "User-Agent": "Go-http-client/2.0",
+ "X-Amzn-Trace-Id": "Root=1-63747872-2fbc85f81c6dde7e5b2091c4"
},
"json": null,
"origin": "47.242.15.156",
"url": "https://httpbin.org/post"
}
The outstanding difference is that curl sends "Content-Length": "197" while the go app sends "Transfer-Encoding": "chunked".
Let's try to modify the go app to send the Content-Length header:
package main
import (
"bytes"
"fmt"
"io"
"log"
"mime/multipart"
"net/http"
"strings"
)
func main() {
source := strings.NewReader(strings.Repeat("a", 1<<21))
buf := new(bytes.Buffer)
multipart := multipart.NewWriter(buf)
part, err := multipart.CreateFormFile("file", "test.txt")
if err != nil {
log.Fatal(err)
}
if _, err := io.Copy(part, source); err != nil {
log.Fatal(err)
}
multipart.Close()
req, err := http.NewRequest(http.MethodPost, "https://api.anonfiles.com/upload", buf)
if err != nil {
log.Fatal(err)
}
req.Header.Add("Content-Type", multipart.FormDataContentType())
// The following line is not required because the http client will set it
// because the request body is a bytes.Buffer.
// req.ContentLength = int64(buf.Len())
client := http.Client{}
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
fmt.Println(string(body))
}
It works!
The disadvantage is that it has to copy the request body into the memory first. It seems to me that this is unavoidable because it needs to know the size of the request body.
I have a secure websocket server running on localhost:443/server-demo ( jetty websocket server).
Now I am writing a go client that can communicate with the websocket server. I am able to connect to the websocket server using right certificates. Here is the sample code.
package main
import (
"crypto/tls"
"crypto/x509"
"fmt"
"io"
"log"
)
func main() {
cert, err := tls.LoadX509KeyPair("nifi-1.10.0-bin/nifi-1.10.0/extras/gen-certs/certs/admin.pem", "nifi-1.10.0-bin/nifi-1.10.0/extras/gen-certs/certs/admin-key.pem")
if err != nil {
log.Fatalf("server: loadkeys: %s", err)
}
config := tls.Config{Certificates: []tls.Certificate{cert}, InsecureSkipVerify: true}
conn, err := tls.Dial("tcp", "127.0.0.1:443", &config)
if err != nil {
log.Fatalf("client: dial: %s", err)
}
defer conn.Close()
log.Println("client: connected to: ", conn.RemoteAddr())
state := conn.ConnectionState()
for _, v := range state.PeerCertificates {
fmt.Println(x509.MarshalPKIXPublicKey(v.PublicKey))
fmt.Println(v.Subject)
}
log.Println("client: handshake: ", state.HandshakeComplete)
log.Println("client: mutual: ", state.NegotiatedProtocolIsMutual)
message := "Hello\n"
n, err := io.WriteString(conn, message)
if err != nil {
log.Fatalf("client: write: %s", err)
}
log.Printf("client: wrote %q (%d bytes)", message, n)
reply := make([]byte, 256)
n, err = conn.Read(reply)
log.Printf("client: read %q (%d bytes)", string(reply[:n]), n)
log.Print("client: exiting")
}
The above code throws this error:
"HTTP/1.1 400 No URI\r\nContent-Type: text/html;charset=iso-8859-1\r\nContent-Length: 49\r\nConnection: close\r\nServer: Jetty(9.4.19.v20190610)\r\n\r\n<h1>Bad Message 400</h1><pre>reason: No URI</pre>" (188 bytes)
My question is after making the connection how can I send message to particular URI? i.e I want to send a message to wss://localhost:443/server-demo.
The code in a question does not establish a WebSocket connection to the server.
To establish the WebSocket connection, the application must write a WebSocket handshake to conn and receive the handshake response. See the RFC for the details.
Most applications use a websocket package than handles all of these details. The gorilla/websocket package is a popular choice.
This code should get you started with gorilla:
cert, err := tls.LoadX509KeyPair("nifi-1.10.0-bin/nifi-1.10.0/extras/gen-certs/certs/admin.pem", "nifi-1.10.0-bin/nifi-1.10.0/extras/gen-certs/certs/admin-key.pem")
if err != nil {
log.Fatalf("server: loadkeys: %s", err)
}
config := tls.Config{Certificates: []tls.Certificate{cert}, InsecureSkipVerify: true}
d := websocket.Dialer{
TLSClientConfig: &config,
}
c, _, err := d.Dial("wss://localhost:443/server-demo", nil)
if err != nil {
log.Fatal(err)
}
defer c.Close()
// Use `c` to send and receive messages
I'm getting dial tcp : socket: too many open files error when i do load test.By setting ulimit its working fine but is there any other solution without setting ulimit?
Code:
package main
import (
"fmt"
"io"
"io/ioutil"
"encoding/json"
"net/http"
)
type Struct_Response struct {
Meta struct {
Requestid string
}
}
var HttpClient = &http.Client{}
func main(){
apiUrl := "http://example.com"
JsonStr :="teststr"
conn_token :="1233333333333"
req, err := http.NewRequest("POST", apiUrl, bytes.NewBuffer(JsonStr))
if err!=nil{
fmt.Println(err)
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("requestid", "1234")
req.Header.Set("Authorization", "Bearer "+conn_token)
req.Header.Set("Connection", "close")
resp, err := HttpClient.Do(req)
req.Close=true
if resp!=nil && resp.StatusCode==200 {
body, _ := ioutil.ReadAll(resp.Body)
var Responce Struct_Response
err := json.Unmarshal([]byte(string(body)), &Responce)
if err != nil {
fmt.Println(err)
}
io.Copy(ioutil.Discard, resp.Body)
resp.Body.Close()
}
}
Thanks in advance.
Your problem may be that you're not cleanly closing connections which causes delays to be added before reusing TCP port client port numbers.
In your code example above, the response body is only consumed and closed when the status is 200. You should always consume/close the response body when present.
I am working in a Go project, and I need to perform some operations over an external API: GET, PUT, POST and DELETE. Currently I am using net/http, and I created a &http.Client{} to make GET and PUT. That is working as expected.
Now I need to perform a DELETE and I cannot find anything about it. Is it supported? Basically, I need to call a URL like this:
somedomain.com/theresource/:id
Method: DELETE
How can I perform that?
Here is a small example of how to do it:
package main
import (
"fmt"
"io/ioutil"
"net/http"
)
func sendRequest() {
// Request (DELETE http://www.example.com/bucket/sample)
// Create client
client := &http.Client{}
// Create request
req, err := http.NewRequest("DELETE", "http://www.example.com/bucket/sample", nil)
if err != nil {
fmt.Println(err)
return
}
// Fetch Request
resp, err := client.Do(req)
if err != nil {
fmt.Println(err)
return
}
defer resp.Body.Close()
// Read Response Body
respBody, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println(err)
return
}
// Display Results
fmt.Println("response Status : ", resp.Status)
fmt.Println("response Headers : ", resp.Header)
fmt.Println("response Body : ", string(respBody))
}