Getting Go's HTTP post to emulate curl -d - go

I am attempting to post content to an nginx server through Go. I have verified that I am able to correctly POST this content through curl, specifically using this command:
$ curl http://example.com/myendpoint -d "Some Text"
I am able to see this POST, and process it correctly. However, when I try to perform a POST with Go, it is rejected by the server. In the nginx access logs, I see these two lines:
127.0.0.1 - - [30/Jan/2014:05:57:34 +0000] "POST /myendpoint HTTP/1.1" 400 0 "-" "Go 1.1 package http"
127.0.0.1 - - [30/Jan/2014:05:57:39 +0000] "Some Text" 400 172 "-" "-"
The code that I have tried is below:
r, err := http.Post(uri, "application/x-www-form-urlencoded", bytes.NewReader(data))
if err != nil {
log.Printf("HTTP NOTIFICATION ERROR: %s\n", err)
return
}
r.Body.Close()
Is there something that I am doing wrong? Thanks!

I can't see any particular issue with the code snippets you provided, but you may not be encoding the data as required by application/x-www-form-urlencoded's specification.
Try using PostForm instead, which will perform this on your behalf:
import (
"net/url"
)
// ...
r, err := http.PostForm(uri, url.Values{"key": {"Value"}, "id": {"123"}})

If it is more convenient for you to convert your post data to []byte, vs url.Values, you can use http.Client. The code below shows how you can use the Client to process a new http request.
var client http.Client
req, err := http.NewRequest("POST", myURL, data) // data is of type []byte
if err != nil {
// todo: do something
}
req.Header.Add("Content-Type", "application/x-www-form-urlencoded")
resp, err := client.Do(req)
if err != nil {
// todo: do something
}
defer resp.Body.Close()
// process your response.

Related

Golang send get 400 but curl is correct

I use golang send http get to a url it return 400 ,but i using the curl ie will give me the correct 200.below is my code
func TestParseUrl(t *testing.T) {
originUrl := "https://auth0.openai.com/authorize?client_id=TdJIcbe16WoTHtN95nyywh5E4yOo6ItG&scope=openid%20email%20profile%20offline_access%20model.request%20model.read%20organization.read&response_type=code&redirect_uri=https%3A%2F%2Fexplorer.api.openai.com%2Fapi%2Fauth%2Fcallback%2Fauth0&audience=https%3A%2F%2Fapi.openai.com%2Fv1&state=I_b9zgmBonH9_nyKm3pF45sBPZeYhEIVdYNduPXP1KU&code_challenge=DjrdWVogz4KP6iQtmtbByQzqeFIO0_rckquiiEwCgxc&code_challenge_method=S256"
// decode url
decodedUrl, err := url.QueryUnescape(originUrl)
if err != nil {
fmt.Println("decode err")
}
// send request
response, err := http.Get(decodedUrl)
if err != nil {
fmt.Println("send failureļ¼š", err)
return
}
defer response.Body.Close()
// out put ans
fmt.Println(response.Status)
}
the golang out put
2023-02-18T03:21:21+08:00 warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)
=== RUN TestParseUrl
400 Bad Request
--- PASS: TestParseUrl (0.60s)
PASS
curl script
curl 'https://auth0.openai.com/authorize?client_id=TdJIcbe16WoTHtN95nyywh5E4yOo6ItG&scope=openid%20email%20profile%20offline_access%20model.request%20model.read%20organization.read&response_type=code&redirect_uri=https%3A%2F%2Fexplorer.api.openai.com%2Fapi%2Fauth%2Fcallback%2Fauth0&audience=https%3A%2F%2Fapi.openai.com%2Fv1&state=8vWnTRiDQxTo16Gz8TOu64QHCXTBxCraLMGKhR-TIMA&code_challenge=kWV8VapTwVi2EcjJnX0uk-vQPUn3k7BlrBGHTgdCFRc&code_challenge_method=S256' \
--compressed
curl out put
enter image description here
i want fix this problom , idont know why golang will show 400 , could any one help me
decodedUrl seems to be the issue to me, I would
// keep the code above this
fmt.Println(originUrl)
// decode url
decodedUrl, err := url.QueryUnescape(originUrl)
if err != nil {
fmt.Println("decode err")
}
//then curl the url that is printed here
fmt.Println(decodedUrl) // I expect curling this will be a bad request as well

What whould be the best way to forward a request by adding headers?

I just started to use Golang and I want to remake my already working NodeJS/TypeScript app in Go.
One endpoint of my API simply adds server-side generated authorization headers and sends a request to a remote API. Basically filling those headers for me by calling my API instead of the remote API.
This is what I am currently writing
func Endpoint(ctx *fiber.Ctx) error {
url := "https://api.twitch.tv" + ctx.OriginalURL()
req, _ := http.NewRequest(http.MethodGet, url, nil)
req.Header.Set("Authorization", "Bearer ---------")
req.Header.Set("Client-Id", "---------")
client := &http.Client{}
res, err := client.Do(req)
// temporary error handling
if err != nil {
log.Fatalln(err)
}
body, err := ioutil.ReadAll(res.Body)
// temporary error handling
if err != nil {
log.Fatalln(err)
}
var forwardedBody interface{}
json.Unmarshal(body, &forwardedBody)
return ctx.Status(fiber.StatusOK).JSON(forwardedBody)
}
I'd like to know if I am on the right steps, because making a request, parsing the JSON response with ioutil then unmarshall it to send it back seems kind of overboard for the simplicity of what I am trying to achieve ?
Edit: Thank you for the help, this is what I will be going for
func Endpoint(ctx *fiber.Ctx) error {
url := "https://api.twitch.tv" + ctx.OriginalURL()
req, _ := http.NewRequest(http.MethodGet, url, nil)
req.Header.Set("Authorization", "Bearer ---------")
req.Header.Set("Client-ID", "---------")
client := &http.Client{}
res, err := client.Do(req)
if err != nil {
return ctx.SendStatus(fiber.StatusBadRequest)
}
ctx.Set("Content-Type", "application/json; charset=utf-8")
return ctx.Status(res.StatusCode).SendStream(res.Body)
}
You can use httputil.ReverseProxy. Which takes a base URL and forwards requests to the base URL, concatenating the path.
ReverseProxy is an HTTP Handler that takes an incoming request and sends it to another server, proxying the response back to the client.
http.Handle("/", &httputil.ReverseProxy{
Director: func(r *http.Request) {
r.URL.Scheme = "https"
r.URL.Host = "go.dev"
r.Host = r.URL.Host
r.Header.Set("X-Foo", "Bar")
},
})
If you are not serving this from the root path / you can use StripPrefix.
http.HandleFunc("/foo/", http.StripPrefix("/foo/", proxy)
There is also a helper function NewSingleHostReverseProxy, which possibly removes the need to configure the proxy struct yourself. But I think it will be better to set the Host header along with your custom header.
You don't need to attempt to parse the data as JSON. This will be problematic if any of your endpoints don't return JSON, anyway, so just inject the body directly into the response:
body, err := ioutil.ReadAll(res.Body)
// temporary error handling
if err != nil {
log.Fatalln(err)
}
// Inject the body from the inner response into the actual response so it can be returned
ctx.Response().SetBody(body)
return cx.Status(fiber.StatusOK)

Golang http client is redirected to signin for bitbucket private url instead of giving auth failed

Trying to download a bitbucket private repo https://bitbucket.org/md-shabbir/test-repo/get/master.tar.gz with golang http client without providing any auth.
package main
import (
"net/http"
"fmt"
)
func CheckRedirect(req *http.Request, via []*http.Request) error {
fmt.Println("Redirect URL: ", req.URL)
return nil
}
func main() {
client := &http.Client{}
client.CheckRedirect = CheckRedirect
req, err := http.NewRequest("GET", "https://bitbucket.org/md-shabbir/test-repo/get/master.tar.gz", nil)
if err != nil {
fmt.Println("Error: ", err)
return
}
res, err := client.Do(req)
if err != nil {
fmt.Println("Error: ", err)
return
}
fmt.Println("Status code: ", res.StatusCode)
}
Output:
Redirect URL: https://bitbucket.org/account/signin/?next=/md-shabbir/test-repo/get/master.tar.gz
Redirect URL: https://bitbucket.org/socialauth/login/atlassianid/?next=%2Fmd-shabbir%2Ftest-repo%2Fget%2Fmaster.tar.gz
Status code: 200
I am expecting the request should return http code 401 with auth failed but it is giving 200.
However I tried to access the same url with curl, wget and python too and these all are giving expected return code 401.
Your client is redirected to the login page that returns a StatusCode of 200.
The first hit is redirected (302 with Location: /account/signin/?next=/md-shabbir/test-repo/get/master.tar.gz), and also the second one (302 Found with Location: https://bitbucket.org/socialauth/login/atlassianid/?next=%2Fmd-shabbir%2Ftest-repo%2Fget%2Fmaster.tar.gz).
This login page returns the status code of 200 which is the one you see.
If you don't want your client to follow these redirects, you could return a non-nil error from CheckRedirect.
I'm not sure if this will really solve the problem (as in BitBucket might still return 302), but you could also not continue on any redirects to this URL and treat them as errors.
If using https in Golange enable ssl , Try this below code like.
http.DefaultTransport.(*http.Transport).TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
_, err := http.Get("https://bitbucket.org/md-shabbir/test-repo/get/master.tar.gz")
if err != nil {
fmt.Println(err)
}

Unexpected error response from GitHub API 422 when attempting to create issue

When posting an issue to GitHub API V3 I am getting an unexpected response. Namely 422 Unprocessable Entity. However the detail of the error is for the Search endpoint rather that the POST create endpoint.
{"message":"Validation Failed","errors":[{"resource":"Search","field":"q","code":"missing"}],"documentation_url":"https://developer.github.com/v3/search"}
My instinct is that I have messed up the json but it is pretty simple and I can't see the issue.
I have tried various solutions posted here and elsewhere but not found what I am doing wrong. This is a coding exercise rather than anything intended for production but driving me mildly insane.
Tested in Debug what the Request body is just before being posted.
{"title":"Hello World","body":"dfsdfsdf\n"}
Tried removing the body as it is optional, same issue.
Tested in Debug that request is of type POST
Tested in Debug that authorization header is correct.
Removed authorization key and received 401 as expected.
The posting function:
func CreateIssue (issue *NewIssue) (*IssueDetailsResult, error){
issueJson, err := json.Marshal(issue)
if err != nil {
log.Fatal(err)
os.Exit(1)
}
req, err := http.NewRequest("POST", github.IssuesURL, bytes.NewBuffer(issueJson))
req.Header.Set("Authorization", "token "+os.Getenv("UPGITUSER"))
req.Header.Set( "Content-Type", "application/json")
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
os.Exit(1)
}
if resp.StatusCode != http.StatusCreated {
bodyBytes, _ := ioutil.ReadAll(resp.Body)
body := string(bodyBytes)
resp.Body.Close()
return nil, fmt.Errorf("create issue failed:%s", resp.Status + "\ntext: " + body)
}
var result IssueDetailsResult
if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
resp.Body.Close()
return nil, err
}
resp.Body.Close()
return &result, nil
}
Would expect 201 from GitHubAPI.
The response is a strong indicator that the request is being sent to the wrong endpoint.
You can use net/http/httputil's DumpRequestOut to inspect the requests you are about to send and to ensure that they are what you expect them to be.

Printf not working in http handler in golang

I am trying to write a web server using Go-bootstrap library and I have written my own handler in handler/users which is called on a post request on
localhost/app/signup. I am trying to print the json data to the terminal but fmt.Printf() prints nothing. The files are as follows:
In handler/users
func AppPostSignup(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
fmt.Println("In AppPostSignup")
data := map[string]interface{}{}
body, _ := ioutil.ReadAll(r.Body)
json.Unmarshal(body, &data)
db := context.Get(r, "db").(*sqlx.DB)
email := data["Email"]
password := data["Password"]
passwordAgain := data["PasswordAgain"]
fmt.Printf("\ntype : %T\nData: %v", email, email)
_, err := dal.NewUser(db).Signup(nil, email.(string), password.(string), passwordAgain.(string))
if err != nil {
libhttp.HandleErrorJson(w, err)
return
}
}
In main.go
router.Handle("/", MustLogin(http.HandlerFunc(handlers.GetHome))).Methods("GET")
router.HandleFunc("/signup", handlers.GetSignup).Methods("GET")
router.HandleFunc("/signup", handlers.PostSignup).Methods("POST")
router.HandleFunc("/login", handlers.GetLogin).Methods("GET")
router.HandleFunc("/login", handlers.PostLogin).Methods("POST")
router.HandleFunc("/app/signup", handlers.AppPostSignup).Methods("POST")
router.HandleFunc("/authenticate", handlers.Authenticate).Methods("POST")
router.HandleFunc("/logout", handlers.GetLogout).Methods("GET")
What did I do wrong?
I just tested your code and I saw the following printed to the console:
In AppPostSignup
My guess is that you tried to test your /app/signup endpoint using a GET request, when you have it marked only as a POST request.
If you really want it to be a POST request, then you can test it with curl like so:
curl -X POST localhost:8888/app/signup
Note, that request is missing a valid body, but you will be able to see your message printed.

Resources