I use golang send http get to a url it return 400 ,but i using the curl ie will give me the correct 200.below is my code
func TestParseUrl(t *testing.T) {
originUrl := "https://auth0.openai.com/authorize?client_id=TdJIcbe16WoTHtN95nyywh5E4yOo6ItG&scope=openid%20email%20profile%20offline_access%20model.request%20model.read%20organization.read&response_type=code&redirect_uri=https%3A%2F%2Fexplorer.api.openai.com%2Fapi%2Fauth%2Fcallback%2Fauth0&audience=https%3A%2F%2Fapi.openai.com%2Fv1&state=I_b9zgmBonH9_nyKm3pF45sBPZeYhEIVdYNduPXP1KU&code_challenge=DjrdWVogz4KP6iQtmtbByQzqeFIO0_rckquiiEwCgxc&code_challenge_method=S256"
// decode url
decodedUrl, err := url.QueryUnescape(originUrl)
if err != nil {
fmt.Println("decode err")
}
// send request
response, err := http.Get(decodedUrl)
if err != nil {
fmt.Println("send failureļ¼", err)
return
}
defer response.Body.Close()
// out put ans
fmt.Println(response.Status)
}
the golang out put
2023-02-18T03:21:21+08:00 warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)
=== RUN TestParseUrl
400 Bad Request
--- PASS: TestParseUrl (0.60s)
PASS
curl script
curl 'https://auth0.openai.com/authorize?client_id=TdJIcbe16WoTHtN95nyywh5E4yOo6ItG&scope=openid%20email%20profile%20offline_access%20model.request%20model.read%20organization.read&response_type=code&redirect_uri=https%3A%2F%2Fexplorer.api.openai.com%2Fapi%2Fauth%2Fcallback%2Fauth0&audience=https%3A%2F%2Fapi.openai.com%2Fv1&state=8vWnTRiDQxTo16Gz8TOu64QHCXTBxCraLMGKhR-TIMA&code_challenge=kWV8VapTwVi2EcjJnX0uk-vQPUn3k7BlrBGHTgdCFRc&code_challenge_method=S256' \
--compressed
curl out put
enter image description here
i want fix this problom , idont know why golang will show 400 , could any one help me
decodedUrl seems to be the issue to me, I would
// keep the code above this
fmt.Println(originUrl)
// decode url
decodedUrl, err := url.QueryUnescape(originUrl)
if err != nil {
fmt.Println("decode err")
}
//then curl the url that is printed here
fmt.Println(decodedUrl) // I expect curling this will be a bad request as well
Related
Trying to download a bitbucket private repo https://bitbucket.org/md-shabbir/test-repo/get/master.tar.gz with golang http client without providing any auth.
package main
import (
"net/http"
"fmt"
)
func CheckRedirect(req *http.Request, via []*http.Request) error {
fmt.Println("Redirect URL: ", req.URL)
return nil
}
func main() {
client := &http.Client{}
client.CheckRedirect = CheckRedirect
req, err := http.NewRequest("GET", "https://bitbucket.org/md-shabbir/test-repo/get/master.tar.gz", nil)
if err != nil {
fmt.Println("Error: ", err)
return
}
res, err := client.Do(req)
if err != nil {
fmt.Println("Error: ", err)
return
}
fmt.Println("Status code: ", res.StatusCode)
}
Output:
Redirect URL: https://bitbucket.org/account/signin/?next=/md-shabbir/test-repo/get/master.tar.gz
Redirect URL: https://bitbucket.org/socialauth/login/atlassianid/?next=%2Fmd-shabbir%2Ftest-repo%2Fget%2Fmaster.tar.gz
Status code: 200
I am expecting the request should return http code 401 with auth failed but it is giving 200.
However I tried to access the same url with curl, wget and python too and these all are giving expected return code 401.
Your client is redirected to the login page that returns a StatusCode of 200.
The first hit is redirected (302 with Location: /account/signin/?next=/md-shabbir/test-repo/get/master.tar.gz), and also the second one (302 Found with Location: https://bitbucket.org/socialauth/login/atlassianid/?next=%2Fmd-shabbir%2Ftest-repo%2Fget%2Fmaster.tar.gz).
This login page returns the status code of 200 which is the one you see.
If you don't want your client to follow these redirects, you could return a non-nil error from CheckRedirect.
I'm not sure if this will really solve the problem (as in BitBucket might still return 302), but you could also not continue on any redirects to this URL and treat them as errors.
If using https in Golange enable ssl , Try this below code like.
http.DefaultTransport.(*http.Transport).TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
_, err := http.Get("https://bitbucket.org/md-shabbir/test-repo/get/master.tar.gz")
if err != nil {
fmt.Println(err)
}
I want to create an api for uploading images in my go-gin application. This is my current api code :
func (s *Service) ImageCreate(c *gin.Context) {
token := c.MustGet(tokenKey).(*models.Token)
now := time.Now()
file, err := c.FormFile("file")
if err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "01> " + err.Error()})
return
}
}
When I call my application from the browser it returns this error:
{"error":"01\u003e invalid URL escape \"%$\\xf4\""}
In postman api it's working without error. What is wrong? How I can fix it for the browser?
Don't use Postman.
curl -X POST http://localhost:8080/server-image/upload-file -F "file=#/Users/shikuanxu/Downloads/images/course-outline.png" -H "Content-Type: multipart/form-data"
Try again
Hi I am creating a POST request in GO but it terminates before actually finishing, for example I am trying to pull a docker image and this can be done with curl by the following:
curl -X POST https://address/images/create?fromImage=imagename
this returns the following
{"status":"Pulling from imagename","id":"latest"}
{"status":"Already exists","progressDetail":{},"id":"3d30e94188f7"}
{"status":"Already exists","progressDetail":{},"id":"bf4e27765153"}
{"status":"Already exists","progressDetail":{},"id":"67280fd39fba"}
.... many of those
{"status":"Pull complete","progressDetail":{},"id":"21c062e2346f"}
{"status":"Digest: sha256:24f26a1344fca6d5ee1adcdsf2d01b20d7823c560ed9d2193466e36bd1f049088"}
{"status":"Pulling from imagename","id":"20161005"}
{"status":"Digest: sha256:f527dsfds88676eb25d8f7de5406f46cbc3a995345ddb4bb3d08fcf110458fe3cf"}
{"status":"Status: Downloaded newer image for imagename"}
and the image is pulled successfully
but If I try from go
func PullImage(imagename string, uuid string) error {
logFields := log.Fields{
"handler": "PullImage",
"uuid": uuid,
}
log.WithFields(logFields).Debugf("imagename:%v", imagename)
url := fmt.Sprintf("https://%s/images/create?fromImage%s", sconf.Docker.Endpoint, imagename)
req, err := http.NewRequest("POST", url, nil)
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)
log.WithFields(logFields).Infof("call url:%s", url)
if err != nil {
log.WithFields(logFields).Errorf("Error in call url (%s) :%s", url, err)
return errors.New(fmt.Sprintf("Error in call url (%s) :%s", url, err))
}
var pullresbody interface{}
err = json.NewDecoder(resp.Body).Decode(&pullresbody)
if err != nil {
log.WithFields(logFields).Errorf("Could not unmarshal json: %s", err)
return errors.New(fmt.Sprintf("Could not unmarshal json: %s", err))
}
log.WithFields(logFields).Infof("response of url %s:%+v", url, resp)
log.WithFields(logFields).Infof("response body:%+v", pullresbody)
return nil
}
then I get this in the logs:
map[status:Pulling from imagename]
and the image is not pulled so the connection is stopped before really finishing how can I fix this?
The Decoder will only decode one object from a stream at a time. To get all objects, you'll need something like
var pullRespBody interface{}
dec := json.NewDecoder(resp.Body)
var err error
for err == nil {
err = dec.Decode(&pullRespBody)
// Check err...
log.WithFields(logFields).Infof("response body:%+v", pullRespBody)
// Do something else with pullRespBody...
}
// Deal with err...
I am attempting to post content to an nginx server through Go. I have verified that I am able to correctly POST this content through curl, specifically using this command:
$ curl http://example.com/myendpoint -d "Some Text"
I am able to see this POST, and process it correctly. However, when I try to perform a POST with Go, it is rejected by the server. In the nginx access logs, I see these two lines:
127.0.0.1 - - [30/Jan/2014:05:57:34 +0000] "POST /myendpoint HTTP/1.1" 400 0 "-" "Go 1.1 package http"
127.0.0.1 - - [30/Jan/2014:05:57:39 +0000] "Some Text" 400 172 "-" "-"
The code that I have tried is below:
r, err := http.Post(uri, "application/x-www-form-urlencoded", bytes.NewReader(data))
if err != nil {
log.Printf("HTTP NOTIFICATION ERROR: %s\n", err)
return
}
r.Body.Close()
Is there something that I am doing wrong? Thanks!
I can't see any particular issue with the code snippets you provided, but you may not be encoding the data as required by application/x-www-form-urlencoded's specification.
Try using PostForm instead, which will perform this on your behalf:
import (
"net/url"
)
// ...
r, err := http.PostForm(uri, url.Values{"key": {"Value"}, "id": {"123"}})
If it is more convenient for you to convert your post data to []byte, vs url.Values, you can use http.Client. The code below shows how you can use the Client to process a new http request.
var client http.Client
req, err := http.NewRequest("POST", myURL, data) // data is of type []byte
if err != nil {
// todo: do something
}
req.Header.Add("Content-Type", "application/x-www-form-urlencoded")
resp, err := client.Do(req)
if err != nil {
// todo: do something
}
defer resp.Body.Close()
// process your response.
I am trying to debug a very unusual error I am receiving for a simple REST library I wrote.
I am using the standard net/http package to make Get, Post, Put, Delete requests but my tests occasionally fail when I make multiple requests successively. My test looks like this:
func TestGetObject(t *testing.T) {
firebaseRoot := New(firebase_url)
body, err := firebaseRoot.Get("1")
if err != nil {
t.Errorf("Error: %s", err)
}
t.Logf("%q", body)
}
func TestPushObject(t *testing.T) {
firebaseRoot := New(firebase_url)
msg := Message{"testing", "1..2..3"}
body, err := firebaseRoot.Push("/", msg)
if err != nil {
t.Errorf("Error: %s", err)
}
t.Logf("%q", body)
}
And I am making the request like this:
// Send HTTP Request, return data
func (f *firebaseRoot) SendRequest(method string, path string, body io.Reader) ([]byte, error) {
url := f.BuildURL(path)
// create a request
req, err := http.NewRequest(method, url, body)
if err != nil {
return nil, err
}
// send JSON to firebase
resp, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("Bad HTTP Response: %v", resp.Status)
}
defer resp.Body.Close()
b, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, err
}
return b, nil
}
Sometimes it works, but most of the time I get 1 or 2 failures:
--- FAIL: TestGetObject (0.00 seconds)
firebase_test.go:53: Error: Get https://go-firebase-test.firebaseio.com/1.json: EOF
firebase_test.go:55: ""
--- FAIL: TestPushObject (0.00 seconds)
firebase_test.go:63: Error: Post https://go-firebase-test.firebaseio.com/.json: EOF
firebase_test.go:65: ""
FAIL
exit status 1
FAIL github.com/chourobin/go.firebase 3.422s
The failures happen when I make more than 1 request. If I comment out everything except for the PUT request, the tests consistently pass. Once I include a second test, such as GET, one or the other fails (sometimes both pass).
I experienced this reliably. You need to set Req.Close to true (the defer on resp.Body.Close() syntax used in the examples is not enough). Like this:
client := &http.Client{}
req, err := http.NewRequest(method, url, httpBody)
// NOTE this !!
req.Close = true
req.Header.Set("Content-Type", "application/json")
req.SetBasicAuth("user", "pass")
resp, err := client.Do(req)
if err != nil {
// whatever
}
defer resp.Body.Close()
response, err = ioutil.ReadAll(resp.Body)
if err != nil {
// Whatever
}
I agree with the assertion that you shouldn't be hitting outside servers in your unit tests, why not just use the built-in http.Server and serve up the content that you want to test. (There is actually the httptest package to help with this)
I recently ran into this same problem while trying to crawl sitemaps, and this is what I have found so far:
Go by default will send requests with the header Connection: Keep-Alive and persist connections for re-use. The problem that I ran into is that the server is responding with Connection: Keep-Alive in the response header and then immediately closing the connection.
As a little background as to how go implements connections in this case (you can look at the full code in net/http/transport.go). There are two goroutines, one responsible for writing and one responsible for reading (readLoop and writeLoop) In most circumstances readLoop will detect a close on the socket, and close down the connection. The problem here occurs when you initiate another request before the readLoop actually detects the close, and the EOF that it reads get interpreted as an error for that new request rather than a close that occurred prior to the request.
Given that this is the case the reason why sleeping in between requests works is that it gives readLoop time to detect the close on the connection before your new request and shut it down, so that your new request will initiate a new connection. (And the reason why it would intermittently fail is because there is some amount code running between your requests and depending of scheduling of goroutines, sometimes the EOF will be properly handled before your next request, sometimes not). And the req.Close = true, solution works because it prevents the connection from being re-used.
There is a ticket related to this situation: https://code.google.com/p/go/issues/detail?id=4677 (and a dupe ticket that I created that allowed me to reliably reproduce this: https://code.google.com/p/go/issues/detail?id=8122)
I'm going to guess there is no problem with your code. The most likely cause of your problem is because the server is closing the connection. Rate limiting is one possible reason for this.
Your test shouldn't be relying on an external service that's very brittle and not hermetic. Instead you should think about spinning up a test server locally.
My experience with this error was when I entered absolutely empty input for my JSON API!
I should send {} as empty JSON, but I sent so this error happened
I encountered this issue while sending an invalid body to a GET request
I could reproduce by making a request similar to below:
var requestBody interface{}
requestData, _ := json.Marshal(requestBody)
payload = strings.NewReader(string(requestData))
req, err := http.NewRequest("GET", url, payload)
...
I was developing an image download app when this problem occurs.
Tried request.Close=true but not work.
60% requests resulted in a EOF error.
I thought maybe it is an image server problem, not my code.
But php code works fine.
Then I use
var client = &http.Client{
Transport: &http.Transport{},
}
client.Do(request)
to make request, instead of
http.DefaultClient.Do(request)
problem gone.
Not sure why,I guess something with RoundTripper