What I'm doing is trying to submit a URL to scan to urlscan.io. I can do a search but have issues with submissions, particularly correctly sending the right headers/encoded data.
from their site on how to submit a url:
curl -X POST "https://urlscan.io/api/v1/scan/" \
-H "Content-Type: application/json" \
-H "API-Key: $apikey" \
-d "{\"url\": \"$url\", \"public\": \"on\"}"
This works to satisfy the Api key header requirement but
req.Header.Add("API-Key", authtoken)
This is my attempt that fails
data := make(url.Values)
data.Add("url", myurltoscan)
The URL property I have struggled with tremendously.
This is my error:
"message": "Missing URL properties",
"description": "The URL supplied was not OK, please specify it including the protocol, host and path (e.g. http://example.com/bar)",
"status": 400
url.Value is a map[string][]string containing values used in query parameters or POST form. You would need it if you were trying to do something like:
curl -X GET https://urlscan.io/api/v1/scan?url=<urltoscan>
or
curl -X POST -F 'url=<urltoscan>' https://urlscan.io/api/v1/scan
See documentation https://golang.org/pkg/net/url/#Values.
To send a regular POST request with JSON data, you can encode the JSON as bytes and send with http.Post:
var payload = []byte(`{"url":"<your-url>","public":"on"}`)
req, err := http.NewRequest("POST", url,
bytes.NewBuffer(payload))
req.Header.Set("API-Key", authtoken)
req.Header.Set("Content-Type", "application/json")
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
Related
I'm trying to make a HTTP request using http.Get(url) in Go and I want to open the response in a browser. I'm using browser.OpenURL() to launch the system browser, but I cannot figure out how to obtain the response url.
In Python, using the requests library, it is an attribute of the response object.
I can obtain and open it in a browser (using the browser library) like so:
response = requests.get(endpoint)
browser.open(response.url)
How can I accomplish this using http/net library in Go? The response object is a struct that doesn't contain that attribute.
I am trying to call the Spotify API to authenticate an app, and this requires opening a browser window for user input. So far I've got this:
func getAuth(endpoint *url.Url) {
request, _ := http.NewRequest("GET", endpoint.string(), nil)
client := &http.Client{}
resp, err := client.Do(request)
if err != nil {
panic(err)
}
headers := resp.Header
page, _ := ioutil.ReadAll(resp.Body)
Where can I obtain the response URL or how can I handle the response so that it opens it in a browser?
Go will update the Request struct on the response if there is a redirect.
resp.Request.URL is what you are looking for.
// Request is the request that was sent to obtain this Response.
// Request's Body is nil (having already been consumed).
// This is only populated for Client requests.
Request *Request
Just get the redirect URL from response header.
redirectURL := resp.Header.Get("Location")
I'm currently trying to interface with an OAuth api with the example curl command curl -u {client_id}:{client_secret} -d grant_type=client_credentials https://us.battle.net/oauth/token. My current go file is:
package main
import (
"bytes"
"fmt"
"mime/multipart"
"net/http"
)
func checkErr(err error) bool {
if err != nil {
panic(err)
}
return true
}
func authcode(id string, secret string, cli http.Client) string {
//un(trace("authcode"))
var form bytes.Buffer
w := multipart.NewWriter(&form)
_, err := w.CreateFormField("grant_type=client_credentials")
checkErr(err)
req, err := http.NewRequest("POST", "https://us.battle.net/oauth/token", &form)
checkErr(err)
req.SetBasicAuth(id, secret)
resp, err := cli.Do(req)
checkErr(err)
defer resp.Body.Close()
json := make([]byte, 1024)
_, err = resp.Body.Read(json)
checkErr(err)
return string(json)
}
func main() {
//un(trace("main"))
const apiID string = "user"
const apiSecret string = "password"
apiClient := &http.Client{}
auth := authcode(apiID, apiSecret, *apiClient)
fmt.Printf("%s", auth)
}
When I run this I get a response of {"error":"invalid_request","error_description":"Missing grant type"}
For reference, the api flow states:
"To request access tokens, an application must make a POST request with the following multipart form data to the token URI: grant_type=client_credentials
The application must pass basic HTTP auth credentials using the client_id as the user and client_secret as the password."
and the expected response is a json string containing an access token, token type, expiration in seconds, and the scope of functions available with said token
From curl manual we have:
-d, --data <data>
(HTTP) Sends the specified data in a POST request to the HTTP server, in the same way that a browser does when a user has filled in an HTML form and
presses the submit button. This will cause curl to pass the data to the server using the content-type application/x-www-form-urlencoded. Compare to
-F, --form.
Note the content-type application/x-www-form-urlencoded part.
as opposed to:
-F, --form <name=content>
(HTTP SMTP IMAP) For HTTP protocol family, this lets curl emulate a filled-in form in which a user has pressed the submit button. This causes curl to
POST data using the Content-Type multipart/form-data according to RFC 2388.
Therefore based on your curl, mime/multipart is probably not what you're looking for and you should be using Client.PostForm, from the manual of which we have:
The Content-Type header is set to application/x-www-form-urlencoded. To set other headers, use NewRequest and Client.Do.
I am using oauth2 to access a third party API. I can get the access token alright, but when I try to call the API by passing the bearer token in the request headers it gives me 401 (Unauthorized) error. Although it works well when I try to do it via POSTMAN by passing headers as (Authorization: Bearer ). But it does not work using go.
Here is the code sample.
url := "http://api.kounta.com/v1/companies/me.json"
var bearer = "Bearer " + <ACCESS TOKEN HERE>
req, err := http.NewRequest("GET", url, nil)
req.Header.Add("authorization", bearer)
client := urlfetch.Client(context)
resp, err := client.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
body, _ := ioutil.ReadAll(resp.Body)
writer.Write([]byte(body)) // Gives 401 Unauthorized error, though same works using POSTMAN
I was able to solve the problem. Actually the problem was two way.
1) The API end point was doing a redirect (302), which was causing a 302 response and then the other API was being called.
2) GO by default does not forward the headers, thus my bearer token was being lost in the middle.
FIX:
I had to override the client's CheckRedirect function and manually pass the headers to the new request.
client.CheckRedirect = checkRedirectFunc
Here is how I forwarded the headers manually.
func checkRedirectFunc(req *http.Request, via []*http.Request) error {
req.Header.Add("Authorization", via[0].Header.Get("Authorization"))
return nil
}
I'm building 2 apis. One make request to another.
To call the api that receives requests, we need to pass a X-Token Header. I'm doing this with Golang
client := &http.Client{
Transport: &http.Transport{
TLSClientConfig: &tls.Config{
InsecureSkipVerify: true,
},
},
}
req, err := http.NewRequest("GET", "https://localhost:8086/v2/example", nil)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"Error": err.Error()})
}
req.Header.Add("accept", "application/json")
req.Header.Add("content-type", "application/json")
req.Header.Add("x-token", "a2e63ee01401aaeca78be023dfbb8c59")
resp, err := client.Do(req)
In the other API, i get the http header with gin like this:
token := c.Request.Header.Get("x-token")
I dont know why my header arrives with another value and no X-Token. Thanks!
Result of fmt.Printf("%+v", c.Request.Header):
map[User-Agent:[Go-http-client/1.1] Referer:[https://localhost:8086/v2/example] Accept-Encoding:[gzip]]
I don't know where is my x-token, accept and content-type headers....
IMPORTANT
If i make a request with x-token header on Postman to the requested API i get the right header.
If i change the request address on the API that makes requests, e.g httpbin, i get the right header too....
Helo, guys! i found the solution....
I don't know why yet... but i think golang don't handle no trailing slash url's....
https://localhost:8086/v2/example
is different of
https://localhost:8086/v2/example/
That was my problem....
I just copy and past the golang generated code of postman... and that was the "biggest" difference....
Thanks mr. Postman...
Need to accept OPTIONS method coming from mobile device,
attempted multiple ways to do so and getting strange behavior:
when trying this I get 403 from the client:
(client sends OPTIONS before POST)
import (
"net/http"
"github.com/gorilla/handlers"
"github.com/gorilla/mux"
)
func main() {
r := mux.NewRouter()
r.HandleFunc("/users", UserEndpoint)
r.HandleFunc("/projects", ProjectEndpoint)
methods := handlers.AllowedMethods([]string{"OPTIONS", "DELETE", "GET", "HEAD", "POST"}
http.ListenAndServe(":8000", handlers.CORS(methods)(r))
}
if I omit the methods:
http.ListenAndServe(":8000", handlers.CORS()(r))
I get 403 not authorized
Also played around with it, removed the GET method:
methods := handlers.AllowedMethods([]string{"OPTIONS"}
http.ListenAndServe(":8000", handlers.CORS(methods)(r))
but still could
get a 200 GET when tried from rest client in browser (chromes DHC)
but if I remove the OPTIONS:
methods := handlers.AllowedMethods([]string{"DELETE", "GET", "HEAD", "POST"}
http.ListenAndServe(":8000", handlers.CORS(methods)(r))
I get 405
First example is based on gorilla handler docs
Any ideas on this issues?
Thanks
You really need to understand the request being made, but I had a similar problem and resolved it with:
handlers.CORS(
handlers.AllowedOrigins([]string{"*"}),
handlers.AllowedMethods([]string{"POST"}),
handlers.AllowedHeaders([]string{"Content-Type", "X-Requested-With"}),
)(router)
The request I needed to make (which mimics a preflight) was:
curl -H "Origin: http://example.com" \
-H "Access-Control-Request-Method: POST" \
-H "Access-Control-Request-Headers: X-Requested-With" \
-X OPTIONS --verbose http://127.0.0.1:8080/products
It was really the AllowedHeaders func that made all the difference. As soon as I added that, the 403 error disappeared.
If you look at cors.go Options are specially handled:
corsOptionMethod string = "OPTIONS"
...
if r.Method == corsOptionMethod {
if ch.ignoreOptions {
ch.h.ServeHTTP(w, r)
return
}
if _, ok := r.Header[corsRequestMethodHeader]; !ok {
w.WriteHeader(http.StatusBadRequest)
return
}
method := r.Header.Get(corsRequestMethodHeader)
if !ch.isMatch(method, ch.allowedMethods) {
w.WriteHeader(http.StatusMethodNotAllowed)
return
}
...
So 405 is http.StatusMethodNotAllowed, so maybe it is not CORs request header?
There is also an IngoreOptions method for handling Options independely: http://www.gorillatoolkit.org/pkg/handlers#IgnoreOptions - maybe that will work in your case and you can just ignore it, or process Options on your own.