Converting curl request to Golang - go

So this curl request to the paypal payouts API works:
curl --silent -v 'https://api.sandbox.paypal.com/v1/oauth2/token' \
-H "Accept: application/json" \
-H "Accept-Language: en_US" \
-u "${client_id}:${client_secret}" \
-d "grant_type=client_credentials"
one thing I am confused about: the -d option is for data in the body of the HTTP request - does the -d option make it a POST request or is the curl request above a GET request with a body? I would guess that latter but I am unsure given the output of curl --help.
In golang I have:
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
)
const (
PayPalTestClientID = "Aeit5RskDRN8eUUMB0Ud3RjA_z6feWMUHktwlJZMeQMo9A9ulbKK"
PayPalTestSecret = "EAAqyzrOTUWf-OFJCB4BxgXT4xuravL7pnkC8Tn20HYtZExd1mFO"
)
func main() {
//reader := bytes.NewBuffer()
req, err := http.NewRequest("GET", "https://api.sandbox.paypal.com/v1/oauth2/token", nil)
if err != nil {
log.Fatal(err)
}
req.Header.Set("Accept", "application/json")
req.Header.Set("Accept-Language", "en_US")
req.Header.Set("Authorization", fmt.Sprintf("Basic %s:%s", PayPalTestClientID, PayPalTestSecret))
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
var v interface{}
err = json.Unmarshal(body, &v);
if err != nil {
log.Fatal(err)
}
log.Print(v)
}
the client/secret were obfuscated, so they won't work as written above. But using the real creds I get:
2020/01/31 16:05:07 map[error:invalid_client error_description:Client
Authentication failed]
The real creds do work with the curl command tho.

Note: The credentials provided are valid? Cause i receive a sonorus 401, Authentication Failed.
NOTE: Using the -d in cURL, you are going to send a POST request instead of a GET. Due to this behaviour, your proably want to send a POST request instead of a GET
You can use my little http library: https://github.com/alessiosavi/Requests/
package main
import requests "github.com/alessiosavi/Requests"
func main() {
req, err := requests.InitRequest("https://postman-echo.com/basic-auth", "GET", []byte{}, false, false)
if err != nil {
fmt.Println("ERROR! ", err)
}
req.CreateHeaderList("Accept", "application/json", "Accept-Language", "en_US", "Authorization", "postman:password")
client := &http.Client{}
resp := req.ExecuteRequest(client)
fmt.Println(resp.Dump())
}
You can change the data (URL, post data, headers) with the one that you need for authenticate to the service.
In your case, will be something like this:
package main
import requests "github.com/alessiosavi/Requests"
const (
ID= "Aeit5RskDRN8eUUMB0Ud3RjA_z6feWMUHktwlJZMeQMo9A9ulbKK"
SECRET= "EAAqyzrOTUWf-OFJCB4BxgXT4xuravL7pnkC8Tn20HYtZExd1mFO"
)
func main() {
req, err := requests.InitRequest("https://api.sandbox.paypal.com/v1/oauth2/token", "GET", []byte{"grant_type=client_credentials"}, false, false)
if err != nil {
fmt.Println("ERROR! ", err)
}
req.CreateHeaderList("Accept", "application/json", "Accept-Language", "en_US", "Authorization", ID+":"+SECRET)
client := &http.Client{}
resp := req.ExecuteRequest(client)
fmt.Println(resp.Dump())
}

Related

https POST not working as expected in golang , but works fine in Python

I am trying to implement a python code from the JIRA REST API examples:
https://developer.atlassian.com/cloud/jira/platform/rest/v3/api-group-jql/#api-rest-api-3-jql-parse-post
My python code (which works as expected):
import requests
from requests.auth import HTTPBasicAuth
import json
url = "https://my-url.com/rest/api/2/search"
auth = HTTPBasicAuth("user1", "pwd1")
headers = {
"Accept": "application/json",
"Content-Type": "application/json"
}
payload = json.dumps( {
"jql": "my-query-string"
}
response = requests.request("POST", url, data=payload, headers=headers, auth=auth, verify=False)
print(json.dumps(json.loads(response.text), sort_keys=True, indent=4, separators=(",", ": ")))
I'm trying to transform this to a golang code as below:
package main
import (
"io/ioutil"
"fmt"
"log"
"time"
"net/http"
"net/url"
}
func main() {
timeout := time.Duration(500 * time.Second)
client := http.Client{
Timeout: timeout,
}
req, err := http.NewRequest("POST", "https://my-url.com/rest/api/2/search", nil)
if err != nil {
log.Fatalln(err)
}
req.SetBasicAuth("user1", "pwd1")
req.Header.Set("Content-Type", "application/json")
q := url.Values{}
q.Add("jql", "my-query-string")
req.URL.RawQuery = q.Encode()
fmt.Println(req.URL.String())
resp, err := client.Do(req)
if err != nil {
log.Fatalln(err)
}
defer resp.Body.Close()
data, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatalln(err)
}
log.Println(string(data))
The code builds with no issues. When I run the go code, I get this error:
2021/04/17 19:36:31 {"errorMessages":["No content to map to Object due to end of input"]}
I have 2 questions :
a. How can I fix the above error ?
b. I also want to include concurrency in the same code, i.e the same POST request will actually be executed for 5 different query strings (concurrently) and fetch the results, how can i achieve that ?
For POST requests you need to send the data as json. Note that in Go setting a request's Content-Type header does not automagically convert whatever you give it to the specified type.
An example sending json.
package main
import (
"strings"
"net/http"
"io/ioutil"
"fmt"
)
func main() {
body := strings.NewReader(`{"jql": "project = HSP"}`)
req, err := http.NewRequest("POST", "https://your-domain.atlassian.com/rest/api/2/search", body)
if err != nil {
panic(err)
}
req.SetBasicAuth("email#example.com", "<api_token>")
req.Header.Set("Accept", "application/json")
req.Header.Set("Content-Type", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
out, err := ioutil.ReadAll(resp.Body)
if err != nil {
panic(err)
}
fmt.Println(string(out))
}
If you want to use query parameters you should use the endpoint with the GET method.
package main
import (
"net/http"
"net/url"
"io/ioutil"
"fmt"
)
func main() {
query := url.Values{"jql": {"project = HSP"}}
req, err := http.NewRequest("GET", "https://your-domain.atlassian.com/rest/api/2/search?" + query.Encode(), nil)
if err != nil {
panic(err)
}
req.SetBasicAuth("email#example.com", "<api_token>")
req.Header.Set("Accept", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
out, err := ioutil.ReadAll(resp.Body)
if err != nil {
panic(err)
}
fmt.Println(string(out))
}

Golang X509 Keypair ignored

We are using Golang to communicate with an external service using HTTPS. When attempting the request using cURL, we get success. When using Go, the certificate appears to be ignored, yielding a 403 from the external service.
We have been unable to spot any differences in the cURL-request vs the Golang code. Can somebody help us find a difference?
The cURL-request gives us a proper JSON response. The Go code gives:
2020/09/07 15:05:57 request error: perform request: api response: 403 Forbidden
Working cURL-request (user agent for debug purposes):
curl -X GET --http1.1 -i -v --key client.key.pem --cacert ca.pem --cert client.pem "https://[redacted]/path/to/endpoint" -H "Accept: application/json; charset=utf-8" -H "User-Agent: Apache-HttpClient/4.5.5 (Java/12.0.1)" -H "X-Identifier: [redacted]" -H "Accept-Encoding: gzip, deflate" -H "Connection: Keep-Alive"
Golang code yielding a 403:
(note: files ca.pem, client.pem (cert) and client.key.pem must be in same directory. run script as go run catest.go --url "https://[redacted]/path/to/endpoint" --identifier [redacted])
package main
import (
"crypto/tls"
"crypto/x509"
"flag"
"fmt"
"io/ioutil"
"log"
"net/http"
)
func main() {
// Get URL to call.
url := flag.String("url", "", "URL to call")
identifier := flag.String("identifier", "", "the X-Identifier value")
flag.Parse()
if url == nil || identifier == nil || *url == "" || *identifier == "" {
log.Fatal("'url' and 'identifier' arguments must be provided")
}
// Set up certificates
caPEM, err := ioutil.ReadFile("ca.pem")
if err != nil {
log.Fatalf("unable to read 'ca.pem' in current directory: %v", err)
}
clientPEM, err := ioutil.ReadFile("client.pem")
if err != nil {
log.Fatalf("unable to read 'client.pem' in current directory: %v", err)
}
clientKeyPEM, err := ioutil.ReadFile("client.key.pem")
if err != nil {
log.Fatalf("unable to read 'client.key.pem' in current directory: %v", err)
}
// Make calls.
client, err := configureClient(caPEM, clientPEM, clientKeyPEM)
if err != nil {
log.Fatalf("unable to setup client: %v", err)
}
_, err = performRequest(client, *url, *identifier)
if err != nil {
log.Fatalf("request error: %v", err)
}
log.Printf("request successful")
}
func configureClient(caCertPEM, clientCertPEM, clientKeyPEM []byte) (*http.Client, error) {
// Load the CA certificate.
caCertPool, err := x509.SystemCertPool()
if err != nil {
return nil, fmt.Errorf("configure client: load cert pool: %w", err)
}
// Append root CA cert from parameter
ok := caCertPool.AppendCertsFromPEM(caCertPEM)
if !ok {
return nil, fmt.Errorf("configure client: could not append ca certificate")
}
// Load the client certificate.
clientCert, err := tls.X509KeyPair(clientCertPEM, clientKeyPEM)
if err != nil {
return nil, fmt.Errorf("configure client: load client certificate: %w", err)
}
// Setup HTTPS client.
tlsConfig := &tls.Config{
RootCAs: caCertPool,
Certificates: []tls.Certificate{clientCert},
Renegotiation: tls.RenegotiateOnceAsClient,
}
tlsConfig.BuildNameToCertificate()
transport := &http.Transport{TLSClientConfig: tlsConfig}
client := &http.Client{Transport: transport}
return client, nil
}
func performRequest(client *http.Client, u, identifier string) ([]byte, error) {
if client == nil {
return nil, fmt.Errorf("perform request: nil client")
}
// Prepare request
req, err := http.NewRequest(http.MethodGet, u, nil)
if err != nil {
return nil, fmt.Errorf("perform request: create GET request: %w", err)
}
// Add same headers as cURL.
req.Header.Add("Accept", "application/json; charset=utf-8")
req.Header.Add("User-Agent", "Apache-HttpClient/4.5.5 (Java/12.0.1)")
req.Header.Add("Accept-Encoding", "gzip, deflate")
req.Header.Add("Connection", "Keep-Alive")
req.Header.Add("X-Identifier", identifier)
// Send request
resp, err := client.Do(req)
if err != nil {
return nil, fmt.Errorf("perform request: client do: %w", err)
}
defer resp.Body.Close()
switch resp.StatusCode {
case http.StatusOK:
break
case http.StatusUnauthorized:
return nil, fmt.Errorf("perform request: api response: unauthorized")
case http.StatusBadRequest:
return nil, fmt.Errorf("perform request: api response: bad request")
default:
return nil, fmt.Errorf("perform request: api response: %v", resp.Status)
}
data, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, fmt.Errorf("perform request: read response body: %w", err)
}
return data, nil
}

How to obtain access token for use with google storage URL

I am trying to read from this google storage bucket:
https://storage.googleapis.com/images.eng.channelmeter.com/avatars/d2d48e49-82be-4cf6-be6e-11ada43c7339
But I am getting this error:
the error is like so:
<Error>
<Code>AccessDenied</Code>
<Message>Access denied.</Message>
<Details>
Anonymous caller does not have storage.objects.get access to images.eng.channelmeter.com/avatars/d2d48e49-82be-4cf6-be6e-11ada43c7339.
</Details>
</Error>
How can I obtain an access token that I can append to url? Something like this:
https://storage.googleapis.com/images.eng.channelmeter.com/avatars/d2d48e49-82be-4cf6-be6e-11ada43c7339?access_token="XXX"
I assume I can make some call using an OAuth library to get a temporary / one-time access token, anyone know how?
Update:
I can get an AccessToken using this technique: https://tanaikech.github.io/2018/12/11/retrieving-access-token-using-service-account-by-googles-oauth2-package-for-golang/
but then when I add the ?access_token=xxx to the URL, I now just get:
<Error>
<Code>AccessDenied</Code>
<Message>Access denied.</Message>
</Error>
damn.
I reproduced your use case. Assuming you have a key.json file of a service account with the wright permissions on the bucket.
To authorize requests from the command line or for testing, you can
use the curl command with the following syntax:
curl -H "Authorization: Bearer ACCESS_TOKEN"
"https://storage.googleapis.com/storage/v1/b/example-bucket/o"
For local testing, you can use the gcloud auth application-default
print-access-token command to generate a token.
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"os"
"net/http"
"golang.org/x/oauth2"
"golang.org/x/oauth2/google"
"golang.org/x/oauth2/jwt"
"log"
)
func serviceAccount(credentialFile string) (*oauth2.Token, error) {
b, err := ioutil.ReadFile(credentialFile)
if err != nil {
return nil, err
}
var c = struct {
Email string `json:"client_email"`
PrivateKey string `json:"private_key"`
}{}
json.Unmarshal(b, &c)
config := &jwt.Config{
Email: c.Email,
PrivateKey: []byte(c.PrivateKey),
Scopes: []string{
"https://www.googleapis.com/auth/cloud-platform",
},
TokenURL: google.JWTTokenURL,
}
token, err := config.TokenSource(oauth2.NoContext).Token()
if err != nil {
return nil, err
}
return token, nil
}
func main() {
token, err := serviceAccount("key.json") // Please set here
if err != nil {
fmt.Println(err)
os.Exit(1)
}
url := "https://storage.googleapis.com/storage/v1/b/your-bucket/o/your-file?alt=media"
// Create a Bearer string by appending string access token
var bearer = "Bearer " + token.AccessToken
// Create a new request using http
req, err := http.NewRequest("GET", url, nil)
// add authorization header to the req
req.Header.Add("Authorization", bearer)
// Send req using http Client
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
log.Println("Error on response.\n[ERRO] -", err)
}
body, _ := ioutil.ReadAll(resp.Body)
log.Println(string([]byte(body)))
}

Consume a DELETE endpoint from Go

I am working in a Go project, and I need to perform some operations over an external API: GET, PUT, POST and DELETE. Currently I am using net/http, and I created a &http.Client{} to make GET and PUT. That is working as expected.
Now I need to perform a DELETE and I cannot find anything about it. Is it supported? Basically, I need to call a URL like this:
somedomain.com/theresource/:id
Method: DELETE
How can I perform that?
Here is a small example of how to do it:
package main
import (
"fmt"
"io/ioutil"
"net/http"
)
func sendRequest() {
// Request (DELETE http://www.example.com/bucket/sample)
// Create client
client := &http.Client{}
// Create request
req, err := http.NewRequest("DELETE", "http://www.example.com/bucket/sample", nil)
if err != nil {
fmt.Println(err)
return
}
// Fetch Request
resp, err := client.Do(req)
if err != nil {
fmt.Println(err)
return
}
defer resp.Body.Close()
// Read Response Body
respBody, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println(err)
return
}
// Display Results
fmt.Println("response Status : ", resp.Status)
fmt.Println("response Headers : ", resp.Header)
fmt.Println("response Body : ", string(respBody))
}

Post request with data in go

I'm trying to access an API like this:
package main
import (
"bytes"
"fmt"
"io/ioutil"
"net/http"
"net/url"
)
func main() {
apiUrl := "https://example.com/api/"
data := url.Values{}
data.Set("api_token", "MY_KEY")
data.Add("action", "list_projects")
req, _ := http.NewRequest("POST", apiUrl, bytes.NewBufferString(data.Encode()))
client := &http.Client{}
resp, err := client.Do(req)
defer resp.Body.Close()
if err == nil {
body, _ := ioutil.ReadAll(resp.Body)
fmt.Println(resp.Status)
fmt.Println(string(body))
}
}
But the response from an API tells me there was no data in POST request.
If I do it like this with curl, it works:
$ curl -X POST "https://example.com/api/" -d "api_token=MY_KEY" -d "action=list_projects"
You may want to use this form of request
resp, err := http.PostForm("http://example.com/form",
url.Values{"key": {"Value"}, "id": {"123"}})
or use the right mime type :
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
and encode data
strings.NewReader(data.Encode())
It's better if you test err != nil and return if necessary. This code may not work cause the request failed.
defer resp.Body.Close()
instead use this pattern:
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
fmt.Println(err)
return
}
defer resp.Body.Close()
body, _ := ioutil.ReadAll(resp.Body)
fmt.Println(resp.Status)
fmt.Println(string(body))
So you can see in the console if the request failed or not

Resources