We are using Golang to communicate with an external service using HTTPS. When attempting the request using cURL, we get success. When using Go, the certificate appears to be ignored, yielding a 403 from the external service.
We have been unable to spot any differences in the cURL-request vs the Golang code. Can somebody help us find a difference?
The cURL-request gives us a proper JSON response. The Go code gives:
2020/09/07 15:05:57 request error: perform request: api response: 403 Forbidden
Working cURL-request (user agent for debug purposes):
curl -X GET --http1.1 -i -v --key client.key.pem --cacert ca.pem --cert client.pem "https://[redacted]/path/to/endpoint" -H "Accept: application/json; charset=utf-8" -H "User-Agent: Apache-HttpClient/4.5.5 (Java/12.0.1)" -H "X-Identifier: [redacted]" -H "Accept-Encoding: gzip, deflate" -H "Connection: Keep-Alive"
Golang code yielding a 403:
(note: files ca.pem, client.pem (cert) and client.key.pem must be in same directory. run script as go run catest.go --url "https://[redacted]/path/to/endpoint" --identifier [redacted])
package main
import (
"crypto/tls"
"crypto/x509"
"flag"
"fmt"
"io/ioutil"
"log"
"net/http"
)
func main() {
// Get URL to call.
url := flag.String("url", "", "URL to call")
identifier := flag.String("identifier", "", "the X-Identifier value")
flag.Parse()
if url == nil || identifier == nil || *url == "" || *identifier == "" {
log.Fatal("'url' and 'identifier' arguments must be provided")
}
// Set up certificates
caPEM, err := ioutil.ReadFile("ca.pem")
if err != nil {
log.Fatalf("unable to read 'ca.pem' in current directory: %v", err)
}
clientPEM, err := ioutil.ReadFile("client.pem")
if err != nil {
log.Fatalf("unable to read 'client.pem' in current directory: %v", err)
}
clientKeyPEM, err := ioutil.ReadFile("client.key.pem")
if err != nil {
log.Fatalf("unable to read 'client.key.pem' in current directory: %v", err)
}
// Make calls.
client, err := configureClient(caPEM, clientPEM, clientKeyPEM)
if err != nil {
log.Fatalf("unable to setup client: %v", err)
}
_, err = performRequest(client, *url, *identifier)
if err != nil {
log.Fatalf("request error: %v", err)
}
log.Printf("request successful")
}
func configureClient(caCertPEM, clientCertPEM, clientKeyPEM []byte) (*http.Client, error) {
// Load the CA certificate.
caCertPool, err := x509.SystemCertPool()
if err != nil {
return nil, fmt.Errorf("configure client: load cert pool: %w", err)
}
// Append root CA cert from parameter
ok := caCertPool.AppendCertsFromPEM(caCertPEM)
if !ok {
return nil, fmt.Errorf("configure client: could not append ca certificate")
}
// Load the client certificate.
clientCert, err := tls.X509KeyPair(clientCertPEM, clientKeyPEM)
if err != nil {
return nil, fmt.Errorf("configure client: load client certificate: %w", err)
}
// Setup HTTPS client.
tlsConfig := &tls.Config{
RootCAs: caCertPool,
Certificates: []tls.Certificate{clientCert},
Renegotiation: tls.RenegotiateOnceAsClient,
}
tlsConfig.BuildNameToCertificate()
transport := &http.Transport{TLSClientConfig: tlsConfig}
client := &http.Client{Transport: transport}
return client, nil
}
func performRequest(client *http.Client, u, identifier string) ([]byte, error) {
if client == nil {
return nil, fmt.Errorf("perform request: nil client")
}
// Prepare request
req, err := http.NewRequest(http.MethodGet, u, nil)
if err != nil {
return nil, fmt.Errorf("perform request: create GET request: %w", err)
}
// Add same headers as cURL.
req.Header.Add("Accept", "application/json; charset=utf-8")
req.Header.Add("User-Agent", "Apache-HttpClient/4.5.5 (Java/12.0.1)")
req.Header.Add("Accept-Encoding", "gzip, deflate")
req.Header.Add("Connection", "Keep-Alive")
req.Header.Add("X-Identifier", identifier)
// Send request
resp, err := client.Do(req)
if err != nil {
return nil, fmt.Errorf("perform request: client do: %w", err)
}
defer resp.Body.Close()
switch resp.StatusCode {
case http.StatusOK:
break
case http.StatusUnauthorized:
return nil, fmt.Errorf("perform request: api response: unauthorized")
case http.StatusBadRequest:
return nil, fmt.Errorf("perform request: api response: bad request")
default:
return nil, fmt.Errorf("perform request: api response: %v", resp.Status)
}
data, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, fmt.Errorf("perform request: read response body: %w", err)
}
return data, nil
}
Related
I've got a http.Client in go and I want it to update every content type for every response to application/json (even though it might not be the case) even before it process the response.
Which attribute shall I override?
Context: the underlying issue there's a bug in the third party API where the real content type is application/json but it's set to the other thing (incorrectly).
Code snippet:
...
requestURL := fmt.Sprintf("http://localhost:%d", serverPort)
req, err := http.NewRequest(http.MethodGet, requestURL, nil)
if err != nil {
fmt.Printf("client: could not create request: %s\n", err)
os.Exit(1)
}
res, err := http.DefaultClient.Do(req)
if err != nil {
fmt.Printf("client: error making http request: %s\n", err)
os.Exit(1)
}
fmt.Printf("client: got response!\n")
fmt.Printf("client: status code: %d\n", res.StatusCode)
resBody, err := ioutil.ReadAll(res.Body)
if err != nil {
fmt.Printf("client: could not read response body: %s\n", err)
os.Exit(1)
}
fmt.Printf("client: response body: %s\n", resBody)
}
package main
import (
"fmt"
"net/http"
)
type MyRoundTripper struct {
httprt http.RoundTripper
}
func (rt MyRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
res, err := rt.httprt.RoundTrip(req)
if err != nil {
fmt.Printf("Error: %v", err)
} else {
res.Header.Set("Content-Type", "application/json")
}
return res, err
}
func main() {
client := &http.Client{Transport: MyRoundTripper{http.DefaultTransport}}
resp, err := client.Get("https://example.com")
if err != nil {
// handle error
}
fmt.Printf("%+v\n", resp.Header)
}
I have a Vertex AI model deployed on an endpoint and want to do some prediction from my app in Golang.
To do this I create code inspired by this example : https://cloud.google.com/go/docs/reference/cloud.google.com/go/aiplatform/latest/apiv1?hl=en
const file = "MY_BASE64_IMAGE"
func main() {
ctx := context.Background()
c, err := aiplatform.NewPredictionClient(cox)
if err != nil {
log.Printf("QueryVertex NewPredictionClient - Err:%s", err)
}
defer c.Close()
parameters, err := structpb.NewValue(map[string]interface{}{
"confidenceThreshold": 0.2,
"maxPredictions": 5,
})
if err != nil {
log.Printf("QueryVertex structpb.NewValue parameters - Err:%s", err)
}
instance, err := structpb.NewValue(map[string]interface{}{
"content": file,
})
if err != nil {
log.Printf("QueryVertex structpb.NewValue instance - Err:%s", err)
}
reqP := &aiplatformpb.PredictRequest{
Endpoint: "projects/PROJECT_ID/locations/LOCATION_ID/endpoints/ENDPOINT_ID",
Instances: []*structpb.Value{instance},
Parameters: parameters,
}
resp, err := c.Predict(cox, reqP)
if err != nil {
log.Printf("QueryVertex Predict - Err:%s", err)
}
log.Printf("QueryVertex Res:%+v", resp)
}
I put the path to my service account JSON file on GOOGLE_APPLICATION_CREDENTIALS environment variable.
But when I run my test app I obtain this error message:
QueryVertex Predict - Err:rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type "text/html; charset=UTF-8"
QueryVertex Res:<nil>
As #DazWilkin suggested, configure the client option to specify the specific regional endpoint with a port 443:
option.WithEndpoint("<region>-aiplatform.googleapis.com:443")
Try like below:
func main() {
ctx := context.Background()
c, err := aiplatform.NewPredictionClient(
ctx,
option.WithEndpoint("<region>-aiplatform.googleapis.com:443"),
)
if err != nil {
log.Printf("QueryVertex NewPredictionClient - Err:%s", err)
}
defer c.Close()
.
.
I'm unfamiliar with Google's (Vertex?) AI Platform and unable to test this hypothesis but it appears that the API uses location-specific endpoints.
Can you try configuring the client's ClientOption to specify the specific regional endpoint, i.e.:
url := fmt.Sprintf("https://%s-aiplatform.googleapis.com", location)
opts := []option.ClientOption{
option.WithEndpoint(url),
}
And:
package main
import (
"context"
"fmt"
"log"
"os"
aiplatform "cloud.google.com/go/aiplatform/apiv1"
"google.golang.org/api/option"
aiplatformpb "google.golang.org/genproto/googleapis/cloud/aiplatform/v1"
"google.golang.org/protobuf/types/known/structpb"
)
const file = "MY_BASE64_IMAGE"
func main() {
// Values from the environment
project := os.Getenv("PROJECT")
location := os.Getenv("LOCATION")
endpoint := os.Getenv("ENDPOINT")
ctx := context.Background()
// Configure the client with a region-specific endpoint
url := fmt.Sprintf("https://%s-aiplatform.googleapis.com", location)
opts := []option.ClientOption{
option.WithEndpoint(url),
}
c, err := aiplatform.NewPredictionClient(ctx, opts...)
if err != nil {
log.Fatal(err)
}
defer c.Close()
parameters, err := structpb.NewValue(map[string]interface{}{
"confidenceThreshold": 0.2,
"maxPredictions": 5,
})
if err != nil {
log.Fatal(err)
}
instance, err := structpb.NewValue(map[string]interface{}{
"content": file,
})
if err != nil {
log.Printf("QueryVertex structpb.NewValue instance - Err:%s", err)
}
rqst := &aiplatformpb.PredictRequest{
Endpoint: fmt.Sprintf("projects/%s/locations/%s/endpoints/%s",
project,
location,
endpoint,
),
Instances: []*structpb.Value{
instance,
},
Parameters: parameters,
}
resp, err := c.Predict(ctx, rqst)
if err != nil {
log.Fatal(err)
}
log.Printf("QueryVertex Res:%+v", resp)
}
Try to do something like this
[...]
url := fmt.Sprintf("%s-aiplatform.googleapis.com:443", location)
[..]
I’m trying to create a ssh server in go using the x/crypto/ssh module but i can’t manage to make the public key authentification work.
I tried the ExampleNewServerConn() function in the ssh/example_test.go file (in the https://go.googlesource.com/crypto repo) but the public key method doesn’t work, it looks like the server isn’t advertising the right algorithms because i get this line when trying to connect with a ssh client :
debug1: send_pubkey_test: no mutual signature algorithm
If i add -o PubkeyAcceptedKeyTypes=+ssh-rsa the public key login works, but this rsa method is deprecated, i would like to use another public key type, how can i do that ?
Thanks in advance.
Edit : here is the code that i used to test
package main
import (
"fmt"
"io/ioutil"
"log"
"net"
"golang.org/x/crypto/ssh"
terminal "golang.org/x/term"
)
func main() {
authorizedKeysBytes, err := ioutil.ReadFile("authorized_keys")
if err != nil {
log.Fatalf("Failed to load authorized_keys, err: %v", err)
}
authorizedKeysMap := map[string]bool{}
for len(authorizedKeysBytes) > 0 {
pubKey, _, _, rest, err := ssh.ParseAuthorizedKey(authorizedKeysBytes)
if err != nil {
log.Fatal(err)
}
authorizedKeysMap[string(pubKey.Marshal())] = true
authorizedKeysBytes = rest
}
config := &ssh.ServerConfig{
PasswordCallback: func(c ssh.ConnMetadata, pass []byte) (*ssh.Permissions, error) {
if c.User() == "testuser" && string(pass) == "tiger" {
return nil, nil
}
return nil, fmt.Errorf("password rejected for %q", c.User())
},
PublicKeyCallback: func(c ssh.ConnMetadata, pubKey ssh.PublicKey) (*ssh.Permissions, error) {
if authorizedKeysMap[string(pubKey.Marshal())] {
return &ssh.Permissions{
// Record the public key used for authentication.
Extensions: map[string]string{
"pubkey-fp": ssh.FingerprintSHA256(pubKey),
},
}, nil
}
return nil, fmt.Errorf("unknown public key for %q", c.User())
},
}
privateBytes, err := ioutil.ReadFile("id_rsa")
if err != nil {
log.Fatal("Failed to load private key: ", err)
}
private, err := ssh.ParsePrivateKey(privateBytes)
if err != nil {
log.Fatal("Failed to parse private key: ", err)
}
config.AddHostKey(private)
listener, err := net.Listen("tcp", "0.0.0.0:2022")
if err != nil {
log.Fatal("failed to listen for connection: ", err)
}
nConn, err := listener.Accept()
if err != nil {
log.Fatal("failed to accept incoming connection: ", err)
}
conn, chans, reqs, err := ssh.NewServerConn(nConn, config)
if err != nil {
log.Fatal("failed to handshake: ", err)
}
log.Printf("logged in with key %s", conn.Permissions.Extensions["pubkey-fp"])
go ssh.DiscardRequests(reqs)
for newChannel := range chans {
if newChannel.ChannelType() != "session" {
newChannel.Reject(ssh.UnknownChannelType, "unknown channel type")
continue
}
channel, requests, err := newChannel.Accept()
if err != nil {
log.Fatalf("Could not accept channel: %v", err)
}
go func(in <-chan *ssh.Request) {
for req := range in {
req.Reply(req.Type == "shell", nil)
}
}(requests)
term := terminal.NewTerminal(channel, "> ")
go func() {
defer channel.Close()
for {
line, err := term.ReadLine()
if err != nil {
break
}
fmt.Println(line)
}
}()
}
}
I found why the client and the server can’t communicate, the rsa-sha2 algorithms are not yet implemented in the x/crypto library. There is an issue about it on github : https://github.com/golang/go/issues/49952 .
A temporary solution is to add
replace golang.org/x/crypto => github.com/rmohr/crypto v0.0.0-20211203105847-e4ed9664ac54
at the end of your go.mod file, it uses a x/crypto fork from #rmohr that works with rsa-sha2.
This is the easy way to do it, let letsencrypt handle the certificates for you :)
func main() {
r := mux.NewRouter()
r.HandleFunc("/index", index)
certManager := autocert.Manager{
Prompt: autocert.AcceptTOS,
HostPolicy: autocert.HostWhitelist("www.example.com"), // replace with your domain
Cache: autocert.DirCache("certs"),
}
srv := &http.Server{
Handler: r,
Addr: ":https",
WriteTimeout: 5 * time.Second,
ReadTimeout: 5 * time.Second,
TLSConfig: &tls.Config{
GetCertificate: certManager.GetCertificate,
},
}
go http.ListenAndServe(":http", certManager.HTTPHandler(nil)) //nolint
log.Fatal(srv.ListenAndServeTLS("", ""))
}
Hi I am interested in adding certificates to the certPool while the server is running, but it seems like it's not picked up. Do I need to relaunch the server for this to work?
package main
import (
"crypto/tls"
"crypto/x509"
"fmt"
"io/ioutil"
"log"
"net/http"
)
func main() {
handler := http.NewServeMux()
// verify the cert if given (or check for jwt token)
handler.HandleFunc("/auth", func(w http.ResponseWriter, r *http.Request) {
if certs := r.TLS.PeerCertificates; len(certs) > 0 {
cert := certs[0]
fmt.Println(cert.Subject.CommonName)
w.WriteHeader(http.StatusOK)
fmt.Fprintln(w, "Cert Valid")
return
}
// ...
})
// the cert pool that holds the client CAs
certPool := x509.NewCertPool()
// post a new CA and add it to the pool
handler.HandleFunc("/ca", func(rw http.ResponseWriter, r *http.Request) {
caCertFile, err := ioutil.ReadAll(r.Body)
if err != nil {
log.Printf("error reading CA certificate: %v", err)
rw.WriteHeader(http.StatusUnprocessableEntity)
return
}
certPool.AppendCertsFromPEM(caCertFile)
rw.WriteHeader(http.StatusCreated)
})
server := http.Server{
Addr: ":9090",
Handler: handler,
TLSConfig: &tls.Config{
ClientAuth: tls.VerifyClientCertIfGiven,
ClientCAs: certPool,
MinVersion: tls.VersionTLS12,
},
}
if err := server.ListenAndServeTLS("certs/server/tls.crt", "certs/server/tls.key"); err != nil {
log.Fatalf("error listening to port: %v", err)
}
}
I am posting the cert with curl
curl -k -X POST https://localhost:9090/ca -d #test-ca.pem
Based, on one comment, I have tried the below but it doesn't seem to work either.
certFile, err := ioutil.ReadFile("certs/server/tls.crt")
if err != nil {
log.Fatal(err)
}
keyFile, err := ioutil.ReadFile("certs/server/tls.key")
if err != nil {
log.Fatal(err)
}
cert, err := tls.X509KeyPair(certFile, keyFile)
if err != nil {
log.Fatal(err)
}
listener, err := tls.Listen("tcp", ":9090", &tls.Config{
ClientAuth: tls.VerifyClientCertIfGiven,
ClientCAs: certPool,
Certificates: []tls.Certificate{cert},
})
if err != nil {
log.Fatalf("error creating listener: %v", err)
}
if err = http.Serve(listener, handler); err != nil {
log.Fatalf("error serving: %v", err)
}
I have defined two funcs that do slightly different things but are syntactically the same.
Functions in question send POST requests to an api.
The duplication occurs in constructing the request, adding headers, etc.
How can I refactor the code to remove said duplication.
package main
import (
"bytes"
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
"net/http/httputil"
)
type token struct {
Token string
}
type config struct {
Foo string
}
func main() {
token, err := getAuthToken()
if err != nil {
log.Fatal(err)
}
config, err := getConfig("foo", token)
if err != nil {
log.Fatal(err)
}
_ = config
}
func getAuthToken() (string, error) {
endpoint := "foo"
body := struct {
UserName string `json:"username"`
Password string `json:"password"`
}{
UserName: "foo",
Password: "bar",
}
jsnBytes, err := json.Marshal(body)
if err != nil {
return "", err
}
req, err := http.NewRequest("POST", endpoint, bytes.NewReader(jsnBytes))
if err != nil {
return "", fmt.Errorf("Unable to create request. %v", err)
}
req.Header.Add("Content-Type", "application/json")
dump, err := httputil.DumpRequest(req, true)
if err != nil {
return "", fmt.Errorf("Could not dump request. ", err)
}
log.Println("Request: ", string(dump))
client := http.Client{}
log.Println("Initiating http request")
resp, err := client.Do(req)
if err != nil {
return "", fmt.Errorf("HTTP Error: %v", err)
}
defer resp.Body.Close()
bytes, err := ioutil.ReadAll(resp.Body)
if err != nil {
return "", fmt.Errorf("Error reading response body: %v", err)
}
var token token
err = json.Unmarshal(bytes, &token)
if err != nil {
return "", fmt.Errorf("Could not unamrshal json. ", err)
}
return token.Token, nil
}
func getConfig(id string, token string) (*config, error) {
endpoint := "foo"
body := struct {
ID string `json:"id"`
}{
ID: id,
}
jsnBytes, err := json.Marshal(body)
if err != nil {
return nil, err
}
req, err := http.NewRequest("POST", endpoint, bytes.NewReader(jsnBytes))
if err != nil {
return nil, fmt.Errorf("Unable to create request. %v", err)
}
req.Header.Add("Authorization", "Bearer "+token)
req.Header.Add("Content-Type", "application/json")
dump, err := httputil.DumpRequest(req, true)
if err != nil {
return nil, fmt.Errorf("Could not dump request. ", err)
}
log.Println("Request: ", string(dump))
client := http.Client{}
log.Println("Initiating http request")
resp, err := client.Do(req)
if err != nil {
return nil, fmt.Errorf("HTTP Error: %v", err)
}
defer resp.Body.Close()
bytes, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, fmt.Errorf("Error reading response body: %v", err)
}
var config config
err = json.Unmarshal(bytes, &config)
if err != nil {
return nil, fmt.Errorf("Could not unamrshal json. ", err)
}
return &config, nil
}
I would say the essence of sending the request is that you are sending a body to an endpoint and parsing a result. The headers are then optional options that you can add to the request along the way. With this in mind I would make a single common function for sending the request with this signature:
type option func(*http.Request)
func sendRequest(endpoint string, body interface{}, result interface{}, options ...option) error {
Note this is using functional options which Dave Cheney did an excellent description of here:
https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis
The complete code then becomes:
https://play.golang.org/p/GV6FeipIybA
package main
import (
"bytes"
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
"net/http/httputil"
)
type token struct {
Token string
}
type config struct {
Foo string
}
func main() {
token, err := getAuthToken()
if err != nil {
log.Fatal(err)
}
config, err := getConfig("foo", token)
if err != nil {
log.Fatal(err)
}
_ = config
}
func getAuthToken() (string, error) {
endpoint := "foo"
body := struct {
UserName string `json:"username"`
Password string `json:"password"`
}{
UserName: "foo",
Password: "bar",
}
var token token
err := sendRequest(endpoint, body, &token)
if err != nil {
return "", err
}
return token.Token, nil
}
func getConfig(id string, token string) (*config, error) {
endpoint := "foo"
body := struct {
ID string `json:"id"`
}{
ID: id,
}
var config config
err := sendRequest(endpoint, body, &config, header("Content-Type", "application/json"))
if err != nil {
return nil, err
}
return &config, nil
}
type option func(*http.Request)
func header(key, value string) func(*http.Request) {
return func(req *http.Request) {
req.Header.Add(key, value)
}
}
func sendRequest(endpoint string, body interface{}, result interface{}, options ...option) error {
jsnBytes, err := json.Marshal(body)
if err != nil {
return err
}
req, err := http.NewRequest("POST", endpoint, bytes.NewReader(jsnBytes))
if err != nil {
return fmt.Errorf("Unable to create request. %v", err)
}
req.Header.Add("Content-Type", "application/json")
for _, option := range options {
option(req)
}
dump, err := httputil.DumpRequest(req, true)
if err != nil {
return fmt.Errorf("Could not dump request. ", err)
}
log.Println("Request: ", string(dump))
client := http.Client{}
log.Println("Initiating http request")
resp, err := client.Do(req)
if err != nil {
return fmt.Errorf("HTTP Error: %v", err)
}
defer resp.Body.Close()
bytes, err := ioutil.ReadAll(resp.Body)
if err != nil {
return fmt.Errorf("Error reading response body: %v", err)
}
err = json.Unmarshal(bytes, result)
if err != nil {
return fmt.Errorf("Could not unamrshal json. ", err)
}
return nil
}
The way I would do this is to extract the two parts that are common to both request executions: 1) create a request and 2) execute the request.
Gist with new code using HTTP Bin as an example
Creating the request includes setting up the endpoint, headers and marshaling the request body to JSON. In your case, you're also dumping the request to the log, that can also go in there. This is how it would look like:
func buildRequest(endpoint string, body interface{}, extraHeaders map[string]string) (*http.Request, error) {
jsnBytes, err := json.Marshal(body)
if err != nil {
return nil, err
}
req, err := http.NewRequest("POST", endpoint, bytes.NewReader(jsnBytes))
if err != nil {
return nil, err
}
req.Header.Add("Content-Type", "application/json")
for name, value := range extraHeaders {
req.Header.Add(name, value)
}
dump, err := httputil.DumpRequest(req, true)
if err != nil {
return nil, err
}
log.Println("Request: ", string(dump))
return req, nil
}
If you have no extra headers, you can pass nil as the third argument here.
The second part to extract is actually executing the request and unmarshalling the data. This is how the executeRequest would look like:
func executeRequest(req *http.Request, responseBody interface{}) error {
client := http.Client{}
log.Println("Initiating http request")
resp, err := client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
bytes, err := ioutil.ReadAll(resp.Body)
if err != nil {
return err
}
log.Printf("Response is: %s\n", string(bytes))
err = json.Unmarshal(bytes, &responseBody)
return err
}