I am trying to route my requests through a proxy and also sending cert.pem in TLS config. Below code is throwing this error - proxyconnect tcp: tls: first record does not look like a TLS handshake. When I change the proxy URL from https to HTTP, the same code works. However proxy URL with https works in python. Below is my code so far
certs := x509.NewCertPool()
pemFile, err := ioutil.ReadFile("cert.pem")
if err != nil {
return
}
certs.AppendCertsFromPEM(pemFile)
tlsConfig := &tls.Config{
RootCAs: certs,
}
proxyUrl, err := url.Parse("https://someproxyurlhere.com:8080")
if err != nil {
return
}
t := &http.Transport{
TLSClientConfig: tlsConfig,
Proxy: http.ProxyURL(proxyUrl),
}
client := http.Client{
Transport: t,
}
reqBody := "some JSON body here"
buff, err := json.Marshal(reqBody)
if err != nil {
return
}
req, err := http.NewRequest(http.MethodPost, "https://someurlhere.com", bytes.NewBuffer(buff))
if err != nil {
return
}
res, err := client.Do(req)
if err != nil {
// Error here - proxyconnect tcp: tls: first record does not look like a TLS handshake
return
}
defer res.Body.Close()
Python code
import requests
os.environ['HTTPS_PROXY'] = 'https://someproxyurlhere.com:8080'
response = requests.post("https://someurlhere.com",
json={'key': 'value'},
verify='cert.pem')
print(str(response.content))
When I change the proxy URL from https to HTTP, the same code works.
This is because you are using a HTTP proxy which need to be accessed by HTTP and not HTTPS even for https://.. URLs. This is how HTTPS proxying usually works. Note that while in theory a proxy could sniff if it gets a plain or a TLS connection in practice proxies (and servers) use different ports for HTTP and HTTPS - thus if it works on one port with HTTP it will very likely not work on the same port with HTTPS.
proxyconnect tcp: tls: first record does not look like a TLS handshake.
This is because the proxy answers with an plain HTTP error to the strange HTTP request (which is actually the start of the TLS handshake).
However proxy URL with https works in python.
While it looks that it works it actually doesn't. Python requests will still use plain HTTP to the proxy even if https:// was given as URL.
You can try to do a simple TLS connection to the proxy itself, for example with Python or with openssl s_client. It will very likely fail with some handshake error because the proxy isn't actually expecting TLS.
Related
I am trying to mock out a response from a HTTPS web service responding to my Golang package.
In my unit test I am using net/http/httptest.
My function is
ts := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "hello")
}))
defer ts.Close()
tr := &http.Transport{TLSClientConfig: &TLS.Config{InsecureSkipVerify: true}}
client := ts.Client()
client= &http.Client{Transport:tr}
res, err := client.Get(ts.URL)
When I step through the code and my package makes the call to the mock service, it errors with error (crypto/x509.UnknownAuthorityError)
I thought the skip verify is meant ignore verifying any certificates? Yes, I am mocking out certificates when my package is instantiated, but for the purpose of the test, I am not interested in the certificates all I want to do is verify the response coming back and the code after.
What am I doing wrong with mocking a HTTPS request response?
I'm interfacing from my code certain 3rd party software (particularly chirpstack v4) which provides API via GPRC. When deployed locally, it answers via HTTP and I get the response all right.
However in cluster we have the same server deployed with HTTPS (with letsencrypt certificate, not something private) so I'm trying to add corresponding transport layer security settings, but to my surprise I got then
rpc error: code = Internal desc = unexpected HTTP status code received from server: 400 (Bad Request); malformed header: missing HTTP content-type
I tried adding SetHeader to context with content-type: application/grpc but this won't change anything so I'm not sure it is really about header (moreover that it works with plain HTTP). So I wonder, perhaps anyone can point me some mistake in initialization of transport layer security?
// this is used with plain HTTP
//opts := []grpc.DialOption{grpc.WithTransportCredentials(insecure.NewCredentials())}
// this for case of working via HTTPS
opts := []grpc.DialOption{grpc.WithTransportCredentials(credentials.NewTLS(&tls.Config{}))}
// also tried NewClientTLSFromCert(x509.SystemCertPool()) - the same
serverAddr := "our-public-address:443"
ctx := context.Background()
conn, err := grpc.Dial(serverAddr, opts...)
if err != nil {
println("Dial error:", err.Error())
return
}
cli := api.NewInternalServiceClient(conn)
req := &api.LoginRequest{Email: "admin", Password: "admin"}
resp, err := cli.Login(ctx, req) // error arises here
if err != nil {
println("login failed", err.Error())
}
Thanks in advance for hints and suggestions!
I have a specific use case, where I have to use HTTP2 protocol with a proxy configured. This proxy function uses the request to determine, which proxy URL to use.
Since http2.transport{} struct does not have a proxy field, I was using:
httpTransport := &http.Transport{
Proxy: *ProxyFunction, // var ProxyFunction *func(*http.Request) (*url.URL, error)
ForceAttemptHTTP2: true,
}
http2.ConfigureTransport(httpTransport)
But, this does not allow me to send "http://" requests as HTTP2. Only "https://" were sent as HTTP2, "http://" requests were using HTTP1.1
This leads me to
httpTransport := &http.Transport{
Proxy: *config.ProxyFunction,
ForceAttemptHTTP2: true,
}
h2, _ := http2.ConfigureTransports(httpTransport)
h2.AllowHTTP = true
But using this config, on request, I receive
error getting: Get http://google.com: http2: no cached connection was available, similar to https://github.com/golang/go/issues/26479
Since there is no official support, what would be the right approach to achieve this?
Can I use a dialer to achieve this?
Note:
proxy URL is obtained from ProxyFunction *func(*http.Request) (*url.URL, error) and changes based on the request
https is not a requirement and need not be supported.
go version go1.13.7
I connect a client (or a couple of clients) to the websockets endpoint in API Gateway.
Then, I try to post a message back to the client using these guidelines: https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-how-to-call-websocket-api-connections.html
// Send sends a message to a connection ID.
func Send(domain, stage, connectionID, message string) (events.APIGatewayProxyResponse, error) {
session := session.Must(session.NewSession())
endpoint := fmt.Sprintf("https://%s/%s/#connections/%s", domain, stage, connectionID)
apiClient := apigatewaymanagementapi.New(session, aws.NewConfig().WithEndpoint(endpoint))
connectionInput := apigatewaymanagementapi.PostToConnectionInput{
ConnectionId: aws.String(connectionID),
Data: []byte(message),
}
_, err := apiClient.PostToConnection(&connectionInput)
if err != nil {
fmt.Println(err.Error())
return events.APIGatewayProxyResponse{StatusCode: 500}, err
}
return events.APIGatewayProxyResponse{StatusCode: 200}, nil
}
It doesn't matter whether I invoke the Send function locally or a client sends a message and API Gateway invokes my publish Lambda where I loop through the connections and invoke Send for each of them.
The result is always the same.
NotFoundException:
status code: 404, request id: 7bb1546a-c2a7-4e98-92a0-fcc7ae175d7c
Things I've tried:
Escaped #connections and the actual connectionID
Made sure the client connection hasn't timed out
Made sure I have the correct AWS credentials in my environment variables
Made sure my Lambda has permissions to invoke API Gateway
Made sure the endpoint is in the correct format: https://{api-id}.execute-api.{region}.amazonaws.com/{stage}/#connections/{connection_id}
How can I successfully send messages to the clients?
Turns out this line
endpoint := fmt.Sprintf("https://%s/%s/#connections/%s", domain, stage, connectionID)
needs to turn into this
endpoint := fmt.Sprintf("https://%s/%s/", domain, stage)
I'm building 2 apis. One make request to another.
To call the api that receives requests, we need to pass a X-Token Header. I'm doing this with Golang
client := &http.Client{
Transport: &http.Transport{
TLSClientConfig: &tls.Config{
InsecureSkipVerify: true,
},
},
}
req, err := http.NewRequest("GET", "https://localhost:8086/v2/example", nil)
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"Error": err.Error()})
}
req.Header.Add("accept", "application/json")
req.Header.Add("content-type", "application/json")
req.Header.Add("x-token", "a2e63ee01401aaeca78be023dfbb8c59")
resp, err := client.Do(req)
In the other API, i get the http header with gin like this:
token := c.Request.Header.Get("x-token")
I dont know why my header arrives with another value and no X-Token. Thanks!
Result of fmt.Printf("%+v", c.Request.Header):
map[User-Agent:[Go-http-client/1.1] Referer:[https://localhost:8086/v2/example] Accept-Encoding:[gzip]]
I don't know where is my x-token, accept and content-type headers....
IMPORTANT
If i make a request with x-token header on Postman to the requested API i get the right header.
If i change the request address on the API that makes requests, e.g httpbin, i get the right header too....
Helo, guys! i found the solution....
I don't know why yet... but i think golang don't handle no trailing slash url's....
https://localhost:8086/v2/example
is different of
https://localhost:8086/v2/example/
That was my problem....
I just copy and past the golang generated code of postman... and that was the "biggest" difference....
Thanks mr. Postman...