I just created a lambda and have given it the default VPC, Security Group, and Subnets. Gave it a role which has AWSLambdaVPCAccessExecutionRole. Verified outbound rules show 0.0.0.0/0 for all ports and protocols. Verified that lambda.amazonaws.com is a trusted entity on the policy.
Gave it code that is this (which works locally):
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
)
func main() {
res, err := http.Get("http://www.google.com/robots.txt")
if err != nil {
log.Fatal(err)
}
robots, err := ioutil.ReadAll(res.Body)
res.Body.Close()
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s", robots)
}
And I use the "test" function in lambda, and all I get for my troubles is:
Get "http://www.google.com/robots.txt": dial tcp 172.217.1.196:80: i/o timeout
I've tried looking through every AWS document on the subject, and it doesn't appear that I'm missing something, but maybe somebody else would know?
I wasn't originally trying to access the google robots.txt file, but after getting the same result no matter what I tried, I figured I would pull an example straight from the documentation to rule out anything I'm doing. At least I know my code is executing, otherwise we wouldn't get this far. Any ideas at all for what to try?
The one thing I overlooked in here is the subnets need a public NAT gateway… not an internet gateway. Once we added those, it was able to reach outside the AWS walls and downloaded the needed files.
Related
I'd like to profile my Go HTTP server application to see if there are places where it can be optimized. I'm using the fasthttp package with the fasthttp/router package, and I'm struggling to figure out how to hook up pprof.
The basic setup looks like this, obviously very abridged:
func main() {
flag.Parse()
r := router.New()
r.GET("/", index)
r.GET("/myroute", myrouteFn)
h := r.Handler
if err := limitedListenAndServe(*addr, fasthttplogger.Tiny(h)); err != nil {
log.Fatalf("Error in ListenAndServe: %s", err)
}
}
First, I tried following a fairly straightforward guide like this one, and added this line in my main() function (per the guide) in addition to the corresponding import. That changed the above to this:
func main() {
flag.Parse()
r := router.New()
r.GET("/", index)
r.GET("/myroute", myrouteFn)
h := r.Handler
if err := limitedListenAndServe(*addr, fasthttplogger.Tiny(h)); err != nil {
log.Fatalf("Error in ListenAndServe: %s", err)
}
defer profile.Start().Stop()
}
After doing that, I ran my program, made a bunch of requests that I was interested in profiling, and then terminated the server. That created a cpu.pprof file, but the file was empty (zero bytes) when I went to run it through the go tool pprof command to generate the graph.
After a bit more sleuthing I found this Gist that I suspect would work if I were using totally vanilla fasthttp.
Trying to combine that with my application, I'm still stuck. Conceptually, I think the solution is to use the fasthttpadaptor to convert the net/http/pprof to a fasthttp route. But the Gist that I was looking at is using its own router to do the mux, I'd and rather not rewrite all the routes using a different router in my server.
How would I go about making the profiling data available here?
You can use net/http/pprof for profiling.
fasthttp provides custom implementation for the same. You can use that same as the net/http/pprof.
To use this, register the handler as:
import "github.com/valyala/fasthttp/pprofhandler"
...
r.GET("/debug/pprof/{profile:*}", pprofhandler.PprofHandler)
Then you can use this to profile.
go tool pprof http://host:port/debug/pprof/profile
Or you can also visit http://host:port/debug/pprof/ to see more types of profiles available.
I have a very strange problem with a simple HTTP Get Request in Golang.
Every request in Golang to https://www.alltron.ch/json/searchSuggestion?searchTerm=notebook needs about 6-8 seconds (!)
If same request fired in Chrome, with Postman or with Powershell it needs less than a second.
Does somebody has a clue why this happens?
My Code:
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
)
func main() {
client := &http.Client{}
req, _ := http.NewRequest("GET", "https://www.alltron.ch/json/searchSuggestion?searchTerm=notebook", nil)
response, err := client.Do(req)
if err != nil && response == nil {
log.Fatalf("Error on request. %v", err)
}
defer response.Body.Close()
body, err := ioutil.ReadAll(response.Body)
if err != nil {
log.Fatalf("Couldn't get response body. %v", err)
}
fmt.Print(string(body))
}
The site you are trying to access is behind the Akamai CDN:
$ dig www.alltron.ch
...
www.alltron.ch. 152 IN CNAME competec.botmanager.edgekey.net.
competec.botmanager.edgekey.net. 7052 IN CNAME e9179.f.akamaiedge.net.
e9179.f.akamaiedge.net. 162 IN A 2.20.176.40
Akamai offers its customers a detection of web clients which are not browsers so that the customers can keep bots away or slowing bots down.
As can be seen from Strange CURL issue with a particular website SSL certificate and Scraping attempts getting 403 error this kind of detection mainly cares about having a Accept-Language header, having a Connection header with the value Keep-Alive and having a User-Agent which matches Mozilla/....
This means the following code changes result in an immediate response:
req, _ := http.NewRequest("GET", "https://www.alltron.ch/json/searchSuggestion?searchTerm=notebook", nil)
req.Header.Set("Connection","Keep-Alive")
req.Header.Set("Accept-Language","en-US")
req.Header.Set("User-Agent","Mozilla/5.0")
Still, the site obviously does not like bots and you should adhere to these wishes and not stress the site too much (like doing lots of information scraping). And, the bot detection done by Akamai might change without notice, i.e. even if this code fixes the problem now it might no longer work in the future. Such changes will be especially true if many clients bypass the bot detection.
try to disable cache in your chrome and compare to golang
I am trying to understand GoLang "Go" together with gRPC and to make a simple service scalable.
Lets say I have a client1 that calls a service1(adds numbers) that calls service2(determines if the result is prime), and service2 returns the result to service1 that returns the result to client1 all via gRPC.
When I use protocol buffers "proto3" and generate the Go code via protoc.
I get generated methods that call the service in one particular way.
I see no distinction to call the methods asynchronously "Go".
And the underlying call seems to be "Invoke" which I believe is synchronous,the call returns once a result is received.
How do I make service1 "performant", I know I can run this in a cluster and have copies, but that would mean I can only serve clients as per the amount of instances within the cluster.
I want a "single" service to be able to serve multiple clients(e.g. 1000) .
Here is a simple server and I am not sure if this is performant or not:
I do know that the getprime function does dial every time,
and this could probably be moved to make this dial persist and be re-used; But more importantly I want to make a simple performant scaleable service and get a good understanding.
(A)
Perhaps the whole design is incorrect and the service1 should just return
as soon as the instruction is received "ack", do the addition and send the next request to sercice2 which determines if the answer is prime or not; again service2 just responds with an acknowledgement of the request being received. Once prime is determined by the service2 a call is made to the client with an answer.
If (A) above is the better approach, then still please explain the bottlenecks below; what happens when multiple clients are processed?
The call to "Listen" does what, "blocks, or does not block", etc.
package main
import (
pb "demo/internal/pkg/proto_gen/calc"
"fmt"
"golang.org/x/net/context"
"google.golang.org/grpc"
"google.golang.org/grpc/reflection"
"log"
"net"
)
const (
port = ":8080"
)
type service struct {
}
func (s *service) Calculate(ctx context.Context, req *pb.Instruction) (*pb.Response, error) {
var answer float64
answer = req.Number1 + req.Number2
// call service prime
p := getprime(int(answer))
pa := pb.PrimeAnswer{Prime: p}
return &pb.Response{Answer: answer, Prime: &pa}, nil
}
const (
primeAddress = "127.0.0.1:8089"
)
func getprime(number int) bool {
conn, err := grpc.Dial(primeAddress, grpc.WithInsecure())
if err != nil {
log.Fatalf("Did not connect to prime service: %v", err)
}
defer conn.Close()
client := pb.NewPrimeServiceClient(conn)
p := pb.PrimeMessage{"", float64(number)}
r, err := client.Prime(context.Background(), &p)
if err != nil {
log.Fatalf("Call to prime service failed: %v", err)
}
return r.Prime
}
func main() {
lis, err := net.Listen("tcp", port)
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
s := grpc.NewServer()
pb.RegisterCalculatorServer(s, &service{})
reflection.Register(s)
if err := s.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
}
Thanks for your question. It is true that gRPC-Go is sync only; that is your Unary RPC(the one in your example) will return only when the RPC has finished (got a response from the server).
About performance:
The Dial operation establishes an underlying connection which may be expensive. So it not wise to do it every time getprime is called. A better way is to create a client, keep it around and make calls to the prime server on it. This way only first RPC incurs the cost of connection.
For each RPC request a server gets we launch a goroutine to process that request. So in general, this should scale fairly well.
About (A): It is not uncommon for a service handler to make an RPC call to yet another server and wait for its response before returning back.
Note that there's no way for a server to make call to the client.
To phrase what JimB said as an answer: "Synchronous" means that the function that makes the remote call waits for a reply before continuing, not that the whole server or client does. The server is normally multithreaded, even when processing synchronous calls; it can accept and work on a second call while it's responding to the first.
And similarly, if a client has multiple concurrent tasks that each have a gRPC call running, that won't block the process. Clients like that could include net/http servers serving end users, or gRPC servers handling multiple RPCs.
Where you might add explicit go statements is if you want to do something else from the specific function making the RPC call. For example, if you want to issue several RPC calls at once then wait for all their results to come in, you could write code following the examples of fan-out calls.
I'm trying to create (and later close) a simple TCP port forward over SSH with Go. I'm new to Golang and statically typed languages. (Coming from Ruby.)
In a terminal I would simply run ssh -L 9000:localhost:9999 user#server.com and this accomplishes what I need. I want to do the same, programmatically with Go.
I have tried using this example as a starting point and this recent test to try to understand what to do, but now I have a pile of confusing jumbled code when it seems like this is actually a very simple thing to do.
Any help would be very much appreciated! :-)
I finally figured out how to do this, I got hints from schmichael in an IRC channel. Thanks to all!
EDIT: A little explanation:
A big part of the problem I was having was that I did not realize a local net.Listener (not just a local net.Conn) needed setup to receive a local request and create the net.Conn before forwarding the bytes.
Also, there exist both port forwards and reverse port forwards and I hadn't previously thought in detail about the fact that a regular port forward also sends bytes back, so copying the remote reader to local writer was not something I had implemented, yet it's very much needed.
Here is an attempt to relate the essence of what this code does:
Listen on local port 9000.
Upon attempted read from local port 9000: (listener.Accept()),
Accept connection and return a local io.Reader and io.Writer and,
Connect to remote server and,
Connect to remote port 9999 returning a io.Reader and io.Writer.
Continually copy local io.Reader bytes to remote io.Writer,
Continually copy remote io.Reader bytes to local io.Writer.
Here is the code:
package main
// Forward from local port 9000 to remote port 9999
import (
"io"
"log"
"net"
"golang.org/x/crypto/ssh"
)
var (
username = "root"
password = "password"
serverAddrString = "192.168.1.100:22"
localAddrString = "localhost:9000"
remoteAddrString = "localhost:9999"
)
func forward(localConn net.Conn, config *ssh.ClientConfig) {
// Setup sshClientConn (type *ssh.ClientConn)
sshClientConn, err := ssh.Dial("tcp", serverAddrString, config)
if err != nil {
log.Fatalf("ssh.Dial failed: %s", err)
}
// Setup sshConn (type net.Conn)
sshConn, err := sshClientConn.Dial("tcp", remoteAddrString)
// Copy localConn.Reader to sshConn.Writer
go func() {
_, err = io.Copy(sshConn, localConn)
if err != nil {
log.Fatalf("io.Copy failed: %v", err)
}
}()
// Copy sshConn.Reader to localConn.Writer
go func() {
_, err = io.Copy(localConn, sshConn)
if err != nil {
log.Fatalf("io.Copy failed: %v", err)
}
}()
}
func main() {
// Setup SSH config (type *ssh.ClientConfig)
config := &ssh.ClientConfig{
User: username,
Auth: []ssh.AuthMethod{
ssh.Password(password),
},
}
// Setup localListener (type net.Listener)
localListener, err := net.Listen("tcp", localAddrString)
if err != nil {
log.Fatalf("net.Listen failed: %v", err)
}
for {
// Setup localConn (type net.Conn)
localConn, err := localListener.Accept()
if err != nil {
log.Fatalf("listen.Accept failed: %v", err)
}
go forward(localConn, config)
}
}
I have used your (damick) example code to build a tiny open source tool: SSHTunnel
https://github.com/SommerEngineering/SSHTunnel
Therefore, the code is freely available at GitHub for anyone: Please feel free to use it for learning purposes or for anything else :) I have mentioned your nickname and also linked to this question.
Best regards,
Thorsten.
I'v finished a simple SSH port forward tool called mallory.
It provides HTTP proxy instead of SOCKS proxy, which is really similar to ssh -D.
The core code is similar to damick's answer.
Create ClientConfig
ssh.Dial to remote SSH server with the config and return Client
Now you can use Client.Dial to forward anything you like.
Dial initiates a connection to the addr from the remote host. The resulting connection has a zero LocalAddr() and RemoteAddr().
If you want to serve a SOCKS proxy server, use Client.Dial to connect to the remote server.
I Wrote a tool,Called gosshtool,with this tool,you can easy to create a simple TCP port forward over SSH with Go.https://github.com/scottkiss/gosshtool
I use this tool implemented a port forward server example project:
https://github.com/scottkiss/gooverssh
Right now I'm fetching urls from indiegogo as part of a side project using the basic get request template found [here][1]. I then translate the byte data into a string using
responseText, err:= ioutil.ReadAll(response.Body)
trueText:= string(responseText)
with appropriate error handling where needed
It works fine for repeated attempts at getting and some other urls of varying length(at least as large as the previous url and some longer than the next).
Strangely, when I attempt to get it breaks and throws a runtime error of
panic: runtime error: index out of range
and exits with a status of 2. I'm curious as to what the issue could be.
I know it isn't indiegogo getting angry about my once a minute requests and cutting my connection because I can request continiously for 20 minutes at with no issue. Give it a bit of downtime and it still completely breaks on
Thanks for the assistance
EDIT, it appears as though it was a malformed bit of html in some of the pages that messed with a loop I was running based on the content that managed to break go in the runtime on only some urls. Thanks for the help
[1]:
There is no error when getting from the url and converting the body to the Go string type. For example,
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
)
func main() {
url := "http://www.indiegogo.com/projects/culcharge-smallest-usb-charge-and-data-cable-for-iphone-and-android"
res, err := http.Get(url)
if err != nil {
log.Fatal(err)
}
body, err := ioutil.ReadAll(res.Body)
res.Body.Close()
if err != nil {
log.Fatal(err)
}
text := string(body)
fmt.Println(len(body), len(text))
}
Output:
66363 66363
You didn't provide us with a small fragment of code which compiles, runs, and fails in the manner you describe. That leaves us all guessing.