I have two AWS lambdas written in go. One lambda invokes the other like this:
payload, err := json.Marshal(request)
if err != nil {
log.Printf("ERROR: could not marshal request [%v] into model.ChildLambdaRequest - %v\n", request, err)
return false
}
log.Printf("--- debug sending payload: %s", payload)
// Invoke Child
result, err := client.Invoke(&lambda.InvokeInput{
FunctionName: aws.String(lambdaName),
Payload: payload,
})
if err != nil {
log.Printf("ERROR: could not invoke Lambda function client [%v] - %v\n", lambdaName, err)
return false
}
The child lambda completes like this:
return model.EventResponse{Success: true}, nil
I know this is doing two things: 1.) it's finishing the execution and 2.) it's returning a value. Is there any way to separate these two actions, so that I can return a value at the top, but complete execution at a later time?
Yes, you need to pass InvocationType: "Event" to the Invoke call, documented here. The default invocation type is RequestResponse which waits for the response from the invoked Lambda.
It's possible with Destinations in Lambda.
Although I would check out AWS Step Functions, which gives you this functionality together with built-in retries and error handling.
It works with existing Lambda functions and has a pretty easy to use interface.
Last option that could work, you send a message from the first function to an SQS Queue,
and your second lambda is triggered by the SQS Queue. This also gives you retry capabilities.
Related
I want to watch changes to Pods continuously using the client-go Kubernetes SDK. I am using the below code to watch the changes:
func (c *Client) watchPods(namespace string, restartLimit int) {
fmt.Println("Watch Kubernetes Pods")
watcher, err := c.Clientset.CoreV1().Pods(namespace).Watch(context.Background(),
metav1.ListOptions{
FieldSelector: "",
})
if err != nil {
fmt.Printf("error create pod watcher: %v\n", err)
return
}
for event := range watcher.ResultChan() {
pod, ok := event.Object.(*corev1.Pod)
if !ok || !checkValidPod(pod) {
continue
}
owner := getOwnerReference(pod)
for _, c := range pod.Status.ContainerStatuses {
if reflect.ValueOf(c.RestartCount).Int() >= int64(restartLimit) {
if c.State.Waiting != nil && c.State.Waiting.Reason == "CrashLoopBackOff" {
doSomething()
}
if c.State.Terminated != nil {
doSomethingElse()
}
}
}
}
}
The code is watching changes to the Pods, but it exits after some time. I want to run this continuously. I also want to know how much load it puts on the API Server and what is the best way to run a control loop for looking for changes.
In Watch, a long poll connection is established with the API server. Upon establishing a connection, the API server sends an initial batch of events and any subsequent changes. The connection will be dropped after a timeout occurs.
I would suggest using an Informer instead of setting up a watch, as it is much more optimized and easier to setup. While creating an informer, you can register specific functions which will be invoked when pods get created, updated and deleted. Even in informers you can target specific pods using a labelSelector, similar to watch. You can also create shared informers, which are shared across multiple controllers in the cluster. This results in reducing the load on the API server.
Below are few links to get you started:
https://aly.arriqaaq.com/kubernetes-informers/
https://www.cncf.io/blog/2019/10/15/extend-kubernetes-via-a-shared-informer/
https://pkg.go.dev/k8s.io/client-go/informers
I'm experiencing some unexpected errors while trying to subscribe on Podio Push service. I use golang concurrency pattern defined here and here is the bayeux client library used for subscription.
Basically the flow tries to retrieve the item first and then subscribe into push object provided with the item object. There is channel object where i store each task (taskLoad: ~each item_id with credentials it needs for retrieval)
item := new(podio.Item)
item, err = podio.GetItem(itemId)
if err != nil {
log.Errorf("PODIO", "Could not get item %d -> %s", itemId, err)
return
}
and, later inside another func
messages := make(chan *bayeux.Message)
server := GetBayeux()
defer server.Close()
if err = push.Subscribe(server, messages); err != nil {
// log err, with item details
log.Errorf("PODIO", "%s", err, push)
// re-enqueue the task in taskLoad channel
go enqueueTask(true, messages, sigrepeat, timer)
// release sigwait channel on subscription error
<-sigwait
return
}
here GetBayeux func is just a singleton which wraps the client
func GetBayeux() *bayeux.Client {
bayeuxOnce.Do(func() {
Bayeux = bayeux.NewClient("https://push.podio.com/faye", nil)
})
return Bayeux
}
there is about ~15000 items to listen and I should subscribe to each of them but unfortunately sometimes I got one of these errors while processing subscriptions
401:k9jx3v4qq276k9q84gokpqirhjrfbyd:Unknown client [{"channel":"/item/9164xxxxx","signature":"46bd8ab3ef2a31297d8f4f5ddxxxx","timestamp":"2018-01-02T14:34:02Z","expires_in":21600}]
OR
HTTP Status 400 [{"channel":"/item/9158xxxxx","signature":"31bf8b4697ca2fda69bd7fd532d08xxxxxx","timestamp":"2018-01-02T14:37:02Z","expires_in":21600}]
OR
[WRN] Bayeux connect failed: HTTP Status 400
OR
Bayeux connect failed: Post https://push.podio.com/faye: http2: server sent GOAWAY and closed the connection; LastStreamID=1999, ErrCode=NO_ERROR, debug=""
So now, i'd like to know why i got these errors and most of all how i can fix them to ensure to listen to all items in the scope.
If anyone knows, is there any limitation about concurrent access into podio push service?
Thanks
UPDATE 2019-01-07
it was the singleton that messed the process. as it was in a goroutine context there was some subscriptions that was not allowed because the server has been closed by another goroutine. The fix was to exposing Unsubscribe method and use it instead of Close method which disconnect the client from the server.
defer server.Close()
became
defer push.Unsubscribe(server)
In one of my services that happens to be my loadbalancer, I am getting the following error when calling the server method in one of my deployed services:
rpc error: code = Unimplemented desc = unknown service
fooService.FooService
I have a few other services set up with gRPC and they work fine. It just seems to be this one and I am wondering if that is because it is the loadbalancer?
func GetResult(w http.ResponseWriter, r *http.Request) {
conn, errOne := grpc.Dial("redis-gateway:10006", grpc.WithInsecure())
defer conn.Close()
rClient := rs.NewRedisGatewayClient(conn)
result , errTwo := rClient.GetData(context.Background(), &rs.KeyRequest{Key: "trump", Value: "trumpVal"}, grpc.WithInsecure())
fmt.Fprintf(w, "print result: %s \n", result) //prints nil
fmt.Fprintf(w, "print error one: %v \n", errOne) // prints nil
fmt.Fprintf(w, "print error two: %s \n", errTwo) // prints error
}
The error says there is no service called fooService.FooService which is true because the dns name for the service I am calling is called foo-service. However it is the exact same setup for my other services that use gRPC and work fine. Also my proto files are correctly configured so that is not an issue.
server I am calling:
func main() {
lis, err := net.Listen("tcp", ":10006")
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
grpcServer := grpc.NewServer()
newServer := &RedisGatewayServer{}
rs.RegisterRedisGatewayServer(grpcServer, newServer)
if err := grpcServer.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
}
The function I am trying to access from client:
func (s *RedisGatewayServer) GetData(ctx context.Context, in *rs.KeyRequest)(*rs.KeyRequest, error) {
return in, nil
}
My docker and yaml files are all correct also with the right naming and ports.
I had this exact problem, and it was because of a very simple mistake: I had put the call to the service registration after the server start. The code looked like this:
err = s.Serve(listener)
if err != nil {
log.Fatalf("[x] serve: %v", err)
}
primepb.RegisterPrimeServiceServer(s, &server{})
Obviously, the registration should have been called before the server was ran:
primepb.RegisterPrimeServiceServer(s, &server{})
err = s.Serve(listener)
if err != nil {
log.Fatalf("[x] serve: %v", err)
}
thanks #dolan's for the comment, it solved the problem.
Basically we have to make sure, the method value should be same in both server and client (you can even copy the method name from the pb.go file generated from server side)
func (cc *ClientConn) Invoke(ctx context.Context, method string, args, reply interface{}, opts ...CallOption) error {
this invoke function will be there inside all the methods which you have implemented in gRPC service.
for me, my client was connecting to the wrong server port.
My server listens to localhost:50052
My client connects to localhost:50051
and hence I see this error.
I had the same problem - I was able to perform rpc call from Postman to the service whereas apigateway was able to connect to the service but on method call It gave error code 12 unknown service and the reason was in my proto files client was under package X whereas server was under package Y.
Quite silly but yeah making the package for both proto under apigateway and service solved my problem.
There are many scenarios in which this can happen. The underlying issue seems to be the same - the GRPC server is reachable but cannot service client requests.
The two scenarios I faced that are not documented in previous answers are:
1. Client and server are not running the same contract version
A breaking change introduced in the contract can cause this error.
In my case, the server was running the older version of the contract while the client was running the latest one.
A breaking change meant that the server could not resolve the service my client was asking for thus returning the unimplemented error.
2. The client is connecting to the wrong GRPC server
The client reached the incorrect server that doesn't implement the contract.
Consider this scenario if you're running multiple different GRPC services. You might be mistakingly dialing the wrong one.
I am using go package for pub/sub. On my API dashboard I see this error(google.pubsub.v1.Subscriber.StreamingPull - error code 503). Per docs(https://cloud.google.com/pubsub/docs/reference/error-codes) it seems it is transient condition but better to implement backoff strategy(https://cloud.google.com/storage/docs/exponential-backoff). the question is I am not able to wrap my head where this error code is coming on Receive method.
Here is func:
err = sub.Receive(ctx, func(ctx context.Context, m *pubsub.Message) {
// Dump message
// log.Printf("Got message: %s", m.Data)
// Decoding coming message
err = json.NewDecoder(bytes.NewReader(m.Data)).Decode(&msg)
if err != nil {
log.Printf("Error decoding - %v", err)
}
// See streaming messages
log.Printf(" %s : %s : Product updated for Product Id(%d) : Product Title(%s)",
msg.AuthID,
msg.TraceID,
msg.Product.ID,
msg.Product.Title,
)
//
// Some business logic
//
// Acknowledge on recieve method
m.Ack()
})
if err != context.Canceled {
// if err != nil {
return errors.Wrap(err, "Error occurred on recieve data from topic: blah")
}
The Cloud Pub/Sub Go client library will retry this transient error on its own, you shouldn't need to handle it. Internally, the client library uses StreamingPull, where it sends a request and receives messages from the service as they are available. Occasionally, there can be a disconnection event requiring the connection to be reestablished. This is why you see the 503 error in the API dashboard. It should not be the case that your code sees an error in this scenario since the underlying library is handling it (including the use of exponential backoff, where relevant).
I am experimenting with Go on AWS lambda, and i found that each function requires a binary to be uploaded for execution.
My question is that, is it possible to have a single binary that can have two different Handler functions, which can be loaded by two different lambda functions.
for example
func Handler(request events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
fmt.Println("Received body in Handler 1: ", request.Body)
return events.APIGatewayProxyResponse{Body: request.Body, StatusCode: 200}, nil
}
func Handler1(request events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
fmt.Println("Received body in Handler 2: ", request.Body)
return events.APIGatewayProxyResponse{Body: request.Body, StatusCode: 200}, nil
}
func EndPoint1() {
lambda.Start(Handler)
}
func EndPoint2() {
lambda.Start(Handler1)
}
and calling the EndPoints in main in such a way that it registers both the EndPoints and the same binary would be uploaded to both the functions MyFunction1 and MyFunction2.
I understand that having two different binary is good because it reduces the load/size of each function.
But this is just an experimentation.
Thanks in advance :)
I believe it is not possible since Lambda console says the handler is the name of the executable file:
Handler: The executable file name value. For example, "myHandler" would call the main function in the package “main” of the myHandler executable program.
So one single executable file is unable to host two different handlers.
you can do the workaround to have same executable to upload to two different lambda like following
func main() {
switch cfg.LambdaCommand {
case "select_lambda_1":
lambda1handler()
case "select_lambda_2":
lambda2handler()
I've had success by adding an environment variable to the functions in the template.yaml, and making the main() check the variable and call the appropriate handler.