Golang vesion: 1.18.3
Logging libs used:
"log"
"github.com/kdar/logrus-cloudwatchlogs"
"github.com/sirupsen/logrus"
I'm writing a AWS Lambda using Go. It sits behind an AWS APIGateway as a REST API.
I'm trying to log the requests got into it and the responses it sends back.
Let's say the example request payload is:
const jsonData = `{"name": "Yomiko","address": {"city": "Tokyo","street": "Shibaura St"},"children":[{"lastName": "Takayashi"}],"isEmployed": false}`
When I log it using standard "log" library as below,
log.Println("JSON Data: ", jsonData)
It's printed nicely in the targeted CloudWatch Stream maintaining the JSON format as below,
Image 1: CloudWatch log with standard "log" library
However, when I used logrus with logrus-cloudwatchlogs as below,
hook, err := logrus_cloudwatchlogs.NewHook(transactionLogGroup, transactionLogStream, sess)
l := logrus.New()
l.Hooks.Add(hook)
l.Out = ioutil.Discard
l.Formatter = &logrus.JSONFormatter{}
l.Println("JSON Data: ", jsonData)
It adds backslashes as shown below and loses the nice JSON Format,
Image 2: CloudWatch log with "logrus" library
My question for you good people is,
How can I persevere the nice JSON format as shown in Image 1 when I use the "logrus" library? Please note that using standard "log" library is not an option here.
Related
Handling errors in a gRPC service commonly requires both a status message and error codes. Both have two definitions:
Google APIs definition (googleapis/go-genproto) - the generated Go packages for common protocol buffer types, and the generated gRPC code for Google's gRPC APIs
gRPC definition (grpc/grpc-go) - the Go implementation of gRPC
The Go packages for both definitions of Status and Codes are:
Google APIs
Status: google.golang.org/genproto/googleapis/rpc/status
Codes: google.golang.org/genproto/googleapis/rpc/code
gRPC
Status: google.golang.org/grpc/status
Codes: google.golang.org/grpc/codes
Since I'm a client of my own gRPC service and not a client of an existing Google gRPC API, I want to use the gRPC definitions of Status and Code.
However, the gRPC proto file for Status is actually copied from Google APIs definition. See https://github.com/grpc/grpc/tree/master/src/proto/grpc/status. The go_package of status.proto is also unchanged, so both the Google API and gRPC definitions use the following Go package
option go_package = "google.golang.org/genproto/googleapis/rpc/status;status";
The upshot is the only way to use Status when defining an API is by importing it with
import "google/rpc/status.proto";
...and importing the language bindings in Go with
import (
"google.golang.org/genproto/googleapis/rpc/status"
)
// Go server code...
But as stated earlier, this is wrong since I'm not a client of a Google API, but rather my own gRPC service. Therefore the language bindings should be imported with
import (
"google.golang.org/grpc/status"
)
// Go server code...
As expected if I switch to importing the gRPC language bindings and try and return a Status message to the API client, I get a compile error
cannot use &(status.Status literal)
(value of type *"google.golang.org/grpc/internal/status".Status) as
*"google.golang.org/genproto/googleapis/rpc/status".Status value
This is caused by my .proto file using the Google API definition of Status while the server implementation (in Go) uses the gRPC definition.
The problem impacts error codes since Google APIs uses signed 32 bit integers (int32) whereas gRPC uses unsigned 32 bit integers (uint32).
Questions
Is my assertion that I should be using the gRPC definition of Status and Codes correct?
If my assertion is correct, how can I use the gRPC definition of Status when it's packaged for Google APIs?
We need to distinguish a few cases. Some of them are obvious, some are not.
Just returning Status from a gRPC handler
If your proto schema (.proto files) doesn't define messages that use Status or Code directly, then the gRPC handlers can satisfy the return type error simply with "google.golang.org/grpc/status".Error(), or Newf().Err(). And that's about it.
Example:
// implements SomeServiceServer unary RPC GetFoo
func (s *SomeService) GetFoo(ctx context.Context, req *grpc.FooRequest) (*grpc.FooResponse, error) {
// status is "google.golang.org/grpc/status"
return nil, status.Error(codes.Unimplemented, "coming soon")
Using Status in your .proto files
In this case, you are forced to use the googleapis implementation. As you already have seen, the status.proto Go package is defined as:
option go_package = "google.golang.org/genproto/googleapis/rpc/status;status";
So let's say you have the following .proto file, where the imported status.proto is just a copy-paste of the gRPC status.proto as per this question:
syntax = "proto3";
package test;
import "status.proto";
option go_package = ".;main";
message Foo {
string a = 1;
google.rpc.Status status = 2;
}
with directory structure as:
/protos
|_ status.proto
|_ test.proto
and you compile the above with:
cd protos && protoc -I=. --go_out=. test.proto
breathe ...then the generated Go code will have the following import
import (
status "google.golang.org/genproto/googleapis/rpc/status"
)
and you must satisfy that by go get google.golang.org/genproto.
So about your first question, you can only use Status from googleapis in proto files, because that's how status.proto declares its Go package.
Using generated googleapis Status in Go code
Since the imported Go package is from googleapis that is what you must use in your Go code in order to initialize such messages:
package main
import (
"fmt"
googleapis "google.golang.org/genproto/googleapis/rpc/status"
)
func main() {
foo := &Foo{
A: "foo",
Status: &googleapis.Status{
Code: int32(code.Code_OK),
Message: "all good",
},
}
// fmt.Println(foo)
}
Yes but I must use grpc-go Status in my Go code
You can't. protoc generates code with the packages described above. If you absolutely NEED to construct these Status fields using grpc-go, you can use Status.Proto:
package main
import (
"fmt"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
)
func main() {
foo := &Foo{
A: "foo",
Status: status.New(codes.OK, "all good").Proto(),
}
fmt.Println(foo)
}
Just for the record, the opposite is also possible with status.FromProto:
package main
import (
"fmt"
googleapis_codes "google.golang.org/genproto/googleapis/rpc/code"
googleapis "google.golang.org/genproto/googleapis/rpc/status"
"google.golang.org/grpc/status"
)
func main() {
gapisStatus := &googleapis.Status{
Code: int32(googleapis_codes.Code_OK),
Message: "all good",
}
grpcStatus := status.FromProto(gapisStatus)
fmt.Println(grpcStatus)
}
As a less well-behaved alternative, you can simply copy-paste the status.proto sources into your project and manually change the go_package:
option go_package = "google.golang.org/grpc/status;status";
This way protoc will generate the Go code with this import, and your own sources will be able to follow suit. Of course this means you now have your own fork of status.proto.
I have a model in Vertex AI, from the logs it seems that Vertex AI has ingested the log into message field within jsonPayload field, but i would like to structure the jsonPayload field such that every key in message will be a field within jsonPayload, i.e: flatten/extract message
The logs in Stackdriver follow a defined LogEntry schema. Cloud Logging uses structured logs where log entries use the jsonPayload field to add structures to their payload.
For Vertex AI, the parameters are passed inside the message field which we see in the logs. These structures of the logs are predefined. However if you want to extract the fields that are present inside the message block you can refer to the below mentioned workarounds:
1. Create a sink :
You can export your logs to a Cloud Storage bucket, Bigquery,Pub/Sub etc.
If you use Bigquery as the sink, then in Bigquery you can use the JSON functions to extract the required data.
2. Download the logs and write your custom code :
You can download the log files and then write your custom logic to extract data as per your requirements.
You can refer to the client library (python) to write the custom logic and python JSON functions.
Using the gcloud logging client to write structure logs into a Vertex AI endpoint:
(Make sure you have a service account with premissions to write logs into gcloud, and, for clean logs make sure you don't stream any other logs into stderr or stdout)
import json
import logging
from logging import Handler, LogRecord
import google.cloud.logging_v2 as logging_v2
from google.api_core.client_options import ClientOptions
from google.oauth2 import service_account
data_to_write_to_endpoint = {key1: value1, ...}
#Json key for a Service account permitted to write logs into the gcp
# project where your endpoint is
credentials = service_account.Credentials.from_service_account_info(
json.loads(SERVICE_ACOUNT_KEY_JSON)
)
client = logging_v2.client.Client(
credentials=credentials, client_options=ClientOptions(api_endpoint="logging.googleapis.com",),
)
# This represent your Vertex AI endpoint
resource = logging_v2.Resource(
type="aiplatform.googleapis.com/Endpoint",
labels={"endpoint_id": YOUR_ENDPOINT_ID, "location": ENDPOINT_REGION},
)
logger = client.logger("LOGGER NAME")
logger.log_struct(
info=data_to_write_to_endpoint,
severity=severity,
resource=resource,
)
I'm using https://github.com/kubernetes/client-go and all works well.
I have a manifest (YAML) for the official Kubernetes Dashboard: https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta4/aio/deploy/recommended.yaml
I want to mimic kubectl apply of this manifest in Go code, using client-go.
I understand that I need to do some (un)marshalling of the YAML bytes into the correct API types defined in package: https://github.com/kubernetes/api
I have successfully Createed single API types to my cluster, but how do I do this for a manifest that contains a list of types that are not the same? Is there a resource kind: List* that supports these different types?
My current workaround is to split the YAML file using csplit with --- as the delimiter
csplit /path/to/recommended.yaml /---/ '{*}' --prefix='dashboard.' --suffix-format='%03d.yaml'
Next, I loop over the new (14) parts that were created, read their bytes, switch on the type of the object returned by the UniversalDeserializer's decoder and call the correct API methods using my k8s clientset.
I would like to do this to programmatically to make updates to any new versions of the dashboard into my cluster. I will also need to do this for the Metrics Server and many other resources. The alternative (maybe simpler) method is to ship my code with kubectl installed to the container image and directly call kubectl apply -f -; but that means I also need to write the kube config to disk or maybe pass it inline so that kubectl can use it.
I found this issue to be helpful: https://github.com/kubernetes/client-go/issues/193
The decoder lives here: https://github.com/kubernetes/apimachinery/tree/master/pkg/runtime/serializer
It's exposed in client-go here: https://github.com/kubernetes/client-go/blob/master/kubernetes/scheme/register.go#L69
I've also taken a look at the RunConvert method that is used by kubectl: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/convert/convert.go#L139 and assume that I can provide my own genericclioptions.IOStreams to get the output?
It looks like RunConvert is on a deprecation path
I've also looked at other questions tagged [client-go] but most use old examples or use a YAML file with a single kind defined, and the API has changed since.
Edit: Because I need to do this for more than one cluster and am creating clusters programmatically (AWS EKS API + CloudFormation/eksctl), I would like to minimize the overhead of creating ServiceAccounts across many cluster contexts, across many AWS accounts. Ideally, the only authentication step involved in creating my clientset is using aws-iam-authenticator to get a token using cluster data (name, region, CA cert, etc). There hasn't been a release of aws-iam-authenticator for a while, but the contents of master allow for the use of a third-party role cross-account role and external ID to be passed. IMO, this is cleaner than using a ServiceAccount (and IRSA) because there are other AWS services the application (the backend API which creates and applies add-ons to these clusters) needs to interact with.
Edit: I have recently found https://github.com/ericchiang/k8s. It's definitely simpler to use than client-go, at a high-level, but doesn't support this behavior.
It sounds like you've figured out how to deserialize YAML files into Kubernetes runtime.Objects, but the problem is dynamically deploying a runtime.Object without writing special code for each Kind.
kubectl achieves this by interacting with the REST API directly. Specifically, via resource.Helper.
In my code, I have something like:
import (
meta "k8s.io/apimachinery/pkg/api/meta"
"k8s.io/cli-runtime/pkg/resource"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/rest"
"k8s.io/client-go/restmapper"
"k8s.io/apimachinery/pkg/runtime"
)
func createObject(kubeClientset kubernetes.Interface, restConfig rest.Config, obj runtime.Object) error {
// Create a REST mapper that tracks information about the available resources in the cluster.
groupResources, err := restmapper.GetAPIGroupResources(kubeClientset.Discovery())
if err != nil {
return err
}
rm := restmapper.NewDiscoveryRESTMapper(groupResources)
// Get some metadata needed to make the REST request.
gvk := obj.GetObjectKind().GroupVersionKind()
gk := schema.GroupKind{Group: gvk.Group, Kind: gvk.Kind}
mapping, err := rm.RESTMapping(gk, gvk.Version)
if err != nil {
return err
}
name, err := meta.NewAccessor().Name(obj)
if err != nil {
return err
}
// Create a client specifically for creating the object.
restClient, err := newRestClient(restConfig, mapping.GroupVersionKind.GroupVersion())
if err != nil {
return err
}
// Use the REST helper to create the object in the "default" namespace.
restHelper := resource.NewHelper(restClient, mapping)
return restHelper.Create("default", false, obj, &metav1.CreateOptions{})
}
func newRestClient(restConfig rest.Config, gv schema.GroupVersion) (rest.Interface, error) {
restConfig.ContentConfig = resource.UnstructuredPlusDefaultContentConfig()
restConfig.GroupVersion = &gv
if len(gv.Group) == 0 {
restConfig.APIPath = "/api"
} else {
restConfig.APIPath = "/apis"
}
return rest.RESTClientFor(&restConfig)
}
I was able to get this working in one of my projects. I had to use much of the source code from kubectl's apply command to get it working correctly.
https://github.com/billiford/go-clouddriver/blob/master/pkg/kubernetes/client.go#L63
I wanted to add trace id to logging done for each request to the micro service.I want this in similar as for springboot application we can set trace id in MDC and fetch it and use it while logging.
I have done some research and I found that MDC equivalent in go lang is context. So, I have set the trace id in my context. Now the problem is where ever I have to log with trace id ,I need to pass context to that function which is very ugly way. I am looking for a better solution for this problem.
func HandlerFunction(f gin.HandlerFunc) gin.HandlerFunc{
return func(cxt *gin.Context) {
reqraceId := cxt.Request.Header.Get("trace-id")
requid , _ := uuid.NewRandom()
if reqTraceId == "" {
c.Request.Header.Set("trace-id", requid.String())
}
f(c)
}
}
It might be worth reading up on context.Context particularly this article which has a section that says:
At Google, we require that Go programmers pass a Context parameter as the first argument to every function on the call path between incoming and outgoing requests.
TL;DR - it's fine to pass the context, but what's the best way?
There's two main patterns
Ask the context to give you a logger
Give the logger the context
Context can be used to store values:
context.WithValue(ctx, someKey, someValue)
This means we can either do:
somepackage.Log(ctx).Info("hello world")
// or
sompackage.Info(ctx, "hello world")
The implementation of these two sample APIs could interact with the context to retrieve the values required with out needing to worry about the extra information that would have been in MDC at any of the logging call sites.
From my side I found that using the default log package we could set a prefix as log.SetPrefix(traceId), doing so, the log will print the trace id as the prefix in the actual and sub-functions/structs.
import (
"log"
"github.com/google/uuid"
)
func (hdl *HTTPHandler) example() {
var traceId string = uuid.NewString()
log.SetPrefix(traceId + " - ")
log.SetFlags(log.LstdFlags)
// ...
// ...
log.Println("......")
}
This issue can also be solved using a dependency injection container.
We can implement "request-scoped" injections, and as a result, for each request, we will recreate all dependency tree that uses request-scoped dependency(logger, error reporter, clients which send requests to another service with context propagation).
But as I understood using dependency injection containers is not a best practice in go and not an "idiomatic" way.
Also, this approach can have some performance and memory issues since we will recreate objects for each request.
How do I change the IP address of the DNS server?
In situation, I set Google DNS server in Windows Network Settins.
And I use LookupTXT function in Golang for getting DNS txt request.
But LookupTXT parameter is just the query string.
Any help or pointers would be highly appreciated. Thanks!
This is not straigtforward to do using golang at this point. You can however use a third party DNS package that allows configuring the resolver. First install the package:
go get github.com/bogdanovich/dns_resolver
Here is an example using it and the google resolvers 8.8.8.8 and 8.8.4.4:
package main
import (
"log"
"github.com/bogdanovich/dns_resolver"
)
func main() {
resolver := dns_resolver.New([]string{"8.8.8.8", "8.8.4.4"})
// In case of i/o timeout
resolver.RetryTimes = 5
ip, err := resolver.LookupHost("google.com")
if err != nil {
log.Fatal(err.Error())
}
log.Println(ip)
// Output [216.58.192.46]
}
Source
There is an open issue in golang here, so hopefully it becomes easier to do it with the builtin net package: https://github.com/golang/go/issues/12503. It could just be a documentation problem, as it is possible now, I just can't find an example.
EDIT: actually that package only supports lookupHost: https://github.com/bogdanovich/dns_resolver/blob/master/dns_resolver.go#L51-L79
So a PR would be required to add a TXT resolver.
2nd Edit: I made a PR with txt lookup here. That project hasn't been touched in years though so it may never get accepted.