Using Go and AWS-SDK
I'm attempting to query route53 CNAME and A records as listed in the AWS Console under Route53 -> Hosted Zones. I'm able to query using the following code, but it requires the (cryptic) HostedZoneId I have to know ahead of time.
Is there a different function, or a HostedZoneId lookup based on the Domain Name such as XXX.XXX.com ?
AWSLogin(instance)
svc := route53.New(instance.AWSSession)
listParams := &route53.ListResourceRecordSetsInput{
HostedZoneId: aws.String("Z2798GPJN9CUFJ"), // Required
// StartRecordType: aws.String("CNAME"),
}
respList, err := svc.ListResourceRecordSets(listParams)
if err != nil {
fmt.Println(err.Error())
return
}
// Pretty-print the response data.
fmt.Println("All records:")
fmt.Println(respList)
edit: oh, additionally, the StartRecordType with the value "CNAME" throws a validation error, so I'm not sure what I should be using there.
You first have to do a lookup to get the HostedZoneID. Here is the func I wrote for it. :
func GetHostedZoneIdByNameLookup(awsSession string, HostedZoneName string) (HostedZoneID string, err error) {
svc := route53.New(awsSession)
listParams := &route53.ListHostedZonesByNameInput{
DNSName: aws.String(HostedZoneName), // Required
}
req, resp := svc.ListHostedZonesByNameRequest(listParams)
err = req.Send()
if err != nil {
return "", err
}
HostedZoneID = *resp.HostedZones[0].Id
// remove the /hostedzone/ path if it's there
if strings.HasPrefix(HostedZoneID, "/hostedzone/") {
HostedZoneID = strings.TrimPrefix(HostedZoneID, "/hostedzone/")
}
return HostedZoneID, nil
}
Related
I'm currently working on an api gateway that forwards requests to an grpc client. The routes use a middleware. Within that middleware the Validate function from the dialed grpc client is used. The function is in the last code snippet at the end.
The gateway is based on a gin router, which, as far as I understand handles each request in a separat go routine. I can send as many requests simultaneously to either to endpoint /protected/1 or to endpoint /protected/2 and they get handled correctly.
The Validate function is having problems handling request sent simultaneously to /protected/1 and /protected/2. Sending two request to both endpoints simultaneously results in that either both, none or only one of the requests is handled correctly. The error message that I receive is actually from the gorm function trying to query the user in the Validate function (ERROR: relation "users" does not exist (SQLSTATE 42P01)). I use a gorm database object that is connected to a postgres database. This behaviour suggests that some how a resource is shared. Using run -race does not give any insights.
So my questions are, why does this setup not work to send requests simultaneously to both endpoints?
I tried to make a working minimal example, but somehow I could nor reproduce the error in a minimal setting, hence I would like to share my code snippets here.
Important snippets of the the Api Gateway implementation
func main() {
c, err := config.LoadConfig()
...
r := gin.Default()
auth.RegisterRoutes(r, &c)
r.Run(":" + c.Port)
}
whereas the routes are defined as
func RegisterRoutes(r *gin.Engine, c *config.Config){
svc := &ServiceClient{
Client: InitServiceClient(c),
}
a := InitAuthMiddleware(svc)
// routes
protected := r.Group("/protected")
protected.Use(a.AuthRequired)
protected.GET("/1", svc.DoStuff)
protected.GET("/2", svc.DoStuff)
}
// dummy handler
func (svc *ServiceClient) DoStuff(ctx *gin.Context) {
handler.DoStuff(ctx, svc.Client)
}
The grpc client is dialed with the following function
type ServiceClient struct {
Client pb.AuthServiceClient
}
func InitServiceClient(c *config.Config) pb.AuthServiceClient {
// using WithInsecure() because no SSL running
cc, err := grpc.Dial(c.AuthSvcUrl, grpc.WithInsecure())
if err != nil {
fmt.Println("Could not connect:", err)
}
return pb.NewAuthServiceClient(cc)
}
and the middleware is implemented in the following way:
type AuthMiddlewareConfig struct {
svc *ServiceClient
}
func InitAuthMiddleware(svc *ServiceClient) AuthMiddlewareConfig {
return AuthMiddlewareConfig{svc}
}
func (c *AuthMiddlewareConfig) AuthRequired(ctx *gin.Context) {
access_token, _ := ctx.Cookie("access_token")
res, err := c.svc.Client.Validate(context.Background(), &pb.ValidateRequest{
Token: access_token,
TokenType: "ACCESS_TOKEN",
Url: ctx.Request.URL.String(),
})
if err != nil || res.Status != http.StatusOK {
ctx.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{"status": http.StatusUnauthorized, "error": res.Error})
return
}
ctx.Set("userId", res.UserId)
ctx.Next()
}
Important snippets of the Auth Service:
func serve(c *conf.Configuration) {
/*
Dial returns:
type Repository struct {
DB *gorm.DB
}
*/
h := storage.Dial(&c.DB)
serviceUri := fmt.Sprintf(":%s", c.API.Port)
lis, err := net.Listen("tcp", serviceUri)
if err != nil {
logrus.Fatal("Failed to listen on: ", err)
}
s := api.Server{
R: h,
//
}
grpcServer := grpc.NewServer()
pb.RegisterAuthServiceServer(grpcServer, &s)
if err := grpcServer.Serve(lis); err != nil {
logrus.Fatalln("Failed to serve:", err)
}
}
and the validate function is implemented by
type Server struct {
R storage.Repository
//
// #https://github.com/grpc/grpc-go/issues/3794:
pb.UnimplementedAuthServiceServer
}
func (s *Server) Validate(ctx context.Context, req *pb.ValidateRequest) (*pb.ValidateResponse, error) {
// token validation stuff works
var user models.User
// causes error sometimes
if result := s.R.DB.Where(&models.User{Id: claims.Id}).First(&user); result.Error != nil {
return &pb.ValidateResponse{
Status: http.StatusNotFound,
Error: "User not found",
}, nil
}
// send response
}
EDIT
For completeness here the function that connects to the postgres client.
func openDb(c *conf.PostgresConfiguration, gormConfig *gorm.Config, database string) *gorm.DB {
connectionString := fmt.Sprintf("postgres://%s:%s#%s:%s/%s", c.User, c.Password, c.Host, c.Port, database)
db, err := gorm.Open(postgres.Open(connectionString), gormConfig)
if err != nil {
logrus.Fatal(fmt.Sprintf("Unable to connect to database '%s' given url: ", database), err)
}
logrus.Info(fmt.Sprintf("Connected to database %s", database))
return db
}
func Dial(c *conf.PostgresConfiguration) Repository {
gormDefaultConfig := &gorm.Config{}
gormAppConfig := &gorm.Config{}
// connect to default database
logrus.Info(fmt.Sprintf("Use default database %s for initialization", c.DefaultDatabase))
defaultDb := openDb(c, gormDefaultConfig, c.DefaultDatabase)
// check if database exists and create it if necessary
var dbexists bool
dbSQL := fmt.Sprintf("SELECT EXISTS (SELECT FROM pg_database WHERE datname = '%s') AS dbexists;", c.AppDatabase)
defaultDb.Raw(dbSQL).Row().Scan(&dbexists)
if !dbexists {
logrus.Info(fmt.Sprintf("Created database %s", c.AppDatabase))
db := defaultDb.Exec(fmt.Sprintf("CREATE DATABASE %s;", c.AppDatabase))
if db.Error != nil {
logrus.Fatal("Unable to create app database: ", db.Error)
}
}
// connect to app databse
appDb := openDb(c, gormAppConfig, c.AppDatabase)
// check if schema exists and create it if necessary
var schemaexists bool
schemaSQL := fmt.Sprintf("SELECT EXISTS(SELECT FROM pg_namespace WHERE nspname = '%s') AS schemaexisits;", c.AppDatabaseSchema)
appDb.Raw(schemaSQL).Row().Scan(&schemaexists)
// create app specfic database, if not already existing
if !schemaexists {
// create service specific schema
db := appDb.Exec(fmt.Sprintf("CREATE SCHEMA %s;", c.AppDatabaseSchema))
if db.Error != nil {
logrus.Fatal(fmt.Sprintf("Unable to create database schema %s", c.AppDatabaseSchema), db.Error)
}
logrus.Info(fmt.Sprintf("Created database schema %s", c.AppDatabaseSchema))
}
db := appDb.Exec(fmt.Sprintf(`set search_path='%s';`, c.AppDatabaseSchema))
logrus.Info(fmt.Sprintf("Use existing database schema %s", c.AppDatabaseSchema))
if db.Error != nil {
logrus.Fatal(fmt.Sprintf("Unable to set search_path for database to schema %s", c.AppDatabaseSchema), db.Error)
}
// migrate table
appDb.AutoMigrate(&models.User{})
return Repository{appDb}
}
Edit2
Changing the connection string in the openDb function to connectionString := fmt.Sprintf("postgres://%s:%s#%s:%s/%s?search_path=%s", c.User, c.Password, c.Host, c.Port, database, c.AppDatabaseSchema) fixes the issue (see how the search_path is part of the url).
Thank you!
I'm using Google's logadmin package which comes in the logging package and trying to retrieve some logs by using a filter. Pretty simple... or so I thought. But the iterator never returns iterator.Done.
By looking at the opened issues in the package on GitHub and searching I couldn't find anything about it.
My code:
iter := client.Entries(ctx, logadmin.NewestFirst(),
logadmin.Filter(
fmt.Sprintf(`resource.type="cloud_function" resource.labels.function_name="%s"`, logger)))
var entries []string
for {
select {
case <-ctx.Done():
return nil
default:
entry, err := iter.Next()
log.Print(fmt.Sprintf("[%s] [LEVEL]: %s\n [PAYLOAD]: %s\n [TIMESTAMP]: %s\n\n", logger,
entry.Severity, entry.Payload, entry.Timestamp.String()))
if err == iterator.Done {
return nil
}
if err != nil {
return err
}
}
}
I manage to solve the problem by adding a timestamp to my filter as follows:
now := time.Now().Add(-24 * time.Hour).Format(time.RFC3339)
filter := fmt.Sprintf(`resource.type="cloud_function" resource.labels.function_name="%s" timestamp > %s`, logger, now)
I have a dynamodb field that looks like this:
[ { "S" : "test#gmail.com" }, { "S" : "test2#gmail.com" } ]
I am trying to run a scan to return any record that e.g. contain test#gmail.com . I am not sure I should use contains to do this, it's currently not returning any records, any pointers as to what I should use?
My go is setup like this:
type Site struct {
ID string `json:"id"`
Site string `json:"site"`
Emails []string `json:"emails,omitempty"`
}
func (ds *datastore) GetEmail(email string, out interface{}) error {
filt := expression.Name("emails").Contains(strings.ToLower(email))
fmt.Println("Get Email", filt)
//filt := expression.Contains(expression.Name("emails"), expression.Value(email))
proj := expression.NamesList(
expression.Name("emails"),
expression.Name("site"),
)
expr, err := expression.NewBuilder().
WithFilter(filt).
WithProjection(proj).
Build()
if err != nil {
fmt.Println(err)
}
scanInput := &dynamodb.ScanInput{
ExpressionAttributeNames: expr.Names(),
ExpressionAttributeValues: expr.Values(),
FilterExpression: expr.Filter(),
ProjectionExpression: expr.Projection(),
TableName: aws.String(ds.TableName),
}
result, err := ds.DDB.Scan(scanInput)
if err != nil {
fmt.Println("what is the err", err)
return err
}
if len(result.Items) == 0 {
fmt.Println("No Email found")
return errors.New(http.StatusText(http.StatusNotFound))
}
err = ds.Marshaler.UnmarshalMap(result.Items[0], out)
return err
}
If you're doing a contains on a partial email it won't match when the filter is applied to a set. It will have to be an exact email match.
{
"Email": "test#gmail.com"
}
// This will match a contains on "test#g"
{
"Emails": ["test#gmail.com", "another#gmail.com"]
}
// this will not match a contains on "test#g" but will match a contains of "test#gmail.com"
See contains: https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Condition.html
Also note you're doing a scan. Scans perform poorly in Dynamodb as soon as your data is of any significant size. Think of storing your data in a different format so you query it via partition keys or use an AWS RDS as an alternative.
I am facing an issue where i have made an api in Go every thing work fine but i am not getting data in postman. When i print the data in logs i am getting the data properly but it is showing blank data in postman.
authorizeModel.go
func GetSkillList() map[string]interface{} {
db := GetDB()
var (
// id int
skillName string
)
type SkillList struct {
name string
}
skillList := SkillList{}
skillArr := []SkillList{}
rows, err := db.Query("select DISTINCT(name) as name from skills where company_id IN ('2') and name != 'Skill Needed' order by name")
if err != nil {
log.Fatal(err)
}
defer rows.Close()
for rows.Next() {
err := rows.Scan(&skillName)
if err != nil {
log.Fatal(err)
}
skillList.name = skillName
skillArr = append(skillArr, skillList)
}
response := u.Message(true, "Skill list retrieved successfully")
response["data"] = skillArr
log.Println(skillArr)
response["authorization"] = false
return response
}
authController.go
var SkillTagList = func(w http.ResponseWriter, r *http.Request) {
resp := models.GetSkillList()
u.Respond(w, resp)
}
routes.go
router.HandleFunc("/api/v1/authorize/skillTagList", controllers.SkillTagList).Methods("POST")
If you see authorizeModel.go i have printed my data in logs i am getting that data successfully in logs. But see the postman screenshot below.
You have to rename name to Name
I'm not sure what is u.Respond(), so I will assume it's a helper function of some framework that you are using, and I will assume u.Respond() is internally using json.Marshal.
If your struct has unexported fields(fields name starting with lowercase letter, in your case name), json.Marshal cannot access those field, and the result won't have name field. That is why you are getting empty objects in JSON.
Given the following data structure which has been created in the database and there is valid data in the rows in the appropriate tables:-
type Deployment struct {
gorm.Model
Name string `gorm:"unique_index:idx_name"`
RestAPIUser string
RestAPIPass string
Servers []Server
model *Model
}
type Server struct {
gorm.Model
DeploymentID uint
Hostname string `gorm:"unique_index:idx_hostname"`
RestPort string
Version string
}
I'm trying to select all Deployments and have GORM automatically fill the Servers for each Deployment.
Unfortunately, it doesn't do this. I've tried several variations of using the Associations() func but I can't seem to get it to work. I seem to have to do this manually:-
func (m *Model) GetDeployments() ([]Deployment, error) {
deployments := []Deployment{}
err := m.db.Find(&deployments).Error
if err != nil {
return nil, err
}
deploymentsWithServers := []Deployment{}
for _, d := range deployments {
servers := []Server{}
err := m.db.Model(&d).Association("Servers").Find(&servers).Error
if err != nil {
return nil, err
}
d.Servers = servers
deploymentsWithServers = append(deploymentsWithServers, d)
}
return deploymentsWithServers, nil
}
Does anyone have any suggestions how I can get GORM to fill the Servers field automatically? Thanks!
Try
m.db.Preload("Servers").Find(&Deployment{})