pq.NewListener/Listen blocks with pq: syntax error at or near "listen" - cockroachdb

I have minio/s3 object store with lambda notifications into cockroachdb (postgres db). I am trying to monitor these events with below golang code.
package main
import (
"database/sql"
"encoding/json"
"fmt"
"github.com/lib/pq"
"time"
)
const (
//crdbConnectStr = "dbname=alerts user=crdbuser1 host=localhost port=26257 sslmode=disable connect_timeout=5"
crdbConnectStr = "postgres://crdbuser1#localhost:26257/alerts?sslmode=disable"
dbDriver = "postgres"
)
func monitorEvents() {
_, err := sql.Open(dbDriver, crdbConnectStr)
if err != nil {
fmt.Printf("connection open to crdb failed - %v\n", err.Error())
}
fmt.Printf("sql open on crdb OK\n")
reportProblem := func(ev pq.ListenerEventType, err error) {
if err != nil {
fmt.Printf("NewListener - event : %v, err - %v\n", ev, err.Error())
}
}
minReconnect := 2 * time.Second
maxReconnect := 20 * time.Second
listener := pq.NewListener(crdbConnectStr, minReconnect, maxReconnect, reportProblem)
err = listener.Listen("monitor")
if err != nil {
fmt.Printf("Listen error - %v\n", err.Error())
return
}
fmt.Printf("begin monitoring events in CRDB\n")
for {
waitForAlertEvents(listener)
}
}
// Record holds json data from object.
type Record struct {
Data struct {
Value struct {
Records []struct {
S3 struct {
Bucket struct {
Name string `json:"name"`
} `json:"bucket"`
Object struct {
Key string `json:"key"`
} `json:"object"`
} `json:"s3"`
} `json:"Records"`
} `json:"value"`
} `json:"data"`
}
func waitForAlertEvents(l *pq.Listener) {
for {
select {
case n := <-l.Notify:
fmt.Printf("Received data from channel [%v]\n", n.Channel)
// Prepare notification payload for pretty print
fmt.Println(n.Extra)
record := Record{}
jerr := json.Unmarshal([]byte(n.Extra), &record)
if jerr != nil {
fmt.Println("Error processing JSON: ", jerr)
return
}
bucket := record.Data.Value.Records[0].S3.Bucket.Name
object := record.Data.Value.Records[0].S3.Object.Key
fmt.Printf("received event on bucket: %v, object: %v\n", bucket, object)
return
case <-time.After(60 * time.Second):
fmt.Println("Received no events for 90 seconds, checking connection")
go func() {
l.Ping()
}()
return
}
}
}
func main() {
monitorAlerts()
}
When I run this program, I see the below errors and it stucks.
[root]# ./alerts
sql open on crdb OK
NewListener - event : 3, err - pq: syntax error at or near "listen"
NewListener - event : 3, err - pq: syntax error at or near "listen"
NewListener - event : 3, err - pq: syntax error at or near "listen"
Connection to cockroachdb manually works OK.
[root]# cockroach sql --insecure --user=crdbuser1
crdbuser1#:26257/defaultdb> show databases; database_name
+---------------+
alerts
(1 row)
Time: 1.22359ms
crdbuser1#:26257/defaultdb> set database=alerts;
SET
Time: 363.994µs
crdbuser1#:26257/alerts> show tables;
table_name
+------------+
alertstab
(1 row)
Time: 1.399014ms
crdbuser1#:26257/alerts>
Any thoughts as why the error pq: syntax error at or near "listen". Also I was looking at
pq sources, the error most likely related to notify.go#L756

The error indicates that CockroachDB does not support the LISTEN and NOTIFY statements.
You will need to find a different way of doing this. The closest thing in CRDB would be Change Data Capture, but that is more about data streaming than custom notifications.
You can find some discussion about LISTEN / NOTIFY for CRDB in this issue, but there is no firm plan to date.

Related

GORM and PostgreSQL: search_path is used only when doing the first request

I'm currently working on an api gateway that forwards requests to an grpc client. The routes use a middleware. Within that middleware the Validate function from the dialed grpc client is used. The function is in the last code snippet at the end.
The gateway is based on a gin router, which, as far as I understand handles each request in a separat go routine. I can send as many requests simultaneously to either to endpoint /protected/1 or to endpoint /protected/2 and they get handled correctly.
The Validate function is having problems handling request sent simultaneously to /protected/1 and /protected/2. Sending two request to both endpoints simultaneously results in that either both, none or only one of the requests is handled correctly. The error message that I receive is actually from the gorm function trying to query the user in the Validate function (ERROR: relation "users" does not exist (SQLSTATE 42P01)). I use a gorm database object that is connected to a postgres database. This behaviour suggests that some how a resource is shared. Using run -race does not give any insights.
So my questions are, why does this setup not work to send requests simultaneously to both endpoints?
I tried to make a working minimal example, but somehow I could nor reproduce the error in a minimal setting, hence I would like to share my code snippets here.
Important snippets of the the Api Gateway implementation
func main() {
c, err := config.LoadConfig()
...
r := gin.Default()
auth.RegisterRoutes(r, &c)
r.Run(":" + c.Port)
}
whereas the routes are defined as
func RegisterRoutes(r *gin.Engine, c *config.Config){
svc := &ServiceClient{
Client: InitServiceClient(c),
}
a := InitAuthMiddleware(svc)
// routes
protected := r.Group("/protected")
protected.Use(a.AuthRequired)
protected.GET("/1", svc.DoStuff)
protected.GET("/2", svc.DoStuff)
}
// dummy handler
func (svc *ServiceClient) DoStuff(ctx *gin.Context) {
handler.DoStuff(ctx, svc.Client)
}
The grpc client is dialed with the following function
type ServiceClient struct {
Client pb.AuthServiceClient
}
func InitServiceClient(c *config.Config) pb.AuthServiceClient {
// using WithInsecure() because no SSL running
cc, err := grpc.Dial(c.AuthSvcUrl, grpc.WithInsecure())
if err != nil {
fmt.Println("Could not connect:", err)
}
return pb.NewAuthServiceClient(cc)
}
and the middleware is implemented in the following way:
type AuthMiddlewareConfig struct {
svc *ServiceClient
}
func InitAuthMiddleware(svc *ServiceClient) AuthMiddlewareConfig {
return AuthMiddlewareConfig{svc}
}
func (c *AuthMiddlewareConfig) AuthRequired(ctx *gin.Context) {
access_token, _ := ctx.Cookie("access_token")
res, err := c.svc.Client.Validate(context.Background(), &pb.ValidateRequest{
Token: access_token,
TokenType: "ACCESS_TOKEN",
Url: ctx.Request.URL.String(),
})
if err != nil || res.Status != http.StatusOK {
ctx.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{"status": http.StatusUnauthorized, "error": res.Error})
return
}
ctx.Set("userId", res.UserId)
ctx.Next()
}
Important snippets of the Auth Service:
func serve(c *conf.Configuration) {
/*
Dial returns:
type Repository struct {
DB *gorm.DB
}
*/
h := storage.Dial(&c.DB)
serviceUri := fmt.Sprintf(":%s", c.API.Port)
lis, err := net.Listen("tcp", serviceUri)
if err != nil {
logrus.Fatal("Failed to listen on: ", err)
}
s := api.Server{
R: h,
//
}
grpcServer := grpc.NewServer()
pb.RegisterAuthServiceServer(grpcServer, &s)
if err := grpcServer.Serve(lis); err != nil {
logrus.Fatalln("Failed to serve:", err)
}
}
and the validate function is implemented by
type Server struct {
R storage.Repository
//
// #https://github.com/grpc/grpc-go/issues/3794:
pb.UnimplementedAuthServiceServer
}
func (s *Server) Validate(ctx context.Context, req *pb.ValidateRequest) (*pb.ValidateResponse, error) {
// token validation stuff works
var user models.User
// causes error sometimes
if result := s.R.DB.Where(&models.User{Id: claims.Id}).First(&user); result.Error != nil {
return &pb.ValidateResponse{
Status: http.StatusNotFound,
Error: "User not found",
}, nil
}
// send response
}
EDIT
For completeness here the function that connects to the postgres client.
func openDb(c *conf.PostgresConfiguration, gormConfig *gorm.Config, database string) *gorm.DB {
connectionString := fmt.Sprintf("postgres://%s:%s#%s:%s/%s", c.User, c.Password, c.Host, c.Port, database)
db, err := gorm.Open(postgres.Open(connectionString), gormConfig)
if err != nil {
logrus.Fatal(fmt.Sprintf("Unable to connect to database '%s' given url: ", database), err)
}
logrus.Info(fmt.Sprintf("Connected to database %s", database))
return db
}
func Dial(c *conf.PostgresConfiguration) Repository {
gormDefaultConfig := &gorm.Config{}
gormAppConfig := &gorm.Config{}
// connect to default database
logrus.Info(fmt.Sprintf("Use default database %s for initialization", c.DefaultDatabase))
defaultDb := openDb(c, gormDefaultConfig, c.DefaultDatabase)
// check if database exists and create it if necessary
var dbexists bool
dbSQL := fmt.Sprintf("SELECT EXISTS (SELECT FROM pg_database WHERE datname = '%s') AS dbexists;", c.AppDatabase)
defaultDb.Raw(dbSQL).Row().Scan(&dbexists)
if !dbexists {
logrus.Info(fmt.Sprintf("Created database %s", c.AppDatabase))
db := defaultDb.Exec(fmt.Sprintf("CREATE DATABASE %s;", c.AppDatabase))
if db.Error != nil {
logrus.Fatal("Unable to create app database: ", db.Error)
}
}
// connect to app databse
appDb := openDb(c, gormAppConfig, c.AppDatabase)
// check if schema exists and create it if necessary
var schemaexists bool
schemaSQL := fmt.Sprintf("SELECT EXISTS(SELECT FROM pg_namespace WHERE nspname = '%s') AS schemaexisits;", c.AppDatabaseSchema)
appDb.Raw(schemaSQL).Row().Scan(&schemaexists)
// create app specfic database, if not already existing
if !schemaexists {
// create service specific schema
db := appDb.Exec(fmt.Sprintf("CREATE SCHEMA %s;", c.AppDatabaseSchema))
if db.Error != nil {
logrus.Fatal(fmt.Sprintf("Unable to create database schema %s", c.AppDatabaseSchema), db.Error)
}
logrus.Info(fmt.Sprintf("Created database schema %s", c.AppDatabaseSchema))
}
db := appDb.Exec(fmt.Sprintf(`set search_path='%s';`, c.AppDatabaseSchema))
logrus.Info(fmt.Sprintf("Use existing database schema %s", c.AppDatabaseSchema))
if db.Error != nil {
logrus.Fatal(fmt.Sprintf("Unable to set search_path for database to schema %s", c.AppDatabaseSchema), db.Error)
}
// migrate table
appDb.AutoMigrate(&models.User{})
return Repository{appDb}
}
Edit2
Changing the connection string in the openDb function to connectionString := fmt.Sprintf("postgres://%s:%s#%s:%s/%s?search_path=%s", c.User, c.Password, c.Host, c.Port, database, c.AppDatabaseSchema) fixes the issue (see how the search_path is part of the url).
Thank you!

How to write in streaming to bigquery from Apache Beam?

I'm trying to use the Apache Beam Go SDK for Dataflow to read from PubSub and write to BigQuery (in streaming so).
The following code without the window is running but nothing is written into BigQuery.
The following code with the window is not running : I got Error processing pipeline. in Dataflow.
package main
import (
"context"
"encoding/json"
"flag"
"time"
"github.com/apache/beam/sdks/go/pkg/beam"
"github.com/apache/beam/sdks/go/pkg/beam/io/bigqueryio"
"github.com/apache/beam/sdks/go/pkg/beam/io/pubsubio"
"github.com/apache/beam/sdks/go/pkg/beam/log"
"github.com/apache/beam/sdks/go/pkg/beam/x/beamx"
"github.com/apache/beam/sdks/go/pkg/beam/x/debug"
)
type (
ProcessedTask struct {
TaskId string `json:"task_id" bigquery:"task_id"`
Connector string `json:"connector" bigquery:"connector"`
Success bool `json:"success" bigquery:"success"`
Retry bool `json:"retry" bigquery:"retry"`
Message string `json:"message" bigquery:"message"`
FinishedAt time.Time `json:"finished_at" bigquery:"finished_at"`
}
)
func buildPipeline(s beam.Scope) {
rawProcessedTasks := pubsubio.Read(s, "project", "topic", &pubsubio.ReadOptions{Subscription: "subscription"})
processedTasks := beam.ParDo(s, func(ctx context.Context, data []byte) (ProcessedTask, error) {
var task ProcessedTask
if err := json.Unmarshal(data, &task); err != nil {
log.Error(ctx, err)
return task, err
}
return task, nil
}, rawProcessedTasks)
debug.Printf(s, "Task : %#v", processedTasks)
/* The window I added */
windowedProcessedTasks := beam.WindowInto(s, window.NewFixedWindows(time.Minute), processedTasks)
/* end of the added window */
bigqueryio.Write(s, "project", "table", processedTasks)
}
func main() {
flag.Parse()
beam.Init()
p, s := beam.NewPipelineWithRoot()
buildPipeline(s)
ctx := context.Background()
if err := beamx.Run(ctx, p); err != nil {
log.Exitf(ctx, "Failed to execute pipeline: %v", err)
}
}
I tried to add a window since in the code, the data is buffered until it achieve 10000 rows or 10485760 bytes of data.
I would like to write in streaming or write each x minutes. I don't have a lot of events, I'm not sure to achieve that limit in a week and I need it to be near-realtime.
Does anyone have suggestions?

Parsing prometheus metrics from file and updating counters

I've a go application that gets run periodically by a batch. Each run, it should read some prometheus metrics from a file, run its logic, update a success/fail counter, and write metrics back out to a file.
From looking at How to parse Prometheus data as well as the godocs for prometheus, I'm able to read in the file, but I don't know how to update app_processed_total with the value returned by expfmt.ExtractSamples().
This is what I've done so far. Could someone please tell me how should I proceed from here? How can I typecast the Vector I got into a CounterVec?
package main
import (
"fmt"
"net/http"
"strings"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
dto "github.com/prometheus/client_model/go"
"github.com/prometheus/common/expfmt"
"github.com/prometheus/common/model"
)
var (
fileOnDisk = prometheus.NewRegistry()
processedTotal = prometheus.NewCounterVec(prometheus.CounterOpts{
Name: "app_processed_total",
Help: "Number of times ran",
}, []string{"status"})
)
func doInit() {
prometheus.MustRegister(processedTotal)
}
func recordMetrics() {
go func() {
for {
processedTotal.With(prometheus.Labels{"status": "ok"}).Inc()
time.Sleep(5 * time.Second)
}
}()
}
func readExistingMetrics() {
var parser expfmt.TextParser
text := `
# HELP app_processed_total Number of times ran
# TYPE app_processed_total counter
app_processed_total{status="ok"} 300
`
parseText := func() ([]*dto.MetricFamily, error) {
parsed, err := parser.TextToMetricFamilies(strings.NewReader(text))
if err != nil {
return nil, err
}
var result []*dto.MetricFamily
for _, mf := range parsed {
result = append(result, mf)
}
return result, nil
}
gatherers := prometheus.Gatherers{
fileOnDisk,
prometheus.GathererFunc(parseText),
}
gathering, err := gatherers.Gather()
if err != nil {
fmt.Println(err)
}
fmt.Println("gathering: ", gathering)
for _, g := range gathering {
vector, err := expfmt.ExtractSamples(&expfmt.DecodeOptions{
Timestamp: model.Now(),
}, g)
fmt.Println("vector: ", vector)
if err != nil {
fmt.Println(err)
}
// How can I update processedTotal with this new value?
}
}
func main() {
doInit()
readExistingMetrics()
recordMetrics()
http.Handle("/metrics", promhttp.Handler())
http.ListenAndServe("localhost:2112", nil)
}
I believe you would need to use processedTotal.WithLabelValues("ok").Inc() or something similar to that.
The more complete example is here
func ExampleCounterVec() {
httpReqs := prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "http_requests_total",
Help: "How many HTTP requests processed, partitioned by status code and HTTP method.",
},
[]string{"code", "method"},
)
prometheus.MustRegister(httpReqs)
httpReqs.WithLabelValues("404", "POST").Add(42)
// If you have to access the same set of labels very frequently, it
// might be good to retrieve the metric only once and keep a handle to
// it. But beware of deletion of that metric, see below!
m := httpReqs.WithLabelValues("200", "GET")
for i := 0; i < 1000000; i++ {
m.Inc()
}
// Delete a metric from the vector. If you have previously kept a handle
// to that metric (as above), future updates via that handle will go
// unseen (even if you re-create a metric with the same label set
// later).
httpReqs.DeleteLabelValues("200", "GET")
// Same thing with the more verbose Labels syntax.
httpReqs.Delete(prometheus.Labels{"method": "GET", "code": "200"})
}
This is taken from the Promethus examples on Github
To use the value of vector you can do the following:
vectorFloat, err := strconv.ParseFloat(vector[0].Value.String(), 64)
if err != nil {
panic(err)
}
processedTotal.WithLabelValues("ok").Add(vectorFloat)
This is assuming you will only ever get a single vector value in your response. The value of the vector is stored as a string but you can convert it to a float with the strconv.ParseFloat method.

Parallel execution of goroutine query in gorm.io/gorm does not work

I have 2 entities:
type Service struct {
ID string
Name string
}
type ConnectedService struct {
ID int
ServiceID string
Service Service
}
I want to get the associated Service entity when I get the ConnectedService.
In gorm.v1 everything worked, in the new version of gorm.v2, when I try to make a request in parallel threads, I get the error:
ERROR: invalid reference to FROM-clause entry for table "connected_services" (SQLSTATE 42P01)
[4.020ms] [rows:0] SELECT "connected_services"."id","connected_services"."service_id" FROM "connected_services" Service WHERE "connected_services"."id" = 3 ORDER BY "connected_services"."id" LIMIT 1
FATA[0000] ERROR: invalid reference to FROM-clause entry for table "connected_services" (SQLSTATE 42P01)
This is due to the fact that gorm.v2 added contexts and sessions to requests:
Method Chain Safety/Goroutine Safety
To reduce GC allocs, GORM V2 will share Statement when using method chains, and will only create new Statement instances for new initialized *gorm.DB or after a New Session Method, to reuse a *gorm.DB, you need to make sure it just after a New Session Method, for example:
But I can't do it. Method for getting connected service:
type Repository struct { db *gorm.DB }
func (r *Repository) GetConnectedService(id int) (*ConnectedService, error) {
cs := &ConnectedService{}
if err := r.db.WithContext(context.Background()).Joins("Service").First(cs, id).Error; err != nil {
return nil, err
}
return cs, nil
}
Init db and repository:
func main() {
dsn := fmt.Sprintf("host=localhost port=5433 user=guest password=guest dbname=mydb sslmode=disable")
db, _ := gorm.Open(postgres.Open(dsn), &gorm.Config{})
repo := &Repository{db: db}
. . .
}
I call it like this:
import (
"gorm.io/driver/postgres"
"gorm.io/gorm"
)
func main() {
. . .
for i := 1; i < 10; i++ {
go func(i int) {
cs, err := repo.GetConnectedService(i)
if err != nil {
logrus.Errorf("get connected service: %s", err.Error())
return
}
logrus.Printf("%+v", cs)
}(i)
}
. . .
}
Without <.Joins("Service")> everything works fine, but as soon as we add this, an error crashes.

Semantics - Passing outer interface type value to inner interface variable

Below is the prototype code for 3 layer API(with some design limitations):
// Sample program demonstrating interface composition.
package main
import (
"errors"
"fmt"
"io"
"math/rand"
"time"
)
func init() {
rand.Seed(time.Now().UnixNano())
}
// =============================================================================
// Data is the structure of the data we are copying.
type Data struct {
Line string
}
// =============================================================================
// Puller declares behavior for pulling data.
type Puller interface {
Pull(d *Data) error
}
// Storer declares behavior for storing data.
type Storer interface {
Store(d *Data) error
}
// PullStorer declares behavior for both pulling and storing.
type PullStorer interface {
Puller
Storer
}
// =============================================================================
// Xenia is a system we need to pull data from.
type Xenia struct {
Host string
Timeout time.Duration
}
// Pull knows how to pull data out of Xenia.
func (*Xenia) Pull(d *Data) error {
switch rand.Intn(10) {
case 1, 9:
return io.EOF
case 5:
return errors.New("Error reading data from Xenia")
default:
d.Line = "Data"
fmt.Println("In:", d.Line)
return nil
}
}
// Pillar is a system we need to store data into.
type Pillar struct {
Host string
Timeout time.Duration
}
// Store knows how to store data into Pillar.
func (*Pillar) Store(d *Data) error {
fmt.Println("Out:", d.Line)
return nil
}
// =============================================================================
// System wraps Xenia and Pillar together into a single system.
type System struct {
Xenia
Pillar
}
// =============================================================================
// pull knows how to pull bulks of data from any Puller.
func pull(p Puller, data []Data) (int, error) {
for i := range data {
if err := p.Pull(&data[i]); err != nil {
return i, err
}
}
return len(data), nil
}
// store knows how to store bulks of data from any Storer.
func store(s Storer, data []Data) (int, error) {
for i := range data {
if err := s.Store(&data[i]); err != nil {
return i, err
}
}
return len(data), nil
}
// Copy knows how to pull and store data from any System.
func Copy(ps PullStorer, batch int) error {
data := make([]Data, batch)
for {
i, err := pull(ps, data)
if i > 0 {
if _, err := store(ps, data[:i]); err != nil {
return err
}
}
if err != nil {
return err
}
}
}
// =============================================================================
func main() {
sys := System{
Xenia: Xenia{
Host: "localhost:8000",
Timeout: time.Second,
},
Pillar: Pillar{
Host: "localhost:9000",
Timeout: time.Second,
},
}
if err := Copy(&sys, 3); err != io.EOF {
fmt.Println(err)
}
}
My understanding is,
If you pass u, the interface contains type of u, and a pointer to a copy of u. If you pass &u, the interface contains the type of &u and the address of u.
So, below is my understanding for variables p, s & ps, from above code:
But, I would like to confirm,
1) On pull(ps, data), does first element of p contain type *System or Xenia type, as first element?
2) On store(ps, data[:i]), does first element of s contain type *System or Pillar type?
The type is *System.
You can confirm this by adding simple print statement, e.g. fmt.Printf("%T\n", p)

Resources