I'm using Strapi with an RDS database cluster backing it. The database cluster has a writer endpoint and a reader endpoint, and the readers can be scaled horizontally, but there can only ever be one writer. Currently, with just very basic Strapi database configuration, 100% of traffic goes through the writer.
I'd like to configure Strapi so that any management through the portal is done using the writer connection, but any querying via API (which an external server does to fetch content) is done via the reader connection. This would be a pretty big benefit for me because those API calls never need to change data, and it would allow me to scale more easily.
I've looked through the documentation but not seen anything obvious. I've also tried scanning through the code that I already have, but since actually connecting to the database is abstracted so far away from us, I'm not even sure where to start.
Related
I am building an application using couchbase as my primary db.
I want to make the application scalable enough to handle multiple requests at times concurrently.
How do you create connection pools for couchbase in Go?
Postgres has pgxpool.
I'll give a bit more detail about how gocb works. Under the hood of gocb is another SDK called gocbcore (direct usage of gocbcore is not supported) which is a fully asynchronous API. Gocb provides a synchronous API over gocbcore as well as making the API quite a lot more user friendly.
What this means is that if you issue requests across multiple goroutines then you can get multiple requests written to the network at a time. This is effectively how the gocb bulk API works - https://github.com/couchbase/gocb/blob/master/collection_bulk.go. Both of these approaches are documented at https://docs.couchbase.com/go-sdk/current/howtos/concurrent-async-apis.html.
If you still don't get enough throughput then you can look at using one of these approaches alongside increasing the number of connections that the SDK makes to each node by using the kv_pool_size query string option in your connection string, i.e. couchbases://10.112.212.101?kv_pool_size=2 however I'd recommend only changing this if the above approaches are not providing the throughput that you need. The SDK is designed to be highly performant anyway.
go-couchbase has already have a connection pool mechanism: conn_pool.go (even though there are a few issues linked to it, like issue 91).
You can see it tested in conn_pool_test.go, and in pools.go itself.
dnault points out in the comments to the more recent couchbase/gocb, which does have a Cluster instead of pool of connections.
I'm trying to build an app with realtime text editing and am stuck on how to best architecturally proceed. Currently, I'm using graphql with apollo and hasura to fetch user profile information along with document metadata and content.
To save content, we are currently just debouncing graphql mutations of the entire document. The drawback here is a user can navigate away without the document content being written back. As a result, I want to move to an approach based on websockets.
For an app that involves realtime text editing, where on each stroke we're sending a delta update, should I try to
use graphql subscriptions with a custom resolver since I'm already using apollo and hasura, or
use a vanilla, separate websocket for each document and avoid using graphql completely?
In both cases, instead of directly writing back to postgres, we would use a redis cache while the websocket connection is live, and then persist the cache content back to postgres when the connection is closed. The former approach would take advantage of already having apollo client, and authentication through hasura; while the latter approach would avoid sending binary blobs (delta updates) over graphql but require more setup.
I have a small web and mobile application partly running on a webserver written in PHP (Symfony). I have a few clients using the application, and slowly expanding to more clients.
My back-end architecture looks like this at the moment:
Database is Cloud SQL running on GCP (every client has it's own
database instance)
Files are stored on Cloud Storage (GCP) or S3 (AWS), depending on the client. (every client has it's own bucket)
PHP application is running in a Compute Engine VM (GCP), (every client has it's own VM)
Now the thing is, in the PHP code, the only thing client specific is a settings file with the database credentials and the Storage/S3 keys in it. All the other code is exactly the same for every client. And mostly the different VMs sit idle all day, waiting on a few hours usage per client.
I'm trying to find a way to avoid having to create and maintain a VM for every customer. How could I rearchitect my back-end so I can keep separate Databases and Storage Buckets per client, but only scale up my VM's when capacity is needed?
I'm hearing alot about Docker, was thinking about keeping db credentials and keys in a Redis DB or Cloud Datastore, was looking at Heroku, AppEngine, Elastic Beanstalk, ...
This is my ideal scenario as I see it now
An incoming request is done, hits a load balancer
From the request, determine which client the request is for
Find the correct settings file, or credentials from a DB
Inject the settings file in an unused "container"
Handle the request
Make the container idle again
And somewhere in there, determine based on the the amount of incoming requests or traffic, if I need to spin up or spin down containers to handle the extra or reduced (temporary) load.
All this information overload has me stuck, I have no idea what direction to choose, and I fail seeing how implementing any of the above technologies will actually fix my problem.
There are several ways do it with minimum efforts:
Rewrite loading of config file depending from customer
Make several back-end web sites on one VM (best choice i think)
I'm new to CouchDb and am trying to comprehend how to properly make use of it. I'm coming from MongoDB where I would always write a web layer and put it in front of mongo so that I could allow users to access the data inside of it, etc. In fact, this is how I've used all databases for every web site that I've ever written. So, looking at Couch, I see that it's native API is HTTP and that it has built in things like OAuth support, and other features that hint to me that perhaps I should no longer have my code layer sitting in front of Couch, but instead write Views and things and just give out accounts to Couch to my users? I'm thinking in terms of like an HTTP-based API for a site of mine, or something that users would consume my data through. Opening up Couch like this seems odd to me, though. Is OAuth, in Couch's sense, meant more for remote access for software that I'd write and run internal to my own network "officially", or is it literally meant for the end users?
I know there might be things that could only be done through a code layer on top of CouchDB, like if you wanted additional non-database related things to occur during API requests, also. So thinking along those lines I think I will still need a code layer, anyway.
Dealer's choice.
Nodejitsu has a great writeup on this sort of topic here.
Not knowing your application specifics I'll take a broad approach...
Back-end
If you want to prevent users from ever seeing your database then make it back-end. You can pipe everything through something like node.js and present only what the user needs to see and they'll never know anything about the database.
See Resource View Presenter
Front-end
If you are not concerned about data security, you can host an entire app on CouchDB; see CouchApp. This approach has the benefit of using the replication mechanism to control publishing your site/data. The drawback here is that you will almost certainly run into some technical limitations that will require moving CouchDB closer to the backend.
Bl-end
Have the app server present the interface and the client pull the data from the database separately. This gives the most flexibility but can be a bag of hurt because even with good design this could lead to supportability and scalability issues.
My recommendation
Use CouchDB on the backend. If you need mobile clients to synchronize then use a secondary DB publicly exposed for this purpose and selectively sync this data to wherever it needs to go.
Simply put, no.
There's no way to secure Couch properly on a public facing site. There's no way to discriminate access at a fine enough granular level. If someone has access to any of the data, they have access to all of the data.
Not all data on a site is meant for public consumption, save for the most trivial of sites.
We have an application provided by a third party which takes a stream of market data (provided by said third party), and writes it into a JDBC compatible database.
The only configuration parameters it has are the JDBC connection string, plus settings allowing us to pick what pieces of data we'd like to be stored in this database.
This is very good for static data, but we'd like to feed this data into our internal ActiveMQ messaging fabric (in addition to writing it into the DB).
The database updates are triggered by pushes of market data to us. I'd like to have this application write the data directly to a set of MQ topics by implementing some kind of jdbc "facade" that would re-route the data directly into MQ.
What I don't want to do is poll the database for new information - as I want to keep the same fluidity of the data (e.g. fast moving stocks will generate a lot more data than slow moving - and we'd want to retain this).
Advice and pointers are very much welcome!
Camel is the answer, but potentially only if you're ok with polling the database. It's great for integration issues like this. If there was some other trigger that you could work with, you could use that to cause the database to be read.