I'm confused about transactions with FastAPI and async SQLAlchemy (i.e. version 1.4+). The FastAPI tutorial for SQL databases uses request-scope transactions created via a FastAPI dependency. This is what I'd expect.
But then the documentation has another tutorial for async SQL databases in which I can't see any sign of transactions. It does use a PyPi dependency called databases which provides "simple asyncio support for a range of databases". I don't know if it does transactions internally but I don't see how they could be request-scoped without help from the calling application.
And then again I see a blog article about FastAPI and async SQLAlchemy which appears to do things the original way with request-scoped transactions and FastAPI dependencies. Which is the "correct" approach? Does it matter if one is using Core or ORM mode or is that no longer a thing in SQLAlchemy 1.4?
databases is a separate library that have different logic of work. It take from SQLAlchemy only query building and do not use SQLAlchemy Sessions.
If you are interesting how to handle transactions in databases code will be something like that:
from databases import Database
from sqlalchemy import insert, select
async def create_something(db: Database, something: SomethingCreateSchema) -> SomethingSchema:
async with db.transaction(): # <- here
something_id = await db.execute(query=insert(SomethingDBModel), values=something.dict())
db_something = await db.fetch_one(query=select(SomethingDBModel).filter(User.id == user_id))
return SomethingSchema(**db_something)
In this case all DB calls will be in same transaction
Related
For AWS Lambda handlers, does it matter if the code is async vs sync (eg async def vs def), with respect to the concurrency performance of multiple calls to that Lambda function? That is, if I write a synchronous function, will that function be called "asynchronously" anyway due to the way AWS internally handles Lambda invocations from separate calls? Or will the function start processing UserA, and queue UserB's request until it has completed UserA?
Backstory: I'm writing some Python + SQLAlchemy Lambda functions. SQLAlchemy 1.4 just released asyncio support, but there will be some re-write overhead on my part. I'm wondering if I can just write traditional synchronous Python code without concurrency consideration on Lambda (where concurrency is something I'd definitely need to consider for custom Flask / FastAPI code).
If you have more than one request to invoke a lambda at the same time the service will attempt to run multiple instances of the lambda. You may run into throttling due to the concurrency limits on either the lambda itself, or the account. Lambda does not queue requests to the same lambda function. A function is never processing more than one request at a time. Even in cases where the system is providing a queuing mechanism to handle the requests, they are not a thread queue, like you might have with a server. They are really just internal SQS queues that are invoking the lambda with one request at a time.
I'm starting with kafka and I need to control the inserts in a specific Oracle table, send the new records through kafka at the moment. I have no control over the database, so, in principle, Debizium is excluded. How can I do this? Without using triggers.
I've made a producer read data from Oracle with a java program in eclipse but, that would make constant requests to the database. I use java for simulated a ETL with consumer.
PS: I work with Windows but that's secondary.
If I understand your problem correctly, you are trying to route inserts from Kafka to Oracle Database. There could be few possibilities:
You implement Kafka consumer and as soon as your kafka cluster gets a message consumer makes a insert. You could reuse your java code here- just remove the polling part. Please visit here
If you have kafka deployed in a cloud environment and are using it as a service(aws msk) you would have the option to handling the events. Again you can use java program or can write a python script to make inserts. Please visit here
I would like to understand your throughput requirements, whether you really need kafka as a distributed messaging system or a simple aws sqs would work just fine. If you can use sqs things would be straightforward for you. You create a queue and you write a listener in
python or java
boto3 is an excellent python library for working with sqs
What is the difference between PubSub and Methods in Meteor?!
Can I put Methods in Server folder like Publishs?
To me seen like the same, but Methods is more reactive.
They are two different sides of the same coin. Here's a drawing of the data lifecycle in meteor:
Publish - Which data is sent from the server
Subscribe - Which data the client requests publications for
Methods - How to manipulate data from the client on the server
Note - this will typically be run on both on the client and the server. The client will make a prediction as to what the server will do so it can update right away. Then latency compensation will kick in when the method is run on the server and the canonical decision is made.
What is the difference between PubSub and Methods in Meteor?!
Publications are reactive and they provide a cursor. Subscription gets you the matching publication on clientside in a minimongo database. On the other hand, methods must be called instead of subscribed and they are mainly designed to execute server side tasks that you don't want to handle client side for many possible reasons.
More details here for publications: https://www.discovermeteor.com/blog/understanding-meteor-publications-and-subscriptions/
And here for methods:
http://meteortips.com/first-meteor-tutorial/methods/
Can I put Methods in Server folder like Publishs?
Yes you can and you should. For example, put them into server\methods
To me seen like the same, but Methods is more reactive.
This is the exact contrary. They are not the same, even if you can achieve similar results with both. Methods are by design not reactive, and pub/sub are.
Most of the articles on the web dealing with WebSockets are about in-memory Chat.
I'm interested in kind of less instant Chat, that is persistent, like a blog's post's comments.
I have a cluster of two servers handling client requests.
I wonder what could be the best strategy to handle pushing of database update to corresponding clients.
As I'm using Heroku to handle this cluster (of 2 web dynos), I obviously read this tutorial aiming to build a Chat Room shared between all clients.
It uses Redis in order to centralize coming messages; each server listening for new messages to propagate to web clients through websocket connections.
My use case differs in that I've got a Neo4j database, persisting into it each message written by any client.
My goal is to notify each client from a specific room that a new message/comment has just been persisted by a client.
With an architecture similar to the tutorial linked above, how could I filter only new messages to propagate to user? Is there an easy and efficient way to tell Redis:
"(WebSocket saying) When my client initiates the websocket connection, I take care to make a query for all persisted messages and sent them to client, however I want you (Redis) to feed me with all NEW messages, that I didn't send to client, so that I will be able to provide them."
How to prevent Redis from publishing the whole conversation each time a websocket connection is made? It would lead to duplications since the database query already provided the existing contents at the moment.
This is actually a pretty common scenario, where you have three components:
A cluster of stateless web servers that maintain open connections with all clients (load balanced across the cluster, obviously)
A persistent main data storage - Neo4j in your case
A messaging/queueing backend for broadcasting messages across channels (thus across the server cluster) - Redis
Your requirement is for new clients to receive an initial feed of the recent messages, and any consequent messages in real-time. All of this is implemented in your connection handlers.
Essentially, this is what your (pseudo-)code should look like:
class ConnectionHandler:
redis = redis.get_connection()
def on_init():
self.send("hello, here are all the recent messages")
recent_msgs = fetch_msgs_from_neo4j()
self.send(recent_msgs)
redis.add_listener(on_msg)
self.send("now listening on new messages")
def on_msg(msg):
self.send("new message: ")
self.send(msg)
The exact implementation really depends on your environment, but this is the general flow of things.
Is it possible to somehow subscribe to 'add to db' event using Node.js? Database is currently populated via Ruby on Rails. Thanks.
MongoDB wise
There's an ongoing discussion about triggers on mongodb Jira.
But for now you're stuck with storing auto-increment values alongside with your data, and using indexed queries to check if there's anything new.
Rails wise
I'm assuming you're using Mongoid. Use callbacks or observers to send messages to a fast capped collection / unix socket / whatever. Other ODMs shouldn't be too different.
You need to notify the Node.js process from the Rails app when you insert something in the DB.
Listen on a socket/port from Node.js process
From Rails write on that socket when record added
From Node, process each message on the socket