I want to use the JOOQ DSL in Quarkus to build my SQL (and hopefully execute them).
Therefore I added the following Quarkus JOOQ extension.
Since I want to use the reactive PG SQL Client in my project, I'm asking myself if e.g. the fetch() method of JOOQ will block the thread ? Is it compatible with the reactive vertx client under the hood or does it use a blocking one ? Looks like the latter one since it doesn't return a future or anything like that.
In that case I propably should only use JOOQ for creating the SQL string.
Which parts of the jOOQ API can be used reactively
jOOQ's ResultQuery<R> extends Publisher<R>, so you can just place a jOOQ query in any reactive stream implementation. There are 3 main Publisher subtypes in jOOQ:
ResultQuery<R> extends Publisher<R>
RowCountQuery extends Publisher<Integer>
Batch extends Publisher<Integer>
And starting with jOOQ 3.17, there will also be a way to create transactional Publisher types.
With this in mind, in the reactive world, you will never need to call any of the traditional jOOQ blocking execution methods. You'll always implicitly execute jOOQ queries via some reactive streams integration.
Avoiding calls to blocking API
Starting with jOOQ 3.17, all the blocking API (e.g. ResultQuery.fetch()) will be annotated as org.jetbrains.annotations.Blocking, so you get IDE support to warn you that you're about to do something that might not make sense in your non-blocking context.
Backing implementation
For any of this to work, you need to provide jOOQ with an R2DBC connection. R2DBC is an SPI that enables interoperability between client libraries like jOOQ and R2DBC drivers, like r2dbc-postgres. Just like JDBC, it works as an SPI, not strictly an API. Besides, it integrates also directly with the reactive streams SPI, which has been integrated in the JDK 9 via the Flow API.
There might be future work to support alternative non-blocking drivers in the future, however R2DBC seemed to be the most interoperable choice at the time the reactive support was added, and I do hope that the Vert.x and R2DBC teams will find ways to cooperate more tightly in the future. The Vert.x SQL client, for example, does not implement the reactive streams SPI directly, Red Hat does not seem too interested (yet) in moving forward with this issue here: https://github.com/eclipse-vertx/vertx-sql-client/issues/249
So, for now, this means that you have to either:
Use jOOQ with R2DBC, which is what jOOQ supports (not sure if Quarkus will support R2DBC, though I don't see any reason why it shouldn't)
Use jOOQ to generate SQL only and run the SQL with Vert.x (you'll lose a lot of type safety and convenience, as well as access to advanced features like MULTISET, which relies on jOOQ executing your query)
A side note on reactive execution
Of course, it's always important to think about whether you really need to go reactive. In my personal experience, this is mostly a matter of programming style, not actual performance and/or load requirements. Sticking with the blocking paradigm and JDBC will greatly simplify your every day work, and I doubt you'll notice a measurable difference in production.
I'm looking for a solution to do the same thing, I haven't tested it yet but I came across this repo:
https://github.com/jklingsporn/vertx-jooq
https://github.com/jklingsporn/quarkus-jooq-reactive-example
It may help to be fully reative using vert.x in quarkus.
Related
I'm trying to use kotlin coroutines with "old-style" Spring JPA repository.
I create a new coroutines scope and run all JPA calls in "async".
I see that even with non reactive JDBC I improve my throughput.
But I wonder, may be exists some coroutines wrapper on Spring JPA repository?
Something created with reflection and Spring "magic"?
First I'd like to clarify one thing to prevent a possible confusion:
If you're using Spring Data JPA, then you should know that this framework uses JDBC driver underhood, that is actually a blocking API, that means all the database calls make the calling thread block until the total result is completed and ready to be consumed.
Having that knowledge I presume you're using suspend functions with coroutines that run on Dispatcher.IO for making such calls.
This dispatcher provides you with one thread (as far as I know, it scales up to 64 threads) for each call. And that thread actually blocks while making your database call, which means coroutines and suspend does not make any magic in this kind of situation except for switching your blocking call to another thread (which eventually will be blocked).
Maybe you should take a look at r2dbc (reactive SQL driver) and use CoroutineCrudRepository<T, ID> from Spring Data instead of using standard JpaRepostitory<T, ID>.
CoroutineCrudRepository<T, ID> has all the methods with suspend which means you can use them for creating "truly" async API without blocking at all.
However, r2dbc may be not suitable for your use cases since it has a lot of limitations, such as mapping relations, caching and etc.
UPDATED:
As far as I know there is no Spring-way to automatically wrap blocking calls, but you can take a look at this library
Spring boot 2.1.5
Project Reactor 3.2.9
I am setting up a bunch of rest reactive APIs using the above-mentioned frameworks and I am running into an annoying problem with MDC (mapped diagnostic context). My applications are in JAVA.
MDC relies on thread locals to store the current query's mapped context to put in the logs. That system, obviously, is not perfect and contradicts the reactive pattern since the different steps of your execution will be executed through different threads.
I have run into the same problem with the Play Reactive framework but found a workaround there by copying the mapped context transparently from one actor to another.
For spring and reactor, I could not find a satisfying solution yet.
Some random examples found on the internet:
First - It works but forces you to use a bunch of utility methods
Same thing
Second - It tries to copy the context during the onNext publisher event but seems to lose some features on the way of doing that. The signal context, for example, is lost.
I am in need of a proper solution to deal with this:
A library which would make the link between MDC and reactor?
A way to tweak reactor/spring to achieve it transparently?
Any advice?
"I could not find a satisfying solution yet."
Working with contexts is the only solution for the moment. Since as you said threadlocals goes against everything that has to do with reactive programming. Using thread local as a storage point during a request is a resource heavy way of solving things and in my opinion poor design. Unless logging frameworks themselves come up with a better solution to the problem we developers must pass the data through the context to accommodate for the logging frameworks blocking nature.
Reactive programming is a paradigm shift in the programming world. Other things like database drivers, that use threadlocal to rollback transactions are also in big trouble. the JDBC database driver spec is defined as blocking in nature, and atm. there has been attempts by spring and the R2DBC project to define a new JDBC driver spec that is inherently non/blocking. This means that all vendors must rewrite ther database driver implementations from scratch.
Reactive program is so new that lots of libraries need to rewrite entire codebases. The logging frameworks as we know it needs to be rewritten from the ground up which is a huge task. And the context in reactive is actually something that should not even be in reactive programming, it was implemented just to accommodate for MDC problems.
It's actually a lot of overhead needing to pass data from thread to thread.
So what can we do?
push on logging frameworks, and/or help logging frameworks to rewrite their codebase
Accept that there is no "tweak" that will magically fix this
use the context and the way suggested in the blogposts
Project reactor context
I found many examples of using spring boot reactive with document databases, but none with SQL databases.
I see that it may not support sql databases yet, probably because some missing feature on the jpa/jdbc stack.
I also see that there is no point to use reactive services that depend on the a sql database with no reactive support.
The question here is: Is there any ongoing development to make this happen (reactive jpa)?
There is a reactive feature built into many RDBMSs called "Change Data Capture" which writes data to an async transaction log for an enabled table. Usually, reactive systems built to stream this data are built on top of that feature. For example, a well-known open source tool that does this is Debezium. You can find other open source projects online that do something similar, or to write your own using the simple CDC functions that are usually provided to support it.
Is it possible to use Hibernate and RDBMS(Mysql, Postgres etc) with ReactiveCrudRepository instead of CrudRepository? I have tried some samples with Spring Data Jpa and Hibernate, but couldn't get it done. I was only able to find a few samples on ReactiveCrudRepository for MongoDB and cassandra.
Is it possible to use Hibernate and Mysql with ReactiveCrudRepository instead of CrudRepository?
TL;DR:
Not with Hibernate and MySQL, but with R2DBC and Postgres, Microsoft SQL Server or H2.
Take a look at Spring Data R2DBC.
Long Version
Why not JPA?
With Hibernate/JPA included this won't happen in the foreseeable future.
JPA is based on the idea that you load part of your data model into memory, manipulate the resulting object model and let JPA transform these changes.
All this within a single transaction.
This is kind of the opposite how one deals with a reactive store where you try to make atomic changes and try to decouple the loading, processing and storing and all this without blocking.
Why not JDBC?
So we have to look at the technology level below JPA: JDBC.
But JDBC is still blocking: You send a SQL statement to your database and then JDBC will block until you get the result.
And again this goes against the idea of reactive: Never block.
One could wrap this in a thread pool to mitigate this to some extent, but that is more of a workaround than a solution.
Why R2DBC?
There are some suitable drivers for some databases that could be used for reactive repositories.
But they are proprietary and thereby not a good basis for something that really should eventually work across all (relevant) relational databases.
For some time the Spring Data team hoped that ADBA would fill that gap.
But discussions on the mailing list made it clear that ADBA was not aiming for reactive but only for asynchronous.
Again not what we needed for a reactive repository abstraction.
So early in 2018 various people living at the intersection or reactive and relational decided that we need a standard for reactive database access.
R2DBC (Reactive Relational Database Connectivity)
is a proposal for such a standard.
The hope is that it either helps convincing Oracle to move ADBA to a reactive approach or if that doesn't happen it becomes the standard itself.
And with already three implementations available chances for the second option look promising.
R2DBC itself is mainly an SPI, i.e. an API that is to be implemented by database providers.
The SPI is designed in a way that puts minimal requirements on implementers.
But this also makes R2DBC somewhat cumbersome to use.
The idea is that other libraries will step up and build libraries designed for usability on top of that SPI, as it happened with JDBC.
Spring Data R2DBC
Spring Data R2DBC is one such library and it offers what you asked for: Support for ReactiveCrudRepository although it is independent of JPA/Hibernate and there is no support for MySQL yet.
State of the projects
Both R2DBC and Spring Data R2DBC didn't have a production release yet and it will take at least several months to get there.
Spring Data R2DBC just released the first milestone.
See the release article for its current capabilities.
R2DBC is on its 6th milestone. See the release article for details.
See also this answer: Why does Spring not provide reactive (non-blocking) clients for relational databases?
Original answer as a reference for archeologists:
As of now (Jan 2017) it is not possible.
The currently relevant release for the reactive part of Spring Data is Spring Data Kay M1 (You can check if there is a newer version available on the project home page)
And a blog post from the Spring Data team about that release and specifically the reactive parts in it starts with (emphasis mine):
Spring Data Kay M1 is the first release ever that comes with support for reactive data access. Its initial set of supported stores — MongoDB, Apache Cassandra, and Redis — all ship reactive drivers already, which made them very natural candidates for such a prototype.
The reason is that there is no standard non-blocking way to access a relational database. So only those that support this kind of API are supported right now.
One could implement a ReactiveCrudRepository using JPA or JDBC and delegate the work to a thread pool. This would provide an async API on the outside, but would still consume the resources for the Threads and block between independent data accesses, so only a small part of the benefits of the reactive approach would get realized.
Hibernate started a new Hibernate Reactive subproject for reactive streams support which provides Hibernate/JPA similar APIs to access RDBMS. But unfortunately at the moment, Spring Data does not support it. So there is no a ReactiveCrudRepoisoty for Hibernate Reactive.
But you can integrate Hibernate with Spring yourself and get reactive support.
Define a persistence.xml file, note the provider class must be specified as the one in Hibernate Reactive.
Declare a Mutiny.SessionFactory bean.
Then inject it in your repository class.
I have created a complete example demos Hibernate Reactive + Spring.
Update: Till now Spring team has no plan to support it, if you are willing to taste other framework, check Quarkus and Micronaunt, both have seamless Hibernate Reactive support. Check my Quarkus Hibernate Reactive example and Micronaut Hibernate Reactive example.
According to quote from previous answer
One could implement a ReactiveCrudRepository using JPA or JDBC and delegating the work to a thread pool. This would provide an async API on the outside, but would still consume the resources for the Threads and block between independent data accesses, so only a small part of the benefits of the reactive approach would get realized.
James Ward claims it can be non-blocking. I mean I asked him:
yeah ok, but isn't ScalikeJDBC-Async doing exactly the same? just putting query invocation into another thread pool?
and he replied
No because ScalalikeJDBC-Async uses https://github.com/mauricio... which is actually a non-blocking (NIO) JDBCish database driver.
source
So you can be reactive by replacing hibernate + spring data with postgresql-async (should work with mysql).
you could try with quarkus framework and panache mongo hibernate reactive repositories. https://quarkus.io/guides/mongodb-panache .It is easy manage a reactive repository over mongoDB, It is later but hope helps.
Im working with the Spring Framework 3.0.5 and the Hibernate Framework and Im starting to use now Springs Transactionmanagement. I have some questions, just to understand how Springs Transactionmanagement works.
1)
I read this things in the Spring reference:
a) Consistent programming model across different transaction APIs such as Java Transaction API (JTA), JDBC, Hibernate, Java Persistence API (JPA), and Java Data Objects (JDO).
b) Spring resolves the disadvantages of global and local transactions. It enables application developers to use a consistent programming model in any environment. You write your code once, and it can benefit from different transaction management strategies in different environments.
c) Gone are the days when the only alternative to using EJB CMT or JTA was to write code with local transactions such as those on JDBC connections, and face a hefty rework if you need that code to run within global, container-managed transactions. With the Spring Framework, only some of the bean definitions in your configuration file, rather than your code, need to change.
From a) I understand that I can use those APIs with Spring without changing the code
From b) I understand that I can use global or local transactions *without changing the code
From c) I understand that while switching between different APIs and global/local transactions I need to change the code
Now I wonder what is correct?
=> Do I need to change the code? When switching between different APIs? When switching between local and global transactions? (Or does it maybe depend on prorgammatic and declarative transaction management?)
2)
I also got an additional question: I really wonder what the use of programmatic transaction management is? Everywhere I read that declarative transactionmanagement is recommended
I read this in spring reference too:
d) With programmatic transaction management, developers work with the Spring Framework transaction abstraction, which can run over any underlying transaction infrastructure. With the preferred declarative model, developers typically write little or no code related to transaction management, and hence do not depend on the Spring Framework transaction API, or any other transaction API.
From d) I understand: with programmatic transaction management I can use any underlying transaction infrastructure... which means what? the different APIs mentioned above?
and: with declarative I do not depend on any api
=> isnt this the same? when I can use any underlying api, I do not depend on any api. I do not really understand this.
where is the difference? I only know that the declarative transaction management is more lightweight, that I have not to start the transaction by my self and catch the exception and handle it and so on. But what is the use of programmatic transaction management then?
Thank you for answering! :-)
You're over-thinking this a bit. The Spring API provides an abstract transaction model that has the same API and semantics regardless of which underlying transaction technology you use. In order to switch from one technology to another, you generally have to alter your Spring config, but the idea is that you never needs to to alter your business logic. So whether you're using local, in-VM JDBC transactions or fully distributed, two-phase-commit XA JPA-style transactions, the API usage within your Spring code is the same. Only the configuration changes.
The difference between declarative and programmatic transaction management is that with the former, you use annotations or XML config to say which bits of code are supposed to be transactional. With programmatic style, you specifically enclose transactional logic using method calls into the Spring API. Note that if you use the declarative style, then Spring will wrap your code in generated logic which uses the programmatic style. The latter is simply a more explicit and low-level version of the former. It gives you more control, but it's more verbose.