I've an application which stores entites in MariaDB. The entity has some relations to other entities and in one of the child entities is a binary attribute. See the following:
Parent
\ 1-n ChildA
- attrA1
- ...
\ 1-n ChildB
- attrB1
- binaryAttr
Storing the binary files in the DB has an impact on the DB size of course and thereby an impact on our backup concept. Besides this binary data doesn't necessarily have to be in the DB as well.
I'm thinking about combining MariaDB with an S3 compatible object store, so that the structured data is persisted in DB and the binary files in the object store.
We're are using Spring Boot (Data-JPA) with Apache Camel.
The naïve approach would be to store the binary file with a uuid-key in the object store and afterwards persisting the rest of the entity with the reference (uuid) to the DB.
But there's no easy transaction handling for this, nor a transparent handling of the entity (I've to handle persisting entity and binary data seperately).
Is there a java / spring based framework to overcome the drawbacks (transaction handling, transparent handling) of two different "datasources"?
What is the best practice to handle such a scenario or is it totally unrecommended?
Is there an extension mechanism for Spring / Hibernate to handle object store persistency within the hibernate orm mapping process?
Is it possible to implement one repository for persisting / loading entity and binary data at once?
Can anyone give me some direction?
I have a CRUD based application, which uses Spring Boot REST Services and JPA. For JPA we have POJO objects mapped to RBMS - PostgreSQL.
Some of my pages require data to be fetched using joins of multiple POJO objects. I just wanted to know what is a good architectural practice to do the same. Following are some of the options i have been informed of, but not sure what are the pros and cons of each especially for a large data volume application.
Use Transient Variables in POJOs and use JPA joins
Use additional Spring View Objects to combine POJOs
Write native/HQL to join tables/POJOs
Any insight would be helpful. If any more details required from me, would be glad to provide.
I think it's better to go with Entity Mappings.
This will enable you to easily fetch the parent and its nested entities using either JPA methods or using hibernate.
You can also specify the fetch type to actually control the behaviour of this fetch.
In case, you are looking for any complex joins or fetch patterns, Entity Graphs and HQL will be very useful.
When would you use Spring Data JPA over Spring Boot with JOOQ and vice versa?
I know that Spring Data JPA can be used for completing basic CRUD queries, but not really for complex join queries while using JOOQ makes it easier?
EDIT: Can you use both Spring data jpa with jooq?
There is no easy answer to your question. I have given a couple of talks on that topic. Sometimes there are good reasons to have both in a project.
Edit: IMHO Abstraction over the database in regards of dialects and datatypes is not the main point here!! jOOQ does a pretty good job to generate SQL for a given target dialect - and so does JPA / Hibernate. I would even say that jOOQ goes an extra mile to emulate functions for databases that don't have all the bells and whistles like Postgres or Oracle.
The question here is "Do I want to be able to express a query myself with everything SQL has to offer or am I happy with what JPA can express?"
Here's an example to run both together. I have a Spring Data JPA provided repository here with a custom extension (interface and implementation are necessary). I let the Spring context inject both the JPA EntityManager as well as the jOOQ context. I then use jOOQ to create queries and run them through JPA.
Why? Because expressing the query in question is not possible with JPA ("Give me the thing I listened the most" which is not the one having the highest number of count, but could be several).
The reason I run the query through JPA is simple: A downstream use case might require me to pass JPA entities to it. jOOQ can of course run this query itself and you could work on records or map the stuff anyway u like. But as you specifically asked about maybe using both technologies, I thought this is a good example:
import java.util.List;
import javax.persistence.EntityManager;
import javax.persistence.Query;
import org.jooq.DSLContext;
import org.jooq.Field;
import org.jooq.Record;
import org.jooq.SelectQuery;
import org.jooq.conf.ParamType;
import org.jooq.impl.DSL;
import org.springframework.data.repository.CrudRepository;
import static ac.simons.bootiful_databases.db.tables.Genres.GENRES;
import static ac.simons.bootiful_databases.db.tables.Plays.PLAYS;
import static ac.simons.bootiful_databases.db.tables.Tracks.TRACKS;
import static org.jooq.impl.DSL.count;
import static org.jooq.impl.DSL.rank;
import static org.jooq.impl.DSL.select;
public interface GenreRepository extends
CrudRepository<GenreEntity, Integer>, GenreRepositoryExt {
List<GenreEntity> findAllByOrderByName();
}
interface GenreRepositoryExt {
List<GenreWithPlaycount> findAllWithPlaycount();
List<GenreEntity> findWithHighestPlaycount();
}
class GenreRepositoryImpl implements GenreRepositoryExt {
private final EntityManager entityManager;
private final DSLContext create;
public GenreRepositoryImpl(EntityManager entityManager, DSLContext create) {
this.entityManager = entityManager;
this.create = create;
}
#Override
public List<GenreWithPlaycount> findAllWithPlaycount() {
final Field<Integer> cnt = count().as("cnt");
return this.create
.select(GENRES.GENRE, cnt)
.from(PLAYS)
.join(TRACKS).onKey()
.join(GENRES).onKey()
.groupBy(GENRES.GENRE)
.orderBy(cnt)
.fetchInto(GenreWithPlaycount.class);
}
#Override
public List<GenreEntity> findWithHighestPlaycount() {
/*
select id, genre
from (
select g.id, g.genre, rank() over (order by count(*) desc) rnk
from plays p
join tracks t on p.track_id = t.id
join genres g on t.genre_id = g.id
group by g.id, g.genre
) src
where src.rnk = 1;
*/
final SelectQuery<Record> sqlGenerator =
this.create.select()
.from(
select(
GENRES.ID, GENRES.GENRE,
rank().over().orderBy(count().desc()).as("rnk")
).from(PLAYS)
.join(TRACKS).onKey()
.join(GENRES).onKey()
.groupBy(GENRES.ID, GENRES.GENRE)
).where(DSL.field("rnk").eq(1)).getQuery();
// Retrieve sql with named parameter
final String sql = sqlGenerator.getSQL(ParamType.NAMED);
// and create actual hibernate query
final Query query = this.entityManager.createNativeQuery(sql, GenreEntity.class);
// fill in parameter
sqlGenerator.getParams().forEach((n, v) -> query.setParameter(n, v.getValue()));
// execute query
return query.getResultList();
}
}
I spoke about this a couple of times. There is no silver bullet in those tech, sometimes it's a very thin judgement:
The full talk is here: https://speakerdeck.com/michaelsimons/live-with-your-sql-fetish-and-choose-the-right-tool-for-the-job
As well as the recorded version of it: https://www.youtube.com/watch?v=NJ9ZJstVL9E
The full working example is here https://github.com/michael-simons/bootiful-databases.
IMHO if you want a performing and maintainable application which uses a database at its core, you don't want to abstract away the fact that you are using a database. JOOQ gives you full control because you can read and write the actual query in your code but with type safety.
JPA embraces the OO model and this simply does not match the way a database works in all cases, which could result in unexpected queries such as N+1 because you put the wrong annotation on a field. If you are not paying enough attention this will lead to performance issues when scaling your application. JPA Criteria helps a bit but it still way harder to write and read.
As a result, with JPA you are first writing your query in SQL and then use half a day to translate it to Criteria. After years of working with both frameworks I would use JOOQ even for simple a CRUD application (because there is no such thing as a simple CRUD application :-)).
Edit: I don't think that you can mix JPA with JOOQ, question is, why would you want to? They are both using a different approach so just choose one. It's difficult enough to learn the intricacies of one framework.
Spring Data JPA gives you the following:
An ORM layer, allowing you to treat database tables as if they were Java objects. It allows you to write code that is largely database-agnostic (you can use MySQL, Oracle, SQL Server, etc) and that avoids much of the error-prone code that you get when writing bare SQL.
The Unit of Work pattern. One reason why you see so many articles on C# explaining what a unit of work is, and practically none for Java, is because of JPA. Java has had this for 15 years; C#, well, you never know.
Domain-driven design repositories. DDD is an approach to object-oriented software that does away with the anaemic domain model you so often see in database-driven applications, with entity object only having data and accessor methods (anaemic model), and all business logic in service classes. There's more to it, but this is the most important bit that pertains to Spring Data JPA.
Integration into the Spring ecosystem, with inversion of control, dependency injection, etc.
jOOQ, on the other hand, is a database mapping library that implements the active record pattern. It takes an SQL-centric approach to database operations, and uses a domain-specific language for that purpose.
As happens so often, there is no one correct or superior choice. Spring Data JPA works very well if you don't care about your database. If you're happy not to do any complicated queries, then Spring Data JPA will be enough. However, once you need to do joins between tables, you notice that a Spring Data JPA repository really isn't a very good match for certain operations.
As #michael-simons mentioned, combining the two can sometimes be the best solution.
Here's an official explanation when JOOQ fits:
https://www.jooq.org/doc/latest/manual/getting-started/jooq-and-jpa/
Just because you're using jOOQ doesn't mean you have to use it for everything!
When introducing jOOQ into an existing application that uses JPA, the
common question is always: "Should we replace JPA by jOOQ?" and "How
do we proceed doing that?"
Beware that jOOQ is not a replacement for JPA. Think of jOOQ as a
complement. JPA (and ORMs in general) try to solve the object graph
persistence problem. In short, this problem is about
Loading an entity graph into client memory from a database
Manipulating that graph in the client Storing the modification back to
the database As the above graph gets more complex, a lot of tricky
questions arise like:
What's the optimal order of SQL DML operations for loading and storing
entities? How can we batch the commands more efficiently? How can we
keep the transaction footprint as low as possible without compromising
on ACID? How can we implement optimistic locking? jOOQ only has some
of the answers. While jOOQ does offer updatable records that help
running simple CRUD, a batch API, optimistic locking capabilities,
jOOQ mainly focuses on executing actual SQL statements.
SQL is the preferred language of database interaction, when any of the
following are given:
You run reports and analytics on large data sets directly in the
database You import / export data using ETL You run complex business
logic as SQL queries Whenever SQL is a good fit, jOOQ is a good fit.
Whenever you're operating and persisting the object graph, JPA is a
good fit.
And sometimes, it's best to combine both
Spring Data JPA does support #Query idiom with the ability to run native queries (by setting nativeQuery flag) where we can write & see the query (simple & complex, with joins or otherwise) right there with the repository & reuse them easily.
Given the above,
When would you use Spring Data JPA over Spring Boot with JOOQ and vice versa?
I would simply use Spring Data JPA unless i am not using the Spring ecosystem itself. Other reason might be that i prefer the fluent style..
I know that Spring Data JPA can be used for completing basic CRUD queries, but not really for complex join queries
As i noted above, Spring Data JPA does provide the ability to use complex and/or join queries. In addition via the custom repository mechanism (example already in #Michael Simons post above that uses JOOQ) provides even more flexibility. So its a full fledged solution by itself.
Can you use both Spring data jpa with jooq?
Already answered wonderfully by #Michael Simons above.
In our application, we need to load large CSV files and fetch some data out of it. For example, getting the distinct values from the CSV file. For this, we decided to go with in-memory DB's like H2, as there is no need to store the data in persistent storage.
However, the file is so dynamic that the columns may not be the same. I need to load the file to the H2 database to a table that is temporary for that session.
Tech Stack is Spring boot and H2.
The examples I see on forums is using a standard entity that knows what fields the table has. However my case the table columns will be dynamic
I tried the below in spring boot
public interface ImportCSVRepository extends JpaRepository<Object, String>
with
#Query(value = "CREATE TABLE TEST AS SELECT * FROM CSVREAD('test.csv');", nativeQuery = true)
But this gives unmanaged entity error. I understand why the error is thrown. However I am not sure how to achieve this. Also please clarify if I should use Spring-batch ?
You can use JdbcTemplate to manually create tables and query/update the data in them.
An example of how to create a table with JdbcTemplate
Dynamically creating tables and defining new entities (or modifying existing ones) is hardly possible with spring-data repositories and #Entity-ies. You probably should also check some NoSQL dbs like MongoDb - it's easier to define documents (or key-value objects - Redis) with dynamic structures in them.
On my domain I have the usual entities (User, Company, etc) and also "entities" that doesn't change, I mean they are fixed values but stored on data base. My backend is Mongo so I make use of MongoRepository. I'm also using Spring Data Rest.
Let's say I have defined Sector as entity, which is nothing more than a String wrapped on a Java object.
So this is how I define the repository.
#RepositoryRestResource
public interface SectorRepo extends MongoRepository<Sector,String>{
}
The thing is that this seems to be inappropriate, as I should not define an object that only wraps an string and treat it as an entity, it isn't. The only purpose for Sector collection is to be loaded on a combo box, nothing more.
The problem gets serious when you have more and more of these non-entities objects.
How I should approach this situation so I can still use MongoRepository + Spring Data Rest?
This is similar to couple of other questions. Please see my answers for both. Hope it helps
Spring Data MongoDB eliminate POJO's
Storing a JSON schema in mongodb with spring