i want to copy many data from serveral db which on diff machine to a centre db.
i think the spring batch may be a choice to fit my requirement.
so. should be make a lot of job to accomplish the whole task, the jobs will like this:
job A: copy from db1 to db111;
job B: copy from db2 to db111;
job C: copy form db3 to db111;
etc...
and the tables in db1, db2, db3...is quite different.
so far, i know how to create datasources at spring boot startup time, but i don't know how to create datasource in job instance at runtime. is any idea about this? (if can support spring data jpa will be better)
or is any other way better then spring batch?
thanks.
A datasource is a set of connections to a DB so in your scenario , there are just multiple kinds of DBs or multiple DBs of same kind & for both scenarios - you will have to create one datasource for each db & then use that in piece of code wherever you need it.
Step 1 - So you write one configuration class for each database to set up one datasource. At propertly file level, you won't be able to use default properties but your custom ones where you prefix properties with db names to distinguish.
You need to define transaction managers etc for each datasource & you uniquely name each datasource.
Step 2 : Next step is to use appropriate datasource with appropriate dao classes. In above configuration class, if you use JPA , those configs would already be there including entity packages, repository packages etc etc. JdbcTemplate takes datasource in constructor etc.
All in All - scenario is similar to single datasource scenario & you will have to set up all datasources in advance at app start up but with appropriate qualified bean names & then use those data-sources wherever you need.
This Answer is what works for me
Related
For a multi tenant application I need to create I want to evaluate how convenient is Slick for creating queries against Postgres different schemas (not to confuse with schema tables).
I'm having a hard time finding how to configure TableQuery to use dynamically the schema provided by the user. TableQuery[Users].resul should return different datasets depending on me querying tenant A or tenant B.
Is it possible with current Slick versions?
TableQuery itself will not need to be configured, as its methods only return queries and actions. Actions are run by a DatabaseDef instance, and that is what will need to be configured to access different schemas/databases/etc. Slick official documentation describes a simple way to create an instance of a DatabaseDef, which by default uses the Typesage Config library:
val db = Database.forConfig("mydb")
where "mydb" specifies a key in a property file Typesafe Config is looking at. You can create and manipulate Config instances programmatically as well, and create db instances from those. I suspect you will have to do something along the lines of creating a new Config instance (there is the convenient withValue() method to copy a Config and replace a config value at the specified key) and use that to create a new db instance for each new schema you are interested in querying.
In our application, we need to load large CSV files and fetch some data out of it. For example, getting the distinct values from the CSV file. For this, we decided to go with in-memory DB's like H2, as there is no need to store the data in persistent storage.
However, the file is so dynamic that the columns may not be the same. I need to load the file to the H2 database to a table that is temporary for that session.
Tech Stack is Spring boot and H2.
The examples I see on forums is using a standard entity that knows what fields the table has. However my case the table columns will be dynamic
I tried the below in spring boot
public interface ImportCSVRepository extends JpaRepository<Object, String>
with
#Query(value = "CREATE TABLE TEST AS SELECT * FROM CSVREAD('test.csv');", nativeQuery = true)
But this gives unmanaged entity error. I understand why the error is thrown. However I am not sure how to achieve this. Also please clarify if I should use Spring-batch ?
You can use JdbcTemplate to manually create tables and query/update the data in them.
An example of how to create a table with JdbcTemplate
Dynamically creating tables and defining new entities (or modifying existing ones) is hardly possible with spring-data repositories and #Entity-ies. You probably should also check some NoSQL dbs like MongoDb - it's easier to define documents (or key-value objects - Redis) with dynamic structures in them.
In a Spring Boot Application project, I have 2 data sources:
a MySQL database (aka "db1")
a MongoDB database (aka "db2")
I'm using Spring Data JPA and Spring Data MongoDB, and it's working great... one at a time.
Saying db1 handles "Players", and db2 handles "Teams" (with a list of players' ID). Is it possible to make the relationship between those 2 heterogeneous entities working? (i.e. #ManyToOne, #Transactional, Lazy/Eager, etc.)
For example, I want to be able to write:
List<Player> fooPlayers = teamDao.findOneById(foo).getPlayers();
EDIT: If possible, I'd like to find a solution working with any spring data project
Unfortunately your conundrum has no solution in spring data.
what may be a possibility is that you create an interface (DAO) class of your own. That DAO class would have implementations to to query both of your DBs. A very crude and short example would be
your DAO
{
yourFind (id)
{
this would find in db2 and return a relevant list of objects
findOneByID(id)
get the player from the above retrieved list and query db1
getPlayer(player)
}
}
i hope this points you in the right direction
Is there any problem using both JDBC (JdbcTemplate) & JPA (EntityManager) in data access layer ?
I am planning to use JDBC to access the stored-procedures/routines.
These stored-procedures will be returning multiple cursors by joining multiple tables (which is not registered as JPA entities).
These JDBC actions are pure read-only.
I am not combining JPA & JDBC actions in the same transactions as given here
It is okay for me . Use the right tools for the job . For example , if I want to do some report queries which the data are spanned over many different entities or want to use some powerful database features (eg window function , common table expression ) which are not supported or difficult to be achieved by JPA , I would prefer to use JDBC to issue native SQL directly to get the job done.
The architecture CQRS also uses this idea which has two different separate models for updating information (command action) and reading information (query action) .For example , JPA can be used for command action while native JDBC is used for the query action.
I've got the problem, that I use a property in the persistence.xml which forces Hibernate to look only for tables in the given schema.
<property name="hibernate.default_schema" value="FOO"/>
Because we are using now 4 different schemas the actual solution is to generate 4 war files with a modified persistence.xml.
That not very elegant.
Does anybody know, how I can configure the schema with a property or by manipulation the JDBC connection string?
I'm using Oracle 10g, 10_2_3 Patch.
Thanks a lot.
You could create four different users on the oracle database for the four different applications, the JDBC connection would include the user.
The for the user, you can create synonyms and permissions for the tables.
E.g.
create or replace synonym USER1.tablename FOR SCHEMA1.tablename;
create or replace synonym USER2.tablename FOR SCHEMA1.tablename;
create or replace synonym USER3.tablename FOR SCHEMA2.tablename;
And when you are accessing the tables from hibernate, just leave the schema off. When logged in as USER1, it'll use SCHEMA1, etc.
That way you don't have to create four different WAR files with four different persistence.xml files. Just deploy the app four times with different jdbc connections.
Hope that helps.
If you don't want to generate four different WARs then put this property in a hibernate.properties file and put that file on the class path (but outside the webapp) for each webapp.
See this - https://www.hibernate.org/429.html
I created a method called deduceSchema that I run when I'm setting up the SessionFactory. It opens a jdbc connection using the data source (because you don't have a Hibernate session yet) and queries "select user from dual" to get the logged in user. This will be accurate if the user you log in as also owns the tables. If not, I use a jndi environment variable to override.
Once I have the schema, I modify the Hibernate configuration to set it for each table although this is only necessary if the logged in user is different than the schema:
for (Iterator iter = configuration.getTableMappings(); iter.hasNext();) {
Table table = (Table) iter.next();
table.setSchema(schema);
}