The official tutorial of dbunit already give a good example for exporting dataset from a single database schema.
Is there any way to export different tables from different schemas into one single dataset (say Table_A from schema_A, Table_B from schema_B)?
The exported dataset, when written into an xml file, would be like this:
<?xml version='1.0' encoding='UTF-8'?>
<dataset schema:schemaA schema:schemaB>
<schemaA:tableA ..... />
<schemaA:tableA ..... />
<schemaB:tableB ..... />
</dataset>
I've just got the same problem and to fix it you need to set the FEATURE_QUALIFIED_TABLE_NAMES properties:
See below the same example code with the change (I removed some part of the code because I don't need full database export):
public static void main(String[] args) throws Exception
{
// database connection
Class driverClass = Class.forName("com.microsoft.sqlserver.jdbc.SQLServerDriver");
Connection jdbcConnection = DriverManager.getConnection(
"jdbc:sqlserver://<server>:1433;DatabaseName=<dbName>", "<usr>", "<passwd>");
IDatabaseConnection connection = new DatabaseConnection(jdbcConnection);
Properties props = new Properties();
props.put(DatabaseConfig.FEATURE_QUALIFIED_TABLE_NAMES, "true");
connection.getConfig().setPropertiesByString(props);
// dependent tables database export: export table X and all tables that
// have a PK which is a FK on X, in the right order for insertion
String[] depTableNames = TablesDependencyHelper.getAllDependentTables( connection, "vehicle.Vehicle_Series_Model_SMA" );
IDataSet depDataset = connection.createDataSet( depTableNames );
FlatXmlDataSet.write(depDataset, new FileOutputStream("vehicle.Vehicle_Series_Model_SMA.xml"));
}
Related
Apache beam provides the JDBCIO connector to connect to CloudSql postgreSQL. My job reads an event from pub/sub. The event body is as below:
tableName,
list<value>
I need to write to the table based on the table name that I get in from my message.
The JDBCIO has prepared statement which will let me parameterize the values in my insert query. But I need to generate the insert query dynamically based on the information present in the event.
pipeline
.apply(PubsubIO.readStrings().fromSubscription())
.apply(convertToKV())
.apply(JdbcIO.<List<String>>>write()
.withDataSourceConfiguration(JdbcIO.DataSourceConfiguration.create(
"com.mysql.jdbc.Driver", "jdbc:mysql://hostname:3306/mydb")
.withUsername("username")
.withPassword("password"))
.withStatement("insert into Person values(?, ?)")
.withPreparedStatementSetter(new JdbcIO.PreparedStatementSetter<KV<Integer, String>>() {
public void setParameters(KV<Integer, String> element, PreparedStatement query)
throws SQLException {
i=0
for each element in list
query.setInt(i, element.get(i);
i++;
}
})
);
I should be able to create the SQL statement dynamically based on the input event from the pcollection.
My select statement should be dynamically generated based on the list value and the table name. Please let me know whether we can do this or not.
Update:-
im trying to manually call the jdbc driver inside the parDo function but getting the below error.
No suitable driver found for jdbcURL.
Please let me know if im missing anyting:
#Setup
public void doAnyRequiredSetup() throws SQLException
{
LoggingContextUtil.installContext(loggingContext);
connection=DriverManager.getConnection(JdbcUrl,user,password);
statement=connection.createStatement();
if (LOGGER.isDebugEnabled()) {
LOGGER.debug("In doAnyRequiredSetup logging Context is now set and JDBC connection is .");
}
}
#SuppressWarnings("unchecked")
#ProcessElement
public void processElement(ProcessContext context)
{
JsonNode element=context.element();
try {
String query=formatQuery(baseQuery);
boolean result=statement.execute(query);
if(LOGGER.isDebugEnabled()) {
LOGGER.debug("Executed query : "+query+" and the result is "+ result);
}
} catch (IllegalArgumentException | SQLException e) {
ErrorMessage em = new ErrorMessage(element.toString(), "Insert Query Failed", e.getMessage());
context.output(ValidateTagHelper.FAILURE_TAG,em);
}
}
You can not have dynamic queries on JdbcIO based on the input elements.
The ParDo has to reset as you like, you can rewrite your ParDo in which you would call the JDBC driver manually.
If find this other workaround, you can split the input PColleciton into multiple outputs. That will work if your use case is limited to some predefined set of queries that you can choose from based on the input. This way you split the input into multiple PCollections and then attach differently configured IOs to each.
https://cloud.google.com/blog/products/gcp/guide-to-common-cloud-dataflow-use-case-patterns-part-1
You can try to read pubsub messages with attributes and in attributes, you can pass the table name and values in the form of Key-value pair.
PCollection<PubsubMessage> pubsubMessage = pipeline
.apply(PubsubIO.readMessagesWithAttributes().fromSubscription("")
This issue occurred in jdbc batch insert. I queried from an Oracle datasource, parsed the resultset and then inserted into another Oracle datasource. I have got the connect metadata and printed the current username along with url, both are invalid.
But when it went to batch update, I got the ora-00942 exception. I'm pretty sure all above works fine in database. Has anyone encountered this exception and can you give me some advice?
EDIT:
Ok, I got a table named photos for example in REMOTE_USER and I queried from it. It gave me a resultset, then I parse it after that INSERT it to LOCAL_USER.photos. I did query the LOCAL_USER.photos where I logon in from PL/SQL Developer. The interesting thing was I could do the select command but not the insert. Below is some part of code.
conn = datasource.getConnection(); // notice that it was target datasource
DatabaseMetaData connMetaData = conn.getMetaData();
String userName = connMetaData.getUserName();
resultSet = ds.getResultSet();
ResultSetMetaData metaData = resultSet.getMetaData();
int count = metaData.getColumnCount();
String insertSql = generateInsertSql(count, metaData, userName);
// this was generated through metaData , the output should be
// "insert into LOCAL_USER.photos(col1,col2) values(?,...)"
logger.error("insert clause is {}", insertSql);
ps = conn.prepareStatement(insertSql);
conn.setAutoCommit(false);
while (resultSet.next()) { // this was the original datasource
stageTotalNum++;
for (int i = 1; i <= count; i++) {
Object object = resultSet.getObject(i);
dealClobColumn(ps, i, object);
}
ps.addBatch();
if (stageTotalNum % 500L == 0L) {
ps.executeBatch(); // throws batchupdateexception.
ps.clearBatch();
conn.commit();
}
}
ps.executeBatch();
conn.commit();
It should be the blob type column which I didn't handle it the right way.
First I queried from original datasource then got the blob column of the resultset by
conn.getObject(index) . Next I insert the blob column into target datasource by conn.setObject. Of course that way wasn't working at all, so I changed to the following:
conn.setBlob(rs.getBlob(index)).
Although it worked fine in my own environemnt, but when the application ran in remote server, it kept annoying about the 'table or view does not exists'.The third version is:
conn.setBinaryStream(rs.getBlob(index).getBinaryStream());
Ok, this time it worked both my pc and remote server. Thanks to #codeLover's advice and link, it really hepled me and saved my time. Appreciated it!
I'm using Model-First approach and Oracle database.
Update2: Fixed Now
On including seed data, I'm getting this error "DeleteDatabase is not supported by the provider"
UPDATE1 If I change seed data type from
public class MySeedData : DropCreateDatabaseAlways<ToolContext> to
public class MySeedData : DropCreateDatabaseIfModelChanges<ToolContext>
this error is replaced by another error:
Exception Model compatibility cannot be checked because the DbContext instance was not created using Code First patterns. DbContext instances created from an ObjectContext or using an EDMX file cannot be checked for compatibility
Seed Data
public class MySeedData : DropCreateDatabaseAlways<ToolContext>
{
protected override void Seed(ToolContext context)
{
base.Seed(context);
var category = new List<CategoryValue>
{
new CategoryValue{Id=1, Name = "Associate"},
new CategoryValue{Id =2, Name = "Professional"},
new CategoryValue{Id=3, Name = "Master"},
new CategoryValue{Id = 4, Name = "Product"},
new CategoryValue{Id = 5, Name = "Portfolio"}
};
category.ForEach(cert => context.CategoryValues.Add(cert));
context.SaveChanges();
}
}
Web.Config
<connectionStrings>
<add name="LMSPriorToolContext"
connectionString="metadata=res://*/Models.LMSPriorToolModel.csdl|res://*/Models.LMSPriorToolModel.ssdl|res://*/Models.LMSPriorToolModel.msl;
provider=Oracle.DataAccess.Client;
provider connection string="DATA SOURCE=DEV;PASSWORD=1234;PERSIST SECURITY INFO=True;USER ID=abc""
providerName="System.Data.EntityClient" />
</connectionStrings>
Application_Start()
Database.SetInitializer<ToolContext>(new SeedData());
Main.cs: EXCEPTION RAISED IN THIS FILE I know when you first try to access database, seed data method or scripts are executed.
using (var dbContext = new ToolContext())
{
var items = dbContext.CategoryValues;
foreach(CategoryValue category in **items**) // **Getting error here**
{
Console.WriteLine(category.Name);
}
}
REFERENCES Seems like I'm missing something here as there is nothing related to Oracle or ODAC
Stack Trace
at System.Data.Common.DbProviderServices.DbDeleteDatabase(DbConnection connection, Nullable`1 commandTimeout, StoreItemCollection storeItemCollection)
at System.Data.Objects.ObjectContext.DeleteDatabase()
at System.Data.Entity.Internal.DatabaseOperations.DeleteIfExists(ObjectContext objectContext)
at System.Data.Entity.Database.Delete()
at System.Data.Entity.DropCreateDatabaseAlways`1.InitializeDatabase(TContext context)
at System.Data.Entity.Database.<>c__DisplayClass2`1.<SetInitializerInternal>b__0(DbContext c)
at System.Data.Entity.Internal.InternalContext.<>c__DisplayClass8.<PerformDatabaseInitialization>b__6()
at System.Data.Entity.Internal.InternalContext.PerformInitializationAction(Action action)
Please suggest what I'm missing here.
Way of seeding data using DropCreateDatabaseAlways or DropCreateDatabaseIfModelChanges is not supported in Model-First approach.
Change seed data class to:
public class ToolSeedData : IDatabaseInitializer<ToolContext>
{
public void InitializeDatabase(ToolContext context)
{
var category = new List<CategoryValue>
{
new CategoryValue{Id=1, Name = "Associate"},
new CategoryValue{Id =2, Name = "Professional"},
new CategoryValue{Id=3, Name = "Master"},
new CategoryValue{Id = 4, Name = "Product"},
new CategoryValue{Id = 5, Name = "Portfolio"}
};
category.ForEach(cert => context.CategoryValues.Add(cert));
context.SaveChanges();
}
Possible error if you don't use it:
DeleteDatabase is not supported by the provider
Exception Model compatibility cannot be checked because the DbContext instance was not created using Code First patterns.
DbContext instances created from an ObjectContext or using an EDMX
file cannot be checked for compatibility
Microsoft link Seeding database does not work
Hope this helps someone else.
I have to run a jdbc update query using preparedstatement/statement without knowing the column type.
I have a query say ' update table set status=? where id=? '
and i am getting a map of values like {("status"="123"), ("id"="546")}
Now I don't know the column type, is there any generic way to run this query using jdbc?
Instead of running - ps.setString(1,map.get(("status"));
beacause i don't know the column type of status field in DB (it may be a int also)
Please help me in solving this without using spring jdbc templates.
ps.setString(1,map.get(("status")) will work for integer also, you just has to take care that value you are putting in integer column is of int type.
Following code explains that:
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.SQLException;
public class SOJDBC
{
public static void main(String rgs[])
{
Connection con = DBConnection.getConnection("TEMP");
try
{
PreparedStatement pstmt = con.prepareStatement("insert into STUDENT(ROLLNO,NAME,AGE) values(?,?,?)");
pstmt.setString(1, "1"); //column type is integer, will work because the value is of int type
pstmt.setString(2, "Bhushan");
pstmt.setString(3, "25"); //column type is integer, will work because the value is of int type
pstmt.executeUpdate();
}
catch (SQLException e)
{
e.printStackTrace();
}
}
}
Is there any way I can get resultset object from one of jdbctemplate query methods?
I have a code like
List<ResultSet> rsList = template.query(finalQuery, new RowMapper<ResultSet>() {
public ResultSet mapRow(ResultSet rs, int rowNum) throws SQLException {
return rs;
}
}
);
I wanted to execute my sql statement stored in finalQuery String and get the resultset. The query is a complex join on 6 to 7 tables and I am select 4-5 columns from each table and wanted to get the metadata of those columns to transform data types and data to downstream systems.
If it is a simple query and I am fetching form only one table I can use RowMapper#mapRow and inside that maprow method i can call ResultsetExtractor.extractData to get list of results; but in this case I have complex joins in my query and I am trying to get resultset Object and from that resultset metadata...
The above code is not good because for each result it will return same resultset object and I dont want to store them in list ...
Once more thing is if maprow is called for each result from my query will JDBCTemplate close the rs and connection even though my list has reference to RS object?
Is there any simple method like jdbcTemplate.queryForResultSet(sql) ?
Now I have implemented my own ResultSet Extractor to process and insert data into downstream systems
sourceJdbcTemplate.query(finalQuery, new CustomResultSetProcessor(targetTable, targetJdbcTemplate));
This CustomResultSetProcessor implements ResultSetExtractor and in extractData method I am calling 3 different methods 1 is get ColumnTypes form rs.getMetaData() and second is getColumnTypes of target metadata by running
SELECT NAME, COLTYPE, TBNAME FROM SYSIBM.SYSCOLUMNS WHERE TBNAME ='TABLENAME' AND TABCREATOR='TABLE CREATOR'
and in 3rd method I am building the insert statement (prepared) form target columntypes and finally calling that using
new BatchPreparedStatementSetter()
{
#Override
public void setValues(PreparedStatement insertStmt, int i) throws SQLException{} }
Hope this helps to others...
Note that the whole point of Spring JDBC Template is that it automatically closes all resources, including ResultSet, after execution of callback method. Therefore it would be better to extract necessary data inside a callback method and allow Spring to close the ResultSet after it.
If result of data extraction is not a List, you can use ResultSetExtractor instead of RowMapper:
SomeComplexResult r = template.query(finalQuery,
new ResultSetExtractor<SomeComplexResult>() {
public SomeResult extractData(ResultSet) {
// do complex processing of ResultSet and return its result as SomeComplexResult
}
});
Something like this would also work:
Connection con = DataSourceUtils.getConnection(dataSource); // your datasource
Statement s = con.createStatement();
ResultSet rs = s.executeQuery(query); // your query
ResultSetMetaData rsmd = rs.getMetaData();
Although I agree with #axtavt that ResultSetExtractor is preferred in Spring environment, it does force you to execute the query.
The code below does not require you to do so, so that the client code is not required to provide the actual arguments for the query parameters:
public SomeResult getMetadata(String querySql) throws SQLException {
Assert.hasText(querySql);
DataSource ds = jdbcTemplate.getDataSource();
Connection con = null;
PreparedStatement ps = null;
try {
con = DataSourceUtils.getConnection(ds);
ps = con.prepareStatement(querySql);
ResultSetMetaData md = ps.getMetaData(); //<-- the query is compiled, but not executed
return processMetadata(md);
} finally {
JdbcUtils.closeStatement(ps);
DataSourceUtils.releaseConnection(con, ds);
}
}