Run SQL script on JDBC connection, minimal approach - jdbc

Long story short: I want to run a SQL script on an HSQLDB database.
I want to follow a minimalistic approach, which means:
Absolutely no manual parsing of SQL
No additional dependencies except for general Utilities. I make the distinction here because, for example I refuse to pull in Ibatis or Hibernate which are larger scope frameworks, but I will accept an apache commons or guava type utils library.
The library MUST BE AVAILABLE ON MAVEN. No small-time pet-project stuff.
(EDIT 12/5/15) Must have the ability to execute SQL file from classpath.
To give you some context:
try {
connection = DriverManager.getConnection("jdbc:hsqldb:file:mydb", "sa", "");
// Run script here
} catch (SQLException e) {
throw new RuntimeException("Unable to load database", e);
}
A one-liner would be great. Something like:
FancyUtils.runScript(connection, new File("myFile.sql"));
I did find org.hsqldb.persist.ScriptRunner but it takes a Database object as an argument and I can't seem to figure out how to get an instance. Also, I don't like the description of "Restores the state of a Database", so does that mean my database will be cleared first? That's definitely not what I want.

I just tried using the SqlFile object in SqlTool and it worked for me. The Maven dependency I used was
<dependency>
<groupId>org.hsqldb</groupId>
<artifactId>sqltool</artifactId>
<version>2.4.1</version>
</dependency>
The SQL script file I wanted to execute was "C:/Users/Public/test/hsqldbCommands.sql":
INSERT INTO table1 (id, textcol) VALUES (2, 'stuff');
INSERT INTO table1 (id, textcol) VALUES (3, 'more stuff');
and my Java test code was
package hsqldbMaven;
import java.io.File;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import org.hsqldb.cmdline.SqlFile;
public class HsqldbMavenMain {
public static void main(String[] args) {
String connUrl = "jdbc:hsqldb:file:C:/Users/Public/test/hsqldb/personal";
String username = "SA";
String password = "";
try (Connection conn = DriverManager.getConnection(connUrl, username, password)) {
// clear out previous test data
try (Statement st = conn.createStatement()) {
st.executeUpdate("DELETE FROM table1 WHERE ID > 1");
}
System.out.println("Before:");
dumpTable(conn);
// execute the commands in the .sql file
SqlFile sf = new SqlFile(new File("C:/Users/Public/test/hsqldbCommands.sql"));
sf.setConnection(conn);
sf.execute();
System.out.println();
System.out.println("After:");
dumpTable(conn);
try (Statement st = conn.createStatement()) {
st.execute("SHUTDOWN");
}
} catch (Exception e) {
e.printStackTrace(System.err);
}
}
private static void dumpTable(Connection conn) throws SQLException {
try (
Statement st = conn.createStatement();
ResultSet rs = st.executeQuery("SELECT id, textcol FROM table1")) {
while (rs.next()) {
System.out.printf("%d - %s%n", rs.getInt("id"), rs.getString("textcol"));
}
}
}
}
producing
Before:
1 - Hello world!
After:
1 - Hello world!
2 - stuff
3 - more stuff
Edit: 2018-08-26
If you want to bundle your SQL script file into the project as a resource then see the example in the other answer.
Note also that this approach is not restricted to HSQLDB databases. It can be used for other databases as well (e.g., MySQL, SQL Server).

This uses the SqlTool library, but reads the script directly from the classpath by using the SqlFile class:
try(InputStream inputStream = getClass().getResourceAsStream("/script.sql")) {
SqlFile sqlFile = new SqlFile(new InputStreamReader(inputStream), "init", System.out, "UTF-8", false, new File("."));
sqlFile.setConnection(connection);
sqlFile.execute();
}

Even though iBatis was mentioned by the OP as a non-requirement, I still want to recommend MyBatis - the iBatis fork by the original creators.
The core library (org.mybatis:mybatis) requires no dependencies (all of its dependencies are optional) and while larger than HSQLDB SqlTool, at 1.7MB binary it is not horribly big for most uses and is continuously maintained (the last release, 3.5, was last month as of this writing).
You can initialize ScriptRunner with a JDBC Connection, then call runScript(new InputStreamReader(sqlinputst, Standard chartered.UTF_8)) to run whatever SQL script you can get an input steam of.

Related

Package org.apache.flink.api.java.io.jdbc does not exist

I want to use the JDBC connector in an Apache Flink application. But maven doesn't find the flink JDBC package.
I added the following dependency to my pom.xml in the "build-jar" section:
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-jdbc_2.11</artifactId>
<version>1.13.1</version>
</dependency>
The jar files were downloaded by maven and are available in the local maven directory.
My code looks like this.
// standard, not relevant flink imports
import org.apache.flink.api.java.io.jdbc.JDBCInputFormat;
import org.apache.flink.api.java.io.jdbc.JDBCOutputFormat;
public class BatchLayerExec {
public static void main( final String[] args ) {
//Definition of Strings for the connection to the database
try {
ExecutionEnvironment environment = ExecutionEnvironment.getExecutionEnvironment();
final TypeInformation<?>[] fieldTypes =
new TypeInformation<?>[] { ... };
final RowTypeInfo rowTypeInfo = new RowTypeInfo(fieldTypes);
//Define Input Format Builder
JDBCInputFormat.JDBCInputFormatBuilder inputBuilder = JDBCInputFormat
.buildJDBCInputFormat()
.setDrivername(driverName)
.setDBUrl(dbURL + sourceDB)
.setQuery(selectQuery)
.setRowTypeInfo(rowTypeInfo)
.setUsername(dbUser)
.setPassword(dbPassword)
.setRowTypeInfo(rowTypeInfo);
DataSet<Row> sourceTable = environment.createInput(inputBuilder.finish());
// Transformation
// ...
// Print for debugging
transformedTable.print();
// Output transformed data to output table
//Define Output Format Builder
JDBCOutputFormat.JDBCOutputFormatBuilder outputBuilder = JDBCOutputFormat
.buildJDBCOutputFormat()
.setDrivername(driverName)
.setDBUrl(dbURL + sourceDB)
.setQuery(insertQuery)
.setSqlTypes(new int[] { ... })
.setUsername(dbUser)
.setPassword(dbPassword);
//Define dataSink
transformedTable.output(outputBuilder.finish());
environment.execute();
} catch(final Exception e) {
System.out.println(e);
}
}
}
But during the build process with mvn clean package -Pbuild-jar, I get the error message:
package org.apache.flink.api.java.io.jdbc does not exist.
I removed some not relevant definitions and steps in the code (see comments). Please comment if you need more information.
I found out that the package org.apache.flink.api.java.io.jdbc is deprecated.
Importing the package org.apache.flink.connector.jdbc works.
EDIT
Note that this requires changing the JDBCInputFormat and JDBCOutputFormat classes to JdbcInputFormat and JdbcOutputFormat.

Benchmarking spring data vs JDBI in select from postgres Database

I wanted to compare the performence for Spring data vs JDBI
I used the following versions
Spring Boot 2.2.4.RELEASE
vs
JDBI 3.13.0
the test is fairly simple select * from admin table and convert to a list of Admin object
here is the relevant details
with spring boot
public interface AdminService extends JpaRepository<Admin, Integer> {
}
and for JDBI
public List<Admin> getAdmins() {
String sql = "Select admin_id as adminId, username from admins";
Handle handle = null;
try {
handle = Sql2oConnection.getInstance().getJdbi().open();
return handle.createQuery(sql).mapToBean(Admin.class).list();
}catch(Exception ex) {
log.error("Could not select admins from admins: {}", ex.getMessage(), ex );
return null;
} finally {
handle.close();
}
}
the test class is executed using junit 5
#Test
#DisplayName("How long does it take to run 1000 queries")
public void loadAdminTable() {
System.out.println("Running load test");
Instant start = Instant.now();
for(int i= 0;i<1000;i++) {
adminService.getAdmins(); // for spring its findAll()
for(Admin admin: admins) {
if(admin.getAdminId() == 654) {
System.out.println("just to simulate work with the data");
}
}
}
Instant end = Instant.now();
Duration duration = Duration.between(start, end);
System.out.println("Total duration: " + duration.getSeconds());
}
i was quite shocked to get the following results
Spring Data: 2 seconds
JDBI: 59 seconds
any idea why i got these results? i was expecting JDBI to be faster
The issue was that spring manages the connection life cycle for us and for a good reason
after reading the docs of JDBI
There is a performance penalty every time a connection is allocated
and released. In the example above, the two insertFullContact
operations take separate Connection objects from your database
connection pool.
i changed the test code of the JDBI test to the following
#Test
#DisplayName("How long does it take to run 1000 queries")
public void loadAdminTable() {
System.out.println("Running load test");
String sql = "Select admin_id as adminId, username from admins";
Handle handle = null;
handle = Sql2oConnection.getInstance().getJdbi().open();
Instant start = Instant.now();
for(int i= 0;i<1000;i++) {
List<Admin> admins = handle.createQuery(sql).mapToBean(Admin.class).list();
if(!admins.isEmpty()) {
for(Admin admin: admins) {
System.out.println(admin.getUsername());
}
}
}
handle.close();
Instant end = Instant.now();
Duration duration = Duration.between(start, end);
System.out.println("Total duration: " + duration.getSeconds());
}
this way the connection is opened once and the query runs 1000 times
the final result was 1 second
twice as fast as spring
On the one hand you seem to make some basic mistakes of benchmarking:
You are not warming up the JVM.
You are not using the results in any way.
Therefore what you are seeing might just be effects of different optimisations of the VM.
Look into JMH in order to improve your benchmarks.
Benchmarks with an external resource are extra hard, because you have so many more parameters to control.
One big question is for example if the connection to the database is realistically slow as in most production systems the database will be on a different machine at least virtually, quite possibly on different hardware.
Is that true in your test as well?
Assuming your results are real, the next step is to investigate where the extra time gets spent.
I would expect the most time to be spent with executing the SQL statements and obtaining the result via the network.
Therefore you should inspect what SQL statements actually get executed.
This might point you to one possible answer that JPA is doing lots of lazy loading and hasn't even loaded most of you really need.

Jfreechart and jdbc.CategoryDataset

My goal is to write a small program that reaches into our Public Safety database and retrieves ambulance response times. Then I want to take that data and generate a histogram.
I am having a very difficult time getting JFreechart to print a histogram of ambulance response times. When I run my code Netbeans generates an error message stating:
Exception in thread "main" java.lang.ClassCastException: org.jfree.data.jdbc.JDBCCategoryDataset cannot be cast to org.jfree.data.xy.IntervalXYDataset
at EMSResearch.histogram1.main(histogram1.java:39)
C:\Users\kulpandm\AppData\Local\NetBeans\Cache\8.1\executor-snippets\run.xml:53: Java returned: 1
BUILD FAILED (total time: 7 seconds)
I have absolutely no idea how to solve this issue. If I can get a histogram to print out I can then move on and hack away at making it look good for my reports. Shown below is my code:
package EMSResearch;
import java.io.File;
import java.io.IOException;
import java.sql.*;
import java.util.logging.Level;
import java.util.logging.Logger;
import org.jfree.chart.ChartFactory;
import org.jfree.chart.ChartUtilities;
import org.jfree.chart.JFreeChart;
import org.jfree.chart.plot.PlotOrientation;
import org.jfree.data.jdbc.JDBCCategoryDataset;
import org.jfree.data.xy.IntervalXYDataset;
public class histogram1
{
public static void main(String[] args) throws ClassNotFoundException
{
try
{
Connection conn = null;
conn = DriverManager.getConnection("jdbc:sqlserver://MyURL;database=MyDatabase;integratedsecurity=false;user=MyUser;password=MyPassword");
//dataset = new JDBCCategoryDataset(conn);
String query = "SELECT PUN_UnitID\n"
+ ", DATEDIFF(SECOND, IIU_tenroute, IIU_tArrive) AS [ResponseTime]\n"
+ "FROM IIncidentUnitSummary\n"
+ "WHERE IIU_tDispatch > 'may 15, 2016'\n"
+ "AND PUN_UnitID LIKE 'M[0-4]_'\n"
+ "AND IIU_tEnroute IS NOT NULL\n"
+ "AND IIU_tArrive IS NOT NULL\n"
+ "AND DATEDIFF(SECOND, IIU_tEnroute, IIU_tArrive) BETWEEN 0 AND 1800";
JDBCCategoryDataset dataset = new JDBCCategoryDataset(conn, query);
dataset.executeQuery(query);
conn.close();
//the following blows up.
JFreeChart chart;
chart = ChartFactory.createHistogram("data", "ws range", "freq", (IntervalXYDataset) dataset, PlotOrientation.VERTICAL, true, false, false);
int width = 560;
int height = 370;
File histChart = new File("c:\\histChart2.jpeg");
try
{
ChartUtilities.saveChartAsJPEG(histChart, chart, width, height);
} catch (IOException ex)
{
Logger.getLogger(chart2.class.getName()).log(Level.SEVERE, null, ex);
}
} catch (SQLException ex)
{
ex.printStackTrace();
}
}
}
Can you please show me what I am doing incorrectly?
What can I do to fix this? (please be specific since I am not an expert programmer).
A CategoryDataset is not an IntervalXYDataset; they provide completely different things to clients.
A CategoryDataset can be used to create a bar chart. Invoking ChartFactory.createBarChart() using your JDBCCategoryDataset should at least give you something to see. Refer to this related example.
The API suggests "writing your own code to read data from a ResultSet." I'd use the data to populate a HistogramDataset, seen here. Because HistogramDataset implements IntervalXYDataset, it is suitable for use with ChartFactory.createHistogram(). Where the example has one series for each primary color, it looks like you might want one series for each PUN_UnitID.
Conceptually, it may be easier to SELECT DISTINCT PUN_UnitID that meet your LIKE clause in an initial query. Then query the response time values for each PUN_UnitID and invoke dataset.addSeries(PUN_UnitID, values, bins).
Alternatively, loop through the ResultSet, adding entries to a Map<String, List<Integer>>, where the key is the PUN_UnitID the value is aList of response times. Use the Map entries to construct the HistogramDataset as before.
If latency becomes a problem, update the dataset in using a SwingWorker, as shown here.

Application not loading data from hibernate

Our application has 2 servers A and B and requests are managed by a load balancer.
Code is same inside two weblogic servers but
when same page is loaded from one server it is getting displayed
but same page loaded from second server its giving
Error 500--Internal Server Error
war file is same in both weblogic servers but when when I check logs I can see that some exception is observed.
org.hibernate.HibernateException: Problem while trying to load or access OracleTypes.CURSOR value
at org.hibernate.dialect.Oracle8iDialect.registerResultSetOutParameter(Oracle8iDialect.java:399)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1586)
at org.hibernate.loader.Loader.doQuery(Loader.java:696)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
at org.hibernate.loader.Loader.doList(Loader.java:2228)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2125)
at org.hibernate.loader.Loader.list(Loader.java:2120)
at org.hibernate.loader.custom.CustomLoader.list(CustomLoader.java:312)
at org.hibernate.impl.SessionImpl.listCustomQuery(SessionImpl.java:1722)
at org.hibernate.impl.AbstractSessionImpl.list(AbstractSessionImpl.java:165)
at org.hibernate.impl.SQLQueryImpl.list(SQLQueryImpl.java:175)
so I directly went through the dialect code in Oracle8iDialect.java inside the hibernate-3.2.7.ga jar file.
Hibernate uses the following code to load the ORACLE TYPES class.
public int registerResultSetOutParameter(CallableStatement statement, int col) throws SQLException {
// register the type of the out param - an Oracle specific type
statement.registerOutParameter( col, getOracleCursorTypeSqlType() );
col++;
return col;
}
So there is no code description inside the Oracle8iDialect.java that throws the above Exception “Problem while trying to load or access OracleTypes.CURSOR value” so I investigated that there is one more class with the same name Oracle8iDialect inside the z_easybeans-uberjar-hibernate-1.1.0-M3-JONAS.jar file. I think that the same class is conflicted between the two jar files by the class loader. So at runtime web logic pick up the oracle8idialect class file in z_easybeans-uberjar-hibernate-1.1.0-M3-JONAS.jar instead of correct class in hibernate-3.2.7.ga jar.
dialect code in Oracle8iDialect.java inside the z_easybeans-uberjar-hibernate-1.1.0-M3-JONAS.jar.
public int registerResultSetOutParameter(java.sql.CallableStatement statement,int col) throws SQLException {
if(oracletypes_cursor_value==0) {
try {
Class types = ReflectHelper.classForName("oracle.jdbc.driver.OracleTypes");
oracletypes_cursor_value = types.getField("CURSOR").getInt(types.newInstance());
} catch (Exception se) {
throw new HibernateException("Problem while trying to load or access OracleTypes.CURSOR value",se);
}
}
// register the type of the out param - an Oracle specific type
statement.registerOutParameter(col, oracletypes_cursor_value);
col++;
return col;
}
May be there is a different version of hibernate is used in this jar and that causes the conflict in second server
Any one Please provide us a solution for this problem.
I have found the Answer by help of a friend.
Changed dialect code in Oracle8iDialect.java inside the z_easybeans-uberjar-hibernate-1.1.0-M3-JONAS.jar.
public int registerResultSetOutParameter(java.sql.CallableStatement statement,int col) throws SQLException {
if(oracletypes_cursor_value==0) {
try {
Class types = ReflectHelper.classForName("**oracle.jdbc.OracleTypes**");
oracletypes_cursor_value = types.getField("CURSOR").getInt(**null**);
} catch (Exception se) {
throw new HibernateException("Problem while trying to load or access OracleTypes.CURSOR value",se);
}
}
// register the type of the out param - an Oracle specific type
statement.registerOutParameter(col, oracletypes_cursor_value);
col++;
return col;
}
and compiled the particular dialect file and added the class file to z_easybeans-uberjar-hibernate-1.1.0-M3-JONAS.jar after deleting old class file
then made war and activated it from weblogic.
Then it worked normally.

Profiling Hadoop

UPDATE:
I had mailed Shevek, founder of Karmasphere, for help. He had given a presentation on hadoop profiling at ApacheCon 2011. He advised to look for Throwable. Catch block for Throwable shows :
localhost: java.lang.IncompatibleClassChangeError: class com.kannan.mentor.sourcewalker.ClassInfoGatherer has interface org.objectweb.asm.ClassVisitor as super class
localhost: at java.lang.ClassLoader.defineClass1(Native Method)
localhost: at java.lang.ClassLoader.defineClass(ClassLoader.java:792)
Hadoop has ASM3.2 jar and I am using 5.0. In 5.0, ClassVisitor is a Super Class and in 3.2 it is an Interface. I am planning to change my profiler to 3.2. Is there any other better way to fix this issue?
BTW, Shevek is super cool. A Founder and CEO, responding to some
anonymous guys emails. Imagine that.
END UPDATE
I am trying to profile Hadoop (JobTracker, Name Node, Data Node etc). Created a profiler using ASM5. Tested it on Spring and everything works fine.
Then tested the profiler on Hadoop in pseudo-distributed mode.
#Override
public byte[] transform(ClassLoader loader, String className,
Class<?> classBeingRedefined, ProtectionDomain protectionDomain,
byte[] classfileBuffer) throws IllegalClassFormatException {
try {
/*1*/ System.out.println(" inside transformer " + className);
ClassReader cr = new ClassReader(classfileBuffer);
ClassWriter cw = new ClassWriter(ClassWriter.COMPUTE_MAXS);
/* c-start */ // CheckClassAdapter cxa = new CheckClassAdapter(cw);
ClassVisitor cv = new ClassInfoGatherer(cw);
/* c-end */ cr.accept(cv, ClassReader.EXPAND_FRAMES);
byte[] b = cw.toByteArray();
/*2*/System.out.println(" inside transformer - returning" + b.length);
return b;
} catch (Exception e) {
System.out.println( " class might not be found " + e.getMessage()) ;
try {
throw new ClassNotFoundException(className, e);
} catch (ClassNotFoundException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
}
return null;
}
I can see the first sysout statement printed but not the second one. There is no error either. If I comment out from /* c-start / to / c-stop*/ and replace cw with classFileBuffer, I can see the second sysout statement. The moment I uncomment line
ClassVisitor cv = new ClassInfoGatherer(cw);
ClassInfoGatherer constructor:
public ClassInfoGatherer(ClassVisitor cv) {
super(ASM5, cv);
}
I am not seeing the second sysout statement.
What am i doing wrong here. Is Hadoop swallowing my sysouts. Tried sys err too. Even if that is the case why can i see the first sysout statement?
Any suggestion would be helpful. I think I am missing something simple and obvious here...but can't figure it out.
following lines were added to hadoop-env.sh
export HADOOP_NAMENODE_OPTS="-javaagent:path to jar $HADOOP_NAMENODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-path to jar $HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-javaagent:path to jar $HADOOP_DATANODE_OPTS"
export HADOOP_BALANCER_OPTS="-javaagent:path to jar $HADOOP_BALANCER_OPTS"
export HADOOP_JOBTRACKER_OPTS="-javaagent:path to jar $HADOOP_JOBTRACKER_OPTS"
Hadoop had asm 3.2 and I was using ASM 5. In ASM5, ClassVisitor is a Superclass and in 3.2 it is an interface. For some reason, the error was a Throwable (credits to Shevek) and catch block was only catching exceptions. The throwable error wasn't captured in any of hadoop logs. So, it was very tough to debug.
Used jar jar links to fix asm version issues and everything works fine now.
If you are using Hadoop and something is not working and no logs to show any error's, then please try to catch Throwable.
Arun

Resources