I am trying to dump database tables to a file using Spring jdbcTemplate. I can't use pg_dump or psql becase it's a custom data dump.
There are some large tables that causes OutOfMemoryError.
Theorically setting fetchSize to some number > 0 would not eat up the memory as stated.
I have tried:
jdbcTemplate.setFetchSize() and rs.setFetchSize().
private void dumpTable(final String qry, final CSVPrinter csvPrinter) {
jdbcTemplate.queryForStream(qry, (rs, rowNum) -> getObjects(rs)).forEach(objs -> {
try {
csvPrinter.printRecord(objs);
} catch (IOException e) {
throw new IllegalStateException("Error", e);
}
});
}
Query looks like this:
"select %s from %s t where exists(select 1 from log.log_table l where l.schema = '%s' and l.table = '%s' and l.key = t.%s and l.location in (0,%d))";
Also tried jdbcTemplate.query() with RowCallbackHandler.
This solved the issue (on application.properties):
spring.datasource.hikari.auto-commit=false
I can confirm that it's needed for postgresql.
Related
I want to rename a Hive table if it exists, and not generate an error if it doesn't.
I need something like
ALTER TABLE IF EXISTS t1 RENAME TO t2;
but this doesn't run ("cannot recognize input near 'if' 'exists' 'rename' in alter table statement"), and neither do the variations that I've tried. This isn't covered in the docs (https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-RenameTable), maybe because it's not possible.
Does anyone know how to do this, or a workaround (e.g. try/catch, if it existed in Hive)?
I'm on Hive 1.2.
IF EXIST clause does not work in Hive CLI as of now. You can write program something like below for condition checking.
public class HiveAlterRenameTo {
private static String driverName = "org.apache.hadoop.hive.jdbc.HiveDriver";
public static void main(String[] args) throws SQLException {
// Register driver and create driver instance
Class.forName(driverName);
// get connection
Connection con = DriverManager.getConnection("jdbc:hive://localhost:10000/userdb", "", "");
// create statement
Statement stmt = con.createStatement();
// execute statement
Resultset res = stmt.executeQuery("SELECT count(*) FROM <Table_name> ;");
if (res > 0) {
// execute statement
stmt.executeQuery("ALTER TABLE employee RENAME TO emp;");
System.out.println("Table Renamed Successfully");
}
else {
System.out.println("Table Not exist");
}
con.close();
}
I am executing multiple queries concurrently and retrieving the results. But, the queries belong to multiple tables so, when resultset is retrieved, it is difficult to identify that a resultset belong to which table.
Can anyone help here as to how to identify the table names for each query resultset?
I tried below code but table name is blank!!!!
public static void getColumnNames(ResultSet rs) throws SQLException {
if (rs == null) {
return;
}
// get result set meta data
ResultSetMetaData rsMetaData = rs.getMetaData();
int numberOfColumns = rsMetaData.getColumnCount();
// get the column names; column indexes start from 1
for (int i = 1; i < numberOfColumns + 1; i++) {
String columnName = rsMetaData.getColumnName(i);
// Get the name of the column's table name
String tableName = rsMetaData.getTableName(i);
System.out.println("column name=" + columnName + " table=" + tableName + "");
}
}
I am calling this method like this:
jdbcTemplate.query(sql, new ResultSetExtractor<ResultSet>() {
#Override
public ResultSet extractData(ResultSet resultSet) throws SQLException,
DataAccessException {
getColumnNames(resultSet);
return resultSet;
}
});
Please advise, what is done wrong here? :(
You're not doing anything wrong here. The problem is caused by the method itself in connection with your DBMS or your JDBC driver, respectively.
See this doc please. 'table name or "" if not applicable' suggests that in your case the DBMS/driver does not provide the required information, causing the method to return an empty string.
I'm afraid, you'll have to find another way to detect which query the result originated from.
I'm trying to insert some data to Cassandra with Astyanax, by I'm getting a lot of "Re-preparing already prepared query" warnings even if have caching enabled:
22:08:03,703 WARN Cluster:1702 - Re-preparing already prepared query INSERT INTO test.test (key,c0,c1,c2,c3,c4,c5,c6,c7,c8,c9) VALUES (?,?,?,?,?,?,?,?,?,?,?) . Please note that preparing the same query more than once is generally an anti-pattern and will likely affect performance. Consider preparing the statement only once.
22:08:03,707 WARN Cluster:1702 - Re-preparing already prepared query INSERT INTO test.test (key,c0,c1,c2,c3,c4,c5,c6,c7,c8,c9) VALUES (?,?,?,?,?,?,?,?,?,?,?) . Please note that preparing the same query more than once is generally an anti-pattern and will likely affect performance. Consider preparing the statement only once.
22:08:03,708 WARN Cluster:1702 - Re-preparing already prepared query INSERT INTO test.test (key,c0,c1,c2,c3,c4,c5,c6,c7,c8,c9) VALUES (?,?,?,?,?,?,?,?,?,?,?) . Please note that preparing the same query more than once is generally an anti-pattern and will likely affect performance. Consider preparing the statement only once.
Source code:
Connect: (executed once)
#Override
public void connect() throws ClientException {
AstyanaxContext<Keyspace> context = new AstyanaxContext.Builder()
.forCluster(clusterName)
.forKeyspace(keyspaceName)
.withHostSupplier(new Supplier<List<Host>>() {
#Override
public List<Host> get() {
return Collections.singletonList(new Host(host, 9160));
}
})
.withAstyanaxConfiguration(
new AstyanaxConfigurationImpl().setDiscoveryType(NodeDiscoveryType.DISCOVERY_SERVICE)
.setDiscoveryDelayInSeconds(60000))
.withConnectionPoolConfiguration(new JavaDriverConfigBuilder().build())
.buildKeyspace(CqlFamilyFactory.getInstance());
context.start();
keyspace = context.getClient();
columnFamilyTemplate = new ColumnFamily<String, String>(columnFamily,
StringSerializer.get(), StringSerializer.get());
try {
columnFamilyTemplate.describe(keyspace);
} catch (ConnectionException e) {
throw new ClientException(e);
}
insert = keyspace.prepareMutationBatch().withCaching(true);
}
Insert: (executed multiple times)
insert.discardMutations();
final ColumnListMutation<String> row = insert.withRow(columnFamilyTemplate, key);
for (Map.Entry<String, String> pair : columnValues.entrySet()) {
final String column = pair.getKey();
final String value = pair.getValue();
row.putColumn(column, value, null);
}
try {
insert.withCaching(true).execute();
} catch (ConnectionException e) {
throw new ClientException(e);
}
The warning message suggests that the caching is not actually working. Any idea how to fix it?
This query is returning the record with Min Create time Stamp for the Person Pers_ID when I run it in SQL Developer and the same query is not returning any value from Java JDBC connection.
Can you please help?
select PERS_ID,CODE,BEG_DTE
from PRD_HIST H
where PERS_ID='12345'
and CODE='ABC'
and CRTE_TSTP=(
select MIN(CRTE_TSTP)
from PRD_HIST S
where H.PERS_ID=S.PERS_ID
and PERS_ID='12345'
and EFCT_END_DTE is null
)
Java Code
public static List<String[]> getPersonwithMinCreateTSTP(final String PERS_ID,final String Category,final Connection connection){
final List<String[]> personRecords = new ArrayList<String[]>();
ResultSet resultSet = null;
Statement statement = null;
String PersID=null;
String ReportCode=null;
String effBegDate=null;
try{
statement = connection.createStatement();
final String query="select PERS_ID,CODE,EFCT_BEG_DTE from PRD_HIST H where PERS_ID='"+PERS_ID+"'and CODE='"+Category+"'and CRTE_TSTP=(select MIN(CRTE_TSTP) from PRD_HIST S where H.PERS_ID=S.PERS_ID and PERS_ID='"+PERS_ID+"' and EFCT_END_DTE is null)";
if (!statement.execute(query)) {
//print error
}
resultSet = statement.getResultSet();
while (resultSet.next()) {
PersID=resultSet.getString("PERS_ID");
ReportCode=resultSet.getString("CODE");
effBegDate=resultSet.getString("EFCT_BEG_DTE");
final String[] personDetails={PersID,ReportCode,effBegDate};
personRecords.add(personDetails);
}
} catch (SQLException sqle) {
CTLoggerUtil.logError(sqle.getMessage());
}finally{ // Finally is added to close the connection and resultset
try {
if (resultSet!=null) {
resultSet.close();
}if (statement!=null) {
statement.close();
}
} catch (SQLException e) {
//print error
}
}
return personRecords;
}
Print out your SQL SELECT statement from your java program and paste it into SQL*Plus and see what is happening. It's likely you're not getting your variables set to what you think you are. In fact, you're likely to see the error when you print out the SELECT statement without even running it - lower case values when upper is needed, etc.
If you still can't see it, post the actual query from your java code here.
I came here with similar problem - just thought I'd post my solution for others following - I hadn't run "COMMIT" after the inserts I'd made (via sqlplus) - doh!
The database table has records but the JDBC client can't retrieve the records.
Means the JDBC client doesn't have the select privileges. Please run the below query on command line:
grant all on emp to hr;
Oracle JDBC supports two distinct models for update batching: Standard Batching and Oracle Specific Batching.
According to oracle 11g JDBC Developer Guide, in any single application, you can use one model or the other, but not both. Oracle JDBC driver will throw exceptions when you mix these.
In my standalone application, the above statement does not hold true. I want to know if I am missing something.
In my application I create a OracleDataSource and do the following
connection = datasource.getConnection();
preparedStatement = connection.prepareStatement("update CAR set CAR_NAME=?, OBJECT_VERSION=? where CAR_ID=?");
for(Car car : cars) {
preparedStatement.setString(1, car.getName());
preparedStatement.setInt(2, car.getVersion() + 1);
preparedStatement.setLong(3, car.getId());
preparedStatement.addBatch();
}
System.out.println("Update Batch : " + Arrays.toString(preparedStatement.executeBatch()));
for(Car car : cars) {
car.setName("v car " + car.getId());
}
//Oracle Update Batching
connection.setAutoCommit(false);
PreparedStatement preparedStatement =
connection.prepareStatement("update CAR set CAR_NAME=?, OBJECT_VERSION=? where CAR_ID=?");
//Change batch size for this statement to 3
((OraclePreparedStatement)preparedStatement).setExecuteBatch (10);
for(Car car : cars) {
preparedStatement.setString(1, car.getName());
preparedStatement.setInt(2, car.getVersion() + 1);
preparedStatement.setLong(3, car.getId());
System.out.println("Execute Update Count " + preparedStatement.executeUpdate());
}
System.out.println("Update Count : " + ((OraclePreparedStatement)preparedStatement).sendBatch()); // JDBC sends the queued request
connection.commit();
preparedStatement.close();
The above code runs well and I could see both the update batches using different batching models getting executed well. Is there anything which I missed out or my interpretation of jdbc developer guide is incorrect?
Thanks in advance
Yes, they write the truth :-)
But this aply to one instance of PreparedStatement
I looked at decompile sources of OraclePreparedStatement:
public void addBatch() throws SQLException {
synchronized(connection){
setJdbcBatchStyle();
processCompletedBindRow(currentRank + 2, currentRank > 0 && sqlKind.isPlsqlOrCall());
currentRank++;
}
}
final void setJdbcBatchStyle() throws SQLException {
if(m_batchStyle == 1){
SQLException sqlexception = DatabaseError.createSqlException(getConnectionDuringExceptionHandling(), 90, "operation cannot be mixed with Oracle-style batching");
sqlexception.fillInStackTrace();
throw sqlexception;
} else{
m_batchStyle = 2;
return;
}
}
So, they realy check mixing of batch modes for instance of OraclePreparedStatement