[Snowflake-jdbc]It hangs when get info from resetset object of connection.getMetadata().getColumns(...) - jdbc

I try to test the jdbc connection of snowflake with codes below
Connection conn = .......
.......
ResultSet rs = conn.getMetaData().getColumns(**null**, "PUBLIC", "TAB1", null); // 1. set parameters to get metadata of table TAB1
while (rs.next()) { // 2. It hangs here if the first parameter is null in above liune; otherwise(set the corrent db name), it works fine
System.out.println( "precision:" + rs.getInt(7)
+ ",col type name:" + rs.getString(6)
+ ",col type:" + rs.getInt(5)
+ ",col name:" + rs.getString(4)
+ ",CHAR_OCTET_LENGTH:" + rs.getInt(16)
+ ",buf LENGTH:" + rs.getString(8)
+ ",SCALE:" + rs.getInt(9));
}
.......
I debug the codes above in Intellij IDEA, and find that the debugger can't get the details of the object, it always shows "Evaluating..."
The JDBC driver I used is snowflake-jdbc-3.12.5.jar
Is it a bug?

When the catalog (database) argument is null, the JDBC code effectively runs the following SQL, which you can verify in your Snowflake account's Query History UIs/Views:
show columns in account;
This is an expensive metadata query to run due to no filters and the wide requested breadth (columns across the entire account).
Depending on how many databases exist in your organization's account, it may require several minutes or upto an hour of execution to return back results, which explains the seeming "hang". On a simple test with about 50k+ tables dispersed across 100+ of databases and schemas, this took at least 15 minutes to return back results.
I debug the codes above in Intellij IDEA, and find that the debugger can't get the details of the object, it always shows "Evaluating..."
This may be a weirdness with your IDE, but in a pinch you can use the Dump Threads (Ctrl + Escape, or Ctrl + Break) option in IDEA to provide a single captured thread dump view. This should help show that the JDBC client thread isn't hanging (as in, its not locked or starved), it is only waiting on the server to send back results.

There is no issue with the 3.12.5 jar.I just tested the same version in Eclipse, I can inspect all the objects . Could be an issue with your IDE.
ResultSet columns = metaData.getColumns(null, null, "TESTTABLE123",null);
while (columns.next()){
System.out.print("Column name and size: "+columns.getString("COLUMN_NAME"));
System.out.print("("+columns.getInt("COLUMN_SIZE")+")");
System.out.println(" ");
System.out.println("COLUMN_DEF : "+columns.getString("COLUMN_DEF"));
System.out.println("Ordinal position: "+columns.getInt("ORDINAL_POSITION"));
System.out.println("Catalog: "+columns.getString("TABLE_CAT"));
System.out.println("Data type (integer value): "+columns.getInt("DATA_TYPE"));
System.out.println("Data type name: "+columns.getString("TYPE_NAME"));
System.out.println(" ");
}

Related

TKProf (Oracle event 10046) on Spring Boot/JDBC

I'm trying to start oracle tracing through invoking direct JDBC calls. I'm obtaining my connection from Spring (boot/jdbc). Then I run the TKProf commands through statements... execute the query and print to the log.
The 3 statements below are returning false. If I use this same statements through Intellij's console I will get the intended results and my *.trc file is properly generated.
try (final Connection connection = DataSourceUtils.getConnection(dataSource)) {
log.debug(query);
final Long maxCount = findMaxCount();
boolean traceIdSet = connection.createStatement().execute("ALTER SESSION SET TRACEFILE_IDENTIFIER = '" + traceId + "'");
boolean traceEnabled = connection.createStatement().execute("ALTER SESSION SET EVENTS '10046 trace name context forever, level 8'");
final PreparedStatement stmt = connection.prepareStatement(query);
map(consumer, stmt.executeQuery(query));
boolean traceIdOff = connection.createStatement().execute("ALTER SESSION SET EVENTS '10046 trace name context off'");
log.debug("|" + traceIdSet + "|" + traceEnabled + "|" + traceIdOff + "| ____________________ DONE __________________________");
} catch (SQLException e) {
log.error("Error Performing the Query", e);
}
It has to be something in my configuration... I mean, java thin driver can do it because I can do it over the IDE... so I have to be missing some other stuff, maybe a Spring Boot convention that I should change.
Could you please help, any input is valuable.
Thanks!
My bad, the real issue was that I wasn't getting proper response from...
SELECT value FROM v$diag_info
Where I couldn't found the trace file, but only some others...
Nevertheless the trc files are in place, so there is no problem with Spring Boot/JDBC for enabling TKProf.

How do I make uCanAccess use Samba authentication, with special characters in username or password?

TL;DR: What Database.FileFormat constant should I use for an MS Access 2000-2003 database, when creating the Database object?
I have built a SAMBA test application using jCIFS. It allows me to create/overwrite files if given the correct authentication credentials, regardless of on which PC in the domain I use it.
I also have an application that uses uCanAccess/jackcess to connect to an MDB on a network share. However (from what I understand), it uses the credentials of the logged-in user, a number of whom have read-only access. Only system/network administrators have write permission.
The database in question is not password-protected. (I don't need to enter a password when opening it.)
My intention is to have the app ask for the administrator's Samba credentials before it writes to the DB, using those in the uCanAccess connection, so that it doesn't throw a java.nio.channels.NonWritableChannelException, as per the below stack trace:
java.nio.channels.NonWritableChannelException
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:747)
at com.healthmarketscience.jackcess.impl.PageChannel.writePage(PageChannel.java:310)
at com.healthmarketscience.jackcess.impl.PageChannel.writePage(PageChannel.java:247)
at com.healthmarketscience.jackcess.impl.TableImpl.writeDataPage(TableImpl.java:1980)
at com.healthmarketscience.jackcess.impl.TableImpl.addRows(TableImpl.java:2229)
at com.healthmarketscience.jackcess.impl.TableImpl.addRow(TableImpl.java:2067)
at net.ucanaccess.converters.UcanaccessTable.addRow(UcanaccessTable.java:44)
at net.ucanaccess.commands.InsertCommand.insertRow(InsertCommand.java:101)
at net.ucanaccess.commands.InsertCommand.persist(InsertCommand.java:148)
at net.ucanaccess.jdbc.UcanaccessConnection.flushIO(UcanaccessConnection.java:315)
at net.ucanaccess.jdbc.UcanaccessConnection.commit(UcanaccessConnection.java:205)
at net.ucanaccess.jdbc.AbstractExecute.executeBase(AbstractExecute.java:217)
at net.ucanaccess.jdbc.Execute.execute(Execute.java:46)
at net.ucanaccess.jdbc.UcanaccessPreparedStatement.execute(UcanaccessPreparedStatement.java:228)
at myapp.db.Digger.addTransaction(Digger.java:993)
at myapp.tasks.TransactionRunnable.run(TransactionRunnable.java:42)
at java.lang.Thread.run(Thread.java:745)
Update: I have tried using the smbFileChannel class by Gord Thompson and J. T. Alhborn, shown here. My code, based off the main class shown in that answer, looks like this:
// Ask the user for login credentials and the path to the database
String smbURL = (chosenDir.endsWith("/") ? chosenDir : chosenDir + '/')
+ dbName;
System.out.println("DB Path to use for URL: " + smbURL);
URL u = new URL(smbURL);
try (
// construct the SMB DB URL
SmbFileChannel sfc = new SmbFileChannel(smbURL);
Database db = new DatabaseBuilder().setChannel(sfc)
.setFileFormat(Database.FileFormat.GENERIC_JET4).create();
) {
// Model the table
Table tbl = new TableBuilder("Transactions")
.addColumn(new ColumnBuilder("TransactionID", DataType.LONG).setAutoNumber(true))
.addColumn(new ColumnBuilder("ControllerID", DataType.LONG).setAutoNumber(false))
.addColumn(new ColumnBuilder("ReaderID", DataType.LONG).setAutoNumber(false))
.addColumn(new ColumnBuilder("Event", DataType.LONG).setAutoNumber(false))
.addColumn(new ColumnBuilder("Timestamp", DataType.SHORT_DATE_TIME).setAutoNumber(false))
.addColumn(new ColumnBuilder("Number", DataType.LONG).setAutoNumber(false))
.addIndex(new IndexBuilder(IndexBuilder.PRIMARY_KEY_NAME).addColumns("TransactionID").setPrimaryKey())
.toTable(db);
// Add the row
Map<String, Object> values = new HashMap<>();
values.put("ControllerID", cid);
values.put("ReaderID", rid);
values.put("Event", evtNum);
values.put("Timestamp", ts); // Long; must be converted to DataType.SHORT_DATE_TIME
values.put("Number", accNum);
tbl.addRowFromMap(values);
} catch (IOException IOEx) {
System.err.println(
"Failed to write record to Transactions table in database: "
+ IOEx.getMessage()
);
IOEx.printStackTrace(System.err);
} catch (Exception ex) {
System.err.println(
'[' + ex.getClass().getSimpleName() + "]: Failed to write record to "
+ "Transactions table in database: " + ex.getMessage()
);
ex.printStackTrace(System.err);
}
Executing the above code results in the following output:
DB Path to use for URL: smb://machine.vpnName/Storage/me/dbs/DBName.mdb
Failed to write record to Transactions table in database: Logon failure: account currently disabled.
jcifs.smb.SmbAuthException: Logon failure: account currently disabled.
at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:546)
at jcifs.smb.SmbTransport.send(SmbTransport.java:663)
at jcifs.smb.SmbSession.sessionSetup(SmbSession.java:390)
at jcifs.smb.SmbSession.send(SmbSession.java:218)
at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
at jcifs.smb.SmbFile.connect(SmbFile.java:957)
at jcifs.smb.SmbFile.connect0(SmbFile.java:880)
at jcifs.smb.SmbFile.open0(SmbFile.java:975)
at jcifs.smb.SmbFile.open(SmbFile.java:1009)
at jcifs.smb.SmbRandomAccessFile.<init>(SmbRandomAccessFile.java:57)
at jcifs.smb.SmbRandomAccessFile.<init>(SmbRandomAccessFile.java:42)
at samba.SmbFileChannel.<init>(SmbFileChannel.java:30)
at samba.SambaLanWriteTest.writeTest(SambaLanWriteTest.java:130)
at samba.SambaLanWriteTest.main(SambaLanWriteTest.java:181)
I have write access to a test copy of the database file in question when using Windows File Explorer. I am choosing that one when prompted.
Update 2: I realised that I neglected to add my username and password to the smb:// URL, as Thompson's example shows. I changed to code to this:
String smbCred = "smb://" + auth.getUsername() + ":" + auth.getPassword() + "#",
fixer = chosenDir.replace("\\", "/").replace("smb://", smbCred),
smbURL = fixer + dbName;
System.out.println("DB Path to use for URL: " + smbURL);
// URL u = new URL(smbURL);
The next problem I had was that my password contains special illegal characters (such as '#', ':', ';', '=' and '?'). I escaped these by using java.net.URLEncoder.encode() on auth.getUsername() and auth.getPassword() so the code doesn't throw a MalformedURLException when creating the SmbChannel. However, the next exception I encountered is as follows:
Failed to write record to Transactions table in database: File format GENERIC_JET4 [VERSION_4] does not support file creation for null
java.io.IOException: File format GENERIC_JET4 [VERSION_4] does not support file creation for null
at com.healthmarketscience.jackcess.impl.DatabaseImpl.create(DatabaseImpl.java:444)
What Database.FileFormat constant should I use for an MS Access 2000-2003 database, when creating the Database object?
It turns out that I needed to use Database.FileFormat.V2000.
After that, it was all plain sailing (although I still need to work out how to get the Long timestamp to convert correctly).

Netezza Batch Insert is very slow even in Batch execute mode

I am referring to this documentation. http://www-01.ibm.com/support/docview.wss?uid=swg21981328. As per the article if we use executeBatch method then inserts will be faster (The Netezza JDBC driver may detect a batch insert, and under the covers convert this to an external table load and external table load will be faster). I had to execute millions of insert statements and i am getting only a speed of 500 records per minute per connection max. Is there any better way to load data faster to netezza via jdbc connection? I am using spark and jdbc connection to insert the records.Why external table via loading is not happening even when i am executing in batches. Given below is the spark code i am using,
Dataset<String> insertQueryDataSet.foreachPartition( partition -> {
Connection conn = NetezzaConnector.getSingletonConnection(url, userName, pwd);
conn.setAutoCommit(false);
int commitBatchCount = 0;
int insertBatchCount = 0;
Statement statement = conn.createStatement();
//PreparedStatement preparedStmt = null;
while(partition.hasNext()){
insertBatchCount++;
//preparedStmt = conn.prepareStatement(partition.next());
statement.addBatch(partition.next());
//statement.addBatch(partition.next());
commitBatchCount++;
if(insertBatchCount % 10000 == 0){
LOGGER.info("Before executeBatch.");
int[] execCount = statement.executeBatch();
LOGGER.info("After execCount." + execCount.length);
LOGGER.info("Before commit.");
conn.commit();
LOGGER.info("After commit.");
}
}
//execute remaining statements
statement.executeBatch();
int[] execCount = statement.executeBatch();
LOGGER.info("After execCount." + execCount.length);
conn.commit();
conn.close();
});
I tried this approach(batch insert) but found very slow,
So I put all data in CSV & do external table load for each csv.
InsertReq="Insert into "+ tablename + " select * from external '"+ filepath + "' using (maxerrors 0, delimiter ',' unase 2000 encoding 'internal' remotesource 'jdbc' escapechar '\' )";
Jdbctemplate.execute(InsertReq);
Since I was using java so JDBC as source & note that csv file path is in single quotes .
Hope this helps.
If you find better than this approach, don't forget to post. :)

Why does h2 ignore slf4j messages on the first connection when LOG is set?

See sample code & output below (with Slf4j/logback on stdout). I can't find any bug reports on this. I'm using h2 version 1.3.176 (last stable), in-memory mode. It doesn't seem to matter what value is set for the LOG (0, 1 or 2) but just has to be set.
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
public class H2TraceTest {
public static void main(String[] args) throws SQLException {
System.out.println("Query connection 1");
Connection myConn = DriverManager.getConnection("jdbc:h2:mem:tracetest;TRACE_LEVEL_FILE=4;LOG=2");
myConn.createStatement().execute("SELECT 1");
System.out.println("Query connection 2");
DriverManager.getConnection("jdbc:h2:mem:tracetest").createStatement().execute("SELECT 1");
System.out.println("Query connection 1 again");
myConn.createStatement().execute("SELECT 1");
System.out.println("End");
}
}
Output:
Query connection 1
Query connection 2
16:17:02.955 INFO h2database - jdbc[3]
/**/Connection conn2 = DriverManager.getConnection("jdbc:h2:mem:tracetest", "", "");
16:17:02.958 DEBUG h2database - jdbc[3]
/**/Statement stat2 = conn2.createStatement();
16:17:02.959 DEBUG h2database - jdbc[3]
/**/stat2.execute("SELECT 1");
16:17:02.959 INFO h2database - jdbc[3]
/*SQL #:1*/SELECT 1;
Query connection 1 again
End
I know that the H2 documentation says about TRACE_LEVEL_FILE: it affects all connections. But thats not (fully) correct:
Every connection keeps a lazy reference to the logging system. And if you change that with the special marker TRACE_LEVEL_FILE=4, then that reference isn't changed for all existing connections - but only for those who do their first logging after that change.
So if you use the connection string "jdbc:h2:mem:tracetest;TRACE_LEVEL_FILE=4" everything is as expected, because your session will write no logging message before changing the logging system. Unfortunately the LOG=2 in jdbc:h2:mem:tracetest;TRACE_LEVEL_FILE=4;LOG=2 is evaluated first, because both parameter are written into and read from an unordered Map. And because LOG=2 is generating a log statement, the reference to the log adapter (=4) is never applied to the current session. Only to the next one.
What can you do:
Use only "jdbc:h2:mem:tracetest;TRACE_LEVEL_FILE=4" - LOG=2 is the default anyway. If you need any other log mode you can use connection.createStatement().executeUpdate("SET LOG 1")
Add some default parameters to the connection string until the TRACE_LEVEL_FILE parameter is the first parameter in the map (not really reliable, as the order may depend on the VM)
Discard the first connection at once
Fill in a bug report and wait for the fix (or fix it yourself), as I think this is somehow a bug
I know this is an old question but here is a reliable way to do it (i.e. you can ensure that TRACE_LEVEL_FILE is set to 4 first:
String url = "jdbc:h2:mem:tracetest;INIT=SET TRACE_LEVEL_FILE=4\\;SET DB_CLOSE_DELAY=-1/* for example, i.e. do other stuff */";

Password Changer using VAccess

Hey I am working on a password changer. User logs in ( successfully), loads a global var with user initials, then launch a password expired form. I try and use those initials on the password expired form to retrieve user info from DB.
vaUserLog.FieldValue("USERINIT") = UserInitials
vaUserLog.GetEqual
vaStat = vaUserLog.Status
vaStat keeps giving me an error of 4. I am using pervasive v9. Connection with VA looks like:
With vaUserLog
.RefreshLocations = True
.DdfPath = DataPath
.TableName = "USERLOG"
.Location = "USERLOG.MKD"
.Open
If .Status <> 0 Then
ErrMsg = "Error Opening File " + .TableName + " - Status " + str$(.Status) + vbCrLf + "Contact IT Department"
End If
End With
In DB table, USERINIT is Char, 3. UserInitials is a String.
Probably missing something small but can't think right now. Any help is appreciate. Lemme know if you require more info.
Cheers
Status 4 means that the record could not be found. In your case, it could be the case of the value being searched is wrong, there's a different padding (spaces versus binary zero), or that the UserInitials value just isn't in the data file.
You can use MKDE Tracing to see what's actually being passed to the PSQL engine. Once you've done that, make sure the value you're using works through the Function Executor where you can open the file and perform a GetEqual.
Here are my suggestions:
- Make sure you're pointing to the right data files.
- Make sure you're passing the right value into the GetEqual (by setting the FieldValue).

Resources