Is there a way of retrieve the nodes names of a vertica cluster using jdbc?
in order to achieve a fault-tolerect connection strategy?
Can it retrieved from this retConnection Connection instance?
Connection retConnection = null;
.
.
.
retConnection = DriverManager.getConnection
(
connectionString,
connectionProperties
);
Thanks.
What about...
rs = stmt.executeQuery("SELECT node_name FROM v_catalog.nodes") ;
Related
I am trying to connect to Snowflake using R in databricks, my connection works and I can make queries and retrieve data successfully, however my problem is that it can take more than 25 minutes to simply connect, but once connected all my queries are quick thereafter.
I am using the sparklyr function 'spark_read_source', which looks like this:
query<- spark_read_source(
sc = sc,
name = "query_tbl",
memory = FALSE,
overwrite = TRUE,
source = "snowflake",
options = append(sf_options, client_Q)
)
where 'sf_options' are a list of connection parameters which look similar to this;
sf_options <- list(
sfUrl = "https://<my_account>.snowflakecomputing.com",
sfUser = "<my_user>",
sfPassword = "<my_pass>",
sfDatabase = "<my_database>",
sfSchema = "<my_schema>",
sfWarehouse = "<my_warehouse>",
sfRole = "<my_role>"
)
and my query is a string appended to the 'options' arguement e.g.
client_Q <- 'SELECT * FROM <my_database>.<my_schema>.<my_table>'
I can't understand why it is taking so long, if I run the same query from RStudio using a local spark instance and 'dbGetQuery', it is instant.
Is spark_read_source the problem? Is it an issue between Snowflake and Databricks? Or something else? Any help would be great. Thanks.
The H2 Database has a list of commands starting with SET, in particular SET DB_CLOSE_DELAY. I would like to find out what the value of DB_CLOSE_DELAY is. I am using JDBC. Setting is easy
cx.createStatement.execute("SET DB_CLOSE_DELAY 0")
but none of the following returns the actual value of DB_CLOSE_DELAY:
cx.createStatement.executeQuery("DB_CLOSE_DELAY")
cx.createStatement.executeQuery("VALUES(#DB_CLOSE_DELAY)")
cx.createStatement.executeQuery("GET DB_CLOSE_DELAY")
cx.createStatement.executeQuery("SHOW DB_CLOSE_DELAY")
Help would be greatly appreciated.
You can access this and other settings in the INFORMATION_SCHEMA.SETTINGS table - for example:
String url = "jdbc:h2:mem:;DB_CLOSE_DELAY=3";
Connection conn = DriverManager.getConnection(url, "sa", "the password goes here");
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery("SELECT * FROM INFORMATION_SCHEMA.SETTINGS where name = 'DB_CLOSE_DELAY'");
while (rs.next()) {
System.out.println(rs.getString("name"));
System.out.println(rs.getString("value"));
}
In this test, I use an unnamed in-memory database, and I explicitly set the delay to 3 seconds when I create the DB.
The output from the print statements is:
DB_CLOSE_DELAY
3
I am referring to this documentation. http://www-01.ibm.com/support/docview.wss?uid=swg21981328. As per the article if we use executeBatch method then inserts will be faster (The Netezza JDBC driver may detect a batch insert, and under the covers convert this to an external table load and external table load will be faster). I had to execute millions of insert statements and i am getting only a speed of 500 records per minute per connection max. Is there any better way to load data faster to netezza via jdbc connection? I am using spark and jdbc connection to insert the records.Why external table via loading is not happening even when i am executing in batches. Given below is the spark code i am using,
Dataset<String> insertQueryDataSet.foreachPartition( partition -> {
Connection conn = NetezzaConnector.getSingletonConnection(url, userName, pwd);
conn.setAutoCommit(false);
int commitBatchCount = 0;
int insertBatchCount = 0;
Statement statement = conn.createStatement();
//PreparedStatement preparedStmt = null;
while(partition.hasNext()){
insertBatchCount++;
//preparedStmt = conn.prepareStatement(partition.next());
statement.addBatch(partition.next());
//statement.addBatch(partition.next());
commitBatchCount++;
if(insertBatchCount % 10000 == 0){
LOGGER.info("Before executeBatch.");
int[] execCount = statement.executeBatch();
LOGGER.info("After execCount." + execCount.length);
LOGGER.info("Before commit.");
conn.commit();
LOGGER.info("After commit.");
}
}
//execute remaining statements
statement.executeBatch();
int[] execCount = statement.executeBatch();
LOGGER.info("After execCount." + execCount.length);
conn.commit();
conn.close();
});
I tried this approach(batch insert) but found very slow,
So I put all data in CSV & do external table load for each csv.
InsertReq="Insert into "+ tablename + " select * from external '"+ filepath + "' using (maxerrors 0, delimiter ',' unase 2000 encoding 'internal' remotesource 'jdbc' escapechar '\' )";
Jdbctemplate.execute(InsertReq);
Since I was using java so JDBC as source & note that csv file path is in single quotes .
Hope this helps.
If you find better than this approach, don't forget to post. :)
I am new to SoapUI framework. I am trying to use soapUI framework for testing REST API. While testing for REST API, I need to verify data from backend database as well like Hive and Cassandra.
I could do the setup for SoapUI and could test a query on Cassandra using groovy script that is provided SoapUI framework. But when I searched for connecting to hive using SoapUI, I couldn't find any reference for that. Also on there sites, JDBC drivers are not provided but hive is not mentioned there.
So is there any option to connect to hive from SoapUI framework?
Should I think about using Hive JDBC driver from SoapUI?
Thanks for your help!
I believe you should be able to use it for different data bases using following ways:
JDBC test step
Groovy Script (you should be able to use almost java code)
Either of the ways, copy the drivers/libraries into SOAPUI_HOME/bin/ext directory and restart SoapUI
Here is the link for client code (in Java) to connect to Hive.
Sample connection code from above link(so should be able to use in groovy) :
try {
Class.forName(driverName);
} catch (ClassNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
System.exit(1);
}
Connection con = DriverManager.getConnection("jdbc:hive://localhost:10000/default", "", "");
Statement stmt = con.createStatement();
String tableName = "testHiveDriverTable";
stmt.executeQuery("drop table " + tableName);
ResultSet res = stmt.executeQuery("create table " + tableName + " (key int, value string)");
// show tables
String sql = "show tables '" + tableName + "'";
System.out.println("Running: " + sql);
res = stmt.executeQuery(sql);
I have a stored procedure in an Oracle 10g database, in my java code, i call it with:
CallableStatement cs = bdr.prepareCall("Begin ADMBAS01.pck_basilea_reportes.cargar_reporte(?,?,?,?,?); END;", ResultSet.TYPE_SCROLL_INSENSITIVE, ResultSet.CONCUR_READ_ONLY);
cs.setInt(1, this.reportNumber);
cs.registerOutParameter(2, OracleTypes.CURSOR);
cs.registerOutParameter(3, OracleTypes.INTEGER);
cs.registerOutParameter(4, OracleTypes.VARCHAR);
cs.setDate(5, new java.sql.Date(this.fecha1.getTime()));
cs.execute();
ResultSet rs = (ResultSet)cs.getObject(2);
i do obtain an ResultSet with correct records in it, but when i try an "scroll_insensitive - only" operation, (like absolute(1) ). I keep getting an SQLException stating that it doesn't work on FORWARD only resultSet.
So how can i obtain this ResultSet with scroll_insensitive capabilites?
Thanks in Advance.
The result set type is merely a suggestion to the driver, which the driver can ignore or downgrade to FORWARD_ONLY if it can't comply. See here for details.