I want to get subquery from impala table as one dataset.
Code like this:
String subQuery = "(select to_timestamp(unix_timestamp(now())) as ts from my_table) t"
Dataset<Row> ds = spark.read().jdbc(myImpalaUrl, subQuery, prop);
But result is error:
Caused by: java.sql.SQLDataException: [Cloudera][JDBC](10140) Error converting value to Timestamp.
I can use unix_timestamp function,but to_timestmap failed, why?
I found code in org.apache.spark.sql.execution.datasources.jdbc.JDBC.compute() exists some problem:
sqlText = s"SELECT $columnList FROM ${options.table} $myWhereClause"
$columList contains " like "col_name" , when I delete " it work fine.
I solve this problem by add dialect, default dialect will add "" to column name,
JdbcDialect ImpalaDialect = new JdbcDialect(){
#Override
public boolean canHandle(String url) {
return url.startsWith("jdbc:impala") || url.contains("impala");
}
#Override
public String quoteIdentifier(String colName) {
return colName;
}
};
JdbcDialects.registerDialect(ImpalaDialect);
Related
I have a Database table called ProgramData. their i have a data column called Id and executed. id set to be as auto increment.
Table structure is like this.
What i want is according to id executed column need to be updated. following is my code segment.
public void saveDtvProgDataExecuted()
{
ProgramData programeData = new ProgramData();
String SQL = "UPDATE program_data SET executed=1 WHERE programeData.id = ?";
this.jdbcTemplate.update(SQL);
}
If i run this code this gives me error like bad SQL grammar [UPDATE program_data SET executed=1 WHERE programeData.id = ?]; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '?' at line 1
Problem is you’re not passing the ID value to the jdbctemplate.
You should use
this.jdbctemplate.update(SQL, id);
Where id is the id of the record you’re updating.
Please refer to the documentation for more information:
http://static.springsource.org/spring/docs/3.1.x/spring-framework-reference/htmlsingle/spring-framework-reference.html#jdbc-updates
TRY THIS statement while you are passing ? in your sql query it need to be set while execution.
String SQL = "UPDATE program_data SET executed=1 WHERE programeData.id = ?";
this.jdbcTemplate.update(SQL,new PreparedStatementCallback<Boolean>(){
#Override
public Boolean doInPreparedStatement(PreparedStatement ps)
throws SQLException, DataAccessException {
ps.setInt(1,"here you need to pass value of programeData.id);
return ps.execute();
}
});
I am fairly new to spring ,I am looking to check if a certain email id exists in database or not , using Spring Jdbc Template ,I looked here but could'nt find the proper answer .I am looking something like ,SELECT count(*) from table where email=?
Any help will be appreciated.
You can do something as below if you are using jdbctemplate and new version of spring
private boolean isEmailIdExists(String email) {
String sql = "SELECT count(*) FROM table WHERE email = ?";
int count = jdbcTemplate.queryForObject(sql, new Object[] { email }, Integer.class);
return count > 0;
}
queryForObject method of jdbcTemplate accepts the sql query as the first parameter, second argument is an array of objects for the sql query place holders and the third argument is the expected return value from the sql query.
In this case we only have one place holder and hence I gave the second argument as new Object[] { email } and the result we are expecting is a count which is a Integer and hence I gave it as Integer.class
I kind of got this answer from https://www.mkyong.com/spring/jdbctemplate-queryforint-is-deprecated/
You can go through it if you are interested.
private boolean isEmailIdExists(String email) {
return jdbcTemplate.queryForObject("SELECT EXISTS(SELECT FROM table WHERE email = ?)", Boolean.class, email);
}
http://www.postgresqltutorial.com/postgresql-exists/
I am executing multiple queries concurrently and retrieving the results. But, the queries belong to multiple tables so, when resultset is retrieved, it is difficult to identify that a resultset belong to which table.
Can anyone help here as to how to identify the table names for each query resultset?
I tried below code but table name is blank!!!!
public static void getColumnNames(ResultSet rs) throws SQLException {
if (rs == null) {
return;
}
// get result set meta data
ResultSetMetaData rsMetaData = rs.getMetaData();
int numberOfColumns = rsMetaData.getColumnCount();
// get the column names; column indexes start from 1
for (int i = 1; i < numberOfColumns + 1; i++) {
String columnName = rsMetaData.getColumnName(i);
// Get the name of the column's table name
String tableName = rsMetaData.getTableName(i);
System.out.println("column name=" + columnName + " table=" + tableName + "");
}
}
I am calling this method like this:
jdbcTemplate.query(sql, new ResultSetExtractor<ResultSet>() {
#Override
public ResultSet extractData(ResultSet resultSet) throws SQLException,
DataAccessException {
getColumnNames(resultSet);
return resultSet;
}
});
Please advise, what is done wrong here? :(
You're not doing anything wrong here. The problem is caused by the method itself in connection with your DBMS or your JDBC driver, respectively.
See this doc please. 'table name or "" if not applicable' suggests that in your case the DBMS/driver does not provide the required information, causing the method to return an empty string.
I'm afraid, you'll have to find another way to detect which query the result originated from.
Here is my code which uses jdbcTemplate
String SQL = "select branch from branchTable where branch_name = '" + branch + "'";
ArrayList<String> branchList = (ArrayList<String>) jdbcTemplateObject.query(SQL, new RowMapper() {
public Object mapRow(ResultSet resultSet, int i) throws SQLException {
return resultSet.getString("city_desc");
}
});
return branchList;
Now i want to be able to use preparedstatement with a query like "select branch from branchTable where branch_name = ?"
How can i do that with jdbcTemplate ?
Examples i searched show demonstration on how to use it with update or insert query, but not with select query..
Please help.
JdbcTemplate has another query() method which takes arguments of the prepared statement as parameter:
jdbcTemplateObject.query(SQL, new Object[] {branchName}, new RowMapper() {...});
Note that:
SQL should be named sql
You should use List and not ArrayList. Nothing in the javadoc guarantees that an ArrayList is returned. And you shouldn't care about the concrete type of list returned.
You should use a RowMapper<String> and not a raw RowMapper.
I'm writing an integration test in Grails using GORM.
I want to do something like the following:
delete from Statistic
where stat_date = TO_DATE(:month_year, 'MON-YYYY')
But I get the following error:
java.sql.SQLException: Unexpected token: TO_DATE in statement [delete
from statistics where stat_date=TO_DATE(?, 'MON-YYYY')]
I think the error is caused by the in memory database used by GORM (is it H2?) not supporting the to_date function.
Any ideas on how to write the delete SQL so that it works in a test and in live?
As I only really care about the Month and Year one thought I had would be to delete the records where the stat_date is between the first and last date of the given month.
Can any one think of a better way?
This still comes up as number 1 on google searches so here's what worked for me.
My unit tests/local environment build and populate a schema using sql files. I created the following alias in the sql file
-- TO_DATE
drop ALIAS if exists TO_DATE;
CREATE ALIAS TO_DATE as '
import java.text.*;
#CODE
java.util.Date toDate(String s, String dateFormat) throws Exception {
return new SimpleDateFormat(dateFormat).parse(s);
}
';
Notice the use of single quotes instead of $$ in h2 user defined functions as that is the only format that worked for me.
I had to adjust bluesman's answer in order to make it work for the date formats that are commonly used in our Oracle sql.
This version supports dateFormats like 'DD-MON-YYYY'
-- TO_DATE
drop ALIAS if exists TO_DATE;
CREATE ALIAS TO_DATE as '
import java.text.*;
#CODE
java.util.Date toDate(String s, String dateFormat) throws Exception {
if (dateFormat.contains("MON")) {
dateFormat = dateFormat.replace("MON", "MMM");
}
if (dateFormat.contains("Y")) {
dateFormat = dateFormat.replaceAll("Y", "y");
}
if (dateFormat.contains("D")) {
dateFormat = dateFormat.replaceAll("D", "d");
}
return new SimpleDateFormat(dateFormat).parse(s);
}
';
I found the tips on this blog post http://javatechniques.com/blog/dateformat-and-simpledateformat-examples/ helpful in figuring out how to translate the Oracle date formats into SimpleDateFormat's formats.
Yes, H2 doesn't support TO_DATE, it's in 1.4.x roadmap. Instead you can use the EXTRACT function that exists both in Oracle DB and H2.
java.util.Date toDate(String dateTime, String dateFormat) throws Exception {
if (dateFormat.contains("MON")) {
dateFormat = dateFormat.replace("MON", "MMM");
}
if (dateFormat.contains("Y")) {
dateFormat = dateFormat.replaceAll("Y", "y");
}
if (dateFormat.contains("D")) {
dateFormat = dateFormat.replaceAll("D", "d");
}
if (dateFormat.contains("HH")) {
dateFormat = dateFormat.replaceAll("HH", "hh");
}
if (dateFormat.contains("hh24")) {
dateFormat = dateFormat.replaceAll("hh24", "hh");
}
if (dateFormat.contains("MI") || dateFormat.contains("mi")) {
dateFormat = dateFormat.replaceAll("MI", "mi").replaceAll("mi", "mm");
}
if (dateFormat.contains("SS")) {
dateFormat = dateFormat.replaceAll("SS", "ss");
}
return new SimpleDateFormat(dateFormat).parse(dateTime);
}
Or you can define your own TO_DATE like
CREATE ALIAS TO_DATE AS $$
java.util.Date to_date(String value, String format) throws java.text.ParseException {
java.text.DateFormat dateFormat = new java.text.SimpleDateFormat(format);
return dateFormat.parse(value);
}
$$;
see http://www.h2database.com/html/features.html#user_defined_functions