I am trying to query hive using java.sql.PreparedStatement and getting an empty result set, Same query giving proper resultset when executed using java.sql.Statement. I am using hive jdbc 1.2.2 jar and hives server is in Hortonworks hdp stack.
Yes, it does:
public class HivePreparedStatement extends HiveStatement implements java.sql.PreparedStatement
As can be seen, internally Hive does implement the JDBC interface PreparedStatement and thus, the driver supports this JDBC feature.
For reference see: https://hive.apache.org/javadocs/r1.2.2/api/org/apache/hive/jdbc/HivePreparedStatement.html
Hope it helps.
Only formally.
https://github.com/apache/hive/blob/ab4c53de82d4aaa33706510441167f2df55df15e/jdbc/src/java/org/apache/hive/jdbc/HivePreparedStatement.java#L116
private String updateSql(String sql, HashMap<Integer, String> parameters) throws SQLException {
List<String> parts = this.splitSqlStatement(sql);
StringBuilder newSql = new StringBuilder((String)parts.get(0));
for(int i = 1; i < parts.size(); ++i) {
if (!parameters.containsKey(i)) {
throw new SQLException("Parameter #" + i + " is unset");
}
newSql.append((String)parameters.get(i));
newSql.append((String)parts.get(i));
}
return newSql.toString();
}
Related
i want use this query 'ANALYZE TABLE {tableName}' but i think mybatis supports only CRUD.
how to use 'ANALYZE TABLE' in mybatis?
Just declare it as a normal select and specify Map as the return type.
#Select("analyze table ${tableName}")
Map<String, Object> analyzeTable(String tableName);
#Test
public void testAnalyzeTable() {
try (SqlSession sqlSession = sqlSessionFactory.openSession()) {
Mapper mapper = sqlSession.getMapper(Mapper.class);
Map<String, Object> result = mapper.analyzeTable("users");
assertEquals("test.users", result.get("Table"));
assertEquals("analyze", result.get("Op"));
assertEquals("status", result.get("Msg_type"));
assertEquals("OK", result.get("Msg_text"));
}
}
Tested using...
MariaDB 10.4.10
MariaDB Connector/J 2.5.4
I used Elasticsearch Connector as a Sink to insert data into Elasticsearch (see : https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/connectors/elasticsearch.html).
But, I did not found any connector to get data from Elasticsearch as source.
Is there any connector or example to use Elasticsearch documents as source in a Flink pipline?
Regards,
Ali
I don't know of an explicit ES source for Flink. I did see one user talking about using elasticsearch-hadoop as a HadoopInputFormat with Flink, but I don't know if that worked for them (see their code).
I finaly defined a simple read from ElasticSearch function
public static class ElasticsearchFunction
extends ProcessFunction<MetricMeasurement, MetricPrediction> {
public ElasticsearchFunction() throws UnknownHostException {
client = new PreBuiltTransportClient(settings)
.addTransportAddress(new TransportAddress(InetAddress.getByName("YOUR_IP"), PORT_NUMBER));
}
#Override
public void processElement(MetricMeasurement in, Context context, Collector<MetricPrediction> out) throws Exception {
MetricPrediction metricPrediction = new MetricPrediction();
metricPrediction.setMetricId(in.getMetricId());
metricPrediction.setGroupId(in.getGroupId());
metricPrediction.setBucket(in.getBucket());
// Get the metric measurement from Elasticsearch
SearchResponse response = client.prepareSearch("YOUR_INDEX_NAME")
.setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
.setQuery(QueryBuilders.termQuery("YOUR_TERM", in.getMetricId())) // Query
.setPostFilter(QueryBuilders.rangeQuery("value").from(0L).to(50L)) // Filter
.setFrom(0).setSize(1).setExplain(true)
.get();
SearchHit[] results = response.getHits().getHits();
for(SearchHit hit : results){
String sourceAsString = hit.getSourceAsString();
if (sourceAsString != null) {
ObjectMapper mapper = new ObjectMapper();
MetricMeasurement obj = mapper.readValue(sourceAsString, MetricMeasurement.class);
obj.getMetricId();
metricPrediction.setPredictionValue(obj.getValue());
}
}
out.collect(metricPrediction);
}
}
Hadoop Compatibility + Elasticsearch Hadoop
https://github.com/cclient/flink-connector-elasticsearch-source
I would like to create a Spark SQL DataFrame from the results of a query performed over CSV data (on HDFS) with Apache Drill. I successfully configured Spark SQL to make it connect to Drill via JDBC:
Map<String, String> connectionOptions = new HashMap<String, String>();
connectionOptions.put("url", args[0]);
connectionOptions.put("dbtable", args[1]);
connectionOptions.put("driver", "org.apache.drill.jdbc.Driver");
DataFrame logs = sqlc.read().format("jdbc").options(connectionOptions).load();
Spark SQL performs two queries: the first one to get the schema, and the second one to retrieve the actual data:
SELECT * FROM (SELECT * FROM dfs.output.`my_view`) WHERE 1=0
SELECT "field1","field2","field3" FROM (SELECT * FROM dfs.output.`my_view`)
The first one is successful, but in the second one Spark encloses fields within double quotes, which is something that Drill doesn't support, so the query fails.
Did someone managed to get this integration working?
Thank you!
you can add JDBC Dialect for this and register the dialect before using jdbc connector
case object DrillDialect extends JdbcDialect {
def canHandle(url: String): Boolean = url.startsWith("jdbc:drill:")
override def quoteIdentifier(colName: java.lang.String): java.lang.String = {
return colName
}
def instance = this
}
JdbcDialects.registerDialect(DrillDialect)
This is how the accepted answer code looks in Java:
import org.apache.spark.sql.jdbc.JdbcDialect;
public class DrillDialect extends JdbcDialect {
#Override
public String quoteIdentifier(String colName){
return colName;
}
public boolean canHandle(String url){
return url.startsWith("jdbc:drill:");
}
}
Before creating the Spark Session register the Dialect:
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.jdbc.JdbcDialects;
public static void main(String[] args) {
JdbcDialects.registerDialect(new DrillDialect());
SparkSession spark = SparkSession
.builder()
.appName("Drill Dialect")
.getOrCreate();
//More Spark code here..
spark.stop();
}
Tried and tested with Spark 2.3.2 and Drill 1.16.0. Hope it helps you too!
How do I set batch size in spring JDBC batch update to improve performance?
Listed below is my code snippet.
public void insertListOfPojos(final List<Student> myPojoList) {
String sql = "INSERT INTO " + "Student " + "(age,name) " + "VALUES "
+ "(?,?)";
try {
jdbcTemplateObject.batchUpdate(sql,
new BatchPreparedStatementSetter() {
#Override
public void setValues(PreparedStatement ps, int i)
throws SQLException {
Student myPojo = myPojoList.get(i);
ps.setString(2, myPojo.getName());
ps.setInt(1, myPojo.getAge());
}
#Override
public int getBatchSize() {
return myPojoList.size();
}
});
} catch (Exception e) {
System.out.println("Exception");
}
}
I read that with Hibernate you can provide your batch size in the
configuration xml.
For example,
<property name="hibernate.jdbc.batch_size" value="100"/>.
Is there something similar in Spring's jdbc?
There is no option for jdbc that looks like Hibernate; I think you have to get a look to specif RDBMS vendor driver options when preparing connection string.
About your code you have to use
BatchPreparedStatementSetter.getBatchSize()
or
JdbcTemplate.batchUpdate(String sql, final Collection<T> batchArgs, final int batchSize, final ParameterizedPreparedStatementSetter<T> pss)
if you use JDBC directly, you decide yourself how much statements are used in one commit, while using one of the provided JDBCWriters you decide the batch* size with the configured commit-rate
*afaik the actual spring version uses the prepared statement batch methods under the hood, see https://github.com/SpringSource/spring-framework/blob/master/spring-jdbc/src/main/java/org/springframework/jdbc/core/JdbcTemplate.java#L549
i am using JDBC template for getting data from database in Spring MVC.
my query is:
SELECT count(A.MEETING_ID),ITEM_TBL.REG_EMAIL FROM ITEM_TBL,MEETINGS_TBL WHERE ITEM_TBL.MEETING_ID=MEETINGS_TBL.MEETING_ID
GROUP BY ITEM_TBL.REG_EMAIL
this is returning rows like:
11 nishant#gmail.com
12 abhilasha#yahoo.com
13 shiwani#in.com
i want to store these value into Hash MAP. Can you please help how can i do this using JDBC TEMPLATE?
Thanks
You need ResultExtractor.
You can achieve that using below code.
String sql = "SELECT count(A.MEETING_ID),ITEM_TBL.REG_EMAIL FROM ITEM_TBL,MEETINGS_TBL WHERE ITEM_TBL.MEETING_ID=MEETINGS_TBL.MEETING_ID
GROUP BY ITEM_TBL.REG_EMAIL";
ResultExtractor mapExtractor = new ResultSetExtractor() {
public Object extractData(ResultSet rs) throws SQLException {
Map<String, String> mapOfKeys = new HashMap<String, String>();
while (rs.next()) {
String key = rs.getString("MEETING_ID");
String obj = rs.getString("REG_EMAIL");
/* set the business object from the resultset */
mapOfKeys.put(key, obj);
}
return mapOfKeys;
}
};
Map map = (HashMap) jdbcTemplate.query(sql.toString(), mapExtractor);