I understand that the OTHER data type can store any Serializable object. However when I try to store an instance of String or Boolean it fails with an exception.
Is this a misunderstanding on my part, or a bug in H2?
Here's repro code.
import org.h2.jdbc.JdbcSQLException;
import java.io.Serializable;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.SQLException;
public class ScratchSpace {
public static void main(String[] args) throws SQLException, ClassNotFoundException {
Class.forName("org.h2.Driver");
Connection conn = DriverManager.getConnection("jdbc:h2:~/test");
conn.createStatement().execute("drop table if exists test;");
conn.createStatement().execute("create table test (key VARCHAR, value OTHER)");
testInsert(conn, "key1", "foobar");
testInsert(conn, "key2", Boolean.TRUE);
testInsert(conn, "key3", new MyClass("foobar"));
conn.close();
}
private static void testInsert(Connection conn, String key, Serializable value) throws SQLException {
try (PreparedStatement statement = conn.prepareStatement("insert into test (key, value) values (?, ?)")) {
statement.setString(1, key);
statement.setObject(2, value);
statement.executeUpdate();
System.out.println("Insert of value={" + value + "} succeeded");
} catch (JdbcSQLException e) {
System.out.println("Insert of value={" + value + "} failed: " + e.getMessage());
e.printStackTrace();
}
}
public static class MyClass implements Serializable {
private final String s;
public MyClass(String s) {
this.s = s;
}
}
}
Here's a stack trace when I try to store String in the OTHER column:
org.h2.jdbc.JdbcSQLException: Hexadecimal string contains non-hex character: "foobar"; SQL statement:
insert into test (key, value) values (?, ?) -- (?1, ?2) [90004-188]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
at org.h2.message.DbException.get(DbException.java:179)
at org.h2.message.DbException.get(DbException.java:155)
at org.h2.util.StringUtils.convertHexToBytes(StringUtils.java:983)
at org.h2.value.Value.convertTo(Value.java:867)
at org.h2.table.Column.convert(Column.java:148)
at org.h2.command.dml.Insert.insertRows(Insert.java:143)
at org.h2.command.dml.Insert.update(Insert.java:114)
at org.h2.command.CommandContainer.update(CommandContainer.java:78)
at org.h2.command.Command.executeUpdate(Command.java:254)
at org.h2.jdbc.JdbcPreparedStatement.executeUpdateInternal(JdbcPreparedStatement.java:157)
at org.h2.jdbc.JdbcPreparedStatement.executeUpdate(JdbcPreparedStatement.java:143)
at com.barbarysoftware.pokercopilot.ScratchSpace.testInsert(ScratchSpace.java:29)
at com.barbarysoftware.pokercopilot.ScratchSpace.main(ScratchSpace.java:19)
Related
getting this error for the below code
java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
at oracle.jdbc.driver.OracleSql.main(OracleSql.java:1717)
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.Statement;
public class Z1 {
public static void main(String[] args) throws Exception
{
// String query = "select * from Employee";
//String sql = "CREATE TABLE people(\r\n"+ "id INT NOT NULL PRIMARY KEY,\r\n"+ "name VARCHAR2(50) \r\n"+ ");";
Class.forName("oracle.jdbc.driver.OracleDriver");
//Class.forName("com.mysql.jdbc.Driver");
Connection con = DriverManager.getConnection("jdbc:oracle:thin:#localhost:1521:","system","palak");
Statement stmt = con.createStatement();
String sql = "insert into test1 values(100)";
stmt.execute(sql);
System.out.println("dOne");
}
}
I have a custom DB Appender which extends the logback DBAppender. My application has 100 parallel threads. These threads also log the events into database through the custom db appender while processing the requests.
The performance of the logback is so slow that with the logs disabled, the execution finishes in 20 mins. Whereas with the logs enabled, it takes almost 5 hours to finish most of the time waiting for the logs to be written to proceed to the next step as my logging is synchronous.
At first I thought the slow performance could be due to the synchronized doAppend method of logback appender. However, I later noticed that DB appender extends the
UnsynchronizedAppenderBase(public abstract class DBAppenderBase<E>
/* */ extends UnsynchronizedAppenderBase<E>).
Can anyone help me understand what could be the reason for this poor performance?
My DB Appender:
package com.test;
import java.io.StringReader;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.Map;
import org.omg.CORBA.TRANSACTION_UNAVAILABLE;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import ch.qos.logback.classic.db.DBAppender;
import ch.qos.logback.classic.spi.ILoggingEvent;
public class CustomDBAppender extends DBAppender {
final static Logger logger = LoggerFactory.getLogger("com.test");
protected String insertPropertiesSQL;
protected String insertExceptionSQL;
protected String insertSQL;
private CustomDBNameResolver dbNameResolver;
// default constructor
public CustomDBAppender() {
logger.debug("DB Appender instantiated");
}
// set dbNameResolver for getting table information
public void setCsiDbNameResolver(CustomDBNameResolver dbNameResolver) {
this.dbNameResolver = dbNameResolver;
}
#Override
public void start() {
super.start();
if (dbNameResolver == null)
dbNameResolver = new CustomDBNameResolver();
insertSQL = CustomSQLBuilder.buildInsertSQL(dbNameResolver);
}
#Override
protected void subAppend(ILoggingEvent event, Connection connection, PreparedStatement insertStatement)
throws Throwable {
logEvents(insertStatement, event);
// System.out.println("=== INSERT STATEMENT === " +
// insertStatement.toString());
int updateCount = -1;
try {
updateCount = insertStatement.executeUpdate();
} catch (Exception e) {
logger.error(" ERROR ");
e.printStackTrace();
}
logger.info(" updateCount = " + updateCount);
if (updateCount != 1) {
logger.error(" ERROR IT IS ");
addWarn("Failed to insert loggingEvent");
}
}
protected void secondarySubAppend(ILoggingEvent event, Connection connection, long eventId) throws Throwable {
}
// #override
private void logEvents(PreparedStatement stmt, ILoggingEvent event) throws SQLException {
logData(event, stmt);
}
private void logData(ILoggingEvent event, PreparedStatement stmt) throws SQLException {
try {
if (null != event && null != logMsgArgs && (event.getLoggerName().indexOf("com.test.beans.") != -1)) {
payload = null == logMsgArgs[0] ? "NA"
: (logMsgArgs[0].toString()).substring(0,
(logMsgArgs[0].toString().length() >= 4000 ? 3999 : logMsgArgs[0].toString().length()));
msgHeader = null == logMsgArgs[1] ? "NA"
: (logMsgArgs[1].toString()).substring(0,
(logMsgArgs[1].toString().length() >= 4000 ? 3999 : logMsgArgs[1].toString().length()));
currentStage = null == logMsgArgs[2] ? -1 : Integer.parseInt(logMsgArgs[2].toString());
currentEvent = null == logMsgArgs[3] ? null : logMsgArgs[3].toString();
trxId = null == logMsgArgs[4] ? null : logMsgArgs[4].toString();
batchId = null == logMsgArgs[5] ? null : logMsgArgs[5].toString();
headerId = null == logMsgArgs[6] ? null : logMsgArgs[6].toString();
}
stmt.setString(1, logLevel);
stmt.setObject(2, trxId);
stmt.setString(3, logMessage);
stmt.setDate(4, getSqlDate());
stmt.setString(5, event.getLoggerName());
stmt.setString(6, event.getThreadName());
stmt.setString(7, event.getLoggerContextVO().getName());
stmt.setString(8, logMessage.substring(0, logMessageLength));
stmt.setString(9, payload);
stmt.setString(10, msgHeader);
stmt.setInt(11, currentStage);
stmt.setString(12, currentEvent);
stmt.setString(13, batchId);
stmt.setString(14, headerId);
} catch (Exception e) {
// TODO Auto-generated catch block
System.out.println("Exception while printing log:" + e.getMessage());
e.printStackTrace();
} finally {
}
}
private Object nvl(Object input) {
if (null == input) {
return null;
}
return input;
}
private java.sql.Date getSqlDate() {
Date now = new Date();
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSSX");
String nowDateStr = sdf.format(now);
logger.info("nowDateStr = " + nowDateStr);
java.sql.Date toDB = null;
try {
toDB = new java.sql.Date(sdf.parse(nowDateStr).getTime());
logger.info("toDB = " + toDB);
} catch (ParseException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return toDB;
}
#Override
protected String getInsertSQL() {
return insertSQL;
}
protected void insertProperties(Map<String, String> mergedMap, Connection connection, long eventId)
throws SQLException {
}
}
/* End of appender */
I have populated a Hbase table with rowid and vrious information pertaining to tweet such as clean-text,url,hashtag etc. as follows
902221655086211073 column=clean-tweet:clean-text-cta, timestamp=1514793745304, value=democrat mayor order hurricane harvey stand houston
However while populating I noticed that the some of the rows are empty like
902487280543305728 column=clean-tweet:clean-text-cta, timestamp=1514622371008, value=
Now how do I find the count of rows that are having data?
Please help me in this
There is no provision to do this in HBase shell as of now. May be you can use a simple code like this to get a number of records with no value for the provided column qualifier.
CountAndFilter [tableName] [columnFamily] [columnQualifier]
import java.io.IOException;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Admin;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.client.Table;
import org.apache.hadoop.hbase.util.Bytes;
public class CountAndFilter {
private static Connection conn;
private static int recordsWithoutValue = 0;
public static Admin getConnection() throws IOException {
if (conn == null) {
conn = ConnectionFactory.createConnection(HBaseConfiguration.create());
}
return conn.getAdmin();
}
public static void main(String args[]) throws IOException {
getConnection();
scan(args[0], args[1], args[2]);
System.out.println("Records with empty value : " + recordsWithoutValue);
}
public static void scan(String tableName, String columnFamily, String columnQualifier) throws IOException {
Table table = conn.getTable(TableName.valueOf(tableName));
ResultScanner rs = table.getScanner(new Scan().addColumn(Bytes.toBytes(columnFamily), Bytes.toBytes(columnQualifier)));
Result res = null;
try {
while ((res = rs.next()) != null) {
if (res.containsEmptyColumn(Bytes.toBytes(columnFamily), Bytes.toBytes(columnQualifier))){
recordsWithoutValue++;
}
}
} finally {
rs.close();
}
}
}
Does somebody implemented confluent-kafka messages deserializer to consume kafka messages by spring "#KafkaListener"-s ?
Here is my answer, which I've implemented based on: "io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer"
import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.Arrays;
import java.util.Map;
import javax.xml.bind.DatatypeConverter;
import org.apache.avro.Schema;
import org.apache.avro.io.DatumReader;
import org.apache.avro.io.DecoderFactory;
import org.apache.avro.specific.SpecificDatumReader;
import org.apache.avro.specific.SpecificRecordBase;
import org.apache.kafka.common.errors.SerializationException;
import org.apache.kafka.common.serialization.Deserializer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class AvroConfluentDeserializer<T extends SpecificRecordBase> implements Deserializer<T> {
private static final Logger LOG = LoggerFactory.getLogger(AvroConfluentDeserializer.class);
protected static final byte MAGIC_BYTE = 0x0;
protected static final int idSize = 4;
private final DecoderFactory decoderFactory = DecoderFactory.get();
protected final Class<T> targetType;
public AvroConfluentDeserializer(Class<T> targetType) {
this.targetType = targetType;
}
#Override
public void close() {
// No-op
}
#Override
public void configure(Map<String, ?> arg0, boolean arg1) {
// No-op
}
#Override
public T deserialize(String topic, byte[] data) {
try {
T result = null;
if (data != null) {
LOG.info("data='{}'", DatatypeConverter.printHexBinary(data));
result = (T) deserializePayload(data, targetType.newInstance().getSchema());
LOG.info("deserialized data='{}'", result);
}
return result;
} catch (Exception ex) {
throw new SerializationException(
"Can't deserialize data '" + Arrays.toString(data) + "' from topic '" + topic + "'", ex);
}
}
protected T deserializePayload(byte[] payload, Schema schema) throws SerializationException {
int id = -1;
try {
ByteBuffer buffer = getByteBuffer(payload);
id = buffer.getInt();
int length = buffer.limit() - 1 - idSize;
int start = buffer.position() + buffer.arrayOffset();
DatumReader<T> reader = new SpecificDatumReader<T>(schema);
return reader.read(null, decoderFactory.binaryDecoder(buffer.array(), start, length, null));
} catch (IOException | RuntimeException e) {
throw new SerializationException("Error deserializing Avro message for id " + id, e);
}
}
private ByteBuffer getByteBuffer(byte[] payload) {
ByteBuffer buffer = ByteBuffer.wrap(payload);
if (buffer.get() != MAGIC_BYTE) {
throw new SerializationException("Unknown magic byte!");
}
return buffer;
}
}
I am trying to run a simple Java program which fetches data from the oracle database and display it. I connected the oracle database. Here is my code:
DataHandler Class:
import java.sql.Connection;
import java.sql.SQLException;
import java.sql.Statement;
import java.sql.ResultSet;
import oracle.jdbc.pool.OracleDataSource;
public class DataHandler {
public DataHandler() {
super();
}
String jdbcUrl = "jdbc:oracle:thin:#localhost:1521:ORCL";
//I already added the above line but still getting error.
String userid = "scott";
String password = "tiger";
Connection conn;
public void getDBConnection() throws SQLException{
OracleDataSource ds;
ds = new OracleDataSource();
ds.setUser(jdbcUrl);
conn = ds.getConnection(userid,password);
}
Statement stmt;
ResultSet rset;
String query;
String sqlString;
public ResultSet getAllEmployees() throws SQLException{
getDBConnection();
stmt = conn.createStatement(ResultSet.TYPE_SCROLL_SENSITIVE, ResultSet.CONCUR_READ_ONLY);
query = "SELECT * FROM emp ORDER BY empno";
System.out.println("\nExecuting query: " + query);
rset = stmt.executeQuery(query);
return rset;
}
}
and the JavaClient Class as
JavaCLient CLass:
import java.sql.ResultSet;
public class JavaClient {
public JavaClient() {
super();
}
public static void main(String[] args) throws Exception{
DataHandler datahandler = new DataHandler();
ResultSet rset = datahandler.getAllEmployees();
while (rset.next()) {
System.out.println(rset.getInt(1) + " " +
rset.getString(2) + " " +
rset.getString(3) + " " +
rset.getString(4));
}
}
}
I get no compilation error but while running it I get following exception error
Error:
Exception in thread "main" java.sql.SQLException: Invalid Oracle URL specified: OracleDataSource.makeURL
at oracle.jdbc.pool.OracleDataSource.makeURL(OracleDataSource.java:1277)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:185)
at student_attendence_iem.DataHandler.getDBConnection(DataHandler.java:22)
at student_attendence_iem.DataHandler.getAllEmployees(DataHandler.java:31)
at student_attendence_iem.JavaClient.main(JavaClient.java:9)
Process exited with exit code 1.
Please help me. Thanks in advance. :)
You have not set URL of your database.
Add setURL(url) method which takes URL of database as parameter. Below is the code.
OracleDataSource ds;
ds = new OracleDataSource();
ds.setURL(jdbcUrl);
Also, with ds.setUser(jdbcUrl); you are trying to setUser with the URL of database which is wrong.
You don't have to setUser like this as you are already doing that in the following line of code conn = ds.getConnection(userid,password);