JUL to SLF4j with oracle jdbc driver - oracle

I am trying to redirect java.util.logging messages logged by the oracle jdbc driver and the oracle ucp (universal connection pool) library, but not able to do so.
The messages logged by my application using JUL get logged but the ones logged by the oracle libraries are not getting logged.
My intent here is redirect JUL messages to Logback to have more fine grained logging through configuration. i.e. enabling logging at class level instead of package level which I assume is not possible in JUL configuration (java.util.config file).
Below is the sample test code. Do you have any suggestions on the above two points?
import oracle.ucp.jdbc.PoolDataSourceImpl;
import org.slf4j.bridge.SLF4JBridgeHandler;
import java.sql.Connection;
import java.sql.SQLException;
import java.util.Properties;
import java.util.logging.Logger;
public class JavaUtilToSlf4jApp {
private static Logger logger = Logger.getLogger(JavaUtilToSlf4jApp.class.getName());
public static void main(String[] args) {
SLF4JBridgeHandler.install();
startConnectionPool();
logger.info("Info Message");
}
private static void startConnectionPool() {
PoolDataSourceImpl pds = new PoolDataSourceImpl();
try {
pds.setConnectionPoolName("Pool Name");
pds.setConnectionFactoryClassName("oracle.jdbc.pool.OracleDataSource");
pds.setConnectionFactoryProperties(getOracleDataSourceProperties());
pds.setDataSourceName("Datasource Name");
pds.setServerName("machine-name");
pds.setPortNumber(1521);
pds.setMinPoolSize(1);
pds.setMaxPoolSize(1);
pds.setMaxIdleTime(1800);
pds.setValidateConnectionOnBorrow(true);
pds.setUser("user");
pds.setPassword("password");
pds.startPool();
} catch (SQLException e) {
throw new RuntimeException("Cannot create project datasource ", e);
}
try {
Connection connection = pds.getConnection();
} catch (SQLException e) {
throw new RuntimeException(e);
}
logger.info("Connection established");
}
private static Properties getOracleDataSourceProperties() {
Properties p = new Properties();
p.put("driverType", "oci");
p.put("networkProtocol", "tcp");
p.put("serviceName", "servicename");
return p;
}
}

If it helps. I took this approach in a Spring boot application, using the ojdbc_g version of the driver jar file:
Set the system property: System.setProperty("oracle.jdbc.Trace", "true").
Initialize the Oracle driver/datasource with whatever mechanism you have (manual, Spring, conn pool, etc.).
Once the datasource has been initialized, programmatically set the level for the "oracle.jdbc" JUL logger:
Logger ol = Logger.getLogger("oracle.jdbc");
ol.setLevel(Level.FINE);
The remaining part would be to figure out how to map the JUL logging levels to SLF4J logging levels.
Worked inside my Spring boot application. Cheers!

This was a complete mess, and after I got some logging, it still didn't work that well.
First off, you definitely need to be using the "_g" version of the jar, which is the jar that was compiled with the debug option, and logging turned on. If you don't use this driver, it'll be like getting blood from a stone.
Second, you need to add the java parameter -Doracle.jdbc.Trace=true.
Third, you need to define the package in your logging file (like logback.xml):
<logger name="oracle" level="INFO" additivity="false">
<appender-ref ref="SERVER_FILE" />
</logger>
This got me the following results:
11:17:45.393 [UCP-worker-thread-3] INFO oracle.jdbc.driver - SQL: select count(*) from mytable
11:17:45.956 [main] INFO oracle.jdbc.driver - SQL:
select myfield
from mytable
where myotherfield='myvalue'
11:17:46.159 [main] INFO oracle.jdbc.driver - SQL: begin :1 := dbms_pickler.get_type_shape(:2,:3,:4,:5,:6,:7); end;
Please let me know if you resolved your issues, how, and if any of this makes sense. There's poor and conflicting comments all of the web on this specific issue.

Related

Add logs to spans using OTEL instrumentation with Jaegar backend

At present, Open Telemetry (OTEL) spans have no mechanism to add logs as found in implementations such as Jaegar.
So is there a workaround to add application logs to a span?
As we saw here, jaegar backend interprets OTEL exceptions in way where the contents of the exception are put in as Logs in the associated span.
Now, exceptions are a form of events, and it seems jaegar backend interprets OTEL events as Logs. So we can replicate this behavior by:
Creating a custom log appender
Inside, create an OTEL event and populate logging details in it.
Add the event to the current span.
This span will be interpreted by jaegar backend in a way where all the events are put in as individual log items in that span.
Custom Log Appender
Below is a basic LogAppender i wrote based on SpanLogsAppender.java from the spring-cloud project.
import ch.qos.logback.classic.Level;
import ch.qos.logback.classic.spi.ILoggingEvent;
import ch.qos.logback.classic.spi.IThrowableProxy;
import ch.qos.logback.classic.spi.ThrowableProxy;
import ch.qos.logback.core.AppenderBase;
import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.api.common.AttributesBuilder;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.StatusCode;
public class SpanLogsAppender extends AppenderBase<ILoggingEvent> {
/**
* This is called only for configured levels.
* It will not be executed for DEBUG level if root logger is INFO.
*/
#Override
protected void append(ILoggingEvent event) {
final Span currentSpan = Span.current();
AttributesBuilder builder = Attributes.builder();
if (currentSpan != null) {
builder.put("logger", event.getLoggerName())
.put("level", event.getLevel().toString())
.put("message", event.getFormattedMessage());
currentSpan.addEvent("LogEvent", builder.build());
if (Level.ERROR.equals(event.getLevel())) {
currentSpan.setStatus(StatusCode.ERROR);
}
IThrowableProxy throwableProxy = event.getThrowableProxy();
if (throwableProxy instanceof ThrowableProxy) {
Throwable throwable = ((ThrowableProxy)throwableProxy).getThrowable();
if (throwable != null) {
currentSpan.recordException(throwable);
}
}
}
}
}
My local versions:
spring boot : 2.5.1
io.opentelemetry.opentelemetry-api : 1.2.0
jaegar backend: 1.18 (windows)

How to fix 'java.lang.NoClassDefFoundError: org/apache/tinkerpop/gremlin/process/traversal/dsl/graph/GraphTraversalSource$GraphTraversalSourceStub'?

I am trying to initialize an in-memory graph using TinkerGraph.
Firstly, i have defined the bean in my context xml file and tried to initialise the TinkerGraph.
My intention is to unit test the classes that i have created for forming the gremlin queries, the end queries that i get from these classes are in the form of a string, so in order to execute them through the TinkerGraph, i have the used the approach given in the following post:
Get Gremlin query as a String and execute it in java without submitting it to the GremlinServer
I would also like to know whether the approach that i have taken is the preferred approach, for running the gremlin queries as part of the unit testing?
Following are the dependencies i have included in the pom.xml:
<dependency>
<groupId>org.apache.tinkerpop</groupId>
<artifactId>tinkergraph-gremlin</artifactId>
<version>3.2.4</version>
</dependency>
<dependency>
<groupId>org.apache.tinkerpop</groupId>
<artifactId>gremlin-groovy</artifactId>
<version>3.0.2-incubating</version>
</dependency>
EmbeddedGremlinQueryEngine is as follows:
import org.apache.tinkerpop.gremlin.driver.Result;
import org.apache.tinkerpop.gremlin.driver.ResultSet;
import org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine;
import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
import org.apache.tinkerpop.gremlin.structure.Graph;
import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import javax.script.Bindings;
import javax.script.ScriptEngine;
import javax.script.ScriptException;
import java.util.List;
public class UcsEmbeddedGremlinQueryEngine implements GremlinEngine{
private static final Logger logger = LoggerFactory.getLogger(UcsEmbeddedGremlinQueryEngine.class);
private GraphTraversalSource graphTraversalSource = null;
private Graph graph = null;
private ScriptEngine engine = null;
private Bindings bindings = null;
public UcsEmbeddedGremlinQueryEngine() {
graph = TinkerGraph.open();
graphTraversalSource = graph.traversal();
engine = new GremlinGroovyScriptEngine();
bindings = engine.createBindings();
bindings.put("g", graphTraversalSource);
}
public void shutdown() throws Exception {
if (graph != null){
graph.close();
}
logger.info("TinkerGraph shutdown complete.");
}
#Override
public List<Result> query(String query) {
List<Result> res = null;
try {
ResultSet results = (ResultSet) engine.eval(query, bindings);
res = results.all().join();
for (Result r : res) {
System.out.println("result: " + r + '\n');
}
} catch (ScriptException e) {
e.printStackTrace();
}
return res;
}
// This function reads the initScript and run them as gremlin queries.
public synchronized void initialize() {
logger.debug("Initializing embedded TinkerGraph. This will only take a few seconds....");
//TODO include the execution of queries as part of initialisation
}
}
Stack trace is as follows:
java.lang.NoClassDefFoundError: org/apache/tinkerpop/gremlin/process/traversal/dsl/graph/GraphTraversalSource$GraphTraversalSourceStub
at org.apache.tinkerpop.gremlin.groovy.loaders.StepLoader.load(StepLoader.groovy:54)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:236)
at org.apache.tinkerpop.gremlin.groovy.loaders.GremlinLoader.load(GremlinLoader.groovy:28)
at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.<init>(GremlinGroovyScriptEngine.java:189)
at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.<init>(GremlinGroovyScriptEngine.java:172)
at com.intuit.gro.mcsdata.gemlinengine.UcsEmbeddedGremlinQueryEngine.<init>(UcsEmbeddedGremlinQueryEngine.java:28)
EmbeddedGremlinQueryEngine is defined as a bean in the xml file, when the bean is loaded i get the error as
Constructor threw exception; nested exception is java.lang.NoClassDefFoundError: org/apache/tinkerpop/gremlin/process/traversal/dsl/graph/GraphTraversalSource$GraphTraversalSourceStub
I don't understand how the GraphTraversalSourceStub comes into picture during initialization, I was not able to find any information about it.
Any help would be appreciated.
I think your problem is that you're:
using really really old versions of TinkerPop
the old versions you're using are probably incompatible
I'm not sure if you have a reason for using 3.2.4, but if so, make sure gremlin-groovy is also 3.2.4. Note that the 3.2.x line of code is largely not maintained at this point with the last release being 3.2.11 about 6 months ago. If you are developing a new application then I highly recommend that you simply utilize the latest version of 3.4.2 which released a few weeks ago.
As for your testing approach, I suppose that's fine. If you have test Gremlin strings then you really don't have much other choice in the matter, short of using Gremlin Server. Obviously providing a test harness for GremlinGroovyScriptEngine is a lot easier to do.
For version 3.4.0 you need at least the following classpath.
set cp=%cp%;C:\pathToM2Repo\org\apache\tinkerpop\gremlin-core\3.4.0\gremlin-core-3.4.0.jar
set cp=%cp%;C:\pathToM2Repo\org\apache\tinkerpop\gremlin-driver\3.4.0\gremlin-driver-3.4.0.jar
set cp=%cp%;C:\pathToM2Repo\org\apache\tinkerpop\gremlin-groovy\3.4.0\gremlin-groovy-3.4.0.jar
set cp=%cp%;C:\pathToM2Repo\org\apache\tinkerpop\gremlin-server\3.4.0\gremlin-server-3.4.0.jar
set cp=%cp%;C:\pathToM2Repo\org\apache\tinkerpop\gremlin-shaded\3.4.0\gremlin-shaded-3.4.0.jar
set cp=%cp%;C:\pathToM2Repo\org\apache\tinkerpop\tinkergraph-gremlin\3.4.0\tinkergraph-gremlin-3.4.0.jar
set cp=%cp%;C:\pathToM2Repo\org\apache\tinkerpop\tinkerpop\3.4.0\tinkerpop-3.4.0.jar
set cp=%cp%;C:\pathToM2Repo\commons-configuration\commons-configuration\1.10\commons-configuration-1.10.jar
set cp=%cp%;C:\pathToM2Repo\org\slf4j\slf4j-api\1.7.25\slf4j-api-1.7.25.jar
set cp=%cp%;C:\pathToM2Repo\org\apache\logging\log4j\log4j-slf4j-impl\2.11.1\log4j-slf4j-impl-2.11.1.jar
set cp=%cp%;C:\pathToM2Repo\org\apache\logging\log4j\log4j-api\2.11.1\log4j-api-2.11.1.jar
set cp=%cp%;C:\pathToM2Repo\org\apache\logging\log4j\log4j-core\2.11.1\log4j-core-2.11.1.jar
set cp=%cp%;C:\pathToM2Repo\org\codehaus\groovy\groovy\2.5.4\groovy-2.5.4-indy.jar
set cp=%cp%;C:\pathToM2Repo\org\codehaus\groovy\groovy-json\2.5.4\groovy-json-2.5.4-indy.jar
set cp=%cp%;C:\pathToM2Repo\org\codehaus\groovy\groovy-xml\2.5.5\groovy-xml-2.5.5.jar
set cp=%cp%;C:\pathToM2Repo\org\codehaus\groovy\groovy-templates\2.5.5\groovy-templates-2.5.5.jar
set cp=%cp%;C:\pathToM2Repo\org\javatuples\javatuples\1.2\javatuples-1.2.jar
set cp=%cp%;C:\pathToM2Repo\commons-collections\commons-collections\3.2.2\commons-collections-3.2.2.jar
set cp=%cp%;C:\pathToM2Repo\io\netty\netty-all\4.1.31.Final\netty-all-4.1.31.Final.jar

Oracle Database JDBC driver cannot read wallet file from Spark

Objective
I'm trying to write to Oracle's ADWC (basically oracle database) from a Spark application running on Yarn. The only way to connect to this database is by using an Oracle Wallet file, which is basically a Java keystore.
Problem
The problem arises when the JDBC driver tries to read the wallet from HDFS. If I include the hdfs:// prefix the parser in the JDBC driver throws an error and if I don't then it cannot find the file.
Previous Attempts
including the directory in the connect string (prefixed and non) jdbc:oracle:thin:#luigi_low?TNS_ADMIN=/user/spark/wallet_LUIGI
including the directory as an spark.driver.extraJavaOptions with -Doracle.net.tns_admin and -Doracle.net.wallet_location
All the code is on GitHub, and specifically, the error messages are here https://github.com/sblack4/kafka-scala-jdbc/blob/master/ERROR.md
I've got a working example of the same connection here https://github.com/sblack4/scala-jdbc-adwc
help me StackOverflow. you are my only hope
If you need any more clarification don't hesitate :)
update (SparkFiles attempt)
the code is on a separate branch of the same repository, https://github.com/sblack4/kafka-scala-jdbc/tree/sparkfiles
This error message mystifies me as it seems my JDBC library has stopped trying to read the wallet files. It may be unrelated to the previous problem
Exception in thread "main" java.sql.SQLRecoverableException: IO Error: Invalid connection string format, a valid format is: "host:port:sid"
I've deleted the other JDBC libraries from my classpath through Ambari as this error could be related to spark picking up an older version of my JDBC library
Here's some code that will help diagnose what the issues is.
It checks and configures everything required to connect.
JDBC Driver version
JCE Installed
Classpath dependencies
Configures
tns_admin
ssl settings
trust/key stores
This is a slimmed down version of what's in sqldev/sqlcl
import java.security.NoSuchAlgorithmException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.util.Collections;
import java.util.Properties;
import javax.crypto.Cipher;
import oracle.jdbc.OracleConnection;
public class JDBCTest {
public static void fail(String msg){
System.err.println(String.join("", Collections.nCopies(20, "*")));
System.err.println(msg);
System.err.println(String.join("", Collections.nCopies(20, "*")));
System.exit(1);
}
public static void main(String[] args) throws SQLException {
System.out.println("JDBC Driver Version:" + oracle.jdbc.OracleDriver.getDriverVersion());
// Check JDBC Driver Version
if (!oracle.jdbc.OracleDriver.getDriverVersion().startsWith("18.")) {
fail(" DRIVER TOOO OLD!!!");
}
// Check JCE Installed
int maxKeySize = 0;
try {
maxKeySize = Cipher.getMaxAllowedKeyLength("AES");
} catch (NoSuchAlgorithmException e) {
}
if (maxKeySize < 129 ) {
fail(" JCE Policy not unlimited!!!");
}
// Check Classpath
String cp = System.getProperty("java.class.path");
String[] cpFiles = {"ojdbc8.jar","oraclepki.jar","osdt_cert.jar","osdt_core.jar"};
for (String file:cpFiles){
if ( cp.indexOf(file) == -1 ){
fail("CLASSPATH Missing:" + file);
}
}
// Wallet unziped location
String unzippedWalletLocation = "/Users/klrice/workspace/12.2JDBC/wallet";
String conString = "jdbc:oracle:thin:#sqldev_medium";
Properties props = new Properties();
props.setProperty("oracle.net.wallet_location",unzippedWalletLocation);
props.setProperty(OracleConnection.CONNECTION_PROPERTY_THIN_NET_CONNECT_TIMEOUT, "2000");
// unzipped includes a tnsnames.ora
props.setProperty("oracle.net.tns_admin",unzippedWalletLocation);
props.setProperty("javax.net.ssl.trustStore","truststore.jks");
props.setProperty("javax.net.ssl.trustStorePassword","<password>");
props.setProperty("javax.net.ssl.keyStore","keystore.jks");
props.setProperty("javax.net.ssl.keyStorePassword","<password>");
props.setProperty("oracle.net.ssl_server_dn_match","true");
props.setProperty("oracle.net.ssl_version","1.2");
props.setProperty("user", "ADMIN");
props.setProperty("password", "<password>");
try {
// now Connect
Connection conn = DriverManager.getConnection(conString,props);
} catch (Exception e){
e.printStackTrace();
fail(e.getLocalizedMessage());
}
System.out.println("SUCCESS!!");
}
}
Are you using 18.3 JDBC drivers? Passing TNS_ADMIN as part of the connection URL requires 18.3 JDBC driver. Also, are you attempting to connect within the corporate network. In that case, you will need to pass HTTPS_PROXY and HTTPS_PROXY_PORT in the connection URL. Let us know. Happy to help with the problem.

How to configure log4j for an xtext gradle build?

When I start the gradle build in one of my modules, it prints an error-message to std-error:
:m28_presentation_api:generateXtext
Error initializing JvmElement
That's not very helpful and I hope, that I can configure log4j to print more details about the exception.
I think this message is logged by JvmTypesBuilder.initializeSafely()
LOG.error("Error initializing JvmElement", e);
Versions:
I am using xtext 2.13: in the MANIFEST.MF file, I see the
log4j version: 1.2.15
gradle version: 4.6
xtext-gradle-plugin version: 1.0.21
According to the log4j V1 docs, it should be enough when I add a log4j.properties file to the classpath: so I just save this file in src/main/java.
But it seems that this is not used/found - or maybe I did something wrong in the configuration file:
log4j.rootLogger=stderr
log4j.appender.stderr=org.apache.log4j.ConsoleAppender
log4j.appender.stderr.layout=org.apache.log4j.PatternLayout
# Pattern to output the caller's file name and line number.
log4j.appender.stderr.layout.ConversionPattern=%5p [%t] (%F:%L) - %m%n
When I now start the build, I'd expect a different log-output for the error-message, but it prints the same message as before. So obviously my log-config is not used for some reasons.
What am I missing?
Or can someone maybe point me to an example project?
it looks like LOG.error() does not print a stacktrace by default. maybe you can actively change your code e.g.
class MyDslJvmModelInferrer extends AbstractModelInferrer {
#Inject extension JvmTypesBuilder
private static Logger LOG = Logger.getLogger(MyDslJvmModelInferrer);
def dispatch void infer(Model element, IJvmDeclaredTypeAcceptor acceptor, boolean isPreIndexingPhase) {
acceptor.accept(element.toClass(element.name, [
try {
throw new IllegalArgumentException("mimimi")
} catch (Exception e) {
e.printStackTrace
throw e
}
]))
}
}
This is not really an answer, but a workaround (maybe helpful for others).
I gave up on the log4j configuration after wasting some hours and applied this workaround which took only some minutes and revealed the real problem.
What I've done, is to create my own JvmTypesBuilder, override the initializeSafely method and directly print the stack-trace to stderr:
import org.eclipse.xtext.xbase.jvmmodel.JvmTypesBuilder
import org.eclipse.emf.ecore.EObject
import org.eclipse.xtext.xbase.lib.Procedures.Procedure1
import org.apache.log4j.Logger
class JvmTypesBuilderTm extends JvmTypesBuilder {
private static Logger LOG = Logger.getLogger(JvmTypesBuilder)
// TODO: nasty workaround because I cannot figure out how to configure the logging correctly
override <T extends EObject> initializeSafely(T targetElement, Procedure1<? super T> initializer) {
if(targetElement !== null && initializer !== null) {
try {
initializer.apply(targetElement);
} catch (Exception e) {
LOG.error("Error initializing JvmElement: "+targetElement.toString, e);
e.printStackTrace
}
}
return targetElement;
}
}
Then I just replaced all occurrences of the JvmTypesBuilder in my code with the new one. With the stacktrace, it was easy to find the real issue in my code.

jdbc connection pool using ThreadpoolExecutor in spring boot

I have an application that runs through multiple databases and for each database runs select query on all tables and dumps it to hadoop.
My design is to create one datasource connection at a time and use the connection pool obtained to run select queries in multiple threads. Once done for this datasource, close the connection and create new one.
Here is the Async code
#Component
public class MySampleService {
private final static Logger LOGGER = Logger
.getLogger(MySampleService.class);
#Async
public Future<String> callAsync(JdbcTemplate template, String query) throws InterruptedException {
try {
jdbcTemplate.query(query);
//process the results
return new AsyncResult<String>("success");
}
catch (Exception ex){
return new AsyncResult<String>("failed");
}
}
Here is the caller
public String taskExecutor() throws InterruptedException, ExecutionException {
Future<String> asyncResult1 = mySampleService.callAsync(jdbcTemplate,query1);
Future<String> asyncResult2 = mySampleService.callAsync(jdbcTemplate,query2);
Future<String> asyncResult3 = mySampleService.callAsync(jdbcTemplate,query3);
Future<String> asyncResult4 = mySampleService.callAsync(jdbcTemplate,query4);
LOGGER.info(asyncResult1.get());
LOGGER.info(asyncResult2.get());
LOGGER.info(asyncResult3.get());
LOGGER.info( asyncResult4.get());
//now all threads finished, close the connection
jdbcTemplate.getConnection().close();
}
I am wondering if this is a right way to do it or do any exiting/optimized solution that out of box I am missing. I can't use spring-data-jpa since my queries are complex.
Thanks
Spring Boot docs:
Production database connections can also be auto-configured using a
pooling DataSource. Here’s the algorithm for choosing a specific
implementation:
We prefer the Tomcat pooling DataSource for its performance and concurrency, so if that is available we always choose it.
Otherwise, if HikariCP is available we will use it.
If neither the Tomcat pooling datasource nor HikariCP are available and if Commons DBCP is available we will use it, but we
don’t recommend it in production.
Lastly, if Commons DBCP2 is available we will use it.
If you use the spring-boot-starter-jdbc or
spring-boot-starter-data-jpa ‘starters’ you will automatically get a
dependency to tomcat-jdbc.
So you should be provided with sensible defaults.

Resources