Azure Databricks Logging Configuration problems - spring

Question

I'm running Spring Boot 2.7.4 with logback 1.2.11 and also slf4j 1.7.36, and I have a console appender in my logback-spring.xml:
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.7.4</version>
<relativePath/>
</parent>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>log4j-over-slf4j</artifactId>
</dependency>
<dependency>
<groupId>com.databricks</groupId>
<artifactId>databricks-jdbc</artifactId>
<version>2.6.29</version>
</dependency>
<pattern>%mask(${CONSOLE_LOG_PATTERN})</pattern>
<pattern>%clr(%d{${LOG_DATEFORMAT_PATTERN:-yyyy-MM-dd HH:mm:ss.SSS}}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){magenta} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %mask(%m%n)</pattern>
My entire application is logging just fine, but I needed to add the Databricks JDBC drivers, and I get this in my log when I execute a SQL from both the Spring JPA framework and with the JDBC Template:
ERROR StatusLogger Unrecognized format specifier [d]
ERROR StatusLogger Unrecognized conversion specifier [d] starting at position 16 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [thread]
ERROR StatusLogger Unrecognized conversion specifier [thread] starting at position 25 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [level]
ERROR StatusLogger Unrecognized conversion specifier [level] starting at position 35 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [logger]
ERROR StatusLogger Unrecognized conversion specifier [logger] starting at position 47 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [msg]
ERROR StatusLogger Unrecognized conversion specifier [msg] starting at position 54 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [n]
ERROR StatusLogger Unrecognized conversion specifier [n] starting at position 56 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [d]
ERROR StatusLogger Unrecognized conversion specifier [d] starting at position 16 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [thread]
ERROR StatusLogger Unrecognized conversion specifier [thread] starting at position 25 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [level]
ERROR StatusLogger Unrecognized conversion specifier [level] starting at position 35 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [logger]
ERROR StatusLogger Unrecognized conversion specifier [logger] starting at position 47 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [msg]
ERROR StatusLogger Unrecognized conversion specifier [msg] starting at position 54 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [n]
ERROR StatusLogger Unrecognized conversion specifier [n] starting at position 56 in conversion pattern.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.databricks.client.jdbc42.internal.io.netty.util.internal.ReflectionUtil (file:/C:/Users/[REDACTED]/com/databricks/databricks-jdbc/2.6.29/databricks-jdbc-2.6.29.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of com.databricks.client.jdbc42.internal.io.netty.util.internal.ReflectionUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
I found this issue that other people had, and it seems like some dependency is bringing in lo4j configurations from somewhere, like some .dat file or a log4j properties file, but I cannot find these file s anywhere in the Databricks jars. There are some log4j files in the jar, but not sure if these are what is causing the issue or not.
Log4j-charsets.properties
Log4j-config.xsd
Log4j-events.dtd
Log4j-events.xsd
Log4j-levels.xsd
It's clearly conflicting with my logging pattern, because it cannot understand the format pattern for "msg" or "thread" etc.
When I debug the my code and get my datasource, and getParentLogger(), it's using java.util.logging.Logger, just like the rest of my datasources (using Spring JPA/Hibernate).
I've tried to exclude any logging from the dependency and cannot figure out how to force it to use the logging for my project:
<dependency>
<groupId>com.databricks</groupId>
<artifactId>databricks-jdbc</artifactId>
<version>2.6.29</version>
<exclusions>
<exclusion>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</exclusion>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty-buffer</artifactId>
</exclusion>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty-common</artifactId>
</exclusion>
<exclusion>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</exclusion>
</exclusions>
</dependency>
I tried adding some of the other dependencies, like these:
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>jul-to-slf4j</artifactId>
</dependency>
And this doesn't fix the issue either.
I have also tried to turn off the logging for the StatusLogger in my logback-spring.xml, but I still see the errors in red text in the log (ANSI is turned on).
<logger name="org.apache.logging.log4j.status.StatusLogger" level="OFF" additivity="false">
<appender-ref ref="CONSOLE" />
</logger>
<logger name="com.databricks.*" level="OFF" additivity="false">
<appender-ref ref="CONSOLE" />
</logger>
It only happens once when I use the datasource, and hibernate is printing the SQL and everything just fine, so I don't know if I can suppress this somehow, but ideally, I would like it to use the logging framework that I'm using for everything else and not try to override my logging patterns.

Answer 1

This is apparently related to Arrow Serialization, and can be fixed by disabling it via your JDBC URL.
Add ;EnableArrow=0 to the end of your URL.
Warning: I do not know the overall implications of disabling this serialization, but I know this will also allow you to use the driver in the latest versions of Java as well.