Connecting Springboot application to Azure databricks - spring-boot

I'm trying to connect SpringBoot Application to Azure Databricks.
Below is something I have tried....
application.properties
spring.datasource.url = jdbc:spark://adb-**********.*.azuredatabricks.net:**/default;transportMode=http;ssl=1;httpPath=sql/protocolv1/o/******/******-*****-abcd341
spring.datasource.username = username
spring.datasource.password = Generated Token
pom.xml
Below are some dependencies I'm using...
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.5.2</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>1.5.2</version>
</dependency>
<dependency>
<groupId>com.databricks</groupId>
<artifactId>spark-avro_2.10</artifactId>
<version>2.0.1</version>
</dependency>
I'm getting below error..
***************************
APPLICATION FAILED TO START
***************************
Description:
Cannot determine embedded database driver class for database type NONE
Action:
If you want an embedded database please put a supported one on the classpath. If you have database settings to be loaded from a particular profile you may need to active it (no profiles are currently active).
Suggest me if I'm missing any maven dependency.
Thanks in Advance..

To connect from the Spring Boot you need to use JDBC driver, not Spark jars (remove them - you don't need them). You can get JDBC driver as described in documentation, or very recently - directly via Maven using following coordinates:
<dependency>
<groupId>com.databricks</groupId>
<artifactId>databricks-jdbc</artifactId>
<version>2.6.25-1</version>
</dependency>
and then use standard JDBC APIs exposed by Spring. I have a simple example that uses JdbcTemplate to access data in Databricks - you just need to construct JDBC URL correctly:
String host = "";
String httpPath = "";
String token = "";
String jdbcUrl = "jdbc:databricks://" + host +
":443/default;transportMode=http;ssl=1;httpPath=" +
httpPath + ";AuthMech=3;UID=token;PWD=" + token;
and then just access data:
// define data source
SimpleDriverDataSource ds = new SimpleDriverDataSource();
ds.setDriver(new Driver());
ds.setUrl(jdbcUrl);
JdbcTemplate jdbcTemplate = new JdbcTemplate(ds);
// query data
List<Map<String, Object>> data = jdbcTemplate.queryForList(query);
for (Map<String, Object> row: data) {
....
}
P.S. You may omit username or at least set it to the token value...

Try adding the "spring.datasource.driverClassName" , and let me know if that helps you to proceed

Related

Debezium MySQL Connector- com.mysql.cj.CharsetMapping.getStaticCollationNameForCollationIndex(Ljava/lang/Integer;)Ljava/lang/String

Debezium mysql connector fail at final stage of snapshotting.
The project is on maven/quarkus , and I want to use debezium/infinispan for cache invalidation .
The Observer and configuration looks like this :
public void startEmbeddedEngine(#Observes #Initialized(ApplicationScoped.class) Object init) throws IOException {
File dbHistoryTempFile = File.createTempFile("offsets", ".dat");
File offsetStorageTempFile = File.createTempFile("dbhistory", ".dat");
final Properties props = new Properties();
props.setProperty("name", "cache-invalidation-engine");
props.setProperty("connector.class", "io.debezium.connector.mysql.MySqlConnector");
props.setProperty("offset.storage.file.filename", offsetStorageTempFile.getAbsolutePath());
props.setProperty("offset.flush.interval.ms", "0");
props.setProperty("database.hostname", "localhost");
props.setProperty("database.port", "3306");
props.setProperty("database.user", "root");
props.setProperty("database.password", "password");
props.setProperty("database.server.id", "1");
props.setProperty("database.server.name", "new_feature");
props.setProperty("database.history",
"io.debezium.relational.history.FileDatabaseHistory");
props.setProperty("database.history.file.filename", dbHistoryTempFile.getAbsolutePath());
props.setProperty("database.include.list", "database");
// props.setProperty("database.history.file.filename","C:/Users/a.pogonet/AppData/Local/Temp/db.dat");
props.setProperty("snapshot.mode", "never");
// props.setProperty("include.unknown.datatypes", "true");
// props.setProperty("include.schema.changes", "false");
DebeziumEngine<ChangeEvent<String, String>> engine = DebeziumEngine.create(Json.class)
.using(props)
.notifying(record -> {
System.out.println(record);
}).build();
executorService = Executors.newSingleThreadExecutor();
executorService.execute(engine);
}
2021-12-09 14:18:34,137 INFO [io.deb.con.mys.MySqlStreamingChangeEventSource] (blc-localhost:3306) Stopped reading binlog after 0 events, no new offset was recorded
Exception in thread "blc-localhost:3306" java.lang.NoSuchMethodError: com.mysql.cj.CharsetMapping.getStaticCollationNameForCollationIndex(Ljava/lang/Integer;)Ljava/lang/String;
at io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser.extractCharset(MySqlAntlrDdlParser.java:404)
at io.debezium.connector.mysql.antlr.listener.CreateAndAlterDatabaseParserListener.enterCreateDatabaseOption(CreateAndAlterDatabaseParserListener.java:49)
at io.debezium.ddl.parser.mysql.generated.MySqlParser$CreateDatabaseOptionContext.enterRule(MySqlParser.java:5912)
at io.debezium.antlr.ProxyParseTreeListenerUtil.delegateEnterRule(ProxyParseTreeListenerUtil.java:46)
at io.debezium.connector.mysql.antlr.listener.MySqlAntlrDdlParserListener.enterEveryRule(MySqlAntlrDdlParserListener.java:89)
at org.antlr.v4.runtime.tree.ParseTreeWalker.enterRule(ParseTreeWalker.java:41)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:25)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)
at io.debezium.antlr.AntlrDdlParser.parse(AntlrDdlParser.java:87)
at io.debezium.connector.mysql.MySqlDatabaseSchema.parseDdl(MySqlDatabaseSchema.java:216)
at io.debezium.connector.mysql.MySqlDatabaseSchema.parseStreamingDdl(MySqlDatabaseSchema.java:202)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleQueryEvent(MySqlStreamingChangeEventSource.java:573)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$14(MySqlStreamingChangeEventSource.java:827)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:349)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$25(MySqlStreamingChangeEventSource.java:855)
at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1125)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:973)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:599)
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:857)
at java.base/java.lang.Thread.run(Thread.java:834)
The Quarkus BOM is enforcing a version of the MySQL driver and AFAICS Debezium is using methods that are not available anymore in this version of the driver.
What I recommend you is to use the version of the driver used by the Debezium connector by overriding the version manually in your project.
I can't tell you exactly which version to use given I have no idea which Debezium version you are using. But the version is there for Debezium 1.8: https://github.com/debezium/debezium/blob/1.8/pom.xml#L120 .
That being said, Quarkus 2.5.1.Final+ is also using MySQL Connector 8.0.27 so if you use these versions, I believe it should work.
you need update mysql driver to 8.0.27
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>8.0.27</version>
<exclusions>
<exclusion>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
</exclusion>
</exclusions>
</dependency>

Micrometer StackdriverMeterRegistry only publishes custom metrics to GCP Monitoring, not automatic instrumentation metrics?

I have this example https://quarkus.io/guides/micrometer (micrometer quickstart directory) running which uses Quarkus and Micrometer together. The example uses Prometheus as the MeterRegistry but I changed it to use the StackdriverMeterRegistry in hopes the same auto instrumentation that shows up in Prometheus would show up in Google Cloud Monitoring.
However, I only see the custom metrics I made appear into Google Cloud Monitoring, and not the auto instrumentation provided by micrometer.
I am unsure if I should think that this is just an issue with the Micrometer StackdriverMeterRegistry library itself or if I am doing something wrong. Any guidance is appreciated.
Code changes:
// Update the constructor to create the gauge
ExampleResource(MeterRegistry registry) {
/* Code for micrometer */
StackdriverConfig stackdriverConfig = new StackdriverConfig() {
#Override
public String projectId() {
return "projectId";
}
#Override
public String get(String key) {
return null;
}
};
this.registry = StackdriverMeterRegistry.builder(stackdriverConfig).build();
registry.config().commonTags("application", "projectId");
registry.gaugeCollectionSize("example.list.size", Tags.empty(), list);
}
Added to pom.xml
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-stackdriver</artifactId>
</dependency>
After tinkering and speaking directly with the Micrometer team I found out the issue. The documentation is a bit confusing but I had imported the StackDriver extension wrong and the default registry being used for the quarkus project was getting all the auto instrumentation but not the StackDriver one. So this default registry needed to be changed to the StackDriver one.
I have uploaded a basic example of using Quarkus StackDriver and Micrometer together using the basic example found on the Micrometer Quarkus documentation page.
https://github.com/jayleenli/quarkus-micrometer-stackdriver-quickstart
The changes:
Add to pom.xml
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-micrometer</artifactId>
</dependency>
<dependency>
<groupId>io.quarkiverse.micrometer.registry</groupId>
<artifactId>quarkus-micrometer-registry-stackdriver</artifactId>
<version>2.2.2</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-core</artifactId>
<version>1.7.3</version>
</dependency>
Then add some Quarkus properties, I used application.properties but there are other ways you can do this.
application.properties
quarkus.micrometer.export.stackdriver.enabled=true
quarkus.micrometer.export.stackdriver.default-registry=true
quarkus.micrometer.export.stackdriver.project-id=fake-id
quarkus.micrometer.export.stackdriver.publish=true
quarkus.micrometer.export.stackdriver.resource-type=global
quarkus.micrometer.export.stackdriver.step=1m
In main class
#Path("/")
public class ExampleResource {
#ConfigProperty(name = "quarkus.micrometer.export.stackdriver.enabled")
boolean enabled;
#ConfigProperty(name = "quarkus.micrometer.export.stackdriver.default-registry")
boolean export;
#ConfigProperty(name="quarkus.micrometer.export.stackdriver.project-id")
String projectId;
#ConfigProperty(name="quarkus.micrometer.export.stackdriver.publish")
boolean publish;
#ConfigProperty(name="quarkus.micrometer.export.stackdriver.resource-type")
String resourceType;
#ConfigProperty(name="quarkus.micrometer.export.stackdriver.step")
String step;

Configure HTTPS in Spring Boot Apache Camel REST API with keystore having multiple certs using camel-jetty component

I am trying to configure https in my apache camel Spring Boot REST application (using apache-camel v3.11.1, springboot v2.5.3) with keystore having multiple certificates.
Problem:
Application run failed
org.apache.camel.RuntimeCamelException: java.lang.IllegalStateException: KeyStores with multiple certificates are not supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory. (Use org.eclipse.jetty.util.ssl.SslContextFactory$Server or org.eclipse.jetty.util.ssl.SslContextFactory$Client instead)
at org.apache.camel.RuntimeCamelException.wrapRuntimeCamelException(RuntimeCamelException.java:51) ~[camel-api-3.11.1.jar:3.11.1]
Project setup:
pom.xml: (dependencies only, to show that I am not using spring-boot-web-starter)
..
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.springboot</groupId>
<artifactId>camel-spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.springboot</groupId>
<artifactId>camel-jetty-starter</artifactId>
</dependency>
..
..<!-- all other required dependencies are in place-->
..
</dependencies>
..
application.properties
#camel.component.jetty.keystore=keystore-with-one-certificate.jks # WORKS
camel.component.jetty.keystore=keystore-with-multiple-certificates.jks # DOESN'T WORK
camel.component.jetty.ssl-key-password=password
camel.component.jetty.ssl-password=password
Rest Route:
restConfiguration()
.component("jetty")
.scheme("https")
.port("8080");
rest()
.path("/api")
.get("/{name}")
..
..
.to("direct:x");
Looked at answers in the below posts, but still not able to resolve the exception that I get,
https://stackoverflow.com/a/60598953/6363894,
https://stackoverflow.com/a/55499113/6363894
I know that exception clearly states to use org.eclipse.jetty.util.ssl.SslContextFactory$Server, but I don't understand how/where to use SslContextFactory.Server object.
SslContextFactory.Server sslContextFactory = new SslContextFactory.Server();
sslContextFactory.setKeyStoreResource(findKeyStorePath());
sslContextFactory.setKeyStorePassword("password");
sslContextFactory.setKeyManagerPassword("password");
sslContextFactory.setNeedClientAuth(true);
Also I've created a bean for sslContextParameters and added that to restConfiguration as below, this time application runs successfully but then when I test, SSL handshake fails.
restConfiguration()
.component("jetty")
.endpointProperty("sslContextParameters", "#sslContextParameters")
.scheme("https")
.port("8080");
#Bean(name = "sslContextParameters")
public SSLContextParameters setSSLContextParameters() {
KeyStoreParameters ksp = new KeyStoreParameters();
ksp.setResource("keystore-with-multiple-certificates.jks");
ksp.setPassword("password");
KeyManagersParameters kmp = new KeyManagersParameters();
kmp.setKeyStore(ksp);
kmp.setKeyPassword("password");
SSLContextServerParameters scsp = new SSLContextServerParameters();
scsp.setClientAuthentication("REQUIRE");
SSLContextParameters scp = new SSLContextParameters();
scp.setServerParameters(scsp);
scp.setKeyManagers(kmp);
return scp;
}
Any help on how to configure SslContextFactory.Server object with the restConfigurations() or any other way I can achieve this? I'll update the post, if any more details are required.

How to configure DataSource for a Spring-Boot Application in a Standalone (war) and in an embedded Tomcat?

I have a Spring-Boot-Aplication with the following dependencyManagement:
<dependencyManagement>
<dependencies>
<dependency>
<!-- Import dependency management from Spring Boot -->
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-dependencies</artifactId>
<version>2.1.5.RELEASE</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
and the following dependencies:
spring-boot-starter-jersey
spring-boot-starter-jdbc(exclusion:tomcat-jdbc)
HikariCP(version:3.3.1)
ojdbc7
On Tomcat I configured a JNDI-Datasource as:
<Resource name="jdbc/myDS"
type="javax.sql.DataSource"
driverClassName="oracle.jdbc.driver.OracleDriver"
username="Superuser"
password="secret"
url="jdbc:oracle:thin:#xxxDbX"
../>
In the .properties-file I added the following properties:
spring.datasource.type=org.apache.tomcat.jdbc.pool.DataSource
spring.datasource.jndi-name=jdbc/myDS
As Spring-Boot is able to configure a DataSource from the properties, I let it do so and I do write no extra code for a DataSource.
Deployed in a Standalone Tomcat it works perfectly.
Logically Spring Boot can not find the JNDI-Resource in an embedded Tomcat and starting the application as a Spring-Boot-Application I got:
***************************
APPLICATION FAILED TO START
***************************
Description:
Failed to bind properties under 'spring.datasource.type' to java.lang.Class<javax.sql.DataSource>:
Property: spring.datasource.type
Value: org.apache.tomcat.jdbc.pool.DataSource
Origin: class path resource [application.properties]:12:24
Reason: No converter found capable of converting from type [java.lang.String] to type [java.lang.Class<javax.sql.DataSource>]
Action:
Update your application's configuration
I would like to be able to start the application as a Spring-Boot-Application and also build a war-file which can be deployed in any Standalone Tomcat.
Is this possible by adding properties for a second DataSource in case the application is started as a Spring-Boot-Application or I am obliged to have a second .properties file?
The solution that worked for me is to add a custom-properties to use for the DataSource in the embedded Tomcat Server like so:
# for a dedicated Tomcat
spring.datasource.jndi-name=jdbc/dirserver
# for the embedded Tomcat
embedded.datasource.driver-class-name=oracle.jdbc.OracleDriver
embedded.datasource.url=jdbc:oracle:thin:#//myServer:1521/xxxxx
embedded.datasource.username=superuser
embedded.datasource.password=topsecret
and to define #Bean DataSource in the class annotated with #SpringBootApplication:
#SpringBootApplication
public class MySbApplication extends SpringBootServletInitializer {
private static final Logger lg = LoggerFactory.getLogger(MySbApplication.class);
#Value("${embedded.datasource.username}")
String username;
#Value("${embedded.datasource.password}")
String password;
#Value("${embedded.datasource.driver-class-name}")
String driverClassName;
#Value("${embedded.datasource.url}")
String url;
#Bean(destroyMethod = "")
public DataSource oracledataSoutŕce() throws SQLException {
final OracleDataSource dataSource = new OracleDataSource();
dataSource.setUser(username);
dataSource.setPassword(password);
dataSource.setURL(url);
dataSource.setImplicitCachingEnabled(true);
dataSource.setFastConnectionFailoverEnabled(true);
return dataSource;
}
}
I willl add a link to a sample project in Github.

spring boot elastic search -configure data source

I am tryinging to configure spring data boot sand ES project
in my pom.xml i have :
#Configuration
#EnableElasticsearchRepositories(basePackages = "com.yoyo.elastic.repository")
public class ElasticConfiguration {
#Bean
public NodeBuilder nodeBuilder() {
return new NodeBuilder();
}
#Bean
public ElasticsearchOperations elasticsearchTemplate() throws IOException {
File tmpDir = File.createTempFile("elastic", Long.toString(System.nanoTime()));
System.out.println("Temp directory: " + tmpDir.getAbsolutePath());
final Client client = nodeBuilder().local(true).node().client();
return new ElasticsearchTemplate(client);
}
}
in my pom xml I have this dep :
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-elasticsearch</artifactId>
</dependency>
which should supplay the driver but i keep on getting :
Description:
Cannot determine embedded database driver class for database type NONE
Action:
If you want an embedded database please put a supported one on the classpath. If you have database settings to be loaded from a particular profile you may need to active it (no profiles are currently active).
I had the same issue when trying to run some exercises with Spring Boot and ElasticSearch.
Right now I figured out that if you have the
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
Alongside spring-boot-starter-data-elasticsearch and don't add additional config classes (where you would configure the DataSource) spring boot will complain.
Other solution would be to actually add a datasource property to application.properties and configure standalone database (like H2)

Resources