Google BigTable access error - maven

I need to read from BigTable in my java application. I am using the google-cloud-bigtable hbase client. I have added the dependency to my pom file:
<dependency>
<groupId>com.google.cloud.bigtable</groupId>
<artifactId>bigtable-hbase-1.x</artifactId>
<version>1.3.0</version>
<exclusions>
<exclusion>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</exclusion>
<exclusion>
<groupId>com.google.api.grpc</groupId>
<artifactId>grpc-google-common-protos</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
</exclusions>
</dependency>
And attempt to create a connection:
Connection connection = BigtableConfiguration.connect(projectID, instanceID));
When I run the code I receive the following stack trace:
java.lang.IllegalStateException: Could not find an appropriate constructor for com.google.cloud.bigtable.hbase1_x.BigtableConnection
at com.google.cloud.bigtable.hbase.BigtableConfiguration.connect(BigtableConfiguration.java:128)
at com.google.cloud.bigtable.hbase.BigtableConfiguration.connect(BigtableConfiguration.java:113)
This issue suggests that this can be solved by setting the environment variable GOOGLE_APPLICATION_CREDENTIALS. I have set this up as an environment variable on systen and as a maven setting on the Eclipse run configuration but still receive the same error.
Thanks for looking.
UPDATE
Longer stack trace as requested. Please don't be mislead by mentions of dataflow, this is attempting to access BigTable in the standard manner rather than as part of a dataflow pipeline.
java.lang.IllegalStateException: Could not find an appropriate constructor for com.google.cloud.bigtable.hbase1_x.BigtableConnection
at com.google.cloud.bigtable.hbase.BigtableConfiguration.connect(BigtableConfiguration.java:128)
at com.google.cloud.bigtable.hbase.BigtableConfiguration.connect(BigtableConfiguration.java:113)
at com.ps.pt.BinReader.getBinDetails(BinReader.java:27)
at dataflows.MessageTypesTests.testBinReader(MessageTypesTests.java:136)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:538)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:760)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:460)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:206)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.google.cloud.bigtable.hbase.BigtableConfiguration.connect(BigtableConfiguration.java:125)
... 26 more
Caused by: java.lang.NoSuchMethodError: com.google.cloud.bigtable.config.BigtableOptions$Builder.setAdminHost(Ljava/lang/String;)Lcom/google/cloud/bigtable/config/BigtableOptions$Builder;
at com.google.cloud.bigtable.hbase.BigtableOptionsFactory.fromConfiguration(BigtableOptionsFactory.java:301)
at org.apache.hadoop.hbase.client.AbstractBigtableConnection.<init>(AbstractBigtableConnection.java:136)
at org.apache.hadoop.hbase.client.AbstractBigtableConnection.<init>(AbstractBigtableConnection.java:111)
at com.google.cloud.bigtable.hbase1_x.BigtableConnection.<init>(BigtableConnection.java:49)
... 31 more

The most relevant part of the stack trace is here:
Caused by: java.lang.NoSuchMethodError: com.google.cloud.bigtable.config.BigtableOptions$Builder.setAdminHost(Ljava/lang/String;)Lcom/google/cloud/bigtable/config/BigtableOptions$Builder;
at com.google.cloud.bigtable.hbase.BigtableOptionsFactory.fromConfiguration(BigtableOptionsFactory.java:301)
From these lines, we can see that the BigtableOptionsFactory, which is declared in the bigtable-hbase jar, is trying to call BigtableOptions.setAdminHost, which is declared in bigtable-client-core. Given the NoSuchMethodError, that means that when we compiled the bigtable-hbase jar at 1.3.0, the BigtableOptions.setAdminHost method was available in the bigtable-client-core 1.3.0 jar. It is therefore very likely that your versions of the two jars don't match. In fact, setAdminHost was added as a method in https://github.com/googleapis/java-bigtable-hbase/commit/6c0756973b12e5fd2a9546a49342bb7e96ae28b3, which was merged in version 1.1.0, so your bigtable-client-core is probably a version from before that. Can you verify that?

Related

java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory - Related to Appium

java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
at io.appium.java_client.service.local.AppiumDriverLocalService.<clinit>(AppiumDriverLocalService.java:57)
at BaseTest.globalSetup(basetest.java:14)
at androidtest.globalSetup(androidtest.java:17)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:133)
at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:62)
at org.testng.internal.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:385)
at org.testng.internal.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:321)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:317)
at org.testng.SuiteRunner.run(SuiteRunner.java:276)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1212)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1134)
at org.testng.TestNG.runSuites(TestNG.java:1063)
at org.testng.TestNG.run(TestNG.java:1031)
at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:66)
at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:109)
Caused by: java.lang.ClassNotFoundException: org.slf4j.LoggerFactory
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 21 more
Below is what i have where the code fails at globalSetUp()
Below is also relevant code
and Appium server says
Hello all, trying to get appium working for the first time as a novice, would like to know what i can do over at pom.xml or what can i do to resolve this problem.
Thanks!
Add a SLF4J implementation:
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.7.30</version>
</dependency>

Gatling using JDBC feeder connecting to SQL Server with Windows Authentication throws error "java.lang.NoSuchMethodError"

Learning from this documentation , I tried to use Gatling JDBC feeder to grab data from our SQL Server with Windows Authentication.
I add this dependency in my POM.xml
<!-- https://mvnrepository.com/artifact/dev.code-n-roll.gatling/jdbc-gatling -->
<dependency>
<groupId>dev.code-n-roll.gatling</groupId>
<artifactId>jdbc-gatling_2.12</artifactId>
<version>2.3.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/io.gatling/gatling-jdbc -->
<dependency>
<groupId>io.gatling</groupId>
<artifactId>gatling-jdbc</artifactId>
<version>3.4.2</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.microsoft.sqlserver/mssql-jdbc -->
<dependency>
<groupId>com.microsoft.sqlserver</groupId>
<artifactId>mssql-jdbc</artifactId>
<version>7.4.1.jre8</version>
</dependency>
My connection string looks like this
import io.gatling.jdbc.Predef._
val databaseUrl = "jdbc:sqlserver://CentralSQLServer:1433;databaseName=CustomerDB;integratedSecurity=true"
I added integratedSecurity=true because the access method to the SQL server is using Windows Authentication, not using username & password. This works fine when I use vanilla Java with JDBC.
Then I define the JDBC Feeder
val mySQLFeeder = jdbcFeeder(databaseUrl, "", "", "SELECT * FROM [CustomerDB].[dbo].[Users]")
I put empty String in username and password sections , since I don't have it.
Running the Gatling test using mvn gatling:test , I see this error is thrown by Gatling
02:05:29.920 [ERROR] i.g.a.Gatling$ - Run crashed
java.lang.NoSuchMethodError: io.gatling.core.feeder.SourceFeederBuilder$.apply(Lio/gatling/core/feeder/FeederSource;Lio/gatling/core/config/GatlingConfiguration;)Lio/gatling/core/feeder/SourceFeederBuilder;
at io.gatling.jdbc.Predef$.jdbcFeeder(Predef.scala:26)
at testCases.Delete.<init>(Delete.scala:49)
... 16 common frames omitted
Wrapped by: java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at io.gatling.app.Runner.run0(Runner.scala:74)
at io.gatling.app.Runner.run(Runner.scala:60)
at io.gatling.app.Gatling$.start(Gatling.scala:80)
at io.gatling.app.Gatling$.fromArgs(Gatling.scala:46)
at io.gatling.app.Gatling$.main(Gatling.scala:38)
at io.gatling.app.Gatling.main(Gatling.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at io.gatling.mojo.MainWithArgsInFile.runMain(MainWithArgsInFile.java:50)
at io.gatling.mojo.MainWithArgsInFile.main(MainWithArgsInFile.java:33)
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at io.gatling.mojo.MainWithArgsInFile.runMain(MainWithArgsInFile.java:50)
at io.gatling.mojo.MainWithArgsInFile.main(MainWithArgsInFile.java:33)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at io.gatling.app.Runner.run0(Runner.scala:74)
at io.gatling.app.Runner.run(Runner.scala:60)
at io.gatling.app.Gatling$.start(Gatling.scala:80)
at io.gatling.app.Gatling$.fromArgs(Gatling.scala:46)
at io.gatling.app.Gatling$.main(Gatling.scala:38)
at io.gatling.app.Gatling.main(Gatling.scala)
... 6 more
Caused by: java.lang.NoSuchMethodError: io.gatling.core.feeder.SourceFeederBuilder$.apply(Lio/gatling/core/feeder/FeederSource;Lio/gatling/core/config/GatlingConfiguration;)Lio/gatling/core/feeder/SourceFeederBuilder;
at io.gatling.jdbc.Predef$.jdbcFeeder(Predef.scala:26)
at testCases.Delete.<init>(Delete.scala:49)
... 16 more
It looks like Gatling JDBCFeeder method complains about my connection string using integratedSecurity=true since in the documentation we should use username and password. The thing is there are no username and password in my case.
Is there any way to solve this issue?
Thanks.
Thanks #GeorgeLeung and #StephaneLandelle .
The issue is caused by dependency version conflict.
The gatling-jdbc version must align with the gatling-core version.
Change the dependencies in POM.xml to this below does solve the issue.
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<encoding>UTF-8</encoding>
<gatling.version>3.3.1</gatling.version>
<gatling-maven-plugin.version>3.0.5</gatling-maven-plugin.version>
<scala-maven-plugin.version>4.3.1</scala-maven-plugin.version>
<maven-jar-plugin.version>3.2.0</maven-jar-plugin.version>
<scala.version>2.12.10</scala.version>
</properties>
<dependencies>
<dependency>
<groupId>io.gatling.highcharts</groupId>
<artifactId>gatling-charts-highcharts</artifactId>
<version>${gatling.version}</version>
</dependency>
<dependency>
<groupId>io.gatling</groupId>
<artifactId>gatling-app</artifactId>
<version>${gatling.version}</version>
</dependency>
<dependency>
<groupId>io.gatling</groupId>
<artifactId>gatling-recorder</artifactId>
<version>${gatling.version}</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.microsoft.sqlserver/mssql-jdbc -->
<dependency>
<groupId>com.microsoft.sqlserver</groupId>
<artifactId>mssql-jdbc</artifactId>
<version>7.4.1.jre8</version>
</dependency>
</dependencies>
The gatling-app does include the gatling-jdbc , so we don't need to extra add it.

java.lang.NoClassDefFoundError: org/springframework/orm/hibernate5/HibernateTransactionManager

I am trying to integrate spring with hibernate and I have spring-orm .4.3.6 jar file in my project.But still I am getting below error :
java.lang.NoClassDefFoundError: org/springframework/orm/hibernate5/HibernateTransactionManager
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.getDeclaredMethods(Class.java:1975)
at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:613)
at org.springframework.util.ReflectionUtils.doWithMethods(ReflectionUtils.java:524)
at org.springframework.util.ReflectionUtils.doWithMethods(ReflectionUtils.java:510)
at org.springframework.util.ReflectionUtils.getUniqueDeclaredMethods(ReflectionUtils.java:570)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getTypeForFactoryMethod(AbstractAutowireCapableBeanFactory.java:697)
Tried googling it.But did not find answer.
Can someone help?
This class is coming out of the spring-orm dependency (notice the hibernate5 package in there. There is equally hibernate3 and hibernate4 packages in the same jar not to break compatibility).
The Maven Coordinates are:
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-orm</artifactId>
<version>4.3.10.RELEASE</version>
</dependency>

java.lang.reflect.InvocationTargetException java.lang.NoClassDefFoundError: com/google/common/io/LimitInputStream

I am using hadoop version 2.5.0.Getting this exception while running mapreduce job on this line
boolean b = job.waitForCompletion(true);
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:53)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoClassDefFoundError: com/google/common/io/LimitInputStream
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:467)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1292)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1313)
Updated
My issue is hadoop is needing LimitInputStream class which is removed from guava16.0 and is upto guava14.0.
On adding org.springframework.boot:spring-boot-gradle-plugin:1.2.7.RELEASE
I get this error .How do I solve this
You have to force the guava version to 14.0
<dependencyManagement>
<dependencies>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>14.0</version>
</dependency>
</dependencies>
</dependencyManagement>

How to find which scala version needs to be used for each kafka release

currently we are using kafka-0.7.2(https://clojars.org/org.clojars.paul/core-kafka_2.8.0) and scala version 2.8.0 in production.
we are excited to see kafka 0.8.1.1 so we followed the producer(https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example) code and consumer(https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example) code in the mentioned blog.
This code leads to an error of missing scala.serializer class so I searched and got info that scala-2.10.3(Eclipse scala.object cannot be resolved) will solve this issue.
while running this code again I got an error of
[WARNING]
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.NoClassDefFoundError: scala/reflect/ClassManifest
at kafka.utils.Log4jController$.<init>(Log4jController.scala:29)
at kafka.utils.Log4jController$.<clinit>(Log4jController.scala)
at kafka.utils.Logging$class.$init$(Logging.scala:29)
will linger despite being asked to die via interruption
[WARNING] NOTE: 3 thread(s) did not finish despite being asked to via interruption. This is not a problem with exec:java, it is a problem with the running code. Although not serious, it should be remedied.
[WARNING] Couldn't destroy threadgroup org.codehaus.mojo.exec.ExecJavaMojo$IsolatedThreadGroup[name=packageName,maxpri=10]
java.lang.IllegalThreadStateException
at java.lang.ThreadGroup.destroy(ThreadGroup.java:775)
at org.codehaus.mojo.exec.ExecJavaMojo.execute(ExecJavaMojo.java:328)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
I don't know why this error occurred,
changed pom.xml
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.9.2</artifactId>
<version>0.8.1.1</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<groupId>com.sun.jmx</groupId>
<artifactId>jmxri</artifactId>
</exclusion>
<exclusion>
<groupId>com.sun.jdmk</groupId>
<artifactId>jmxtools</artifactId>
</exclusion>
<exclusion>
<groupId>javax.jms</groupId>
<artifactId>jms</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.10.3</version>
</dependency>
I also got info from one of the site that kafka 0.8.1.1 will have a support for different version of scala,but i don't know why it was not supported for this scala version(2.10.3).
How can we find for x kafka version, y scala version will work
i also figured out that in gradle.properties of kafka-0.8.1.1
group=org.apache.kafka
version=0.8.1.1
scalaVersion=2.8.0
task=build
here mentioned scalaVersion of 2.8.0 is meant for ?
Thanks in advance
I have removed the scala dependency,since its using scala internally.now its worked properly and also some changes in property name of zookeeper connections.

Resources