Trying to debug AWS Glue scripts locally using Glue ETL library.
I have installed aws-glue-libs and spark-3.1.1-amzn-0-bin-3.2.1-amzn-3.tgz.
When I run job.init(), I get the following error trace:
py4j.protocol.Py4JJavaError: An error occurred while calling z:com.amazonaws.services.glue.util.Job.init.
: java.lang.NoClassDefFoundError: com/typesafe/config/ConfigMergeable
at com.amazonaws.services.glue.util.Job$.init(Job.scala:93)
at com.amazonaws.services.glue.util.Job.init(Job.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: com.typesafe.config.ConfigMergeable
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 13 more
This error wasn't resolved per say. But i found a workaround. Instead of running my scripts from pycharm, i run them using gluesparksubmit bash command. Now it doesn't throw error at job.init(). Still trying to figure out how to get access to data catalog when running glue scripts from local machine.
If it was for ConfigMergeable, check if proper jar file - config-1.3.3.jar - exists in your /opt/spark/jars dir.
Whole idea is that jars in /opt/spark/jars and ./aws-glue-libs/jarsv1 should match.
When I do a mvn -version I get the following error
FYI:- I saw the other related posts and none of it is resolving my prob
Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/maven/cli/MavenCli : Unsupported major.minor version 51.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:637)
at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClassFromSelf(ClassRealm.java:401)at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass(SelfFirstStrategy.java:42)
at org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass(ClassRealm.java:271)
at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:254)
at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:239)
at org.codehaus.plexus.classworlds.launcher.Launcher.getMainClass(Launcher.java:144)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:266)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
My bash profile is set as follows:
export JAVA_HOME=/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
export M2_HOME=/Users/ashvithavenkateshkumar/Downloads/apache-maven-3.3.9
export PATH=$PATH:$M2_HOME/bin
Can someone please see what is going wrong here.
Also I am getting the error java home not found if the path is given as JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.6.0_65.jdk/Contents/Home
but if I give as JAVA_HOME=/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home then it works fine. (The difference is if i remove the _65 and jdk in front of the version)
I am testing Tomcat7 showing data from the example table "malaga_plagues". I have done changes but, when I try test it again, I obtain the following error creating .war file.
[root#host-192-168-192-78 AMS_Widhoc]# ../../../apache-maven-3.3.3/bin/mvn package
Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/maven/cli/MavenCli : Unsupported major.minor version 51.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:643)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:277)
at java.net.URLClassLoader.access$000(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:212)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClassFromSelf(ClassRealm.java:401)
at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass(SelfFirstStrategy.java:42)
at org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass(ClassRealm.java:271)
at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:254)
at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:239)
at org.codehaus.plexus.classworlds.launcher.Launcher.getMainClass(Launcher.java:144)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:266)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
[root#host-192-168-192-78 AMS_Widhoc]#
One month ago this was working fine, but now I have resumed the example and it does not work. I reinstalled Orion CB in mi VM but I didn't change anything in Tomcat.
Could you help me? Thank you.
Maven 3.3.3 is only compatible with Java 7+, and it seems that you are trying to run it with Java 6. Please check your JAVA_HOME environment variable, it should point to a Java 7 JDK.
I'm having issues with the Eclipse plugin of sonar. When executing it I'm getting the following exception in the Eclipse console:
Exception in thread "main" org.sonar.runner.impl.RunnerException: Unable to execute Sonar
at org.sonar.runner.impl.BatchLauncher$1.delegateExecution(BatchLauncher.java:91)
at org.sonar.runner.impl.BatchLauncher$1.run(BatchLauncher.java:75)
at java.security.AccessController.doPrivileged(Native Method)
at org.sonar.runner.impl.BatchLauncher.doExecute(BatchLauncher.java:69)
at org.sonar.runner.impl.BatchLauncher.execute(BatchLauncher.java:50)
at org.sonar.runner.impl.BatchLauncherMain.execute(BatchLauncherMain.java:41)
at org.sonar.runner.impl.BatchLauncherMain.main(BatchLauncherMain.java:59)
Caused by: org.sonar.api.utils.HttpDownloader$HttpException: Fail to download [http://<server>/batch_bootstrap/db?project=<project>]. Response code: 500
at org.sonar.api.utils.HttpDownloader$BaseHttpDownloader$HttpInputSupplier.getInput(HttpDownloader.java:281)
at org.sonar.api.utils.HttpDownloader$BaseHttpDownloader$HttpInputSupplier.getInput(HttpDownloader.java:235)
at com.google.common.io.ByteStreams.copy(ByteStreams.java:116)
at com.google.common.io.Files.copy(Files.java:231)
at org.sonar.batch.bootstrap.ServerClient.download(ServerClient.java:69)
at org.sonar.batch.bootstrap.PreviewDatabase.downloadDatabase(PreviewDatabase.java:101)
at org.sonar.batch.bootstrap.PreviewDatabase.start(PreviewDatabase.java:69)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.picocontainer.lifecycle.ReflectionLifecycleStrategy.invokeMethod(ReflectionLifecycleStrategy.java:110)
at org.picocontainer.lifecycle.ReflectionLifecycleStrategy.start(ReflectionLifecycleStrategy.java:89)
at org.picocontainer.injectors.AbstractInjectionFactory$LifecycleAdapter.start(AbstractInjectionFactory.java:84)
at org.picocontainer.behaviors.AbstractBehavior.start(AbstractBehavior.java:169)
at org.picocontainer.behaviors.Stored$RealComponentLifecycle.start(Stored.java:132)
at org.picocontainer.behaviors.Stored.start(Stored.java:110)
at org.picocontainer.DefaultPicoContainer.potentiallyStartAdapter(DefaultPicoContainer.java:1015)
at org.picocontainer.DefaultPicoContainer.startAdapters(DefaultPicoContainer.java:1008)
at org.picocontainer.DefaultPicoContainer.start(DefaultPicoContainer.java:766)
at org.sonar.api.platform.ComponentContainer.startComponents(ComponentContainer.java:91)
at org.sonar.api.platform.ComponentContainer.execute(ComponentContainer.java:77)
at org.sonar.batch.bootstrapper.Batch.startBatch(Batch.java:92)
at org.sonar.batch.bootstrapper.Batch.execute(Batch.java:74)
at org.sonar.runner.batch.IsolatedLauncher.execute(IsolatedLauncher.java:45)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.sonar.runner.impl.BatchLauncher$1.delegateExecution(BatchLauncher.java:87)
... 6 more
We're running SonarQube Server 4.1.2, the plugin version is 3.3.0.
Any ideas?
Update
Could this be related to poor DB performance (even though it's not some timeout exception)? Sonar runs on an Oracle DB, and we just updated sonar which resolved issues with excessively long query times (but weirdly enough only for logged in users). I added login information in the plugin settings though, so i assumed it should work.
When opening a source file in Eclipse, I'm getting messages like
Retrieve issues of resource ResultUtil.java...
Done in 43625ms
which sounds like the query would be running way longer that desirable.
Update 2:
We realized that even though the project existed in SonarQube, and was analyzed, it was not provisioned (the project was created with a pre 4.0 version of SonarQube). To rule out that caused the issue we provisioned a new project in SonarQube, ran the analysis on that and configured the eclipse plugin to use this new project.
This did not resolve the issue, either we're getting the same exception, or no further message at all. Calling the URL in a web-browser yields the same result (i.e. endlessly says "waiting for <server>".
The "SonarQube Web Browser" view of the eclipse plugin correctly shows issues, which I take as a sign that the plugin is configured correctly. Also, for some reason querying the issues from SonarQube is now a lot faster, so I would rule out that this caused the issue.
Any more suggestions on what we could check?
Update 3
We tried restarting SonarQube and then tried to analyze the project using the eclipse plugin. This failed with the above exception. Checking the SonarQube logs revealed these issues:
2014.03.19 11:17:35 ERROR [o.s.c.p.DbTemplate] Fail to copy table rules
org.h2.jdbc.JdbcBatchUpdateException: Unique index or primary key violation: "RULES_PLUGIN_KEY_AND_NAME ON PUBLIC.RULES(PLUGIN_RULE_KEY, PLUGIN_NAME)"; SQL statement:
INSERT INTO rules(ID,PLUGIN_RULE_KEY,PLUGIN_NAME,DESCRIPTION,PRIORITY,CARDINALITY,PARENT_ID,PLUGIN_CONFIG_KEY,NAME,STATUS,LANGUAGE,CREATED_AT,UPDATED_AT) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?) [23505-172]
at org.h2.jdbc.JdbcPreparedStatement.executeBatch(JdbcPreparedStatement.java:1167) ~[h2-1.3.172.jar:1.3.172]
at org.apache.commons.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:297) ~[commons-dbcp-1.4.jar:1.4]
at org.apache.commons.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:297) ~[commons-dbcp-1.4.jar:1.4]
at org.sonar.core.persistence.DbTemplate.copyTableColumns(DbTemplate.java:100) [sonar-core-4.1.2.jar:na]
at org.sonar.core.persistence.DbTemplate.copyTableColumns(DbTemplate.java:54) [sonar-core-4.1.2.jar:na]
at org.sonar.core.persistence.DbTemplate.copyTable(DbTemplate.java:49) [sonar-core-4.1.2.jar:na]
at org.sonar.core.persistence.PreviewDatabaseFactory.copy(PreviewDatabaseFactory.java:87) [sonar-core-4.1.2.jar:na]
at org.sonar.core.persistence.PreviewDatabaseFactory.createNewDatabaseForDryRun(PreviewDatabaseFactory.java:63) [sonar-core-4.1.2.jar:na]
at org.sonar.core.preview.PreviewCache.generateNewDB(PreviewCache.java:121) [sonar-core-4.1.2.jar:na]
at org.sonar.core.preview.PreviewCache.getDatabaseForPreview(PreviewCache.java:81) [sonar-core-4.1.2.jar:na]
at org.sonar.server.ui.JRubyFacade.createDatabaseForPreview(JRubyFacade.java:471) [JRubyFacade.class:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_45]
(... snip ...)
followed by
2014.03.19 11:17:35 ERROR [o.s.s.ui.JRubyFacade] Fail to render: http://hulk:9000/batch_bootstrap/db?project=E2PR-12.0
attempt to unlock read lock, not locked by current thread
java.util.concurrent.locks.ReentrantReadWriteLock$Sync.unmatchedUnlockException(ReentrantReadWriteLock.java:447)
java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(ReentrantReadWriteLock.java:431)
java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1340)
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock(ReentrantReadWriteLock.java:883)
org.sonar.core.preview.PreviewCache.getDatabaseForPreview(PreviewCache.java:92)
org.sonar.server.ui.JRubyFacade.createDatabaseForPreview(JRubyFacade.java:471)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:606)
(... snip ...)
Subsequent analysis runs triggered by the eclipse plugin seem to hang at 11:28:46.317 DEBUG - Download: http://hulk:9000/batch_bootstrap/db?project=E2PR-12.0 (no proxy)
I can provide the logfile if that would help you to find out what's wrong.
It looks like your project has not been analyzed in SonarQube prior to running preview analyses in Eclipse. See http://docs.codehaus.org/display/SONAR/Configuring+SonarQube+in+Eclipse.
So, after a lot of digging I was able to resolve this issue. What pointed into the right direction was this discussion: http://comments.gmane.org/gmane.comp.java.sonar.general/33747
In the end this was caused by duplicate manual rules in the RULES table. I have no idea how they ended up there, but they were not visible from the UI, which made it kinda hard to spot.
After deleting these rules in the DB I'm now able to trigger the analysis from eclipse.
I have an application which requires a couple different versions of the same libraries. When running through intellij this isn't a problem because it exports the classpath in a way that avoid the conflicts. I then use the maven shade plugin to create an uber-jar when compiling for production.
However when using the surefire plugin to run my unit tests the conflicts appear again. Does anyone know a way that I can ensure that the sure-fire plugin creates the class path in a way to avoid these conflicts?
Thanks!
UPDATE: Below is the error I am getting (Although it is fairly specific to my application). Basically there are two versions of Saxon HE and PE which are both being loaded. When running the surefire tests it causes this exception:
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.jorsek.appserver.bootstrap.Main.invokeMain(Main.java:130)
at com.jorsek.appserver.bootstrap.Main.run(Main.java:449)
at framework.TestUtils.startupDatabase(TestUtils.java:41)
at framework.AbstractTestInstance.startDatabase(AbstractTestInstance.java:41)
at framework.AbstractTestInstance.startDatabase(AbstractTestInstance.java:55)
at org.jorsek.exist.versioning.AllSetslTest.setUpBeforeClass(AllSetslTest.java:80)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:59)
at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:120)
at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:103)
at org.apache.maven.surefire.Surefire.run(Surefire.java:169)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:350)
at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1021)
Caused by: java.util.ServiceConfigurationError: javax.xml.transform.TransformerFactory: Provider com.saxonica.config.ProfessionalTransformerFactory could not be instantiated: java.lang.VerifyError: (class: com/saxonica/config/ProfessionalConfiguration, method: newExpressionParser signature: (Ljava/lang/String;ZLnet/sf/saxon/value/DecimalValue;)Lnet/sf/saxon/expr/ExpressionParser;) Wrong return type in function
at java.util.ServiceLoader.fail(ServiceLoader.java:207)
at java.util.ServiceLoader.access$100(ServiceLoader.java:164)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:360)
at java.util.ServiceLoader$1.next(ServiceLoader.java:428)
at org.exist.validation.XmlLibraryChecker.check(XmlLibraryChecker.java:162)
at com.jorsek.appserver.jetty.JettyStart.<init>(JettyStart.java:95)
at com.jorsek.appserver.jetty.JettyStart.main(JettyStart.java:77)
... 30 more
Caused by: java.lang.VerifyError: (class: com/saxonica/config/ProfessionalConfiguration, method: newExpressionParser signature: (Ljava/lang/String;ZLnet/sf/saxon/value/DecimalValue;)Lnet/sf/saxon/expr/ExpressionParser;) Wrong return type in function
at com.saxonica.config.ProfessionalTransformerFactory.<init>(ProfessionalTransformerFactory.java:24)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:357)
at java.lang.Class.newInstance(Class.java:310)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:356)
... 34 more
Perhaps you can override the other dependency on test scope as well? Or if need be, just override them both on default scope. Or is there some reason why you cannot do either of the above?
Or an alternative of course is to start dabbling with the classloader during your test runs, like shown here.