i have a naven application in which i use Birt 4.6. Below my dependencies.
<dependency>
<groupId>org.eclipse.birt.ojdbc</groupId>
<artifactId>odajdbc</artifactId>
<version>4.6.0-201606072122</version>
</dependency>
<dependency>
<groupId>org.eclipse.birt.runtime</groupId>
<artifactId>org.eclipse.birt.runtime</artifactId>
<version>4.6.0-20160607</version>
<exclusions>
<exclusion>
<groupId>org.eclipse.birt.runtime</groupId>
<artifactId>org.apache.xerces</artifactId>
</exclusion>
<exclusion>
<artifactId>org.apache.poi</artifactId>
<groupId>org.eclipse.birt.runtime</groupId>
</exclusion>
</exclusions>
</dependency>
I am able to connect with the database and generate the reports. Those are the good news.
Unfortunately, i noticed in my log file that there is an exception thrown. The exception can seen below
2017-01-10 14:57:15,446 SEVERE [org.eclipse.birt.report.data.oda.jdbc.JDBCDriverManager] (default task-6) DriverClassLoader failed to load class: oracle.jdbc.driver.OracleDriver: java.lang.ClassNotFoundException: oracle.jdbc.driver.OracleDriver
at org.eclipse.birt.core.framework.URLClassLoader.findClass1(URLClassLoader.java:188)
at org.eclipse.birt.core.framework.URLClassLoader$1.run(URLClassLoader.java:156)
at org.eclipse.birt.core.framework.URLClassLoader$1.run(URLClassLoader.java:1)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.birt.core.framework.URLClassLoader.findClass(URLClassLoader.java:151)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.eclipse.birt.report.data.oda.jdbc.JDBCDriverManager.loadExtraDriver(JDBCDriverManager.java:1064)
at org.eclipse.birt.report.data.oda.jdbc.JDBCDriverManager.findDriver(JDBCDriverManager.java:859)
at org.eclipse.birt.report.data.oda.jdbc.JDBCDriverManager.loadAndRegisterDriver(JDBCDriverManager.java:986)
at org.eclipse.birt.report.data.oda.jdbc.JDBCDriverManager.loadAndRegisterDriver(JDBCDriverManager.java:958)
at org.eclipse.birt.report.data.oda.jdbc.JDBCDriverManager.doConnect(JDBCDriverManager.java:285)
at org.eclipse.birt.report.data.oda.jdbc.JDBCDriverManager.getConnection(JDBCDriverManager.java:236)
at org.eclipse.birt.report.data.oda.jdbc.Connection.connectByUrl(Connection.java:254)
at org.eclipse.birt.report.data.oda.jdbc.Connection.open(Connection.java:163)
at org.eclipse.datatools.connectivity.oda.consumer.helper.OdaConnection.open(OdaConnection.java:250)
at org.eclipse.birt.data.engine.odaconsumer.ConnectionManager.openConnection(ConnectionManager.java:165)
at org.eclipse.birt.data.engine.executor.DataSource.newConnection(DataSource.java:224)
at org.eclipse.birt.data.engine.executor.DataSource.open(DataSource.java:212)
at org.eclipse.birt.data.engine.impl.DataSourceRuntime.openOdiDataSource(DataSourceRuntime.java:217)
at org.eclipse.birt.data.engine.impl.QueryExecutor.openDataSource(QueryExecutor.java:437)
at org.eclipse.birt.data.engine.impl.QueryExecutor.prepareExecution(QueryExecutor.java:325)
at org.eclipse.birt.data.engine.impl.PreparedQuery.doPrepare(PreparedQuery.java:463)
at org.eclipse.birt.data.engine.impl.PreparedDataSourceQuery.produceQueryResults(PreparedDataSourceQuery.java:190)
at org.eclipse.birt.data.engine.impl.PreparedDataSourceQuery.execute(PreparedDataSourceQuery.java:178)
at org.eclipse.birt.data.engine.impl.PreparedOdaDSQuery.execute(PreparedOdaDSQuery.java:179)
at org.eclipse.birt.report.data.adapter.impl.DataRequestSessionImpl.execute(DataRequestSessionImpl.java:651)
at org.eclipse.birt.report.engine.data.dte.DteDataEngine.doExecuteQuery(DteDataEngine.java:152)
at org.eclipse.birt.report.engine.data.dte.AbstractDataEngine.execute(AbstractDataEngine.java:286)
at org.eclipse.birt.report.engine.executor.ExecutionContext.executeQuery(ExecutionContext.java:1947)
at org.eclipse.birt.report.engine.executor.QueryItemExecutor.executeQuery(QueryItemExecutor.java:80)
at org.eclipse.birt.report.engine.executor.TableItemExecutor.execute(TableItemExecutor.java:62)
at org.eclipse.birt.report.engine.internal.executor.dup.SuppressDuplicateItemExecutor.execute(SuppressDuplicateItemExecutor.java:43)
at org.eclipse.birt.report.engine.internal.executor.wrap.WrappedReportItemExecutor.execute(WrappedReportItemExecutor.java:46)
at org.eclipse.birt.report.engine.internal.executor.l18n.LocalizedReportItemExecutor.execute(LocalizedReportItemExecutor.java:34)
at org.eclipse.birt.report.engine.layout.html.HTMLBlockStackingLM.layoutNodes(HTMLBlockStackingLM.java:65)
at org.eclipse.birt.report.engine.layout.html.HTMLPageLM.layout(HTMLPageLM.java:92)
at org.eclipse.birt.report.engine.layout.html.HTMLReportLayoutEngine.layout(HTMLReportLayoutEngine.java:100)
at org.eclipse.birt.report.engine.api.impl.RunAndRenderTask.doRun(RunAndRenderTask.java:181)
at org.eclipse.birt.report.engine.api.impl.RunAndRenderTask.run(RunAndRenderTask.java:77)
For some reason the JDBCDriverManager struggles to find the correct driver, throws the exception , finally finds the driver connects to the database and generates the report.
I did a debug on JDBCDriverManager and hope that the information below does help a bit.
The app goes through the doConnect() function JDBCDriverManager. Inside there the Connection getJndiDSConnection( driverClass, jndiNameUrl, connectionProperties ); returns null . Same happens for the getJndiDSConnection in the doConnect. also returns null
Then the loadAndRegisterDriver( driverClass, driverClassPath ); is called with following arguments oracle.jdbc.driver.OracleDriver and null
Inside the loadAndRegisterDriver the findDriver( className, driverClassPath, refreshClassLoader ) is called with following arguments oracle.jdbc.driver.OracleDriver , null, false
On the next step driverClass = loadExtraDriver( className, true, refresh, driverClassPath ); is called with oracle.jdbc.driver.OracleDriver , true , false , null which throws the ClassNotFoundException mentioned above.
Final step, we are still inside findDriver method where the driver = this.getDriverInstance( driverClass, refresh ); method is called and finally returns oracle.jdbc.driver.OracleDriver .
After step 5 everything works fine. As i mentioned, the exception appears only one time and still the connection with the database is created and the reports are generated. After that, no matter how many times i create a report, the exception is never thrown again.
I would like here to add some further info about the findDriver method. This method tries in several ways to get the driver. First is
// Driver not in plugin class path; find it in drivers directory
driverClass = loadExtraDriver( className, true, refresh, driverClassPath );
Which returns null and then gives a try to get the driver from the context
driverClass = Class.forName( className, true, Thread.currentThread( ).getContextClassLoader());
This times it finally achieves to retrieve the driver.
What am i missing? It is clear that it cannot load it from the plugins since i do not have any plugins directory. Is there a way to overcome this exception ?
As Mark mentioned, there was no need to add as a dependency the org.eclipse.birt.ojdbc . I stopped using the org.eclipse.birt.report.data.oda.jdbc_4.6.0.v201606072122.jar and used my local ojdbc driver.
<dependency>
<groupId>com.oracle</groupId>
<artifactId>ojdbc6</artifactId>
<version>11.2.0.4.0</version>
</dependency>
The above fixes the exception we get on the first try to load the driver.
Adding ojdbc7.jar under the WEB-INF path of the Birt Viewer folder (Web/App Server side) solved the problem for me:
[1] ../lib
[2] ../platform/plugins/org.eclipse.birt.report.data.oda.jdbc_<VERSION>/drivers
Logs
Before adding [2] above (was having only [1]):
20-Mar-2017 14:12:26.752 SEVERE [http-nio-8080-exec-4] org.eclipse.birt.report.data.oda.jdbc.JDBCDriverManager.loadExtraDriver DriverClassLoader failed to load class: oracle.jdbc.driver.OracleDriver java.lang.ClassNotFoundException: oracle.jdbc.driver.OracleDriver
After adding [2] above (was having only [1]):
20-Mar-2017 14:49:42.196 INFO [http-nio-8080-exec-4] org.eclipse.birt.report.data.oda.jdbc.JDBCDriverManager$DriverClassLoader.refreshFileURL JDBCDriverManager: found JAR file ojdbc7.jar. URL=file:../WEB-INF/platform/plugins/org.eclipse.birt.report.data.oda.jdbc_4.6.0.v201606072122/drivers/ojdbc7.jar
Related
I am trying to upgrade my JMeter DSL implementation to the latest JMeter version(5.4.3). But I got an issue with ConcurrencyThreadGroup object creation, it throws an exception. See the below exception
Using following versions
JMeter 5.4.3
jmeter-plugins-standard 1.4.0
Method implementation
public ConcurrencyThreadGroup getConcurrencyThreadGroup(String name, String targetConcurrency,
String rampUpTime, String rampUpStepCount, String timeUnit, String holdTargetTime,
boolean setEnabled
) {
ConcurrencyThreadGroup concurrencyThreadGroup = new ConcurrencyThreadGroup();
concurrencyThreadGroup.setName(name);
concurrencyThreadGroup.setTargetLevel(targetConcurrency);
concurrencyThreadGroup.setRampUp(rampUpTime);
concurrencyThreadGroup.setSteps(rampUpStepCount);
concurrencyThreadGroup.setUnit(timeUnit);
concurrencyThreadGroup.setHold(holdTargetTime);
concurrencyThreadGroup.setEnabled(setEnabled);
concurrencyThreadGroup.setProperty("TestElement.test_class", ConcurrencyThreadGroup.class.getName());
concurrencyThreadGroup.setProperty("TestElement.gui_class", ConcurrencyThreadGroupGui.class.getName());
return concurrencyThreadGroup;
}
Observing below exception when try to execute
at org.apache.jmeter.reporters.ResultCollector.<init>(ResultCollector.java:167)
at org.apache.jmeter.reporters.ResultCollector.<init>(ResultCollector.java:157)
at com.blazemeter.jmeter.reporters.FlushingResultCollector.<init>(FlushingResultCollector.java:7)
at com.blazemeter.jmeter.threads.AbstractDynamicThreadGroupModel.<init>(AbstractDynamicThreadGroupModel.java:28)
at com.blazemeter.jmeter.threads.AbstractDynamicThreadGroup.<init>(AbstractDynamicThreadGroup.java:23)
at com.blazemeter.jmeter.threads.concurrency.ConcurrencyThreadGroup.<init>(ConcurrencyThreadGroup.java:11)
at org.qa.perf.jmeter.api.threadgroups.QADSLThreadGroup.getConcurrencyThreadGroup(QADSLThreadGroup.java:136)
at org.qa.perf.jmeter.dsl.QADSLJMeterDSL.concurrencyThreadGroup(QADSLJMeterDSL.java:93)
at org.qa.perf.dsl.sample.threadgroup.ConcurrencyThreadGroupSampleTest$1.prepareJMeterTestPlan(ConcurrencyThreadGroupSampleTest.java:20)
at org.qa.perf.jmeter.engine.QADSLPerfTestClient.executeTest(QADSLPerfTestClient.java:29)
at org.qa.perf.dsl.sample.threadgroup.ConcurrencyThreadGroupSampleTest.testConcurrencyThreadGroupSampleTest(ConcurrencyThreadGroupSampleTest.java:29)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:599)
at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:174)
at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822)
at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.testng.TestRunner.privateRun(TestRunner.java:764)
at org.testng.TestRunner.run(TestRunner.java:585)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337)
at org.testng.SuiteRunner.run(SuiteRunner.java:286)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
at org.testng.TestNG.runSuites(TestNG.java:1069)
at org.testng.TestNG.run(TestNG.java:1037)
at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:66)
at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:109)
Caused by: java.lang.NullPointerException
at org.apache.jmeter.samplers.SampleSaveConfiguration.<clinit>(SampleSaveConfiguration.java:287)
... 38 more
Appreciate any clue or solution to solve this issue.
You're supposed to show your full code and full stacktrace as the partials unfortunately don't tell the full story.
Most probably you didn't load JMeter Properties which are responsible for the Results File Configuration so my expectation is that you need to call the following function somewhere in the beginning of your code:
org.apache.jmeter.util.JMeterUtils.loadJMeterProperties("/path/to/your/jmeter.properties")
More information:
JMeter API
Five Ways To Launch a JMeter Test without Using the JMeter GUI
jmeter-from-code example project
I am trying to add elements to TitanDB using SparkStreaming (collecting messages from a Kafka queue). But it seems that it's harder than expected.
Here the definition of the Titan connection:
val confPath: String = "titan-cassandra-es-spark.properties"
val conn: TitanModule = new TitanModule(confPath)
Titan module is a Serializable class that configure the TitanDB connection:
...
val configurationFilePath: String = confFilePath
val configuration = new PropertiesConfiguration(configurationFilePath)
val gConn: TitanGraph = TitanFactory.open(configuration)
...
When I execute the sparkStreaming job that collect messages (json) from a Kafka queue, it receive the message and trying to add it into TitanDB, it explodes with the following stackTrace.
Do you guys know if adding data into TitanDB is feasible with SparkStreaming?
Do you know what could be the solution for this?
18:03:50,596 ERROR JobScheduler:95 - Error running job streaming job 1464624230000 ms.0
org.apache.spark.SparkException: Task not serializable
at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:304)
at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:294)
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:122)
at org.apache.spark.SparkContext.clean(SparkContext.scala:2055)
at org.apache.spark.rdd.RDD$$anonfun$foreach$1.apply(RDD.scala:911)
at org.apache.spark.rdd.RDD$$anonfun$foreach$1.apply(RDD.scala:910)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
at org.apache.spark.rdd.RDD.foreach(RDD.scala:910)
at salvob.SparkConsumer$$anonfun$main$1.apply(SparkConsumer.scala:200)
at salvob.SparkConsumer$$anonfun$main$1.apply(SparkConsumer.scala:132)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:50)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:50)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:50)
at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:426)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:49)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:49)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:49)
at scala.util.Try$.apply(Try.scala:161)
at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:224)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:224)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:224)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:223)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.NotSerializableException: org.apache.commons.configuration.PropertiesConfiguration
Serialization stack:
- object not serializable (class: org.apache.commons.configuration.PropertiesConfiguration, value: org.apache.commons.configuration.PropertiesConfiguration#2cef9ce8)
- field (class: salvob.TitanModule, name: configuration, type: class org.apache.commons.configuration.PropertiesConfiguration)
- object (class salvob.TitanModule, salvob.TitanModule#20d984db)
- field (class: salvob.SparkConsumer$$anonfun$main$1$$anonfun$apply$3, name: conn$1, type: class salvob.TitanModule)
- object (class salvob.SparkConsumer$$anonfun$main$1$$anonfun$apply$3, <function1>)
at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)
at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:47)
at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:101)
at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:301)
... 28 more
Exception in thread "main" org.apache.spark.SparkException: Task not serializable
at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:304)
at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:294)
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:122)
at org.apache.spark.SparkContext.clean(SparkContext.scala:2055)
at org.apache.spark.rdd.RDD$$anonfun$foreach$1.apply(RDD.scala:911)
at org.apache.spark.rdd.RDD$$anonfun$foreach$1.apply(RDD.scala:910)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
at org.apache.spark.rdd.RDD.foreach(RDD.scala:910)
at salvob.SparkConsumer$$anonfun$main$1.apply(SparkConsumer.scala:200)
at salvob.SparkConsumer$$anonfun$main$1.apply(SparkConsumer.scala:132)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:50)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:50)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:50)
at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:426)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:49)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:49)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:49)
at scala.util.Try$.apply(Try.scala:161)
at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:224)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:224)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:224)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:223)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.NotSerializableException: org.apache.commons.configuration.PropertiesConfiguration
Serialization stack:
- object not serializable (class: org.apache.commons.configuration.PropertiesConfiguration, value: org.apache.commons.configuration.PropertiesConfiguration#2cef9ce8)
- field (class: salvob.TitanModule, name: configuration, type: class org.apache.commons.configuration.PropertiesConfiguration)
- object (class salvob.TitanModule, salvob.TitanModule#20d984db)
- field (class: salvob.SparkConsumer$$anonfun$main$1$$anonfun$apply$3, name: conn$1, type: class salvob.TitanModule)
- object (class salvob.SparkConsumer$$anonfun$main$1$$anonfun$apply$3, <function1>)
at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)
at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:47)
at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:101)
at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:301)
... 28 more
Spark Streaming produces RDDs. Processing of the data inside the RDDs happens on the worker nodes. The code you write inside rdd.map(), is serialised along with the objects which are referenced inside that block and sent to the worker node for processing.
So ideal way to use the graph instance through Spark is the following :
streamRdd.map(kafkaTuple => {
// create graph instance
// use graph instance to add / modify graph
// close graph instance
})
But this will create a new graph instance for each row. As an optimisation, you can create the graph instance per instance
rdd.foreachPartition((rddRows: Iterator[kafkaTuple]) => {
val graph: TitanGraph = // create titan instance
val trans: TitanTransaction = graph.newTransaction()
rddRows.foreach(graphVertex => {
// do graph insertion in the above transaction
})
createVertexTrans.commit()
graph.close()
})
graph.newTransaction() here helps in multi threaded graph updates. Other wise you will get lock exceptions.
Only thing is that, according to what I have read so far, There is no direct support for multi node update. From what I saw, Titan Transaction updates HBase with a lock whenever it tries to modify a vertex. So other partitions will fail when they try to do any updates. You will have to build an external synchronisation mechanism or repartition your rdd into a single partition and then use the above code to do updates.
Make sure that all classes that could be passed to other slave machines are Serializable. It's quite important. Do not initialize any variables outside of these passed classes.
I have used Apache Spark (Not Streaming) and it worked well. It wasn't quite easy to get it right since Titan uses a version of Spark. So there would be some dependency conflicts. This is the only version that would work
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.2.2</version>
</dependency>
This is how I started the cluster.
SparkConf conf = new SparkConf()
.setAppName(AbstractSparkImporter.class.getCanonicalName())
.setMaster("spark_cluster_name");
this.sc = new JavaSparkContext(conf);
this.numPartitions=new Integer(num);
Then parse the data
JavaRDD<T> javaRDD = initRetriever(); // init JavaRDD
javaRDD.foreachPartition(iter->{
Graph graph= initGraph();
Parser<T> parser= initParser(graph);
while(iter.hasNext()){
try {
parser.parse(iter); // extends serializable !
} catch (Exception e) {
logger.error("Failed in importing all vertices ", e);
graph.tx().rollback();
}
}
graph.tx().commit();
});
I might be able to release this module on Github if it's necessary.
I want to set up a Gatling testcase to put messages on a Oracle AQ. But I have no idea how to set up the the following:
val jmsConfig = jms
.connectionFactoryName(???)
.url("tcp://localhost:10002")
.credentials("user", "secret")
.contextFactory(???)
.listenerCount(1)
.usePersistentDeliveryMode
What is the connection factory name and what is the context factory?
I managed to get it working using the oracle.jms.AQjmsInitialContextFactory. The above mentioned oracle.jms.AQjmsFactory is not an InitialContextFactory so that wont work.
Make sure to add at least version 11+ of the Oracle AQ dependencies for the AQjmsInitialContextFactory to be found.
You database user should of-course have the correct privileges to be able to insert messages into the queue (table).
Gatling expects your to have request-reply semantics, so it will wait for a reply to be received. I actually wanted to abort waiting for a reply after a specified period, but I have no clue how to do that. So if anyone know how to.. please tell me :-)
MySimulation.scala
val jmsConfig = jms
.connectionFactoryName("ConnectionFactory") // MUST!!!! be called ConnectionFactory, the AQjmsFactory expects this naming convention!
.url("jdbc:oracle:thin:#host:1521:SID")
.credentials("user", "password")
.contextFactory("oracle.jms.AQjmsInitialContextFactory")
.listenerCount(1)
.usePersistentDeliveryMode
// TODO check how to set a timeout on the reply
val jmsScenario = scenario("JMS DSL test")
.repeat(1) {
exec(
jms("req reply testing")
.reqreply
.queue("AQ_ADMIN.QUEUE_NAME")
.textMessage("some message")
.check(simpleCheck(checkBodyTextCorrect))
)
}
def checkBodyTextCorrect(m: Message) = {
// this assumes that the service just does an "uppercase" transform on the text
m match {
case tm: TextMessage => tm.getText == "text that should be in the reply message"
case _ => false
}
}
setUp(jmsScenario.inject(atOnceUsers(1)).protocols(jmsConfig));
jndi.properties
I had to add the jndi.properties to the classpath:
db_url=jdbc:oracle:thin:#host:1521:SID
pom.xml
Dependencies (maven):
<dependency>
<groupId>oracle</groupId>
<artifactId>aqapi</artifactId>
<version>11.2.0.3</version>
</dependency>
<dependency>
<groupId>com.oracle</groupId>
<artifactId>ojdbc6</artifactId>
<version>11.2.0.2.0</version>
</dependency>
<dependency>
<groupId>javax.transaction</groupId>
<artifactId>jta</artifactId>
<version>1.1</version>
</dependency>
"contextFactory" is the class name of your ContextFactory. Doc seems to state it's "oracle.jms.AQjmsFactory".
"connectionFactoryName" is the key used for the JNDI look-up. Doc again seems to state it's "cn=OracleDBConnections".
I'm trying to write a prototype for a project that involves having java use carrot2 as a metasearch engine for several sources, such as bing and google , etc.
I've got a maven project with dependency :
<dependency>
<groupId>org.carrot2</groupId>
<artifactId>carrot2-core</artifactId>
<version>3.9.3</version>
</dependency>
I'm trying to run the following :
/* A controller to manage the processing pipeline. */
Controller controller = ControllerFactory.createSimple();
/* Input data for clustering, the query and number of results in this case. */
Map<String, Object> attributes = new HashMap<String, Object>();
attributes.put(AttributeNames.QUERY, "sugar");
attributes.put(AttributeNames.RESULTS, 100);
/* Perform processing */
ProcessingResult result = controller.process(attributes,
Bing3DocumentSource.class, LingoClusteringAlgorithm.class);
/* Documents fetched from the document source, clusters created by Carrot2. */
List<Document> documents = result.getDocuments();
List<Cluster> clusters = result.getClusters();
What I get is :
Exception in thread "main" org.carrot2.core.ComponentInitializationException: Could not instantiate component class: org.carrot2.source.microsoft.Bing3DocumentSource
at org.carrot2.core.SimpleProcessingComponentManager.prepare(SimpleProcessingComponentManager.java:68)
at org.carrot2.core.Controller.process(Controller.java:341)
at org.carrot2.core.Controller.process(Controller.java:246)
at com.jbaysolutions.metasearch.Test.main(Test.java:41)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
Caused by: java.lang.InstantiationException: org.carrot2.source.microsoft.Bing3DocumentSource
at java.lang.Class.newInstance(Class.java:359)
at org.carrot2.core.SimpleProcessingComponentManager.prepare(SimpleProcessingComponentManager.java:55)
... 8 more
Am I using the API correctly ? I've tried going over the documentation of carrot2 but it goes very little into the usage of the API, and also the examples don't seam to work.
Could really use some help here
Answer from Dawid Weiss on the carrot2 mailing list:
You're trying to instantiate an abstract class. Won't fly unless
you're Chuck Norris.
Why not look at the examples distributed with the project? There is
an example that uses Bing there.
https://github.com/carrot2/carrot2/blob/master/applications/carrot2-examples/examples/org/carrot2/examples/clustering/ClusteringDataFromDocumentSources.java#L111
All the examples are here, packaged and ready:
http://project.carrot2.org/download-java-api.html
If you're planning to use Bing make sure you use your own appkey,
please (and thanks).
The part in question is :
ProcessingResult result = controller.process(attributes,
Bing3DocumentSource.class, LingoClusteringAlgorithm.class);
That should instead read :
ProcessingResult result = controller.process(attributes,
Bing3WebDocumentSource.class, LingoClusteringAlgorithm.class);
The application is based on OSGI.
I have a custom annotated converter:
package com.domain.bundle1.web.camel.converters;
import ...;
#Converter
public class FooTransferObjectConverter {
public FooTransferObjectConverter() {
}
#Converter
public static FooTransferObject toFooTransferObject(Foo foo, Exchange exchange) throws Exception {
// some magic
return fooTransferObject;
}
}
Also i declared package where it plased in TypeConverter file:
http://i.stack.imgur.com/U3QQH.png
which contains:
com.domain.bundle1.web.camel.converters
And camel-context file contains next code:
<log loggingLevel="INFO" message="Converting to FooTransferObject" />
<convertBodyTo type="com.domain.bundle2.model.FooTransferObject" />
<log loggingLevel="INFO" message="Converted!" />
Before converting, body of message is a Foo object.
But when process reaches converting, then throws an exception:
Failed delivery for (MessageId: ID-EPUALVIW0567-55536-1401106375216-26-5 on ExchangeId: ID-EPUALVIW0567-55536-1401106375216-26-6).
Exhausted after delivery attempt: 1 caught: org.apache.camel.InvalidPayloadException: No body available of type: com.domain.bundle2.model.FooTransferObject but has value: Foo{97, Wall, null, null} of type: com.domain.bundle3.model.Foo on: Message: Foo{97, Wall, null, null}.
Caused by: Error during type conversion from type: com.domain.bundle3.model.Foo to the required type: com.domain.bundle2.model.FooTransferObject with value Foo{97, Wall, null, null} due 6 counts of IllegalAnnotationExceptions. Exchange[Message: Foo{97, Wall, null, null}]. Caused by: [org.apache.camel.TypeConversionException - Error during type conversion from type: Foo{97, Wall, null, null} to the required type: com.domain.bundle2.model.FooTransferObjec with value....
then exception cached by custom handler,
and then I found this:
Caused by: javax.xml.bind.MarshalException
- with linked exception:
[com.sun.istack.internal.SAXException2: A cycle is detected in the object graph. This will cause infinitely deep XML: freebaseball SpeedKick -> fr????f????tb??ll Sp????dK??ck -> free
football SpeedKick ]
at com.sun.xml.internal.bind.v2.runtime.MarshallerImpl.write(MarshallerImpl.java:311)[:1.7.0_40]
at com.sun.xml.internal.bind.v2.runtime.MarshallerImpl.marshal(MarshallerImpl.java:236)[:1.7.0_40]
at javax.xml.bind.helpers.AbstractMarshallerImpl.marshal(AbstractMarshallerImpl.java:95)
at org.apache.camel.converter.jaxb.FallbackTypeConverter.marshall(FallbackTypeConverter.java:238)
at org.apache.camel.converter.jaxb.FallbackTypeConverter.convertTo(FallbackTypeConverter.java:95)
... 163 more
Caused by: com.sun.istack.internal.SAXException2: A cycle is detected in the object graph. This will cause infinitely deep XML: freebaseball SpeedKick -> fr????f????tb??ll Sp????dK??c
k -> freebaseball SpeedKick
at com.sun.xml.internal.bind.v2.runtime.XMLSerializer.reportError(XMLSerializer.java:237)[:1.7.0_40]
How do You think what's a problem? How can I see loaded converters in TypeConverterRegistry?
I have already solved my problem. FallbackTypeConverter started work, because camel didn't load my custom regular type converter.
I checked the map of converters in TypeConverterRegister in debug mode, and didn't find my FooTransferObjectConverter.
The problem was in file TypeConverter. I just added name of converter class to path and after that it loaded to registry.
com.domain.bundle1.web.camel.converters.FooTransferObjectConverter
Camel version in application - 2.11.1. In camel docs written next:
In Camel 2.8 we improved the type converter loader to support
specifying the FQN class name of the converter classes. This has the
advantage of avoiding having to scan packages for #Converter classes.
Instead it loads the #Converter class directly. This is a highly
recommend approach to use going forward.
But i tryed run the application from chapter 3 (from 'Camel in action' book) with custom converter. And file TypeConverter contained only from package path.