Flink JDBCInputFormat cannot find method 'setRowTypeInfo' - jdbc

I want to use flink-jdbc to get data from mysql。
I have seen an example on Apache flink website
// Read data from a relational database using the JDBC input format
DataSet<Tuple2<String, Integer> dbData =
env.createInput(
JDBCInputFormat.buildJDBCInputFormat()
.setDrivername("org.apache.derby.jdbc.EmbeddedDriver")
.setDBUrl("jdbc:derby:memory:persons")
.setQuery("select name, age from persons")
.setRowTypeInfo(new RowTypeInfo(BasicTypeInfo.STRING_TYPE_INFO, BasicTypeInfo.INT_TYPE_INFO))
.finish()
);
But when i try to write a demo, i can't find the method 'setRowTypeInfo'.
It was like this
import org.apache.flink.api.common.typeinfo.BasicTypeInfo
import org.apache.flink.api.java.ExecutionEnvironment
import org.apache.flink.api.java.io.jdbc.JDBCInputFormat
import org.apache.flink.api.scala._
/**
* Created by lulijun on 17/7/7.
*/
object FlinkJDBC {
def main(args:Array[String]): Unit = {
val env = ExecutionEnvironment.createLocalEnvironment()
val dbData = env.createInput(
JDBCInputFormat.buildJDBCInputFormat
.setDrivername("com.mysql.jdbc.Driver")
.setDBUrl("XXX")
.setUsername("xxx")
.setPassword("XXX")
.setQuery("select name, age from persons")
.setRowTypeInfo(new Nothing(BasicTypeInfo.STRING_TYPE_INFO, BasicTypeInfo.INT_TYPE_INFO))
.finish)
dbData.print()
env.execute()
}
}
The "setRowTypeInfo" method is always red, and the IDEA prompts
"cannot resolve symbol setRowTypeInfo"
The jar version of flink-jdbc i used is 1.0.0.
<dependencies>
<!-- Use this dependency if you are using the DataSet API -->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-scala_2.10</artifactId>
<version>1.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients_2.10</artifactId>
<version>1.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-jdbc</artifactId>
<version>1.0.0</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.36</version>
</dependency>
</dependencies>
I have searched a lot, and most of the people use the method exactly like the official document, but on one mentioned this problem.
I doubt whether I used the wrong version of flink-jdbc, but I cannot get any information about the right way to use flink-jdbc.
If you know the problem, please teach me.Thank you.

I changed the flink-jdbc version from 1.0.0 to 1.3.0 and the problem solved.
But when I search flink-jdbc on maven websit
https://mvnrepository.com/search?q=flink-jdbc, I can't get the right information in the first few pages, It makes me thought the version of flink-jdbc do not need to be matched with other flink jars.
But the truth is flink-jdbc/1.1.3 use class RowTypeInfo of package api.table, but flink-jdbc/1.3.0 use class RowTypeInfo of package api.java.They have close ties with each other.
We must make sure the version is matched.

Related

Error while trying to do POStagging: Error while loading a tagger model (probably missing model file)

I am trying to use StanfordNLP for croatian using windows command prompt. I have downloaded the specific model for this language (hr_set_models) with .pt files.
I have created the .properties file but I get the following message:
Exception in thread "main" edu.stanford.nlp.io.RuntimeIOException: Error while loading a tagger model (probably missing model file)
There is no problem for the tokenizer model and the file hr_set_tagger.pt is in the folder.
I see that in the model folder there is also a file named hr_set.pretrain.pt, I do not know if I should use it in the .properties file.
Thanks in advance!
Bellow is the .properties file I have created.
annotators = tokenize, ssplit, pos, lemma, depparse
# tokenize
tokenize.model = hr_set_models/hr_set_tokenizer.pt
# pos
pos.model = hr_set_models/hr_set_tagger.pt
# lemma
lemma.model = hr_set_models/hr_set_lemmatizer.pt
#depparse
depparse.model = hr_set_models/hr_set_parser.pt
You need to use the full Python system. There are no Java models for Croatian, so you shouldn't be using the Stanford CoreNLP server.
There is more documentation here: https://stanfordnlp.github.io/stanfordnlp/pipeline.html
Try to use
<dependencies>
<dependency>
<groupId>edu.stanford.nlp</groupId>
<artifactId>stanford-corenlp</artifactId>
<version>3.6.0</version>
</dependency>
<dependency>
<groupId>edu.stanford.nlp</groupId>
<artifactId>stanford-corenlp</artifactId>
<version>3.6.0</version>
<classifier>models</classifier>
</dependency>
</dependencies>

How to use Evaluators in Java to score on a PMML using org.apache.spark?

I've implemented the code for scoring on a provided PMML file and a csv data file (Linear Regression) using Spark and Java. For this I've used jpmml-evaluator-spark and spark-mllib_2.11 maven artifacts, and it works fine.
Now, I'm looking at replacing jpmml-evaluator-spark library, which is AGPL licensed, to something similar may be bundled within org-apache-spark (or any other fully open source option)
I don't see Evaluators for scoring on a PMML available in org.apache.spark group of dependencies. Please confirm if this is correct and suggest some alternative.
https://github.com/jpmml/jpmml-evaluator-spark
This is the PMML evaluator library for the Apache Spark cluster computing system (http://spark.apache.org/) and is AGPL.
Also refer to: http://spark.apache.org/docs/latest/ml-guide.html
These suggest that whatever is packaged along with apache spark includes algorithms and model creation and training, but scoring on the model is not available here & has its dependencies included in the jpmml-evaluator-spark only.
import org.apache.spark.ml.Transformer;
import org.apache.spark.sql.Dataset;
import org.jpmml.evaluator.Evaluator;
import org.jpmml.evaluator.EvaluatorBuilder;
import org.jpmml.evaluator.LoadingModelEvaluatorBuilder;
import org.jpmml.evaluator.spark.TransformerBuilder;
...
...
...
EvaluatorBuilder evaluatorBuilder = new LoadingModelEvaluatorBuilder().setLocatable(false)
.setVisitors(new DefaultVisitorBattery()).load(pmmlInputStream);
Evaluator evaluator = evaluatorBuilder.build();
evaluator.verify();
TransformerBuilder pmmlTransformerBuilder = new TransformerBuilder(evaluator).withLabelCol("Predicted_SpeciesCategory").exploded(true);
Transformer pmmlTransformer = pmmlTransformerBuilder.build();
Dataset<?> resultDataset = pmmlTransformer.transform(csvDataset);
...
...
Maven dependencies:
<dependency>
<groupId>org.jpmml</groupId>
<artifactId>jpmml-evaluator-spark</artifactId>
<version>1.2.2</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.4.3</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.11</artifactId>
<version>2.4.3</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.jpmml</groupId>
<artifactId>jpmml-sparkml</artifactId>
<version>1.5.4</version>
</dependency>
This code still has dependency on org.jpmml library, which I wish to remove. Looking for an alternative using org.apache.spark library to achieve similar results.
You could use the PMML4S-Spark to evaluate a PMML model against Spark, for example:
import org.pmml4s.spark.ScoreModel
val model = ScoreModel.fromInputStream(pmmlInputStream)
val resultDataset = model.transform(csvDataset)
If you want to use PMML4S-Spark in Java, it's also easy to use and similar as Scala, for example:
import org.pmml4s.spark.ScoreModel;
import org.apache.spark.sql.Dataset;
ScoreModel model = ScoreModel.fromInputStream(pmmlInputStream);
Dataset<?> resultDataset = model.transform(csvDataset);
BTW, PMML4S-Spark's license is APL 2.0.
My answer might be totally irrelevant to your question, but since I faced an issue and was coming to this question again and again - I don't want others to face it. I reached to the solution some how by going through many stackoverflows...
The problem
I was using spark in java using the old dependency maven code:
<dependency>
<groupId>org.jpmml</groupId>
<artifactId>pmml-evaluator-metro</artifactId>
<version>1.6.3</version>
</dependency>
Which I thought is perfect and will work. But that was not recognizing TransformerBuilder as one of its libraries.
The dependency code given below should solve your problem if your problem is related to TransformerBuilder:
<dependency>
<groupId>org.jpmml</groupId>
<artifactId>jpmml-evaluator-spark</artifactId>
<version>1.3.0</version>
</dependency>
That was it. You're welcome in advance 😉

ignite-indexing and H2 version

When I use the spatial index module in ignite1.6.0 , I found it depends 1.3.175 version of the H2, but I need to use the 1.4.X h2 version.
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<version>1.3.175</version>
<scope>compile</scope>
</dependency>
This method org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing # start will call org.h2.constant.SysProperties and org.h2.util.Utils, in front of the class in the 1.3.176 version of the above have been It does not exist, the latter class is missing serializer variables.
if (SysProperties.serializeJavaObject) {
U.warn(log, "Serialization of Java objects in H2 was enabled.");
SysProperties.serializeJavaObject = false;
}
if (Utils.serializer != null)
U.warn(log, "Custom H2 serialization is already configured, will override.");
Utils.serializer = h2Serializer();
Is there any way to solve it?
Ignite depends on H2 1.3.175 and you can't use any other version. If you already have some code that depends on 1.4, you should isolate Ignite-related code in a separate module in your project. This way different versions of H2 will coextist.

Always getting exception "Wrong type at constant pool index" with Cucumber-Java8

I am trying to set-up an example project for the Java8 dialect of Cucumber. My problem is, that I don't get it running. I always get the following hierarchy of exceptions:
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.068 sec <<< FAILURE! - in soy.wimmer.CucumberIT
Feature: Cucumber with Java8 Time elapsed: 0.051 sec <<< ERROR!
cucumber.runtime.CucumberException: Failed to instantiate class soy.wimmer.CucumberStepdefs
[…]
Caused by: java.lang.reflect.InvocationTargetException: null
[…]
Caused by: cucumber.runtime.CucumberException: java.lang.IllegalArgumentException: Wrong type at constant pool index
[…]
Caused by: java.lang.IllegalArgumentException: Wrong type at constant pool index
at sun.reflect.ConstantPool.getMemberRefInfoAt0(Native Method)
at sun.reflect.ConstantPool.getMemberRefInfoAt(ConstantPool.java:47)
at cucumber.runtime.java8.ConstantPoolTypeIntrospector.getTypeString(ConstantPoolTypeIntrospector.java:37)
at cucumber.runtime.java8.ConstantPoolTypeIntrospector.getGenericTypes(ConstantPoolTypeIntrospector.java:27)
at cucumber.runtime.java.Java8StepDefinition.<init>(Java8StepDefinition.java:45)
at cucumber.runtime.java.JavaBackend.addStepDefinition(JavaBackend.java:162)
at cucumber.api.java8.En.Given(En.java:190)
at soy.wimmer.CucumberStepdefs.<init>(CucumberStepdefs.java:8)
[…]
Results :
Tests in error:
Failed to instantiate class soy.wimmer.CucumberStepdefs
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
I have no clue why I get this error nor how to fix it.
I have packaged everything in a Maven project. The layout is like that:
./src/test/java/soy/wimmer/CucumberIT.java
./src/test/java/soy/wimmer/CucumberStepdefs.java
./src/test/resources/cucumber/cucumber-java8.feature
./pom.xml
The dependencies I include in the pom.xml are:
<dependencies>
<dependency>
<groupId>info.cukes</groupId>
<artifactId>cucumber-java8</artifactId>
<version>1.2.3</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>info.cukes</groupId>
<artifactId>cucumber-junit</artifactId>
<version>1.2.3</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
</dependencies>
Additionally the pom.xml only loads the compiler and the failsafe plugin.
My definition of CucumberIT.java:
package soy.wimmer;
import cucumber.api.CucumberOptions;
import cucumber.api.junit.Cucumber;
import org.junit.runner.RunWith;
#RunWith(Cucumber.class)
#CucumberOptions(features = "classpath:cucumber")
public class CucumberIT {
}
My feature definition:
Feature: Cucumber with Java8
As a developer
I want to use Cucumber-java8
So that I have nicer step definitions
Scenario: Let's try it
Given I have some dummy code
When I try to test it
Then it should work with cucumber-java8
And this are my step definitions:
package soy.wimmer;
import cucumber.api.PendingException;
import cucumber.api.java8.En;
public class CucumberStepdefs implements En {
public CucumberStepdefs() {
Given("^I have some dummy code$", () -> {
// Write code here that turns the phrase above into concrete actions
throw new PendingException();
});
When("^I try to test it$", () -> {
// Write code here that turns the phrase above into concrete actions
throw new PendingException();
});
Then("^it should work with cucumber-java(\\d+)$", (Integer arg1) -> {
// Write code here that turns the phrase above into concrete actions
throw new PendingException();
});
}
}
Any idea what I'm doing wrong here?
The problem is caused because the Java8 dialect of Cucumber uses implementation details of Oracle's JDK8.
I was using OpenJDK8 as packaged by Debian which causes a different organisation of the constant pool. When I try the same with Oracle's JDK8 everything works as expected.
If you want to try it yourself, I published the complete example project on github: https://github.com/mawis/cucumber-java8-test
I also reported a bug at the issue tracker of cucumber-jvm here: https://github.com/cucumber/cucumber-jvm/issues/912
You might check the issue tracker to see if the problem will have been fixed in the future.
For now if you want to use cucumber-java8 it seems you have to use Oracle's implementation of the JDK.
(The fame for solving this problem belongs to Holger with his comments to the question. I just wanted to write this answer as a summary.)
Just use 1.2.5 version which has been recently released. It solved the bug referenced by accepted answer.

eye.candy.sixties not found?

When I try to run my report, I'm getting this exception:
Chart theme 'eye.candy.sixties' not found.
net.sf.jasperreports.engine.JRRuntimeException: Chart theme 'eye.candy.sixties' not found.
Sure enough, I couldn't find the theme defined anywhere in jasper-4.0.2.jar. What library do I need to get the default ireport chart themes?
I had this problem with charts using the 'aegean' theme in a web application.
I copied the jasperreports-chart-themes-4.x.x.jar eg
jasperreports-server-cp-4.0.0/ireport/ireport/modules/ext/jasperreports-chart-themes-4.0.0.jar
into my WEB-INF/lib and the charts worked.
<dependency>
<groupId>net.sf.jasperreports</groupId>
<artifactId>jasperreports-chart-themes</artifactId>
<version>${jasperReport.version}</version>
</dependency>
<dependency>
<groupId>net.sf.jasperreports</groupId>
<artifactId>jasperreports-fonts</artifactId>
<version>${jasperReport.version}</version>
</dependency>
You would have to build a project and a jar with the themes manually. There doesn't seem to be an easy library you could just include.

Resources