I am experiencing an issue when trying to using the "com.google.protobuf.Any" message type in my protobuf definitions along with the official "protobuf" deserializer provided by Flink in version 1.16.
It seems that the following message definition
syntax = "proto2";
option java_package = "com.company.model.proto";
import "google/protobuf/any.proto";
message MessageWrapper {
optional google.protobuf.Any message = 1;
}
with the Flink table definition
tableEnv.executeSql("CREATE TABLE trade_records (" +
" kk_trade_id STRING," +
" message ROW<`value` BYTES>, " +
" PRIMARY KEY (kk_trade_id) NOT ENFORCED" +
") WITH (" +
" 'connector' = 'upsert-kafka'," +
" 'value.protobuf.message-class-name' = 'com.company.model.proto.Common$MessageWrapper'," +
" 'key.format' = 'raw'," +
" 'key.fields-prefix' = 'kk_'," +
" 'value.fields-include' = 'EXCEPT_KEY'," +
" 'value.format' = 'protobuf'" +
...
");"
);
triggers this error:
Caused by: org.apache.flink.api.common.InvalidProgramException: Program cannot be compiled. This is a bug. Please file an issue.
at org.apache.flink.formats.protobuf.util.PbCodegenUtils.compileClass(PbCodegenUtils.java:265)
at org.apache.flink.formats.protobuf.deserialize.ProtoToRowConverter.<init>(ProtoToRowConverter.java:118)
... 14 more
Caused by: org.codehaus.commons.compiler.CompileException: Line 20, Column 36: "com.company.model.proto.Common" declares no member type "Any"
at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:13014)
at org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:6873)
I have narrowed this down to an issue in deriving the correct type when Flink generates the source code to feed to the Janino compiler:
public class GeneratedProtoToRow_07c0170e88dd4e5ba9f3e2c33217562a{
...
com.company.model.proto.Common.Any message22 = message12.getMessage();
I believe this type should read com.google.protobuf.Any instead. Is there a known compiler option or some other workaround for this problem?
Related
I'm trying to use Flink to work with Oracle. Just do a simple task copy data from table to a new one.
EnvironmentSettings settings = EnvironmentSettings.inStreamingMode();
TableEnvironment tEnv = TableEnvironment.create(settings);
tEnv.executeSql("CREATE TABLE ExistedTable(\n" +
" quoteid BIGINT,\n" +
" requestid BIGINT,\n" +
" createddt DATE,\n" +
" PRIMARY KEY (quoteid) NOT ENFORCED\n" +
") WITH (\n" +
" 'connector' = 'jdbc',\n" +
" 'url' = 'jdbc:oracle:thin:#xxx.xxx.xxx.xxx:1521:DBNAME',\n" +
" 'table-name' = 'TableName',\n" +
" 'driver' = 'oracle.jdbc.driver.OracleDriver',\n" +
" 'username' = 'UserName',\n" +
" 'password' = 'Password'\n" +
")");
tEnv.executeSql("CREATE TABLE NewTable (\n" +
" quoteid BIGINT,\n" +
" requestid BIGINT,\n" +
" createddt DATE,\n" +
" PRIMARY KEY (quoteid) NOT ENFORCED\n" +
") WITH (\n" +
" 'connector' = 'jdbc',\n" +
" 'url' = 'jdbc:oracle:thin:#xxx.xxx.xxx.xxx:1521:DBNAME',\n" +
" 'table-name' = 'NewTableName',\n" +
" 'driver' = 'oracle.jdbc.driver.OracleDriver',\n" +
" 'username' = 'UserName',\n" +
" 'password' = 'Password'\n" +
")");
Table data= tEnv.from("ExistedTable");
data.executeInsert("NewTable");
When running I've got error
The program finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Unable to create a source for reading table 'default_catalog.default_database.xxx'.
Table options are:
'connector'='jdbc'
'driver'='oracle.jdbc.OracleDriver'
'password'='******'
'table-name'='xxx'
'url'='jdbc:oracle:thin:#xxx:1521:xxx'
'username'='xxx'
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
Caused by: org.apache.flink.table.api.ValidationException: Unable to create a source for reading table 'default_catalog.default_database.xxx'.
Is there any error in my sqlconnection. I couldn't found any example for working with oracle.
Thanks,
Which version of Flink are you using? Support for Oracle JDBC is available since Flink 1.15, which hasn't been released yet.
I am trying using script with query to update particular documents , but m not sure which class I need to use of jest client which work's with update by query
Here is my code
String h = " { "
+ " \"script\": { \n"
+ " \"inline\": \"ctx._source.order=params.prasad\", \n "
+ "\"params\": { \"prasad\":["+column+"] \n"
+ " } , \n"
+ "\"lang\": \"painless\" \n"
+ "},\n"
+ "\"query\": { \n"
+ "\"term\": { \n"
+ "\"orgid\": "+orgId+" \n"
+ "} \n"
+ "}";
LocalJestClientInstance.getInstance().getClient().execute(new Update.Builder(h).index(index).type(type).build());
This is not updating my documents.
Uri generating is this uri=loadfields/loadfields_type/_update,method=POST
Instead of _update how to get the _update_by_query
Jest didn't have it before but it looks like it does have it now.
My workaround was to use a second client as well, a standard http client and parse the JSON output just for this functionality that was missing.
I am running a test suite in Soap UI where I am trying to call one ruby script from groovy script. The step is getting executed successfully but still the script is not able to move on to the next step as it gives this error after running.
Have searched in google about this error, but found no proper resolution. Moreover the error itself is not very explanatory.
Will appreciate any kind of help.
Below is the groovy script which is calling "ap-v4-batch_DEV_QA.rb" ruby script.
This ruby script opens a browser and performs the task successfully and closes the browser. We expect the step to be marked as Passed so that it can move on to the next step, but it gives the error mentioned at the bottom.
Groovy Script:
String script = "webdriver/v4/ap-v4-batch_DEV_QA.rb";
String argv0 = com.eviware.soapui.SoapUI.globalProperties.getPropertyValue("GLOB_DefaultIP");
String argv1 = "com.wupay.batch.process.tasks.PaymentFileParsingTask_RunOnce";
String argv2 = "";
String argv3 = "";
String argv4 = "";
/* Nothing needs to be modified below */
String commandLine = "ruby " + com.eviware.soapui.SoapUI.globalProperties.getPropertyValue("GLOB_ScriptLocation") + "/" + script + " " + argv0 + " " + argv1 + " " + argv2 + " " + argv3 + " " + argv4;
log.info("Running command line: " + commandLine);
java.lang.Runtime runtime = java.lang.Runtime.getRuntime();
java.lang.Process p = runtime.exec(commandLine);
def propertyStep = testRunner.testCase.getTestStepByName("Properties");
java.io.BufferedReader stdInput =
new java.io.BufferedReader(new java.io.InputStreamReader(p.getInputStream()));
java.io.BufferedReader stdError =
new java.io.BufferedReader(new java.io.InputStreamReader(p.getErrorStream()));
String s = null;
String e = null;
StringBuffer eb = new StringBuffer();
while ((e = stdError.readLine()) != null) {
eb.append(e);
log.error("Ruby: " + e);
}
while ((s = stdInput.readLine()) != null) {
log.info("Ruby: " + s);
if(s.startsWith("#prop")) {
String[] propSplit = s.split(":", 3);
testRunner.testCase.setPropertyValue(propSplit[1], propSplit[2]);
}
}
p.waitFor();
log.info("Ruby: exit value " + p.exitValue());
if(eb.length() > 0) {
throw new Exception(eb.toString());
}
Error:
java.lang.Exception: C:/Ruby23/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:133:in require':require "watir-webdriver"is deprecated. Please, userequire "watir". java.lang.Exception: C:/Ruby23/lib/ruby/site_ruby/2.3.0/rubygems/core_ext/kernel_require.rb:133:inrequire': require "watir-webdriver" is deprecated. Please, use require "watir". error at line: 57
I have finally resolved the issue.
The issue was that ruby script was not accepting require "watir-webdriver".
I installed watir and replaced require "watir-webdriver" with require "watir".
now I am not getting the above mentioned error.
Thanks anyways!
Regards,
Faraz
I get the error "expression was too complex to be solved in reasonable time" for the following code.
I saw other threads regarding the error with much complexer expressions then mine and it was stated, that the compiler is buggy.
Is this also true for the following simple code ?
Problem:
let value1:Int = 1;
let value2:Int = 3;
var sql="The Number 2 is in "
+ " between " + String(iFrom) + " and " + String(iTo) + ".";
but this works
let value1:Int = 1;
let value2:Int = 3;
var sql="The Number 2 is in "
+ " between " + String(iFrom) + " and " + String(iTo);
The difference is only the "." at the end of concatenation.
If its a bug:
The process SourceKitService is running at max and slowes everything down. Can compilation at runtime be disabled ?
One thing you could try that I've had some success with is to try specifically typing your sql variable.
var sql: String ="The Number 2 is in "
+ " between " + String(iFrom) + " and " + String(iTo) + ".";
I need a tool that's commandline and which supports retrieving version information from the resources of a PE executable and also patch the binary with a new version. What I need to do is to automate incrementing the version of a PE binary.
Any ideas?
Resource Tools library [1, 2] with COM interface. You can use it from VBScript/JScript and automate patching of version information as I already do.
Import/browse DLL type library to see COM interface syntax and below is code snippet for automation of VersionInfo patching.
WshShell = WScript.CreateObject("WScript.Shell");
WScript.Echo("Current Directory: " + WshShell.CurrentDirectory);
Image = new ActiveXObject("AlaxInfo.ResourceTools.Image");
Image.Initialize(WshShell.CurrentDirectory + "\\Sample.dll");
WScript.Echo("Product Version: " + Image.VersionInfo.ProductVersionString);
WScript.Echo("File Version: " + Image.VersionInfo.FileVersionString);
VersionString = Image.VersionInfo.FileVersionString;
VersionString = VersionString.replace(" ", "");
VersionString = VersionString.replace(",", ".");
if(/([0-9]+)\.([0-9]+)\.([0-9]+)\.([0-9]+)/i.exec(VersionString) == null)
throw "Incorrect version string";
V1 = new Number(RegExp.$1);
V2 = new Number(RegExp.$2);
V3 = new Number(RegExp.$3);
V4 = new Number(RegExp.$4);
WScript.Echo("File Version Components: " + V1 + ", " + V2 + ", " + V3 + ", " + V4);
V4++;
VersionString = V1 + "." + V2 + "." + V3 + "." + V4;
WScript.Echo("New File Version: " + VersionString);
Image.VersionInfo.SetFileVersion(V1 * 65536 + V2, V3 * 65536 + V4);
Image.VersionInfo.SetString(0, "FileVersion", VersionString);
Image.VersionInfo.Update();
Image.EndUpdate(false);
The library is basically a wrapper over BeginUpdateResource function and friends providing you with automation interface into patching not only numbers but other version info components, strings, bitmaps.