I have a custom processor for AWSCredentialsProviderControllerService controller service. this has been build on top of 1.8 version .
the structure for custom processor in flow.xml.gz file is:
<name>AWSCredentialsProviderControllerService100</name> <class>org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService</class>
<bundle>
<group>com.xxx.xx1234</group>
<artifact>nifi-custom-ping-credentials-controller-service</artifact>
<version>1.0.0</version>
</bundle>
There are also default AWSCredentialsProviderControllerService controller service of 1.8version is present which is having below configuration in flow.xml
<name>AWSCredentialsProviderControllerService180</name> <class>org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService</class>
<bundle>
<group>org.apache.nifi</group>
<artifact>nifi-aws-nar</artifact>
<version>1.8.0</version>
So I am upgrading nifi cluster , means putting this flow.xml.gz file from 1.8 cluster to 1.12 cluster .
After the cluster up I am seeing the default AWSCredentialsProviderControllerService(1.8 version) controller is not auto upgraded to 1.12 bundle and getting invalid with error:
Error:
missing controller service validated against "any property" is invalid because controler service of this type org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService, but this is not a valid reporting task type.
log I am seeing :
2020-10-14 17:14:56,042 ERROR [main] o.a.nifi.controller.ExtensionBuilder Could not create Controller Service of type org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService for ID 25defb18-0175-1000-5bb4-febb1b1a21db due to: Unable to find bundle for coordinate org.apache.nifi:nifi-aws-nar:1.8.0; creating "Ghost" implementation
2020-10-14 17:14:56,042 INFO [main] o.a.nifi.groups.StandardProcessGroup StandardControllerServiceNode[service=GhostControllerService[id=25defb18-0175-1000-5bb4-febb1b1a21db, type=org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService], versionedComponentId=null, processGroup=StandardProcessGroup[identifier=8cb90667-0174-1000-8741-3bfe7f19db7f], active=false] added to StandardProcessGroup[identifier=8cb90667-0174-1000-8741-3bfe7f19db7f]
The is no issue in the custom processor (nifi-custom-ping-credentials-controller-service 1.0.0) as 1.0.0 version nar file is present in the 1.12 cluster
Also no issue with other 1.8 version processor & controller service , all are auto upgraded to 1.12 version.
Could be please let me know what should be done avoid this type of issue in upgrade?
Related
My team and I updated a spring boot application in order to work with oauth2, but with the updated we jumped from Spring Boot 1.5.2 to Spring Boot 2.5.8. The oauth2 is now implemented although the new version of the app cannot found any client encryption keys from the config server. Checking the logs of the last version I have:
INFO Nov 17 16:30:51 [main] org.springframework.cloud.bootstrap.config.PropertySourceBootstrapConfiguration: LOCAL_HOST:, APP_ID:, OPERATION_NAME:, TRANSACTION_ID:, CALL_PATH:, REMOTE_HOST:, USER: - Located property source: CompositePropertySource [name='configService', propertySources=[MapPropertySource [name='fulfillments-event-validation-job'], MapPropertySource [name='config-internal']]]
After this log there are several com.esrx.inf.config.client.CustomEnvironmentDecryptApplicationInitializer* logs where it loads the values of config server and decrypts them
But in the new version I have this logs:
INFO Nov 18 02:25:53 [main] org.springframework.cloud.bootstrap.config.PropertySourceBootstrapConfiguration: LOCAL_HOST:, APP_ID:, OPERATION_NAME:, TRANSACTION_ID:, CALL_PATH:, REMOTE_HOST:, USER: - Located property source: [BootstrapPropertySource {name='bootstrapProperties-configClient'}, BootstrapPropertySource {name='bootstrapProperties-fulfillments-event-validation-job'}, BootstrapPropertySource {name='bootstrapProperties-config-internal'}]
After this log there are not even one log for com.esrx.inf.config.client.CustomEnvironmentDecryptApplicationInitializer*
The old app can load and decrypt the config server values, the new version doesn't even load the values of config server.
old version
new version
spring-boot-starter-test 1.5.2
2.5.8
it does not have this one
spring-cloud-starter-bootstrap 3.0.5
spring-cloud-starter-config 1.3.0
3.0.6
configService
bootstrapProperties-configClient
fulfillments-event-validation-job
bootstrapProperties-fulfillments-event-validation-job
config-internal
bootstrapProperties-config-internal
So I'm thinking the error is with the values of the new version, for some reason is prepending en every value the string "bootstrapProperties-". You know any related to this issue?
I changed the configuration of my pom and application.properties several times.
At the end I stayed with spring-cloud-starter-bootstrap in order for the app to run.
Trying to persist hive table from storm-hive client, Getting following logs in HiveMetastoreServer logs.
020-02-26 23:20:27,748 ERROR org.apache.thrift.server.TThreadPoolServer: [pool-8-thread-178]: Error occurred during processing of message.
java.lang.IllegalStateException: Unexpected **DataOperationType: UNSET** agentInfo=Unknown txnid:1641
at org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:906) ~[hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2]
at org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:781) ~[hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2]
Try to use explode instead of unnest! check this:
https://stackoverflow.com/a/51846380/9185215
I have downgraded storm-hive client to 1.2.3 from 2.1.0. And also excluded hive dependency jars from storm-hive 1.2.3 and added hive client version 2.1.1 to match my cloudera environment.
When running my Beam pipeline locally it all works as expected but when trying to run it on the DataflowRunner I suddenly get the error below. Honestly I don't even know where to start evaluating this because the DataflowRunner seems to be a black box.
Jan 14, 2019 11:26:51 AM org.apache.beam.runners.dataflow.DataflowRunner fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 165 files. Enable logging at DEBUG level to see which files will be staged.
Exception in thread "main" java.lang.IncompatibleClassChangeError: Class org.apache.beam.model.pipeline.v1.RunnerApi$StandardPTransforms$Primitives does not implement the requested interface com.google.protobuf.ProtocolMessageEnum
at org.apache.beam.runners.core.construction.BeamUrns.getUrn(BeamUrns.java:27)
at org.apache.beam.runners.core.construction.PTransformTranslation.<clinit>(PTransformTranslation.java:58)
at org.apache.beam.runners.core.construction.UnconsumedReads$1.visitValue(UnconsumedReads.java:49)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:666)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:649)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:649)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:649)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:311)
at org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245)
at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458)
at org.apache.beam.runners.core.construction.UnconsumedReads.ensureAllReadsConsumed(UnconsumedReads.java:40)
at org.apache.beam.runners.dataflow.DataflowRunner.replaceTransforms(DataflowRunner.java:868)
at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:660)
at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:173)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
at (my code: pipe.run().waitUntilFinish();)
check the versions of beam etc and upgrade your dependencies where possible.
I had the same error and after seeing you get this error, I thought it must be a dependency conflict as it didn't exist before.
I'm using scio to deploy to dataflow and just referenced what they're using. https://github.com/spotify/scio/blob/v0.7.1/build.sbt
I updated guava and protobuf also.
I know you're using java, but try updating beam to 2.9.0 and maybe guava, protobuf...
Using HDP 2.4 and HAWQ 2.0
Wanted to read json data kept in HDFS path into HAWQ external table?
Followed below steps to add new json plugin into PXF and read data.
Download plugin "json-pxf-ext-3.0.1.0-1.jar" from
https://bintray.com/big-data/maven/pxf-plugins/view#
Copy the plugin into path /usr/lib/pxf.
Create External table
CREATE EXTERNAL TABLE ext_json_mytestfile ( created_at TEXT,
id_str TEXT, text TEXT, source TEXT, "user.id" INTEGER,
"user.location" TEXT,
"coordinates.type" TEXT,
"coordinates.coordinates[0]" DOUBLE PRECISION,
"coordinates.coordinates[1]" DOUBLE PRECISION)
LOCATION ('pxf://localhost:51200/tmp/hawq_test.json'
'?FRAGMENTER=org.apache.hawq.pxf.plugins.hdfs.HdfsDataFragmenter'
'&ACCESSOR=org.apache.hawq.pxf.plugins.json.JsonAccessor'
'&RESOLVER=org.apache.hawq.pxf.plugins.json.JsonResolver'
'&ANALYZER=org.apache.hawq.pxf.plugins.hdfs.HdfsAnalyzer')
FORMAT 'CUSTOM' (FORMATTER='pxfwritable_import')
LOG ERRORS INTO err_json_mytestfile SEGMENT REJECT LIMIT 10 ROWS;
When execute the above DDL table create successfully. After that trying to execute select query
select * from ext_json_mytestfile;
But getting error: -
ERROR: remote component error (500) from 'localhost:51200': type Exception report message java.lang.ClassNotFoundException: org.apache.hawq.pxf.plugins.json.JsonAccessor description The server encountered an internal error that prevented it from fulfilling this request. exception javax.servlet.ServletException: java.lang.ClassNotFoundException: org.apache.hawq.pxf.plugins.json.JsonAccessor (libchurl.c:878) (seg4 sandbox.hortonworks.com:40000 pid=117710) (dispatcher.c:1801)
DETAIL: External table ext_json_mytestfile
Any help would be much appreciated.
It seems that referenced jar file has old package name as com.pivotal.*. The JSON PXF extension is still incubating, the jar pxf-json-3.0.0.jar is built for JDK 1.7 as Single node HDB VM is using JDK 1.7 and uploaded to dropbox.
https://www.dropbox.com/s/9ljnv7jiin866mp/pxf-json-3.0.0.jar?dl=0
Echo'ing the details of the above comments so that the steps are performed correctly to ensure the PXF service recognize the jar file. The below steps assume that Hawq/HDB is managed by Ambari. If not, the manual steps as mentioned by the previous updates should work.
Copy the pxf-json-3.0.0.jar to /usr/lib/pxf/ of all your HAWQ nodes (master and segments).
In Ambari managed PXF, add the below line by going through Ambari Admin -> PXF -> Advanced pxf-public-classpath
/usr/lib/pxf/pxf-json-3.0.0.jar
In Ambari managed PXF, add this snippet to your pxf profile xml at the end by going through Ambari Admin -> PXF -> Advanced pxf-profiles
<profile>
<name>Json</name>
<description>
JSON Accessor</description>
<plugins>
<fragmenter>org.apache.hawq.pxf.plugins.hdfs.HdfsDataFragmenter</fragmenter>
<accessor>org.apache.hawq.pxf.plugins.json.JsonAccessor</accessor>
<resolver>org.apache.hawq.pxf.plugins.json.JsonResolver</resolver>
</plugins>
</profile>
Restart PXF service via Ambari
Did you add the jar file location to /etc//conf/pxf-public.classpath?
Did you try:
copying PXF JSON jar file to /usr/lib/pxf
updating /etc/pxf/conf/pxf-profiles.xml to include the Json plug-in profile if not already present
(per comment above) updating the /etc/pxf/conf/pxf-public.classpath
restarting the PXF service either via Ambari or command line (sudo service pxf-service restart)
likely didn't add json jar in classpath.
Create External Table DDL will always succeed as it was just a definition.
Only when you run queries, HAWQ will check the run time jar dependencies.
Yes, the jar json-pxf-ext-3.0.1.0-1.jar" from https://bintray.com/big-data/maven/pxf-plugins/view# has old package name as com.pivotal.*. The previous update has edited with details to download the correct jar from dropbox
I am tring to connect HBASE with jasperreports-server-cp-6.0.1. I have hadoop 2.5.2 and hbase-1.0.1 installed on my system.
I have installed HBasePlugin-0.5.1.nbm plugin in iReport 5.6.0.
I have followed all the steps given in: http://community.jaspersoft.com/wiki/hadoop-hbase
When I write the following Query:
{ "tableName" : "blogposts", "deserializerClass" : "com.jaspersoft.hbase.deserialize.impl.ShellDeserializer" }
In iReport, I am getting the following error:
Message:
net.sf.jasperreports.engine.JRException: No deserializer defined
Level:
SEVERE
Stack Trace:
No deserializer defined
com.jaspersoft.hadoop.hbase.query.HBaseQueryWrapper.<init>(HBaseQueryWrapper.java:152)
com.jaspersoft.hadoop.hbase.HBaseFieldsProvider.getFields(HBaseFieldsProvider.java:50)
com.jaspersoft.ireport.hbase.designer.HBaseFieldsProvider.getFields(HBaseFieldsProvider.java:57)
com.jaspersoft.ireport.hbase.connection.HBaseConnection.readFields(HBaseConnection.java:185)
com.jaspersoft.ireport.designer.wizards.ConnectionSelectionWizardPanel.validate(ConnectionSelectionWizardPanel.java:146)
org.openide.WizardDescriptor$7.run(WizardDescriptor.java:1357)
org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:572)
org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:997)
Could you please help me with this error (I also tried with iReport 4.0.2, but I received the same error)?
Both iReport and the HBase connector are outdated.
Try using the Apache Phoenix JDBC driver which is compatible with the latest release (6.2) of the Jaspersoft products:
http://community.jaspersoft.com/wiki/how-use-apache-phoenix-jdbc-driver-run-reports-hbase
Thanks!