Good day,
I'm trying to upload nifi template and getting this kind of error:
org.apache.nifi.processors.kite.ConvertAvroSchema is not known to this NiFi instance
I suppose that my Nifi Instance misses some component - but where can I get it and where on file system I need to add missing file?
I'm little bit confused.
Thanks!
As stated here, since 1.10.0 kite-nar was taken out of the default binaries because of space limitations.
You can get it here.
Related
If I created a NiFi flow template in NiFi v1.11.1, what is the best way to update that template flow for a newer NiFi build version say v1.15.0?
The XML template file has several processors referencing v1.11.1 and when loading it in v1.15.0, it is sometimes unable to load.
Do I need to re-construct this flow template in the newer NiFi build or should I manually modify the XML template file?
You do not need to start manually changing the template XML.
Update NiFi and your flow will update with it.
If things have changed, e.g. a processor has a new, required property, it will be left in an Invalid state with a warning tip that tells you that you need to add the missing property.
I've worked the whole day on a Nifi Flow in a local docker container. Once finished I've downloaded the flow as json file and killed the container. I now want it to import into my Nifi instance on Kubernetes. Unfortunately, it seems that the way to go is using templates. So I guess the download flow as JSON file function is a one way road? Or what is the purpose of this functionality?
Is there a ways to convert this JSON to a template.xml? Otherwise I have to redo all my work.
You can upload the flow definition when creating the Process Group. Use the "Browse" icon:
You need NiFi Registry to import:
Good resource: https://community.cloudera.com/t5/Community-Articles/How-to-import-a-flow-to-NiFi-registry-in-CDP-Cloud/ta-p/308335
Personally, I do not like posts by Timothy Spann, it may be useful but lack "lots" of explanations.
Summary:
Install NiFi Registry
Connect NiFi with the registry
Import Json file manually or using NiFi Toolkit or NiPyAPI to do it programmatically.
We are working with H2O version 3.22.0.1. We created a process in java 10 that communicates with the REST API utilizing jersey version 2.27 with gson 2.3.1. The process invokes ImportFiles, followed by ParseSetup and Parse. Everything works well up until that point. Then the process invokes 3/ModelBuilders/gbm/parameters. From examining the log, it appears that the H2O server responds as expected. However, gson throws a JsonSyntaxException caused by the following:
java.lang.IllegalStateException: Expected BEGIN_OBJECT but was BEGIN_ARRAY at line 1 column 4115 path $.parameters
Upon further analysis, it appears that the H2O server is providing a GBMV3 object with an array of ModelParameterSchemaV3 objects, while the GBMV3 class, as defined in the library that our client uses, extends SharedTreeV3, which extends ModelBuilderSchema, which has a single instance of ModelParametersSchemaV3. There is an apparent discrepancy between the way the GBMV3 object provided by the H2O server is composed, and the way the class is defined in the H2O library. One has an array of ModelParameterSchemaV3 objects, while the other has a single instance of ModelParametersSchemaV3. Is that the case? If so, could you please help us understand what we may be doing wrong, and how to correct it?
See the files located at: https://1drv.ms/f/s!AsSlPHvlhJI1hIpB2M5X49J5L-h1qw
Run the H2O server. Import the CSV file in H2O Flow. SetupParse and Parse the data. Run the test procedure. Thank you for your kind assistance.
Thanks for the detailed description. To better understand your problem - would you be able to provide a simplified example of how you are calling H2O-3 using the Java bindings?
You might be hitting a bug so if you are able to give us a reproducer we could expedite a fix for this issue.
I have NiFi template exported as xml. I am using rest-api to upload template to a NiFi instance. Now, I want to update/add some properties (say, password) of the template from rest-api (or any other option available, programatically).
I read the docs and various community threads without success. Referred links:
How to set props of processor
Update nifi flow on the fly
Open for any approach,
Thanks
I think there is a bit of confusion in your wording. Correct me if I'm wrong but I believe what you want to do is:
Create a template in one location
Export it
Upload it to another NiFi instance
Add the template to the canvas (so now it's just components on your NiFi canvas)
Edit the properties of the components that were added
There are generally two different reasons you would want to edit the properties after importing a template: the properties are specific to the instance you're running on; they were sensitive properties.
With the addition of the "variable registry" in NiFi-0.7.0 you can have multiple files that at NiFi's start-up are read in to give custom variables to use. Here is a section about it in the NiFi docs. This allows you to have custom variables to reference via Expression Language (EL) specific to each environment you run on.
The "variable registry" doesn't help for the sensitive properties though, because the EL used to reference them doesn't get exported with the template (since the property is sensitive). You will need to use the rest-api to update the processor properties explicitly. The NiFi docs give the exact call to use to update a processor (under Processors -> Put). Upgrading the variable registry to work securely is on the NiFi roadmap.
If I was completely off and you simply want to modify a template after importing it into a NiFi instance. You would have to add the template to your graph, delete the template from the listing, re-create it using the components on your graph. After templates are imported/created they are immutable.
I encountered an error while adding a new service (Service Type = HDFS) using Cloudera Manager (Free Edition). The error message says as follows:
Could not create process: com.cloudera.cmf.service.config.ConfigFileSpec$GenerateException: Unable to process template:couldn't find the template hadoop/topology.py.vm
I checked /var/log/cloudera-scm-server/cloudera-scm-server.log and found a line like below.
org.apache.velocity.exception.ResourceNotFoundException: Unable to find resource '/WEB-INF/templates/hadoop/topology.py.vm'
I guess that a certain war file does not contain hadoop-metrics.properties.vm (Velocity template file?) although it should do and that this might be related to WHIRR-370.
Could you help me to solve this problem, please?
May I ask which version of Cloudera Manager is being used? Does this error occurred just after you try to add add the service of after some time when the service is added?
Based on the error, it seems some of the configuration is missing that why service addition failed. So I would like to know how did you install Hadoop on this cluster?
If you download the virtual machine and compare from your installation, you can compare the folder for completeness and missing content. It does work for me always.