I have written one Custom NiFi processor which uses some Hadoop classes, processes flow files and serializes flow files to and from Avro.
As per my knowledge
to use Hadoop classes without packaging them as part of NAR I will need nifi-hadoop-nar bundle AND
to serialize flow file contents to/from Avro, I will need RecordSetWriter and reader which are part of record-serialization-services-api.
Please correct me if my assumption is wrong.
I have written code, tested it and everything works as expected until I try to deploy NAR to NiFi.
When I deploy NAR and restart NiFi, it throws java.lang.ClassNotFoundException: org.apache.nifi.serialization.RecordSetWriterFactory exception.
My NAR's pom.xml looks like below:
<dependencies>
<dependency>
<groupId>com.my_company.is.data.tools</groupId>
<artifactId>custom-data-movement-processors</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-hadoop-nar</artifactId>
<version>1.5.0</version>
<type>nar</type>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-record-serialization-service-api</artifactId>
<version>1.5.0</version>
<scope>provided</scope>
</dependency>
</dependencies>
and processor's pom.xml file looks like below:
<dependencies>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-record-serialization-services</artifactId>
<version>1.5.0</version>
<scope>provided</scope>
<dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-record-serialization-services-api</artifactId>
<version>1.5.0</version>
<scope>provided</scope>
<dependency>
</dependencies>
Since I have marked nifi-record-serialization-services-api as provided, it doesn't get bundled in resulting NAR file.
Now I can do a quick fix and remove scope altogether and create NAR BUT then NiFi complains about next class not found error.
I would like to know:
In NiFi project's code I found multiple nested bundles which have relation something like this nifi-nar-bundles -> nifi--bundle -> . How to refer two bundles in my processor? As per my knowledge custom processor can have only one parent NAR and NAR = bundle?
What is the preferred way to specify multiple NiFi bundles in pom.xml
Ideally I do not want to bundle Hadoop related dependencies or NiFi dependencies in my NAR file. or is it mandetory in NiFi to bundle these dependencies?
<packaging>nar</packaging>
<dependencies>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-hadoop-nar</artifactId>
<version>1.5.0</version>
<type>nar</type>
</dependency>
should be enough to get in resulted nar META-INF\MANIFEST.MF
attributes Nar-Dependency-*
Nar-Dependency-Id: nifi-hadoop-nar
Nar-Dependency-Group: org.apache.nifi
Nar-Dependency-Version: 1.5.0
Related
I use Spring Boot with Jetty and JSP/JSTL:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-tomcat</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jetty</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>jstl</artifactId>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>apache-jsp</artifactId>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>apache-jstl</artifactId>
</dependency>
During startup, I get a number of warnings like:
WARN 18439 --- [ main] o.a.tomcat.util.scan.StandardJarScanner : Failed to scan [file:/home/user/.m2/repository/.../guava.jar] from classloader hierarchy
There are spurious jars in these warnings. Why tomcat scans anything if it is absent from the configuration files and it is Jetty which runs? Is the scan needed for e.g. JSP/JSTL? Can it be disabled? I can't use this because no Tomcat libraries are available in the application.
EDIT: Tomcat (?) sources are apparently called by Jasper, used in turn by Jetty. Thus, I put this into pom.xml in order to disable Jar scanning by Jetty:
<plugin>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-maven-plugin</artifactId>
<configuration>
<webApp>
<!-- no need to scan anything as we're using servlet 2.5 and moreover, we're not using Servl
<!-- for more relevant information regarding scanning of classes refer to https://java.net/j
<webInfIncludeJarPattern>^$</webInfIncludeJarPattern>
<containerIncludeJarPattern>^$</containerIncludeJarPattern>
<!--<webInfIncludeJarPattern>.*/spring-[^/]*\.jar$|.*/.*jsp-api-[^/]\.jar$|./.*jsp-[^/]\.jar
</webApp>
</configuration>
</plugin>
but it did not have any effect.
Your org.apache.tomcat.util.scan.StandardJarScanner is coming from the apache-jsp-<ver>.jar on the classpath.
Apache Jasper JSP needs to scan your classpath for TLDs when it does the JSPC steps.
Since you are using spring-boot, there's no real separation of WebAppClassLoader and SystemClassLoader, so it's forced to scan the entire classloader hierarchy to find those TLDs.
This scan is independent of Jetty, so that means the webInfIncludeJarPattern and containerIncludeJarPattern configurations will have no impact.
The best advice here is to precompile your JSP's with jspc during build-time and just have the runtime Jasper JSP never have the need to kick off JSPC and it's associated TLD scan.
If you still want runtime JSPC, consider setting the following property at Server startup, before your Context has started to initialize.
System.setProperty(
org.apache.tomcat.util.scan.Constants.SKIP_JARS_PROPERTY,
"*.jar");
That will tell the StandardJarScanFilter to skip all files ending in .jar.
Note that there is also a Constants.SCAN_JARS_PROPERTY which you get to specify specific jars to scan. (I don't know if this property supports globs, or relative references)
TL;DR: the GCP secrets are not resolved in bootstrap file but the sql starter requires an instance connection name and database name on bootstrap
I'm trying to incorporate GCP Secretmanager in a Spring Boot application that is running on Google App Engine and using the GCP SQL.
However the ${sm:// prefix doesn't seem to be resolved at bootstrap time.
For reference, this is my part of my pom. (I'm using the com.google.cloud dependencies) And I enable the spring profile "gcp"
<parent>
<artifactId>spring-boot-starter-parent</artifactId>
<groupId>org.springframework.boot</groupId>
<relativePath/>
<version>2.4.2</version> <!-- lookup parent from repository -->
</parent>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>spring-cloud-gcp-dependencies</artifactId>
<version>2.0.1</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<!-- ... -->
<!-- cloud -->
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>spring-cloud-gcp-starter</artifactId>
</dependency>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>spring-cloud-gcp-starter-sql-postgresql</artifactId>
</dependency>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>spring-cloud-gcp-starter-secretmanager</artifactId>
</dependency>
And in my bootstrap (for example)
spring:
cloud:
gcp:
sql:
database-name: ${sm://some-fancy-db-secret}
instance-connection-name: ${sm://some-cool-connection-name}
When deploying I get an exception stating that a database-name needs to be defined.
If I fill in the plain properties it works just fine.
Even the ${sm://db-username} works inside my application-gcp.yml file.
When I move the property from the bootstrap file it also fails.
It seems it needs the connection when bootstrapping. (I'm a bit in the dark about that)
There's no fancy multi-project going on, and yes the secret exists.
I have a feeling I'm missing something stupid here or there's a version mismatch somewhere.
(The Codelab doesn't seem to be mentioning anything special at all.)
I also checked this question. However the proposed answer doesn't seem to be valid anymore. The com.google.cloud.spring.autoconfigure.secretmanager.GcpSecretManagerProperties don't even contain a prefix property, besides it works just fine in my normal properties file.
Has been fixed in the latest release.
I am migrating existing Spring project into Spring boot.unable to run spring boot application its showing following error.
The error log says there is a conflict on tomcat-embed-core.
In eclipse Dependency hierarchy of porm.xml is given below
i exclude the maven architect ,and try to run the application its showing following error
porm.xml
<modelVersion>4.0.0</modelVersion>
<artifactId>MyService</artifactId>
<packaging>jar</packaging>
<properties>
<java.version>1.8</java.version>
<!-- 2.1.3.RELEASE -->
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<springframework.boot.version>2.1.7.RELEASE</springframework.boot.version>
</properties>
<name>MyService</name>
<url>http://maven.apache.org</url>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>com.sybase.jdbc3.jdbc</groupId>
<artifactId>jconn3</artifactId>
<version>${jconn3.version}</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.tomcat</groupId>
<artifactId>tomcat-juli</artifactId>
<version>${tomcat.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.tomcat</groupId>
<artifactId>tomcat-jdbc</artifactId>
<version>${tomcat.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</pluginManagement>
</build>
</project>
what was wrong in this porm.xml
Where is
${tomcat.version}
defined?
That version probably does not match the tomcat version that auto magically is included with spring boot items.
And thus the conflict.
Go here:
https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-starter-web/2.1.7.RELEASE
And start following the COMPILE dependencies, and you'll find the versions that are auto included with 2.1.7.RELEASE. and you have to alter the other includes that are overwriting the springboot auto include tomcat versions.
Again, follow the COMPILED dependency trail.
So below is what you should find by crawling the COMPILED dependency trail (from immediately above in my answer)
https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-starter-tomcat/2.1.7.RELEASE
And you'll find you need to set
tomcat.version to
9.0.22
By defining tomcat.version as 8.x, you are breaking it.
Another way to put it
You have to go ~way~ back to springboot 1.5.2.RELEASE or 1.5.3.RELEASE
https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-starter-tomcat/1.5.2.RELEASE
https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-starter-tomcat/1.5.3.RELEASE
(Again, in the two above links, looked at the COMPILE dependencies)
To find a version of tomcat (that is auto included with springboot) that gets close to tomcat 8.5.x (where 8.5.x is the one you are attempting to use)
That's pretty old.
The principal you are missing is that springboot auto includes dependencies. And anything else you import has to play nice with everything springboot auto includes.
And your current value for tomcat.version is NOT playing nice with everything springboot 2.1.7.RELEASE is auto including.
And now that you've been through all of that. You'll find you'll make your life easier if you engage the springboot world more completely.
Alot of times, springboot will have a (sub)package that will bring in the thing you really desire.
spring-boot-starter-jdbc
https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-starter-jdbc/2.1.7.RELEASE
You would probably be better off bringing that package in, vs hand-picking ones. Aka, get rid of your "tomcat-jdbc" include and see if the spring-boot-starter-jdbc can give you what you want.
The curse/blessing of spring-boot is that it is its own universe. But if you engage, you probably want to play by its rules more often than not.
PS
It is pom.xml, not porm.xml
Try adding spring-boot-starter-tomcat as a dependency:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-tomcat</artifactId>
<scope>provided</scope>
</dependency>
Remove tomcat-juli and tomcat-jdbc dependencies. If you need JDBC support, add the corresponding starter:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>
If you use JSP views, you will probably need the following dependencies as well:
<dependency>
<groupId>org.apache.tomcat.embed</groupId>
<artifactId>tomcat-embed-jasper</artifactId>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>jstl</artifactId>
</dependency>
Also, pay attention to your dependencies versions. Spring Boot's parent POM defines version management for many common artifacts so you don't need to set the <version></version> for these libraries. See https://docs.spring.io/spring-boot/docs/2.1.7.RELEASE/reference/htmlsingle/#appendix-dependency-versions
I would like to find/create a pom.xml containing all libraries included in tomEE, using "provided" scope. Goal of this is to make it as "pom parent" of a webproject, and have no risk to use other library version, or other implementation.
Does a such pom.xml exist? or is there a simple way to create it?
thanks in advance
Clément
Because JavaEE is a specification, there are several available.
I use this one for several reasons. This is in my organization's parent pom so all the projects automatically pull it in:
<dependencies>
<dependency>
<groupId>org.apache.openejb</groupId>
<artifactId>javaee-api</artifactId>
<version>6.0-2</version>
<scope>provided</scope>
</dependency>
</dependencies>
Thanks to exabrial, the maven dependency i was looking for is this one :
<dependency>
<groupId>org.apache.openejb</groupId>
<artifactId>tomee-webapp</artifactId>
<version>1.6.0</version>
</dependency>
I am working on a big set of projects strongly related to each other and looking for a way to reduce to minimimun the maintenance overhead using a good maven configuration.
One of the scenarios which I am currently working on is a set of several small projects inside the same "company project", most of them depending on the others, with a big list of external dependencies in each one.
In order to make it cleaner I created a parent pom, set all of them as modules and created a big list of dependencies inside the parent's dependencyManagement section allowing me to remove most of the version tags inside the project poms.
The next stage for me is to reduce the amount of resulting jars by making use of some plugin, such as one-jar or assembly, in oder to get a signle jar from one of those projects containing all the others inside. However, this led me to having an enourmous jar with tons of external libraries inside, most of them already provided by the running environment.
I know that adding the <scope>provided</scope> tag to the declaration of external dependencies would do the job, but I would like to avoid having dozens of these tags in my poms.
So the questions are:
Is there any way to define a default scope, so I can use compile scope only in the libs which I want included?
Then, instead of having this:
<dependencyManagement>
<dependency>
<groupId>any.external</groupId>
<artifactId>lib</artifactId>
<version>1.0</version>
<scope>provided</scope> <!--Skip Inclusion-->
</dependency>
<dependency>
<groupId>any.external</groupId>
<artifactId>cool-lib</artifactId>
<version>1.0</version>
<scope>provided</scope> <!--Skip Inclusion-->
</dependency>
<dependency>
<groupId>another.external</groupId>
<artifactId>lib</artifactId>
<version>1.0</version>
<scope>provided</scope> <!--Skip Inclusion-->
</dependency>
<dependency>
<groupId>my.company</groupId>
<artifactId>some-dependency</artifactId>
<version>1.0</version>
</dependency>
</dependencyManagement>
I could have this (note that the difference will come when we have only 10 internal projects to include, but more than 100 external libs):
<dependencyManagement>
<dependency>
<groupId>any.external</groupId>
<artifactId>lib</artifactId>
<version>1.0</version>
</dependency>
<dependency>
<groupId>any.external</groupId>
<artifactId>cool-lib</artifactId>
<version>1.0</version>
</dependency>
<dependency>
<groupId>another.external</groupId>
<artifactId>lib</artifactId>
<version>1.0</version>
</dependency>
<dependency>
<groupId>my.company</groupId>
<artifactId>some-dependency</artifactId>
<version>1.0</version>
<scope>compile</scope><!-- Include It! -->
</dependency>
</dependencyManagement>
Or better: Maybe I'm skipping something and there is already a way to tell maven: "Include only the jars generated within the current project"?
Remember that all modules are "brothers", included as modules of the same parent, and compiled at the same time.
Many thanks in advance!
Carles