NiFi custom processor introduces duplicates of standard artifacts - apache-nifi

I've written a Custom Processor to handle Multipart Posts in advance of this being introduced in 1.12.
The processor and nar that I have work fine, but the nar bundle has introduced duplicates of a lot of the standard processors with a version number matching that of my Custom Processor.
This started to happen after I introduced the SSLContextService controller. Adding the Controller service required the addition of
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-standard-services-api-nar</artifactId>
<version>1.11.4</version>
<type>nar</type>
</dependency>
As per the guide on the apache nifi wiki, but the completed nar seems to contain "2.1.1-SNAPSHOT" version of standard nifi processors such as AttributesToJson or PutRecord and dozens of others.
Excluding this dependency causes the nar build to fail as the SSLContexService class is missing.
Using a provided scope builds a nar of approximately the same size, which then causes NiFi to fail to start (so far I haven't found an error message telling my why, it just dies at startup).
Does anyone know how to stop these duplicates being created? The whole nar pom is;
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.jontia</groupId>
<artifactId>PostMultipartFormData</artifactId>
<version>2.1.1-SNAPSHOT</version>
</parent>
<artifactId>nifi-multipart-nar</artifactId>
<version>2.1.1-SNAPSHOT</version>
<packaging>nar</packaging>
<properties>
<maven.javadoc.skip>true</maven.javadoc.skip>
<source.skip>true</source.skip>
</properties>
<dependencies>
<dependency>
<groupId>com.jontia</groupId>
<artifactId>nifi-multipart-processors</artifactId>
<version>2.1.1-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-standard-services-api-nar</artifactId>
<version>1.11.4</version>
<type>nar</type>
</dependency>
</dependencies>
</project>
Processor Pom;
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.jontia</groupId>
<artifactId>PostMultipartFormData</artifactId>
<version>2.1.1-SNAPSHOT</version>
</parent>
<artifactId>nifi-multipart-processors</artifactId>
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-api</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-utils</artifactId>
<version>1.11.4</version>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-ssl-context-service-api</artifactId>
<version>1.11.4</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-proxy-configuration-api</artifactId>
<version>1.11.4</version>
<scope>provided</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.nifi/nifi-processor-utils -->
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-processor-utils</artifactId>
<version>1.11.4</version>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-standard-processors</artifactId>
<version>1.11.4</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.9</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId>
<version>4.4.11</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpmime</artifactId>
<version>4.5.12</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.13.3</version>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-mock</artifactId>
<version>1.11.4</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.12</version>
</dependency>
</dependencies>
</project>

The processors pom.xml has a dependency on nifi-standard-processors:
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-standard-processors</artifactId>
<version>1.11.4</version>
</dependency>
So when your NAR is built it pulls in the JAR of the standard processors so now they are all bundled again in your NAR and also in the standard NAR.
You should not depend on standard processors. If there is some code in there that you need then it should be refactored into some type of reusable common module, or if you are trying to extend a processor you should really just copy it into your NAR and make whatever modifications are needed.

Related

Vert.x annotation processor problem with module-info.java

I'm trying to create a new project in Java that is composed by several modules (following the Java Modules concept introduced on Java 9). In one of those modules, I'm trying to use vert.x codegen to generate a service that will be exposed to the other modules.
Now that problem is that when I try to compile the service module I got the following error:
module-info.java:[5,22] module not found: io.vertx.codegen.
For your reference, I will put my module configuration (as stated on module-info.java) and my POM file.
Module-info.java:
module mo.com.test.daomodule {
requires io.vertx.core;
requires io.vertx.client.sql.mssql;
requires io.vertx.client.sql;
requires io.vertx.codegen;
requires io.vertx.serviceproxy;
requires org.apache.logging.log4j.core;
requires org.apache.logging.log4j;
exports mo.com.test.dao.db.service;
exports mo.com.test.dao.db;
}
My pom:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>mo.com.test</groupId>
<artifactId>MainProject</artifactId>
<version>1.0-SNAPSHOT</version>
</parent>
<properties>
<vertx.version>4.2.1</vertx.version>
<jdbc.version>9.4.0.jre11</jdbc.version>
<log4j2api.version>2.14.1</log4j2api.version>
<log4j2core.version>2.14.1</log4j2core.version>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-stack-depchain</artifactId>
<version>${vertx.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-core</artifactId>
</dependency>
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-json-schema</artifactId>
</dependency>
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-jdbc-client</artifactId>
</dependency>
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-mssql-client</artifactId>
</dependency>
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-codegen</artifactId>
<version>${vertx.version}</version>
<classifier>processor</classifier>
</dependency>
<dependency>
<groupId>io.vertx</groupId>
<artifactId>vertx-service-proxy</artifactId>
<version>${vertx.version}</version>
</dependency>
<dependency>
<groupId>com.microsoft.sqlserver</groupId>
<artifactId>mssql-jdbc</artifactId>
<version>${jdbc.version}</version>
<type>jar</type>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>${log4j2api.version}</version>
<type>jar</type>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>${log4j2core.version}</version>
<type>jar</type>
</dependency>
</dependencies>
<groupId>mo.com.test</groupId>
<artifactId>daomodule</artifactId>
<version>1.0</version>
<packaging>jar</packaging>
<name>daomodule</name>
</project>
By checking above, I have added the necessary imports in the POM file and add include the vert.x modules as required in the module-info.java.
No my question is, what can I do to solve this compilation problem? Is there anything missing on my POM or my module-info.java?

Added maven dependencies is giving

After adding Maven dependencies I'm getting a compilation error.
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com</groupId>
<artifactId>Sample1</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>Sample1</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>6.8</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>3.141.59</version>
</dependency>
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>6.14.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi</artifactId>
<version>3.17</version>
</dependency>
<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi-ooxml</artifactId>
<version>3.15</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
<dependency>
<groupId>com.relevantcodes</groupId>
<artifactId>extentreports</artifactId>
<version>2.41.2</version>
</dependency>
</dependencies>
</project>
The console shows a warning:
Compiler Error Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project Sample1: Compilation failure
[ERROR] No compiler is provided in this environment. Perhaps you are running on a JRE rather than a JDK?
Looks like your IDE is pointed to JRE. Add JDK path in your IDE
For Eclipse try these steps:
Windows --> perfernces --> java--> installed JREs.
Add your JDK and click on the newly added JDK
for reference

Spring Boot unable to find jsp files on ElasticBeanstalk AWS

It works good when the application is running from Eclipse. But the problem is starting after i packaging the app to JAR or WAR and running it.
Getting 404 on every pages.
Here what i have done so far:
Here my POM file:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>springtry</groupId>
<artifactId>spring</artifactId>
<packaging>jar</packaging>
<version>0.0.1-SNAPSHOT</version>
<name>spring Maven Webapp</name>
<url>http://maven.apache.org</url>
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.4.3.RELEASE</version>
</parent>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-solr</artifactId>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
</dependency>
<!-- JSTL for JSP -->
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>jstl</artifactId>
<scope>provided</scope>
</dependency>
<!-- Need this to compile JSP -->
<dependency>
<groupId>org.apache.tomcat.embed</groupId>
<artifactId>tomcat-embed-jasper</artifactId>
<scope>provided</scope>
</dependency>
<!-- Need this to compile JSP, tomcat-embed-jasper version is not working,
no idea why -->
<dependency>
<groupId>org.eclipse.jdt.core.compiler</groupId>
<artifactId>ecj</artifactId>
<version>4.6.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>okhttp</artifactId>
<version>3.5.0</version>
</dependency>
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>6.10</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
So to resolve the issue i have changed my elastic beanstalk from Java to Tomcat. And now it works with WAR.
Using the Java i was not able to run the webapp
For jar packaging, the jsp files should be in src/main/resources/META-INF/resources/.
For war packaging src/main/webapp/ is good.
Also, I guess with war packaging, you are probably not implementing SpringBootServletInitializer with it's method configure(...) in your config main class.
A complete example is here

To remove extra dependencies inherited from spring-boot-starter-parent

<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.2.6.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
When spring-boot-starter-parent is used as parent, then it includes many unnecessary dependencies. Like in my case it fetches the dependencies for
(spring-mobile-device , spring-social-facebook , spring-social-linkedin , org.apache.activemq ) which i don't want.
If I open my effective POM(with get's auto created) , it's contains hell lot of dependencies which i don't want.
Is there any way to remove the unnecessary dependencies which are not used in the project? Is there a tool to identify such dependencies?
Any help is appreciated :)
Thanks
POM.xml :
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>demo</groupId>
<version>0.0.1-SNAPSHOT</version>
<name>demo</name>
<description>demo</description>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.2.5.RELEASE</version>
<relativePath />
</parent>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-thymeleaf</artifactId>
</dependency>
<!-- <dependency> <groupId>com.google.code.gson</groupId> <artifactId>gson</artifactId>
<version>2.2.4</version>$NO-MVN-MAN-VER$ </dependency> -->
<!-- <dependency> <groupId>com.sun.jersey</groupId> <artifactId>jersey-client</artifactId>
<version>1.17.1</version> </dependency> -->
<!-- <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId>
<version>4.4.1</version>$NO-MVN-MAN-VER$ </dependency> <dependency> <groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId> <version>4.4.1</version> </dependency> -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.4</version>
</dependency>
<!-- <dependency> <groupId>joda-time</groupId> <artifactId>joda-time</artifactId>
<version>2.8</version> </dependency> -->
<!-- <dependency> <groupId>com.rabbitmq</groupId> <artifactId>amqp-client</artifactId>
<version>3.5.4</version> </dependency> -->
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.4</version>
</dependency>
<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-core</artifactId>
<version>4.5.1</version>
</dependency>
<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-analyzers-common</artifactId>
<version>4.5.1</version>
</dependency>
<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-queryparser</artifactId>
<version>4.5.1</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
</dependency>
<dependency>
<groupId>dom4j</groupId>
<artifactId>dom4j</artifactId>
<version>1.6.1</version>
</dependency>
</dependencies>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<start-class>demo.Application</start-class>
<java.version>1.7</java.version>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
<artifactId>DEMO</artifactId>
spring-boot-starter-parent doesn't declare any dependencies. However, via spring-boot-dependencies, it does provide dependency management for a wide-range of potential dependencies.
From Maven's documentation:
The dependency management section is a mechanism for centralizing dependency information. When you have a set of projects that inherits a common parent it's possible to put all information about the dependency in the common POM and have simpler references to the artifacts in the child POMs
The references to artifacts can be simpler as, thanks to the dependency management, there's no need to declare a version. For example:
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<dependency>
<dependencies>
If you don't specify a version, Maven will use the version that's configured in the dependency management.
Only when you add an entry in your pom's <dependencies> section will your project actually have that dependency and Maven will download it as part of the build.

Jongo Maven dependency causing run time issue

I have a Maven project where I am using the Jongo dependency with the following POM.
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.xxx</groupId>
<artifactId>mytracker</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>mytracker</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.7</maven.compiler.source>
<maven.compiler.target>1.7</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
</dependency>
<dependency>
<groupId>com.yammer.dropwizard</groupId>
<artifactId>dropwizard-core</artifactId>
<version>0.6.2</version>
</dependency>
<dependency>
<groupId>net.vz.mongodb.jackson</groupId>
<artifactId>mongo-jackson-mapper</artifactId>
<version>1.4.2</version>
</dependency>
<dependency>
<groupId>org.jongo</groupId>
<artifactId>jongo</artifactId>
<version>1.2</version>
</dependency>
<dependency>
<groupId>org.joda</groupId>
<artifactId>joda-money</artifactId>
<version>0.10.0</version>
</dependency>
<dependency>
<groupId>com.sun.jersey.jersey-test-framework</groupId>
<artifactId>jersey-test-framework-core</artifactId>
<version>1.19</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-client</artifactId>
<version>1.19</version>
</dependency>
<dependency>
<groupId>com.sun.jersey.test.framework</groupId>
<artifactId>jersey-test-framework</artifactId>
<version>1.0.3.1</version>
</dependency>
<dependency>
<groupId>javax</groupId>
<artifactId>javaee-api</artifactId>
<version>7.0</version>
</dependency>
</dependencies>
</project>
Now when I run my Main class , I get the following error in eclipse , although the project is built successfully using Maven , this error shows up when I run the Main class. I am using JDK 1.7 ,any ideas ?
Exception in thread "main" java.lang.VerifyError: (class: org/jongo/marshall/jackson/configuration/AbstractMappingBuilder, method: <init> signature: ()V) Incompatible argument to function
at org.jongo.Jongo.<init>(Jongo.java:32)
at com.mycompany.mytracker.CashbackService.run(CashbackService.java:44)
at com.mycompany.mytracker.CashbackService.run(CashbackService.java:18)
at com.yammer.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:38)
at com.yammer.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:58)
at com.yammer.dropwizard.cli.Cli.run(Cli.java:53)
at com.yammer.dropwizard.Service.run(Service.java:61)
at com.mycompany.mytracker.CashbackService.main(CashbackService.java:22)
I removed the following dependency and it solved my problem.
<dependency>
<groupId>net.vz.mongodb.jackson</groupId>
<artifactId>mongo-jackson-mapper</artifactId>
<version>1.4.2</version>
</dependency>

Resources