Transitive Dependency: Using Elasticsearch Rest High Client problem in AEM - maven

Question

I am trying to use Java High Level Rest Client in Adobe Experience Manager to finish project of comparison between Lucene, Solr and Elasticsearch search engines.
I am having some problems with elasticsearh implementation.
Here is the code:
Dependency in the parent pom.xml (the same is defined in core pom.xml)
<!-- Elasticseach dependencies -->
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-high-level-client</artifactId>
<version>7.4.0</version>
</dependency>
The only line of code that I am using that is from dependencies above
try (RestHighLevelClient client = new
RestHighLevelClient(RestClient.builder(new HttpHost(server, port,
protocol),
new HttpHost(server, secondPort, protocol)));)
{
}
catch (ElasticsearchException e)
{
LOG.error("Exception: " + e);
}
protocol = "http", server = "localhost", port = 9200, secondPort =
9201
Error
Dependencies from IntelliJ
I know that there is usually problem with dependencies versions, but all are 7.4.0 in this case. Also elasticsearch 7.4.0v is running locally on 3 nodes.
This project is done on We.Retail project so it is easy to replicate. Also all the code with this error is available here:
https://github.com/tadijam64/search-engines-comparison-on-we-retail/tree/elasticsearch-integration
AEM 6.4v.
Any info or idea is appreciated.
UPDATE
I tried with adding the following to embed these dependencies externally since they are not OSGi dependencies:
<build>
<plugins>
<plugin>
<groupId>org.apache.felix</groupId>
<artifactId>maven-scr-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.apache.felix</groupId>
<artifactId>maven-bundle-plugin</artifactId>
<extensions>true</extensions>
<configuration>
<instructions>
<Embed-Dependency>org.apache.servicemix.bundles.solr-solrj, log4j, noggit, zookeeper,
elasticsearch-rest-high-level-client
</Embed-Dependency>
<Embed-Transitive>true</Embed-Transitive>
<Embed-Directory>OSGI-INF/lib</Embed-Directory>
<Export-Package>we.retail.core.model*</Export-Package>
<Import-Package>
*;resolution:=optional
</Import-Package>
<Private-Package>we.retail.core*</Private-Package>
<Sling-Model-Packages>
we.retail.core.model
</Sling-Model-Packages>
</instructions>
</configuration>
</plugin>
</plugins>
</build>
The error remains. I also tried adding it to the "export-package", but nothing helps.
And by Elasticsearch documentation, all I need to use Elasticsearch is
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-high-level-client</artifactId>
<version>7.4.0</version>
</dependency>
but then NoClassDefFoundErrors occurs. It seems like a problem with transitive dependencies maybe. Not sure, but any idea is appreciated.
Some other suggestions can be found here: https://forums.adobe.com/thread/2653586
I have also tried adding it's transitive dependencies like org.elasticsearch and org.elasticsearch.client, but it does not work. The same error, just other class.
AEM version 6.4, Java version: jdk1.8.0_191.jdk

Answer 1

So my guess was right, transitive dependencies were not included altho <Embed-Transitive>true</Embed-Transitive> exists.
The following is necessary when running elasticsearch as a search engine on AEM the problem:
I have added all transitive dependencies in pom.xml (versions are defined in parent/pom.xml):
<!-- Elasticsearch -->
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-high-level-client</artifactId>
</dependency>
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-client</artifactId>
</dependency>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
</dependency>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch-x-content</artifactId>
</dependency>
<dependency>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>rank-eval-client</artifactId>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-imaging</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>lang-mustache-client</artifactId>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpasyncclient</artifactId>
</dependency>
It is important to add all third-party dependencies as <Embed-Dependency> inside maven-bundle-plugin like this:
<plugin>
<groupId>org.apache.felix</groupId>
<artifactId>maven-bundle-plugin</artifactId>
<extensions>true</extensions>
<configuration>
<instructions>
<Embed-Dependency>org.apache.servicemix.bundles.solr-solrj, noggit,
elasticsearch-rest-high-level-client,
elasticsearch,
elasticsearch-rest-client,
elasticsearch-x-content,
elasticsearch-core,
rank-eval-client,
lang-mustache-client,
httpasyncclient;
</Embed-Dependency>
<Embed-Transitive>true</Embed-Transitive>
<Embed-Directory>OSGI-INF/lib</Embed-Directory>
<Export-Package>we.retail.core.model*</Export-Package>
<Import-Package>
*;resolution:=optional
</Import-Package>
<Private-Package>
we.retail.core*
</Private-Package>
<Sling-Model-Packages>
we.retail.core.model
</Sling-Model-Packages>
<_fixupmessages>"Classes found in the wrong directory";is:=warning</_fixupmessages>
</instructions>
</configuration>
</plugin>
Important to notice:
All third-party dependencies (the ones outside of OSGi) must be included in the "Embed-Dependency"
"Embed-Transitive" must be set to true to include transitive dependencies
"Import-Package" must include "*;resolution:=optional" to exclude all dependencies that could not be resolved so that the program can run
normally
For some reason, there was an error in compile time when "elasticsearch" dependency was added which is not important for this
task, so I've decided to ignore it this way:
<_fixupmessages>"Classes found in the wrong directory";is:=warning</_fixupmessages>
Though challenging, I finally resolved it. There are many similar or the same problems on Google, so I hope this will help someone. Thanks to everyone that tried to help.