Unable to install Hadoop and Spark through Gremlin Shell - hadoop

I am trying to install hadoop-gremlin and spark-gremlin through the Gremlin shell.
I downloaded Gremlin console 3.2.0-incubating.
When I run bin/gremlin.sh, the gremlin console launches without error. However, I am unable to install spark-gremlin or hadoop-gremlin.
gremlin> :install org.apache.tinkerpop spark-gremlin 3.2.0-incubating
==>Error grabbing Grapes -- [unresolved dependency: com.github.jeremyh#jBCrypt;jbcrypt-0.4: not found]
gremlin> :install org.apache.tinkerpop hadoop-gremlin 3.2.0-incubating
==>Error grabbing Grapes -- [unresolved dependency: com.github.jeremyh#jBCrypt;jbcrypt-0.4: not found]
The following (seemingly relevant) jar exists: $HOME/lib/apache-gremlin-console-3.2.0-incubating/lib/jBCrypt-jbcrypt-0.4.jar. Additionally, I set these environment variables before running the console:
GREMLINHOME=$HOME/lib/apache-gremlin-console-3.2.0-incubating
export HADOOP_GREMLIN_LIBS=$GREMLINHOME/ext/spark-gremlin/plugin:$GREMLINHOME/ext/hadoop-gremlin/plugin:$GREMLINHOME/ext/gremlin-groovy/plugin:$GREMLINHOME/lib
export HADOOP_HOME=/usr/hdp/current/hadoop-client/client
export CLASSPATH=$HADOOP_HOME/*:$HADOOP_HOME/lib/*:$HADOOP_HOME/etc/hadoop
While this question appears very similar, it seems the solution is very specific to neo4j.
Please let me know if there is any other information I can provide.

You'll need to set a Grapes configuration file under ~/.groovy/grapeConfig.xml
<ivysettings>
<settings defaultResolver="downloadGrapes"/>
<resolvers>
<chain name="downloadGrapes">
<filesystem name="cachedGrapes">
<ivy pattern="${user.home}/.groovy/grapes/[organisation]/[module]/ivy-[revision].xml"/>
<artifact pattern="${user.home}/.groovy/grapes/[organisation]/[module]/[type]s/[artifact]-[revision].[ext]"/>
</filesystem>
<ibiblio name="codehaus" root="http://repository.codehaus.org/" m2compatible="true"/>
<ibiblio name="central" root="http://central.maven.org/maven2/" m2compatible="true"/>
<ibiblio name="jitpack" root="https://jitpack.io" m2compatible="true"/>
<ibiblio name="java.net2" root="http://download.java.net/maven/2/" m2compatible="true"/>
</chain>
</resolvers>
</ivysettings>
The jBCrypt dependency is coming from the jitpack repository.
You can read more about this in the Apache TinkerPop documentation.

Related

right working of ivy with archiva repository

I'm trying to understanding and using ivy with archiva, but i have this strange thing that i can't explain,
Build.xml
...some xml code
<target name="resolveArchiva" description="--> go take the jar">
<ivy:settings file="./archivaIvySetting.xml" >
<credentials host="my host to Archiva" realm="my repository's name"
username="user" passwd="passwd"/>
</ ivy:settings>
<ivy:retrieve />
</target>
some other code....
Ivy.xml
<ivy-module version="2.0">
<info organisation="archiva" module="my-repo"/>
<dependencies>
<dependency org="annogen" name="annogen" rev="0.1.0"/>
<dependency org="commons-lang" name="commons-lang" rev="2.1" />
</dependencies>
</ivy-module>
arvchivaIvySetting.xml
<ivysettings>
<settings defaultResolver="archiva" />
<resolvers>
<ibiblio name="archiva" m2compatible="true" root="repositories/internal/"/>
</resolvers>
</ivysettings>
the problem it's that in my repo there is an artifact for annogen but it's not present the common-lang artifact, despite that, the build says that it find the artifacts for both the dependencies, and that's true because i find both .jar in my lib project. How is it possible? it's clear that my setting doesn't point to the archiva rep but to the maven ones but how can i change it?

"com.springsource" prefix on artifact/module causing conflicts

I have a dependency on my project like this:
<dependency org="org.springframework"
name="org.springframework.web.servlet"
rev="3.1.2.RELEASE" />
And it wants to pull things like this:
org.apache.commons#com.springsource.org.apache.commons.logging;1.1.1
which causes issues with other libraries that use a newer version of commons-logging. So I work around it by excluding it like this:
<exclude org="org.apache.commons"
module="com.springsource.org.apache.commons.logging" />
but having to do that kind of defeats the purpose of the conflict management Ivy has built-in. Is there a better way to handle this, or do I have to track down all the potential conflicts manually?
Disclaimer: I'm pretty new to Ivy and dependency management, so I have no idea if this concept extends into the Maven world as well. If not, I apologize for the false tagging.
I'm assuming you are using the EBR repositories in your ivysettings.xml file.
You get those com.springsource dependencies because all the libraries in EBR are valid OSGi bundles and SpringSource transformed normal jars in bundles by prefixing the bundle names with com.springsource.
Also, I'm assuming your project doesn't make use of an OSGi container and you don't actually need OSGi bundles in your project. To get back to resolving normal jars and not OSGi bundles you need not to use the EBR repositories.
In your ivysettings.xml comment the resolvers that point to EBR repositories, for example:
<!-- <resolvers>
<chain name="chained">
<url name="com.springsource.repository.bundles.release">
<ivy pattern="http://repository.springsource.com/ivy/bundles/release/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
<artifact pattern="http://repository.springsource.com/ivy/bundles/release/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
</url>
<url name="com.springsource.repository.bundles.external">
<ivy pattern="http://repository.springsource.com/ivy/bundles/external/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
<artifact pattern="http://repository.springsource.com/ivy/bundles/external/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
</url>
</chain>
</resolvers> -->
And replace those with the ibiblio resolver:
<settings defaultResolver="central"/>
<resolvers>
<ibiblio name="central" m2compatible="true"/>
</resolvers>
Also, you need to change any dependency that, initially, was taken from EBR repositories to the format the usual Maven central repository has. For example, your dependency on org.springframework.web.servlet needs to be changed in ivy.xml to:
<dependency org="org.springframework"
name="spring-web"
rev="3.1.2.RELEASE" />
With the changes above, Ivy will resolve the dependencies from Maven central, as well.

Get latest google drive api by Ivy

I have add resolver to ivysettings.xml
<ibiblio name="google-api-services" root="http://mavenrepo.google-api-java-client.googlecode.com/hg" m2compatible="true" />
But get a error
module not found: com.google.apis#google-api-services-drive;v2-rev13-1.8.0-beta
It resolves fine for me..... Perhaps there is a problem with your ivysettings file?
The following example configures the google api services repo and Maven Central.
<ivysettings>
<settings defaultResolver="repos" />
<resolvers>
<chain name="repos">
<ibiblio name="central" m2compatible="true"/>
<ibiblio name="google-api-services" m2compatible="true" root="http://mavenrepo.google-api-java-client.googlecode.com/hg"/>
</chain>
</resolvers>
</ivysettings>

How can I use a <filesystem> resolver that doesn't copy artifacts to the cache in Ivy?

I have a <filesystem> resolver in my ivysettings.xml, along with the central M2 repository, and it all works OK. However, I was wondering whether there is a way to bypass the cache entirely for the dependencies found with the filesystem resolver. I don't need to have them so many times around on my filesystem (once in the directory searched by the resolver, once in the cache, and once in each project's lib folder…).
Found it. The name of a cache with the useOrigin="true" attribute can be specified on the resolver:
<ivysettings>
<settings defaultResolver="main"/>
<caches>
<cache name="main" basedir="${ivy.settings.dir}/ivycache" />
<cache name="nocache" useOrigin="true" />
</caches>
<resolvers>
<chain name="main">
<filesystem name="filesystem" cache="nocache">
<artifact pattern="${ivy.settings.dir}/ivyrep/[artifact].[ext]" />
</filesystem>
<ibiblio name="ibiblio" m2compatible="true" usepoms="false" />
</chain>
</resolvers>
</ivysettings>

Ivy via Nexus proxy

does anyone knows how do I specify in Ivy something like mirror/mirrorOf in Maven? I'm working with a local Maven proxy (Nexus) and need the tool to specify which of the parent repositories should Nexus proxy be accessing.
In Maven I do simply:
<mirrors>
<mirror>
<id>central-mirror</id>
<mirrorOf>central</mirrorOf>
<url>http://localhost:8081/content/repositories/central</url>
</mirror>
</mirrors>
but I can't find this kind of option in Ivy.
You need to create a public resolver that does what you want (more details # Ivy docs)
Basically save the following snippet under $USERHOME/.ivy2/ivysettings-public.xml. This should do the trick.
<ivysettings>
<resolvers>
<ibiblio name="public" m2compatible="true" root="http://localhost:8081/content/groups/public"/>
</resolvers>
</ivysettings>
The unmodified standard installation has 'nexus' in the URL!
If you need to deploy artifacts, I think the solution is to do something similar to the shared resolver (see link to docs above), but I haven't tried.
I changed your local URL to resolve to the standard 'content/groups/public' which is better since in the maven settings fragment above you're passing all calls through the mirror, not just the ones to central. Just add any additional repositories to that group in the Nexus UI as they come up and you should be fine.
If your project loads it's own ivysettings which doesn't honor the defaults, then these settings will not get loaded and you're again back at zero :(
I don't think such an option exists directly. You could try implementing a chain, and put your Nexus repository ahead of central in that chain. If I understand how chains work correctly (that's a big if), Ivy will check your repository before central, so as long as your repository has the relevant contents central won't be needed.
See the tutorial for details.
This is how I made it work (The answer from #Heron did not work for me):
Create a file with this content:
<ivysettings>
<settings defaultResolver="default"/>
<property name="m2-pattern" value="${user.home}/.m2/repository/[organisation]/[module]/[revision]/[module]-[revision](-[classifier]).[ext]" override="false" />
<resolvers>
<chain name="default">
<ibiblio name="public" m2compatible="true" root="http://nexus-server:8081/nexus/content/groups/public"/>
</chain>
</resolvers>
</ivysettings>
Refere to it from the ant build:
<ivy:settings file="/Users/wdb/.ivy2/ivysettings-public.xml" />
Ivy is now able to resolve dependencies from my nexus repository.
I have done the same but with Archiva, what is very similar. You only have to declare in a new chain the following:
<chain name="private">
<url name="archiva" m2compatible="true">
<ivy pattern="http://..../archiva/repository/internal/[organisation]/[module]/[revision]/ivy.xml" />
<artifact pattern="http://..../archiva/repository/internal/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
<artifact pattern="http://..../archiva/repository/internal/[organisation]/[module]/[revision]/[artifact].[ext]" />
</url>
</chain>
Archiva manages Maven 2 repositories (artifacts with Maven meta data)
there isn't usually Ivy meta data (ivy.xml).
And the Maven 2 layout is [organisation]/[module]/[revision]/[artifact]-[revision].[ext].
We have only to provide the following information
<url name="archiva" m2compatible="true">
<artifact pattern="http://..../archiva/repository/internal/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
</url>
</chain>
or
<settings defaultResolver="archiva"/>
<resolvers>
<ibiblio name="archiva" m2compatible="true" root="http://.../archiva/repository/internal/[organization]/[module]/[revision]/[artifact]-[revision].[ext]"/>
</resolvers>

Resources