Storm pulling in stale dependencies - apache-storm

Question

We are trying to upgrade from Storm 0.10.0 to 1.0.2 and our project uses kafka-spout
Between versions, backtype.storm package-names were changed to org.apache.storm package-names by the storm-community.
As part of our own upgrade, we had to change storm-kafka's version also to 1.0.2
But when I run the topology on storm, I get the error about missing classes from older version:
apache-storm-1.0.2/bin/storm \
jar \
$jarFile \
org.apache.storm.flux.Flux \
$yamlFile \
--remote
+- Apache Storm -+
+- data FLow User eXperience -+
Version: 1.0.2
Parsing file: topology-config.yaml
333 [main] INFO o.a.s.f.p.FluxParser - loading YAML from input stream...
335 [main] INFO o.a.s.f.p.FluxParser - Not performing property substitution.
335 [main] INFO o.a.s.f.p.FluxParser - Not performing environment variable substitution.
Exception in thread "main" java.lang.NoClassDefFoundError: backtype/storm/spout/MultiScheme
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.storm.flux.FluxBuilder.buildObject(FluxBuilder.java:291)
at org.apache.storm.flux.FluxBuilder.buildComponents(FluxBuilder.java:350)
at org.apache.storm.flux.FluxBuilder.buildTopology(FluxBuilder.java:75)
at org.apache.storm.flux.Flux.runCli(Flux.java:153)
at org.apache.storm.flux.Flux.main(Flux.java:98)
Caused by: java.lang.ClassNotFoundException: backtype.storm.spout.MultiScheme
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more
I am not sure from where backtype.storm is creeping into my code-base.
I have looked all over the code-base (and mvn dependency:tree) for 0.10.0 and backtype.storm but those things from older-version are no place to be found.
I even excluded all storm-kafka versions from my child-dependencies and then added an explicit dependency on storm-kafka but it did not change a thing.
Maven does not seem to be at fault because I do not see any 0.10* versions in my ~/.m2/repository:
ls ~/.m2/repository/org/apache/storm/*
~/.m2/repository/org/apache/storm/flux:
1.0.2
~/.m2/repository/org/apache/storm/flux-core:
1.0.2
~/.m2/repository/org/apache/storm/storm:
1.0.2
~/.m2/repository/org/apache/storm/storm-kafka:
1.0.2
Clearly storm is doing something wrong here.

Answer 1

Check your topology-config.yaml, Class is loaded dynamically loaded after picking the class name from the yaml file.

Answer 2

So the issue was because of using shaded jar files as dependencies in our code.
One of the shaded jar file used an older version of the storm-dependency and somehow this was not being shown in the mvn dependency:tree
On excluding the storm dependencies from the shaded jar dependency, the above error went away.
TL;DR: I actual exploded the eventual jar file (using jar -xvf final.jar) and saw duplicate classes in backtype/storm and org/apache/storm packages and the classes in backtype/storm were getting picked up perhaps because backtype comes before org lexicographically? (Not sure on this one because I would have expected the package name to take precedence in such cases).