I'm following this tutorial: https://learn.microsoft.com/en-us/azure/hdinsight/storm/apache-storm-develop-java-topology
What I've done so far is
maven setting
vi *.java files (in src/main/java/com/microsoft/example directory)
RandomSentenceSpout.java
SplitSentence.java
WordCount.java
WordCountTopology.java
mvn compile
jar cf storm.jar *.class (in target/classes/com/microsoft/example directory)
RandomSentenceSpout.class SplitSentence.class WordCount.class WordCountTopology.class
The above 4 files were used to make storm.jar file
Then, I tried
storm jar ./storm.jar com.microsoft.example.WordCountTopology WordCountTopology
and
storm jar ./storm.jar WordCountTopology
, but both of these failed, saying:
Error: Could not find or load main class com.microsoft.example.WordCountTopology
or
Error: Could not find or load main class WordCountTopology
According to a document, it says
Syntax: storm jar topology-jar-path class ...
Runs the main method of class with the specified arguments. The storm
jars and configs in ~/.storm are put on the classpath. The process is
configured so that StormSubmitter will upload the jar at
topology-jar-path when the topology is submitted.
I cannot find where to fix.
How can I resolve this?
I think your jar file does not contain class WordCountTopology. You can check it with jar tf storm.jar | grep WordCountTopology.
Looks like your jar does not contain a Manifest file which keeps information about the main class.
Try including the Manifest file or you can run the below java command to include the Manifest file
Hope this works!
jar cvfe storm.jar mainClassNameWithoutDotClassExtn *.class
Related
I've done a successfull mvn clean install on the project I built and the structure appears correct, all classes included and manifest is under META-INF including class-paths and main-class. Not sure what's not matching up here, but the class contents are valid when I checked contents using javap. Main method is present in the redacted_automatedSupport class and is also public.
Error:
host MINGW64 ~/Desktop/Projects/redacted_Automated_Support/target (master)
$ java -jar redacted_automatedSupport-1.0.jar
Error: Could not find or load main class support.redacted_automatedSupport
host MINGW64 ~/Desktop/Projects/redacted_Automated_Support/target (master)
$ java -cp redacted_automatedSupport-1.0.jar support.redacted_automatedSupport
Error: Could not find or load main class support.redacted_automatedSupport
host MINGW64 ~/Desktop/Projects/redacted_Automated_Support/target (master)
$
Manifest contents:
Manifest-Version: 1.0
Archiver-Version: Plexus Archiver
Built-By: bennet.vella
Class-Path: aws-java-sdk-s3-1.11.696.jar aws-java-sdk-kms-1.11.696.jar
aws-java-sdk-core-1.11.696.jar commons-logging-1.1.3.jar httpclient-
4.5.9.jar httpcore-4.4.11.jar commons-codec-1.11.jar ion-java-1.0.2.j
ar jackson-databind-2.6.7.3.jar jackson-annotations-2.6.0.jar jackson
-core-2.6.7.jar jackson-dataformat-cbor-2.6.7.jar joda-time-2.8.1.jar
jmespath-java-1.11.696.jar groovy-3.0.1.jar groovy-json-3.0.1.jar gr
oovy-dateutil-3.0.1.jar
Created-By: Apache Maven 3.6.3
Build-Jdk: 1.8.0_231
Main-Class: support.redacted_automatedSupport
Jar Archive Contents (renamed to zip to browse):
Support Jar Contents (removed some unecessary data, all names match):
It is not a complete answer, since it doesn't target the Maven issue I was having, but I did solve the IntelliJ problem I had with the wrong manifest file - and that's because I was creating the manifest in src/main/java when it should have been src/main/resources. This should hopefully alleviate some users' problems.
I have not however resolved how to properly build and include all relevant dependent jars using Maven - intelliJ does this successfully.
The question is simple, but I spent the last 2 days trying to deploy my app. And so far it doesn't.
I have a single CSS file for my style, and when I execute the jar, CSS is not found (404) or the jar won't package.
As stated here: Spring Boot Executable jar structure
"Do not use the src/main/webapp folder if your application will be packaged as a jar"
and
"You should place your static resources in src/main/resources instead."
so put the CSS here:
src/main/resources/styles.css
In Vaadin documentation (which is very pour on how to package...) I import the CSS like this:
#StyleSheet("styles.css")
Source : https://vaadin.com/docs/v11/flow/importing-dependencies/tutorial-include-css.html
Then I package my project:
mvn clean package -Pproduction
I get this error:
[ERROR] Failed to execute goal com.vaadin:vaadin-maven-plugin:11.0.0:package-for-production (default) on project importparcoursup: Execution default of goal com.vaadin:vaadin-maven-plugin:11.0.0:package-for-production failed: An import that ends with 'styles.css' cannot be resolved: the corresponding file 'C:\Workspace\lasteclipeandjava10\parcoursup\target\frontend\styles.css' was not found.
[ERROR] Double check the corresponding import and verify the following:
[ERROR] * the import string is correct
[ERROR] * the file imported is either present in 'frontend://' directory of the project or in one of the project WebJar dependencies or in one of the regular jar dependencies
[ERROR] * if the file is present in one of the regular jar dependencies, it should be located in META-INF/resources/frontend directory in the jar
Can someone provide a simple example of a 'springboot + Vaadin10' app packaged as a jar with static resources inside ?
I tried so many configurations (put the CSS in META-INF, include webapp resources in the maven build process...) but after 2 days, I still can't deploy my app on the server!
finally the solution
css has to be here:
src/main/resources/META-INF/resources/frontend/styles.css
then declared as:
#StyleSheet("frontend://styles.css")
This could be helpful too even though I still miss an example:
Vaadin 10 makes some changes to the way it loads static resources,
such as application templates, custom styles and any additional
JavaScript files. The gist of it is that such files should be put in
src/main/webapp/frontend/ when building a .war file and
src/main/resources/META-INF/resources/frontend/ when building a .jar
file.
Link to Vaadin Dokumentation: Vaadin 10 and static resources
I have been trying spark using spark-shell. All my data is in sql.
I used to include external jars using the --jars flag like /bin/spark-shell --jars /path/to/mysql-connector-java-5.1.23-bin.jar --master spark://sparkmaster.com:7077
I have included it in class path by changing the bin/compute-classpath.sh file
I was running succesfully with this config.
Now when I am running a standalone job through jobserver. I am getting the following error message
result: {
"message" : "com.mysql.jdbc.Driver"
"errorClass" : "java.lang.classNotFoundException"
"stack" :[.......]
}
I have included the jar file in my local.conf file as below.
context-settings{
.....
dependent-jar-uris = ["file:///absolute/path/to/the/jarfile"]
......
}
All of your dependencies should be included in your spark-jobserver application JAR (e.g. create an "uber-jar"), or be included on the classpath of the Spark executors. I recommend configuring the classpath, as it's faster and requires less disk-space since the third-party library dependencies don't need to be copied to each worker whenever your application runs.
Here are the steps to configure the worker (executor) classpath on Spark 1.3.1:
Copy the third-party JAR(s) to each of your Spark workers and the Spark master
Place the JAR(s) in the same directory on each host (e.g. /home/ec2-user/lib
Add the following line to the Spark /root/spark/conf/spark-defaults.conf file on the Spark master:
spark.executor.extraClassPath /root/ephemeral-hdfs/conf:/home/ec2-user/lib/name-of-your-jar-file.jar
Here's an example of my own modifications to use the Stanford NLP library:
spark.executor.extraClassPath /root/ephemeral-hdfs/conf:/home/ec2-user/lib/stanford-corenlp-3.4.1.jar:/home/ec2-user/lib/stanford-corenlp-3.4.1-models.jar
You might not be having /path/to/mysql-connector-java-5.1.23-bin.jar in your workers.
You can either copy required dependency to all spark workers or
Bundle the submitting jar with required dependencies.
I use maven for building the jar. The scope of dependencies must be run-time.
curl --data-binary #/PATH/jobs_jar_2.10-1.0.jar 192.168.0.115:8090/jars/job_to_be_registered
For posting dependency jar
curl -d "" 'http://192.168.0.115:8090/contexts/new_context?dependent-jar-uris=file:///path/dependent.jar'
This works for jobserver 1.6.1
I'm trying to build a simple WordCount jar project which utilizes Hadoop-lzo library but cannot seem to get the following command to work, even though the class I'm referencing is within hadoop classpath:
$ javac -cp `hadoop classpath` *.java
LzoWordCount.java:76: cannot find symbol
symbol : class LzoTextInputFormat
location: class LzoWordCount
job.setInputFormatClass(LzoTextInputFormat.class);
^
1 error
Any ideas?
I assume you have correctly installed your LZO libraries (you should have libgplcompression.so in your lib/natives/Linux**-**/ and the jar file in your lib/ folder)
Since you have them the correct class should be LzoDeprecatedTextInputFormat.class or LzoTextInputFormat.class depending on wich API you use (According to your post you are using it right Job with LzoTextInputFormat).
So your problem could be in your java.library.path, wich should include path to your jar file. You can set it up in your .bash_profile or in you bin/hadoop file.
hope that helps.
I have set up Hadoop on my laptop and ran the example program given in the installation guide successfully. But, I am not able to run a program.
rohit#renaissance1:~/hadoop/ch2$ hadoop MaxTemperature input/ncdc/sample.txt output
Exception in thread "main" java.lang.NoClassDefFoundError: MaxTemperature
Caused by: java.lang.ClassNotFoundException: MaxTemperature
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
Could not find the main class: MaxTemperature. Program will exit.
The book said that we should set a Hadoop Classpath by writing
rohit#renaissance1:~/hadoop/ch2$ export HADOOP_CLASSPATH=build/classes
The main class is defined in MaxTemperature.java file that I am executing. How do we set the Hadoop Classpath? Do we have to do it for all program execution or only once? Where should I put the input folder. My code is at /home/rohit/hadoop/ch2 and my Hadoop installation is at /home/hadoop.
You should package your application into a JAR file, that's much easier and less error-prone than fiddling with classpath folders.
In your case, you must also compile the .java file. You said it's MaxTemparature.java, but there must also be a MaxTemperature.class before you can run it.
First compile the Java files as told by walid:
javac -classpath path-to-hadoop-0.19.2-core.jar .java-files -d folder-to-contain-classes
Create jar file of application classes using the following command:
jar cf filename.jar *.class
In either of the, whether you are exporting the classes into jar file or using specific folder to store class files , you should define HADOOP_CLASSPATH pointing to that particular class file or folder containing class file. So that at the time of running Hadoop command it should know where to look specified for main class.
set HADOOP_CLASSPATH
export HADOOP_CLASSPATH=path-to-filename.jar
or
export HADOOP_CLASSPATH=path-to-folder-containing-classes
Run using Hadoop command:
hadoop main-class args
I found this problem as well when going thru the Hadoop Book (O'Reilly). I fixed it by setting the HADOOP_CLASSPATH variable in the hadoop-env.sh file in your configuration directory.
here is the ansewer in 3 steps:
1:
javac -verbose -classpath C:\\hadoop\\hadoop-0.19.2-core.jar MaxTemperature*.java -d build/classes
2:
put *.class in build/classes
3:
export HADOOP_CLASSPATH=${HADOOP_HOME}/path/to/build/classes
(you have to create the build/classes directory)
Best Regards
walid
You do not necessarily need a jar file, but did you put MaxTemperature in a package?
If so, say your MaxTemperature.class file is in yourdir/bin/yourpackage/, all you need to do is:
export HADOOP_CLASSPATH=yourdir/bin
hadoop yourpackage.MaxTemperature
after you make your class a jar file:
hadoop jar MaxTemperature.jar MaxTemperature
basicly :
hadoop jar jarfile main [args]