How to add a new module to Spark source and make it work in spark-shell? - maven

I'm using IDEA15 and I want to add a new module to the spark source.
I clicked File->new->module and chose a maven module. Then I set the "Add as module to..." option and the "Parent" to "Spark Project Parent POM". After typing in the module name I clicked "Finish".
Then I added some code to my new module and build it using the follwing command
"build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package"
The project was built successfully but in the spark-shell I can't import my newly added classes.
I wonder what's wrong with what I've done and how can I add a new module and then import it in the spark-shell?
Thanks a lot!
PS: I'm sure there's no problem with my code. I added my code in the mllib module and it worked.
Maybe some dependency is missing but I don't know how to fix it.

The Maven build created a jar file from your build (should be in the target/ directory inside your project folder).
When you start the Spark shell, you can define the jar files to include in your shell. You can include your jar there like this:
spark-shell --jars /path/to/your/project.jar
Also you could give it a try, to install your project inside your local Maven repository, it is possible that the Spark shell can pick it up (so you don't have to specify --jars each time you run the it).
For this, run your Maven build command with clean install instead of clean package at the end.

Related

Run mvn command on module from existing source

I imported several maven modules on IntelliJ IDEA by using the option File/New/Module From Existing Source. This is working fine but I'm not able to run mvn command lines on one specific module by its module name.
I was able to do it by specifying the path to the pom.xml file by using -f option:
mvn -f "path/to/pom.xml" clean
But I would like to avoid having specifying the path every time I want to run a mvn command. Is their any way to run the command by specifiying the name of the module ?
Thank you.
If you use "Run Anything" then it's possible to select module at the top right corner
You can perform maven install, maven clean etc for a complete module or sub modules of a project using top right option in IntelliJ.
Maven-->select module/submodule folder-->Plugins-->select the option:- deploy, compile, install, clean etc.

Export a Eclipse RCP product from command Line

I tried different way to export it but i couldn't succeed in it.
I need a way to export my product file which is located in my workspace.
If you need to implement do that using command lines, you must compile the product using maven with tycho. You will need to implement every module as a artifact in Maven.
After the configuration you only need to run:
mvn clean install
Please read this link:
https://www.eclipse.org/tycho/
Here is the tutorial to configure your project:
https://www.vogella.com/tutorials/EclipseTycho/article.html

Error: Unable to access jarfile build/libs/gs-spring-boot-0.1.0.jar?

I follow the instructions in https://spring.io/guides/gs/spring-boot/#scratch, but when it says to run:
./gradlew build && java -jar build/libs/gs-spring-boot-0.1.0.jar
the build fails with the above error.
There is message before the failure that says:
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
See https://docs.gradle.org/4.8.1/userguide/command_line_interface.html#sec:command_line_warnings
but everyone online says that's just a warning.
The build doesn't appear to create or download build/libs/gs-spring-boot-0.1.0.jar.
Currently completely blocked on first attempt to use Gradle.
I just had this problem.
The tutorial is in error in what you need to run. It should be
$ gradlew build && java -jar build/libs/gs-rest-service-0.1.0.jar
I think that they updated the code, but forgot to update the tutorial.
I had the same issue when build a simple project with Maven on Intellij IDEA. (Ubuntu 18.04.2).
Just typed terminal (in project directory):
$ sudo mvn package
$ java -jar ./target/(your-project-name)-(<version> at pom.xml).jar
For example my project name is hello-world-spring and version name in pom.xml is <version>0.0.1-SNAPSHOT</version>, I have to type:
$ sudo mvn package
$ java -jar ./target/hello-world-spring-0.0.1-SNAPSHOT.jar
Maybe this method can work for gradle as well.
Please check the path of the jar file build/libs/gs-spring-boot-0.1.0.jar. For your case, the jar might be in a different folder. If your code is in a module in the main project, then the jar will be in the build folder of the module.
If you git clone the repo, then the tutorial works. If you "To start from scratch, move on to Build with Gradle.", then the tutorial doesn't work. There are missing setup steps.
I got the same issue and I changed the command to java -jar target/rest-service-0.0.1-SNAPSHOT.jar (I checked the .jar file in target folder and found that the file name was incorrect).
Parent folder of my project was having spaces in it's name, i changed it to the underscore and it worked.
Looked at the command line as it was in the official guide:
./gradlew clean build && java -jar build/libs/gs-actuator-service-0.1.0.jar
First, the above command line has two parts:
(1) ./gradlew clean build //Use gradle wrapper to build
(2) java -jar build/libs/gs-actuator-service-0.1.0.jar //To run an application packaged as a JAR file
Now, one might run into issues with one part or both parts. Separating them and running just on thing at a time helped troubleshoot.
(1) didn't work for my Windows, I did the following instead and that built the application successfully.
.\gradlew.bat clean build
Now moving to (2) java -jar build/libs/gs-actuator-service-0.1.0.jar
It literally means that "Run a jar file that is called gs-actuator-service-0.1.0.jar under this directory/path: build/libs/" Again, for Windows, this translates to build\libs\ , and there's one more thing that may catch you: The jar file name can be slightly different depending on how it was actually named by the configuration in initial/setting.gradle:
rootProject.name = 'actuator-service'
Note that the official guide changed it from 'gs-actuator-service' to 'actuator-service' in their sample code but hasn't updated the tutorial accordingly. But now you know where the jar file name comes from, that doesn't matter anymore, and you have the choice to rename it however you want.
Having all the factors adjusted, below is what eventually worked in my case:
java -jar build\libs\actuator-service-0.0.1-SNAPSHOT.jar
or
java -jar C:\MyWorkspace\Spring\gs-actuator-service\initial\build\libs\actuator-service-0.0.1-SNAPSHOT.jar //with fully qualified path
If you are curious where does "-0.0.1-SNAPSHOT" come from, here it is:
in build.gradle
version = '0.0.1-SNAPSHOT'
Again, you have the choice to modify it however you want. For example, if I changed it to 0.0.2-SNAPSHOT, the command line should be adjusted accordingly
java -jar build\libs\actuator-service-0.0.2-SNAPSHOT.jar
Reference: https://docs.oracle.com/javase/tutorial/deployment/jar/basicsindex.html
Because you are trying to execute .jar file that doesn't exist. After building the project go to ./build/libs and check the name of freshly built .jar file and then in your project directory run:
./gradlew build && java -jar build/libs/name-of-your-jar-file.jar
or you can set version property to empty string in your build.gradle file
version = ''
after that:
./gradlew build && java -jar build/libs/your-project-name.jar
For Windows, these commands solved the problem: "Error: Unable to access jarfile springboot.jar":
cd target
java -jar springboot-0.0.1-SNAPSHOT.jar
run ./mvnw package
Now a folder named target is created and you can see a jar file inside it.
then execute java -jar target/<jarfilename>

org.apache.flink.api.java.io.jdbc.JDBCInputFormat NOT INSIDE FLINK JARS

I have created a new Java project in
eclipse-jee-kepler-SR2-win32-x86_64.
I have included the Jars in
flink-0.8.1\lib.
I have created the standard WordCount and it works.
I have modified my WordCount to take input from text files and csv files and it works.
all the imports work perfectly.
then i tried import org.apache.flink.api.java.io.jdbc.JDBCInputFormat.
Eclipse doesn't find it?
Why does Eclipse not find the import?
Because inside the jar flink-java-0.8.1.jar there is no directory io/jdbc.
I tried the same thing with flink-0.9.0-bin-hadoop27 and in the jar flink-dist-0.9.0.jar there is no org/apache/flink/api/java/io/jdbc directory. I uncompressed the jar and searched for the string "jdbcinputformat" with 0 results. I searched the string "jdbc" and it is only mentioned in org/apache/log4j, org/eclipse/jetty, and in other places that are not org.apache.flink.api.java.io
So my question is: Where do I find the class JDBCInputFormat?
What can I do to access SqlServer2012 in Flink (apart from accessing it outside Flink, create csv files, and then reading them in Flink (It sounds horrible to me since there should be a class specific for that))?
The corresponding module is not included. In order to use it, you need to build Flink from scratch. Run the following commands:
git clone https://github.com/apache/flink.git
cd flink
mvn -DskipTests clean install
This builds the latest snapshot for flink-0.10-SNAPSHOT. If you want to use stable version 0.9 run different git clone command:
git clone -b release-0.9 https://github.com/apache/flink.git
In your current project, you need to change the used Flink version in your pom file accordingly, eg, 0.10-SNAPSHOT or 0.9-SNAPSHOT.

How to Build a maven project using script file?

I have created a maven project in STS.I completed the development and testing code for my project.If now I want to run or build this project, then I have to do the following
Right click on the project-->Run as-->Run on Server (or)
Right click on the project-->Run as-->Maven Build
If I want to run the test code then
Right click on the class file-->Run as-->Run JUnit
But I want to create a text file I mean script file to run all these commands when I run this script file from the cmd prompt. I have found out on a web site that I should create a PowerShell file, So I don't cognize how to compose a script file like this, is there any example file for it ?
Please, anybody can help me
You can just run mvn clean install on your project root folder (i.e. where your pom.xml file is) in cmd prompt. This command will trigger your project default build lifecycle covering a number of build phases including:
validate
compile
test
package
integration-test
integration
verify
install
During these build phases, Maven will validate and compile your project, run tests (if any) against your codes, package the resultant binaries into say, a JAR file, run integration tests (if any) against your JAR, verify it, and then install the verified package to your local .m2 repository.
If you really want a script, then just add mvn clean install to your batch file.

Resources