QUESTION
I am not a maven pro and I got stuck trying to package a Kafka Connector. There are two options for packaging it:
Either you produce a folder with a jar that contains the connector + all the dependency jars - all the kafka-specific jars
Or build a fat jar with all of the dependencies (and I also assume without the kafka-specific jars again, but it is not explicit in the docs).
I am following docs on confluent webpage and the connector I am trying to package is this one on github.
What I tried, after cloning the repo with git, is the following mvn clean package. But this seems to create only a single jar of the original project with the dependencies in mvn cache (~/.m2/repository/).
Google also has link on how to create a fat jar, but I would need somehow to specify which jars I want to exclude from the fat jar.
Thanks
UPDATE
Now I am running:
connect-standalone /etc/kafka/connect-standalone.properties /etc/kafka/connect-cdc-mssql-source.properties
Where /etc/kafka/connect-standalone.properties contains the following line:
plugin.path=/shared_win_files
And ls -al /shared_win_files contains the following:
kafka-connect-cdc-mssql-0.0.1-SNAPSHOT.jar
And jar tvf kafka-connect-cdc-mssql-0.0.1-SNAPSHOT.jar contains the following:
6996 Thu Sep 07 14:47:24 BST 2017 com/github/jcustenborder/kafka/connect/cdc/mssql/MsSqlSourceConnector.class
where MsSqlSourceConnector.classis basically this class here which implements the Connector.
But when I try to run the connector with the command above, I get an error
Failed to find any class that implements Connector and which name matches com.github.jcustenborder.kafka.connect.cdc.mssql.MsSqlSourceConnector
It gives a massive list with all available plugins, but mine is not in there.
Currently, an easy way to package your connector with maven is to use maven-assembly-plugin. This basically entails two main steps:
Define one or more assembly descriptors and save them under src/assembly.
Doc: http://maven.apache.org/plugins/maven-assembly-plugin/descriptor-refs.html
Example: https://github.com/confluentinc/kafka-connect-elasticsearch/blob/master/src/assembly/package.xml
In the descriptor, among other things, you may choose the packaging format of your archive, files and directories to include or exclude, as well as specific settings regarding your projects dependencies.
Include the plugin in your project's pom.xml
Example: https://github.com/confluentinc/kafka-connect-elasticsearch/blob/master/pom.xml
This mainly requires you to define the configuration and execution sections of the maven-assembly-plugin section. Additionally you can associate calls to specific assembly plugin descriptors with certain maven profiles that you may define.
Finally, stay tuned because packaging your Kafka Connect plugins (connectors, transforms, converters) might be significantly simplified soon.
Following Konstantine's answer how to package jars.
The remaining problem was that when specifying plugin.path=/abc in KafkaConnect config, you have to be careful.
You can either put a fat jar like this:
/abc/fatjar.jar
Or you have to create another folder in abc and put all the related jars into that folder like this:
/abc/my-connector-a/connector.jar
/abc/my-connector-a/connector-dependency.jar
...
As in my case, it was treating jars as separate plugins.
Related
I really don't have much experience in developing let alone using build tools.
I was assigned a task to build dependencies locally and get the jar files.
say I have a list of deps (GAV) like this:-
1. org.jetbrains.kotlin:kotlin-stdlib:1.6.0-RC
2. com.auth0:java-jwt:3.18.2, etc
3. openapi4j:openapi-operation-validator:1.0.7, etc
So i was able to download the source code url from maven repository and source code from github programmatically, for example :-
org.jetbrains.kotlin:kotlin-stdlib:1.6.0-RC - https://github.com/JetBrains/kotlin
com.auth0:java-jwt:3.18.2 - https://github.com/auth0/java-jwt
openapi4j-openapi-operation-validator-1.0.7 https://github.com/openapi4j/openapi4j
but there are many build.gradle files in different directories, how do I know which directory should I move into before running the gradle build command.
Things I have already tried and failed:-
For deps like this openapi4j:openapi-operation-validator:1.0.7, i can directly go into the openapi-operation-validator folder in the Github repo (https://github.com/openapi4j/openapi4j ) and run the gradle build command, but not all projects are structured like that I guess?
For deps like this com.auth0:java-jwt:3.18.2, the artifactId (java-jwt
) is already present in the github path (https://github.com/auth0/java-jwt), so i can run the gradle build command on the root github repo.
From the spring guides , among all the Gradle.build files available I can check which file has:-
jar {
archiveBaseName = <artifactId>
archiveVersion = <version>
}
, then I can move to that dir and run Gradle build, but not all build.gradle files have this.
None of the above approaches are concrete, is there any other firm approach that I can use to tackle the problem?
Your approach is generally correct.
You need to find the source code in github/gitlab/wherever, read the readme file and try to build it with whatever build tool was used there.
This may or may not work.
I am new to Bazel. I need to download external dependency jar files using Bazel. Where to configure to download external jar files. I tried to add jar files in BUILD and WORKSPACE files , but not working.
Workspace file
maven_server(name="myserver",url="https://mvnrepository.com/artifact/org.apache.flink/flink-java")
maven_jar(name="flink-java",artifact="org.apache.flink:filnk-java:jar:1.2.0",server="myserver")
bind(name="flink-java",actual="#flink-java//jar")
Build file
java_binary(
name="read_files",
srcs = glob(["ReadFiles.java"]),
main_class="com.ibm.cdo.gts.contracts.pipeline.preprocess.ReadFiles",
deps=["//external:flink-java"],
)
The logical mistake you have there is that the workspace name (flink-java) is illegal see here and here. What is needed is to change the - to _ in maven_jar and of course the reference in the bind.
maven_jar(name="flink_java",artifact="org.apache.flink:flink-java:jar:1.2.0")
bind(name="flink-java",actual="#flink_java//jar")
Two more mistakes you had in the above were:
In the artifact coordinates you wrote filnk-java when you needed flink-java
The mvnrepository URL you used doesn't seem to be legal. I tried a few variants but couldn't get it to work. Additionally from the site they seem to link downloads to Maven Central itself so I'm not sure they server the jars. Having said that since Bazel fallbacks to Maven Central I just omitted the maven_server and the build passes. If you need help with the maven_server part and you need to use a private Maven repository please double check the URL first.
PS: It would have really helped if you added to your question the Bazel outputs since I needed to create my own workspace to know what were the problems you encountered.
(Not found in the Activator documentation)
It seems that it is possible to have Activator also use an existing local Maven repository by adding the following entry (in bold) in file build.sbt:
resolvers += Seq(
"Local Maven Repository" at "file://q:/repositories/maven",
"Sonatype OSS Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots"
)
I am not sure it works but anyway, the problem with this approach is that the project structure must already have been created (and therefore a local repository created and automatically populated by downloads), hence my question : is it possible to tell Activator before it creates the project structure that it should use some local Maven repository ?
Thanks in advance for any hint.
Activator makes use of the sbt-launcher. You can use the sbt-launcher to control which repositories sbt makes use of by default for each project and for the launcher itself.
If you'd like to modify the activator launcher itself, unzip the jar file and take a look at the sbt/boot.properties file included. You can use the format outlined at sbt's launcher docs to add your local maven repository to the list.
A simpler option in the future (but not enabled in our current properties file) is the launcher's ability to have an override repository configuration file. See: Sbt's proxy configuration docs. This file would allow you to specify the repositories you wish activator to use by default. We disabled this to ensure the offline repository which activator uses is added by default. However, I'll open a ticket to re-enable this feature. That way, you should be able to just create a ~/.sbt/repositories file with the following contents:
[repositories]
activator-local: file://${activator.local.repository-${activator.home-${user.home}/.activator}/repository}, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
local
maven-local
maven-central
typesafe-releases: http://typesafe.artifactoryonline.com/typesafe/releases
typesafe-ivy-releases: http://typesafe.artifactoryonline.com/typesafe/ivy-releases, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
Note: the ~/.sbt/boot directory will always exist. This is created to ensure that no other process deletes jar files we use while running, so we copy these out of the local cache. If we didn't, you'd see some really fun error messages.
How can I create a simple bundle with an Activator in bndtools?
It keeps saying that:
The JAR is empty: The instructions for the JAR named com.myproj did not cause any content to be included, this is likely wrong bnd.bnd /com.myproj Unknown Bndtools Problem Marker
Unused Private-Package instructions, no such package(s) on the class path: [com.myproj] bnd.bnd /com.myproj Unknown Bndtools Problem Marker
The way I create this project in Eclipse is:
Create new "Bndtools OSGi project"
Right click, configure - Convert to Maven project
Create Activator.java in package com.myproj.
Add com.myproj to private packages
Set activator to com.Activator
Here is my bnd file:
Bundle-Activator: com.myproj.Activator
Private-Package: com.myproj
My generated jar is empty. Any tips?
P.S.: Here is my eclipse project (exported as a zip-archive) in case it sheds any light on things: https://dl.dropbox.com/u/9162958/scraper.zip
My guess is that "Convert to Maven project" is the trouble. This likely has changed the Eclipse classpath for the project from the bnd default bin folder to 'target/classes'. Can you confirm that it works without converting to maven?
bnd can work with other places for the bin folder, you must set the ${bin} property (preferably in cnf/build.bnd). There are some writeups how to use bndtools with maven. The reason that bnd does not follow Eclipse's settings here is that they are not available without Eclipse and a design goal of bnd is that it builds anywhere: the bnd file must therefore be the final arbiter of information.
Anyway one more tip ... activators are not the right way to build OSGi builds since they are an evil singleton. Declarative services is far superior and we should actually have used a similar mechanism when we designed OSGi.
My setup:
Eclipse Luna 4.4.0 (20140612-0600)
Bndtools 2.3.0.REL-20140510-023245
Here is how I made it work:
I downloaded you exported scraper.zip.
Created and empty workspace in Eclipse.
Imported your project from the ZIP archive into the empty workspace.
The default cnf project was automatically created.
By default, bnd is configured to use the bin directory for compiled *.class files while your Eclipse project is configured to use target/classes. Therefore, I had to change this settings in cnf/build.bnd by adding a single line:
########################
## BND BUILD SETTINGS ##
########################
bin: target/classes
Now, after cleaning and rebuilding the project, bnd creates generated/scraper.jar that contains your Activator.class.
Notes:
You could also adapt your project configuration to use the bin directory instead of target/classes but I assume that you will use Maven later on.
When using bndtools, it sometimes helps when you start with an empty workspace and import your projects one-by-one.
There is a bug in bndtools 2.4 which causes some problems if there are multiple source directories per project. Therefore, I'm still using version 2.3
How can a corporate Maven repository be used (to the exclusion of other repositories) with sbt 0.11.x, as described in how do I get sbt to use a local maven proxy repository (Nexus)? ? There is no mention of ivyRepositories in the new sbt wiki at github, so I'm assuming the accepted solution there is out of date.
Step 1: Follow the instructions at Detailed Topics: Proxy Repositories, which I have summarised and added to below:
(If you are using Artifactory, you can skip this step.) Create an entirely separate Maven proxy repository (or group) on your corporate Maven repository, to proxy ivy-style repositories such as these two important ones:
http://repo.typesafe.com/typesafe/ivy-releases/
http://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/
This is needed because some repository managers cannot handle Ivy-style and Maven-style repositories being mixed together.
Create a file repositories, listing both your main corporate repository and any extra one that you created in step 1, in the format shown below:
[repositories]
my-maven-proxy-releases: http://repo.example.com/maven-releases/
my-ivy-proxy-releases: http://repo.example.com/ivy-releases/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
Either save that file in the .sbt directory inside your home directory, or specify it on the sbt command line (you will need to specify if you have disabled sharing):
sbt -Dsbt.repository.config=<path-to-your-repo-file>
Good news for those using older versions of sbt: Even though, in the sbt 0.12.0 launcher jar at least, the boot properties files for older sbt versions don't contain the required line (the one that mentions repository.config), it will still work for those versions of sbt if you edit those files to add the required line, and repackage them into the sbt 0.12.0 launcher jar! This is because the feature is implemented in the launcher, not in sbt itself. And the sbt 0.12.0 launcher is claimed to be able to launch all versions of sbt, right back to 0.7!
Step 2: To make sure external repositories are not being used, remove the default repositories from your resolvers. This can be done in one of three ways:
Add the command line option -Dsbt.override.build.repos=true mentioned on the Detailed Topics page above. This will cause the repositories you specified in the file to override any repositories specified in any of your sbt files. This might only work in sbt 0.12 and above, though - I haven't tried it yet.
Having the same effect as 1, you can use overrideBuildResolvers := true, with the advantage that you can control the projects where it is applicable, depending on which scope (a project / ThisBuild / Global) you define it in. This works in sbt 0.13.
Use fullResolvers := Seq( resolver(s) for your corporate maven repositories ) in your build files, instead of resolvers ++= or resolvers := or whatever you used to use.
Finally, note that the sbt launcher script has a bug in reading the sbtopts file, so if you decide to put your common sbt command-line options in there, make sure the last line of the file ends in a newline (Emacs in particular can fail to ensure this, unless configured to do so).
An alternative for Step 2 of the accepted answer (am using sbt 0.13.1):
Add file .sbtopts to the project root directory with contents:
-Dsbt.override.build.repos=true
Another alternative is to add this line in $SBT_HOME/conf/.sbtopts, but this would force the setting for all projects.
Unpack the sbt-launcher.jar and copy the sbt.boot.properties file to a location of your choice. Change the launch script to use this file. In the file, change the repositories section to only contain your local repo and the corporate one. The distinction between Maven and Ivy comes from the given pattern (no pattern means Maven pattern by default).
Here is an example:
[repositories]
local
corporate: http://inhouse.acme.com/releases/