I am new to Bazel. I need to download external dependency jar files using Bazel. Where to configure to download external jar files. I tried to add jar files in BUILD and WORKSPACE files , but not working.
Workspace file
maven_server(name="myserver",url="https://mvnrepository.com/artifact/org.apache.flink/flink-java")
maven_jar(name="flink-java",artifact="org.apache.flink:filnk-java:jar:1.2.0",server="myserver")
bind(name="flink-java",actual="#flink-java//jar")
Build file
java_binary(
name="read_files",
srcs = glob(["ReadFiles.java"]),
main_class="com.ibm.cdo.gts.contracts.pipeline.preprocess.ReadFiles",
deps=["//external:flink-java"],
)
The logical mistake you have there is that the workspace name (flink-java) is illegal see here and here. What is needed is to change the - to _ in maven_jar and of course the reference in the bind.
maven_jar(name="flink_java",artifact="org.apache.flink:flink-java:jar:1.2.0")
bind(name="flink-java",actual="#flink_java//jar")
Two more mistakes you had in the above were:
In the artifact coordinates you wrote filnk-java when you needed flink-java
The mvnrepository URL you used doesn't seem to be legal. I tried a few variants but couldn't get it to work. Additionally from the site they seem to link downloads to Maven Central itself so I'm not sure they server the jars. Having said that since Bazel fallbacks to Maven Central I just omitted the maven_server and the build passes. If you need help with the maven_server part and you need to use a private Maven repository please double check the URL first.
PS: It would have really helped if you added to your question the Bazel outputs since I needed to create my own workspace to know what were the problems you encountered.
Related
I really don't have much experience in developing let alone using build tools.
I was assigned a task to build dependencies locally and get the jar files.
say I have a list of deps (GAV) like this:-
1. org.jetbrains.kotlin:kotlin-stdlib:1.6.0-RC
2. com.auth0:java-jwt:3.18.2, etc
3. openapi4j:openapi-operation-validator:1.0.7, etc
So i was able to download the source code url from maven repository and source code from github programmatically, for example :-
org.jetbrains.kotlin:kotlin-stdlib:1.6.0-RC - https://github.com/JetBrains/kotlin
com.auth0:java-jwt:3.18.2 - https://github.com/auth0/java-jwt
openapi4j-openapi-operation-validator-1.0.7 https://github.com/openapi4j/openapi4j
but there are many build.gradle files in different directories, how do I know which directory should I move into before running the gradle build command.
Things I have already tried and failed:-
For deps like this openapi4j:openapi-operation-validator:1.0.7, i can directly go into the openapi-operation-validator folder in the Github repo (https://github.com/openapi4j/openapi4j ) and run the gradle build command, but not all projects are structured like that I guess?
For deps like this com.auth0:java-jwt:3.18.2, the artifactId (java-jwt
) is already present in the github path (https://github.com/auth0/java-jwt), so i can run the gradle build command on the root github repo.
From the spring guides , among all the Gradle.build files available I can check which file has:-
jar {
archiveBaseName = <artifactId>
archiveVersion = <version>
}
, then I can move to that dir and run Gradle build, but not all build.gradle files have this.
None of the above approaches are concrete, is there any other firm approach that I can use to tackle the problem?
Your approach is generally correct.
You need to find the source code in github/gitlab/wherever, read the readme file and try to build it with whatever build tool was used there.
This may or may not work.
QUESTION
I am not a maven pro and I got stuck trying to package a Kafka Connector. There are two options for packaging it:
Either you produce a folder with a jar that contains the connector + all the dependency jars - all the kafka-specific jars
Or build a fat jar with all of the dependencies (and I also assume without the kafka-specific jars again, but it is not explicit in the docs).
I am following docs on confluent webpage and the connector I am trying to package is this one on github.
What I tried, after cloning the repo with git, is the following mvn clean package. But this seems to create only a single jar of the original project with the dependencies in mvn cache (~/.m2/repository/).
Google also has link on how to create a fat jar, but I would need somehow to specify which jars I want to exclude from the fat jar.
Thanks
UPDATE
Now I am running:
connect-standalone /etc/kafka/connect-standalone.properties /etc/kafka/connect-cdc-mssql-source.properties
Where /etc/kafka/connect-standalone.properties contains the following line:
plugin.path=/shared_win_files
And ls -al /shared_win_files contains the following:
kafka-connect-cdc-mssql-0.0.1-SNAPSHOT.jar
And jar tvf kafka-connect-cdc-mssql-0.0.1-SNAPSHOT.jar contains the following:
6996 Thu Sep 07 14:47:24 BST 2017 com/github/jcustenborder/kafka/connect/cdc/mssql/MsSqlSourceConnector.class
where MsSqlSourceConnector.classis basically this class here which implements the Connector.
But when I try to run the connector with the command above, I get an error
Failed to find any class that implements Connector and which name matches com.github.jcustenborder.kafka.connect.cdc.mssql.MsSqlSourceConnector
It gives a massive list with all available plugins, but mine is not in there.
Currently, an easy way to package your connector with maven is to use maven-assembly-plugin. This basically entails two main steps:
Define one or more assembly descriptors and save them under src/assembly.
Doc: http://maven.apache.org/plugins/maven-assembly-plugin/descriptor-refs.html
Example: https://github.com/confluentinc/kafka-connect-elasticsearch/blob/master/src/assembly/package.xml
In the descriptor, among other things, you may choose the packaging format of your archive, files and directories to include or exclude, as well as specific settings regarding your projects dependencies.
Include the plugin in your project's pom.xml
Example: https://github.com/confluentinc/kafka-connect-elasticsearch/blob/master/pom.xml
This mainly requires you to define the configuration and execution sections of the maven-assembly-plugin section. Additionally you can associate calls to specific assembly plugin descriptors with certain maven profiles that you may define.
Finally, stay tuned because packaging your Kafka Connect plugins (connectors, transforms, converters) might be significantly simplified soon.
Following Konstantine's answer how to package jars.
The remaining problem was that when specifying plugin.path=/abc in KafkaConnect config, you have to be careful.
You can either put a fat jar like this:
/abc/fatjar.jar
Or you have to create another folder in abc and put all the related jars into that folder like this:
/abc/my-connector-a/connector.jar
/abc/my-connector-a/connector-dependency.jar
...
As in my case, it was treating jars as separate plugins.
I am trying to build a p2 repository using Tycho and the tycho-p2-repository-plugin in an eclipse-repository packaging type. When I run the build, it is giving me the following exception:
Failed to execute goal org.eclipse.tycho:tycho-p2-repository-plugin:0.13.0:assemble-repository (default-assemble-repository) on project com.mycompany: No content specified for p2 repository
Can someone point me to the cause of this problem?
You need either of the following files in the root of the eclipse-repository project:
A category.xml files with at least one feature referenced in it
A *.product file
These files define what should be included in the p2 repository, so if there is none of these, the p2 repository would be empty. This is probably not what you wanted, so Tycho fails the build. (Admittedly, the error message could be more helpful...)
I had this problem too, unfortunately it was one of many problems and I do not remember which thing I did to fix it.
Make sure you are using 0.14.1 of tycho.
My working version is a very empty pom.xml, much like Packaging Types
I have not needed to configure the eclipse-repository via tycho-p2-repository-plugin.
Do you have a valid target definition defined in your reactor build somewhere?
Is your *.product file valid? The product file needs to be in the same directory as your eclipse-repository pom, I think it gets found via black magic.
Also make sure your product feature is separate from your eclipse-repository.
under PDE builds you could have the *.product file and the with the feature that is the root for your product.
How can a corporate Maven repository be used (to the exclusion of other repositories) with sbt 0.11.x, as described in how do I get sbt to use a local maven proxy repository (Nexus)? ? There is no mention of ivyRepositories in the new sbt wiki at github, so I'm assuming the accepted solution there is out of date.
Step 1: Follow the instructions at Detailed Topics: Proxy Repositories, which I have summarised and added to below:
(If you are using Artifactory, you can skip this step.) Create an entirely separate Maven proxy repository (or group) on your corporate Maven repository, to proxy ivy-style repositories such as these two important ones:
http://repo.typesafe.com/typesafe/ivy-releases/
http://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/
This is needed because some repository managers cannot handle Ivy-style and Maven-style repositories being mixed together.
Create a file repositories, listing both your main corporate repository and any extra one that you created in step 1, in the format shown below:
[repositories]
my-maven-proxy-releases: http://repo.example.com/maven-releases/
my-ivy-proxy-releases: http://repo.example.com/ivy-releases/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
Either save that file in the .sbt directory inside your home directory, or specify it on the sbt command line (you will need to specify if you have disabled sharing):
sbt -Dsbt.repository.config=<path-to-your-repo-file>
Good news for those using older versions of sbt: Even though, in the sbt 0.12.0 launcher jar at least, the boot properties files for older sbt versions don't contain the required line (the one that mentions repository.config), it will still work for those versions of sbt if you edit those files to add the required line, and repackage them into the sbt 0.12.0 launcher jar! This is because the feature is implemented in the launcher, not in sbt itself. And the sbt 0.12.0 launcher is claimed to be able to launch all versions of sbt, right back to 0.7!
Step 2: To make sure external repositories are not being used, remove the default repositories from your resolvers. This can be done in one of three ways:
Add the command line option -Dsbt.override.build.repos=true mentioned on the Detailed Topics page above. This will cause the repositories you specified in the file to override any repositories specified in any of your sbt files. This might only work in sbt 0.12 and above, though - I haven't tried it yet.
Having the same effect as 1, you can use overrideBuildResolvers := true, with the advantage that you can control the projects where it is applicable, depending on which scope (a project / ThisBuild / Global) you define it in. This works in sbt 0.13.
Use fullResolvers := Seq( resolver(s) for your corporate maven repositories ) in your build files, instead of resolvers ++= or resolvers := or whatever you used to use.
Finally, note that the sbt launcher script has a bug in reading the sbtopts file, so if you decide to put your common sbt command-line options in there, make sure the last line of the file ends in a newline (Emacs in particular can fail to ensure this, unless configured to do so).
An alternative for Step 2 of the accepted answer (am using sbt 0.13.1):
Add file .sbtopts to the project root directory with contents:
-Dsbt.override.build.repos=true
Another alternative is to add this line in $SBT_HOME/conf/.sbtopts, but this would force the setting for all projects.
Unpack the sbt-launcher.jar and copy the sbt.boot.properties file to a location of your choice. Change the launch script to use this file. In the file, change the repositories section to only contain your local repo and the corporate one. The distinction between Maven and Ivy comes from the given pattern (no pattern means Maven pattern by default).
Here is an example:
[repositories]
local
corporate: http://inhouse.acme.com/releases/
I realize that this question is pretty much the exact question found here. However, seeing as that question is 1.5 years old (or so), I would like to revisit it. How does one add local dependencies using leiningen? Surely this capability must exist by now?
Create a private Maven Repository, and then, add the following to your project.clj
:repositories {"local" ~(str (.toURI (java.io.File. "your_local_repository")))}
If the jars are based on your own projects, you can use lein install to put them into your local .m2, or use the checkout-dependencies feature.
You can also use the extra-classpaths feature, etc.
I found that the easiest (albeit somewhat hacky) solution is to do the following:
For an existing project that you're using as a dependency:
In your local project that has the dependency you want to modify, ensure you run lein deps
Clone the repo of this dependency so you can modify it locally (obv. make sure you're using the same tag as the version you specify in your project.clj file)
Run lein uberjar in this dependency dir (where the relevant project.clj file lives)
Copy the generated standalone jar in target/ to the exact path/file of your local maven files... (something like: ~/.m2/repository/project/.../file.jar); Ensure that you backup the original jar file so you can restore it later on if that is desirable
For development of your own project:
Within the project or plugin you're developing, simply run lein install
Find out where your local maven repo is (see above for an example path)
Enter dependency information in your test project like you would for any other leiningen project
Again, this is a quick hack and perhaps not the way you'd go about doing serious local development, but I found it easy enough for what I wanted. Check out lein help tutorial for much more info