We're wanting to set up Elastic Search with Appbase.io's abc tool (pointed at one of our existing PostgreSQL dynos). We don't want to compile either of these tools, but Heroku's buildpack documentation strongly suggests that there are flags that are set with the java buildpack sets that we want/need. The most simple solution would probably be to set up an empty pom file, but I don't know if that would actually work. (I assume it's not that hard to do--no java knowledge here.)
I plan to use Amiel's fix of the apt buildpack to actually install Elastic Search, and a custom buildpack to grab & install abc.
Is this a reasonable approach?
What are the gottchas we need to watch out for?
Related
I'm currently developping a Spring Native application, it's building using paketo buildpack and generating a Docker image.
I was wondering if it's possible to customize the generated Docker image by adding third party tools (like a Datadog agent for example).
Also, for now the generated container image is installed locally, is it possible to send it directly in another Docker repo ?
I'm currently developping a Spring Native application, it's building using paketo buildpack and generating a Docker image. I was wondering if it's possible to customize the generated Docker image by adding third party tools (like a Datadog agent for example).
This applies to Spring Boot apps, but really also any other app you can build with buildpacks.
There are a couple of options:
You can customize the base image that you use (called a stack).
You can add additional buildpacks which will perform more customizations during the build.
#2 is obviously easier if there is a buildpack that provides the functionality that you require. In regards to Datadog specifically, the Paketo buildpack now has a Datadog Buildpack you can use with Java and Node.js apps.
It's more work, but you can also create a buildpack if you are looking to add specific functionality. I wouldn't recommend this if you have one application that needs the functionality, but if you have lots of applications it can be worth the effort.
A colleague of mine put this basic sample buildpack together, which installs and configures a fictitious APM agent. It is a pretty concise example of this scenario.
#1 is also possible. You can create your own base image and stack. The process isn't that hard, especially if you base it on a well-known and trusted image that is getting regular updates. The Paketo team also has the jam create-stack command which you can use to streamline the process.
What's more difficult with both options is that you need to keep them up-to-date. That requires some CI to watch for software updates & publish new versions of your buildpack or stack. If you cannot commit to this, then both are a bad idea because your customization will get out of date and potentially cause security problems down the road.
UPDATE
You can bundle dependencies with your application. This option works well if you have static binaries you need to include, perhaps a cli you call to from your application.
In this case, you'd just create a folder in your project called binaries/ (or whatever you want) and place the static binaries in there (make sure to download versions compatible with the container image you're using, Paketo is Ubuntu Bionic at the time I write this). Then when you call the cli commands from your application, simply use the full path to them. That would be /workspace/binaries or /workspace/<path to binaries in your project>.
You can use the apt buildpack to install packages with apt. This is a generic buildpack that you provide a list of apt packages to and it will install them.
This can work in some cases, but the main drawback is that buildpacks don't run as root, so this buildpack cannot install these packages into their standard locations. It attempts to work around this by setting env variables like PATH, LD_LIBRARY_PATH, etc to help other applications find the packages that have been installed.
This works ok most of the time, but you may encounter situations where an application is not able to locate something that you install with the apt buildpack. Worth noting if you see problems when trying this approach.
END OF UPDATE
For what it's worth, this is a common scenario that is a bit painful to work through. Fortunately, there is an RFC that should make the process easier in the future.
Also, for now the generated container image is installed locally, is it possible to send it directly in another Docker repo ?
You can docker push it or you can add the --publish flag to pack build and it will send the image to whatever registry you tell it to use.
https://stackoverflow.com/a/28349540/1585136
The publish flag works the same way, you need to name your image [REGISTRYHOST/][USERNAME/]NAME[:TAG].
For me what worked was in my build.gradle file (I'm using kotlin) I added this:
bootBuildImage {
val ecrRepository: String? by project
buildpacks = listOf("urn:cnb:builder:paketo-buildpacks/java", "urn:cnb:builder:paketo-buildpacks/datadog")
imageName = "$ecrRepository:${project.version}"
environment = mapOf("BP_JVM_VERSION" to "17.*", "BP_DATADOG_ENABLED" to "true")
isPublish = true
docker {
val ecrPassword: String? by project
publishRegistry {
url = ecrRepository
username = "AWS"
password = ecrPassword
}
}
}
notice the buildpacks part where I added first the base default oci and then the datadog oci. I also added on the environment the BP_DATADOG_ENABLED to true, so that it adds the agent.
This is an inverse of my question Install a custom feature or module in Opendaylight?. I am looking to take the Hello World app and add the ability to do a feature:install for the following features:
feature:install
odl-restconf
odl-mdsal-apidocs
odl-openflowplugin-flow-services
odl-openflowplugin-app-table-miss-enforcer
odl-openflowplugin-nxm-extensions
odl-restconf-all
odl-openflowplugin-flow-services
I assume it comes down to listing these features properly in one of the many pom files.
For the record, currently we run the ODL 0.12.1 by downloading the .tar.gz from ODL's nexus server, shell into karaf, and run the feature:install command against all the above features and I am able to do the install.
I'd like to be able to run that same feature:install for all those features as well, but in the Hello World example, karaf can't find those features.
There are some similar questions out there (such as this How to add new features to OpenDayLight Karaf?) however the answers weren't specific enough and seem generic to Karaf. For example, the answer there seems to be about modifying the values of org.ops4j.pax.url.mvn.repositories however when I look at the ODL 0.12.1 integration/distribution repo, I do not see this value being used at all.
I think what you are looking for is the featuresBoot config in the
etc/org.apache.karaf.features.cfg file.
I've got a hello world opendaylight app (created following the tutorials) which compiles using a mvn clean install, and appears when I run the karaf package that is also generated.
However I am unable to get it to run in another ODL install (downloaded the binary with all the other packages from the website), and even using a
bundle:install mvn:org.andrew.test
results in unable to install bundle (tried copying to deploy and system/org/andrew....)
How do you get a bundle which can be used in another install?
Why do you want to bundle:install instead of feature:install ?
What most existing ODL projects do for you, and what the example generated by the archetype should also show you how to do for your custom org.andrew.test one (have you used the archetype? try it..) is that there is a local karaf/ artifact which correctly depends on the features/odl-something feature of your example, and lets you install it, which will install your bundle/s.
In theory and if you really know what you are doing, you can also get it to work for what you call "in another install" in your question, but you have to use repo-app and what not - most people do not use it like that AFAIK (at least in ODL development); so I wouldn't bother, if I were you.
If you want to learn more about this in general outside of OpenDaylight, the general Karaf documentation manual is not bad. Beware that in ODL we've tweaked a few things though; for example, we have (intentionally) disabled the direct installation from ~/.m2/repository (for better isolation).
The Heroku buildpack api documentation makes no mention of support for a config_vars property in the YAML hash exported by the release script. The only properties documented there are addons and default_process_types.
However, I have seen multiple custom buildpacks making use of such a property.
So my question is, is this some undocumented, working feature? Or is it a feature that has been removed and therefore undocumented?
If it's working, how exactly is it supposed to work? Does it just add the values as environment variables when the slug is running, or does it add them to the configuration you'd see through the management interface?
My guess is that it isn't working because the buildpack I'm using includes a PATH variable that isn't reflected in the PATH I see when I log in to the ad hoc console. However it's possible that I misunderstand how the feature works, or that my release script is broken.
It turns out that it's an undocumented feature which currently works.
Just like the behavior of the addons property provided by the release script, the config_vars property provided by the same script will only be processed the FIRST time the app is deployed.
It creates entries in the config vars which are visible through the management tool, and these will then be available as environment variables in the app.
So if you change the value of config_vars in your custom buildpack, the change will NOT be reflected in any existing apps that have already been deployed using that buildpack. This includes if you change to a different branch of the Git repo where your custom buildpack is stored which has a different release script.
I am looking for extending WebSphere Liberty buildpack included in Bluemix with some third party libraries from our application architecture, so the size of EAR file will decrease a lot and cf push command will be more fast and agile. Is it possible?
I know there is a WebSphere Liberty buildpack open sourced at Cloudfoundry.org but as far as I know, it is not so powerful like the one included in Bluemix and we would loose some interesting features.
Thanks!
The Liberty buildpack you find on GitHub should be pretty similar to the one Bluemix uses especially in terms of features. Beta features are included and auto-configuration should work so in theory if you do cf push appName -b https://github.com/cloudfoundry/ibm-websphere-liberty-buildpack.git -p myapp.war it should work the same way if you did cf push appName -p myapp.war.
If you want to modify the buildpack you can fork it and add the jars you want although I am unsure on the process for adding the jars. Maybe someone else can add an answer that points you in the right direction.
You can find an alternative buildpack for Websphere Liberty here. As you will notice, it is actually a CloudFoundry buildpack -- normally all of them will work in Bluemix. To use it, add a buildpack: line to your manifest.yml or a -b option to cf command line tool.
There is not much detail in your question as to what libraries or modifications you need, but this buildpack is in github and you can freely fork it and modify it. The buildpack prepares the runtime, and you should be able to add your own downloads in some hooks and replicate the directory structure you must be using locally. It is also very well documented, so you should find your way through the configurations if you are doing anything non-standard.
You are welcome to share your own new buildpack too!
If you want to be more efficient during the development cycle where you constantly push changes, you can try the 'development mode' support in the Bluemix Liberty buildpack. This allows you to push incremental file changes without even restarting the Liberty server (not to mention the whole application container, aka no cf push). See the doc here: https://www.ng.bluemix.net/docs/manageapps/eclipsetools/eclipsetools.html#incrementalpublish. You can also do remote debugging with development mode.
To customize a Liberty server in Bluemix, you can also use the server package command of a local Liberty server, and then cf push the generated zip.
See https://www.ibm.com/developerworks/community/blogs/msardana/entry/developing_with_bluemix_customizing_the_liberty_build_pack_to_add_your_configurations for details
That, of course, defeats your goal of smaller deployment times, but I wanted to add the possibility for completeness and to prevent someone from forking the buildpack without needing it.