I've an ECS cluster running Fargate instances with Springboot apps & want to enable tracing with least number of code changes. There're the two approaches I started looking at:
Use AWS-Xray : Steps -> Add dependencies, add aWSXRayServletFilter, run X-Ray daemon in a separate container.
Use Spring Cloud Sleuth : Steps -> Add dependency & property, integrate with X-Ray
So the second approach saves you number of steps in modifying your code, the issues is I couldn't find any good doc to integrate Spring Cloud Sleuth with X-Ray, can anyone point me to correct direction?
I tried reading number of docs including: https://cloud.spring.io/spring-cloud-sleuth/spring-cloud-sleuth.html
I came across this when looking for a solution for option two. AFAIK, you still have to use the X-Ray daemon. I had to look across multiple GitHub repos and issues to solve the problem so am providing the solution here.
I used Gradle for my solution but this can be easily translated to Maven as well.
Add the BOM for spring cloud
dependencyManagement {
imports {
mavenBom("org.springframework.cloud:spring-cloud-dependencies:2021.0.3")
}
}
Add the following dependencies to the project.
implementation 'org.springframework.cloud:spring-cloud-starter-sleuth'
implementation 'org.springframework.cloud:spring-cloud-sleuth-zipkin'
implementation 'io.zipkin.aws:zipkin-reporter-xray-udp:0.23.4'
Then Add configuration to define the Bean which is used for reporting to the X-Ray daemon.
#Configuration
public class TracingConfiguration {
#Bean(ZipkinAutoConfiguration.REPORTER_BEAN_NAME)
Reporter<Span> reporter() {
return XRayUDPReporter.create();
}
}
Define the propagation type as aws for Sleuth as per the documentation.
spring.sleuth.propagation.type=aws
I haven't tried it yet, but from the documentation you can combine the following
An amazon/aws-xray-daemon
zipkin-aws with the experimental X-Ray Storage. They have a Docker image for zipkin-aws. You need to point it to the XRay daemon. This will be running as a Zipkin server listening on port 9411.
Then you use Spring Cloud Sleuth's instrumentation and AsyncZipkinSender.
By doing this approach, you can decouple yourself from AWS as long as you have a different zipkin server.
currently AWS X-Ray SDK doesn't have integration with Spring Cloud Sleuth. In order to using AWS X-Ray, the first approach would be the best way to do it.
Related
When creating a new Spring Boot project using Spring Initializr and adding Azure Support, it adds a dependency to com.microsoft.azure:azure-spring-boot-starter.
implementation 'com.microsoft.azure:azure-spring-boot-starter'
The spring cloud documentation says:
The Azure Support entry contains auto-configuration support for Azure
managed services [...]
Now I'd like to send and receive messages from Azure Service Bus and the documentation wants me to add a dependency to azure-servicebus.
implementation 'com.microsoft.azure:azure-spring-boot-starter'
implementation 'com.microsoft.azure:azure-servicebus'
Okay fine. When I now switch over to Microsoft and read the documentation about Service Bus there, it mentions two libraries, where Microsoft states the second one is dated and legacy.
azure-messaging-servicebus (latest) implementation 'com.azure:azure-messaging-servicebus:7.0.0'
azure-servicebus (legacy) 'com.microsoft.azure:azure-servicebus'
Question
Is the Spring Boot documentation just outdated?
I have read somewhere that libraries in com.microsoft.com are for managing the resource itself, while libraries in com.azure are for managing the data. Is this true?
What is meant by auto-configuration support?
As you can see I am very confused which dependencies I need to add. Also I don't get the difference between packages from com.azure and com.microsoft.azure.
Can someone please shed some light on this?
Let me preface this with the fact that I am still very new to Apache Camel. I'm still trying to understand how it all works, and what needs to be done (and HOW to do it) to achieve a particular effect.
I am trying to develop a Spring Boot application that will use Apache Camel to handle the transmission (and possibly also receipt) of data to/from a number of possible sources and destinations. The purpose of the application is to provide a means to produce/generate network traffic, at the network application level, that will be fed into another Spring Boot application - let's call this the target. We are trying to observe and measure the effects various network loads have on the target.
We would like to be able to transmit data via a number of protocols, including: ftp, http/s, file systems (nfs), various mail protocols (smtp, pop) and data streaming protocols for voice and video. There may be other protocols added at a later time. The data itself is irrelevant, we just need to be able to transmit data via various protocols with various loads.
These applications/services will be running in a containerized environment (Docker) that will be run within our local development and test environment, as well as possibly in a cloud environment, such as AWS. We have used Docker, Ansible, Terraform and are currently working towards using Kubernetes and Istio to manage the configuration, deployment, and operation of these applications.
We need to be able to provide specific configurations of Camel routes for particular deployments.
It would appear that the preferred method to configure Camel routes is via Java DSL, rather than XML DSL. The Camel documentation and nearly every other source of information I've found have a strong bias towards using Java DSL. Examples of XML DSL route configuration are far and few.
My initial impression is that going the Java DSL route (excuse the pun), would not work well with our need to be able to deploy a Camel application with a specific route configuration. It seems like you are required to have Java DSL defined route configurations hardwired into the code.
We think that it will be easier to provide a specific route configuration via an XML file that can be included in a deployment, hence why I've been trying to investigate and experiment with XML DSL. Perhaps we are mistaken in this regard.
My question to the community is: Considering what I've described above, can the Java DSL approach be used to meet the requirements as I've described them? Can we use Java DSL in a way that allows for dynamic route configuration? Keep in mind we would not be attempting to change configuration during operation, just in the course of performing a deployment.
If Java DSL could be used for this purpose, it would be very much appreciated if pointers to documentation, examples, etc. could be provided.
For your use cases you could use XML DSL also. Anyhow below book covers most aspects Camel development with examples. In this book authors describes XML DSL use for most of java DSL examples.
https://www.manning.com/books/camel-in-action-second-edition
In below github repository you can find the source code for all the examples listed in above book.
https://github.com/camelinaction/camelinaction2
Simple tutorial and github repository for Apache Camel using Spring boot.
https://www.baeldung.com/apache-camel-spring-boot
https://github.com/eugenp/tutorials/tree/master/spring-boot-modules/spring-boot-camel
Maven Plugin for build and deployment of spring boot container application into Kubernetes cluster
https://maven.fabric8.io/
In case if your company can afford some funding for your effort look at below link which provides commercial offerings around Camel.
https://camel.apache.org/manual/latest/commercial-camel-offerings.html
Thanks
Madhu Gupta
Our team has a few projects which use the Java DSL for building routes. In order to make them dynamic, there are control structures for iterating and setting endpoints based off configurations. That works for us because the routes are basically all the same, just with different sources and sinks.
If you could dynamically add/change the XML DSL files in a way that doesn't involve redeploying your application, that might be a viable route to follow. One might, for example, change the camel.springboot.xml-routes property to point to a folder which changes as needed.
I see there are many Github pages for gradle kubernetes plugin like
https://github.com/bmuschko/gradle-kubernetes-plugin
https://github.com/kolleroot/gradle-kubernetes-plugin
https://github.com/qaware/gradle-cloud-deployer
None of these having any concrete example how to connect to kubernetes from gradle and create a new deployment and service I tried all about git link in gradle but no luck...
Since I also faced a lack of plugins that deal with Kubernetes I started working on a Gradle plugin to make deploying resources to a Kubernetes cluster easier: https://github.com/kuberig-io/kuberig.
In the user manual you will find details about how to connect to a kubernetes cluster here:
https://kuberig-io.github.io/kuberig/#/initializing-an-environment
It also includes an example of how to define a deployment here: https://kuberig-io.github.io/kuberig/#/defining-a-deployment
And a service here:
https://kuberig-io.github.io/kuberig/#/defining-a-service
It may also be useful to go through the quickstart first https://kuberig-io.github.io/kuberig/#/quick-start.
Hope it can be of use to you.
I have a springboot 2 application, and I want to display metrics in AWS Cloudwatch.
I have included micrometer cloudwatch dependency in pom.
Here setting is documented for various metric systems but not cloudwatch.
Whar else configurations I need to do for cloudwatch?
First of all you may have to add some additional dependecies. I needed the following:
org.springframework.boot - spring-boot-starter-actuator
org.springframework.cloud - spring-cloud-starter-aws
io.micrometer - micrometer-core
io.micrometer - micrometer-registry-cloudwatch
Boot was not able to manage the versions for these dependencies except for actuator in my case, so you might have to figure out the right versions for you.
Firthermore some application properties have to be set:
# disable unwanted features to prevent autoconfigure which will produce errors and abort of application startup eventually
# alternatively you can try to configure those features correctly if you intend to use them
cloud.aws.stack.auto=false
# enable micrometer for cloudwatch (only where there is actually access to it)
management.metrics.export.cloudwatch.enabled=true
# set the namespace that will contain the metrics for this application
management.metrics.export.cloudwatch.namespace=test
# set max batch size to the actual maximum (if not a bug in certain versions of micrometer for cloudwatch will send
# batches that are too big)
management.metrics.export.cloudwatch.batchSize=20
The next step will be in AWS. The role associated with your EC2-instance (or whatever you are using) needs to have the permission CloudWatch:PutMetricData.
Using this configuration should enable CloudWatch-Monitoring for your Spring-Boot-Application.
One of the sources I encountered stated that you should use:
cloud.aws.credentials.instanceProfile=false
This will prevent Spring Boot from automatically obtaining credentials that are necessary to push metrics to CloudWatch. You could also provide own credentials another way, but I didn't try that.
It's well understood that in a microservices architecture, configuration must be externalized.
Tools like zookeeper, etcd or consul are excellent options to store that configuration. However a new layer on top of those services is required in order to provide new functionalities that are fundamental in a configuration server. Ex. versioning; change history; "draft" / published configuration, etc...
I've found spring config server, which is an interesting project and addresses all these concerns using git for handling the above mentioned requirements. However, I'd like avoid using git due to additional required setup. ex. replication, etc...
Do you know any other options other then spring config server?
Our findings have been the same - configuration must be externalized and the notion of "code as config" through the lessons learned from implementing time consuming Puppet/Chef systems.
We're building a microservices and API orchestration system at LunchBadger. We also uses git - but it's encapsulated into our system that we provide as a service, because we want configuration to be externalized AND dovetail into whatever CI/CD pipeline infrastructure you may have or are looking to adopt. We also provide visualization on top of microservices and APIs so that you can get an idea of the topology of your once monolithic app in the form of many microservices.
You can look into Microconfig.IO and it's Microconfig Server. It's deployed via docker and configured via 3 env vars. It still use git repo for config storage, but that's actually a proper way to go. As a bonus you get a powerful templating functionality designed explicitly to handle app configuration