multi output binding in spring cloud data flow - spring

I'm trying to setup a multi-destination bindings but for some reason, messages from the second channel are going to the first exchange.queue. For example:
spring:
cloud:
stream:
bindings:
output:
destination: exchange1
producer.requiredGroups: queue1
output-other:
destination: exchange2
producer.requiredGroups: queue2
I also used org.springframework.cloud.stream.messaging.Source for the default Output and created a dedicated Source Binding for output-other channel
public interface OtherSource {
String OUTPUT = "output-other";
#Output(OtherSource.OUTPUT)
MessageChannel output();
}
and the producer class
#EnableBinding(Source.class)
public class OutputSender {
private final Source source;
public void send(Output1 obj) {
Message message = toMessage(obj);
this.source.output().send(message);
}
}
this works as expected. messages are sent to the correct queue (exchange1.queue1)
second producer:
#EnableBinding(OtherSource.class)
public class OutputOtherSender {
OtherSource source;
public void send(Output2 obj) {
Message message = toMessage(obj)
this.source.output().send(obj);
}
}
2 issues with this setup:
exchange2.queue2 is not created (something wrong with application.yml configuration?)
messages that are sent using OtherSource are going to exchange1.queue1
Dependencies
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.1.6.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream</artifactId>
<version>2.2.0.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-rabbit</artifactId>
<version>2.2.0.RELEASE</version>
</dependency>

By default, the stream applications in Spring Cloud Data Flow are linear, meaning the applications are bound to one another using single output->single input.
If you want to have your stream using multiple input/output destinations, then you have to manually configure your bindings(using the Spring Cloud Stream properties) and define your application as app type - a special type for a streaming app in SCDF that lets the user configure the bindings manually.
For more information on this, you can refer: https://dataflow.spring.io/docs/feature-guides/streams/stream-application-dsl/

Related

Spring Boot, Sleuth, OTEL, and Honeycomb

I have a scenario where I have Spring Boot integrated with OTEL and shipping to Honeycomb.io. I am trying to add an environment tag to each trace. I have created a class:
#Component
public class EnvironmentSpanProcessor implements SpanProcessor {
#Value("${ENVIRONMENT")
private String environment;
Queue<SpanData> spans = new LinkedBlockingQueue<>(50);
#Override
public void onStart(Context context, ReadWriteSpan readWriteSpan) {
readWriteSpan.setAttribute("env", environment);
}
#Override
public boolean isStartRequired() {
return false;
}
#Override
public void onEnd(ReadableSpan readableSpan) {
this.spans.add(readableSpan.toSpanData());
}
#Override
public boolean isEndRequired() {
return true;
}
}
I have set break points in this class, and they never hit on startup, even though the bean can be seen in actuator. I have put breakpoints on:
SdkTracerProvider otelTracerProvider(SpanLimits spanLimits, ObjectProvider<List<SpanProcessor>> spanProcessors,
SpanExporterCustomizer spanExporterCustomizer, ObjectProvider<List<SpanExporter>> spanExporters,
Sampler sampler, Resource resource, SpanProcessorProvider spanProcessorProvider) {
SdkTracerProviderBuilder sdkTracerProviderBuilder = SdkTracerProvider.builder().setResource(resource)
.setSampler(sampler).setSpanLimits(spanLimits);
List<SpanProcessor> processors = spanProcessors.getIfAvailable(ArrayList::new);
processors.addAll(spanExporters.getIfAvailable(ArrayList::new).stream()
.map(e -> spanProcessorProvider.toSpanProcessor(spanExporterCustomizer.customize(e)))
.collect(Collectors.toList()));
processors.forEach(sdkTracerProviderBuilder::addSpanProcessor);
return sdkTracerProviderBuilder.build();
}
in OtelAutoConfiguration and am not seeing them firing either on startup.
My pom.xml relevant section is:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-sleuth</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-sleuth-brave</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-sleuth-otel-autoconfigure</artifactId>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-extension-trace-propagators</artifactId>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-netty-shaded</artifactId>
<version>1.47.0</version>
</dependency>
And my configuration from application.yaml
sleuth:
enabled: true
web:
additional-skip-pattern: /readiness|/liveness
client.skip-pattern: /readiness
sampler:
probability: 1.0
rate: 100
propagation:
type: OT_TRACER
otel:
config:
trace-id-ratio-based: 1.0
log.exporter.enabled: true
exporter:
otlp:
endpoint: https://api.honeycomb.io
headers:
x-honeycomb-team: ${TELEMETRY_API_KEY}
x-honeycomb-dataset: app-telemetry
sleuth-span-filter:
enabled: true
resource:
enabled: true
I am getting traces, so it appears the system itself is working, however I cannot get my env tag added.
Preemptive thank you to #marcingrzejszczak for the help so far on my gist: https://gist.github.com/fpmoles/b880ccfdef2d2138169ed398e87ec396
I'm unsure why your span processor is not being picked up by Spring and being added to your list of processors being registered with the tracer provider.
An alternative way to set process consistent values, like environment, would be to set it as a resource attribute. This is more desireable because it's set once and delivered once per batch of spans sent to the configured backend (eg Honeycomb). Using a span processor adds the same attribute to every span.
This can be done in a few different ways:
If using AutoConfigure, you can set via system property or environment variable
Set directly on the resource during your otelTracerProvider method:
resource.setAttribute("environment", "${environment}");
FYI Honeycomb has OTel Java SDK & Agent distros to help simplify sending data that reduces required configuration and sets sensible defaults.

Resilience4j Circuit Breaker is not working

I am facing a issue with the circuit breaker implementation using Spring Cloud Resilience4j.
Following some tutorial, I have tried to add the necessary dependencies in the project.
Also, tried to add the configurations but, still the circuit is not opening and fallback method is not getting called.
For the use case, I am calling an external API from my service and if that external API is down then after few calls I need to enable the circuit breaker.
Please find the code pieces from the different files.
I am a newbie to circuit breaker pattern. Any help will be highly appreciated.
pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.5.5</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<properties>
<java.version>11</java.version>
<spring-cloud.version>2020.0.4</spring-cloud.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-circuitbreaker-resilience4j</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-aop</artifactId>
</dependency>
<dependencies>
</project>
Application properties
resilience4j.circuitbreaker.instances.test-api.register-health-indicator=true
resilience4j.circuitbreaker.instances.test-api.minimum-number-of-calls=4
resilience4j.circuitbreaker.instances.test-api.failure-rate-threshold=50
resilience4j.circuitbreaker.instances.test-api.permitted-number-of-calls-in-half-open-state=3
resilience4j.circuitbreaker.instances.test-api.wait-duration-in-open-state=30s
resilience4j.circuitbreaker.instances.test-api.automatic-transition-from-open-to-half-open-enabled=true
resilience4j.circuitbreaker.instances.test-api.record-exceptions=com.testapi.exception.ServiceUnavailableError
Service Class Code Piece
#CircuitBreaker(name = "test-api", fallbackMethod = "storeResponseFallback")
public TestResponse storeResponse(String apiURL, HttpEntity<String> entityrequest) {
TestResponse testResponse = new TestResponse();
Optional<ResponseEntity<TestResponse>> response = Optional.empty();
Future<ResponseEntity<TestResponse>> responseFuture;
ExecutorService executor = Executors.newFixedThreadPool(10);
log.debug("Calling Extrenal API, Request Body: {}", entityrequest.toString());
try {
//Service call returns a future
responseFuture = executor.submit(() -> restTemplate.postForEntity(apiURL, entityrequest, TestResponse.class));
response = Optional.ofNullable(responseFuture.get());
log.info("Got response from external API");
if ((response.isPresent()) && (response.get().hasBody())) {
testResponse = response.get().getBody();
}
} catch (Exception exception) {
log.error("External api call got failed with an error");
Thread.currentThread().interrupt();
throw new ServiceUnavailableError();
}
return testResponse;
}
public TestResponse storeResponseFallback(ServiceUnavailableError ex) {
log.error("Executing Fallback Method For General exceptions");
throw new ServiceUnavailableError();
}
ServiceUnavailableError Java file
#Data
#AllArgsConstructor
#NoArgsConstructor
public class ServiceUnavailableError extends RuntimeException{
private static final long serialVersionUID = 2382122402994502766L;
private String message;
}
The signature of your fallback method is wrong. It should contain all the parameters of the actual method ( in your case storeResponseFallback is the fallback method and storeResponse is the actual method), along with the exception. Please make sure to remove the try catch block. You do not want to handle the exception yourself, rather you should let circuit breaker to handle it for you.
Please take a look at the following code which is from given link
https://resilience4j.readme.io/docs/getting-started-3
#CircuitBreaker(name = BACKEND, fallbackMethod = "fallback")
public Mono<String> method(String param1) {
return Mono.error(new NumberFormatException());
}
private Mono<String> fallback(String param1, IllegalArgumentException e) {
return Mono.just("test");
}
Try using the following yaml file
I used the following configuration with your existing code,I used yaml instead of properties file. this seems to stay in open state and call only the fallback method.
resilience4j.circuitbreaker:
configs:
default:
slidingWindowSize: 4
permittedNumberOfCallsInHalfOpenState: 10
waitDurationInOpenState: 10000
failureRateThreshold: 60
eventConsumerBufferSize: 10
registerHealthIndicator: true
someShared:
slidingWindowSize: 3
permittedNumberOfCallsInHalfOpenState: 10
instances:
test-api:
baseConfig: default
waitDurationInOpenState: 500000
backendB:
baseConfig: someShared
Here is the updated fallback method
public TestResponse storeResponseFallback(String apiURL, String entityrequest, java.lang.Throwable t) {
log.error("Executing Fallback Method For General exceptions "+t.getMessage());
return new TestResponse("Frm Fallback");// Making sure to send a blank response
}
I have prepared the video, where I have defined main service and target service and I am preparing the bean from config and making use of Try.of() please check the video if it help.
you will see that the fallback method is working fine.
https://www.youtube.com/watch?v=8yJ0xek6l6Y&t=31s

Micrometer StackdriverMeterRegistry only publishes custom metrics to GCP Monitoring, not automatic instrumentation metrics?

I have this example https://quarkus.io/guides/micrometer (micrometer quickstart directory) running which uses Quarkus and Micrometer together. The example uses Prometheus as the MeterRegistry but I changed it to use the StackdriverMeterRegistry in hopes the same auto instrumentation that shows up in Prometheus would show up in Google Cloud Monitoring.
However, I only see the custom metrics I made appear into Google Cloud Monitoring, and not the auto instrumentation provided by micrometer.
I am unsure if I should think that this is just an issue with the Micrometer StackdriverMeterRegistry library itself or if I am doing something wrong. Any guidance is appreciated.
Code changes:
// Update the constructor to create the gauge
ExampleResource(MeterRegistry registry) {
/* Code for micrometer */
StackdriverConfig stackdriverConfig = new StackdriverConfig() {
#Override
public String projectId() {
return "projectId";
}
#Override
public String get(String key) {
return null;
}
};
this.registry = StackdriverMeterRegistry.builder(stackdriverConfig).build();
registry.config().commonTags("application", "projectId");
registry.gaugeCollectionSize("example.list.size", Tags.empty(), list);
}
Added to pom.xml
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-stackdriver</artifactId>
</dependency>
After tinkering and speaking directly with the Micrometer team I found out the issue. The documentation is a bit confusing but I had imported the StackDriver extension wrong and the default registry being used for the quarkus project was getting all the auto instrumentation but not the StackDriver one. So this default registry needed to be changed to the StackDriver one.
I have uploaded a basic example of using Quarkus StackDriver and Micrometer together using the basic example found on the Micrometer Quarkus documentation page.
https://github.com/jayleenli/quarkus-micrometer-stackdriver-quickstart
The changes:
Add to pom.xml
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-micrometer</artifactId>
</dependency>
<dependency>
<groupId>io.quarkiverse.micrometer.registry</groupId>
<artifactId>quarkus-micrometer-registry-stackdriver</artifactId>
<version>2.2.2</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-core</artifactId>
<version>1.7.3</version>
</dependency>
Then add some Quarkus properties, I used application.properties but there are other ways you can do this.
application.properties
quarkus.micrometer.export.stackdriver.enabled=true
quarkus.micrometer.export.stackdriver.default-registry=true
quarkus.micrometer.export.stackdriver.project-id=fake-id
quarkus.micrometer.export.stackdriver.publish=true
quarkus.micrometer.export.stackdriver.resource-type=global
quarkus.micrometer.export.stackdriver.step=1m
In main class
#Path("/")
public class ExampleResource {
#ConfigProperty(name = "quarkus.micrometer.export.stackdriver.enabled")
boolean enabled;
#ConfigProperty(name = "quarkus.micrometer.export.stackdriver.default-registry")
boolean export;
#ConfigProperty(name="quarkus.micrometer.export.stackdriver.project-id")
String projectId;
#ConfigProperty(name="quarkus.micrometer.export.stackdriver.publish")
boolean publish;
#ConfigProperty(name="quarkus.micrometer.export.stackdriver.resource-type")
String resourceType;
#ConfigProperty(name="quarkus.micrometer.export.stackdriver.step")
String step;

Configure HTTPS in Spring Boot Apache Camel REST API with keystore having multiple certs using camel-jetty component

I am trying to configure https in my apache camel Spring Boot REST application (using apache-camel v3.11.1, springboot v2.5.3) with keystore having multiple certificates.
Problem:
Application run failed
org.apache.camel.RuntimeCamelException: java.lang.IllegalStateException: KeyStores with multiple certificates are not supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory. (Use org.eclipse.jetty.util.ssl.SslContextFactory$Server or org.eclipse.jetty.util.ssl.SslContextFactory$Client instead)
at org.apache.camel.RuntimeCamelException.wrapRuntimeCamelException(RuntimeCamelException.java:51) ~[camel-api-3.11.1.jar:3.11.1]
Project setup:
pom.xml: (dependencies only, to show that I am not using spring-boot-web-starter)
..
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.springboot</groupId>
<artifactId>camel-spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.springboot</groupId>
<artifactId>camel-jetty-starter</artifactId>
</dependency>
..
..<!-- all other required dependencies are in place-->
..
</dependencies>
..
application.properties
#camel.component.jetty.keystore=keystore-with-one-certificate.jks # WORKS
camel.component.jetty.keystore=keystore-with-multiple-certificates.jks # DOESN'T WORK
camel.component.jetty.ssl-key-password=password
camel.component.jetty.ssl-password=password
Rest Route:
restConfiguration()
.component("jetty")
.scheme("https")
.port("8080");
rest()
.path("/api")
.get("/{name}")
..
..
.to("direct:x");
Looked at answers in the below posts, but still not able to resolve the exception that I get,
https://stackoverflow.com/a/60598953/6363894,
https://stackoverflow.com/a/55499113/6363894
I know that exception clearly states to use org.eclipse.jetty.util.ssl.SslContextFactory$Server, but I don't understand how/where to use SslContextFactory.Server object.
SslContextFactory.Server sslContextFactory = new SslContextFactory.Server();
sslContextFactory.setKeyStoreResource(findKeyStorePath());
sslContextFactory.setKeyStorePassword("password");
sslContextFactory.setKeyManagerPassword("password");
sslContextFactory.setNeedClientAuth(true);
Also I've created a bean for sslContextParameters and added that to restConfiguration as below, this time application runs successfully but then when I test, SSL handshake fails.
restConfiguration()
.component("jetty")
.endpointProperty("sslContextParameters", "#sslContextParameters")
.scheme("https")
.port("8080");
#Bean(name = "sslContextParameters")
public SSLContextParameters setSSLContextParameters() {
KeyStoreParameters ksp = new KeyStoreParameters();
ksp.setResource("keystore-with-multiple-certificates.jks");
ksp.setPassword("password");
KeyManagersParameters kmp = new KeyManagersParameters();
kmp.setKeyStore(ksp);
kmp.setKeyPassword("password");
SSLContextServerParameters scsp = new SSLContextServerParameters();
scsp.setClientAuthentication("REQUIRE");
SSLContextParameters scp = new SSLContextParameters();
scp.setServerParameters(scsp);
scp.setKeyManagers(kmp);
return scp;
}
Any help on how to configure SslContextFactory.Server object with the restConfigurations() or any other way I can achieve this? I'll update the post, if any more details are required.

spring boot elastic search -configure data source

I am tryinging to configure spring data boot sand ES project
in my pom.xml i have :
#Configuration
#EnableElasticsearchRepositories(basePackages = "com.yoyo.elastic.repository")
public class ElasticConfiguration {
#Bean
public NodeBuilder nodeBuilder() {
return new NodeBuilder();
}
#Bean
public ElasticsearchOperations elasticsearchTemplate() throws IOException {
File tmpDir = File.createTempFile("elastic", Long.toString(System.nanoTime()));
System.out.println("Temp directory: " + tmpDir.getAbsolutePath());
final Client client = nodeBuilder().local(true).node().client();
return new ElasticsearchTemplate(client);
}
}
in my pom xml I have this dep :
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-elasticsearch</artifactId>
</dependency>
which should supplay the driver but i keep on getting :
Description:
Cannot determine embedded database driver class for database type NONE
Action:
If you want an embedded database please put a supported one on the classpath. If you have database settings to be loaded from a particular profile you may need to active it (no profiles are currently active).
I had the same issue when trying to run some exercises with Spring Boot and ElasticSearch.
Right now I figured out that if you have the
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
Alongside spring-boot-starter-data-elasticsearch and don't add additional config classes (where you would configure the DataSource) spring boot will complain.
Other solution would be to actually add a datasource property to application.properties and configure standalone database (like H2)

Resources