Hystrix Dashboard metrics getting reset - microservices

I am very new to Hystrix and Turbine.
I have to read the metrics of all my services from Hystrix dashboard, for that i have added maven dependencies and #EnableHystrixDashboard on service class, able to read the metrics but the metrics are getting reset for every 10 sec which is useless for my client, i want to increase the timeout interval, for that i did configuration like this in my yml file
hystrix
metrics:
rollingStats:
timeInMilliseconds: 50000000
but it is not working. Could please help me here to fix the issue.
#EnableHystrixDashboard
#EnableAutoConfiguration(exclude = { JmxAutoConfiguration.class, DataSourceAutoConfiguration.class})
public class ProxyApplication
{
public static void main(String[] args)
{
ApplicationContext ctx = SpringApplication.run(ProxyApplication.class,args);
}
}
[Used Dependencies]

Related

Enforce Spring Boot Application to Fail When Cannot Connect to Kafka

I want to make my spring boot application fail on start if it cannot connect to the kafka broker. My application is only publishing messages to topics. I added this line to my properties file but no luck so far spring.kafka.admin.fail-fast=true.
Addition on how to speed up fail-fast
TL;DR Spring Boot 2.4.5 auto configuration does not let you speed up fail-fast using env parameters. Add this to your #Configuration to get 10 seconds timeout:
#Bean
public KafkaAdmin kafkaAdmin(#Autowired KafkaProperties properties) {
KafkaAdmin kafkaAdmin = new KafkaAdmin(properties.buildAdminProperties());
kafkaAdmin.setFatalIfBrokerNotAvailable(properties.getAdmin().isFailFast());
/* speed up fail fast */
kafkaAdmin.setOperationTimeout(5);
kafkaAdmin.setCloseTimeout(5);
return kafkaAdmin;
}
More detailed answer
Fail-fast occurs when initialize() method of class org.springframework.kafka.core.KafkaAdmin is executed. This method may block:
If new topics are found, it blocks on topic creation for up to operationTimeout
If topic creation failed, it blocks on releasing resources for up to closeTimeout
By default those values are 30 and 10 seconds accordingly (hardcoded in the class mentioned above). You can redefine them using set methods: setOperationTimeout(int sec), setCloseTimeout(int sec).
What about Spring Boot? KafkaAdmin bean is created in class org.springframework.boot.autoconfigure.kafka.KafkaAutoConfiguration. As you can guess, Spring Boot simply does not set timeouts:
#Bean
#ConditionalOnMissingBean
public KafkaAdmin kafkaAdmin() {
KafkaAdmin kafkaAdmin = new KafkaAdmin(this.properties.buildAdminProperties());
kafkaAdmin.setFatalIfBrokerNotAvailable(this.properties.getAdmin().isFailFast());
return kafkaAdmin;
}
fail-fast will only work if there is at least one NewTopic bean in the context (so the admin will try to check if the topic exists and create it if not).
#SpringBootApplication
public class So55177700Application {
public static void main(String[] args) {
SpringApplication.run(So55177700Application.class, args);
}
#Bean
public NewTopic topic() {
return new NewTopic("so55177700", 1, (short) 1);
}
}
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2019-03-15 09:42:49.555 ERROR 41793 --- [ main] o.s.boot.SpringApplication : Application run failed
java.lang.IllegalStateException: Could not configure topics

Report Metrics from Kafka to Actuator

I'm trying to get some metrics (client lag, ...) from kafka to provide it for consumption by prometheus.
My approach would be to write a simple springboot application which exposes the metrics for prometheus. I understand that kafka provides metrics to all its consumers via the interface MetricsReporter.
So I implemented a class which should do exactly that:
public class MonitoringIntegration implements MetricsReporter {
#Override
public void init(List<KafkaMetric> list) {
System.out.println("init");
for (KafkaMetric kafkaMetric : list) {
System.out.println(kafkaMetric.metricName());
System.out.println(kafkaMetric.metricValue());
}
}
#Override
public void metricChange(KafkaMetric kafkaMetric) {
System.out.println("Metric Change");
System.out.println(kafkaMetric.metricName());
System.out.println(kafkaMetric.metricValue());
}
#Override
public void metricRemoval(KafkaMetric kafkaMetric) {
System.out.println("Removal");
System.out.println(kafkaMetric.metricName());
System.out.println(kafkaMetric.metricValue());
}
#Override
public void close() {
System.out.println("close");
}
#Override
public void configure(Map<String, ?> map) {
System.out.println("Configuring");
System.out.println(map);
}
}
I registered this class with a bean:
#Configuration
public class MetricConfiguration {
#Bean
public ProducerFactory<?, ?> kafkaProducerFactory(KafkaProperties properties) {
Map<String, Object> producerProperties = properties.buildProducerProperties();
producerProperties.put(CommonClientConfigs.METRIC_REPORTER_CLASSES_CONFIG,
MonitoringIntegration.class.getName());
return new DefaultKafkaProducerFactory<>(producerProperties);
}
#Bean
public ConsumerFactory<?, ?> kafkaConsumerFactory(KafkaProperties properties) {
Map<String, Object> consumererProperties = properties.buildConsumerProperties();
consumererProperties.put(CommonClientConfigs.METRIC_REPORTER_CLASSES_CONFIG,
MonitoringIntegration.class.getName());
return new DefaultKafkaConsumerFactory<>(consumererProperties);
}
}
When I start the application some metrics will be printed out to cmd, but they have all default values (0.0, infinite, ..) and they will only be provided once after the application started.
Why am I not getting the metrics? What did I do wrong?
Cheers,
Fabian
Spring Kafka already exposes Kafka metrics as a JMX metrics. You dont need to update/send the metrics to Prometheus. Prometheus server will automatically read from your application's "/prometheus" endpoint. Enable Spring Actuator with Prometheus in your Spring project and configure the Prometheus server to read from it.
Here is a great example using Spring Boot - https://www.callicoder.com/spring-boot-actuator-metrics-monitoring-dashboard-prometheus-grafana/
MetricsReporter is not used to "report" metric values as they change. Check the docs. (For some reason I cant find the latest API).
https://archive.apache.org/dist/kafka/0.8.2-beta/java-doc/org/apache/kafka/common/metrics/MetricsReporter.html
A plugin interface to allow things to listen as new metrics are created so they can be reported.
metricChange() method will only be called when a metric is changed. This is the reason you see the first few outputs during application startup, because the metrics were created.
The consumer metrics support are only available on spring boot 2.1+ versions.
Auto-configuration Support For New Metrics
Metrics coverage has been improved to include:
Hibernate metrics
Spring Framework’s WebClient
Kafka consumer metrics
Log4j2 metrics
Jetty server thread pool metrics
Server-side Jersey HTTP request metrics
https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-2.1-Release-Notes#auto-configuration-support-for-new-metrics
I recommend you to upgrade to newer versions. But if you really need to use Spring Boot prior versions, you can check my kafka metrics micrometer implementation at:
https://github.com/luiztoscano/spring-boot-kmetrics

Testing Hystrix fallback through Feign API: com.netflix.client.ClientException: Load balancer does not have available server for client

When testing the Hystrix fallback behavior of my Feign API, I get an error, when I expect it to succeed.
Feign interface:
This is the api to the external service.
#FeignClient(name = "book", fallback = BookAPI.BookAPIFallback.class)
public interface BookAPI {
#RequestMapping("/")
Map<String, String> getBook();
#Component
class BookAPIFallback implements BookAPI {
#Override
#RequestMapping("/")
public Map<String, String> getBook() {
Map<String, String> fallbackmap = new HashMap<>();
fallbackmap.put("book", "fallback book");
return fallbackmap;
}
}
}
Test class
This test exists just to verify fallback behavior:
#RunWith(SpringRunner.class)
#SpringBootTest(webEnvironment = NONE)
public class BookServiceClientTest {
#MockBean
RestTemplate restTemplate;// <---- #LoadBalanced bean
#Autowired
private BookServiceClient bookServiceClient;
#Before
public void setup() {
when(restTemplate.getForObject(anyString(), any()))
.thenThrow(new RuntimeException("created a mock failure"));
}
#Test
public void fallbackTest() {
assertThat(bookServiceClient.getBook())
.isEqualTo(new BookAPI.BookAPIFallback().getBook().get("book")); // <--- I thought this should work
}
}
config files
application.yml
These files show configuration that might be relevant:
feign:
hystrix:
enabled: true
test/application.yml
eureka:
client:
enabled: false
The Question
Everything works fine when running the apps.
But when running this test, I get the below error.
Naturally, it's a test, so I'm trying to bypass the lookup anyway.
java.lang.RuntimeException: com.netflix.client.ClientException: Load balancer does not have available server for client: book
at org.springframework.cloud.netflix.feign.ribbon.LoadBalancerFeignClient.execute(LoadBalancerFeignClient.java:71)
at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:97)
What am I missing?
Addendums
Application class
#SpringBootApplication
#EnableCircuitBreaker
#EnableDiscoveryClient
#EnableFeignClients
public class LibraryApplication {
public static void main(String[] args) {
SpringApplication.run(LibraryApplication.class, args);
}
}
LibraryController
#Controller
public class LibraryController {
private final BookServiceClient bookService;
public LibraryController(BookServiceClient bookServiceClient) {
this.bookService = bookServiceClient;
}
#GetMapping("/")
String getLibrary(Model model) {
model.addAttribute("msg", "Welcome to the Library");
model.addAttribute("book", bookService.getBook());
return "library";
}
}
There are no other classes.
so! I was able to recreate the issue, thanks for adding more code, had to play about with it a tad as I was unsure what the BookClientService looked like and it wouldn't make sense for it to implement the BookAPI as that would be an internal call e.g. in your application and not an external API call with Feign.
Anyway,
I pushed my version of what you provided here.
https://github.com/Flaw101/feign-testing
The issue was resolved when I renamed the second application.yml which lives in the src/test/resources folder to application-test.yml which will merge the properties.
The issue was caused by the fact the second property source, the testing one, overrides the initial application.yml and disables hystrix, because Hystrix is disabled there is no fallback to go to and it throws the root cause of what would cause the fallback, a lack of a server to call to for the Book API. Renaming it to application-test will always be loaded into spring test contexts. You could resolve it with the use of inlined properties or profiles.
I've added another test disabling feign /w hystrix within the test which re-creates the error you are recieving.

How to prevent Spring Boot daemon/server application from closing/shutting down immediately?

My Spring Boot application is not a web server, but it's a server using custom protocol (using Camel in this case).
But Spring Boot immediately stops (gracefully) after started. How do I prevent this?
I'd like the app to stop if Ctrl+C or programmatically.
#CompileStatic
#Configuration
class CamelConfig {
#Bean
CamelContextFactoryBean camelContext() {
final camelContextFactory = new CamelContextFactoryBean()
camelContextFactory.id = 'camelContext'
camelContextFactory
}
}
I found the solution, using org.springframework.boot.CommandLineRunner + Thread.currentThread().join(), e.g.:
(note: code below is in Groovy, not Java)
package id.ac.itb.lumen.social
import org.slf4j.LoggerFactory
import org.springframework.boot.CommandLineRunner
import org.springframework.boot.SpringApplication
import org.springframework.boot.autoconfigure.SpringBootApplication
#SpringBootApplication
class LumenSocialApplication implements CommandLineRunner {
private static final log = LoggerFactory.getLogger(LumenSocialApplication.class)
static void main(String[] args) {
SpringApplication.run LumenSocialApplication, args
}
#Override
void run(String... args) throws Exception {
log.info('Joining thread, you can press Ctrl+C to shutdown application')
Thread.currentThread().join()
}
}
As of Apache Camel 2.17 there is a cleaner answer. To quote http://camel.apache.org/spring-boot.html:
To keep the main thread blocked so that Camel stays up, either include the spring-boot-starter-web dependency, or add camel.springboot.main-run-controller=true to your application.properties or application.yml file.
You will want the following dependency too:
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-spring-boot-starter</artifactId>
<version>2.17.0</version>
</dependency>
Clearly replace <version>2.17.0</version> or use the camel BOM to import dependency-management information for consistency.
An example implementation using a CountDownLatch:
#Bean
public CountDownLatch closeLatch() {
return new CountDownLatch(1);
}
public static void main(String... args) throws InterruptedException {
ApplicationContext ctx = SpringApplication.run(MyApp.class, args);
final CountDownLatch closeLatch = ctx.getBean(CountDownLatch.class);
Runtime.getRuntime().addShutdownHook(new Thread() {
#Override
public void run() {
closeLatch.countDown();
}
});
closeLatch.await();
}
Now to stop your application, you can look up the process ID and issue a kill command from the console:
kill <PID>
Spring Boot leaves the task of running the application to the protocol around which the application is implemented. See, for example, this guide:
Also required are some housekeeping objects like a CountDownLatch to keep the main thread alive...
So the way of running a Camel service, for example, would to be to run Camel as a standalone application from your main Spring Boot application class.
This is now made even simpler.
Just add camel.springboot.main-run-controller=true to your application.properties
All threads are completed, the program will close automatically.
So, register an empty task with #Scheduled will create a loop thread to prevent shutdown.
file application.yml
spring:
main:
web-application-type: none
file DemoApplication.java
#SpringBootApplication
#EnableScheduling
public class DemoApplication {
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
}
file KeepAlive.java
#Component
public class KeepAlive {
private static final Logger log = LoggerFactory.getLogger(ScheduledTasks.class);
private static final SimpleDateFormat dateFormat = new SimpleDateFormat("HH:mm:ss");
#Scheduled(fixedRate = 1 * 1000 * 60) // 1 minute
public void reportCurrentTime() {
log.info("Keepalive at time {}", dateFormat.format(new Date()));
}
}
My project is NON WEB Spirng Boot.
My elegant solution is create a daemon thread by CommandLineRunner.
Then, Application do not shutdown immediately.
#Bean
public CommandLineRunner deQueue() {
return args -> {
Thread daemonThread;
consumer.connect(3);
daemonThread = new Thread(() -> {
try {
consumer.work();
} catch (InterruptedException e) {
logger.info("daemon thread is interrupted", e);
}
});
daemonThread.setDaemon(true);
daemonThread.start();
};
}
To keep the java process alive when not deploying a web application set the webEnvironment property to false like so:
SpringApplication sa = new SpringApplication();
sa.setWebEnvironment(false); //important
ApplicationContext ctx = sa.run(ApplicationMain.class, args);
for springboot app to run continously it has to be run in a container, otherwise it is just like any java app all threads are done it finishes,
you can add
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
and it will turn it into webapp, if not you are responsible keeping it alive in your implementation

Spring cloud aws (messaging) - not getting message

I am attempting to get spring cloud to work with messaging using auto configure.
My properties file contains:
cloud.aws.credentials.accessKey=xxxxxxxxxx
cloud.aws.credentials.secretKey=xxxxxxxxxx
cloud.aws.region.static=us-west-2
My Configuration class is as follows:
#ComponentScan
#EnableAutoConfiguration
public class Application {
public static void main(String[] args) throws Exception {
SpringApplication.run(Application.class, args);
}
}
My Listener class:
#RestController
public class OrderListener {
#MessageMapping("orderQueue")
public void orderListener(Order order){
System.out.println("Order Name " + order.getName());
System.out.println("Order Url" + order.getUrl());
}
}
However, nothing seems to print out. I've verified that my queue is in the right region and there is a message on the queue thats ready to be received.
Can someone please provide some guidance?
I see three possible reasons why it is not working
The OrderListener class is not scanned by the component scanner. In order to be scanned this class must be in the same package as your Application class or in a sub-package.
The spring-cloud-aws-autoconfigure artifact is missing on you classpath and therefore the AmazonSQS client is not automatically configured and the queues are not registered.
The message you have in your queue is missing a message header contentType with value application/json (for more information about this see question Spring Cloud - SQS).

Resources