To consume messages from Kafka, I am using the below configuration in my local. It is working fine.
spring.kafka.bootstrap-servers: localhost:9092
spring.kafka.consumer.key-deserializer: org.apache.kafka.common.serialization.IntegerDeserializer
spring.kafka.consumer.value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.group-id: order-events
spring.kafka.admin.properties.bootstrap.servers: localhost:9092
In DEV, SIT Kubernates cluster, we are using Amazon MSK. The below details are shared.
Host: z3.kafka-central-1.amazonaws.com:9096,z1.kafka-central-1.amazonaws.com:9096,z2.kafka-central-1.amazonaws.com:9096
Configuration: 3 Partitions, 3 Replicas, 3 Brokers, 3 Different AZs, SASL/SCRAM authentication, retention.ms=604800000, max.message.bytes=2097164
VPC Id: vpc-123sdfsdf234
AWS Account 123456789
CIDR 10.20.1.1/24
Username user-msk-kafka-user
Password XXXXXXXX
What are the properties, i should configure in spring application properties file.
Your MSK VPC CIDR 10.20.1.1/24 is private.
Make sure your Dev/SIT the environment you are using to connect to MSK is in same VPC vpc-123sdfsdf234. If it's inside same VPC, check MSK security group to make sure your CIDR range 10.20.1.1/24 is whitelisted with the required ports.
Otherwise create MSK with Public IP's.
Related
I'm working on a project for a large company with millions of users. We are attempting to convert their REST based architecture to an event based architecture. The current architecture involves a service, we'll call it Service-A, that makes 7 REST calls when a user logs in.
Rather than calling out to the 7 services for that data when the user logs in we want to modify those 7 services to produce events when there are updates to the data. Then we will have Service-A listen to 7 different kafka topics and save those updates to the database.
It is a Java Spring Boot application. We are using AWS MSK to host our kafka cluster and we are using AWS Glue for the schema registry. I can configure my consumer in Service-A to listen to 7 topics but I don't know how to get Service-A to check 7 different schemas when consuming a message from one of those 7 topics.
So far, the only configuration I've found for the kafka consumer is one property that takes one schema name.
Here is my config yaml:
spring:
kafka:
listener:
ack-mode: manual_immediate
consumer:
enable-auto-commit: false
group-id: my-group
key-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
properties:
spring.json.trusted.packages: com.app.somepackage.domain
spring.deserializer.key.delegate.class: org.apache.kafka.common.serialization.StringDeserializer
spring.deserializer.value.delegate.class: com.amazonaws.services.schemaregistry.deserializers.avro.AWSKafkaAvroDeserializer
auto-offset-reset: earliest
bootstrap-servers: <my-msk-url>
properties:
region: us-west-2
schemaName: my-first-schema
registry.name: my-registry-name
avroRecordType: SPECIFIC_RECORD
I am trying to deploy Spring Boot microservices using Docker using Appmesh and EC2. I have deployed two sample microservices (https://github.com/amitgct/appmesh-hello) namely: caller-service and called-service using docker on a single EC2 instance and configured appmesh accordingly by following guide https://docs.aws.amazon.com/app-mesh/latest/userguide/getting-started-ec2.html. Currently, my applications are running on ec2 but they cannot communicate with each other and getting error on calling called-service from caller-service i.e. Unknown host. Can anyone tell me how can I specify hostname and register service with that host on EC2 and App mesh. (Note: I don't want to use kubernetes, ECS, AWS cloud map, AWS Route53) . If can provide example also then very thankful to you. Please help.
https://www.appmeshworkshop.com/servicediscovery/
here's a step by step process shown, and this is for http protocol...
but if you change the listeners section in virtual routes to tcp then it should work for TCP messages as well - for those systems which works on tcp protocol - example Akka Clusters
I am trying to register a spring boot app to a consul cluster.
I have 3 node consul cluster 1 master 2 agents.
I have a load-balancer in front of 2 consul agents, so that it is HA.
In my application.yml. I ask the services to join via load-balancer
spring
cloud:
consul:
enabled: true
port: loadbalancer_port
host: http://loadbalancer
discovery:
instance-id: ${info.app.environment}:${spring.application.name}
tags:
- ${spring.profiles.active}
Now, when my service restarts it is creating a duplicate entry in consul.
I figured that, because it is being registered in 2 different agents.
Does this mean, I cant have HA consul with loadbalancer ? or should I ask the services to register to particular agents with out load-balancer?
Please help!!
Consul is designed to have a Consul client agent deployed on each server in your data center (see Consul Reference Architecture). Instead of registering services centrally, services running on a machine are registered with the local/co-located Consul agent. The agents then submit the list of services registered against them to the Consul servers, which then aggregates this info from each agent to form the service catalog. The catalog maintains the high-level view of the cluster, including which services are available, which nodes run those services, health information, etc.
TLDR; Remove the load balancer and register the services directly with the agents in order to avoid this issue where service registrations are duplicated across hosts.
How can we have two AWS kinesis connections using spring-cloud-stream-binder-kinesis?
1st connection: spring application and AWS kinesis stream in the same AWS account.
2nd connection: other AWS kinesis stream sitting in a different AWS account.
Is it possible to have two different connections from a spring application to two different kinesis streams in different AWS accounts?
If it is yes, How do we implement this?
See Connecting to Multiple Systems.
By default, binders share the application’s Spring Boot auto-configuration, so that one instance of each binder found on the classpath is created. If your application should connect to more than one broker of the same type, you can specify multiple binder configurations, each with different environment settings.
I’m having a bit of trouble getting Turbine to work in Spring Cloud. In a nutshell, I can’t determine how to configure it to aggregate circuits from more than one application at a time.
I have 6 separate services, a eureka server, and a turbine server running in standalone mode. I can see from my Eureka server that all of the services are registered, including turbine. My turbine server is up and running, and I can see its /hystrix page without issue. But when I try to use it to examine turbine.stream, I only see the FIRST server that is listed in turbine.appConfig, the rest are ignored.
This is my Turbine server’s application.yml, or at least the relevant parts:
---
eureka:
client:
serviceUrl:
defaultZone: http://localhost:8010/eureka/
server:
port: 8030
info:
component: Turbine
turbine:
clusterNameExpression: new String(“default”)
appConfig: sentence,subject,verb,article,adjective,noun
management:
port: 8990
When I run this and access the hystrix dashboard on my turbine instance, asking for the turbine.stream, the ONLY circuit breakers listed in the output are for the first service listed in appConfig, the “sentence” service in this case. Curiously, if I re-arrange the order of these services and put another one first (like “noun”), I see only the circuits for THAT service. Only the first service in the list is displayed.
I’ll admit to being a little confused on some of the terminology, like streams, clusters, etc., so I could be missing some basic concept here, but my understanding is that Turbine could digest streams from more than one service and aggregate them in a single display. Suggestions would be appreciated.
I don't have enough reputation to comment, so I have to write this in an answer :)
I had the exactly same problem:
There are two services "test-service" and "other-service", each with it's own working hystrix-stream
and there is one Turbine-Application, which is configured like this:
turbine:
clusterNameExpression: new String("default")
appConfig: test-service,other-service
All of my services are running on my local machine.
Result is: My Hystrix-Dashboard just shows the metrics from "test-service".
Reason:
It seems to be, that a Turbine-Client which is configured the described way doesn't handle multiple services when they are running at the same host.
This is explained here:
https://github.com/Netflix/Hystrix/issues/117#issuecomment-14262713
Turbine maintains state of all these instances in order to maintain persistent connections to them and it does rely on the "hostname" and if the host name is the same then it won't instantiate a new connection to that same server (on a different port).
So the main point is, that all of your services must be registered with different hostnames. How you could do this on your local machine is described below.
UPDATE 2015-06-12/2016-01-23: Workaround for local testing
Change your hostfile:
# ...
127.0.0.1 localhost
127.0.0.1 localdomain1
127.0.0.1 localdomain2
# ...
127.0.0.1 localdomainx
And then set the hostname for your clients each to a different domain-entry like this:
application.yml:
eureka:
instance:
hostname: localdomainx