How to update configurations using spring cloud config server and spring cloud bus with rabbit mq? - spring

I could not find any up to date and / or working documentation and / or example on how to set this up so what I did is researching and try and error. Yet I have not been able to get it running.
Im having a config-server with the following dependencies:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-config-server</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-bus-amqp</artifactId>
<version>2.2.1.RELEASE</version>
</dependency>
The config-server has the following bootstrap.yml:
spring:
application:
name: config-server
rabbitmq:
host: ${RABBIT_MQ_HOST:localhost}
port: ${RABBIT_MQ_PORT:5672}
username: ${RABBIT_MQ_URSER_NAME:guest}
password: ${RABBIT_MQ_URSER_PASSWORD:guest}
And the following application.yml
spring:
cloud:
config:
server:
git:
uri: https://github.com/MyGit/config-repository.git
cloneOnStart: true
management:
endpoints:
web:
exposure:
include: "*"
server:
port: 8888
All my clients are having the following dependencies:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-bus-amqp</artifactId>
<version>2.2.1.RELEASE</version>
</dependency>
Their bootstrap.yml for example looks like this:
spring:
application:
name: user-service
cloud:
config:
uri: http://${CONFIG_HOST:localhost}:${CONFIG_PORT:8888}
rabbitmq:
host: ${RABBIT_MQ_HOST:localhost}
port: ${RABBIT_MQ_PORT:5672}
username: ${RABBIT_MQ_URSER_NAME:guest}
password: ${RABBIT_MQ_URSER_PASSWORD:guest}
And their application.yml like this:
management:
endpoints:
web:
exposure:
include: "*"
server:
port: 8000
eureka:
client:
registerWithEureka: true
fetchRegistry: true
serviceUrl:
defaultZone: http://${EUREKA_HOST:localhost}:${EUREKA_PORT:8761}/eureka
Im using springs #ConfigurationProperties, for example:
#Component
#ConfigurationProperties("user")
#Getter
#Setter
public class UserConfiguration {
private String role;
}
I'm starting all my services (including rabbitmq) using docker and docker-compose:
version: "3"
services:
rabbitmq:
image: bitnami/rabbitmq:latest
container_name: rabbitmq
environment:
RABBITMQ_USERNAME: admin
RABBITMQ_PASSWORD: admin
ports:
- "4369:4369"
- "5672:5672"
- "15672:15672"
- "25672:25672"
config-server:
build:
context: config-server
dockerfile: Dockerfile
container_name: config-server
depends_on:
- rabbitmq
environment:
RABBIT_MQ_HOST: "rabbitmq"
RABBIT_MQ_PORT: "5672"
RABBIT_MQ_URSER_NAME: "admin"
RABBIT_MQ_URSER_PASSWORD: "admin"
ports:
- "8888:8888"
eureka-server:
build:
context: eureka-server
dockerfile: Dockerfile
container_name: eureka-server
depends_on:
- rabbitmq
- config-server
environment:
RABBIT_MQ_HOST: "rabbitmq"
RABBIT_MQ_PORT: "5672"
RABBIT_MQ_URSER_NAME: "admin"
RABBIT_MQ_URSER_PASSWORD: "admin"
CONFIG_HOST: "config-server"
CONFIG_PORT: "8888"
ports:
- "8761:8761"
zuul-gateway:
build:
context: zuul-gateway
dockerfile: Dockerfile
container_name: zuul-gateway
depends_on:
- rabbitmq
- config-server
- eureka-server
environment:
RABBIT_MQ_HOST: "rabbitmq"
RABBIT_MQ_PORT: "5672"
RABBIT_MQ_URSER_NAME: "admin"
RABBIT_MQ_URSER_PASSWORD: "admin"
CONFIG_HOST: "config-server"
CONFIG_PORT: "8888"
EUREKA_HOST: "eureka-server"
EUREKA_PORT: "8761"
ports:
- "8765:8765"
user-service:
build:
context: user-service
dockerfile: Dockerfile
container_name: user-service
depends_on:
- rabbitmq
- config-server
- eureka-server
environment:
SPRING_PROFILES_ACTIVE: "development"
RABBIT_MQ_HOST: "rabbitmq"
RABBIT_MQ_PORT: "5672"
RABBIT_MQ_URSER_NAME: "admin"
RABBIT_MQ_URSER_PASSWORD: "admin"
CONFIG_HOST: "config-server"
CONFIG_PORT: "8888"
EUREKA_HOST: "eureka-server"
EUREKA_PORT: "8761"
ports:
- "8000:8000"
user-service-2:
build:
context: user-service
dockerfile: Dockerfile
container_name: user-service-2
depends_on:
- rabbitmq
- config-server
- eureka-server
environment:
SPRING_PROFILES_ACTIVE: "development"
RABBIT_MQ_HOST: "rabbitmq"
RABBIT_MQ_PORT: "5672"
RABBIT_MQ_URSER_NAME: "admin"
RABBIT_MQ_URSER_PASSWORD: "admin"
CONFIG_HOST: "config-server"
CONFIG_PORT: "8888"
EUREKA_HOST: "eureka-server"
EUREKA_PORT: "8761"
ports:
- "8001:8000"
user-service-3:
build:
context: user-service
dockerfile: Dockerfile
container_name: user-service-3
depends_on:
- rabbitmq
- config-server
- eureka-server
environment:
SPRING_PROFILES_ACTIVE: "development"
RABBIT_MQ_HOST: "rabbitmq"
RABBIT_MQ_PORT: "5672"
RABBIT_MQ_URSER_NAME: "admin"
RABBIT_MQ_URSER_PASSWORD: "admin"
CONFIG_HOST: "config-server"
CONFIG_PORT: "8888"
EUREKA_HOST: "eureka-server"
EUREKA_PORT: "8761"
ports:
- "8002:8000"
When all my services are started, at first everything seems to be fine. All services are registered with eureka, all services are callable on localhost and they can talk to each other.
But the config refresh is not working.
When I change something in my config repository and commit and push it it has no effect on my services. The configuration they have stays the same. When I manually call the bus refresh on my config-server:
http://localhost:8888/bus/refresh
with POST, like it is described in nearly every guide I could find so far, I get the response:
{
"timestamp": "2020-03-30T09:33:57.818+0000",
"status": 405,
"error": "Method Not Allowed",
"message": "Request method 'POST' not supported",
"path": "/bus/refresh"
}
When I use GET instead, I get:
{
"name": "bus",
"profiles": [
"refresh"
],
"label": null,
"version": "03759e798f3516da6a18fc8b61a265d37ddeff4e",
"state": null,
"propertySources": []
}
and it also has no effect on the confiuration of my services.
And when I call the bus refresh on any of my services I get:
{
"timestamp": "2020-03-30T09:35:39.643+0000",
"status": 404,
"error": "Not Found",
"message": "No message available",
"path": "/bus/refresh"
}
What do I need to do to get the auto configuration refresh working?
Manually calling the /actuator/refresh endpoints on the services works. They are then pulling the new config. So it seems like the rabbit mq simply is not working.

I found the solution.
Acutally my setup and configuration files are totally fine, I just called the wrong endpoint. Instead of calling:
http://localhost:8888/bus/refresh
I have to call:
http://localhost:8888/actuator/bus-refresh
This works on any service I started. It will automatically send a message to rabbitmq which will then publish the refresh to all consumers. All configurations are then updating.
Though I still dont know why bus/refresh is not working. It is used in many examples / tutorials.

Related

Eureka Server is Working but not the services Springboot Microservices Docker-compose

I have created Microservices using Spring Boot and Eureka . I have used API Gateway for the microservices .
All the microservices ( eureka clients ) are visible on eureka server but giving an error like the below
api-Gateway port : 8999
product-service : 9001
product-detail-service : 9002
eureka-server : 8761
api-gatway application.properties
server.port =8999
spring.application.name = api-gateway
eureka.client.instance.preferIpAddress = true
eureka.client.serviceUrl.defaultZone= http://localhost:8761/eureka
spring.cloud.gateway.routes[0].id=product-service
spring.cloud.gateway.routes[0].uri=lb://product-service
spring.cloud.gateway.routes[0].predicates[0]=Path=/product/**
spring.cloud.gateway.routes[1].id=product-detail-service
spring.cloud.gateway.routes[1].uri=lb://product-detail-service
spring.cloud.gateway.routes[1].predicates[0]=Path=/productDetail/**
eureka-server application.properties
server.port=8761
eureka.client.register-with-eureka = false
eureka.server.waitTimeInMsWhenSyncEmpty = 0
product-detail-service application.properties
server.port=9002
spring.application.name = product-detail-service
eureka.instance.preferIpAddress = true
product-service application.properties
server.port = 9001
spring.application.name = product-service
eureka.client.instance.preferIpAddress = true
docker-compose.yml
version: '3.8'
services:
api-server:
build: ../apigateway
ports:
- 8999:8999
environment:
- eureka.client.service-url.defaultZone=http://eureka-server:8761/eureka
depends_on:
- product-service
- product-detail-service
eureka-server:
build: ../eureka_server
ports:
- 8761:8761
depends_on:
- product-service
- product-detail-service
product-service:
build: ../product_service
ports:
- 9001:9001
environment:
- eureka.client.service-url.defaultZone=http://eureka-server:8761/eureka
depends_on:
- product-detail-service
product-detail-service:
build: ../product_details_service
ports:
- 9002:9002
environment:
- eureka.client.service-url.defaultZone=http://eureka-server:8761/eureka
my docker images are created successfully and are running fine without docker-compose .
I have used networks and much more but still not resolved
Please help I am trying to solve the issue from 3 days

Laravel + Vite + Sail over HTTPS. Possible?

After several hours to try things, I think I'm too much confuse to understand what's going wrong... The title explain perfectly what I'm trying to make working.
My docker-compose.yml:
version: '3'
services:
mysite.test:
build:
context: ./docker/8.1
dockerfile: Dockerfile
args:
WWWGROUP: '${WWWGROUP}'
image: sail-8.1/app
extra_hosts:
- 'host.docker.internal:host-gateway'
ports:
- '${APP_PORT:-80}:80'
- '443:443' //added for test but not working...
- '${HMR_PORT:-8080}:8080'
- '5173:5173' //Vite port
environment:
WWWUSER: '${WWWUSER}'
LARAVEL_SAIL: 1
XDEBUG_MODE: '${SAIL_XDEBUG_MODE:-off}'
XDEBUG_CONFIG: '${SAIL_XDEBUG_CONFIG:-client_host=host.docker.internal}'
volumes:
- '.:/var/www/html'
networks:
- sail
depends_on:
- mysql
- minio
mysql:
image: 'mysql/mysql-server:8.0'
ports:
- '${FORWARD_DB_PORT:-3306}:3306'
environment:
MYSQL_ROOT_PASSWORD: '${DB_PASSWORD}'
MYSQL_ROOT_HOST: "%"
MYSQL_DATABASE: '${DB_DATABASE}'
MYSQL_USER: '${DB_USERNAME}'
MYSQL_PASSWORD: '${DB_PASSWORD}'
MYSQL_ALLOW_EMPTY_PASSWORD: 1
volumes:
- 'sail-mysql:/var/lib/mysql'
- './vendor/laravel/sail/database/mysql/create-testing-database.sh:/docker-entrypoint-initdb.d/10-create-testing-database.sh'
networks:
- sail
healthcheck:
test: ["CMD", "mysqladmin", "ping", "-p${DB_PASSWORD}"]
retries: 3
timeout: 5s
networks:
sail:
driver: bridge
volumes:
sail-nginx:
driver: local
sail-mysql:
driver: local
My vite.config.js:
import { defineConfig } from 'vite';
import laravel from 'laravel-vite-plugin';
export default defineConfig({
server: {
https: true,
host: '0.0.0.0'
},
plugins: [
laravel({
input: [
'resources/css/app.css',
'resources/js/app.js',
],
refresh: true
}),
],
});
My .env:
(...)
APP_URL=https://mysite.test
APP_SERVICE=mysite.test
(...)
The result of that configuration is it working with http://mysite.test but not in https. That return:
This site can’t be reached
mysite.test unexpectedly closed the connection.
Does anyone have a tips for me? 🙏
Thank you!
I managed to get this working with Caddy using the following gist
https://gist.github.com/gilbitron/36d48eb751875bebdf43da0a91c9faec
After all of the above I added the vite port to docker-compose.yml under laravel.test service.
ports:
- '5173:5173'
Also vite itself needs an ssl certificate
vite.config.js:
import { defineConfig } from 'vite';
import laravel from 'laravel-vite-plugin';
import vue from '#vitejs/plugin-vue';
import basicSsl from '#vitejs/plugin-basic-ssl'
export default defineConfig({
server: {
https: true,
host: '0.0.0.0',
hmr: {
host: 'localhost'
},
},
plugins: [
basicSsl(),
laravel({
input: 'resources/js/app.js',
refresh: true,
}),
vue({
template: {
transformAssetUrls: {
base: null,
includeAbsolute: false,
},
},
}),
],
});
For some reason my app generates http links, not https. So I added the following to the boot method of my AppServiceProvider.php
\URL::forceScheme('https');
Using the share command you can run a https tunnel through expose. Your domain will look like
https://[something].expose.dev
Getting a real globally trusted ssl certificate can only be done by a certificate authority like for example Let's Encrypt. They need to verify you own the domain, either by a http challenge or a dns challenge.
As you cannot really own a .test domain, your only option left is sign a certificate yourself and add it to your own computer plus it's root certificate. If you do this only your computer will show the connection as secure though.

Docker: Eureka clients are not registered in Eureka Server

I have searched over stackoverflow & tried alot of solutions but with no result.
I have 2 containers one is for eureka server and another one for spring cloud gateway.
I'm running these 2 containers with docker-compose but spring-cloud-gateway didn't register to eureka server.
I'm using https. On localhost everything is working as expected but when I jumped to containers this is the problem of unregistered clients.
eureka.server.yml:
eureka:
client:
fetch-registry: false
register-with-eureka: false
instance:
secure-port-enabled: true
non-secure-port-enabled: false
server:
port: 8761
ssl:
enabled: true
key-alias: statement
key-store: classpath:statement-keystore.p12
key-store-password: secret
key-store-type: PKCS12
spring:
application:
name: netflix-eureka
spring.cloud.gateway.yml:
eureka:
instance:
nonSecurePortEnabled: false
securePortEnabled: true
securePort: 8765
prefer-ip-address: true
client:
registerWithEureka: true
fetchRegistry: true
serviceUrl:
defaultZone: https://localhost:8761/eureka
level:
org:
springframework:
cloud:
gateway: DEBUG
reactor:
netty:
http:
client: DEBUG
server:
port: 8765
ssl:
enabled: true
key-alias: statement
key-store: classpath:statement-keystore.p12
key-store-password: secret
key-store-type: PKCS12
spring:
application:
name: api-gateway
cloud:
config:
enabled: false
gateway:
discovery:
locator:
enabled: true
httpclient:
wiretap: true
httpserver:
wiretap: true
globalcors:
corsConfigurations:
'[/**]':
allowedOrigins: "https://localhost:4200"
allowedHeaders: "*"
allowedMethods:
- GET
- POST
- PUT
- DELETE
spring.cloud.gateway.Dockerfile:
FROM openjdk:11-jre-slim
LABEL Description="SPring cloud gateway" Version="0.0.1"
ARG VERSION=0.0.1
VOLUME /tmp
ADD target/api-gateway-${VERSION}-SNAPSHOT.jar app.jar
RUN sh -c 'touch /app.jar'
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
eureka.server.Dockerfile:
FROM openjdk:11-jre-slim
LABEL Description="Eureka Server" Version="1.0"
ARG VERSION=1.0
VOLUME /tmp
ADD target/eureka-server-${VERSION}-SNAPSHOT.jar app.jar
RUN sh -c 'touch /app.jar'
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
docker-compose.yml:
version: '3.7'
services:
netflix-eureka:
image: abraham/netflix-eureka
container_name: "netflix-eureka"
ports:
- "8761:8761"
networks:
- statement-network
api-gateway:
image: abraham/api-gateway
container_name: "api-gateway"
ports:
- "8765:8765"
networks:
- statement-network
environment:
eureka.client.service-url.default-zone: https://netflix-eureka:8761/eureka
depends_on:
- netflix-eureka
links:
- netflix-eureka
networks:
abraham-network:

Could not connect Logstash to Kafka via compose file

I'm using compose file to create data pipeline between Logstash and Kafka. But this message shows up in logstash container. Could someone help me out?
The message:
[WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-0, groupId=logstash] Connection to node 2 could not be established. Broker may not be available.
My compose file:
version: "3"
services:
zookeeper:
image: confluentinc/cp-zookeeper:6.2.0
container_name: zookeeper
ports:
- "2181:2181"
networks:
- kafkanet
environment:
ZOOKEEPER_CLIENT_PORT: "2181"
ZOOKEEPER_TICK_TIME: "2000"
ZOOKEEPER_SYNC_LIMIT: "2"
kafkaserver:
image: confluentinc/cp-kafka:6.2.0
container_name: kafka
ports:
- "9092:9092"
networks:
- kafkanet
environment:
KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://localhost:9092"
KAFKA_BROKER_ID: "2"
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: "1"
depends_on:
- zookeeper
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.4.0
container_name: elasticsearch
ports:
- 9200:9200
- 9300:9300
networks:
- kafkanet
kibana:
image: docker.elastic.co/kibana/kibana:6.4.0
container_name: kibana
ports:
- 5601:5601
networks:
- kafkanet
depends_on: [ 'elasticsearch' ]
# Logstash Docker Image
logstash:
image: docker.elastic.co/logstash/logstash:6.4.0
container_name: logstash
networks:
- kafkanet
depends_on: [ 'elasticsearch', 'kafkaserver' ]
volumes:
- './logstash/config:/usr/share/logstash/pipeline/'
networks:
kafkanet:
driver: bridge
./logstash/config/logstash.conf
input {
kafka {
bootstrap_servers => "kafkaserver:9092"
topics => ["sit.catalogue.item","uat.catalogue.item"]
auto_offset_reset => "earliest"
decorate_events => true
}
}
output {
elasticsearch {
hosts => ["elasticsearch:9200"]
index => "%{[indexPrefix]}-logs-%{+YYYY.MM.dd}"
}
}
Your advertised listener in Kafka is not right. It should be kafkaserver.
So instead of
KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://localhost:9092"
You need
KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafkaserver:9092"
For more details, see this blog that I wrote.
BTW if you're pushing data from Kafka to Elasticsearch you should check out Kafka Connect as another option.

Spring on docker, environment variables not evaluated with configuration placeholders

I have a Spring project and use an application.yml file to run configure it. When using placeholders in the configuration file and run it as a built docker image it will not evaluate the placeholder, whereas this works fine when running the jar without docker. What could be wrong here?
org.postgresql.util.PSQLException: FATAL: password authentication failed for user "${POSTGRES_DB_USER}"
application.yml
server:
compression:
enabled: true
mime-types: application/json,application/xml,text/html,text/xml,text/plain
context-path: /
port: 8085
logging:
config: config/log4j2-spring.xml
spring:
datasource:
url: jdbc:h2:./data/cdr
username: sa
password:
jpa:
hibernate:
ddl-auto: create-drop
security:
user:
password: secretpassword
---
spring:
profiles: docker
datasource:
url: jdbc:postgresql://postgres/databasename
username: ${POSTGRES_DB_USER}
password: ${POSTGRES_DB_PASS}
docker-compose.yml
version: '2'
services:
application:
image: domain.com:3000/application:0-SNAPSHOT
volumes:
- application:/application/logs
ports:
- 8085:8085
links:
- postgres
environment:
PASSWORD: secretpassword
postgres:
image: sameersbn/postgresql:9.5-2
volumes:
- postgres-data:/var/lib/postgresql
environment:
DB_NAME: databasename
DB_USER: user
DB_PASS: secretpassword
volumes:
postgres-data:
driver: local
application-logs:
driver: local
I think you want this:
version: '2'
services:
application:
image: domain.com:3000/application:0-SNAPSHOT
volumes:
- application-logs:/application/logs
ports:
- 8085:8085
environment:
POSTGRES_DB_USER: user
POSTGRES_DB_PASS: secretpassword
postgres:
image: sameersbn/postgresql:9.5-2
volumes:
- postgres-data:/var/lib/postgresql
environment:
DB_NAME: databasename
DB_USER: user
DB_PASS: secretpassword
volumes:
postgres-data: {}
application-logs: {}

Resources