Spring Cloud Config - Vault and JDBC backend with JDBC creds in Vault - spring-boot

I am attempting to modify our current Spring Cloud Config server which has only a JDBC backend to include a Vault backend in order make the JDBC connection credentials secret.
VAULT:
Listener 1: tcp (addr: "127.0.0.1:8400", cluster address: "127.0.0.1:8401", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
C:\apps\HashiCorp>vault kv get secret/my-secrets
=============== Data ===============
Key Value
--- -----
spring.datasource.password yadayadayada
spring.datasource.username cobar
bootstrap.yml
server:
port: 8888
spring:
application:
name: config-server
cloud:
config:
allowOverride: true
server:
jdbc:
sql: SELECT prop_key, prop_value from CloudProperties where application=? and profile=? and label=?
order: 2
#https://cloud.spring.io/spring-cloud-config/reference/html/#vault-backend
vault:
scheme: http
host: localhost
port: 8400
defaultKey: my-secrets
order: 1
application.yml
spring:
main:
banner-mode: off
allow-bean-definition-overriding: true
datasource:
url: jdbc:mysql://localhost/bootdb?createDatabaseIfNotExist=true&autoReconnect=true&useSSL=false
#username: cobar
#password: yadayadayada
driverClassName: com.mysql.jdbc.Driver
hikari:
connection-timeout: 60000
maximum-pool-size: 5
cloud:
vault:
scheme: http
host: localhost
port: 8400
defaultKey: my-secrets
token: root.RIJQjZ4jRZUS8mskzfCON88K
The spring.datasource username and password are not being retrieved from the vault.
2021-12-01 12:43:39.927 INFO 5992 --- [ restartedMain]: The following profiles are active: jdbc,vault
2021-12-01 12:43:46.123 ERROR 5992 --- [ restartedMain] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Exception during pool initialization.
Login failed for user ''. ClientConnectionId:a32

Move properties from bootstrap to application context.
Call Vault endpoint to obtain secrets and use these to configure Datasource to JDBC backend.
#Slf4j
#SpringBootApplication
#EnableConfigServer
public class ConfigServerApplication {
public static final String VAULT_URL_FRMT = "%s://%s:%s/v1/secret/%s";
#Autowired
private Environment env;
public static void main(String[] args) {
SpringApplication app = new SpringApplication(ConfigServerApplication.class);
app.addListeners(new ApplicationPidFileWriter());
app.addListeners(new WebServerPortFileWriter());
app.run(args);
}
#Order(1)
#Bean("restTemplate")
public RestTemplate restTemplate() {
return new RestTemplate();
}
#Configuration
public class JdbcConfig {
#Autowired
private RestTemplate restTemplate;
#Bean
public DataSource getDataSource() {
Secrets secrets = findSecrets();
DataSourceBuilder dataSourceBuilder = DataSourceBuilder.create();
dataSourceBuilder.url(secrets.getData().get("spring.datasource.url"));
dataSourceBuilder.username(secrets.getData().get("spring.datasource.username"));
dataSourceBuilder.password(secrets.getData().get("spring.datasource.password"));
return dataSourceBuilder.build();
}
private Secrets findSecrets() {
HttpHeaders httpHeaders = new HttpHeaders();
httpHeaders.set("X-Vault-Token", env.getProperty("spring.cloud.vault.token"));
HttpEntity request = new HttpEntity(httpHeaders);
String url = String.format(VAULT_URL_FRMT,
env.getProperty("spring.cloud.vault.scheme"),
env.getProperty("spring.cloud.vault.host"),
env.getProperty("spring.cloud.vault.port"),
env.getProperty("spring.cloud.vault.defaultKey")
);
return restTemplate.exchange(url, HttpMethod.GET, request, Secrets.class, 1).getBody();
}
}
}
#Getter
#Setter
public class Secrets implements Serializable {
private String request_id;
private String lease_id;
private boolean renewable;
private Duration lease_duration;
private Map<String, String> data;
}
Now you have a Cloud Config with a JDBC backend you can keep the Database properties secret.

Related

Amazon RDS Read Replica configuration Postgres database from an spring boot application deployed on PCF?

Hi All currently we are have deployed our springboot code to pcf which is running on aws.
we are using aws database - where we have cup service and VCAP_SERVICES which hold the parameter of db.
Below our configuration to get datasource
#Bean
public DataSource dataSource() {
if (dataSource == null) {
dataSource = connectionFactory().dataSource();
configureDataSource(dataSource);
}
return dataSource;
}
#Bean
public JdbcTemplate jdbcTemplate() {
return new JdbcTemplate(dataSource());
}
private void configureDataSource(DataSource dataSource) {
org.apache.tomcat.jdbc.pool.DataSource tomcatDataSource = asTomcatDatasource(dataSource);
tomcatDataSource.setTestOnBorrow(true);
tomcatDataSource.setValidationQuery("SELECT 1");
tomcatDataSource.setValidationInterval(30000);
tomcatDataSource.setTestWhileIdle(true);
tomcatDataSource.setTimeBetweenEvictionRunsMillis(60000);
tomcatDataSource.setRemoveAbandoned(true);
tomcatDataSource.setRemoveAbandonedTimeout(60);
tomcatDataSource.setMaxActive(Environment.getAsInt("MAX_ACTIVE_DB_CONNECTIONS", tomcatDataSource.getMaxActive()));
}
private org.apache.tomcat.jdbc.pool.DataSource asTomcatDatasource(DataSource dataSource) {
Objects.requireNonNull(dataSource, "There is no DataSource configured");
DataSource targetDataSource = ((DelegatingDataSource)dataSource).getTargetDataSource();
return (org.apache.tomcat.jdbc.pool.DataSource) targetDataSource;
}
Now when we have read replicas created , what configuration do i need to modify so our spring boot application uses the read replicas?
is Just #Transactional(readOnly = true) on the get call is enough - that it will be automatically taken care? or do i need to add some more configuration
#Repository
public class PostgresSomeRepository implements SomeRepository {
#Autowired
public PostgresSomeRepository(JdbcTemplate jdbcTemplate, RowMapper<Consent> rowMapper) {
this.jdbcTemplate = jdbcTemplate;
this.rowMapper = rowMapper;
}
#Override
#Transactional(readOnly = true)
public List<SomeValue> getSomeGetCall(List<String> userIds, String applicationName, String propositionName, String since, String... types) {
//Some Logic
try {
return jdbcTemplate.query(sql, rowMapper, paramList.toArray());
} catch (DataAccessException ex) {
throw new ErrorGettingConsent(ex.getMessage(), ex);
}
}
}
Note:we have not added any spring aws jdbc dependency
Let's assume the cloud service name is my_db.
Map the cloud service to the application config appication-cloud.yml used by default in the CF (BTW this is better than using the connector because you can customize the datasource)
spring:
datasource:
type: com.zaxxer.hikari.HikariDataSource
# my_db
url: ${vcap.services.my_db.credentials.url}
username: ${vcap.services.my_db.credentials.username}
password: ${vcap.services.my_db.credentials.password}
hikari:
poolName: Hikari
auto-commit: false
data-source-properties:
cachePrepStmts: true
prepStmtCacheSize: 250
prepStmtCacheSqlLimit: 2048
useServerPrepStmts: true
jpa:
generate-ddl: false
show-sql: true
put the service to the application manifest.yml:
---
applications:
- name: my-app
env:
SPRING_PROFILES_ACTIVE: "cloud" # by default
services:
- my_db

Error while using custom loadbalancer for spring cloud loadbalancer with healthcheck configuration

I am using a static list (SimpleDiscoveryClient) to loadbalance using spring cloud loadbalancer. Using StickySession Loadbalancer rule in https://github.com/fitzoh/spring-cloud-commons/blob/e10997b6141ff560479ef7065c3547f1f59360c8/spring-cloud-loadbalancer/src/main/java/org/springframework/cloud/loadbalancer/core/StickySessionLoadBalancer.java.
My WebClientConfig class:
#Configuration
#LoadBalancerClient(name = "testservice", configuration = CustomLoadBalancerConfiguration.class)
public class WebClientConfig {
#LoadBalanced
#Bean
WebClient.Builder webClientBuilder() {
return WebClient.builder();
}
}
Custom LoadBalancer Configuration class:
public class CustomLoadBalancerConfiguration {
#Bean
ReactorLoadBalancer<ServiceInstance> StickySessionLoadBalancer(Environment environment,
LoadBalancerClientFactory loadBalancerClientFactory) {
String name = environment.getProperty(LoadBalancerClientFactory.PROPERTY_NAME);
return new StickySessionLoadBalancer(loadBalancerClientFactory
.getLazyProvider(name, ServiceInstanceListSupplier.class),
name);
}
#Bean
public ServiceInstanceListSupplier discoveryClientServiceInstanceListSupplier(
ConfigurableApplicationContext context) {
return ServiceInstanceListSupplier.builder()
.withDiscoveryClient()
.withHealthChecks()
.build(context);
}
}
Posting my yml here :
spring:
application:
name: sample
cloud:
discovery:
client:
health-indicator:
enabled: false
simple:
instances:
testservice:
- uri: http://localhost:8082
- uri: http://localhost:8081
loadbalancer:
configurations: health-check
cache:
enabled: false
health-check:
path:
default: /actuator/health
interval: 10000
gateway:
routes:
- id: testrouting
path: /user/*
uri: lb://testservice
predicates:
- Method=GET,POST
- Path=/user/**
It's all according to the official documentation. But with the customloadbalancer rule (Stickysession Loadbalancer), the healthchecks to servers are not happening to checkif the servers are alive or not. The server list is always empty (all servers are marked as not alive).

FeignException ServiceUnavailable: Load balancer does not contain an instance for the service match-result-service

I fail to understand, nor find any explanation about how can I fix the exception in subject.
This is the main code I use:
Eureka Server
application:
...
#SpringBootApplication
#EnableEurekaServer
public class BbSimApplication {
public static void main(String[] args) {
SpringApplication.run(BbSimApplication.class, args);
}
}
application.yml:
server.port : 8088
spring:
application:
name : bbsim-discovery-server
eureka:
server:
evictionIntervalTimerInMs: 3000
response-cache-update-interval-ms: 3000
wait-time-in-ms-when-sync-empty: 0
peer-node-read-timeout-ms : 10000
client:
registerWithEureka: false
fetchRegistry: false
service-url:
defaultZone: http://localhost:${server.port}/eureka
Controller class:
#RestController
public class WebController {
#Autowired private WebService webService;
#GetMapping("/matchresultbyids")
public MatchStats matchResult(#RequestParam Long homeId, #RequestParam Long awayId){
return webService.matchResult(homeId, awayId);
}
}
Manager Service
application:
#SpringBootApplication
#EnableFeignClients
#EnableDiscoveryClient
public class BBSimManagerServiceApplication {
public static void main(String[] args) {
SpringApplication.run(BBSimManagerServiceApplication.class, args);
}
}
application.yaml:
server.port: 9903
spring:
application.name: bbsim-manager-service
eureka:
client:
serviceUrl:
defaultZone: ${EUREKA_URI:http://localhost:8088/eureka}
registryFetchIntervalSeconds: 1
# register-with-eureka: true
# fetch-registry: true
instance:
leaseRenewalIntervalInSeconds: 1
Client interface:
#FeignClient("match-result-service")
public interface MatchResultClient {
#GetMapping("/matchresultbyids")
MatchStats getMatchResult();
}
Controller class:
#RestController
public class BbsimManagerController {
#Autowired
MatchResultClient matchStatsClient;
#GetMapping("/matchresultbyids")
public MatchStats matchResult(){
return matchStatsClient.getMatchResult();
}
}
MatchResult Service
application:
#SpringBootApplication
#EnableDiscoveryClient
public class MatchResultServiceApplication {
public static void main(String[] args) {
SpringApplication.run(MatchResultServiceApplication.class, args);
}
}
application.yml:
server.port: 9901
spring:
application:
name: match-result-service
eureka:
client:
serviceUrl:
defaultZone: http://localhost:8088/eureka/
registryFetchIntervalSeconds: 1
instance:
leaseRenewalIntervalInSeconds: 1
Controller:
#RestController
public class WebController {
#Autowired private WebService webService;
#GetMapping("/matchresultbyids")
public MatchStats matchResult(){
return webService.matchResult();
}
}
When I try to execute:
http://localhost:9903/matchresultbyids
I get the exception:
o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is feign.FeignException$ServiceUnavailable: [503] during [GET] to [http://match-result-service/matchresultbyids?homeId=0&awayId=1] [MatchResultClient#getMatchResult()]: [Load balancer does not contain an instance for the service match-result-service]] with root cause
feign.FeignException$ServiceUnavailable: [503] during [GET] to [http://match-result-service/matchresultbyids?homeId=0&awayId=1] [MatchResultClient#getMatchResult()]: [Load balancer does not contain an instance for the service match-result-service]
Can you advise me what is wrong and how to fix it?
Thank you all.
It may be related to OpenFeign bug. You can try to upgrade it to the version >= v3.0.6.
If you use spring-cloud-dependencies the version >= v2020.0.5 should be also ok.

Unable to update hikari datasource configuration in spring boot app

Hikari datasource configurations always taking as default values. even though i provide the actual configurations in application.yml file.
MainApp.java
#SpringBootApplication
public class MainApp {
public static void main(String[] args) {
SpringApplication.run(MainApp.class, args);
}
#Bean
#Primary
public DataSourceProperties dataSourceProperties() {
return new DataSourceProperties();
}
//I use below method to set password from different approach instead of taking from app.yml
#Bean
public DataSource dataSource(DataSourceProperties properties) {
DataSource ds = properties.initializeDataSourceBuilder()
.password("setting a password from vault")
.build();
return ds;
}
}
application.yml
spring:
application:
name: demo
datasource:
hikari:
connection-timeout: 20000
minimum-idle: 5
maximum-pool-size: 12
idle-timeout: 300000
max-lifetime: 1200000
auto-commit: true
driverClassName: com.microsoft.sqlserver.jdbc.SQLServerDriver
url: jdbc:sqlserver://ip:port;databaseName=sample
username: username
Using Spring 2.1.1.RELEASE and when i start the application i logged the Hikari logs and it says all default values. So i did debugged on the second bean datasource and its actually HikariDatasource and except default values rest of them are empty and not reflecting the given values from the application.yml. Please recommend or comment if i misconfigured anything or done any mistake!

Zuul Implementing WeightedResponseTimeRule Error - VIP address for client null is null

I'm encountering this VIP address null white implementing weightedResponseTimeRule for my Zuul gateway.
Did i do it correctly? implementing a config that will implement weightedRule.
I want to route my request to my 2 or more instance.
Error creating bean with name 'ribbonServerList' defined in org.springframework.cloud.netflix.ribbon.eureka.EurekaRibbonClientConfiguration: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.netflix.loadbalancer.ServerList]: Factory method 'ribbonServerList' threw exception; nested exception is java.lang.NullPointerException: VIP address for client null is null
Currently my Config for Ribbon
#Configuration
public class GatewayRibbonConfiguration {
#Bean
public IClientConfig ribbonClientConfig(){
return new DefaultClientConfigImpl();
}
#Bean
public IPing ribbonPing(IClientConfig config) {
return new PingUrl();
}
#Bean
public IRule ribbonRule(IClientConfig config) {
return new WeightedResponseTimeRule();
}
}
Application
#SpringBootApplication
#EnableDiscoveryClient
#EnableZuulProxy
public class GatewayApplication {
public static void main(String[] args) {
SpringApplication.run(GatewayApplication.class, args);
}
#Bean
public Prefilter prefilter(){
return new Prefilter();
}
}
bootstrap.yml
spring:
application:
name: gateway
eureka:
client:
serviceUrl:
defaultZone: http://localhost:8761/eureka/
my properties
server:
port: 8080
hystrix:
command:
default:
execution:
isolation:
thread:
timeoutInMilliseconds: 20000
ribbon:
ReadTimeout: 20000
ConnectTimeout: 20000
zuul:
prefix: /api
ignoredServices: '*'
host:
connect-timeout-millis: 20000
socket-timeout-millis: 20000
routes:
kicks-service:
path: /kicks/**
serviceId: kicks-service
stripPrefix: false
sensitiveHeaders:
kicks-inventory:
path: /inventory/**
serviceId: kicks-inventory
stripPrefix: false
sensitiveHeaders:

Resources