Redis SSL configuration is not clear in case of SpringBoot with lettuce-core - spring-boot

Spring-boot: 2.0.3.RELEASE
lettuce-core: 5.0.4.RELEASE
spring-data-redis: 2.0.8.RELEASE
So first of all my issue is solved I just don't understand why and it bothers me, so some clarification around this is appreciated.
I have this Redis Configuration in one of the micro-service I'm working on:
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import io.lettuce.core.RedisClient;
import io.lettuce.core.RedisURI;
import io.lettuce.core.api.StatefulRedisConnection;
import io.lettuce.core.resource.ClientResources;
import io.lettuce.core.resource.DefaultClientResources;
#Configuration
public class LettuceConfig {
#Bean(destroyMethod = "shutdown")
ClientResources clientResources() {
return DefaultClientResources.create();
}
#Bean(destroyMethod = "shutdown")
RedisClient redisClient(ClientResources clientResources) {
String host = System.getenv("spring_redis_host") != null ? System.getenv("spring_redis_host") : "127.0.0.1";
String port = System.getenv("spring_redis_port") != null ? System.getenv("spring_redis_port") : "6379";
String password = System.getenv("spring_redis_password") != null ? System.getenv("spring_redis_password") : "";
Boolean isLocal = host.equals("127.0.0.1");
RedisURI redisUri = RedisURI.Builder.redis(host).withSsl(!isLocal).withPassword(password).withPort(Integer.parseInt(port))
.build();
return RedisClient.create(clientResources, redisUri);
}
#Bean(destroyMethod = "close")
StatefulRedisConnection<String, String> connection(RedisClient redisClient) {
return redisClient.connect();
}
}
After I added actuator to the project I realized that the service couldn't connect to the remote Redis in AWS and the service throwed the following Exception:
java.io.IOException: Connection reset by peer
After a quick google search most of the answers suggested that the problem is around the SSL connection I have, but I was confused since the RedisURI contains the withSsl function call.
Just out of curiosity I added the following property to my application.properties file, since I'm using property based Redis configuration in the other services and I knew it works.
spring.redis.ssl=true
Now this solved my problem, but I don't understand why this approach is working and the RedisURI version is not.
Could somebody provide me an explanation, who has a clear understanding about the following situation please?
I can share more from the logs if needed. Thank you!

Related

How to redirect Prometheus Metrics to the default spring boot server

I am trying to expose a custom Gauge metric from my Spring Boot Application. I am using Micrometer with the Prometheus registry to do so. I have set up the PrometheusRegistry and configs as per - Micrometer Samples - Github but it creates one more HTTP server for exposing the Prometheus metrics. I need to redirect or expose all the metrics to the Spring boot's default context path - /actuator/prometheus instead of a new context path on a new port. I have implemented the following code so far -
PrometheusRegistry.java -
package com.xyz.abc.prometheus;
import java.io.IOException;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.time.Duration;
import com.sun.net.httpserver.HttpServer;
import io.micrometer.core.lang.Nullable;
import io.micrometer.prometheus.PrometheusConfig;
import io.micrometer.prometheus.PrometheusMeterRegistry;
public class PrometheusRegistry {
public static PrometheusMeterRegistry prometheus() {
PrometheusMeterRegistry prometheusRegistry = new PrometheusMeterRegistry(new PrometheusConfig() {
#Override
public Duration step() {
return Duration.ofSeconds(10);
}
#Override
#Nullable
public String get(String k) {
return null;
}
});
try {
HttpServer server = HttpServer.create(new InetSocketAddress(8081), 0);
server.createContext("/sample-data/prometheus", httpExchange -> {
String response = prometheusRegistry.scrape();
httpExchange.sendResponseHeaders(200, response.length());
OutputStream os = httpExchange.getResponseBody();
os.write(response.getBytes());
os.close();
});
new Thread(server::start).run();
} catch (IOException e) {
throw new RuntimeException(e);
}
return prometheusRegistry;
}
}
MicrometerConfig.java -
package com.xyz.abc.prometheus;
import io.micrometer.core.instrument.MeterRegistry;
public class MicrometerConfig {
public static MeterRegistry carMonitoringSystem() {
// Pick a monitoring system here to use in your samples.
return PrometheusRegistry.prometheus();
}
}
Code snippet where I am creating a custom Gauge metric. As of now, it's a simple REST API to test - (Please read the comments in between)
#SuppressWarnings({ "unchecked", "rawtypes" })
#RequestMapping(value = "/sampleApi", method= RequestMethod.GET)
#ResponseBody
//This Timed annotation is working fine and this metrics comes in /actuator/prometheus by default
#Timed(value = "car.healthcheck", description = "Time taken to return healthcheck")
public ResponseEntity healthCheck(){
MeterRegistry registry = MicrometerConfig.carMonitoringSystem();
AtomicLong n = new AtomicLong();
//Starting from here none of the Gauge metrics shows up in /actuator/prometheus path instead it goes to /sample-data/prometheus on port 8081 as configured.
registry.gauge("car.gauge.one", Tags.of("k", "v"), n);
registry.gauge("car.gauge.two", Tags.of("k", "v1"), n, n2 -> n2.get() - 1);
registry.gauge("car.help.gauge", 89);
//This thing never works! This gauge metrics never shows up in any URI configured
Gauge.builder("car.gauge.test", cpu)
.description("car.device.cpu")
.tags("customer", "demo")
.register(registry);
return new ResponseEntity("Car is working fine.", HttpStatus.OK);
}
I need all the metrics to show up inside - /actuator/prometheus instead of a new HTTP Server getting created. I know that I am explicitly creating a new HTTP Server so metrics are popping up there. Please let me know how to avoid creating a new HTTP Server and redirect all the prometheus metrics to the default path - /actuator/prometheus. Also if I use Gauge.builder to define a custom gauge metrics, it never works. Please explain how I can make that work also. Let me know where I am doing wrong.
Thank you.
Every time you call MicrometerConfig.carMonitoringSystem(); it is creating a new prometheus registry (and trying to start a new server)
You need to inject the MeterRegistry in your class that is creating the gauge and use the injected MeterRegistry that way.

aggregator spring cloud stream with timeout

I want to make an application that receives messages, stores those messages in a list, and later with and schedule releases those messages every x amount of time.
I know spring cloud stream has an aggregator that already does this, but I think I need it to be done manually because I need to keep a unique message based upon a key and only replace the old message if it matches a specific condition ( I think of it as a Set aggregator with conditions)
what I have tried so far.
also in this link https://github.com/chalimbu/AggregatorQuestionStack
Processor.
import org.springframework.cloud.stream.annotation.EnableBinding
import org.springframework.cloud.stream.annotation.Input
import org.springframework.cloud.stream.annotation.Output
import org.springframework.cloud.stream.messaging.Processor
import org.springframework.scheduling.annotation.Scheduled
#EnableBinding(Processor::class)
class SetAggregatorProcessor(val storageService: StorageService) {
#Input
public fun inputMessage(input: Map<String,Any>){
storageService.messages.add(input)
}
#Output
#Scheduled(fixedDelay = 20000)
public fun produceOutput():List<Map<String,Any>>{
val message= storageService.messages
storageService.messages.clear()
return message;
}
}
Memory storage.
import org.springframework.stereotype.Service
#Service
class StorageService {
public var messages: MutableList<Map<String,Any>> = mutableListOf()
}
This code generates the following error when I start pushing messages.
Caused by: org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:139) ~[spring-integration-core-5.5.8.jar:5.5.8]
The idea is to deploy this app as part of the spring cloud stream (dataflow) platform.
I prefer the declarative approach(over the functional approach), but if somebody knows how to do it with the reactor way, I could settle for it.
Thanks for any help or advice.
thanks to this example(https://github.com/spring-cloud/spring-cloud-stream-samples/blob/main/processor-samples/sensor-average-reactive-kafka/src/main/java/sample/sensor/average/SensorAverageProcessorApplication.java) I was able to figure something out using flux in case someone else needs it
#Configuration
class SetAggregatorProcessor : Function<Flux<Map<String, Any>>, Flux<MutableList<Map<String, Any>>>> {
override fun apply(data: Flux<Map<String, Any>>):Flux<MutableList<Map<String, Any>>> {
return data.window(Duration.ofSeconds(20)).flatMap { window: Flux<Map<String, Any>> ->
this.aggregateList(window)
}
}
private fun aggregateList(group: Flux<Map<String, Any>>): Mono<MutableList<Map<String, Any>>>? {
return group.reduce(
mutableListOf(),
BiFunction<MutableList<Map<String, Any>>, Map<String, Any>, MutableList<Map<String, Any>>> {
acumulator: MutableList<Map<String, Any>>, element: Map<String, Any> ->
acumulator.add(element)
acumulator
}
)
}
}
update https://github.com/chalimbu/AggregatorQuestionStack/tree/main/src/main/kotlin/com/project/co/SetAggregator

Attaching AWS documentDB to Spring Boot application

I've recently tried using the new AWS DocumentDB service as my DB in a Spring application.
The cluster has been created in the same VPC as the EKS on which I deploy my application. Security groups allow connections between all nodes in the VPC.
AWS exposes a mongo URI like this for my DB cluster:
mongodb://<my-user>:<insertYourPassword>#<my-cluster-endpoint>:27017/?ssl_ca_certs=rds-combined-ca-bundle.pem&replicaSet=rs0
My question:
How do I make my Spring code work with this kind of connection?
I have tried adding the followig to my application.properties file:
spring.data.mongodb.uri=mongodb://<my-user>:<insertYourPassword>#<my-cluster-endpoint>:27017/admin?ssl_ca_certs=rds-combined-ca-bundle.pem&replicaSet=rs00
spring.data.mongodb.database=admin
server.ssl.key-store=classpath:rds-combined-ca-bundle.pem
And placing the PEM file in /src/main/resources
However the code still fails to connect to the DB cluster.
I get this message as an error: No server chosen by com.mongodb.client.internal.MongoClientDelegate
Followed by a Exception in monitor thread while connecting to server ...
And finally a timeout exception: com.mongodb.MongoSocketReadTimeoutException: Timeout while receiving message
It looks kind of like a security group issue but I have no problem connecting with mongo shell from the same EC2 running the Spring application Pod.
Any ideas?
As mentioned in the documentation,
By design, you access Amazon DocumentDB (with MongoDB compatibility) resources from an Amazon EC2 instance within the same Amazon VPC as the Amazon DocumentDB resources. However, suppose that your use case requires that you or your application access your Amazon DocumentDB resources from outside the cluster's Amazon VPC. In that case, you can use SSH tunneling (also known as "port forwarding") to access your Amazon DocumentDB resources.
Connect from outside VPC
Your Amazon DocumentDB cluster should be running in your default virtual private cloud (VPC). To interact with your Amazon DocumentDB cluster, you must launch an Amazon Elastic Compute Cloud (Amazon EC2) instance into your default VPC, in the same AWS Region where you created your Amazon DocumentDB cluster.
Follow the guide to connect to the cluster
AWS DocumentDB cluster
GitHub Reference: spring-boot-aws-documentdb
Update:
To connect through SSL, use below logic by setting SSL_CERTIFICATE pointing to aws region specific intermediate certificate.
This can be downloaded from SSL certs and copy it to base directory.
Alternatively, you can provide absolute path to the variable SSL_CERTIFICATE.
private static final String SSL_CERTIFICATE = "rds-ca-2015-us-east-1.pem";
private static final String KEY_STORE_TYPE = "JKS";
private static final String KEY_STORE_PROVIDER = "SUN";
private static final String KEY_STORE_FILE_PREFIX = "sys-connect-via-ssl-test-cacerts";
private static final String KEY_STORE_FILE_SUFFIX = ".jks";
private static final String DEFAULT_KEY_STORE_PASSWORD = "changeit";
public static void main(String[] args) {
SSLContextHelper.setSslProperties();
SpringApplication.run(Application.class, args);
}
protected static class SSLContextHelper{
/**
* This method sets the SSL properties which specify the key store file, its type and password:
* #throws Exception
*/
private static void setSslProperties() {
try {
System.setProperty("javax.net.ssl.trustStore", createKeyStoreFile());
} catch (Exception e) {
e.printStackTrace();
}
System.setProperty("javax.net.ssl.trustStoreType", KEY_STORE_TYPE);
System.setProperty("javax.net.ssl.trustStorePassword", DEFAULT_KEY_STORE_PASSWORD);
}
private static String createKeyStoreFile() throws Exception {
return createKeyStoreFile(createCertificate()).getPath();
}
/**
* This method generates the SSL certificate
* #return
* #throws Exception
*/
private static X509Certificate createCertificate() throws Exception {
CertificateFactory certFactory = CertificateFactory.getInstance("X.509");
URL url = new File(SSL_CERTIFICATE).toURI().toURL();
if (url == null) {
throw new Exception();
}
try (InputStream certInputStream = url.openStream()) {
return (X509Certificate) certFactory.generateCertificate(certInputStream);
}
}
/**
* This method creates the Key Store File
* #param rootX509Certificate - the SSL certificate to be stored in the KeyStore
* #return
* #throws Exception
*/
private static File createKeyStoreFile(X509Certificate rootX509Certificate) throws Exception {
File keyStoreFile = File.createTempFile(KEY_STORE_FILE_PREFIX, KEY_STORE_FILE_SUFFIX);
try (FileOutputStream fos = new FileOutputStream(keyStoreFile.getPath())) {
KeyStore ks = KeyStore.getInstance(KEY_STORE_TYPE, KEY_STORE_PROVIDER);
ks.load(null);
ks.setCertificateEntry("rootCaCertificate", rootX509Certificate);
ks.store(fos, DEFAULT_KEY_STORE_PASSWORD.toCharArray());
}
return keyStoreFile;
}
}
connection output:
019-01-17 13:33:22.316 INFO 3598 --- [onaws.com:27017] org.mongodb.driver.cluster : Canonical address mongodb.cktoiipu3bbd.us-east-1.docdb.amazonaws.com:27017 does not match server address. Removing mongodb.cluster-cktoiipu3bbd.us-east-1.docdb.amazonaws.com:27017 from client view of cluster
2019-01-17 13:33:22.401 INFO 3598 --- [onaws.com:27017] org.mongodb.driver.connection : Opened connection [connectionId{localValue:2}] to mongodb.cktoiipu3bbd.us-east-1.docdb.amazonaws.com:27017
2019-01-17 13:33:22.403 INFO 3598 --- [onaws.com:27017] org.mongodb.driver.cluster : Monitor thread successfully connected to server with description ServerDescription{address=mongodb.cktoiipu3bbd.us-east-1.docdb.amazonaws.com:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 6, 0]}, minWireVersion=0, maxWireVersion=6, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=null, roundTripTimeNanos=2132149, setName='rs0', canonicalAddress=mongodb.cktoiipu3bbd.us-east-1.docdb.amazonaws.com:27017, hosts=[mongodb.cktoiipu3bbd.us-east-1.docdb.amazonaws.com:27017], passives=[], arbiters=[], primary='mongodb.cktoiipu3bbd.us-east-1.docdb.amazonaws.com:27017', tagSet=TagSet{[]}, electionId=7fffffff0000000000000001, setVersion=null, lastWriteDate=Thu Jan 17 13:33:21 UTC 2019, lastUpdateTimeNanos=516261208876}
2019-01-17 13:33:22.406 INFO 3598 --- [onaws.com:27017] org.mongodb.driver.cluster : Discovered replica set primary mongodb.cktoiipu3bbd.us-east-1.docdb.amazonaws.com:27017
2019-01-17 13:33:22.595 INFO 3598 --- [ main] com.barath.app.CustomerService : Saving the customer with customer details com.barath.app.Customer#6c130c45
2019-01-17 13:33:22.912 INFO 3598 --- [ main] org.mongodb.driver.connection : Opened connection [connectionId{localValue:3}] to mongodb.cktoiipu3bbd.us-east-1.docdb.amazonaws.com:27017
2019-01-17 13:33:23.936 INFO 3598 --- [ main] pertySourcedRequestMappingHandlerMapping : Mapped URL path [/v2/api-docs] onto method [public org.springframework.http.ResponseEntity<springfox.documentation.spring.web.json.Json> springfox.documentation.swagger2.web.Swagger2Controller.getDocumentation(java.lang.String,javax.servlet.http.HttpServletRequest)]
The answer provided by #Sunny Pelletier worked for me with a mashup of #Frank's answer in our Java setup.
So for me, I wanted a solution that worked for our local docker setup and for any of our AWS environments that have active profiles and other env vars set in our environment via the CDK.
I first started with a simple Configuration POJO to setup my properties outside the spring.data.mongo.* paradigm. You don't have to do this and can just let Spring handle it as it normally does to create the MongoClient.
My default local dev application.yml and corresponding config class.
mongo:
user: mongo
password: mongo
host: localhost
port: 27017
database: my-service
#Data
#Configuration
#ConfigurationProperties(prefix = "mongo")
public class MongoConnectConfig {
private int port;
private String host;
private String user;
private String database;
private String password;
}
Then, I created two AbstractMongoClientConfiguration child classes; one for local and one for non-local. The key here is that I didn't create my own MongoClient. The reason is because I want all the good Spring Boot initialization stuff that you get with the framework. For example, the auto-registration of all the converters and such.
Instead, I leveraged the customization hook provided by AbstractMongoClientConfiguration.configureClientSettings(MongoClientSettings.Builder builder) to then aggregate the custom settings like the .pem piece.
The other part is that I leveraged profiles to enable/disable the configurations to make it "seamless" for local developers; we don't use any profiles other than default for local development so it's easier to get setup without having to "know" so much from the start.
import com.mongodb.ConnectionString;
import com.mongodb.MongoClientSettings;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;
import org.springframework.data.mongodb.config.AbstractMongoClientConfiguration;
import org.springframework.data.mongodb.repository.config.EnableMongoRepositories;
#Slf4j
#Configuration
#RequiredArgsConstructor
#Profile({"!dev && !qa && !prod"})
#EnableMongoRepositories(basePackages = "co.my.data.repositories")
public class LocalDevMongoConfig extends AbstractMongoClientConfiguration {
private final MongoConnectConfig config;
#Override
public String getDatabaseName() {
return config.getDatabase();
}
#Override
protected void configureClientSettings(MongoClientSettings.Builder builder) {
log.info("Applying Local Dev MongoDB Configuration");
builder.applyConnectionString(new ConnectionString(getConnectionString()));
}
//mongodb://${mongo.user}:${mongo.password}#${mongo.host}:${mongo.port}/${mongo.database}?authSource=admin
private String getConnectionString() {
return String.format("mongodb://%s:%s#%s:%s/%s?authSource=admin",
config.getUser(),
config.getPassword(),
config.getHost(),
config.getPort(),
config.getDatabase()
);
}
}
import com.mongodb.ConnectionString;
import com.mongodb.MongoClientSettings;
import lombok.RequiredArgsConstructor;
import lombok.SneakyThrows;
import lombok.extern.slf4j.Slf4j;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;
import org.springframework.core.io.ClassPathResource;
import org.springframework.data.mongodb.config.AbstractMongoClientConfiguration;
import org.springframework.data.mongodb.repository.config.EnableMongoRepositories;
import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManagerFactory;
import java.io.ByteArrayInputStream;
import java.io.File;
import java.nio.file.Files;
import java.security.KeyStore;
import java.security.cert.CertificateFactory;
import java.util.Arrays;
import java.util.stream.Collectors;
#Slf4j
#Configuration
#RequiredArgsConstructor
#Profile({"dev || qa || prod"})
#EnableMongoRepositories(basePackages = "co.my.data.repositories")
public class DocumentDbMongoConfig extends AbstractMongoClientConfiguration {
private final MongoConnectConfig config;
#Override
public String getDatabaseName() {
return config.getDatabase();
}
#SneakyThrows
#Override
protected void configureClientSettings(MongoClientSettings.Builder builder) {
log.info("Applying AWS DocumentDB Configuration");
builder.applyConnectionString(new ConnectionString(getConnectionString()));
var endOfCertificateDelimiter = "-----END CERTIFICATE-----";
File resource = new ClassPathResource("certs/rds-combined-ca-bundle.pem").getFile();
String pemContents = new String(Files.readAllBytes(resource.toPath()));
var allCertificates = Arrays.stream(pemContents
.split(endOfCertificateDelimiter))
.filter(line -> !line.isBlank())
.map(line -> line + endOfCertificateDelimiter)
.collect(Collectors.toUnmodifiableList());
var certificateFactory = CertificateFactory.getInstance("X.509");
var keyStore = KeyStore.getInstance(KeyStore.getDefaultType());
// This allows us to use an in-memory key-store
keyStore.load(null);
for (int i = 0; i < allCertificates.size(); i++) {
var certString = allCertificates.get(i);
var caCert = certificateFactory.generateCertificate(new ByteArrayInputStream(certString.getBytes()));
keyStore.setCertificateEntry(String.format("AWS-certificate-%s", i), caCert);
}
var trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
trustManagerFactory.init(keyStore);
var sslContext = SSLContext.getInstance("TLS");
sslContext.init(null, trustManagerFactory.getTrustManagers(), null);
builder.applyToSslSettings(ssl -> {
ssl.enabled(true).context(sslContext);
});
}
/**
* Partly based on the AWS Console "Connectivity & security " section in the DocumentDB Cluster View.
* Since we register the pem above, we don't need to add the ssl & sslCAFile piece
* mongodb://${user}:${password}#${host}:${port}/?replicaSet=rs0&readPreference=secondaryPreferred&retryWrites=false
*/
private String getConnectionString() {
return String.format("mongodb://%s:%s#%s:%s/%s?replicaSet=rs0&readPreference=secondaryPreferred&retryWrites=false",
config.getUser(),
config.getPassword(),
config.getHost(),
config.getPort(),
config.getDatabase()
);
}
}
Lastly, we place the rds-combined-ca-bundle.pem in the src/main/resources/certs/ folder.
Side Notes:
Again, I believe you should be able to get away with using the default spring.data* properties and your MongoClient should have used them.
Ignore the #SneakyThrows here, I just did that for code brevity purposes, handle your checked exceptions as you see fit.
I guess we can see why Kotlin syntax can be considered "cleaner" huh? :)
I can confirm the solution provided by #Barath allows you to secure the AWS DocumentDB TLS connection inside the Java application itself. This is a much cleaner approach compared to the one described by AWS on https://docs.aws.amazon.com/documentdb/latest/developerguide/connect_programmatically.html which requires you to run a script on your server which is more complicated and difficult for automated deploys etc.
To further set up the connection itself in the Spring application I used the following #Configuration class, which allows you to connect to a local MongoDB for testing during development, and the AWS one once deployed with settings from the properties file.
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;
import org.springframework.core.convert.converter.Converter;
import org.springframework.data.mongodb.config.AbstractMongoClientConfiguration;
import org.springframework.data.mongodb.core.convert.MongoCustomConversions;
import org.springframework.data.mongodb.repository.config.EnableMongoRepositories;
#Configuration
#EnableMongoRepositories(basePackages = "YOUR.PACKAGE.WITH.repository")
public class MongoDbConfig extends AbstractMongoClientConfiguration {
#Value("${spring.profiles.active}")
private String activeProfile;
#Value("${mongodb.host:localhost}")
private String dbUri;
#Value("${mongodb.port:27017}")
private int dbPort;
#Value("${mongodb.database.name:YOUR_DOCUMENTDB_NAME}")
private String dbName;
#Value("${mongodb.username:}")
private String dbUser;
#Value("${mongodb.password:}")
private String dbPassword;
#Override
public String getDatabaseName() {
return dbName;
}
#Override
public MongoClient mongoClient() {
ConnectionString connectionString = new ConnectionString(getConnectionString());
MongoClientSettings mongoClientSettings = MongoClientSettings.builder()
.applyConnectionString(connectionString)
.build();
return MongoClients.create(mongoClientSettings);
}
private String getConnectionString() {
if (activeProfile.contains("local")) {
return String.format("mongodb://%s:%s/%s", dbUri, dbPort, dbName);
}
return String.format("mongodb://%s:%s#%s:%s/%s?ssl=true&replicaSet=rs0&readpreference=secondaryPreferred&retrywrites=false",
dbUser, dbPassword, dbUri, dbPort, dbName);
}
}
I actually faced the same issue as you did, but now AWS uses rds-combined-ca-bundle.pem which combines together many certificates into one.
If you don't want to create a trust-store using their outdated documentation, you can do it yourself and have the rds-combined-ca-bundle.pem into your application generating the key-store at runtime.
I managed to get this to work with this code sample. This has been tested with spring:2.4, mongo-driver: 4.1.1 and documentDB using mongo 4.0 compatibility.
val endOfCertificateDelimiter = "-----END CERTIFICATE-----"
// rds-combined-ca-bundle.pem contains more than one certificate. We need to add them all to the trust-store independantly.
val allCertificates = ClassPathResource("certificates/rds-combined-ca-bundle.pem").file.readText()
.split(endOfCertificateDelimiter)
.filter { it.isNotBlank() }
.map { it + endOfCertificateDelimiter }
val certificateFactory = CertificateFactory.getInstance("X.509")
val keyStore = KeyStore.getInstance(KeyStore.getDefaultType())
keyStore.load(null) // This allows us to use an in-memory key-store
allCertificates.forEachIndexed { index, certificate ->
val caCert = certificateFactory.generateCertificate(certificate.byteInputStream()) as X509Certificate
keyStore.setCertificateEntry("AWS-certificate-$index", caCert)
}
val trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm())
trustManagerFactory.init(keyStore)
val sslContext = SSLContext.getInstance("TLS")
sslContext.init(null, trustManagerFactory.trustManagers, null)
builder.applyToSslSettings {
it.enabled(true)
.context(sslContext)
}
Here is a solution that worked for me just call the setSslProperties method before you connect to your documentdb.
/**
* This method sets the SSL properties which specify the key store file, its type and password.
*
* #throws Exception
*/
private static void setSslProperties() throws Exception {
System.setProperty("javax.net.ssl.trustStore", createKeyStoreFile());
System.setProperty("javax.net.ssl.trustStoreType", KEY_STORE_TYPE);
System.setProperty("javax.net.ssl.trustStorePassword", DEFAULT_KEY_STORE_PASSWORD);
}
/**
* This method returns the path of the Key Store File needed for the SSL verification during the IAM Database Authentication to
* the db instance.
*
* #return
* #throws Exception
*/
private static String createKeyStoreFile() throws Exception {
return createKeyStoreFile(createCertificate()).getPath();
}
/**
* This method generates the SSL certificate.
*
* #return
* #throws Exception
*/
private static X509Certificate createCertificate() throws Exception {
final CertificateFactory certFactory = CertificateFactory.getInstance("X.509");
final ClassLoader classLoader = MyClass.class.getClassLoader();
final InputStream is = classLoader.getResourceAsStream(SSL_CERTIFICATE);
return (X509Certificate) certFactory.generateCertificate(is);
}
/**
* This method creates the Key Store File.
*
* #param rootX509Certificate - the SSL certificate to be stored in the KeyStore
* #return
* #throws Exception
*/
private static File createKeyStoreFile(final X509Certificate rootX509Certificate) throws Exception {
final File keyStoreFile = File.createTempFile(KEY_STORE_FILE_PREFIX, KEY_STORE_FILE_SUFFIX);
try (final FileOutputStream fos = new FileOutputStream(keyStoreFile.getPath())) {
final KeyStore ks = KeyStore.getInstance(KEY_STORE_TYPE, KEY_STORE_PROVIDER);
ks.load(null);
ks.setCertificateEntry("rootCaCertificate", rootX509Certificate);
ks.store(fos, DEFAULT_KEY_STORE_PASSWORD.toCharArray());
}
return keyStoreFile;
}
Here are the constants.
public static final String SSL_CERTIFICATE = "rds-ca-2019-root.pem";
public static final String KEY_STORE_TYPE = "JKS";
public static final String KEY_STORE_PROVIDER = "SUN";
public static final String KEY_STORE_FILE_PREFIX = "sys-connect-via-ssl-test-cacerts";
public static final String KEY_STORE_FILE_SUFFIX = ".jks";
public static final String DEFAULT_KEY_STORE_PASSWORD = "changeit";
Here is the link for rds-ca-2019-root.pem file place that file inder resources folder.
let me know this works for you.
Here is a sample
setSslProperties();
final MongoCredential credential = MongoCredential.createCredential(userName, mongoProps.getDatabaseName(), password.toCharArray());
final MongoClientSettings settings = MongoClientSettings.builder()
.credential(credential)
.readPreference(ReadPreference.secondaryPreferred())
.retryWrites(false)
.applyToSslSettings(builder -> builder.enabled(true))
.applyToConnectionPoolSettings(connPoolBuilder ->
ConnectionPoolSettings.builder().
maxSize(1).build())
.applyToClusterSettings(builder ->
builder.hosts(Arrays.asList(new ServerAddress(clusterEndPoint, 27017))))
.build();
mongoClient = MongoClients.create(settings);
As pointed out by #mmr25 in comments to #Barath answer, The solution only works for when service needs to only connect to documentDB. You start getting "Gettting PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested" for other http requests.
To address this issue we need to only enable sslcontext for documentdb connections. To do we can use Netty as HttpClient for mongodb connections. To enable netty we need to add following maven dependency to your spring boot project:
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative-boringssl-static</artifactId>
<version>2.0.53.Final</version>
</dependency>
and put your pem file in your resources folder and define following beans in one of the class annotated with #Configutation annotations.
#Slf4j
#Configuration
public class MongoDbConfiguration {
private static final String AWS_PUBLIC_KEY_NAME = "rds-ca-2019-root.pem";
private final String mongoConnectionUri;
private final String databaseName;
public MongoDbConfiguration(#Value("${spring.data.mongodb.uri}") String mongoConnectionUri, #Value("${spring.data.mongodb.database}") String databaseName) {
this.mongoConnectionUri = mongoConnectionUri;
this.databaseName = databaseName;
}
#Bean
#Primary
#SneakyThrows
#Profile("!default")
public MongoClient mongoClient() {
SslContext sslContext = SslContextBuilder.forClient()
.sslProvider(SslProvider.OPENSSL)
.trustManager(new ClassPathResource(AWS_PUBLIC_KEY_NAME).getInputStream())
.build();
ConnectionString connectionString = new ConnectionString(mongoConnectionUri);
return MongoClients.create(
MongoClientSettings.builder()
.applyConnectionString(connectionString)
.applyToSslSettings(builder -> {
builder.enabled((null == connectionString.getSslEnabled()) ? false : connectionString.getSslEnabled());
builder.invalidHostNameAllowed((null == connectionString.getSslInvalidHostnameAllowed()) ? false : connectionString.getSslInvalidHostnameAllowed());
})
.streamFactoryFactory(NettyStreamFactoryFactory.builder()
.sslContext(sslContext)
.build())
.build());
}
}
Import Statements:
import com.mongodb.ConnectionString;
import com.mongodb.MongoClientSettings;
import com.mongodb.client.MongoClient;
import com.mongodb.client.MongoClients;
import com.mongodb.connection.netty.NettyStreamFactoryFactory;
import io.netty.handler.ssl.SslContext;
import io.netty.handler.ssl.SslContextBuilder;
import io.netty.handler.ssl.SslProvider;
import lombok.SneakyThrows;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.context.annotation.Profile;
import org.springframework.core.io.ClassPathResource;
import org.springframework.data.mongodb.MongoDatabaseFactory;
import org.springframework.data.mongodb.MongoTransactionManager;
Now you should be able to connect to your documentdb and other http connection should also work as expected.
Reference: https://www.mongodb.com/docs/drivers/java/sync/current/fundamentals/connection/tls/#customize-tls-ssl-configuration-through-the-netty-sslcontext
The Simple solution is you can remove the TLS (SSL) option in AWS, then you can remove the "ssl_ca_certs=rds-combined-ca-bundle.pem" from your connection string. But if the application required the SSL DB connectivity, then you can use the
AWS Guide

Spring UTF-8 replace chars "ě", "č", "ř" to "?"

I have problems with UTF-8 encoding.
I have a Spring-Boot REST Server. When I perform a PUT request to save an article and there are special characters in the content (like "ě", "č" or "ř") they will be replace by "?" characters. But if I edit an article using phpmyadmin and perform a GET request it returns "ě", "č" and "ř" correctly.
package cz.flay.fellcms.http;
import cz.flay.fellcms.dao.ArticlesRepository;
import cz.flay.fellcms.entities.Article;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpStatus;
import org.springframework.web.bind.annotation.*;
#CrossOrigin
#RestController
#RequestMapping(path = "/api/articles", produces = "application/json;charset=UTF-8")
public class ArticleRestController {
#Autowired
private ArticlesRepository articlesRepository;
#CrossOrigin
#GetMapping(path = "/all")
public #ResponseBody Iterable<Article> getAll(){
return articlesRepository.getAll();
}
#CrossOrigin
#GetMapping(path = "/newest")
public #ResponseBody Iterable<Article> getNewest(){
return articlesRepository.getNewest();
}
#CrossOrigin
#PutMapping(path = "/save")
public #ResponseBody HttpStatus saveArticle(#RequestBody Article article) {
articlesRepository.save(article);
return HttpStatus.OK;
}
#CrossOrigin
#GetMapping(path = "/get")
public #ResponseBody Article getArticle(#RequestParam int id) {
return articlesRepository.findOne(id);
}
}
I tried to log an article out to the console. Then the special charactes are displayed correctly.
I tried a lot of things that I found in other discussions but nothing helped me.
Thanks for any help !
The way it worked for me was to enter the following in the mysql configuration file (my.cnf)
[mysql]
default-character-set=utf8
[mysqld]
character-set-server=utf8
The mysql configuration file is usually in /etc/mysql/my.cnf in linux (I tried it in Ubuntu 14.04). If it doesn't exist there you can create the file using sudo touch /etc/mysql/my.cnf; then edit the file.
Edit:
For Win Users: File is located in /mysql/bin/my.ini
Try this in your properties file;
spring.datasource.url = jdbc:mysql://localhost:3306/db_name?useUnicode=yes&characterEncoding=UTF-8

Running cucumber-groovy features against a spring boot api

I've been attempting to get cucumber-groovy working with spring-boot, but it's not been going well. I get the error org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost:8080/applicants": Connection refused; nested exception is java.net.ConnectException: Connection refused which seems to indicate that it's hitting the endpoint, but that the service isn't running.
I've read that I need to have a cucumber.xml file, but my project is not using any xml config, it's all annotations, so instead I've got this:
package support
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
#Configuration
#ComponentScan(basePackages = "com.base.package")
public class CucumberConfiguration {}
I've added it to the World, but this seems to be the wrong way of doing things (i.e. I don't know how to add an annotation on groovy step defs).
package support
import com.thing.app.Application
import org.junit.runner.RunWith
import org.springframework.boot.test.SpringApplicationContextLoader
import org.springframework.boot.test.WebIntegrationTest
import org.springframework.test.context.ContextConfiguration
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner
import org.springframework.test.context.web.WebAppConfiguration
import static cucumber.api.groovy.Hooks.*
//#RunWith(SpringJUnit4ClassRunner)
//#ContextConfiguration(classes = Application, loader = SpringApplicationContextLoader)
//#WebAppConfiguration
//#WebIntegrationTest
#ContextConfiguration(classes = CucumberConfiguration)
public class AbstractTest {
}
World() {
new AbstractTest()
}
Before() {}
After() {}
I left in my other annotations to kind of show what I've done so far. None of it has worked.
I've also tried setting up an AbstractDefs class as seen here https://github.com/jakehschwartz/spring-boot-cucumber-example/tree/master/src/test/java/demo, but that also hasn't worked, mostly because I'm not using the cucumber-java style of things, but instead the cucumber-groovy style, which doesn't use step definition classes.
Edit: Just discovered I was doing things wrong by having an env.groovy, I'm used to the ruby cucumber, so I'm having trouble finding all the little problems. Still am having the same issue though, I don't know how to execute in a Spring context.
You can instantiate Spring test context with io.cucumber.spring.SpringFactory and register adapter in World to allow groovy script has access to Spring beans:
env.groovy:
#ContextConfiguration(classes = TestConfiguration, loader = SpringBootContextLoader)
class CucumberContextConfiguration {
}
//adapter bypassing World properties to Spring context
class SpringFactoryWorldAdapter {
private final SpringFactory factory;
SpringFactoryWorldAdapter(SpringFactory factory) {
this.factory = factory;
}
#Override
Object getProperty(String s) {
return factory.testContextManager.getContext().getBean(s);
}
}
def factory; //Keep state to prevent repeated context initialization
World { args ->
if (factory == null) {
factory = new SpringFactory()
factory.addClass(CucumberContextConfiguration)
factory.start()
}
new SpringFactoryWorldAdapter(factory)
}

Resources