hapi-fhir-cli upload examples command giving an error - hl7-fhir

I'm trying to upload test data to the local JPA fhir server using hapi-fhir-cli. But while uploading the resources, I'm getting the following error.
2020-09-03 17:33:26.486 [main] INFO c.u.f.c.ExampleDataUploader 1 good references
2020-09-03 17:33:26.511 [main] INFO c.u.f.c.ExampleDataUploader Final bundle: 18 entries
2020-09-03 17:33:26.527 [main] INFO c.u.f.c.ExampleDataUploader About to upload 11 examples in a transaction, 2 remaining
2020-09-03 17:33:26.637 [main] INFO c.u.f.c.ExampleDataUploader Final bundle: 62 KB
2020-09-03 17:33:26.641 [main] INFO c.u.f.c.ExampleDataUploader Uploading bundle to server: http://127.0.0.1:8080/hapi-fhir-jpaserver/fhir
2020-09-03 17:33:26.960 [main] ERROR c.u.f.c.ExampleDataUploader Failed to upload bundle:HTTP 0: Failed to retrieve the server metadata statement during client initialization. URL used was http://127.0.0.1:8080/hapi-fhir-jpaserver/fhir/metadata
Even if I replace http://127.0.0.1:8080/hapi-fhir-jpaserver/fhir/metadata by public hapi fhir test server, i.e. http://hapi.fhir.org/baseR4, I'm getting the same error. I'm getting the above error after running the following hapi-fhir-cli command.
hapi-fhir-5.1.0-cli>hapi-fhir-cli upload-examples -t http://127.0.0.1:8080/hapi-fhir-jpaserver/fhir -v dstu2 -l 40
If I change the version to dstu3 or r4, I get the validation error, i.e. bundle type=transaction not found in valueset defined at hl7 website, even if it's defined.
Does anyone have any idea about both of these errors? Any help would be appreciated. Thanks.

Can you show where you are creating your client code (please).
But the two suggestions I have:
Are you setting the FhirContext to the right version? Do you need a bearer token?
//import ca.uhn.fhir.context.FhirContext;
private FhirContext getContext() {
return FhirContext.forR4();
}
Note, creating the context (the call to "forR4" is expensive, so you want to minimize the number of times you call that).
//// import ca.uhn.fhir.rest.client.api.IGenericClient;
private IGenericClient generateIGenericClient(FhirContext fhirContext, GenericClientCreateArgs createArgs) {
IGenericClient client = fhirContext.newRestfulGenericClient(createArgs.getServerBase());
if (null != createArgs && createArgs.getBearerToken().isPresent()) {
String token = createArgs.getBearerToken().get();
if (StringUtils.isNotBlank(token)) {
BearerTokenAuthInterceptor authInterceptor = new BearerTokenAuthInterceptor(token);
client.registerInterceptor(authInterceptor);
}
}
return client;
}
and my "args" holder class:
import java.util.Optional;
public final class GenericClientCreateArgs {
private String serverBase;
private Optional<String> bearerToken;
public String getServerBase() {
return serverBase;
}
public void setServerBase(String serverBase) {
this.serverBase = serverBase;
}
public Optional<String> getBearerToken() {
return bearerToken;
}
public void setBearerToken(Optional<String> bearerToken) {
this.bearerToken = bearerToken;
}
}

Related

How to configure Spring Boot App to connect to Axon Server and register event handler

First, I have a spring boot application using axon 4.5.5 libraries connecting to server 4.2.4. Is this supported?
This spring boot app is suppose to listen to several events coming from the (main app emitted to the) axon server and here is my sprint boot axon client configuration in application.yml below.
Second, this application connects to the axon server but fails to handle any events causing all the said events to be blacklisted. I have trace it to the event handler registration is probably causing the issue. We are using the EventStore as the StreamableMessageSource when calling registerTrackingEventProcessor().
Do you guys have any ideas why the registered event handlers are not firing? I can see the spring boot app is connected to the axon server on the dashboard as well as the main app that fires the events. I can also see the fired events when searching it in the dashboard and their blacklisting in the axon server log. So my guess it is the configuration causing issues.
Here is my library versions (from pom.xml):
Version 4.5.5
axon-test
axon-spring-boot-autoconfigure
axon-spring-boot-starter
axon-modelling
axon-metrics
axon-messaging
axon-eventsourcing
Version 2.6.0
spring-boot-starter-web
Here is my application.yml axon fragment:
axon:
axonserver:
client-id: reporting-etl-client
component-name: reporting-etl
query-threads: ${AXON_QUERY_THREADS_MAX:50}
servers: ${AXON_AXONSERVER_SERVERS:axonserver}
serializer:
events: jackson
and AxonConfig.java:
package com.fedeee.reporting.config;
import com.fedeee.reporting.axon.events.EventHandlerProjector;
import com.thoughtworks.xstream.XStream;
import org.axonframework.axonserver.connector.query.AxonServerQueryBus;
import org.axonframework.config.EventProcessingConfigurer;
import org.axonframework.config.ProcessingGroup;
import org.axonframework.eventhandling.EventBus;
import org.axonframework.eventhandling.TrackingEventProcessorConfiguration;
import org.axonframework.eventhandling.gateway.DefaultEventGateway;
import org.axonframework.eventhandling.gateway.EventGateway;
import org.axonframework.queryhandling.DefaultQueryGateway;
import org.axonframework.queryhandling.QueryBus;
import org.axonframework.queryhandling.QueryGateway;
import org.axonframework.serialization.AbstractXStreamSerializer;
import org.axonframework.serialization.Serializer;
import org.axonframework.serialization.xml.XStreamSerializer;
import org.reflections.Reflections;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.util.Assert;
import java.util.Set;
#Configuration
public class AxonConfig {
private static final Logger LOG = LoggerFactory.getLogger(AxonConfig.class);
/**
* Correctly configuring the XStream serializer to avoid security warnings.
*/
#Autowired
public void configureXStream(Serializer serializer) {
if (serializer instanceof AbstractXStreamSerializer) {
XStream xStream = ((XStreamSerializer) serializer).getXStream();
XStream.setupDefaultSecurity(xStream);
xStream.allowTypesByWildcard(new String[] {"com.fedeee.pkg.api.events.**", "org.axonframework.**"});
}
}
/**
*
* #param configurer
* #param context
*/
#Autowired
public void configure(EventProcessingConfigurer configurer, ApplicationContext context) {
LOG.info("Setting up TrackingEventProcessors for threads, batch size and other configurations..."
+ " annotated with #ProcessingGroup...");
// find classes in the com.fedeee.* package that has methods annotated with #ProcessingGroup to configure
Reflections reflections = new Reflections("com.fedeee.reporting.axon.events");
Set<Class<?>> annotatedClasses = reflections.getTypesAnnotatedWith(ProcessingGroup.class);
// Configure each identified class
annotatedClasses.stream().forEach(annotatedClass -> {
// Locate the appropriate spring bean to get appropriate values from each one.
String beanName = annotatedClass.getName().substring(annotatedClass.getName().lastIndexOf(".") + 1);
beanName = beanName.substring(0,1).toLowerCase() + beanName.substring(1);
Object projObj = context.getBean(beanName);
if (projObj instanceof EventHandlerProjector) {
EventHandlerProjector projector = (EventHandlerProjector) projObj;
LOG.info("Configuring EventHandlerProjector Bean '{}' with maxThreads: {} and batchSize: {}.",
beanName, projector.getMaxThreads(), projector.getBatchSize());
ProcessingGroup pgAnnotation = annotatedClass.getAnnotation(ProcessingGroup.class);
String processingGroup = pgAnnotation.value();
configurer.registerTrackingEventProcessor(
processingGroup,
org.axonframework.config.Configuration::eventStore,
conf -> TrackingEventProcessorConfiguration.forParallelProcessing(projector.getMaxThreads())
.andBatchSize(projector.getBatchSize())
).registerHandlerInterceptor(processingGroup, configuration -> new EventHandlerLoggingInterceptor());
// Enable logging for EventHandlers
LOG.info(".. '{}' successfully configured with processing group '{}'.", beanName, processingGroup);
} else {
LOG.info(".. '{}' failed to configure with any processing group.", beanName);
}
});
// TODO: handle tracking event processor initialization. See the axon mailing list thread:
// *****************************************************************************************************************
// https://groups.google.com/forum/#!topic/axonframework/eyw0rRiSzUw
// In that thread there is a discussion about properly initializing the token store to avoid recreating query models.
// I still need to understand more about this...
// *****************************************************************************************************************
}
// #Autowired
// public void configureErrorHandling(
// EventProcessingConfigurer configurer, ErrorHandler errorHandler
// ) {
// configurer.registerDefaultListenerInvocationErrorHandler(c -> errorHandler);
// }
#Autowired
public void registerInterceptors(QueryBus queryBus) {
Assert.notNull(queryBus, "Invalid configuration, queryBus is null!");
if (AxonServerQueryBus.class.isAssignableFrom(queryBus.getClass())) {
queryBus.registerHandlerInterceptor(InterceptorSupport.authorizationHandlerInterceptor());
}
}
#Bean
public QueryGateway queryGateway(QueryBus queryBus) {
return DefaultQueryGateway.builder().queryBus(queryBus).build();
}
#Bean
public EventGateway eventGateway(EventBus eventBus) {
return DefaultEventGateway.builder().eventBus(eventBus).build();
}
}
Here is the EventHandlerProjector.java:
package com.fedeee.reporting.axon.events;
/**
* Defines a contract to ensure we specify the number of threads and batch size to be allowed by the event
*/
public interface EventHandlerProjector {
/**
* Specifies the number of records per batch to be handled by the tracking processor.
* #return
*/
public default Integer getBatchSize() {
return 1;
}
/**
* Specifies the maximumn number of threads the tracking processor can be specified to use.
* #return
*/
public default Integer getMaxThreads() {
return 1;
}
}
And finally the event handler class:
package com.fedeee.reporting.axon.events.pkg;
import com.fedeee.api.UserInfo;
import com.fedeee.pkg.api.events.*;
import com.fedeee.reporting._shared.utils.MdcAutoClosable;
import com.fedeee.reporting.axon.events.EventHandlerProjector;
import com.fedeee.reporting.recover.packageevents.PackageCreatedDto;
import com.fedeee.reporting.recover.packageevents.RecoverPackageCreatedRequest;
import com.fedeee.reporting.recover.packageevents.RecoverPackageEditedDto;
import com.fedeee.reporting.recover.packageevents.RecoverPackageEventsService;
import com.fedeee.reporting.recover.translators.PackageEventTranslator;
import com.fedeee.reporting.recover.translators.UserInfoTranslator;
import com.fedeee.reporting.recover.user.RecoverUserDto;
import org.axonframework.config.ProcessingGroup;
import org.axonframework.eventhandling.EventHandler;
import org.axonframework.eventhandling.Timestamp;
import org.axonframework.messaging.annotation.MetaDataValue;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;
import java.time.Instant;
import static com.fedeee.api.AxonMessageMetadataKeys.USER_INFO;
/**
* Event TrackingProcessor for PackageRecord-based Events handled by Axon.
*
* Configurations can be provided with the given batchSize and maxThreads options via .env or docker-compose.
*
* IMPORTANT! <code>AxonConfig</code> looks for the <em>#ProcessingGroup</em> annotation to set everything up properly.
*/
#ProcessingGroup(value = "Package-Record")
#Component
public class PackageRecordProjector implements EventHandlerProjector {
#Value("${reporting-etl.tp-batch-size.package-record:1}")
private Integer batchSize;
#Value("${reporting-etl.tp-max-threads.package-record:5}")
private Integer maxThreads;
private RecoverPackageEventsService recoverPackageEventsService;
#Autowired
public PackageRecordProjector(RecoverPackageEventsService recoverPackageEventsService) {
super();
this.recoverPackageEventsService = recoverPackageEventsService;
}
private final Logger LOG = LoggerFactory.getLogger(this.getClass());
/**
* Event handler to handle packages created in Recover.
*
* This replaces the REST endpoint exposed and used by Recover in RecoverPackageEventsController.created().
*
* #param event
* #param occurrenceInstant
* #param userInfo
*/
#EventHandler
void on(PackageCreated event, #Timestamp Instant occurrenceInstant, #MetaDataValue(USER_INFO) UserInfo userInfo) {
try (MdcAutoClosable mdc = new MdcAutoClosable()) {
mdcInit(event, userInfo, mdc);
LOG.info("Handling PackageCreated event...");
PackageCreatedDto createdDto = PackageEventTranslator.from(event, occurrenceInstant, userInfo);
RecoverUserDto recoverUserDto = UserInfoTranslator.from(userInfo);
RecoverPackageCreatedRequest request = new RecoverPackageCreatedRequest(createdDto, recoverUserDto);
/* Once we are ready, comment this in and make appropriate changes to RecoverPackageEventsController to
disallow duplication via the REST endpoint. (There are comments in there already. :) )
*/
recoverPackageEventsService.save(request);
LOG.info("Finished handling PackageCreated event.");
} catch (Exception e) {
LOG.info("An Exception has been thrown : ", e);
throw e;
}
}
#EventHandler
void on(PackageTypeCorrected event, #Timestamp Instant occurrenceInstant, #MetaDataValue(USER_INFO) UserInfo userInfo) {
try (MdcAutoClosable mdc = new MdcAutoClosable()) {
mdcInit(event, userInfo, mdc);
// TODO: not implemented (Recover and Reporting-ETL)
LOG.info("Finished handling PackageTypeCorrected event.");
} catch (Exception e) {
LOG.info("An Exception has been thrown : ", e);
throw e;
}
}
#EventHandler
void on(PackageDeleted event, #Timestamp Instant occurrenceInstant, #MetaDataValue(USER_INFO) UserInfo userInfo) {
try (MdcAutoClosable mdc = new MdcAutoClosable()) {
mdcInit(event, userInfo, mdc);
// TODO: not implemented (Reporting-ETL)
LOG.info("Finished handling PackageDeleted event.");
} catch (Exception e) {
LOG.info("An Exception has been thrown : ", e);
throw e;
}
}
// TODO: not implemented (Recover and Reporting-ETL)
// #EventHandler
// void on(PackageIntegrated event, #Timestamp Instant occurrenceInstant, #MetaDataValue(USER_INFO) UserInfo userInfo) {
// try (MdcAutoClosable mdc = new MdcAutoClosable()) {
// mdcInit(event, userInfo, mdc);
// } catch (Exception e) {
// LOG.info("An Exception has been thrown : ", e);
// throw e;
// }
// }
#EventHandler
void on(DataCaptureStarted event, #Timestamp Instant occurrenceInstant, #MetaDataValue(USER_INFO) UserInfo userInfo) {
try (MdcAutoClosable mdc = new MdcAutoClosable()) {
mdcInit(event, userInfo, mdc);
RecoverPackageEditedDto editedDto = PackageEventTranslator.from(event, occurrenceInstant, userInfo);
/* Once we are ready, comment this in and make appropriate changes to RecoverPackageEventsController to
disallow duplication via the REST endpoint. (There are comments in there already. :) )
*/
recoverPackageEventsService.save(editedDto);
LOG.info("Finished handling DataCaptureStarted event.");
} catch (Exception e) {
LOG.info("An Exception has been thrown : ", e);
throw e;
}
}
#EventHandler
void on(DataCaptureEnded event, #Timestamp Instant occurrenceInstant, #MetaDataValue(USER_INFO) UserInfo userInfo) {
try (MdcAutoClosable mdc = new MdcAutoClosable()) {
mdcInit(event, userInfo, mdc);
RecoverPackageEditedDto editedDto = PackageEventTranslator.from(event, occurrenceInstant, userInfo);
/* Once we are ready, comment this in and make appropriate changes to RecoverPackageEventsController to
disallow duplication via the REST endpoint. (There are comments in there already. :) )
*/
recoverPackageEventsService.update(event.getPackageId(), editedDto);
LOG.info("Finished handling DataCaptureEnded event.");
} catch (Exception e) {
LOG.info("An Exception has been thrown : ", e);
throw e;
}
}
#Override public Integer getBatchSize() {
return batchSize;
}
#Override public Integer getMaxThreads() {
return maxThreads;
}
private void mdcInit(PackageEvent event, UserInfo userInfo, MdcAutoClosable mdc) {
mdc.put("PackageId", event.getPackageId());
mdc.put("UserId", userInfo.getUserId());
LOG.info("Handling package record event: {}", event);
}
}
Here is the logs for today 2023/01/27...
.
.
.
2023-01-27 17:19:32.924 DEBUG 8 --- [MessageBroker-4] i.a.a.message.command.CommandCache : Checking timed out commands
2023-01-27 17:19:37.924 DEBUG 8 --- [MessageBroker-4] i.a.axonserver.message.query.QueryCache : Checking timed out queries
2023-01-27 17:19:37.924 DEBUG 8 --- [MessageBroker-3] i.a.a.message.command.CommandCache : Checking timed out commands
2023-01-27 17:19:40.299 DEBUG 8 --- [ool-5-thread-16] i.a.axonserver.grpc.PlatformService : Registered client : ClientComponent{client='reporting-etl-client', component='reporting-etl', context='default'}
2023-01-27 17:19:40.299 INFO 8 --- [ool-5-thread-16] i.a.a.logging.TopologyEventsLogger : Application connected: reporting-etl, clientId = reporting-etl-client, context = default
2023-01-27 17:19:40.299 DEBUG 8 --- [ool-5-thread-16] i.a.a.c.version.ClientVersionsCache : Version update received from client reporting-etl-client.default to version 4.4.
2023-01-27 17:19:40.332 INFO 8 --- [ool-5-thread-15] i.a.a.message.event.EventDispatcher : Starting tracking event processor for : - 209301
2023-01-27 17:19:42.925 DEBUG 8 --- [MessageBroker-7] i.a.axonserver.message.query.QueryCache : Checking timed out queries
2023-01-27 17:19:42.925 DEBUG 8 --- [MessageBroker-3] i.a.a.message.command.CommandCache : Checking timed out commands
.
.
.
2023-01-27 18:56:08.163 DEBUG 8 --- [MessageBroker-1] i.a.a.message.command.CommandCache : Checking timed out commands
2023-01-27 18:56:08.163 DEBUG 8 --- [MessageBroker-7] i.a.axonserver.message.query.QueryCache : Checking timed out queries
2023-01-27 18:56:09.242 DEBUG 8 --- [pool-5-thread-9] i.a.a.message.command.CommandDispatcher : Dispatch com.fedeee.pkg.api.commands.StartDataCapture to: recover-api-client.default
2023-01-27 18:56:09.257 DEBUG 8 --- [ool-5-thread-10] i.a.a.message.command.CommandDispatcher : Sending response to: io.axoniq.axonserver.message.command.CommandInformation#17b317ff
2023-01-27 18:56:09.294 DEBUG 8 --- [st-dispatcher-2] i.a.a.grpc.GrpcQueryDispatcherListener : Send request io.axoniq.axonserver.grpc.SerializedQuery#158b1b9a, with priority: 0
2023-01-27 18:56:09.294 DEBUG 8 --- [st-dispatcher-2] i.a.a.grpc.GrpcQueryDispatcherListener : Remaining time for message: 72152808-ca89-4565-82dd-2675e52686e2 - 3600000ms
2023-01-27 18:56:09.300 DEBUG 8 --- [pool-5-thread-5] i.a.axonserver.message.query.QueryCache : Remove messageId 72152808-ca89-4565-82dd-2675e52686e2
2023-01-27 18:56:09.300 DEBUG 8 --- [pool-5-thread-5] i.a.a.message.query.QueryDispatcher : No (more) information for 72152808-ca89-4565-82dd-2675e52686e2 on completed
2023-01-27 18:56:09.300 DEBUG 8 --- [st-dispatcher-2] i.a.a.grpc.GrpcQueryDispatcherListener : Send request io.axoniq.axonserver.grpc.SerializedQuery#6e93a34b, with priority: 0
2023-01-27 18:56:09.301 DEBUG 8 --- [st-dispatcher-2] i.a.a.grpc.GrpcQueryDispatcherListener : Remaining time for message: 53aa1974-4012-452a-b451-2957003e4b9f - 3599999ms
2023-01-27 18:56:09.306 DEBUG 8 --- [pool-5-thread-4] i.a.axonserver.message.query.QueryCache : Remove messageId 53aa1974-4012-452a-b451-2957003e4b9f
2023-01-27 18:56:09.306 DEBUG 8 --- [pool-5-thread-4] i.a.a.message.query.QueryDispatcher : No (more) information for 53aa1974-4012-452a-b451-2957003e4b9f on completed
2023-01-27 18:56:09.319 DEBUG 8 --- [ool-5-thread-13] i.a.a.m.q.s.handler.DirectUpdateHandler : SubscriptionQueryResponse for subscription Id 55748e24-e26f-4863-a5d0-a4eeff43da69 send to client.
2023-01-27 18:56:10.509 DEBUG 8 --- [st-dispatcher-2] i.a.a.grpc.GrpcQueryDispatcherListener : Send request io.axoniq.axonserver.grpc.SerializedQuery#fd8d154, with priority: 0
2023-01-27 18:56:10.510 DEBUG 8 --- [st-dispatcher-2] i.a.a.grpc.GrpcQueryDispatcherListener : Remaining time for message: d2d91224-557f-4735-933b-e4195e7e42f9 - 3599999ms
2023-01-27 18:56:10.514 DEBUG 8 --- [ool-5-thread-15] i.a.axonserver.message.query.QueryCache : Remove messageId d2d91224-557f-4735-933b-e4195e7e42f9
2023-01-27 18:56:10.514 DEBUG 8 --- [ool-5-thread-15] i.a.a.message.query.QueryDispatcher : No (more) information for d2d91224-557f-4735-933b-e4195e7e42f9 on completed
2023-01-27 18:56:13.163 DEBUG 8 --- [MessageBroker-2] i.a.a.message.command.CommandCache : Checking timed out commands
2023-01-27 18:56:13.163 DEBUG 8 --- [MessageBroker-5] i.a.axonserver.message.query.QueryCache : Checking timed out queries
.
.
.
As you might be running into known bugs, it's better to use 4.6 of server and framework, that said, it should likely work since the server API is pretty stable.
Could you also share the com.fedeee.reporting.axon.events.EventHandlerProjector class? Has that class #EventHandler annotated methods? Are there more classes with #EventHandler annotated methods?
I assume you are using Spring Boot and the Axon Framework starter, is did indeed the case?

Spring Data Redis - Custom error handling

I have a Spring Boot rest api using Spring Data Redis.
My aim is to make redis 'transparent' for the application. Is redis is there, use it, if not do not throw any exception and runs normally.
My first step was to create a custom cache error handler
#Configuration
public class CustomCacheConfiguration extends CachingConfigurerSupport {
#Override
public CacheErrorHandler errorHandler() {
return new CustomCacheErrorHandler();
}
}
and
#Slf4j
public class CustomCacheErrorHandler implements CacheErrorHandler {
#Override
public void handleCacheGetError(RuntimeException exception, Cache cache, Object key) {
log.error(String.format("Unable to get data from cache : %s", exception.getMessage()));
}
#Override
public void handleCachePutError(RuntimeException exception, Cache cache, Object key, Object value) {
log.error(String.format("Unable to put data into the cache : %s", exception.getMessage()));
}
#Override
public void handleCacheEvictError(RuntimeException exception, Cache cache, Object key) {
log.error(String.format("Unable to evict data from cache : %s", exception.getMessage()));
}
#Override
public void handleCacheClearError(RuntimeException exception, Cache cache) {
log.error(String.format("Unable to clear data from cache : %s", exception.getMessage()));
}
}
It is working fine. If redis is not up no more exception, just an error in the logs and the api keeps working normally.
But it remains one problem. If redis is going down after the application is started. In that case :
the api hangs when trying to lookup in the cache an takes more than 2 minutes to respond in place of less than few seconds.
in the logs (not even every minute) I can see :
2022-07-01 10:01:11.636 INFO 7004 --- [xecutorLoop-1-1] i.l.core.protocol.ConnectionWatchdog : Reconnecting, last destination was localhost/<unresolved>:6379
2022-07-01 10:01:11.638 WARN 7004 --- [ioEventLoop-4-7] i.l.core.protocol.ConnectionWatchdog : Cannot reconnect to [localhost/<unresolved>:6379]: Connection refused: no further information: localhost/127.0.0.1:6379
Is there a way of resolving point 1 and avoiding point 2 ?

Netty how to test Handler which uses Remote Address of a client

I have a Netty TCP Server with Spring Boot 2.3.1 with the following handler :
#Slf4j
#Component
#RequiredArgsConstructor
#ChannelHandler.Sharable
public class QrReaderProcessingHandler extends ChannelInboundHandlerAdapter {
private final CarParkPermissionService permissionService;
private final Gson gson = new Gson();
private String remoteAddress;
#Override
public void channelActive(ChannelHandlerContext ctx) {
ctx.fireChannelActive();
remoteAddress = ctx.channel().remoteAddress().toString();
if (log.isDebugEnabled()) {
log.debug(remoteAddress);
}
ctx.writeAndFlush("Your remote address is " + remoteAddress + ".\r\n");
}
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) {
log.info("CLIENT_IP: {}", remoteAddress);
String stringMsg = (String) msg;
log.info("CLIENT_REQUEST: {}", stringMsg);
String lowerCaseMsg = stringMsg.toLowerCase();
if (RequestType.HEARTBEAT.containsName(lowerCaseMsg)) {
HeartbeatRequest heartbeatRequest = gson.fromJson(stringMsg, HeartbeatRequest.class);
log.debug("heartbeat request: {}", heartbeatRequest);
HeartbeatResponse response = HeartbeatResponse.builder()
.responseCode("ok")
.build();
ctx.writeAndFlush(response + "\n\r");
}
}
Request DTO:
#Data
#Builder
#NoArgsConstructor
#AllArgsConstructor
public class HeartbeatRequest {
private String messageID;
}
Response DTO:
#Data
#Builder
#NoArgsConstructor
#AllArgsConstructor
public class HeartbeatResponse {
private String responseCode;
}
Logic is quite simple. Only I have to know the IP address of the client.
I need to test it as well.
I have been looking for many resources for testing handlers for Netty, like
Testing Netty with EmbeddedChannel
How to unit test netty handler
However, it didn't work for me.
For EmbeddedChannel I have following error - Your remote address is embedded.
Here is code:
#ActiveProfiles("test")
#RunWith(MockitoJUnitRunner.class)
public class ProcessingHandlerTest_Embedded {
#Mock
private PermissionService permissionService;
private EmbeddedChannel embeddedChannel;
private final Gson gson = new Gson();
private ProcessingHandler processingHandler;
#Before
public void setUp() {
processingHandler = new ProcessingHandler(permissionService);
embeddedChannel = new EmbeddedChannel(processingHandler);
}
#Test
public void testHeartbeatMessage() {
// given
HeartbeatRequest heartbeatMessage = HeartbeatRequest.builder()
.messageID("heartbeat")
.build();
HeartbeatResponse response = HeartbeatResponse.builder()
.responseCode("ok")
.build();
String request = gson.toJson(heartbeatMessage).concat("\r\n");
String expected = gson.toJson(response).concat("\r\n");
// when
embeddedChannel.writeInbound(request);
// then
Queue<Object> outboundMessages = embeddedChannel.outboundMessages();
assertEquals(expected, outboundMessages.poll());
}
}
Output:
22:21:29.062 [main] INFO handler.ProcessingHandler - CLIENT_IP: embedded
22:21:29.062 [main] INFO handler.ProcessingHandler - CLIENT_REQUEST: {"messageID":"heartbeat"}
22:21:29.067 [main] DEBUG handler.ProcessingHandler - heartbeat request: HeartbeatRequest(messageID=heartbeat)
org.junit.ComparisonFailure:
<Click to see difference>
However, I don't know how to do exact testing for such a case.
Here is a snippet from configuration:
#Bean
#SneakyThrows
public InetSocketAddress tcpSocketAddress() {
// for now, hostname is: localhost/127.0.0.1:9090
return new InetSocketAddress("localhost", nettyProperties.getTcpPort());
// for real client devices: A05264/172.28.1.162:9090
// return new InetSocketAddress(InetAddress.getLocalHost(), nettyProperties.getTcpPort());
}
#Component
#RequiredArgsConstructor
public class QrReaderChannelInitializer extends ChannelInitializer<SocketChannel> {
private final StringEncoder stringEncoder = new StringEncoder();
private final StringDecoder stringDecoder = new StringDecoder();
private final QrReaderProcessingHandler readerServerHandler;
private final NettyProperties nettyProperties;
#Override
protected void initChannel(SocketChannel socketChannel) {
ChannelPipeline pipeline = socketChannel.pipeline();
// Add the text line codec combination first
pipeline.addLast(new DelimiterBasedFrameDecoder(1024 * 1024, Delimiters.lineDelimiter()));
pipeline.addLast(new ReadTimeoutHandler(nettyProperties.getClientTimeout()));
pipeline.addLast(stringDecoder);
pipeline.addLast(stringEncoder);
pipeline.addLast(readerServerHandler);
}
}
How to test handler with IP address of a client?
Two things that could help:
Do not annotate with #ChannelHandler.Sharable if your handler is NOT sharable. This can be misleading. Remove unnecessary state from handlers. In your case you should remove the remoteAddress member variable and ensure that Gson and CarParkPermissionService can be reused and are thread-safe.
"Your remote address is embedded" is NOT an error. It actually is the message written by your handler onto the outbound channel (cf. your channelActive() method)
So it looks like it could work.
EDIT
Following your comments here are some clarifications regarding the second point. I mean that:
your code making use of EmbeddedChannel is almost correct. There is just a misunderstanding on the expected results (assert).
To make the unit test successful, you just have either:
to comment this line in channelActive(): ctx.writeAndFlush("Your remote ...")
or to poll the second message from Queue<Object> outboundMessages in testHeartbeatMessage()
Indeed, when you do this:
// when
embeddedChannel.writeInbound(request);
(1) You actually open the channel once, which fires a channelActive() event. You don't have a log in it but we see that the variable remoteAddress is not null afterwards, meaning that it was assigned in the channelActive() method.
(2) At the end of the channelActive() method, you eventually already send back a message by writing on the channel pipeline, as seen at this line:
ctx.writeAndFlush("Your remote address is " + remoteAddress + ".\r\n");
// In fact, this is the message you see in your failed assertion.
(3) Then the message written by embeddedChannel.writeInbound(request) is received and can be read, which fires a channelRead() event. This time, we see this in your log output:
22:21:29.062 [main] INFO handler.ProcessingHandler - CLIENT_IP: embedded
22:21:29.062 [main] INFO handler.ProcessingHandler - CLIENT_REQUEST: {"messageID":"heartbeat"}
22:21:29.067 [main] DEBUG handler.ProcessingHandler - heartbeat request: HeartbeatRequest(messageID=heartbeat)
(4) At the end of channelRead(ChannelHandlerContext ctx, Object msg), you will then send a second message (the expected one):
HeartbeatResponse response = HeartbeatResponse.builder()
.responseCode("ok")
.build();
ctx.writeAndFlush(response + "\n\r");
Therefore, with the following code of your unit test...
Queue<Object> outboundMessages = embeddedChannel.outboundMessages();
assertEquals(expected, outboundMessages.poll());
... you should be able to poll() two messages:
"Your remote address is embedded"
"{ResponseCode":"ok"}
Does it make sense for you?

Spring: Can't map YAML to list of objects

I have a yaml file with errors and error messages. I'm attempting to load them into a class, RefreshErrors within a common jar used by my Spring Boot microservices. The common jar does have Spring Boot on the classpath, but there is no SpringBootApplication - it's simply included by other applications. So that may be why I'm not getting any values in RefreshErrors?
Here's the yaml, which is in application.yml, located under src/main/resources:
refreshErrorsProperties:
refreshErrors:
-
recordStatus: "Error"
errorCode: "502 Bad Gateway"
errorMessage: "AWS service is tempoarily not available. ESS has been notified. Please try refreshing again after the service becomes available."
-
recordStatus: "Error"
errorCode: "503 Service Temporarily Unavailable"
errorMessage: "AWS service is tempoarily not available. ESS has been notified. Please try refreshing again after the service becomes available."
-
...
And here's my configuration for the class I want the yaml mapped to:
#Data
#Component
#ConfigurationProperties(prefix="refreshErrors")
public class RefreshErrorsProperties {
private List<RefreshErrors> refreshErrors = new ArrayList<>();
}
The RefreshErrors class:
#Data
#AllArgsConstructor
public class RefreshErrors {
private String errorCode;
private String recordStatus;
private String errorMessage;
}
I autowire RefreshErrors in another class (see below) and I do get an object reference, but all the values inside (like errorCode etc) are empty. Any help would be appreciated!
#Autowired
public void setRefreshErrorsProperties(RefreshErrorsProperties refreshErrorsProperties) {
RefreshJobDetailHelper.refreshErrorsProperties = refreshErrorsProperties;
}
...
RefreshErrors error;
if (exception != null) {
String fullException = ExceptionUtils.getStackTrace(exception);
error = refreshErrorsProperties.getRefreshErrors().stream()
.filter(f -> fullException.contains(f.getErrorCode()))
.findAny()
.orElseGet((Supplier<? extends RefreshErrors>) new RefreshErrors("Error", ERROR.getValue(), "An error has occurred while refreshing this record. ESS has been notified. Please try refreshing again after the issue has been resolved."));
} else {
error = new RefreshErrors("No data", ERROR_NO_DATA.getValue(), "Data is not available in the source for this product, domain, and effective date.");
}
The issue here is with the prefix #ConfigurationProperties(prefix = "refreshErrors"). It should be like #ConfigurationProperties, with no prefix, the prefix is required when you need to map properties under the prefix, but refreshErrors is the property you want to map and not the prefix. The below code works fine.
I have refactored a bit :
properties file:
refresh-errors:
-
record-status: "Error"
error-code: "502 Bad Gateway"
error-message: "AWS service is tempoarily not available. ESS has been notified. Please try refreshing again after the service becomes available."
-
record-status: "Error"
error-code: "503 Service Temporarily Unavailable"
error-message: "AWS service is tempoarily not available. ESS has been notified. Please try refreshing again after the service becomes available."
PropertiesMapperClass :
#Data
#Component
#ConfigurationProperties
public class RefreshErrorsProperties {
private List<RefreshErrors> refreshErrors = new ArrayList<>();
#Data
public static class RefreshErrors {
private String errorCode;
private String recordStatus;
private String errorMessage;
}
}
if you don't change your annotation,the yaml file node refreshErrors's attribute should not be a list,
the yaml file should be
refresh-errors:
record-status: "Error"
error-code: "502 Bad Gateway"
error-message: "AWS service ..."
which may not be what you want,
if you don't change your yaml file,
the code should be like Mohit Singh shows

Attaching AWS documentDB to Spring Boot application

I've recently tried using the new AWS DocumentDB service as my DB in a Spring application.
The cluster has been created in the same VPC as the EKS on which I deploy my application. Security groups allow connections between all nodes in the VPC.
AWS exposes a mongo URI like this for my DB cluster:
mongodb://<my-user>:<insertYourPassword>#<my-cluster-endpoint>:27017/?ssl_ca_certs=rds-combined-ca-bundle.pem&replicaSet=rs0
My question:
How do I make my Spring code work with this kind of connection?
I have tried adding the followig to my application.properties file:
spring.data.mongodb.uri=mongodb://<my-user>:<insertYourPassword>#<my-cluster-endpoint>:27017/admin?ssl_ca_certs=rds-combined-ca-bundle.pem&replicaSet=rs00
spring.data.mongodb.database=admin
server.ssl.key-store=classpath:rds-combined-ca-bundle.pem
And placing the PEM file in /src/main/resources
However the code still fails to connect to the DB cluster.
I get this message as an error: No server chosen by com.mongodb.client.internal.MongoClientDelegate
Followed by a Exception in monitor thread while connecting to server ...
And finally a timeout exception: com.mongodb.MongoSocketReadTimeoutException: Timeout while receiving message
It looks kind of like a security group issue but I have no problem connecting with mongo shell from the same EC2 running the Spring application Pod.
Any ideas?
As mentioned in the documentation,
By design, you access Amazon DocumentDB (with MongoDB compatibility) resources from an Amazon EC2 instance within the same Amazon VPC as the Amazon DocumentDB resources. However, suppose that your use case requires that you or your application access your Amazon DocumentDB resources from outside the cluster's Amazon VPC. In that case, you can use SSH tunneling (also known as "port forwarding") to access your Amazon DocumentDB resources.
Connect from outside VPC
Your Amazon DocumentDB cluster should be running in your default virtual private cloud (VPC). To interact with your Amazon DocumentDB cluster, you must launch an Amazon Elastic Compute Cloud (Amazon EC2) instance into your default VPC, in the same AWS Region where you created your Amazon DocumentDB cluster.
Follow the guide to connect to the cluster
AWS DocumentDB cluster
GitHub Reference: spring-boot-aws-documentdb
Update:
To connect through SSL, use below logic by setting SSL_CERTIFICATE pointing to aws region specific intermediate certificate.
This can be downloaded from SSL certs and copy it to base directory.
Alternatively, you can provide absolute path to the variable SSL_CERTIFICATE.
private static final String SSL_CERTIFICATE = "rds-ca-2015-us-east-1.pem";
private static final String KEY_STORE_TYPE = "JKS";
private static final String KEY_STORE_PROVIDER = "SUN";
private static final String KEY_STORE_FILE_PREFIX = "sys-connect-via-ssl-test-cacerts";
private static final String KEY_STORE_FILE_SUFFIX = ".jks";
private static final String DEFAULT_KEY_STORE_PASSWORD = "changeit";
public static void main(String[] args) {
SSLContextHelper.setSslProperties();
SpringApplication.run(Application.class, args);
}
protected static class SSLContextHelper{
/**
* This method sets the SSL properties which specify the key store file, its type and password:
* #throws Exception
*/
private static void setSslProperties() {
try {
System.setProperty("javax.net.ssl.trustStore", createKeyStoreFile());
} catch (Exception e) {
e.printStackTrace();
}
System.setProperty("javax.net.ssl.trustStoreType", KEY_STORE_TYPE);
System.setProperty("javax.net.ssl.trustStorePassword", DEFAULT_KEY_STORE_PASSWORD);
}
private static String createKeyStoreFile() throws Exception {
return createKeyStoreFile(createCertificate()).getPath();
}
/**
* This method generates the SSL certificate
* #return
* #throws Exception
*/
private static X509Certificate createCertificate() throws Exception {
CertificateFactory certFactory = CertificateFactory.getInstance("X.509");
URL url = new File(SSL_CERTIFICATE).toURI().toURL();
if (url == null) {
throw new Exception();
}
try (InputStream certInputStream = url.openStream()) {
return (X509Certificate) certFactory.generateCertificate(certInputStream);
}
}
/**
* This method creates the Key Store File
* #param rootX509Certificate - the SSL certificate to be stored in the KeyStore
* #return
* #throws Exception
*/
private static File createKeyStoreFile(X509Certificate rootX509Certificate) throws Exception {
File keyStoreFile = File.createTempFile(KEY_STORE_FILE_PREFIX, KEY_STORE_FILE_SUFFIX);
try (FileOutputStream fos = new FileOutputStream(keyStoreFile.getPath())) {
KeyStore ks = KeyStore.getInstance(KEY_STORE_TYPE, KEY_STORE_PROVIDER);
ks.load(null);
ks.setCertificateEntry("rootCaCertificate", rootX509Certificate);
ks.store(fos, DEFAULT_KEY_STORE_PASSWORD.toCharArray());
}
return keyStoreFile;
}
}
connection output:
019-01-17 13:33:22.316 INFO 3598 --- [onaws.com:27017] org.mongodb.driver.cluster : Canonical address mongodb.cktoiipu3bbd.us-east-1.docdb.amazonaws.com:27017 does not match server address. Removing mongodb.cluster-cktoiipu3bbd.us-east-1.docdb.amazonaws.com:27017 from client view of cluster
2019-01-17 13:33:22.401 INFO 3598 --- [onaws.com:27017] org.mongodb.driver.connection : Opened connection [connectionId{localValue:2}] to mongodb.cktoiipu3bbd.us-east-1.docdb.amazonaws.com:27017
2019-01-17 13:33:22.403 INFO 3598 --- [onaws.com:27017] org.mongodb.driver.cluster : Monitor thread successfully connected to server with description ServerDescription{address=mongodb.cktoiipu3bbd.us-east-1.docdb.amazonaws.com:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 6, 0]}, minWireVersion=0, maxWireVersion=6, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=null, roundTripTimeNanos=2132149, setName='rs0', canonicalAddress=mongodb.cktoiipu3bbd.us-east-1.docdb.amazonaws.com:27017, hosts=[mongodb.cktoiipu3bbd.us-east-1.docdb.amazonaws.com:27017], passives=[], arbiters=[], primary='mongodb.cktoiipu3bbd.us-east-1.docdb.amazonaws.com:27017', tagSet=TagSet{[]}, electionId=7fffffff0000000000000001, setVersion=null, lastWriteDate=Thu Jan 17 13:33:21 UTC 2019, lastUpdateTimeNanos=516261208876}
2019-01-17 13:33:22.406 INFO 3598 --- [onaws.com:27017] org.mongodb.driver.cluster : Discovered replica set primary mongodb.cktoiipu3bbd.us-east-1.docdb.amazonaws.com:27017
2019-01-17 13:33:22.595 INFO 3598 --- [ main] com.barath.app.CustomerService : Saving the customer with customer details com.barath.app.Customer#6c130c45
2019-01-17 13:33:22.912 INFO 3598 --- [ main] org.mongodb.driver.connection : Opened connection [connectionId{localValue:3}] to mongodb.cktoiipu3bbd.us-east-1.docdb.amazonaws.com:27017
2019-01-17 13:33:23.936 INFO 3598 --- [ main] pertySourcedRequestMappingHandlerMapping : Mapped URL path [/v2/api-docs] onto method [public org.springframework.http.ResponseEntity<springfox.documentation.spring.web.json.Json> springfox.documentation.swagger2.web.Swagger2Controller.getDocumentation(java.lang.String,javax.servlet.http.HttpServletRequest)]
The answer provided by #Sunny Pelletier worked for me with a mashup of #Frank's answer in our Java setup.
So for me, I wanted a solution that worked for our local docker setup and for any of our AWS environments that have active profiles and other env vars set in our environment via the CDK.
I first started with a simple Configuration POJO to setup my properties outside the spring.data.mongo.* paradigm. You don't have to do this and can just let Spring handle it as it normally does to create the MongoClient.
My default local dev application.yml and corresponding config class.
mongo:
user: mongo
password: mongo
host: localhost
port: 27017
database: my-service
#Data
#Configuration
#ConfigurationProperties(prefix = "mongo")
public class MongoConnectConfig {
private int port;
private String host;
private String user;
private String database;
private String password;
}
Then, I created two AbstractMongoClientConfiguration child classes; one for local and one for non-local. The key here is that I didn't create my own MongoClient. The reason is because I want all the good Spring Boot initialization stuff that you get with the framework. For example, the auto-registration of all the converters and such.
Instead, I leveraged the customization hook provided by AbstractMongoClientConfiguration.configureClientSettings(MongoClientSettings.Builder builder) to then aggregate the custom settings like the .pem piece.
The other part is that I leveraged profiles to enable/disable the configurations to make it "seamless" for local developers; we don't use any profiles other than default for local development so it's easier to get setup without having to "know" so much from the start.
import com.mongodb.ConnectionString;
import com.mongodb.MongoClientSettings;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;
import org.springframework.data.mongodb.config.AbstractMongoClientConfiguration;
import org.springframework.data.mongodb.repository.config.EnableMongoRepositories;
#Slf4j
#Configuration
#RequiredArgsConstructor
#Profile({"!dev && !qa && !prod"})
#EnableMongoRepositories(basePackages = "co.my.data.repositories")
public class LocalDevMongoConfig extends AbstractMongoClientConfiguration {
private final MongoConnectConfig config;
#Override
public String getDatabaseName() {
return config.getDatabase();
}
#Override
protected void configureClientSettings(MongoClientSettings.Builder builder) {
log.info("Applying Local Dev MongoDB Configuration");
builder.applyConnectionString(new ConnectionString(getConnectionString()));
}
//mongodb://${mongo.user}:${mongo.password}#${mongo.host}:${mongo.port}/${mongo.database}?authSource=admin
private String getConnectionString() {
return String.format("mongodb://%s:%s#%s:%s/%s?authSource=admin",
config.getUser(),
config.getPassword(),
config.getHost(),
config.getPort(),
config.getDatabase()
);
}
}
import com.mongodb.ConnectionString;
import com.mongodb.MongoClientSettings;
import lombok.RequiredArgsConstructor;
import lombok.SneakyThrows;
import lombok.extern.slf4j.Slf4j;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;
import org.springframework.core.io.ClassPathResource;
import org.springframework.data.mongodb.config.AbstractMongoClientConfiguration;
import org.springframework.data.mongodb.repository.config.EnableMongoRepositories;
import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManagerFactory;
import java.io.ByteArrayInputStream;
import java.io.File;
import java.nio.file.Files;
import java.security.KeyStore;
import java.security.cert.CertificateFactory;
import java.util.Arrays;
import java.util.stream.Collectors;
#Slf4j
#Configuration
#RequiredArgsConstructor
#Profile({"dev || qa || prod"})
#EnableMongoRepositories(basePackages = "co.my.data.repositories")
public class DocumentDbMongoConfig extends AbstractMongoClientConfiguration {
private final MongoConnectConfig config;
#Override
public String getDatabaseName() {
return config.getDatabase();
}
#SneakyThrows
#Override
protected void configureClientSettings(MongoClientSettings.Builder builder) {
log.info("Applying AWS DocumentDB Configuration");
builder.applyConnectionString(new ConnectionString(getConnectionString()));
var endOfCertificateDelimiter = "-----END CERTIFICATE-----";
File resource = new ClassPathResource("certs/rds-combined-ca-bundle.pem").getFile();
String pemContents = new String(Files.readAllBytes(resource.toPath()));
var allCertificates = Arrays.stream(pemContents
.split(endOfCertificateDelimiter))
.filter(line -> !line.isBlank())
.map(line -> line + endOfCertificateDelimiter)
.collect(Collectors.toUnmodifiableList());
var certificateFactory = CertificateFactory.getInstance("X.509");
var keyStore = KeyStore.getInstance(KeyStore.getDefaultType());
// This allows us to use an in-memory key-store
keyStore.load(null);
for (int i = 0; i < allCertificates.size(); i++) {
var certString = allCertificates.get(i);
var caCert = certificateFactory.generateCertificate(new ByteArrayInputStream(certString.getBytes()));
keyStore.setCertificateEntry(String.format("AWS-certificate-%s", i), caCert);
}
var trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
trustManagerFactory.init(keyStore);
var sslContext = SSLContext.getInstance("TLS");
sslContext.init(null, trustManagerFactory.getTrustManagers(), null);
builder.applyToSslSettings(ssl -> {
ssl.enabled(true).context(sslContext);
});
}
/**
* Partly based on the AWS Console "Connectivity & security " section in the DocumentDB Cluster View.
* Since we register the pem above, we don't need to add the ssl & sslCAFile piece
* mongodb://${user}:${password}#${host}:${port}/?replicaSet=rs0&readPreference=secondaryPreferred&retryWrites=false
*/
private String getConnectionString() {
return String.format("mongodb://%s:%s#%s:%s/%s?replicaSet=rs0&readPreference=secondaryPreferred&retryWrites=false",
config.getUser(),
config.getPassword(),
config.getHost(),
config.getPort(),
config.getDatabase()
);
}
}
Lastly, we place the rds-combined-ca-bundle.pem in the src/main/resources/certs/ folder.
Side Notes:
Again, I believe you should be able to get away with using the default spring.data* properties and your MongoClient should have used them.
Ignore the #SneakyThrows here, I just did that for code brevity purposes, handle your checked exceptions as you see fit.
I guess we can see why Kotlin syntax can be considered "cleaner" huh? :)
I can confirm the solution provided by #Barath allows you to secure the AWS DocumentDB TLS connection inside the Java application itself. This is a much cleaner approach compared to the one described by AWS on https://docs.aws.amazon.com/documentdb/latest/developerguide/connect_programmatically.html which requires you to run a script on your server which is more complicated and difficult for automated deploys etc.
To further set up the connection itself in the Spring application I used the following #Configuration class, which allows you to connect to a local MongoDB for testing during development, and the AWS one once deployed with settings from the properties file.
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;
import org.springframework.core.convert.converter.Converter;
import org.springframework.data.mongodb.config.AbstractMongoClientConfiguration;
import org.springframework.data.mongodb.core.convert.MongoCustomConversions;
import org.springframework.data.mongodb.repository.config.EnableMongoRepositories;
#Configuration
#EnableMongoRepositories(basePackages = "YOUR.PACKAGE.WITH.repository")
public class MongoDbConfig extends AbstractMongoClientConfiguration {
#Value("${spring.profiles.active}")
private String activeProfile;
#Value("${mongodb.host:localhost}")
private String dbUri;
#Value("${mongodb.port:27017}")
private int dbPort;
#Value("${mongodb.database.name:YOUR_DOCUMENTDB_NAME}")
private String dbName;
#Value("${mongodb.username:}")
private String dbUser;
#Value("${mongodb.password:}")
private String dbPassword;
#Override
public String getDatabaseName() {
return dbName;
}
#Override
public MongoClient mongoClient() {
ConnectionString connectionString = new ConnectionString(getConnectionString());
MongoClientSettings mongoClientSettings = MongoClientSettings.builder()
.applyConnectionString(connectionString)
.build();
return MongoClients.create(mongoClientSettings);
}
private String getConnectionString() {
if (activeProfile.contains("local")) {
return String.format("mongodb://%s:%s/%s", dbUri, dbPort, dbName);
}
return String.format("mongodb://%s:%s#%s:%s/%s?ssl=true&replicaSet=rs0&readpreference=secondaryPreferred&retrywrites=false",
dbUser, dbPassword, dbUri, dbPort, dbName);
}
}
I actually faced the same issue as you did, but now AWS uses rds-combined-ca-bundle.pem which combines together many certificates into one.
If you don't want to create a trust-store using their outdated documentation, you can do it yourself and have the rds-combined-ca-bundle.pem into your application generating the key-store at runtime.
I managed to get this to work with this code sample. This has been tested with spring:2.4, mongo-driver: 4.1.1 and documentDB using mongo 4.0 compatibility.
val endOfCertificateDelimiter = "-----END CERTIFICATE-----"
// rds-combined-ca-bundle.pem contains more than one certificate. We need to add them all to the trust-store independantly.
val allCertificates = ClassPathResource("certificates/rds-combined-ca-bundle.pem").file.readText()
.split(endOfCertificateDelimiter)
.filter { it.isNotBlank() }
.map { it + endOfCertificateDelimiter }
val certificateFactory = CertificateFactory.getInstance("X.509")
val keyStore = KeyStore.getInstance(KeyStore.getDefaultType())
keyStore.load(null) // This allows us to use an in-memory key-store
allCertificates.forEachIndexed { index, certificate ->
val caCert = certificateFactory.generateCertificate(certificate.byteInputStream()) as X509Certificate
keyStore.setCertificateEntry("AWS-certificate-$index", caCert)
}
val trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm())
trustManagerFactory.init(keyStore)
val sslContext = SSLContext.getInstance("TLS")
sslContext.init(null, trustManagerFactory.trustManagers, null)
builder.applyToSslSettings {
it.enabled(true)
.context(sslContext)
}
Here is a solution that worked for me just call the setSslProperties method before you connect to your documentdb.
/**
* This method sets the SSL properties which specify the key store file, its type and password.
*
* #throws Exception
*/
private static void setSslProperties() throws Exception {
System.setProperty("javax.net.ssl.trustStore", createKeyStoreFile());
System.setProperty("javax.net.ssl.trustStoreType", KEY_STORE_TYPE);
System.setProperty("javax.net.ssl.trustStorePassword", DEFAULT_KEY_STORE_PASSWORD);
}
/**
* This method returns the path of the Key Store File needed for the SSL verification during the IAM Database Authentication to
* the db instance.
*
* #return
* #throws Exception
*/
private static String createKeyStoreFile() throws Exception {
return createKeyStoreFile(createCertificate()).getPath();
}
/**
* This method generates the SSL certificate.
*
* #return
* #throws Exception
*/
private static X509Certificate createCertificate() throws Exception {
final CertificateFactory certFactory = CertificateFactory.getInstance("X.509");
final ClassLoader classLoader = MyClass.class.getClassLoader();
final InputStream is = classLoader.getResourceAsStream(SSL_CERTIFICATE);
return (X509Certificate) certFactory.generateCertificate(is);
}
/**
* This method creates the Key Store File.
*
* #param rootX509Certificate - the SSL certificate to be stored in the KeyStore
* #return
* #throws Exception
*/
private static File createKeyStoreFile(final X509Certificate rootX509Certificate) throws Exception {
final File keyStoreFile = File.createTempFile(KEY_STORE_FILE_PREFIX, KEY_STORE_FILE_SUFFIX);
try (final FileOutputStream fos = new FileOutputStream(keyStoreFile.getPath())) {
final KeyStore ks = KeyStore.getInstance(KEY_STORE_TYPE, KEY_STORE_PROVIDER);
ks.load(null);
ks.setCertificateEntry("rootCaCertificate", rootX509Certificate);
ks.store(fos, DEFAULT_KEY_STORE_PASSWORD.toCharArray());
}
return keyStoreFile;
}
Here are the constants.
public static final String SSL_CERTIFICATE = "rds-ca-2019-root.pem";
public static final String KEY_STORE_TYPE = "JKS";
public static final String KEY_STORE_PROVIDER = "SUN";
public static final String KEY_STORE_FILE_PREFIX = "sys-connect-via-ssl-test-cacerts";
public static final String KEY_STORE_FILE_SUFFIX = ".jks";
public static final String DEFAULT_KEY_STORE_PASSWORD = "changeit";
Here is the link for rds-ca-2019-root.pem file place that file inder resources folder.
let me know this works for you.
Here is a sample
setSslProperties();
final MongoCredential credential = MongoCredential.createCredential(userName, mongoProps.getDatabaseName(), password.toCharArray());
final MongoClientSettings settings = MongoClientSettings.builder()
.credential(credential)
.readPreference(ReadPreference.secondaryPreferred())
.retryWrites(false)
.applyToSslSettings(builder -> builder.enabled(true))
.applyToConnectionPoolSettings(connPoolBuilder ->
ConnectionPoolSettings.builder().
maxSize(1).build())
.applyToClusterSettings(builder ->
builder.hosts(Arrays.asList(new ServerAddress(clusterEndPoint, 27017))))
.build();
mongoClient = MongoClients.create(settings);
As pointed out by #mmr25 in comments to #Barath answer, The solution only works for when service needs to only connect to documentDB. You start getting "Gettting PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested" for other http requests.
To address this issue we need to only enable sslcontext for documentdb connections. To do we can use Netty as HttpClient for mongodb connections. To enable netty we need to add following maven dependency to your spring boot project:
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative-boringssl-static</artifactId>
<version>2.0.53.Final</version>
</dependency>
and put your pem file in your resources folder and define following beans in one of the class annotated with #Configutation annotations.
#Slf4j
#Configuration
public class MongoDbConfiguration {
private static final String AWS_PUBLIC_KEY_NAME = "rds-ca-2019-root.pem";
private final String mongoConnectionUri;
private final String databaseName;
public MongoDbConfiguration(#Value("${spring.data.mongodb.uri}") String mongoConnectionUri, #Value("${spring.data.mongodb.database}") String databaseName) {
this.mongoConnectionUri = mongoConnectionUri;
this.databaseName = databaseName;
}
#Bean
#Primary
#SneakyThrows
#Profile("!default")
public MongoClient mongoClient() {
SslContext sslContext = SslContextBuilder.forClient()
.sslProvider(SslProvider.OPENSSL)
.trustManager(new ClassPathResource(AWS_PUBLIC_KEY_NAME).getInputStream())
.build();
ConnectionString connectionString = new ConnectionString(mongoConnectionUri);
return MongoClients.create(
MongoClientSettings.builder()
.applyConnectionString(connectionString)
.applyToSslSettings(builder -> {
builder.enabled((null == connectionString.getSslEnabled()) ? false : connectionString.getSslEnabled());
builder.invalidHostNameAllowed((null == connectionString.getSslInvalidHostnameAllowed()) ? false : connectionString.getSslInvalidHostnameAllowed());
})
.streamFactoryFactory(NettyStreamFactoryFactory.builder()
.sslContext(sslContext)
.build())
.build());
}
}
Import Statements:
import com.mongodb.ConnectionString;
import com.mongodb.MongoClientSettings;
import com.mongodb.client.MongoClient;
import com.mongodb.client.MongoClients;
import com.mongodb.connection.netty.NettyStreamFactoryFactory;
import io.netty.handler.ssl.SslContext;
import io.netty.handler.ssl.SslContextBuilder;
import io.netty.handler.ssl.SslProvider;
import lombok.SneakyThrows;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.context.annotation.Profile;
import org.springframework.core.io.ClassPathResource;
import org.springframework.data.mongodb.MongoDatabaseFactory;
import org.springframework.data.mongodb.MongoTransactionManager;
Now you should be able to connect to your documentdb and other http connection should also work as expected.
Reference: https://www.mongodb.com/docs/drivers/java/sync/current/fundamentals/connection/tls/#customize-tls-ssl-configuration-through-the-netty-sslcontext
The Simple solution is you can remove the TLS (SSL) option in AWS, then you can remove the "ssl_ca_certs=rds-combined-ca-bundle.pem" from your connection string. But if the application required the SSL DB connectivity, then you can use the
AWS Guide

Resources