What is the best way to test that a spring application context fails to start? - spring

Question

I use the spring-boot-starter-web and spring-boot-starter-test.
Let's say I have a class for binding configuration properties:
#ConfigurationProperties(prefix = "dummy")
public class DummyProperties {
#URL
private String url;
// getter, setter ...
}
Now I want to test that my bean validation is correct. The context should fail to start (with a specfic error message) if the property dummy.value is not set or if it contains an invalid URL. The context should start if the property contains a valid URL. (The test would show that #NotNull is missing.)
A test class would look like this:
#RunWith(SpringJUnit4ClassRunner.class)
#SpringApplicationConfiguration(classes = MyApplication.class)
#IntegrationTest({ "dummy.url=123:456" })
public class InvalidUrlTest {
// my test code
}
This test would fail because the provided property is invalid. What would be the best way to tell Spring/JUnit: "yep, this error is expected". In plain JUnit tests I would use the ExpectedException.

Answer 1

The best way to test Spring application context is to use ApplicationContextRunner
It is described in Spring Boot Reference Documentation:
https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-developing-auto-configuration.html#boot-features-test-autoconfig
And there is a quick guide about it:
https://www.baeldung.com/spring-boot-context-runner
Sample usage
private static final String POSITIVE_CASE_CONFIG_FILE =
"classpath:some/path/positive-case-config.yml";
private static final String NEGATIVE_CASE_CONFIG_FILE =
"classpath:some/path/negative-case-config.yml";
#Test
void positiveTest() {
ApplicationContextRunner contextRunner = new ApplicationContextRunner()
.withInitializer(new ConfigDataApplicationContextInitializer())//1
.withInitializer(new ConditionEvaluationReportLoggingListener(LogLevel.DEBUG))//2
.withUserConfiguration(MockBeansTestConfiguration.class)//3
.withPropertyValues("spring.config.location=" + POSITIVE_CASE_CONFIG_FILE)//4
.withConfiguration(AutoConfigurations.of(BookService.class));//5
contextRunner
.run((context) -> {
Assertions.assertThat(context).hasNotFailed();//6
});
}
#Test
void negativeTest() {
ApplicationContextRunner contextRunner = new ApplicationContextRunner()
.withInitializer(new ConfigDataApplicationContextInitializer())//1
.withInitializer(new ConditionEvaluationReportLoggingListener(LogLevel.DEBUG))//2
.withUserConfiguration(MockBeansTestConfiguration.class)//3
.withPropertyValues("spring.config.location=" + NEGATIVE_CASE_CONFIG_FILE)//4
.withConfiguration(AutoConfigurations.of(BookService.class));//5
contextRunner
.run((context) -> {
assertThat(context)
.hasFailed();
assertThat(context.getStartupFailure())
.isNotNull();
assertThat(context.getStartupFailure().getMessage())
.contains("Some exception message");
assertThat(extractFailureCauseMessages(context))
.contains("Cause exception message");
});
}
private List<String> extractFailureCauseMessages(AssertableApplicationContext context) {
var failureCauseMessages = new ArrayList<String>();
var currentCause = context.getStartupFailure().getCause();
while (!Objects.isNull(currentCause)) {//7
failureCauseMessages.add(currentCause.getMessage());
currentCause = currentCause.getCause();
}
return failureCauseMessages;
}
Explanation with examples of similar definitions from Junit5 Spring Boot Test Annotations:
Triggers loading of config files like application.properties or application.yml
Logs ConditionEvaluationReport using given log level when application context fails
Provides class that specifies mock beans, ie. we have #Autowired BookRepository in our BookService and we provide mock BookRepository in MockBeansTestConfiguration. Similar to #Import({MockBeansTestConfiguration.class}) in test class and #TestConfiguration in class with mock beans in normal Junit5 Spring Boot Test
Equivalent of #TestPropertySource(properties = { "spring.config.location=" + POSITIVE_CASE_CONFIG_FILE})
Triggers spring auto configuration for given class, not direct equivalent, but it is similar to using #ContextConfiguration(classes = {BookService.class}) or #SpringBootTest(classes = {BookService.class}) together with #Import({BookService.class}) in normal test
Assertions.class from AssertJ library, there should be static import for Assertions.assertThat, but I wanted to show where this method is from
There should be static import for Objects.isNull, but I wanted to show where this method is from
MockBeansTestConfiguration class:
#TestConfiguration
public class MockBeansTestConfiguration {
private static final Book SAMPLE_BOOK = Book.of(1L, "Stanisław Lem", "Solaris", "978-3-16-148410-0");
#Bean
public BookRepository mockBookRepository() {
var bookRepository = Mockito.mock(BookRepository.class);//1
Mockito.when(bookRepository.findByIsbn(SAMPLE_BOOK.getIsbn()))//2
.thenReturn(SAMPLE_BOOK);
return bookRepository;
}
}
Remarks:
1,2. There should be static import, but I wanted to show where this method is from

Answer 2

Why is that an integration test to begin with? Why are you starting a full blown Spring Boot app for that?
This looks like unit testing to me. That being said, you have several options:
Don't add #IntegrationTest and Spring Boot will not start a web server to begin with (use #PropertySource to pass value to your test but it feels wrong to pass an invalid value to your whole test class)
You can use spring.main.web-environment=false to disable the web server (but that's silly given the point above)
Write a unit test that process that DummyProperties of yours. You don't even need to start a Spring Boot application for that. Look at our own test suite
I'd definitely go with the last one. Maybe you have a good reason to have an integration test for that?

Answer 3

I think the easiest way is:
public class InvalidUrlTest {
#Rule
public DisableOnDebug testTimeout = new DisableOnDebug(new Timeout(5, TimeUnit.SECONDS));
#Rule
public ExpectedException expected = ExpectedException.none();
#Test
public void shouldFailOnStartIfUrlInvalid() {
// configure ExpectedException
expected.expect(...
MyApplication.main("--dummy.url=123:456");
}
// other cases
}