I’d like to have spring auto configuration which adds LogstashTcpSocketAppender.
What I have done:
The LogstashTcpSocketAppender was added in to the LoggerContext from the LogstashAutoConfiguration.java
#Configuration
#ConditionalOnProperty(name = "logging.logstash.url")
#RequiredArgsConstructor
public class LogstashAutoConfiguration {
#Value("${spring.application.name:null}")
private String applicationName;
#Value("${logging.logstash.url}")
private String logstashUrl;
#Bean
public LogstashTcpSocketAppender logstashAppender() {
LoggerContext loggerContext = (LoggerContext) LoggerFactory.getILoggerFactory();
LogstashTcpSocketAppender logstashTcpSocketAppender = new LogstashTcpSocketAppender();
logstashTcpSocketAppender.setName("LOGSTASH");
logstashTcpSocketAppender.setContext(loggerContext);
logstashTcpSocketAppender.addDestination(logstashUrl);
LogstashEncoder encoder = new LogstashEncoder();
encoder.setIncludeMdc(true);
encoder.getFieldNames().setLevelValue(null);
encoder.setCustomFields(String.format("{\"app_name\":\"%s\"}", applicationName));
logstashTcpSocketAppender.setEncoder(encoder);
logstashTcpSocketAppender.start();
loggerContext.getLogger(Logger.ROOT_LOGGER_NAME).addAppender(logstashTcpSocketAppender);
return logstashTcpSocketAppender;
}
}
After a while, the Spring Boot trigger a event will make the app reconfig. (for example I use Consul so I just change property in key/value storage, then spring refresh my context)
It call the initializeWithConventions in the AbstractLoggingSystem.java
then it will call the loadConfiguration in the LogbackLoggingSystem.java
then it will stopAndReset(loggerContext). here it will stop all the appenders, and it will resetAllListeners();, which will clear the all logback listeners. (so I cannot use logback listeners for addition appenders again)
Are there correct way add Appender through spring auto configuration?
How can I prevent remove LogstashTcpSocketAppender from LoggerContext when spring makes the app reconfig?
In the above Configuration class, the intent is to add an appender for Logstash and that takes care of sending the logs.
In case of an environment variable changes or the context gets refreshed, you can listen to the relevant events, then check if your Logstash appender is configured or not. Make sure you add the Logstash appender only if it is missing.
Here is the class that would do the same.
#Configuration
#ConditionalOnProperty(name = "logging.logstash.url")
public class LogstashAppenderConfiguration {
#Value("${spring.application.name:null}")
private String applicationName;
#Value("${logging.logstash.url}")
private String logstashUrl;
#EventListener(ContextRefreshedEvent.class)
public void onContextRefreshedEvent(ContextRefreshedEvent event) {
this.addLogStashAppenderIfMissing();
}
#EventListener(RefreshScopeRefreshedEvent.class)
public void onRefreshScopeRefreshedEvent(RefreshScopeRefreshedEvent event) {
this.addLogStashAppenderIfMissing();
}
#EventListener(EnvironmentChangeEvent.class)
public void onEnvironmentChangeEvent(EnvironmentChangeEvent event) {
this.addLogStashAppenderIfMissing();
}
public void addLogStashAppenderIfMissing() {
LoggerContext loggerContext = (LoggerContext) LoggerFactory.getILoggerFactory();
synchronized (this) {
if (Objects.isNull(loggerContext.getLogger(Logger.ROOT_LOGGER_NAME).getAppender("LOGSTASH"))) {
LogstashTcpSocketAppender logstashTcpSocketAppender = new LogstashTcpSocketAppender();
logstashTcpSocketAppender.setName("LOGSTASH");
logstashTcpSocketAppender.setContext(loggerContext);
logstashTcpSocketAppender.addDestination(logstashUrl);
LogstashEncoder encoder = new LogstashEncoder();
encoder.setIncludeMdc(true);
encoder.getFieldNames().setLevelValue(null);
encoder.setCustomFields(String.format("{\"app_name\":\"%s\"}", applicationName));
logstashTcpSocketAppender.setEncoder(encoder);
logstashTcpSocketAppender.start();
loggerContext.getLogger(Logger.ROOT_LOGGER_NAME).addAppender(logstashTcpSocketAppender); ;
}
}
}
}
Related
I have a number of microservices, each running in its own container in a load balanced environment. I have a need for each instance of these microservices to create a rabbitmq queue when it starts up and delete it when it stops. I have currently defined the following property in my application properties file:
config_queue: config_${PID}
My message queue listener looks like this:
public class ConfigListener {
Logger logger = LoggerFactory.getLogger(ConfigListener.class);
// https://www.programcreek.com/java-api-examples/index.php?api=org.springframework.amqp.rabbit.annotation.RabbitListener
#RabbitListener(bindings = #QueueBinding(
value = #Queue(value = "${config_queue}",
autoDelete = "true"),
exchange = #Exchange(value = AppConstants.TOPIC_CONFIGURATION,
type= ExchangeTypes.FANOUT)
))
public void configChanged(String message){
... application logic
}
}
All this works great when I run the microservice. A queue with prefix config and process id gets created and is auto deleted when I stop the service.
However, when I run this service and others in their individual docker containers, all services have the same PID and that is 1.
Does anybody have any idea how I can create specify a queue that is unique to that instance.
Thanks in advance for your help.
Use an AnonymousQueue instead:
#SpringBootApplication
public class So72030217Application {
public static void main(String[] args) {
SpringApplication.run(So72030217Application.class, args);
}
#RabbitListener(queues = "#{configQueue.name}")
public void listen(String in) {
System.out.println(in);
}
}
#Configuration
class Config {
#Bean
FanoutExchange fanout() {
return new FanoutExchange("config");
}
#Bean
Queue configQueue() {
return new AnonymousQueue(new Base64UrlNamingStrategy("config_"));
}
#Bean
Binding binding() {
return BindingBuilder.bind(configQueue()).to(fanout());
}
}
AnonymousQueues are auto-delete and use a Base64 encoded UUID in the name.
From reading the spring-boot docs, it seems like the standard way to customize the Jetty server is to implement a class like the following:
#Component
public class JettyServerCustomizer
implements WebServerFactoryCustomizer<JettyServletWebServerFactory> {
#Autowired
private ServerProperties serverProperties;
#Override
public void customize(final JettyServletWebServerFactory factory) {
factory.addServerCustomizers((server) -> {
// Customize
});
}
}
I'm specifically interested in modifying the SSLContextFactory.
Tracing through the spring-boot code, right before the customizers are called, ssl is configured:
if (getSsl() != null && getSsl().isEnabled()) {
customizeSsl(server, address);
}
for (JettyServerCustomizer customizer : getServerCustomizers()) {
customizer.customize(server);
}
customizeSsl is a private method so cannot be overridden easily:
private void customizeSsl(Server server, InetSocketAddress address) {
new SslServerCustomizer(address, getSsl(), getSslStoreProvider(), getHttp2()).customize(server);
}
One option is to create the context factory and connector ourselves in the customizer, and then overwrite the connectors on the server. This would probably work but it feels like we are re-creating a bunch of code that spring-boot is already doing just to be able to call a method on the SSLContextFactory.
It seems like if we could somehow provider our own SslServerCustomizer then we could do the custom configuration we want.
Does anyone know of a better way to do this?
On my case it works just fine as:
#SpringBootApplication
#ComponentScan(basePackages = { "org.demo.jetty.*" })
public class DemoWebApplication {
public static void main(String[] args) {
SpringApplication.run(DemoWebApplication.class, args);
}
#Bean
public ConfigurableServletWebServerFactory webServerFactory() {
JettyServletWebServerFactory factory = new JettyServletWebServerFactory();
factory.setContextPath("/demo-app");
factory.addServerCustomizers(getJettyConnectorCustomizer());
return factory;
}
private JettyServerCustomizer getJettyConnectorCustomizer() {
return server -> {
final HttpConfiguration httpConfiguration = new HttpConfiguration();
httpConfiguration.setSecureScheme("https");
httpConfiguration.setSecurePort(44333);
SslContextFactory.Server sslContextFactory = new SslContextFactory.Server();
sslContextFactory.setKeyStoreType("PKCS12");
sslContextFactory.setKeyStorePath("C:/jetty-demo/demo_cert.p12");
sslContextFactory.setKeyStorePassword("*****");
sslContextFactory.setKeyManagerPassword("****");
final HttpConfiguration httpsConfiguration = new HttpConfiguration(httpConfiguration);
httpsConfiguration.addCustomizer(new SecureRequestCustomizer());
ServerConnector httpsConnector = new ServerConnector(server,
new SslConnectionFactory(sslContextFactory, HttpVersion.HTTP_1_1.asString()),
new HttpConnectionFactory(httpsConfiguration));
httpsConnector.setPort(44333);
server.setConnectors(new Connector[] { httpsConnector });
server.setStopAtShutdown(true);
server.setStopTimeout(5_000);
};
}
}
You can define also a HTTP connector and add it to the customized section
...
ServerConnector connector = new ServerConnector(server);
connector.addConnectionFactory(new HttpConnectionFactory(httpConfiguration));
connector.setPort(8081);
server.setConnectors(new Connector[]{connector, httpsConnector});
...
I'm pretty new to spring-integration anyway I'm using it in order to receive mails and elaborate them.
I used this spring configuration class:
#Configuration
#EnableIntegration
#PropertySource(value = { "classpath:configuration.properties" }, encoding = "UTF-8", ignoreResourceNotFound = false)
public class MailReceiverConfiguration {
private static final Log logger = LogFactory.getLog(MailReceiverConfiguration.class);
#Autowired
private EmailTransformerService emailTransformerService;
// Configurazione AE
#Bean
public MessageChannel inboundChannelAE() {
return new DirectChannel();
}
#Bean(name= {"aeProps"})
public Properties aeProps() {
Properties javaMailPropertiesAE = new Properties();
javaMailPropertiesAE.put("mail.store.protocol", "imap");
javaMailPropertiesAE.put("mail.debug", Boolean.TRUE);
javaMailPropertiesAE.put("mail.auth.debug", Boolean.TRUE);
javaMailPropertiesAE.put("mail.smtp.socketFactory.fallback", "false");
javaMailPropertiesAE.put("mail.imap.socketFactory.class", "javax.net.ssl.SSLSocketFactory");
return javaMailPropertiesAE;
}
#Bean(name="mailReceiverAE")
public MailReceiver mailReceiverAE(#Autowired MailConfigurationBean mcb, #Autowired #Qualifier("aeProps") Properties javaMailPropertiesAE) throws Exception {
return ConfigurationUtil.getMailReceiver("imap://USERNAME:PASSWORD#MAILSERVER:PORT/INBOX", new BigDecimal(2), javaMailPropertiesAE);
}
#Bean
#InboundChannelAdapter( autoStartup = "true",
channel = "inboundChannelAE",
poller = {#Poller(fixedRate = "${fixed.rate.ae}",
maxMessagesPerPoll = "${max.messages.per.poll.ae}") })
public MailReceivingMessageSource pollForEmailAE(#Autowired MailReceiver mailReceiverAE) {
MailReceivingMessageSource mrms = new MailReceivingMessageSource(mailReceiverAE);
return mrms;
}
#Transformer(inputChannel = "inboundChannelAE", outputChannel = "transformerChannelAE")
public MessageBean transformitAE( MimeMessage mailMessage ) throws Exception {
// amministratore email inbox
MessageBean messageBean = emailTransformerService.transformit(mailMessage);
return messageBean;
}
#Splitter(inputChannel = "transformerChannelAE", outputChannel = "nullChannel")
public List<Message<?>> splitIntoMessagesAE(final MessageBean mb) {
final List<Message<?>> messages = new ArrayList<Message<?>>();
for (EmailFragment emailFragment : mb.getEmailFragments()) {
Message<?> message = MessageBuilder.withPayload(emailFragment.getData())
.setHeader(FileHeaders.FILENAME, emailFragment.getFilename())
.setHeader("directory", emailFragment.getDirectory()).build();
messages.add(message);
}
return messages;
}
}
So far so good.... I start my micro-service and there is this component listening on the specified mail server and mails are downloaded.
Now I have this requirement: mail server configuration (I mean the string "imap://USERNAME:PASSWORD#MAILSERVER:PORT/INBOX") must be taken from a database and it can be configurable. In any time a system administrator can change it and the mail receiver must use the new configuration.
As far as I understood I should create a new instance of MailReceiver when a new configuration is present and use it in the InboundChannelAdapter
Is there any best practice in order to do it? I found this solution: ImapMailReceiver NO STORE attempt on READ-ONLY folder (Failure) [THROTTLED];
In this solution I can inject the ThreadPoolTaskScheduler if I define it in my Configuration class; I can also inject the DirectChannel but every-time I should create a new MailReceiver and a new ImapIdleChannelAdapter without considering this WARN message I get when the
ImapIdleChannelAdapter starts:
java.lang.RuntimeException: No beanfactory at org.springframework.integration.expression.ExpressionUtils.createStandardEvaluationContext(ExpressionUtils.java:79) at org.springframework.integration.mail.AbstractMailReceiver.onInit(AbstractMailReceiver.java:403)
Is there a better way to satisfy my scenario?
Thank you
Angelo
The best way to do this is to use the Java DSL and dynamic flow registration.
Documentation here.
That way, you can unregister the old flow and register a new one, each time the configuration changes.
It will automatically handle injecting dependencies such as the bean factory.
Sample spring configuration is as below.
<int-mail:imap-idle-channel-adapter id="mailAdapter"
store-uri="imaps://${"username"}:${"password"}#imap-server:993/INBOX"
java-mail-properties="javaMailProperties"
channel="emails"
should-delete-messages="false"
should-mark-messages-as-read="true">
</int-mail:imap-idle-channel-adapter>
I wish to keep the password field encrypted in properties file and decrypt it in the code. I am not sure on how to set mailReceiver property of ImapIdleChannelAdapter to my custom version of ImapMailReceiver.
Please let me know if there is any way to do this.
All of my configurations are in XML as described above.
Above solution of adding the defifnation did not work may be I am doing something wrong. Then I tried using XML + Java configuration, as below.
#Configuration
public class EmailConfiguration {
#Bean
public ImapIdleChannelAdapter customAdapter() {
ImapIdleChannelAdapter adapter = new ImapIdleChannelAdapter(mailReceiver());
adapter.setOutputChannel(outputChannel());
adapter.setErrorChannel(errorChannel());
adapter.setAutoStartup(true);
adapter.setShouldReconnectAutomatically(true);
adapter.setTaskScheduler(taskScheduler());
return adapter;
}
#Bean
public TaskImapMailReceiver mailReceiver() {
TaskImapMailReceiver mailReceiver = new TaskImapMailReceiver("imaps://[username]:[password]#imap.googlemail.com:993/inbox");
mailReceiver.setShouldDeleteMessages(false);
mailReceiver.setShouldMarkMessagesAsRead(true);
//mailReceiver.setJavaMailProperties(javaMailProperties());
mailReceiver.setMaxFetchSize(Integer.MAX_VALUE);
return mailReceiver;
}
}
Also created empty errorChannel,outputChannel etc. I observed that Spring creates two instances one with xml config and other with java #Configuration. Where it was expected to use only java configuration. If I remove the xml config tag
then it provides sigle imap instance with my mailReceiver but runs only once does not go periodic. also does not show IMAPS logs.
Just wondering if I need to do so much to encrypt the password. Is somthing wrong with my approach.
Use Java configuration instead of XML...
#Configuration
public class MyConfigClass {
#Bean
public MyMailReceiver receiver() {
...
}
#Bean
public ImapIdleChannelAdapter adapter() {
ImapIdleChannelAdapter adapter = new ImapIdleChannelAdapter(receiver());
...
return adapter;
}
}
If you are using XML for everything else, simply add this class as a <bean/> to your XML.
EDIT
Here's an example that works fine for me...
#SpringBootApplication
public class So42298254Application {
public static void main(String[] args) {
SpringApplication.run(So42298254Application.class, args);
}
#Bean
public TestMailServer.ImapServer imapServer() {
return TestMailServer.imap(0);
}
#Bean
public ImapMailReceiver receiver() {
ImapMailReceiver imapMailReceiver = new ImapMailReceiver(imapUrl("user", "pw"));
imapMailReceiver.setHeaderMapper(new DefaultMailHeaderMapper()); // converts the MimeMessage to a String
imapMailReceiver.setUserFlag("testSIUserFlag"); // needed by the SI test server
Properties javaMailProperties = new Properties();
javaMailProperties.put("mail.debug", "true");
imapMailReceiver.setJavaMailProperties(javaMailProperties);
return imapMailReceiver;
}
private String imapUrl(String user, String pw) {
return "imap://"
+ user + ":" + pw
+ "#localhost:" + imapServer().getPort() + "/INBOX";
}
#Bean
public ImapIdleChannelAdapter adapter() {
ImapIdleChannelAdapter adapter = new ImapIdleChannelAdapter(receiver());
adapter.setOutputChannelName("handleMail");
return adapter;
}
#ServiceActivator(inputChannel = "handleMail")
public void handle(String mail, #Header(MailHeaders.FROM) Object from) {
System.out.println(mail + " from:" + from);
imapServer().resetServer(); // so we'll get the email again
}
}
My intention was to use encrypted passwords in properties files.
So I changed my approach of getting into email receiving classes. I added inherited PropertyPlaceholderConfigurer and implemented method convertPropertyValue() as below.
public class EncryptationAwarePropertyPlaceholderConfigurer extends PropertyPlaceholderConfigurer {
private static final Logger logger = LoggerFactory.getLogger(EncryptationAwarePropertyPlaceholderConfigurer.class);
#Override
protected String convertPropertyValue(String originalValue) {
if (originalValue.contains("{<ENC>}") && originalValue.contains("{</ENC>}")) {
String encryptedTaggedValue = originalValue.substring(originalValue.indexOf("{<ENC>}"), originalValue.indexOf("{</ENC>}") + 8);
String encryptedValue = originalValue.substring(originalValue.indexOf("{<ENC>}") + 7, originalValue.indexOf("{</ENC>}"));
try {
String decryptedValue = EncrypDecriptUtil.decrypt(encryptedValue);//EncrypDecriptUtil is my class for encription and decryption
originalValue = originalValue.replace(encryptedTaggedValue, decryptedValue);
} catch (GeneralSecurityException e) {
logger.error("failed to decrypt property returning original value as in properties file.", e);
}
}
return originalValue;
}
}
And changed properties file to enclose encrypted value in custuom ENC tag
as
mail.imap.task.url=imap://username:{<ENC>}encryptedPassword{</ENC>}#imap.googlemail.com:993/inbox
I am using a properties File to store some configuration properties, that are accessed this way:
#Value("#{configuration.path_file}")
private String pathFile;
Is it possible (with Spring 3) to use the same #Value annotation, but loading the properties from a database instead of a file ?
Assuming you have a table in your database stored key/value pairs:
Define a new bean "applicationProperties" - psuedo-code follows...
public class ApplicationProperties {
#AutoWired
private DataSource datasource;
public getPropertyValue(String key) {
// transact on your datasource here to fetch value for key
// SNIPPED
}
}
Inject this bean where required in your application. If you already have a dao/service layer then you would just make use of that.
Yes, you can keep your #Value annotation, and use the database source with the help of EnvironmentPostProcessor.
As of Spring Boot 1.3, we're able to use the EnvironmentPostProcessor to customize the application's Environment before application context is refreshed.
For example, create a class which implements EnvironmentPostProcessor:
public class ReadDbPropertiesPostProcessor implements EnvironmentPostProcessor {
private static final String PROPERTY_SOURCE_NAME = "databaseProperties";
private String[] CONFIGS = {
"app.version"
// list your properties here
};
#Override
public void postProcessEnvironment(ConfigurableEnvironment environment, SpringApplication application) {
Map<String, Object> propertySource = new HashMap<>();
try {
// the following db connections properties must be defined in application.properties
DataSource ds = DataSourceBuilder
.create()
.username(environment.getProperty("spring.datasource.username"))
.password(environment.getProperty("spring.datasource.password"))
.url(environment.getProperty("spring.datasource.url"))
.driverClassName("com.mysql.jdbc.Driver")
.build();
try (Connection connection = ds.getConnection();
// suppose you have a config table storing the properties name/value pair
PreparedStatement preparedStatement = connection.prepareStatement("SELECT value FROM config WHERE name = ?")) {
for (int i = 0; i < CONFIGS.length; i++) {
String configName = CONFIGS[i];
preparedStatement.setString(1, configName);
ResultSet rs = preparedStatement.executeQuery();
while (rs.next()) {
propertySource.put(configName, rs.getString("value"));
}
// rs.close();
preparedStatement.clearParameters();
}
}
environment.getPropertySources().addFirst(new MapPropertySource(PROPERTY_SOURCE_NAME, propertySource));
} catch (Throwable e) {
throw new RuntimeException(e);
}
}
}
Finally, don't forget to put your spring.factories in META-INF. An example:
org.springframework.boot.autoconfigure.EnableAutoConfiguration=
com.baeldung.environmentpostprocessor.autoconfig.PriceCalculationAutoConfig
Although not having used spring 3, I'd assume you can, if you make a bean that reads the properties from the database and exposes them with getters.