Add list element to Spring properties at runtime - spring-boot

I have an application.yml file with the following:
topics:
input:
- name: topic1
partitions: 3
replicas: 1
- name: topic2
partitions: 6
replicas: 2
I would like to be able to update or add a new topic object at runtime.
I have tried the following for updating an existing object:
java -jar app.jar --topics.input[0].name="topicX"
and the following for adding another object to the list:
java -jar app.jar --topics.input[2].name="topicX" --topics.input[2].partitions=6 --topics.input[2].replicas=2
I am accessing the properties in the following way:
#Component
#ConfigurationProperties(prefix = "topics")
#Validated
public class TopicsConfiguration {
#NotEmpty
#NotNull
private List<TopicConfiguration> input = new ArrayList<>();
public List<TopicConfiguration> getInputTopics() {
return input;
}
public void setFacts(List<TopicConfiguration> input) {
this.input = input;
}
}
Where TopicConfiguration is just a POJO with the 3 fields listed.
When I don't try and modify any of the property objects at runtime this works exactly as I expect, however I can not update the property list at all. When I try and update an existing object I get an NPE. When I try and add a new object to the list I get:
Property: topics.input[2].name
Value: lmnop
Origin: "topics.input[2].name" from property source "commandLineArgs"
Reason: The elements [topics.input[2].name,topics.input[2].partitions,topics.input[2].replicas] were left unbound.
I would just like to know if there is any way to update or add an element to the list at runtime so that users of my project don't have to modify application.yml if they want to update this configuration.

okey so i did some research and some testing and came up with the following:
you cannot update a defined list in application.yml
if you run:
java -jar myApp.jar -my-list[2].name=foo
it will fail, because as soon as you want to pass the list the list will override the current list in application.yml and you are trying to pass in the 3rd item in the list when there is nothing at index 0 and 1.
it is not a dynamic list, it as an array.
You need to pass the entire list:
java -jar myApp.jar -my-list[0].name=foo -my-list[1].name=bar -my-list[2].name=foobar
So if you are going to pass a list in from cmd, you must always define the list from scratch.

So you want to reload your application.properties or application.yml in your Spring Boot application on the fly without having to make another code commit to another deployment? If I understood that right then here is how you can do it.
Read the these links and also google more on Spring Boot Actuator Reload Properties or Spring Scheduling. We can't provide you an out-written code to get this working but there is plenty of examples to examine and try out.
Links:
Actuator Refresh
Spring Scheduling
If I were you, I would prefer Actuator method as it avoids confusion and is cleaner.

Related

spring - why define variable values of properties in application.properties

I am trying to understand something about the following application.properties syntax in spring
some-api:
url: ${variable.url:http://localhost:8080}
I know that to get the value of the above we use (for example)
#Value("${some-api.url}")
private String url;
what's the point of declaring ${variable.url:VALUE} when I reference it with some-api.url ? where do you use this ?
also can you call this value in pom.xml ?
In your example properties file you are referring another property, like this is how your application.yml must be looking
variable:
url: http://host
some-api:
url: ${variable.url:http://localhost:8080}
and vaue after : is the default value when variable.url is not defined.
also can you call this value in pom.xml ?
No, you need some maven plugin which can read your properties file in order to do that.

Adding Custom "trace id with Alpha numeric values and spiting it out in application Log "

I am using Sleuth 2.1.3.
I want to add a custom "trace ID" as "correlation id" with alpha numeric value and want to spit in logs with spanid and parent id.
If i use below implementation for creating new custom trace id. does it get printed in logs ?
I tried below implementation but does not see any custom trace in log
https://github.com/openzipkin/zipkin-aws/blob/release-0.11.2/brave-propagation-aws/src/main/java/brave/propagation/aws/AWSPropagation.java
Tracing.newBuilder().propagationFactory(
ExtraFieldPropagation.newFactoryBuilder(B3Propagation.FACTORY)
.addField("x-vcap-request-id")
.addPrefixedFields("x-baggage-", Arrays.asList("country-code", "user-id"))
.build()
);
I tried with above code from https://cloud.spring.io/spring-cloud-sleuth/reference/html/#propagation but didnt see any custom trace id in log
You've passed in the B3Propagation.FACTORY as the implementation of the propagation factory so you're explicitly stating that you want the default B3 headers. You've said that you want some other field that is alphanumeric to be also propagated. Then in a log parsing tool you can define that you want to use your custom field as the trace id, but it doesn't mean that the deafult X-B3-TraceId field will be changed. If you want to use your custom field as trace id that Sleuth understands, you need to change the logging format and implement a different propagation factory bean.
One of the way which worked for me is
using ExtraFieldPropagation
and adding those keys in sleuth properties under propagation-keys
and whitelisted-keys
sample code
' #Autowired Tracer tracer;
Span currentSpan = tracer.nextSpan().start();
ExtraFieldPropagation.set(
"customkey", "customvalue");
sleuth:
log:
slf4j:
whitelisted-mdc-key : customkey
propagation:
tag:
enabled: true
propagation-keys : customkey '

Spring Boot - Programmatically load properties files, including YAML

I'm trying to programmatically read Spring-style properties files, including YAML-formatted files.
What I have somewhat works, but it ends up flattening the YAML and so I end up with keys that are all mixed up and when I try to map it to a specific object, it gets confused.
Here is what I have (in Kotlin):
import my.project.Properties
import java.util.Properties as JavaProperties
fun loadConfig(configFiles: List<String>): Properties =
JavaProperties().apply {
putAll(appProperties.defaultConfig)
(if (configFiles.isEmpty()) appProperties.defaultConfigFiles else configFiles)
.forEach { PropertiesLoaderUtils.fillProperties(this, FileSystemResource(it)) }
}.let {
// throws exception because it was flattened
objectMapper.convertValue(it, Properties::class.java)
}
If I have a config file like this:
a:
b:
c: 1
I would want it to end up something equivalent to this in the Java Properties:
mapOf("a" to mapOf("b" to mapOf("c" to 1)))
Instead, I just get something like this:
mapOf("c" to 1)
It totally flattens it out and throws away the hierarchy entirely.
I'd prefer not to just use a YAML parser and do it myself, because I would like to allow the "normal" Spring style property files as well.
Note: These have to be loaded at run-time because the application is actually for running against something else, so I can't just let Spring handle it automatically like a normally would.

How to create Apis on Spring, that are using up to date environment values

I want to create API, which takes part of its input from environment (urls etc..). Basically a Jar file.
I also wan't the values being auto updated when application.properties changes. When there is change, this is called:
org.springframework.cloud.endpoint.RefreshEndpoint#refresh
However, I consider it bad practice to have magic env variable keys like 'server.x.url' in the Api contract between client application and the jar. (Problem A)
That's why I'd like to use Api like this. But there's problem of old values.
public class MyC {
TheAPI theApi=null;
void MyC(){
theApi = new TheApi();
theApi.setUrl( env.get("server.x.url") );
}
doStuff() {
theApi.doStuff(); // fails as theApi has obsolete value of server.x.url, Problem B
}
So either I have ugly API contract or I get obsolete values in the API calls.
I'm sure there must be Spring way of solving this, but I cant get it to my head just now.

Spring, property file, empty values

I have configured spring security with a ldap server (but continue reading, it's not a problem if you have no knowledge about it, this is really a spring problem). All runs like a charm. Here is the line I use for that:
<ldap-server ldif="" root="" manager-dn="" manager-password="" url="" id="ldapServer" />
If I fill ldif and root attributes, it will run an embeded server:
<ldap-server ldif="classpath://ldap.ldif" root="dc=springframework,dc=org" manager-dn="" manager-password="" url="" id="ldapServer" />
If I fill other fields, it will run a distant server:
<ldap-server ldif="" root="" manager-dn="dc=admin,dc=springframeworg,dc=org" manager-password="password" url="ldap://myldapserver.com/dc=springframeworg,dc=org" id="ldapServer" />
All this stuff run correctly. Now I want to use Spring mechanism to load such parameters from a property file:
So I replace attribute values like this:
<ldap-server ldif="${ldap.ldif.path}" root="${ldap.ldif.root}" manager-dn="${ldap.server.manager.dn}" manager-password="${ldap.server.manager.password}" url="${ldap.server.url}" id="ldapServer" />
and create a property file with:
ldap.server.url=
ldap.server.manager.dn=
ldap.server.manager.password=
ldap.ldif.path=
ldap.ldif.root=
Now, the funny part of the problem. If I fill the following properties in the file:
ldap.server.url=ldap://myldapserver.com/dc=springframeworg,dc=org
ldap.server.manager.dn=dc=admin,dc=springframeworg,dc=org
ldap.server.manager.password=password
ldap.ldif.path=
ldap.ldif.root=
It runs a distant server as expected.
If I fill the property file like this:
ldap.server.url=
ldap.server.manager.dn=
ldap.server.manager.password=
ldap.ldif.path= classpath:ldap.ldif
ldap.ldif.root= dc=springframeworg,dc=org
It does not run, complaining that the ldap url is missing. But the problem is that if I change the spring configuration from:
<ldap-server ldif="${ldap.ldif.path}" root="${ldap.ldif.root}" manager-dn="${ldap.server.manager.dn}" manager-password="${ldap.server.manager.password}" url="${ldap.server.url}" id="ldapServer" />
to (by just removing the reference to the variable ${ldap.server.url})
<ldap-server ldif="${ldap.ldif.path}" root="${ldap.ldif.root}" manager-dn="${ldap.server.manager.dn}" manager-password="${ldap.server.manager.password}" url="" id="ldapServer" />
It runs !
My thoughs are that spring does not replace the attribute value with the property config one if this one is empty. But I find it strange.
Can you give me some clue to understand that ? And what's the best to do to configure my ldap server via a property file ?
EDIT: this is due to a poor design choice (look at accepted answer), an issue has been opened on jira :
https://jira.springsource.org/browse/SEC-1966
Ok, I think this is a spring security bug.
If I debug and look at the class LdapServerBeanDefinition, there is a method called "parse". Here is an extract:
public BeanDefinition parse(Element elt, ParserContext parserContext) {
String url = elt.getAttribute(ATT_URL);
RootBeanDefinition contextSource;
if (!StringUtils.hasText(url)) {
contextSource = createEmbeddedServer(elt, parserContext);
} else {
contextSource = new RootBeanDefinition();
contextSource.setBeanClassName(CONTEXT_SOURCE_CLASS);
contextSource.getConstructorArgumentValues().addIndexedArgumentValue(0, url);
}
contextSource.setSource(parserContext.extractSource(elt));
String managerDn = elt.getAttribute(ATT_PRINCIPAL);
String managerPassword = elt.getAttribute(ATT_PASSWORD);
if (StringUtils.hasText(managerDn)) {
if(!StringUtils.hasText(managerPassword)) {
parserContext.getReaderContext().error("You must specify the " + ATT_PASSWORD +
" if you supply a " + managerDn, elt);
}
contextSource.getPropertyValues().addPropertyValue("userDn", managerDn);
contextSource.getPropertyValues().addPropertyValue("password", managerPassword);
}
...
}
If I debug here, all variables (url, managerDn, managerPassword...) are not replaced by the value specified in the property file. And so, url has the value ${ldap.server.url}, managerDn has the value ${ldap.server.manager.dn} and so on.
The method parse creates a bean, a context source that will be used further. And when this bean will be used, place holders will be replaced.
Here, we got the bug. The parse method check if url is empty or not. The problem is that url is not empty here because it has the value ${ldap.server.url}. So, the parse method creates a context source as a distant server.
When the created source will be used, it will replace the ${ldap.server.url} by empty value (like specified in the property file). And....... Bug !
I don't know really how to solve this for the moment, but I now understand why it bugs ;)
I cannot explain it, but I think you can fix your problem using defaulting syntax, available since Spring 3.0.0.RC1 (see).
In the chageg log you can read: PropertyPlaceholderConfigurer supports "${myKey:myDefaultValue}" defaulting syntax
Anyway, I think that the problem is because "" is valid value, but no value in the property file don't.
I think that url="" works because url attribute is of type xs:token in spring-security XSD and empty string is converted to null (xs:token is removing any leading or trailing spaces, so "" can be recognized as no value). Maybe the value of ${ldap.server.url} is resolved as empty string and that is why you've got an error.
You can try use Spring profiles to define different configurations of ldap server (see Spring Team Blog for details about profiles)
I believe there is an issue here while using place holders. The following will most probably solve the problem:
Create a class which extends PropertyPlaceHolderConfigurer and override its method convertPropertyValue()
in the method you can return the property as empty string if you find anything other than a string which is of type LDAP url i.e. ldap://myldapserver.com/dc=springframeworg,dc=org
Also you need to configure your new specialization of class PropertyPlaceHolderConfigurer in the context file.
Hope this helps.
You can define empty String in the application.properties file as following:
com.core.estimation.stopwords=\ \

Resources