I am currently working on a small Kafka Streams project with the goal to automatically set up a Kafka cluster and run a streams app on it. For convenience it would be very helpful if I could set the bootstrap.servers property outside of the .java file of the application, if possible even after compilation.
The documentation's only way to set properties is the following:
import java.util.Properties;
import org.apache.kafka.streams.StreamsConfig;
Properties settings = new Properties();
// Set a few key parameters
settings.put(StreamsConfig.APPLICATION_ID_CONFIG, "my-first-streams-application");
settings.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-broker1:9092");
// Any further settings
settings.put(... , ...);
But what if I don't know the hostname and port of the cluster yet when writing the code? The reason of this is that I want to create a tool that takes the configuration parameters of a Kafka cluster and a Kafka Streams application to then automatically set up all necessary VMs and services and execute the application. Someone else should be able to use this tool without having to change the Java code of the streams application.
Any ideas? The only other option I have in mind is parsing through the code and changing the property. If possible I'd like not to do that to unknown code...
Yes, you can always make the brokers and other configuration as runtime configuration. Infact that is the best practice to decouple the code with the cluster configuration since these configuration may subject to change. You can create a application.properties or application.yaml file with all the runtime args.
1.(With springboot) application.yaml example :
application-id: my-application1
bootstrap-servers: server1:9092,server2:9092
schema-registry-url: http://localhost:8081
default-key-serde: String
default-value-serde: JsonNode
auto-offset-reset: earliest
default-state-dir: /data/state
num-threads: 4
And add code to read properties in following way :
https://www.mkyong.com/spring-boot/spring-boot-configurationproperties-example/
If you are using spring boot project, you can run your jar in following way with application.yaml
java -jar myjar.jar --spring.config.location=path-to-yaml-file
2. ( Without Spring) application.properties
java -cp ... -Dmy.app.properties=/path/to/app.properties mypackage.myclass
Then you can read the System properties.
String propertiesPath = System.getProperty( "app.properties" );
final Properties myProps;
if ( propertiesPath != null ){
final FileInputStream in = new FileInputStream( propertiesPath );
try{
myProps = Properties.load( in );
}finally{
in.close( );
}
}
Related
I have spring boot micro-service with database credentials define in the application properties.
spring.datasource.url=<<url>>
spring.datasource.username=<<username>>
spring.datasource.password=<<password>>
We do not use spring data source to create the connection manually. Only Spring create the database connection with JPA.(org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaAutoConfiguration)
We only provide the application properties, but spring create the connections automatically to use with the database connection pool.
Our requirement to enhance the security without using db properties in clear text. Two possible methods.
Encrypt the database credentials
Use the AWS secret manager. (then get the credential with the application load)
For the option1, jasypt can be used, since we are just providing the properties only and do not want to create the data source manually, how to do to understand by the spring framework is the problem. If better I can get some working sample or methods.
Regarding the option-2,
first we need to define secretName.
use the secertName and get the database credentials from AWS secret manager.
update the application.properties programatically to understand by spring framework. (I need to know this step)
I need to use either option1 and option2. Mentioned the issues with each option.
What you could do is use environment variables for your properties. You can use them like this:
spring.datasource.url=${SECRET_URL}
You could then retrieve these and start your Spring process using a ProcessBuilder. (Or set the variables any other way)
I have found the solution for my problem.
We need to define org.springframework.context.ApplicationListenerin spring.factories file. It should define the required application context listener like below.
org.springframework.context.ApplicationListener=com.sample.PropsLoader
PropsLoader class is like this.
public class PropsLoader implements ApplicationListener<ApplicationEnvironmentPreparedEvent> {
#Override
public void onApplicationEvent(ApplicationEnvironmentPreparedEvent event) {
ConfigurableEnvironment environment = event.getEnvironment();
String appEnv = environment.getProperty("application.env");
//set new properties based on the application environment.
// calling other methods and depends on the enviornment and get the required value set
Properties props = new Properties();
props.put("new_property", "value");
environment.getPropertySources().addFirst(new PropertiesPropertySource("props", props));
}
}
spring.factories file should define under the resources package and META-INF
folder.
This will set the application context with new properties before loading any other beans.
Created a Spring Boot application that will need to migrate from "Local Dev" to "Test", "QA" and "Prod" environments.
Application currently uses a "application.properties" for database connectivity and Kafka configuration.
I am wanting to deploy to "Test" and realized that the properties will not work for that enviornment. After reading the ref docs, it looks like I can simply copy the application.properties file and add a new one application-test.properties, so on, and then run the standalone jar with a -Dspring.profiles.active=test and that seems to work.
But by the time I am done, that means I h ave 4 different appliction-XXXXX.properties files in the jar which may or may not be bad. I know the ultimate configuration would be to use Spring Config server, but right now we are not there with regards to this.
Can anyone validate that using multiple properties files is viable and will work for a bit, or if I am looking at th is all wrong. I do not want to have configuration on the servers in each environment, as I am thinking these mini-services should be self-contained.
Any input would be appreciated.
in a word, your configuration file should be outside your source code.
#PropertySource(value = {"classpath:system.properties"})
public class EnvironmentConfig {
#Bean
public static PropertySourcesPlaceholderConfigurer properties() {
return new PropertySourcesPlaceholderConfigurer();
}
}
Let's say it's named "system.properties", which will be uploaded to server at deployment stage under your application classpath.
I am working on a Java EE 7 application. I am using Payara micro to deploy my WAR files. Now, I need JDBC connectivity in my application, but I need to keep the database IP/username/password somewhere I can change later on, without re-uploading and deploying WAR file again.
Could anybody please tell me how can I achieve this?
EDIT:
I came accross a solution to that on SO:
https://stackoverflow.com/a/6296375/1931698
But, I am looking for a solution without all that plumbing. Inheriting DataSource just to have connection info in some external file looks like overkill.
EDIT:
Also, it would be really helpful if I can just provide a configuration panel to the user, where he / she can enter JDBC connection info. Is there a way to change that info at runtime (effectively discarding existing connection pool and creating a new one)?
Payara Micro should allow system property replacement in Datasource definitions using the syntax ${system.property.name} you can use that to define the database user name and password as well as the connect strings.
There is also environment variable support in 171.1 onwards using the syntax ${ENV=env.name} where env.name is the environment variable name.
Make use of java.utils.Properties with a seperate myproperties.properties file in your src folder (in classpath).
Properties prop = null;
try {
prop = new Properties();
InputStream inputStream = getClass().getClassLoader()
.getResourceAsStream("myProperty.properties");
if (inputStream != null) {
prop.load(inputStream);
}
} catch (IOException e) {
errorLog.error("Property file not found in classpath:
ClientChecking Class.");
}
String userName = prop.getProperty("USERNAME");
myproperties.properties file must contain:
USERNAME=user123
If you are using Apache tomcat for hosting, then you can find and edit the propertyfile from path WebContent(Root dir)/WEB-INF/classes/
This question already has answers here:
How can I reload properties file in Spring 4 using annotations?
(3 answers)
Closed 6 years ago.
I would like to maintain a list of application properties like service endpoints, application variables, etc. in a Spring application. These properties should be able to updated dynamically (possibly through an web page by system administrator).
Does spring has an inbuilt feature to accomplish this requirement?
I am not sure spring has an implementation for updating the properties file dynamically.
You can do something like reading the properties file using FileInputStream into a Properties object. Then you will be able to update the properties. Later you can write back the properties to the same file using the FileOutputStream.
// reading the existing properties
FileInputStream in = new FileInputStream("propertiesFile");
Properties props = new Properties();
props.load(in);
in.close();
// writing back the properties after updation
FileOutputStream out = new FileOutputStream("propertiesFile");
props.setProperty("property", "value");
props.store(out, null);
out.close();
Externalizing properties, take a look here
Spring loads these properties which can be configured at runtime and accessed in your application in different ways.
Add your own implementation of a PropertySource to your Environment.
Warning: Properties used by #ConfigurationProperties and #Value annotations are only read once on application startup, so changing the actual property values at runtime will have no effect (until restarted).
I am not sure, but check if you can make use #ConfigurationProperties of Spring boot framework.
#ConfigurationProperties(locations = "classpath:application.properties", ignoreUnknownFields = false, prefix = "spring.datasource")
You can keep this application.properties file in you classpath
Change the properties in this file without redeploying the application
Java Experts - I am just trying to explore my view. Corrections are always welcome.
Edit - I read a good examples on #PropertySource here
I'm trying to set up Spring Security to work with Spring Boot's embedded Tomcat instance. There are quite a few basic samples that do this but I'm stuck where they leave off -- they do basic authentication over HTTP (not HTTPS).
I could probably make it work if I had access to the Tomcat configuration files (server.xml) but since Spring Boot uses an embedded Tomcat instance (which is otherwise a huge convenience), I dont have access to the Tomcat configuration files (at least, not to my knowledge).
There may be an application.properties setting for this but I haven't been able to track it down. I've seen references to a server.contextPath field in application.properties that I suspect may have something to do with replacement Tomcat config files. Even if it is related, I wouldn't know where to begin anyway -- all of the Tomcat SSL instructions I've seen start with editing an existing server.xml file, not building one from scratch.
Can this be done with Spring Boot (either by somehow specifying a snippet of server.xml or through other means)? If not, what would be the simplest way to do this? I understand that I may need to exclude the Tomcat component of Spring Boot but I'd prefer to avoid that if possible.
Starting with Spring Boot 1.2, you can configure SSL using application.properties or application.yml. Here's an example for application.properties:
server.port = 8443
server.ssl.key-store = classpath:keystore.jks
server.ssl.key-store-password = secret
server.ssl.key-password = another-secret
Same thing with application.yml:
server:
port: 8443
ssl:
key-store: classpath:keystore.jks
key-store-password: secret
key-password: another-secret
Here's a link to the current reference documentation.
For external keystores, prefix with "file:"
server.ssl.key-store=file:config/keystore
It turns out that there is a way to do this, although I'm not sure I've found the 'proper' way since this required hours of reading source code from multiple projects. In other words, this might be a lot of dumb work (but it works).
First, there is no way to get at the server.xml in the embedded Tomcat, either to augment it or replace it. This must be done programmatically.
Second, the 'require_https' setting doesn't help since you can't set cert info that way. It does set up forwarding from http to https, but it doesn't give you a way to make https work so the forwarding isnt helpful. However, use it with the stuff below, which does make https work.
To begin, you need to provide an EmbeddedServletContainerFactory as explained in the Embedded Servlet Container Support docs. The docs are for Java but the Groovy would look pretty much the same. Note that I haven't been able to get it to recognize the #Value annotation used in their example but its not needed. For groovy, simply put this in a new .groovy file and include that file on the command line when you launch spring boot.
Now, the instructions say that you can customize the TomcatEmbeddedServletContainerFactory class that you created in that code so that you can alter web.xml behavior, and this is true, but for our purposes its important to know that you can also use it to tailor server.xml behavior. Indeed, reading the source for the class and comparing it with the Embedded Tomcat docs, you see that this is the only place to do that. The interesting function is TomcatEmbeddedServletContainerFactory.addConnectorCustomizers(), which may not look like much from the Javadocs but actually gives you the Embedded Tomcat object to customize yourself. Simply pass your own implementation of TomcatConnectorCustomizer and set the things you want on the given Connector in the void customize(Connector con) function. Now, there are about a billion things you can do with the Connector and I couldn't find useful docs for it but the createConnector() function in this this guys personal Spring-embedded-Tomcat project is a very practical guide. My implementation ended up looking like this:
package com.deepdownstudios.server
import org.springframework.boot.context.embedded.tomcat.TomcatConnectorCustomizer
import org.springframework.boot.context.embedded.EmbeddedServletContainerFactory
import org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainerFactory
import org.apache.catalina.connector.Connector;
import org.apache.coyote.http11.Http11NioProtocol;
import org.springframework.boot.*
import org.springframework.stereotype.*
#Configuration
class MyConfiguration {
#Bean
public EmbeddedServletContainerFactory servletContainer() {
final int port = 8443;
final String keystoreFile = "/path/to/keystore"
final String keystorePass = "keystore-password"
final String keystoreType = "pkcs12"
final String keystoreProvider = "SunJSSE"
final String keystoreAlias = "tomcat"
TomcatEmbeddedServletContainerFactory factory =
new TomcatEmbeddedServletContainerFactory(this.port);
factory.addConnectorCustomizers( new TomcatConnectorCustomizer() {
void customize(Connector con) {
Http11NioProtocol proto = (Http11NioProtocol) con.getProtocolHandler();
proto.setSSLEnabled(true);
con.setScheme("https");
con.setSecure(true);
proto.setKeystoreFile(keystoreFile);
proto.setKeystorePass(keystorePass);
proto.setKeystoreType(keystoreType);
proto.setProperty("keystoreProvider", keystoreProvider);
proto.setKeyAlias(keystoreAlias);
}
});
return factory;
}
}
The Autowiring will pick up this implementation an run with it. Once I fixed my busted keystore file (make sure you call keytool with -storetype pkcs12, not -storepass pkcs12 as reported elsewhere), this worked. Also, it would be far better to provide the parameters (port, password, etc) as configuration settings for testing and such... I'm sure its possible if you can get the #Value annotation to work with Groovy.
If you don't want to implement your connector customizer, you can build and import the library (https://github.com/ycavatars/spring-boot-https-kit) which provides predefined connector customizer. According to the README, you only have to create your keystore, configure connector.https.*, import the library and add #ComponentScan("org.ycavatars.sboot.kit"). Then you'll have HTTPS connection.
And here's an example of the customizer implemented in Groovy:
https://github.com/UniconLabs/orville/blob/master/web/src/main/groovy/org/apereo/openregistry/config/TomcatSslConfiguration.groovy