I have a spring boot application where I am using some aws services.
The code in openly available in Git.
I don't want to commit AWS secret and access keys which are part of application.properties. I can't add to .gitignore as I want to commit other values of application.properties.
Many are committing to this repo. We are adding these aws keys in local and making sure its not added as part of any commit.
I want to make sure the aws keys in application.properties should not come to git at any cost. Which is the best way to manage these secret keys.?
You shouldn't be placing AWS API keys in application.properties at all. If the application is running on AWS it should be using the IAM role of the server it is running on. If it is not running on AWS it should probably be using environment variables.
Please review the documentation on this subject here.
Thanks to #Mark B. I would prefer using Java system properties as we can maintain them at application level. Env variables will be at system level which is not really needed and it may lead to conflicts.
while running an spring-boot jar with mvn, it can be done as below
mvn spring-boot:run -Dspring-boot.run.jvmArguments="-Daws.accessKeyId=XXXXXXXXXXX -Daws.secretKey=XXXXXXXXXXX"
if running for IDE like Eclipse or IntelliJ, it should be added as VM Options.
-Daws.accessKeyId=XXXXXXXXXXX -Daws.secretKey=XXXXXXXXXXXX
After this AWS client object can be built as usual.
As an example,
SNS client can be build by
AmazonSNS snsClient = AmazonSNSClient.builder().withRegion(Regions.US_EAST_1).build();
SES client can be built by
AmazonSimpleEmailService emailClient = AmazonSimpleEmailServiceClientBuilder.standard().withRegion(Regions.US_EAST_1).build();
Related
Testcontainers can manage dockerized service dependencies, like a database, Kafka, Elasticsearch, and so on for integration testing.
Can I configure my Spring Boot application to manage these service dependencies during local development?
For example, my Spring Boot application needs a MySQL database.
I would like to integrate it with Testcontainers to provide a Docker container with MySQL not only during the tests execution, but at application startup during local development too.
Testcontainers provides an API to manage applications and services in Docker containers. It's incredibly useful for integration testing, where having a programmatically configured, isolated, repeatable environments is an essential requirement for trustworthy tests.
Because of that Testcontainers has integrations with the frameworks like Spring and Quarkus, and tes frameworks like JUnit, Spock, etc to automatically tie the lifecycle of your containerized dependencies to the lifecycle of the tests.
However, Testcontainers API is generic and doesn't have to run during the tests. For example, Quarkus has a feature called Dev Services which automatically creates a container for your database (or other service dependencies, for example Kafka, Redis, etc) when your application tries to access the database, but the configuration is not present.
You can think about it like this, if you have the data access repository classes initialized and wired, but no datasource.url in the config -- it'll spin up the database using testcontainers and configure the app to use it (just like it would happen during tests, but instead used for local development).
Spring Boot doesn't have an automated feature like that currently, there's an open issue to investigate these local development setups with Testcontainers.
If you're open to manually add a feature for your particular application, you can look at the prototype linked from that issue here: https://github.com/joshlong/testcontainers-auto-services-prototype
It's a bit more involved because it integrates with the Spring DevTools, but here are the essential parts that need to be taken care of:
Check that you need to use the database (in your application it can be a given).
Verify the configuration to use the database is absent (if the database is already configured you don't need to spin up a new one)
Create a container using Testcontainers API, either using an appropriate module or the GenericContainer with any Docker image.
Provide the configuration back to the application. For the database that would be the jdbcUrl, username, password, database name, r2dbcUrl and any other relevant properties.
You can take a look at the video with Josh Long where this concept was tried: https://www.youtube.com/watch?v=1PUshxvTbAc&t=2450s
It would also work in the production environments, but the usefulness of the ephemeral Databases, might be limited.
I am using Spring Cloud Config Server first time and have a basic query.
Spring Config server externalises the configuration to a separate git repository.
Why would I create a separate repository just for the configurations?
Is not it advisable to have mono repository with all application code and configurations in a single repo than creating a separate one just for configurations.
We have multiple micro services all present in the same repository. Should not the config server to be one of the micro service present in the same repository where the other application code is?
So, in my multi-module gradle project, I can make config-server as one of the module and give the same repository name as git backed url in config-server. Is this advisable? If yes, where should I keep the configurations in config-server? Inside resources?
Thank you.
When working with microservices it is advisible to have one repository for each microservice. The config server is a microservice as well, therefore it should be put in a separate repository.
Each microservice should have its own independent code repository and your application configuration should never be in the same repository as your source code.
You can read more about this here: Heroku's The Twelve-Factor App. Here you can find 12 best practices to use when building microservices, but for this question I recommend looking at
1st factor: The codebase
3rd factor: The config
I have a spring boot project with 4 microservices (Eureka service registry, Config server, a Zuul gateway and a userservice) in one repository with a parent project where I have a docker-compose.yml which reads the Dockerfiles in the microservices project and uses the "application-docker.yml" and "bootstrap-docker.yml"
What I'd like to do is to trigger a jenkins pipeline after a commit in git so that it will compile and deploy the microservices in Docker. Eventually I'd like to have a production configuration that deploys the images in Kubernetes maybe AWS.
Now, in order to work, the microservices need to start in order:
configserver
eureka service registry
gateway , etc..
What is the best practise?
If I have separate repositories per microservice, I think I can figure it out. It should be easy to deploy a single microservice assuming that configserver and eureka service registry are already up and running, in reality they should never change.
If I have a single repository, and I keep developing new microservices, do I need to have separate jenkins file per microservices or can I have a jenkinsfile in the parent project and use docker-compose?
How does it work? Any articles online that can help (couldn't find any). Does it make sense?
Or do I need to look at Jenkins X ?
Thanks!
I would recommend using separate repositories for each microservice. You use microservices to prevent monoliths and have small well-defined services; it only seems appropriate to also separate them by space i.e. store them in separate repositories (making it for example easier to reuse one).
You would then have to provide a Jenkinsfile in each repo. These would be mostly identical.
If you want fast release cycles you could automatically deploy a single service upon release.
Alternatively you could use an additional release train module that handles the full deployment.
In both cases I would use a docker-compose file that handles the interconnection between the services.
You can enforce the right order by using 'depends_on, links, volumes_from, and network_mode: "service:..."'. For a full reference see the docker documentation.
If you want to keep your single repository your Jenkinsfile(s) would have to be quite hacky, I suppose... After each commit you would either
build all modules --> monolithic behaviour
somehow determine which modules have changed (e.g. looking at the git log) --> same behaviour as with multiple modules but very hackily
The Docker-Compose File
If you want to release all modules at a specific point of time you could use a Release Train module where the docker-compose.yml resides next to a Jenkinsfile. Then when you want to ship your application you can start this Jenkins-job.
If you want to ship each service as soon as it is released, independently from the others, you would need to access the docker-compose.yml from each module. You could do this manually (since the files won't change too often) or create a docker module that you use as a git-submodule in all your services.
We use a generic docker-compose.yml for this, where every version is replaced by a variable:
example-service:
image: example.service:${EXAMPLE_SERVICE_VERSION}
Then to start that specific service in jenkins we use the command
export EXAMPLE_SERVICE_VERSION=1.1.1
docker-compose -p example-project -f docker-compose.yml up -d example-service
The question I'm going to ask might be a bit more of a 'proof of concept' area, so I fully expect to find out that it's not possible to achieve what I want.
Background:
I have deployed AWS Aurora database cluster using Serverless Framework. Cluster connection details are stored as follows:
Endpoint URL - Stored as String in the AWS Parameter Store and exists among CloudFormation outputs.
Master Password - Randomly generated during the Serverless deployment and stored as a SecureString in the AWS Parameter Store; decryptable by given lambdas only (in other words - no-one knows the password)
Master Username - hardcoded, not going to change
Serverless Build invokes Flyway migration scripts to create the Aurora database schema. This is done using AWS lambda-backed custom resource, thus it has access to SSM (Parameter Store) to initialise DB connection.
What I'm trying to achieve:
Maven build that invokes JOOQ code generation using the given Aurora DB (which is based on MySQL) above.
What I've tried and problems I've faced:
During the Serverless deployment create a read-only user for the Aurora DB. The login details could as well be hardcoded. However, at this stage, Maven build is unaware about the cluster endpoint URL.
Have embedded MySQL database during Maven build. Replicate Flyway migration to this embedded database as well. However, all my attempts to integrate Flyway & JOOQ to embedded MySQL DB have failed due to connection issues. Additionally, this approach will have further complications when I integrate my build process to a CI pipeline.
I was considering having AWS Lambda being invoked during the Maven build. It would have to return a packaged JAR that contains JOOQ generated code and would add the JAR to the sources. Haven't tried it as it seems to be too much overhead for what is required.
Another concern I have for this design - currently Aurora cluster runs on the security group that allows ingress from anywhere (for debug purposes). However, this will change to only access ingress from its own group that will restrict access to only lambdas sharing same security group.
Has anyone tried to achieve anything similar? Do you have any suggestions for my approach?
I want to use spring cloud config to externalise application properties. I have configured config-server and client, but i don't want to use git(enterprise) as source, this is due to the fact that git may be down for the maintenance. Therefore, i thought about releasing application properties to nexus as application.properties.tar.gz and do the following:
1.) When config-server starts, download the release version from nexus and then unpack the tar.gz to a file system on the server where config-server is running.
2.) The config server which will pick the unpacked properties files.
Please note that the nexus url, application.properties.tar.gz version and location are all passed as VM options to config server.
Is this a sensible option? if this not best option, please could you suggest any other options?
Thanks
kankalam
Unless the Git server is down for maintenance so frequently, I wouldn't go with that option, it seems to be an overkill. Also the Nexus server might go down, so there's no advantage on doing that.
The config server does not read the configuration from the repo once and again, the repository is cloned when configuration is requested. From the docs:
The default strategy for locating property sources is to clone a git
repository (at spring.cloud.config.server.git.uri) and use it to
initialize a mini SpringApplication. The mini-application’s
Environment is used to enumerate property sources and publish them via
a JSON endpoint.
With that in mind, you need to check that the git server you use is available before you start the application or before you refresh its context. If you think it might be a problem, you still have two options:
Set up a dedicated Git server for your application (Check out Gogs as a lightweight option).
Go the "native" way and load the files from the file system. This way you don't have a version control of the configuration files, but you could do it separately into your project.