I am trying to set up a spring-cloud-config-server for production.
I want to read multiple git repos so I am providing following configuration in application.yml of config-server placed in src/main/resources
spring:
application:
name: config-server
profiles:
active: git
cloud:
config:
server:
git:
uri: https://somedomain.com/project1/project1.git
username: project1user
password: project1password
repos:
project2:
pattern: project2/*
uri: https://somedomain.com/project2/project2.git
username: project2user
password: project2password
searchPaths:
- 'src/main/resources'
Now, I want to externalize this configuration of the config server.
I could provide main git repo (https://somedomain.com/project1/project1.git) properties by environment variables like following
spring.cloud.config.server.git.uri=https://somedomain.com/project1/project1.git
spring.cloud.config.server.git.username=project1user
spring.cloud.config.server.git.password=project1password
But what about other git repo properties. Passing complex map-like structure would be pretty tedious if passed via environment variables.
What is the best possible way to pass this configuration of other repos?
Passing some configuration as an environment variable has other disadvantages like those properties could not be refreshed on runtime.
Is there any possibility that additional repo configuration is picked by from some configuration file in main git (https://somedomain.com/project1/project1.git) itself?
The Spring Cloud Config Server provides a configuration with multiples repositories and they can be accessed by url like we do for one git repo. The config server will fetch each configuration by a pattern, so your label {application} in the path will be the key to find the correct repository. Like in mine I did:
spring:
cloud:
config:
server:
git:
uri: https://github.com/solivaf/config-properties-foo
Now we should add our additional repositories as you can see below:
spring:
cloud:
config:
server:
git:
uri: https://github.com/solivaf/config-properties-foo
repos:
appFoo:
pattern: app-foo
uri: https://github.com/solivaf/config-properties-bar
Restart the config server and perform the requests below.
$ curl localhost:8080/fooapp/prod
{
"name": "fooapp",
"profiles": [
"prod"
],
"label": null,
"version": "8686fb74f9af0aead98bd20d6e20e84a37028781",
"state": null,
"propertySources": [
{
"name": "https://github.com/solivaf/config-properties-foo/application-prod.yml",
"source": {
"bar.foo": "testProdPropertiesYml"
}
},
{
"name": "https://github.com/solivaf/config-properties-foo/application.yml",
"source": {
"bar.foo": "testPropertiesYml"
}
}
]
}
Now we can see the repository used to application fooapp, as we don’t have any pattern mapped to this application, the config server will use the default app, now if we specify the pattern app-foo which is mapped in our config-server property file we should get another repository as response.
$ curl localhost:8080/app-foo/prod
{
"name": "app-foo",
"profiles": [
"prod"
],
"label": null,
"version": "f34ced0565042be4cf87c937c1dab2703e0b8ed2",
"state": null,
"propertySources": [
{
"name": "https://github.com/solivaf/config-properties-bar/app-foo-prod.yml",
"source": {
"foo.bar": "testProdPropertiesYml"
}
},
{
"name": "https://github.com/solivaf/config-properties-bar/application-prod.yml",
"source": {
"foo.bar": "testProdPropertiesYml"
}
},
{
"name": "https://github.com/solivaf/config-properties-bar/application.yml",
"source": {
"foo.bar": "testPropertiesYml"
}
}
]
}
Now, we have the correct repository which was mapped in our property file and all files which represents our app-foo application. The response order represents the hierarchy of the file being the most priority the first file in the list.
Related
When accessing my Spring Actuator /info endpoint I receive the following information:
{
"git": {
"branch": "8743b52063cd84097a65d1633f5c74f5",
"commit": {
"id": "b3j2924",
"time": "05.07.2021 # 10:00:00 UTC"
}
},
"build": {
"encoding": {
"source": "UTF-8"
},
"version": "1.0",
"artifact": "my-artifact",
"name": "my-app",
"time": 0451570333.122000000,
"group": "my.group"
}
}
My project does not maintain a META-INF/build-info.properties file.
I now wanted to write a unit-test for that exact output but get the following error:
java.lang.AssertionError:
Expecting actual:
"{"git":{"branch":"8743b52063cd84097a65d1633f5c74f5","commit":{"id":"b3j2924","time":"05.07.2021 # 10:00:00 UTC"}}}"
to contain:
"build"
The whole build block is missing in the output.
My questions are the following:
What needs to be done to access the build information during a local unit-test run without providing a META-INF/build-info.properties.
From where does Spring Actuator retrieve the actual build information when my project does not has a META-INF/build-info.properties file so it gives me the output from above?
The build-info.properties file is typically generated at build time by Spring Boot's Maven or Gradle plugins.
I'm deploying Docker swarm with ansible and I would like to ensure the ingress network has been created. In that aim, I configured the following task :
- name: Ensure ingress network exists
docker_network:
state: present
name: ingress
driver: overlay
driver_options:
ingress: true
And I'm getting the following error :
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: docker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.41/networks/ingress/disconnect: Not Found ("No such container: ingress-endpoint")
fatal: [swarm-srv-1]: FAILED! => {"changed": false, "msg": "An unexpected docker error occurred: 404 Client Error for http+docker://localhost/v1.41/networks/ingress/disconnect: Not Found (\"No such container: ingress-endpoint\")"}
I've tried to add some arguments likes :
scope: swarm
force: yes
But no changes... I've also tried to delete the ingress with ansible (state: absent), but I always get the same error.
Note that I don't face any issue when trying to delete a recreate the ingress network manually on the swarm : docker network rm ingress
I don't know how to resolve that issue...Any help would be appreciated. Thanks !
Here are some informations that may help...
# docker version
Version: 20.10.6
API version: 1.41
Go version: go1.13.15
Git commit: 370c289
Built: Fri Apr 9 22:47:35 2021
OS/Arch: linux/amd64
# docker inspect ingress
[
{
"Name": "ingress",
"Id": "yb2tkhep8vtaj9q7w3mssc9lx",
"Created": "2021-05-19T05:53:27.524446929-04:00",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.0.0/24",
"Gateway": "10.0.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": true,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"ingress-sbox": {
"Name": "ingress-endpoint",
"EndpointID": "dfdc0f123d21a196c7a815c7e0a886924d0799ae5f3be2d38b64d527ed4620b1",
"MacAddress": "02:42:0a:00:00:02",
"IPv4Address": "10.0.0.2/24",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4096"
},
"Labels": {},
"Peers": [
{
"Name": "8f8932d6f99f",
"IP": "(ip address here)"
},
{
"Name": "28b9ca95dcf0",
"IP": "(ip address here)"
},
{
"Name": "f7c48c8af2f5",
"IP": "(ip address here)"
}
]
}
]
I had the exact same issue when trying to customize the IP range of the ingress network. It looks like the docker_network module does not support modification of swarm specific networks: there is a open Github issue for this.
I went for the ugly workaround of removing the network by executing it through a shell (docker network rm ingress command) and adding it again. When adding it with the docker_network module, I found that adding also seems not be working (fails to set the ingress property of the network). So I ended up doing both remove- and create operation through a shell command.
Since the removal will trigger a confirmation dialogue:
WARNING! Before removing the routing-mesh network, make sure all the nodes in your swarm run the same docker engine version. Otherwise, removal may not be effective and functionality of newly create ingress networks will be impaired.
Are you sure you want to continue? [y/N]
I used the expect module to confirm the dialogue:
- name: remove default ingress network
ansible.builtin.expect:
command: docker network rm ingress
responses:
"[y/N]": "y"
- name: create customized ingress network
shell: "docker network create --ingress --subnet {{ docker_ingress_network }} --driver overlay ingress"
It is not perfect but it works.
There was one last problem I experienced: when running it on an existing swarm I ended up having network issues on the node where I did run this (somehow the docker_gwbridge network on that node could not handle the change). The fix for this was to fully remove the node and re-join the swarm (regenerates the docker_gwbridge).
I am following the below to set up a config server in PCF and access the application in PCF.
https://github.com/pcf-guides/gs-configuration-server
I have added the config server from the PCF marketplace and it is bind with the app.
The config server is pointed to the following Git repository:
https://github.com/pcf-guides/configuration-server-config-repo.git
The VCAP_SERVICES is like below :
"p-config-server": [
{
"binding_name": null,
"credentials": {
"access_token_uri": "https://p-spring-cloud-
services.uaa.run.pivotal.io/oauth/token",
"client_id": "p-config-server-d837bb39-4cf1-47ce-994b-03257852a7f6",
"client_secret": "XXsWzX6IhKME",
"uri": "https://config-9f0e115f-dbb0-42e8-981a-e70e2cd62570.cfapps.io"
},
"instance_name": "sconfigserver",
"label": "p-config-server",
"name": "sconfigserver",
"plan": "trial",
"provider": null,
"syslog_drain_url": null,
"tags": [
"configuration",
"spring-cloud"
Whenever I am trying to access the application authentication error is coming :
Full authentication is required to access this resource.
I dont have any security configuration in the application.
In your application.properties file, you should add the following
management.security.enabled=false
This will disable security altogether.
I tried this in my main application class and worked fine for me
#SpringBootApplication(exclude = {SecurityAutoConfiguration.class , ManagementWebSecurityAutoConfiguration.class})
I wanted to fetch properties from two git repos. one is https://username#bitbucket.my.domain.com/share.git - which will have a property file contains some common key value pair and the other one is https://username#bitbucket.my.domain.com/service.git - it will have property files of all the micro services.
While I am deploying the service only one yml file (which is in https://username#bitbucket.my.domain.com/share.git repo) is read by the config server. What I am missing? How to read the property file from another repo i.e. https://username#bitbucket.my.domain.com/service.git too?
I wanted to deploy the service in PCF. So I configured the config-server in PCF with the following json.
{
"count": 1,
"git": {
"label": "feature",
"uri": "https://username#bitbucket.my.domain.com/share.git",
"username": "username",
"password": "password",
"repos": {
"configserver": {
"password": "password",
"label": "feature",
"uri": "https://username#bitbucket.my.domain.com/service.git"
"username": "username"
}
}
}
}
Name of my service is LogDemo and spring profile is active. I have created two yml files and placed in the corresponding repo. (I have given same name two both the files like LogDemo-active.yml). While I am deploying the service only one yml file (which is in https://username#bitbucket.my.domain.com/share.git repo) is read by the config server. /env is giving me the following:
{
"profiles": [
"active",
"cloud"
],
"server.ports": {
"local.server.port": 8080
},
"configService:configClient": {
"config.client.version": "234e59d4a9f80f035f00fdf07e6f9f16e5560a55"
},
"configService:https://username#bitbucket.my.domain.com/share.git/LogDemo-active.yml": {
"key1": "value1",
"key2": "value2"
},
...................
...................
What I am missing? How to read the property file from other repo i.e. https://username#bitbucket.my.domain.com/service.git too?
Below is my bootstrap.yml
spring:
application:
name: LogDemo
mvc:
view:
prefix: /
suffix: .jsp
Here is my manifest file
---
inherit: baseManifest.yml
applications:
- name: LogDemo
host: LogDemo
env:
LOG_LEVEL: INFO
spring.profiles.active: active
TZ: America/New_York
memory: 1024M
domain: my.domain.com
services:
- config-server-comp
When using multiple repos, the repos that will be applied depend on the pattern's defined for those repos. The default pattern is <repo-name>/*. Thus changing the repo name to LogDemo will activate the repo for your app, because the app name, spring.application.name, is LogDemo.
If one or more patterns match, then the repo for the matched patterns will be used. If no pattern matches then the default is used.
Full details are described in the docs here.
https://cloud.spring.io/spring-cloud-config/single/spring-cloud-config.html#_pattern_matching_and_multiple_repositories
If you don't need or want the pattern matching feature, you can use the [composite backend](
https://docs.pivotal.io/spring-cloud-services/2-0/common/config-server/composite-backends.html). The composite backend allows you to define multiple Git repositories. See the first config example here.
https://docs.pivotal.io/spring-cloud-services/2-0/common/config-server/composite-backends.html#general-configuration
I have added a Redis ElastiCache section to my s-resource-cf.json (a CloudFormation template), and selected its hostname as an output.
"Resources": {
...snip...
"Redis": {
"Type": "AWS::ElastiCache::CacheCluster",
"Properties": {
"AutoMinorVersionUpgrade": "true",
"AZMode": "single-az",
"CacheNodeType": "cache.t2.micro",
"Engine": "redis",
"EngineVersion": "2.8.24",
"NumCacheNodes": "1",
"PreferredAvailabilityZone": "eu-west-1a",
"PreferredMaintenanceWindow": "tue:00:30-tue:01:30",
"CacheSubnetGroupName": {
"Ref": "cachesubnetdefault"
},
"VpcSecurityGroupIds": [
{
"Fn::GetAtt": [
"sgdefault",
"GroupId"
]
}
]
}
}
},
"Outputs": {
"IamRoleArnLambda": {
"Description": "ARN of the lambda IAM role",
"Value": {
"Fn::GetAtt": [
"IamRoleLambda",
"Arn"
]
}
},
"RedisEndpointAddress": {
"Description": "Redis server host",
"Value": {
"Fn::GetAtt": [
"Redis",
"Address"
]
}
}
}
I can get CloudFormation to output the Redis server host when running sls resources deploy, but how can I access that output from within a Lambda function?
There is nothing in this starter project template that refers to that IamRoleArnLambda, which came with the example project. According to the docs, templates are only usable for project configuration, they are not accessible from Lambda functions:
Templates & Variables are for Configuration Only
Templates and variables are used for configuration of the project only. This information is not usable in your lambda functions. To set variables which can be used by your lambda functions, use environment variables.
So, then how do I set an environment variable to the hostname of the ElastiCache server after it has been created?
You can set environment variables in the environment section of a function's s-function.json file. Furthermore, if you want to prevent those variables from being put into version control (for example, if your code will be posted to a public GitHub repo), you can put them in the appropriate files in your _meta/variables directory and then reference those from your s-function.json files. Just make sure you add a _meta line to your .gitignore file.
For example, in my latest project I needed to connect to a Redis Cloud server, but didn't want to commit the connection details to version control. I put variables into my _meta/variables/s-variables-[stage]-[region].json file, like so:
{
"redisUrl": "...",
"redisPort": "...",
"redisPass": "..."
}
…and referenced the connection settings variables in that function's s-function.json file:
"environment": {
"REDIS_URL": "${redisUrl}",
"REDIS_PORT": "${redisPort}",
"REDIS_PASS": "${redisPass}"
}
I then put this redis.js file in my functions/lib directory:
module.exports = () => {
const redis = require('redis')
const jsonify = require('redis-jsonify')
const redisOptions = {
host: process.env.REDIS_URL,
port: process.env.REDIS_PORT,
password: process.env.REDIS_PASS
}
return jsonify(redis.createClient(redisOptions))
}
Then, in any function that needed to connect to that Redis database, I imported redis.js:
redis = require('../lib/redis')()
(For more details on my Serverless/Redis setup and some of the challenges I faced in getting it to work, see this question I posted yesterday.)
update
CloudFormation usage has been streamlined somewhat since that comment was posted in the issue tracker. I have submitted a documentation update to http://docs.serverless.com/docs/templates-variables, and posted a shortened version of my configuration in a gist.
It is possible to refer to a CloudFormation output in a s-function.json Lambda configuration file, in order to make those outputs available as environment variables.
s-resource-cf.json output section:
"Outputs": {
"redisHost": {
"Description": "Redis host URI",
"Value": {
"Fn::GetAtt": [
"RedisCluster",
"RedisEndpoint.Address"
]
}
}
}
s-function.json environment section:
"environment": {
"REDIS_HOST": "${redisHost}"
},
Usage in a Lambda function:
exports.handler = function(event, context) {
console.log("Redis host: ", process.env.REDIS_HOST);
};
old answer
Looks like a solution was found / implemented in the Serverless issue tracker (link). To quote HyperBrain:
CF Output variables
To have your lambda access the CF output variables you have to give it the cloudformation:describeStacks access rights in the lambda IAM role.
The CF.loadVars() promise will add all CF output variables to the process'
environment as SERVERLESS_CF_OutVar name. It will add a few ms to the
startup time of your lambda.
Change your lambda handler as follows:
// Require Serverless ENV vars
var ServerlessHelpers = require('serverless-helpers-js');
ServerlessHelpers.loadEnv();
// Require Logic
var lib = require('../lib');
// Lambda Handler
module.exports.handler = function(event, context) {
ServerlessHelpers.CF.loadVars()
.then(function() {
lib.respond(event, function(error, response) {
return context.done(error, response);
});
})
};