I am trying to load secret (and its values) in my local using localstack (docker-compose).
I am able to successfully start local stack and also able to create secret using below:
aws --endpoint-url=http://127.0.0.1:4566 secretsmanager create-secret --name /secret/spring-boot-app --secret-string '{"username":"username","my_pwd":"password"}'
But when I am trying to access the value in Spring Boot App (Version 2.7.8, JDK 11), I am getting different errors (unable to resolve placeholders, secret manager cannot find secret etc).
Feel that getting mixed up with using the right dependencies and properties/configuration, to use secret that is sucessfully created in localstack.
Below is my application.yml
spring:
config:
import: aws-secretsmanager:/spring/spring-boot-app
aws:
secretsmanager:
enabled: true
endpoint: http://127.0.0.1:4566
cloud:
aws:
secretsmanager:
endpoint: http://localhost:4566
credentials:
accessKey: dummy
secretKey: dummy
profile-name: dummy
region:
static: us-east-1
stack:
auto: true
test:
value: ${my_pwd}
Basically it is unable to resolve the 'my_wd' or 'username' in any spring bean.
Tried different dependencies:
<dependency>
<groupId>io.awspring.cloud</groupId>
<artifactId>spring-cloud-starter-aws-secrets-manager-config</artifactId>
<version>2.3.0</version>
</dependency>
-----------------------or ------
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-secretsmanager</artifactId>
<version>1.12.398</version>
</dependency>
Related
I'm working with microservice architecture and have spring cloud config service and another microservice.
profiles:
active: vault
cloud:
# Configuration for a vault server running in dev mode
vault:
scheme: http
host: 127.0.0.1
port: 8200
connection-timeout: 5000
read-timeout: 15000
authentication: TOKEN
token: s.E4gdoIYAKxMvCE56MP5Etmvy
kv:
enabled: true
backend: secret
backend-version: 2
profile-separator: /
generic:
enabled: false
application-name: myapp
Config server dependency
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-vault-config</artifactId>
<version>2.1.5.RELEASE</version>
</dependency>
this is into .yml into the config service. Then into the .yml for my microservice i have db.username property which I want to resolve from Vault but I can't. Do you have any ideas?
username: db.username
password: secret/apm-transaction-service/dev/db.user
#Value("${db.username}")
this value is resolved into the java code but not into the .yml
Now for each microservice which I have I want to resolve the secrets from the configuration service without making any changes into the microservices. Currently reading native .ymls from the config service and want to add one more source :)
ApplicationStartupRunner run method Started !!root
if you are using spring-boot, for the value in .yml file to be resolved it has to be a variable. you must use ${db.username} in the yaml file
Spring Boot version: 2.1.6.RELEASE
Spring Cloud Version: Greenwich.SR1
Objective is to set up two instances of Eureka Servers in a cluster and have all microservices registered to both, in order to achieve HA. Currently I am testing this on my local machine running Ubuntu 18.04.
So as mentioned in the official docs, I have set up the peer to peer awareness of Eureka Instances. On bringing both the Eureka Server Instances up, on the dashboard of Instance-1(Port 8080) under available replicas showing Instance-2 name and on dashboard of Instance-2(Port 8081) it is showing Instance-1 name. So far so good.
I have deployed them on external tomcat server as war and not running them as jar. War name is "eureka-naming-server". So my access URL becomes: http://localhost:8080/eureka-naming-server/eureka and http://localhost:8081/eureka-naming-server/eureka.
I have set up a load balancer using Apache 2 and mod_jk. So instead of accessing them using their individual URLs i.e. http://localhost:8080/eureka-naming-server/eureka, http://localhost:8081/eureka-naming-server/eureka, I can access them using Apache URL as http://localhost:80/eureka-naming-server/eureka and I can see load-balancing happening properly.
Now the issue is that once I give Eureka URL as http://localhost:80/eureka-naming-server/eureka in the microservice, it registers only on one instance depending on which Eureka instance the load-balancer has redirected the request. But even though peer to peer awareness is setup, microservice is not getting registered on the other eureka instance.
In Eureak servers, I'm using the below dependency:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-eureka-server</artifactId>
</dependency>
Eureka-1 props:
spring:
application:
name: eureka-naming-server
jmx:
default-domain: eureka-naming-server
eureka:
client:
service-url:
defaultZone: http://eureka-server-2:8081/eureka-naming-server/eureka
#register-with-eureka: false
#fetch-registry: false
instance:
hostname: eureka-server-1
Eureka-2 props:
spring:
application:
name: eureka-naming-server
jmx:
default-domain: eureka-naming-server
eureka:
client:
service-url:
defaultZone: http://eureka-server-1:8080/eureka-naming-server/eureka
#register-with-eureka: false
#fetch-registry: false
instance:
hostname: eureka-server-2
In microservices, I'm using below dependencies:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-ribbon</artifactId>
</dependency>
Microservice props:
eureka:
instance:
metadata-map:
configPath: /config
client:
service-url:
defaultZone: http://localhost/eureka-naming-server/eureka
In the machine's /etc/hosts file I have set it up as:
127.0.0.1 eureka-server-1
127.0.0.1 eureka-server-2
The eureka instance on which it is getting registered I can see the below in the logs:
WARN 13748 --- [nio-8009-exec-3] Registered instance CONFIG-SERVER/192.168.1.16:config-server:8888 with status UP (replication=false)
The other Eureka instance where it is not registered, when it receives the heart-beat because of the load-balancer, shows this in the logs:
WARN 13748 --- [nio-8009-exec-3] c.n.e.registry.AbstractInstanceRegistry : DS: Registry: lease doesn't exist, registering resource: CONFIG-SERVER - 192.168.1.16:config-server:8888
2019-12-26 19:00:34.995 WARN 13748 --- [nio-8009-exec-3] c.n.eureka.resources.InstanceResource : Not Found (Renew): CONFIG-SERVER - 192.168.1.16:config-server:8888
Eureka-1 Dashboard
Eureka-2 Dashboard
I have already tried suggestions mentioned here and here. But nothing seems to work.
So what could I be missing here?
Issue:
spring:
application:
name: eureka-naming-server
You put the same name for an application so it's displayed like this.
you need to change it to,
Eureka-1 props:
spring:
application:
name: eureka-naming-server-1
Eureka-2 props:
spring:
application:
name: eureka-naming-server-2
It may also be worth checking your security configuration for your eureka servers. If you do have spring boot security enabled, you will need to be sure that you include the corresponding username/password in the defaultZone portion of your config.
i.e. in your eureka server 1 config: eureka.client.serviceUrl.defaultZone=http://:#eureka-server-2:8761/eureka
(and visa versa for your eureka server 2 config)
I only mention this because I had left out my own username/password values in that config value and it was not registering my microservices on all my eureka instances. After make this change, it worked as I was expecting.
I have configured bootstrap.yml for spring config
spring:
application:
name: cce-auth
cloud:
config:
uri: http://temp.com:8888
It works fine but I need URI value Dynamically, for example if I will publish this .war in test environment this URL must be http://test-temp.com:8888.
So for this I have solution create config.txt file in server and using I/O Stream reed/write this string to bootstrap.yml.
But problem is loading, spring loads http://localhost:8888 before I'm writing in bootstrap.yml.
So my reason is to create dynamically URI for config server.
DO you have any idea?
Define active_profile in bootstrap.yml file
spring:
profiles:
active: ${activatedProperties}
Then create bootstrap-${activatedProperties}.yml for each environment, etc..bootstrap-dev.yml, bootstrap-pre.yml, bootstrap-prod.yml
For example:
spring:
application:
name: servicename_prod
cloud:
config:
uri: https://admin:123456#test.com:8888
server:
port: 8443
add plugin to pom.xml file :
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-config</artifactId>
</dependency>
When run java, define environment on it: for example, run with prod environment.
java -Dserver.port=8443 -Dspring.profiles.active=prod -jar ....
In my PCF logs config-client app gives
"ConfigServicePropertySourceLocator : Could not locate PropertySource:
401 Unauthorized"
I have gitlab repository to provide properties files in PCF config server.
If I check Config Server -> manage -> it gives "
Config server is online"
Where am I missing the configuration?
bootstrap.yml config-client-app
spring:
application:
name: <config-client-name>
cloud:
config:
enabled: true
name: <config-server-instance-name>
uri: <uri details from view credential modal>
# username: <tried different permutations to get it working>
# password: <tried different permutations to get it working>
management:
security:
enabled: false
manifest.yml config-client-app
applications:
- name: <app-name>
host: <app-name>
memory: 1024M
instances: 1
path: /target/<app-name>-0.0.1-SNAPSHOT.jar
buildpack: java_buildpack_offline
services:
- <pcf config server instance name>
env:
SPRING_PROFILES_ACTIVE: dev
I have done lots of work around to get some result but no luck, sample app works fine.
I have tried my own config-server and config client app with github repository and it worked.
But in PCF I am totally stuck please help.
Doesn't a look anything wrong in your configuration. if you are seeing Config server is online in PCF that means config service is intialized properly and ready for use.
But let's try with the sample app which is working fine for you.
Add following information in you build.gradle if you are using gradle build(or add in pom.xml accordingly.)
SCS(Spring Cloud services) (mentioned versions are compatible with springBootVersion = '2.0.6.RELEASE' Check for comparability if you are using a different version of spring-boot.)
imports {
mavenBom "io.pivotal.spring.cloud:spring-cloud-services-dependencies:2.0.1.RELEASE"
mavenBom "org.springframework.cloud:spring-cloud-dependencies:Finchley.RELEASE"
}
include config-client dependency
compile("io.pivotal.spring.cloud:spring-cloud-services-starter-config-client")
Now let's move to bootstrap.yml:
only following information is enough to pick the properties
spring:
application:
name: <config-client-name>
No change required in application.yml if you are using the same active profile while pushing to PCF.
#Avhi - your answer helped me.
My mistake was I was using wrong dependency, and I was adding unnecessary info. in bootstrap.yml
spring:
application:
name: <config-client-name>
cloud:
config:
enabled: true
name: <config-server-instance-name>
uri: <uri details from view credential modal>
# username: <tried different permutations to get it working>
# password: <tried different permutations to get it working>
management:
security:
enabled: false
name: <config-server-instance-name> we should configure the property file name that we are going to point from client application, I was providing config-server instance name.
Note: I have learnt that we can specify multiple properties files separated by comma.
uri: <uri details from view credential modal>
# username: <tried different permutations to get it working>
# password: <tried different permutations to get it working>
The above three properties were useless and I wasted most of my time because of this.
We need not have these values set unless we want to bind config-server service in our localhost:8080 app.
There's no harm in providing service name in our manifest.yml and as it is not the part of application, it won't create any trouble. BTW we can use CLI or PCF App Manager web console to create and bind service to our app.
After setting up all, we can expect spring security web page to enter username & password (that obviously we don't know). if spring security is configured then fine or we can disable while testing.
I did two things to disable and activate all endpoints of actuator (as I am using spring-boot 2.x)
added dependencies as below:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-rsa</artifactId>
<!-- <version>1.0.5.RELEASE</version> -->
</dependency>
and created SecurityConfiguration to disable security explicitly
#Configuration
#EnableWebSecurity
public class SecurityConfiguration extends WebSecurityConfigurerAdapter{
#Override
protected void configure(HttpSecurity http) throws Exception{
http.csrf().disable();
http.authorizeRequests().antMatchers("/").permitAll();
}
}
To make all the actuators endpoints accessible I have done the following:
management:
endpoints:
web:
exposure:
include: "*"
I'm trying to get Spring Cloud's Config Server setup with a BitBucket private repository and haven't had any luck. No matter what configuration I use, I always seem to get a 404 returned when trying to load a configuration.
I've also tried setting breakpoints in JGitEnvironmentRepository but it never seems to get called outside of afterPropertiesSet. If I manually triggering the findOne(application,profile,label) while debugging, I get an error that Branch name <null> not allowed. If I specify "master" for label property, then I get the dreaded Ref master cannot be resolved error.
The app loads fine but no results. From all the documentation I've read, it seems like this should work out of the box. Any help would be appreciated.
bootstrap.yml
server:
port: 8888
spring:
application:
name: config-service
cloud:
bus.amqp.enabled: false
config:
enabled: false
failFast: true
server:
prefix: /configs
git :
uri: https://bitbucket.org/[team]/[repo].git
username: [user]
password: [pass]
Repo files
- demo.app.yml
Attempted URL
http://localhost:8888/configs/demo.app
you need to add a profile name to the url. The default profile is 'default'. http://localhost:8888/configs/demo.app/default
this only works with the 1.0.0.RELEASE version to me, but with the 1.0.1.RELEASE version I found a problem on Windows (there is a problem with the file separators so the config server didn't find the YAML files in the local cloned repository). Maybe this is the related issue: Spring Cloud Config | Git Based | Not working on windows machine
If you still have any issues, you can refer to the below client configuration. I have successfully configured the bitbucket repository.
Spring Boot Version: 1.4.1.RELEASE
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>Camden.RELEASE</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
-----------------------------------------------------------------------
bootstrap.yml
spring:
application:
name: client-config
cloud:
config:
failFast: true
label: master
profile: default
username: <username>
password: <password>
server:
bootstrap: true
uri: http://localhost:8888