Skaffold dev doesn't do automatic file sync for spring boot app - spring-boot

I have a spring boot app that generates a jar artifact.I want to configure skaffold to make use of automatic file sync. Below are my config files :
pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.5.13</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.example</groupId>
<artifactId>rest-service-complete</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>rest-service-complete</name>
<description>Demo project for Spring Boot</description>
<properties>
<java.version>1.8</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>${project.parent.version}</version>
</plugin>
</plugins>
</build>
</project>
Dockerfile
VOLUME /tmp
ARG DEPENDENCY=target/dependency
COPY ${DEPENDENCY}/BOOT-INF/lib /app/lib
COPY ${DEPENDENCY}/META-INF /app/META-INF
COPY ${DEPENDENCY}/BOOT-INF/classes /app
ENTRYPOINT ["java","-cp","app:app/lib/*","com.example.restservice.RestServiceApplication"]
deployment.yaml
kind: Deployment
metadata:
labels:
app: demo
name: demo
spec:
replicas: 1
selector:
matchLabels:
app: demo
strategy: {}
template:
metadata:
labels:
app: demo
spec:
containers:
- image: mariatamas89/spring-boot-hw
name: spring-boot-hw
imagePullPolicy: Always
resources: {}
imagePullSecrets:
- name: myregistrykey
status: {}
---
apiVersion: v1
kind: Service
metadata:
labels:
app: demo
name: demo
spec:
ports:
- name: 8080-8080
port: 8080
protocol: TCP
targetPort: 8080
selector:
app: demo
type: LoadBalancer
status:
loadBalancer: {}
skaffold.yaml
kind: Config
metadata:
name: complete
build:
tagPolicy:
sha256: { }
local:
push: true
artifacts:
- image: mariatamas89/spring-boot-hw
sync:
infer:
- '/target/dependency/app/com/example/*.java'
docker:
dockerfile: Dockerfile
noCache: true
deploy:
kubectl:
manifests:
- deployment.yaml
I run skaffold dev , app starts correctly , but when I do a change to a java file , nothing happens in the skaffold cli, the change is not picked up. Any ideas how to make skaffold file sync work?Thank you.

Related

Traefik throwing error 502 with Kotlin Spring Boot application

Ok so I'm facing an annoying issue with Traefik in Docker: it constantly throws an error 502: Bad Gateway when I'm trying to reach my HTTP service.
All I'm trying to do is add it as a simple load balancer to build 3 replicas of my Spring Boot app. The app works well when I remove anything concerning Traefik.
My Spring Boot Controller maps /api/v1 endpoint, which is working well without load balancing. I'm trying to map this same endpoint to Traefik, which is using port :80.
Dashboard at :8080 works just fine, I have no error whatsoever about my configuration mentioned in it; only problems when I'm reaching anything under :80.
I have to mention that this exact same configuration (docker-compose, traefik conf, postgres db, and spring boot app) DOES WORK fine with the exact same docker-compose.yml AND traefik.yml, which makes my problem even weirder. Maybe the issue comes from the fact that my Spring Boot app is made of Kotlin and not the one where it's working?. Anyway, here are my files:
# Dockerfile (default one generated)
FROM maven:latest
WORKDIR /usr/src/app
COPY . /usr/src/app
RUN mvn package
ENV PORT 5000
EXPOSE $PORT
CMD [ "sh", "-c", "mvn -Dserver.port=${PORT} spring-boot:run" ]
# docker-compose.yml (works as-is in my other app)
version: "3.8"
services:
project:
build: .
deploy:
replicas: 3
labels:
- "traefik.http.routers.project.rule=PathPrefix(`/api/v1`)" # same endpoint as in my Spring controller
- "traefik.http.middlewares.project.stripprefix.forceSlash=false"
- "traefik.http.routers.project.middlewares=project#docker"
traefik:
image: traefik:latest
restart: unless-stopped
ports:
- "80:80"
- "8080:8080"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./traefik.yml:/etc/traefik/traefik.yml
postgres:
image: postgres:latest
container_name: postgres
ports:
- "5432:5432"
environment:
POSTGRES_PASSWORD: example
# rabbitmq:
# image: rabbitmq:management-alpine
# container_name: rabbitmq
# ports:
# - "5672:5672" # AMQP
# - "15672:15672" # HTTP (Management)
# traefik.yml
global:
checkNewVersion: true
sendAnonymousUsage: false
providers:
docker: {}
log:
level: DEBUG # temporarily
api:
insecure: true
entryPoints:
insecure:
address: ":80"
secure:
address: ":443"
# src/main/resources/application.yml
spring:
sql.init.mode: always
datasource:
url: jdbc:postgresql://localhost:5432,postgres:5432/postgres
username: postgres
password: example
jpa:
open-in-view: false
<!-- pom.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.example</groupId>
<artifactId>project</artifactId>
<version>1.0.0</version>
<packaging>jar</packaging>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.6.4</version>
<relativePath/>
</parent>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>17</java.version>
<maven.compiler.source>17</maven.compiler.source>
<maven.compiler.target>17</maven.compiler.target>
<kotlin.version>1.6.10</kotlin.version>
</properties>
<dependencies>
<!-- Kotlin -->
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-reflect</artifactId>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib-jdk8</artifactId>
</dependency>
<!-- Spring Boot -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-amqp</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<!-- PostgreSQL -->
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
</dependency>
</dependencies>
<build>
<sourceDirectory>${project.basedir}/src/main/kotlin</sourceDirectory>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-plugin</artifactId>
<configuration>
<args>
<arg>-Xjsr305=strict</arg>
</args>
<compilerPlugins>
<plugin>spring</plugin>
</compilerPlugins>
</configuration>
<dependencies>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-allopen</artifactId>
<version>${kotlin.version}</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>
</project>
I'm using JDK 17, and docker-compose up --build to start everything. Thanks!

Spring Gateway deployment on external tomcat

I am developing a micro-service solution with spring boot. I have 3 micro services:
authentication services (manages users, roles, permissions)
project service (services used to do CRUD actions on Project entity)
gateway (used as entry point of the solution, also used to validate token and route to services)
I want to deploy these on a tomcat server as war placed in the webapps (knowing that they have an embedded tomcat).
I am able to deploy the authentication and project services and they are reachable over the URI ** localhost:8080/authentication/* ** and ** /project/* *, but I'm unable to reach the gateway on ** localhost:8080/gateway/ **, it returns a 404
I am sure that the gateway is running cause it created the tables in the DB after deployment.
**Knowing that in eclipse it works fine (since i run each service on a port specified in the application.yml) i don't understand why when deploying the gateway on tomcat it is unreachable and always return a 404. I can see it in the tomcat manager as deployed.
Here is the pom.xml of the gateway :
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.4.1</version>
<relativePath /> <!-- lookup parent from repository -->
</parent>
<groupId>com.myProject</groupId>
<artifactId>gateway</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>gateway</name>
<description>Gateway</description>
<profiles>
<profile>
<id>dev</id>
<properties>
<activatedProperties>dev</activatedProperties>
</properties>
</profile>
</profiles>
<properties>
<java.version>11</java.version>
<springfox-swagger.version>3.0.0</springfox-swagger.version>
</properties>
<dependencies>
<dependency>
<groupId>com.nimbusds</groupId>
<artifactId>nimbus-jose-jwt</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>com.oracle.database.jdbc</groupId>
<artifactId>ojdbc8</artifactId>
<version>21.1.0.0</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-gateway</artifactId>
<version>3.0.0</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-tomcat</artifactId>
<scope>provided</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<skipTests>true</skipTests>
</configuration>
</plugin>
</plugins>
</build>
</project>
and here is my application.yml file :
spring:
cloud:
gateway:
routes:
- id: incident
uri: http://localhost:8080/project
predicates:
- Path=/gateway/v1/project/**
filters:
- RewritePath=/gateway/(?<RID>.*), /$\{RID}
- JwtFilter
- RemoveRequestHeader=Authorization
- id: authentication
uri: http://localhost:8080/auth
predicates:
- Path=/gateway/auth/**
filters:
- RewritePath=/gateway/(?<RID>.*), /$\{RID}
- id: userManagement
uri: http://localhost:8080/auth
predicates:
- Path=/gateway/v1/user/**,/gateway/v1/permission/**,/gateway/v1/role/**,/gateway/v1/role-permission/**,/gateway/v1/user-role/**
filters:
- RewritePath=/gateway/(?<RID>.*), /$\{RID}
- JwtFilter
- RemoveRequestHeader=Authorization
datasource:
url: jdbc:oracle:thin:#//xxx.xxx.xxx.xxx:1521/ORCLCDB.localdomain
username: xxx
password: xxx
jpa:
hibernate:
ddl-auto: update
id:
new_generator_mappings=true
properties:
hibernate:
temp:
use_jdbc_metadata_defaults: false
dialect: org.hibernate.dialect.Oracle12cDialect
physical_naming_strategy: org.hibernate.boot.model.naming.PhysicalNamingStrategyStandardImpl
show-sql: true
logging:
level:
org.springframework: TRACE
org.hibernate.type: TRACE
org.apache.tomcat: INFO
org.apache.catalina: INFO
pattern:
console: "%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"
file: "%d %p %c{1.} [%t] %m%n"
file:
name: ./tomcat-logs/gateway
path: ./tomcat-logs/gateway
Spring Cloud Gateway is itself a Spring Boot WebFlux application. It's support only embedded server. Please check spring documentation
https://docs.spring.io/spring-boot/docs/current/reference/html/howto.html#howto-traditional-deployment

ADD failed: stat /var/lib/docker/tmp/docker-builder : : no such file or directory

I am new to Docker/Spring Boot. I am trying do the auto builds from GITHUB repo.
Below is my GitHub repo :
https://github.com/rajatgupta828/TestApplication
And below is my docker repo :
https://hub.docker.com/repository/docker/rajatgupta56/apptest
When I am building image from my local after Maven Build, it is working fine, But when I am trying to auto build using GITHUB I am getting below logs :
Cloning into '.'...
Warning: Permanently added the RSA host key for IP address '140.82.113.3' to the list of known hosts.
Reset branch 'master'
Your branch is up-to-date with 'origin/master'.
KernelVersion: 4.4.0-1060-aws
Components: [{u'Version': u'18.03.1-ee-3', u'Name': u'Engine', u'Details': {u'KernelVersion': u'4.4.0-1060-aws', u'Os': u'linux', u'BuildTime': u'2018-08-30T18:42:30.000000000+00:00', u'ApiVersion': u'1.37', u'MinAPIVersion': u'1.12', u'GitCommit': u'b9a5c95', u'Arch': u'amd64', u'Experimental': u'false', u'GoVersion': u'go1.10.2'}}]
Arch: amd64
BuildTime: 2018-08-30T18:42:30.000000000+00:00
ApiVersion: 1.37
Platform: {u'Name': u''}
Version: 18.03.1-ee-3
MinAPIVersion: 1.12
GitCommit: b9a5c95
Os: linux
GoVersion: go1.10.2
Starting build of index.docker.io/rajatgupta56/apptest:0.0.1-SNAPSHOT...
Step 1/8 : FROM openjdk:8-jdk-alpine
---> a3562aa0b991
Step 2/8 : LABEL maintainer="Rajatgupta828#gmail.com"
---> Running in 4e55966615dc
Removing intermediate container 4e55966615dc
---> 670b63467701
Step 3/8 : VOLUME /tmp/${pwd}
---> Running in 3a33ba4471a8
Removing intermediate container 3a33ba4471a8
---> 182122656d6a
Step 4/8 : WORKDIR /
Removing intermediate container 9344531c842f
---> ae012d63578d
Step 5/8 : EXPOSE 8080
---> Running in 0be952ee63fc
Removing intermediate container 0be952ee63fc
---> 5065a1cd7212
Step 6/8 : ARG JAR_FILE=target/ApplicationTest-0.0.1-SNAPSHOT.jar
---> Running in a6def98cd21d
Removing intermediate container a6def98cd21d
---> 5a77e9b6d578
Step 7/8 : ADD ${JAR_FILE} appliation-test.jar
ADD failed: stat /var/lib/docker/tmp/docker-builder579146423/target/ApplicationTest-0.0.1-SNAPSHOT.jar: no such file or directory
My POM File is :
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.2.7.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.application</groupId>
<artifactId>ApplicationTest</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>ApplicationTest</name>
<description>Demo project for Spring Boot</description>
<properties>
<java.version>1.8</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-rest</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>org.junit.vintage</groupId>
<artifactId>junit-vintage-engine</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>dev</id>
<properties>
<activatedProperties>dev</activatedProperties>
</properties>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
</profile>
<profile>
<id>prod</id>
<properties>
<activatedProperties>prod</activatedProperties>
</properties>
</profile>
</profiles>
</project>
And my DockerFile is :
# Start with a base image containing Java runtime
FROM openjdk:8-jdk-alpine
#Add Maintainer Info
LABEL maintainer="Rajatgupta828#gmail.com"
# Add a volume pointing to /tmp
VOLUME /tmp/${pwd}
WORKDIR /
# Make port 8080 available to the world outside this container
EXPOSE 8080
# The application's jar file
ARG JAR_FILE=target/ApplicationTest-0.0.1-SNAPSHOT.jar
# Add the application's jar to the container
ADD ${JAR_FILE} appliation-test.jar
# Run the jar file
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/appliation-test.jar"]
I am following below tutorial :
https://www.callicoder.com/spring-boot-docker-example/
Dockerfile is located in my root library :
My Project

Springboot and Mongo containers are not connecting

I'm running 2 containers on my local machine using docker compose. The first one is mongo container and the second is spring boot application.
This is my spring boot's Dockerfile
VOLUME /tmp
ADD target/app.jar app.jar
EXPOSE 8080
RUN bash -c "touch /app.jar"
ENTRYPOINT ["java","-Dspring.data.mongodb.uri=mongodb://mongo/test", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
This is my spring boot's application properties
# mongodb configuration0
dockerspring.data.mongodb.uri= mongodb://mongo:27000/test
This is my docker-compose.yml
version: '3.1'
services:
level-2-springboot:
image: level-2:latest
restart: always
container_name: level_2_microservice
ports:
- 8081:8080
working_dir: /tmp
depends_on:
- mongo
mongo:
image: mongo
container_name: mongo
ports:
- 27000:27017
restart: always
This is my pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.2.6.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.yash</groupId>
<artifactId>spring-sync-ms</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>spring-sync</name>
<description>Demo project for Spring Boot</description>
<properties>
<java.version>1.8</java.version>
<maven-jar-plugin.version>3.1.1</maven-jar-plugin.version>
</properties>
<dependencies>
<dependency>
<groupId>org.mongodb</groupId>
<artifactId>mongo-java-driver</artifactId>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>org.junit.vintage</groupId>
<artifactId>junit-vintage-engine</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
<finalName>app</finalName>
</build>
</project>
I execute this to create image for my spring boot app
docker build -t level-2 .
After I see my image, I execute docker-compose up and they work just fine no error
Here is the containers running
But when i do a POST or GET request to localhost:8081/api/employees/ or localhost:8080/api/employees/ I'd get this message from Postman
Could not get any response
Below is my project structure
If I run the app on eclipse and mongo manually and do POST/GET request, I'd get 201 OK, but when I do it using docker in containers, they dont talk to each other. Anyone know whats causing this?
I've consulted the internet and i think i got what you need.
The question that follows is “how does our service container talk to
the Mongo container?” For that, we get into container linking.
When you run a container, you can pass an optional –link parameter,
specifying the running container name that the new container needs to
be able to communicate with.
So with our command
docker run -P -d --name employee --link mongodb microservicedemo/employee
we fire up our new container image, exposing its ports (-P) in the background
(-d), naming it employee (–name), and telling it to link to the
container named “mongodb” (–link). Linking these does the following:
Creates a hosts file entry in the employee container pointing at the
running location of the MongoDB container Inserts a number of
environment variables in the employee container to assist with any
other programmatic access needed.
To see them run: docker exec employee bash -c 'env |grep MONGODB'
Allows containers to communicate directly over ports exposed, so there is no need to
worry about the hose machine part mapping.
see:
https://www.3pillarglobal.com/insights/building-a-microservice-architecture-with-spring-boot-and-docker-part-iii
https://www.3pillarglobal.com/insights/building-a-microservice-architecture-with-spring-boot-and-docker-part-ii
This:
# mongodb configuration0
dockerspring.data.mongodb.uri= mongodb://mongo:27000/test
Should be:
# mongodb configuration0
dockerspring.data.mongodb.uri= mongodb://mongo:27017/test
The two services in your docker-compose.yml expose Ports 8081 and 27000 to the host machine.
But inside the network created by docker-compose they expose Ports 8080 and 27017.
As your Mongo-Container runs in the docker-compose network, it has to use Port 27017 to connect to the database.

Spring Cloud Data Flow Streams Error: Unable to access jarfile

I launch Spring cloud data flow with docker-compose base on this website.
https://dataflow.spring.io/docs/installation/local/docker/
I created 3 apps, Source, Processor & Sink.
I registered them on the Spring Data Flow Cloud Dashboard.
Then I created a stream with the source connecting to processor connecting to sink.
When I deployed the apps, and opened http://localhost:9393/streams/logs/{name-of-stream},
I get the following error,
{
logs: {
testdir-stream.sender1-v3: "Error: Unable to access jarfile /testdir/data-sender-0.0.1-SNAPSHOT.jar ",
testdir-stream.sink1-v3: "Error: Unable to access jarfile /testdir/data-sink-0.0.1-SNAPSHOT.jar ",
testdir-stream.processor1-v3: "Error: Unable to access jarfile /testdir/data-processor-0.0.1-SNAPSHOT.jar "
}
}
Apparently, the Spring Cloud Data Flow (SCDF) server cannot access(or executing it or something?) the jarfile. I have chmod +x the entire testdir folder in the docker container such that the permissions is 'drwxr-xrwx'.
However, SCDF is still unable to access the jar. How can I solve this problem? Thanks.
EDIT
I am running docker-compose -f ./docker-compose.yml -f ./docker-compose-prometheus.yml up
docker-compose.yml
version: '3'
services:
mysql:
image: mysql:5.7.25
container_name: dataflow-mysql
environment:
MYSQL_DATABASE: dataflow
MYSQL_USER: root
MYSQL_ROOT_PASSWORD: rootpw
expose:
- 3306
kafka-0:
image: confluentinc/cp-kafka:5.2.1
container_name: dataflow-kafka-0
expose:
- "9092"
environment:
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-0:9092
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
- KAFKA_ADVERTISED_HOST_NAME=kafka-0
- KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
depends_on:
- zookeeper
kafka-1:
image: confluentinc/cp-kafka:5.2.1
container_name: dataflow-kafka-1
expose:
- "9093"
environment:
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-1:9093
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
- KAFKA_ADVERTISED_HOST_NAME=kafka-1
- KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
depends_on:
- zookeeper
zookeeper:
image: confluentinc/cp-zookeeper:5.2.1
container_name: dataflow-kafka-zookeeper
expose:
- "2181"
environment:
- ZOOKEEPER_CLIENT_PORT=2181
dataflow-server:
image: springcloud/spring-cloud-dataflow-server:${DATAFLOW_VERSION:?DATAFLOW_VERSION is not set!}
container_name: dataflow-server
volumes:
- 'C:\Users\Desmond\Desktop\Desmond\Project\springcloud:/root/apps'
ports:
- "9393:9393"
environment:
- spring.cloud.dataflow.applicationProperties.stream.spring.cloud.stream.kafka.binder.brokers=PLAINTEXT://kafka-0:9092,PLAINTEXT://kafka-1:9093
- spring.cloud.dataflow.applicationProperties.stream.spring.cloud.stream.kafka.streams.binder.brokers=PLAINTEXT://kafka-0:9092,PLAINTEXT://kafka-1:9093
- spring.cloud.dataflow.applicationProperties.stream.spring.cloud.stream.kafka.binder.zkNodes=zookeeper:2181
- spring.cloud.dataflow.applicationProperties.stream.spring.cloud.stream.kafka.streams.binder.zkNodes=zookeeper:2181
- spring.cloud.skipper.client.serverUri=http://skipper-server:7577/api
- SPRING_DATASOURCE_URL=jdbc:mysql://mysql:3306/dataflow
- SPRING_DATASOURCE_USERNAME=root
- SPRING_DATASOURCE_PASSWORD=rootpw
- SPRING_DATASOURCE_DRIVER_CLASS_NAME=org.mariadb.jdbc.Driver
depends_on:
- kafka-0
- kafka-1
entrypoint: "./wait-for-it.sh mysql:3306 -- java -jar /maven/spring-cloud-dataflow-server.jar"
app-import:
image: springcloud/openjdk:2.0.0.RELEASE
container_name: dataflow-app-import
depends_on:
- dataflow-server
command: >
/bin/sh -c "
./wait-for-it.sh -t 180 dataflow-server:9393;
wget -qO- 'http://dataflow-server:9393/apps' --post-data='uri=${STREAM_APPS_URI:-https://dataflow.spring.io/kafka-maven-latest&force=true}';
echo 'Stream apps imported'
wget -qO- 'http://dataflow-server:9393/apps' --post-data='uri=${TASK_APPS_URI:-https://dataflow.spring.io/task-maven-latest&force=true}';
echo 'Task apps imported'"
skipper-server:
image: springcloud/spring-cloud-skipper-server:${SKIPPER_VERSION:?SKIPPER_VERSION is not set!}
container_name: skipper
ports:
- "7577:7577"
- "9000-9010:9000-9010"
- "20000-20105:20000-20105"
environment:
- SPRING_CLOUD_SKIPPER_SERVER_PLATFORM_LOCAL_ACCOUNTS_DEFAULT_PORTRANGE_LOW=20000
- SPRING_CLOUD_SKIPPER_SERVER_PLATFORM_LOCAL_ACCOUNTS_DEFAULT_PORTRANGE_HIGH=20100
- SPRING_DATASOURCE_URL=jdbc:mysql://mysql:3306/dataflow
- SPRING_DATASOURCE_USERNAME=root
- SPRING_DATASOURCE_PASSWORD=rootpw
- SPRING_DATASOURCE_DRIVER_CLASS_NAME=org.mariadb.jdbc.Driver
entrypoint: "./wait-for-it.sh mysql:3306 -- java -Djava.security.egd=file:/dev/./urandom -jar /spring-cloud-skipper-server.jar"
docker-compose-prometheus.yml
version: '3'
# Extends the default docker-compose.yml with Prometheus/Grafana monitoring configuration
# Usage: docker-compose -f ./docker-compose.yml -f ./docker-compose-prometheus.yml up
services:
dataflow-server:
environment:
- |
SPRING_APPLICATION_JSON=
{
"spring.cloud.dataflow.applicationProperties":{
"task.management.metrics.export.prometheus":{
"enabled":true,
"rsocket.enabled":true,
"rsocket.host":"prometheus-rsocket-proxy",
"rsocket.port":7001
},
"stream.management.metrics.export.prometheus":{
"enabled":true,
"rsocket.enabled":true,
"rsocket.host":"prometheus-rsocket-proxy",
"rsocket.port":7001
}
},
"spring.cloud.dataflow.grafana-info.url":"http://localhost:3000"
}
prometheus-rsocket-proxy:
image: micrometermetrics/prometheus-rsocket-proxy:0.8.0
container_name: prometheus-rsocket-proxy
expose:
- '9096'
- '7001'
- '8081'
ports:
- '9096:9096'
- '7001:7001'
- '8081:8081'
environment:
- server.port=9096
grafana:
image: springcloud/spring-cloud-dataflow-grafana-prometheus:${DATAFLOW_VERSION:?DATAFLOW_VERSION is not set! Use 'export DATAFLOW_VERSION=local-server-image-tag'}
container_name: grafana
ports:
- '3000:3000'
prometheus:
image: springcloud/spring-cloud-dataflow-prometheus-local:${DATAFLOW_VERSION:?DATAFLOW_VERSION is not set! Use 'export DATAFLOW_VERSION=local-server-image-tag'}
container_name: prometheus
ports:
- '9090:9090'
Pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.1.7.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.example</groupId>
<artifactId>data-sender</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>data-sender</name>
<description>Test Proj</description>
<properties>
<java.version>1.8</java.version>
<spring-cloud.version>Greenwich.SR2</spring-cloud.version>
<mockito-junit-jupiter.version>2.19.0</mockito-junit-jupiter.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-test-support</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.4</version>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<version>${junit-jupiter.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-junit-jupiter</artifactId>
<version>${mockito-junit-jupiter.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</dependency>
<!-- https://mvnrepository.com/artifact/com.googlecode.json-simple/json-simple -->
<dependency>
<groupId>com.googlecode.json-simple</groupId>
<artifactId>json-simple</artifactId>
<version>1.1.1</version>
</dependency>
<dependency>
<groupId>net.lingala.zip4j</groupId>
<artifactId>zip4j</artifactId>
<version>1.3.2</version>
</dependency>
<!-- https://mvnrepository.com/artifact/commons-io/commons-io -->
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.6</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.9</version>
</dependency>
<!-- prometheus-rsocket monitoring -->
<dependency>
<groupId>org.springframework.cloud.stream.app</groupId>
<artifactId>app-starters-micrometer-common</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer.prometheus</groupId>
<artifactId>prometheus-rsocket-spring</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-cloudfoundry-connector</artifactId>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${spring-cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
For accessing custom applications from your local machine, you need to configure both the dataflow-server and the skipper-skipper Docker containers to access to your local file system. You have already mounted the source folder to dataflow-server but you need to do the same with the skipper-server definition like this:
skipper-server:
image: springcloud/spring-cloud-skipper-server:${SKIPPER_VERSION:?SKIPPER_VERSION is not set!}
container_name: skipper
volumes:
- 'C:\Users\Desmond\Desktop\Desmond\Project\springcloud:/root/apps'
...
We will clarify this in our documentation #128
Two additional points regarding prometheus-rsocket monitoring:
Since you are using the new prometheus-rsocket configuration you will need to import the Einstein-BUILD-SNAPSHOT-stream-applications-kafka-maven app starters instead of kafka-maven-latest. You can do this by setting the following property before starting docker-compose:
export STREAM_APPS_URI=https://dataflow.spring.io/Einstein-BUILD-SNAPSHOT-stream-applications-kafka-maven
The SR4 app starter release is expected very soon, but until then you need to use the BUILD-SNAPSHOT to leverage the prometheus metrics.
Can you please share how you have implemented your custom apps? Do they contain the following dependencies?
<dependencyManagement>
<dependencies>
...
<dependency>
<groupId>org.springframework.cloud.stream.app</groupId>
<artifactId>app-starters-core-dependencies</artifactId>
<version>2.1.2.BUILD-SNAPSHOT</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
....
<dependency>
<groupId>org.springframework.cloud.stream.app</groupId>
<artifactId>app-starters-micrometer-common</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer.prometheus</groupId>
<artifactId>prometheus-rsocket-spring</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-cloudfoundry-connector</artifactId>
</dependency>
</dependencies>
<repositories>
<repository>
<snapshots>
<enabled>true</enabled>
</snapshots>
<id>spring-snapshots</id>
<name>Spring Snapshots</name>
<url>https://repo.spring.io/snapshot</url>
</repository>
...
</repositories>
If your apps's parent POM is org.springframework.cloud.stream.app:app-starters-build:2.1.2.BUILD-SNAPSHOT above dependencies are already injected. Otherwise you have to add them manually.
UPDATE -- adding a simple pom configuration to enable prometheus-rsocket metrics:
<properties>
<spring-cloud.version>Greenwich.SR2</spring-cloud.version>-->
<prometheus-rsocket-spring.version>0.8.0</prometheus-rsocket-spring.version>
</properties>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer.prometheus</groupId>
<artifactId>prometheus-rsocket-spring</artifactId>
<version>${prometheus-rsocket-spring.version}</version>
</dependency>
<dependency>
<groupId>io.micrometer.prometheus</groupId>
<artifactId>prometheus-rsocket-client</artifactId>
<version>${prometheus-rsocket-spring.version}</version>
</dependency>
I believe container configs are wrong for skipper server as that's the one running those containers if local setup is used. It should work if same volume is used in skipper as it's now done with dataflow server.
For getting those logs, dataflow simply requests those from a skipper and error originates from there.

Resources