Skaffold dev works with minikube only. Other on-prem cluster fails - spring-boot

Question

I have a Spring Boot app with jib-maven configured
POM
<plugin>
<groupId>com.google.cloud.tools</groupId>
<artifactId>jib-maven-plugin</artifactId>
<version>2.1.0</version>
<configuration>
<from>
<image>openjdk:11-jre-slim</image>
</from>
<to>
<image>registry.demo/${project.artifactId}</image>
<tags>
<tag>${project.version}</tag>
</tags>
<tags>
<tag>latest</tag>
</tags>
</to>
<container>
<jvmFlags>
<jvmFlag>-XX:+UseContainerSupport</jvmFlag>
<jvmFlag>-XX:MinRAMPercentage=60.0</jvmFlag>
<jvmFlag>-XX:MaxRAMPercentage=90.0</jvmFlag>
<jvmFlag> -XshowSettings:vm</jvmFlag>
</jvmFlags>
<mainClass>com.demo.DemoApplication</mainClass>
</container>
</configuration>
SKAFFOLD.YAML
apiVersion: skaffold/v2beta1
kind: Config
metadata:
name: springtokube
build:
artifacts:
- image: registry.demo/springtokube
jib:
project: com.demo:springtokube
local:
push: true
concurrency: 1
useBuildkit: false
useDockerCLI: true
deploy:
kubectl:
manifests:
- deployment.yaml
ALSO SET INSECURE REGISTRY
skaffold config set --global insecure-registries registry.demo
But when using minikube I can run successfully
skaffold dev
When using other cluster (ON-PREM) I get
FATA[0016] exiting dev mode because first build failed: build failed: building [registry.demo/springtokube]: build artifact: getting image: GET http://registry.demo/v2/: : Not Found
What might be the problem?
I restarted today using kubectl context
skaffold debug --no-prune=false --cache-artifacts=false
And It Failed
Listing files to watch...
Generating tags...
- registry.demo/springtokube -> registry.demo/springtokube:cf60c31
Found [minikube] context, using local docker daemon.
Building [registry.demo/springtokube]...
.............
...............
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.294 s - in com.demo.springtokube.SpringtokubeApplicationTests
2020-04-15 08:45:48.277 INFO 30662 --- [extShutdownHook] o.s.s.concurrent.ThreadPoolTaskExecutor : Shutting down ExecutorService 'applicationTaskExecutor'
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- maven-jar-plugin:3.1.2:jar (default-jar) # springtokube ---
[INFO] Building jar: ....../springtokube/target/springtokube.jar
[INFO]
[INFO] --- spring-boot-maven-plugin:2.2.6.RELEASE:repackage (repackage) # springtokube ---
[INFO] Replacing main artifact with repackaged archive
[INFO]
[INFO] --- jib-maven-plugin:2.1.0:build (default-cli) # springtokube ---
[INFO]
[INFO] Containerizing application to registry.demo/springtokube:cf60c31, registry.demo/springtokube...
[WARNING] Base image 'openjdk:11-jre-slim' does not use a specific image digest - build may not be reproducible
[INFO] Getting manifest for base image openjdk:11-jre-slim...
[INFO] Building dependencies layer...
[INFO] Building resources layer...
[INFO] Building classes layer...
[INFO] Using credentials from Docker config (~/.docker/config.json) for registry.demo/springtokube:cf60c31
[WARNING] Cannot verify server at https://registry.demo/v2/. Attempting again with no TLS verification.
[WARNING] Cannot verify server at https://registry.demo/v2/springtokube/blobs/sha256:1fb3fb86aa52691fa3705554da5ba07dcb556f62a93ba7efab0e397ca3db092c. Attempting again with no TLS verification.
[WARNING] Cannot verify server at https://registry.demo/v2/springtokube/blobs/sha256:88a7d9887f9fdeb5a4736d07c64818453e00e71fe916b13f413eb6e545445a68. Attempting again with no TLS verification.
[WARNING] Cannot verify server at https://registry.demo/v2/springtokube/blobs/sha256:a6c851c4b90b9eb7af89d240dd4f438dba9feba5c78600fed7eadddf8cb7b647. Attempting again with no TLS verification.
[INFO] The base image requires auth. Trying again for openjdk:11-jre-slim...
[INFO] Using credentials from Docker config (~/.docker/config.json) for openjdk:11-jre-slim
[INFO] Using base image with digest: sha256:01669f539159a1b5dd69c4782be9cc7da0ac1f4ddc5e2c2d871ef1481efd693e
[INFO]
[INFO] Container entrypoint set to [java, -XX:+UseContainerSupport, -XX:MinRAMPercentage=60.0, -XX:MaxRAMPercentage=90.0, -XshowSettings:vm, -cp, /app/resources:/app/classes:/app/libs/*, com.demo.springtokube.SpringtokubeApplication]
[INFO]
[INFO] Built and pushed image as registry.demo/springtokube:cf60c31, registry.demo/springtokube
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 20.058 s
[INFO] Finished at: 2020-04-15T08:45:57+03:00
[INFO] ------------------------------------------------------------------------
Pruning images...
FATA[0024] exiting dev mode because first build failed: build failed: building [registry.demo/springtokube]: build artifact: getting image: GET http://registry.demo/v2/: : Not Found
I thought the minikube works. But disabling cache fails to build
if I run
skaffold debug OR skaffold dev
Works Fine
But if I run with cache disabled
skaffold debug --no-prune=false --cache-artifacts=false
FAILS it shows the logs above

Answer 1

After days of struggling I found a solution.
Following Brian de Alwis suggestions I was able to make Skaffold work with Self Signed Certificate.
Skaffold build or dev does not use certificate put in.
/etc/docker/certs.d/myregistrydomain.com/ca.crt
The path is used by docker client only.
The solution was to put yout registry certificate into
/usr/local/share/ca-certificates/myregistrydomain.com.crt
Then
update-ca-certificates
Check The link for more info
If you are using self signed certificate no need for insecure registry in your scaffold yaml file
apiVersion: skaffold/v2beta1
kind: Config
metadata:
name: springtokube
build:
# insecureRegistries:
# - myregistrydomain.com
Or Running skaffold with
skaffold dev --insecure-registry=myregistrydomain.com
Hope this help someone else struggling to make skaffold works with self signed certificate