containerd pass incorrect hostPath from host - containerd

everybody!
I try use hostPath in deployment, instead docker I use containerd (kubernetes version 1.21 and containerd version 1.5
, but creating pod failed with error, I guess may be I have some issue with config.toml of containerd
Error: failed to create containerd task: failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/run/containerd/io.containerd.runtime.v2.task/k8s.io/host_dir/config/services-config.json" to rootfs at "/app/config/local.json": stat /run/containerd/io.containerd.runtime.v2.task/k8s.io/container_inside_dir/service/config/services-config.json: no such file or directory: unknown

Related

gitlab-runner shows me an error on my server with docker

I discover docker. I installed gitlab-runner with a docker image and it works.
i linked gitlab.com to my runner and it works.
However, in the .gitlab-ci.yml when I docker builds I have an error:
dial tcp: lookup docker on x.x.x.9:53: no such host
$ docker info
Client:
Context: default
Debug Mode: false
Server:
ERROR: error during connect: Get http://docker:2375/v1.24/info: dial tcp: lookup docker on x.x.x.9:53: no such host
errors pretty printing info
I can't build my spring-boot image thank

Cannot connect to the Docker daemon at tcp://xxx.xxx.xx.xxx:2375. Is the docker daemon running?

I'm using Gitlab and Docker to get continuous integration to my spring boot application and I'm getting this error:
Cannot connect to the Docker daemon at tcp://xxx.xxx.xx.xxx:2375. Is the docker daemon running?
.development.env:
export SPRING_ACTIVE_PROFILE='development'
export DOCKER_REPO='DOCKER_HUB_ID/app_name:dev'
export APP_NAME='app_name_dev'
export PORT='8080'
export SERVER_IP='xxx.xxx.xx.xxx' #SERVER_IP
export SERVER_SSH_KEY="$DEV_SSH_PRIVATE_KEY"
export DOCKER_HOST='tcp://xxx.xxx.xx.xxx:2375' #SERVER_IP
.gitlab-ci.yml
services:
- docker:19.03.7-dind
stages:
- build and push docker image
docker build:
image: docker:stable
stage: build and push docker image
before_script:
- source .${CI_COMMIT_REF_NAME}.env #.development.env
script:
- docker build --build-arg SPRING_ACTIVE_PROFILE=$SPRING_ACTIVE_PROFILE -t $DOCKER_REPO .
- docker login -u $DOCKER_USER -p $DOCKER_PASSWORD docker.io
- docker push $DOCKER_REPO
This is the whole logs from gitlab:
Running with gitlab-runner 13.5.0 (ece86343)
on gitlab-server JuhWVkPJ
Preparing the "docker" executor
00:38
Using Docker executor with image docker:stable ...
Starting service docker:19.03.7-dind ...
Pulling docker image docker:19.03.7-dind ...
Using docker image sha256:14af3ba31e635475ec8f7fbe17470424514777621e627a91c41bbbe028dbae16 for docker:19.03.7-dind with digest docker#sha256:2683fcdf7480ea101415833f7793fb058c5f20227890a953b0a70bfc350af5bc ...
Waiting for services to be up and running...
*** WARNING: Service runner-juhwvkpj-project-13-concurrent-0-7c99eb8ace2e2ae6-docker-0 probably didn't start properly.
Health check error:
service "runner-juhwvkpj-project-13-concurrent-0-7c99eb8ace2e2ae6-docker-0-wait-for-service" timeout
Health check container logs:
Service container logs:
2020-12-30T03:14:07.879506461Z Generating RSA private key, 4096 bit long modulus (2 primes)
2020-12-30T03:14:08.459745140Z ..............................................++++
2020-12-30T03:14:08.673203110Z ..................++++
2020-12-30T03:14:08.673231544Z e is 65537 (0x010001)
2020-12-30T03:14:08.713960023Z Generating RSA private key, 4096 bit long modulus (2 primes)
2020-12-30T03:14:08.851463609Z ..............++++
2020-12-30T03:14:09.403244538Z .....................................................++++
2020-12-30T03:14:09.403286293Z e is 65537 (0x010001)
2020-12-30T03:14:09.516423752Z Signature ok
2020-12-30T03:14:09.516463300Z subject=CN = docker:dind server
2020-12-30T03:14:09.516471290Z Getting CA Private Key
2020-12-30T03:14:09.536975767Z /certs/server/cert.pem: OK
2020-12-30T03:14:09.553642146Z Generating RSA private key, 4096 bit long modulus (2 primes)
2020-12-30T03:14:09.927078677Z ...................................................++++
2020-12-30T03:14:10.107451624Z ...................++++
2020-12-30T03:14:10.108457646Z e is 65537 (0x010001)
2020-12-30T03:14:10.156096074Z Signature ok
2020-12-30T03:14:10.156125397Z subject=CN = docker:dind client
2020-12-30T03:14:10.156302268Z Getting CA Private Key
2020-12-30T03:14:10.178703934Z /certs/client/cert.pem: OK
2020-12-30T03:14:10.194290163Z mount: permission denied (are you root?)
2020-12-30T03:14:10.194438175Z Could not mount /sys/kernel/security.
2020-12-30T03:14:10.194456604Z AppArmor detection and --privileged mode might break.
2020-12-30T03:14:10.195933829Z mount: permission denied (are you root?)
*********
Pulling docker image docker:stable ...
Using docker image sha256:b0757c55a1fdbb59c378fd34dde3e12bd25f68094dd69546cf5ca00ddbaa7a33 for docker:stable with digest docker#sha256:fd4d028713fd05a1fb896412805daed82c4a0cc84331d8dad00cb596d7ce3e3a ...
Preparing environment
00:01
Running on runner-juhwvkpj-project-13-concurrent-0 via gitlab-server...
Getting source from Git repository
00:03
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/abdallah/harvis/.git/
Checking out 5568bbc9 as DM_Module...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:36
$ source .${CI_COMMIT_REF_NAME}.env
$ whoami
root
$ docker build --build-arg SPRING_ACTIVE_PROFILE=$SPRING_ACTIVE_PROFILE -t $DOCKER_REPO .
Cannot connect to the Docker daemon at tcp://xxx.xxx.xx.xxx:2375. Is the docker daemon running?
Cleaning up file based variables
00:01
ERROR: Job failed: exit code 1
I think the error is in DOCKER_HOST, what should I assign it?
I'd appreciate any help or suggestion.
The Docker daemon can listen for Docker Engine API requests via three different types of Socket: unix, tcp, and fd
your docker daemon is trying to connect via tcp (2375), and it seems not enabled.
you need to start docker daemon with -H tcp://<ip>:2375
or put it in /etc/docker/daemon.json
"hosts": ["tcp://<ip>:2375", "unix:///var/run/docker.sock"],
EDIT
Binding to 0.0.0.0 is dangerous as David pointed out.

Windows Docker Desktop Linux mode - docker container time skew

Question: How can I map docker container time to my local PC time to sync the time inside the docker container?
From my windows 10 PC, I am running Linux mode docker desktop version 2.2.0.4 (43472), docker Engine 19.03.8.
All the docker containers created are showing massive time skew from that of the host:
From centos 8 docker container:
[root# /]# date
Thu May 7 01:18:16 UTC 2020
From docker host running Window Doker desktop on Windows 10 PC:
PS> date
14 May 2020 14:42:17
I tried to create a new container with -v option as below:
docker container run -it -v c:\docker_volumes\docker1:/storage -v /etc/localtime:/etc/localtime:ro --name centos7-squid centos:7.7.1908 /bin/bash
I get the error below
Unable to find image 'centos:7.7.1908' locally
7.7.1908: Pulling from library/centos
f34b00c7da20: Pull complete Digest: sha256:50752af5182c6cd5518e3e91d48f7ff0cba93d5d760a67ac140e2d63c4dd9efc
Status: Downloaded newer image for centos:7.7.1908
C:\Program Files\Docker\Docker\resources\bin\docker.exe: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\"/etc/localtime\\" to rootfs \\"/var/lib/docker/overlay2/c7e86cffdc46c354f19b25fa97146ce8f2caee653793219719b043c97040d1b7/merged\\" at \\"/var/lib/docker/overlay2/c7e86cffdc46c354f19b25fa97146ce8f2caee653793219719b043c97040d1b7/merged/usr/share/zoneinfo/UTC\\" caused \\"not a directory\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type.
I fixed it by setting the hardware clock of the virtual machine running docker:
docker run --rm --privileged alpine hwclock -s
credit:
https://blog.jverkamp.com/2017/11/15/clock-drift-in-docker-containers/

Minikube start issue on AWS EC2 ubuntu bionic

I have got AWS EC2 instance started.
Logged onto the instance using Putty client.
Below all commands I tried on EC2 instance which is Ubuntu 18.04.
Configuration details of EC2 are as follows
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.4 LTS
Release: 18.04
Codename: bionic
I have installed docker and virtualbox
~$ docker --version
Docker version 19.03.8, build afacb8b7f0
$ dpkg -l | grep 'virtualbox'
ii virtualbox-5.2 5.2.30-130521~Ubuntu~bionic amd64 Oracle VM VirtualBox
$ lscpu | grep Virtualization
Virtualization type: full
I tried to start minikube.
$ minikube start
* minikube 1.9.2 is available! Download it: https://github.com/kubernetes/minikube/releases/tag/v1.9.2
* To disable this notice, run: 'minikube config set WantUpdateNotification false'
! minikube v1.9.0 on Ubuntu 18.04
* Using the virtualbox driver based on existing profile
* Creating virtualbox VM (CPUs=2, Memory=1993MB, Disk=20000MB) ...
! StartHost failed, but will try again: creating host: create: precreate: This computer doesn't have VT-X/AMD-v enabled. Enabling it in the BIOS is mandatory
* Creating virtualbox VM (CPUs=2, Memory=1993MB, Disk=20000MB) ...
* StartHost failed again: creating host: create: precreate: This computer doesn't have VT-X/AMD-v enabled. Enabling it in the BIOS is mandatory
- Run: "minikube delete", then "minikube start --alsologtostderr -v=1" to try again with more logging
*
E0408 16:46:12.812503 5285 style.go:173] unable to parse "X Unable to start VM after repeated tries. Please try {{'minikube delete' if possible\n": template: X Unable to start VM after repeated tries. Please try {{'minikube delete' if possible
:1: malformed character constant: 'minikube delete' - returning raw string.
X Unable to start VM after repeated tries. Please try {{'minikube delete' if possible
* Error: [VBOX_VTX_DISABLED] creating host: create: precreate: This computer doesn't have VT-X/AMD-v enabled. Enabling it in the BIOS is mandatory
* Suggestion: Virtualization support is disabled on your computer. If you are running minikube within a VM, try '--driver=none'. Otherwise, consult your systems BIOS manual for how to enable virtualization.
* Related issues:
- https://github.com/kubernetes/minikube/issues/3900
- https://github.com/kubernetes/minikube/issues/4730
Based on error message, I also tried below command:
$ minikube start --no-vtx-check
* minikube v1.9.0 on Ubuntu 18.04
* Automatically selected the virtualbox driver
* Creating virtualbox VM (CPUs=2, Memory=1993MB, Disk=20000MB) ...
! StartHost failed, but will try again: creating host: create: creating: Unable to start the VM: /usr/bin/VBoxManage startvm minikube --type headless failed:
VBoxManage: error: VT-x is not available (VERR_VMX_NO_VMX)
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component ConsoleWrap, interface IConsole
Details: 00:00:00.455234 Power up failed (vrc=VERR_VMX_NO_VMX, rc=NS_ERROR_FAILURE (0X80004005))
* Deleting "minikube" in virtualbox ...
* Creating virtualbox VM (CPUs=2, Memory=1993MB, Disk=20000MB) ...
* StartHost failed again: creating host: create: creating: Unable to start the VM: /usr/bin/VBoxManage startvm minikube --type headless failed:
VBoxManage: error: VT-x is not available (VERR_VMX_NO_VMX)
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component ConsoleWrap, interface IConsole
Details: 00:00:00.270050 Power up failed (vrc=VERR_VMX_NO_VMX, rc=NS_ERROR_FAILURE (0X80004005))
- Run: "minikube delete", then "minikube start --alsologtostderr -v=1" to try again with more logging
*
E0408 16:52:12.324862 5411 style.go:173] unable to parse "X Unable to start VM after repeated tries. Please try {{'minikube delete' if possible\n": template: X Unable to start VM after repeated tries. Please try {{'minikube delete' if possible
:1: malformed character constant: 'minikube delete' - returning raw string.
X Unable to start VM after repeated tries. Please try {{'minikube delete' if possible
* Error: [VBOX_VERR_VMX_NO_VMX] creating host: create: creating: Unable to start the VM: /usr/bin/VBoxManage startvm minikube --type headless failed:
VBoxManage: error: VT-x is not available (VERR_VMX_NO_VMX)
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component ConsoleWrap, interface IConsole
Details: 00:00:00.270050 Power up failed (vrc=VERR_VMX_NO_VMX, rc=NS_ERROR_FAILURE (0X80004005))
* Suggestion: Your host does not support virtualization. If you are running minikube within a VM, try '--driver=none'. Otherwise, enable virtualization in your BIOS
* Related issues:
- https://github.com/kubernetes/minikube/issues/1994
- https://github.com/kubernetes/minikube/issues/5326
I went to all those jira specified in error log but of no use. Most of those doesn't talk about issue within Amazon EC2 ubuntu 18.04 .
Questions:
how do I enable VT-X/AMD-v in BIOS in logged in EC2 instance which is ubuntu 18.04? Is there any command I can try? note: I don't have access to go to AWS EC2 instance GUI.
is AWS EC2 Ubuntu 18 instance compatible with virtualbox 5 or 6 to start minikube ?
Questions: how do I enable VT-X/AMD-v in BIOS in logged in EC2 instance which is ubuntu 18.04? Is there any command I can try? note: I don't have access to go to AWS EC2 instance GUI.
You don't, EC2 is a virtual server and You have no access to the BIOS.
To use Minikube on EC2 you need to start it without vm driver so just execute $ minikube start --vm-driver=none. This is described here if you insist on using Minikube.
Instead of installing Minikube I would recommend using kubeadm and setting single control-plane cluster with kubeadm.

minikube/kubectl, not running locally in Win10-pro-64-bit-laptop

GitHub issue link :
https://github.com/kubernetes/minikube/issues/1994
Docker on Windows installed already, docker is running fine and the containers are running fine.
Docker stopped and made running as well , for running K8s setup. Both scenarios, same status- minikube did not start.
I installed Google Cloud SDK and then started minikube/kubectl for setting K8S, FYI.
Hyper-V was running all along these setup, in my laptop.
Trying to run K8S locally on my laptop using {kubernetes.io/docs/getting-started-guides/minikube/} and observed the issues below.
Started kubectl cluster-info.
C:>kubectl cluster-info
Kubernetes master is running at https://192.168.99.100:8443
and then exited after sometime.
minikube version command was run.
Output is as follows :
C:>minikube-windows-amd64 version
minikube version: v0.21.0
minikube start command was run.
Output is as follows :
C:>minikube-windows-amd64 start
Starting local Kubernetes v1.7.0 cluster...
Starting VM...
E0921 18:52:12.921824 10592 start.go:129] Error starting host: Error starting stopped host: Unable to start the VM: C:\Program Files\Oracle\VirtualBox\VBoxManage.exe startvm minikube --type headless failed:
VBoxManage.exe: error: VT-x is not available (VERR_VMX_NO_VMX)
VBoxManage.exe: error: Details: code E_FAIL (0x80004005), component ConsoleWrap, interface IConsole
Details: 00:00:01.552591 Power up failed (vrc=VERR_VMX_NO_VMX, rc=E_FAIL (0X80004005)).
Retrying.
E0921 18:52:12.924743 10592 start.go:135] Error starting host: Error starting stopped host: Unable to start the VM: C:\Program Files\Oracle\VirtualBox\VBoxManage.exe startvm minikube --type headless failed:
VBoxManage.exe: error: VT-x is not available (VERR_VMX_NO_VMX)
VBoxManage.exe: error: Details: code E_FAIL (0x80004005), component ConsoleWrap, interface IConsole
Details: 00:00:01.552591 Power up failed (vrc=VERR_VMX_NO_VMX, rc=E_FAIL (0X80004005))
I then ran kubectl cluster-info dump. Output is as follows :
C:>kubectl cluster-info dump
Unable to connect to the server: dial tcp 192.168.99.100:8443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
Then, I ran status of minikube. Output is as follows :
C:>minikube-windows-amd64 status
minikube: Saved
localkube:
kubectl:
Minikube on Windows uses VirtualBox per default and fails to create your virtual machine because Hyper-V disables VT-x access for other hypervisors. You need to either configure minikube to use Hyper-V or disable Hyper-V and run minikube start again.

Resources