How to get OpenEBS Mayastor clustered storage to work on microk8s? - microk8s

Question

I have tried all sorts of things to get OpenEBS Mayastor clustered storage to work on microk8s without much success. So rather than give up completely I thought I would detail one of my failed attempts and see if anyone could figure out what I am doing wrong. Thanks in advance for any help you can give me :-)
Failed Attempt
Here is the results of following the steps posted on at https://microk8s.io/docs/addon-mayastor.
VM Setup:
3 VM running Ubuntu 22.04 with 16GB ram on a vSphere hypervisor. I have used these same VM to create a 3 node microk8s cluster with good success in the past.
Microk8s removal:
removed microk8s on all 3 nodes.
microk8s stop
sudo snap remove microk8s --purge
sudo reboot
Microk8s fresh install:
https://microk8s.io/docs/setting-snap-channel
snap info microk8s
latest/stable: v1.26.0 2022-12-17 (4390) 176MB classic
On all 3 nodes:
sudo snap install microk8s --classic --channel=1.26/stable
sudo usermod -a -G microk8s $USER
sudo chown -f -R $USER ~/.kube
newgrp microk8s
sudo reboot
verify everything is ok
microk8s status
microk8s inspect
**Do what inspect tells you to do:**
WARNING: IPtables FORWARD policy is DROP. Consider enabling traffic forwarding with: sudo iptables -P FORWARD ACCEPT
The change can be made persistent with: sudo apt-get install iptables-persistent
sudo iptables -S
sudo iptables-legacy -S
sudo iptables -P FORWARD ACCEPT
sudo apt-get install iptables-persistent
sudo systemctl is-enabled netfilter-persistent.service
sudo reboot
microk8s inspect
still get the IPtable FORWARD warning on 2 of the 3 nodes.
hopefully it is not that important.
ping all the ip addresses in cluster from every node.
Followed the directions at https://microk8s.io/docs/addon-mayastor
step 1:
sudo sysctl vm.nr_hugepages=1024
echo 'vm.nr_hugepages=1024' | sudo tee -a /etc/sysctl.conf
sudo nvim /etc/sysctl.conf
step 2:
sudo apt install linux-modules-extra-$(uname -r)
sudo modprobe nvme_tcp
echo 'nvme-tcp' | sudo tee -a /etc/modules-load.d/microk8s-mayastor.conf
sudo nvim /etc/modules-load.d/microk8s-mayastor.conf
step 3:
microk8s enable dns
microk8s enable helm3
thought we might need rbac so I enabled that also.
microk8s enable rbac
Created 3 node cluster.
from main node.
sudo microk8s add-node
go to 2nd node.
microk8s join 10.1.0.116:25000/0c902af525c13fbfb5e7c37cff29b29a/acf13be17a96
from main node.
sudo microk8s add-node
go to 3rd node.
microk8s join 10.1.0.116:25000/36134181872079c649bed48d969a006d/acf13be17a96
microk8s status
enable the mayastor add-on:
from main node.
sudo microk8s enable core/mayastor --default-pool-size 20G
go to 2nd node.
sudo microk8s enable core/mayastor --default-pool-size 20G
Addon core/mayastor is already enabled
go to 3rd node.
sudo microk8s enable core/mayastor --default-pool-size 20G
Addon core/mayastor is already enabled
Wait for the mayastor control plane and data plane pods to come up:
sudo microk8s.kubectl get pod -n mayastor
NAME READY STATUS RESTARTS AGE
mayastor-csi-962jf 0/2 ContainerCreating 0 2m6s
mayastor-csi-l4zxx 0/2 ContainerCreating 0 2m5s
mayastor-8pcc4 0/1 Init:0/3 0 2m6s
msp-operator-74ff9cf5d5-jvxqb 0/1 Init:0/2 0 2m5s
mayastor-lt8qq 0/1 Init:0/3 0 2m5s
etcd-operator-mayastor-65f9967f5-mpkrw 0/1 ContainerCreating 0 2m5s
mayastor-csi-6wb7x 0/2 ContainerCreating 0 2m5s
core-agents-55d76bb877-8nffd 0/1 Init:0/1 0 2m5s
csi-controller-54ccfcfbcc-m94b7 0/3 Init:0/1 0 2m5s
mayastor-9q4gl 0/1 Init:0/3 0 2m5s
rest-77d69fb479-qsvng 0/1 Init:0/2 0 2m5s
# Still waiting
sudo microk8s.kubectl get pod -n mayastor
NAME READY STATUS RESTARTS AGE
mayastor-8pcc4 0/1 Init:0/3 0 32m
msp-operator-74ff9cf5d5-jvxqb 0/1 Init:0/2 0 32m
mayastor-lt8qq 0/1 Init:0/3 0 32m
core-agents-55d76bb877-8nffd 0/1 Init:0/1 0 32m
csi-controller-54ccfcfbcc-m94b7 0/3 Init:0/1 0 32m
mayastor-9q4gl 0/1 Init:0/3 0 32m
rest-77d69fb479-qsvng 0/1 Init:0/2 0 32m
mayastor-csi-962jf 2/2 Running 0 32m
mayastor-csi-l4zxx 2/2 Running 0 32m
etcd-operator-mayastor-65f9967f5-mpkrw 1/1 Running 1 32m
mayastor-csi-6wb7x 2/2 Running 0 32m
etcd-6tjf7zb9dh 0/1 Init:0/1 0 30m
Went to the trouble-shooting section at https://microk8s.io/docs/addon-mayastor
microk8s.kubectl logs -n mayastor daemonset/mayastor
output was:
Found 3 pods, using pod/mayastor-8pcc4
Defaulted container "mayastor" out of: mayastor, registration-probe (init), etcd-probe (init), initialize-pool (init)
Error from server (BadRequest): container "mayastor" in pod "mayastor-8pcc4" is waiting to start: PodInitializing