cenots 7: minikube v1.33.1 faces coredns CrashLoopBackOff.

[charleech]

What Happened?

minikube on CentOS 7.7 does not work.

Software information

$ minikube version
minikube version: v1.33.1
commit: 5883c09216182566a63dff4c326a6fc9ed2982ff

$ docker version
Client: Docker Engine - Community
 Version:           26.1.2
 API version:       1.45
 Go version:        go1.21.10
 Git commit:        211e74b
 Built:             Wed May  8 14:01:02 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.1.2
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.10
  Git commit:       ef1912d
  Built:            Wed May  8 13:59:55 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.31
  GitCommit:        e377cd56a71523140ca6ae87e30244719194a521
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Steps

$ export MINIKUBE_HOME=/tmp/minikube_home

$ minikube start \
   --driver=docker \
   --network minikube

😄  minikube v1.33.1 on Centos 7.7.1908
    ▪ MINIKUBE_IN_STYLE=true
    ▪ MINIKUBE_HOME=/tmp/minikube_home
✨  Using the docker driver based on user configuration
📌  Using Docker driver with root privileges
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🚜  Pulling base image v0.0.44 ...
🔥  Creating docker container (CPUs=2, Memory=8000MB) ...
❗  This container is having trouble accessing https://registry.k8s.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🐳  Preparing Kubernetes v1.30.0 on Docker 26.1.1 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring Calico (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

The coredns status is CrashLoopBackOff

$  kubectl get pod -A -o wide

NAMESPACE     NAME                               READY   STATUS             RESTARTS     AGE   IP             NODE       NOMINATED NODE   READINESS GATES
kube-system   coredns-7db6d8ff4d-mp9pd           0/1     CrashLoopBackOff   2 (3s ago)   26s   10.244.0.2     minikube   <none>           <none>
kube-system   etcd-minikube                      1/1     Running            0            39s   192.188.49.2   minikube   <none>           <none>
kube-system   kube-apiserver-minikube            1/1     Running            0            39s   192.188.49.2   minikube   <none>           <none>
kube-system   kube-controller-manager-minikube   1/1     Running            0            39s   192.188.49.2   minikube   <none>           <none>
kube-system   kube-proxy-cknfg                   1/1     Running            0            26s   192.188.49.2   minikube   <none>           <none>
kube-system   kube-scheduler-minikube            1/1     Running            0            39s   192.188.49.2   minikube   <none>           <none>
kube-system   storage-provisioner                1/1     Running            0            38s   192.188.49.2   minikube   <none>           <none>

describe pod

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  17m                   default-scheduler  Successfully assigned kube-system/coredns-7db6d8ff4d-mp9pd to minikube
  Normal   Created    16m (x4 over 17m)     kubelet            Created container coredns
  Normal   Started    16m (x4 over 17m)     kubelet            Started container coredns
  Normal   Pulled     16m (x5 over 17m)     kubelet            Container image "registry.k8s.io/coredns/coredns:v1.11.1" already present on machine
  Warning  BackOff    2m21s (x82 over 17m)  kubelet            Back-off restarting failed container coredns in pod coredns-7db6d8ff4d-mp9pd_kube-system(fbae1fb2-d70f-4683-ae4b-2b11dd0e7d14)

Try to print the log

$ kubectl -n kube-system logs -f coredns-7db6d8ff4d-mp9pd
Listen: listen tcp :53: bind: permission denied

Attach the log file

log.txt

Operating System

Redhat/Fedora

Driver

Docker

[charleech]

I can confirm that after rolling back to minikube version 1.32.0, this issue does not occur.

$ minikube version
minikube version: v1.32.0
commit: 8220a6eb95f0a4d75f7f2d7b14cef975f050512d

$ docker version
Client: Docker Engine - Community
 Version:           26.1.2
 API version:       1.45
 Go version:        go1.21.10
 Git commit:        211e74b
 Built:             Wed May  8 14:01:02 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.1.2
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.10
  Git commit:       ef1912d
  Built:            Wed May  8 13:59:55 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.31
  GitCommit:        e377cd56a71523140ca6ae87e30244719194a521
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ export MINIKUBE_HOME=/tmp/minikube_home

$ minikube start \
   --driver=docker \
   --network minikube

😄  minikube v1.32.0 on Centos 7.7.1908
    ▪ MINIKUBE_IN_STYLE=true
    ▪ MINIKUBE_HOME=/tmp/minikube_home
✨  Using the docker driver based on user configuration
📌  Using Docker driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=8000MB) ...
❗  This container is having trouble accessing https://registry.k8s.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🐳  Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🔎  Verifying Kubernetes components...
🌟  Enabled addons: default-storageclass, storage-provisioner

❗  /usr/local/bin/kubectl is version 1.30.0, which may have incompatibilities with Kubernetes 1.28.3.
    ▪ Want kubectl v1.28.3? Try 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

$  kubectl get pods -A -o wide
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE   IP             NODE       NOMINATED NODE   READINESS GATES
kube-system   coredns-5dd5756b68-lt5m5           1/1     Running   0          15s   10.244.0.2     minikube   <none>           <none>
kube-system   etcd-minikube                      1/1     Running   0          27s   192.188.49.2   minikube   <none>           <none>
kube-system   kube-apiserver-minikube            1/1     Running   0          27s   192.188.49.2   minikube   <none>           <none>
kube-system   kube-controller-manager-minikube   1/1     Running   0          27s   192.188.49.2   minikube   <none>           <none>
kube-system   kube-proxy-b8dn2                   1/1     Running   0          15s   192.188.49.2   minikube   <none>           <none>
kube-system   kube-scheduler-minikube            1/1     Running   0          27s   192.188.49.2   minikube   <none>           <none>
kube-system   storage-provisioner                1/1     Running   0          26s   192.188.49.2   minikube   <none>           <none>

log.txt

[Dranaxel]

Not totally sure it's related but I'm trying to install minikube with cilium in the same version and got a coredns error too:

╰─$ minikube version
minikube version: v1.33.0
commit: 86fc9d54fca63f295d8737c8eacdbb7987e89c67

╰─$ docker version                                                                                                                 127 ↵
Client: Docker Engine - Community
 Version:           26.1.3
 API version:       1.45
 Go version:        go1.21.10
 Git commit:        b72abbb
 Built:             Thu May 16 08:33:48 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.1.3
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.10
  Git commit:       8e96db1
  Built:            Thu May 16 08:33:48 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.32
  GitCommit:        8b3b7ca2e5ce38e8f31a34f35b2b68ceb8470d89
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

I think cilium is crashing bc of the coredns errors

╰─$ k get pods -A
NAMESPACE     NAME                               READY   STATUS               RESTARTS         AGE
kube-system   cilium-l9prq                       0/1     CrashLoopBackOff     11 (76s ago)     39m
kube-system   cilium-mhl56                       0/1     CrashLoopBackOff     10 (3m14s ago)   32m
kube-system   cilium-nmrfz                       0/1     PostStartHookError   11 (8s ago)      36m
kube-system   cilium-operator-86f4c5579c-q6s75   1/1     Running              5 (31m ago)      39m
kube-system   coredns-7db6d8ff4d-wt9s6           0/1     CrashLoopBackOff     17 (9s ago)      39m
kube-system   etcd-minikube                      1/1     Running              0                40m
kube-system   kube-apiserver-minikube            1/1     Running              1 (38m ago)      40m
kube-system   kube-controller-manager-minikube   1/1     Running              0                40m
kube-system   kube-proxy-ffnbj                   1/1     Running              0                36m
kube-system   kube-proxy-jpr6w                   1/1     Running              0                32m
kube-system   kube-proxy-kk54f                   1/1     Running              0                39m
kube-system   kube-scheduler-minikube            1/1     Running              0                40m
kube-system   storage-provisioner                1/1     Running              6 (30m ago)      40m

[Dranaxel]

Yes, downgrading to 1.32.0 fixed the issue

[HuXinjing]

Same question.

[medyagh]

HuXinjing are you using Centos 7 as well ? I am suspecting this is a centos only issue, can everybody confirm they are all on centos

[HuXinjing]

Yes, my server is centos7. Thanks for your in time reply when I'm quitting, lol, and I'll have a try on my pc, which is ubuntu 2304.

…

---Original--- From: "Medya ***@***.***> Date: Wed, Jun 5, 2024 00:53 AM To: ***@***.***>; Cc: ***@***.******@***.***>; Subject: Re: [kubernetes/minikube] minikube v1.33.1 faces corednsCrashLoopBackOff. (Issue #18950) HuXinjing are you using Centos 7 as well ? I am suspecting this is a centos only issue, can everybody confirm they are all on centos — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[prezha]

hi @charleech, i think that the issue is not minikube v1.33.1 vs v1.32.0 but it's about the centos 7 and the default kubernetes version that comes with the respective minikube version

i think you'd be able to confirm that with the (latest) minikube v1.33.1 whilst specifying the older (ie, <v1.29.0) kubernetes version (eg, k8s v1.28.3 that's used by minikube v1.32.0 - as shown in your comment above as well):

minikube start --container-runtime=docker --kubernetes-version=1.28.3 -p k8s-1.28.3

i expect this to work for @Dranaxel and @HuXinjing too

more details about this issue is in the comment of #18421

can you please run this and let us know if it works for you?

[prezha]

btw, i was able to replicate the issue with the latest minikube (v1.33.1) running on centos 7 (vm), but on the other hand, it works fine on centos (stream) 8 (vm), so i think it's specifically related to the centos 7 that's going end-of-life by the end of this months (june 2024) and therefore probably will not get the necessary fixes backported (eg, setting net.ipv4.ip_unprivileged_port_start)

$ minikube start
😄  minikube v1.33.1 on Centos 8 (kvm/amd64)
✨  Automatically selected the docker driver
📌  Using Docker driver with root privileges
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🚜  Pulling base image v0.0.44 ...
💾  Downloading Kubernetes v1.30.0 preload ...
    > preloaded-images-k8s-v18-v1...:  342.90 MiB / 342.90 MiB  100.00% 23.57 M
    > gcr.io/k8s-minikube/kicbase...:  481.58 MiB / 481.58 MiB  100.00% 16.31 M
🔥  Creating docker container (CPUs=2, Memory=2200MB) ...
🐳  Preparing Kubernetes v1.30.0 on Docker 26.1.1 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
💡  kubectl not found. If you need it, try: 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

$ minikube kubectl -- get pods -A -owide
    > kubectl.sha256:  64 B / 64 B [-------------------------] 100.00% ? p/s 0s
    > kubectl:  49.07 MiB / 49.07 MiB [--------------] 100.00% 1.73 MiB p/s 29s
NAMESPACE     NAME                               READY   STATUS    RESTARTS      AGE   IP             NODE       NOMINATED NODE   READINESS GATES
kube-system   coredns-7db6d8ff4d-j62tq           1/1     Running   0             81s   10.244.0.2     minikube   <none>           <none>
kube-system   etcd-minikube                      1/1     Running   0             97s   192.168.49.2   minikube   <none>           <none>
kube-system   kube-apiserver-minikube            1/1     Running   0             97s   192.168.49.2   minikube   <none>           <none>
kube-system   kube-controller-manager-minikube   1/1     Running   0             96s   192.168.49.2   minikube   <none>           <none>
kube-system   kube-proxy-9d4dg                   1/1     Running   0             82s   192.168.49.2   minikube   <none>           <none>
kube-system   kube-scheduler-minikube            1/1     Running   0             95s   192.168.49.2   minikube   <none>           <none>
kube-system   storage-provisioner                1/1     Running   1 (50s ago)   94s   192.168.49.2   minikube   <none>           <none>

[charleech]

@prezha

Thank you for your advice.

I'm requesting my engineer to set up a new VM with the latest version of Oracle Linux 9.x. Once it's ready, I'll test it and update you on the results. This process should be completed by Friday.

[charleech]

I can confirm that after upgrading the OS to newer/latest version (for me it is an Oracle Linux 9.4), the coredns is started properly.