Error configuring minikube with docker driver and custom network

[mtatheonly]

What Happened?

When starting minikube with docker driver and with a custom profile and custom network, there's an error regarding to the network passed as a parameter. If the network name is the same as the profile name, there's no error.
I believe the error is caused in the function HostIP from file pkg/minikube/cluster/ip.go. There's no reference to network name in the function. The only name the function gets is the clusterName.

(Running with docker 24.0.5 in Manjaro)

Attach the log file

log.txt

Operating System

Other

Driver

Docker

[kundan2707]

/kind support

[spowelljr]

Hi @mtatheonly, I wasn't able to reproduce this, I did the following:

$ docker network create --driver=bridge --subnet=192.168.49.0/24 --gateway=192.168.49.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=65535 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=minikube cats123
ba1025a270855cb5c0659f611671e125e590c0cdb319aa073a3e788d926e5070

$ minikube start --network cats123 -p p2
😄  [p2] minikube v1.31.2 on Darwin 13.6 (arm64)
✨  Automatically selected the docker driver. Other choices: qemu2, ssh
📌  Using Docker Desktop driver with root privileges
👍  Starting control plane node p2 in cluster p2
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=4000MB) ...
🐳  Preparing Kubernetes v1.28.2 on Docker 24.0.6 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "p2" cluster and "default" namespace by default

Could you explain what you're doing

[mtatheonly]

tried same command, another network name:

❯ docker network create --driver=bridge --subnet=192.168.49.0/24 --gateway=192.168.49.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=65535 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=minikube abcde123
2e7f334091b6401798a1449bb0aff8e793f80581fa240e1c2a8e52dba7ddbe95

~ 
❯ minikube start --network abcde123 -p p2                                                                                                                                                                                                        
😄  [p2] minikube v1.31.2 on Arch "23.0.2"
✨  Automatically selected the docker driver. Other choices: kvm2, ssh
📌  Using Docker driver with root privileges
👍  Starting control plane node p2 in cluster p2
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=4900MB) ...
🐳  Preparing Kubernetes v1.27.4 on Docker 24.0.4 ...E0928 18:25:21.207212 3247703 start.go:131] Unable to get host IP: network inspect: docker network inspect p2 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}": exit status 1
stdout:


stderr:
Error response from daemon: network p2 not found

    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "p2" cluster and "default" namespace by default

docker version                                                                                                                                                                                                                                     
Client:
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.6
 Git commit:        ced0996600
 Built:             Wed Jul 26 21:44:58 2023
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.6
  Git commit:       a61e2b4c9c
  Built:            Wed Jul 26 21:44:58 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.6
  GitCommit:        091922f03c2762540fd057fba91260237ff86acb.m
 runc:
  Version:          1.1.9
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

minikube version  
minikube version: v1.31.2
commit: fd7ecd9c4599bef9f04c0986c4a0187f98a4396e

[spowelljr]

Thanks for the info, I looked into it, and while you are correct that dockerNetworkInspect is being passed the cluster name and not the network name, it should fall back when the network doesn't exist.

For you this is resolving to false: if strings.Contains(rr.Output(), "No such network")

If you run docker network inspect on a non-existing network what is your output? And also try with abcde123

$ docker network inspect cat
[]
Error: No such network: cat

[mtatheonly]

❯ docker network inspect qwerty                                                                                                                                                                                                                      
[]
Error response from daemon: network qwerty not found

~ 
❯ docker network inspect abcde123                                                        
[
    {
        "Name": "abcde123",
        "Id": "2e7f334091b6401798a1449bb0aff8e793f80581fa240e1c2a8e52dba7ddbe95",
        "Created": "2023-09-28T18:24:34.498373916-03:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "192.168.49.0/24",
                    "Gateway": "192.168.49.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "95d87f64c1e9f5ff05f992ee8a2a49f21d5b6272ea665b9fff7152d1d7cf1260": {
                "Name": "p2",
                "EndpointID": "8b7ae03343656df441c5aa0a04e8640f4540d62ad71ea79b1bc0493bf71e25d7",
                "MacAddress": "02:42:c0:a8:31:02",
                "IPv4Address": "192.168.49.2/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "--icc": "",
            "--ip-masq": "",
            "com.docker.network.driver.mtu": "65535"
        },
        "Labels": {
            "created_by.minikube.sigs.k8s.io": "true",
            "name.minikube.sigs.k8s.io": "minikube"
        }
    }
]

[mtatheonly]

I've searched within docker project and the string minikube is searching for in docker inspect network output is not returned (anymore?)

https://github.com/moby/moby/blob/7d30dafd69ed06e94a0e1b87f74a1f7c9a98dfeb/libnetwork/error.go#L11

func (nsn ErrNoSuchNetwork) Error() string {
	return fmt.Sprintf("network %s not found", string(nsn))
}

It's being returned "network %s not found" instead of "No such network"

[spowelljr]

Thanks for the help debugging this issue! The machine I was using had an old Docker version (20.10.21), but I confirmed on another machine using 24.0.6 that it returns network %s not found and was able to reproduce:

$ minikube start --network abcde123 -p p2
😄  [p2] minikube v1.31.2 on Debian 11.7 (amd64)
✨  Automatically selected the docker driver. Other choices: qemu2, ssh
📌  Using Docker driver with root privileges
👍  Starting control plane node p2 in cluster p2
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=8000MB) ...
🐳  Preparing Kubernetes v1.28.2 on Docker 24.0.6 ...E0929 17:13:22.689810 2528575 start.go:131] Unable to get host IP: network inspect: docker network inspect p2 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}": exit status 1
stdout:


stderr:
Error response from daemon: network p2 not found

    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "p2" cluster and "default" namespace by default

I'll create a PR shortly to fix this, thanks again for your help debugging