Node local DNS creates dummy interface without IP address

[negz]

Hello,

I've just started experimenting with the new node local DNS cache. For reasons I haven't yet determined the NetIfManager manages to create the nodelocaldns dummy IP in my setup, but fails to allocate it an IP address. NetIfManager does not check the error return value when assigning IPs, so this manifests as follows:

$ kubectl --kubeconfig=/Users/negz/tfk-negz.kubecfg -n kube-system logs nodelocaldns-ltlfp --previous
2018/12/21 03:22:52 2018-12-21T03:22:52.066Z [INFO] Tearing down
2018/12/21 03:22:53 2018-12-21T03:22:53.064Z [INFO] Setting up networking for node cache
listen tcp 169.254.20.10:8080: bind: cannot assign requested address

I see when inspect the dummy interface that it's missing an IP:

# ip address show dev nodelocaldns
23: nodelocaldns: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1300 qdisc noqueue state UNKNOWN group default 
    link/ether 72:b5:c3:81:f6:2f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::70b5:c3ff:fe81:f62f/64 scope link 
       valid_lft forever preferred_lft forever

[negz]

Scratch that - I didn't read the code closely enough and confused EnsureDummyDevice, which does not check for errors when setting an IP, with AddDummyDevice which does.

Nevertheless, somehow the device is being created without an IP address and also without producing any error while configuring the device.

[dannyk81]

@negz hitting the same issue here (using CoreOS)

Were you able to resolve this?

[dannyk81]

Something is very bizarre here... on one occasion, it was able to set the IP (just on one node, they are all identical and I didn't do any kind of changes):

$ kubectl logs -n kube-system nodelocaldns-mc52t
2019/01/19 17:04:34 2019-01-19T17:04:34.16Z [INFO] Tearing down
2019/01/19 17:04:34 2019-01-19T17:04:34.25Z [INFO] Setting up networking for node cache
ip6.arpa.:53 on 169.254.25.10
.:53 on 169.254.25.10
green.k8s.<domain>.com.:53 on 169.254.25.10
in-addr.arpa.:53 on 169.254.25.10
2019-01-19T17:04:34.292Z [INFO] CoreDNS-1.2.6
2019-01-19T17:04:34.292Z [INFO] linux/amd64, go1.11.2,
CoreDNS-1.2.6
linux/amd64, go1.11.2,
 [INFO] plugin/reload: Running configuration MD5 = 3338a3593b32fb7a729debfe1c9253fa

26: nodelocaldns: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
    link/ether b6:2d:01:30:ad:27 brd ff:ff:ff:ff:ff:ff
    inet 169.254.25.10/32 brd 169.254.25.10 scope global nodelocaldns
       valid_lft forever preferred_lft forever
    inet6 fe80::b42d:1ff:fe30:ad27/64 scope link
       valid_lft forever preferred_lft forever

I deleted the above Pod to see if it will work again, after the pod was recreated on the same node (part of a DaemonSet) it failed again. 🤷‍♂️

[dannyk81]

This seems to be OS/Kernel related.

I can reproduce this on CoreOS running build 1688.5.3 (kernel 4.14.32) or build 1967.3.0 (kernel 4.14.88) - this is latest stable.

However this works just fine on a Debian Jessie with Kernel 4.9.0.

/edit: going to try with latest CoreOS alpha

/edit 2: same issue with latest CoreOS alpha (2023.0.0)

[negz]

@dannyk81 We never solved this. Ended up writing a mutating admission webhook (https://github.com/planetlabs/legion) that we used to inject a small caching CoreDNS sidecar container into pods.

[dannyk81]

thanks @negz!

Could you confirm which OS/Kernel combo are you/were you using for this? I wonder if it's CoreOS as well... since I can't reproduce this on Debian.

[negz]

It was CoreOS. I can’t say for sure which kernel version but it would have most likely been the stable CoreOS release at the time of writing.

[dannyk81]

Thanks, that confirms it.

Considering I tried 4 or 5 different versions, seems like a general issue with that OS.

[dannyk81]

@prameshj any chance you could take a look at this? seems like node local cache is broken on CoreOS.

[prameshj]

Is the 169.254.20.10 ip address used in some other interface on coreOS? Can you list the interfaces on the host and share the output?
Is assigning a different address working - maybe if we try assigning the clusterDNS ip(10.0.010 by default) to the nodelocaldns interface? Just to test if an ip address from a different subnet can be assigned successfully.

Also is the cluster being created using kubeadm?

[dannyk81]

Hi @prameshj, thanks for looking into this 😄 please see details below.

Is the 169.254.20.10 ip address used in some other interface on coreOS? Can you list the interfaces on the host and share the output?

The ip address (169.254.25.10) is not used by any other interface, here's ip a output:

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:94:96:ba brd ff:ff:ff:ff:ff:ff
    inet 10.230.142.57/24 brd 10.230.142.255 scope global dynamic ens192
       valid_lft 537847sec preferred_lft 537847sec
    inet6 fe80::250:56ff:fe94:96ba/64 scope link
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:6a:13:76:79 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::42:6aff:fe13:7679/64 scope link
       valid_lft forever preferred_lft forever
6: kube-ipvs0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
    link/ether 6e:44:e0:0c:b3:6e brd ff:ff:ff:ff:ff:ff
    inet 10.233.0.1/32 brd 10.233.0.1 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.233.0.3/32 brd 10.233.0.3 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.233.47.95/32 brd 10.233.47.95 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.233.34.203/32 brd 10.233.34.203 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.233.63.6/32 brd 10.233.63.6 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet6 fe80::6c44:e0ff:fe0c:b36e/64 scope link
       valid_lft forever preferred_lft forever
9: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1440 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
    inet 10.233.106.64/32 brd 10.233.106.64 scope global tunl0
       valid_lft forever preferred_lft forever
10: calif938b616ee1@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link
       valid_lft forever preferred_lft forever
20: cali790afe0195d@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link
       valid_lft forever preferred_lft forever
292: nodelocaldns: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
    link/ether 52:e5:a0:82:28:9c brd ff:ff:ff:ff:ff:ff
    inet6 fe80::50e5:a0ff:fe82:289c/64 scope link
       valid_lft forever preferred_lft forever

Is assigning a different address working - maybe if we try assigning the clusterDNS ip(10.0.010 by default) to the nodelocaldns interface? Just to test if an ip address from a different subnet can be assigned successfully.

I tried that (with 10.10.10.10 and other IPs), the result is the same:

# ./node-cache -localip 10.10.10.10 -conf Corefile
2019/01/20 14:40:13 2019-01-20T14:40:13.091Z [INFO] Tearing down
2019/01/20 14:40:13 2019-01-20T14:40:13.197Z [INFO] Setting up networking for node cache
listen tcp 10.10.10.10:8080: bind: cannot assign requested address

Also is the cluster being created using kubeadm?

Yes, it is. I'm using Kubespray to deploy the cluster and it uses kubeadm. It's deploying K8s v1.13.2.

For debugging, I extracted the node-cache binary from the container and launched it with strace on the host with root user, getting same error.

strace -s 100 -f -o strace-out -x ./node-cache -localip 10.10.10.10 -conf Corefile <-- content of strace-out attached.

strace-out.gz

[prameshj]

Thanks Danny! I think this is the relevant strace section. I was able to map the netlink request and the parameters, but I wasn't thorough.

Bind new ipaddress. unix.RTM_NEWADDR = x14

1593 sendto(5, "\x30\x00\x00\x00\x14\x00\x05\x06\x07\x00\x00\x00\x00\x00\x00\x00\x02\x20\x00\x00\x19\x00\x00\x00\x08\x00\x02\x00\x0a\x0a\x0a\x0a\x08\x00\x01\x00\x0a\x0a\x0a\x0a\x08\x00\x04\x00\x0a\x0a\x0a\x0a", 48, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 48
1593 getsockname(5, {sa_family=AF_NETLINK, pid=1593, groups=00000000}, [12]) = 0
1593 recvfrom(5, "\x24\x00\x00\x00\x02\x00\x00\x01\x07\x00\x00\x00\x39\x06\x00\x00\x00\x00\x00\x00\x30\x00\x00\x00\x14\x00\x05\x06\x07\x00\x00\x00\x00\x00\x00\x00", 4096, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, [12]) = 36
1593 close(5)

Would it be possible for you to by hand create a dummy interface, assign it an ip and run strace on those 2 commands ? It will be easier to debug this.

Interestingly, kube-ipvs0 is also a dummy interface, created by the same code, but different version of the github repo. I see that kube-ipvs0 has some service ip addresses bound to it.

ipvs code uses this, much older commit.
Nodelocal dns uses this

I wonder if something changed between these 2 commits that is causing the error.

[dannyk81]

@prameshj, attaching straces of two commands

1. strace -s 100 -f -o strace-add-dummy -x ip link add dummy0 type dummy --> strace-add-dummy.txt
2. strace -s 100 -f -o strace-add-addr -x ip addr add 10.10.10.10/32 dev dummy0 --> strace-add-addr.txt

The interface was created and address added:

# ip a show dev dummy0
437: dummy0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 76:88:e6:73:3f:e5 brd ff:ff:ff:ff:ff:ff
    inet 10.10.10.10/32 scope global dummy0
       valid_lft forever preferred_lft forever
    inet6 fe80::7488:e6ff:fe73:3fe5/64 scope link
       valid_lft forever preferred_lft forever