docker network inspect output is flaky/wrong

[ahmetb]

When I run the following on my Swarm cluster:

docker network create frontend
docker run -d --net frontend --name fe1 nginx
docker run -d --net frontend --name fe2 nginx
docker run -d --net frontend --name fe3 nginx
docker run -d --net frontend --name fe4 nginx

and execute docker network inspect frontend I pretty much always get different output every time (pay attention to Containers→IPv4Address):

$ docker network inspect frontend
[
    {
        "Name": "frontend",
        "Id": "acde7eaf4651c0da70c8077d47882ccac8d17d071ebd69f2fc212d15a697804f",
        "Scope": "global",
        "Driver": "overlay",
        "IPAM": {
            "Driver": "default",
            "Config": [
                {}
            ]
        },
        "Containers": {},
        "Options": {}
    }
]

[
    {
        "Name": "frontend",
        "Id": "acde7eaf4651c0da70c8077d47882ccac8d17d071ebd69f2fc212d15a697804f",
        "Scope": "global",
        "Driver": "overlay",
        "IPAM": {
            "Driver": "default",
            "Config": [
                {}
            ]
        },
        "Containers": {
            "f34345e2351b98150387bc4136324c73ab5d8df3162a1e4a8674b9d55d24c07d": {
                "EndpointID": "4e13bbd62e91eb7a3803b82460a5c207c07e2c300a15bdf3736f17a43f8691da",
                "MacAddress": "02:42:0a:00:00:05",
                "IPv4Address": "10.0.0.5/24",
                "IPv6Address": ""
            }
        },
        "Options": {}
    }
]

[
    {
        "Name": "frontend",
        "Id": "acde7eaf4651c0da70c8077d47882ccac8d17d071ebd69f2fc212d15a697804f",
        "Scope": "global",
        "Driver": "overlay",
        "IPAM": {
            "Driver": "default",
            "Config": [
                {}
            ]
        },
        "Containers": {
            "2c2e8c1eb9e3f673fce36cf0c28b542f2a10174adc51fa23c59dafb7546c7c41": {
                "EndpointID": "de9fc2b4df99c83a84cde04fca0253d8c37cf0c8a4b7bf55d7a82f3f2fd9e594",
                "MacAddress": "02:42:0a:00:00:04",
                "IPv4Address": "10.0.0.4/24",
                "IPv6Address": ""
            }
        },
        "Options": {}
    }
]

[
    {
        "Name": "frontend",
        "Id": "acde7eaf4651c0da70c8077d47882ccac8d17d071ebd69f2fc212d15a697804f",
        "Scope": "global",
        "Driver": "overlay",
        "IPAM": {
            "Driver": "default",
            "Config": [
                {}
            ]
        },
        "Containers": {
            "9d4bac5e5b7daf59b7a474fc9cf04078fe84e84d4a126f1ef40e70d4fb20b416": {
                "EndpointID": "693b23d5aa5bfbe5bfb9e37b45ed8622ca7802d7f5ed9578de41bb873315b989",
                "MacAddress": "02:42:0a:00:00:06",
                "IPv4Address": "10.0.0.6/24",
                "IPv6Address": ""
            }
        },
        "Options": {}
    }
]

[
    {
        "Name": "frontend",
        "Id": "acde7eaf4651c0da70c8077d47882ccac8d17d071ebd69f2fc212d15a697804f",
        "Scope": "global",
        "Driver": "overlay",
        "IPAM": {
            "Driver": "default",
            "Config": [
                {}
            ]
        },
        "Containers": {
            "f358b856f8d62138183174e5b0bdd066919ac7ca5bd5f118363dc3e9ef50884c": {
                "EndpointID": "66927023d38a311d6dda1d05b1d826e50ca3b5d45b93e79637c3cf0ca3c97041",
                "MacAddress": "02:42:0a:00:00:03",
                "IPv4Address": "10.0.0.3/24",
                "IPv6Address": ""
            }
        },
        "Options": {}
    }
]

as shown above it mostly returns a random container and sometimes it does not return any "Containers" at all.

It never returned more than one container in this response as far as I can tell (I ran the command probably 30 times).

[ahmetb]

I must also add, if I do docker inspect <container> I can see all networks the container is connected to just fine. I'm guessing there's something wrong with Swarm's aggregation here.

[ahmetb]

I got it to return multiple containers in the "Containers" section only in one case (repro below), but it still returns empty list sometimes even for this case:

docker run -d --net frontend --name fe1 nginx
docker run -d --net frontend --name fe2 nginx
docker run -d --net frontend --name fe nginx

docker run -d --net backend --name be1 nginx
docker run -d --net backend --name be2 nginx
docker run -d --net backend --name be nginx

docker network connect frontend-backend be
docker network connect frontend-backend fe

docker network inspect frontend-backend

[abronan]

See #1262 (comment)

In the case of an overlay network, docker network inspect returns the container attached to this network for a given engine. Thus the API response for inspect is bounded to a single engine. Because we proxy the request in a random way, when inspecting an overlay, it gets back a varying and incomplete information.

Instead of randomly proxy'ing the request, we must aggregate the information from all the nodes and build our own response. But this has scalability issues on a large number of nodes.

First solution would be to cache the container entries for a given overlay at the Swarm level when we receive container connect/disconnect or create events from docker (and as of now libnetwork does not emit any events, hopefully this is going to be fixed for 1.10).

Second solution would be for libnetwork to store the container informations in the distributed key/value store and get back the complete inspect information for an overlay no matter on which node you are (but this assumes that docker steps beyond its responsibilities which might be wrong/not ideal).

[abronan]

/cc @mavenugo @mrjana