Fix duplicate mount points for multiple --volumes-from in docker run

[yongtang]

- What I did
This fix tries to fix the issue raised in #21845.

The issue with #21845 is that if multiple --volumes-from with the same destination has been specified, then one volume will be overridden by the other. This will mess up reference and prevent the overridden volume from being removed at the end.

Issue #21845 was observed with docker-compose though it is possible to emulate the same behavior with docker alone:

$ cat Dockerfile
FROM busybox
VOLUME ["/tmp/data"]
$ docker build -t vimage .
$ docker run --name=data1 vimage true
$ docker run --name=data2 vimage true
$ docker run --name=app --volumes-from=data1 --volumes-from=data2 -d busybox top
$ docker rm -f -v $(docker ps -aq)
$ docker volume ls
$ docker volume rm ...

- How I did it

This fix tries to address the issue by return an error if duplicate mount points was used with --volumes-from.

- How to verify it

An integration test has been added.

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

This fix fixes #21845.

/cc @cpuguy83 @vdemeester @thaJeztah

Signed-off-by: Yong Tang yong.tang.github@outlook.com

[AkihiroSuda]

I wonder similar checking about mountPoints is needed for this loop (for _, cfg := range hostConfig.Mounts {}) as well: https://github.com/yongtang/docker/blob/d231e33926a58a11974bbdd9a038e79fac2217df/daemon/volumes.go#L164-L206

[yongtang]

Thanks @AkihiroSuda for the reminder. I will double check for this case.

[thaJeztah]

CI looks to be failing

02:45:57 
02:45:57 ----------------------------------------------------------------------
02:45:57 FAIL: docker_deprecated_api_v124_test.go:87: DockerSuite.TestDeprecatedContainerAPIStartVolumesFrom
02:45:57 
02:45:57 docker_deprecated_api_v124_test.go:110:
02:45:57     c.Assert(status, checker.Equals, http.StatusNoContent)
02:45:57 ... obtained int = 500
02:45:57 ... expected int = 204
02:45:57 

[thaJeztah]

Interesting, these work:

$ docker run -v /bla -v /bla --rm alpine sh
$ docker run -v /bla -v bla:/bla --rm alpine sh
$ docker run -v /bla --tmpfs /bla --rm alpine sh

These don't

$ docker run -v bla:/bla -v /bla --tmpfs /bla --rm alpine sh
docker: Error response from daemon: Duplicate mount point '/bla'.
See 'docker run --help'.

$ docker run -v bla:/bla --tmpfs /bla --rm alpine sh
docker: Error response from daemon: Duplicate mount point '/bla'.
See 'docker run --help'.

Not sure if it's all expected (i.e., can it be a problem as well for anonymous volumes?),
but thought I'd post it here for reference

[cpuguy83]

This is really a behavior change.
I think this will end up breaking people who have already brought in a volume and then also do --volumes-from on a container with the same volume.

[cpuguy83]

I think we'll need to pick which one wins and log an error for the other one.
And the one we pick is whatever happens to win right now.

[yongtang]

@cpuguy83 @thaJeztah Got that, thanks. Was wondering how to best handle this situation. Will take a look and make sure the previous behavior is preserved.

[yongtang]

Update on this PR:

In addition to the issue described above, the following will also prevent volumes from removed:

$ docker build -t vimage .
$ docker run --name=data1 vimage true
$ docker run --name=data2 vimage true
$ docker run --name=app --volumes-from=data1 --volumes-from=data2 -v /tmp/data:/tmp/data -d busybox top
$ docker rm -f -v $(docker ps -aq)
$ docker volume ls
$ docker volume rm ...

The bind -v /tmp/data:/tmp/data will override the mount point as well.

Both the original issue and the bind override issue have been fixed.

Will continue to investigate other scenarios.

[yongtang]

Update of the PR (case 3):
In addition to the above two mentioned cases, it is also possible to specify bind with HostConfig.Mounts in API:

$ docker build -t vimage .
$ docker run --name=data1 vimage true
$ docker run --name=data2 vimage true
$ curl --unix-socket /var/run/docker.sock ...
$ docker rm -f -v $(docker ps -aq)
$ docker volume ls
$ docker volume rm ...

This case will also trigger volume is in use - ... error. This case is different from case (2) in that HostConfig.Mounts is only available in API. Case (2) and Case (3) happens in different places in registerMountpoint().

[yongtang]

@cpuguy83 @AkihiroSuda @thaJeztah. The PR has been updated with different possible combinations of scenarios covered. Please take a look and let me know if there are any issues.

[vdemeester]

ping @thaJeztah @cpuguy83

[cpuguy83]

Is this actually fixed now that references are stored as a map instead of a slice?

[LK4D4]

These seven lines are the same three times; maybe we should have a helper for it.

[yongtang]

Thanks @LK4D4. The PR has been updated with helper func. Please take a look.

[yongtang]

Thanks @cpuguy83 for the review. To preserve the behavior we need to deference the old volume that points to the same mount point (when old and new volume point to the same mount point). So the change is needed for the fix.

[LK4D4]

LGTM

[thaJeztah]

Not sure if it's the same part that's causing this error, but;

docker run -ti --rm --tmpfs /dev/shm:rw,nosuid,nodev,size=8g busybox
docker: Error response from daemon: linux mounts: Duplicate mount point '/dev/shm'.

As mentioned here; #6758 (comment)

[cpuguy83]

LGTM

This looks like it's fixing a real problem with refcounting.
However our code looks really awful here because we're essentially leaking volumes.
Not sure what to do here at the moment, but we should figure out how to do this better.

[cpuguy83]

Actually, sorry this looks like it should be Debug, not Error

[cpuguy83]

LGTM

[yongtang]

Thanks @cpuguy83. I agree the current way of handling it is less than ideal. I will spend some time to investigate it and see if we could come up with something better.