collect events continuously from an engine

[dongluochen]

Swarm manager collects events from each engine thru events API. Event stream from an engine may be broken for various reasons. While manager reconnects to collect events, events from that engine between reconnects are lost. This PR remembers the timestamp of last event from an engine, and uses it with --since to get the missing events.

Signed-off-by: Dong Chen dongluo.chen@docker.com

[dongluochen]

The test failure on Docker 1.9 is caused by incompatible event API change.

# time="2017-05-31T23:55:03Z" level=error msg="error monitoring events, will restart" error="Error response from daemon: strconv.ParseInt: parsing \"1496274901.000000000\": invalid syntax" id="VKIJ:X5FZ:QGBP:UJ2G:4QCA:PEFE:GSFK:PFQP:EKL2:6S5S:6ZO5:IF5W|127.0.0.1:5482" name=node-0 

Docker 1.9 and before only accepts --since with int64 value as second.

  since, err := httputils.Int64ValueOrDefault(r, "since", -1)

From Docker 1.10, --since takes second and nanosecond in the form of sec.nano.

   since, sinceNano, err := timetypes.ParseTimestamps(r.Form.Get("since"), -1)

[dongluochen]

We have several options:

1. Don't take this change. Wait until Docker 1.9 is deprecated from Swarm support version.
2. Deprecate Docker 1.9 and take this change in this release.
3. Make EventMonitor aware of Docker version. It uses different --since based on version. So far only Engine has Docker version. This diverts from current approach.
4. Make --since only take second parameter.
   4.1 Only take second from time. It'd result in duplicate events at reconnect.
   4.2 Round the second part of time to the next second. It'd miss events.

cc @wsong @alexmavr @nishanttotla

[nishanttotla]

How about we deprecate 1.9 with the upcoming release? I would even propose that we deprecate all versions before 1.12 soon, because 1.12 is almost a year old now. WDYT?

[dongluochen]

@nishanttotla I'm ok to deprecate support for Docker 1.9 which is 2 years old. I'm a little hesitated to deprecate support for Docker-1.10 and 1.11. Production takes a long time to evolve.

[nishanttotla]

Docker 1.9 is now deprecated and removed from the CI.

We can move ahead with this PR. @dongluochen let me know if you're able to rebase, otherwise I can carry it.