Shapes and annotations do not handle Date objects or strings with time zones

[harbulot]

At the moment (version 1.25.1), data traces can handle Date objects, which will then be displayed with what seems to be the browser's timezone settings. For example, date = new Date("2017-03-30T00:00:00.000Z") used on the X-axis in a trace, with a browser in the UTC+1 timezone will be displayed as "Mar 30, 2017, 01:00", which is indeed correct.

However, that same date object cannot be used for x0/x1 in a shape on the same object.

When using date.toISOString() instead (both for data and shapes), the timezone indicator (e.g. Z) at the end of the string is ignored and the string is considered to be within the local time zone.

Here is a jsfiddle to demonstrate the problem: https://jsfiddle.net/tkhmx8Le/

Effectively, the problem is two-fold (and might lead to two separate issues):

• It would be good if shapes and annotations also supported date objects.
• ISO-8601 date parsing should take into account the Z (or generally +xxxx or +xx:xx) timezone indicator before building its internal UTC representation. This is certainly related to Support ISO 8601 timestamps #1003 (and subsequent work).

[alexcjohnson]

Thanks for the report @harbulot

Yes, that's certainly inconsistent. Using Date objects within data traces is supported for backward compatibility, but - again for backward compatibility - it works in a strange way. I think you understand it correctly, but just for clarity for other folks reading this: we interpret the date in the local time zone but then discard the time zone and treat it as UTC, so for example if you supply a Date that amounts to Mar 30, 2017, 01:00 in your local time zone, we treat it as Mar 30, 2017 01:00 UTC and draw the axis as UTC.

Because of this strange behavior, it's not recommended to use Date objects to build plots. The present behavior is included for backward compatibility to a time before shapes existed and when annotations needed to be positioned using epoch milliseconds (in the local time zone).

With that caveat, I would be willing to consider it a bug that shapes and annotations do not support positioning by Date objects. It shouldn't be difficult to fix that, although we might not get to it right away (I'd be happy to review a PR to do it though!).

Re: time zone parsing - unless and until we build real time zone handling into our date axes, which would involve allowing separate time zones for the data (for strings that do not include an explicit time zone component) and the axis display, we are going to continue ignoring this field. The reason is that most of the time people expect to see dates displayed "WYSIWYG", ie '2017-03-30 09:00+0500' is typically going to be viewed by someone in the +5 time zone, so should be displayed at 09:00, even though we're internally turning this into UTC. If we do add time zone support, I expect it will continue to work this way unless and until an explicit time zone is specified for the axis, the data, or both.

[harbulot]

@alexcjohnson Thanks for these details.

we interpret the date in the local time zone but then discard the time zone and treat it as UTC, so for example if you supply a Date that amounts to Mar 30, 2017, 01:00 in your local time zone, we treat it as Mar 30, 2017 01:00 UTC and draw the axis as UTC.

Actually, I don't think this is the case. I was building a date from an ISO string (with Z time zone), but I've just changed the jsfiddle (https://jsfiddle.net/tkhmx8Le/1/ ) to build a date object (from a local date/time).
I get the same behaviour on the graph using this as a starting point:

• var startDate = new Date("2017-03-30T00:00:00.000Z"); (that's in UTC).
• var startDate = new Date(2017, 2, 30, 1, 0, 0); (implicitly the same moment in time, but expressed in GMT+1, for the timezone of my browser).

This first date/time in data is displayed as "Mar 30, 2017, 01:00" in the graph.

When I explicitly provide the string "2017-03-30T00:00:00.000Z" (using toISOString(), which is in the Z timezone), then the curve is shifted by one hour.

Date objects are effectively internally stored in UTC anyway as far as I'm aware (although sometimes they have some additional timezone information). Similarly to Java, the only reliable methods to identify a moment in time are getTime() and perhaps getISOString().

I understand what you're saying with your "2017-03-30 09:00+0500" example, but I'd consider that a timezone presentation issue (typically, that of the browser, by default). Essentially, I think we should be able to supply the graph with 2017-03-30 09:00+0500 or 2017-03-30T04:00:00.000Z (representing the same moment in time), and then format the values in whichever timezone is chosen (by the default, the timezone of the browser).

[alexcjohnson]

@harbulot I'm pretty sure we're saying the same thing actually about how things work at present.

To my knowledge, js Dates don't really have any timezone information - they are internally represented as milliseconds since the start of 1970 in UTC, but always displayed in the system timezone unless you use one of the UTC methods. So for me (US east) I can make two objects:

var a = new Date("2017-03-30T00:00:00.000Z");
var b = new Date(2017, 2, 29, 20, 0, 0);

but these are in fact identical objects, there's no way to tell that one was created as UTC and the other local:

+a===+b
> true
a.toString()
> "Wed Mar 29 2017 20:00:00 GMT-0400 (EDT)"
b.toString()
> "Wed Mar 29 2017 20:00:00 GMT-0400 (EDT)"
a.getTimezoneOffset()
> 240
b.getTimezoneOffset()
> 240

Now, when you supply a date string, we DON'T use the built-in date parser - that would open us up to too many browser-dependent issues but also limit the portability of the plot JSON across timezones. In our string->date parser we explicitly drop any timezone information in the string, so where you see this date on the plot is exactly where you specified assuming you're in the same timezone as the date. Until we add explicit timezone support, this is the behavior we need to support (even though it has known problems, like if you provide data with different timezones, they will not be correctly placed relative to each other).

[harbulot]

I've just looked into it a bit further. I think it's not so much that Plotly handles date/times as UTC. Rather, it seems to discard all potential timezone information altogether and assume there's only one timezone, making it the "unique" timezone.

For example, "2017-03-30 05:00:00-0800" and "2017-03-30 14:00:00+0100" should be the same moment in time. Yet, they're interpreted as "2017-03-30 05:00" and "2017-03-30 14:00" (in what's effectively the unique timezone).

Essentially, it's not handling timezones at all, which creates oddities for DST changes (like last week-end in the UK): https://jsfiddle.net/tkhmx8Le/3/
(This is skipping a point, for otherwise valid representations.)

[alexcjohnson]

Rather, it seems to discard all potential timezone information altogether and assume there's only one timezone, making it the "unique" timezone.

Exactly. And unless and until we build real timezone support, that's the only consistent way we can handle it.

[harbulot]

@alexcjohnson Thank you. Yes, you're correct.

What I find surprising is dropping this timezone information altogether. Maybe we're not facing the same data in general, but I always either put the timezone information in my JSON serialisation or use the seconds/milliseconds since Epoch, which is really the only way to identify a moment in time precisely. Not doing so creates many problems, especially when faced with periods when there a change Daylight Saving Time.

Rather, it seems to discard all potential timezone information altogether and assume there's only one timezone, making it the "unique" timezone.

Exactly. And unless and until we build real timezone support, that's the only consistent way we can handle it.

That makes sense. I would suggest perhaps a better way would be to store the data internally as milliseconds since 1970-01-01 UTC and format it depending on some settings (or using the default browser's locale representation).

[alexcjohnson]

I would suggest perhaps a better way would be to store the data internally as milliseconds since 1970-01-01 UTC and format it depending on some settings (or using the default browser's locale representation).

Yes, where the "some settings" is the timezone support we don't yet have. At that point that's exactly what we'll do (we already do store the data internally as UTC epoch milliseconds).

[finger563]

Possibly related to this issue:

[alexcjohnson]

@finger563 that doesn't give us a whole lot to go on - can you make a runnable example (preferably in codepen or some such) that shows this error?

[finger563]

@alexcjohnson Yea I realize that, I'm working on getting a minimal reproducible example (that separated from the huge codebase my plotly is embedded into) :)

[brian428]

I'd just like to add that the direction of using date strings over Date objects seems terribly inefficient. Obviously, Plotly has to convert these into Date objects somewhere, so that just adds overhead. But worse, we have apps that do large amounts of Date-related processing on the same data that drives the charts. If I have to constantly convert these to Dates, do something, then convert them back to strings to hand to Plotly, that's a ton of wasted time. Especially if you're dealing with hundreds of thousands of data points (which we are).

[alexcjohnson]

using date strings over Date objects seems terribly inefficient. Obviously, Plotly has to convert these into Date objects somewhere, so that just adds overhead.

Internally, we actually don't use Date objects except as an intermediary during string conversion. We use plain numbers which, among other things, give us a bit more precision than a Date object, and play nicer with WebGL. Our other key consideration here is portability - unfortunately JSON has no concept of a date, so strings are the clear choice for storage and communicating with other languages.

If your app sends an array of Date objects to Plotly, it will likely be faster than sending in date strings. I haven't benchmarked this myself, but we certainly don't do anything as roundabout as converting to a string and then back to a Date - Here is where that happens (notice the timezone hack, we can't do anything about that until v2). If you do any benchmarking of string vs Date inputs I'd love to see the results!

[brian428]

Right, I just meant that somewhere, Plotly has to be turning the string into a Date, even if it's just to call getTime() on it. That date parsing is overhead that just isn't there if you start with a Date in the first place.

But, any Date-related overhead in Plotly is secondary to me. The real concern would be if I have to constantly convert date strings and Dates back and forth when I'm doing Date-related processing on the underlying data. If I've got 250,000 data points, converting that to a Date and back to a string for every point and for every calculation is a lot of unnecessary overhead (and, likely, memory consumption).