Add outgoing metadata to TestContext #2457
Description
Now that v3 has a TestContext object, we should provide a way for test authors (and extensibility authors who are writing test extensions) a way to record additional information that can be reported (which I'll call "outgoing metadata").
One example of such a request is UI-based testing systems which would like to attach screenshots to failed test records.
Let's assume a model that is like traits (aka "incoming metadata"): introduce name/value pairs of string => string, where the data is then added to the "results" object model, which then in turn gets placed into XML on the way out for reporting (this XML can be written directly to disk, and is also the basis of our transformation system to produce other report files, like competing/compatible XML report files and an HTML-based report).
I believe the use of strings here rather than arbitrary types (i.e., string => object) places a reasonable limit on the outgoing data to be able to ensure that it is serializable; using arbitrary objects then places an additional serialization burden on the end user, and at the end of the day, still needs to get translated into a string to be placed into XML. The usefulness of a Base64 encoding of a .NET binary serialized object is effectively zero; forcing the user to think about how to make a string value useful, therefore, ensures that this design consideration is not overlooked. It makes more sense, then, to just use string as the value type in the key/value pair, rather than object + yet another arbitrary serialization system.
What remains, then, is the decision about how much we should expect from the end user about the string values, and whether we should attempt to suggest or enforce some restrictions. For example, should they be limited in size? Should we suggest or mandate that they be human readable? If we don't mandate human readability and/or limit the size, does that mean we shouldn't render their values into the HTML report, which is intended to be relatively compact and human readable?
In the aforementioned example of screenshots, I feel a bit torn by this requirement and how best to support it (indirectly).
There are two obvious (and perhaps other non-obvious) implementations here.
- Write the screenshot to disk and use the outgoing string value to point to the screenshot file.
- Use a compacted version of the screenshot, and then Base64 encode that into the outgoing string value.
Option 1 is more readable in report form, but has an inherent external dependency (the file on disk) that may not have been preserved in some circumstances, like a continuous integration environment. Option 2 is not reasonable readable in a generic report, and would require some additional transformation during display that would not necessarily be inherent in the built-in HTML reporting.
As such, I'm looking for feedback on the design of this outgoing metadata system. I like the self-contained nature of the XML, and am concerned that introducing the extra concept of files necessarily complicates that model. If we directly support files, what's the best way to represent this? Perhaps the outgoing metadata is a trio of information ("key", "value", and "value-data-type") which can be used to help reports determine what to do with that information? Do we also need to provide an arbitrary external storage mechanism for values, or should we assume that specific value types are actually just Base64-encoded binary values?