|
1 | 1 | # The `rustdoc` test suite |
2 | 2 |
|
3 | | -This page is specifically about the test suite named `rustdoc`. |
4 | | -For other test suites used for testing rustdoc, see [Rustdoc tests](../rustdoc.md#tests). |
| 3 | +This page is about the test suite named `rustdoc` used to test the HTML output of `rustdoc`. |
| 4 | +For other rustdoc-specific test suites, see [Rustdoc test suites]. |
5 | 5 |
|
6 | | -The `rustdoc` test suite is specifically used to test the HTML output of rustdoc. |
| 6 | +Each test file in this test suite is simply a Rust source file `file.rs` sprinkled with |
| 7 | +so-called *directives* located inside normal Rust code comments. |
| 8 | +These come in two flavors: *Compiletest* and *HtmlDocCk*. |
7 | 9 |
|
8 | | -This is achieved by means of `htmldocck.py`, a custom checker script that leverages [XPath]. |
| 10 | +To learn more about the former, read [Compiletest directives]. |
| 11 | +For the latter, continue reading. |
9 | 12 |
|
10 | | -[XPath]: https://en.wikipedia.org/wiki/XPath |
| 13 | +Internally, [`compiletest`] invokes the supplementary checker script [`htmldocck.py`]. |
11 | 14 |
|
12 | | -## Directives |
13 | | -Directives to htmldocck are similar to those given to `compiletest` in that they take the form of `//@` comments. |
| 15 | +[Rustdoc test suites]: ../tests/compiletest.md#rustdoc-test-suites |
| 16 | +[`compiletest`]: ../tests/compiletest.md |
| 17 | +[`htmldocck.py`]: https://github.com/rust-lang/rust/blob/master/src/etc/htmldocck.py |
14 | 18 |
|
15 | | -In addition to the directives listed here, |
16 | | -`rustdoc` tests also support most |
17 | | -[compiletest directives](../tests/directives.html). |
| 19 | +## HtmlDocCk Directives |
18 | 20 |
|
19 | | -All `PATH`s in directives are relative to the rustdoc output directory (`build/TARGET/test/rustdoc/TESTNAME`), |
20 | | -so it is conventional to use a `#![crate_name = "foo"]` attribute to avoid |
21 | | -having to write a long crate name multiple times. |
22 | | -To avoid repetition, `-` can be used in any `PATH` argument to re-use the previous `PATH` argument. |
| 21 | +Directives to HtmlDocCk are assertions that place constraints on the generated HTML. |
| 22 | +They look similar to those given to `compiletest` in that they take the form of `//@` comments |
| 23 | +but ultimately, they are completey distinct and processed by different programs. |
23 | 24 |
|
24 | | -All arguments take the form of quoted strings |
25 | | -(both single and double quotes are supported), |
26 | | -with the exception of `COUNT` and the special `-` form of `PATH`. |
| 25 | +[XPath] is used to query parts of the HTML document tree. |
| 26 | + |
| 27 | +**Introductory example**: |
| 28 | + |
| 29 | +```rust,ignore (illustrative) |
| 30 | +//@ has file/type.Alias.html |
| 31 | +//@ has - '//*[@class="rust item-decl"]//code' 'type Alias = Option<i32>;' |
| 32 | +pub type Alias = Option<i32>; |
| 33 | +``` |
| 34 | + |
| 35 | +Here, we check that documentation generated for crate `file` contains a page for the |
| 36 | +public type alias `Alias` where the code block that is found at the top contains the |
| 37 | +expected rendering of the item. The `//*[@class="rust item-decl"]//code` is an XPath |
| 38 | +expression. |
27 | 39 |
|
28 | | -Directives are assertions that place constraints on the generated HTML. |
| 40 | +Conventionally, you place these directives directly above the thing they are meant to test. |
| 41 | +Technically speaking however, they don't need to be as HtmlDocCk only looks for the directives. |
29 | 42 |
|
30 | | -All directives (except `files`) can be negated by putting a `!` in front of their name. |
| 43 | +All directives take a `PATH` argument. |
| 44 | +To avoid repetition, `-` can be passed to it to re-use the previous `PATH` argument. |
| 45 | +Since the path contains the name of the crate, it is conventional to add a |
| 46 | +`#![crate_name = "foo"]` attribute to the crate root to shorten the resulting path. |
| 47 | + |
| 48 | +All arguments take the form of shell-style (single or double) quoted strings, |
| 49 | +with the exception of `COUNT` and the special `-` form of `PATH`. |
| 50 | + |
| 51 | +All directives (except `files`) can be *negated* by putting a `!` in front of their name. |
| 52 | +Before you add negated directives, please read about [their caveats](#caveats). |
31 | 53 |
|
32 | 54 | Similar to shell commands, |
33 | 55 | directives can extend across multiple lines if their last char is `\`. |
34 | 56 | In this case, the start of the next line should be `//`, with no `@`. |
35 | 57 |
|
36 | | -For example, `//@ !has 'foo/struct.Bar.html'` checks that crate `foo` does not have a page for a struct named `Bar` in the crate root. |
| 58 | +Use the special string `{{channel}}` in XPaths, `PATTERN` arguments and [snapshot files](#snapshot) |
| 59 | +if you'd like to refer to the URL `https://doc.rust-lang.org/CHANNEL` where `CHANNEL` refers to the |
| 60 | +current release channel (e.g, `stable` or `nightly`). |
| 61 | + |
| 62 | +Listed below are all possible directives: |
| 63 | + |
| 64 | +[XPath]: https://en.wikipedia.org/wiki/XPath |
37 | 65 |
|
38 | 66 | ### `has` |
39 | 67 |
|
40 | | -Usage 1: `//@ has PATH` |
41 | | -Usage 2: `//@ has PATH XPATH PATTERN` |
| 68 | +> Usage 1: `//@ has PATH` |
42 | 69 |
|
43 | | -In the first form, `has` checks that a given file exists. |
| 70 | +Check that the file given by `PATH` exists. |
44 | 71 |
|
45 | | -In the second form, `has` is an alias for `matches`, |
46 | | -except `PATTERN` is a whitespace-normalized[^1] string instead of a regex. |
| 72 | +> Usage 2: `//@ has PATH XPATH PATTERN` |
47 | 73 |
|
48 | | -### `matches` |
| 74 | +Checks that the text of each element / attribute / text selected by `XPATH` in the |
| 75 | +whitespace-normalized[^1] file given by `PATH` matches the |
| 76 | +(also whitespace-normalized) string `PATTERN`. |
| 77 | + |
| 78 | +**Tip**: If you'd like to avoid whitespace normalization and/or if you'd like to match with a regex, |
| 79 | +use `matches` instead. |
49 | 80 |
|
50 | | -Usage: `//@ matches PATH XPATH PATTERN` |
| 81 | +### `hasraw` |
51 | 82 |
|
52 | | -Checks that the text of each element selected by `XPATH` in `PATH` matches the python-flavored regex `PATTERN`. |
| 83 | +> Usage: `//@ hasraw PATH PATTERN` |
53 | 84 |
|
54 | | -### `matchesraw` |
| 85 | +Checks that the contents of the whitespace-normalized[^1] file given by `PATH` |
| 86 | +matches the (also whitespace-normalized) string `PATTERN`. |
55 | 87 |
|
56 | | -Usage: `//@ matchesraw PATH PATTERN` |
| 88 | +**Tip**: If you'd like to avoid whitespace normalization and / or if you'd like to match with a |
| 89 | +regex, use `matchesraw` instead. |
57 | 90 |
|
58 | | -Checks that the contents of the file `PATH` matches the regex `PATTERN`. |
| 91 | +### `matches` |
59 | 92 |
|
60 | | -### `hasraw` |
| 93 | +> Usage: `//@ matches PATH XPATH PATTERN` |
61 | 94 |
|
62 | | -Usage: `//@ hasraw PATH PATTERN` |
| 95 | +Checks that the text of each element / attribute / text selected by `XPATH` in the |
| 96 | +file given by `PATH` matches the Python-flavored[^2] regex `PATTERN`. |
63 | 97 |
|
64 | | -Same as `matchesraw`, except `PATTERN` is a whitespace-normalized[^1] string instead of a regex. |
| 98 | +### `matchesraw` |
| 99 | + |
| 100 | +> Usage: `//@ matchesraw PATH PATTERN` |
| 101 | +
|
| 102 | +Checks that the contents of the file given by `PATH` matches the |
| 103 | +Python-flavored[^2] regex `PATTERN`. |
65 | 104 |
|
66 | 105 | ### `count` |
67 | 106 |
|
68 | | -Usage: `//@ count PATH XPATH COUNT` |
| 107 | +> Usage: `//@ count PATH XPATH COUNT` |
69 | 108 |
|
70 | | -Checks that there are exactly `COUNT` matches for `XPATH` within the file `PATH`. |
| 109 | +Checks that there are exactly `COUNT` matches for `XPATH` within the file given by `PATH`. |
71 | 110 |
|
72 | 111 | ### `snapshot` |
73 | 112 |
|
74 | | -Usage: `//@ snapshot NAME PATH XPATH` |
| 113 | +> Usage: `//@ snapshot NAME PATH XPATH` |
75 | 114 |
|
76 | | -Creates a snapshot test named NAME. |
77 | | -A snapshot test captures a subtree of the DOM, at the location |
78 | | -determined by the XPath, and compares it to a pre-recorded value |
79 | | -in a file. The file's name is the test's name with the `.rs` extension |
80 | | -replaced with `.NAME.html`, where NAME is the snapshot's name. |
| 115 | +Checks that the element / text selected by `XPATH` in the file given by `PATH` matches the |
| 116 | +pre-recorded subtree or text (the "snapshot") in file `FILE_STEM.NAME.html` where `FILE_STEM` |
| 117 | +is the file stem of the test file. |
81 | 118 |
|
82 | | -htmldocck supports the `--bless` option to accept the current subtree |
83 | | -as expected, saving it to the file determined by the snapshot's name. |
84 | | -compiletest's `--bless` flag is forwarded to htmldocck. |
| 119 | +Pass the `--bless` option to `compiletest` to accept the current subtree/text as expected. |
| 120 | +This will overwrite the aforementioned file (or create it if it doesn't exist). It will |
| 121 | +automatically normalize the channel-dependent URL `https://doc.rust-lang.org/CHANNEL` to |
| 122 | +the special string `{{channel}}`. |
85 | 123 |
|
86 | 124 | ### `has-dir` |
87 | 125 |
|
88 | | -Usage: `//@ has-dir PATH` |
| 126 | +> Usage: `//@ has-dir PATH` |
89 | 127 |
|
90 | | -Checks for the existence of directory `PATH`. |
| 128 | +Checks for the existence of the directory given by `PATH`. |
91 | 129 |
|
92 | 130 | ### `files` |
93 | 131 |
|
94 | | -Usage: `//@ files PATH ENTRIES` |
| 132 | +> Usage: `//@ files PATH ENTRIES` |
| 133 | +
|
| 134 | +Checks that the directory given by `PATH` contains exactly `ENTRIES`. |
| 135 | +`ENTRIES` is a Python-like list of strings inside a quoted string. |
| 136 | + |
| 137 | +**Example**: `//@ files "foo/bar" '["index.html", "sidebar-items.js"]'` |
| 138 | + |
| 139 | +[^1]: Whitespace normalization means that all spans of consecutive whitespace are replaced with a single space. |
| 140 | +[^2]: They are Unicode aware (flag `UNICODE` is set), match case-sensitively and in single-line mode. |
| 141 | + |
| 142 | +## Compiletest Directives (Brief) |
| 143 | + |
| 144 | +As mentioned in the introduction, you also have access to [compiletest directives]. |
| 145 | +Most importantly, they allow you to register auxiliary crates and |
| 146 | +to pass flags to the `rustdoc` binary under test. |
| 147 | +It's *strongly recommended* to read that chapter if you don't know anything about them yet. |
| 148 | + |
| 149 | +Here are some details that are relevant to this test suite specifically: |
95 | 150 |
|
96 | | -Checks that the directory `PATH` contains exactly `ENTRIES`. |
97 | | -`ENTRIES` is a python list of strings inside a quoted string, |
98 | | -as if it were to be parsed by `eval`. |
99 | | -(note that the list is actually parsed by `shlex.split`, |
100 | | -so it cannot contain arbitrary python expressions). |
| 151 | +* While you can use both `//@ compile-flags` and `//@ doc-flags` to pass flags to `rustdoc`, |
| 152 | + prefer to user the latter to show intent. The former is meant for `rustc`. |
| 153 | +* Add `//@ build-aux-docs` to the test file that has auxiliary crates to not only compile the |
| 154 | + auxiliaries with `rustc` but to also document them with `rustdoc`. |
101 | 155 |
|
102 | | -Example: `//@ files "foo/bar" '["index.html", "sidebar-items.js"]'` |
| 156 | +## Caveats |
103 | 157 |
|
104 | | -[^1]: Whitespace normalization means that all spans of consecutive whitespace are replaced with a single space. The files themselves are also whitespace-normalized. |
| 158 | +Testing for the absence of an element or a piece of text is quite fragile and not very future proof. |
| 159 | + |
| 160 | +It's not unusual that the *shape* of the generated HTML document tree changes from time to time. |
| 161 | +This includes for example renamings of CSS classes. |
| 162 | + |
| 163 | +Whenever that happens, *positive* checks will either continue to match the intended element / |
| 164 | +attribute / text (if their XPath expression is general / loose enough) and |
| 165 | +thus continue to test the correct thing or they won't in which case they would fail thereby |
| 166 | +forcing the author of the change to look at them. |
| 167 | + |
| 168 | +Compare that to *negative* checks (e.g., `//@ !has PATH XPATH PATTERN`) which won't fail if their |
| 169 | +XPath expression "no longer" matches. The author who changed "the shape" thus won't get notified and |
| 170 | +as a result someone else can unintentionally reintroduce `PATTERN` into the generated docs without |
| 171 | +the original negative check failing. |
| 172 | + |
| 173 | +**Note**: Please avoid the use of *negated* checks! |
| 174 | + |
| 175 | +**Tip**: If you can't avoid it, please **always** pair it with an analogous positive check in the |
| 176 | +immediate vicinity, so people changing "the shape" have a chance to notice and to update the |
| 177 | +negated check! |
105 | 178 |
|
106 | 179 | ## Limitations |
107 | | -`htmldocck.py` uses the xpath implementation from the standard library. |
| 180 | + |
| 181 | +HtmlDocCk uses the XPath implementation from the Python standard library. |
108 | 182 | This leads to several limitations: |
| 183 | + |
109 | 184 | * All `XPATH` arguments must start with `//` due to a flaw in the implementation. |
110 | 185 | * Many XPath features (functions, axies, etc.) are not supported. |
111 | 186 | * Only well-formed HTML can be parsed (hopefully rustdoc doesn't output mismatched tags). |
112 | 187 |
|
| 188 | +Furthmore, compiletest [revisions] are not supported. |
| 189 | + |
| 190 | +[revisions]: ../tests/compiletest.md#revisions |
| 191 | +[compiletest directives]: ../tests/directives.md |
0 commit comments