1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
|
When reviewing a test, make sure the test follows the
[format][format] and [style][style] guidelines.
In addition, the test should be checked for the following:
## All tests
<input type="checkbox">
The test passes when it's supposed to pass
<input type="checkbox">
The test fails when it's supposed to fail
<input type="checkbox">
The test is testing what it thinks it's testing
<input type="checkbox">
The spec backs up the expected behavior in the test.
<input type="checkbox">
The test is automated as either [reftest][reftest] or a
[script test][scripttest] unless there's a very good reason why the
test must be manual.
<input type="checkbox">
The test does not use external resources.
<input type="checkbox">
The test does not use proprietary features (vendor-prefixed or otherwise).
## Reftests Only
<input type="checkbox">
The test has a [self-describing][selftest] statement
<input type="checkbox">
The self-describing statement is accurate, precise, simple, and
self-explanatory. Your mother/husband/roommate/brother/bus driver
should be able to say whether the test passed or failed within a few
seconds, and not need to spend several minutes thinking or asking
questions.
<input type="checkbox">
The reference file is accurate and will render pixel-perfect
identically to the test on all platforms.
<input type="checkbox">
The reference file uses a different technique that won't fail in
the same way as the test.
<input type="checkbox">
The title is descriptive but not too wordy.
<input type="checkbox">
The test is as cross-platform as reasonably possible, working
across different devices, screen resolutions, paper sizes, etc. If
there are limitations (e.g. the test will only work on 96dpi
devices, or screens wider than 200 pixels) that these are documented
in the instructions.
## Script Tests Only
<input type="checkbox">
The test uses the most specific asserts possible (e.g. doesn't use
`assert_true` for everything).
<input type="checkbox">
The number of tests in each file and the test names are consistent
across runs and browsers. It is best to avoid the pattern where there is
a test that asserts that the feature is supported and bails out without
running the rest of the tests in the file if it isn't.
<input type="checkbox">
The test avoids patterns that make it less likely to be stable.
In particular, tests should avoid setting internal timeouts, since the
time taken to run it may vary on different devices; events should be used
instead (if at all possible).
<input type="checkbox">
The test uses `idlharness.js` if it covers the use case.
<input type="checkbox">
Tests in a single file are separated by one empty line.
## In-depth Checklist
<input type="checkbox">
A test does not use self-closing start tag ("/" (U+002F)) when using the
HTML syntax.
<input type="checkbox">
The test does not use the Unicode byte order mark (BOM U+FEFF). The test
uses Unix line endings (LF, no CR). The executable bit is not set
unnecessarily.
<input type="checkbox">
For indentation, spaces are preferred over tabs.
<input type="checkbox">
The test does not contain trailing whitespace (whitespace at the end of
lines).
<input type="checkbox">
The test does not contain commented-out code.
<input type="checkbox">
The test does not use `console.*` methods for anything. The
[script test][scripttest] harness never relies on `console.*` methods in
any way, and so use of `console.*` methods in tests is usually just the
equivalent of extra `printf`s in production code; i.e., leftover debugging
that isn't actually useful to the next person running the test. It also
introduces useless overhead when running tests in automation.
<input type="checkbox">
The test is placed in the relevant directory, based on the /TR latest
version link if available.
<input type="checkbox">
If the test needs code running on the server side, the server code must
be written in python, and the python code must be reviewed carefully to
ensure it isn't doing anything dangerous.
[format]: ./test-format-guidelines.html
[style]: ./test-style-guidelines.html
[reftest]: ./reftests.html
[scripttest]: ./testharness-documentation.html
[selftest]: ./test-style-guidelines.html#self-describing
|
