diff options
author | Matt A. Tobin <mattatobin@localhost.localdomain> | 2018-02-02 04:16:08 -0500 |
---|---|---|
committer | Matt A. Tobin <mattatobin@localhost.localdomain> | 2018-02-02 04:16:08 -0500 |
commit | 5f8de423f190bbb79a62f804151bc24824fa32d8 (patch) | |
tree | 10027f336435511475e392454359edea8e25895d /testing/web-platform/tests/docs/reftests.md | |
parent | 49ee0794b5d912db1f95dce6eb52d781dc210db5 (diff) | |
download | UXP-5f8de423f190bbb79a62f804151bc24824fa32d8.tar UXP-5f8de423f190bbb79a62f804151bc24824fa32d8.tar.gz UXP-5f8de423f190bbb79a62f804151bc24824fa32d8.tar.lz UXP-5f8de423f190bbb79a62f804151bc24824fa32d8.tar.xz UXP-5f8de423f190bbb79a62f804151bc24824fa32d8.zip |
Add m-esr52 at 52.6.0
Diffstat (limited to 'testing/web-platform/tests/docs/reftests.md')
-rw-r--r-- | testing/web-platform/tests/docs/reftests.md | 152 |
1 files changed, 152 insertions, 0 deletions
diff --git a/testing/web-platform/tests/docs/reftests.md b/testing/web-platform/tests/docs/reftests.md new file mode 100644 index 000000000..f096bc0c6 --- /dev/null +++ b/testing/web-platform/tests/docs/reftests.md @@ -0,0 +1,152 @@ +A reftest is a test that compares the visual output of one file (the +test case) with the output of one or more other files (the +references). The test and the reference must be carefully written so +that when the test passes they have identical rendering, but different +rendering when the test fails. + +## How to Run Reftests + +Reftests can be run manually simply by opening the test and the +reference file in multiple windows or tabs and either placing them +side-by side or flipping between the two. In automation the comparison +is done in an automated fashion, which can lead to differences too +small for the human eye to notice causing tests to fail. + +## Components of a Reftest + +In the simplest case, a reftest consists of a pair of files called the +*test* and the *reference*. + +The *test* file is the one that makes use of the technology being +tested. It also contains a `link` element with `rel="match"` or +`rel="mismatch"` and `href` attribute pointing to the *reference* file +e.g. `<link rel=match href=references/green-box-ref.html>`. + +The *reference* file is typically written to be as simple as possible, +and does not use the technology under test. It is desirable that the +reference be rendered correctly even in UAs with relatively poor +support for CSS and no support for the technology under test. + +When the `<link>` element in the *test* has `rel="match"`, the test +only passes if the *test* and *reference* have pixel-perfect identical +rendering. `rel="mismatch"` inverts this so the test only passes when +the renderings differ. + +In general the files used in a reftest should follow the +[format][format] and [style][style] guidelines. The *test* should also +be [self-describing][selfdesc], to allow a human to determine whether +the the rendering is as expected. + +Note that references can be shared between tests; this is strongly +encouraged since it permits optimizations when running tests. + +## Controlling When Comparison Occurs + +By default reftest screenshots are taken in response to the `load` +event firing. In some cases it is necessary to delay the screenshot +later than this, for example becase some DOM manipulation is +required to set up the desired test conditions. To enable this, the +test may have a `class="reftest-wait"` attribute specified on the root +element. This will cause the screenshot to be delayed until the `load` +event has fired and the `reftest-wait` class has been removed from the +root element (technical note: the implementation in wptrunner uses +mutation observers so the screenshot will be triggered in the +microtask checkpoint after the class is removed. Because the harness +isn't synchronized with the browser event loop it is dangerous to rely +on precise timing here). + +## Matching Multiple References + +Sometimes it is desirable for a file to match multiple references or, +in rare cases, to allow it to match more than one possible +reference. Note: *this is not currently supported by test runners and +so best avoided if possible until that support improves*. + +Multiple references linked from a single file are interpreted as +multiple possible renderings for that file. `<link rel=[mis]match>` +elements in a reference create further conditions that must be met in +order for the test to pass. For example, consider a situation where +`a.html` has `<link rel=match href=b.html>` and `<link rel=match +href=c.html>`, `b.html` has `<link rel=match href=b1.html>` and `c.html` +has `<link rel=mismatch href=c1.html>`. In this case, to pass we must +either have `a.html`, `b.html` and `b1.html` all rendering identically, or +`a.html` and `c.html` rendering identically, but `c.html` rendering +differently from `c1.html`. + +## Fuzzy Matching + +In some situations a test may have subtle differences in rendering +compared to the reference due to e.g. antialiasing. This may cause the +test to pass on some platforms but fail on others. In this case some +affordance for subtle discrepancies is desirable. However no mechanism +to allow this has yet been standardized. + +## Limitations + +In some cases, a test cannot be a reftest. For example, there is no +way to create a reference for underlining, since the position and +thickness of the underline depends on the UA, the font, and/or the +platform. However, once it's established that underlining an inline +element works, it's possible to construct a reftest for underlining +a block element, by constructing a reference using underlines on a +```<span>``` that wraps all the content inside the block. + +## Example Reftests + +These examples are all [self-describing][selfdesc] tests as they +each have a simple statement on the page describing how it should +render to pass the tests. + +### HTML example + +### Test File + +This test verifies that a right-to-left rendering of **SAW** within a +```<bdo>``` element displays as **WAS**. + +([view page rendering][html-reftest-example]) + +```html +<!DOCTYPE html> +<meta charset="utf-8"> +<title>BDO element dir=rtl</title> +<link rel="help" href="http://www.whatwg.org/specs/web-apps/current-work/#the-bdo-element"> +<meta name="assert" content="BDO element's DIR content attribute renders corrently given value of 'rtl'."> +<link rel="match" href="test-bdo-001.html"> +<p>Pass if you see WAS displayed below.</p> +<bdo dir="rtl">SAW</bdo> +``` + +### Reference File + +The reference file must look exactly like the test file, +except that the code behind it is different. + +* All metadata is removed. +* The ```title``` need not match. +* The markup that created the actual test data is + different: here, the same effect is created with + very mundane, dependable technology. + +([view page rendering][html-reffile-example]) + +```html +<!DOCTYPE html> +<meta charset="utf-8"> +<title>HTML Reference File</title> +<p>Pass if you see WAS displayed below.</p> +<p>WAS</p> +``` + +[testharness]: ./testharness-documentation.html +[format]: ./test-format-guidelines.html +[style]: ./test-style-guidelines.html +[selfdesc]: ./test-style-guidelines.html#self-describing-tests +[reference-links]: ./test-templates.html#reference-links +[html-reftest-example]: ./html-reftest-example.html +[html-reffile-example]: ./html-reffile-example.html +[css-reftest-example]: http://test.csswg.org/source/css21/borders/border-bottom-applies-to-009.xht +[css-reffile-example]: http://test.csswg.org/source/css21/borders/border-bottom-applies-to-001-ref.xht +[svg-reftest-example]: http://test.csswg.org/source/css-transforms-1/translate/svg-translate-001.html +[svg-reffile-example]: http://test.csswg.org/source/css-transforms-1/translate/reference/svg-translate-ref.html +[indicating-failure]: ./test-style-guidelines.html#failure |