diff options
| author | Matt A. Tobin <mattatobin@localhost.localdomain> | 2018-02-02 04:16:08 -0500 |
|---|---|---|
| committer | Matt A. Tobin <mattatobin@localhost.localdomain> | 2018-02-02 04:16:08 -0500 |
| commit | 5f8de423f190bbb79a62f804151bc24824fa32d8 (patch) | |
| tree | 10027f336435511475e392454359edea8e25895d /testing/web-platform/tests/docs/manual-test.md | |
| parent | 49ee0794b5d912db1f95dce6eb52d781dc210db5 (diff) | |
| download | UXP-5f8de423f190bbb79a62f804151bc24824fa32d8.tar UXP-5f8de423f190bbb79a62f804151bc24824fa32d8.tar.gz UXP-5f8de423f190bbb79a62f804151bc24824fa32d8.tar.lz UXP-5f8de423f190bbb79a62f804151bc24824fa32d8.tar.xz UXP-5f8de423f190bbb79a62f804151bc24824fa32d8.zip | |
Add m-esr52 at 52.6.0
Diffstat (limited to 'testing/web-platform/tests/docs/manual-test.md')
| -rw-r--r-- | testing/web-platform/tests/docs/manual-test.md | 72 |
1 files changed, 72 insertions, 0 deletions
diff --git a/testing/web-platform/tests/docs/manual-test.md b/testing/web-platform/tests/docs/manual-test.md new file mode 100644 index 000000000..4b5469589 --- /dev/null +++ b/testing/web-platform/tests/docs/manual-test.md @@ -0,0 +1,72 @@ +Some testing scenarios are intrinsically difficult to automate and +require a human to run the test and check the pass condition. + +## When to Write Manual Tests + +Whenever possible it's best to write a fully automated test. For a +browser vendor it's possible to run an automated test hundreds of +times a day, but manual tests are likely to be run a handful of times +a year. This makes them significantly less useful for catching +regressions than automated tests. + +However, there are certain scenarios in which this is not yet +possible. For example: + +* Tests that require interaction with browser security UI (e.g. a test + in which a user refuses a geolocation permissions grant) + +* Tests that require interaction with the underlying OS e.g. tests for + drag and drop from the desktop onto the browser + +* Tests that require non-default browser configuration e.g. images + disabled + +* Tests that require interaction with the physical environment + e.g. tests that the vibration API causes the device to vibrate or + that various sensor APIs respond in the expected way. + +There are also some rare cases where it isn't possible to write a layout +test as a reftest, and a manual test must be written instead. + +## Requirements for a Manual Test + +Manual tests are distinguished by their filename; all manual tests +have filenames of the form `name-manual.ext` i.e. a `-manual` +suffix after the main filename but before the extension. + +Manual tests must be fully +[self-describing](test-style-guielines.html#self-describing-tests). It +is particularly important for these tests that it is easy to determine +the result from the information presented to the tester, because a +tester may have hundreds of tests to get through, and little +understanding of the features that they are testing. Therefore +minimalism is a virtue. An ideal self-describing test will have: + +* Step-by-step instructions for performing the test (if required) + +* A clear statement of the test outcome (if it can be automatically + determined after some setup) or of how to determine the outcome. + +Any information other than this (e.g. quotes from the spec) should be +avoided. + +## Using testharness.js for Manual Tests + +A convenient way to present the results of a test that can have the +result determined by script after some manual setup steps is to use +testharness.js to determine and present the result. In this case one +must pass `{explicit_timeout: true}` in a call to `setup()` in order +to disable the automatic timeout of the test. For example: + +```html +<!doctype html> +<title>Manual click on button triggers onclick handler</title> +<script src="/resources/testharness.js"></script> +<script src="/resources/testharnessreport.js"></script> +<script> +setup({explicit_timeout: true}) +</script> +<p>Click on the button below. If a "PASS" result appears the test +passes, otherwise it fails</p> +<button onclick="done()">Click Here</button> +``` |