1 files changed, 242 insertions, 0 deletions
diff --git a/testing/web-platform/harness/README.rst b/testing/web-platform/harness/README.rst
new file mode 100644
index 000000000..fc650eec4
--- /dev/null
+++ b/testing/web-platform/harness/README.rst
@@ -0,0 +1,242 @@
+wptrunner: A web-platform-tests harness
+=======================================
+
+wptrunner is a harness for running the W3C `web-platform-tests testsuite`_.
+
+.. contents::
+
+Installation
+~~~~~~~~~~~~
+
+wptrunner is expected to be installed into a virtualenv using pip. For
+development, it can be installed using the `-e` option::
+
+  pip install -e ./
+
+Running the Tests
+~~~~~~~~~~~~~~~~~
+
+After installation, the command ``wptrunner`` should be available to run
+the tests.
+
+The ``wptrunner`` command  takes multiple options, of which the
+following are most significant:
+
+``--product`` (defaults to `firefox`)
+  The product to test against: `b2g`, `chrome`, `firefox`, or `servo`.
+
+``--binary`` (required if product is `firefox` or `servo`)
+  The path to a binary file for the product (browser) to test against.
+
+``--webdriver-binary`` (required if product is `chrome`)
+  The path to a `driver` binary; e.g., a `chromedriver` binary.
+
+``--certutil-binary`` (required if product is `firefox` [#]_)
+  The path to a `certutil` binary (for tests that must be run over https).
+
+``--metadata`` (required)
+  The path to a directory containing test metadata. [#]_
+
+``--tests`` (required)
+  The path to a directory containing a web-platform-tests checkout.
+
+``--prefs-root`` (required only when testing a Firefox binary)
+  The path to a directory containing Firefox test-harness preferences. [#]_
+
+``--config`` (should default to `wptrunner.default.ini`)
+  The path to the config (ini) file.
+
+.. [#] The ``--certutil-binary`` option is required when the product is
+   ``firefox`` unless ``--ssl-type=none`` is specified.
+
+.. [#] The ``--metadata`` path is to a directory that contains:
+
+  * a ``MANIFEST.json`` file (instructions on generating this file are
+    available in the `detailed documentation
+    <http://wptrunner.readthedocs.org/en/latest/usage.html#installing-wptrunner>`_);
+    and
+  * (optionally) any expectation files (see below)
+
+.. [#] Example ``--prefs-root`` value: ``~/mozilla-central/testing/profiles``.
+
+There are also a variety of other options available; use ``--help`` to
+list them.
+
+-------------------------------
+Example: How to start wptrunner
+-------------------------------
+
+To test a Firefox Nightly build in an OS X environment, you might start
+wptrunner using something similar to the following example::
+
+  wptrunner --metadata=~/web-platform-tests/ --tests=~/web-platform-tests/ \
+    --binary=~/mozilla-central/obj-x86_64-apple-darwin14.3.0/dist/Nightly.app/Contents/MacOS/firefox \
+    --certutil-binary=~/mozilla-central/obj-x86_64-apple-darwin14.3.0/security/nss/cmd/certutil/certutil \
+    --prefs-root=~/mozilla-central/testing/profiles
+
+And to test a Chromium build in an OS X environment, you might start
+wptrunner using something similar to the following example::
+
+  wptrunner --metadata=~/web-platform-tests/ --tests=~/web-platform-tests/ \
+    --binary=~/chromium/src/out/Release/Chromium.app/Contents/MacOS/Chromium \
+    --webdriver-binary=/usr/local/bin/chromedriver --product=chrome
+
+-------------------------------------
+Example: How to run a subset of tests
+-------------------------------------
+
+To restrict a test run just to tests in a particular web-platform-tests
+subdirectory, specify the directory name in the positional arguments after
+the options; for example, run just the tests in the `dom` subdirectory::
+
+  wptrunner --metadata=~/web-platform-tests/ --tests=~/web-platform-tests/ \
+    --binary=/path/to/firefox --certutil-binary=/path/to/certutil \
+    --prefs-root=/path/to/testing/profiles \
+    dom
+
+Output
+~~~~~~
+
+By default wptrunner just dumps its entire output as raw JSON messages
+to stdout. This is convenient for piping into other tools, but not ideal
+for humans reading the output.
+
+As an alternative, you can use the ``--log-mach`` option, which provides
+output in a reasonable format for humans. The option requires a value:
+either the path for a file to write the `mach`-formatted output to, or
+"`-`" (a hyphen) to write the `mach`-formatted output to stdout.
+
+When using ``--log-mach``, output of the full raw JSON log is still
+available, from the ``--log-raw`` option. So to output the full raw JSON
+log to a file and a human-readable summary to stdout, you might start
+wptrunner using something similar to the following example::
+
+  wptrunner --metadata=~/web-platform-tests/ --tests=~/web-platform-tests/ \
+    --binary=/path/to/firefox --certutil-binary=/path/to/certutil \
+    --prefs-root=/path/to/testing/profiles \
+    --log-raw=output.log --log-mach=-
+
+Expectation Data
+~~~~~~~~~~~~~~~~
+
+wptrunner is designed to be used in an environment where it is not
+just necessary to know which tests passed, but to compare the results
+between runs. For this reason it is possible to store the results of a
+previous run in a set of ini-like "expectation files". This format is
+documented below. To generate the expectation files use `wptrunner` with
+the `--log-raw=/path/to/log/file` option. This can then be used as
+input to the `wptupdate` tool.
+
+Expectation File Format
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Metadata about tests, notably including their expected results, is
+stored in a modified ini-like format that is designed to be human
+editable, but also to be machine updatable.
+
+Each test file that requires metadata to be specified (because it has
+a non-default expectation or because it is disabled, for example) has
+a corresponding expectation file in the `metadata` directory. For
+example a test file `html/test1.html` containing a failing test would
+have an expectation file called `html/test1.html.ini` in the
+`metadata` directory.
+
+An example of an expectation file is::
+
+  example_default_key: example_value
+
+  [filename.html]
+    type: testharness
+
+    [subtest1]
+      expected: FAIL
+
+    [subtest2]
+      expected:
+        if platform == 'win': TIMEOUT
+        if platform == 'osx': ERROR
+        FAIL
+
+  [filename.html?query=something]
+    type: testharness
+    disabled: bug12345
+
+The file consists of two elements, key-value pairs and
+sections.
+
+Sections are delimited by headings enclosed in square brackets. Any
+closing square bracket in the heading itself my be escaped with a
+backslash. Each section may then contain any number of key-value pairs
+followed by any number of subsections. So that it is clear which data
+belongs to each section without the use of end-section markers, the
+data for each section (i.e. the key-value pairs and subsections) must
+be indented using spaces. Indentation need only be consistent, but
+using two spaces per level is recommended.
+
+In a test expectation file, each resource provided by the file has a
+single section, with the section heading being the part after the last
+`/` in the test url. Tests that have subsections may have subsections
+for those subtests in which the heading is the name of the subtest.
+
+Simple key-value pairs are of the form::
+
+  key: value
+
+Note that unlike ini files, only `:` is a valid seperator; `=` will
+not work as expected. Key-value pairs may also have conditional
+values of the form::
+
+  key:
+    if condition1: value1
+    if condition2: value2
+    default
+
+In this case each conditional is evaluated in turn and the value is
+that on the right hand side of the first matching conditional. In the
+case that no condition matches, the unconditional default is used. If
+no condition matches and no default is provided it is equivalent to
+the key not being present. Conditionals use a simple python-like expression
+language e.g.::
+
+  if debug and (platform == "linux" or platform == "osx"): FAIL
+
+For test expectations the avaliable variables are those in the
+`run_info` which for desktop are `version`, `os`, `bits`, `processor`,
+`debug` and `product`.
+
+Key-value pairs specified at the top level of the file before any
+sections are special as they provide defaults for the rest of the file
+e.g.::
+
+  key1: value1
+
+  [section 1]
+    key2: value2
+
+  [section 2]
+    key1: value3
+
+In this case, inside section 1, `key1` would have the value `value1`
+and `key2` the value `value2` whereas in section 2 `key1` would have
+the value `value3` and `key2` would be undefined.
+
+The web-platform-test harness knows about several keys:
+
+`expected`
+  Must evaluate to a possible test status indicating the expected
+  result of the test. The implicit default is PASS or OK when the
+  field isn't present.
+
+`disabled`
+  Any value indicates that the test is disabled.
+
+`type`
+  The test type e.g. `testharness`, `reftest`, or `wdspec`.
+
+`reftype`
+  The type of comparison for reftests; either `==` or `!=`.
+
+`refurl`
+  The reference url for reftests.
+
+.. _`web-platform-tests testsuite`: https://github.com/w3c/web-platform-tests