summaryrefslogtreecommitdiffstats
path: root/testing/web-platform/tests/annotation-protocol/README.md
diff options
context:
space:
mode:
Diffstat (limited to 'testing/web-platform/tests/annotation-protocol/README.md')
-rw-r--r--testing/web-platform/tests/annotation-protocol/README.md86
1 files changed, 86 insertions, 0 deletions
diff --git a/testing/web-platform/tests/annotation-protocol/README.md b/testing/web-platform/tests/annotation-protocol/README.md
new file mode 100644
index 000000000..d0ec93573
--- /dev/null
+++ b/testing/web-platform/tests/annotation-protocol/README.md
@@ -0,0 +1,86 @@
+Annotation-Protocol: Tests for the Web Annotation Protocol
+==========================================================
+
+The [Web Annotation Protocol](https://www.w3.org/TR/annotation-protocol)
+specification presents set of messages to allow Annotation clients and servers
+to interact seamlessly.
+
+The purpose of these tests is to help validate that clients send and are
+capable of receiving correctly formatted messages, and that servers are
+able to receive and respond to correctly structured requests.
+
+The general approach for this testing is to enable both manual and
+automated testing. However, since the specification has no actual user
+interface requirements, there is no general automation mechanism that
+can be presented for testing clients. Also the server tests need to be
+pointed at a server implementation to exercise. However, once provided
+the basic information, testing is automated.
+
+Implementors could take advantage of the plumbing we provide here to
+help their implementations talk to the endpoint we provide or exercise
+their endpoint with the provided server tests. This assumes knowledge
+of the requirements of each test / collection of tests so that the input
+data is relevant. Each test or test collection contains information
+sufficient for the task.
+
+With regard to server tests, the browser tests we provide can be
+pointed at an endpoint and will exercise that endpoint using well
+defined messages. This is done semi-automatically, although some set-up
+is required.
+
+Running Tests
+-------------
+
+In the case of this test collection, we will be initially creating manual
+tests. These will automatically determine pass or fail and generate output for
+the main WPT window. The plan is to minimize the number of such tests to
+ease the burden on the testers while still exercising all the features.
+
+The workflow for running these tests is something like:
+
+1. Start up the test driver window and select the annotation-protocol tests -
+ either client or server - then click "Start".
+2. A window pops up that shows a test - the description of which tells the
+ tester what is required. The window will contain fields into which some
+ information is provided.
+3. In the case of client testing the tester (presumably in another window) brings up their
+ annotation client and points it at the supplied endpoint. They they perform the
+ action specified (annotating content in the test window, requesting an annotation from the server, etc.).
+4. The server receives the information from the client, evaluates it, and reports the result of testing.
+ In the event of multi-step messages, the cycle repeats until complete.
+5. Repeat steps 2-4 until done.
+6. Download the JSON format report of test results, which can then be visually
+ inspected, reported on using various tools, or passed on to W3C for
+ evaluation and collection in the Implementation Report via github.
+
+**Remember that while these tests are written to help exercise implementations,
+their other (important) purpose is to increase confidence that there are
+interoperable implementations.** So, implementers are our audience, but these
+tests are not meant to be a comprehensive collection of tests for an implementor.
+The bulk of the tests are manual because there are no UI requirements in the
+Recommendation that would make it possible to effectively stimulate every client portably.
+
+Having said that, because the structure of these "manual" tests is very rigid,
+it is possible for an implementer who understands test automation to use an
+open source tool such as [Selenium](http://www.seleniumhq.org/) to run these
+"manual" tests against their implementation - exercising their implementation
+against content they provide to create annotations and feed the data into our
+test input field and run the test.
+
+Capturing and Reporting Results
+-------------------------------
+
+As tests are run against implementations, if the results of testing are
+submitted to [test-results](https://github.com/w3c/test-results/) then they will
+be automatically included in documents generated by
+[wptreport](https://www.github.com/w3c/wptreport). The same tool can be used
+locally to view reports about recorded results.
+
+Automating Test Execution
+-------------------------
+
+Writing Tests
+-------------
+
+If you are interested in writing tests for this environment, see the
+associated [CONTRIBUTING](CONTRIBUTING.md) document.