diff options
Diffstat (limited to 'testing/web-platform/tests/annotation-model/CONTRIBUTING.md')
| -rw-r--r-- | testing/web-platform/tests/annotation-model/CONTRIBUTING.md | 381 |
1 files changed, 0 insertions, 381 deletions
diff --git a/testing/web-platform/tests/annotation-model/CONTRIBUTING.md b/testing/web-platform/tests/annotation-model/CONTRIBUTING.md deleted file mode 100644 index f591d2a97..000000000 --- a/testing/web-platform/tests/annotation-model/CONTRIBUTING.md +++ /dev/null @@ -1,381 +0,0 @@ -Annotation-model: Guidelines for Contributing Tests -=================================================== - -This document describes the method people should use for authoring tests and -integrating them into the repository. Anyone is welcome to submit new tests to -this collection. If you do, please create the tests following the guidelines -below. Then submit them as a pull request so they can be evaluated - -Structure ---------- - -Tests are organized by major section of the Annotation Model specification. The -folders associated with these are: - -* annotations -* bodiesTargets -* collections -* specificResources - * selectors - * states - -Within these folders, special files ending with the suffix ".test" provide the source -for the test as a set of declarative assertions about the required shape of the conforming -JSON object. These files are transformed using a test generation tool into ".html" files -that are then accessed by the Web Platform Test framework. - -There are a few other folders that provide supporting materials for the tests: - -* common - assertionObjects, conditionObjects, and other supporting materials -* definitions - JSON Schema definitions that can be referenced -* scripts - JavaScript that are included by tests -* tools - supporting scripts and files - -NOTE: The files in the definitions folder are expected to be JSON Schema -definitions - basically commonly used concepts that are referenced by other JSON -Schema files in the system. All of these 'definitions' are preloaded by the -system before any other parts of a test are processed. - -Test Cases ----------- - -Each test is expressed as a simple (or complex) requirement in a test file. -For each section of the document, the requirement is represented as a structure -that describes the nature of the test, and then includes or references minimal -JSON Schema that test the assertions implied by the requirement. - -The structure of a test case is defined using a [JSON-LD -Context](JSONtest-v1.jsonld). That context defines the following terms: - -|Keyword | Values | Meaning -|---------------|-----------------|--------- -|name | string | The name of this test for display purposes -|description | string | A long self-describing paragraph that explains the purpose of the test and the expected input -|ref | URI | An optional reference to the portion of the specification to which the test relates -|testType | `automated`, `manual`, `ref` | The type of test - this informs [WPT](https://github.com/w3c/web-platform-tests) how the test should be controlled and presented -|skipFailures | list of strings | An optional list of assertionType values that, if present, should have their test skipped if the result would be "unexpected". Defaults to the empty list. -|assertions | list of URI, List @@@ATRISK@@@, or AssertionObject | The ordered collection of tests the input should be run against. See [JSON Schema Usage](#jsonSchema) for the structure of the objects. URI is relative to the top level folder of the test collection if it has a slash; relative to the current directory if it does not. @@@@ATRISK@@@@ Lists can be nested to define groups of sub-tests. Assertions / groups can be conditionally skipped. See [Assertion Lists](#assertionLists) for more details. -|content | URI or object | An object containing content to be checked against the referenced assertions, or a URI from which to retrieve that content - -Each test case has a suffix of `.test` and a shape like: - -<pre> -{ - "@context": "https://www.w3.org/ns/JSONtest-v1.jsonld", - "name": "Verify annotation conforms to the model", - "description": "Supply an example annotation that conforms to the basic structure.", - "ref": "https://www.w3.org/TR/annotation-model/#model", - "testType": "manual", - "assertions": [ - "common/has_context.json", - "common/has_id.json", - { - "$schema": "http://json-schema.org/draft-04/schema#", - "title": "Verify annotation has target", - "assertionType": "must", - "expectedResult": "valid", - "errorMessage": "The object was missing a required 'target' property", - "type": "object", - "properties": { - "target": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "array", - "anyOf": [ - { - "type": "string" - } - ] - } - ], - "not": {"type": "object"} - } - }, - "required": ["target"] - } - ] -} -</pre> - -External references are used when the "assertion" is a common one that needs to -be checked on many different test cases (e.g., that there is an @context in the -supplied annotation). - -NOTE: The title property of an assertionObject can contain markdown. This can -help improve readability of the rendered assertions and debugging output. - -NOTE: The content property does not yet have a defined use. One potential use would -be to act as a pointer to a URI that can supply annotations from an implementation. -In that case the URI would take a parameter with the test name as a way of telling -the end point what test is running so it can deliver the right content. - -### <a id="assertionLists">Assertion Lists</a> ### - -The `assertion` list is an ordered list of assertions that will be evaluated -against the submitted content. The list is *required*, and MUST have at least -one entry. Entries in the list have the following types: - -* AssertionObject - -An in-line Object as defined in the section [Assertion -Objects](#assertionObjects). -* URI - -A relative or absolute URI that references a AssertionObject in a .json file. -If the URI is relative but contains no slashes, then it is considered to be -in the current directory. If the URI is relative, contains slashes, but -**does not start with a slash** then it is considered relative to the top of -the tree of the current test collection (e.g., `annotation-model`). -* List @@@ATRISK@@@ - -A nested Assertion List. While nested Assertion Lists are optional, if one -is present it MUST have at least one entry. Entries are as in this list. -Assertion Lists can be nested to any depth (but don't do that - it would be -too hard to maintain). - - -<a id="assertionObjects">Assertion Objects</a> ------------------ - -In this collection of tests, Assertion Objects can be contained inline in the -`.test` files or contained in external files with the suffix `.json`. The -vocabularly and structure is as defined in [JSON Schema -v4](http://json-schema.org/documentation.html) augmented with some additional -properties defined in this section. - -In general each JSON Schema definition provided in this test suite should be as -minimal as possible. This promotes clarity and increases the likelihood that -it is correct. While it is ---possible--- to create JSON Schema files that -enforce many different requirements on a data model, it increases the -complexity and can also reduce the atomicity of tests / sub-tests (because a - test ends up testing more than one thing). Please try to avoid creating -complex JSON Schema. (A notable exception is the situation where multiple - properties of a structure are interdependent.) - -Tools such as [the JSON Schema Creator](http://jsonschema.net/) may be helpful -in creating schema snippets that can be integrated into JSONtest Assertion -Objects. Remember that the JSON Schema you create should be as permissive as -possible to address the various shapes that a give property might take (e.g., a - 'foo' might be a string or an object that contains sub-properties that express - the string, or an array of 1 or more objects with those sub-properties). - -In addition to the validation keys defined in JSON Schema v4, Schema files in -this collection are also permitted to use the following keywords: - -|Keyword | Values | Meaning | -|---------------|-----------------|---------| -|onUnexpectedResult | `failAndContinue`, `failAndSkip`, `failAndAbort`, `passAndContinue`, `passAndSkip`, `passAndAbort` | Action to take when the result is not as expected. Default is `failAndContinue` | -|assertionType | `must`, `may`, `should` | Informs the system about the severity of a failure. The default is `must` | -|assertionFile | URI | An external file that contains an assertion SCHEMA. When this value is supplied, and local properties will override the ones loaded from the external file. -|errorMessage | string | A human readable explanation of what it means if the test fails. | -|expectedResult | `valid`, `invalid` | Tells the framework whether validating against this schema is expected to succeed or fail. The default is `valid` | - - -### Example Assertion Object ### - -<pre> -{ - "$schema": "http://json-schema.org/draft-04/schema#", - "title": "Verify annotation has @context", - "type": "object", - "expectedResult" : "valid", - "properties": { - "@context": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "array", - "anyOf": [ - { - "type": "string" - } - ] - } - ], - "not": {"type": "object"} - } - }, - "required": ["@context"] -} -</pre> - -Note that in the case where a feature is *optional* the JSON Schema MUST be -crafted such that if the attribute is permitted to be missing from the content -(so that the result is `true`), but when the attribute is present in the -content it conforms to any requirements. - - - -<a id="conditionObjects">Condition Objects</a> ------------------ - -A Condition Object is a sub-class of an Assertion Object. It allows the -specification of the evaluation strategy for the assertions referenced by the -object. An object is a Condition Object IFF it has a `assertions` property. In -this case, there MUST NOT be an `assertionFile` property. - - -|Keyword | Values | Meaning | -|---------------|-----------------|---------| -|compareWith | `and`, `or` | How should the result of any referenced assertions be compared. Defaults to `and`. Note that this implies there is also an assertions property with a nested list of assertions to compare. | -|assertions | a list of assertions as in a Test Case above. This is required if there is a compareWith property | - - -An example of a test that would pass if there were an `@context` OR there were an `@id`: - -<pre> -{ - "@context": "https://www.w3.org/ns/JSONtest-v1.jsonld", - "name": "A test that has an 'or' clause", - "description": "A complex test that uses or-ing among a list of assertions", - "ref": "https://www.w3.org/TR/annotation-model/#model", - "testType": "manual", - "assertions": [ - { "$schema": "http://json-schema.org/draft-04/schema#", - "title": "must have context or id", - "description": "A more complex example that allows one of many options to pass", - "assertions": [ - { "title": "Condition Object", - "description": "A pseudo-test that will get a result from the aggregate of its children", - "assertionType": "must", - "expectedResult": "valid", - "errorMessage": "Error: None of the various options were present", - "compareWith": "or", - "assertions": [ - "common/has_context.json", - "common/has_id.json" - ] - } - ] - } - ] -} -</pre> - - -Command Line Tools ------------------- - -### Building the Test Files ### - -The actual .html test case files are generated using the script -tools/make_tests.py. This script will search the directory heriarchy looking for -files ending on `.test` and creating `.html` files from them using the template in -the tools folder. If you want to regenerate the examples too, supply the -`--examples` option to the script. - -Note that when submitting tests to the repository, the `.html` versions must be -included. - -### Testing the Tests ### - -### Driving Tests with Input Files ### - -Complex Examples ----------------- - -This section is a collection of more complex examples to illustrate the -expressive power of the [Assertion List](#assertionLists) structure. These can -be used as templates for creating actual `.test` files. - -### Including and Overriding an Assertion ### - -Assertions can be contained in external `.json` files. It is possible for an -object in an Assertion List to include the external file and override one or -more of its properties: - -<pre> -{ - "@context": "https://www.w3.org/ns/JSONtest-v1.jsonld", - "name": "Permit no target property", - "description": "Ensure there is no 'target' property when there is a 'randomName' property in the Annotation", - "assertions": [ - { - "$schema": "http://json-schema.org/draft-04/schema#", - "title": "Verify annotation has randomName", - "type": "object", - "properties": { - "randomName": { - "type": "string" - } - }, - "required": ["randomName"] - }, - { "assertionFile" : "common/target.json", - "title" : "Require target to be missing", - "expectedResult" : "invalid", - "errorMessage" : "The target MUST not be present when 'randomName' is also present", - } - ] -} -</pre> - -### Nested Assertion Collections with Skip ### - -Assertion Lists can be nested within Assertion Lists. This feature, combined -with the `onUnexpectedResult` property, makes it possible to skip a collection -of tests when an assertion in the list is not satisfied: - -<pre> -{ - "@context": "https://www.w3.org/ns/JSONtest-v1.jsonld", - "name": "If there is no 'target' property, skip some tests", - "description": "When 'target' is not present, other properties related to 'target' are not required", - "assertions": [ - "common/context.json", - [ - { "assertionFile" : "common/target.json", - "errorMessage" : "Target was not present so skip the rest of this section", - "onUnexpectedResult" : "failAndSkip" - }, - "sometest.json", - "sometest2.json", - "sometest3.json" - ] - ] -} ; -</pre> - -### Assertion that finds a specific @context Value ### - -Sometimes you want a property to be flexible, but to have one and only one of a -specific value. This is especially true with, for example, @context in JSON-LD. -One way you might do this is: - -<pre> -{ - "$schema": "http://json-schema.org/draft-04/schema#", - "title": "Verify a specific @context", - "type": "object", - "expectedResult" : "valid", - "properties": { - "@context": { - "anyOf": [ - { - "type": "string" - "enum": [ "http://www.w3.org/ns/anno.jsonld" ] - }, - { - "type": "array", - "minitems": "1", - "uniqueItems": true, - "additionalItems": true, - "items" : [ - { "type": "string", - "enum": [ "http://www.w3.org/ns/anno.jsonld" ] - } - ] - } - ], - "not": {"type": "object"} - } - }, - "required": ["@context"] -} - -</pre> |