Add Basilisk

1 files changed, 429 insertions, 0 deletions

diff --git a/application/basilisk/experiments/docs/manifest.rst b/application/basilisk/experiments/docs/manifest.rst
new file mode 100644
index 000000000..d4fad5243
--- /dev/null
+++ b/application/basilisk/experiments/docs/manifest.rst
@@ -0,0 +1,429 @@
+.. _experiments_manifests:
+
+=====================
+Experiments Manifests
+=====================
+
+*Experiments Manifests* are documents that describe the set of active
+experiments a client may run.
+
+*Experiments Manifests* are fetched periodically by clients. When
+fetched, clients look at the experiments within the manifest and
+determine which experiments are applicable. If an experiment is
+applicable, the client may download and start the experiment.
+
+Manifest Format
+===============
+
+Manifests are JSON documents where the main element is an object.
+
+The *schema* of the object is versioned and defined by the presence
+of a top-level ``version`` property, whose integer value is the
+schema version used by that manifest. Each version is documented
+in the sections below.
+
+Version 1
+---------
+
+Version 1 is the original manifest format.
+
+The following properties may exist in the root object:
+
+experiments
+   An array of objects describing candidate experiments. The format of
+   these objects is documented below.
+
+   An array is used to create an explicit priority of experiments.
+   Experiments listed at the beginning of the array take priority over
+   experiments that follow.
+
+Experiments Objects
+^^^^^^^^^^^^^^^^^^^
+
+Each object in the ``experiments`` array may contain the following
+properties:
+
+id
+   (required) String identifier of this experiment. The identifier should
+   be treated as opaque by clients. It is used to uniquely identify an
+   experiment for all of time.
+
+xpiURL
+   (required) String URL of the XPI that implements this experiment.
+
+   If the experiment is activated, the client will download and install this
+   XPI.
+
+xpiHash
+   (required) String hash of the XPI that implements this experiment.
+
+   The value is composed of a hash identifier followed by a colon
+   followed by the hash value. e.g.
+   `sha1:f677428b9172e22e9911039aef03f3736e7f78a7`. `sha1` and `sha256`
+   are the two supported hashing mechanisms. The hash value is the hex
+   encoding of the binary hash.
+
+   When the client downloads the XPI for the experiment, it should compare
+   the hash of that XPI against this value. If the hashes don't match,
+   the client should not install the XPI.
+
+   Clients may also use this hash as a means of determining when an
+   experiment's XPI has changed and should be refreshed.
+
+startTime
+   Integer seconds since UNIX epoch that this experiment should
+   start. Clients should not start an experiment if *now()* is less than
+   this value.
+
+maxStartTime
+   (optional) Integer seconds since UNIX epoch after which this experiment
+   should no longer start.
+
+   Some experiments may wish to impose hard deadlines after which no new
+   clients should activate the experiment. This property may be used to
+   facilitate that.
+
+endTime
+   Integer seconds since UNIX epoch after which this experiment
+   should no longer run. Clients should cease an experiment when the current
+   time is beyond this value.
+
+maxActiveSeconds
+   Integer seconds defining the max wall time this experiment should be
+   active for.
+
+   The client should deactivate the experiment this many seconds after
+   initial activation.
+
+   This value only involves wall time, not browser activity or session time.
+
+appName
+   Array of application names this experiment should run on.
+
+   An application name comes from ``nsIXULAppInfo.name``. It is a value
+   like ``Firefox``, ``Fennec``, or `B2G`.
+
+   The client should compare its application name against the members of
+   this array. If a match is found, the experiment is applicable.
+
+minVersion
+   (optional) String version number of the minimum application version this
+   experiment should run on.
+
+   A version number is something like ``27.0.0`` or ``28``.
+
+   The client should compare its version number to this value. If the client's
+   version is greater or equal to this version (using a version-aware comparison
+   function), the experiment is applicable.
+
+   If this is not specified, there is no lower bound to versions this
+   experiment should run on.
+
+maxVersion
+   (optional) String version number of the maximum application version this
+   experiment should run on.
+
+   This is similar to ``minVersion`` except it sets the upper bound for
+   application versions.
+
+   If the client's version is less than or equal to this version, the
+   experiment is applicable.
+
+   If this is not specified, there is no upper bound to versions this
+   experiment should run on.
+
+version
+   (optional) Array of application versions this experiment should run on.
+
+   This is similar to ``minVersion`` and ``maxVersion`` except only a
+   whitelisted set of specific versions are allowed.
+
+   The client should compare its version to members of this array. If a match
+   is found, the experiment is applicable.
+
+minBuildID
+   (optional) String minimum Build ID this experiment should run on.
+
+   Build IDs are values like ``201402261424``.
+
+   The client should perform a string comparison of its Build ID against this
+   value. If its value is greater than or equal to this value, the experiment
+   is applicable.
+
+maxBuildID
+   (optional) String maximum Build ID this experiment should run on.
+
+   This is similar to ``minBuildID`` except it sets the upper bound
+   for Build IDs.
+
+   The client should perform a string comparison of its Build ID against
+   this value. If its value is less than or equal to this value, the
+   experiment is applicable.
+
+buildIDs
+   (optional) Array of Build IDs this experiment should run on.
+
+   This is similar to ``minBuildID`` and ``maxBuildID`` except only a
+   whitelisted set of Build IDs are considered.
+
+   The client should compare its Build ID to members of this array. If a
+   match is found, the experiment is applicable.
+
+os
+   (optional) Array of operating system identifiers this experiment should
+   run on.
+
+   Values for this array come from ``nsIXULRuntime.OS``.
+
+   The client will compare its operating system identifier to members
+   of this array. If a match is found, the experiment is applicable to the
+   client.
+
+channel
+   (optional) Array of release channel identifiers this experiment should run
+   on.
+
+   The client will compare its channel to members of this array. If a match
+   is found, the experiment is applicable.
+
+   If this property is not defined, the client should assume the experiment
+   is to run on all channels.
+
+locale
+   (optional) Array of locale identifiers this experiment should run on.
+
+   A locale identifier is a string like ``en-US`` or ``zh-CN`` and is
+   obtained by looking at
+   ``nsIXULChromeRegistry.getSelectedLocale("global")``.
+
+   The client should compare its locale identifier to members of this array.
+   If a match is found, the experiment is applicable.
+
+   If this property is not defined, the client should assume the experiment
+   is to run on all locales.
+
+sample
+   (optional) Decimal number indicating the sampling rate for this experiment.
+
+   This will contain a value between ``0.0`` and ``1.0``. The client should
+   generate a random decimal between ``0.0`` and ``1.0``. If the randomly
+   generated number is less than or equal to the value of this field, the
+   experiment is applicable.
+
+disabled
+   (optional) Boolean value indicating whether an experiment is disabled.
+
+   Normally, experiments are deactivated after a certain time has passed or
+   after the experiment itself determines it no longer needs to run (perhaps
+   it collected sufficient data already).
+
+   This property serves as a backup mechanism to remotely disable an
+   experiment before it was scheduled to be disabled. It can be used to
+   kill experiments that are found to be doing wrong or bad things or that
+   aren't useful.
+
+   If this property is not defined or is false, the client should assume
+   the experiment is active and a candidate for activation.
+
+frozen
+   (optional) Boolean value indicating this experiment is frozen and no
+   longer accepting new enrollments.
+
+   If a client sees a true value in this field, it should not attempt to
+   activate an experiment.
+
+jsfilter
+    (optional) JavaScript code that will be evaluated to determine experiment
+    applicability.
+
+    This property contains the string representation of JavaScript code that
+    will be evaluated in a sandboxed environment using JavaScript's
+    ``eval()``.
+
+    The string is expected to contain the definition of a JavaScript function
+    ``filter(context)``. This function receives as its argument an object
+    holding application state. See the section below for the definition of
+    this object.
+
+    The purpose of this property is to allow experiments to define complex
+    rules and logic for evaluating experiment applicability in a manner
+    that is privacy conscious and doesn't require the transmission of
+    excessive data.
+
+    The return value of this filter indicates whether the experiment is
+    applicable. Functions should return true if the experiment is
+    applicable.
+
+    If an experiment is not applicable, they should throw an Error whose
+    message contains the reason the experiment is not applicable. This
+    message may be logged and sent to remote servers, so it should not
+    contain private or otherwise sensitive data that wouldn't normally
+    be submitted.
+
+    If a falsey (or undefined) value is returned, the client should
+    assume the experiment is not applicable.
+
+    If this property is not defined, the client does not consider a custom
+    JavaScript filter function when determining whether an experiment is
+    applicable.
+
+JavaScript Filter Context Objects
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The object passed to a ``jsfilter`` ``filter()`` function contains the
+following properties:
+
+healthReportSubmissionEnabled
+   This property contains a boolean indicating whether Firefox Health
+   Report has its data submission flag enabled (whether Firefox Health
+   Report is sending data to remote servers).
+
+healthReportPayload
+   This property contains the current Firefox Health Report payload.
+
+   The payload format is documented at :ref:`healthreport_dataformat`.
+
+telemetryPayload
+   This property contains the current Telemetry payload.
+
+The evaluation sandbox for the JavaScript filters may be destroyed
+immediately after ``filter()`` returns. This function should not assume
+async code will finish.
+
+Experiment Applicability and Client Behavior
+============================================
+
+The point of an experiment manifest is to define which experiments are
+available and where and how to run them. This section explains those
+rules in more detail.
+
+Many of the properties in *Experiment Objects* are related to determining
+whether an experiment should run on a given client. This evaluation is
+performed client side.
+
+1. Multiple conditions in an experiment
+---------------------------------------
+
+If multiple conditions are defined for an experiment, the client should
+combine each condition with a logical *AND*: all conditions must be
+satisfied for an experiment to run. If one condition fails, the experiment
+is not applicable.
+
+2. Active experiment disappears from manifest
+---------------------------------------------
+
+If a specific experiment disappears from the manifest, the client should
+continue conducting an already-active experiment. Furthermore, the
+client should remember what the expiration events were for an experiment
+and honor them.
+
+The rationale here is that we want to prevent an accidental deletion
+or temporary failure on the server to inadvertantly deactivate
+supposed-to-be-active experiments. We also don't want premature deletion
+of an experiment from the manifest to result in indefinite activation
+periods.
+
+3. Inactive experiment disappears from manifest
+-----------------------------------------------
+
+If an inactive but scheduled-to-be-active experiment disappears from the
+manifest, the client should not activate the experiment.
+
+If that experiment reappears in the manifest, the client should not
+treat that experiment any differently than any other new experiment. Put
+another way, the fact an inactive experiment disappears and then
+reappears should not be significant.
+
+The rationale here is that server operators should have complete
+control of an inactive experiment up to it's go-live date.
+
+4. Re-evaluating applicability on manifest refresh
+--------------------------------------------------
+
+When an experiment manifest is refreshed or updated, the client should
+re-evaluate the applicability of each experiment therein.
+
+The rationale here is that the server may change the parameters of an
+experiment and want clients to pick those up.
+
+5. Activating a previously non-applicable experiment
+----------------------------------------------------
+
+If the conditions of an experiment change or the state of the client
+changes to allow an experiment to transition from previously
+non-applicable to applicable, the experiment should be activated.
+
+For example, if a client is running version 28 and the experiment
+initially requires version 29 or above, the client will not mark the
+experiment as applicable. But if the client upgrades to version 29 or if
+the manifest is updated to require 28 or above, the experiment will
+become applicable.
+
+6. Deactivating a previously active experiment
+----------------------------------------------
+
+If the conditions of an experiment change or the state of the client
+changes and an active experiment is no longer applicable, that
+experiment should be deactivated.
+
+7. Calculation of sampling-based applicability
+----------------------------------------------
+
+For calculating sampling-based applicability, the client will associate
+a random value between ``0.0`` and ``1.0`` for each observed experiment
+ID. This random value will be generated the first time sampling
+applicability is evaluated. This random value will be persisted and used
+in future applicability evaluations for this experiment.
+
+By saving and re-using the value, the client is able to reliably and
+consistently evaluate applicability, even if the sampling threshold
+in the manifest changes.
+
+Clients should retain the randomly-generated sampling value for
+experiments that no longer appear in a manifest for a period of at least
+30 days. The rationale is that if an experiment disappears and reappears
+from a manifest, the client will not have multiple opportunities to
+generate a random value that satisfies the sampling criteria.
+
+8. Incompatible version numbers
+-------------------------------
+
+If a client receives a manifest with a version number that it doesn't
+recognize, it should ignore the manifest.
+
+9. Usage of old manifests
+-------------------------
+
+If a client experiences an error fetching a manifest (server not
+available) or if the manifest is corrupt, not readable, or compatible,
+the client may use a previously-fetched (cached) manifest.
+
+10. Updating XPIs
+-----------------
+
+If the URL or hash of an active experiment's XPI changes, the client
+should fetch the new XPI, uninstall the old XPI, and install the new
+XPI.
+
+Examples
+========
+
+Here is an example manifest::
+
+   {
+     "version": 1,
+     "experiments": [
+       {
+         "id": "da9d7f4f-f3f9-4f81-bacd-6f0626ffa360",
+         "xpiURL": "https://experiments.mozilla.org/foo.xpi",
+         "xpiHash": "sha1:cb1eb32b89d86d78b7326f416cf404548c5e0099",
+         "startTime": 1393000000,
+         "endTime": 1394000000,
+         "appName": ["Firefox", "Fennec"],
+         "minVersion": "28",
+         "maxVersion": "30",
+         "os": ["windows", "linux", "osx"],
+         "jsfilter": "function filter(context) { return context.healthReportEnabled; }"
+       }
+     ]
+   }