path: root/application/basilisk/experiments/docs/manifest.rst

.. _experiments_manifests:

=====================
Experiments Manifests
=====================

*Experiments Manifests* are documents that describe the set of active
experiments a client may run.

*Experiments Manifests* are fetched periodically by clients. When
fetched, clients look at the experiments within the manifest and
determine which experiments are applicable. If an experiment is
applicable, the client may download and start the experiment.

Manifest Format
===============

Manifests are JSON documents where the main element is an object.

The *schema* of the object is versioned and defined by the presence
of a top-level ``version`` property, whose integer value is the
schema version used by that manifest. Each version is documented
in the sections below.

Version 1
---------

Version 1 is the original manifest format.

The following properties may exist in the root object:

experiments
   An array of objects describing candidate experiments. The format of
   these objects is documented below.

   An array is used to create an explicit priority of experiments.
   Experiments listed at the beginning of the array take priority over
   experiments that follow.

Experiments Objects
^^^^^^^^^^^^^^^^^^^

Each object in the ``experiments`` array may contain the following
properties:

id
   (required) String identifier of this experiment. The identifier should
   be treated as opaque by clients. It is used to uniquely identify an
   experiment for all of time.

xpiURL
   (required) String URL of the XPI that implements this experiment.

   If the experiment is activated, the client will download and install this
   XPI.

xpiHash
   (required) String hash of the XPI that implements this experiment.

   The value is composed of a hash identifier followed by a colon
   followed by the hash value. e.g.
   `sha1:f677428b9172e22e9911039aef03f3736e7f78a7`. `sha1` and `sha256`
   are the two supported hashing mechanisms. The hash value is the hex
   encoding of the binary hash.

   When the client downloads the XPI for the experiment, it should compare
   the hash of that XPI against this value. If the hashes don't match,
   the client should not install the XPI.

   Clients may also use this hash as a means of determining when an
   experiment's XPI has changed and should be refreshed.

startTime
   Integer seconds since UNIX epoch that this experiment should
   start. Clients should not start an experiment if *now()* is less than
   this value.

maxStartTime
   (optional) Integer seconds since UNIX epoch after which this experiment
   should no longer start.

   Some experiments may wish to impose hard deadlines after which no new
   clients should activate the experiment. This property may be used to
   facilitate that.

endTime
   Integer seconds since UNIX epoch after which this experiment
   should no longer run. Clients should cease an experiment when the current
   time is beyond this value.

maxActiveSeconds
   Integer seconds defining the max wall time this experiment should be
   active for.

   The client should deactivate the experiment this many seconds after
   initial activation.

   This value only involves wall time, not browser activity or session time.

appName
   Array of application names this experiment should run on.

   An application name comes from ``nsIXULAppInfo.name``. It is a value
   like ``Firefox``, ``Fennec``, or `B2G`.

   The client should compare its application name against the members of
   this array. If a match is found, the experiment is applicable.

minVersion
   (optional) String version number of the minimum application version this
   experiment should run on.

   A version number is something like ``27.0.0`` or ``28``.

   The client should compare its version number to this value. If the client's
   version is greater or equal to this version (using a version-aware comparison
   function), the experiment is applicable.

   If this is not specified, there is no lower bound to versions this
   experiment should run on.

maxVersion
   (optional) String version number of the maximum application version this
   experiment should run on.

   This is similar to ``minVersion`` except it sets the upper bound for
   application versions.

   If the client's version is less than or equal to this version, the
   experiment is applicable.

   If this is not specified, there is no upper bound to versions this
   experiment should run on.

version
   (optional) Array of application versions this experiment should run on.

   This is similar to ``minVersion`` and ``maxVersion`` except only a
   whitelisted set of specific versions are allowed.

   The client should compare its version to members of this array. If a match
   is found, the experiment is applicable.

minBuildID
   (optional) String minimum Build ID this experiment should run on.

   Build IDs are values like ``201402261424``.

   The client should perform a string comparison of its Build ID against this
   value. If its value is greater than or equal to this value, the experiment
   is applicable.

maxBuildID
   (optional) String maximum Build ID this experiment should run on.

   This is similar to ``minBuildID`` except it sets the upper bound
   for Build IDs.

   The client should perform a string comparison of its Build ID against
   this value. If its value is less than or equal to this value, the
   experiment is applicable.

buildIDs
   (optional) Array of Build IDs this experiment should run on.

   This is similar to ``minBuildID`` and ``maxBuildID`` except only a
   whitelisted set of Build IDs are considered.

   The client should compare its Build ID to members of this array. If a
   match is found, the experiment is applicable.

os
   (optional) Array of operating system identifiers this experiment should
   run on.

   Values for this array come from ``nsIXULRuntime.OS``.

   The client will compare its operating system identifier to members
   of this array. If a match is found, the experiment is applicable to the
   client.

channel
   (optional) Array of release channel identifiers this experiment should run
   on.

   The client will compare its channel to members of this array. If a match
   is found, the experiment is applicable.

   If this property is not defined, the client should assume the experiment
   is to run on all channels.

locale
   (optional) Array of locale identifiers this experiment should run on.

   A locale identifier is a string like ``en-US`` or ``zh-CN`` and is
   obtained by looking at
   ``nsIXULChromeRegistry.getSelectedLocale("global")``.

   The client should compare its locale identifier to members of this array.
   If a match is found, the experiment is applicable.

   If this property is not defined, the client should assume the experiment
   is to run on all locales.

sample
   (optional) Decimal number indicating the sampling rate for this experiment.

   This will contain a value between ``0.0`` and ``1.0``. The client should
   generate a random decimal between ``0.0`` and ``1.0``. If the randomly
   generated number is less than or equal to the value of this field, the
   experiment is applicable.

disabled
   (optional) Boolean value indicating whether an experiment is disabled.

   Normally, experiments are deactivated after a certain time has passed or
   after the experiment itself determines it no longer needs to run (perhaps
   it collected sufficient data already).

   This property serves as a backup mechanism to remotely disable an
   experiment before it was scheduled to be disabled. It can be used to
   kill experiments that are found to be doing wrong or bad things or that
   aren't useful.

   If this property is not defined or is false, the client should assume
   the experiment is active and a candidate for activation.

frozen
   (optional) Boolean value indicating this experiment is frozen and no
   longer accepting new enrollments.

   If a client sees a true value in this field, it should not attempt to
   activate an experiment.

jsfilter
    (optional) JavaScript code that will be evaluated to determine experiment
    applicability.

    This property contains the string representation of JavaScript code that
    will be evaluated in a sandboxed environment using JavaScript's
    ``eval()``.

    The string is expected to contain the definition of a JavaScript function
    ``filter(context)``. This function receives as its argument an object
    holding application state. See the section below for the definition of
    this object.

    The purpose of this property is to allow experiments to define complex
    rules and logic for evaluating experiment applicability in a manner
    that is privacy conscious and doesn't require the transmission of
    excessive data.

    The return value of this filter indicates whether the experiment is
    applicable. Functions should return true if the experiment is
    applicable.

    If an experiment is not applicable, they should throw an Error whose
    message contains the reason the experiment is not applicable. This
    message may be logged and sent to remote servers, so it should not
    contain private or otherwise sensitive data that wouldn't normally
    be submitted.

    If a falsey (or undefined) value is returned, the client should
    assume the experiment is not applicable.

    If this property is not defined, the client does not consider a custom
    JavaScript filter function when determining whether an experiment is
    applicable.

JavaScript Filter Context Objects
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The object passed to a ``jsfilter`` ``filter()`` function contains the
following properties:

healthReportSubmissionEnabled
   This property contains a boolean indicating whether Firefox Health
   Report has its data submission flag enabled (whether Firefox Health
   Report is sending data to remote servers).

healthReportPayload
   This property contains the current Firefox Health Report payload.

   The payload format is documented at :ref:`healthreport_dataformat`.

telemetryPayload
   This property contains the current Telemetry payload.

The evaluation sandbox for the JavaScript filters may be destroyed
immediately after ``filter()`` returns. This function should not assume
async code will finish.

Experiment Applicability and Client Behavior
============================================

The point of an experiment manifest is to define which experiments are
available and where and how to run them. This section explains those
rules in more detail.

Many of the properties in *Experiment Objects* are related to determining
whether an experiment should run on a given client. This evaluation is
performed client side.

1. Multiple conditions in an experiment
---------------------------------------

If multiple conditions are defined for an experiment, the client should
combine each condition with a logical *AND*: all conditions must be
satisfied for an experiment to run. If one condition fails, the experiment
is not applicable.

2. Active experiment disappears from manifest
---------------------------------------------

If a specific experiment disappears from the manifest, the client should
continue conducting an already-active experiment. Furthermore, the
client should remember what the expiration events were for an experiment
and honor them.

The rationale here is that we want to prevent an accidental deletion
or temporary failure on the server to inadvertantly deactivate
supposed-to-be-active experiments. We also don't want premature deletion
of an experiment from the manifest to result in indefinite activation
periods.

3. Inactive experiment disappears from manifest
-----------------------------------------------

If an inactive but scheduled-to-be-active experiment disappears from the
manifest, the client should not activate the experiment.

If that experiment reappears in the manifest, the client should not
treat that experiment any differently than any other new experiment. Put
another way, the fact an inactive experiment disappears and then
reappears should not be significant.

The rationale here is that server operators should have complete
control of an inactive experiment up to it's go-live date.

4. Re-evaluating applicability on manifest refresh
--------------------------------------------------

When an experiment manifest is refreshed or updated, the client should
re-evaluate the applicability of each experiment therein.

The rationale here is that the server may change the parameters of an
experiment and want clients to pick those up.

5. Activating a previously non-applicable experiment
----------------------------------------------------

If the conditions of an experiment change or the state of the client
changes to allow an experiment to transition from previously
non-applicable to applicable, the experiment should be activated.

For example, if a client is running version 28 and the experiment
initially requires version 29 or above, the client will not mark the
experiment as applicable. But if the client upgrades to version 29 or if
the manifest is updated to require 28 or above, the experiment will
become applicable.

6. Deactivating a previously active experiment
----------------------------------------------

If the conditions of an experiment change or the state of the client
changes and an active experiment is no longer applicable, that
experiment should be deactivated.

7. Calculation of sampling-based applicability
----------------------------------------------

For calculating sampling-based applicability, the client will associate
a random value between ``0.0`` and ``1.0`` for each observed experiment
ID. This random value will be generated the first time sampling
applicability is evaluated. This random value will be persisted and used
in future applicability evaluations for this experiment.

By saving and re-using the value, the client is able to reliably and
consistently evaluate applicability, even if the sampling threshold
in the manifest changes.

Clients should retain the randomly-generated sampling value for
experiments that no longer appear in a manifest for a period of at least
30 days. The rationale is that if an experiment disappears and reappears
from a manifest, the client will not have multiple opportunities to
generate a random value that satisfies the sampling criteria.

8. Incompatible version numbers
-------------------------------

If a client receives a manifest with a version number that it doesn't
recognize, it should ignore the manifest.

9. Usage of old manifests
-------------------------

If a client experiences an error fetching a manifest (server not
available) or if the manifest is corrupt, not readable, or compatible,
the client may use a previously-fetched (cached) manifest.

10. Updating XPIs
-----------------

If the URL or hash of an active experiment's XPI changes, the client
should fetch the new XPI, uninstall the old XPI, and install the new
XPI.

Examples
========

Here is an example manifest::

   {
     "version": 1,
     "experiments": [
       {
         "id": "da9d7f4f-f3f9-4f81-bacd-6f0626ffa360",
         "xpiURL": "https://experiments.mozilla.org/foo.xpi",
         "xpiHash": "sha1:cb1eb32b89d86d78b7326f416cf404548c5e0099",
         "startTime": 1393000000,
         "endTime": 1394000000,
         "appName": ["Firefox", "Fennec"],
         "minVersion": "28",
         "maxVersion": "30",
         "os": ["windows", "linux", "osx"],
         "jsfilter": "function filter(context) { return context.healthReportEnabled; }"
       }
     ]
   }