diff options
Diffstat (limited to 'toolkit/components/telemetry/docs')
30 files changed, 4694 insertions, 0 deletions
diff --git a/toolkit/components/telemetry/docs/collection/custom-pings.rst b/toolkit/components/telemetry/docs/collection/custom-pings.rst new file mode 100644 index 000000000..daad87bfe --- /dev/null +++ b/toolkit/components/telemetry/docs/collection/custom-pings.rst @@ -0,0 +1,74 @@ +======================= +Submitting custom pings +======================= + +Custom pings can be submitted from JavaScript using: + +.. code-block:: js + + TelemetryController.submitExternalPing(type, payload, options) + +- ``type`` - a ``string`` that is the type of the ping, limited to ``/^[a-z0-9][a-z0-9-]+[a-z0-9]$/i``. +- ``payload`` - the actual payload data for the ping, has to be a JSON style object. +- ``options`` - optional, an object containing additional options: + - ``addClientId``- whether to add the client id to the ping, defaults to ``false`` + - ``addEnvironment`` - whether to add the environment data to the ping, defaults to ``false`` + - ``overrideEnvironment`` - a JSON style object that overrides the environment data + +``TelemetryController`` will assemble a ping with the passed payload and the specified options. +That ping will be archived locally for use with Shield and inspection in ``about:telemetry``. +If the preferences allow upload of Telemetry pings, the ping will be uploaded at the next opportunity (this is subject to throttling, retry-on-failure, etc.). + +Submission constraints +---------------------- + +When submitting pings on shutdown, they should not be submitted after Telemetry shutdown. +Pings should be submitted at the latest within: + +- the `observer notification <https://developer.mozilla.org/de/docs/Observer_Notifications#Application_shutdown>`_ ``"profile-before-change"`` +- the :ref:`AsyncShutdown phase <AsyncShutdown_phases>` ``sendTelemetry`` + +There are other constraints that can lead to a ping submission getting dropped: + +- invalid ping type strings +- invalid payload types: E.g. strings instead of objects. +- oversized payloads: We currently only drop pings >1MB, but targetting sizes of <=10KB is recommended. + +Tools +===== + +Helpful tools for designing new pings include: + +- `gzipServer <https://github.com/mozilla/gzipServer>`_ - a Python script that can run locally and receives and saves Telemetry pings. Making Firefox send to it allows inspecting outgoing pings easily. +- ``about:telemetry`` - allows inspecting submitted pings from the local archive, including all custom ones. + +Designing custom pings +====================== + +In general, creating a new custom ping means you don't benefit automatically from the existing tooling. Further work is needed to make data show up in re:dash or other analysis tools. + +In addition to the `data collection review <https://wiki.mozilla.org/Firefox/Data_Collection>`_, questions to guide a new pings design are: + +- Submission interval & triggers: + - What events trigger ping submission? + - What interval is the ping submitted in? + - Is there a throttling mechanism? + - What is the desired latency? (submitting "at least daily" still leads to certain latency tails) + - Are pings submitted on a clock schedule? Or based on "time since session start", "time since last ping" etc.? (I.e. will we get sharp spikes in submission volume?) +- Size and volume: + - What’s the size of the submitted payload? + - What's the full ping size including metadata in the pipeline? + - What’s the target population? + - What's the overall estimated volume? +- Dataset: + - Is it opt-out? + - Does it need to be opt-out? + - Does it need to be in a separate ping? (why can’t the data live in probes?) +- Privacy: + - Is there risk to leak PII? + - How is that risk mitigated? +- Data contents: + - Does the submitted data answer the posed product questions? + - Does the shape of the data allow to answer the questions efficiently? + - Is the data limited to whats needed to answer the questions? + - Does the data use common formats? (i.e. can we re-use tooling or analysis know-how) diff --git a/toolkit/components/telemetry/docs/collection/histograms.rst b/toolkit/components/telemetry/docs/collection/histograms.rst new file mode 100644 index 000000000..8d0233dbf --- /dev/null +++ b/toolkit/components/telemetry/docs/collection/histograms.rst @@ -0,0 +1,5 @@ +========== +Histograms +========== + +Recording into histograms is currently documented in `a MDN article <https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Adding_a_new_Telemetry_probe>`_. diff --git a/toolkit/components/telemetry/docs/collection/index.rst b/toolkit/components/telemetry/docs/collection/index.rst new file mode 100644 index 000000000..e4084e62a --- /dev/null +++ b/toolkit/components/telemetry/docs/collection/index.rst @@ -0,0 +1,35 @@ +=============== +Data collection +=============== + +There are different APIs and formats to collect data in Firefox, all suiting different use cases. + +In general, we aim to submit data in a common format where possible. This has several advantages; from common code and tooling to sharing analysis know-how. + +In cases where this isn't possible and more flexibility is needed, we can submit custom pings or consider adding different data formats to existing pings. + +*Note:* Every new data collection must go through a `data collection review <https://wiki.mozilla.org/Firefox/Data_Collection>`_. + +The current data collection possibilities include: + +* :doc:`scalars` allow recording of a single value (string, boolean, a number) +* :doc:`histograms` can efficiently record multiple data points +* ``environment`` data records information about the system and settings a session occurs in +* ``TelemetryLog`` allows collecting ordered event entries (note: this does not have supporting analysis tools) +* :doc:`measuring elapsed time <measuring-time>` +* :doc:`custom pings <custom-pings>` + +.. toctree:: + :maxdepth: 2 + :titlesonly: + :hidden: + :glob: + + scalars + histograms + measuring-time + custom-pings + +Browser Usage Telemetry +~~~~~~~~~~~~~~~~~~~~~~~ +For more information, see :ref:`browserusagetelemetry`. diff --git a/toolkit/components/telemetry/docs/collection/measuring-time.rst b/toolkit/components/telemetry/docs/collection/measuring-time.rst new file mode 100644 index 000000000..918c8a85a --- /dev/null +++ b/toolkit/components/telemetry/docs/collection/measuring-time.rst @@ -0,0 +1,74 @@ +====================== +Measuring elapsed time +====================== + +To make it easier to measure how long operations take, we have helpers for both JavaScript and C++. +These helpers record the elapsed time into histograms, so you have to create suitable histograms for them first. + +From JavaScript +=============== +JavaScript can measure elapsed time using `TelemetryStopwatch.jsm <https://dxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/TelemetryStopwatch.jsm>`_. + +``TelemetryStopwatch`` is a helper that simplifies recording elapsed time (in milliseconds) into histograms (plain or keyed). + +API: + +.. code-block:: js + + TelemetryStopwatch = { + // Start, cancel & finish recording elapsed time into a histogram. + // |aObject| is optional. If specificied, the timer is associated with this + // object, so multiple time measurements can be done concurrently. + start(histogramId, aObject); + cancel(histogramId, aObject); + finish(histogramId, aObject); + // Start, cancel & finished recording elapsed time into a keyed histogram. + // |key| specificies the key to record into. + // |aObject| is optional and used as above. + startKeyed(histogramId, key, aObject); + cancelKeyed(histogramId, key, aObject); + finishKeyed(histogramId, key, aObject); + }; + +Example: + +.. code-block:: js + + TelemetryStopwatch.start("SAMPLE_FILE_LOAD_TIME_MS"); + // ... start loading file. + if (failedToOpenFile) { + // Cancel this if the operation failed early etc. + TelemetryStopwatch.cancel("SAMPLE_FILE_LOAD_TIME_MS"); + return; + } + // ... do more work. + TelemetryStopwatch.finish("SAMPLE_FILE_LOAD_TIME_MS"); + +From C++ +======== + +API: + +.. code-block:: cpp + + // This helper class is the preferred way to record elapsed time. + template<ID id, TimerResolution res = MilliSecond> + class AutoTimer { + // Record into a plain histogram. + explicit AutoTimer(TimeStamp aStart = TimeStamp::Now()); + // Record into a keyed histogram, with key |aKey|. + explicit AutoTimer(const nsCString& aKey, + TimeStamp aStart = TimeStamp::Now()); + }; + + void AccumulateTimeDelta(ID id, TimeStamp start, TimeStamp end = TimeStamp::Now()); + +Example: + +.. code-block:: cpp + + { + Telemetry::AutoTimer<Telemetry::FIND_PLUGINS> telemetry; + // ... scan disk for plugins. + } + // When leaving the scope, AutoTimers destructor will record the time that passed. diff --git a/toolkit/components/telemetry/docs/collection/scalars.rst b/toolkit/components/telemetry/docs/collection/scalars.rst new file mode 100644 index 000000000..2c48601a4 --- /dev/null +++ b/toolkit/components/telemetry/docs/collection/scalars.rst @@ -0,0 +1,140 @@ +======= +Scalars +======= + +Historically we started to overload our histogram mechanism to also collect scalar data, +such as flag values, counts, labels and others. +The scalar measurement types are the suggested way to collect that kind of scalar data. +We currently only support recording of scalars from the parent process. +The serialized scalar data is submitted with the :doc:`main pings <../data/main-ping>`. + +The API +======= +Scalar probes can be managed either through the `nsITelemetry interface <https://dxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/nsITelemetry.idl>`_ +or the `C++ API <https://dxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/Telemetry.h>`_. + +JS API +------ +Probes in privileged JavaScript code can use the following functions to manipulate scalars: + +.. code-block:: js + + Services.telemetry.scalarAdd(aName, aValue); + Services.telemetry.scalarSet(aName, aValue); + Services.telemetry.scalarSetMaximum(aName, aValue); + + Services.telemetry.keyedScalarAdd(aName, aKey, aValue); + Services.telemetry.keyedScalarSet(aName, aKey, aValue); + Services.telemetry.keyedScalarSetMaximum(aName, aKey, aValue); + +These functions can throw if, for example, an operation is performed on a scalar type that doesn't support it +(e.g. calling scalarSetMaximum on a scalar of the string kind). Please look at the `code documentation <https://dxr.mozilla.org/mozilla-central/search?q=regexp%3ATelemetryScalar%3A%3A(Set%7CAdd)+file%3ATelemetryScalar.cpp&redirect=false>`_ for +additional information. + +C++ API +------- +Probes in native code can use the more convenient helper functions declared in `Telemetry.h <https://dxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/Telemetry.h>`_: + +.. code-block:: cpp + + void ScalarAdd(mozilla::Telemetry::ScalarID aId, uint32_t aValue); + void ScalarSet(mozilla::Telemetry::ScalarID aId, uint32_t aValue); + void ScalarSet(mozilla::Telemetry::ScalarID aId, const nsAString& aValue); + void ScalarSet(mozilla::Telemetry::ScalarID aId, bool aValue); + void ScalarSetMaximum(mozilla::Telemetry::ScalarID aId, uint32_t aValue); + + void ScalarAdd(mozilla::Telemetry::ScalarID aId, const nsAString& aKey, uint32_t aValue); + void ScalarSet(mozilla::Telemetry::ScalarID aId, const nsAString& aKey, uint32_t aValue); + void ScalarSet(mozilla::Telemetry::ScalarID aId, const nsAString& aKey, bool aValue); + void ScalarSetMaximum(mozilla::Telemetry::ScalarID aId, const nsAString& aKey, uint32_t aValue); + +The YAML definition file +======================== +Scalar probes are required to be registered, both for validation and transparency reasons, +in the `Scalars.yaml <https://dxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/Scalars.yaml>`_ +definition file. + +The probes in the definition file are represented in a fixed-depth, two-level structure: + +.. code-block:: yaml + + # The following is a group. + a.group.hierarchy: + a_probe_name: + kind: uint + ... + another_probe: + kind: string + ... + ... + group2: + probe: + kind: int + ... + +Group and probe names need to follow a few rules: + +- they cannot exceed 40 characters each; +- group names must be alpha-numeric + ``.``, with no leading/trailing digit or ``.``; +- probe names must be alpha-numeric + ``_``, with no leading/trailing digit or ``_``. + +A probe can be defined as follows: + +.. code-block:: yaml + + a.group.hierarchy: + a_scalar: + bug_numbers: + - 1276190 + description: A nice one-line description. + expires: never + kind: uint + notification_emails: + - telemetry-client-dev@mozilla.com + +Required Fields +--------------- + +- ``bug_numbers``: A list of unsigned integers representing the number of the bugs the probe was introduced in. +- ``description``: A single or multi-line string describing what data the probe collects and when it gets collected. +- ``expires``: The version number in which the scalar expires, e.g. "30"; a version number of type "N" and "N.0" is automatically converted to "N.0a1" in order to expire the scalar also in the development channels. A telemetry probe acting on an expired scalar will print a warning into the browser console. For scalars that never expire the value ``never`` can be used. +- ``kind``: A string representing the scalar type. Allowed values are ``uint``, ``string`` and ``boolean``. +- ``notification_emails``: A list of email addresses to notify with alerts of expiring probes. More importantly, these are used by the data steward to verify that the probe is still useful. + +Optional Fields +--------------- + +- ``cpp_guard``: A string that gets inserted as an ``#ifdef`` directive around the automatically generated C++ declaration. This is typically used for platform-specific scalars, e.g. ``ANDROID``. +- ``release_channel_collection``: This can be either ``opt-in`` (default) or ``opt-out``. With the former the scalar is submitted by default on pre-release channels; on the release channel only if the user opted into additional data collection. With the latter the scalar is submitted by default on release and pre-release channels, unless the user opted out. +- ``keyed``: A boolean that determines whether this is a keyed scalar. It defaults to ``False``. + +String type restrictions +------------------------ +To prevent abuses, the content of a string scalar is limited to 50 characters in length. Trying +to set a longer string will result in an error and no string being set. + +Keyed Scalars +------------- +Keyed scalars are collections of one of the available scalar types, indexed by a string key that can contain UTF8 characters and cannot be longer than 70 characters. Keyed scalars can contain up to 100 keys. This scalar type is for example useful when you want to break down certain counts by a name, like how often searches happen with which search engine. + +Keyed scalars should only be used if the set of keys are not known beforehand. If the keys are from a known set of strings, other options are preferred if suitable, like categorical histograms or splitting measurements up into separate scalars. + +The processor scripts +===================== +The scalar definition file is processed and checked for correctness at compile time. If it +conforms to the specification, the processor scripts generate two C++ headers files, included +by the Telemetry C++ core. + +gen-scalar-data.py +------------------ +This script is called by the build system to generate the ``TelemetryScalarData.h`` C++ header +file out of the scalar definitions. +This header file contains an array holding the scalar names and version strings, in addition +to an array of ``ScalarInfo`` structures representing all the scalars. + +gen-scalar-enum.py +------------------ +This script is called by the build system to generate the ``TelemetryScalarEnums.h`` C++ header +file out of the scalar definitions. +This header file contains an enum class with all the scalar identifiers used to access them +from code through the C++ API. diff --git a/toolkit/components/telemetry/docs/concepts/archiving.rst b/toolkit/components/telemetry/docs/concepts/archiving.rst new file mode 100644 index 000000000..a2c57de43 --- /dev/null +++ b/toolkit/components/telemetry/docs/concepts/archiving.rst @@ -0,0 +1,12 @@ +========= +Archiving +========= + +When archiving is enabled through the relevant pref (``toolkit.telemetry.archive.enabled``), pings submitted to ``TelemetryController`` are also stored locally in the user profile directory, in ``<profile-dir>/datareporting/archived``. + +To allow for cheaper lookup of archived pings, storage follows a specific naming scheme for both the directory and the ping file name: `<YYYY-MM>/<timestamp>.<UUID>.<type>.jsonlz4`. + +* ``<YYYY-MM>`` - The subdirectory name, generated from the ping creation date. +* ``<timestamp>`` - Timestamp of the ping creation date. +* ``<UUID>`` - The ping identifier. +* ``<type>`` - The ping type. diff --git a/toolkit/components/telemetry/docs/concepts/crashes.rst b/toolkit/components/telemetry/docs/concepts/crashes.rst new file mode 100644 index 000000000..c9f69a23b --- /dev/null +++ b/toolkit/components/telemetry/docs/concepts/crashes.rst @@ -0,0 +1,23 @@ +======= +Crashes +======= + +There are many different kinds of crashes for Firefox, there is not a single system used to record all of them. + +Main process crashes +==================== + +If the Firefox main process dies, that should be recorded as an aborted session. We would submit a :doc:`main ping <../data/main-ping>` with the reason ``aborted-session``. +If we have a crash dump for that crash, we should also submit a :doc:`crash ping <../data/crash-ping>`. + +The ``aborted-session`` information is first written to disk 60 seconds after startup, any earlier crashes will not trigger an ``aborted-session`` ping. +Also, the ``aborted-session`` is updated at least every 5 minutes, so it may lag behind the last session state. + +Crashes during startup should be recorded in the next sessions main ping in the ``STARTUP_CRASH_DETECTED`` histogram. + +Child process crashes +===================== + +If a Firefox plugin, content or gmplugin process dies unexpectedly, this is recorded in the main pings ``SUBPROCESS_ABNORMAL_ABORT`` keyed histogram. + +If we catch a crash report for this, then additionally the ``SUBPROCESS_CRASHES_WITH_DUMP`` keyed histogram is incremented. diff --git a/toolkit/components/telemetry/docs/concepts/index.rst b/toolkit/components/telemetry/docs/concepts/index.rst new file mode 100644 index 000000000..a49466f8d --- /dev/null +++ b/toolkit/components/telemetry/docs/concepts/index.rst @@ -0,0 +1,23 @@ +======== +Concepts +======== + +There are common concepts used throughout Telemetry: + +* :doc:`pings <pings>` - the packets we use to submit data +* :doc:`sessions & subsessions <sessions>` - how we slice a users' time in the browser +* *measurements* - how we :doc:`collect data <../collection/index>` +* *opt-in* & *opt-out* - the different sets of data we collect +* :doc:`submission <submission>` - how we send data to the servers +* :doc:`archiving <archiving>` - retaining ping data locally +* :doc:`crashes <crashes>` - the different data crashes generate + +.. toctree:: + :maxdepth: 2 + :titlesonly: + :glob: + :hidden: + + pings + crashes + * diff --git a/toolkit/components/telemetry/docs/concepts/pings.rst b/toolkit/components/telemetry/docs/concepts/pings.rst new file mode 100644 index 000000000..db7371b32 --- /dev/null +++ b/toolkit/components/telemetry/docs/concepts/pings.rst @@ -0,0 +1,32 @@ +.. _telemetry_pings: + +===================== +Telemetry pings +===================== + +A *Telemetry ping* is the data that we send to Mozillas Telemetry servers. + +That data is stored as a JSON object client-side and contains common information to all pings and a payload specific to a certain *ping types*. + +The top-level structure is defined by the :doc:`common ping format <../data/common-ping>` format. +It contains: + +* some basic information shared between different ping types +* the :doc:`environment data <../data/environment>` (optional) +* the data specific to the *ping type*, the *payload*. + +Ping types +========== + +We send Telemetry with different ping types. The :doc:`main <../data/main-ping>` ping is the ping that contains the bulk of the Telemetry measurements for Firefox. For more specific use-cases, we send other ping types. + +Pings sent from code that ships with Firefox are listed in the :doc:`data documentation <../data/index>`. + +Important examples are: + +* :doc:`main <../data/main-ping>` - contains the information collected by Telemetry (Histograms, hang stacks, ...) +* :doc:`saved-session <../data/main-ping>` - has the same format as a main ping, but it contains the *"classic"* Telemetry payload with measurements covering the whole browser session. This is only a separate type to make storage of saved-session easier server-side. This is temporary and will be removed soon. +* :doc:`crash <../data/crash-ping>` - a ping that is captured and sent after Firefox crashes. +* ``activation`` - *planned* - sent right after installation or profile creation +* ``upgrade`` - *planned* - sent right after an upgrade +* :doc:`deletion <../data/deletion-ping>` - sent when FHR upload is disabled, requesting deletion of the data associated with this user diff --git a/toolkit/components/telemetry/docs/concepts/sessions.rst b/toolkit/components/telemetry/docs/concepts/sessions.rst new file mode 100644 index 000000000..088556978 --- /dev/null +++ b/toolkit/components/telemetry/docs/concepts/sessions.rst @@ -0,0 +1,40 @@ +======== +Sessions +======== + +A *session* is the time from when Firefox starts until it shut down. +A session can be very long-running. E.g. for Mac users that are used to always put their laptops into sleep-mode, Firefox may run for weeks. +We slice the sessions into smaller logical units called *subsessions*. + +Subsessions +=========== + +The first subsession starts when the browser starts. After that, we split the subsession for different reasons: + +* ``daily``, when crossing local midnight. This keeps latency acceptable by triggering a ping at least daily for most active users. +* ``environment-change``, when a change to the *environment* happens. This happens for important changes to the Firefox settings and when addons activate or deactivate. + +On a subsession split, a :doc:`main ping <../data/main-ping>` with that reason will be submitted. We store the reason in the pings payload, to see what triggered it. + +A session always ends with a subsession with one of two reason: + +* ``shutdown``, when the browser was cleanly shut down. To avoid delaying shutdown, we only save this ping to disk and send it at the next opportunity (typically the next browsing session). +* ``aborted-session``, when the browser crashed. While Firefox is active, we write the current ``main`` ping data to disk every 5 minutes. If the browser crashes, we find this data on disk on the next start and send it with this reason. + +.. image:: subsession_triggers.png + +Subsession data +=============== + +A subsessions data consists of: + +* general information: the date the subsession started, how long it lasted, etc. +* specific measurements: histogram & scalar data, etc. + +This has some advantages: + +* Latency - Sending a ping with all the data of a subsession immediately after it ends means we get the data from installs faster. For ``main`` pings, we aim to send a ping at least daily by starting a new subsession at local midnight. +* Correlation - By starting new subsessions when fundamental settings change (i.e. changes to the *environment*), we can correlate a subsessions data better to those settings. + + + diff --git a/toolkit/components/telemetry/docs/concepts/submission.rst b/toolkit/components/telemetry/docs/concepts/submission.rst new file mode 100644 index 000000000..165917d40 --- /dev/null +++ b/toolkit/components/telemetry/docs/concepts/submission.rst @@ -0,0 +1,34 @@ +========== +Submission +========== + +*Note:* The server-side behaviour is documented in the `HTTP Edge Server specification <https://wiki.mozilla.org/CloudServices/DataPipeline/HTTPEdgeServerSpecification>`_. + +Pings are submitted via a common API on ``TelemetryController``. +If a ping fails to successfully submit to the server immediately (e.g. because +of missing internet connection), Telemetry will store it on disk and retry to +send it until the maximum ping age is exceeded (14 days). + +*Note:* the :doc:`main pings <../data/main-ping>` are kept locally even after successful submission to enable the HealthReport and SelfSupport features. They will be deleted after their retention period of 180 days. + +Submission logic +================ + +Sending of pending pings starts as soon as the delayed startup is finished. They are sent in batches, newest-first, with up +to 10 persisted pings per batch plus all unpersisted pings. +The send logic then waits for each batch to complete. + +If it succeeds we trigger the next send of a ping batch. This is delayed as needed to only trigger one batch send per minute. + +If ping sending encounters an error that means retrying later, a backoff timeout behavior is +triggered, exponentially increasing the timeout for the next try from 1 minute up to a limit of 120 minutes. +Any new ping submissions and "idle-daily" events reset this behavior as a safety mechanism and trigger immediate ping sending. + +Status codes +============ + +The telemetry server team is working towards `the common services status codes <https://wiki.mozilla.org/CloudServices/DataPipeline/HTTPEdgeServerSpecification#Server_Responses>`_, but for now the following logic is sufficient for Telemetry: + +* `2XX` - success, don't resubmit +* `4XX` - there was some problem with the request - the client should not try to resubmit as it would just receive the same response +* `5XX` - there was a server-side error, the client should try to resubmit later diff --git a/toolkit/components/telemetry/docs/concepts/subsession_triggers.png b/toolkit/components/telemetry/docs/concepts/subsession_triggers.png Binary files differnew file mode 100644 index 000000000..5717b00a9 --- /dev/null +++ b/toolkit/components/telemetry/docs/concepts/subsession_triggers.png diff --git a/toolkit/components/telemetry/docs/data/addons-malware-ping.rst b/toolkit/components/telemetry/docs/data/addons-malware-ping.rst new file mode 100644 index 000000000..18502d748 --- /dev/null +++ b/toolkit/components/telemetry/docs/data/addons-malware-ping.rst @@ -0,0 +1,42 @@ + +Add-ons malware ping +==================== + +This ping is generated by an add-on created by Mozilla and shipped to users on older versions of Firefox (44-46). The ping contains information about the profile that might have been altered by a third party malicious add-on. + +Structure: + +.. code-block:: js + + { + type: "malware-addon-states", + ... + clientId: <UUID>, + environment: { ... }, + // Common ping data. + payload: { + // True if the blocklist was disabled at startup time. + blocklistDisabled: <bool>, + // True if the malicious add-on exists and is enabled. False if it + // exists and is disabled or null if the add-on was not found. + mainAddonActive: <bool | null>, + // A value of the malicious add-on block list state, or null if the + // add-on was not found. + mainAddonBlocked: <int | null>, + // True if a malicious user.js file was found in the profile. + foundUserJS: <bool>, + // If a malicious secmodd.db file was found the extension ID that the // file contained.. + secmoddAddon: <string | null>, . + // A list of IDs for extensions which were hidden by malicious CSS. + hiddenAddons: [ + <string>, + ... + ], + // A mapping of installed add-on IDs with known malicious + // update URL patterns to their exact update URLs. + updateURLs: { + <extensionID>: <updateURL>, + ... + } + } + } diff --git a/toolkit/components/telemetry/docs/data/common-ping.rst b/toolkit/components/telemetry/docs/data/common-ping.rst new file mode 100644 index 000000000..445557efd --- /dev/null +++ b/toolkit/components/telemetry/docs/data/common-ping.rst @@ -0,0 +1,42 @@ + +Common ping format +================== + +This defines the top-level structure of a Telemetry ping. +It contains basic information shared between different ping types, which enables proper storage and processing of the raw pings server-side. + +It also contains optional further information: + +* the :doc:`environment data <../data/environment>`, which contains important info to correlate the measurements against +* the ``clientId``, a UUID identifying a profile and allowing user-oriented correlation of data + +*Note:* Both are not submitted with all ping types due to privacy concerns. This and the data it that can be correlated against is inspected under the `data collection policy <https://wiki.mozilla.org/Firefox/Data_Collection>`_. + +Finally, the structure also contains the `payload`, which is the specific data submitted for the respective *ping type*. + +Structure: + +.. code-block:: js + + { + type: <string>, // "main", "activation", "deletion", "saved-session", ... + id: <UUID>, // a UUID that identifies this ping + creationDate: <ISO date>, // the date the ping was generated + version: <number>, // the version of the ping format, currently 4 + + application: { + architecture: <string>, // build architecture, e.g. x86 + buildId: <string>, // "20141126041045" + name: <string>, // "Firefox" + version: <string>, // "35.0" + displayVersion: <string>, // "35.0b3" + vendor: <string>, // "Mozilla" + platformVersion: <string>, // "35.0" + xpcomAbi: <string>, // e.g. "x86-msvc" + channel: <string>, // "beta" + }, + + clientId: <UUID>, // optional + environment: { ... }, // optional, not all pings contain the environment + payload: { ... }, // the actual payload data for this ping type + } diff --git a/toolkit/components/telemetry/docs/data/core-ping.rst b/toolkit/components/telemetry/docs/data/core-ping.rst new file mode 100644 index 000000000..7f38f2f7e --- /dev/null +++ b/toolkit/components/telemetry/docs/data/core-ping.rst @@ -0,0 +1,191 @@ + +"core" ping +============ + +This mobile-specific ping is intended to provide the most critical +data in a concise format, allowing for frequent uploads. + +Since this ping is used to measure retention, it should be sent +each time the browser is opened. + +Submission will be per the Edge server specification:: + + /submit/telemetry/docId/docType/appName/appVersion/appUpdateChannel/appBuildID + +* ``docId`` is a UUID for deduping +* ``docType`` is “core” +* ``appName`` is “Fennec” +* ``appVersion`` is the version of the application (e.g. "46.0a1") +* ``appUpdateChannel`` is “release”, “beta”, etc. +* ``appBuildID`` is the build number + +Note: Counts below (e.g. search & usage times) are “since the last +ping”, not total for the whole application lifetime. + +Structure: + +.. code-block:: js + + { + "v": 7, // ping format version + "clientId": <string>, // client id, e.g. + // "c641eacf-c30c-4171-b403-f077724e848a" + "seq": <positive integer>, // running ping counter, e.g. 3 + "locale": <string>, // application locale, e.g. "en-US" + "os": <string>, // OS name. + "osversion": <string>, // OS version. + "device": <string>, // Build.MANUFACTURER + " - " + Build.MODEL + // where manufacturer is truncated to 12 characters + // & model is truncated to 19 characters + "arch": <string>, // e.g. "arm", "x86" + "profileDate": <pos integer>, // Profile creation date in days since + // UNIX epoch. + "defaultSearch": <string>, // Identifier of the default search engine, + // e.g. "yahoo". + "distributionId": <string>, // Distribution identifier (optional) + "created": <string>, // date the ping was created + // in local time, "yyyy-mm-dd" + "tz": <integer>, // timezone offset (in minutes) of the + // device when the ping was created + "sessions": <integer>, // number of sessions since last upload + "durations": <integer>, // combined duration, in seconds, of all + // sessions since last upload + "searches": <object>, // Optional, object of search use counts in the + // format: { "engine.source": <pos integer> } + // e.g.: { "yahoo.suggestion": 3, "other.listitem": 1 } + "experiments": [<string>, /* … */], // Optional, array of identifiers + // for the active experiments + } + +Field details +------------- + +device +~~~~~~ +The ``device`` field is filled in with information specified by the hardware +manufacturer. As such, it could be excessively long and use excessive amounts +of limited user data. To avoid this, we limit the length of the field. We're +more likely have collisions for models within a manufacturer (e.g. "Galaxy S5" +vs. "Galaxy Note") than we are for shortened manufacturer names so we provide +more characters for the model than the manufacturer. + +distributionId +~~~~~~~~~~~~~~ +The ``distributionId`` contains the distribution ID as specified by +preferences.json for a given distribution. More information on distributions +can be found `here <https://wiki.mozilla.org/Mobile/Distribution_Files>`_. + +It is optional. + +defaultSearch +~~~~~~~~~~~~~ +On Android, this field may be ``null``. To get the engine, we rely on +``SearchEngineManager#getDefaultEngine``, which searches in several places in +order to find the search engine identifier: + +* Shared Preferences +* The distribution (if it exists) +* The localized default engine + +If the identifier could not be retrieved, this field is ``null``. If the +identifier is retrieved, we attempt to create an instance of the search +engine from the search plugins (in order): + +* In the distribution +* From the localized plugins shipped with the browser +* The third-party plugins that are installed in the profile directory + +If the plugins fail to create a search engine instance, this field is also +``null``. + +This field can also be ``null`` when a custom search engine is set as the +default. + +sessions & durations +~~~~~~~~~~~~~~~~~~~~ +On Android, a session is the time when Firefox is focused in the foreground. +`sessions` tracks the number of sessions since the last upload and +`durations` is the accumulated duration in seconds of all of these +sessions. Note that showing a dialog (including a Firefox dialog) will +take Firefox out of focus & end the current session. + +An implementation that records a session when Firefox is completely hidden is +preferrable (e.g. to avoid the dialog issue above), however, it's more complex +to implement and so we chose not to, at least for the initial implementation. + +profileDate +~~~~~~~~~~~ +On Android, this value is created at profile creation time and retrieved or, +for legacy profiles, taken from the package install time (note: this is not the +same exact metric as profile creation time but we compromised in favor of ease +of implementation). + +Additionally on Android, this field may be ``null`` in the unlikely event that +all of the following events occur: + +#. The times.json file does not exist +#. The package install date could not be persisted to disk + +The reason we don't just return the package install time even if the date could +not be persisted to disk is to ensure the value doesn't change once we start +sending it: we only want to send consistent values. + +searches +~~~~~~~~ +In the case a search engine is added by a user, the engine identifier "other" is used, e.g. "other.<source>". + +Sources in Android are based on the existing UI telemetry values and are as +follows: + +* actionbar: the user types in the url bar and hits enter to use the default + search engine +* listitem: the user selects a search engine from the list of secondary search + engines at the bottom of the screen +* suggestion: the user clicks on a search suggestion or, in the case that + suggestions are disabled, the row corresponding with the main engine + +Other parameters +---------------- + +HTTP "Date" header +~~~~~~~~~~~~~~~~~~ +This header is used to track the submission date of the core ping in the format +specified by +`rfc 2616 sec 14.18 <https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.18>`_, +et al (e.g. "Tue, 01 Feb 2011 14:00:00 GMT"). + + +Version history +--------------- +* v7: added ``sessionCount`` & ``sessionDuration`` +* v6: added ``searches`` +* v5: added ``created`` & ``tz`` +* v4: ``profileDate`` will return package install time when times.json is not available +* v3: added ``defaultSearch`` +* v2: added ``distributionId`` +* v1: initial version + +Notes +~~~~~ + +* ``distributionId`` (v2) actually landed after ``profileDate`` (v4) but was + uplifted to 46, whereas ``profileDate`` landed on 47. The version numbers in + code were updated to be increasing (bug 1264492) and the version history docs + rearranged accordingly. + +Android implementation notes +---------------------------- +On Android, the uploader has a high probability of delivering the complete data +for a given client but not a 100% probability. This was a conscious decision to +keep the code simple. The cases where we can lose data: + +* Resetting the field measurements (including incrementing the sequence number) + and storing a ping for upload are not atomic. Android can kill our process + for memory pressure in between these distinct operations so we can just lose + a ping's worth of data. That sequence number will be missing on the server. +* If we exceed some number of pings on disk that have not yet been uploaded, + we remove old pings to save storage space. For those pings, we will lose + their data and their sequence numbers will be missing on the server. + +Note: we never expect to drop data without also dropping a sequence number so +we are able to determine when data loss occurs. diff --git a/toolkit/components/telemetry/docs/data/crash-ping.rst b/toolkit/components/telemetry/docs/data/crash-ping.rst new file mode 100644 index 000000000..3cdbc6030 --- /dev/null +++ b/toolkit/components/telemetry/docs/data/crash-ping.rst @@ -0,0 +1,144 @@ + +"crash" ping +============ + +This ping is captured after the main Firefox process crashes, whether or not the crash report is submitted to crash-stats.mozilla.org. It includes non-identifying metadata about the crash. + +The environment block that is sent with this ping varies: if Firefox was running long enough to record the environment block before the crash, then the environment at the time of the crash will be recorded and ``hasCrashEnvironment`` will be true. If Firefox crashed before the environment was recorded, ``hasCrashEnvironment`` will be false and the recorded environment will be the environment at time of submission. + +The client ID is submitted with this ping. + +Structure: + +.. code-block:: js + + { + version: 1, + type: "crash", + ... common ping data + clientId: <UUID>, + environment: { ... }, + payload: { + crashDate: "YYYY-MM-DD", + sessionId: <UUID>, // may be missing for crashes that happen early + // in startup. Added in Firefox 48 with the + // intention of uplifting to Firefox 46 + crashId: <UUID>, // Optional, ID of the associated crash + stackTraces: { ... }, // Optional, see below + metadata: { // Annotations saved while Firefox was running. See nsExceptionHandler.cpp for more information + ProductName: "Firefox", + ReleaseChannel: <channel>, + Version: <version number>, + BuildID: "YYYYMMDDHHMMSS", + AvailablePageFile: <size>, // Windows-only, available paging file + AvailablePhysicalMemory: <size>, // Windows-only, available physical memory + AvailableVirtualMemory: <size>, // Windows-only, available virtual memory + BlockedDllList: <list>, // Windows-only, see WindowsDllBlocklist.cpp for details + BlocklistInitFailed: 1, // Windows-only, present only if the DLL blocklist initialization failed + CrashTime: <time>, // Seconds since the Epoch + ContainsMemoryReport: 1, // Optional + EventLoopNestingLevel: <levels>, // Optional, present only if >0 + IsGarbageCollecting: 1, // Optional, present only if set to 1 + MozCrashReason: <reason>, // Optional, contains the string passed to MOZ_CRASH() + OOMAllocationSize: <size>, // Size of the allocation that caused an OOM + SecondsSinceLastCrash: <duration>, // Seconds elapsed since the last crash occurred + SystemMemoryUsePercentage: <percentage>, // Windows-only, percent of memory in use + TelemetrySessionId: <id>, // Active telemetry session ID when the crash was recorded + TextureUsage: <usage>, // Optional, usage of texture memory in bytes + TotalPageFile: <size>, // Windows-only, paging file in use + TotalPhysicalMemory: <size>, // Windows-only, physical memory in use + TotalVirtualMemory: <size>, // Windows-only, virtual memory in use + UptimeTS: <duration>, // Seconds since Firefox was started + User32BeforeBlocklist: 1, // Windows-only, present only if user32.dll was loaded before the DLL blocklist has been initialized + }, + hasCrashEnvironment: bool + } + } + +Stack Traces +------------ + +The crash ping may contain a ``stackTraces`` field which has been populated +with stack traces for all threads in the crashed process. The format of this +field is similar to the one used by Socorro for representing a crash. The main +differences are that redundant fields are not stored and that the module a +frame belongs to is referenced by index in the module array rather than by its +file name. + +Note that this field does not contain data from the application; only bare +stack traces and module lists are stored. + +.. code-block:: js + + { + status: <string>, // Status of the analysis, "OK" or an error message + crash_info: { // Basic crash information + type: <string>, // Type of crash, SIGSEGV, assertion, etc... + address: <addr>, // Crash address crash, hex format, see the notes below + crashing_thread: <index> // Index in the thread array below + }, + main_module: <index>, // Index of Firefox' executable in the module list + modules: [{ + base_addr: <addr>, // Base address of the module, hex format + end_addr: <addr>, // End address of the module, hex format + code_id: <string>, // Unique ID of this module, see the notes below + debug_file: <string>, // Name of the file holding the debug information + debug_id: <string>, // ID or hash of the debug information file + filename: <string>, // File name + version: <string>, // Library/executable version + }, + ... // List of modules ordered by base memory address + ], + threads: [{ // Stack traces for every thread + frames: [{ + module_index: <index>, // Index of the module this frame belongs to + ip: <ip>, // Program counter, hex format + trust: <string> // Trust of this frame, see the notes below + }, + ... // List of frames, the first frame is the topmost + ] + }] + } + +Notes +~~~~~ + +Memory addresses and instruction pointers are always stored as strings in +hexadecimal format (e.g. "0x4000"). They can be made of up to 16 characters for +64-bit addresses. + +The crash type is both OS and CPU dependent and can be either a descriptive +string (e.g. SIGSEGV, EXCEPTION_ACCESS_VIOLATION) or a raw numeric value. The +crash address meaning depends on the type of crash. In a segmentation fault the +crash address will be the memory address whose access caused the fault; in a +crash triggered by an illegal instruction exception the address will be the +instruction pointer where the invalid instruction resides. +See `breakpad <https://chromium.googlesource.com/breakpad/breakpad/+/c99d374dde62654a024840accfb357b2851daea0/src/processor/minidump_processor.cc#675>`_'s +relevant code for further information. + +Since it's not always possible to establish with certainty the address of the +previous frame while walking the stack, every frame has a trust value that +represents how it was found and thus how certain we are that it's a real frame. +The trust levels are (from least trusted to most trusted): + ++---------------+---------------------------------------------------+ +| Trust | Description | ++===============+===================================================+ +| context | Given as instruction pointer in a context | ++---------------+---------------------------------------------------+ +| prewalked | Explicitly provided by some external stack walker | ++---------------+---------------------------------------------------+ +| cfi | Derived from call frame info | ++---------------+---------------------------------------------------+ +| frame_pointer | Derived from frame pointer | ++---------------+---------------------------------------------------+ +| cfi_scan | Found while scanning stack using call frame info | ++---------------+---------------------------------------------------+ +| scan | Scanned the stack, found this | ++---------------+---------------------------------------------------+ +| none | Unknown, this is most likely not a valid frame | ++---------------+---------------------------------------------------+ + +The ``code_id`` field holds a unique ID used to distinguish between different +versions and builds of the same module. See `breakpad <https://chromium.googlesource.com/breakpad/breakpad/+/24f5931c5e0120982c0cbf1896641e3ef2bdd52f/src/google_breakpad/processor/code_module.h#60>`_'s +description for further information. This field is populated only on Windows. diff --git a/toolkit/components/telemetry/docs/data/deletion-ping.rst b/toolkit/components/telemetry/docs/data/deletion-ping.rst new file mode 100644 index 000000000..c4523ce54 --- /dev/null +++ b/toolkit/components/telemetry/docs/data/deletion-ping.rst @@ -0,0 +1,19 @@ + +"deletion" ping +=============== + +This ping is generated when a user turns off FHR upload from the Preferences panel, changing the related ``datareporting.healthreport.uploadEnabled`` preference. This requests that all associated data from that user be deleted. + +This ping contains the client id and no environment data. + +Structure: + +.. code-block:: js + + { + version: 4, + type: "deletion", + ... common ping data + clientId: <UUID>, + payload: { } + }
\ No newline at end of file diff --git a/toolkit/components/telemetry/docs/data/environment.rst b/toolkit/components/telemetry/docs/data/environment.rst new file mode 100644 index 000000000..ff0d204a4 --- /dev/null +++ b/toolkit/components/telemetry/docs/data/environment.rst @@ -0,0 +1,373 @@ + +Environment +=========== + +The environment consists of data that is expected to be characteristic for performance and other behavior and not expected to change too often. + +Changes to most of these data points are detected (where possible and sensible) and will lead to a session split in the :doc:`main-ping`. +The environment data may also be submitted by other ping types. + +*Note:* This is not submitted with all ping types due to privacy concerns. This and other data is inspected under the `data collection policy <https://wiki.mozilla.org/Firefox/Data_Collection>`_. + +Some parts of the environment must be fetched asynchronously at startup. We don't want other Telemetry components to block on waiting for the environment, so some items may be missing from it until the async fetching finished. +This currently affects the following sections: + +- profile +- addons + + +Structure: + +.. code-block:: js + + { + build: { + applicationId: <string>, // nsIXULAppInfo.ID + applicationName: <string>, // "Firefox" + architecture: <string>, // e.g. "x86", build architecture for the active build + architecturesInBinary: <string>, // e.g. "i386-x86_64", from nsIMacUtils.architecturesInBinary, only present for mac universal builds + buildId: <string>, // e.g. "20141126041045" + version: <string>, // e.g. "35.0" + vendor: <string>, // e.g. "Mozilla" + platformVersion: <string>, // e.g. "35.0" + xpcomAbi: <string>, // e.g. "x86-msvc" + hotfixVersion: <string>, // e.g. "20141211.01" + }, + settings: { + addonCompatibilityCheckEnabled: <bool>, // Whether application compatibility is respected for add-ons + blocklistEnabled: <bool>, // true on failure + isDefaultBrowser: <bool>, // null on failure, not available on Android + defaultSearchEngine: <string>, // e.g. "yahoo" + defaultSearchEngineData: {, // data about the current default engine + name: <string>, // engine name, e.g. "Yahoo"; or "NONE" if no default + loadPath: <string>, // where the engine line is located; missing if no default + origin: <string>, // 'default', 'verified', 'unverified', or 'invalid'; based on the presence and validity of the engine's loadPath verification hash. + submissionURL: <string> // missing if no default or for user-installed engines + }, + searchCohort: <string>, // optional, contains an identifier for any active search A/B experiments + e10sEnabled: <bool>, // whether e10s is on, i.e. browser tabs open by default in a different process + e10sCohort: <string>, // which e10s cohort was assigned for this user + telemetryEnabled: <bool>, // false on failure + locale: <string>, // e.g. "it", null on failure + update: { + channel: <string>, // e.g. "release", null on failure + enabled: <bool>, // true on failure + autoDownload: <bool>, // true on failure + }, + userPrefs: { + // Only prefs which are changed from the default value are listed + // in this block + "pref.name.value": value // some prefs send the value + "pref.name.url": "<user-set>" // For some privacy-sensitive prefs + // only the fact that the value has been changed is recorded + }, + attribution: { // optional, only present if the installation has attribution data + // all of these values are optional. + source: <string>, // referring partner domain, when install happens via a known partner + medium: <string>, // category of the source, such as "organic" for a search engine + campaign: <string>, // identifier of the particular campaign that led to the download of the product + content: <string>, // identifier to indicate the particular link within a campaign + }, + }, + profile: { + creationDate: <integer>, // integer days since UNIX epoch, e.g. 16446 + resetDate: <integer>, // integer days since UNIX epoch, e.g. 16446 - optional + }, + partner: { // This section may not be immediately available on startup + distributionId: <string>, // pref "distribution.id", null on failure + distributionVersion: <string>, // pref "distribution.version", null on failure + partnerId: <string>, // pref mozilla.partner.id, null on failure + distributor: <string>, // pref app.distributor, null on failure + distributorChannel: <string>, // pref app.distributor.channel, null on failure + partnerNames: [ + // list from prefs app.partner.<name>=<name> + ], + }, + system: { + memoryMB: <number>, + virtualMaxMB: <number>, // windows-only + isWow64: <bool>, // windows-only + cpu: { + count: <number>, // desktop only, e.g. 8, or null on failure - logical cpus + cores: <number>, // desktop only, e.g., 4, or null on failure - physical cores + vendor: <string>, // desktop only, e.g. "GenuineIntel", or null on failure + family: <number>, // desktop only, null on failure + model: <number, // desktop only, null on failure + stepping: <number>, // desktop only, null on failure + l2cacheKB: <number>, // L2 cache size in KB, only on windows & mac + l3cacheKB: <number>, // desktop only, L3 cache size in KB + speedMHz: <number>, // desktop only, cpu clock speed in MHz + extensions: [ + <string>, + ... + // as applicable: + // "MMX", "SSE", "SSE2", "SSE3", "SSSE3", "SSE4A", "SSE4_1", + // "SSE4_2", "AVX", "AVX2", "EDSP", "ARMv6", "ARMv7", "NEON" + ], + }, + device: { // This section is only available on mobile devices. + model: <string>, // the "device" from FHR, null on failure + manufacturer: <string>, // null on failure + hardware: <string>, // null on failure + isTablet: <bool>, // null on failure + }, + os: { + name: <string>, // "Windows_NT" or null on failure + version: <string>, // e.g. "6.1", null on failure + kernelVersion: <string>, // android/b2g only or null on failure + servicePackMajor: <number>, // windows only or null on failure + servicePackMinor: <number>, // windows only or null on failure + windowsBuildNumber: <number>, // windows 10 only or null on failure + windowsUBR: <number>, // windows 10 only or null on failure + installYear: <number>, // windows only or null on failure + locale: <string>, // "en" or null on failure + }, + hdd: { + profile: { // hdd where the profile folder is located + model: <string>, // windows only or null on failure + revision: <string>, // windows only or null on failure + }, + binary: { // hdd where the application binary is located + model: <string>, // windows only or null on failure + revision: <string>, // windows only or null on failure + }, + system: { // hdd where the system files are located + model: <string>, // windows only or null on failure + revision: <string>, // windows only or null on failure + }, + }, + gfx: { + D2DEnabled: <bool>, // null on failure + DWriteEnabled: <bool>, // null on failure + //DWriteVersion: <string>, // temporarily removed, pending bug 1154500 + adapters: [ + { + description: <string>, // e.g. "Intel(R) HD Graphics 4600", null on failure + vendorID: <string>, // null on failure + deviceID: <string>, // null on failure + subsysID: <string>, // null on failure + RAM: <number>, // in MB, null on failure + driver: <string>, // null on failure + driverVersion: <string>, // null on failure + driverDate: <string>, // null on failure + GPUActive: <bool>, // currently always true for the first adapter + }, + ... + ], + // Note: currently only added on Desktop. On Linux, only a single + // monitor is returned representing the entire virtual screen. + monitors: [ + { + screenWidth: <number>, // screen width in pixels + screenHeight: <number>, // screen height in pixels + refreshRate: <number>, // refresh rate in hertz (present on Windows only). + // (values <= 1 indicate an unknown value) + pseudoDisplay: <bool>, // networked screen (present on Windows only) + scale: <number>, // backing scale factor (present on Mac only) + }, + ... + ], + features: { + compositor: <string>, // Layers backend for compositing (eg "d3d11", "none", "opengl") + + // Each the following features can have one of the following statuses: + // "unused" - This feature has not been requested. + // "unavailable" - Safe Mode or OS restriction prevents use. + // "blocked" - Blocked due to an internal condition such as safe mode. + // "blacklisted" - Blocked due to a blacklist restriction. + // "disabled" - User explicitly disabled this default feature. + // "failed" - This feature was attempted but failed to initialize. + // "available" - User has this feature available. + "d3d11" { // This feature is Windows-only. + status: <string>, + warp: <bool>, // Software rendering (WARP) mode was chosen. + textureSharing: <bool> // Whether or not texture sharing works. + version: <number>, // The D3D11 device feature level. + blacklisted: <bool>, // Whether D3D11 is blacklisted; use to see whether WARP + // was blacklist induced or driver-failure induced. + }, + "d2d" { // This feature is Windows-only. + status: <string>, + version: <string>, // Either "1.0" or "1.1". + }, + }, + }, + }, + addons: { + activeAddons: { // the currently enabled addons + <addon id>: { + blocklisted: <bool>, + description: <string>, // null if not available + name: <string>, + userDisabled: <bool>, + appDisabled: <bool>, + version: <string>, + scope: <integer>, + type: <string>, // "extension", "service", ... + foreignInstall: <bool>, + hasBinaryComponents: <bool> + installDay: <number>, // days since UNIX epoch, 0 on failure + updateDay: <number>, // days since UNIX epoch, 0 on failure + signedState: <integer>, // whether the add-on is signed by AMO, only present for extensions + isSystem: <bool>, // true if this is a System Add-on + }, + ... + }, + theme: { // the active theme + id: <string>, + blocklisted: <bool>, + description: <string>, + name: <string>, + userDisabled: <bool>, + appDisabled: <bool>, + version: <string>, + scope: <integer>, + foreignInstall: <bool>, + hasBinaryComponents: <bool> + installDay: <number>, // days since UNIX epoch, 0 on failure + updateDay: <number>, // days since UNIX epoch, 0 on failure + }, + activePlugins: [ + { + name: <string>, + version: <string>, + description: <string>, + blocklisted: <bool>, + disabled: <bool>, + clicktoplay: <bool>, + mimeTypes: [<string>, ...], + updateDay: <number>, // days since UNIX epoch, 0 on failure + }, + ... + ], + activeGMPlugins: { + <gmp id>: { + version: <string>, + userDisabled: <bool>, + applyBackgroundUpdates: <integer>, + }, + ... + }, + activeExperiment: { // section is empty if there's no active experiment + id: <string>, // id + branch: <string>, // branch name + }, + persona: <string>, // id of the current persona, null on GONK + }, + } + +build +----- + +buildId +~~~~~~~ +Firefox builds downloaded from mozilla.org use a 14-digit buildId. Builds included in other distributions may have a different format (e.g. only 10 digits). + +Settings +-------- + +defaultSearchEngine +~~~~~~~~~~~~~~~~~~~ +Note: Deprecated, use defaultSearchEngineData instead. + +Contains the string identifier or name of the default search engine provider. This will not be present in environment data collected before the Search Service initialization. + +The special value ``NONE`` could occur if there is no default search engine. + +The special value ``UNDEFINED`` could occur if a default search engine exists but its identifier could not be determined. + +This field's contents are ``Services.search.defaultEngine.identifier`` (if defined) or ``"other-"`` + ``Services.search.defaultEngine.name`` if not. In other words, search engines without an ``.identifier`` are prefixed with ``other-``. + +defaultSearchEngineData +~~~~~~~~~~~~~~~~~~~~~~~ +Contains data identifying the engine currently set as the default. + +The object contains: + +- a ``name`` property with the name of the engine, or ``NONE`` if no + engine is currently set as the default. + +- a ``loadPath`` property: an anonymized path of the engine xml file, e.g. + jar:[app]/omni.ja!browser/engine.xml + (where 'browser' is the name of the chrome package, not a folder) + [profile]/searchplugins/engine.xml + [distribution]/searchplugins/common/engine.xml + [other]/engine.xml + +- an ``origin`` property: the value will be ``default`` for engines that are built-in or from distribution partners, ``verified`` for user-installed engines with valid verification hashes, ``unverified`` for non-default engines without verification hash, and ``invalid`` for engines with broken verification hashes. + +- a ``submissionURL`` property with the HTTP url we would use to search. + For privacy, we don't record this for user-installed engines. + +``loadPath`` and ``submissionURL`` are not present if ``name`` is ``NONE``. + +searchCohort +~~~~~~~~~~~~ + +If the user has been enrolled into a search default change experiment, this contains the string identifying the experiment the user is taking part in. Most user profiles will never be part of any search default change experiment, and will not send this value. + +userPrefs +~~~~~~~~~ + +This object contains user preferences. + +Each key in the object is the name of a preference. A key's value depends on the policy with which the preference was collected. There are two such policies, "value" and "state". For preferences collected under the "value" policy, the value will be the preference's value. For preferences collected under the "state" policy, the value will be an opaque marker signifying only that the preference has a user value. The "state" policy is therefore used when user privacy is a concern. + +The following is a partial list of collected preferences. + +- ``browser.search.suggest.enabled``: The "master switch" for search suggestions everywhere in Firefox (search bar, urlbar, etc.). Defaults to true. + +- ``browser.urlbar.suggest.searches``: True if search suggestions are enabled in the urlbar. Defaults to false. + +- ``browser.urlbar.userMadeSearchSuggestionsChoice``: True if the user has clicked Yes or No in the urlbar's opt-in notification. Defaults to false. + +- ``browser.zoom.full``: True if zoom is enabled for both text and images, that is if "Zoom Text Only" is not enabled. Defaults to true. Collection of this preference has been enabled in Firefox 50 and will be disabled again in Firefox 53 (`Bug 979323 <https://bugzilla.mozilla.org/show_bug.cgi?id=979323>`_). + +- ``security.sandbox.content.level``: The meanings of the values are OS dependent, but 0 means not sandboxed for all OS. Details of the meanings can be found in the `Firefox prefs file <http://hg.mozilla.org/mozilla-central/file/tip/browser/app/profile/firefox.js>`_. + +attribution +~~~~~~~~~~~ + +This object contains the attribution data for the product installation. + +Attribution data is used to link installations of Firefox with the source that the user arrived at the Firefox download page from. It would indicate, for instance, when a user executed a web search for Firefox and arrived at the download page from there, directly navigated to the site, clicked on a link from a particular social media campaign, etc. + +The attribution data is included in some versions of the default Firefox installer for Windows (the "stub" installer) and stored as part of the installation. All platforms other than Windows and also Windows installations that did not use the stub installer do not have this data and will not include the ``attribution`` object. + +partner +------- + +If the user is using a partner repack, this contains information identifying the repack being used, otherwise "partnerNames" will be an empty array and other entries will be null. The information may be missing when the profile just becomes available. In Firefox for desktop, the information along with other customizations defined in distribution.ini are processed later in the startup phase, and will be fully applied when "distribution-customization-complete" notification is sent. + +Distributions are most reliably identified by the ``distributionId`` field. Partner information can be found in the `partner repacks <https://github.com/mozilla-partners>`_ (`the old one <http://hg.mozilla.org/build/partner-repacks/>`_ is deprecated): it contains one private repository per partner. +Important values for ``distributionId`` include: + +- "MozillaOnline" for the Mozilla China repack. +- "canonical", for the `Ubuntu Firefox repack <http://bazaar.launchpad.net/~mozillateam/firefox/firefox.trusty/view/head:/debian/distribution.ini>`_. +- "yandex", for the Firefox Build by Yandex. + +system +------ + +os +~~ + +This object contains operating system information. + +- ``name``: the name of the OS. +- ``version``: a string representing the OS version. +- ``kernelVersion``: an Android/B2G only string representing the kernel version. +- ``servicePackMajor``: the Windows only major version number for the installed service pack. +- ``servicePackMinor``: the Windows only minor version number for the installed service pack. +- ``windowsBuildNumber``: the Windows build number, only available for Windows >= 10. +- ``windowsUBR``: the Windows UBR number, only available for Windows >= 10. This value is incremented by Windows cumulative updates patches. +- ``installYear``: the Windows only integer representing the year the OS was installed. +- ``locale``: the string representing the OS locale. + +addons +------ + +activeAddons +~~~~~~~~~~~~ + +Starting from Firefox 44, the length of the following string fields: ``name``, ``description`` and ``version`` is limited to 100 characters. The same limitation applies to the same fields in ``theme`` and ``activePlugins``. diff --git a/toolkit/components/telemetry/docs/data/heartbeat-ping.rst b/toolkit/components/telemetry/docs/data/heartbeat-ping.rst new file mode 100644 index 000000000..413da0376 --- /dev/null +++ b/toolkit/components/telemetry/docs/data/heartbeat-ping.rst @@ -0,0 +1,63 @@ + +"heartbeat" ping +================= + +This ping is submitted after a Firefox Heartbeat survey. Even if the user exits +the browser, closes the survey window, or ignores the survey, Heartbeat will +provide a ping to Telemetry for sending during the same session. + +The payload contains the user's survey response (if any) as well as timestamps +of various Heartbeat events (survey shown, survey closed, link clicked, etc). + +The ping will also report the "surveyId", "surveyVersion" and "testing" +Heartbeat survey parameters (if they are present in the survey config). +These "meta fields" will be repeated verbatim in the payload section. + +The environment block and client ID are submitted with this ping. + +Structure: + +.. code-block:: js + + { + type: "heartbeat", + version: 4, + clientId: <UUID>, + environment: { /* ... */ } + // ... common ping data + payload: { + version: 1, + flowId: <string>, + ... timestamps below ... + offeredTS: <integer epoch timestamp>, + learnMoreTS: <integer epoch timestamp>, + votedTS: <integer epoch timestamp>, + engagedTS: <integer epoch timestamp>, + closedTS: <integer epoch timestamp>, + expiredTS: <integer epoch timestamp>, + windowClosedTS: <integer epoch timestamp>, + // ... user's rating below + score: <integer>, + // ... survey meta fields below + surveyId: <string>, + surveyVersion: <integer>, + testing: <boolean> + } + } + +Notes: + +* Pings will **NOT** have all possible timestamps, timestamps are only reported for events that actually occurred. +* Timestamp meanings: + * offeredTS: when the survey was shown to the user + * learnMoreTS: when the user clicked on the "Learn More" link + * votedTS: when the user voted + * engagedTS: when the user clicked on the survey-provided button (alternative to voting feature) + * closedTS: when the Heartbeat notification bar was closed + * expiredTS: indicates that the survey expired after 2 hours of no interaction (threshold regulated by "browser.uitour.surveyDuration" pref) + * windowClosedTS: the user closed the entire Firefox window containing the survey, thus ending the survey. This timestamp will also be reported when the survey is ended by the browser being shut down. +* The surveyId/surveyVersion fields identify a specific survey (like a "1040EZ" tax paper form). The flowID is a UUID that uniquely identifies a single user's interaction with the survey. Think of it as a session token. +* The self-support page cannot include additional data in this payload. Only the the 4 flowId/surveyId/surveyVersion/testing fields are under the self-support page's control. + +See also: :doc:`common ping fields <common-ping>` + diff --git a/toolkit/components/telemetry/docs/data/index.rst b/toolkit/components/telemetry/docs/data/index.rst new file mode 100644 index 000000000..a0467e9a1 --- /dev/null +++ b/toolkit/components/telemetry/docs/data/index.rst @@ -0,0 +1,18 @@ +================== +Data documentation +================== + +.. toctree:: + :maxdepth: 2 + :titlesonly: + :glob: + + common-ping + environment + main-ping + deletion-ping + crash-ping + *-ping + addons-malware-ping + +The `mozilla-pipeline-schemas repository <https://github.com/mozilla-services/mozilla-pipeline-schemas/>`_ contains schemas for some of the pings. diff --git a/toolkit/components/telemetry/docs/data/main-ping.rst b/toolkit/components/telemetry/docs/data/main-ping.rst new file mode 100644 index 000000000..445090af9 --- /dev/null +++ b/toolkit/components/telemetry/docs/data/main-ping.rst @@ -0,0 +1,609 @@ + +"main" ping +=========== + +.. toctree:: + :maxdepth: 2 + +This is the "main" Telemetry ping type, whose payload contains most of the measurements that are used to track the performance and health of Firefox in the wild. +It includes the histograms and other performance and diagnostic data. + +This ping is triggered by different scenarios, which is documented by the ``reason`` field: + +* ``aborted-session`` - this ping is regularly saved to disk (every 5 minutes), overwriting itself, and deleted at shutdown. If a previous aborted session ping is found at startup, it gets sent to the server. The first aborted-session ping is generated as soon as Telemetry starts +* ``environment-change`` - the :doc:`environment` changed, so the session measurements got reset and a new subsession starts +* ``shutdown`` - triggered when the browser session ends +* ``daily`` - a session split triggered in 24h hour intervals at local midnight. If an ``environment-change`` ping is generated by the time it should be sent, the daily ping is rescheduled for the next midnight +* ``saved-session`` - the *"classic"* Telemetry payload with measurements covering the whole browser session (only submitted for a transition period) + +Most reasons lead to a session split, initiating a new *subsession*. We reset important measurements for those subsessions. + +After a new subsession split, the ``internal-telemetry-after-subsession-split`` topic is notified to all the observers. *This is an internal topic and is only meant for internal Telemetry usage.* + +*Note:* ``saved-session`` is sent with a different ping type (``saved-session``, not ``main``), but otherwise has the same format as discussed here. + +Structure: + +.. code-block:: js + + { + version: 4, + + info: { + reason: <string>, // what triggered this ping: "saved-session", "environment-change", "shutdown", ... + revision: <string>, // the Histograms.json revision + timezoneOffset: <integer>, // time-zone offset from UTC, in minutes, for the current locale + previousBuildId: <string>, // null if this is the first run, or the previous build ID is unknown + + sessionId: <uuid>, // random session id, shared by subsessions + subsessionId: <uuid>, // random subsession id + previousSessionId: <uuid>, // session id of the previous session, null on first run. + previousSubsessionId: <uuid>, // subsession id of the previous subsession (even if it was in a different session), + // null on first run. + + subsessionCounter: <unsigned integer>, // the running no. of this subsession since the start of the browser session + profileSubsessionCounter: <unsigned integer>, // the running no. of all subsessions for the whole profile life time + + sessionStartDate: <ISO date>, // daily precision + subsessionStartDate: <ISO date>, // daily precision, ISO date in local time + sessionLength: <integer>, // the session length until now in seconds, monotonic + subsessionLength: <integer>, // the subsession length in seconds, monotonic + + flashVersion: <string>, // obsolete, use ``environment.addons.activePlugins`` + addons: <string>, // obsolete, use ``environment.addons`` + }, + + processes: {...}, + childPayloads: [...], // only present with e10s; reduced payloads from content processes, null on failure + simpleMeasurements: {...}, + + // The following properties may all be null if we fail to collect them. + histograms: {...}, + keyedHistograms: {...}, + chromeHangs: {...}, + threadHangStats: [...], + log: [...], + webrtc: {...}, + gc: {...}, + fileIOReports: {...}, + lateWrites: {...}, + addonDetails: {...}, + addonHistograms: {...}, + UIMeasurements: [...], + slowSQL: {...}, + slowSQLstartup: {...}, + } + +info +---- + +sessionLength +~~~~~~~~~~~~~ +The length of the current session so far in seconds. +This uses a monotonic clock, so this may mismatch with other measurements that +are not monotonic like calculations based on ``Date.now()``. + +If the monotonic clock failed, this will be ``-1``. + +subsessionLength +~~~~~~~~~~~~~~~~ +The length of this subsession in seconds. +This uses a monotonic clock, so this may mismatch with other measurements that are not monotonic (e.g. based on Date.now()). + +If ``sessionLength`` is ``-1``, the monotonic clock is not working. + +processes +--------- +This section contains per-process data. + +Structure: + +.. code-block:: js + + "processes" : { + ... other processes ... + "parent": { + scalars: {...}, + }, + "content": { + histograms: {...}, + keyedHistograms: {...}, + }, + } + +histograms and keyedHistograms +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +This section contains histograms and keyed histograms accumulated on content processes. Histograms recorded on a content child process have different character than parent histograms. For instance, ``GC_MS`` will be much different in ``processes.content`` as it has to contend with web content, whereas the instance in ``payload.histograms`` has only to contend with browser JS. Also, some histograms may be absent if never recorded on a content child process (``EVENTLOOP_UI_ACTIVITY`` is parent-process-only). + +This format was adopted in Firefox 51 via bug 1218576. + +scalars +~~~~~~~ +This section contains the :doc:`../collection/scalars` that are valid for the current platform. Scalars are not created nor submitted if no data was added to them, and are only reported with subsession pings. Scalar data is only currently reported for the main process. Their type and format is described by the ``Scalars.yaml`` file. Its most recent version is available `here <https://dxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/Scalars.yaml>`_. The ``info.revision`` field indicates the revision of the file that describes the reported scalars. + +childPayloads +------------- +The Telemetry payloads sent by child processes, recorded on child process shutdown (event ``content-child-shutdown`` observed). They are reduced session payloads, only available with e10s. Among some other things, they don't contain histograms, keyed histograms, addon details, addon histograms, or UI Telemetry. + +Note: Child payloads are not collected and cleared with subsession splits, they are currently only meaningful when analysed from ``saved-session`` or ``main`` pings with ``reason`` set to ``shutdown``. + +Note: Before Firefox 51 and bug 1218576, content process histograms and keyedHistograms were in the individual child payloads instead of being aggregated into ``processes.content``. + +simpleMeasurements +------------------ +This section contains a list of simple measurements, or counters. In addition to the ones highlighted below, Telemetry timestamps (see `here <https://dxr.mozilla.org/mozilla-central/search?q=%22TelemetryTimestamps.add%22&redirect=false&case=true>`_ and `here <https://dxr.mozilla.org/mozilla-central/search?q=%22recordTimestamp%22&redirect=false&case=true>`_) can be reported. + +totalTime +~~~~~~~~~ +A non-monotonic integer representing the number of seconds the session has been alive. + +uptime +~~~~~~ +A non-monotonic integer representing the number of minutes the session has been alive. + +addonManager +~~~~~~~~~~~~ +Only available in the extended set of measures, it contains a set of counters related to Addons. See `here <https://dxr.mozilla.org/mozilla-central/search?q=%22AddonManagerPrivate.recordSimpleMeasure%22&redirect=false&case=true>`_ for a list of recorded measures. + +UITelemetry +~~~~~~~~~~~ +Only available in the extended set of measures. For more see :ref:`uitelemetry`. + +startupInterrupted +~~~~~~~~~~~~~~~~~~ +A boolean set to true if startup was interrupted by an interactive prompt. + +js +~~ +This section contains a series of counters from the JavaScript engine. + +Structure: + +.. code-block:: js + + "js" : { + "setProto": <unsigned integer>, // Number of times __proto__ is set + "customIter": <unsigned integer> // Number of times __iterator__ is used (i.e., is found for a for-in loop) + } + +maximalNumberOfConcurrentThreads +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +An integer representing the highest number of threads encountered so far during the session. + +startupSessionRestoreReadBytes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Windows-only integer representing the number of bytes read by the main process up until the session store has finished restoring the windows. + +startupSessionRestoreWriteBytes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Windows-only integer representing the number of bytes written by the main process up until the session store has finished restoring the windows. + +startupWindowVisibleReadBytes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Windows-only integer representing the number of bytes read by the main process up until after a XUL window is made visible. + +startupWindowVisibleWriteBytes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Windows-only integer representing the number of bytes written by the main process up until after a XUL window is made visible. + +debuggerAttached +~~~~~~~~~~~~~~~~ +A boolean set to true if a debugger is attached to the main process. + +shutdownDuration +~~~~~~~~~~~~~~~~ +The time, in milliseconds, it took to complete the last shutdown. + +failedProfileLockCount +~~~~~~~~~~~~~~~~~~~~~~ +The number of times the system failed to lock the user profile. + +savedPings +~~~~~~~~~~ +Integer count of the number of pings that need to be sent. + +activeTicks +~~~~~~~~~~~ +Integer count of the number of five-second intervals ('ticks') the user was considered 'active' (sending UI events to the window). An extra event is fired immediately when the user becomes active after being inactive. This is for some mouse and gamepad events, and all touch, keyboard, wheel, and pointer events (see `EventStateManager.cpp <https://dxr.mozilla.org/mozilla-central/rev/e6463ae7eda2775bc84593bb4a0742940bb87379/dom/events/EventStateManager.cpp#549>`_). +This measure might be useful to give a trend of how much a user actually interacts with the browser when compared to overall session duration. It does not take into account whether or not the window has focus or is in the foreground. Just if it is receiving these interaction events. +Note that in ``main`` pings, this measure is reset on subsession splits, while in ``saved-session`` pings it covers the whole browser session. + +pingsOverdue +~~~~~~~~~~~~ +Integer count of pending pings that are overdue. + +histograms +---------- +This section contains the histograms that are valid for the current platform. ``Flag`` and ``count`` histograms are always created and submitted, with their default value being respectively ``false`` and ``0``. Other histogram types (`see here <https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Adding_a_new_Telemetry_probe#Choosing_a_Histogram_Type>`_) are not created nor submitted if no data was added to them. The type and format of the reported histograms is described by the ``Histograms.json`` file. Its most recent version is available `here <https://dxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/Histograms.json>`_. The ``info.revision`` field indicates the revision of the file that describes the reported histograms. + +keyedHistograms +--------------- +This section contains the keyed histograms available for the current platform. + +As of Firefox 48, this section does not contain empty keyed histograms anymore. + +threadHangStats +--------------- +Contains the statistics about the hangs in main and background threads. Note that hangs in this section capture the [C++ pseudostack](https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Profiling_with_the_Built-in_Profiler#Native_stack_vs._Pseudo_stack) and an incomplete JS stack, which is not 100% precise. + +To avoid submitting overly large payloads, some limits are applied: + +* Identical, adjacent "(chrome script)" or "(content script)" stack entries are collapsed together. If a stack is reduced, the "(reduced stack)" frame marker is added as the oldest frame. +* The depth of the reported stacks is limited to 11 entries. This value represents the 99.9th percentile of the thread hangs stack depths reported by Telemetry. + +Structure: + +.. code-block:: js + + "threadHangStats" : [ + { + "name" : "Gecko", + "activity" : {...}, // a time histogram of all task run times + "hangs" : [ + { + "stack" : [ + "Startup::XRE_Main", + "Timer::Fire", + "(content script)", + "IPDL::PPluginScriptableObject::SendGetChildProperty", + ... up to 11 frames ... + ], + "nativeStack": [...], // optionally available + "histogram" : {...}, // the time histogram of the hang times + "annotations" : [ + { + "pluginName" : "Shockwave Flash", + "pluginVersion" : "18.0.0.209" + }, + ... other annotations ... + ] + }, + ], + }, + ... other threads ... + ] + +chromeHangs +----------- +Contains the statistics about the hangs happening exclusively on the main thread of the parent process. Precise C++ stacks are reported. This is only available on Nightly Release on Windows, when building using "--enable-profiling" switch. + +Some limits are applied: + +* Reported chrome hang stacks are limited in depth to 50 entries. +* The maximum number of reported stacks is 50. + +Structure: + +.. code-block:: js + + "chromeHangs" : { + "memoryMap" : [ + ["wgdi32.pdb", "08A541B5942242BDB4AEABD8C87E4CFF2"], + ["igd10iumd32.pdb", "D36DEBF2E78149B5BE1856B772F1C3991"], + ... other entries in the format ["module name", "breakpad identifier"] ... + ], + "stacks" : [ + [ + [ + 0, // the module index or -1 for invalid module indices + 190649 // the offset of this program counter in its module or an absolute pc + ], + [1, 2540075], + ... other frames, up to 50 ... + ], + ... other stacks, up to 50 ... + ], + "durations" : [8, ...], // the hang durations (in seconds) + "systemUptime" : [692, ...], // the system uptime (in minutes) at the time of the hang + "firefoxUptime" : [672, ...], // the Firefox uptime (in minutes) at the time of the hang + "annotations" : [ + [ + [0, ...], // the indices of the related hangs + { + "pluginName" : "Shockwave Flash", + "pluginVersion" : "18.0.0.209", + ... other annotations as key:value pairs ... + } + ], + ... + ] + }, + +log +--- +This section contains a log of important or unusual events reported through Telemetry. + +Structure: + +.. code-block:: js + + "log": [ + [ + "Event_ID", + 3785, // the timestamp (in milliseconds) for the log entry + ... other data ... + ], + ... + ] + + +webrtc +------ +Contains special statistics gathered by WebRTC related components. + +So far only a bitmask for the ICE candidate type present in a successful or +failed WebRTC connection is getting reported through C++ code as +IceCandidatesStats, because the required bitmask is too big to be represented +in a regular enum histogram. Further this data differentiates between Loop +(aka Firefox Hello) connections and everything else, which is categorized as +WebRTC. + +Note: in most cases the webrtc and loop dictionaries inside of +IceCandidatesStats will simply be empty as the user has not used any WebRTC +PeerConnection at all during the ping report time. + +Structure: + +.. code-block:: js + + "webrtc": { + "IceCandidatesStats": { + "webrtc": { + "34526345": { + "successCount": 5 + }, + "2354353": { + "failureCount": 1 + } + }, + "loop": { + "2349346359": { + "successCount": 3 + }, + "73424": { + "successCount": 1, + "failureCount": 5 + } + } + } + }, + +gc +-- +Contains statistics about selected garbage collections. To avoid +bloating the ping, only a few GCs are included. There are two +selection strategies. We always save the two GCs with the worst +max_pause time. Additionally, in content processes, two collections +are selected at random. If a GC runs for C milliseconds and the total +time for all GCs since the session began is T milliseconds, then the +GC has a C/T probablility of being selected for one of these "slots". + +Structure: + +.. code-block:: js + + "gc": { + "random": [ + { + // Timestamps are in milliseconds since startup. All the times here + // are wall-clock times, which may not be monotonically increasing. + "timestamp": 294872.2, + // All durations are in milliseconds. + "max_pause": 73.629, + "total_time": 364.951, // Sum of all slice times. + "zones_collected": 9, + "total_zones": 9, + "total_compartments": 309, + "minor_gcs": 44, + "store_buffer_overflows": 19, + "mmu_20ms": 0, + "mmu_50ms": 0, + // Reasons include "None", "NonIncrementalRequested", + // "AbortRequested", "KeepAtomsSet", "IncrementalDisabled", + // "ModeChange", "MallocBytesTrigger", "GCBytesTrigger", + // "ZoneChange". + "nonincremental_reason": "None", + "allocated": 37, // In megabytes. + "added_chunks": 54, + "removed_chunks": 12, + // Total number of slices (some of which may not appear + // in the "slices" array). + "num_slices": 15, + // We record at most 4 slices. + "slices": [ + { + "slice": 0, // The index of this slice. + "pause": 23.221, // How long the slice took. + "when": 0, // Milliseconds since the start of the GC. + "reason": "SET_NEW_DOCUMENT", + // GC state when the slice started + "initial_state": "NotActive", + // GC state when the slice ended + "final_state": "Mark", + // Budget is either "Xms", "work(Y)", or + // "unlimited". + "budget": "10ms", + // Number of page faults during the slice. + "page_faults": 0, + "start_timestamp": 294875, + "end_timestamp": 294879, + // Time taken by each phase. There are at most 65 possible + // phases, but usually only a few phases run in a given slice. + "times": { + "wait_background_thread": 0.012, + "mark_discard_code": 2.845, + "purge": 0.723, + "mark": 9.831, + "mark_roots": 0.102, + "buffer_gray_roots": 3.095, + "mark_cross_compartment_wrappers": 0.039, + "mark_c_and_js_stacks": 0.005, + "mark_runtime_wide_data": 2.313, + "mark_embedding": 0.117, + "mark_compartments": 2.27, + "unmark": 1.063, + "minor_gcs_to_evict_nursery": 8.701, + ... + } + }, + { ... }, + ], + // Sum of the phase times across all slices, including + // omitted slices. As before, there are <= 65 possible phases. + "totals": { + "wait_background_thread": 0.012, + "mark_discard_code": 2.845, + "purge": 0.723, + "mark": 9.831, + "mark_roots": 0.102, + "buffer_gray_roots": 3.095, + "mark_cross_compartment_wrappers": 0.039, + "mark_c_and_js_stacks": 0.005, + "mark_runtime_wide_data": 2.313, + "mark_embedding": 0.117, + "mark_compartments": 2.27, + "unmark": 1.063, + "minor_gcs_to_evict_nursery": 8.701, + ... + } + }, + ... // Up to four more selected GCs follow. + ], + "worst": [ + ... // Same as above, but the 2 worst GCs by max_pause. + ] + }, + +fileIOReports +------------- +Contains the statistics of main-thread I/O recorded during the execution. Only the I/O stats for the XRE and the profile directories are currently reported, neither of them disclosing the full local path. + +Structure: + +.. code-block:: js + + "fileIOReports": { + "{xre}": [ + totalTime, // Accumulated duration of all operations + creates, // Number of create/open operations + reads, // Number of read operations + writes, // Number of write operations + fsyncs, // Number of fsync operations + stats, // Number of stat operations + ], + "{profile}": [ ... ], + ... + } + +lateWrites +---------- +This sections reports writes to the file system that happen during shutdown. The reported data contains the stack and the loaded libraries at the time the writes happened. + +Structure: + +.. code-block:: js + + "lateWrites" : { + "memoryMap" : [ + ["wgdi32.pdb", "08A541B5942242BDB4AEABD8C87E4CFF2"], + ... other entries in the format ["module name", "breakpad identifier"] ... + ], + "stacks" : [ + [ + [ + 0, // the module index or -1 for invalid module indices + 190649 // the offset of this program counter in its module or an absolute pc + ], + [1, 2540075], + ... other frames ... + ], + ... other stacks ... + ], + }, + +addonDetails +------------ +This section contains per-addon telemetry details, as reported by each addon provider. The XPI provider is the only one reporting at the time of writing (`see DXR <https://dxr.mozilla.org/mozilla-central/search?q=setTelemetryDetails&case=true>`_). Telemetry does not manipulate or enforce a specific format for the supplied provider's data. + +Structure: + +.. code-block:: js + + "addonDetails": { + "XPI": { + "adbhelper@mozilla.org": { + "scan_items": 24, + "scan_MS": 3, + "location": "app-profile", + "name": "ADB Helper", + "creator": "Mozilla & Android Open Source Project", + "startup_MS": 30 + }, + ... + }, + ... + } + +addonHistograms +--------------- +This section contains the histogram registered by the addons (`see here <https://dxr.mozilla.org/mozilla-central/rev/584870f1cbc5d060a57e147ce249f736956e2b62/toolkit/components/telemetry/nsITelemetry.idl#303>`_). This section is not present if no addon histogram is available. + +UITelemetry +----------- +See the ``UITelemetry data format`` documentation. + +slowSQL +------- +This section contains the informations about the slow SQL queries for both the main and other threads. The execution of an SQL statement is considered slow if it takes 50ms or more on the main thread or 100ms or more on other threads. Slow SQL statements will be automatically trimmed to 1000 characters. This limit doesn't include the ellipsis and database name, that are appended at the end of the stored statement. + +Structure: + +.. code-block:: js + + "slowSQL": { + "mainThread": { + "Sanitized SQL Statement": [ + 1, // the number of times this statement was hit + 200 // the total time (in milliseconds) that was spent on this statement + ], + ... + }, + "otherThreads": { + "VACUUM /* places.sqlite */": [ + 1, + 330 + ], + ... + } + }, + +slowSQLStartup +-------------- +This section contains the slow SQL statements gathered at startup (until the "sessionstore-windows-restored" event is fired). The structure of this section resembles the one for `slowSQL`_. + +UIMeasurements +-------------- +This section contains UI specific telemetry measurements and events. This section is mainly populated with Android-specific data and events (`see here <https://dxr.mozilla.org/mozilla-central/search?q=regexp%3AUITelemetry.%28addEvent|startSession|stopSession%29&redirect=false&case=false>`_). + +Structure: + +.. code-block:: js + + "UIMeasurements": [ + { + "type": "event", // either "session" or "event" + "action": "action.1", + "method": "menu", + "sessions": [], + "timestamp": 12345, + "extras": "settings" + }, + { + "type": "session", + "name": "awesomescreen.1", + "reason": "commit", + "start": 123, + "end": 456 + } + ... + ], diff --git a/toolkit/components/telemetry/docs/data/sync-ping.rst b/toolkit/components/telemetry/docs/data/sync-ping.rst new file mode 100644 index 000000000..775ab008a --- /dev/null +++ b/toolkit/components/telemetry/docs/data/sync-ping.rst @@ -0,0 +1,182 @@ + +"sync" ping +=========== + +This is an aggregated format that contains information about each sync that occurred during a timeframe. It is submitted every 12 hours, and on browser shutdown, but only if the syncs property would not be empty. The ping does not contain the enviroment block, nor the clientId. + +Each item in the syncs property is generated after a sync is completed, for both successful and failed syncs, and contains measurements pertaining to sync performance and error information. + +A JSON-schema document describing the exact format of the ping's payload property can be found at `services/sync/tests/unit/sync\_ping\_schema.json <https://dxr.mozilla.org/mozilla-central/source/services/sync/tests/unit/sync_ping_schema.json>`_. + +Structure: + +.. code-block:: js + + { + version: 4, + type: "sync", + ... common ping data + payload: { + version: 1, + discarded: <integer count> // Number of syncs discarded -- left out if zero. + why: <string>, // Why did we submit the ping? Either "shutdown" or "schedule". + // Array of recorded syncs. The ping is not submitted if this would be empty + syncs: [{ + when: <integer milliseconds since epoch>, + took: <integer duration in milliseconds>, + uid: <string>, // Hashed FxA unique ID, or string of 32 zeros. + deviceID: <string>, // Hashed FxA Device ID, hex string of 64 characters, not included if the user is not logged in. + didLogin: <bool>, // Optional, is this the first sync after login? Excluded if we don't know. + why: <string>, // Optional, why the sync occured, excluded if we don't know. + + // Optional, excluded if there was no error. + failureReason: { + name: <string>, // "httperror", "networkerror", "shutdownerror", etc. + code: <integer>, // Only present for "httperror" and "networkerror". + error: <string>, // Only present for "othererror" and "unexpectederror". + from: <string>, // Optional, and only present for "autherror". + }, + + // Optional, excluded if we couldn't get a valid uid or local device id + devices: [{ + os: <string>, // OS string as reported by Services.appinfo.OS, + version: <string>, // Firefox version, as reported by Services.appinfo.version + id: <string>, // Hashed FxA device id for device + }], + + // Internal sync status information. Omitted if it would be empty. + status: { + sync: <string>, // The value of the Status.sync property, unless it indicates success. + service: <string>, // The value of the Status.service property, unless it indicates success. + }, + // Information about each engine's sync. + engines: [ + { + name: <string>, // "bookmarks", "tabs", etc. + took: <integer duration in milliseconds>, // Optional, values of 0 are omitted. + + status: <string>, // The value of Status.engines, if it holds a non-success value. + + // Optional, excluded if all items would be 0. A missing item indicates a value of 0. + incoming: { + applied: <integer>, // Number of records applied + succeeded: <integer>, // Number of records that applied without error + failed: <integer>, // Number of records that failed to apply + newFailed: <integer>, // Number of records that failed for the first time this sync + reconciled: <integer>, // Number of records that were reconciled + }, + + // Optional, excluded if it would be empty. Records that would be + // empty (e.g. 0 sent and 0 failed) are omitted. + outgoing: [ + { + sent: <integer>, // Number of outgoing records sent. Zero values are omitted. + failed: <integer>, // Number that failed to send. Zero values are omitted. + } + ], + // Optional, excluded if there were no errors + failureReason: { ... }, // Same as above. + + // Optional, excluded if it would be empty or if the engine cannot + // or did not run validation on itself. + validation: { + // Optional validator version, default of 0. + version: <integer>, + checked: <integer>, + took: <non-monotonic integer duration in milliseconds>, + // Entries with a count of 0 are excluded, the array is excluded if no problems are found. + problems: [ + { + name: <string>, // The problem identified. + count: <integer>, // Number of times it occurred. + } + ], + // Format is same as above, this is only included if we tried and failed + // to run validation, and if it's present, all other fields in this object are optional. + failureReason: { ... }, + } + } + ] + }] + } + } + +info +---- + +discarded +~~~~~~~~~ + +The ping may only contain a certain number of entries in the ``"syncs"`` array, currently 500 (it is determined by the ``"services.sync.telemetry.maxPayloadCount"`` preference). Entries beyond this are discarded, and recorded in the discarded count. + +syncs.took +~~~~~~~~~~ + +These values should be monotonic. If we can't get a monotonic timestamp, -1 will be reported on the payload, and the values will be omitted from the engines. Additionally, the value will be omitted from an engine if it would be 0 (either due to timer inaccuracy or finishing instantaneously). + +syncs.uid +~~~~~~~~~ + +This property containing a hash of the FxA account identifier, which is a 32 character hexidecimal string. In the case that we are unable to authenticate with FxA and have never authenticated in the past, it will be a placeholder string consisting of 32 repeated ``0`` characters. + +syncs.why +~~~~~~~~~ + +One of the following values: + +- ``startup``: This is the first sync triggered after browser startup. +- ``schedule``: This is a sync triggered because it has been too long since the last sync. +- ``score``: This sync is triggered by a high score value one of sync's trackers, indicating that many changes have occurred since the last sync. +- ``user``: The user manually triggered the sync. +- ``tabs``: The user opened the synced tabs sidebar, which triggers a sync. + +syncs.status +~~~~~~~~~~~~ + +The ``engine.status``, ``payload.status.sync``, and ``payload.status.service`` properties are sync error codes, which are listed in `services/sync/modules/constants.js <https://dxr.mozilla.org/mozilla-central/source/services/sync/modules/constants.js>`_, and success values are not reported. + +syncs.failureReason +~~~~~~~~~~~~~~~~~~~ + +Stores error information, if any is present. Always contains the "name" property, which identifies the type of error it is. The types can be. + +- ``httperror``: Indicates that we recieved an HTTP error response code, but are unable to be more specific about the error. Contains the following properties: + + - ``code``: Integer HTTP status code. + +- ``nserror``: Indicates that an exception with the provided error code caused sync to fail. + + - ``code``: The nsresult error code (integer). + +- ``shutdownerror``: Indicates that the sync failed because we shut down before completion. + +- ``autherror``: Indicates an unrecoverable authentication error. + + - ``from``: Where the authentication error occurred, one of the following values: ``tokenserver``, ``fxaccounts``, or ``hawkclient``. + +- ``othererror``: Indicates that it is a sync error code that we are unable to give more specific information on. As with the ``syncStatus`` property, it is a sync error code, which are listed in `services/sync/modules/constants.js <https://dxr.mozilla.org/mozilla-central/source/services/sync/modules/constants.js>`_. + + - ``error``: String identifying which error was present. + +- ``unexpectederror``: Indicates that some other error caused sync to fail, typically an uncaught exception. + + - ``error``: The message provided by the error. + +- ``sqlerror``: Indicates that we recieved a ``mozIStorageError`` from a database query. + + - ``code``: Value of the ``error.result`` property, one of the constants listed `here <https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Reference/Interface/MozIStorageError#Constants>`_. + +syncs.engine.name +~~~~~~~~~~~~~~~~~ + +Third-party engines are not reported, so only the following values are allowed: ``addons``, ``bookmarks``, ``clients``, ``forms``, ``history``, ``passwords``, ``prefs``, and ``tabs``. + +syncs.engine.validation.problems +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For engines that can run validation on themselves, an array of objects describing validation errors that have occurred. Items that would have a count of 0 are excluded. Each engine will have its own set of items that it might put in the ``name`` field, but there are a finite number. See ``BookmarkProblemData.getSummary`` in `services/sync/modules/bookmark\_validator.js <https://dxr.mozilla.org/mozilla-central/source/services/sync/modules/bookmark_validator.js>`_ for an example. + +syncs.devices +~~~~~~~~~~~~~ + +The list of remote devices associated with this account, as reported by the clients collection. The ID of each device is hashed using the same algorithm as the local id. diff --git a/toolkit/components/telemetry/docs/data/uitour-ping.rst b/toolkit/components/telemetry/docs/data/uitour-ping.rst new file mode 100644 index 000000000..8d17ec55a --- /dev/null +++ b/toolkit/components/telemetry/docs/data/uitour-ping.rst @@ -0,0 +1,26 @@ + +"uitour-tag" ping +================= + +This ping is submitted via the UITour setTreatmentTag API. It may be used by +the tour to record what settings were made by a user or to track the result of +A/B experiments. + +The client ID is submitted with this ping. + +Structure: + +.. code-block:: js + + { + version: 1, + type: "uitour-tag", + clientId: <string>, + payload: { + tagName: <string>, + tagValue: <string> + } + } + +See also: :doc:`common ping fields <common-ping>` + diff --git a/toolkit/components/telemetry/docs/fhr/architecture.rst b/toolkit/components/telemetry/docs/fhr/architecture.rst new file mode 100644 index 000000000..6a857d329 --- /dev/null +++ b/toolkit/components/telemetry/docs/fhr/architecture.rst @@ -0,0 +1,226 @@ +.. _healthreport_architecture: + +============ +Architecture +============ + +``healthreporter.jsm`` contains the main interface for FHR, the +``HealthReporter`` type. An instance of this is created by the +``data_reporting_service`. + +``providers.jsm`` contains numerous ``Metrics.Provider`` and +``Metrics.Measurement`` used for collecting application metrics. If you +are looking for the FHR probes, this is where they are. + +Storage +======= + +Firefox Health Report stores data in 3 locations: + +* Metrics measurements and provider state is stored in a SQLite database + (via ``Metrics.Storage``). +* Service state (such as the IDs of documents uploaded) is stored in a + JSON file on disk (via OS.File). +* Lesser state and run-time options are stored in preferences. + +Preferences +=========== + +Preferences controlling behavior of Firefox Health Report live in the +``datareporting.healthreport.*`` branch. + +Service and Data Control +------------------------ + +The follow preferences control behavior of the service and data upload. + +service.enabled + Controls whether the entire health report service runs. The overall + service performs data collection, storing, and submission. + + This is the primary kill switch for Firefox Health Report + outside of the build system variable. i.e. if you are using an + official Firefox build and wish to disable FHR, this is what you + should set to false to prevent FHR from not only submitting but + also collecting data. + +uploadEnabled + Whether uploading of data is enabled. This is the preference the + checkbox in the preferences UI reflects. If this is + disabled, FHR still collects data - it just doesn't upload it. + +service.loadDelayMsec + How long (in milliseconds) after initial application start should FHR + wait before initializing. + + FHR may initialize sooner than this if the FHR service is requested. + This will happen if e.g. the user goes to ``about:healthreport``. + +service.loadDelayFirstRunMsec + How long (in milliseconds) FHR should wait to initialize on first + application run. + + FHR waits longer than normal to initialize on first application run + because first-time initialization can use a lot of I/O to initialize + the SQLite database and this I/O should not interfere with the + first-run user experience. + +documentServerURI + The URI of a Bagheera server that FHR should interface with for + submitting documents. + + You typically do not need to change this. + +documentServerNamespace + The namespace on the document server FHR should upload documents to. + + You typically do not need to change this. + +infoURL + The URL of a page containing more info about FHR, it's privacy + policy, etc. + +about.reportUrl + The URL to load in ``about:healthreport``. + +about.reportUrlUnified + The URL to load in ``about:healthreport``. This is used instead of ``reportUrl`` for UnifiedTelemetry when it is not opt-in. + +service.providerCategories + A comma-delimited list of category manager categories that contain + registered ``Metrics.Provider`` records. Read below for how provider + registration works. + +If the entire service is disabled, you lose data collection. This means +that **local** data analysis won't be available because there is no data +to analyze! Keep in mind that Firefox Health Report can be useful even +if it's not submitting data to remote servers! + +Logging +------- + +The following preferences allow you to control the logging behavior of +Firefox Health Report. + +logging.consoleEnabled + Whether to write log messages to the web console. This is true by + default. + +logging.consoleLevel + The minimum log level FHR messages must have to be written to the + web console. By default, only FHR warnings or errors will be written + to the web console. During normal/expected operation, no messages of + this type should be produced. + +logging.dumpEnabled + Whether to write log messages via ``dump()``. If true, FHR will write + messages to stdout/stderr. + + This is typically only enabled when developing FHR. + +logging.dumpLevel + The minimum log level messages must have to be written via + ``dump()``. + +State +----- + +currentDaySubmissionFailureCount + How many submission failures the client has encountered while + attempting to upload the most recent document. + +lastDataSubmissionFailureTime + The time of the last failed document upload. + +lastDataSubmissionRequestedTime + The time of the last document upload attempt. + +lastDataSubmissionSuccessfulTime + The time of the last successful document upload. + +nextDataSubmissionTime + The time the next data submission is scheduled for. FHR will not + attempt to upload a new document before this time. + +pendingDeleteRemoteData + Whether the client currently has a pending request to delete remote + data. If true, the client will attempt to delete all remote data + before an upload is performed. + +FHR stores various state in preferences. + +Registering Providers +===================== + +Firefox Health Report providers are registered via the category manager. +See ``HealthReportComponents.manifest`` for providers defined in this +directory. + +Essentially, the category manager receives the name of a JS type and the +URI of a JSM to import that exports this symbol. At run-time, the +providers registered in the category manager are instantiated. + +Providers are registered via the category manager to make registration +simple and less prone to errors. Any XPCOM component can create a +category manager entry. Therefore, new data providers can be added +without having to touch core Firefox Health Report code. Additionally, +category manager registration means providers are more likely to be +registered on FHR's terms, when it wants. If providers were registered +in code at application run-time, there would be the risk of other +components prematurely instantiating FHR (causing a performance hit if +performed at an inopportune time) or semi-complicated code around +observers or listeners. Category manager entries are only 1 line per +provider and leave FHR in control: they are simple and safe. + +Document Generation and Lifecycle +================================= + +FHR will attempt to submit a JSON document containing data every 24 wall +clock hours. + +At upload time, FHR will query the database for **all** information from +the last 180 days and assemble this data into a JSON document. We +attempt to upload this JSON document with a client-generated UUID to the +configured server. + +Before we attempt upload, the generated UUID is stored in the JSON state +file on local disk. At this point, the client assumes the document with +that UUID has been successfully stored on the server. + +If the client is aware of other document UUIDs that presumably exist on +the server, those UUIDs are sent with the upload request so the client +can request those UUIDs be deleted. This helps ensure that each client +only has 1 document/UUID on the server at any one time. + +Importance of Persisting UUIDs +------------------------------ + +The choices of how, where, and when document UUIDs are stored and updated +are very important. One should not attempt to change things unless she +has a very detailed understanding of why things are the way they are. + +The client is purposefully very conservative about forgetting about +generated UUIDs. In other words, once a UUID is generated, the client +deliberately holds on to that UUID until it's very confident that UUID +is no longer stored on the server. The reason we do this is because +*orphaned* documents/UUIDs on the server can lead to faulty analysis, +such as over-reporting the number of Firefox installs that stop being +used. + +When uploading a new UUID, we update the state and save the state file +to disk *before* an upload attempt because if the upload succeeds but +the response never makes it back to the client, we want the client to +know about the uploaded UUID so it can delete it later to prevent an +orphan. + +We maintain a list of UUIDs locally (not simply the last UUID) because +multiple upload attempts could fail the same way as the previous +paragraph describes and we have no way of knowing which (if any) +actually succeeded. The safest approach is to assume every document +produced managed to get uploaded some how. + +We store the UUIDs on a file on disk and not anywhere else because we +want storage to be robust. We originally stored UUIDs in preferences, +which only flush to disk periodically. Writes to preferences were +apparently getting lost. We switched to writing directly to files to +eliminate this window. diff --git a/toolkit/components/telemetry/docs/fhr/dataformat.rst b/toolkit/components/telemetry/docs/fhr/dataformat.rst new file mode 100644 index 000000000..b067f9d0c --- /dev/null +++ b/toolkit/components/telemetry/docs/fhr/dataformat.rst @@ -0,0 +1,1997 @@ +.. _healthreport_dataformat: + +============== +Payload Format +============== + +Currently, the Firefox Health Report is submitted as a compressed JSON +document. The root JSON element is an object. A *version* field defines +the version of the payload which in turn defines the expected contents +the object. + +As of 2013-07-03, desktop submits Version 2, and Firefox for Android submits +Version 3 payloads. + +Version 3 +========= + +Version 3 is a complete rebuild of the document format. Events are tracked in +an "environment". Environments are computed from a large swath of local data +(e.g., add-ons, CPU count, versions), and a new environment comes into being +when one of its attributes changes. + +Client documents, then, will include descriptions of many environments, and +measurements will be attributed to one particular environment. + +A map of environments is present at the top level of the document, with the +current named "current" in the map. Each environment has a hash identifier and +a set of attributes. The current environment is completely described, and has +its hash present in a "hash" attribute. All other environments are represented +as a tree diff from the current environment, with their hash as the key in the +"environments" object. + +A removed add-on has the value 'null'. + +There is no "last" data at present. + +Daily data is hierarchical: by day, then by environment, and then by +measurement, and is present in "data", just as in v2. + +Leading by example:: + + { + "lastPingDate": "2013-06-29", + "thisPingDate": "2013-07-03", + "version": 3, + "environments": { + "current": { + "org.mozilla.sysinfo.sysinfo": { + "memoryMB": 1567, + "cpuCount": 4, + "architecture": "armeabi-v7a", + "_v": 1, + "version": "4.1.2", + "name": "Android" + }, + "org.mozilla.profile.age": { + "_v": 1, + "profileCreation": 15827 + }, + "org.mozilla.addons.active": { + "QuitNow@TWiGSoftware.com": { + "appDisabled": false, + "userDisabled": false, + "scope": 1, + "updateDay": 15885, + "foreignInstall": false, + "hasBinaryComponents": false, + "blocklistState": 0, + "type": "extension", + "installDay": 15885, + "version": "1.18.02" + }, + "{dbbf9331-b713-6eda-1006-205efead09dc}": { + "appDisabled": false, + "userDisabled": "askToActivate", + "scope": 8, + "updateDay": 15779, + "foreignInstall": true, + "blocklistState": 0, + "type": "plugin", + "installDay": 15779, + "version": "11.1 r115" + }, + "desktopbydefault@bnicholson.mozilla.org": { + "appDisabled": false, + "userDisabled": true, + "scope": 1, + "updateDay": 15870, + "foreignInstall": false, + "hasBinaryComponents": false, + "blocklistState": 0, + "type": "extension", + "installDay": 15870, + "version": "1.1" + }, + "{6e092a7f-ba58-4abb-88c1-1a4e50b217e4}": { + "appDisabled": false, + "userDisabled": false, + "scope": 1, + "updateDay": 15828, + "foreignInstall": false, + "hasBinaryComponents": false, + "blocklistState": 0, + "type": "extension", + "installDay": 15828, + "version": "1.1.0" + }, + "{46551EC9-40F0-4e47-8E18-8E5CF550CFB8}": { + "appDisabled": false, + "userDisabled": true, + "scope": 1, + "updateDay": 15879, + "foreignInstall": false, + "hasBinaryComponents": false, + "blocklistState": 0, + "type": "extension", + "installDay": 15879, + "version": "1.3.2" + }, + "_v": 1 + }, + "org.mozilla.appInfo.appinfo": { + "_v": 3, + "appLocale": "en_us", + "osLocale": "en_us", + "distribution": "", + "acceptLangIsUserSet": 0, + "isTelemetryEnabled": 1, + "isBlocklistEnabled": 1 + }, + "geckoAppInfo": { + "updateChannel": "nightly", + "id": "{aa3c5121-dab2-40e2-81ca-7ea25febc110}", + "os": "Android", + "platformBuildID": "20130703031323", + "platformVersion": "25.0a1", + "vendor": "Mozilla", + "name": "fennec", + "xpcomabi": "arm-eabi-gcc3", + "appBuildID": "20130703031323", + "_v": 1, + "version": "25.0a1" + }, + "hash": "tB4Pnnep9yTxnMDymc3dAB2RRB0=", + "org.mozilla.addons.counts": { + "extension": 4, + "plugin": 1, + "_v": 1, + "theme": 0 + } + }, + "k2O3hlreMeS7L1qtxeMsYWxgWWQ=": { + "geckoAppInfo": { + "platformBuildID": "20130630031138", + "appBuildID": "20130630031138", + "_v": 1 + }, + "org.mozilla.appInfo.appinfo": { + "_v": 2, + } + }, + "1+KN9TutMpzdl4TJEl+aCxK+xcw=": { + "geckoAppInfo": { + "platformBuildID": "20130626031100", + "appBuildID": "20130626031100", + "_v": 1 + }, + "org.mozilla.addons.active": { + "QuitNow@TWiGSoftware.com": null, + "{dbbf9331-b713-6eda-1006-205efead09dc}": null, + "desktopbydefault@bnicholson.mozilla.org": null, + "{6e092a7f-ba58-4abb-88c1-1a4e50b217e4}": null, + "{46551EC9-40F0-4e47-8E18-8E5CF550CFB8}": null, + "_v": 1 + }, + "org.mozilla.addons.counts": { + "extension": 0, + "plugin": 0, + "_v": 1 + } + } + }, + "data": { + "last": {}, + "days": { + "2013-07-03": { + "tB4Pnnep9yTxnMDymc3dAB2RRB0=": { + "org.mozilla.appSessions": { + "normal": [ + { + "r": "P", + "d": 2, + "sj": 653 + }, + { + "r": "P", + "d": 22 + }, + { + "r": "P", + "d": 5 + }, + { + "r": "P", + "d": 0 + }, + { + "r": "P", + "sg": 3560, + "d": 171, + "sj": 518 + }, + { + "r": "P", + "d": 16 + }, + { + "r": "P", + "d": 1079 + } + ], + "_v": "4" + } + }, + "k2O3hlreMeS7L1qtxeMsYWxgWWQ=": { + "org.mozilla.appSessions": { + "normal": [ + { + "r": "P", + "d": 27 + }, + { + "r": "P", + "d": 19 + }, + { + "r": "P", + "d": 55 + } + ], + "_v": "4" + }, + "org.mozilla.searches.counts": { + "bartext": { + "google": 1 + }, + "_v": "4" + }, + "org.mozilla.experiment": { + "lastActive": "some.experiment.id" + "_v": "1" + } + } + } + } + } + } + +App sessions in Version 3 +------------------------- + +Sessions are divided into "normal" and "abnormal". Session objects are stored as discrete JSON:: + + "org.mozilla.appSessions": { + _v: 4, + "normal": [ + {"r":"P", "d": 123}, + ], + "abnormal": [ + {"r":"A", "oom": true, "stopped": false} + ] + } + +Keys are: + +"r" + reason. Values are "P" (activity paused), "A" (abnormal termination). +"d" + duration. Value in seconds. +"sg" + Gecko startup time (msec). Present if this is a clean launch. This + corresponds to the telemetry timer *FENNEC_STARTUP_TIME_GECKOREADY*. +"sj" + Java activity init time (msec). Present if this is a clean launch. This + corresponds to the telemetry timer *FENNEC_STARTUP_TIME_JAVAUI*, + and includes initialization tasks beyond initial + *onWindowFocusChanged*. + +Abnormal terminations will be missing a duration and will feature these keys: + +"oom" + was the session killed by an OOM exception? +"stopped" + was the session stopped gently? + +Version 3.2 +----------- + +As of Firefox 35, the search counts measurement is now bumped to v6, including the *activity* location for the search activity. + +Version 3.1 +----------- + +As of Firefox 27, *appinfo* is now bumped to v3, including *osLocale*, +*appLocale* (currently always the same as *osLocale*), *distribution* (a string +containing the distribution ID and version, separated by a colon), and +*acceptLangIsUserSet*, an integer-boolean that describes whether the user set +an *intl.accept_languages* preference. + +The search counts measurement is now at version 5, which indicates that +non-partner searches are recorded. You'll see identifiers like "other-Foo Bar" +rather than "other". + + +Version 3.2 +----------- + +In Firefox 32, Firefox for Android includes a device configuration section +in the environment description:: + + "org.mozilla.device.config": { + "hasHardwareKeyboard": false, + "screenXInMM": 58, + "screenLayout": 2, + "uiType": "default", + "screenYInMM": 103, + "_v": 1, + "uiMode": 1 + } + +Of these, the only keys that need explanation are: + +uiType + One of "default", "smalltablet", "largetablet". +uiMode + A mask of the Android *Configuration.uiMode* value, e.g., + *UI_MODE_TYPE_CAR*. +screenLayout + A mask of the Android *Configuration.screenLayout* value. One of the + *SCREENLAYOUT_SIZE_* constants. + +Note that screen dimensions can be incorrect due to device inaccuracies and platform limitations. + +Other notable differences from Version 2 +---------------------------------------- + +* There is no default browser indicator on Android. +* Add-ons include a *blocklistState* attribute, as returned by AddonManager. +* Searches are now version 4, and are hierarchical: how the search was started + (bartext, barkeyword, barsuggest), and then counts per provider. + +Version 2 +========= + +Version 2 is the same as version 1 with the exception that it has an additional +top-level field, *geckoAppInfo*, which contains basic application info. + +geckoAppInfo +------------ + +This field is an object that is a simple map of string keys and values +describing basic application metadata. It is very similar to the *appinfo* +measurement in the *last* section. The difference is this field is almost +certainly guaranteed to exist whereas the one in the data part of the +payload may be omitted in certain scenarios (such as catastrophic client +error). + +Its keys are as follows: + +appBuildID + The build ID/date of the application. e.g. "20130314113542". + +version + The value of nsXREAppData.version. This is the application's version. e.g. + "21.0.0". + +vendor + The value of nsXREAppData.vendor. Can be empty an empty string. For + official Mozilla builds, this will be "Mozilla". + +name + The value of nsXREAppData.name. For official Firefox builds, this + will be "Firefox". + +id + The value of nsXREAppData.ID. + +platformVersion + The version of the Gecko platform (as opposed to the app version). For + Firefox, this is almost certainly equivalent to the *version* field. + +platformBuildID + The build ID/date of the Gecko platfor (as opposed to the app version). + This is commonly equivalent to *appBuildID*. + +os + The name of the operating system the application is running on. + +xpcomabi + The binary architecture of the build. + +updateChannel + The name of the channel used for application updates. Official Mozilla + builds have one of the values {release, beta, aurora, nightly}. Local and + test builds have *default* as the channel. + +Version 1 +========= + +Top-level Properties +-------------------- + +The main JSON object contains the following properties: + +lastPingDate + UTC date of the last upload. If this is the first upload from this client, + this will not be present. + +thisPingDate + UTC date when this payload was constructed. + +version + Integer version of this payload format. Currently only 1 is defined. + +clientID + An identifier that identifies the client that is submitting data. + + This property may not be present in older clients. + + See :ref:`healthreport_identifiers` for more info on identifiers. + +clientIDVersion + Integer version associated with the generation semantics for the + ``clientID``. + + If the value is ``1``, ``clientID`` is a randomly-generated UUID. + + This property may not be present in older clients. + +data + Object holding data constituting health report. + +Data Properties +--------------- + +The bulk of the health report is contained within the *data* object. This +object has the following keys: + +days + Object mapping UTC days to measurements from that day. Keys are in the + *YYYY-MM-DD* format. e.g. "2013-03-14" + +last + Object mapping measurement names to their values. + + +The value of *days* and *last* are objects mapping measurement names to that +measurement's values. The values are always objects. Each object contains +a *_v* property. This property defines the version of this measurement. +Additional non-underscore-prefixed properties are defined by the measurement +itself (see sections below). + +Example +------- + +Here is an example JSON document for version 1:: + + { + "version": 1, + "thisPingDate": "2013-03-11", + "lastPingDate": "2013-03-10", + "data": { + "last": { + "org.mozilla.addons.active": { + "masspasswordreset@johnathan.nightingale": { + "userDisabled": false, + "appDisabled": false, + "version": "1.05", + "type": "extension", + "scope": 1, + "foreignInstall": false, + "hasBinaryComponents": false, + "installDay": 14973, + "updateDay": 15317 + }, + "places-maintenance@bonardo.net": { + "userDisabled": false, + "appDisabled": false, + "version": "1.3", + "type": "extension", + "scope": 1, + "foreignInstall": false, + "hasBinaryComponents": false, + "installDay": 15268, + "updateDay": 15379 + }, + "_v": 1 + }, + "org.mozilla.appInfo.appinfo": { + "_v": 1, + "appBuildID": "20130309030841", + "distributionID": "", + "distributionVersion": "", + "hotfixVersion": "", + "id": "{ec8030f7-c20a-464f-9b0e-13a3a9e97384}", + "locale": "en-US", + "name": "Firefox", + "os": "Darwin", + "platformBuildID": "20130309030841", + "platformVersion": "22.0a1", + "updateChannel": "nightly", + "vendor": "Mozilla", + "version": "22.0a1", + "xpcomabi": "x86_64-gcc3" + }, + "org.mozilla.profile.age": { + "_v": 1, + "profileCreation": 12444 + }, + "org.mozilla.appSessions.current": { + "_v": 3, + "startDay": 15773, + "activeTicks": 522, + "totalTime": 70858, + "main": 1245, + "firstPaint": 2695, + "sessionRestored": 3436 + }, + "org.mozilla.sysinfo.sysinfo": { + "_v": 1, + "cpuCount": 8, + "memoryMB": 16384, + "architecture": "x86-64", + "name": "Darwin", + "version": "12.2.1" + } + }, + "days": { + "2013-03-11": { + "org.mozilla.addons.counts": { + "_v": 1, + "extension": 15, + "plugin": 12, + "theme": 1 + }, + "org.mozilla.places.places": { + "_v": 1, + "bookmarks": 757, + "pages": 104858 + }, + "org.mozilla.appInfo.appinfo": { + "_v": 1, + "isDefaultBrowser": 1 + } + }, + "2013-03-10": { + "org.mozilla.addons.counts": { + "_v": 1, + "extension": 15, + "plugin": 12, + "theme": 1 + }, + "org.mozilla.places.places": { + "_v": 1, + "bookmarks": 757, + "pages": 104857 + }, + "org.mozilla.searches.counts": { + "_v": 1, + "google.urlbar": 4 + }, + "org.mozilla.appInfo.appinfo": { + "_v": 1, + "isDefaultBrowser": 1 + } + } + } + } + } + +Measurements +============ + +The bulk of payloads consists of measurement data. An individual measurement +is merely a collection of related values e.g. *statistics about the Places +database* or *system information*. + +Each measurement has an integer version number attached. When the fields in +a measurement or the semantics of data within that measurement change, the +version number is incremented. + +All measurements are defined alphabetically in the sections below. + +org.mozilla.addons.addons +------------------------- + +This measurement contains information about the currently-installed add-ons. + +Version 2 +^^^^^^^^^ + +This version adds the human-readable fields *name* and *description*, both +coming directly from the Addon instance as most properties in version 1. +Also, all plugin details are now in org.mozilla.addons.plugins. + +Version 1 +^^^^^^^^^ + +The measurement object is a mapping of add-on IDs to objects containing +add-on metadata. + +Each add-on contains the following properties: + +* userDisabled +* appDisabled +* version +* type +* scope +* foreignInstall +* hasBinaryComponents +* installDay +* updateDay + +With the exception of *installDay* and *updateDay*, all these properties +come direct from the Addon instance. See https://developer.mozilla.org/en-US/docs/Addons/Add-on_Manager/Addon. +*installDay* and *updateDay* are the number of days since UNIX epoch of +the add-ons *installDate* and *updateDate* properties, respectively. + +Notes +^^^^^ + +Add-ons that have opted out of AMO updates via the +*extensions._id_.getAddons.cache.enabled* preference are, since Bug 868306 +(Firefox 24), included in the list of submitted add-ons. + +Example +^^^^^^^ +:: + + "org.mozilla.addons.addons": { + "_v": 2, + "{d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d}": { + "userDisabled": false, + "appDisabled": false, + "name": "Adblock Plus", + "version": "2.4.1", + "type": "extension", + "scope": 1, + "description": "Ads were yesterday!", + "foreignInstall": false, + "hasBinaryComponents": false, + "installDay": 16093, + "updateDay": 16093 + }, + "{e4a8a97b-f2ed-450b-b12d-ee082ba24781}": { + "userDisabled": true, + "appDisabled": false, + "name": "Greasemonkey", + "version": "1.14", + "type": "extension", + "scope": 1, + "description": "A User Script Manager for Firefox", + "foreignInstall": false, + "hasBinaryComponents": false, + "installDay": 16093, + "updateDay": 16093 + } + } + +org.mozilla.addons.plugins +-------------------------- + +This measurement contains information about the currently-installed plugins. + +Version 1 +^^^^^^^^^ + +The measurement object is a mapping of plugin IDs to objects containing +plugin metadata. + +The plugin ID is constructed of the plugins filename, name, version and +description. Every plugin has at least a filename and a name. + +Each plugin contains the following properties: + +* name +* version +* description +* blocklisted +* disabled +* clicktoplay +* mimeTypes +* updateDay + +With the exception of *updateDay* and *mimeTypes*, all these properties come +directly from ``nsIPluginTag`` via ``nsIPluginHost``. +*updateDay* is the number of days since UNIX epoch of the plugins last modified +time. +*mimeTypes* is the list of mimetypes the plugin supports, see +``nsIPluginTag.getMimeTypes()``. + +Example +^^^^^^^ + +:: + + "org.mozilla.addons.plugins": { + "_v": 1, + "Flash Player.plugin:Shockwave Flash:12.0.0.38:Shockwave Flash 12.0 r0": { + "mimeTypes": [ + "application/x-shockwave-flash", + "application/futuresplash" + ], + "name": "Shockwave Flash", + "version": "12.0.0.38", + "description": "Shockwave Flash 12.0 r0", + "blocklisted": false, + "disabled": false, + "clicktoplay": false + }, + "Default Browser.plugin:Default Browser Helper:537:Provides information about the default web browser": { + "mimeTypes": [ + "application/apple-default-browser" + ], + "name": "Default Browser Helper", + "version": "537", + "description": "Provides information about the default web browser", + "blocklisted": false, + "disabled": true, + "clicktoplay": false + } + } + +org.mozilla.addons.counts +------------------------- + +This measurement contains information about historical add-on counts. + +Version 1 +^^^^^^^^^ + +The measurement object consists of counts of different add-on types. The +properties are: + +extension + Integer count of installed extensions. +plugin + Integer count of installed plugins. +theme + Integer count of installed themes. +lwtheme + Integer count of installed lightweigh themes. + +Notes +^^^^^ + +Add-ons opted out of AMO updates are included in the counts. This differs from +the behavior of the active add-ons measurement. + +If no add-ons of a particular type are installed, the property for that type +will not be present (as opposed to an explicit property with value of 0). + +Example +^^^^^^^ + +:: + + "2013-03-14": { + "org.mozilla.addons.counts": { + "_v": 1, + "extension": 21, + "plugin": 4, + "theme": 1 + } + } + + + +org.mozilla.appInfo.appinfo +--------------------------- + +This measurement contains basic XUL application and Gecko platform +information. It is reported in the *last* section. + +Version 2 +^^^^^^^^^ + +In addition to fields present in version 1, this version has the following +fields appearing in the *days* section: + +isBlocklistEnabled + Whether the blocklist ping is enabled. This is an integer, 0 or 1. + This does not indicate whether the blocklist ping was sent but merely + whether the application will try to send the blocklist ping. + +isTelemetryEnabled + Whether Telemetry is enabled. This is an integer, 0 or 1. + +Version 1 +^^^^^^^^^ + +The measurement object contains mostly string values describing the +current application and build. The properties are: + +* vendor +* name +* id +* version +* appBuildID +* platformVersion +* platformBuildID +* os +* xpcomabi +* updateChannel +* distributionID +* distributionVersion +* hotfixVersion +* locale +* isDefaultBrowser + +Notes +^^^^^ + +All of the properties appear in the *last* section except for +*isDefaultBrowser*, which appears under *days*. + +Example +^^^^^^^ + +This example comes from an official OS X Nightly build:: + + "org.mozilla.appInfo.appinfo": { + "_v": 1, + "appBuildID": "20130311030946", + "distributionID": "", + "distributionVersion": "", + "hotfixVersion": "", + "id": "{ec8030f7-c20a-464f-9b0e-13a3a9e97384}", + "locale": "en-US", + "name": "Firefox", + "os": "Darwin", + "platformBuildID": "20130311030946", + "platformVersion": "22.0a1", + "updateChannel": "nightly", + "vendor": "Mozilla", + "version": "22.0a1", + "xpcomabi": "x86_64-gcc3" + }, + +org.mozilla.appInfo.update +-------------------------- + +This measurement contains information about the application update mechanism +in the application. + +Version 1 +^^^^^^^^^ + +The following daily values are reported: + +enabled + Whether automatic application update checking is enabled. 1 for yes, + 0 for no. +autoDownload + Whether automatic download of available updates is enabled. + +Notes +^^^^^ + +This measurement was merged to mozilla-central for JS FHR on 2013-07-15. + +Example +^^^^^^^ + +:: + + "2013-07-15": { + "org.mozilla.appInfo.update": { + "_v": 1, + "enabled": 1, + "autoDownload": 1, + } + } + +org.mozilla.appInfo.versions +---------------------------- + +This measurement contains a history of application version numbers. + +Version 2 +^^^^^^^^^ + +Version 2 reports more fields than version 1 and is not backwards compatible. +The following fields are present in version 2: + +appVersion + An array of application version strings. +appBuildID + An array of application build ID strings. +platformVersion + An array of platform version strings. +platformBuildID + An array of platform build ID strings. + +When the application is upgraded, the new version and/or build IDs are +appended to their appropriate fields. + +Version 1 +^^^^^^^^^ + +When the application version (*version* from *org.mozilla.appinfo.appinfo*) +changes, we record the new version on the day the change was seen. The new +versions for a day are recorded in an array under the *version* property. + +Notes +^^^^^ + +If the application isn't upgraded, this measurement will not be present. +This means this measurement will not be present for most days if a user is +on the release channel (since updates are typically released every 6 weeks). +However, users on the Nightly and Aurora channels will likely have a lot +of these entries since those builds are updated every day. + +Values for this measurement are collected when performing the daily +collection (typically occurs at upload time). As a result, it's possible +the actual upgrade day may not be attributed to the proper day - the +reported day may lag behind. + +The app and platform versions and build IDs should be identical for most +clients. If they are different, we are possibly looking at a *Frankenfox*. + +Example +^^^^^^^ + +:: + + "2013-03-27": { + "org.mozilla.appInfo.versions": { + "_v": 2, + "appVersion": [ + "22.0.0" + ], + "appBuildID": [ + "20130325031100" + ], + "platformVersion": [ + "22.0.0" + ], + "platformBuildID": [ + "20130325031100" + ] + } + } + +org.mozilla.appSessions.current +------------------------------- + +This measurement contains information about the currently running XUL +application's session. + +Version 3 +^^^^^^^^^ + +This measurement has the following properties: + +startDay + Integer days since UNIX epoch when this session began. +activeTicks + Integer count of *ticks* the session was active for. Gecko periodically + sends out a signal when the session is active. Session activity + involves keyboard or mouse interaction with the application. Each tick + represents a window of 5 seconds where there was interaction. +totalTime + Integer seconds the session has been alive. +main + Integer milliseconds it took for the Gecko process to start up. +firstPaint + Integer milliseconds from process start to first paint. +sessionRestored + Integer milliseconds from process start to session restore. + +Example +^^^^^^^ + +:: + + "org.mozilla.appSessions.current": { + "_v": 3, + "startDay": 15775, + "activeTicks": 4282, + "totalTime": 249422, + "main": 851, + "firstPaint": 3271, + "sessionRestored": 5998 + } + +org.mozilla.appSessions.previous +-------------------------------- + +This measurement contains information about previous XUL application sessions. + +Version 3 +^^^^^^^^^ + +This measurement contains per-day lists of all the sessions started on that +day. The following properties may be present: + +cleanActiveTicks + Active ticks of sessions that were properly shut down. +cleanTotalTime + Total number of seconds for sessions that were properly shut down. +abortedActiveTicks + Active ticks of sessions that were not properly shut down. +abortedTotalTime + Total number of seconds for sessions that were not properly shut down. +main + Time in milliseconds from process start to main process initialization. +firstPaint + Time in milliseconds from process start to first paint. +sessionRestored + Time in milliseconds from process start to session restore. + +Notes +^^^^^ + +Sessions are recorded on the date on which they began. + +If a session was aborted/crashed, the total time may be less than the actual +total time. This is because we don't always update total time during periods +of inactivity and the abort/crash could occur after a long period of idle, +before we've updated the total time. + +The lengths of the arrays for {cleanActiveTicks, cleanTotalTime}, +{abortedActiveTicks, abortedTotalTime}, and {main, firstPaint, sessionRestored} +should all be identical. + +The length of the clean sessions plus the length of the aborted sessions should +be equal to the length of the {main, firstPaint, sessionRestored} properties. + +It is not possible to distinguish the main, firstPaint, and sessionRestored +values from a clean vs aborted session: they are all lumped together. + +For sessions spanning multiple UTC days, it's not possible to know which +days the session was active for. It's possible a week long session only +had activity for 2 days and there's no way for us to tell which days. + +Example +^^^^^^^ + +:: + + "org.mozilla.appSessions.previous": { + "_v": 3, + "cleanActiveTicks": [ + 78, + 1785 + ], + "cleanTotalTime": [ + 4472, + 88908 + ], + "main": [ + 32, + 952 + ], + "firstPaint": [ + 2755, + 3497 + ], + "sessionRestored": [ + 5149, + 5520 + ] + } + +org.mozilla.crashes.crashes +--------------------------- + +This measurement contains a historical record of application crashes. + +Version 6 +^^^^^^^^^ + +This version adds tracking for out-of-memory (OOM) crashes in the main process. +An OOM crash will be counted as both main-crash and main-crash-oom. + +This measurement will be reported on each day there was a crash or crash +submission. Records may contain the following fields, whose values indicate +the number of crashes, hangs, or submissions that occurred on the given day: + +* content-crash +* content-crash-submission-succeeded +* content-crash-submission-failed +* content-hang +* content-hang-submission-succeeded +* content-hang-submission-failed +* gmplugin-crash +* gmplugin-crash-submission-succeeded +* gmplugin-crash-submission-failed +* main-crash +* main-crash-oom +* main-crash-submission-succeeded +* main-crash-submission-failed +* main-hang +* main-hang-submission-succeeded +* main-hang-submission-failed +* plugin-crash +* plugin-crash-submission-succeeded +* plugin-crash-submission-failed +* plugin-hang +* plugin-hang-submission-succeeded +* plugin-hang-submission-failed + +Version 5 +^^^^^^^^^ + +This version adds support for Gecko media plugin (GMP) crashes. + +This measurement will be reported on each day there was a crash or crash +submission. Records may contain the following fields, whose values indicate +the number of crashes, hangs, or submissions that occurred on the given day: + +* content-crash +* content-crash-submission-succeeded +* content-crash-submission-failed +* content-hang +* content-hang-submission-succeeded +* content-hang-submission-failed +* gmplugin-crash +* gmplugin-crash-submission-succeeded +* gmplugin-crash-submission-failed +* main-crash +* main-crash-submission-succeeded +* main-crash-submission-failed +* main-hang +* main-hang-submission-succeeded +* main-hang-submission-failed +* plugin-crash +* plugin-crash-submission-succeeded +* plugin-crash-submission-failed +* plugin-hang +* plugin-hang-submission-succeeded +* plugin-hang-submission-failed + +Version 4 +^^^^^^^^^ + +This version follows up from version 3, adding submissions which are now +tracked by the :ref:`crashes_crashmanager`. + +This measurement will be reported on each day there was a crash or crash +submission. Records may contain the following fields, whose values indicate +the number of crashes, hangs, or submissions that occurred on the given day: + +* main-crash +* main-crash-submission-succeeded +* main-crash-submission-failed +* main-hang +* main-hang-submission-succeeded +* main-hang-submission-failed +* content-crash +* content-crash-submission-succeeded +* content-crash-submission-failed +* content-hang +* content-hang-submission-succeeded +* content-hang-submission-failed +* plugin-crash +* plugin-crash-submission-succeeded +* plugin-crash-submission-failed +* plugin-hang +* plugin-hang-submission-succeeded +* plugin-hang-submission-failed + +Version 3 +^^^^^^^^^ + +This version follows up from version 2, building on improvements to +the :ref:`crashes_crashmanager`. + +This measurement will be reported on each day there was a +crash. Records may contain the following fields, whose values indicate +the number of crashes or hangs that occurred on the given day: + +* main-crash +* main-hang +* content-crash +* content-hang +* plugin-crash +* plugin-hang + +Version 2 +^^^^^^^^^ + +The switch to version 2 coincides with the introduction of the +:ref:`crashes_crashmanager`, which provides a more robust source of +crash data. + +This measurement will be reported on each day there was a crash. The +following fields may be present in each record: + +mainCrash + The number of main process crashes that occurred on the given day. + +Yes, version 2 does not track submissions like version 1. It is very +likely submissions will be re-added later. + +Also absent from version 2 are plugin crashes and hangs. These will be +re-added, likely in version 3. + +Version 1 +^^^^^^^^^ + +This measurement will be reported on each day there was a crash. The +following properties are reported: + +pending + The number of crash reports that haven't been submitted. +submitted + The number of crash reports that were submitted. + +Notes +^^^^^ + +Main process crashes are typically submitted immediately after they +occur (by checking a box in the crash reporter, which should appear +automatically after a crash). If the crash reporter submits the crash +successfully, we get a submitted crash. Else, we leave it as pending. + +A pending crash does not mean it will eventually be submitted. + +Pending crash reports can be submitted post-crash by going to +about:crashes. + +If a pending crash is submitted via about:crashes, the submitted count +increments but the pending count does not decrement. This is because FHR +does not know which pending crash was just submitted and therefore it does +not know which day's pending crash to decrement. + +Example +^^^^^^^ + +:: + + "org.mozilla.crashes.crashes": { + "_v": 1, + "pending": 1, + "submitted": 2 + }, + "org.mozilla.crashes.crashes": { + "_v": 2, + "mainCrash": 2 + } + "org.mozilla.crashes.crashes": { + "_v": 4, + "main-crash": 2, + "main-crash-submission-succeeded": 1, + "main-crash-submission-failed": 1, + "main-hang": 1, + "plugin-crash": 2 + } + +org.mozilla.healthreport.submissions +------------------------------------ + +This measurement contains a history of FHR's own data submission activity. +It was added in Firefox 23 in early May 2013. + +Version 2 +^^^^^^^^^ + +This is the same as version 1 except an additional field has been added. + +uploadAlreadyInProgress + A request for upload was initiated while another upload was in progress. + This should not occur in well-behaving clients. It (along with a lock + preventing simultaneous upload) was added to ensure this never occurs. + +Version 1 +^^^^^^^^^ + +Daily counts of upload events are recorded. + +firstDocumentUploadAttempt + An attempt was made to upload the client's first document to the server. + These are uploads where the client is not aware of a previous document ID + on the server. Unless the client had disabled upload, there should be at + most one of these in the history of the client. + +continuationUploadAttempt + An attempt was made to upload a document that replaces an existing document + on the server. Most upload attempts should be attributed to this as opposed + to *firstDocumentUploadAttempt*. + +uploadSuccess + The upload attempt recorded by *firstDocumentUploadAttempt* or + *continuationUploadAttempt* was successful. + +uploadTransportFailure + An upload attempt failed due to transport failure (network unavailable, + etc). + +uploadServerFailure + An upload attempt failed due to a server-reported failure. Ideally these + are failures reported by the FHR server itself. However, intermediate + proxies, firewalls, etc may trigger this depending on how things are + configured. + +uploadClientFailure + An upload attempt failued due to an error/exception in the client. + This almost certainly points to a bug in the client. + +The result for an upload attempt is always attributed to the same day as +the attempt, even if the result occurred on a different day from the attempt. +Therefore, the sum of the result counts should equal the result of the attempt +counts. + +org.mozilla.hotfix.update +------------------------- + +This measurement contains results from the Firefox update hotfix. + +The Firefox update hotfix bypasses the built-in application update mechanism +and installs a modern Firefox. + +Version 1 +^^^^^^^^^ + +The fields in this measurement are dynamically created based on which +versions of the update hotfix state file are found on disk. + +The general format of the fields is ``<version>.<thing>`` where ``version`` +is a hotfix version like ``v20140527`` and ``thing`` is a key from the +hotfix state file, e.g. ``upgradedFrom``. Here are some of the ``things`` +that can be defined. + +upgradedFrom + String identifying the Firefox version that the hotfix upgraded from. + e.g. ``16.0`` or ``17.0.1``. + +uninstallReason + String with enumerated values identifying why the hotfix was uninstalled. + Value will be ``STILL_INSTALLED`` if the hotfix is still installed. + +downloadAttempts + Integer number of times the hotfix started downloading an installer. + Download resumes are part of this count. + +downloadFailures + Integer count of times a download supposedly completed but couldn't + be validated. This likely represents something wrong with the network + connection. The ratio of this to ``downloadAttempts`` should be low. + +installAttempts + Integer count of times the hotfix attempted to run the installer. + This should ideally be 1. It should only be greater than 1 if UAC + elevation was cancelled or not allowed. + +installFailures + Integer count of total installation failures this client experienced. + Can be 0. ``installAttempts - installFailures`` implies install successes. + +notificationsShown + Integer count of times a notification was displayed to the user that + they are running an older Firefox. + +org.mozilla.places.places +------------------------- + +This measurement contains information about the Places database (where Firefox +stores its history and bookmarks). + +Version 1 +^^^^^^^^^ + +Daily counts of items in the database are reported in the following properties: + +bookmarks + Integer count of bookmarks present. +pages + Integer count of pages in the history database. + +Example +^^^^^^^ + +:: + + "org.mozilla.places.places": { + "_v": 1, + "bookmarks": 388, + "pages": 94870 + } + +org.mozilla.profile.age +----------------------- + +This measurement contains information about the current profile's age (and +in version 2, the profile's most recent reset date) + +Version 2 +^^^^^^^^^ + +*profileCreation* and *profileReset* properties are present. Both define +the integer days since UNIX epoch that the current profile was created or +reset accordingly. + +Version 1 +^^^^^^^^^ + +A single *profileCreation* property is present. It defines the integer +days since UNIX epoch that the current profile was created. + +Notes +^^^^^ + +It is somewhat difficult to obtain a reliable *profile born date* due to a +number of factors, but since Version 2, improvements have been made - on a +"profile reset" we copy the profileCreation date from the old profile and +record the time of the reset in profileReset. + +Example +^^^^^^^ + +:: + + "org.mozilla.profile.age": { + "_v": 2, + "profileCreation": 15176 + "profileReset": 15576 + } + +org.mozilla.searches.counts +--------------------------- + +This measurement contains information about searches performed in the +application. + +Version 6 (mobile) +^^^^^^^^^^^^^^^^^^ + +This adds two new search locations: *widget* and *activity*, corresponding to the search widget and search activity respectively. + +Version 2 +^^^^^^^^^ + +This behaves like version 1 except we added all search engines that +Mozilla has a partner agreement with. Like version 1, we concatenate +a search engine ID with a search origin. + +Another difference with version 2 is we should no longer misattribute +a search to the *other* bucket if the search engine name is localized. + +The set of search engine providers is: + +* amazon-co-uk +* amazon-de +* amazon-en-GB +* amazon-france +* amazon-it +* amazon-jp +* amazondotcn +* amazondotcom +* amazondotcom-de +* aol-en-GB +* aol-web-search +* bing +* eBay +* eBay-de +* eBay-en-GB +* eBay-es +* eBay-fi +* eBay-france +* eBay-hu +* eBay-in +* eBay-it +* google +* google-jp +* google-ku +* google-maps-zh-TW +* mailru +* mercadolibre-ar +* mercadolibre-cl +* mercadolibre-mx +* seznam-cz +* twitter +* twitter-de +* twitter-ja +* yahoo +* yahoo-NO +* yahoo-answer-zh-TW +* yahoo-ar +* yahoo-bid-zh-TW +* yahoo-br +* yahoo-ch +* yahoo-cl +* yahoo-de +* yahoo-en-GB +* yahoo-es +* yahoo-fi +* yahoo-france +* yahoo-fy-NL +* yahoo-id +* yahoo-in +* yahoo-it +* yahoo-jp +* yahoo-jp-auctions +* yahoo-mx +* yahoo-sv-SE +* yahoo-zh-TW +* yandex +* yandex-ru +* yandex-slovari +* yandex-tr +* yandex.by +* yandex.ru-be + +And of course, *other*. + +The sources for searches remain: + +* abouthome +* contextmenu +* searchbar +* urlbar + +The measurement will only be populated with providers and sources that +occurred that day. + +If a user switches locales, searches from default providers on the older +locale will still be supported. However, if that same search engine is +added by the user to the new build and is *not* a default search engine +provider, its searches will be attributed to the *other* bucket. + +Version 1 +^^^^^^^^^ + +We record counts of performed searches grouped by search engine and search +origin. Only search engines with which Mozilla has a business relationship +are explicitly counted. All other search engines are grouped into an +*other* bucket. + +The following search engines are explicitly counted: + +* Amazon.com +* Bing +* Google +* Yahoo +* Other + +The following search origins are distinguished: + +about:home + Searches initiated from the search text box on about:home. +context menu + Searches initiated from the context menu (highlight text, right click, + and select "search for...") +search bar + Searches initiated from the search bar (the text field next to the + Awesomebar) +url bar + Searches initiated from the awesomebar/url bar. + +Due to the localization of search engine names, non en-US locales may wrongly +attribute searches to the *other* bucket. This is fixed in version 2. + +Example +^^^^^^^ + +:: + + "org.mozilla.searches.counts": { + "_v": 1, + "google.searchbar": 3, + "google.urlbar": 7 + }, + +org.mozilla.searches.engines +---------------------------- + +This measurement contains information about search engines. + +Version 1 +^^^^^^^^^ + +This version debuted with Firefox 31 on desktop. It contains the +following properties: + +default + Daily string identifier or name of the default search engine provider. + + This field will only be collected if Telemetry is enabled. If + Telemetry is enabled and then later disabled, this field may + disappear from future days in the payload. + + The special value ``NONE`` could occur if there is no default search + engine. + + The special value ``UNDEFINED`` could occur if a default search + engine exists but its identifier could not be determined. + + This field's contents are + ``Services.search.defaultEngine.identifier`` (if defined) or + ``"other-"`` + ``Services.search.defaultEngine.name`` if not. + In other words, search engines without an ``.identifier`` + are prefixed with ``other-``. + +Version 2 +^^^^^^^^^ + +Starting with Firefox 40, there is an additional optional value: + +cohort + Daily cohort string identifier, recorded if the user is part of + search defaults A/B testing. + +org.mozilla.sync.sync +--------------------- + +This daily measurement contains information about the Sync service. + +Values should be recorded for every day FHR measurements occurred. + +Version 1 +^^^^^^^^^ + +This version debuted with Firefox 30 on desktop. It contains the following +properties: + +enabled + Daily numeric indicating whether Sync is configured and enabled. 1 if so, + 0 otherwise. + +preferredProtocol + String version of the maximum Sync protocol version the client supports. + This will be ``1.1`` for for legacy Sync and ``1.5`` for clients that + speak the Firefox Accounts protocol. + +actualProtocol + The actual Sync protocol version the client is configured to use. + + This will be ``1.1`` if the client is configured with the legacy Sync + service or if the client only supports ``1.1``. + + It will be ``1.5`` if the client supports ``1.5`` and either a) the + client is not configured b) the client is using Firefox Accounts Sync. + +syncStart + Count of sync operations performed. + +syncSuccess + Count of sync operations that completed successfully. + +syncError + Count of sync operations that did not complete successfully. + + This is a measure of overall sync success. This does *not* reflect + recoverable errors (such as record conflict) that can occur during + sync. This is thus a rough proxy of whether the sync service is + operating without error. + +org.mozilla.sync.devices +------------------------ + +This daily measurement contains information about the device type composition +for the configured Sync account. + +Version 1 +^^^^^^^^^ + +Version 1 was introduced with Firefox 30. + +Field names are dynamic according to the client-reported device types from +Sync records. All fields are daily last seen integer values corresponding to +the number of devices of that type. + +Common values include: + +desktop + Corresponds to a Firefox desktop client. + +mobile + Corresponds to a Fennec client. + +org.mozilla.sync.migration +-------------------------- + +This daily measurement contains information about sync migration (that is, the +semi-automated process of migrating a legacy sync account to an FxA account.) + +Measurements will start being recorded after a migration is offered by the +sync server and stop after migration is complete or the user elects to "unlink" +their sync account. In other words, it is expected that users with Sync setup +for FxA or with sync unconfigured will not collect data, and that for users +where data is collected, the collection will only be for a relatively short +period. + +Version 1 +^^^^^^^^^ + +Version 1 was introduced with Firefox 37 and includes the following properties: + +state + Corresponds to either a STATE_USER_* string or a STATE_INTERNAL_* string in + FxaMigration.jsm. This reflects a state where we are waiting for the user, + or waiting for some internal process to complete on the way to completing + the migration. + +declined + Corresponds to the number of times the user closed the migration infobar. + +unlinked + Set if the user declined to migrate and instead "unlinked" Sync from the + browser. + +accepted + Corresponds to the number of times the user explicitly elected to start or + continue the migration - it counts how often the user clicked on any UI + created specifically for migration. The "ideal" UX for migration would see + this at exactly 1, some known edge-cases (eg, browser restart required to + finish) could expect this to be 2, and anything more means we are doing + something wrong. + +org.mozilla.sysinfo.sysinfo +--------------------------- + +This measurement contains basic information about the system the application +is running on. + +Version 2 +^^^^^^^^^ + +This version debuted with Firefox 29 on desktop. + +A single property was introduced. + +isWow64 + If present, this property indicates whether the machine supports WoW64. + This property can be used to identify whether the host machine is 64-bit. + + This property is only present on Windows machines. It is the preferred way + to identify 32- vs 64-bit support in that environment. + +Version 1 +^^^^^^^^^ + +The following properties may be available: + +cpuCount + Integer number of CPUs/cores in the machine. +memoryMB + Integer megabytes of memory in the machine. +manufacturer + The manufacturer of the device. +device + The name of the device (like model number). +hardware + Unknown. +name + OS name. +version + OS version. +architecture + OS architecture that the application is built for. This is not the + actual system architecture. + +Example +^^^^^^^ + +:: + + "org.mozilla.sysinfo.sysinfo": { + "_v": 1, + "cpuCount": 8, + "memoryMB": 8192, + "architecture": "x86-64", + "name": "Darwin", + "version": "12.2.0" + } + + +org.mozilla.translation.translation +----------------------------------- + +This daily measurement contains information about the usage of the translation +feature. It is a special telemetry measurement which will only be recorded in +FHR if telemetry is enabled. + +Version 1 +^^^^^^^^^ + +Daily counts are reported in the following properties: + +translationOpportunityCount + Integer count of the number of opportunities there were to translate a page. +missedTranslationOpportunityCount + Integer count of the number of missed opportunities there were to translate a page. + A missed opportunity is when the page language is not supported by the translation + provider. +pageTranslatedCount + Integer count of the number of pages translated. +charactersTranslatedCount + Integer count of the number of characters translated. +detectedLanguageChangedBefore + Integer count of the number of times the user manually adjusted the detected + language before translating. +detectedLanguageChangedAfter + Integer count of the number of times the user manually adjusted the detected + language after having first translated the page. +targetLanguageChanged + Integer count of the number of times the user manually adjusted the target + language. +deniedTranslationOffer + Integer count of the number of times the user opted-out offered + page translation, either by the Not Now button or by the notification's + close button in the "offer" state. +autoRejectedTranlationOffer + Integer count of the number of times the user is not offered page + translation because they had previously clicked "Never translate this + language" or "Never translate this site". +showOriginalContent + Integer count of the number of times the user activated the Show Original + command. + +Additional daily counts broken down by language are reported in the following +properties: + +translationOpportunityCountsByLanguage + A mapping from language to count of opportunities to translate that + language. +missedTranslationOpportunityCountsByLanguage + A mapping from language to count of missed opportunities to translate that + language. +pageTranslatedCountsByLanguage + A mapping from language to the counts of pages translated from that + language. Each language entry will be an object containing a "total" member + along with individual counts for each language translated to. + +Other properties: + +detectLanguageEnabled + Whether automatic language detection is enabled. This is an integer, 0 or 1. +showTranslationUI + Whether the translation feature UI will be shown. This is an integer, 0 or 1. + +Example +^^^^^^^ + +:: + + "org.mozilla.translation.translation": { + "_v": 1, + "detectLanguageEnabled": 1, + "showTranslationUI": 1, + "translationOpportunityCount": 134, + "missedTranslationOpportunityCount": 32, + "pageTranslatedCount": 6, + "charactersTranslatedCount": "1126", + "detectedLanguageChangedBefore": 1, + "detectedLanguageChangedAfter": 2, + "targetLanguageChanged": 0, + "deniedTranslationOffer": 3, + "autoRejectedTranlationOffer": 1, + "showOriginalContent": 2, + "translationOpportunityCountsByLanguage": { + "fr": 100, + "es": 34 + }, + "missedTranslationOpportunityCountsByLanguage": { + "it": 20, + "nl": 10, + "fi": 2 + }, + "pageTranslatedCountsByLanguage": { + "fr": { + "total": 6, + "es": 5, + "en": 1 + } + } + } + + +org.mozilla.experiments.info +---------------------------------- + +Daily measurement reporting information about the Telemetry Experiments service. + +Version 1 +^^^^^^^^^ + +Property: + +lastActive + ID of the final Telemetry Experiment that is active on a given day, if any. + + +Version 2 +^^^^^^^^^ + +Adds an additional optional property: + +lastActiveBranch + If the experiment uses branches, the branch identifier string. + +Example +^^^^^^^ + +:: + + "org.mozilla.experiments.info": { + "_v": 2, + "lastActive": "some.experiment.id", + "lastActiveBranch": "control" + } + +org.mozilla.uitour.treatment +---------------------------- + +Daily measurement reporting information about treatment tagging done +by the UITour module. + +Version 1 +^^^^^^^^^ + +Daily text values in the following properties: + +<tag>: + Array of discrete strings corresponding to calls for setTreatmentTag(tag, value). + +Example +^^^^^^^ + +:: + + "org.mozilla.uitour.treatment": { + "_v": 1, + "treatment": [ + "optin", + "optin-DNT" + ], + "another-tag": [ + "foobar-value" + ] + } + +org.mozilla.passwordmgr.passwordmgr +----------------------------------- + +Daily measurement reporting information about the Password Manager + +Version 1 +^^^^^^^^^ + +Property: + +numSavedPasswords + number of passwords saved in the Password Manager + +enabled + Whether or not the user has disabled the Password Manager in prefernces + +Example +^^^^^^^ + +:: + + "org.mozilla.passwordmgr.passwordmgr": { + "_v": 1, + "numSavedPasswords": 5, + "enabled": 0, + } + +Version 2 +^^^^^^^^^ + +More detailed measurements of login forms & their behavior + +numNewSavedPasswordsInSession + Number of passwords saved to the password manager this session. + +numSuccessfulFills + Number of times the password manager filled in password fields for user this session. + +numTotalLoginsEncountered + Number of times a login form was encountered by the user in the session. + +Example +^^^^^^^ + +:: + "org.mozilla.passwordmgr.passwordmgr": { + "_v": 2, + "numSavedPasswords": 32, + "enabled": 1, + "numNewSavedPasswords": 5, + "numSuccessfulFills": 11, + "numTotalLoginsEncountered": 23, + } diff --git a/toolkit/components/telemetry/docs/fhr/identifiers.rst b/toolkit/components/telemetry/docs/fhr/identifiers.rst new file mode 100644 index 000000000..82ad0e49e --- /dev/null +++ b/toolkit/components/telemetry/docs/fhr/identifiers.rst @@ -0,0 +1,83 @@ +.. _healthreport_identifiers: + +=========== +Identifiers +=========== + +Firefox Health Report records some identifiers to keep track of clients +and uploaded documents. + +Identifier Types +================ + +Document/Upload IDs +------------------- + +A random UUID called the *Document ID* or *Upload ID* is generated when the FHR +client creates or uploads a new document. + +When clients generate a new *Document ID*, they persist this ID to disk +**before** the upload attempt. + +As part of the upload, the client sends all old *Document IDs* to the server +and asks the server to delete them. In well-behaving clients, the server +has a single record for each client with a randomly-changing *Document ID*. + +Client IDs +---------- + +A *Client ID* is an identifier that **attempts** to uniquely identify an +individual FHR client. Please note the emphasis on *attempts* in that last +sentence: *Client IDs* do not guarantee uniqueness. + +The *Client ID* is generated when the client first runs or as needed. + +The *Client ID* is transferred to the server as part of every upload. The +server is thus able to affiliate multiple document uploads with a single +*Client ID*. + +Client ID Versions +^^^^^^^^^^^^^^^^^^ + +The semantics for how a *Client ID* is generated are versioned. + +Version 1 + The *Client ID* is a randomly-generated UUID. + +History of Identifiers +====================== + +In the beginning, there were just *Document IDs*. The thinking was clients +would clean up after themselves and leave at most 1 active document on the +server. + +Unfortunately, this did not work out. Using brute force analysis to +deduplicate records on the server, a number of interesting patterns emerged. + +Orphaning + Clients would upload a new payload while not deleting the old payload. + +Divergent records + Records would share data up to a certain date and then the data would + almost completely diverge. This appears to be indicative of profile + copying. + +Rollback + Records would share data up to a certain date. Each record in this set + would contain data for a day or two but no extra data. This could be + explained by filesystem rollback on the client. + +A significant percentage of the records on the server belonged to +misbehaving clients. Identifying these records was extremely resource +intensive and error-prone. These records were undermining the ability +to use Firefox Health Report data. + +Thus, the *Client ID* was born. The intent of the *Client ID* was to +uniquely identify clients so the extreme effort required and the +questionable reliability of deduplicating server data would become +problems of the past. + +The *Client ID* was originally a randomly-generated UUID (version 1). This +allowed detection of orphaning and rollback. However, these version 1 +*Client IDs* were still susceptible to use on multiple profiles and +machines if the profile was copied. diff --git a/toolkit/components/telemetry/docs/fhr/index.rst b/toolkit/components/telemetry/docs/fhr/index.rst new file mode 100644 index 000000000..497385dd8 --- /dev/null +++ b/toolkit/components/telemetry/docs/fhr/index.rst @@ -0,0 +1,34 @@ +================================ +Firefox Health Report (Obsolete) +================================ + +**Firefox Health Report (FHR) is obsolete and no longer ships with Firefox. +This documentation will live here for a few more cycles.** + +Firefox Health Report is a background service that collects application +metrics and periodically submits them to a central server. The core +parts of the service are implemented in this directory. However, the +actual XPCOM service is implemented in the +``data_reporting_service`. + +The core types can actually be instantiated multiple times and used to +power multiple data submission services within a single Gecko +application. In other words, everything in this directory is effectively +a reusable library. However, the terminology and some of the features +are very specific to what the Firefox Health Report feature requires. + +.. toctree:: + :maxdepth: 1 + + architecture + dataformat + identifiers + +Legal and Privacy Concerns +========================== + +Because Firefox Health Report collects and submits data to remote +servers and is an opt-out feature, there are legal and privacy +concerns over what data may be collected and submitted. **Additions or +changes to submitted data should be signed off by responsible +parties.** diff --git a/toolkit/components/telemetry/docs/index.rst b/toolkit/components/telemetry/docs/index.rst new file mode 100644 index 000000000..5d30c5e92 --- /dev/null +++ b/toolkit/components/telemetry/docs/index.rst @@ -0,0 +1,25 @@ +.. _telemetry: + +========= +Telemetry +========= + +Telemetry is a feature that allows data collection. This is being used to collect performance metrics and other information about how Firefox performs in the wild. + +Client-side, this consists of: + +* data collection in `Histograms <https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Adding_a_new_Telemetry_probe>`_, :doc:`collection/scalars` and other data structures +* assembling :doc:`concepts/pings` with the general information and the data payload +* sending them to the server and local ping retention + +*Note:* the `data collection policy <https://wiki.mozilla.org/Firefox/Data_Collection>`_ documents the process and requirements that are applied here. + +.. toctree:: + :maxdepth: 5 + :titlesonly: + + concepts/index + collection/index + data/index + internals/index + fhr/index diff --git a/toolkit/components/telemetry/docs/internals/index.rst b/toolkit/components/telemetry/docs/internals/index.rst new file mode 100644 index 000000000..e912ea49a --- /dev/null +++ b/toolkit/components/telemetry/docs/internals/index.rst @@ -0,0 +1,9 @@ +========= +Internals +========= + +.. toctree:: + :maxdepth: 2 + :titlesonly: + + preferences diff --git a/toolkit/components/telemetry/docs/internals/preferences.rst b/toolkit/components/telemetry/docs/internals/preferences.rst new file mode 100644 index 000000000..c8af2f2d5 --- /dev/null +++ b/toolkit/components/telemetry/docs/internals/preferences.rst @@ -0,0 +1,119 @@ +Preferences +=========== + +Telemetry behaviour is controlled through the preferences listed here. + +Default behaviors +----------------- + +Sending only happens on official builds (i.e. with ``MOZILLA_OFFICIAL`` set) with ``MOZ_TELEMETRY_REPORTING`` defined. +All other builds drop all outgoing pings, so they will also not retry sending them later. + +Preferences +----------- + +``toolkit.telemetry.unified`` + + This controls whether unified behavior is enabled. If true: + + * Telemetry is always enabled and recording *base* data. + * Telemetry will send additional ``main`` pings. + +``toolkit.telemetry.enabled`` + + If ``unified`` is off, this controls whether the Telemetry module is enabled. + If ``unified`` is on, this controls whether to record *extended* data. + This preference is controlled through the `Preferences` dialog. + + Note that the default value here of this pref depends on the define ``RELEASE_OR_BETA`` and the channel. + If ``RELEASE_OR_BETA`` is set, ``MOZ_TELEMETRY_ON_BY_DEFAULT`` gets set, which means this pref will default to ``true``. + This is overridden by the preferences code on the "beta" channel, the pref also defaults to ``true`` there. + +``datareporting.healthreport.uploadEnabled`` + + Send the data we record if user has consented to FHR. This preference is controlled through the `Preferences` dialog. + +``toolkit.telemetry.archive.enabled`` + + Allow pings to be archived locally. This can only be enabled if ``unified`` is on. + +``toolkit.telemetry.server`` + + The server Telemetry pings are sent to. + +``toolkit.telemetry.log.level`` + + This sets the Telemetry logging verbosity per ``Log.jsm``, with ``Trace`` or ``0`` being the most verbose and the default being ``Warn``. + By default logging goes only the console service. + +``toolkit.telemetry.log.dump`` + + Sets whether to dump Telemetry log messages to ``stdout`` too. + +Data-choices notification +------------------------- + +``toolkit.telemetry.reportingpolicy.firstRun`` + + This preference is not present until the first run. After, its value is set to false. This is used to show the infobar with a more aggressive timeout if it wasn't shown yet. + +``datareporting.policy.firstRunURL`` + + If set, a browser tab will be opened on first run instead of the infobar. + +``datareporting.policy.dataSubmissionEnabled`` + + This is the data submission master kill switch. If disabled, no policy is shown or upload takes place, ever. + +``datareporting.policy.dataSubmissionPolicyNotifiedTime`` + + Records the date user was shown the policy. This preference is also used on Android. + +``datareporting.policy.dataSubmissionPolicyAcceptedVersion`` + + Records the version of the policy notified to the user. This preference is also used on Android. + +``datareporting.policy.dataSubmissionPolicyBypassNotification`` + + Used in tests, it allows to skip the notification check. + +``datareporting.policy.currentPolicyVersion`` + + Stores the current policy version, overrides the default value defined in TelemetryReportingPolicy.jsm. + +``datareporting.policy.minimumPolicyVersion`` + + The minimum policy version that is accepted for the current policy. This can be set per channel. + +``datareporting.policy.minimumPolicyVersion.channel-NAME`` + + This is the only channel-specific version that we currently use for the minimum policy version. + +Testing +------- + +The following prefs are for testing purpose only. + +``toolkit.telemetry.initDelay`` + + Delay before initializing telemetry (seconds). + +``toolkit.telemetry.minSubsessionLength`` + + Minimum length of a telemetry subsession (seconds). + +``toolkit.telemetry.collectInterval`` + + Minimum interval between data collection (seconds). + +``toolkit.telemetry.scheduler.tickInterval`` + + Interval between scheduler ticks (seconds). + +``toolkit.telemetry.scheduler.idleTickInterval`` + + Interval between scheduler ticks when the user is idle (seconds). + +``toolkit.telemetry.idleTimeout`` + + Timeout until we decide whether a user is idle or not (seconds). |
