path: root/toolkit/components/telemetry/docs/data/main-ping.rst

"main" ping
===========

.. toctree::
   :maxdepth: 2

This is the "main" Telemetry ping type, whose payload contains most of the measurements that are used to track the performance and health of Firefox in the wild.
It includes the histograms and other performance and diagnostic data.

This ping is triggered by different scenarios, which is documented by the ``reason`` field:

* ``aborted-session`` - this ping is regularly saved to disk (every 5 minutes), overwriting itself, and deleted at shutdown. If a previous aborted session ping is found at startup, it gets sent to the server. The first aborted-session ping is generated as soon as Telemetry starts
* ``environment-change`` - the :doc:`environment` changed, so the session measurements got reset and a new subsession starts
* ``shutdown`` - triggered when the browser session ends
* ``daily`` - a session split triggered in 24h hour intervals at local midnight. If an ``environment-change`` ping is generated by the time it should be sent, the daily ping is rescheduled for the next midnight
* ``saved-session`` - the *"classic"* Telemetry payload with measurements covering the whole browser session (only submitted for a transition period)

Most reasons lead to a session split, initiating a new *subsession*. We reset important measurements for those subsessions.

After a new subsession split, the ``internal-telemetry-after-subsession-split`` topic is notified to all the observers. *This is an internal topic and is only meant for internal Telemetry usage.*

*Note:* ``saved-session`` is sent with a different ping type (``saved-session``, not ``main``), but otherwise has the same format as discussed here.

Structure:

.. code-block:: js

    {
      version: 4,

      info: {
        reason: <string>, // what triggered this ping: "saved-session", "environment-change", "shutdown", ...
        revision: <string>, // the Histograms.json revision
        timezoneOffset: <integer>, // time-zone offset from UTC, in minutes, for the current locale
        previousBuildId: <string>, // null if this is the first run, or the previous build ID is unknown

        sessionId: <uuid>,  // random session id, shared by subsessions
        subsessionId: <uuid>,  // random subsession id
        previousSessionId: <uuid>, // session id of the previous session, null on first run.
        previousSubsessionId: <uuid>, // subsession id of the previous subsession (even if it was in a different session),
                                      // null on first run.

        subsessionCounter: <unsigned integer>, // the running no. of this subsession since the start of the browser session
        profileSubsessionCounter: <unsigned integer>, // the running no. of all subsessions for the whole profile life time

        sessionStartDate: <ISO date>, // daily precision
        subsessionStartDate: <ISO date>, // daily precision, ISO date in local time
        sessionLength: <integer>, // the session length until now in seconds, monotonic
        subsessionLength: <integer>, // the subsession length in seconds, monotonic

        flashVersion: <string>, // obsolete, use ``environment.addons.activePlugins``
        addons: <string>, // obsolete, use ``environment.addons``
      },

      processes: {...},
      childPayloads: [...], // only present with e10s; reduced payloads from content processes, null on failure
      simpleMeasurements: {...},

      // The following properties may all be null if we fail to collect them.
      histograms: {...},
      keyedHistograms: {...},
      chromeHangs: {...},
      threadHangStats: [...],
      log: [...],
      webrtc: {...},
      gc: {...},
      fileIOReports: {...},
      lateWrites: {...},
      addonDetails: {...},
      addonHistograms: {...},
      UIMeasurements: [...],
      slowSQL: {...},
      slowSQLstartup: {...},
    }

info
----

sessionLength
~~~~~~~~~~~~~
The length of the current session so far in seconds.
This uses a monotonic clock, so this may mismatch with other measurements that
are not monotonic like calculations based on ``Date.now()``.

If the monotonic clock failed, this will be ``-1``.

subsessionLength
~~~~~~~~~~~~~~~~
The length of this subsession in seconds.
This uses a monotonic clock, so this may mismatch with other measurements that are not monotonic (e.g. based on Date.now()).

If ``sessionLength`` is ``-1``, the monotonic clock is not working.

processes
---------
This section contains per-process data.

Structure:

.. code-block:: js

    "processes" : {
      ... other processes ...
      "parent": {
        scalars: {...},
      },
      "content": {
        histograms: {...},
        keyedHistograms: {...},
      },
    }

histograms and keyedHistograms
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This section contains histograms and keyed histograms accumulated on content processes. Histograms recorded on a content child process have different character than parent histograms. For instance, ``GC_MS`` will be much different in ``processes.content`` as it has to contend with web content, whereas the instance in ``payload.histograms`` has only to contend with browser JS. Also, some histograms may be absent if never recorded on a content child process (``EVENTLOOP_UI_ACTIVITY`` is parent-process-only).

This format was adopted in Firefox 51 via bug 1218576.

scalars
~~~~~~~
This section contains the :doc:`../collection/scalars` that are valid for the current platform. Scalars are not created nor submitted if no data was added to them, and are only reported with subsession pings. Scalar data is only currently reported for the main process. Their type and format is described by the ``Scalars.yaml`` file. Its most recent version is available `here <https://dxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/Scalars.yaml>`_. The ``info.revision`` field indicates the revision of the file that describes the reported scalars.

childPayloads
-------------
The Telemetry payloads sent by child processes, recorded on child process shutdown (event ``content-child-shutdown`` observed). They are reduced session payloads, only available with e10s. Among some other things, they don't contain histograms, keyed histograms, addon details, addon histograms, or UI Telemetry.

Note: Child payloads are not collected and cleared with subsession splits, they are currently only meaningful when analysed from ``saved-session`` or ``main`` pings with ``reason`` set to ``shutdown``.

Note: Before Firefox 51 and bug 1218576, content process histograms and keyedHistograms were in the individual child payloads instead of being aggregated into ``processes.content``.

simpleMeasurements
------------------
This section contains a list of simple measurements, or counters. In addition to the ones highlighted below, Telemetry timestamps (see `here <https://dxr.mozilla.org/mozilla-central/search?q=%22TelemetryTimestamps.add%22&redirect=false&case=true>`_ and `here <https://dxr.mozilla.org/mozilla-central/search?q=%22recordTimestamp%22&redirect=false&case=true>`_) can be reported.

totalTime
~~~~~~~~~
A non-monotonic integer representing the number of seconds the session has been alive.

uptime
~~~~~~
A non-monotonic integer representing the number of minutes the session has been alive.

addonManager
~~~~~~~~~~~~
Only available in the extended set of measures, it contains a set of counters related to Addons. See `here <https://dxr.mozilla.org/mozilla-central/search?q=%22AddonManagerPrivate.recordSimpleMeasure%22&redirect=false&case=true>`_ for a list of recorded measures.

UITelemetry
~~~~~~~~~~~
Only available in the extended set of measures. For more see :ref:`uitelemetry`.

startupInterrupted
~~~~~~~~~~~~~~~~~~
A boolean set to true if startup was interrupted by an interactive prompt.

js
~~
This section contains a series of counters from the JavaScript engine.

Structure:

.. code-block:: js

    "js" : {
      "setProto": <unsigned integer>, // Number of times __proto__ is set
      "customIter": <unsigned integer> // Number of times __iterator__ is used (i.e., is found for a for-in loop)
    }

maximalNumberOfConcurrentThreads
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
An integer representing the highest number of threads encountered so far during the session.

startupSessionRestoreReadBytes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Windows-only integer representing the number of bytes read by the main process up until the session store has finished restoring the windows.

startupSessionRestoreWriteBytes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Windows-only integer representing the number of bytes written by the main process up until the session store has finished restoring the windows.

startupWindowVisibleReadBytes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Windows-only integer representing the number of bytes read by the main process up until after a XUL window is made visible.

startupWindowVisibleWriteBytes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Windows-only integer representing the number of bytes written by the main process up until after a XUL window is made visible.

debuggerAttached
~~~~~~~~~~~~~~~~
A boolean set to true if a debugger is attached to the main process.

shutdownDuration
~~~~~~~~~~~~~~~~
The time, in milliseconds, it took to complete the last shutdown.

failedProfileLockCount
~~~~~~~~~~~~~~~~~~~~~~
The number of times the system failed to lock the user profile.

savedPings
~~~~~~~~~~
Integer count of the number of pings that need to be sent.

activeTicks
~~~~~~~~~~~
Integer count of the number of five-second intervals ('ticks') the user was considered 'active' (sending UI events to the window). An extra event is fired immediately when the user becomes active after being inactive. This is for some mouse and gamepad events, and all touch, keyboard, wheel, and pointer events (see `EventStateManager.cpp <https://dxr.mozilla.org/mozilla-central/rev/e6463ae7eda2775bc84593bb4a0742940bb87379/dom/events/EventStateManager.cpp#549>`_).
This measure might be useful to give a trend of how much a user actually interacts with the browser when compared to overall session duration. It does not take into account whether or not the window has focus or is in the foreground. Just if it is receiving these interaction events.
Note that in ``main`` pings, this measure is reset on subsession splits, while in ``saved-session`` pings it covers the whole browser session.

pingsOverdue
~~~~~~~~~~~~
Integer count of pending pings that are overdue.

histograms
----------
This section contains the histograms that are valid for the current platform. ``Flag`` and ``count`` histograms are always created and submitted, with their default value being respectively ``false`` and ``0``. Other histogram types (`see here <https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Adding_a_new_Telemetry_probe#Choosing_a_Histogram_Type>`_) are not created nor submitted if no data was added to them. The type and format of the reported histograms is described by the ``Histograms.json`` file. Its most recent version is available `here <https://dxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/Histograms.json>`_. The ``info.revision`` field indicates the revision of the file that describes the reported histograms.

keyedHistograms
---------------
This section contains the keyed histograms available for the current platform.

As of Firefox 48, this section does not contain empty keyed histograms anymore.

threadHangStats
---------------
Contains the statistics about the hangs in main and background threads. Note that hangs in this section capture the [C++ pseudostack](https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Profiling_with_the_Built-in_Profiler#Native_stack_vs._Pseudo_stack) and an incomplete JS stack, which is not 100% precise.

To avoid submitting overly large payloads, some limits are applied:

* Identical, adjacent "(chrome script)" or "(content script)" stack entries are collapsed together. If a stack is reduced, the "(reduced stack)" frame marker is added as the oldest frame.
* The depth of the reported stacks is limited to 11 entries. This value represents the 99.9th percentile of the thread hangs stack depths reported by Telemetry.

Structure:

.. code-block:: js

    "threadHangStats" : [
      {
        "name" : "Gecko",
        "activity" : {...}, // a time histogram of all task run times
        "hangs" : [
          {
            "stack" : [
              "Startup::XRE_Main",
              "Timer::Fire",
              "(content script)",
              "IPDL::PPluginScriptableObject::SendGetChildProperty",
              ... up to 11 frames ...
            ],
            "nativeStack": [...], // optionally available
            "histogram" : {...}, // the time histogram of the hang times
            "annotations" : [
              {
                "pluginName" : "Shockwave Flash",
                "pluginVersion" : "18.0.0.209"
              },
              ... other annotations ...
            ]
          },
        ],
      },
      ... other threads ...
     ]

chromeHangs
-----------
Contains the statistics about the hangs happening exclusively on the main thread of the parent process. Precise C++ stacks are reported. This is only available on Nightly Release on Windows, when building using "--enable-profiling" switch.

Some limits are applied:

* Reported chrome hang stacks are limited in depth to 50 entries.
* The maximum number of reported stacks is 50.

Structure:

.. code-block:: js

    "chromeHangs" : {
      "memoryMap" : [
        ["wgdi32.pdb", "08A541B5942242BDB4AEABD8C87E4CFF2"],
        ["igd10iumd32.pdb", "D36DEBF2E78149B5BE1856B772F1C3991"],
        ... other entries in the format ["module name", "breakpad identifier"] ...
       ],
      "stacks" : [
        [
          [
            0, // the module index or -1 for invalid module indices
            190649 // the offset of this program counter in its module or an absolute pc
          ],
          [1, 2540075],
          ... other frames, up to 50 ...
         ],
         ... other stacks, up to 50 ...
      ],
      "durations" : [8, ...], // the hang durations (in seconds)
      "systemUptime" : [692, ...], // the system uptime (in minutes) at the time of the hang
      "firefoxUptime" : [672, ...], // the Firefox uptime (in minutes) at the time of the hang
      "annotations" : [
        [
          [0, ...], // the indices of the related hangs
          {
            "pluginName" : "Shockwave Flash",
            "pluginVersion" : "18.0.0.209",
            ... other annotations as key:value pairs ...
          }
        ],
        ...
      ]
    },

log
---
This section contains a log of important or unusual events reported through Telemetry.

Structure:

.. code-block:: js

    "log": [
      [
        "Event_ID",
        3785, // the timestamp (in milliseconds) for the log entry
        ... other data ...
      ],
      ...
    ]


webrtc
------
Contains special statistics gathered by WebRTC related components.

So far only a bitmask for the ICE candidate type present in a successful or
failed WebRTC connection is getting reported through C++ code as
IceCandidatesStats, because the required bitmask is too big to be represented
in a regular enum histogram. Further this data differentiates between Loop
(aka Firefox Hello) connections and everything else, which is categorized as
WebRTC.

Note: in most cases the webrtc and loop dictionaries inside of
IceCandidatesStats will simply be empty as the user has not used any WebRTC
PeerConnection at all during the ping report time.

Structure:

.. code-block:: js

    "webrtc": {
      "IceCandidatesStats": {
        "webrtc": {
          "34526345": {
            "successCount": 5
          },
          "2354353": {
            "failureCount": 1
          }
        },
        "loop": {
          "2349346359": {
            "successCount": 3
          },
          "73424": {
            "successCount": 1,
            "failureCount": 5
          }
        }
      }
    },

gc
--
Contains statistics about selected garbage collections. To avoid
bloating the ping, only a few GCs are included. There are two
selection strategies. We always save the two GCs with the worst
max_pause time. Additionally, in content processes, two collections
are selected at random. If a GC runs for C milliseconds and the total
time for all GCs since the session began is T milliseconds, then the
GC has a C/T probablility of being selected for one of these "slots".

Structure:

.. code-block:: js

    "gc": {
      "random": [
        {
          // Timestamps are in milliseconds since startup. All the times here
          // are wall-clock times, which may not be monotonically increasing.
          "timestamp": 294872.2,
          // All durations are in milliseconds.
          "max_pause": 73.629,
          "total_time": 364.951, // Sum of all slice times.
          "zones_collected": 9,
          "total_zones": 9,
          "total_compartments": 309,
          "minor_gcs": 44,
          "store_buffer_overflows": 19,
          "mmu_20ms": 0,
          "mmu_50ms": 0,
          // Reasons include "None", "NonIncrementalRequested",
          // "AbortRequested", "KeepAtomsSet", "IncrementalDisabled",
          // "ModeChange", "MallocBytesTrigger", "GCBytesTrigger",
          // "ZoneChange".
          "nonincremental_reason": "None",
          "allocated": 37, // In megabytes.
          "added_chunks": 54,
          "removed_chunks": 12,
          // Total number of slices (some of which may not appear
          // in the "slices" array).
          "num_slices": 15,
          // We record at most 4 slices.
          "slices": [
            {
              "slice": 0,  // The index of this slice.
              "pause": 23.221,  // How long the slice took.
              "when": 0,  // Milliseconds since the start of the GC.
              "reason": "SET_NEW_DOCUMENT",
              // GC state when the slice started
              "initial_state": "NotActive",
              // GC state when the slice ended
              "final_state": "Mark",
              // Budget is either "Xms", "work(Y)", or
              // "unlimited".
              "budget": "10ms",
              // Number of page faults during the slice.
              "page_faults": 0,
              "start_timestamp": 294875,
              "end_timestamp": 294879,
              // Time taken by each phase. There are at most 65 possible
              // phases, but usually only a few phases run in a given slice.
              "times": {
                "wait_background_thread": 0.012,
                "mark_discard_code": 2.845,
                "purge": 0.723,
                "mark": 9.831,
                "mark_roots": 0.102,
                "buffer_gray_roots": 3.095,
                "mark_cross_compartment_wrappers": 0.039,
                "mark_c_and_js_stacks": 0.005,
                "mark_runtime_wide_data": 2.313,
                "mark_embedding": 0.117,
                "mark_compartments": 2.27,
                "unmark": 1.063,
                "minor_gcs_to_evict_nursery": 8.701,
                ...
              }
            },
            { ... },
          ],
          // Sum of the phase times across all slices, including
          // omitted slices. As before, there are <= 65 possible phases.
          "totals": {
            "wait_background_thread": 0.012,
            "mark_discard_code": 2.845,
            "purge": 0.723,
            "mark": 9.831,
            "mark_roots": 0.102,
            "buffer_gray_roots": 3.095,
            "mark_cross_compartment_wrappers": 0.039,
            "mark_c_and_js_stacks": 0.005,
            "mark_runtime_wide_data": 2.313,
            "mark_embedding": 0.117,
            "mark_compartments": 2.27,
            "unmark": 1.063,
            "minor_gcs_to_evict_nursery": 8.701,
            ...
          }
        },
        ... // Up to four more selected GCs follow.
      ],
      "worst": [
        ... // Same as above, but the 2 worst GCs by max_pause.
      ]
    },

fileIOReports
-------------
Contains the statistics of main-thread I/O recorded during the execution. Only the I/O stats for the XRE and the profile directories are currently reported, neither of them disclosing the full local path.

Structure:

.. code-block:: js

    "fileIOReports": {
      "{xre}": [
        totalTime, // Accumulated duration of all operations
        creates, // Number of create/open operations
        reads, // Number of read operations
        writes, // Number of write operations
        fsyncs, // Number of fsync operations
        stats, // Number of stat operations
      ],
      "{profile}": [ ... ],
      ...
    }

lateWrites
----------
This sections reports writes to the file system that happen during shutdown. The reported data contains the stack and the loaded libraries at the time the writes happened.

Structure:

.. code-block:: js

    "lateWrites" : {
      "memoryMap" : [
        ["wgdi32.pdb", "08A541B5942242BDB4AEABD8C87E4CFF2"],
        ... other entries in the format ["module name", "breakpad identifier"] ...
       ],
      "stacks" : [
        [
          [
            0, // the module index or -1 for invalid module indices
            190649 // the offset of this program counter in its module or an absolute pc
          ],
          [1, 2540075],
          ... other frames ...
         ],
         ... other stacks ...
      ],
    },

addonDetails
------------
This section contains per-addon telemetry details, as reported by each addon provider. The XPI provider is the only one reporting at the time of writing (`see DXR <https://dxr.mozilla.org/mozilla-central/search?q=setTelemetryDetails&case=true>`_). Telemetry does not manipulate or enforce a specific format for the supplied provider's data.

Structure:

.. code-block:: js

    "addonDetails": {
      "XPI": {
        "adbhelper@mozilla.org": {
          "scan_items": 24,
          "scan_MS": 3,
          "location": "app-profile",
          "name": "ADB Helper",
          "creator": "Mozilla & Android Open Source Project",
          "startup_MS": 30
        },
        ...
      },
      ...
    }

addonHistograms
---------------
This section contains the histogram registered by the addons (`see here <https://dxr.mozilla.org/mozilla-central/rev/584870f1cbc5d060a57e147ce249f736956e2b62/toolkit/components/telemetry/nsITelemetry.idl#303>`_). This section is not present if no addon histogram is available.

UITelemetry
-----------
See the ``UITelemetry data format`` documentation.

slowSQL
-------
This section contains the informations about the slow SQL queries for both the main and other threads. The execution of an SQL statement is considered slow if it takes 50ms or more on the main thread or 100ms or more on other threads. Slow SQL statements will be automatically trimmed to 1000 characters. This limit doesn't include the ellipsis and database name, that are appended at the end of the stored statement.

Structure:

.. code-block:: js

    "slowSQL": {
      "mainThread": {
        "Sanitized SQL Statement": [
          1, // the number of times this statement was hit
          200  // the total time (in milliseconds) that was spent on this statement
        ],
        ...
      },
      "otherThreads": {
        "VACUUM /* places.sqlite */": [
          1,
          330
        ],
        ...
      }
    },

slowSQLStartup
--------------
This section contains the slow SQL statements gathered at startup (until the "sessionstore-windows-restored" event is fired). The structure of this section resembles the one for `slowSQL`_.

UIMeasurements
--------------
This section contains UI specific telemetry measurements and events. This section is mainly populated with Android-specific data and events (`see here <https://dxr.mozilla.org/mozilla-central/search?q=regexp%3AUITelemetry.%28addEvent|startSession|stopSession%29&redirect=false&case=false>`_).

Structure:

.. code-block:: js

    "UIMeasurements": [
      {
        "type": "event", // either "session" or "event"
        "action": "action.1",
        "method": "menu",
        "sessions": [],
        "timestamp": 12345,
        "extras": "settings"
      },
      {
        "type": "session",
        "name": "awesomescreen.1",
        "reason": "commit",
        "start": 123,
        "end": 456
      }
      ...
    ],