diff options
Diffstat (limited to 'gfx/doc/Silk.md')
| -rw-r--r-- | gfx/doc/Silk.md | 246 |
1 files changed, 246 insertions, 0 deletions
diff --git a/gfx/doc/Silk.md b/gfx/doc/Silk.md new file mode 100644 index 000000000..c71e79e43 --- /dev/null +++ b/gfx/doc/Silk.md @@ -0,0 +1,246 @@ +Silk Architecture Overview +================= + +#Architecture +Our current architecture is to align three components to hardware vsync timers: + +1. Compositor +2. RefreshDriver / Painting +3. Input Events + +The flow of our rendering engine is as follows: + +1. Hardware Vsync event occurs on an OS specific *Hardware Vsync Thread* on a per monitor basis. +2. The *Hardware Vsync Thread* attached to the monitor notifies the **CompositorVsyncDispatchers** and **RefreshTimerVsyncDispatcher**. +3. For every Firefox window on the specific monitor, notify a **CompositorVsyncDispatcher**. The **CompositorVsyncDispatcher** is specific to one window. +4. The **CompositorVsyncDispatcher** notifies a **CompositorWidgetVsyncObserver** when remote compositing, or a **CompositorVsyncScheduler::Observer** when compositing in-process. +5. If remote compositing, a vsync notification is sent from the **CompositorWidgetVsyncObserver** to the **VsyncBridgeChild** on the UI process, which sends an IPDL message to the **VsyncBridgeParent** on the compositor thread of the GPU process, which then dispatches to **CompositorVsyncScheduler::Observer**. +6. The **RefreshTimerVsyncDispatcher** notifies the Chrome **RefreshTimer** that a vsync has occured. +7. The **RefreshTimerVsyncDispatcher** sends IPC messages to all content processes to tick their respective active **RefreshTimer**. +8. The **Compositor** dispatches input events on the *Compositor Thread*, then composites. Input events are only dispatched on the *Compositor Thread* on b2g. +9. The **RefreshDriver** paints on the *Main Thread*. + +The implementation is broken into the following sections and will reference this figure. Note that **Objects** are bold fonts while *Threads* are italicized. + +<img src="silkArchitecture.png" width="900px" height="630px" /> + +#Hardware Vsync +Hardware vsync events from (1), occur on a specific **Display** Object. +The **Display** object is responsible for enabling / disabling vsync on a per connected display basis. +For example, if two monitors are connected, two **Display** objects will be created, each listening to vsync events for their respective displays. +We require one **Display** object per monitor as each monitor may have different vsync rates. +As a fallback solution, we have one global **Display** object that can synchronize across all connected displays. +The global **Display** is useful if a window is positioned halfway between the two monitors. +Each platform will have to implement a specific **Display** object to hook and listen to vsync events. +As of this writing, both Firefox OS and OS X create their own hardware specific *Hardware Vsync Thread* that executes after a vsync has occured. +OS X creates one *Hardware Vsync Thread* per **CVDisplayLinkRef**. +We do not currently support multiple displays, so we use one global **CVDisplayLinkRef** that works across all active displays. +On Windows, we have to create a new platform *thread* that waits for DwmFlush(), which works across all active displays. +Once the thread wakes up from DwmFlush(), the actual vsync timestamp is retrieved from DwmGetCompositionTimingInfo(), which is the timestamp that is actually passed into the compositor and refresh driver. + +When a vsync occurs on a **Display**, the *Hardware Vsync Thread* callback fetches all **CompositorVsyncDispatchers** associated with the **Display**. +Each **CompositorVsyncDispatcher** is notified that a vsync has occured with the vsync's timestamp. +It is the responsibility of the **CompositorVsyncDispatcher** to notify the **Compositor** that is awaiting vsync notifications. +The **Display** will then notify the associated **RefreshTimerVsyncDispatcher**, which should notify all active **RefreshDrivers** to tick. + +All **Display** objects are encapsulated in a **VsyncSource** object. +The **VsyncSource** object lives in **gfxPlatform** and is instantiated only on the parent process when **gfxPlatform** is created. +The **VsyncSource** is destroyed when **gfxPlatform** is destroyed. +There is only one **VsyncSource** object throughout the entire lifetime of Firefox. +Each platform is expected to implement their own **VsyncSource** to manage vsync events. +On Firefox OS, this is through the **HwcComposer2D**. +On OS X, this is through **CVDisplayLinkRef**. +On Windows, it should be through **DwmGetCompositionTimingInfo**. + +#Compositor +When the **CompositorVsyncDispatcher** is notified of the vsync event, the **CompositorVsyncScheduler::Observer** associated with the **CompositorVsyncDispatcher** begins execution. +Since the **CompositorVsyncDispatcher** executes on the *Hardware Vsync Thread* and the **Compositor** composites on the *CompositorThread*, the **CompositorVsyncScheduler::Observer** posts a task to the *CompositorThread*. +The **CompositorBridgeParent** then composites. +The model where the **CompositorVsyncDispatcher** notifies components on the *Hardware Vsync Thread*, and the component schedules the task on the appropriate thread is used everywhere. + +The **CompositorVsyncScheduler::Observer** listens to vsync events as needed and stops listening to vsync when composites are no longer scheduled or required. +Every **CompositorBridgeParent** is associated and tied to one **CompositorVsyncScheduler::Observer**, which is associated with the **CompositorVsyncDispatcher**. +Each **CompositorBridgeParent** is associated with one widget and is created when a new platform window or **nsBaseWidget** is created. +The **CompositorBridgeParent**, **CompositorVsyncDispatcher**, **CompositorVsyncScheduler::Observer**, and **nsBaseWidget** all have the same lifetimes, which are created and destroyed together. + +##Out-of-process Compositors +When compositing out-of-process, this model changes slightly. +In this case there are effectively two observers: a UI process observer (**CompositorWidgetVsyncObserver**), and the **CompositorVsyncScheduler::Observer** in the GPU process. +There are also two dispatchers: the widget dispatcher in the UI process (**CompositorVsyncDispatcher**), and the IPDL-based dispatcher in the GPU process (**CompositorBridgeParent::NotifyVsync**). +The UI process observer and the GPU process dispatcher are linked via an IPDL protocol called PVsyncBridge. +**PVsyncBridge** is a top-level protocol for sending vsync notifications to the compositor thread in the GPU process. +The compositor controls vsync observation through a separate actor, **PCompositorWidget**, which (as a subactor for **CompositorBridgeChild**) links the compositor thread in the GPU process to the main thread in the UI process. + +Out-of-process compositors do not go through **CompositorVsyncDispatcher** directly. +Instead, the **CompositorWidgetDelegate** in the UI process creates one, and gives it a **CompositorWidgetVsyncObserver**. +This observer forwards notifications to a Vsync I/O thread, where **VsyncBridgeChild** then forwards the notification again to the compositor thread in the GPU process. +The notification is received by a **VsyncBridgeParent**. +The GPU process uses the layers ID in the notification to find the correct compositor to dispatch the notification to. + +###CompositorVsyncDispatcher +The **CompositorVsyncDispatcher** executes on the *Hardware Vsync Thread*. +It contains references to the **nsBaseWidget** it is associated with and has a lifetime equal to the **nsBaseWidget**. +The **CompositorVsyncDispatcher** is responsible for notifying the **CompositorBridgeParent** that a vsync event has occured. +There can be multiple **CompositorVsyncDispatchers** per **Display**, one **CompositorVsyncDispatcher** per window. +The only responsibility of the **CompositorVsyncDispatcher** is to notify components when a vsync event has occured, and to stop listening to vsync when no components require vsync events. +We require one **CompositorVsyncDispatcher** per window so that we can handle multiple **Displays**. +When compositing in-process, the **CompositorVsyncDispatcher** is attached to the CompositorWidget for the +window. When out-of-process, it is attached to the CompositorWidgetDelegate, which forwards +observer notifications over IPDL. In the latter case, its lifetime is tied to a CompositorSession +rather than the nsIWidget. + +###Multiple Displays +The **VsyncSource** has an API to switch a **CompositorVsyncDispatcher** from one **Display** to another **Display**. +For example, when one window either goes into full screen mode or moves from one connected monitor to another. +When one window moves to another monitor, we expect a platform specific notification to occur. +The detection of when a window enters full screen mode or moves is not covered by Silk itself, but the framework is built to support this use case. +The expected flow is that the OS notification occurs on **nsIWidget**, which retrieves the associated **CompositorVsyncDispatcher**. +The **CompositorVsyncDispatcher** then notifies the **VsyncSource** to switch to the correct **Display** the **CompositorVsyncDispatcher** is connected to. +Because the notification works through the **nsIWidget**, the actual switching of the **CompositorVsyncDispatcher** to the correct **Display** should occur on the *Main Thread*. +The current implementation of Silk does not handle this case and needs to be built out. + +###CompositorVsyncScheduler::Observer +The **CompositorVsyncScheduler::Observer** handles the vsync notifications and interactions with the **CompositorVsyncDispatcher**. +When the **Compositor** requires a scheduled composite, it notifies the **CompositorVsyncScheduler::Observer** that it needs to listen to vsync. +The **CompositorVsyncScheduler::Observer** then observes / unobserves vsync as needed from the **CompositorVsyncDispatcher** to enable composites. + +###GeckoTouchDispatcher +The **GeckoTouchDispatcher** is a singleton that resamples touch events to smooth out jank while tracking a user's finger. +Because input and composite are linked together, the **CompositorVsyncScheduler::Observer** has a reference to the **GeckoTouchDispatcher** and vice versa. + +###Input Events +One large goal of Silk is to align touch events with vsync events. +On Firefox OS, touchscreens often have different touch scan rates than the display refreshes. +A Flame device has a touch refresh rate of 75 HZ, while a Nexus 4 has a touch refresh rate of 100 HZ, while the device's display refresh rate is 60HZ. +When a vsync event occurs, we resample touch events, and then dispatch the resampled touch event to APZ. +Touch events on Firefox OS occur on a *Touch Input Thread* whereas they are processed by APZ on the *APZ Controller Thread*. +We use [Google Android's touch resampling](http://www.masonchang.com/blog/2014/8/25/androids-touch-resampling-algorithm) algorithm to resample touch events. + +Currently, we have a strict ordering between Composites and touch events. +When a touch event occurs on the *Touch Input Thread*, we store the touch event in a queue. +When a vsync event occurs, the **CompositorVsyncDispatcher** notifies the **Compositor** of a vsync event, which notifies the **GeckoTouchDispatcher**. +The **GeckoTouchDispatcher** processes the touch event first on the *APZ Controller Thread*, which is the same as the *Compositor Thread* on b2g, then the **Compositor** finishes compositing. +We require this strict ordering because if a vsync notification is dispatched to both the **Compositor** and **GeckoTouchDispatcher** at the same time, a race condition occurs between processing the touch event and therefore position versus compositing. +In practice, this creates very janky scrolling. +As of this writing, we have not analyzed input events on desktop platforms. + +One slight quirk is that input events can start a composite, for example during a scroll and after the **Compositor** is no longer listening to vsync events. +In these cases, we notify the **Compositor** to observe vsync so that it dispatches touch events. +If touch events were not dispatched, and since the **Compositor** is not listening to vsync events, the touch events would never be dispatched. +The **GeckoTouchDispatcher** handles this case by always forcing the **Compositor** to listen to vsync events while touch events are occurring. + +###Widget, Compositor, CompositorVsyncDispatcher, GeckoTouchDispatcher Shutdown Procedure +When the [nsBaseWidget shuts down](http://hg.mozilla.org/mozilla-central/file/0df249a0e4d3/widget/nsBaseWidget.cpp#l182) - It calls nsBaseWidget::DestroyCompositor on the *Gecko Main Thread*. +During nsBaseWidget::DestroyCompositor, it first destroys the CompositorBridgeChild. +CompositorBridgeChild sends a sync IPC call to CompositorBridgeParent::RecvStop, which calls [CompositorBridgeParent::Destroy](http://hg.mozilla.org/mozilla-central/file/ab0490972e1e/gfx/layers/ipc/CompositorBridgeParent.cpp#l509). +During this time, the *main thread* is blocked on the parent process. +CompositorBridgeParent::RecvStop runs on the *Compositor thread* and cleans up some resources, including setting the **CompositorVsyncScheduler::Observer** to nullptr. +CompositorBridgeParent::RecvStop also explicitly keeps the CompositorBridgeParent alive and posts another task to run CompositorBridgeParent::DeferredDestroy on the Compositor loop so that all ipdl code can finish executing. +The **CompositorVsyncScheduler::Observer** also unobserves from vsync and cancels any pending composite tasks. +Once CompositorBridgeParent::RecvStop finishes, the *main thread* in the parent process continues shutting down the nsBaseWidget. + +At the same time, the *Compositor thread* is executing tasks until CompositorBridgeParent::DeferredDestroy runs, which flushes the compositor message loop. +Now we have two tasks as both the nsBaseWidget releases a reference to the Compositor on the *main thread* during destruction and the CompositorBridgeParent::DeferredDestroy releases a reference to the CompositorBridgeParent on the *Compositor Thread*. +Finally, the CompositorBridgeParent itself is destroyed on the *main thread* once both references are gone due to explicit [main thread destruction](http://hg.mozilla.org/mozilla-central/file/50b95032152c/gfx/layers/ipc/CompositorBridgeParent.h#l148). + +With the **CompositorVsyncScheduler::Observer**, any accesses to the widget after nsBaseWidget::DestroyCompositor executes are invalid. +Any accesses to the compositor between the time the nsBaseWidget::DestroyCompositor runs and the CompositorVsyncScheduler::Observer's destructor runs aren't safe yet a hardware vsync event could occur between these times. +Since any tasks posted on the Compositor loop after CompositorBridgeParent::DeferredDestroy is posted are invalid, we make sure that no vsync tasks can be posted once CompositorBridgeParent::RecvStop executes and DeferredDestroy is posted on the Compositor thread. +When the sync call to CompositorBridgeParent::RecvStop executes, we explicitly set the CompositorVsyncScheduler::Observer to null to prevent vsync notifications from occurring. +If vsync notifications were allowed to occur, since the **CompositorVsyncScheduler::Observer**'s vsync notification executes on the *hardware vsync thread*, it would post a task to the Compositor loop and may execute after CompositorBridgeParent::DeferredDestroy. +Thus, we explicitly shut down vsync events in the **CompositorVsyncDispatcher** and **CompositorVsyncScheduler::Observer** during nsBaseWidget::Shutdown to prevent any vsync tasks from executing after CompositorBridgeParent::DeferredDestroy. + +The **CompositorVsyncDispatcher** may be destroyed on either the *main thread* or *Compositor Thread*, since both the nsBaseWidget and **CompositorVsyncScheduler::Observer** race to destroy on different threads. +nsBaseWidget is destroyed on the *main thread* and releases a reference to the **CompositorVsyncDispatcher** during destruction. +The **CompositorVsyncScheduler::Observer** has a race to be destroyed either during CompositorBridgeParent shutdown or from the **GeckoTouchDispatcher** which is destroyed on the main thread with [ClearOnShutdown](http://hg.mozilla.org/mozilla-central/file/21567e9a6e40/xpcom/base/ClearOnShutdown.h#l15). +Whichever object, the CompositorBridgeParent or the **GeckoTouchDispatcher** is destroyed last will hold the last reference to the **CompositorVsyncDispatcher**, which destroys the object. + +#Refresh Driver +The Refresh Driver is ticked from a [single active timer](http://hg.mozilla.org/mozilla-central/file/ab0490972e1e/layout/base/nsRefreshDriver.cpp#l11). +The assumption is that there are multiple **RefreshDrivers** connected to a single **RefreshTimer**. +There are two **RefreshTimers**: an active and an inactive **RefreshTimer**. +Each Tab has its own **RefreshDriver**, which connects to one of the global **RefreshTimers**. +The **RefreshTimers** execute on the *Main Thread* and tick their connected **RefreshDrivers**. +We do not want to break this model of multiple **RefreshDrivers** per a set of two global **RefreshTimers**. +Each **RefreshDriver** switches between the active and inactive **RefreshTimer**. + +Instead, we create a new **RefreshTimer**, the **VsyncRefreshTimer** which ticks based on vsync messages. +We replace the current active timer with a **VsyncRefreshTimer**. +All tabs will then tick based on this new active timer. +Since the **RefreshTimer** has a lifetime of the process, we only need to create a single **RefreshTimerVsyncDispatcher** per **Display** when Firefox starts. +Even if we do not have any content processes, the Chrome process will still need a **VsyncRefreshTimer**, thus we can associate the **RefreshTimerVsyncDispatcher** with each **Display**. + +When Firefox starts, we initially create a new **VsyncRefreshTimer** in the Chrome process. +The **VsyncRefreshTimer** will listen to vsync notifications from **RefreshTimerVsyncDispatcher** on the global **Display**. +When nsRefreshDriver::Shutdown executes, it will delete the **VsyncRefreshTimer**. +This creates a problem as all the **RefreshTimers** are currently manually memory managed whereas **VsyncObservers** are ref counted. +To work around this problem, we create a new **RefreshDriverVsyncObserver** as an inner class to **VsyncRefreshTimer**, which actually receives vsync notifications. It then ticks the **RefreshDrivers** inside **VsyncRefreshTimer**. + +With Content processes, the start up process is more complicated. +We send vsync IPC messages via the use of the PBackground thread on the parent process, which allows us to send messages from the Parent process' without waiting on the *main thread*. +This sends messages from the Parent::*PBackground Thread* to the Child::*Main Thread*. +The *main thread* receiving IPC messages on the content process is acceptable because **RefreshDrivers** must execute on the *main thread*. +However, there is some amount of time required to setup the IPC connection upon process creation and during this time, the **RefreshDrivers** must tick to set up the process. +To get around this, we initially use software **RefreshTimers** that already exist during content process startup and swap in the **VsyncRefreshTimer** once the IPC connection is created. + +During nsRefreshDriver::ChooseTimer, we create an async PBackground IPC open request to create a **VsyncParent** and **VsyncChild**. +At the same time, we create a software **RefreshTimer** and tick the **RefreshDrivers** as normal. +Once the PBackground callback is executed and an IPC connection exists, we swap all **RefreshDrivers** currently associated with the active **RefreshTimer** and swap the **RefreshDrivers** to use the **VsyncRefreshTimer**. +Since all interactions on the content process occur on the main thread, there are no need for locks. +The **VsyncParent** listens to vsync events through the **VsyncRefreshTimerDispatcher** on the parent side and sends vsync IPC messages to the **VsyncChild**. +The **VsyncChild** notifies the **VsyncRefreshTimer** on the content process. + +During the shutdown process of the content process, ActorDestroy is called on the **VsyncChild** and **VsyncParent** due to the normal PBackground shutdown process. +Once ActorDestroy is called, no IPC messages should be sent across the channel. +After ActorDestroy is called, the IPDL machinery will delete the **VsyncParent/Child** pair. +The **VsyncParent**, due to being a **VsyncObserver**, is ref counted. +After **VsyncParent::ActorDestroy** is called, it unregisters itself from the **RefreshTimerVsyncDispatcher**, which holds the last reference to the **VsyncParent**, and the object will be deleted. + +Thus the overall flow during normal execution is: + +1. VsyncSource::Display::RefreshTimerVsyncDispatcher receives a Vsync notification from the OS in the parent process. +2. RefreshTimerVsyncDispatcher notifies VsyncRefreshTimer::RefreshDriverVsyncObserver that a vsync occured on the parent process on the hardware vsync thread. +3. RefreshTimerVsyncDispatcher notifies the VsyncParent on the hardware vsync thread that a vsync occured. +4. The VsyncRefreshTimer::RefreshDriverVsyncObserver in the parent process posts a task to the main thread that ticks the refresh drivers. +5. VsyncParent posts a task to the PBackground thread to send a vsync IPC message to VsyncChild. +6. VsyncChild receive a vsync notification on the content process on the main thread and ticks their respective RefreshDrivers. + +###Compressing Vsync Messages +Vsync messages occur quite often and the *main thread* can be busy for long periods of time due to JavaScript. +Consistently sending vsync messages to the refresh driver timer can flood the *main thread* with refresh driver ticks, causing even more delays. +To avoid this problem, we compress vsync messages on both the parent and child processes. + +On the parent process, newer vsync messages update a vsync timestamp but do not actually queue any tasks on the *main thread*. +Once the parent process' *main thread* executes the refresh driver tick, it uses the most updated vsync timestamp to tick the refresh driver. +After the refresh driver has ticked, one single vsync message is queued for another refresh driver tick task. +On the content process, the IPDL **compress** keyword automatically compresses IPC messages. + +### Multiple Monitors +In order to have multiple monitor support for the **RefreshDrivers**, we have multiple active **RefreshTimers**. +Each **RefreshTimer** is associated with a specific **Display** via an id and tick when it's respective **Display** vsync occurs. +We have **N RefreshTimers**, where N is the number of connected displays. +Each **RefreshTimer** still has multiple **RefreshDrivers**. + +When a tab or window changes monitors, the **nsIWidget** receives a display changed notification. +Based on which display the window is on, the window switches to the correct **RefreshTimerVsyncDispatcher** and **CompositorVsyncDispatcher** on the parent process based on the display id. +Each **TabParent** should also send a notification to their child. +Each **TabChild**, given the display ID, switches to the correct **RefreshTimer** associated with the display ID. +When each display vsync occurs, it sends one IPC message to notify vsync. +The vsync message contains a display ID, to tick the appropriate **RefreshTimer** on the content process. +There is still only one **VsyncParent/VsyncChild** pair, just each vsync notification will include a display ID, which maps to the correct **RefreshTimer**. + +#Object Lifetime +1. CompositorVsyncDispatcher - Lives as long as the nsBaseWidget associated with the VsyncDispatcher +2. CompositorVsyncScheduler::Observer - Lives and dies the same time as the CompositorBridgeParent. +3. RefreshTimerVsyncDispatcher - As long as the associated display object, which is the lifetime of Firefox. +4. VsyncSource - Lives as long as the gfxPlatform on the chrome process, which is the lifetime of Firefox. +5. VsyncParent/VsyncChild - Lives as long as the content process +6. RefreshTimer - Lives as long as the process + +#Threads +All **VsyncObservers** are notified on the *Hardware Vsync Thread*. It is the responsibility of the **VsyncObservers** to post tasks to their respective correct thread. For example, the **CompositorVsyncScheduler::Observer** will be notified on the *Hardware Vsync Thread*, and post a task to the *Compositor Thread* to do the actual composition. + +1. Compositor Thread - Nothing changes +2. Main Thread - PVsyncChild receives IPC messages on the main thread. We also enable/disable vsync on the main thread. +3. PBackground Thread - Creates a connection from the PBackground thread on the parent process to the main thread in the content process. +4. Hardware Vsync Thread - Every platform is different, but we always have the concept of a hardware vsync thread. Sometimes this is actually created by the host OS. On Windows, we have to create a separate platform thread that blocks on DwmFlush(). |