diff options
Diffstat (limited to 'testing/web-platform/tests/tools/pytest/doc/en/writing_plugins.rst')
| -rw-r--r-- | testing/web-platform/tests/tools/pytest/doc/en/writing_plugins.rst | 575 |
1 files changed, 575 insertions, 0 deletions
diff --git a/testing/web-platform/tests/tools/pytest/doc/en/writing_plugins.rst b/testing/web-platform/tests/tools/pytest/doc/en/writing_plugins.rst new file mode 100644 index 000000000..cc346aaa8 --- /dev/null +++ b/testing/web-platform/tests/tools/pytest/doc/en/writing_plugins.rst @@ -0,0 +1,575 @@ +.. _plugins: +.. _`writing-plugins`: + +Writing plugins +=============== + +It is easy to implement `local conftest plugins`_ for your own project +or `pip-installable plugins`_ that can be used throughout many projects, +including third party projects. Please refer to :ref:`using plugins` if you +only want to use but not write plugins. + +A plugin contains one or multiple hook functions. :ref:`Writing hooks <writinghooks>` +explains the basics and details of how you can write a hook function yourself. +``pytest`` implements all aspects of configuration, collection, running and +reporting by calling `well specified hooks`_ of the following plugins: + +* :ref:`builtin plugins`: loaded from pytest's internal ``_pytest`` directory. + +* :ref:`external plugins <extplugins>`: modules discovered through + `setuptools entry points`_ + +* `conftest.py plugins`_: modules auto-discovered in test directories + +In principle, each hook call is a ``1:N`` Python function call where ``N`` is the +number of registered implementation functions for a given specification. +All specifications and implementations following the ``pytest_`` prefix +naming convention, making them easy to distinguish and find. + +.. _`pluginorder`: + +Plugin discovery order at tool startup +-------------------------------------- + +``pytest`` loads plugin modules at tool startup in the following way: + +* by loading all builtin plugins + +* by loading all plugins registered through `setuptools entry points`_. + +* by pre-scanning the command line for the ``-p name`` option + and loading the specified plugin before actual command line parsing. + +* by loading all :file:`conftest.py` files as inferred by the command line + invocation: + + - if no test paths are specified use current dir as a test path + - if exists, load ``conftest.py`` and ``test*/conftest.py`` relative + to the directory part of the first test path. + + Note that pytest does not find ``conftest.py`` files in deeper nested + sub directories at tool startup. It is usually a good idea to keep + your conftest.py file in the top level test or project root directory. + +* by recursively loading all plugins specified by the + ``pytest_plugins`` variable in ``conftest.py`` files + + +.. _`pytest/plugin`: http://bitbucket.org/pytest-dev/pytest/src/tip/pytest/plugin/ +.. _`conftest.py plugins`: +.. _`conftest.py`: +.. _`localplugin`: +.. _`conftest`: +.. _`local conftest plugins`: + +conftest.py: local per-directory plugins +---------------------------------------- + +Local ``conftest.py`` plugins contain directory-specific hook +implementations. Hook Session and test running activities will +invoke all hooks defined in ``conftest.py`` files closer to the +root of the filesystem. Example of implementing the +``pytest_runtest_setup`` hook so that is called for tests in the ``a`` +sub directory but not for other directories:: + + a/conftest.py: + def pytest_runtest_setup(item): + # called for running each test in 'a' directory + print ("setting up", item) + + a/test_sub.py: + def test_sub(): + pass + + test_flat.py: + def test_flat(): + pass + +Here is how you might run it:: + + py.test test_flat.py # will not show "setting up" + py.test a/test_sub.py # will show "setting up" + +.. Note:: + If you have ``conftest.py`` files which do not reside in a + python package directory (i.e. one containing an ``__init__.py``) then + "import conftest" can be ambiguous because there might be other + ``conftest.py`` files as well on your PYTHONPATH or ``sys.path``. + It is thus good practice for projects to either put ``conftest.py`` + under a package scope or to never import anything from a + conftest.py file. + + +Writing your own plugin +----------------------- + +.. _`setuptools`: http://pypi.python.org/pypi/setuptools + +If you want to write a plugin, there are many real-life examples +you can copy from: + +* a custom collection example plugin: :ref:`yaml plugin` +* around 20 :ref:`builtin plugins` which provide pytest's own functionality +* many `external plugins <http://plugincompat.herokuapp.com>`_ providing additional features + +All of these plugins implement the documented `well specified hooks`_ +to extend and add functionality. + +.. note:: + Make sure to check out the excellent + `cookiecutter-pytest-plugin <https://github.com/pytest-dev/cookiecutter-pytest-plugin>`_ + project, which is a `cookiecutter template <https://github.com/audreyr/cookiecutter>`_ + for authoring plugins. + + The template provides an excellent starting point with a working plugin, + tests running with tox, comprehensive README and + entry-pointy already pre-configured. + +Also consider :ref:`contributing your plugin to pytest-dev<submitplugin>` +once it has some happy users other than yourself. + + +.. _`setuptools entry points`: +.. _`pip-installable plugins`: + +Making your plugin installable by others +---------------------------------------- + +If you want to make your plugin externally available, you +may define a so-called entry point for your distribution so +that ``pytest`` finds your plugin module. Entry points are +a feature that is provided by `setuptools`_. pytest looks up +the ``pytest11`` entrypoint to discover its +plugins and you can thus make your plugin available by defining +it in your setuptools-invocation: + +.. sourcecode:: python + + # sample ./setup.py file + from setuptools import setup + + setup( + name="myproject", + packages = ['myproject'] + + # the following makes a plugin available to pytest + entry_points = { + 'pytest11': [ + 'name_of_plugin = myproject.pluginmodule', + ] + }, + ) + +If a package is installed this way, ``pytest`` will load +``myproject.pluginmodule`` as a plugin which can define +`well specified hooks`_. + + + + +Requiring/Loading plugins in a test module or conftest file +----------------------------------------------------------- + +You can require plugins in a test module or a conftest file like this:: + + pytest_plugins = "name1", "name2", + +When the test module or conftest plugin is loaded the specified plugins +will be loaded as well. You can also use dotted path like this:: + + pytest_plugins = "myapp.testsupport.myplugin" + +which will import the specified module as a ``pytest`` plugin. + + +Accessing another plugin by name +-------------------------------- + +If a plugin wants to collaborate with code from +another plugin it can obtain a reference through +the plugin manager like this: + +.. sourcecode:: python + + plugin = config.pluginmanager.getplugin("name_of_plugin") + +If you want to look at the names of existing plugins, use +the ``--traceconfig`` option. + +Testing plugins +--------------- + +pytest comes with some facilities that you can enable for testing your +plugin. Given that you have an installed plugin you can enable the +:py:class:`testdir <_pytest.pytester.Testdir>` fixture via specifying a +command line option to include the pytester plugin (``-p pytester``) or +by putting ``pytest_plugins = "pytester"`` into your test or +``conftest.py`` file. You then will have a ``testdir`` fixture which you +can use like this:: + + # content of test_myplugin.py + + pytest_plugins = "pytester" # to get testdir fixture + + def test_myplugin(testdir): + testdir.makepyfile(""" + def test_example(): + pass + """) + result = testdir.runpytest("--verbose") + result.fnmatch_lines(""" + test_example* + """) + +Note that by default ``testdir.runpytest()`` will perform a pytest +in-process. You can pass the command line option ``--runpytest=subprocess`` +to have it happen in a subprocess. + +Also see the :py:class:`RunResult <_pytest.pytester.RunResult>` for more +methods of the result object that you get from a call to ``runpytest``. + +.. _`writinghooks`: + +Writing hook functions +====================== + + +.. _validation: + +hook function validation and execution +-------------------------------------- + +pytest calls hook functions from registered plugins for any +given hook specification. Let's look at a typical hook function +for the ``pytest_collection_modifyitems(session, config, +items)`` hook which pytest calls after collection of all test items is +completed. + +When we implement a ``pytest_collection_modifyitems`` function in our plugin +pytest will during registration verify that you use argument +names which match the specification and bail out if not. + +Let's look at a possible implementation: + +.. code-block:: python + + def pytest_collection_modifyitems(config, items): + # called after collection is completed + # you can modify the ``items`` list + +Here, ``pytest`` will pass in ``config`` (the pytest config object) +and ``items`` (the list of collected test items) but will not pass +in the ``session`` argument because we didn't list it in the function +signature. This dynamic "pruning" of arguments allows ``pytest`` to +be "future-compatible": we can introduce new hook named parameters without +breaking the signatures of existing hook implementations. It is one of +the reasons for the general long-lived compatibility of pytest plugins. + +Note that hook functions other than ``pytest_runtest_*`` are not +allowed to raise exceptions. Doing so will break the pytest run. + + + +firstresult: stop at first non-None result +------------------------------------------- + +Most calls to ``pytest`` hooks result in a **list of results** which contains +all non-None results of the called hook functions. + +Some hook specifications use the ``firstresult=True`` option so that the hook +call only executes until the first of N registered functions returns a +non-None result which is then taken as result of the overall hook call. +The remaining hook functions will not be called in this case. + + +hookwrapper: executing around other hooks +------------------------------------------------- + +.. currentmodule:: _pytest.core + +.. versionadded:: 2.7 (experimental) + +pytest plugins can implement hook wrappers which wrap the execution +of other hook implementations. A hook wrapper is a generator function +which yields exactly once. When pytest invokes hooks it first executes +hook wrappers and passes the same arguments as to the regular hooks. + +At the yield point of the hook wrapper pytest will execute the next hook +implementations and return their result to the yield point in the form of +a :py:class:`CallOutcome` instance which encapsulates a result or +exception info. The yield point itself will thus typically not raise +exceptions (unless there are bugs). + +Here is an example definition of a hook wrapper:: + + import pytest + + @pytest.hookimpl(hookwrapper=True) + def pytest_pyfunc_call(pyfuncitem): + # do whatever you want before the next hook executes + + outcome = yield + # outcome.excinfo may be None or a (cls, val, tb) tuple + + res = outcome.get_result() # will raise if outcome was exception + # postprocess result + +Note that hook wrappers don't return results themselves, they merely +perform tracing or other side effects around the actual hook implementations. +If the result of the underlying hook is a mutable object, they may modify +that result but it's probably better to avoid it. + + +Hook function ordering / call example +------------------------------------- + +For any given hook specification there may be more than one +implementation and we thus generally view ``hook`` execution as a +``1:N`` function call where ``N`` is the number of registered functions. +There are ways to influence if a hook implementation comes before or +after others, i.e. the position in the ``N``-sized list of functions: + +.. code-block:: python + + # Plugin 1 + @pytest.hookimpl(tryfirst=True) + def pytest_collection_modifyitems(items): + # will execute as early as possible + + # Plugin 2 + @pytest.hookimpl(trylast=True) + def pytest_collection_modifyitems(items): + # will execute as late as possible + + # Plugin 3 + @pytest.hookimpl(hookwrapper=True) + def pytest_collection_modifyitems(items): + # will execute even before the tryfirst one above! + outcome = yield + # will execute after all non-hookwrappers executed + +Here is the order of execution: + +1. Plugin3's pytest_collection_modifyitems called until the yield point + because it is a hook wrapper. + +2. Plugin1's pytest_collection_modifyitems is called because it is marked + with ``tryfirst=True``. + +3. Plugin2's pytest_collection_modifyitems is called because it is marked + with ``trylast=True`` (but even without this mark it would come after + Plugin1). + +4. Plugin3's pytest_collection_modifyitems then executing the code after the yield + point. The yield receives a :py:class:`CallOutcome` instance which encapsulates + the result from calling the non-wrappers. Wrappers shall not modify the result. + +It's possible to use ``tryfirst`` and ``trylast`` also in conjunction with +``hookwrapper=True`` in which case it will influence the ordering of hookwrappers +among each other. + + +Declaring new hooks +------------------------ + +.. currentmodule:: _pytest.hookspec + +Plugins and ``conftest.py`` files may declare new hooks that can then be +implemented by other plugins in order to alter behaviour or interact with +the new plugin: + +.. autofunction:: pytest_addhooks + +Hooks are usually declared as do-nothing functions that contain only +documentation describing when the hook will be called and what return values +are expected. + +For an example, see `newhooks.py`_ from :ref:`xdist`. + +.. _`newhooks.py`: https://github.com/pytest-dev/pytest-xdist/blob/974bd566c599dc6a9ea291838c6f226197208b46/xdist/newhooks.py + + +Optionally using hooks from 3rd party plugins +--------------------------------------------- + +Using new hooks from plugins as explained above might be a little tricky +because of the standard :ref:`validation mechanism <validation>`: +if you depend on a plugin that is not installed, validation will fail and +the error message will not make much sense to your users. + +One approach is to defer the hook implementation to a new plugin instead of +declaring the hook functions directly in your plugin module, for example:: + + # contents of myplugin.py + + class DeferPlugin(object): + """Simple plugin to defer pytest-xdist hook functions.""" + + def pytest_testnodedown(self, node, error): + """standard xdist hook function. + """ + + def pytest_configure(config): + if config.pluginmanager.hasplugin('xdist'): + config.pluginmanager.register(DeferPlugin()) + +This has the added benefit of allowing you to conditionally install hooks +depending on which plugins are installed. + +.. _`well specified hooks`: + +.. currentmodule:: _pytest.hookspec + +pytest hook reference +===================== + + +Initialization, command line and configuration hooks +---------------------------------------------------- + +.. autofunction:: pytest_load_initial_conftests +.. autofunction:: pytest_cmdline_preparse +.. autofunction:: pytest_cmdline_parse +.. autofunction:: pytest_namespace +.. autofunction:: pytest_addoption +.. autofunction:: pytest_cmdline_main +.. autofunction:: pytest_configure +.. autofunction:: pytest_unconfigure + +Generic "runtest" hooks +----------------------- + +All runtest related hooks receive a :py:class:`pytest.Item` object. + +.. autofunction:: pytest_runtest_protocol +.. autofunction:: pytest_runtest_setup +.. autofunction:: pytest_runtest_call +.. autofunction:: pytest_runtest_teardown +.. autofunction:: pytest_runtest_makereport + +For deeper understanding you may look at the default implementation of +these hooks in :py:mod:`_pytest.runner` and maybe also +in :py:mod:`_pytest.pdb` which interacts with :py:mod:`_pytest.capture` +and its input/output capturing in order to immediately drop +into interactive debugging when a test failure occurs. + +The :py:mod:`_pytest.terminal` reported specifically uses +the reporting hook to print information about a test run. + +Collection hooks +---------------- + +``pytest`` calls the following hooks for collecting files and directories: + +.. autofunction:: pytest_ignore_collect +.. autofunction:: pytest_collect_directory +.. autofunction:: pytest_collect_file + +For influencing the collection of objects in Python modules +you can use the following hook: + +.. autofunction:: pytest_pycollect_makeitem +.. autofunction:: pytest_generate_tests + +After collection is complete, you can modify the order of +items, delete or otherwise amend the test items: + +.. autofunction:: pytest_collection_modifyitems + +Reporting hooks +--------------- + +Session related reporting hooks: + +.. autofunction:: pytest_collectstart +.. autofunction:: pytest_itemcollected +.. autofunction:: pytest_collectreport +.. autofunction:: pytest_deselected +.. autofunction:: pytest_report_header +.. autofunction:: pytest_report_teststatus +.. autofunction:: pytest_terminal_summary + +And here is the central hook for reporting about +test execution: + +.. autofunction:: pytest_runtest_logreport + +You can also use this hook to customize assertion representation for some +types: + +.. autofunction:: pytest_assertrepr_compare + + +Debugging/Interaction hooks +--------------------------- + +There are few hooks which can be used for special +reporting or interaction with exceptions: + +.. autofunction:: pytest_internalerror +.. autofunction:: pytest_keyboard_interrupt +.. autofunction:: pytest_exception_interact +.. autofunction:: pytest_enter_pdb + + +Reference of objects involved in hooks +====================================== + +.. autoclass:: _pytest.config.Config() + :members: + +.. autoclass:: _pytest.config.Parser() + :members: + +.. autoclass:: _pytest.main.Node() + :members: + +.. autoclass:: _pytest.main.Collector() + :members: + :show-inheritance: + +.. autoclass:: _pytest.main.Item() + :members: + :show-inheritance: + +.. autoclass:: _pytest.python.Module() + :members: + :show-inheritance: + +.. autoclass:: _pytest.python.Class() + :members: + :show-inheritance: + +.. autoclass:: _pytest.python.Function() + :members: + :show-inheritance: + +.. autoclass:: _pytest.runner.CallInfo() + :members: + +.. autoclass:: _pytest.runner.TestReport() + :members: + +.. autoclass:: _pytest.vendored_packages.pluggy._CallOutcome() + :members: + +.. autofunction:: _pytest.config.get_plugin_manager() + +.. autoclass:: _pytest.config.PytestPluginManager() + :members: + :undoc-members: + :show-inheritance: + +.. autoclass:: pluggy.PluginManager() + :members: + +.. currentmodule:: _pytest.pytester + +.. autoclass:: Testdir() + :members: runpytest,runpytest_subprocess,runpytest_inprocess,makeconftest,makepyfile + +.. autoclass:: RunResult() + :members: + +.. autoclass:: LineMatcher() + :members: |