diff options
Diffstat (limited to 'testing/web-platform/tests/tools/pytest/doc/en/goodpractices.rst')
-rw-r--r-- | testing/web-platform/tests/tools/pytest/doc/en/goodpractices.rst | 278 |
1 files changed, 278 insertions, 0 deletions
diff --git a/testing/web-platform/tests/tools/pytest/doc/en/goodpractices.rst b/testing/web-platform/tests/tools/pytest/doc/en/goodpractices.rst new file mode 100644 index 000000000..2d8050bd9 --- /dev/null +++ b/testing/web-platform/tests/tools/pytest/doc/en/goodpractices.rst @@ -0,0 +1,278 @@ +.. highlightlang:: python +.. _`goodpractices`: + +Good Integration Practices +================================================= + + +.. _`test discovery`: +.. _`Python test discovery`: + +Conventions for Python test discovery +------------------------------------------------- + +``pytest`` implements the following standard test discovery: + +* If no arguments are specified then collection starts from :confval:`testpaths` + (if configured) or the current directory. Alternatively, command line arguments + can be used in any combination of directories, file names or node ids. +* recurse into directories, unless they match :confval:`norecursedirs` +* ``test_*.py`` or ``*_test.py`` files, imported by their `test package name`_. +* ``Test`` prefixed test classes (without an ``__init__`` method) +* ``test_`` prefixed test functions or methods are test items + +For examples of how to customize your test discovery :doc:`example/pythoncollection`. + +Within Python modules, ``pytest`` also discovers tests using the standard +:ref:`unittest.TestCase <unittest.TestCase>` subclassing technique. + + +Choosing a test layout / import rules +------------------------------------------ + +``pytest`` supports two common test layouts: + +* putting tests into an extra directory outside your actual application + code, useful if you have many functional tests or for other reasons + want to keep tests separate from actual application code (often a good + idea):: + + setup.py # your setuptools Python package metadata + mypkg/ + __init__.py + appmodule.py + tests/ + test_app.py + ... + + +* inlining test directories into your application package, useful if you + have direct relation between (unit-)test and application modules and + want to distribute your tests along with your application:: + + setup.py # your setuptools Python package metadata + mypkg/ + __init__.py + appmodule.py + ... + test/ + test_app.py + ... + +Important notes relating to both schemes: + +- **make sure that "mypkg" is importable**, for example by typing once:: + + pip install -e . # install package using setup.py in editable mode + +- **avoid "__init__.py" files in your test directories**. + This way your tests can run easily against an installed version + of ``mypkg``, independently from the installed package if it contains + the tests or not. + +- With inlined tests you might put ``__init__.py`` into test + directories and make them installable as part of your application. + Using the ``py.test --pyargs mypkg`` invocation pytest will + discover where mypkg is installed and collect tests from there. + With the "external" test you can still distribute tests but they + will not be installed or become importable. + +Typically you can run tests by pointing to test directories or modules:: + + py.test tests/test_app.py # for external test dirs + py.test mypkg/test/test_app.py # for inlined test dirs + py.test mypkg # run tests in all below test directories + py.test # run all tests below current dir + ... + +Because of the above ``editable install`` mode you can change your +source code (both tests and the app) and rerun tests at will. +Once you are done with your work, you can `use tox`_ to make sure +that the package is really correct and tests pass in all +required configurations. + +.. note:: + + You can use Python3 namespace packages (PEP420) for your application + but pytest will still perform `test package name`_ discovery based on the + presence of ``__init__.py`` files. If you use one of the + two recommended file system layouts above but leave away the ``__init__.py`` + files from your directories it should just work on Python3.3 and above. From + "inlined tests", however, you will need to use absolute imports for + getting at your application code. + +.. _`test package name`: + +.. note:: + + If ``pytest`` finds a "a/b/test_module.py" test file while + recursing into the filesystem it determines the import name + as follows: + + * determine ``basedir``: this is the first "upward" (towards the root) + directory not containing an ``__init__.py``. If e.g. both ``a`` + and ``b`` contain an ``__init__.py`` file then the parent directory + of ``a`` will become the ``basedir``. + + * perform ``sys.path.insert(0, basedir)`` to make the test module + importable under the fully qualified import name. + + * ``import a.b.test_module`` where the path is determined + by converting path separators ``/`` into "." characters. This means + you must follow the convention of having directory and file + names map directly to the import names. + + The reason for this somewhat evolved importing technique is + that in larger projects multiple test modules might import + from each other and thus deriving a canonical import name helps + to avoid surprises such as a test modules getting imported twice. + + +.. _`virtualenv`: http://pypi.python.org/pypi/virtualenv +.. _`buildout`: http://www.buildout.org/ +.. _pip: http://pypi.python.org/pypi/pip + +.. _`use tox`: + +Tox +------ + +For development, we recommend to use virtualenv_ environments and pip_ +for installing your application and any dependencies +as well as the ``pytest`` package itself. This ensures your code and +dependencies are isolated from the system Python installation. + +If you frequently release code and want to make sure that your actual +package passes all tests you may want to look into `tox`_, the +virtualenv test automation tool and its `pytest support +<http://testrun.org/tox/latest/example/pytest.html>`_. +Tox helps you to setup virtualenv environments with pre-defined +dependencies and then executing a pre-configured test command with +options. It will run tests against the installed package and not +against your source code checkout, helping to detect packaging +glitches. + +Continuous integration services such as Jenkins_ can make use of the +``--junitxml=PATH`` option to create a JUnitXML file and generate reports. + + +Integrating with setuptools / ``python setup.py test`` / ``pytest-runner`` +-------------------------------------------------------------------------- + +You can integrate test runs into your setuptools based project +with the `pytest-runner <https://pypi.python.org/pypi/pytest-runner>`_ plugin. + +Add this to ``setup.py`` file: + +.. code-block:: python + + from setuptools import setup + + setup( + #..., + setup_requires=['pytest-runner', ...], + tests_require=['pytest', ...], + #..., + ) + + +And create an alias into ``setup.cfg`` file: + + +.. code-block:: ini + + [aliases] + test=pytest + +If you now type:: + + python setup.py test + +this will execute your tests using ``pytest-runner``. As this is a +standalone version of ``pytest`` no prior installation whatsoever is +required for calling the test command. You can also pass additional +arguments to py.test such as your test directory or other +options using ``--addopts``. + + +Manual Integration +^^^^^^^^^^^^^^^^^^ + +If for some reason you don't want/can't use ``pytest-runner``, you can write +your own setuptools Test command for invoking pytest. + +.. code-block:: python + + import sys + + from setuptools.command.test import test as TestCommand + + + class PyTest(TestCommand): + user_options = [('pytest-args=', 'a', "Arguments to pass to py.test")] + + def initialize_options(self): + TestCommand.initialize_options(self) + self.pytest_args = [] + + def run_tests(self): + #import here, cause outside the eggs aren't loaded + import pytest + errno = pytest.main(self.pytest_args) + sys.exit(errno) + + + setup( + #..., + tests_require=['pytest'], + cmdclass = {'test': PyTest}, + ) + +Now if you run:: + + python setup.py test + +this will download ``pytest`` if needed and then run your tests +as you would expect it to. You can pass a single string of arguments +using the ``--pytest-args`` or ``-a`` command-line option. For example:: + + python setup.py test -a "--durations=5" + +is equivalent to running ``py.test --durations=5``. + + +.. _standalone: +.. _`genscript method`: + +(deprecated) Create a pytest standalone script +----------------------------------------------- + +.. deprecated:: 2.8 + +.. note:: + + ``genscript`` has been deprecated because: + + * It cannot support plugins, rendering its usefulness extremely limited; + * Tooling has become much better since ``genscript`` was introduced; + * It is possible to build a zipped ``pytest`` application without the + shortcomings above. + + There's no planned version in which this command will be removed + at the moment of this writing, but its use is discouraged for new + applications. + +If you are a maintainer or application developer and want people +who don't deal with python much to easily run tests you may generate +a standalone ``pytest`` script:: + + py.test --genscript=runtests.py + +This generates a ``runtests.py`` script which is a fully functional basic +``pytest`` script, running unchanged under Python2 and Python3. +You can tell people to download the script and then e.g. run it like this:: + + python runtests.py + + +.. include:: links.inc |