diff options
Diffstat (limited to 'python/mach/docs')
| -rw-r--r-- | python/mach/docs/commands.rst | 145 | ||||
| -rw-r--r-- | python/mach/docs/driver.rst | 51 | ||||
| -rw-r--r-- | python/mach/docs/index.rst | 75 | ||||
| -rw-r--r-- | python/mach/docs/logging.rst | 100 | ||||
| -rw-r--r-- | python/mach/docs/settings.rst | 140 |
5 files changed, 511 insertions, 0 deletions
diff --git a/python/mach/docs/commands.rst b/python/mach/docs/commands.rst new file mode 100644 index 000000000..af2973dd7 --- /dev/null +++ b/python/mach/docs/commands.rst @@ -0,0 +1,145 @@ +.. _mach_commands: + +===================== +Implementing Commands +===================== + +Mach commands are defined via Python decorators. + +All the relevant decorators are defined in the *mach.decorators* module. +The important decorators are as follows: + +:py:func:`CommandProvider <mach.decorators.CommandProvider>` + A class decorator that denotes that a class contains mach + commands. The decorator takes no arguments. + +:py:func:`Command <mach.decorators.Command>` + A method decorator that denotes that the method should be called when + the specified command is requested. The decorator takes a command name + as its first argument and a number of additional arguments to + configure the behavior of the command. + +:py:func:`CommandArgument <mach.decorators.CommandArgument>` + A method decorator that defines an argument to the command. Its + arguments are essentially proxied to ArgumentParser.add_argument() + +:py:func:`SubCommand <mach.decorators.SubCommand>` + A method decorator that denotes that the method should be a + sub-command to an existing ``@Command``. The decorator takes the + parent command name as its first argument and the sub-command name + as its second argument. + + ``@CommandArgument`` can be used on ``@SubCommand`` instances just + like they can on ``@Command`` instances. + +Classes with the ``@CommandProvider`` decorator **must** have an +``__init__`` method that accepts 1 or 2 arguments. If it accepts 2 +arguments, the 2nd argument will be a +:py:class:`mach.base.CommandContext` instance. + +Here is a complete example: + +.. code-block:: python + + from mach.decorators import ( + CommandArgument, + CommandProvider, + Command, + ) + + @CommandProvider + class MyClass(object): + @Command('doit', help='Do ALL OF THE THINGS.') + @CommandArgument('--force', '-f', action='store_true', + help='Force doing it.') + def doit(self, force=False): + # Do stuff here. + +When the module is loaded, the decorators tell mach about all handlers. +When mach runs, it takes the assembled metadata from these handlers and +hooks it up to the command line driver. Under the hood, arguments passed +to the decorators are being used to help mach parse command arguments, +formulate arguments to the methods, etc. See the documentation in the +:py:mod:`mach.base` module for more. + +The Python modules defining mach commands do not need to live inside the +main mach source tree. + +Conditionally Filtering Commands +================================ + +Sometimes it might only make sense to run a command given a certain +context. For example, running tests only makes sense if the product +they are testing has been built, and said build is available. To make +sure a command is only runnable from within a correct context, you can +define a series of conditions on the +:py:func:`Command <mach.decorators.Command>` decorator. + +A condition is simply a function that takes an instance of the +:py:func:`mach.decorators.CommandProvider` class as an argument, and +returns ``True`` or ``False``. If any of the conditions defined on a +command return ``False``, the command will not be runnable. The +docstring of a condition function is used in error messages, to explain +why the command cannot currently be run. + +Here is an example: + +.. code-block:: python + + from mach.decorators import ( + CommandProvider, + Command, + ) + + def build_available(cls): + """The build needs to be available.""" + return cls.build_path is not None + + @CommandProvider + class MyClass(MachCommandBase): + def __init__(self, build_path=None): + self.build_path = build_path + + @Command('run_tests', conditions=[build_available]) + def run_tests(self): + # Do stuff here. + +It is important to make sure that any state needed by the condition is +available to instances of the command provider. + +By default all commands without any conditions applied will be runnable, +but it is possible to change this behaviour by setting +``require_conditions`` to ``True``: + +.. code-block:: python + + m = mach.main.Mach() + m.require_conditions = True + +Minimizing Code in Commands +=========================== + +Mach command modules, classes, and methods work best when they are +minimal dispatchers. The reason is import bloat. Currently, the mach +core needs to import every Python file potentially containing mach +commands for every command invocation. If you have dozens of commands or +commands in modules that import a lot of Python code, these imports +could slow mach down and waste memory. + +It is thus recommended that mach modules, classes, and methods do as +little work as possible. Ideally the module should only import from +the :py:mod:`mach` package. If you need external modules, you should +import them from within the command method. + +To keep code size small, the body of a command method should be limited +to: + +1. Obtaining user input (parsing arguments, prompting, etc) +2. Calling into some other Python package +3. Formatting output + +Of course, these recommendations can be ignored if you want to risk +slower performance. + +In the future, the mach driver may cache the dispatching information or +have it intelligently loaded to facilitate lazy loading. diff --git a/python/mach/docs/driver.rst b/python/mach/docs/driver.rst new file mode 100644 index 000000000..022ebe657 --- /dev/null +++ b/python/mach/docs/driver.rst @@ -0,0 +1,51 @@ +.. _mach_driver: + +======= +Drivers +======= + +Entry Points +============ + +It is possible to use setuptools' entry points to load commands +directly from python packages. A mach entry point is a function which +returns a list of files or directories containing mach command +providers. e.g.: + +.. code-block:: python + + def list_providers(): + providers = [] + here = os.path.abspath(os.path.dirname(__file__)) + for p in os.listdir(here): + if p.endswith('.py'): + providers.append(os.path.join(here, p)) + return providers + +See http://pythonhosted.org/setuptools/setuptools.html#dynamic-discovery-of-services-and-plugins +for more information on creating an entry point. To search for entry +point plugins, you can call +:py:meth:`mach.main.Mach.load_commands_from_entry_point`. e.g.: + +.. code-block:: python + + mach.load_commands_from_entry_point("mach.external.providers") + +Adding Global Arguments +======================= + +Arguments to mach commands are usually command-specific. However, +mach ships with a handful of global arguments that apply to all +commands. + +It is possible to extend the list of global arguments. In your +*mach driver*, simply call +:py:meth:`mach.main.Mach.add_global_argument`. e.g.: + +.. code-block:: python + + mach = mach.main.Mach(os.getcwd()) + + # Will allow --example to be specified on every mach command. + mach.add_global_argument('--example', action='store_true', + help='Demonstrate an example global argument.') diff --git a/python/mach/docs/index.rst b/python/mach/docs/index.rst new file mode 100644 index 000000000..cd2056333 --- /dev/null +++ b/python/mach/docs/index.rst @@ -0,0 +1,75 @@ +==== +mach +==== + +Mach (German for *do*) is a generic command dispatcher for the command +line. + +To use mach, you install the mach core (a Python package), create an +executable *driver* script (named whatever you want), and write mach +commands. When the *driver* is executed, mach dispatches to the +requested command handler automatically. + +Features +======== + +On a high level, mach is similar to using argparse with subparsers (for +command handling). When you dig deeper, mach offers a number of +additional features: + +Distributed command definitions + With optparse/argparse, you have to define your commands on a central + parser instance. With mach, you annotate your command methods with + decorators and mach finds and dispatches to them automatically. + +Command categories + Mach commands can be grouped into categories when displayed in help. + This is currently not possible with argparse. + +Logging management + Mach provides a facility for logging (both classical text and + structured) that is available to any command handler. + +Settings files + Mach provides a facility for reading settings from an ini-like file + format. + +Components +========== + +Mach is conceptually composed of the following components: + +core + The mach core is the core code powering mach. This is a Python package + that contains all the business logic that makes mach work. The mach + core is common to all mach deployments. + +commands + These are what mach dispatches to. Commands are simply Python methods + registered as command names. The set of commands is unique to the + environment mach is deployed in. + +driver + The *driver* is the entry-point to mach. It is simply an executable + script that loads the mach core, tells it where commands can be found, + then asks the mach core to handle the current request. The driver is + unique to the deployed environment. But, it's usually based on an + example from this source tree. + +Project State +============= + +mach was originally written as a command dispatching framework to aid +Firefox development. While the code is mostly generic, there are still +some pieces that closely tie it to Mozilla/Firefox. The goal is for +these to eventually be removed and replaced with generic features so +mach is suitable for anybody to use. Until then, mach may not be the +best fit for you. + +.. toctree:: + :maxdepth: 1 + + commands + driver + logging + settings diff --git a/python/mach/docs/logging.rst b/python/mach/docs/logging.rst new file mode 100644 index 000000000..ff245cf03 --- /dev/null +++ b/python/mach/docs/logging.rst @@ -0,0 +1,100 @@ +.. _mach_logging: + +======= +Logging +======= + +Mach configures a built-in logging facility so commands can easily log +data. + +What sets the logging facility apart from most loggers you've seen is +that it encourages structured logging. Instead of conventional logging +where simple strings are logged, the internal logging mechanism logs all +events with the following pieces of information: + +* A string *action* +* A dict of log message fields +* A formatting string + +Essentially, instead of assembling a human-readable string at +logging-time, you create an object holding all the pieces of data that +will constitute your logged event. For each unique type of logged event, +you assign an *action* name. + +Depending on how logging is configured, your logged event could get +written a couple of different ways. + +JSON Logging +============ + +Where machines are the intended target of the logging data, a JSON +logger is configured. The JSON logger assembles an array consisting of +the following elements: + +* Decimal wall clock time in seconds since UNIX epoch +* String *action* of message +* Object with structured message data + +The JSON-serialized array is written to a configured file handle. +Consumers of this logging stream can just perform a readline() then feed +that into a JSON deserializer to reconstruct the original logged +message. They can key off the *action* element to determine how to +process individual events. There is no need to invent a parser. +Convenient, isn't it? + +Logging for Humans +================== + +Where humans are the intended consumer of a log message, the structured +log message are converted to more human-friendly form. This is done by +utilizing the *formatting* string provided at log time. The logger +simply calls the *format* method of the formatting string, passing the +dict containing the message's fields. + +When *mach* is used in a terminal that supports it, the logging facility +also supports terminal features such as colorization. This is done +automatically in the logging layer - there is no need to control this at +logging time. + +In addition, messages intended for humans typically prepends every line +with the time passed since the application started. + +Logging HOWTO +============= + +Structured logging piggybacks on top of Python's built-in logging +infrastructure provided by the *logging* package. We accomplish this by +taking advantage of *logging.Logger.log()*'s *extra* argument. To this +argument, we pass a dict with the fields *action* and *params*. These +are the string *action* and dict of message fields, respectively. The +formatting string is passed as the *msg* argument, like normal. + +If you were logging to a logger directly, you would do something like: + +.. code-block:: python + + logger.log(logging.INFO, 'My name is {name}', + extra={'action': 'my_name', 'params': {'name': 'Gregory'}}) + +The JSON logging would produce something like:: + + [1339985554.306338, "my_name", {"name": "Gregory"}] + +Human logging would produce something like:: + + 0.52 My name is Gregory + +Since there is a lot of complexity using logger.log directly, it is +recommended to go through a wrapping layer that hides part of the +complexity for you. The easiest way to do this is by utilizing the +LoggingMixin: + +.. code-block:: python + + import logging + from mach.mixin.logging import LoggingMixin + + class MyClass(LoggingMixin): + def foo(self): + self.log(logging.INFO, 'foo_start', {'bar': True}, + 'Foo performed. Bar: {bar}') diff --git a/python/mach/docs/settings.rst b/python/mach/docs/settings.rst new file mode 100644 index 000000000..b51dc54a2 --- /dev/null +++ b/python/mach/docs/settings.rst @@ -0,0 +1,140 @@ +.. _mach_settings: + +======== +Settings +======== + +Mach can read settings in from a set of configuration files. These +configuration files are either named ``machrc`` or ``.machrc`` and +are specified by the bootstrap script. In mozilla-central, these files +can live in ``~/.mozbuild`` and/or ``topsrcdir``. + +Settings can be specified anywhere, and used both by mach core or +individual commands. + + +Core Settings +============= + +These settings are implemented by mach core. + +* alias - Create a command alias. This is useful if you want to alias a command to something else, optionally including some defaults. It can either be used to create an entire new command, or provide defaults for an existing one. For example: + +.. parsed-literal:: + + [alias] + mochitest = mochitest -f browser + browser-test = mochitest -f browser + + +Defining Settings +================= + +Settings need to be explicitly defined, along with their type, +otherwise mach will throw when trying to access them. + +To define settings, use the :func:`~decorators.SettingsProvider` +decorator in an existing mach command module. E.g: + +.. code-block:: python + + from mach.decorators import SettingsProvider + + @SettingsProvider + class ArbitraryClassName(object): + config_settings = [ + ('foo.bar', 'string'), + ('foo.baz', 'int', 0, set([0,1,2])), + ] + +``@SettingsProvider``'s must specify a variable called ``config_settings`` +that returns a list of tuples. Alternatively, it can specify a function +called ``config_settings`` that returns a list of tuples. + +Each tuple is of the form: + +.. code-block:: python + + ('<section>.<option>', '<type>', default, extra) + +``type`` is a string and can be one of: +string, boolean, int, pos_int, path + +``default`` is optional, and provides a default value in case none was +specified by any of the configuration files. + +``extra`` is also optional and is a dict containing additional key/value +pairs to add to the setting's metadata. The following keys may be specified +in the ``extra`` dict: + * ``choices`` - A set of allowed values for the setting. + +Wildcards +--------- + +Sometimes a section should allow arbitrarily defined options from the user, such +as the ``alias`` section mentioned above. To define a section like this, use ``*`` +as the option name. For example: + +.. parsed-literal:: + + ('foo.*', 'string') + +This allows configuration files like this: + +.. parsed-literal:: + + [foo] + arbitrary1 = some string + arbitrary2 = some other string + + +Documenting Settings +==================== + +All settings must at least be documented in the en_US locale. Otherwise, +running ``mach settings`` will raise. Mach uses gettext to perform localization. + +A handy command exists to generate the localization files: + +.. parsed-literal:: + + mach settings locale-gen <section> + +You'll be prompted to add documentation for all options in section with the +en_US locale. To add documentation in another locale, pass in ``--locale``. + + +Accessing Settings +================== + +Now that the settings are defined and documented, they're accessible from +individual mach commands if the command receives a context in its constructor. +For example: + +.. code-block:: python + + from mach.decorators import ( + Command, + CommandProvider, + SettingsProvider, + ) + + @SettingsProvider + class ExampleSettings(object): + config_settings = [ + ('a.b', 'string', 'default'), + ('foo.bar', 'string'), + ('foo.baz', 'int', 0, {'choices': set([0,1,2])}), + ] + + @CommandProvider + class Commands(object): + def __init__(self, context): + self.settings = context.settings + + @Command('command', category='misc', + description='Prints a setting') + def command(self): + print(self.settings.a.b) + for option in self.settings.foo: + print(self.settings.foo[option]) |