diff options
Diffstat (limited to 'build/docs')
26 files changed, 2880 insertions, 0 deletions
diff --git a/build/docs/androideclipse.rst b/build/docs/androideclipse.rst new file mode 100644 index 000000000..a8c048130 --- /dev/null +++ b/build/docs/androideclipse.rst @@ -0,0 +1,90 @@ +.. _build_androideclipse: + +======================== +Android Eclipse Projects +======================== + +The build system contains alpha support for generating Android Eclipse +project files to aid with development. + +To generate Android Eclipse project files, you'll need to have a fully +built and packaged tree:: + + mach build && mach package + +(This is because Eclipse itself packages an APK containing +``omni.ja``, and ``omni.ja`` is only assembled during packaging.) + +Then, simply generate the Android Eclipse build backend:: + + mach build-backend -b AndroidEclipse + +If all goes well, the path to the generated projects should be +printed (currently, ``$OBJDIR/android_eclipse``). + +To use the generated Android Eclipse project files, you'll need to +have a recent version of Eclipse (see `Tested Versions`_) with the +`Eclipse ADT plugin +<http://developer.android.com/tools/sdk/eclipse-adt.html>`_ +installed. You can then import all the projects into Eclipse using +*File > Import ... > General > Existing Projects into Workspace*. + +Updating Project Files +====================== + +As you pull and update the source tree, your Android Eclipse files may +fall out of sync with the build configuration. The tree should still +build fine from within Eclipse, but source files may be missing and in +rare circumstances Eclipse's index may not have the proper build +configuration. + +To account for this, you'll want to periodically regenerate the +Android Eclipse project files. You can do this by running ``mach build +&& mach package && mach build-backend -b AndroidEclipse`` from the +command line. It's a good idea to refresh and clean build all projects +in Eclipse after doing this. + +In future, we'd like to include an Android Eclipse run configuration +or build target that integrates updating the project files. + +Currently, regeneration rewrites the original project files. **If +you've made any customizations to the projects, they will likely get +overwritten.** We would like to improve this user experience in the +future. + +Troubleshooting +=============== + +If Eclipse's builder gets confused, you should always refresh and +clean build all projects. If Eclipse's builder is continually +confused, you can see a log of what is happening at +``$OBJDIR/android_eclipse/build.log``. + +If you run into memory problems executing ``dex``, you should +`Increase Eclipse's memory limits <http://stackoverflow.com/a/11093228>`_. + +The produced Android Eclipse project files are unfortunately not +portable. Please don't move them around. + +Structure of Android Eclipse projects +===================================== + +The Android Eclipse backend generates several projects spanning Fennec +itself and its tests. You'll mostly interact with the *Fennec* project +itself. + +In future, we'd like to expand this documentation to include some of +the technical details of how the Eclipse integration works, and how to +add additional Android Eclipse projects using the ``moz.build`` +system. + +Tested Versions +=============== + +=============== ==================================== ================= +OS Version Working as of +=============== ==================================== ================= +Mac OS X Luna (Build id: 20130919-0819) February 2014 +Mac OS X Kepler (Build id: 20131219-0014) February 2014 +Mac OS X 10.8.5 Kepler (Build id: 20130919-0819) February 2014 +=============== ==================================== ================= diff --git a/build/docs/build-overview.rst b/build/docs/build-overview.rst new file mode 100644 index 000000000..a81531bc9 --- /dev/null +++ b/build/docs/build-overview.rst @@ -0,0 +1,117 @@ +.. _build_overview: + +===================== +Build System Overview +===================== + +This document provides an overview on how the build system works. It is +targeted at people wanting to learn about internals of the build system. +It is not meant for persons who casually interact with the build system. +That being said, knowledge empowers, so consider reading on. + +The build system is composed of many different components working in +harmony to build the source tree. We begin with a graphic overview. + +.. graphviz:: + + digraph build_components { + rankdir="LR"; + "configure" -> "config.status" -> "build backend" -> "build output" + } + +Phase 1: Configuration +====================== + +Phase 1 centers around the ``configure`` script, which is a bash shell script. +The file is generated from a file called ``configure.in`` which is written in M4 +and processed using Autoconf 2.13 to create the final configure script. +You don't have to worry about how you obtain a ``configure`` file: the build +system does this for you. + +The primary job of ``configure`` is to determine characteristics of the system +and compiler, apply options passed into it, and validate everything looks OK to +build. The primary output of the ``configure`` script is an executable file +in the object directory called ``config.status``. ``configure`` also produces +some additional files (like ``autoconf.mk``). However, the most important file +in terms of architecture is ``config.status``. + +The existence of a ``config.status`` file may be familiar to those who have worked +with Autoconf before. However, Mozilla's ``config.status`` is different from almost +any other ``config.status`` you've ever seen: it's written in Python! Instead of +having our ``configure`` script produce a shell script, we have it generating +Python. + +Now is as good a time as any to mention that Python is prevalent in our build +system. If we need to write code for the build system, we do it in Python. +That's just how we roll. For more, see :ref:`python`. + +``config.status`` contains 2 parts: data structures representing the output of +``configure`` and a command-line interface for preparing/configuring/generating +an appropriate build backend. (A build backend is merely a tool used to build +the tree - like GNU Make or Tup). These data structures essentially describe +the current state of the system and what the existing build configuration looks +like. For example, it defines which compiler to use, how to invoke it, which +application features are enabled, etc. You are encouraged to open up +``config.status`` to have a look for yourself! + +Once we have emitted a ``config.status`` file, we pass into the realm of +phase 2. + +Phase 2: Build Backend Preparation and the Build Definition +=========================================================== + +Once ``configure`` has determined what the current build configuration is, +we need to apply this to the source tree so we can actually build. + +What essentially happens is the automatically-produced ``config.status`` Python +script is executed as soon as ``configure`` has generated it. ``config.status`` +is charged with the task of tell a tool how to build the tree. To do this, +``config.status`` must first scan the build system definition. + +The build system definition consists of various ``moz.build`` files in the tree. +There is roughly one ``moz.build`` file per directory or per set of related directories. +Each ``moz.build`` files defines how its part of the build config works. For +example it says *I want these C++ files compiled* or *look for additional +information in these directories.* config.status starts with the ``moz.build`` +file from the root directory and then descends into referenced ``moz.build`` +files by following ``DIRS`` variables or similar. + +As the ``moz.build`` files are read, data structures describing the overall +build system definition are emitted. These data structures are then fed into a +build backend, which then performs actions, such as writing out files to +be read by a build tool. e.g. a ``make`` backend will write a +``Makefile``. + +When ``config.status`` runs, you'll see the following output:: + + Reticulating splines... + Finished reading 1096 moz.build files into 1276 descriptors in 2.40s + Backend executed in 2.39s + 2188 total backend files. 0 created; 1 updated; 2187 unchanged + Total wall time: 5.03s; CPU time: 3.79s; Efficiency: 75% + +What this is saying is that a total of *1096* ``moz.build`` files were read. +Altogether, *1276* data structures describing the build configuration were +derived from them. It took *2.40s* wall time to just read these files and +produce the data structures. The *1276* data structures were fed into the +build backend which then determined it had to manage *2188* files derived +from those data structures. Most of them already existed and didn't need +changed. However, *1* was updated as a result of the new configuration. +The whole process took *5.03s*. Although, only *3.79s* was in +CPU time. That likely means we spent roughly *25%* of the time waiting on +I/O. + +For more on how ``moz.build`` files work, see :ref:`mozbuild-files`. + +Phase 3: Invokation of the Build Backend +======================================== + +When most people think of the build system, they think of phase 3. This is +where we take all the code in the tree and produce Firefox or whatever +application you are creating. Phase 3 effectively takes whatever was +generated by phase 2 and runs it. Since the dawn of Mozilla, this has been +make consuming Makefiles. However, with the transition to moz.build files, +you may soon see non-Make build backends, such as Tup or Visual Studio. + +When building the tree, most of the time is spent in phase 3. This is when +header files are installed, C++ files are compiled, files are preprocessed, etc. diff --git a/build/docs/build-targets.rst b/build/docs/build-targets.rst new file mode 100644 index 000000000..dacd46c7f --- /dev/null +++ b/build/docs/build-targets.rst @@ -0,0 +1,62 @@ +.. _build_targets: + +============= +Build Targets +============= + +When you build with ``mach build``, there are some special targets that can be +built. This page attempts to document them. + +Partial Tree Targets +==================== + +The targets in this section only build part of the tree. Please note that +partial tree builds can be unreliable. Use at your own risk. + +export + Build the *export* tier. The *export* tier builds everything that is + required for C/C++ compilation. It stages all header files, processes + IDLs, etc. + +compile + Build the *compile* tier. The *compile* tier compiles all C/C++ files. + +libs + Build the *libs* tier. The *libs* tier performs linking and performs + most build steps which aren't related to compilation. + +tools + Build the *tools* tier. The *tools* tier mostly deals with supplementary + tools and compiled tests. It will link tools against libXUL, including + compiled test binaries. + +binaries: + Recompiles and relinks C/C++ files. Only works after a complete normal + build, but allows for much faster rebuilds of C/C++ code. For performance + reasons, however, it skips nss, nspr, icu and ffi. This is targeted to + improve local developer workflow when touching C/C++ code. + +install-manifests + Process install manifests. Install manifests handle the installation of + files into the object directory. + + Unless ``NO_REMOVE=1`` is defined in the environment, files not accounted + in the install manifests will be deleted from the object directory. + +install-tests + Processes the tests install manifest. + +Common Actions +============== + +The targets in this section correspond to common build-related actions. Many +of the actions in this section are effectively frontends to shell scripts. +These actions will likely all be replaced by mach commands someday. + +buildsymbols + Create a symbols archive for the current build. + + This must be performed after a successful build. + +check + Run build system tests. diff --git a/build/docs/cppeclipse.rst b/build/docs/cppeclipse.rst new file mode 100644 index 000000000..4492add47 --- /dev/null +++ b/build/docs/cppeclipse.rst @@ -0,0 +1,54 @@ +.. _build_cppeclipse: + +===================== +Cpp Eclipse Projects +===================== + +For additional information on using Eclipse CDT see +`the MDN page +<https://developer.mozilla.org/en-US/docs/Eclipse_CDT>`_. + +The build system contains alpha support for generating C++ Eclipse +project files to aid with development. + +Please report bugs to bugzilla and make them depend on bug 973770. + +To generate a C++ Eclipse project files, you'll need to have a fully +built tree:: + + mach build + +Then, simply generate the Android Eclipse build backend:: + + mach build-backend -b CppEclipse + +If all goes well, the path to the generated workspace should be +printed (currently, ``$OBJDIR/android_eclipse``). + +To use the generated Android Eclipse project files, you'll need to +have a Eclipse CDT 8.3 (We plan to follow the latest Eclipse release) +`Eclipse CDT plugin +<https://www.eclipse.org/cdt/>`_ +installed. You can then import all the projects into Eclipse using +*File > Import ... > General > Existing Projects into Workspace* +-only- if you have not ran the background indexer. + +Updating Project Files +====================== + +As you pull and update the source tree, your C++ Eclipse files may +fall out of sync with the build configuration. The tree should still +build fine from within Eclipse, but source files may be missing and in +rare circumstances Eclipse's index may not have the proper build +configuration. + +To account for this, you'll want to periodically regenerate the +Android Eclipse project files. You can do this by running ``mach build +&& mach build-backend -b CppEclipse`` from the +command line. + +Currently, regeneration rewrites the original project files. **If +you've made any customizations to the projects, they will likely get +overwritten.** We would like to improve this user experience in the +future. + diff --git a/build/docs/defining-binaries.rst b/build/docs/defining-binaries.rst new file mode 100644 index 000000000..97dadbbd5 --- /dev/null +++ b/build/docs/defining-binaries.rst @@ -0,0 +1,351 @@ +.. _defining_binaries: + +====================================== +Defining Binaries for the Build System +====================================== + +One part of what the build system does is compile C/C++ and link the resulting +objects to produce executables and/or libraries. This document describes the +basics of defining what is going to be built and how. All the following +describes constructs to use in moz.build files. + + +Source files +============ + +Source files to be used in a given directory are registered in the ``SOURCES`` +and ``UNIFIED_SOURCES`` variables. ``UNIFIED_SOURCES`` have a special behavior +in that they are aggregated by batches of 16, requiring, for example, that there +are no conflicting variables in those source files. + +``SOURCES`` and ``UNIFIED_SOURCES`` are lists which must be appended to, and +each append requires the given list to be alphanumerically ordered. + +.. code-block:: python + + UNIFIED_SOURCES += [ + 'FirstSource.cpp', + 'SecondSource.cpp', + 'ThirdSource.cpp', + ] + + SOURCES += [ + 'OtherSource.cpp', + ] + +``SOURCES`` and ``UNIFIED_SOURCES`` can contain a mix of different file types, +for C, C++, and Objective C. + + +Static Libraries +================ + +To build a static library, other than defining the source files (see above), one +just needs to define a library name with the ``Library`` template. + +.. code-block:: python + + Library('foo') + +The library file name will be ``libfoo.a`` on UNIX systems and ``foo.lib`` on +Windows. + +If the static library needs to aggregate other static libraries, a list of +``Library`` names can be added to the ``USE_LIBS`` variable. Like ``SOURCES``, it +requires the appended list to be alphanumerically ordered. + +.. code-block:: python + + USE_LIBS += ['bar', 'baz'] + +If there are multiple directories containing the same ``Library`` name, it is +possible to disambiguate by prefixing with the path to the wanted one (relative +or absolute): + +.. code-block:: python + + USE_LIBS += [ + '/path/from/topsrcdir/to/bar', + '../relative/baz', + ] + +Note that the leaf name in those paths is the ``Library`` name, not an actual +file name. + +Note that currently, the build system may not create an actual library for +static libraries. It is an implementation detail that shouldn't need to be +worried about. + +As a special rule, ``USE_LIBS`` is allowed to contain references to shared +libraries. In such cases, programs and shared libraries linking this static +library will inherit those shared library dependencies. + + +Intermediate (Static) Libraries +=============================== + +In many cases in the tree, static libraries are built with the only purpose +of being linked into another, bigger one (like libxul). Instead of adding all +required libraries to ``USE_LIBS`` for the bigger one, it is possible to tell +the build system that the library built in the current directory is meant to +be linked to that bigger library, with the ``FINAL_LIBRARY`` variable. + +.. code-block:: python + + FINAL_LIBRARY = 'xul' + +The ``FINAL_LIBRARY`` value must match a unique ``Library`` name somewhere +in the tree. + +As a special rule, those intermediate libraries don't need a ``Library`` name +for themselves. + + +Shared Libraries +================ + +Sometimes, we want shared libraries, a.k.a. dynamic libraries. Such libraries +are defined similarly to static libraries, using the ``SharedLibrary`` template +instead of ``Library``. + +.. code-block:: python + + SharedLibrary('foo') + +When this template is used, no static library is built. See further below to +build both types of libraries. + +With a ``SharedLibrary`` name of ``foo``, the library file name will be +``libfoo.dylib`` on OSX, ``libfoo.so`` on ELF systems (Linux, etc.), and +``foo.dll`` on Windows. On Windows, there is also an import library named +``foo.lib``, used on the linker command line. ``libfoo.dylib`` and +``libfoo.so`` are considered the import library name for, resp. OSX and ELF +systems. + +On OSX, one may want to create a special kind of dynamic library: frameworks. +This is done with the ``Framework`` template. + +.. code-block:: python + + Framework('foo') + +With a ``Framework`` name of ``foo``, the framework file name will be ``foo``. +This template however affects the behavior on all platforms, so it needs to +be set only on OSX. + + +Executables +=========== + +Executables, a.k.a. programs, are, in the simplest form, defined with the +``Program`` template. + +.. code-block:: python + + Program('foobar') + +On UNIX systems, the executable file name will be ``foobar``, while on Windows, +it will be ``foobar.exe``. + +Like static and shared libraries, the build system can be instructed to link +libraries to the executable with ``USE_LIBS``, listing various ``Library`` +names. + +In some cases, we want to create an executable per source file in the current +directory, in which case we can use the ``SimplePrograms`` template + +.. code-block:: python + + SimplePrograms([ + 'FirstProgram', + 'SecondProgram', + ]) + +Contrary to ``Program``, which requires corresponding ``SOURCES``, when using +``SimplePrograms``, the corresponding ``SOURCES`` are implied. If the +corresponding ``sources`` have an extension different from ``.cpp``, it is +possible to specify the proper extension: + +.. code-block:: python + + SimplePrograms([ + 'ThirdProgram', + 'FourthProgram', + ], ext='.c') + +Please note this construct was added for compatibility with what already lives +in the mozilla tree ; it is recommended not to add new simple programs with +sources with a different extension than ``.cpp``. + +Similar to ``SimplePrograms``, is the ``CppUnitTests`` template, which defines, +with the same rules, C++ unit tests programs. Like ``SimplePrograms``, it takes +an ``ext`` argument to specify the extension for the corresponding ``SOURCES``, +if it's different from ``.cpp``. + + +Linking with system libraries +============================= + +Programs and libraries usually need to link with system libraries, such as a +widget toolkit, etc. Those required dependencies can be given with the +``OS_LIBS`` variable. + +.. code-block:: python + + OS_LIBS += [ + 'foo', + 'bar', + ] + +This expands to ``foo.lib bar.lib`` when building with MSVC, and +``-lfoo -lbar`` otherwise. + +For convenience with ``pkg-config``, ``OS_LIBS`` can also take linker flags +such as ``-L/some/path`` and ``-llib``, such that it is possible to directly +assign ``LIBS`` variables from ``CONFIG``, such as: + +.. code-block:: python + + OS_LIBS += CONFIG['MOZ_PANGO_LIBS'] + +(assuming ``CONFIG['MOZ_PANGO_LIBS']`` is a list, not a string) + +Like ``USE_LIBS``, this variable applies to static and shared libraries, as +well as programs. + + +Libraries from third party build system +======================================= + +Some libraries in the tree are not built by the moz.build-governed build +system, and there is no ``Library`` corresponding to them. + +However, ``USE_LIBS`` allows to reference such libraries by giving a full +path (like when disambiguating identical ``Library`` names). The same naming +rules apply as other uses of ``USE_LIBS``, so only the library name without +prefix and suffix shall be given. + +.. code-block:: python + + USE_LIBS += [ + '/path/from/topsrcdir/to/third-party/bar', + '../relative/third-party/baz', + ] + +Note that ``/path/from/topsrcdir/to/third-party`` and +``../relative/third-party/baz`` must lead under a subconfigured directory (a +directory with an AC_OUTPUT_SUBDIRS in configure.in), or ``security/nss``. + + +Building both static and shared libraries +========================================= + +When both types of libraries are required, one needs to set both +``FORCE_SHARED_LIB`` and ``FORCE_STATIC_LIB`` boolean variables. + +.. code-block:: python + + FORCE_SHARED_LIB = True + FORCE_STATIC_LIB = True + +But because static libraries and Windows import libraries have the same file +names, either the static or the shared library name needs to be different +than the name given to the ``Library`` template. + +The ``STATIC_LIBRARY_NAME`` and ``SHARED_LIBRARY_NAME`` variables can be used +to change either the static or the shared library name. + +.. code-block:: python + + Library('foo') + STATIC_LIBRARY_NAME = 'foo_s' + +With the above, on Windows, ``foo_s.lib`` will be the static library, +``foo.dll`` the shared library, and ``foo.lib`` the import library. + +In some cases, for convenience, it is possible to set both +``STATIC_LIBRARY_NAME`` and ``SHARED_LIBRARY_NAME``. For example: + +.. code-block:: python + + Library('mylib') + STATIC_LIBRARY_NAME = 'mylib_s' + SHARED_LIBRARY_NAME = CONFIG['SHARED_NAME'] + +This allows to use ``mylib`` in the ``USE_LIBS`` of another library or +executable. + +When refering to a ``Library`` name building both types of libraries in +``USE_LIBS``, the shared library is chosen to be linked. But sometimes, +it is wanted to link the static version, in which case the ``Library`` name +needs to be prefixed with ``static:`` in ``USE_LIBS`` + +:: + + a/moz.build: + Library('mylib') + FORCE_SHARED_LIB = True + FORCE_STATIC_LIB = True + STATIC_LIBRARY_NAME = 'mylib_s' + b/moz.build: + Program('myprog') + USE_LIBS += [ + 'static:mylib', + ] + + +Miscellaneous +============= + +The ``SDK_LIBRARY`` boolean variable defines whether the library in the current +directory is going to be installed in the SDK. + +The ``SONAME`` variable declares a "shared object name" for the library. It +defaults to the ``Library`` name or the ``SHARED_LIBRARY_NAME`` if set. When +linking to a library with a ``SONAME``, the resulting library or program will +have a dependency on the library with the name corresponding to the ``SONAME`` +instead of the ``Library`` name. This only impacts ELF systems. + +:: + + a/moz.build: + Library('mylib') + b/moz.build: + Library('otherlib') + SONAME = 'foo' + c/moz.build: + Program('myprog') + USE_LIBS += [ + 'mylib', + 'otherlib', + ] + +On e.g. Linux, the above ``myprog`` will have DT_NEEDED markers for +``libmylib.so`` and ``libfoo.so`` instead of ``libmylib.so`` and +``libotherlib.so`` if there weren't a ``SONAME``. This means the runtime +requirement for ``myprog`` is ``libfoo.so`` instead of ``libotherlib.so``. + + +Gecko-related binaries +====================== + +Some programs or libraries are totally independent of Gecko, and can use the +above mentioned templates. Others are Gecko-related in some way, and may +need XPCOM linkage, mozglue. These things are tedious. A set of additional +templates exists to ease defining such programs and libraries. They are +essentially the same as the above mentioned templates, prefixed with "Gecko": + + - ``GeckoProgram`` + - ``GeckoSimplePrograms`` + - ``GeckoCppUnitTests`` + - ``GeckoSharedLibrary`` + - ``GeckoFramework`` + +There is also ``XPCOMBinaryComponent`` for XPCOM components, which is a +special kind of library. + +All the Gecko-prefixed templates take the same arguments as their +non-Gecko-prefixed counterparts, and can take a few more arguments +for non-standard cases. See the definition of ``GeckoBinary`` in +build/gecko_templates.mozbuild for more details, but most usecases +should not require these additional arguments. diff --git a/build/docs/environment-variables.rst b/build/docs/environment-variables.rst new file mode 100644 index 000000000..c46339159 --- /dev/null +++ b/build/docs/environment-variables.rst @@ -0,0 +1,31 @@ +.. _environment_variables: + +================================================ +Environment Variables Impacting the Build System +================================================ + +Various environment variables have an impact on the behavior of the +build system. This document attempts to document them. + +AUTOCLOBBER + If defines, the build system will automatically clobber as needed. + The default behavior is to print a message and error out when a + clobber is needed. + + This variable is typically defined in a :ref:`mozconfig <mozconfig>` + file via ``mk_add_options``. + +REBUILD_CHECK + If defined, the build system will print information about why + certain files were rebuilt. + + This feature is disabled by default because it makes the build slower. + +MACH_NO_TERMINAL_FOOTER + If defined, the terminal footer displayed when building with mach in + a TTY is disabled. + +MACH_NO_WRITE_TIMES + If defined, mach commands will not prefix output lines with the + elapsed time since program start. This option is equivalent to + passing ``--log-no-times`` to mach. diff --git a/build/docs/files-metadata.rst b/build/docs/files-metadata.rst new file mode 100644 index 000000000..5af4d96fd --- /dev/null +++ b/build/docs/files-metadata.rst @@ -0,0 +1,178 @@ +.. _mozbuild_files_metadata: + +============== +Files Metadata +============== + +:ref:`mozbuild-files` provide a mechanism for attaching metadata to +files. Essentially, you define some flags to set on a file or file +pattern. Later, some tool or process queries for metadata attached to a +file of interest and it does something intelligent with that data. + +Defining Metadata +================= + +Files metadata is defined by using the +:ref:`Files Sub-Context <mozbuild_subcontext_Files>` in ``moz.build`` +files. e.g.:: + + with Files('**/Makefile.in'): + BUG_COMPONENT = ('Core', 'Build Config') + +This working example says, *for all Makefile.in files in every directory +underneath this one - including this directory - set the Bugzilla +component to Core :: Build Config*. + +For more info, read the +:ref:`docs on Files <mozbuild_subcontext_Files>`. + +How Metadata is Read +==================== + +``Files`` metadata is extracted in :ref:`mozbuild_fs_reading_mode`. + +Reading starts by specifying a set of files whose metadata you are +interested in. For each file, the filesystem is walked to the root +of the source directory. Any ``moz.build`` encountered during this +walking are marked as relevant to the file. + +Let's say you have the following filesystem content:: + + /moz.build + /root_file + /dir1/moz.build + /dir1/foo + /dir1/subdir1/foo + /dir2/foo + +For ``/root_file``, the relevant ``moz.build`` files are just +``/moz.build``. + +For ``/dir1/foo`` and ``/dir1/subdir1/foo``, the relevant files are +``/moz.build`` and ``/dir1/moz.build``. + +For ``/dir2``, the relevant file is just ``/moz.build``. + +Once the list of relevant ``moz.build`` files is obtained, each +``moz.build`` file is evaluated. Root ``moz.build`` file first, +leaf-most files last. This follows the rules of +:ref:`mozbuild_fs_reading_mode`, with the set of evaluated ``moz.build`` +files being controlled by filesystem content, not ``DIRS`` variables. + +The file whose metadata is being resolved maps to a set of ``moz.build`` +files which in turn evaluates to a list of contexts. For file metadata, +we only care about one of these contexts: +:ref:`Files <mozbuild_subcontext_Files>`. + +We start with an empty ``Files`` instance to represent the file. As +we encounter a *files sub-context*, we see if it is appropriate to +this file. If it is, we apply its values. This process is repeated +until all *files sub-contexts* have been applied or skipped. The final +state of the ``Files`` instance is used to represent the metadata for +this particular file. + +It may help to visualize this. Say we have 2 ``moz.build`` files:: + + # /moz.build + with Files('*.cpp'): + BUG_COMPONENT = ('Core', 'XPCOM') + + with Files('**/*.js'): + BUG_COMPONENT = ('Firefox', 'General') + + # /foo/moz.build + with Files('*.js'): + BUG_COMPONENT = ('Another', 'Component') + +Querying for metadata for the file ``/foo/test.js`` will reveal 3 +relevant ``Files`` sub-contexts. They are evaluated as follows: + +1. ``/moz.build - Files('*.cpp')``. Does ``/*.cpp`` match + ``/foo/test.js``? **No**. Ignore this context. +2. ``/moz.build - Files('**/*.js')``. Does ``/**/*.js`` match + ``/foo/test.js``? **Yes**. Apply ``BUG_COMPONENT = ('Firefox', 'General')`` + to us. +3. ``/foo/moz.build - Files('*.js')``. Does ``/foo/*.js`` match + ``/foo/test.js``? **Yes**. Apply + ``BUG_COMPONENT = ('Another', 'Component')``. + +At the end of execution, we have +``BUG_COMPONENT = ('Another', 'Component')`` as the metadata for +``/foo/test.js``. + +One way to look at file metadata is as a stack of data structures. +Each ``Files`` sub-context relevant to a given file is applied on top +of the previous state, starting from an empty state. The final state +wins. + +.. _mozbuild_files_metadata_finalizing: + +Finalizing Values +================= + +The default behavior of ``Files`` sub-context evaluation is to apply new +values on top of old. In most circumstances, this results in desired +behavior. However, there are circumstances where this may not be +desired. There is thus a mechanism to *finalize* or *freeze* values. + +Finalizing values is useful for scenarios where you want to prevent +wildcard matches from overwriting previously-set values. This is useful +for one-off files. + +Let's take ``Makefile.in`` files as an example. The build system module +policy dictates that ``Makefile.in`` files are part of the ``Build +Config`` module and should be reviewed by peers of that module. However, +there exist ``Makefile.in`` files in many directories in the source +tree. Without finalization, a ``*`` or ``**`` wildcard matching rule +would match ``Makefile.in`` files and overwrite their metadata. + +Finalizing of values is performed by setting the ``FINAL`` variable +on ``Files`` sub-contexts. See the +:ref:`Files documentation <mozbuild_subcontext_Files>` for more. + +Here is an example with ``Makefile.in`` files, showing how it is +possible to finalize the ``BUG_COMPONENT`` value.:: + + # /moz.build + with Files('**/Makefile.in'): + BUG_COMPONENT = ('Core', 'Build Config') + FINAL = True + + # /foo/moz.build + with Files('**'): + BUG_COMPONENT = ('Another', 'Component') + +If we query for metadata of ``/foo/Makefile.in``, both ``Files`` +sub-contexts match the file pattern. However, since ``BUG_COMPONENT`` is +marked as finalized by ``/moz.build``, the assignment from +``/foo/moz.build`` is ignored. The final value for ``BUG_COMPONENT`` +is ``('Core', 'Build Config')``. + +Here is another example:: + + with Files('*.cpp'): + BUG_COMPONENT = ('One-Off', 'For C++') + FINAL = True + + with Files('**'): + BUG_COMPONENT = ('Regular', 'Component') + +For every files except ``foo.cpp``, the bug component will be resolved +as ``Regular :: Component``. However, ``foo.cpp`` has its value of +``One-Off :: For C++`` preserved because it is finalized. + +.. important:: + + ``FINAL`` only applied to variables defined in a context. + + If you want to mark one variable as finalized but want to leave + another mutable, you'll need to use 2 ``Files`` contexts. + +Guidelines for Defining Metadata +================================ + +In general, values defined towards the root of the source tree are +generic and become more specific towards the leaves. For example, +the ``BUG_COMPONENT`` for ``/browser`` might be ``Firefox :: General`` +whereas ``/browser/components/preferences`` would list +``Firefox :: Preferences``. diff --git a/build/docs/glossary.rst b/build/docs/glossary.rst new file mode 100644 index 000000000..f846effe5 --- /dev/null +++ b/build/docs/glossary.rst @@ -0,0 +1,48 @@ +======== +Glossary +======== + +.. glossary:: + :sorted: + + object directory + A directory holding the output of the build system. The build + system attempts to isolate all file modifications to this + directory. By convention, object directories are commonly + directories under the source directory prefixed with **obj-**. + e.g. **obj-firefox**. + + mozconfig + A shell script used to configure the build system. + + configure + A generated shell script which detects the current system + environment, applies a requested set of build configuration + options, and writes out metadata to be consumed by the build + system. + + config.status + An executable file produced by **configure** that takes the + generated build config and writes out files used to build the + tree. Traditionally, config.status writes out a bunch of + Makefiles. + + install manifest + A file containing metadata describing file installation rules. + A large part of the build system consists of copying files + around to appropriate places. We write out special files + describing the set of required operations so we can process the + actions effeciently. These files are install manifests. + + clobber build + A build performed with an initially empty object directory. All + build actions must be performed. + + incremental build + A build performed with the result of a previous build in an + object directory. The build should not have to work as hard because + it will be able to reuse the work from previous builds. + + mozinfo + An API for accessing a common and limited subset of the build and + run-time configuration. See :ref:`mozinfo`. diff --git a/build/docs/index.rst b/build/docs/index.rst new file mode 100644 index 000000000..63e053757 --- /dev/null +++ b/build/docs/index.rst @@ -0,0 +1,50 @@ +============ +Build System +============ + +Important Concepts +================== +.. toctree:: + :maxdepth: 1 + + glossary + build-overview + supported-configurations + Mozconfig Files <mozconfigs> + mozbuild-files + mozbuild-symbols + files-metadata + Profile Guided Optimization <pgo> + slow + environment-variables + build-targets + python + test_manifests + mozinfo + preprocessor + jar-manifests + defining-binaries + toolchains + locales + rust + +integrated development environment (IDE) +======================================== +.. toctree:: + :maxdepth: 1 + + androideclipse + cppeclipse + visualstudio + +mozbuild +======== + +mozbuild is a Python package containing a lot of the code for the +Mozilla build system. + +.. toctree:: + :maxdepth: 1 + + mozbuild/index + mozbuild/dumbmake diff --git a/build/docs/jar-manifests.rst b/build/docs/jar-manifests.rst new file mode 100644 index 000000000..89776a101 --- /dev/null +++ b/build/docs/jar-manifests.rst @@ -0,0 +1,97 @@ +.. _jar_manifests: + +============= +JAR Manifests +============= + +JAR Manifests are plaintext files in the tree that are used to package chrome +files into the correct JARs, and create +`Chrome Registration <https://developer.mozilla.org/en-US/docs/Chrome_Registration>`_ +manifests. JAR Manifests are commonly named ``jar.mn``. They are +declared in ``moz.build`` files using the ``JAR_MANIFESTS`` variable. + +``jar.mn`` files are automatically processed by the build system when building a +source directory that contains one. The ``jar``.mn is run through the +:ref:`preprocessor` before being passed to the manifest processor. In order to +have ``@variables@`` expanded (such as ``@AB_CD@``) throughout the file, add +the line ``#filter substitution`` at the top of your ``jar.mn`` file. + +The format of a jar.mn is fairly simple; it consists of a heading specifying +which JAR file is being packaged, followed by indented lines listing files and +chrome registration instructions. + +To see a simple ``jar.mn`` file at work, see ``toolkit/profile/jar.mn``. A much +more complex ``jar.mn`` is at ``toolkit/locales/jar.mn``. + +Shipping Chrome Files +===================== + +To ship chrome files in a JAR, an indented line indicates a file to be packaged:: + + <jarfile>.jar: + path/in/jar/file_name.xul (source/tree/location/file_name.xul) + +The JAR location may be preceded with a base path between square brackets:: + [base/path] <jarfile>.jar: + path/in/jar/file_name.xul (source/tree/location/file_name.xul) + +In this case, the jar will be directly located under the given ``base/bath``, +while without a base path, it will be under a ``chrome`` directory. + +If the JAR manifest and packaged file live in the same directory, the path and +parenthesis can be omitted. In other words, the following two lines are +equivalent:: + + path/in/jar/same_place.xhtml (same_place.xhtml) + path/in/jar/same_place.xhtml + +The source tree location may also be an *absolute* path (taken from the +top of the source tree:: + + path/in/jar/file_name.xul (/path/in/sourcetree/file_name.xul) + +An asterisk marker (``*``) at the beginning of the line indicates that the +file should be processed by the :ref:`preprocessor` before being packaged:: + + * path/in/jar/preprocessed.xul (source/tree/location/file_name.xul) + +Preprocessed files always replace existing files, to ensure that changes in +``#expand`` or ``#include`` directives are picked up. + +There is a special source-directory format for localized files (note the +percent sign in the source file location): this format reads ``localized.dtd`` +from the ``en-US`` directory if building an English version, and reads the +file from the alternate localization source tree +``/l10n/<locale>/path/localized.dtd`` if building a localized version:: + + locale/path/localized.dtd (%localized/path/localized.dtd) + +The source tree location can also use wildcards, in which case the path in +jar is expected to be a base directory. Paths before the wildcard are not +made part of the destination path:: + + path/in/jar/ (source/tree/location/*.xul) + +The above will install all xul files under ``source/tree/location`` as +``path/in/jar/*.xul``. + +Register Chrome +=============== + +`Chrome Registration <https://developer.mozilla.org/en-US/docs/Chrome_Registration>`_ +instructions are marked with a percent sign (``%``) at the beginning of the +line, and must be part of the definition of a JAR file. Any additional percents +signs are replaced with an appropriate relative URL of the JAR file being +packaged:: + + % content global %path/in/jar/ + % overlay chrome://blah/content/blah.xul chrome://foo/content/overlay.xul + +There are two possible locations for a manifest file. If the chrome is being +built into a standalone application, the ``jar.mn`` processor creates a +``<jarfilename>.manifest`` next to the JAR file itself. This is the default +behavior. + +If the build specifies ``USE_EXTENSION_MANIFEST = 1``, the ``jar.mn`` processor +creates a single ``chrome.manifest`` file suitable for registering chrome as +an extension. diff --git a/build/docs/locales.rst b/build/docs/locales.rst new file mode 100644 index 000000000..4705ed963 --- /dev/null +++ b/build/docs/locales.rst @@ -0,0 +1,100 @@ +.. _localization: + +=================== +Localization (l10n) +=================== + +Single-locale language repacks +============================== + +To save on build time, the build system and automation collaborate to allow +downloading a packaged en-US Firefox, performing some locale-specific +post-processing, and re-packaging a locale-specific Firefox. Such artifacts +are termed "single-locale language repacks". There is another concept of a +"multi-locale language build", which is more like a regular build and less +like a re-packaging post-processing step. + +There are scripts in-tree in mozharness to orchestrate these re-packaging +steps for `Desktop +<https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/desktop_l10n.py>`_ +and `Android +<https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/mobile_l10n.py>`_ +but they rely heavily on buildbot information so they are almost impossible to +run locally. + +The following instructions are extracted from the `Android script with hg hash +494289c7 +<https://dxr.mozilla.org/mozilla-central/rev/494289c72ba3997183e7b5beaca3e0447ecaf96d/testing/mozharness/scripts/mobile_l10n.py>`_, +and may need to be updated and slightly modified for Desktop. + +Step by step instructions for Android +------------------------------------- + +This assumes that ``$AB_CD`` is the locale you want to repack with; I tested +with "ar" and "en-GB". + +.. warning:: l10n repacks do not work with artifact builds. Repackaging + compiles no code so supporting ``--disable-compile-environment`` would not + save much, if any, time. + +#. You must have a built and packaged object directory, or a pre-built + ``en-US`` package. + + .. code-block:: shell + + ./mach build + ./mach package + +#. Clone ``l10n-central/$AB_CD`` so that it is a sibling to your + ``mozilla-central`` directory. + + .. code-block:: shell + + $ ls -al + mozilla-central + ... + $ mkdir -p l10n-central + $ hg clone https://hg.mozilla.org/l10n-central/$AB_CD l10n-central/$AB_CD + $ ls -al + mozilla-central + l10n-central/$AB_CD + ... + +#. Copy your ``mozconfig`` to ``mozconfig.l10n`` and add the following. + + :: + + ac_add_options --with-l10n-base=../../l10n-central + ac_add_options --disable-tests + mk_add_options MOZ_OBJDIR=./objdir-l10n + +#. Configure and prepare the l10n object directory. + + .. code-block:: shell + + MOZCONFIG=mozconfig.l10n ./mach configure + MOZCONFIG=mozconfig.l10n ./mach build -C config export + MOZCONFIG=mozconfig.l10n ./mach build buildid.h + +#. Copy your built package and unpack it into the l10n object directory. + + .. code-block:: shell + + cp $OBJDIR/dist/fennec-*en-US*.apk ./objdir-l10n/dist + MOZCONFIG=mozconfig.l10n ./mach build -C mobile/android/locales unpack + +#. Run the ``compare-locales`` script to write locale-specific changes into + ``objdir-l10n/merged``. + + .. code-block:: shell + + MOZCONFIG=mozconfig.l10n ./mach compare-locales --merge-dir objdir-l10n/merged $AB_CD + +#. Finally, repackage using the locale-specific changes. + + .. code-block:: shell + + MOZCONFIG=mozconfig.l10n LOCALE_MERGEDIR=`realpath objdir-l10n/merged` ./mach build -C mobile/android/locales installers-$AB_CD + + (Note the absolute path for ``LOCALE_MERGEDIR``.) You should find a + re-packaged build at ``objdir-l10n/dist/fennec-*$AB_CD*.apk``. diff --git a/build/docs/mozbuild-files.rst b/build/docs/mozbuild-files.rst new file mode 100644 index 000000000..3550c51ef --- /dev/null +++ b/build/docs/mozbuild-files.rst @@ -0,0 +1,176 @@ +.. _mozbuild-files: + +=============== +moz.build Files +=============== + +``moz.build`` files are the mechanism by which tree metadata (notably +the build configuration) is defined. + +Directories in the tree contain ``moz.build`` files which declare +functionality for their respective part of the tree. This includes +things such as the list of C++ files to compile, where to find tests, +etc. + +``moz.build`` files are actually Python scripts. However, their +execution is governed by special rules. This is explained below. + +moz.build Python Sandbox +======================== + +As mentioned above, ``moz.build`` files are Python scripts. However, +they are executed in a special Python *sandbox* that significantly +changes and limits the execution environment. The environment is so +different, it's doubtful most ``moz.build`` files would execute without +error if executed by a vanilla Python interpreter (e.g. ``python +moz.build``. + +The following properties make execution of ``moz.build`` files special: + +1. The execution environment exposes a limited subset of Python. +2. There is a special set of global symbols and an enforced naming + convention of symbols. +3. Some symbols are inherited from previously-executed ``moz.build`` + files. + +The limited subset of Python is actually an extremely limited subset. +Only a few symbols from ``__builtins__`` are exposed. These include +``True``, ``False``, and ``None``. Global functions like ``import``, +``print``, and ``open`` aren't available. Without these, ``moz.build`` +files can do very little. *This is by design*. + +The execution sandbox treats all ``UPPERCASE`` variables specially. Any +``UPPERCASE`` variable must be known to the sandbox before the script +executes. Any attempt to read or write to an unknown ``UPPERCASE`` +variable will result in an exception being raised. Furthermore, the +types of all ``UPPERCASE`` variables is strictly enforced. Attempts to +assign an incompatible type to an ``UPPERCASE`` variable will result in +an exception being raised. + +The strictness of behavior with ``UPPERCASE`` variables is a very +intentional design decision. By ensuring strict behavior, any operation +involving an ``UPPERCASE`` variable is guaranteed to have well-defined +side-effects. Previously, when the build configuration was defined in +``Makefiles``, assignments to variables that did nothing would go +unnoticed. ``moz.build`` files fix this problem by eliminating the +potential for false promises. + +After a ``moz.build`` file has completed execution, only the +``UPPERCASE`` variables are used to retrieve state. + +The set of variables and functions available to the Python sandbox is +defined by the :py:mod:`mozbuild.frontend.context` module. The +data structures in this module are consumed by the +:py:class:`mozbuild.frontend.reader.MozbuildSandbox` class to construct +the sandbox. There are tests to ensure that the set of symbols exposed +to an empty sandbox are all defined in the ``context`` module. +This module also contains documentation for each symbol, so nothing can +sneak into the sandbox without being explicitly defined and documented. + +Reading and Traversing moz.build Files +====================================== + +The process for reading ``moz.build`` files roughly consists of: + +1. Start at the root ``moz.build`` (``<topsrcdir>/moz.build``). +2. Evaluate the ``moz.build`` file in a new sandbox. +3. Emit the main *context* and any *sub-contexts* from the executed + sandbox. +4. Extract a set of ``moz.build`` files to execute next. +5. For each additional ``moz.build`` file, goto #2 and repeat until all + referenced files have executed. + +From the perspective of the consumer, the output of reading is a stream +of :py:class:`mozbuild.frontend.reader.context.Context` instances. Each +``Context`` defines a particular aspect of data. Consumers iterate over +these objects and do something with the data inside. Each object is +essentially a dictionary of all the ``UPPERCASE`` variables populated +during its execution. + +.. note:: + + Historically, there was only one ``context`` per ``moz.build`` file. + As the number of things tracked by ``moz.build`` files grew and more + and more complex processing was desired, it was necessary to split these + contexts into multiple logical parts. It is now common to emit + multiple contexts per ``moz.build`` file. + +Build System Reading Mode +------------------------- + +The traditional mode of evaluation of ``moz.build`` files is what's +called *build system traversal mode.* In this mode, the ``CONFIG`` +variable in each ``moz.build`` sandbox is populated from data coming +from ``config.status``, which is produced by ``configure``. + +During evaluation, ``moz.build`` files often make decisions conditional +on the state of the build configuration. e.g. *only compile foo.cpp if +feature X is enabled*. + +In this mode, traversal of ``moz.build`` files is governed by variables +like ``DIRS`` and ``TEST_DIRS``. For example, to execute a child +directory, ``foo``, you would add ``DIRS += ['foo']`` to a ``moz.build`` +file and ``foo/moz.build`` would be evaluated. + +.. _mozbuild_fs_reading_mode: + +Filesystem Reading Mode +----------------------- + +There is an alternative reading mode that doesn't involve the build +system and doesn't use ``DIRS`` variables to control traversal into +child directories. This mode is called *filesystem reading mode*. + +In this reading mode, the ``CONFIG`` variable is a dummy, mostly empty +object. Accessing all but a few special variables will return an empty +value. This means that nearly all ``if CONFIG['FOO']:`` branches will +not be taken. + +Instead of using content from within the evaluated ``moz.build`` +file to drive traversal into subsequent ``moz.build`` files, the set +of files to evaluate is controlled by the thing doing the reading. + +A single ``moz.build`` file is not guaranteed to be executable in +isolation. Instead, we must evaluate all *parent* ``moz.build`` files +first. For example, in order to evaluate ``/foo/moz.build``, one must +execute ``/moz.build`` and have its state influence the execution of +``/foo/moz.build``. + +Filesystem reading mode is utilized to power the +:ref:`mozbuild_files_metadata` feature. + +Technical Details +----------------- + +The code for reading ``moz.build`` files lives in +:py:mod:`mozbuild.frontend.reader`. The Python sandboxes evaluation results +(:py:class:`mozbuild.frontend.context.Context`) are passed into +:py:mod:`mozbuild.frontend.emitter`, which converts them to classes defined +in :py:mod:`mozbuild.frontend.data`. Each class in this module defines a +domain-specific component of tree metdata. e.g. there will be separate +classes that represent a JavaScript file vs a compiled C++ file or test +manifests. This means downstream consumers of this data can filter on class +types to only consume what they are interested in. + +There is no well-defined mapping between ``moz.build`` file instances +and the number of :py:mod:`mozbuild.frontend.data` classes derived from +each. Depending on the content of the ``moz.build`` file, there may be 1 +object derived or 100. + +The purpose of the ``emitter`` layer between low-level sandbox execution +and metadata representation is to facilitate a unified normalization and +verification step. There are multiple downstream consumers of the +``moz.build``-derived data and many will perform the same actions. This +logic can be complicated, so we have a component dedicated to it. + +:py:class:`mozbuild.frontend.reader.BuildReader`` and +:py:class:`mozbuild.frontend.reader.TreeMetadataEmitter`` have a +stream-based API courtesy of generators. When you hook them up properly, +the :py:mod:`mozbuild.frontend.data` classes are emitted before all +``moz.build`` files have been read. This means that downstream errors +are raised soon after sandbox execution. + +Lots of the code for evaluating Python sandboxes is applicable to +non-Mozilla systems. In theory, it could be extracted into a standalone +and generic package. However, until there is a need, there will +likely be some tightly coupled bits. diff --git a/build/docs/mozbuild-symbols.rst b/build/docs/mozbuild-symbols.rst new file mode 100644 index 000000000..4e9a8853a --- /dev/null +++ b/build/docs/mozbuild-symbols.rst @@ -0,0 +1,7 @@ +.. _mozbuild_symbols: + +======================== +mozbuild Sandbox Symbols +======================== + +.. mozbuildsymbols:: mozbuild.frontend.context diff --git a/build/docs/mozbuild/dumbmake.rst b/build/docs/mozbuild/dumbmake.rst new file mode 100644 index 000000000..7840695ad --- /dev/null +++ b/build/docs/mozbuild/dumbmake.rst @@ -0,0 +1,38 @@ +dumbmake +======== + +*dumbmake* is a simple dependency tracker for make. + +It turns lists of make targets into longer lists of make targets that +include dependencies. For example: + + netwerk, package + +might be turned into + + netwerk, netwerk/build, toolkit/library, package + +The dependency list is read from the plain text file +`topsrcdir/build/dumbmake-dependencies`. The format best described by +example: + + build_this + when_this_changes + +Interpret this to mean that `build_this` is a dependency of +`when_this_changes`. More formally, a line (CHILD) indented more than +the preceding line (PARENT) means that CHILD should trigger building +PARENT. That is, building CHILD will trigger building first CHILD and +then PARENT. + +This structure is recursive: + + build_this_when_either_change + build_this_only_when + this_changes + +This means that `build_this_when_either_change` is a dependency of +`build_this_only_when` and `this_changes`, and `build_this_only_when` +is a dependency of `this_changes`. Building `this_changes` will build +first `this_changes`, then `build_this_only_when`, and finally +`build_this_when_either_change`. diff --git a/build/docs/mozbuild/index.rst b/build/docs/mozbuild/index.rst new file mode 100644 index 000000000..86f38940b --- /dev/null +++ b/build/docs/mozbuild/index.rst @@ -0,0 +1,41 @@ +======== +mozbuild +======== + +mozbuild is a Python package providing functionality used by Mozilla's +build system. + +Modules Overview +================ + +* mozbuild.backend -- Functionality for producing and interacting with build + backends. A build backend is an entity that consumes build system metadata + (from mozbuild.frontend) and does something useful with it (typically writing + out files that can be used by a build tool to build the tree). +* mozbuild.compilation -- Functionality related to compiling. This + includes managing compiler warnings. +* mozbuild.frontend -- Functionality for reading build frontend files + (what defines the build system) and converting them to data structures + which are fed into build backends to produce backend configurations. +* mozpack -- Functionality related to packaging builds. + +Overview +======== + +The build system consists of frontend files that define what to do. They +say things like "compile X" "copy Y." + +The mozbuild.frontend package contains code for reading these frontend +files and converting them to static data structures. The set of produced +static data structures for the tree constitute the current build +configuration. + +There exist entities called build backends. From a high level, build +backends consume the build configuration and do something with it. They +typically produce tool-specific files such as make files which can be used +to build the tree. + +Piecing it all together, we have frontend files that are parsed into data +structures. These data structures are fed into a build backend. The output +from build backends is used by builders to build the tree. + diff --git a/build/docs/mozconfigs.rst b/build/docs/mozconfigs.rst new file mode 100644 index 000000000..1befca9ec --- /dev/null +++ b/build/docs/mozconfigs.rst @@ -0,0 +1,70 @@ +.. _mozconfig: + +=============== +mozconfig Files +=============== + +mozconfig files are used to configure how a build works. + +mozconfig files are actually shell scripts. They are executed in a +special context with specific variables and functions exposed to them. + +API +=== + +Functions +--------- + +The following special functions are available to a mozconfig script. + +ac_add_options +^^^^^^^^^^^^^^ + +This function is used to declare extra options/arguments to pass into +configure. + +e.g.:: + + ac_add_options --disable-tests + ac_add_options --enable-optimize + +mk_add_options +^^^^^^^^^^^^^^ + +This function is used to inject statements into client.mk for execution. +It is typically used to define variables, notably the object directory. + +e.g.:: + + mk_add_options AUTOCLOBBER=1 + +ac_add_options +^^^^^^^^^^^^^^ + +This is a variant of ac_add_options() which only adds configure options +for a specified application. This is only used when building multiple +applications through client.mk. This function is typically not needed. + +Special mk_add_options Variables +-------------------------------- + +For historical reasons, the method for communicating certain +well-defined variables is via mk_add_options(). In this section, we +document what those special variables are. + +MOZ_OBJDIR +^^^^^^^^^^ + +This variable is used to define the :term:`object directory` for the current +build. + +Finding the active mozconfig +============================ + +Multiple mozconfig files can exist to provide different configuration +options for different tasks. The rules for finding the active mozconfig +are defined in the +:py:func:`mozbuild.mozconfig.MozconfigLoader.find_mozconfig` method: + +.. autoclass:: mozbuild.mozconfig.MozconfigLoader + :members: find_mozconfig diff --git a/build/docs/mozinfo.rst b/build/docs/mozinfo.rst new file mode 100644 index 000000000..d74ca75e6 --- /dev/null +++ b/build/docs/mozinfo.rst @@ -0,0 +1,177 @@ +.. _mozinfo: + +======= +mozinfo +======= + +``mozinfo`` is a solution for representing a subset of build +configuration and run-time data. + +``mozinfo`` data is typically accessed through a ``mozinfo.json`` file +which is written to the :term:`object directory` during build +configuration. The code for writing this file lives in +:py:mod:`mozbuild.mozinfo`. + +``mozinfo.json`` is an object/dictionary of simple string values. + +The attributes in ``mozinfo.json`` are used for many purposes. One use +is to filter tests for applicability to the current build. For more on +this, see :ref:`test_manifests`. + +.. _mozinfo_attributes: + +mozinfo.json Attributes +================================= + +``mozinfo`` currently records the following attributes. + +appname + The application being built. + + Value comes from ``MOZ_APP_NAME`` from ``config.status``. + + Optional. + +asan + Whether address sanitization is enabled. + + Values are ``true`` and ``false``. + + Always defined. + +bin_suffix + The file suffix for binaries produced with this build. + + Values may be an empty string, as not all platforms have a binary + suffix. + + Always defined. + +bits + The number of bits in the CPU this build targets. + + Values are typically ``32`` or ``64``. + + Universal Mac builds do not have this key defined. + + Unkown processor architectures (see ``processor`` below) may not have + this key defined. + + Optional. + +buildapp + The path to the XUL application being built. + + For desktop Firefox, this is ``browser``. For Fennec, it's + ``mobile/android``. For B2G, it's ``b2g``. + +crashreporter + Whether the crash reporter is enabled for this build. + + Values are ``true`` and ``false``. + + Always defined. + +datareporting + Whether data reporting (MOZ_DATA_REPORTING) is enabled for this build. + + Values are ``true`` and ``false``. + + Always defined. + +debug + Whether this is a debug build. + + Values are ``true`` and ``false``. + + Always defined. + +healthreport + Whether the Health Report feature is enabled. + + Values are ``true`` and ``false``. + + Always defined. + +mozconfig + The path of the :ref:`mozconfig file <mozconfig>` used to produce this build. + + Optional. + +nightly_build + Whether this is a nightly build. + + Values are ``true`` and ``false``. + + Always defined. + +os + The operating system the build is produced for. Values for tier-1 + supported platforms are ``linux``, ``win``, ``mac``, ``b2g``, and + ``android``. For other platforms, the value is the lowercase version + of the ``OS_TARGET`` variable from ``config.status``. + + Always defined. + +processor + Information about the processor architecture this build targets. + + Values come from ``TARGET_CPU``, however some massaging may be + performed. + + If the build is a universal build on Mac (it targets both 32-bit and + 64-bit), the value is ``universal-x86-x86_64``. + + If the value starts with ``arm``, the value is ``arm``. + + If the value starts with a string of the form ``i[3-9]86]``, the + value is ``x86``. + + Always defined. + +release_or_beta + Whether this is a release or beta build. + + Values are ``true`` and ``false``. + + Always defined. + +sm_promise + Whether spidermonkey promises have been enabled or not. This is set + by adding --enable-sm-promise to the mozconfig file. + + Values are ``true`` and ``false``. + + Always defined. + +tests_enabled + Whether tests are enabled for this build. + + Values are ``true`` and ``false``. + + Always defined. + +toolkit + The widget toolkit in case. The value comes from the + ``MOZ_WIDGET_TOOLKIT`` ``config.status`` variable. + + Always defined. + +topsrcdir + The path to the source directory the build came from. + + Always defined. + +wave + Whether Wave audio support is enabled. + + Values are ``true`` and ``false``. + + Always defined. + +webm + Whether WebM support is enabled. + + Values are ``true`` and ``false``. + + Always defined. diff --git a/build/docs/pgo.rst b/build/docs/pgo.rst new file mode 100644 index 000000000..51d03c625 --- /dev/null +++ b/build/docs/pgo.rst @@ -0,0 +1,40 @@ +.. _pgo: + +=========================== +Profile Guided Optimization +=========================== + +:abbr:`PGO (Profile Guided Optimization)` is the process of adding +probes to a compiled binary, running said binary, then using the +run-time information to *recompile* the binary to (hopefully) make it +faster. + +How PGO Builds Work +=================== + +The supported interface for invoking a PGO build is to evaluate the +*build* target of client.mk with *MOZ_PGO* defined. e.g.:: + + $ make -f client.mk MOZ_PGO=1 + +This is equivalent to:: + + $ make -f client.mk profiledbuild + +Which is roughly equivalent to: + +#. Perform a build with *MOZ_PROFILE_GENERATE=1* and *MOZ_PGO_INSTRUMENTED=1* +#. Package with *MOZ_PGO_INSTRUMENTED=1* +#. Performing a run of the instrumented binaries +#. $ make maybe_clobber_profiledbuild +#. Perform a build with *MOZ_PROFILE_USE=1* + +Differences between toolchains +============================== + +There are some implementation differences depending on the compiler +toolchain being used. + +The *maybe_clobber_profiledbuild* step gets its name because of a +difference. On Windows, this step merely moves some *.pgc* files around. +Using GCC or Clang, it is equivalent to a *make clean*. diff --git a/build/docs/preprocessor.rst b/build/docs/preprocessor.rst new file mode 100644 index 000000000..ab2d8ecf7 --- /dev/null +++ b/build/docs/preprocessor.rst @@ -0,0 +1,244 @@ +.. _preprocessor: + +================= +Text Preprocessor +================= + +The build system contains a text preprocessor similar to the C preprocessor, +meant for processing files which have no built-in preprocessor such as XUL +and JavaScript documents. It is implemented at ``python/mozbuild/mozbuild/preprocessor.py`` and +is typically invoked via :ref:`jar_manifests`. + +While used to preprocess CSS files, the directives are changed to begin with +``%`` instead of ``#`` to avoid conflict of the id selectors. + +Directives +========== + +Variable Definition +------------------- + +define +^^^^^^ + +:: + + #define variable + #define variable value + +Defines a preprocessor variable. + +Note that, unlike the C preprocessor, instances of this variable later in the +source are not automatically replaced (see #filter). If value is not supplied, +it defaults to ``1``. + +Note that whitespace is significant, so ``"#define foo one"`` and +``"#define foo one "`` is different (in the second case, ``foo`` is defined to +be a four-character string). + +undef +^^^^^ + +:: + + #undef variable + +Undefines a preprocessor variable. + +Conditionals +------------ + +if +^^ + +:: + + #if variable + #if !variable + #if variable==string + #if variable!=string + +Disables output if the conditional is false. This can be nested to arbitrary +depths. Note that in the equality checks, the variable must come first, and +the comparison operator must not be surrounded by any whitespace. + +else +^^^^ + +:: + + #else + +Reverses the state of the previous conditional block; for example, if the +last ``#if`` was true (output was enabled), an ``#else`` makes it off +(output gets disabled). + +.. warning:: An ``#else`` is relative to the last conditional block only, + unlike the C preprocessor. + + It does not matter whether any blocks before it were true. This behavior + changed on trunk (Gecko 1.9) on 2006-12-07; see Bug 277122 for details. + +:: + + #if 1 + always included + #elif 1 + never included + #else + always included + #endif + +endif +^^^^^ + +:: + + #endif + +Ends the conditional block. + +ifdef / ifndef +^^^^^^^^^^^^^^ + +:: + + #ifdef variable + #ifndef variable + +An ``#if`` conditional that is true only if the preprocessor variable +variable is defined (in the case of ``ifdef``) or not defined (``ifndef``). + +elif / elifdef / elifndef +^^^^^^^^^^^^^^^^^^^^^^^^^ + +:: + + #elif variable + #elif !variable + #elif variable == string + #elif variable != string + #elifdef variable + #elifndef variable + +A shorthand to mean an ``#else`` combined with the relevant conditional. +The following two blocks are equivalent:: + + #ifdef foo + block 1 + #elifdef bar + block 2 + #endif + +:: + + #ifdef foo + block 1 + #else + #ifdef bar + block 2 + #endif + #endif + +.. warning:: An ``#elif``, ``#elifdef``, or ``#elifndef`` is relative to + the last conditional block only (as well as the condition it implies), + unlike the C preprocessor. It does not matter whether any blocks before + it were true. This behavior changed on trunk (Gecko 1.9) on 2006-12-07. + See Bug 277122 for details. + +File Inclusion +-------------- + +include +^^^^^^^ + +:: + + #include filename + +The file specified by filename is processed as if the contents was placed +at this position. This also means that preprocessor conditionals can even +be started in one file and ended in another (but is highly discouraged). +There is no limit on depth of inclusion, or repeated inclusion of the same +file, or self inclusion; thus, care should be taken to avoid infinite loops. + +includesubst +^^^^^^^^^^^^ + +:: + + #includesubst @variable@filename + +Same as a ``#include`` except that all instances of variable in the included +file is also expanded as in ``#filter`` substitution + +expand +^^^^^^ + +:: + + #expand string + +All variables wrapped in ``__`` are replaced with their value, for this line +only. If the variable is not defined, it expands to an empty string. For +example, if ``foo`` has the value ``bar``, and ``baz`` is not defined, then:: + + #expand This <__foo__> <__baz__> gets expanded + +Is expanded to:: + + This <bar> <> gets expanded + +filter / unfilter +^^^^^^^^^^^^^^^^^ + +:: + + #filter filter1 filter2 ... filterN + #unfilter filter1 filter2 ... filterN + +``#filter`` turns on the given filter. + +Filters are run in alphabetical order on a per-line basis. + +``#unfilter`` turns off the given filter. Available filters are: + +emptyLines + strips blank lines from the output +slashslash + strips everything from the first two consecutive slash (``/``) + characters until the end of the line +spaces + collapses consecutive sequences of spaces into a single space, + and strips leading and trailing spaces +substitution + all variables wrapped in @ are replaced with their value. If the + variable is not defined, it is a fatal error. Similar to ``#expand`` + and ``#filter`` +attemptSubstitution + all variables wrapped in ``@`` are replaced with their value, or an + empty string if the variable is not defined. Similar to ``#expand``. + +literal +^^^^^^^ + +:: + + #literal string + +Output the string (i.e. the rest of the line) literally, with no other fixups. +This is useful to output lines starting with ``#``, or to temporarily +disable filters. + +Other +----- + +#error +^^^^^^ + +:: + + #error string + +Cause a fatal error at this point, with the error message being the +given string. diff --git a/build/docs/python.rst b/build/docs/python.rst new file mode 100644 index 000000000..098541706 --- /dev/null +++ b/build/docs/python.rst @@ -0,0 +1,178 @@ +.. _python: + +=========================== +Python and the Build System +=========================== + +The Python programming language is used significantly in the build +system. If we need to write code for the build system or for a tool +related to the build system, Python is typically the first choice. + +Python Requirements +=================== + +The tree requires Python 2.7.3 or greater but not Python 3 to build. +All Python packages not in the Python distribution are included in the +source tree. So all you should need is a vanilla Python install and you +should be good to go. + +Only CPython (the Python distribution available from www.python.org) is +supported. + +We require Python 2.7.3 (and not say 2.7.2) to build because Python +2.7.3 contains numerous bug fixes, especially around the area of Unicode +handling. These bug fixes are extremely annoying and have to be worked +around. The build maintainers were tired of doing this, so the minimum +version requirement was upped (bug 870420). + +We intend to eventually support Python 3. This will come by way of dual +2.7/3.x compatibility because a single flag day conversion to 3.x will +be too cumbersome given the amount of Python that would need converted. +We will not know which 3.x minor release we are targeting until this +effort is underway. This is tracked in bug 636155. + +Compiled Python Packages +======================== + +There are some features of the build that rely on compiled Python packages +(packages containing C source). These features are currently all +optional because not every system contains the Python development +headers required to build these extensions. + +We recommend you have the Python development headers installed (``mach +bootstrap`` should do this for you) so you can take advantage of these +features. + +Issues with OS X System Python +============================== + +The Python that ships with OS X has historically been littered with +subtle bugs and suboptimalities. Furthermore, OS X up through 10.8 don't +ship with Python 2.7.3 (10.8 ships with 2.7.2). + +OS X 10.8 and below users will be required to install a new Python +distribution. This may not be necessary for OS X 10.9+. However, we +still recommend installing a separate Python because of the history with +OS X's system Python issues. + +We recommend installing Python through Homebrew or MacPorts. If you run +``mach bootstrap``, this should be done for you. + +Virtualenvs +=========== + +The build system relies heavily on +`virtualenvs <http://www.virtualenv.org/en/latest/>`_. Virtualenvs are +standalone and isolated Python environments. The problem a virtualenv +solves is that of dependencies across multiple Python components. If two +components on a system relied on different versions of a package, there +could be a conflict. Instead of managing multiple versions of a package +simultaneously, Python and virtualenvs take the route that it is easier +to just keep them separate so there is no potential for conflicts. + +Very early in the build process, a virtualenv is created inside the +:term:`object directory`. The virtualenv is configured such that it can +find all the Python packages in the source tree. The code for this lives +in :py:mod:`mozbuild.virtualenv`. + +Deficiencies +------------ + +There are numerous deficiencies with the way virtualenvs are handled in +the build system. + +* mach reinvents the virtualenv. + + There is code in ``build/mach_bootstrap.py`` that configures ``sys.path`` + much the same way the virtualenv does. There are various bugs tracking + this. However, no clear solution has yet been devised. It's not a huge + problem and thus not a huge priority. + +* They aren't preserved across copies and packaging. + + If you attempt to copy an entire tree from one machine to another or + from one directory to another, chances are the virtualenv will fall + apart. It would be nice if we could preserve it somehow. Instead of + actually solving portable virtualenvs, all we really need to solve is + encapsulating the logic for populating the virtualenv along with all + dependent files in the appropriate place. + +* .pyc files written to source directory. + + We rely heavily on ``.pth`` files in our virtualenv. A ``.pth`` file + is a special file that contains a list of paths. Python will take the + set of listed paths encountered in ``.pth`` files and add them to + ``sys.path``. + + When Python compiles a ``.py`` file to bytecode, it writes out a + ``.pyc`` file so it doesn't have to perform this compilation again. + It puts these ``.pyc`` files alongside the ``.pyc`` file. Python + provides very little control for determing where these ``.pyc`` files + go, even in Python 3 (which offers customer importers). + + With ``.pth`` files pointing back to directories in the source tree + and not the object directory, ``.pyc`` files are created in the source + tree. This is bad because when Python imports a module, it first looks + for a ``.pyc`` file before the ``.py`` file. If there is a ``.pyc`` + file but no ``.py`` file, it will happily import the module. This + wreaks havoc during file moves, refactoring, etc. + + There are various proposals for fixing this. See bug 795995. + +Installing Python Manually +========================== + +We highly recommend you use your system's package manager or a +well-supported 3rd party package manager to install Python for you. If +these are not available to you, we recommend the following tools for +installing Python: + +* `buildout.python <https://github.com/collective/buildout.python>`_ +* `pyenv <https://github.com/yyuu/pyenv>`_ +* An official installer from http://www.python.org. + +If all else fails, consider compiling Python from source manually. But this +should be viewed as the least desirable option. + +Common Issues with Python +========================= + +Upgrading your Python distribution breaks the virtualenv +-------------------------------------------------------- + +If you upgrade the Python distribution (e.g. install Python 2.7.5 +from 2.7.3, chances are parts of the virtualenv will break. +This commonly manifests as a cryptic ``Cannot import XXX`` exception. +More often than not, the module being imported contains binary/compiled +components. + +If you upgrade or reinstall your Python distribution, we recommend +clobbering your build. + +Packages installed at the system level conflict with build system's +------------------------------------------------------------------- + +It is common for people to install Python packages using ``sudo`` (e.g. +``sudo pip install psutil``) or with the system's package manager +(e.g. ``apt-get install python-mysql``. + +A problem with this is that packages installed at the system level may +conflict with the package provided by the source tree. As of bug 907902 +and changeset f18eae7c3b27 (September 16, 2013), this should no longer +be an issue since the virtualenv created as part of the build doesn't +add the system's ``site-packages`` directory to ``sys.path``. However, +poorly installed packages may still find a way to creep into the mix and +interfere with our virtualenv. + +As a general principle, we recommend against using your system's package +manager or using ``sudo`` to install Python packages. Instead, create +virtualenvs and isolated Python environments for all of your Python +projects. + +Python on $PATH is not appropriate +---------------------------------- + +Tools like ``mach`` will look for Python by performing ``/usr/bin/env +python`` or equivalent. Please be sure the appropriate Python 2.7.3+ +path is on $PATH. On OS X, this likely means you'll need to modify your +shell's init script to put something ahead of ``/usr/bin``. diff --git a/build/docs/rust.rst b/build/docs/rust.rst new file mode 100644 index 000000000..970cf3268 --- /dev/null +++ b/build/docs/rust.rst @@ -0,0 +1,79 @@ +.. _rust: + +============================== +Including Rust Code in Firefox +============================== + +The build system has support for building and linking Rust crates. +Rust code is built using ``cargo`` in the typical way, so it is +straightforward to take an existing Rust crate and integrate it +into Firefox. + +.. important:: + + Rust code is not currently enabled by default in Firefox builds. + This should change soon (`bug 1283898 <https://bugzilla.mozilla.org/show_bug.cgi?id=1283898>`_), + but the option to build without Rust code will likely last a little longer + (`bug 1284816 <https://bugzilla.mozilla.org/show_bug.cgi?id=1284816>`_), + so Rust code cannot currently be used for required components. + + +Linking Rust Crates into libxul +=============================== + +Rust crates that you want to link into libxul should be listed in the +``dependencies`` section of `toolkit/library/rust/shared/Cargo.toml <https://dxr.mozilla.org/mozilla-central/source/toolkit/library/rust/shared/Cargo.toml>`_. +You'll also need to add an ``extern crate`` reference to `toolkit/library/rust/shared/lib.rs <https://dxr.mozilla.org/mozilla-central/source/toolkit/library/rust/shared/lib.rs>`_. +This ensures that the Rust code will be linked properly into libxul as well +as the copy of libxul used for gtests. + +Linking Rust Crates into something else +======================================= + +There currently is not any Rust code being linked into binaries other than +libxul. If you would like to do so, you'll need to create a directory with +a ``Cargo.toml`` file for your crate, and a ``moz.build`` file that contains: + +.. code-block:: python + + RustLibrary('crate_name') + +Where *crate_name* matches the name from the ``[package]`` section of your +``Cargo.toml``. You can refer to `the moz.build file <https://dxr.mozilla.org/mozilla-central/rev/3f4c3a3cabaf94958834d3a8935adfb4a887942d/toolkit/library/rust/moz.build#7>`_ and `the Cargo.toml file <https://dxr.mozilla.org/mozilla-central/rev/3f4c3a3cabaf94958834d3a8935adfb4a887942d/toolkit/library/rust/Cargo.toml>`_ that are used for libxul. + +You can then add ``USE_LIBS += ['crate_name']`` to the ``moz.build`` file +that defines the binary as you would with any other library in the tree. + +.. important:: + + You cannot link a Rust crate into an intermediate library that will wind + up being linked into libxul. The build system enforces that only a single + ``RustLibrary`` may be linked into a binary. If you need to do this, you + will have to add a ``RustLibrary`` to link to any standalone binaries that + link the intermediate library, and also add the Rust crate to the libxul + dependencies as in `linking Rust Crates into libxul`_. + +Where Should I put my Crate? +============================ + +If your crate's canonical home is mozilla-central, you can put it next to the +other code in the module it belongs to. + +If your crate is mirrored into mozilla-central from another repository, and +will not be actively developed in mozilla-central, you can simply list it +as a ``crates.io``-style dependency with a version number, and let it be +vendored into the ``third_party/rust`` directory. + +If your crate is mirrored into mozilla-central from another repository, but +will be actively developed in both locations, you should send mail to the +dev-builds mailing list to start a discussion on how to meet your needs. + + +Crate dependencies +================== + +All dependencies for in-tree Rust crates are vendored into the +``third_party/rust`` directory. Currently if you add a dependency on a new +crate you must run ``mach vendor rust`` to vendor the dependencies into +that directory. In the future we hope to make it so that you only need to +vendor the dependencies in order to build your changes in a CI push. diff --git a/build/docs/slow.rst b/build/docs/slow.rst new file mode 100644 index 000000000..546fe26ce --- /dev/null +++ b/build/docs/slow.rst @@ -0,0 +1,179 @@ +.. _slow: + +============================ +Why the Build System is Slow +============================ + +A common complaint about the build system is that it's slow. There are +many reasons contributing to its slowness. We will attempt to document +them here. + +First, it is important to distinguish between a :term:`clobber build` +and an :term:`incremental build`. The reasons for why each are slow can +be different. + +The build does a lot of work +============================ + +It may not be obvious, but the main reason the build system is slow is +because it does a lot of work! The source tree consists of a few +thousand C++ files. On a modern machine, we spend over 120 minutes of CPU +core time compiling files! So, if you are looking for the root cause of +slow clobber builds, look at the sheer volume of C++ files in the tree. + +You don't have enough CPU cores and MHz +======================================= + +The build should be CPU bound. If the build system maintainers are +optimizing the build system perfectly, every CPU core in your machine +should be 100% saturated during a build. While this isn't currently the +case (keep reading below), generally speaking, the more CPU cores you +have in your machine and the more total MHz in your machine, the better. + +**We highly recommend building with no fewer than 4 physical CPU +cores.** Please note the *physical* in this sentence. Hyperthreaded +cores (an Intel Core i7 will report 8 CPU cores but only 4 are physical +for example) only yield at most a 1.25x speedup per core. + +We also recommend using the most modern CPU model possible. Haswell +chips deliver much more performance per CPU cycle than say Sandy Bridge +CPUs. + +This cause impacts both clobber and incremental builds. + +You are building with a slow I/O layer +====================================== + +The build system can be I/O bound if your I/O layer is slow. Linking +libxul on some platforms and build architectures can perform gigabytes +of I/O. + +To minimize the impact of slow I/O on build performance, **we highly +recommend building with an SSD.** Power users with enough memory may opt +to build from a RAM disk. Mechanical disks should be avoided if at all +possible. + +Some may dispute the importance of an SSD on build times. It is true +that the beneficial impact of an SSD can be mitigated if your system has +lots of memory and the build files stay in the page cache. However, +operating system memory management is complicated. You don't really have +control over what or when something is evicted from the page cache. +Therefore, unless your machine is a dedicated build machine or you have +more memory than is needed by everything running on your machine, +chances are you'll run into page cache eviction and you I/O layer will +impact build performance. That being said, an SSD certainly doesn't +hurt build times. And, anyone who has used a machine with an SSD will +tell you how great of an investment it is for performance all around the +operating system. On top of that, some automated tests are I/O bound +(like those touching SQLite databases), so an SSD will make tests +faster. + +This cause impacts both clobber and incremental builds. + +You don't have enough memory +============================ + +The build system allocates a lot of memory, especially when building +many things in parallel. If you don't have enough free system memory, +the build will cause swap activity, slowing down your system and the +build. Even if you never get to the point of swapping, the build system +performs a lot of I/O and having all accessed files in memory and the +page cache can significantly reduce the influence of the I/O layer on +the build system. + +**We recommend building with no less than 8 GB of system memory.** As +always, the more memory you have, the better. For a bare bones machine +doing nothing more than building the source tree, anything more than 16 +GB is likely entering the point of diminishing returns. + +This cause impacts both clobber and incremental builds. + +You are building on Windows +=========================== + +New processes on Windows are about a magnitude slower to spawn than on +UNIX-y systems such as Linux. This is because Windows has optimized new +threads while the \*NIX platforms typically optimize new processes. +Anyway, the build system spawns thousands of new processes during a +build. Parts of the build that rely on rapid spawning of new processes +are slow on Windows as a result. This is most pronounced when running +*configure*. The configure file is a giant shell script and shell +scripts rely heavily on new processes. This is why configure on Windows +can run over a minute slower on Windows. + +Another reason Windows builds are slower is because Windows lacks proper +symlink support. On systems that support symlinks, we can generate a +file into a staging area then symlink it into the final directory very +quickly. On Windows, we have to perform a full file copy. This incurs +much more I/O. And if done poorly, can muck with file modification +times, messing up build dependencies. As of the summer of 2013, the +impact of symlinks is being mitigated through the use +of an :term:`install manifest`. + +These issues impact both clobber and incremental builds. + +Recursive make traversal is slow +================================ + +The build system has traditionally been built by employing recursive +make. Recursive make involves make iterating through directories / make +files sequentially and executing each in turn. This is inefficient for +directories containing few targets/tasks because make could be *starved* +for work when processing these directories. Any time make is starved, +the build isn't using all available CPU cycles and the build is slower +as a result. + +Work has started in bug 907365 to fix this issue by changing the way +make traverses all the make files. + +The impact of slow recursive make traversal is mostly felt on +incremental builds. Traditionally, most of the wall time during a +no-op build is spent in make traversal. + +make is inefficient +=================== + +Compared to modern build backends like Tup or Ninja, make is slow and +inefficient. We can only make make so fast. At some point, we'll hit a +performance plateau and will need to use a different tool to make builds +faster. + +Please note that clobber and incremental builds are different. A clobber +build with make will likely be as fast as a clobber build with e.g. Tup. +However, Tup should vastly outperform make when it comes to incremental +builds. Therefore, this issue is mostly seen when performing incremental +builds. + +C++ header dependency hell +========================== + +Modifying a *.h* file can have significant impact on the build system. +If you modify a *.h* that is used by 1000 C++ files, all of those 1000 +C++ files will be recompiled. + +Our code base has traditionally been sloppy managing the impact of +changed headers on build performance. Bug 785103 tracks improving the +situation. + +This issue mostly impacts the times of an :term:`incremental build`. + +A search/indexing service on your machine is running +==================================================== + +Many operating systems have a background service that automatically +indexes filesystem content to make searching faster. On Windows, you +have the Windows Search Service. On OS X, you have Finder. + +These background services sometimes take a keen interest in the files +being produced as part of the build. Since the build system produces +hundreds of megabytes or even a few gigabytes of file data, you can +imagine how much work this is to index! If this work is being performed +while the build is running, your build will be slower. + +OS X's Finder is notorious for indexing when the build is running. And, +it has a tendency to suck up a whole CPU core. This can make builds +several minutes slower. If you build with ``mach`` and have the optional +``psutil`` package built (it requires Python development headers - see +:ref:`python` for more) and Finder is running during a build, mach will +print a warning at the end of the build, complete with instructions on +how to fix it. diff --git a/build/docs/supported-configurations.rst b/build/docs/supported-configurations.rst new file mode 100644 index 000000000..cc2c1ea72 --- /dev/null +++ b/build/docs/supported-configurations.rst @@ -0,0 +1,55 @@ +.. _build_supported_configurations: + +======================== +Supported Configurations +======================== + +This page attempts to document supported build configurations. + +Windows +======= + +We support building on Windows XP and newer operating systems using +Visual Studio 2010 and newer. + +The following are not fully supported by Mozilla (but may work): + +* Building without the latest *MozillaBuild* Windows development + environment +* Building with Mingw or any other non-Visual Studio toolchain. + +OS X +==== + +We support building on OS X 10.6 and newer with the OS X 10.6 SDK. + +The tree should build with the following OS X releases and SDK versions: + +* 10.6 Snow Leopard +* 10.7 Lion +* 10.8 Mountain Lion +* 10.9 Mavericks + +The tree requires building with Clang 3.3 and newer. This corresponds to +version of 4.2 of Apple's Clang that ships with Xcode. This corresponds +to Xcode 4.6 and newer. Xcode 4.6 only runs on OS X 10.7.4 and newer. +So, OS X 10.6 users will need to install a non-Apple toolchain. Running +``mach bootstrap`` should install an appropriate toolchain from Homebrew +or MacPorts automatically. + +The tree should build with GCC 4.4 and newer on OS X. However, this +build configuration isn't as widely used (and differs from what Mozilla +uses to produce OS X builds), so it's recommended to stick with Clang. + +Linux +===== + +Linux 2.6 and later kernels are supported. + +Most distributions are supported as long as the proper package +dependencies are in place. Running ``mach bootstrap`` should install +packages for popular Linux distributions. ``configure`` will typically +detect missing dependencies and inform you how to disable features to +work around unsatisfied dependencies. + +Clang 3.3 or GCC 4.4 is required to build the tree. diff --git a/build/docs/test_manifests.rst b/build/docs/test_manifests.rst new file mode 100644 index 000000000..8c85579be --- /dev/null +++ b/build/docs/test_manifests.rst @@ -0,0 +1,207 @@ +.. _test_manifests: + +============== +Test Manifests +============== + +Many test suites have their test metadata defined in files called +**test manifests**. + +Test manifests are divided into two flavors: :ref:`manifestparser_manifests` +and :ref:`reftest_manifests`. + +Naming Convention +================= + +The build system does not enforce file naming for test manifest files. +However, the following convention is used. + +mochitest.ini + For the *plain* flavor of mochitests. + +chrome.ini + For the *chrome* flavor of mochitests. + +browser.ini + For the *browser chrome* flavor of mochitests. + +a11y.ini + For the *a11y* flavor of mochitests. + +xpcshell.ini + For *xpcshell* tests. + +.. _manifestparser_manifests: + +ManifestParser Manifests +========================== + +ManifestParser manifests are essentially ini files that conform to a basic +set of assumptions. + +The `reference documentation <http://mozbase.readthedocs.org/en/latest/manifestparser.html>`_ +for manifestparser manifests describes the basic format of test manifests. + +In summary, manifests are ini files with section names describing test files:: + + [test_foo.js] + [test_bar.js] + +Keys under sections can hold metadata about each test:: + + [test_foo.js] + skip-if = os == "win" + [test_foo.js] + skip-if = os == "linux" && debug + [test_baz.js] + fail-if = os == "mac" || os == "android" + +There is a special **DEFAULT** section whose keys/metadata apply to all +sections/tests:: + + [DEFAULT] + property = value + + [test_foo.js] + +In the above example, **test_foo.js** inherits the metadata **property = value** +from the **DEFAULT** section. + +Recognized Metadata +------------------- + +Test manifests can define some common keys/metadata to influence behavior. +Those keys are as follows: + +head + List of files that will be executed before the test file. (Used in + xpcshell tests.) + +tail + List of files that will be executed after the test file. (Used in + xpcshell tests.) + +support-files + List of additional files required to run tests. This is typically + defined in the **DEFAULT** section. + + Unlike other file lists, *support-files* supports a globbing mechanism + to facilitate pulling in many files with minimal typing. This globbing + mechanism is activated if an entry in this value contains a ``*`` + character. A single ``*`` will wildcard match all files in a directory. + A double ``**`` will descend into child directories. For example, + ``data/*`` will match ``data/foo`` but not ``data/subdir/bar`` where + ``data/**`` will match ``data/foo`` and ``data/subdir/bar``. + + Support files starting with ``/`` are placed in a root directory, rather + than a location determined by the manifest location. For mochitests, + this allows for the placement of files at the server root. The source + file is selected from the base name (e.g., ``foo`` for ``/path/foo``). + Files starting with ``/`` cannot be selected using globbing. + + Some support files are used by tests across multiple directories. In + this case, a test depending on a support file from another directory + must note that dependency with the path to the required support file + in its own **support-files** entry. These use a syntax where paths + starting with ``!/`` will indicate the beginning of the path to a + shared support file starting from the root of the srcdir. For example, + if a manifest at ``dom/base/test/mochitest.ini`` has a support file, + ``dom/base/test/server-script.sjs``, and a mochitest in + ``dom/workers/test`` depends on that support file, the test manifest + at ``dom/workers/test/mochitest.ini`` must include + ``!/dom/base/test/server-script.sjs`` in its **support-files** entry. + +generated-files + List of files that are generated as part of the build and don't exist in + the source tree. + + The build system assumes that each manifest file, test file, and file + listed in **head**, **tail**, and **support-files** is static and + provided by the source tree (and not automatically generated as part + of the build). This variable tells the build system not to make this + assumption. + + This variable will likely go away sometime once all generated files are + accounted for in the build config. + + If a generated file is not listed in this key, a clobber build will + likely fail. + +dupe-manifest + Record that this manifest duplicates another manifest. + + The common scenario is two manifest files will include a shared + manifest file via the ``[include:file]`` special section. The build + system enforces that each test file is only provided by a single + manifest. Having this key present bypasses that check. + + The value of this key is ignored. + + +skip-if + Skip this test if the specified condition is true. + See :ref:`manifest_filter_language`. + +fail-if + Expect test failure if the specified condition is true. + See :ref:`manifest_filter_language`. + +run-sequentially + If present, the test should not be run in parallel with other tests. + + Some test harnesses support parallel test execution on separate processes + and/or threads (behavior varies by test harness). If this key is present, + the test harness should not attempt to run this test in parallel with any + other test. + + By convention, the value of this key is a string describing why the test + can't be run in parallel. + +.. _manifest_filter_language: + +Manifest Filter Language +------------------------ + +Some manifest keys accept a special filter syntax as their values. These +values are essentially boolean expressions that are evaluated at test +execution time. + +The expressions can reference a well-defined set of variables, such as +``os`` and ``debug``. These variables are populated from the +``mozinfo.json`` file. For the full list of available variables, see +the :ref:`mozinfo documentation <mozinfo_attributes>`. + +See +`the source <https://hg.mozilla.org/mozilla-central/file/default/testing/mozbase/manifestparser/manifestparser/manifestparser.py>`_ for the full documentation of the +expression syntax until it is documented here. + +.. todo:: + + Document manifest filter language. + +.. _manifest_file_installation: + +File Installation +----------------- + +Files referenced by manifests are automatically installed into the object +directory into paths defined in +:py:func:`mozbuild.frontend.emitter.TreeMetadataEmitter._process_test_manifest`. + +Relative paths resolving to parent directory (e.g. +``support-files = ../foo.txt`` have special behavior. + +For ``support-files``, the file will be installed to the default destination +for that manifest. Only the file's base name is used to construct the final +path: directories are irrelevant. Files starting with ``/`` are an exception, +these are installed relative to the root of the destination; the base name is +instead used to select the file.. + +For all other entry types, the file installation is skipped. + +.. _reftest_manifests: + +Reftest Manifests +================= + +See `MDN <https://developer.mozilla.org/en-US/docs/Creating_reftest-based_unit_tests>`_. diff --git a/build/docs/toolchains.rst b/build/docs/toolchains.rst new file mode 100644 index 000000000..eba640fa0 --- /dev/null +++ b/build/docs/toolchains.rst @@ -0,0 +1,111 @@ +.. _build_toolchains: + +=========================== +Creating Toolchain Archives +=========================== + +There are various scripts in the repository for producing archives +of the build tools (e.g. compilers and linkers) required to build. + +Clang +===== + +See the ``build/build-clang`` directory. Read ``build/build-clang/README`` +for more. + +Windows +======= + +The ``build/windows_toolchain.py`` script is used to build and manage +Windows toolchain archives containing Visual Studio executables, SDKs, +etc. + +The way Firefox build automation works is an archive containing the +toolchain is produced and uploaded to an internal Mozilla server. The +build automation will download, verify, and extract this archive before +building. The archive is self-contained so machines don't need to install +Visual Studio, SDKs, or various other dependencies. Unfortunately, +Microsoft's terms don't allow Mozilla to distribute this archive +publicly. However, the same tool can be used to create your own copy. + +Configuring Your System +----------------------- + +It is **highly** recommended to perform this process on a fresh installation +of Windows 7 or 10 (such as in a VM). Installing all updates through +Windows Update is not only acceptable - it is encouraged. Although it +shouldn't matter. + +Next, install Visual Studio 2015 Community. The download link can be +found at https://www.visualstudio.com/en-us/products/visual-studio-community-vs.aspx. +Be sure to follow these install instructions: + +1. Choose a ``Custom`` installation and click ``Next`` +2. Select ``Programming Languages`` -> ``Visual C++`` (make sure all sub items are + selected) +3. Under ``Windows and Web Development`` uncheck everything except + ``Universal Windows App Development Tools`` and the items under it + (should be ``Tools (1.3.1)...`` and the ``Windows 10 SDK``). + +Once Visual Studio 2015 Community has been installed, from a checkout +of mozilla-central, run something like the following to produce a ZIP +archive:: + + $ ./mach python build/windows_toolchain.py create-zip vs2015u3 + +The produced archive will be the argument to ``create-zip`` + ``.zip``. + +Firefox for Android with Gradle +=============================== + +To build Firefox for Android with Gradle in automation, archives +containing both the Gradle executable and a Maven repository +comprising the exact build dependencies are produced and uploaded to +an internal Mozilla server. The build automation will download, +verify, and extract these archive before building. These archives +provide a self-contained Gradle and Maven repository so that machines +don't need to fetch additional Maven dependencies at build time. +(Gradle and the downloaded Maven dependencies can be both +redistributed publicly.) + +Archiving the Gradle executable is straight-forward, but archiving a +local Maven repository is not. Therefore a special Task Cluster +Docker image and job exist for producing the required archives. The +Docker image definition is rooted in +``taskcluster/docker/android-gradle-build``. The Task Cluster job +definition is in +``testing/taskcluster/tasks/builds/android_api_15_gradle_dependencies.yml``. +The job runs in a container based on the custom Docker image and +spawns a Sonatype Nexus proxying Maven repository process in the +background. The job builds Firefox for Android using Gradle and the +in-tree Gradle configuration rooted at ``build.gradle``. The spawned +proxying Maven repository downloads external dependencies and collects +them. After the Gradle build completes, the job archives the Gradle +version used to build, and the downloaded Maven repository, and +exposes them as Task Cluster artifacts. + +Here is `an example try job fetching these dependencies +<https://treeherder.mozilla.org/#/jobs?repo=try&revision=75bc98935147&selectedJob=17793653>`_. +The resulting task produced a `Gradle archive +<https://queue.taskcluster.net/v1/task/CeYMgAP3Q-KF8h37nMhJjg/runs/0/artifacts/public%2Fbuild%2Fgradle.tar.xz>`_ +and a `Maven repository archive +<https://queue.taskcluster.net/v1/task/CeYMgAP3Q-KF8h37nMhJjg/runs/0/artifacts/public%2Fbuild%2Fjcentral.tar.xz>`_. +These archives were then uploaded (manually) to Mozilla automation +using tooltool for consumption in Gradle builds. + +To update the version of Gradle in the archive produced, update +``gradle/wrapper/gradle-wrapper.properties``. Be sure to also update +the SHA256 checksum to prevent poisoning the build machines! + +To update the versions of Gradle dependencies used, update +``dependencies`` sections in the in-tree Gradle configuration rooted +at ``build.gradle``. Once you are confident your changes build +locally, push a fresh try build with an invocation like:: + + $ hg push-to-try -m "try: -b o -p android-api-15-gradle-dependencies" + +Then `upload your archives to tooltool +<https://wiki.mozilla.org/ReleaseEngineering/Applications/Tooltool#How_To_Upload_To_Tooltool>`_, +update the in-tree manifests in +``mobile/android/config/tooltool-manifests``, and push a fresh try +build. diff --git a/build/docs/visualstudio.rst b/build/docs/visualstudio.rst new file mode 100644 index 000000000..3fbf28e94 --- /dev/null +++ b/build/docs/visualstudio.rst @@ -0,0 +1,100 @@ +.. _build_visualstudio: + +====================== +Visual Studio Projects +====================== + +The build system contains alpha support for generating Visual Studio +project files to aid with development. + +To generate Visual Studio project files, you'll need to have a configured tree:: + + mach configure + +(If you have built recently, your tree is already configured.) + +Then, simply generate the Visual Studio build backend:: + + mach build-backend -b VisualStudio + +If all goes well, the path to the generated Solution (``.sln``) file should be +printed. You should be able to open that solution with Visual Studio 2010 or +newer. + +Currently, output is hard-coded to the Visual Studio 2010 format. If you open +the solution in a newer Visual Studio release, you will be prompted to upgrade +projects. Simply click through the wizard to do that. + +Structure of Solution +===================== + +The Visual Studio solution consists of hundreds of projects spanning thousands +of files. To help with organization, the solution is divided into the following +trees/folders: + +Build Targets + This folder contains common build targets. The *full* project is used to + perform a full build. The *binaries* project is used to build just binaries. + The *visual-studio* project can be built to regenerate the Visual Studio + project files. + + Performing the *clean* action on any of these targets will clean the + *entire* build output. + +Binaries + This folder contains common binaries that can be executed from within + Visual Studio. If you are building the Firefox desktop application, + the *firefox* project will launch firefox.exe. You probably want one of + these set to your startup project. + +Libraries + This folder contains entries for each static library that is produced as + part of the build. These roughly correspond to each directory in the tree + containing C/C++. e.g. code from ``dom/base`` will be contained in the + ``dom_base`` project. + + These projects don't do anything when built. If you build a project here, + the *binaries* build target project is built. + +Updating Project Files +====================== + +As you pull and update the source tree, your Visual Studio files may fall out +of sync with the build configuration. The tree should still build fine from +within Visual Studio. But source files may be missing and IntelliSense may not +have the proper build configuration. + +To account for this, you'll want to periodically regenerate the Visual Studio +project files. You can do this within Visual Studio by building the +``Build Targets :: visual-studio`` project or by running +``mach build-backend -b VisualStudio`` from the command line. + +Currently, regeneration rewrites the original project files. **If you've made +any customizations to the solution or projects, they will likely get +overwritten.** We would like to improve this user experience in the +future. + +Moving Project Files Around +=========================== + +The produced Visual Studio solution and project files should be portable. +If you want to move them to a non-default directory, they should continue +to work from wherever they are. If they don't, please file a bug. + +Invoking mach through Visual Studio +=================================== + +It's possible to build the tree via Visual Studio. There is some light magic +involved here. + +Alongside the Visual Studio project files is a batch script named ``mach.bat``. +This batch script sets the environment variables present in your *MozillaBuild* +development environment at the time of Visual Studio project generation +and invokes *mach* inside an msys shell with the arguments specified to the +batch script. This script essentially allows you to invoke mach commands +inside the MozillaBuild environment without having to load MozillaBuild. + +While projects currently only utilize the ``mach build`` command, the batch +script does not limit it's use: any mach command can be invoked. Developers +may abuse this fact to add custom projects and commands that invoke other +mach commands. |