summaryrefslogtreecommitdiffstats
path: root/testing/docker/README.md
diff options
context:
space:
mode:
Diffstat (limited to 'testing/docker/README.md')
-rw-r--r--testing/docker/README.md145
1 files changed, 145 insertions, 0 deletions
diff --git a/testing/docker/README.md b/testing/docker/README.md
new file mode 100644
index 000000000..ca4a197c1
--- /dev/null
+++ b/testing/docker/README.md
@@ -0,0 +1,145 @@
+# Docker Images for use in TaskCluster
+
+This folder contains various docker images used in [taskcluster](http://docs.taskcluster.net/) as well as other misc docker images which may be useful for
+hacking on gecko.
+
+## Organization
+
+Each folder describes a single docker image. We have two types of images that can be defined:
+
+1. [Task Images (build-on-push)](#task-images-build-on-push)
+2. [Docker Images (prebuilt)](#docker-registry-images-prebuilt)
+
+These images depend on one another, as described in the [`FROM`](https://docs.docker.com/v1.8/reference/builder/#from)
+line at the top of the Dockerfile in each folder.
+
+Images could either be an image intended for pushing to a docker registry, or one that is meant either
+for local testing or being built as an artifact when pushed to vcs.
+
+### Task Images (build-on-push)
+
+Images can be uploaded as a task artifact, [indexed](#task-image-index-namespace) under
+a given namespace, and used in other tasks by referencing the task ID.
+
+Important to note, these images do not require building and pushing to a docker registry, and are
+build per push (if necessary) and uploaded as task artifacts.
+
+The decision task that is run per push will [determine](#context-directory-hashing)
+if the image needs to be built based on the hash of the context directory and if the image
+exists under the namespace for a given branch.
+
+As an additional convenience, and a precaution to loading images per branch, if an image
+has been indexed with a given context hash for mozilla-central, any tasks requiring that image
+will use that indexed task. This is to ensure there are not multiple images built/used
+that were built from the same context. In summary, if the image has been built for mozilla-central,
+pushes to any branch will use that already built image.
+
+To use within an in-tree task definition, the format is:
+
+```yaml
+image:
+ type: 'task-image'
+ path: 'public/image.tar.zst'
+ taskId: '{{#task_id_for_image}}builder{{/task_id_for_image}}'
+```
+
+##### Context Directory Hashing
+
+Decision tasks will calculate the sha256 hash of the contents of the image
+directory and will determine if the image already exists for a given branch and hash
+or if a new image must be built and indexed.
+
+Note: this is the contents of *only* the context directory, not the
+image contents.
+
+The decision task will:
+1. Recursively collect the paths of all files within the context directory
+2. Sort the filenames alphabetically to ensure the hash is consistently calculated
+3. Generate a sha256 hash of the contents of each file.
+4. All file hashes will then be combined with their path and used to update the hash
+of the context directory.
+
+This ensures that the hash is consistently calculated and path changes will result
+in different hashes being generated.
+
+##### Task Image Index Namespace
+
+Images that are built on push and uploaded as an artifact of a task will be indexed under the
+following namespaces.
+
+* docker.images.v2.level-{level}.{image_name}.latest
+* docker.images.v2.level-{level}.{image_name}.pushdate.{year}.{month}-{day}-{pushtime}
+* docker.images.v2.level-{level}.{image_name}.hash.{context_hash}
+
+Not only can images be browsed by the pushdate and context hash, but the 'latest' namespace
+is meant to view the latest built image. This functions similarly to the 'latest' tag
+for docker images that are pushed to a registry.
+
+### Docker Registry Images (prebuilt)
+
+***Deprecation Warning: Use of prebuilt images should only be used for base images (those that other images
+will inherit from), or private images that must be stored in a private docker registry account. Existing
+public images will be converted to images that are built on push and any newly added image should
+follow this pattern.***
+
+These are images that are intended to be pushed to a docker registry and used by specifying the
+folder name in task definitions. This information is automatically populated by using the 'docker_image'
+convenience method in task definitions.
+
+Example:
+ image: {#docker_image}builder{/docker_image}
+
+Each image has a version, given by its `VERSION` file. This should be bumped when any changes are made that will be deployed into taskcluster.
+Then, older tasks which were designed to run on an older version of the image can still be executed in taskcluster, while new tasks can use the new version.
+
+Each image also has a `REGISTRY`, defaulting to the `REGISTRY` in this directory, and specifying the image registry to which the completed image should be uploaded.
+
+## Building images
+
+Generally, images can be pulled from the [registry](./REGISTRY) rather than
+built locally, however, for developing new images it's often helpful to hack on
+them locally.
+
+To build an image, invoke `build.sh` with the name of the folder (without a trailing slash):
+```sh
+./build.sh base
+```
+
+This is a tiny wrapper around building the docker images via `docker
+build -t $REGISTRY/$FOLDER:$FOLDER_VERSION`
+
+Note: If no "VERSION" file present in the image directory, the tag 'latest' will be used and no
+registry user will be defined. The image is only meant to run locally and will overwrite
+any existing image with the same name and tag.
+
+On completion, if the image has been tagged with a version and registry, `build.sh` gives a
+command to upload the image to the registry, but this is not necessary until the image
+is ready for production usage. Docker will successfully find the local, tagged image
+while you continue to hack on the image definitions.
+
+## Adding a new image
+
+The docker image primitives are very basic building block for
+constructing an "image" but generally don't help much with tagging it
+for deployment so we have a wrapper (./build.sh) which adds some sugar
+to help with tagging/versioning... Each folder should look something
+like this:
+
+```
+ - your_amazing_image/
+ - your_amazing_image/Dockerfile: Standard docker file syntax
+ - your_amazing_image/VERSION: The version of the docker file
+ (required* used during tagging)
+ - your_amazing_image/REGISTRY: Override default registry
+ (useful for secret registries)
+```
+
+## Conventions
+
+In some image folders you will see `.env` files these can be used in
+conjunction with the `--env-file` flag in docker to provide a
+environment with the given environment variables. These are primarily
+for convenience when manually hacking on the images.
+
+You will also see a `system-setup.sh` script used to build the image.
+Do not replicate this technique - prefer to include the commands and options directly in the Dockerfile.