From: Helen Koike helen.koike@collabora.com
Add documentation of kci-gitlab.
Signed-off-by: Shreeya Patel shreeya.patel@collabora.com Signed-off-by: Vignesh Raman vignesh.raman@collabora.com Signed-off-by: Helen Koike helen.koike@collabora.com --- Documentation/ci/gitlab-ci/gitlab-ci.rst | 471 +++++++++++++++++++++++ Documentation/index.rst | 7 + 2 files changed, 478 insertions(+) create mode 100644 Documentation/ci/gitlab-ci/gitlab-ci.rst
diff --git a/Documentation/ci/gitlab-ci/gitlab-ci.rst b/Documentation/ci/gitlab-ci/gitlab-ci.rst new file mode 100644 index 000000000000..940a80006923 --- /dev/null +++ b/Documentation/ci/gitlab-ci/gitlab-ci.rst @@ -0,0 +1,471 @@ +.. SPDX-License-Identifier: GPL-2.0+ + +========================================= +Automated Testing with kci-gitlab +========================================= + + +This documentation outlines kci-gitlab, a GitLab CI/CD workflow for the +Linux Kernel. kci-gitlab pipeline is specifically designed for kernel testing. +It provides kernel developers with an integrated, efficient, and flexible +testing framework using GitLab's CI/CD capabilities. The workflow is designed +to simplify testing for developers, allowing tests to be run on any branch at +any time, without the need for specific infrastructure. Tests are automatically +triggered on each `git push`, with results displayed in the GitLab UI. + +.. image:: images/the-pipeline.png + :alt: GitLab-CI pipeline for kernel testing + :align: center + +Customizations and extensions of the pipeline are possible through the +scenarios. Scenarios can override existing jobs, change configurations, or +define new jobs and stages. See :ref:`extending-the-ci` section. + +.. note:: If you are unfamiliar with GitLab CI/CD basic concepts, please check + the `official documentation https://docs.gitlab.com/ee/ci/`_. + +.. only:: subproject and html + + Indices + ======= + + * :ref:`genindex` + +Setup +----- + +The kci-gitlab pipeline is set up with minimal configuration required. Pushing code to a +GitLab repository automatically triggers the pipeline provided the CI/CD configuration +file path is set as per the instructions given below.. + + .. code-block:: bash + + # Download the Linux kernel source code + git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git + # Create a repository on GitLab and add it as a remote + git remote add gitlab https://gitlab.yourinstance.com/your-username/your-repo.git + # In GitlabCI, go to Settings > CI/CD > General pipelines > CI/CD Configuration file + # and add the following path to it :- + tools/ci/gitlab-ci/gitlab-ci.yml + # Push the code to GitLab + git push gitlab + +.. image:: images/pipelines-on-push.png + :alt: Pipeline triggered on push + :align: center + +Troubleshooting +--------------- + +If the pipeline doesn't trigger automatically, check the following: + +1. **Enable CI/CD in Project Settings:** + + - Go to `Settings > General > Visibility, project features, permissions`. + - Under `Repository`, ensure the `CI/CD` toggle is enabled. + +2. **Enable Container Registry:** + + - Still in `Settings`, find the `Container Registry` section. + - Enable the `Container Registry` toggle. + +3. **CI Minutes and Resources:** + + - If you've exhausted CI minutes or other resources on the Free Tier, + consider setting up a local GitLab runner (see below). + +Setting Up a Local GitLab Runner +-------------------------------- + +You can use your own machine as a runner, instead of the shared runners provided +by your GitLab instance. + +1. **Generate a GitLab Runner Token:** + + - Navigate to `Settings > CI/CD > Runners`. + - Expand the `Runners` section and click on "New project runner". + - Choose "Run untagged jobs" and click "Create runner". + - Copy the provided token. + +.. image:: images/new-project-runner.png + :alt: New project runner button + :align: center + +2. **Launch the Runner:** + + - Ensure Docker is installed and your user is added to the Docker group: + + .. code-block:: bash + + sudo usermod -aG docker <your-user> + + - Log in again to apply the changes. + - Set up the runner: + + .. code-block:: bash + + export GITLAB_RUNNER_TOKEN=<your_token> + export GITLAB_URL=https://gitlab.yourinstance.com # Use this for instances other than gitlab.com + cd tools/ci/gitlab-ci + ./bootstrap-gitlab-runner.sh + + +Current Pipeline Jobs +--------------------- + +stage: container +^^^^^^^^^^^^^^^^ + +**job: debian/x86_64_build debian/arm64_build** + +This job prepares the container for x86_64 and arm64 architectures that will be +used by subsequent jobs. It starts from a base Debian image, installing necessary +tools for building the kernel and running tests. The resulting image is pushed to +the project registry and cached. If the image already exists in the registry, it +won't be rebuilt. + +To force a rebuild, update the `FDO_DISTRIBUTION_TAG` variable in the +`container.yml` file. + +stage: static-checks +^^^^^^^^^^^^^^^^^^^^ + +**job: checkpatch** + +Runs the `checkpatch.pl` script on the last `$ICI_PATCH_SERIES_SIZE` commits. +This variable is determined by: + +- `ICI_PATCH_SERIES_SIZE=` The number of differing patches between target and + source branches for merge requests; Or +- `ICI_PATCH_SERIES_SIZE=$KCI_PATCH_SERIES_SIZE` if `KCI_PATCH_SERIES_SIZE` is + set (see :ref:`how-to-set-variables` below). + +Defaults to 1 and raises a GitLab warning if unable to identify the number of +commits. + +**job: smatch** + +Checks `.c` files in the last `$ICI_PATCH_SERIES_SIZE` commits. Creates a +job based on architecture and configuration mentioned in the scenario specific +yaml files. +If a smatch database exists (see `job: smatch-db-generate` below), it reuses it. + +stage: build +^^^^^^^^^^^^ + +**job: build:arm32 build:arm64 build:x86_64** + +Compiles the kernel for each architecture and configuration +in `container.yml`. + +Raises a GitLab warning if "warning" is found in the build log. + +.. image:: images/job-matrix.png + :alt: build kernel jobs + :align: center + +**job: build-docs** + +Builds documentation. Creates a job for each documentation type. Not run +automatically; requires manual triggering. + +stage: test +^^^^^^^^^^^ + +**job: test-boot** + +Runs boot tests using virtme to launch a virtual machine and execute +`test-boot.sh` script. + +stage: cache +^^^^^^^^^^^^ + +**job: smatch-db-generate** + +Generates a smatch database for use by the `smatch` job. Not run automatically; +requires manual triggering. + +.. _extending-the-ci: + +Extending the CI - Test Scenarios (KCI_SCENARIO) +------------------------------------------------ + +The kci-gitlab pipeline offers flexibility and adaptability through the use of +scenarios, enhancing the CI/CD process with broad customization options. +Key capabilities include: + +- **Overriding Existing Jobs:** Tailor existing jobs to meet specific needs or + conditions. + +- **Changing Configurations:** Dynamically adapt job settings to suit various + environments or subsystem requirements. + +- **Defining New Jobs and Stages:** Introduce new jobs and stages for additional + tests, build processes, or deployment strategies. + +These features are particularly useful when a subsystem has distinct +requirements. For instance, to enable testing different configurations for a +specific architecture, running static checks with varied arguments, or +installing specialized tools to conduct targeted tests. + +Writing a test scenario +^^^^^^^^^^^^^^^^^^^^^^^ + +The kci-gitlab pipeline configuration allows the inclusion of additional `.yml` files +based on the `KCI_SCENARIO` variable. For example, setting `KCI_SCENARIO` to `my-scenario` +includes `my-scenario.yml` from the `scenarios/<my-scenario>/` folder. + +A different container image can be built for the newly added scenario by modifying +the FDO_DISTRIBUTION_TAG. FDO_DISTRIBUTION_EXEC can be used to run scripts to install +the required tools for the container. To illustrate, building a container for a specific +architecture with a custom configuration can be achieved by overriding the `build` job +in the `scenarios/<my-scenario>/my-scenario.yml` file: + +.. code-block:: yaml + + variables: + FDO_DISTRIBUTION_TAG: "<tag>" + FDO_DISTRIBUTION_EXEC: <script> + + build:arm64: + variables: + KCI_KCONFIGS_ENABLE: "CONFIG1 CONFIG2" + KCI_DEFCONFIG: "my/custom/config1" + +We also need to add `scenarios/<my-scenario>/test.yml` to override the test-boot job +and add job to run tests for the scenario. + +This modifies the builds and static checks for `arm64` with different configurations. + +To select this scenario, trigger the pipeline with KCI_SCENARIO=my-scenario. See +:ref:`how-to-set-variables` below. + +Additionally, we need to add `scenarios/<my-scenario>/test.yml` to override the default +test-boot job and add job to run tests for the scenario. + +DRM scenario +^^^^^^^^^^^^ + +Setting `KCI_SCENARIO` to `drm` includes `drm.yml` from the `scenarios/drm/` folder. + +A different container image can be built for the newly added `drm` scenario by modifying +the FDO_DISTRIBUTION_TAG. FDO_DISTRIBUTION_EXEC runs the `prepare-container.sh` script, +which installs the required tools (e.g., deqp-runner, IGT, Rust) for the container. +To illustrate, building a container for a specific architecture with a custom +configuration can be achieved by overriding the build job in the `scenarios/drm/drm.yml` file: + +.. code-block:: yaml + + variables: + FDO_DISTRIBUTION_TAG: "<tag>" + FDO_DISTRIBUTION_EXEC: <prepare-container.sh> + + build:arm64: + variables: + KCI_KCONFIGS_ENABLE: "DRM_VKMS DRM_BOCHS" + +This modifies the builds for x86_64 to enable VKMS driver in the kernel. + +Additionally, we need to add `scenarios/drm/test.yml` to override the default +test-boot job and add job to run tests for the `drm` scenario. + +Below is a pipeline running IGT tests using igt-runner provided by +deqp-runner for :ref:`VKMS <vkms>`. + +.. image:: images/drm-vkms.png + :alt: DRM vkms job + :align: center + +`deqp-runner tool https://gitlab.freedesktop.org/mesa/deqp-runner`_ supports parallel +testing of various test suites, including dEQP, Piglit, GTest, and IGT GPU Tools, +using baseline expectations and known flakes. deqp-runner contains experimental support +for running IGT tests. `IGT GPU Tools https://gitlab.freedesktop.org/drm/igt-gpu-tools`_ +is a collection of tools for development and testing of the DRM drivers. + +Refer to Documentation/gpu/automated_testing.rst for details regarding +fails/flakes/skips files. + +Variables +--------- + +kci-gitlab supports various variables to modify pipeline behavior or for use +in jobs. + +- **CI_ Prefix:** Standard GitLab CI/CD variables (see GitLab documentation). +- **KCI_ Prefix:** Custom variables defined for kernel CI. +- **ICI_ Prefix:** Internal variables used between scripts (not for external + use). + +.. _how-to-set-variables: + +How to Set Variables +-------------------- + +Variables can be set in several ways: + +- **Project Settings:** Under `CI/CD > Variables`. +- **Pipeline UI:** When triggering a pipeline manually. +- **Command Line:** When triggering a pipeline manually (see + :ref:`triggering-pipelines-from-command-line` below). +- **YML Files:** Using the `variables` keyword. +- **Commit Message:** For runtime variables only (see + :ref:`setting-variables-in-the-commit-message` below). + +.. image:: images/variables.png + :alt: Manual creation of pipeline + :align: center + +Variables Precedence +-------------------- + +- **Commit Message Variables:** Highest precedence if evaluated at runtime. +- **Pipeline Variables:** Next in precedence. +- **Project Variables:** Follow pipeline variables. +- **YML File Variables:** Considered after the above levels. + +.. _setting-variables-in-the-commit-message: + +Setting Variables in the Commit Message +--------------------------------------- + +Runtime variables can be set in the commit message. Patterns like +`KCI_VARIABLE=value` are extracted and exported to the job. To avoid including +variables in the git history, add them after three dashes (`---`) in the commit +message, as `git am` ignores text after this line. + +Example: + +.. code-block:: + + Title of my awesome commit + + This is the commit message description of my awesome patch + --- + KCI_PATCH_SERIES_SIZE=4 + +Description of Each Variable +---------------------------- + +**KCI_BUILD_ARCH** + Defines the build architecture to be used in FDO_REPO_SUFFIX for the container + image name. + +**KCI_KERNEL_ARCH** + Defines the architecture to be used in the kernel build jobs and static checks jobs. + Usually set in the `container.yml` or overridden in scenarios/<my-scenario>/my-scenario.yml. + +**KCI_DEFCONFIG** + Defines the config file to be used in the build-kernel and static checks + jobs. Usually set in the `container.yml` or overridden in scenarios/<my-scenario>/my-scenario.yml. + +**KCI_KCONFIGS_{ENABLE,DISABLE,MODULE}** + Defines the extra configs to be enabled, disabled or set as a module, used + in the build-kernel and static checks jobs. Usually set in the `container.yml` or + overridden in scenarios/<my-scenario>/my-scenario.yml. + +**KCI_SCENARIO** + Used to select which extra scenario file to include in the pipeline. See + :ref:`extending-the-ci` section above. Usually set by the user at project or + pipeline level. + +**KCI_CHECKPATCH_OPTIONS** + Used in `checkpatch.pl "$KCI_CHECKPATCH_OPTIONS"` (see checkpatch + documentation). It is commonly used with the --ignore flag to suppress + specific warnings generated by checkpatch.pl. It can also be defined in the + commit message, since it is evaluated in run time. + +**KCI_PATCH_SERIES_SIZE** + Used to define the size of the patch series, see `job: checkpatch` section + above. It is evaluated in run time, and can be set in the commit message. + +.. _triggering-pipelines-from-command-line: + +Triggering Pipelines from Command Line +-------------------------------------- + +Pipelines can be triggered from the command line with custom variables using the +`GitLab CLI tool https://docs.gitlab.com/ee/editor_extensions/gitlab_cli`_. + +Example: + +.. code-block:: bash + + glab auth login + glab ci run -b gitlab-draft -R https://gitlab.collabora.com/koike/linux/ --variables-env KCI_PATCH_SERIES_SIZE:4 + + +Debugging and Replicating Jobs Locally +-------------------------------------- + +When a job fails in GitLab CI/CD, it's handy to replicate the issue in the +same environment used by the GitLab CI/CD runner. This allows for interactive +execution of each step and the use of debugging tools to pinpoint the failure's +root cause. + +Rather than repeatedly modifying scripts and running the entire pipeline for +debugging, you can download the specific Docker image used by the job and run it +locally. + +To do this, first inspect the failed job in GitLab CI/CD. Look for a message +indicating the Docker image used, typically in this format: + + Pulling docker image registry.gitlab.collabora.com/koike/linux/debian/bookworm-slim:2024-02-6-ci-test-1 + +You can then use this image to run the job locally. For example: + +.. code-block:: bash + + IMAGE=registry.gitlab.collabora.com/koike/linux/debian/bookworm-slim:2024-02-6-ci-test-1 + docker pull $IMAGE + docker run --rm -v `pwd`:/linux -w /linux $IMAGE bash + + +Suggestions +----------- + +Send Pipeline Links with Your Patch +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When submitting patches or merge requests, it's highly beneficial to include +links to the related GitLab CI pipelines. This practice enhances the review +process in several ways: + +1. **Immediate Visibility:** Reviewers can immediately see the results of + automated tests, making it easier to assess the patch's impact. + +2. **Increased Confidence:** Successful pipeline runs increase confidence in the + changes, demonstrating that they pass existing tests. + +3. **Efficient Troubleshooting:** If there are issues, pipeline links allow both + authors and reviewers to quickly access logs and test results, facilitating + faster troubleshooting and iteration. + +4. **Transparency:** Providing pipeline links promotes transparency in the + development process, making it clear how changes have been verified. + +To include a pipeline link in your patch or merge request, simply copy the URL +of the pipeline from your GitLab project's CI/CD pipeline page and paste it into +your commit description after three dashes (`---`) or as a reply to your email +patch. + +Always Green Pipeline +^^^^^^^^^^^^^^^^^^^^^ + +Maintaining an "always green" pipeline refers to the practice of ensuring that +the main branch's pipeline is always in a passing state. This approach has +several advantages: + +1. **Reliable Main Branch:** A green pipeline indicates a stable and reliable + main branch, which is crucial for continuous integration practices. + +2. **Immediate Feedback:** Developers receive immediate feedback on their + changes. If a merge causes the pipeline to fail, it's a clear signal that the + change introduced an issue. + +3. **Faster Iteration:** An always green pipeline facilitates faster development + and iteration, as developers can confidently build on top of the latest main + branch without worrying about hidden failures. + +4. **Culture of Responsibility:** It fosters a culture of responsibility, where + developers are encouraged to fix broken builds promptly and avoid merging + changes that could disrupt the pipeline. diff --git a/Documentation/index.rst b/Documentation/index.rst index f9f525f4c0dd..701e70b54780 100644 --- a/Documentation/index.rst +++ b/Documentation/index.rst @@ -101,6 +101,13 @@ Architecture-specific documentation
CPU architectures <arch/index>
+CI: Automated testing documentation +=================================== + +.. toctree:: + :maxdepth: 2 + + ci/gitlab-ci/gitlab-ci
Other documentation ===================