mbox series

[0/3] kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing

Message ID 20240228225527.1052240-1-helen.koike@collabora.com
Headers show
Series kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing | expand

Message

Helen Mae Koike Fornazier Feb. 28, 2024, 10:55 p.m. UTC
Dear Kernel Community,

This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining a
basic test pipeline triggered by code pushes to a GitLab-CI instance. This
initial version includes static checks (checkpatch and smatch for now) and build
tests across various architectures and configurations. It leverages an
integrated cache for efficient build times and introduces a flexible 'scenarios'
mechanism for subsystem-specific extensions.

tl;dr: check this video to see a quick demo: https://youtu.be/TWiTjhjOuzg,
but don't forget to check the "Motivation for this work" below. Your feedback,
whether a simple thumbs up or down, is crucial to determine if it is worthwhile
to pursue this initiative.

GitLab is an Open Source platform that includes integrated CI/CD. The pipeline
provided in this patch is designed to work out-of-the-box with any GitLab
instance, including the gitlab.com Free Tier. If you reach the limits of the
Free Tier, consider using community instances like https://gitlab.freedesktop.org/.
Alternatively, you can set up a local runner for more flexibility. The
bootstrap-gitlab-runner.sh script included with this patch simplifies this
process, enabling you to run tests on your preferred infrastructure, including
your own machine.

For detailed information, please refer to the documentation included in the
patch, or check the rendered version here: https://koike.pages.collabora.com/-/linux/-/jobs/298498/artifacts/artifacts/Documentation-output/ci/gitlab-ci/gitlab-ci.html .


Motivation for this Work
========================

We all know tests are a major topic in the community, so let's mention the
specificities of this approach:

1. **Built-in User Interface:** GitLab CI/CD is growing in popularity and has an
user-friendly interface. Our experience with the upstream DRM-CI in the kernel
tree (see this blog post [https://www.collabora.com/news-and-blog/blog/2024/02/08/drm-ci-a-gitlab-ci-pipeline-for-linux-kernel-testing/] )
has provided insights into how such a system can benefit the wider community.

2. **Distributed Infrastructure:**
The proposed GitLab-CI pipeline is designed with a distributed infrastructure
model, being possible to run in any gitlab instance. 

3. **Reduce regressions:** Fostering a culture where people habitually run
validated tests and post their results can prevent many issues in post-merge
tests.

4. **Collaborative Testing Environment:** The kernel community is already
engaged in numerous testing efforts, including various GitLab-CI pipelines such
as DRM-CI, which I maintain, along with other solutions like KernelCI and
BPF-CI. This proposal is designed to further stimulate contributions to the
evolving testing landscape. Our goal is to establish a comprehensive suite of
common tools and files.

5. **Ownership of QA:** 
Discrepancies between kernel code and outdated tests often lead to misattributed
failures, complicating regression tracking. This issue, often arising from
neglected or deprioritized test updates, creates uncertainty about the source of
failures. Adopting an "always green pipeline" approach, as detailed in this
patch's documentation, encourages timely maintenance and validation of tests.
This ensures that testing accurately reflects the current state of the kernel,
thereby improving the effectiveness of our QA processes.

Additionally, if we discover that this method isn't working for us, we can
easily remove it from the codebase, as it is primarily contained within the ci/
folder.


Future Work
===========

**Expanding Static Checks:**
We have the opportunity to integrate a variety of static analysis tools,
including:
- dtbs_checks
- sparse
- yamllint
- dt-doc-validate
- coccicheck

**Adding Userspace Tests on VMs:**
To further our testing, we can implement userspace tests that run on virtual
machines (VMs), such as:
- kselftests
- kunit tests
- Subsystem-specific tests, customizable in the scenarios.

**Leveraging External Test Labs:**
We can extend our testing to external labs, similar to what DRM-CI currently
does. This includes:
- Lava labs
- Bare metal labs
- Using KernelCI-provided labs

**Other integrations**
- Submit results to KCIDB

**Lightweight Implementation for All Developers:**
We aim to design these tests to be lightweight, ensuring developers with limited
computing resources can still run essential tests. Resource-intensive tests can
be set to trigger manually, rather than automatically, to accommodate diverse
development environments.


Chat Discussions
================

For those interested in further discussions:

**Join Our Slack Channel:**
We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ .
Feel free to join and contribute to the conversation. The KernelCI team has
weekly calls where we also discuss the GitLab-CI pipeline.

**Acknowledgments:**
A special thanks to Nikolai Kondrashov, Tales da Aparecida - both from Red Hat -
and KernelCI community for their valuable feedback and support in this proposal.


I eagerly await your thoughts and suggestions on this initiative.

Also, if you want to see this initiave move faster, we are happy to discuss
funding options.

Best regards,
Helen Koike

Helen Koike (3):
  kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing
  kci-gitlab: Add documentation
  kci-gitlab: docs: Add images

 .gitlab-ci.yml                                |   2 +
 Documentation/ci/gitlab-ci/gitlab-ci.rst      | 404 ++++++++++++++++++
 .../ci/gitlab-ci/images/job-matrix.png        | Bin 0 -> 159752 bytes
 .../gitlab-ci/images/new-project-runner.png   | Bin 0 -> 607737 bytes
 .../ci/gitlab-ci/images/pipelines-on-push.png | Bin 0 -> 532143 bytes
 .../ci/gitlab-ci/images/the-pipeline.png      | Bin 0 -> 91675 bytes
 .../ci/gitlab-ci/images/variables.png         | Bin 0 -> 277518 bytes
 Documentation/index.rst                       |   7 +
 MAINTAINERS                                   |   9 +
 ci/gitlab-ci/bootstrap-gitlab-runner.sh       |  55 +++
 ci/gitlab-ci/ci-scripts/build-docs.sh         |  35 ++
 ci/gitlab-ci/ci-scripts/build-kernel.sh       |  35 ++
 ci/gitlab-ci/ci-scripts/ici-functions.sh      | 104 +++++
 ci/gitlab-ci/ci-scripts/install-smatch.sh     |  13 +
 .../ci-scripts/parse_commit_message.sh        |  27 ++
 ci/gitlab-ci/ci-scripts/run-checkpatch.sh     |  19 +
 ci/gitlab-ci/ci-scripts/run-smatch.sh         |  45 ++
 ci/gitlab-ci/docker-compose.yaml              |  18 +
 ci/gitlab-ci/linux.code-workspace             |  11 +
 ci/gitlab-ci/yml/build.yml                    |  43 ++
 ci/gitlab-ci/yml/cache.yml                    |  26 ++
 ci/gitlab-ci/yml/container.yml                |  36 ++
 ci/gitlab-ci/yml/gitlab-ci.yml                |  71 +++
 ci/gitlab-ci/yml/kernel-combinations.yml      |  18 +
 ci/gitlab-ci/yml/scenarios.yml                |  12 +
 ci/gitlab-ci/yml/scenarios/file-systems.yml   |  21 +
 ci/gitlab-ci/yml/scenarios/media.yml          |  21 +
 ci/gitlab-ci/yml/scenarios/network.yml        |  21 +
 ci/gitlab-ci/yml/static-checks.yml            |  21 +
 29 files changed, 1074 insertions(+)
 create mode 100644 .gitlab-ci.yml
 create mode 100644 Documentation/ci/gitlab-ci/gitlab-ci.rst
 create mode 100644 Documentation/ci/gitlab-ci/images/job-matrix.png
 create mode 100644 Documentation/ci/gitlab-ci/images/new-project-runner.png
 create mode 100644 Documentation/ci/gitlab-ci/images/pipelines-on-push.png
 create mode 100644 Documentation/ci/gitlab-ci/images/the-pipeline.png
 create mode 100644 Documentation/ci/gitlab-ci/images/variables.png
 create mode 100755 ci/gitlab-ci/bootstrap-gitlab-runner.sh
 create mode 100755 ci/gitlab-ci/ci-scripts/build-docs.sh
 create mode 100755 ci/gitlab-ci/ci-scripts/build-kernel.sh
 create mode 100644 ci/gitlab-ci/ci-scripts/ici-functions.sh
 create mode 100755 ci/gitlab-ci/ci-scripts/install-smatch.sh
 create mode 100755 ci/gitlab-ci/ci-scripts/parse_commit_message.sh
 create mode 100755 ci/gitlab-ci/ci-scripts/run-checkpatch.sh
 create mode 100755 ci/gitlab-ci/ci-scripts/run-smatch.sh
 create mode 100644 ci/gitlab-ci/docker-compose.yaml
 create mode 100644 ci/gitlab-ci/linux.code-workspace
 create mode 100644 ci/gitlab-ci/yml/build.yml
 create mode 100644 ci/gitlab-ci/yml/cache.yml
 create mode 100644 ci/gitlab-ci/yml/container.yml
 create mode 100644 ci/gitlab-ci/yml/gitlab-ci.yml
 create mode 100644 ci/gitlab-ci/yml/kernel-combinations.yml
 create mode 100644 ci/gitlab-ci/yml/scenarios.yml
 create mode 100644 ci/gitlab-ci/yml/scenarios/file-systems.yml
 create mode 100644 ci/gitlab-ci/yml/scenarios/media.yml
 create mode 100644 ci/gitlab-ci/yml/scenarios/network.yml
 create mode 100644 ci/gitlab-ci/yml/static-checks.yml

Comments

Laurent Pinchart Feb. 28, 2024, 11:07 p.m. UTC | #1
Hi Helen,

I appreciate the amount of work you've put into this, to improve testing
of the kernel as a whole. I'll need more time to answer, but please see
below for a quick comment already.

On Wed, Feb 28, 2024 at 07:55:24PM -0300, Helen Koike wrote:
> Dear Kernel Community,
> 
> This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining a
> basic test pipeline triggered by code pushes to a GitLab-CI instance. This
> initial version includes static checks (checkpatch and smatch for now) and build
> tests across various architectures and configurations. It leverages an
> integrated cache for efficient build times and introduces a flexible 'scenarios'
> mechanism for subsystem-specific extensions.
> 
> tl;dr: check this video to see a quick demo: https://youtu.be/TWiTjhjOuzg,
> but don't forget to check the "Motivation for this work" below. Your feedback,
> whether a simple thumbs up or down, is crucial to determine if it is worthwhile
> to pursue this initiative.
> 
> GitLab is an Open Source platform that includes integrated CI/CD. The pipeline
> provided in this patch is designed to work out-of-the-box with any GitLab
> instance, including the gitlab.com Free Tier. If you reach the limits of the
> Free Tier, consider using community instances like https://gitlab.freedesktop.org/.
> Alternatively, you can set up a local runner for more flexibility. The
> bootstrap-gitlab-runner.sh script included with this patch simplifies this
> process, enabling you to run tests on your preferred infrastructure, including
> your own machine.
> 
> For detailed information, please refer to the documentation included in the
> patch, or check the rendered version here: https://koike.pages.collabora.com/-/linux/-/jobs/298498/artifacts/artifacts/Documentation-output/ci/gitlab-ci/gitlab-ci.html .
> 
> 
> Motivation for this Work
> ========================
> 
> We all know tests are a major topic in the community, so let's mention the
> specificities of this approach:
> 
> 1. **Built-in User Interface:** GitLab CI/CD is growing in popularity and has an
> user-friendly interface. Our experience with the upstream DRM-CI in the kernel
> tree (see this blog post [https://www.collabora.com/news-and-blog/blog/2024/02/08/drm-ci-a-gitlab-ci-pipeline-for-linux-kernel-testing/] )
> has provided insights into how such a system can benefit the wider community.
> 
> 2. **Distributed Infrastructure:**
> The proposed GitLab-CI pipeline is designed with a distributed infrastructure
> model, being possible to run in any gitlab instance. 
> 
> 3. **Reduce regressions:** Fostering a culture where people habitually run
> validated tests and post their results can prevent many issues in post-merge
> tests.
> 
> 4. **Collaborative Testing Environment:** The kernel community is already
> engaged in numerous testing efforts, including various GitLab-CI pipelines such
> as DRM-CI, which I maintain, along with other solutions like KernelCI and
> BPF-CI. This proposal is designed to further stimulate contributions to the
> evolving testing landscape. Our goal is to establish a comprehensive suite of
> common tools and files.
> 
> 5. **Ownership of QA:** 
> Discrepancies between kernel code and outdated tests often lead to misattributed
> failures, complicating regression tracking. This issue, often arising from
> neglected or deprioritized test updates, creates uncertainty about the source of
> failures. Adopting an "always green pipeline" approach, as detailed in this
> patch's documentation, encourages timely maintenance and validation of tests.
> This ensures that testing accurately reflects the current state of the kernel,
> thereby improving the effectiveness of our QA processes.
> 
> Additionally, if we discover that this method isn't working for us, we can
> easily remove it from the codebase, as it is primarily contained within the ci/
> folder.
> 
> 
> Future Work
> ===========
> 
> **Expanding Static Checks:**
> We have the opportunity to integrate a variety of static analysis tools,
> including:
> - dtbs_checks
> - sparse
> - yamllint
> - dt-doc-validate
> - coccicheck
> 
> **Adding Userspace Tests on VMs:**
> To further our testing, we can implement userspace tests that run on virtual
> machines (VMs), such as:
> - kselftests
> - kunit tests
> - Subsystem-specific tests, customizable in the scenarios.
> 
> **Leveraging External Test Labs:**
> We can extend our testing to external labs, similar to what DRM-CI currently
> does. This includes:
> - Lava labs
> - Bare metal labs
> - Using KernelCI-provided labs
> 
> **Other integrations**
> - Submit results to KCIDB
> 
> **Lightweight Implementation for All Developers:**
> We aim to design these tests to be lightweight, ensuring developers with limited
> computing resources can still run essential tests. Resource-intensive tests can
> be set to trigger manually, rather than automatically, to accommodate diverse
> development environments.
> 
> 
> Chat Discussions
> ================
> 
> For those interested in further discussions:
> 
> **Join Our Slack Channel:**
> We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ .
> Feel free to join and contribute to the conversation. The KernelCI team has
> weekly calls where we also discuss the GitLab-CI pipeline.

Could we communicate using free software please ? Furthermore, it's not
possible to create an account on that slack instance unless you have an
e-mail address affiliated with a small number of companies
(https://kernelci.slack.com/signup#/domain-signup). That's a big no-go
for me.

> **Acknowledgments:**
> A special thanks to Nikolai Kondrashov, Tales da Aparecida - both from Red Hat -
> and KernelCI community for their valuable feedback and support in this proposal.
> 
> 
> I eagerly await your thoughts and suggestions on this initiative.
> 
> Also, if you want to see this initiave move faster, we are happy to discuss
> funding options.
> 
> Best regards,
> Helen Koike
> 
> Helen Koike (3):
>   kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing
>   kci-gitlab: Add documentation
>   kci-gitlab: docs: Add images
> 
>  .gitlab-ci.yml                                |   2 +
>  Documentation/ci/gitlab-ci/gitlab-ci.rst      | 404 ++++++++++++++++++
>  .../ci/gitlab-ci/images/job-matrix.png        | Bin 0 -> 159752 bytes
>  .../gitlab-ci/images/new-project-runner.png   | Bin 0 -> 607737 bytes
>  .../ci/gitlab-ci/images/pipelines-on-push.png | Bin 0 -> 532143 bytes
>  .../ci/gitlab-ci/images/the-pipeline.png      | Bin 0 -> 91675 bytes
>  .../ci/gitlab-ci/images/variables.png         | Bin 0 -> 277518 bytes
>  Documentation/index.rst                       |   7 +
>  MAINTAINERS                                   |   9 +
>  ci/gitlab-ci/bootstrap-gitlab-runner.sh       |  55 +++
>  ci/gitlab-ci/ci-scripts/build-docs.sh         |  35 ++
>  ci/gitlab-ci/ci-scripts/build-kernel.sh       |  35 ++
>  ci/gitlab-ci/ci-scripts/ici-functions.sh      | 104 +++++
>  ci/gitlab-ci/ci-scripts/install-smatch.sh     |  13 +
>  .../ci-scripts/parse_commit_message.sh        |  27 ++
>  ci/gitlab-ci/ci-scripts/run-checkpatch.sh     |  19 +
>  ci/gitlab-ci/ci-scripts/run-smatch.sh         |  45 ++
>  ci/gitlab-ci/docker-compose.yaml              |  18 +
>  ci/gitlab-ci/linux.code-workspace             |  11 +
>  ci/gitlab-ci/yml/build.yml                    |  43 ++
>  ci/gitlab-ci/yml/cache.yml                    |  26 ++
>  ci/gitlab-ci/yml/container.yml                |  36 ++
>  ci/gitlab-ci/yml/gitlab-ci.yml                |  71 +++
>  ci/gitlab-ci/yml/kernel-combinations.yml      |  18 +
>  ci/gitlab-ci/yml/scenarios.yml                |  12 +
>  ci/gitlab-ci/yml/scenarios/file-systems.yml   |  21 +
>  ci/gitlab-ci/yml/scenarios/media.yml          |  21 +
>  ci/gitlab-ci/yml/scenarios/network.yml        |  21 +
>  ci/gitlab-ci/yml/static-checks.yml            |  21 +
>  29 files changed, 1074 insertions(+)
>  create mode 100644 .gitlab-ci.yml
>  create mode 100644 Documentation/ci/gitlab-ci/gitlab-ci.rst
>  create mode 100644 Documentation/ci/gitlab-ci/images/job-matrix.png
>  create mode 100644 Documentation/ci/gitlab-ci/images/new-project-runner.png
>  create mode 100644 Documentation/ci/gitlab-ci/images/pipelines-on-push.png
>  create mode 100644 Documentation/ci/gitlab-ci/images/the-pipeline.png
>  create mode 100644 Documentation/ci/gitlab-ci/images/variables.png
>  create mode 100755 ci/gitlab-ci/bootstrap-gitlab-runner.sh
>  create mode 100755 ci/gitlab-ci/ci-scripts/build-docs.sh
>  create mode 100755 ci/gitlab-ci/ci-scripts/build-kernel.sh
>  create mode 100644 ci/gitlab-ci/ci-scripts/ici-functions.sh
>  create mode 100755 ci/gitlab-ci/ci-scripts/install-smatch.sh
>  create mode 100755 ci/gitlab-ci/ci-scripts/parse_commit_message.sh
>  create mode 100755 ci/gitlab-ci/ci-scripts/run-checkpatch.sh
>  create mode 100755 ci/gitlab-ci/ci-scripts/run-smatch.sh
>  create mode 100644 ci/gitlab-ci/docker-compose.yaml
>  create mode 100644 ci/gitlab-ci/linux.code-workspace
>  create mode 100644 ci/gitlab-ci/yml/build.yml
>  create mode 100644 ci/gitlab-ci/yml/cache.yml
>  create mode 100644 ci/gitlab-ci/yml/container.yml
>  create mode 100644 ci/gitlab-ci/yml/gitlab-ci.yml
>  create mode 100644 ci/gitlab-ci/yml/kernel-combinations.yml
>  create mode 100644 ci/gitlab-ci/yml/scenarios.yml
>  create mode 100644 ci/gitlab-ci/yml/scenarios/file-systems.yml
>  create mode 100644 ci/gitlab-ci/yml/scenarios/media.yml
>  create mode 100644 ci/gitlab-ci/yml/scenarios/network.yml
>  create mode 100644 ci/gitlab-ci/yml/static-checks.yml
Maxime Ripard Feb. 29, 2024, 8:43 a.m. UTC | #2
On Thu, Feb 29, 2024 at 01:07:25AM +0200, Laurent Pinchart wrote:
> > Chat Discussions
> > ================
> > 
> > For those interested in further discussions:
> > 
> > **Join Our Slack Channel:**
> > We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ .
> > Feel free to join and contribute to the conversation. The KernelCI team has
> > weekly calls where we also discuss the GitLab-CI pipeline.
> 
> Could we communicate using free software please ? Furthermore, it's not
> possible to create an account on that slack instance unless you have an
> e-mail address affiliated with a small number of companies
> (https://kernelci.slack.com/signup#/domain-signup). That's a big no-go
> for me.

Yeah, and that list looks super restrictive and arbitrary. Like,
microsoft is there but kernel.org isn't?

I'm sure there's a reason, but we should at the very least open it to
everyone.

Maxime
Nikolai Kondrashov Feb. 29, 2024, 9:26 a.m. UTC | #3
On 2/29/24 01:07, Laurent Pinchart wrote:
> On Wed, Feb 28, 2024 at 07:55:24PM -0300, Helen Koike wrote:
>> **Join Our Slack Channel:**
>> We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ .
>> Feel free to join and contribute to the conversation. The KernelCI team has
>> weekly calls where we also discuss the GitLab-CI pipeline.
> 
> Could we communicate using free software please ? Furthermore, it's not
> possible to create an account on that slack instance unless you have an
> e-mail address affiliated with a small number of companies
> (https://kernelci.slack.com/signup#/domain-signup). That's a big no-go
> for me.

Yes, it's not ideal that we use closed-source software for communication, but 
FWIW I'd be happy to invite you there. Perhaps if you try logging in e.g. with 
a Google account, I'd be able to see and approve your attempt too.

Otherwise, our video meetings are generally open to everyone and run in Jitsi.

Nick
Mark Brown Feb. 29, 2024, 11:09 a.m. UTC | #4
On Thu, Feb 29, 2024 at 09:39:01AM +0000, Sakari Ailus wrote:
> On Thu, Feb 29, 2024 at 01:07:25AM +0200, Laurent Pinchart wrote:

> > > We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ .
> > > Feel free to join and contribute to the conversation. The KernelCI team has
> > > weekly calls where we also discuss the GitLab-CI pipeline.

> > Could we communicate using free software please ? Furthermore, it's not
> > possible to create an account on that slack instance unless you have an
> > e-mail address affiliated with a small number of companies
> > (https://kernelci.slack.com/signup#/domain-signup). That's a big no-go
> > for me.

> I agree. There is no shortage of good alternatives either: we have IRC,
> Matrix and XMPP for instance. Just pick one of them.

And indeed KernelCI does actually already have #kernelci on libera.chat.
Laurent Pinchart Feb. 29, 2024, 11:19 a.m. UTC | #5
On Thu, Feb 29, 2024 at 01:10:16PM +0200, Nikolai Kondrashov wrote:
> On 2/29/24 11:34 AM, Laurent Pinchart wrote:
> > On Thu, Feb 29, 2024 at 11:26:51AM +0200, Nikolai Kondrashov wrote:
> >> On 2/29/24 01:07, Laurent Pinchart wrote:
> >>> On Wed, Feb 28, 2024 at 07:55:24PM -0300, Helen Koike wrote:
> >>>> **Join Our Slack Channel:**
> >>>> We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ .
> >>>> Feel free to join and contribute to the conversation. The KernelCI team has
> >>>> weekly calls where we also discuss the GitLab-CI pipeline.
> >>>
> >>> Could we communicate using free software please ? Furthermore, it's not
> >>> possible to create an account on that slack instance unless you have an
> >>> e-mail address affiliated with a small number of companies
> >>> (https://kernelci.slack.com/signup#/domain-signup). That's a big no-go
> >>> for me.
> >>
> >> Yes, it's not ideal that we use closed-source software for communication, but
> >> FWIW I'd be happy to invite you there. Perhaps if you try logging in e.g. with
> >> a Google account, I'd be able to see and approve your attempt too.
> > 
> > I don't use Google accounts to authenticate to third-party services,
> > sorry. And in any case, that won't make slack free software.
> 
> Of course. You're also welcome to join the #kernelci channel on libera.chat.

Isn't that a bit pointless if it's no the main IM channel ?

> It's much quieter there, though.
Mark Brown Feb. 29, 2024, 11:41 a.m. UTC | #6
On Thu, Feb 29, 2024 at 01:19:19PM +0200, Laurent Pinchart wrote:
> On Thu, Feb 29, 2024 at 01:10:16PM +0200, Nikolai Kondrashov wrote:

> > Of course. You're also welcome to join the #kernelci channel on libera.chat.

> Isn't that a bit pointless if it's no the main IM channel ?

It *was* the original channel and still gets some usage (mostly started
by me admittedly since I've never joined slack for a bunch of reasons
that make it hassle), IIRC the Slack was started because there were some
interns who had trouble figuring out IRC and intermittent connectivity
but people seem to have migrated.
Guillaume Tucker Feb. 29, 2024, 12:20 p.m. UTC | #7
Hello,

On 28/02/2024 23:55, Helen Koike wrote:
> Dear Kernel Community,
> 
> This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining a
> basic test pipeline triggered by code pushes to a GitLab-CI instance. This
> initial version includes static checks (checkpatch and smatch for now) and build
> tests across various architectures and configurations. It leverages an
> integrated cache for efficient build times and introduces a flexible 'scenarios'
> mechanism for subsystem-specific extensions.

This sounds like a nice starting point to me as an additional way
to run tests upstream.  I have one particular question as I see a
pattern through the rest of the email, please see below.

[...]

> 4. **Collaborative Testing Environment:** The kernel community is already
> engaged in numerous testing efforts, including various GitLab-CI pipelines such
> as DRM-CI, which I maintain, along with other solutions like KernelCI and
> BPF-CI. This proposal is designed to further stimulate contributions to the
> evolving testing landscape. Our goal is to establish a comprehensive suite of
> common tools and files.

[...]

> **Leveraging External Test Labs:**
> We can extend our testing to external labs, similar to what DRM-CI currently
> does. This includes:
> - Lava labs
> - Bare metal labs
> - Using KernelCI-provided labs
> 
> **Other integrations**
> - Submit results to KCIDB

[...]

> **Join Our Slack Channel:**
> We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ .
> Feel free to join and contribute to the conversation. The KernelCI team has
> weekly calls where we also discuss the GitLab-CI pipeline.
> 
> **Acknowledgments:**
> A special thanks to Nikolai Kondrashov, Tales da Aparecida - both from Red Hat -
> and KernelCI community for their valuable feedback and support in this proposal.

Where does this fit on the KernelCI roadmap?

I see it mentioned a few times but it's not entirely clear
whether this initiative is an independent one or in some way
linked to KernelCI.  Say, are you planning to use the kci tool,
new API, compiler toolchains, user-space and Docker images etc?
Or, are KernelCI plans evolving to follow this move?

Thanks,
Guillaume
Nikolai Kondrashov Feb. 29, 2024, 2:16 p.m. UTC | #8
On 2/29/24 2:20 PM, Guillaume Tucker wrote:
> Hello,
> 
> On 28/02/2024 23:55, Helen Koike wrote:
>> Dear Kernel Community,
>>
>> This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining a
>> basic test pipeline triggered by code pushes to a GitLab-CI instance. This
>> initial version includes static checks (checkpatch and smatch for now) and build
>> tests across various architectures and configurations. It leverages an
>> integrated cache for efficient build times and introduces a flexible 'scenarios'
>> mechanism for subsystem-specific extensions.
> 
> This sounds like a nice starting point to me as an additional way
> to run tests upstream.  I have one particular question as I see a
> pattern through the rest of the email, please see below.
> 
> [...]
> 
>> 4. **Collaborative Testing Environment:** The kernel community is already
>> engaged in numerous testing efforts, including various GitLab-CI pipelines such
>> as DRM-CI, which I maintain, along with other solutions like KernelCI and
>> BPF-CI. This proposal is designed to further stimulate contributions to the
>> evolving testing landscape. Our goal is to establish a comprehensive suite of
>> common tools and files.
> 
> [...]
> 
>> **Leveraging External Test Labs:**
>> We can extend our testing to external labs, similar to what DRM-CI currently
>> does. This includes:
>> - Lava labs
>> - Bare metal labs
>> - Using KernelCI-provided labs
>>
>> **Other integrations**
>> - Submit results to KCIDB
> 
> [...]
> 
>> **Join Our Slack Channel:**
>> We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ .
>> Feel free to join and contribute to the conversation. The KernelCI team has
>> weekly calls where we also discuss the GitLab-CI pipeline.
>>
>> **Acknowledgments:**
>> A special thanks to Nikolai Kondrashov, Tales da Aparecida - both from Red Hat -
>> and KernelCI community for their valuable feedback and support in this proposal.
> 
> Where does this fit on the KernelCI roadmap?
> 
> I see it mentioned a few times but it's not entirely clear
> whether this initiative is an independent one or in some way
> linked to KernelCI.  Say, are you planning to use the kci tool,
> new API, compiler toolchains, user-space and Docker images etc?
> Or, are KernelCI plans evolving to follow this move?

I would say this is an important part of KernelCI the project, considering its 
aim to improve testing and CI in the kernel. It's not a part of KernelCI the 
service as it is right now, although I would say it would be good to have 
ability to submit KernelCI jobs from GitLab CI and pull results in the same 
pipeline, as we discussed earlier.

Nick
Guillaume Tucker March 1, 2024, 9:56 p.m. UTC | #9
On 29/02/2024 17:28, Nicolas Dufresne wrote:
> Hi,
> 
> Le jeudi 29 février 2024 à 16:16 +0200, Nikolai Kondrashov a écrit :
>> On 2/29/24 2:20 PM, Guillaume Tucker wrote:
>>> Hello,
>>>
>>> On 28/02/2024 23:55, Helen Koike wrote:
>>>> Dear Kernel Community,
>>>>
>>>> This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining a
>>>> basic test pipeline triggered by code pushes to a GitLab-CI instance. This
>>>> initial version includes static checks (checkpatch and smatch for now) and build
>>>> tests across various architectures and configurations. It leverages an
>>>> integrated cache for efficient build times and introduces a flexible 'scenarios'
>>>> mechanism for subsystem-specific extensions.
>>>
>>> This sounds like a nice starting point to me as an additional way
>>> to run tests upstream.  I have one particular question as I see a
>>> pattern through the rest of the email, please see below.
>>>
>>> [...]
>>>
>>>> 4. **Collaborative Testing Environment:** The kernel community is already
>>>> engaged in numerous testing efforts, including various GitLab-CI pipelines such
>>>> as DRM-CI, which I maintain, along with other solutions like KernelCI and
>>>> BPF-CI. This proposal is designed to further stimulate contributions to the
>>>> evolving testing landscape. Our goal is to establish a comprehensive suite of
>>>> common tools and files.
>>>
>>> [...]
>>>
>>>> **Leveraging External Test Labs:**
>>>> We can extend our testing to external labs, similar to what DRM-CI currently
>>>> does. This includes:
>>>> - Lava labs
>>>> - Bare metal labs
>>>> - Using KernelCI-provided labs
>>>>
>>>> **Other integrations**
>>>> - Submit results to KCIDB
>>>
>>> [...]
>>>
>>>> **Join Our Slack Channel:**
>>>> We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ .
>>>> Feel free to join and contribute to the conversation. The KernelCI team has
>>>> weekly calls where we also discuss the GitLab-CI pipeline.
>>>>
>>>> **Acknowledgments:**
>>>> A special thanks to Nikolai Kondrashov, Tales da Aparecida - both from Red Hat -
>>>> and KernelCI community for their valuable feedback and support in this proposal.
>>>
>>> Where does this fit on the KernelCI roadmap?
>>>
>>> I see it mentioned a few times but it's not entirely clear
>>> whether this initiative is an independent one or in some way
>>> linked to KernelCI.  Say, are you planning to use the kci tool,
>>> new API, compiler toolchains, user-space and Docker images etc?
>>> Or, are KernelCI plans evolving to follow this move?
>>
>> I would say this is an important part of KernelCI the project, considering its 
>> aim to improve testing and CI in the kernel. It's not a part of KernelCI the 
>> service as it is right now, although I would say it would be good to have 
>> ability to submit KernelCI jobs from GitLab CI and pull results in the same 
>> pipeline, as we discussed earlier.

Right, I think this needs a bit of disambiguation.  The legacy
KernelCI system from the Linaro days several years ago is really
a service on its own like the many other CIs out there.  However,
the new KernelCI API and related tooling (kci command line, new
web dashboard, modular runtime design etc.) is not that.  It's
about addressing all the community requirements and that includes
being able to run a same test manually in a shell, or in a VM, or
automatically from GitLab CI or using a main generic pipeline
hosted by KernelCI itself.  With this approach, there's no
distinction between "the project" and "the service", and as we
discussed before there shouldn't even be a distinction with
KCIDB.  Just KernelCI.

However I don't really see this happening, unless I'm missing a
part of the story or some upcoming announcement with an updated
roadmap.  For some reason the old and established paradigm seems
unshakeable.  The new KernelCI implementation is starting to look
just like a refresh of the old one with newer components - which
is a huge missed opportunity to really change things IMHO.

This may sound like a bit of a tangent, facilitating GitLab CI
for the upstream kernel is of course significant progress in any
case - no question about that.  My comment is more about why it's
being driven hand-in-hand with KernelCI in what seems like a
diverging direction from KernelCI's announced plans.  Why push
for a GitLab-centered orchestration when there's a more universal
solution being proposed by the project?  I would find it easier
to understand - and I sense I'm not the only one here reading the
thread - if KernelCI wasn't mentioned that many times in the
cover letter and if the scripts didn't have KCI_* in so many
places, basically if this was clearly an independent initiative
such as KUnit, 0-day or regzbot.

> I'd like to add that both CI have a different purpose in the Linux project. This
> CI work is a pre-merge verification. Everyone needs to run checkpatch and
> smatch, this is automating it (and will catch those that forgot or ran it
> incorrectly). But it can go further by effectively testing specific patches on
> real hardware (with pretty narrow filters). It will help catch submission issues
> earlier, and reduce kernelCI regression rate. As a side effect, kernelCI infra
> will endup catching the "integration" issues, which are the issue as a result of
> simultenous changes in different trees. They are also often more complex and
> benefit from the bisection capabilities.
> 
> kernelCI tests are also a lot more intensive, they usually covers everything,
> but they bundle multiple changes per run. The pre-merge tests will be reduced to
> what seems meaningful for the changes. Its important to understand that pre-
> merge CI have a time cost, and we need to make sure CI time does not exceed the
> merge window period.

You're referring to the legacy KernelCI, to illustrate the point
I made earlier.  The plan with the new implementation was to be
able to do pre-merge testing as well as many other things,
basically to provide a platform able to cope with the diversity
of workflows across the kernel subsystems and the complexity of
the "system under test" itself.


Well, let's see how this goes and it does look quite promising.
Evolution is always a chaotic process, especially in a complex
project like this.  I'm not expecting to get all the answers to
the questions I have but it seemed important to raise this point
and seek a bit more clarity around KernelCI.

Guillaume
Gustavo Padovan March 2, 2024, 9:48 p.m. UTC | #10
On Friday, March 01, 2024 18:56 -03, Guillaume Tucker <gtucker@gtucker.io> wrote:

> On 29/02/2024 17:28, Nicolas Dufresne wrote:
> > Hi,
> > 
> > Le jeudi 29 février 2024 à 16:16 +0200, Nikolai Kondrashov a écrit :
> >> On 2/29/24 2:20 PM, Guillaume Tucker wrote:
> >>> Hello,
> >>>
> >>> On 28/02/2024 23:55, Helen Koike wrote:
> >>>> Dear Kernel Community,
> >>>>
> >>>> This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining a
> >>>> basic test pipeline triggered by code pushes to a GitLab-CI instance. This
> >>>> initial version includes static checks (checkpatch and smatch for now) and build
> >>>> tests across various architectures and configurations. It leverages an
> >>>> integrated cache for efficient build times and introduces a flexible 'scenarios'
> >>>> mechanism for subsystem-specific extensions.
> >>>
> >>> This sounds like a nice starting point to me as an additional way
> >>> to run tests upstream.  I have one particular question as I see a
> >>> pattern through the rest of the email, please see below.
> >>>
> >>> [...]
> >>>
> >>>> 4. **Collaborative Testing Environment:** The kernel community is already
> >>>> engaged in numerous testing efforts, including various GitLab-CI pipelines such
> >>>> as DRM-CI, which I maintain, along with other solutions like KernelCI and
> >>>> BPF-CI. This proposal is designed to further stimulate contributions to the
> >>>> evolving testing landscape. Our goal is to establish a comprehensive suite of
> >>>> common tools and files.
> >>>
> >>> [...]
> >>>
> >>>> **Leveraging External Test Labs:**
> >>>> We can extend our testing to external labs, similar to what DRM-CI currently
> >>>> does. This includes:
> >>>> - Lava labs
> >>>> - Bare metal labs
> >>>> - Using KernelCI-provided labs
> >>>>
> >>>> **Other integrations**
> >>>> - Submit results to KCIDB
> >>>
> >>> [...]
> >>>
> >>>> **Join Our Slack Channel:**
> >>>> We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ .
> >>>> Feel free to join and contribute to the conversation. The KernelCI team has
> >>>> weekly calls where we also discuss the GitLab-CI pipeline.
> >>>>
> >>>> **Acknowledgments:**
> >>>> A special thanks to Nikolai Kondrashov, Tales da Aparecida - both from Red Hat -
> >>>> and KernelCI community for their valuable feedback and support in this proposal.
> >>>
> >>> Where does this fit on the KernelCI roadmap?
> >>>
> >>> I see it mentioned a few times but it's not entirely clear
> >>> whether this initiative is an independent one or in some way
> >>> linked to KernelCI.  Say, are you planning to use the kci tool,
> >>> new API, compiler toolchains, user-space and Docker images etc?
> >>> Or, are KernelCI plans evolving to follow this move?
> >>
> >> I would say this is an important part of KernelCI the project, considering its 
> >> aim to improve testing and CI in the kernel. It's not a part of KernelCI the 
> >> service as it is right now, although I would say it would be good to have 
> >> ability to submit KernelCI jobs from GitLab CI and pull results in the same 
> >> pipeline, as we discussed earlier.
> 
> Right, I think this needs a bit of disambiguation.  The legacy
> KernelCI system from the Linaro days several years ago is really
> a service on its own like the many other CIs out there.  However,
> the new KernelCI API and related tooling (kci command line, new
> web dashboard, modular runtime design etc.) is not that.  It's
> about addressing all the community requirements and that includes
> being able to run a same test manually in a shell, or in a VM, or
> automatically from GitLab CI or using a main generic pipeline
> hosted by KernelCI itself.  With this approach, there's no
> distinction between "the project" and "the service", and as we
> discussed before there shouldn't even be a distinction with
> KCIDB.  Just KernelCI.
> 
> However I don't really see this happening, unless I'm missing a
> part of the story or some upcoming announcement with an updated
> roadmap.  For some reason the old and established paradigm seems
> unshakeable.  The new KernelCI implementation is starting to look
> just like a refresh of the old one with newer components - which
> is a huge missed opportunity to really change things IMHO.

Calling that a missed opportunity is a subjective perspective about
the latest developments in KernelCI. The system implementation is
one level less important than the actual kernel community engagement
the project can generate. If one asks people around, the lack of
community engagement with KernelCI is evident.

However, after the recent leadership change in the project there is a
growing effort to bring the kernel community closer to the KernelCI
project with a renewed focus on high quality test results, clean regression
reporting, among other things. Then, with an increased number of community
members involved, we will have the necessary feedback (and funding!) to
evolve the KernelCI infrastructure and technology to new levels.

Otherwise, envisioning something that can solve *all* community testing
needs would never be anything more than a fantasy in people's heads.

- Gus
Jarkko Sakkinen May 21, 2024, 9:28 a.m. UTC | #11
On Thu Feb 29, 2024 at 12:55 AM EET, Helen Koike wrote:
> Dear Kernel Community,
>
> This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining a
> basic test pipeline triggered by code pushes to a GitLab-CI instance. This
> initial version includes static checks (checkpatch and smatch for now) and build
> tests across various architectures and configurations. It leverages an
> integrated cache for efficient build times and introduces a flexible 'scenarios'
> mechanism for subsystem-specific extensions.
>
> tl;dr: check this video to see a quick demo: https://youtu.be/TWiTjhjOuzg,
> but don't forget to check the "Motivation for this work" below. Your feedback,
> whether a simple thumbs up or down, is crucial to determine if it is worthwhile
> to pursue this initiative.
>
> GitLab is an Open Source platform that includes integrated CI/CD. The pipeline
> provided in this patch is designed to work out-of-the-box with any GitLab
> instance, including the gitlab.com Free Tier. If you reach the limits of the
> Free Tier, consider using community instances like https://gitlab.freedesktop.org/.
> Alternatively, you can set up a local runner for more flexibility. The
> bootstrap-gitlab-runner.sh script included with this patch simplifies this
> process, enabling you to run tests on your preferred infrastructure, including
> your own machine.
>
> For detailed information, please refer to the documentation included in the
> patch, or check the rendered version here: https://koike.pages.collabora.com/-/linux/-/jobs/298498/artifacts/artifacts/Documentation-output/ci/gitlab-ci/gitlab-ci.html .
>
>
> Motivation for this Work
> ========================
>
> We all know tests are a major topic in the community, so let's mention the
> specificities of this approach:
>
> 1. **Built-in User Interface:** GitLab CI/CD is growing in popularity and has an
> user-friendly interface. Our experience with the upstream DRM-CI in the kernel
> tree (see this blog post [https://www.collabora.com/news-and-blog/blog/2024/02/08/drm-ci-a-gitlab-ci-pipeline-for-linux-kernel-testing/] )
> has provided insights into how such a system can benefit the wider community.
>
> 2. **Distributed Infrastructure:**
> The proposed GitLab-CI pipeline is designed with a distributed infrastructure
> model, being possible to run in any gitlab instance. 
>
> 3. **Reduce regressions:** Fostering a culture where people habitually run
> validated tests and post their results can prevent many issues in post-merge
> tests.
>
> 4. **Collaborative Testing Environment:** The kernel community is already
> engaged in numerous testing efforts, including various GitLab-CI pipelines such
> as DRM-CI, which I maintain, along with other solutions like KernelCI and
> BPF-CI. This proposal is designed to further stimulate contributions to the
> evolving testing landscape. Our goal is to establish a comprehensive suite of
> common tools and files.
>
> 5. **Ownership of QA:** 
> Discrepancies between kernel code and outdated tests often lead to misattributed
> failures, complicating regression tracking. This issue, often arising from
> neglected or deprioritized test updates, creates uncertainty about the source of
> failures. Adopting an "always green pipeline" approach, as detailed in this
> patch's documentation, encourages timely maintenance and validation of tests.
> This ensures that testing accurately reflects the current state of the kernel,
> thereby improving the effectiveness of our QA processes.
>
> Additionally, if we discover that this method isn't working for us, we can
> easily remove it from the codebase, as it is primarily contained within the ci/
> folder.

Not to criticize but I can do  tests I ever want with either Github
or Gitlab simply by bootstrapping BuildRoot on top of whatever the CI
runner has. So I essentially need just enough deps to make a BR build,
and that's it. And e.g. could run x86 tests on ARM ISA runner with zero
issues. And can even have emulated TPM chip in the QEMU VM by building
swtpm.

I had this for some time running actually Gitlab runner. It does not
currently build QEMU but then it also did that:

https://gitlab.com/jarkkojs/linux-tpmdd-test

Essentially just executing this sequence:

git clone https://gitlab.com/jarkkojs/linux-tpmdd-test.git
cd linux-tpmdd-test
cmake -Bbuild && make -Cbuild buildroot-prepare
make -Cbuild/buildroot/build
build/buildroot/build/images/run-tests.sh

I use TCL's "expect" to make conclusions from the output :-)

I'm assuming that this has a bigger point that I can understand right
now but makes me a bit puzzled given that it is quite trivial problem
to my understanding (if you want to pursue to it). Like one work 
week maybe but not more than that...

Especially it feels weird that it needs kernel to be patched at all and
when I did read the motivation but it has sort of whitepaperish stuff
that does not really explain me the edge of this compared to e.g. to my
ad-hoc but still very usable solution (which is agnostic to runner's CPU
architecture, can emulated hardware and works in any possible hosting
with CI).

So maybe my review comment it this: do not assume that this would be
entirely new thing. It is not, and I'm sure some other people have
done this too in the past rather than just me. Instead this should
explain why this is so great that even kernel tree needs to be patched?

BR, Jarkko