| Message ID | 20240228225527.1052240-1-helen.koike@collabora.com |
|---|---|
| Headers | show |
| Series | kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing | expand |
Hi Helen, I appreciate the amount of work you've put into this, to improve testing of the kernel as a whole. I'll need more time to answer, but please see below for a quick comment already. On Wed, Feb 28, 2024 at 07:55:24PM -0300, Helen Koike wrote: > Dear Kernel Community, > > This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining a > basic test pipeline triggered by code pushes to a GitLab-CI instance. This > initial version includes static checks (checkpatch and smatch for now) and build > tests across various architectures and configurations. It leverages an > integrated cache for efficient build times and introduces a flexible 'scenarios' > mechanism for subsystem-specific extensions. > > tl;dr: check this video to see a quick demo: https://youtu.be/TWiTjhjOuzg, > but don't forget to check the "Motivation for this work" below. Your feedback, > whether a simple thumbs up or down, is crucial to determine if it is worthwhile > to pursue this initiative. > > GitLab is an Open Source platform that includes integrated CI/CD. The pipeline > provided in this patch is designed to work out-of-the-box with any GitLab > instance, including the gitlab.com Free Tier. If you reach the limits of the > Free Tier, consider using community instances like https://gitlab.freedesktop.org/. > Alternatively, you can set up a local runner for more flexibility. The > bootstrap-gitlab-runner.sh script included with this patch simplifies this > process, enabling you to run tests on your preferred infrastructure, including > your own machine. > > For detailed information, please refer to the documentation included in the > patch, or check the rendered version here: https://koike.pages.collabora.com/-/linux/-/jobs/298498/artifacts/artifacts/Documentation-output/ci/gitlab-ci/gitlab-ci.html . > > > Motivation for this Work > ======================== > > We all know tests are a major topic in the community, so let's mention the > specificities of this approach: > > 1. **Built-in User Interface:** GitLab CI/CD is growing in popularity and has an > user-friendly interface. Our experience with the upstream DRM-CI in the kernel > tree (see this blog post [https://www.collabora.com/news-and-blog/blog/2024/02/08/drm-ci-a-gitlab-ci-pipeline-for-linux-kernel-testing/] ) > has provided insights into how such a system can benefit the wider community. > > 2. **Distributed Infrastructure:** > The proposed GitLab-CI pipeline is designed with a distributed infrastructure > model, being possible to run in any gitlab instance. > > 3. **Reduce regressions:** Fostering a culture where people habitually run > validated tests and post their results can prevent many issues in post-merge > tests. > > 4. **Collaborative Testing Environment:** The kernel community is already > engaged in numerous testing efforts, including various GitLab-CI pipelines such > as DRM-CI, which I maintain, along with other solutions like KernelCI and > BPF-CI. This proposal is designed to further stimulate contributions to the > evolving testing landscape. Our goal is to establish a comprehensive suite of > common tools and files. > > 5. **Ownership of QA:** > Discrepancies between kernel code and outdated tests often lead to misattributed > failures, complicating regression tracking. This issue, often arising from > neglected or deprioritized test updates, creates uncertainty about the source of > failures. Adopting an "always green pipeline" approach, as detailed in this > patch's documentation, encourages timely maintenance and validation of tests. > This ensures that testing accurately reflects the current state of the kernel, > thereby improving the effectiveness of our QA processes. > > Additionally, if we discover that this method isn't working for us, we can > easily remove it from the codebase, as it is primarily contained within the ci/ > folder. > > > Future Work > =========== > > **Expanding Static Checks:** > We have the opportunity to integrate a variety of static analysis tools, > including: > - dtbs_checks > - sparse > - yamllint > - dt-doc-validate > - coccicheck > > **Adding Userspace Tests on VMs:** > To further our testing, we can implement userspace tests that run on virtual > machines (VMs), such as: > - kselftests > - kunit tests > - Subsystem-specific tests, customizable in the scenarios. > > **Leveraging External Test Labs:** > We can extend our testing to external labs, similar to what DRM-CI currently > does. This includes: > - Lava labs > - Bare metal labs > - Using KernelCI-provided labs > > **Other integrations** > - Submit results to KCIDB > > **Lightweight Implementation for All Developers:** > We aim to design these tests to be lightweight, ensuring developers with limited > computing resources can still run essential tests. Resource-intensive tests can > be set to trigger manually, rather than automatically, to accommodate diverse > development environments. > > > Chat Discussions > ================ > > For those interested in further discussions: > > **Join Our Slack Channel:** > We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . > Feel free to join and contribute to the conversation. The KernelCI team has > weekly calls where we also discuss the GitLab-CI pipeline. Could we communicate using free software please ? Furthermore, it's not possible to create an account on that slack instance unless you have an e-mail address affiliated with a small number of companies (https://kernelci.slack.com/signup#/domain-signup). That's a big no-go for me. > **Acknowledgments:** > A special thanks to Nikolai Kondrashov, Tales da Aparecida - both from Red Hat - > and KernelCI community for their valuable feedback and support in this proposal. > > > I eagerly await your thoughts and suggestions on this initiative. > > Also, if you want to see this initiave move faster, we are happy to discuss > funding options. > > Best regards, > Helen Koike > > Helen Koike (3): > kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing > kci-gitlab: Add documentation > kci-gitlab: docs: Add images > > .gitlab-ci.yml | 2 + > Documentation/ci/gitlab-ci/gitlab-ci.rst | 404 ++++++++++++++++++ > .../ci/gitlab-ci/images/job-matrix.png | Bin 0 -> 159752 bytes > .../gitlab-ci/images/new-project-runner.png | Bin 0 -> 607737 bytes > .../ci/gitlab-ci/images/pipelines-on-push.png | Bin 0 -> 532143 bytes > .../ci/gitlab-ci/images/the-pipeline.png | Bin 0 -> 91675 bytes > .../ci/gitlab-ci/images/variables.png | Bin 0 -> 277518 bytes > Documentation/index.rst | 7 + > MAINTAINERS | 9 + > ci/gitlab-ci/bootstrap-gitlab-runner.sh | 55 +++ > ci/gitlab-ci/ci-scripts/build-docs.sh | 35 ++ > ci/gitlab-ci/ci-scripts/build-kernel.sh | 35 ++ > ci/gitlab-ci/ci-scripts/ici-functions.sh | 104 +++++ > ci/gitlab-ci/ci-scripts/install-smatch.sh | 13 + > .../ci-scripts/parse_commit_message.sh | 27 ++ > ci/gitlab-ci/ci-scripts/run-checkpatch.sh | 19 + > ci/gitlab-ci/ci-scripts/run-smatch.sh | 45 ++ > ci/gitlab-ci/docker-compose.yaml | 18 + > ci/gitlab-ci/linux.code-workspace | 11 + > ci/gitlab-ci/yml/build.yml | 43 ++ > ci/gitlab-ci/yml/cache.yml | 26 ++ > ci/gitlab-ci/yml/container.yml | 36 ++ > ci/gitlab-ci/yml/gitlab-ci.yml | 71 +++ > ci/gitlab-ci/yml/kernel-combinations.yml | 18 + > ci/gitlab-ci/yml/scenarios.yml | 12 + > ci/gitlab-ci/yml/scenarios/file-systems.yml | 21 + > ci/gitlab-ci/yml/scenarios/media.yml | 21 + > ci/gitlab-ci/yml/scenarios/network.yml | 21 + > ci/gitlab-ci/yml/static-checks.yml | 21 + > 29 files changed, 1074 insertions(+) > create mode 100644 .gitlab-ci.yml > create mode 100644 Documentation/ci/gitlab-ci/gitlab-ci.rst > create mode 100644 Documentation/ci/gitlab-ci/images/job-matrix.png > create mode 100644 Documentation/ci/gitlab-ci/images/new-project-runner.png > create mode 100644 Documentation/ci/gitlab-ci/images/pipelines-on-push.png > create mode 100644 Documentation/ci/gitlab-ci/images/the-pipeline.png > create mode 100644 Documentation/ci/gitlab-ci/images/variables.png > create mode 100755 ci/gitlab-ci/bootstrap-gitlab-runner.sh > create mode 100755 ci/gitlab-ci/ci-scripts/build-docs.sh > create mode 100755 ci/gitlab-ci/ci-scripts/build-kernel.sh > create mode 100644 ci/gitlab-ci/ci-scripts/ici-functions.sh > create mode 100755 ci/gitlab-ci/ci-scripts/install-smatch.sh > create mode 100755 ci/gitlab-ci/ci-scripts/parse_commit_message.sh > create mode 100755 ci/gitlab-ci/ci-scripts/run-checkpatch.sh > create mode 100755 ci/gitlab-ci/ci-scripts/run-smatch.sh > create mode 100644 ci/gitlab-ci/docker-compose.yaml > create mode 100644 ci/gitlab-ci/linux.code-workspace > create mode 100644 ci/gitlab-ci/yml/build.yml > create mode 100644 ci/gitlab-ci/yml/cache.yml > create mode 100644 ci/gitlab-ci/yml/container.yml > create mode 100644 ci/gitlab-ci/yml/gitlab-ci.yml > create mode 100644 ci/gitlab-ci/yml/kernel-combinations.yml > create mode 100644 ci/gitlab-ci/yml/scenarios.yml > create mode 100644 ci/gitlab-ci/yml/scenarios/file-systems.yml > create mode 100644 ci/gitlab-ci/yml/scenarios/media.yml > create mode 100644 ci/gitlab-ci/yml/scenarios/network.yml > create mode 100644 ci/gitlab-ci/yml/static-checks.yml
On Thu, Feb 29, 2024 at 01:07:25AM +0200, Laurent Pinchart wrote: > > Chat Discussions > > ================ > > > > For those interested in further discussions: > > > > **Join Our Slack Channel:** > > We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . > > Feel free to join and contribute to the conversation. The KernelCI team has > > weekly calls where we also discuss the GitLab-CI pipeline. > > Could we communicate using free software please ? Furthermore, it's not > possible to create an account on that slack instance unless you have an > e-mail address affiliated with a small number of companies > (https://kernelci.slack.com/signup#/domain-signup). That's a big no-go > for me. Yeah, and that list looks super restrictive and arbitrary. Like, microsoft is there but kernel.org isn't? I'm sure there's a reason, but we should at the very least open it to everyone. Maxime
On 2/29/24 01:07, Laurent Pinchart wrote: > On Wed, Feb 28, 2024 at 07:55:24PM -0300, Helen Koike wrote: >> **Join Our Slack Channel:** >> We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . >> Feel free to join and contribute to the conversation. The KernelCI team has >> weekly calls where we also discuss the GitLab-CI pipeline. > > Could we communicate using free software please ? Furthermore, it's not > possible to create an account on that slack instance unless you have an > e-mail address affiliated with a small number of companies > (https://kernelci.slack.com/signup#/domain-signup). That's a big no-go > for me. Yes, it's not ideal that we use closed-source software for communication, but FWIW I'd be happy to invite you there. Perhaps if you try logging in e.g. with a Google account, I'd be able to see and approve your attempt too. Otherwise, our video meetings are generally open to everyone and run in Jitsi. Nick
On Thu, Feb 29, 2024 at 09:39:01AM +0000, Sakari Ailus wrote: > On Thu, Feb 29, 2024 at 01:07:25AM +0200, Laurent Pinchart wrote: > > > We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . > > > Feel free to join and contribute to the conversation. The KernelCI team has > > > weekly calls where we also discuss the GitLab-CI pipeline. > > Could we communicate using free software please ? Furthermore, it's not > > possible to create an account on that slack instance unless you have an > > e-mail address affiliated with a small number of companies > > (https://kernelci.slack.com/signup#/domain-signup). That's a big no-go > > for me. > I agree. There is no shortage of good alternatives either: we have IRC, > Matrix and XMPP for instance. Just pick one of them. And indeed KernelCI does actually already have #kernelci on libera.chat.
On Thu, Feb 29, 2024 at 01:10:16PM +0200, Nikolai Kondrashov wrote: > On 2/29/24 11:34 AM, Laurent Pinchart wrote: > > On Thu, Feb 29, 2024 at 11:26:51AM +0200, Nikolai Kondrashov wrote: > >> On 2/29/24 01:07, Laurent Pinchart wrote: > >>> On Wed, Feb 28, 2024 at 07:55:24PM -0300, Helen Koike wrote: > >>>> **Join Our Slack Channel:** > >>>> We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . > >>>> Feel free to join and contribute to the conversation. The KernelCI team has > >>>> weekly calls where we also discuss the GitLab-CI pipeline. > >>> > >>> Could we communicate using free software please ? Furthermore, it's not > >>> possible to create an account on that slack instance unless you have an > >>> e-mail address affiliated with a small number of companies > >>> (https://kernelci.slack.com/signup#/domain-signup). That's a big no-go > >>> for me. > >> > >> Yes, it's not ideal that we use closed-source software for communication, but > >> FWIW I'd be happy to invite you there. Perhaps if you try logging in e.g. with > >> a Google account, I'd be able to see and approve your attempt too. > > > > I don't use Google accounts to authenticate to third-party services, > > sorry. And in any case, that won't make slack free software. > > Of course. You're also welcome to join the #kernelci channel on libera.chat. Isn't that a bit pointless if it's no the main IM channel ? > It's much quieter there, though.
On Thu, Feb 29, 2024 at 01:19:19PM +0200, Laurent Pinchart wrote: > On Thu, Feb 29, 2024 at 01:10:16PM +0200, Nikolai Kondrashov wrote: > > Of course. You're also welcome to join the #kernelci channel on libera.chat. > Isn't that a bit pointless if it's no the main IM channel ? It *was* the original channel and still gets some usage (mostly started by me admittedly since I've never joined slack for a bunch of reasons that make it hassle), IIRC the Slack was started because there were some interns who had trouble figuring out IRC and intermittent connectivity but people seem to have migrated.
Hello, On 28/02/2024 23:55, Helen Koike wrote: > Dear Kernel Community, > > This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining a > basic test pipeline triggered by code pushes to a GitLab-CI instance. This > initial version includes static checks (checkpatch and smatch for now) and build > tests across various architectures and configurations. It leverages an > integrated cache for efficient build times and introduces a flexible 'scenarios' > mechanism for subsystem-specific extensions. This sounds like a nice starting point to me as an additional way to run tests upstream. I have one particular question as I see a pattern through the rest of the email, please see below. [...] > 4. **Collaborative Testing Environment:** The kernel community is already > engaged in numerous testing efforts, including various GitLab-CI pipelines such > as DRM-CI, which I maintain, along with other solutions like KernelCI and > BPF-CI. This proposal is designed to further stimulate contributions to the > evolving testing landscape. Our goal is to establish a comprehensive suite of > common tools and files. [...] > **Leveraging External Test Labs:** > We can extend our testing to external labs, similar to what DRM-CI currently > does. This includes: > - Lava labs > - Bare metal labs > - Using KernelCI-provided labs > > **Other integrations** > - Submit results to KCIDB [...] > **Join Our Slack Channel:** > We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . > Feel free to join and contribute to the conversation. The KernelCI team has > weekly calls where we also discuss the GitLab-CI pipeline. > > **Acknowledgments:** > A special thanks to Nikolai Kondrashov, Tales da Aparecida - both from Red Hat - > and KernelCI community for their valuable feedback and support in this proposal. Where does this fit on the KernelCI roadmap? I see it mentioned a few times but it's not entirely clear whether this initiative is an independent one or in some way linked to KernelCI. Say, are you planning to use the kci tool, new API, compiler toolchains, user-space and Docker images etc? Or, are KernelCI plans evolving to follow this move? Thanks, Guillaume
On 2/29/24 2:20 PM, Guillaume Tucker wrote: > Hello, > > On 28/02/2024 23:55, Helen Koike wrote: >> Dear Kernel Community, >> >> This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining a >> basic test pipeline triggered by code pushes to a GitLab-CI instance. This >> initial version includes static checks (checkpatch and smatch for now) and build >> tests across various architectures and configurations. It leverages an >> integrated cache for efficient build times and introduces a flexible 'scenarios' >> mechanism for subsystem-specific extensions. > > This sounds like a nice starting point to me as an additional way > to run tests upstream. I have one particular question as I see a > pattern through the rest of the email, please see below. > > [...] > >> 4. **Collaborative Testing Environment:** The kernel community is already >> engaged in numerous testing efforts, including various GitLab-CI pipelines such >> as DRM-CI, which I maintain, along with other solutions like KernelCI and >> BPF-CI. This proposal is designed to further stimulate contributions to the >> evolving testing landscape. Our goal is to establish a comprehensive suite of >> common tools and files. > > [...] > >> **Leveraging External Test Labs:** >> We can extend our testing to external labs, similar to what DRM-CI currently >> does. This includes: >> - Lava labs >> - Bare metal labs >> - Using KernelCI-provided labs >> >> **Other integrations** >> - Submit results to KCIDB > > [...] > >> **Join Our Slack Channel:** >> We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . >> Feel free to join and contribute to the conversation. The KernelCI team has >> weekly calls where we also discuss the GitLab-CI pipeline. >> >> **Acknowledgments:** >> A special thanks to Nikolai Kondrashov, Tales da Aparecida - both from Red Hat - >> and KernelCI community for their valuable feedback and support in this proposal. > > Where does this fit on the KernelCI roadmap? > > I see it mentioned a few times but it's not entirely clear > whether this initiative is an independent one or in some way > linked to KernelCI. Say, are you planning to use the kci tool, > new API, compiler toolchains, user-space and Docker images etc? > Or, are KernelCI plans evolving to follow this move? I would say this is an important part of KernelCI the project, considering its aim to improve testing and CI in the kernel. It's not a part of KernelCI the service as it is right now, although I would say it would be good to have ability to submit KernelCI jobs from GitLab CI and pull results in the same pipeline, as we discussed earlier. Nick
On 29/02/2024 17:28, Nicolas Dufresne wrote: > Hi, > > Le jeudi 29 février 2024 à 16:16 +0200, Nikolai Kondrashov a écrit : >> On 2/29/24 2:20 PM, Guillaume Tucker wrote: >>> Hello, >>> >>> On 28/02/2024 23:55, Helen Koike wrote: >>>> Dear Kernel Community, >>>> >>>> This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining a >>>> basic test pipeline triggered by code pushes to a GitLab-CI instance. This >>>> initial version includes static checks (checkpatch and smatch for now) and build >>>> tests across various architectures and configurations. It leverages an >>>> integrated cache for efficient build times and introduces a flexible 'scenarios' >>>> mechanism for subsystem-specific extensions. >>> >>> This sounds like a nice starting point to me as an additional way >>> to run tests upstream. I have one particular question as I see a >>> pattern through the rest of the email, please see below. >>> >>> [...] >>> >>>> 4. **Collaborative Testing Environment:** The kernel community is already >>>> engaged in numerous testing efforts, including various GitLab-CI pipelines such >>>> as DRM-CI, which I maintain, along with other solutions like KernelCI and >>>> BPF-CI. This proposal is designed to further stimulate contributions to the >>>> evolving testing landscape. Our goal is to establish a comprehensive suite of >>>> common tools and files. >>> >>> [...] >>> >>>> **Leveraging External Test Labs:** >>>> We can extend our testing to external labs, similar to what DRM-CI currently >>>> does. This includes: >>>> - Lava labs >>>> - Bare metal labs >>>> - Using KernelCI-provided labs >>>> >>>> **Other integrations** >>>> - Submit results to KCIDB >>> >>> [...] >>> >>>> **Join Our Slack Channel:** >>>> We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . >>>> Feel free to join and contribute to the conversation. The KernelCI team has >>>> weekly calls where we also discuss the GitLab-CI pipeline. >>>> >>>> **Acknowledgments:** >>>> A special thanks to Nikolai Kondrashov, Tales da Aparecida - both from Red Hat - >>>> and KernelCI community for their valuable feedback and support in this proposal. >>> >>> Where does this fit on the KernelCI roadmap? >>> >>> I see it mentioned a few times but it's not entirely clear >>> whether this initiative is an independent one or in some way >>> linked to KernelCI. Say, are you planning to use the kci tool, >>> new API, compiler toolchains, user-space and Docker images etc? >>> Or, are KernelCI plans evolving to follow this move? >> >> I would say this is an important part of KernelCI the project, considering its >> aim to improve testing and CI in the kernel. It's not a part of KernelCI the >> service as it is right now, although I would say it would be good to have >> ability to submit KernelCI jobs from GitLab CI and pull results in the same >> pipeline, as we discussed earlier. Right, I think this needs a bit of disambiguation. The legacy KernelCI system from the Linaro days several years ago is really a service on its own like the many other CIs out there. However, the new KernelCI API and related tooling (kci command line, new web dashboard, modular runtime design etc.) is not that. It's about addressing all the community requirements and that includes being able to run a same test manually in a shell, or in a VM, or automatically from GitLab CI or using a main generic pipeline hosted by KernelCI itself. With this approach, there's no distinction between "the project" and "the service", and as we discussed before there shouldn't even be a distinction with KCIDB. Just KernelCI. However I don't really see this happening, unless I'm missing a part of the story or some upcoming announcement with an updated roadmap. For some reason the old and established paradigm seems unshakeable. The new KernelCI implementation is starting to look just like a refresh of the old one with newer components - which is a huge missed opportunity to really change things IMHO. This may sound like a bit of a tangent, facilitating GitLab CI for the upstream kernel is of course significant progress in any case - no question about that. My comment is more about why it's being driven hand-in-hand with KernelCI in what seems like a diverging direction from KernelCI's announced plans. Why push for a GitLab-centered orchestration when there's a more universal solution being proposed by the project? I would find it easier to understand - and I sense I'm not the only one here reading the thread - if KernelCI wasn't mentioned that many times in the cover letter and if the scripts didn't have KCI_* in so many places, basically if this was clearly an independent initiative such as KUnit, 0-day or regzbot. > I'd like to add that both CI have a different purpose in the Linux project. This > CI work is a pre-merge verification. Everyone needs to run checkpatch and > smatch, this is automating it (and will catch those that forgot or ran it > incorrectly). But it can go further by effectively testing specific patches on > real hardware (with pretty narrow filters). It will help catch submission issues > earlier, and reduce kernelCI regression rate. As a side effect, kernelCI infra > will endup catching the "integration" issues, which are the issue as a result of > simultenous changes in different trees. They are also often more complex and > benefit from the bisection capabilities. > > kernelCI tests are also a lot more intensive, they usually covers everything, > but they bundle multiple changes per run. The pre-merge tests will be reduced to > what seems meaningful for the changes. Its important to understand that pre- > merge CI have a time cost, and we need to make sure CI time does not exceed the > merge window period. You're referring to the legacy KernelCI, to illustrate the point I made earlier. The plan with the new implementation was to be able to do pre-merge testing as well as many other things, basically to provide a platform able to cope with the diversity of workflows across the kernel subsystems and the complexity of the "system under test" itself. Well, let's see how this goes and it does look quite promising. Evolution is always a chaotic process, especially in a complex project like this. I'm not expecting to get all the answers to the questions I have but it seemed important to raise this point and seek a bit more clarity around KernelCI. Guillaume
On Friday, March 01, 2024 18:56 -03, Guillaume Tucker <gtucker@gtucker.io> wrote: > On 29/02/2024 17:28, Nicolas Dufresne wrote: > > Hi, > > > > Le jeudi 29 février 2024 à 16:16 +0200, Nikolai Kondrashov a écrit : > >> On 2/29/24 2:20 PM, Guillaume Tucker wrote: > >>> Hello, > >>> > >>> On 28/02/2024 23:55, Helen Koike wrote: > >>>> Dear Kernel Community, > >>>> > >>>> This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining a > >>>> basic test pipeline triggered by code pushes to a GitLab-CI instance. This > >>>> initial version includes static checks (checkpatch and smatch for now) and build > >>>> tests across various architectures and configurations. It leverages an > >>>> integrated cache for efficient build times and introduces a flexible 'scenarios' > >>>> mechanism for subsystem-specific extensions. > >>> > >>> This sounds like a nice starting point to me as an additional way > >>> to run tests upstream. I have one particular question as I see a > >>> pattern through the rest of the email, please see below. > >>> > >>> [...] > >>> > >>>> 4. **Collaborative Testing Environment:** The kernel community is already > >>>> engaged in numerous testing efforts, including various GitLab-CI pipelines such > >>>> as DRM-CI, which I maintain, along with other solutions like KernelCI and > >>>> BPF-CI. This proposal is designed to further stimulate contributions to the > >>>> evolving testing landscape. Our goal is to establish a comprehensive suite of > >>>> common tools and files. > >>> > >>> [...] > >>> > >>>> **Leveraging External Test Labs:** > >>>> We can extend our testing to external labs, similar to what DRM-CI currently > >>>> does. This includes: > >>>> - Lava labs > >>>> - Bare metal labs > >>>> - Using KernelCI-provided labs > >>>> > >>>> **Other integrations** > >>>> - Submit results to KCIDB > >>> > >>> [...] > >>> > >>>> **Join Our Slack Channel:** > >>>> We have a Slack channel, #gitlab-ci, on the KernelCI Slack instance https://kernelci.slack.com/ . > >>>> Feel free to join and contribute to the conversation. The KernelCI team has > >>>> weekly calls where we also discuss the GitLab-CI pipeline. > >>>> > >>>> **Acknowledgments:** > >>>> A special thanks to Nikolai Kondrashov, Tales da Aparecida - both from Red Hat - > >>>> and KernelCI community for their valuable feedback and support in this proposal. > >>> > >>> Where does this fit on the KernelCI roadmap? > >>> > >>> I see it mentioned a few times but it's not entirely clear > >>> whether this initiative is an independent one or in some way > >>> linked to KernelCI. Say, are you planning to use the kci tool, > >>> new API, compiler toolchains, user-space and Docker images etc? > >>> Or, are KernelCI plans evolving to follow this move? > >> > >> I would say this is an important part of KernelCI the project, considering its > >> aim to improve testing and CI in the kernel. It's not a part of KernelCI the > >> service as it is right now, although I would say it would be good to have > >> ability to submit KernelCI jobs from GitLab CI and pull results in the same > >> pipeline, as we discussed earlier. > > Right, I think this needs a bit of disambiguation. The legacy > KernelCI system from the Linaro days several years ago is really > a service on its own like the many other CIs out there. However, > the new KernelCI API and related tooling (kci command line, new > web dashboard, modular runtime design etc.) is not that. It's > about addressing all the community requirements and that includes > being able to run a same test manually in a shell, or in a VM, or > automatically from GitLab CI or using a main generic pipeline > hosted by KernelCI itself. With this approach, there's no > distinction between "the project" and "the service", and as we > discussed before there shouldn't even be a distinction with > KCIDB. Just KernelCI. > > However I don't really see this happening, unless I'm missing a > part of the story or some upcoming announcement with an updated > roadmap. For some reason the old and established paradigm seems > unshakeable. The new KernelCI implementation is starting to look > just like a refresh of the old one with newer components - which > is a huge missed opportunity to really change things IMHO. Calling that a missed opportunity is a subjective perspective about the latest developments in KernelCI. The system implementation is one level less important than the actual kernel community engagement the project can generate. If one asks people around, the lack of community engagement with KernelCI is evident. However, after the recent leadership change in the project there is a growing effort to bring the kernel community closer to the KernelCI project with a renewed focus on high quality test results, clean regression reporting, among other things. Then, with an increased number of community members involved, we will have the necessary feedback (and funding!) to evolve the KernelCI infrastructure and technology to new levels. Otherwise, envisioning something that can solve *all* community testing needs would never be anything more than a fantasy in people's heads. - Gus
On Thu Feb 29, 2024 at 12:55 AM EET, Helen Koike wrote: > Dear Kernel Community, > > This patch introduces a `.gitlab-ci` file along with a `ci/` folder, defining a > basic test pipeline triggered by code pushes to a GitLab-CI instance. This > initial version includes static checks (checkpatch and smatch for now) and build > tests across various architectures and configurations. It leverages an > integrated cache for efficient build times and introduces a flexible 'scenarios' > mechanism for subsystem-specific extensions. > > tl;dr: check this video to see a quick demo: https://youtu.be/TWiTjhjOuzg, > but don't forget to check the "Motivation for this work" below. Your feedback, > whether a simple thumbs up or down, is crucial to determine if it is worthwhile > to pursue this initiative. > > GitLab is an Open Source platform that includes integrated CI/CD. The pipeline > provided in this patch is designed to work out-of-the-box with any GitLab > instance, including the gitlab.com Free Tier. If you reach the limits of the > Free Tier, consider using community instances like https://gitlab.freedesktop.org/. > Alternatively, you can set up a local runner for more flexibility. The > bootstrap-gitlab-runner.sh script included with this patch simplifies this > process, enabling you to run tests on your preferred infrastructure, including > your own machine. > > For detailed information, please refer to the documentation included in the > patch, or check the rendered version here: https://koike.pages.collabora.com/-/linux/-/jobs/298498/artifacts/artifacts/Documentation-output/ci/gitlab-ci/gitlab-ci.html . > > > Motivation for this Work > ======================== > > We all know tests are a major topic in the community, so let's mention the > specificities of this approach: > > 1. **Built-in User Interface:** GitLab CI/CD is growing in popularity and has an > user-friendly interface. Our experience with the upstream DRM-CI in the kernel > tree (see this blog post [https://www.collabora.com/news-and-blog/blog/2024/02/08/drm-ci-a-gitlab-ci-pipeline-for-linux-kernel-testing/] ) > has provided insights into how such a system can benefit the wider community. > > 2. **Distributed Infrastructure:** > The proposed GitLab-CI pipeline is designed with a distributed infrastructure > model, being possible to run in any gitlab instance. > > 3. **Reduce regressions:** Fostering a culture where people habitually run > validated tests and post their results can prevent many issues in post-merge > tests. > > 4. **Collaborative Testing Environment:** The kernel community is already > engaged in numerous testing efforts, including various GitLab-CI pipelines such > as DRM-CI, which I maintain, along with other solutions like KernelCI and > BPF-CI. This proposal is designed to further stimulate contributions to the > evolving testing landscape. Our goal is to establish a comprehensive suite of > common tools and files. > > 5. **Ownership of QA:** > Discrepancies between kernel code and outdated tests often lead to misattributed > failures, complicating regression tracking. This issue, often arising from > neglected or deprioritized test updates, creates uncertainty about the source of > failures. Adopting an "always green pipeline" approach, as detailed in this > patch's documentation, encourages timely maintenance and validation of tests. > This ensures that testing accurately reflects the current state of the kernel, > thereby improving the effectiveness of our QA processes. > > Additionally, if we discover that this method isn't working for us, we can > easily remove it from the codebase, as it is primarily contained within the ci/ > folder. Not to criticize but I can do tests I ever want with either Github or Gitlab simply by bootstrapping BuildRoot on top of whatever the CI runner has. So I essentially need just enough deps to make a BR build, and that's it. And e.g. could run x86 tests on ARM ISA runner with zero issues. And can even have emulated TPM chip in the QEMU VM by building swtpm. I had this for some time running actually Gitlab runner. It does not currently build QEMU but then it also did that: https://gitlab.com/jarkkojs/linux-tpmdd-test Essentially just executing this sequence: git clone https://gitlab.com/jarkkojs/linux-tpmdd-test.git cd linux-tpmdd-test cmake -Bbuild && make -Cbuild buildroot-prepare make -Cbuild/buildroot/build build/buildroot/build/images/run-tests.sh I use TCL's "expect" to make conclusions from the output :-) I'm assuming that this has a bigger point that I can understand right now but makes me a bit puzzled given that it is quite trivial problem to my understanding (if you want to pursue to it). Like one work week maybe but not more than that... Especially it feels weird that it needs kernel to be patched at all and when I did read the motivation but it has sort of whitepaperish stuff that does not really explain me the edge of this compared to e.g. to my ad-hoc but still very usable solution (which is agnostic to runner's CPU architecture, can emulated hardware and works in any possible hosting with CI). So maybe my review comment it this: do not assume that this would be entirely new thing. It is not, and I'm sure some other people have done this too in the past rather than just me. Instead this should explain why this is so great that even kernel tree needs to be patched? BR, Jarkko