[1/4] Acceptance tests: use an available kernel image package for arm

Which means a newer kernel version.  Expected output was changed
to match the new kernel too.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/acceptance/boot_linux_console.py | 44 +++++++++++++-------------
 tests/acceptance/replay_kernel.py      |  8 ++---
 2 files changed, 26 insertions(+), 26 deletions(-)

On Mon, Sep 07, 2020 at 10:06:13AM +0200, Philippe Mathieu-Daudé wrote:
> [Cc'ing Daniel who usually have good ideas for that
> kind if project-wide problem]
> 
> On 9/7/20 6:19 AM, Cleber Rosa wrote:
> > Which means a newer kernel version.  Expected output was changed
> > to match the new kernel too.
> 
> Nack.
> 
> Acceptance tests are not to test the latest Linux kernel,
> they aim to assert a specific kernel tested by some developer
> still works while QEMU evolves.
> QEMU doesn't have to adapt to the latest kernel;
> QEMU should keep boot an old kernel.
> 
> Testing new kernels is good, you are adding coverage. But
> this break the acceptance testing contract "keep testing
> the same thing over time".
> 
> The problem you are trying to fix is the "where to keep
> assets from public locations where they are being removed?"
> one. Two years ago [*] you suggested to use some storage on
> the avocado-project.org:
> 
>   For Avocado-VT, there are the JeOS images[1], which we
>   keep on a test "assets" directory.  We have a lot of
>   storage/bandwidth availability, so it can be used for
>   other assets proven to be necessary for tests.
> 
>   As long as distribution rights and licensing are not
>   issues, we can definitely use the same server for kernels,
>   u-boot images and what not.
> 
>   [1] - https://avocado-project.org/data/assets/

If I look at stuff under that directory I see a bunch of "Jeos" qcow2
images, and zero information about the corresponding source for the
images, nor any information about the licenses of software included.
IOW what is stored their right now does not appear to comply with the
GPL licensing requirements for providing full and corresponding source.

> It is time to have QEMU assets managed the same way.

I'd rather we didn't do anything relying on binary blobs with no
info about how they were built. Pointing to the 3rd party download
URLs was the easy way to ensure we don't have to worry about licensing
problems.

IIUC, the majority of our acceptance tests needs should be satisfied
with a combination of a Linux kernel and busybox tools. We already
use cross-compilers to build TCG test cases.

I'd like to see us provide a minimal "config" file for each Linux
kernel combo we need to test. We should be able to define a fairly
simple script that do the Linux build, busybox build and then puts
the combination of them into a disk image.  Something 100% automated
and reviewable.

Even a minimal linux build takes some time, so we would need to cache
the binary result in some suitable manner. At least this way we have
a clear record of what we're building & how and can thus satisfy the
license rules.

> 
> [*] https://www.mail-archive.com/avocado-devel@redhat.com/msg00860.html
> 
> > 
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >  tests/acceptance/boot_linux_console.py | 44 +++++++++++++-------------
> >  tests/acceptance/replay_kernel.py      |  8 ++---
> >  2 files changed, 26 insertions(+), 26 deletions(-)
> > 
> > diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> > index aaa781a581..751b47b8fd 100644
> > --- a/tests/acceptance/boot_linux_console.py
> > +++ b/tests/acceptance/boot_linux_console.py
> > @@ -490,12 +490,12 @@ class BootLinuxConsole(LinuxKernelTest):
> >          :avocado: tags=arch:arm
> >          :avocado: tags=machine:cubieboard
> >          """
> > -        deb_url = ('https://apt.armbian.com/pool/main/l/'
> > -                   'linux-4.20.7-sunxi/linux-image-dev-sunxi_5.75_armhf.deb')
> > -        deb_hash = '1334c29c44d984ffa05ed10de8c3361f33d78315'
> > +        deb_url = ('https://apt.armbian.com/pool/main/l/linux-5.8.0-sunxi/'
> > +                   'linux-image-dev-sunxi_20.08_armhf.deb')
> > +        deb_hash = 'ae553a9f7d43b18abfa8f3e64bf2d31878b9be89'
> >          deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
> >          kernel_path = self.extract_from_deb(deb_path,
> > -                                            '/boot/vmlinuz-4.20.7-sunxi')
> > +                                            '/boot/vmlinuz-5.8.0-sunxi')
> >          dtb_path = '/usr/lib/linux-image-dev-sunxi/sun4i-a10-cubieboard.dtb'
> >          dtb_path = self.extract_from_deb(deb_path, dtb_path)
> >          initrd_url = ('https://github.com/groeck/linux-build-test/raw/'
> > @@ -522,7 +522,7 @@ class BootLinuxConsole(LinuxKernelTest):
> >          exec_command_and_wait_for_pattern(self, 'cat /proc/cpuinfo',
> >                                                  'Allwinner sun4i/sun5i')
> >          exec_command_and_wait_for_pattern(self, 'cat /proc/iomem',
> > -                                                'system-control@1c00000')
> > +                                                '1c00000.system-control')
> >          # cubieboard's reboot is not functioning; omit reboot test.
> >  
> >      def test_arm_cubieboard_sata(self):
> > @@ -530,12 +530,12 @@ class BootLinuxConsole(LinuxKernelTest):
> >          :avocado: tags=arch:arm
> >          :avocado: tags=machine:cubieboard
> >          """
> > -        deb_url = ('https://apt.armbian.com/pool/main/l/'
> > -                   'linux-4.20.7-sunxi/linux-image-dev-sunxi_5.75_armhf.deb')
> > -        deb_hash = '1334c29c44d984ffa05ed10de8c3361f33d78315'
> > +        deb_url = ('https://apt.armbian.com/pool/main/l/linux-5.8.0-sunxi/'
> > +                   'linux-image-dev-sunxi_20.08_armhf.deb')
> > +        deb_hash = 'ae553a9f7d43b18abfa8f3e64bf2d31878b9be89'
> >          deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
> >          kernel_path = self.extract_from_deb(deb_path,
> > -                                            '/boot/vmlinuz-4.20.7-sunxi')
> > +                                            '/boot/vmlinuz-5.8.0-sunxi')
> >          dtb_path = '/usr/lib/linux-image-dev-sunxi/sun4i-a10-cubieboard.dtb'
> >          dtb_path = self.extract_from_deb(deb_path, dtb_path)
> >          rootfs_url = ('https://github.com/groeck/linux-build-test/raw/'
> > @@ -573,12 +573,12 @@ class BootLinuxConsole(LinuxKernelTest):
> >          :avocado: tags=arch:arm
> >          :avocado: tags=machine:orangepi-pc
> >          """
> > -        deb_url = ('https://apt.armbian.com/pool/main/l/'
> > -                   'linux-4.20.7-sunxi/linux-image-dev-sunxi_5.75_armhf.deb')
> > -        deb_hash = '1334c29c44d984ffa05ed10de8c3361f33d78315'
> > +        deb_url = ('https://apt.armbian.com/pool/main/l/linux-5.8.0-sunxi/'
> > +                   'linux-image-dev-sunxi_20.08_armhf.deb')
> > +        deb_hash = 'ae553a9f7d43b18abfa8f3e64bf2d31878b9be89'
> >          deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
> >          kernel_path = self.extract_from_deb(deb_path,
> > -                                            '/boot/vmlinuz-4.20.7-sunxi')
> > +                                            '/boot/vmlinuz-5.8.0-sunxi')
> >          dtb_path = '/usr/lib/linux-image-dev-sunxi/sun8i-h3-orangepi-pc.dtb'
> >          dtb_path = self.extract_from_deb(deb_path, dtb_path)
> >  
> > @@ -598,12 +598,12 @@ class BootLinuxConsole(LinuxKernelTest):
> >          :avocado: tags=arch:arm
> >          :avocado: tags=machine:orangepi-pc
> >          """
> > -        deb_url = ('https://apt.armbian.com/pool/main/l/'
> > -                   'linux-4.20.7-sunxi/linux-image-dev-sunxi_5.75_armhf.deb')
> > -        deb_hash = '1334c29c44d984ffa05ed10de8c3361f33d78315'
> > +        deb_url = ('https://apt.armbian.com/pool/main/l/linux-5.8.0-sunxi/'
> > +                   'linux-image-dev-sunxi_20.08_armhf.deb')
> > +        deb_hash = 'ae553a9f7d43b18abfa8f3e64bf2d31878b9be89'
> >          deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
> >          kernel_path = self.extract_from_deb(deb_path,
> > -                                            '/boot/vmlinuz-4.20.7-sunxi')
> > +                                            '/boot/vmlinuz-5.8.0-sunxi')
> >          dtb_path = '/usr/lib/linux-image-dev-sunxi/sun8i-h3-orangepi-pc.dtb'
> >          dtb_path = self.extract_from_deb(deb_path, dtb_path)
> >          initrd_url = ('https://github.com/groeck/linux-build-test/raw/'
> > @@ -629,7 +629,7 @@ class BootLinuxConsole(LinuxKernelTest):
> >          exec_command_and_wait_for_pattern(self, 'cat /proc/cpuinfo',
> >                                                  'Allwinner sun8i Family')
> >          exec_command_and_wait_for_pattern(self, 'cat /proc/iomem',
> > -                                                'system-control@1c00000')
> > +                                                '1c00000.system-control')
> >          exec_command_and_wait_for_pattern(self, 'reboot',
> >                                                  'reboot: Restarting system')
> >          # Wait for VM to shut down gracefully
> > @@ -641,12 +641,12 @@ class BootLinuxConsole(LinuxKernelTest):
> >          :avocado: tags=machine:orangepi-pc
> >          :avocado: tags=device:sd
> >          """
> > -        deb_url = ('https://apt.armbian.com/pool/main/l/'
> > -                   'linux-4.20.7-sunxi/linux-image-dev-sunxi_5.75_armhf.deb')
> > -        deb_hash = '1334c29c44d984ffa05ed10de8c3361f33d78315'
> > +        deb_url = ('https://apt.armbian.com/pool/main/l/linux-5.8.0-sunxi/'
> > +                   'linux-image-dev-sunxi_20.08_armhf.deb')
> > +        deb_hash = 'ae553a9f7d43b18abfa8f3e64bf2d31878b9be89'
> >          deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
> >          kernel_path = self.extract_from_deb(deb_path,
> > -                                            '/boot/vmlinuz-4.20.7-sunxi')
> > +                                            '/boot/vmlinuz-5.8.0-sunxi')
> >          dtb_path = '/usr/lib/linux-image-dev-sunxi/sun8i-h3-orangepi-pc.dtb'
> >          dtb_path = self.extract_from_deb(deb_path, dtb_path)
> >          rootfs_url = ('http://storage.kernelci.org/images/rootfs/buildroot/'
> > diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
> > index b79fc8daf8..80a460814f 100644
> > --- a/tests/acceptance/replay_kernel.py
> > +++ b/tests/acceptance/replay_kernel.py
> > @@ -132,12 +132,12 @@ class ReplayKernel(LinuxKernelTest):
> >          :avocado: tags=arch:arm
> >          :avocado: tags=machine:cubieboard
> >          """
> > -        deb_url = ('https://apt.armbian.com/pool/main/l/'
> > -                   'linux-4.20.7-sunxi/linux-image-dev-sunxi_5.75_armhf.deb')
> > -        deb_hash = '1334c29c44d984ffa05ed10de8c3361f33d78315'
> > +        deb_url = ('https://apt.armbian.com/pool/main/l/linux-5.8.0-sunxi/'
> > +                   'linux-image-dev-sunxi_20.08_armhf.deb')
> > +        deb_hash = 'ae553a9f7d43b18abfa8f3e64bf2d31878b9be89'
> >          deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
> >          kernel_path = self.extract_from_deb(deb_path,
> > -                                            '/boot/vmlinuz-4.20.7-sunxi')
> > +                                            '/boot/vmlinuz-5.8.0-sunxi')
> >          dtb_path = '/usr/lib/linux-image-dev-sunxi/sun4i-a10-cubieboard.dtb'
> >          dtb_path = self.extract_from_deb(deb_path, dtb_path)
> >          initrd_url = ('https://github.com/groeck/linux-build-test/raw/'
> > 
> 
> 

Regards,
Daniel

On Mon, Sep 07, 2020 at 11:59:18AM +0200, Philippe Mathieu-Daudé wrote:
> On 9/7/20 11:39 AM, Daniel P. Berrangé wrote:
> > On Mon, Sep 07, 2020 at 10:06:13AM +0200, Philippe Mathieu-Daudé wrote:
> >> [Cc'ing Daniel who usually have good ideas for that
> >> kind if project-wide problem]
> >>
> >> On 9/7/20 6:19 AM, Cleber Rosa wrote:
> >>> Which means a newer kernel version.  Expected output was changed
> >>> to match the new kernel too.
> >>
> >> Nack.
> >>
> >> Acceptance tests are not to test the latest Linux kernel,
> >> they aim to assert a specific kernel tested by some developer
> >> still works while QEMU evolves.
> >> QEMU doesn't have to adapt to the latest kernel;
> >> QEMU should keep boot an old kernel.
> >>
> >> Testing new kernels is good, you are adding coverage. But
> >> this break the acceptance testing contract "keep testing
> >> the same thing over time".
> >>
> >> The problem you are trying to fix is the "where to keep
> >> assets from public locations where they are being removed?"
> >> one. Two years ago [*] you suggested to use some storage on
> >> the avocado-project.org:
> >>
> >>   For Avocado-VT, there are the JeOS images[1], which we
> >>   keep on a test "assets" directory.  We have a lot of
> >>   storage/bandwidth availability, so it can be used for
> >>   other assets proven to be necessary for tests.
> >>
> >>   As long as distribution rights and licensing are not
> >>   issues, we can definitely use the same server for kernels,
> >>   u-boot images and what not.
> >>
> >>   [1] - https://avocado-project.org/data/assets/
> > 
> > If I look at stuff under that directory I see a bunch of "Jeos" qcow2
> > images, and zero information about the corresponding source for the
> > images, nor any information about the licenses of software included.
> > IOW what is stored their right now does not appear to comply with the
> > GPL licensing requirements for providing full and corresponding source.
> > 
> >> It is time to have QEMU assets managed the same way.
> > 
> > I'd rather we didn't do anything relying on binary blobs with no
> > info about how they were built. Pointing to the 3rd party download
> > URLs was the easy way to ensure we don't have to worry about licensing
> > problems.
> 
> I tried to be very strict including the recipe about how to rebuild
> and description of the source (for licensing) in each commits (Alex
> Bennée once said Debian/Fedora based was OK):

..snip...

Well that looks better than what is done for the JEOS images currently
on avocado-project.org, as I can't tell what distro those came from
at all.

If we're hosting images built by some 3rd party, and we intend to rely
on the 3rd party to satisfy source availability, then we need to be sure
that the 3rd party is themselves still distributing the same images.

IIUC, from Cleber's commit here the original images we're pointing to
are now 404s. If the URLs moved, we just need to update to fix the URLs
to point the new location. If the content was entirely removed though,
we shouldn't mirror it ourselves, because we can't rely on the original
vendor to be providing the source at that point.

Regards,
Daniel

On 9/7/20 12:28 PM, Daniel P. Berrangé wrote:
> On Mon, Sep 07, 2020 at 11:59:18AM +0200, Philippe Mathieu-Daudé wrote:
>> On 9/7/20 11:39 AM, Daniel P. Berrangé wrote:
>>> On Mon, Sep 07, 2020 at 10:06:13AM +0200, Philippe Mathieu-Daudé wrote:
>>>> [Cc'ing Daniel who usually have good ideas for that
>>>> kind if project-wide problem]
>>>>
>>>> On 9/7/20 6:19 AM, Cleber Rosa wrote:
>>>>> Which means a newer kernel version.  Expected output was changed
>>>>> to match the new kernel too.
>>>>
>>>> Nack.
>>>>
>>>> Acceptance tests are not to test the latest Linux kernel,
>>>> they aim to assert a specific kernel tested by some developer
>>>> still works while QEMU evolves.
>>>> QEMU doesn't have to adapt to the latest kernel;
>>>> QEMU should keep boot an old kernel.
>>>>
>>>> Testing new kernels is good, you are adding coverage. But
>>>> this break the acceptance testing contract "keep testing
>>>> the same thing over time".
>>>>
>>>> The problem you are trying to fix is the "where to keep
>>>> assets from public locations where they are being removed?"
>>>> one. Two years ago [*] you suggested to use some storage on
>>>> the avocado-project.org:
>>>>
>>>>   For Avocado-VT, there are the JeOS images[1], which we
>>>>   keep on a test "assets" directory.  We have a lot of
>>>>   storage/bandwidth availability, so it can be used for
>>>>   other assets proven to be necessary for tests.
>>>>
>>>>   As long as distribution rights and licensing are not
>>>>   issues, we can definitely use the same server for kernels,
>>>>   u-boot images and what not.
>>>>
>>>>   [1] - https://avocado-project.org/data/assets/
>>>
>>> If I look at stuff under that directory I see a bunch of "Jeos" qcow2
>>> images, and zero information about the corresponding source for the
>>> images, nor any information about the licenses of software included.
>>> IOW what is stored their right now does not appear to comply with the
>>> GPL licensing requirements for providing full and corresponding source.
>>>
>>>> It is time to have QEMU assets managed the same way.
>>>
>>> I'd rather we didn't do anything relying on binary blobs with no
>>> info about how they were built. Pointing to the 3rd party download
>>> URLs was the easy way to ensure we don't have to worry about licensing
>>> problems.
>>
>> I tried to be very strict including the recipe about how to rebuild
>> and description of the source (for licensing) in each commits (Alex
>> Bennée once said Debian/Fedora based was OK):
> 
> ..snip...
> 
> Well that looks better than what is done for the JEOS images currently
> on avocado-project.org, as I can't tell what distro those came from
> at all.
> 
> If we're hosting images built by some 3rd party, and we intend to rely
> on the 3rd party to satisfy source availability, then we need to be sure
> that the 3rd party is themselves still distributing the same images.
> 
> IIUC, from Cleber's commit here the original images we're pointing to
> are now 404s. If the URLs moved, we just need to update to fix the URLs
> to point the new location. If the content was entirely removed though,
> we shouldn't mirror it ourselves, because we can't rely on the original
> vendor to be providing the source at that point.

Having backups and the SHA1 of the files already commited in our
repository, this is the outcome I prefer.
Let see what other think on this topic.

Thanks for your insights :)

> 
> Regards,
> Daniel
>

Philippe Mathieu-Daudé <philmd@redhat.com> writes:

> On 9/7/20 11:39 AM, Daniel P. Berrangé wrote:
>> On Mon, Sep 07, 2020 at 10:06:13AM +0200, Philippe Mathieu-Daudé wrote:
>>> [Cc'ing Daniel who usually have good ideas for that
>>> kind if project-wide problem]
>>>
>>> On 9/7/20 6:19 AM, Cleber Rosa wrote:
>>>> Which means a newer kernel version.  Expected output was changed
>>>> to match the new kernel too.
>>>
>>> Nack.
>>>
>>> Acceptance tests are not to test the latest Linux kernel,
>>> they aim to assert a specific kernel tested by some developer
>>> still works while QEMU evolves.
>>> QEMU doesn't have to adapt to the latest kernel;
>>> QEMU should keep boot an old kernel.
>>>
>>> Testing new kernels is good, you are adding coverage. But
>>> this break the acceptance testing contract "keep testing
>>> the same thing over time".
>>>
>>> The problem you are trying to fix is the "where to keep
>>> assets from public locations where they are being removed?"
>>> one. Two years ago [*] you suggested to use some storage on
>>> the avocado-project.org:
>>>
>>>   For Avocado-VT, there are the JeOS images[1], which we
>>>   keep on a test "assets" directory.  We have a lot of
>>>   storage/bandwidth availability, so it can be used for
>>>   other assets proven to be necessary for tests.
>>>
>>>   As long as distribution rights and licensing are not
>>>   issues, we can definitely use the same server for kernels,
>>>   u-boot images and what not.
>>>
>>>   [1] - https://avocado-project.org/data/assets/
>> 
>> If I look at stuff under that directory I see a bunch of "Jeos" qcow2
>> images, and zero information about the corresponding source for the
>> images, nor any information about the licenses of software included.
>> IOW what is stored their right now does not appear to comply with the
>> GPL licensing requirements for providing full and corresponding source.
>> 
>>> It is time to have QEMU assets managed the same way.
>> 
>> I'd rather we didn't do anything relying on binary blobs with no
>> info about how they were built. Pointing to the 3rd party download
>> URLs was the easy way to ensure we don't have to worry about licensing
>> problems.
>
> I tried to be very strict including the recipe about how to rebuild
> and description of the source (for licensing) in each commits (Alex
> Bennée once said Debian/Fedora based was OK):

Debian and Fedora both have good project hygiene for tracking sources
for their binary packages and extensive archives which we can work with.
These other projects seem to be more of a moving feast which I think is
proving we can't rely on them as a CI test and at best best efforts for
developer testing.

It seems Armbian do document their build process but it's not quite as
easy as just downloading the source deb.

>
<snip>
>
>> 
>> IIUC, the majority of our acceptance tests needs should be satisfied
>> with a combination of a Linux kernel and busybox tools. We already
>> use cross-compilers to build TCG test cases.
>> 
>> I'd like to see us provide a minimal "config" file for each Linux
>> kernel combo we need to test. We should be able to define a fairly
>> simple script that do the Linux build, busybox build and then puts
>> the combination of them into a disk image.  Something 100% automated
>> and reviewable.
>
> For Debian/Fedora this config is available in their package (or source
> package).
>
>> 
>> Even a minimal linux build takes some time, so we would need to cache
>> the binary result in some suitable manner. At least this way we have
>> a clear record of what we're building & how and can thus satisfy the
>> license rules.
>
> Yes, but if the Debian/Fedora build farms already build/published
> their packages, it is easier to use it directly.
>
> QEMU developers are very reluctant to add tests. I suppose we prefer
> to develop or fix bugs rather than write tests. If we ask full build
> recipe for each tests, I doubt we'll have much testing.

I've pondered this before and while it might have a place for random
weird machines with no mainline distro support it's not a path I'm keen
to go down. This goes especially for some of the more esoteric ARM
hardware where you start relying on hacked up vendor trees with very
specific toolchain requirements.

<snip>