| Message ID | 20190910084349.28817-45-alex.bennee@linaro.org |
|---|---|
| State | Accepted |
| Commit | 93bbbdf6721557f706fc67257d028e4fc16dba58 |
| Headers | show |
| Series | testing updates (fixes, upgrades, caching) | expand |
On Tue, Sep 10, 2019 at 09:43:48AM +0100, Alex Bennée wrote: > From: Cleber Rosa <crosa@redhat.com> > > The LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16 test, > from tests/acceptance/linux_initrd.py, is currently failing to fetch > the "vmlinuz" file. The reason for the failure is that the Fedora > project retires older versions from the "dl.fedoraproject.org" URL, > and keeps them in "archives.fedoraproject.org". As an added note, > that test uses a Fedora 28 image, because of the specific Linux kernel > version requirements of the test. > > For the sake of stability, let's use URLs from the archived and > supposedely ever stable URLs. The good news is that the currently > supported versions are also hosted on the later. This change limits > itself to change the URLs, while keeping the fetched files the same > (as can be evidenced by the unchanged hashes). FWIW, I commented on the oiriginal patch before seeing this PR. I don't think we should be using the archive URL for anything which still exists on download.fedoraproject.org as that's placing undesirable load on the Fedora master servers, instead of using its primary CDN via the mirror services. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Daniel P. Berrangé <berrange@redhat.com> writes: > On Tue, Sep 10, 2019 at 09:43:48AM +0100, Alex Bennée wrote: >> From: Cleber Rosa <crosa@redhat.com> >> >> The LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16 test, >> from tests/acceptance/linux_initrd.py, is currently failing to fetch >> the "vmlinuz" file. The reason for the failure is that the Fedora >> project retires older versions from the "dl.fedoraproject.org" URL, >> and keeps them in "archives.fedoraproject.org". As an added note, >> that test uses a Fedora 28 image, because of the specific Linux kernel >> version requirements of the test. >> >> For the sake of stability, let's use URLs from the archived and >> supposedely ever stable URLs. The good news is that the currently >> supported versions are also hosted on the later. This change limits >> itself to change the URLs, while keeping the fetched files the same >> (as can be evidenced by the unchanged hashes). > > FWIW, I commented on the oiriginal patch before seeing this PR. > I don't think we should be using the archive URL for anything > which still exists on download.fedoraproject.org as that's placing > undesirable load on the Fedora master servers, instead of using its > primary CDN via the mirror services. I think most of the CI services have caching proxies to the outside world to reduce their internetwork load. Would we be able to check with the webmasters to see what sort of effect we are having on their load? The problem of using the normal CDN is if we don't keep on top of updating URLs as the upstream moves the artefacts to archives our CI keeps breaking. Perhaps we should be hosting these ourselves? But that involves infrastructure and also ensuring we meet GPL requirements whereas at the moment we basically offload that work onto the distros. > > > Regards, > Daniel -- Alex Bennée
On Tue, 10 Sep 2019 at 10:07, Daniel P. Berrangé <berrange@redhat.com> wrote: > > On Tue, Sep 10, 2019 at 09:43:48AM +0100, Alex Bennée wrote: > > From: Cleber Rosa <crosa@redhat.com> > > > > The LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16 test, > > from tests/acceptance/linux_initrd.py, is currently failing to fetch > > the "vmlinuz" file. The reason for the failure is that the Fedora > > project retires older versions from the "dl.fedoraproject.org" URL, > > and keeps them in "archives.fedoraproject.org". As an added note, > > that test uses a Fedora 28 image, because of the specific Linux kernel > > version requirements of the test. > > > > For the sake of stability, let's use URLs from the archived and > > supposedely ever stable URLs. The good news is that the currently > > supported versions are also hosted on the later. This change limits > > itself to change the URLs, while keeping the fetched files the same > > (as can be evidenced by the unchanged hashes). > > FWIW, I commented on the oiriginal patch before seeing this PR. > I don't think we should be using the archive URL for anything > which still exists on download.fedoraproject.org as that's placing > undesirable load on the Fedora master servers, instead of using its > primary CDN via the mirror services. Mmm; I think we should take this patch for the moment to fix the Travis CI failures, though -- we can look at a more careful choice of examples/URLs as a followup. thanks -- PMM
On 9/10/19 11:07 AM, Daniel P. Berrangé wrote: > On Tue, Sep 10, 2019 at 09:43:48AM +0100, Alex Bennée wrote: >> From: Cleber Rosa <crosa@redhat.com> >> >> The LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16 test, >> from tests/acceptance/linux_initrd.py, is currently failing to fetch >> the "vmlinuz" file. The reason for the failure is that the Fedora >> project retires older versions from the "dl.fedoraproject.org" URL, >> and keeps them in "archives.fedoraproject.org". As an added note, >> that test uses a Fedora 28 image, because of the specific Linux kernel >> version requirements of the test. >> >> For the sake of stability, let's use URLs from the archived and >> supposedely ever stable URLs. The good news is that the currently >> supported versions are also hosted on the later. This change limits >> itself to change the URLs, while keeping the fetched files the same >> (as can be evidenced by the unchanged hashes). > > FWIW, I commented on the oiriginal patch before seeing this PR. > I don't think we should be using the archive URL for anything > which still exists on download.fedoraproject.org as that's placing > undesirable load on the Fedora master servers, instead of using its > primary CDN via the mirror services. - qemu-doc.texi This documentation is used by distributions and can stay installed for ages. I have no idea how often users copy/paste the doc to use it, but at least the documentation urls will stay working. - tests/acceptance/boot_linux_console.py - tests/acceptance/linux_initrd.py These are cached for local user, and with the addition of '.travis.yml: Cache Avocado cache' in this series, these are now also cached on Travis CI. - tests/vm/fedora This one I don't know, you might be right. Regards, Phil.
diff --git a/qemu-doc.texi b/qemu-doc.texi index 4d828a5135f..b47e89cfca6 100644 --- a/qemu-doc.texi +++ b/qemu-doc.texi @@ -451,15 +451,15 @@ of <protocol>. Example: boot from a remote Fedora 20 live ISO image @example -@value{qemu_system_x86} --drive media=cdrom,file=http://dl.fedoraproject.org/pub/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly +@value{qemu_system_x86} --drive media=cdrom,file=https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly -@value{qemu_system_x86} --drive media=cdrom,file.driver=http,file.url=http://dl.fedoraproject.org/pub/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly +@value{qemu_system_x86} --drive media=cdrom,file.driver=http,file.url=http://archives.fedoraproject.org/pub/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso,readonly @end example Example: boot from a remote Fedora 20 cloud image using a local overlay for writes, copy-on-read, and a readahead of 64k @example -qemu-img create -f qcow2 -o backing_file='json:@{"file.driver":"http",, "file.url":"https://dl.fedoraproject.org/pub/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2",, "file.readahead":"64k"@}' /tmp/Fedora-x86_64-20-20131211.1-sda.qcow2 +qemu-img create -f qcow2 -o backing_file='json:@{"file.driver":"http",, "file.url":"http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.qcow2",, "file.readahead":"64k"@}' /tmp/Fedora-x86_64-20-20131211.1-sda.qcow2 @value{qemu_system_x86} -drive file=/tmp/Fedora-x86_64-20-20131211.1-sda.qcow2,copy-on-read=on @end example diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py index 2504ef01507..8a9a314ab49 100644 --- a/tests/acceptance/boot_linux_console.py +++ b/tests/acceptance/boot_linux_console.py @@ -76,8 +76,9 @@ class BootLinuxConsole(Test): :avocado: tags=arch:x86_64 :avocado: tags=machine:pc """ - kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/' - 'releases/29/Everything/x86_64/os/images/pxeboot/vmlinuz') + kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora' + '/linux/releases/29/Everything/x86_64/os/images/pxeboot' + '/vmlinuz') kernel_hash = '23bebd2680757891cf7adedb033532163a792495' kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash) @@ -250,8 +251,9 @@ class BootLinuxConsole(Test): :avocado: tags=arch:aarch64 :avocado: tags=machine:virt """ - kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/' - 'releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz') + kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora' + '/linux/releases/29/Everything/aarch64/os/images/pxeboot' + '/vmlinuz') kernel_hash = '8c73e469fc6ea06a58dc83a628fc695b693b8493' kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash) @@ -271,8 +273,9 @@ class BootLinuxConsole(Test): :avocado: tags=arch:arm :avocado: tags=machine:virt """ - kernel_url = ('https://download.fedoraproject.org/pub/fedora/linux/' - 'releases/29/Everything/armhfp/os/images/pxeboot/vmlinuz') + kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora' + '/linux/releases/29/Everything/armhfp/os/images/pxeboot' + '/vmlinuz') kernel_hash = 'e9826d741b4fb04cadba8d4824d1ed3b7fb8b4d4' kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash) @@ -318,8 +321,9 @@ class BootLinuxConsole(Test): :avocado: tags=arch:s390x :avocado: tags=machine:s390_ccw_virtio """ - kernel_url = ('https://download.fedoraproject.org/pub/fedora-secondary/' - 'releases/29/Everything/s390x/os/images/kernel.img') + kernel_url = ('https://archives.fedoraproject.org/pub/archive' + '/fedora-secondary/releases/29/Everything/s390x/os/images' + '/kernel.img') kernel_hash = 'e8e8439103ef8053418ef062644ffd46a7919313' kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash) @@ -360,8 +364,9 @@ class BootLinuxConsole(Test): :avocado: tags=arch:ppc64 :avocado: tags=machine:pseries """ - kernel_url = ('https://download.fedoraproject.org/pub/fedora-secondary/' - 'releases/29/Everything/ppc64le/os/ppc/ppc64/vmlinuz') + kernel_url = ('https://archives.fedoraproject.org/pub/archive' + '/fedora-secondary/releases/29/Everything/ppc64le/os' + '/ppc/ppc64/vmlinuz') kernel_hash = '3fe04abfc852b66653b8c3c897a59a689270bc77' kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash) diff --git a/tests/acceptance/linux_initrd.py b/tests/acceptance/linux_initrd.py index 23be5a63aa8..c61d9826a45 100644 --- a/tests/acceptance/linux_initrd.py +++ b/tests/acceptance/linux_initrd.py @@ -54,8 +54,9 @@ class LinuxInitrd(Test): QEMU has supported up to 4 GiB initrd for recent kernel Expect guest can reach 'Unpacking initramfs...' """ - kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/' - 'Everything/x86_64/os/images/pxeboot/vmlinuz') + kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora' + '/linux/releases/28/Everything/x86_64/os/images/pxeboot/' + 'vmlinuz') kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a' kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash) max_size = 2 * (1024 ** 3) + 1 diff --git a/tests/vm/fedora b/tests/vm/fedora index e8fa5bf0d21..7fec1479fb7 100755 --- a/tests/vm/fedora +++ b/tests/vm/fedora @@ -23,7 +23,7 @@ class FedoraVM(basevm.BaseVM): name = "fedora" arch = "x86_64" - base = "http://dl.fedoraproject.org/pub/fedora/linux/releases/30/" + base = "https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/30/" link = base + "Server/x86_64/iso/Fedora-Server-netinst-x86_64-30-1.2.iso" repo = base + "Server/x86_64/os/" full = base + "Everything/x86_64/os/"