[4/7] docs/devel: add information on how to setup build environments

MacOS and Linux are straightforward, but Windows needs a bit more
details.

Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
---
 docs/about/build-platforms.rst   |   4 +-
 docs/devel/build-environment.rst | 114 +++++++++++++++++++++++++++++++
 docs/devel/index-build.rst       |   1 +
 3 files changed, 118 insertions(+), 1 deletion(-)
 create mode 100644 docs/devel/build-environment.rst

On Mon, Nov 18, 2024 at 09:23:54AM -0800, Pierrick Bouvier wrote:
> MacOS and Linux are straightforward, but Windows needs a bit more
> details.
> 
> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
> ---
>  docs/about/build-platforms.rst   |   4 +-
>  docs/devel/build-environment.rst | 114 +++++++++++++++++++++++++++++++
>  docs/devel/index-build.rst       |   1 +
>  3 files changed, 118 insertions(+), 1 deletion(-)
>  create mode 100644 docs/devel/build-environment.rst
> 
> diff --git a/docs/about/build-platforms.rst b/docs/about/build-platforms.rst
> index 6102f00aec0..c1ea53db834 100644
> --- a/docs/about/build-platforms.rst
> +++ b/docs/about/build-platforms.rst
> @@ -29,6 +29,9 @@ The `Repology`_ site is a useful resource to identify
>  currently shipped versions of software in various operating systems,
>  though it does not cover all distros listed below.
>  
> +You can find how to install build dependencies for different systems on the
> +:ref:`setup-build-env` page.
> +
>  Supported host architectures
>  ----------------------------
>  
> @@ -130,7 +133,6 @@ Optional build dependencies
>    cross compilation using ``docker`` or ``podman``, or to use pre-built
>    binaries distributed with QEMU.
>  
> -
>  Windows
>  -------
>  
> diff --git a/docs/devel/build-environment.rst b/docs/devel/build-environment.rst
> new file mode 100644
> index 00000000000..d9a66f5fcc6
> --- /dev/null
> +++ b/docs/devel/build-environment.rst
> @@ -0,0 +1,114 @@
> +
> +.. _setup-build-env:
> +
> +Setup build environment
> +=======================
> +
> +QEMU uses a lot of dependencies on the host system. glib2 is used everywhere in
> +the code base, and most of the other dependencies are optional.
> +
> +We present here simple instructions to enable native builds on most popular
> +systems.
> +
> +You can find additional instructions on `QEMU wiki <https://wiki.qemu.org/>`_:
> +
> +- `Linux <https://wiki.qemu.org/Hosts/Linux>`_
> +- `MacOS <https://wiki.qemu.org/Hosts/Mac>`_
> +- `Windows <https://wiki.qemu.org/Hosts/W32>`_
> +- `BSD <https://wiki.qemu.org/Hosts/BSD>`_


We generally suffer from having information spread over multiple sources,
giving us the burden of keeping the different places consistent, which we
pretty consistently fail at.

It is a good think to add build env docs to qemu.git where we actually
have oversight / review to catch mistakes. If we do this though, IMHO,
we should be deleting the wiki pages and making them into 302 redirects
to our new in-tree docs.

> +
> +Linux
> +-----
> +
> +Fedora
> +++++++
> +
> +::
> +
> +    sudo dnf update && sudo dnf builddep qemu
> +
> +Debian/Ubuntu
> ++++++++++++++
> +
> +You first need to enable `Sources List <https://wiki.debian.org/SourcesList>`_.
> +Then, use apt to install dependencies:
> +
> +::
> +
> +    sudo apt update && sudo apt build-dep qemu
> +
> +MacOS
> +-----
> +
> +You first need to install `Homebrew <https://brew.sh/>`_. Then, use it to
> +install dependencies:
> +
> +::
> +
> +    brew update && brew install $(brew deps --include-build qemu)


The downside in recommending the 'build dep' approach is that it misses
out on deps that have been newly introduced in qemu.git, since whatever
old version of QEMU the distros are shipping.  It also misses deps for
any features the distro vendor has decided to exclude.

Can we put a caveat describing this limitation at the top, so that users
have pointer if things don't quite go the way we expected.

> +
> +Windows
> +-------
> +
> +You first need to install `MSYS2 <https://www.msys2.org/>`_.
> +MSYS2 offers `different environments <https://www.msys2.org/docs/environments/>`_.
> +x86_64 environments are based on GCC, while aarch64 is based on Clang.
> +
> +We recommend to use MINGW64 for windows-x86_64 and CLANGARM64 for windows-aarch64
> +(only available on windows-aarch64 hosts).

Does CLANGARM64 really work with QEMU ?   We go out of our way to actively
block the use of CLang for Windows because of its lack of support for
'gcc_struct' attributes, so I would have expected it to fail

> +
> +Then, you can open a windows shell, and enter msys2 env using:
> +
> +::
> +
> +    c:/msys64/msys2_shell.cmd -defterm -here -no-start -mingw64
> +    # Replace -ucrt64 by -clangarm64 or -ucrt64 for other environments.
> +
> +MSYS2 package manager does not offer a built-in way to install build
> +dependencies. You can start with this list of packages using pacman:
> +

With regards,
Daniel

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Mon, Nov 18, 2024 at 09:23:54AM -0800, Pierrick Bouvier wrote:
>> MacOS and Linux are straightforward, but Windows needs a bit more
>> details.
>> 
>> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
>> ---
>>  docs/about/build-platforms.rst   |   4 +-
>>  docs/devel/build-environment.rst | 114 +++++++++++++++++++++++++++++++
>>  docs/devel/index-build.rst       |   1 +
>>  3 files changed, 118 insertions(+), 1 deletion(-)
>>  create mode 100644 docs/devel/build-environment.rst
>> 
<snip>
>> +Fedora
>> +++++++
>> +
>> +::
>> +
>> +    sudo dnf update && sudo dnf builddep qemu
>> +
>> +Debian/Ubuntu
>> ++++++++++++++
>> +
>> +You first need to enable `Sources List <https://wiki.debian.org/SourcesList>`_.
>> +Then, use apt to install dependencies:
>> +
>> +::
>> +
>> +    sudo apt update && sudo apt build-dep qemu
>> +
>> +MacOS
>> +-----
>> +
>> +You first need to install `Homebrew <https://brew.sh/>`_. Then, use it to
>> +install dependencies:
>> +
>> +::
>> +
>> +    brew update && brew install $(brew deps --include-build qemu)
>
>
> The downside in recommending the 'build dep' approach is that it misses
> out on deps that have been newly introduced in qemu.git, since whatever
> old version of QEMU the distros are shipping.  It also misses deps for
> any features the distro vendor has decided to exclude.
>
> Can we put a caveat describing this limitation at the top, so that users
> have pointer if things don't quite go the way we expected.

At a recent QEMU workshop I presented I pointed to lcitool as a way to
list the minimal dependencies QEMU needs:

  ./tests/lcitool/libvirt-ci/bin/lcitool variables -f yaml debian-12 ./tests/lcitool/projects/qemu-minimal.yml

It wouldn't be the hardest thing to extend lcitool to generate a pkg
install line instead of a list. Although I notice it fails with the full
set of dependancies:

  ✗  ./tests/lcitool/libvirt-ci/bin/lcitool -d tests/lcitool variables -f shell debian-12 ./tests/lcitool/projects/qemu.yml
  Package generic name resolution error: Package libcbor not present in mappings

I note there is already an "install" action but I'm wary of what it does.

>
>> +
>> +Windows
>> +-------
>> +
>> +You first need to install `MSYS2 <https://www.msys2.org/>`_.
>> +MSYS2 offers `different environments <https://www.msys2.org/docs/environments/>`_.
>> +x86_64 environments are based on GCC, while aarch64 is based on Clang.
>> +
>> +We recommend to use MINGW64 for windows-x86_64 and CLANGARM64 for windows-aarch64
>> +(only available on windows-aarch64 hosts).
>
> Does CLANGARM64 really work with QEMU ?   We go out of our way to actively
> block the use of CLang for Windows because of its lack of support for
> 'gcc_struct' attributes, so I would have expected it to fail
>
>> +
>> +Then, you can open a windows shell, and enter msys2 env using:
>> +
>> +::
>> +
>> +    c:/msys64/msys2_shell.cmd -defterm -here -no-start -mingw64
>> +    # Replace -ucrt64 by -clangarm64 or -ucrt64 for other environments.
>> +
>> +MSYS2 package manager does not offer a built-in way to install build
>> +dependencies. You can start with this list of packages using pacman:
>> +
>
> With regards,
> Daniel

On Tue, Nov 19, 2024 at 11:08:12AM +0000, Alex Bennée wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> > On Mon, Nov 18, 2024 at 09:23:54AM -0800, Pierrick Bouvier wrote:
> >> MacOS and Linux are straightforward, but Windows needs a bit more
> >> details.
> >> 
> >> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
> >> ---
> >>  docs/about/build-platforms.rst   |   4 +-
> >>  docs/devel/build-environment.rst | 114 +++++++++++++++++++++++++++++++
> >>  docs/devel/index-build.rst       |   1 +
> >>  3 files changed, 118 insertions(+), 1 deletion(-)
> >>  create mode 100644 docs/devel/build-environment.rst
> >> 
> <snip>
> >> +Fedora
> >> +++++++
> >> +
> >> +::
> >> +
> >> +    sudo dnf update && sudo dnf builddep qemu
> >> +
> >> +Debian/Ubuntu
> >> ++++++++++++++
> >> +
> >> +You first need to enable `Sources List <https://wiki.debian.org/SourcesList>`_.
> >> +Then, use apt to install dependencies:
> >> +
> >> +::
> >> +
> >> +    sudo apt update && sudo apt build-dep qemu
> >> +
> >> +MacOS
> >> +-----
> >> +
> >> +You first need to install `Homebrew <https://brew.sh/>`_. Then, use it to
> >> +install dependencies:
> >> +
> >> +::
> >> +
> >> +    brew update && brew install $(brew deps --include-build qemu)
> >
> >
> > The downside in recommending the 'build dep' approach is that it misses
> > out on deps that have been newly introduced in qemu.git, since whatever
> > old version of QEMU the distros are shipping.  It also misses deps for
> > any features the distro vendor has decided to exclude.
> >
> > Can we put a caveat describing this limitation at the top, so that users
> > have pointer if things don't quite go the way we expected.
> 
> At a recent QEMU workshop I presented I pointed to lcitool as a way to
> list the minimal dependencies QEMU needs:
> 
>   ./tests/lcitool/libvirt-ci/bin/lcitool variables -f yaml debian-12 ./tests/lcitool/projects/qemu-minimal.yml
> 
> It wouldn't be the hardest thing to extend lcitool to generate a pkg
> install line instead of a list. Although I notice it fails with the full
> set of dependancies:
> 
>   ✗  ./tests/lcitool/libvirt-ci/bin/lcitool -d tests/lcitool variables -f shell debian-12 ./tests/lcitool/projects/qemu.yml
>   Package generic name resolution error: Package libcbor not present in mappings

I think your libvirt-ci git submodule might be out of date. The current
generated Dockerfiles include libcbor and its in the mappings.

> I note there is already an "install" action but I'm wary of what it does.

That's related to lcitool functionality for provisioning VMs - kinda like
QEMUs tests/vm/ stuff.

With regards,
Daniel

On 11/19/24 01:24, Daniel P. Berrangé wrote:
> On Mon, Nov 18, 2024 at 09:23:54AM -0800, Pierrick Bouvier wrote:
>> MacOS and Linux are straightforward, but Windows needs a bit more
>> details.
>>
>> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
>> ---
>>   docs/about/build-platforms.rst   |   4 +-
>>   docs/devel/build-environment.rst | 114 +++++++++++++++++++++++++++++++
>>   docs/devel/index-build.rst       |   1 +
>>   3 files changed, 118 insertions(+), 1 deletion(-)
>>   create mode 100644 docs/devel/build-environment.rst
>>
>> diff --git a/docs/about/build-platforms.rst b/docs/about/build-platforms.rst
>> index 6102f00aec0..c1ea53db834 100644
>> --- a/docs/about/build-platforms.rst
>> +++ b/docs/about/build-platforms.rst
>> @@ -29,6 +29,9 @@ The `Repology`_ site is a useful resource to identify
>>   currently shipped versions of software in various operating systems,
>>   though it does not cover all distros listed below.
>>   
>> +You can find how to install build dependencies for different systems on the
>> +:ref:`setup-build-env` page.
>> +
>>   Supported host architectures
>>   ----------------------------
>>   
>> @@ -130,7 +133,6 @@ Optional build dependencies
>>     cross compilation using ``docker`` or ``podman``, or to use pre-built
>>     binaries distributed with QEMU.
>>   
>> -
>>   Windows
>>   -------
>>   
>> diff --git a/docs/devel/build-environment.rst b/docs/devel/build-environment.rst
>> new file mode 100644
>> index 00000000000..d9a66f5fcc6
>> --- /dev/null
>> +++ b/docs/devel/build-environment.rst
>> @@ -0,0 +1,114 @@
>> +
>> +.. _setup-build-env:
>> +
>> +Setup build environment
>> +=======================
>> +
>> +QEMU uses a lot of dependencies on the host system. glib2 is used everywhere in
>> +the code base, and most of the other dependencies are optional.
>> +
>> +We present here simple instructions to enable native builds on most popular
>> +systems.
>> +
>> +You can find additional instructions on `QEMU wiki <https://wiki.qemu.org/>`_:
>> +
>> +- `Linux <https://wiki.qemu.org/Hosts/Linux>`_
>> +- `MacOS <https://wiki.qemu.org/Hosts/Mac>`_
>> +- `Windows <https://wiki.qemu.org/Hosts/W32>`_
>> +- `BSD <https://wiki.qemu.org/Hosts/BSD>`_
> 
> 
> We generally suffer from having information spread over multiple sources,
> giving us the burden of keeping the different places consistent, which we
> pretty consistently fail at.
> 

Agree on that.

> It is a good think to add build env docs to qemu.git where we actually
> have oversight / review to catch mistakes. If we do this though, IMHO,
> we should be deleting the wiki pages and making them into 302 redirects
> to our new in-tree docs.
> 

Sure.
However, there is still some useful information there, that are not 
reflected in this patch.

We could migrate the information later, but for now, I can't spend many 
cycles to try to build all configurations described on the wiki to make 
sure it's up to date.
So I just included information I'm sure it works, and that cover the 
basic needs to build on all major systems.

Hope this is a starting point we can maybe accept, and we can enhance 
the documentation later. But asking to do all this at once is a big 
task, and I feel it's a lot to ask for it as part of the current series.

Would you accept that, or prefer to not include anything before we have 
purged the wikis completely?

>> +
>> +Linux
>> +-----
>> +
>> +Fedora
>> +++++++
>> +
>> +::
>> +
>> +    sudo dnf update && sudo dnf builddep qemu
>> +
>> +Debian/Ubuntu
>> ++++++++++++++
>> +
>> +You first need to enable `Sources List <https://wiki.debian.org/SourcesList>`_.
>> +Then, use apt to install dependencies:
>> +
>> +::
>> +
>> +    sudo apt update && sudo apt build-dep qemu
>> +
>> +MacOS
>> +-----
>> +
>> +You first need to install `Homebrew <https://brew.sh/>`_. Then, use it to
>> +install dependencies:
>> +
>> +::
>> +
>> +    brew update && brew install $(brew deps --include-build qemu)
> 
> 
> The downside in recommending the 'build dep' approach is that it misses
> out on deps that have been newly introduced in qemu.git, since whatever
> old version of QEMU the distros are shipping.  It also misses deps for
> any features the distro vendor has decided to exclude.
> 
> Can we put a caveat describing this limitation at the top, so that users
> have pointer if things don't quite go the way we expected.
>

I can add a note about it. It's especially true for Linux, as on Windows 
(MSYS2) and MacOS (homebrew), they usually track latest release of QEMU, 
so the build list should match, given a reduced time frame after the 
release.

>> +
>> +Windows
>> +-------
>> +
>> +You first need to install `MSYS2 <https://www.msys2.org/>`_.
>> +MSYS2 offers `different environments <https://www.msys2.org/docs/environments/>`_.
>> +x86_64 environments are based on GCC, while aarch64 is based on Clang.
>> +
>> +We recommend to use MINGW64 for windows-x86_64 and CLANGARM64 for windows-aarch64
>> +(only available on windows-aarch64 hosts).
> 
> Does CLANGARM64 really work with QEMU ?   We go out of our way to actively
> block the use of CLang for Windows because of its lack of support for
> 'gcc_struct' attributes, so I would have expected it to fail
> 

I can drop the windows-arm64 part.
For the support, it's a discussion we add previously on the concerned 
series. In current master, it won't even go past the configure step 
because of intentional restriction we have in meson.build.
My initial hope was to remove it, but I understand that we prefer to 
wait for gcc_struct support in clang. So be it, I can live with a local 
fix for that meanwhile :).

>> +
>> +Then, you can open a windows shell, and enter msys2 env using:
>> +
>> +::
>> +
>> +    c:/msys64/msys2_shell.cmd -defterm -here -no-start -mingw64
>> +    # Replace -ucrt64 by -clangarm64 or -ucrt64 for other environments.
>> +
>> +MSYS2 package manager does not offer a built-in way to install build
>> +dependencies. You can start with this list of packages using pacman:
>> +
> 
> With regards,
> Daniel