| Message ID | 20201102185115.7425-1-andrew@daynix.com |
|---|---|
| Headers | show |
| Series | eBPF RSS support for virtio-net | expand |
On 2020/11/3 上午2:51, Andrew Melnychenko wrote: > Basic idea is to use eBPF to calculate and steer packets in TAP. > RSS(Receive Side Scaling) is used to distribute network packets to guest virtqueues > by calculating packet hash. > eBPF RSS allows us to use RSS with vhost TAP. > > This set of patches introduces the usage of eBPF for packet steering > and RSS hash calculation: > * RSS(Receive Side Scaling) is used to distribute network packets to > guest virtqueues by calculating packet hash > * eBPF RSS suppose to be faster than already existing 'software' > implementation in QEMU > * Additionally adding support for the usage of RSS with vhost > > Supported kernels: 5.8+ > > Implementation notes: > Linux TAP TUNSETSTEERINGEBPF ioctl was used to set the eBPF program. > Added eBPF support to qemu directly through a system call, see the > bpf(2) for details. > The eBPF program is part of the qemu and presented as an array of bpf > instructions. > The program can be recompiled by provided Makefile.ebpf(need to adjust > 'linuxhdrs'), > although it's not required to build QEMU with eBPF support. > Added changes to virtio-net and vhost, primary eBPF RSS is used. > 'Software' RSS used in the case of hash population and as a fallback option. > For vhost, the hash population feature is not reported to the guest. > > Please also see the documentation in PATCH 6/6. > > I am sending those patches as RFC to initiate the discussions and get > feedback on the following points: > * Fallback when eBPF is not supported by the kernel Yes, and it could also a lacking of CAP_BPF. > * Live migration to the kernel that doesn't have eBPF support Is there anything that we needs special treatment here? > * Integration with current QEMU build Yes, a question here: 1) Any reason for not using libbpf, e.g it has been shipped with some distros 2) It would be better if we can avoid shipping bytecodes > * Additional usage for eBPF for packet filtering Another interesting topics in to implement mac/vlan filters. And in the future, I plan to add mac based steering. All of these could be done via eBPF. > > Know issues: > * hash population not supported by eBPF RSS: 'software' RSS used Is this because there's not way to write to vnet header in STERRING BPF? > as a fallback, also, hash population feature is not reported to guests > with vhost. > * big-endian BPF support: for now, eBPF is disabled for big-endian systems. Are there any blocker for this? Just some quick questions after a glance of the codes. Will go through them tomorrow. Thanks > > Andrew (6): > Added SetSteeringEBPF method for NetClientState. > ebpf: Added basic eBPF API. > ebpf: Added eBPF RSS program. > ebpf: Added eBPF RSS loader. > virtio-net: Added eBPF RSS to virtio-net. > docs: Added eBPF documentation. > > MAINTAINERS | 6 + > configure | 36 +++ > docs/ebpf.rst | 29 ++ > docs/ebpf_rss.rst | 129 ++++++++ > ebpf/EbpfElf_to_C.py | 67 ++++ > ebpf/Makefile.ebpf | 38 +++ > ebpf/ebpf-stub.c | 28 ++ > ebpf/ebpf.c | 107 +++++++ > ebpf/ebpf.h | 35 +++ > ebpf/ebpf_rss.c | 178 +++++++++++ > ebpf/ebpf_rss.h | 30 ++ > ebpf/meson.build | 1 + > ebpf/rss.bpf.c | 470 ++++++++++++++++++++++++++++ > ebpf/trace-events | 4 + > ebpf/trace.h | 2 + > ebpf/tun_rss_steering.h | 556 +++++++++++++++++++++++++++++++++ > hw/net/vhost_net.c | 2 + > hw/net/virtio-net.c | 120 ++++++- > include/hw/virtio/virtio-net.h | 4 + > include/net/net.h | 2 + > meson.build | 3 + > net/tap-bsd.c | 5 + > net/tap-linux.c | 19 ++ > net/tap-solaris.c | 5 + > net/tap-stub.c | 5 + > net/tap.c | 9 + > net/tap_int.h | 1 + > net/vhost-vdpa.c | 2 + > 28 files changed, 1889 insertions(+), 4 deletions(-) > create mode 100644 docs/ebpf.rst > create mode 100644 docs/ebpf_rss.rst > create mode 100644 ebpf/EbpfElf_to_C.py > create mode 100755 ebpf/Makefile.ebpf > create mode 100644 ebpf/ebpf-stub.c > create mode 100644 ebpf/ebpf.c > create mode 100644 ebpf/ebpf.h > create mode 100644 ebpf/ebpf_rss.c > create mode 100644 ebpf/ebpf_rss.h > create mode 100644 ebpf/meson.build > create mode 100644 ebpf/rss.bpf.c > create mode 100644 ebpf/trace-events > create mode 100644 ebpf/trace.h > create mode 100644 ebpf/tun_rss_steering.h >
On Tue, Nov 3, 2020 at 11:02 AM Jason Wang <jasowang@redhat.com> wrote: > > On 2020/11/3 上午2:51, Andrew Melnychenko wrote: > > Basic idea is to use eBPF to calculate and steer packets in TAP. > > RSS(Receive Side Scaling) is used to distribute network packets to guest > virtqueues > > by calculating packet hash. > > eBPF RSS allows us to use RSS with vhost TAP. > > > > This set of patches introduces the usage of eBPF for packet steering > > and RSS hash calculation: > > * RSS(Receive Side Scaling) is used to distribute network packets to > > guest virtqueues by calculating packet hash > > * eBPF RSS suppose to be faster than already existing 'software' > > implementation in QEMU > > * Additionally adding support for the usage of RSS with vhost > > > > Supported kernels: 5.8+ > > > > Implementation notes: > > Linux TAP TUNSETSTEERINGEBPF ioctl was used to set the eBPF program. > > Added eBPF support to qemu directly through a system call, see the > > bpf(2) for details. > > The eBPF program is part of the qemu and presented as an array of bpf > > instructions. > > The program can be recompiled by provided Makefile.ebpf(need to adjust > > 'linuxhdrs'), > > although it's not required to build QEMU with eBPF support. > > Added changes to virtio-net and vhost, primary eBPF RSS is used. > > 'Software' RSS used in the case of hash population and as a fallback > option. > > For vhost, the hash population feature is not reported to the guest. > > > > Please also see the documentation in PATCH 6/6. > > > > I am sending those patches as RFC to initiate the discussions and get > > feedback on the following points: > > * Fallback when eBPF is not supported by the kernel > > > Yes, and it could also a lacking of CAP_BPF. > > > > * Live migration to the kernel that doesn't have eBPF support > > > Is there anything that we needs special treatment here? > > Possible case: rss=on, vhost=on, source system with kernel 5.8 (everything works) -> dest. system 5.6 (bpf does not work), the adapter functions, but all the steering does not use proper queues. > > > * Integration with current QEMU build > > > Yes, a question here: > > 1) Any reason for not using libbpf, e.g it has been shipped with some > distros > We intentionally do not use libbpf, as it present only on some distros. We can switch to libbpf, but this will disable bpf if libbpf is not installed > 2) It would be better if we can avoid shipping bytecodes > This creates new dependencies: llvm + clang + ... We would prefer byte code and ability to generate it if prerequisites are installed. > > > > * Additional usage for eBPF for packet filtering > > > Another interesting topics in to implement mac/vlan filters. And in the > future, I plan to add mac based steering. All of these could be done via > eBPF. > > No problem, we can cooperate if needed > > > > > Know issues: > > * hash population not supported by eBPF RSS: 'software' RSS used > > > Is this because there's not way to write to vnet header in STERRING BPF? > > Yes. We plan to submit changes for kernel to cooperate with BPF and populate the hash, this work is in progress > > > as a fallback, also, hash population feature is not reported to guests > > with vhost. > > * big-endian BPF support: for now, eBPF is disabled for big-endian > systems. > > > Are there any blocker for this? > No, can be added in v2 > > Just some quick questions after a glance of the codes. Will go through > them tomorrow. > > Thanks > > > > > > Andrew (6): > > Added SetSteeringEBPF method for NetClientState. > > ebpf: Added basic eBPF API. > > ebpf: Added eBPF RSS program. > > ebpf: Added eBPF RSS loader. > > virtio-net: Added eBPF RSS to virtio-net. > > docs: Added eBPF documentation. > > > > MAINTAINERS | 6 + > > configure | 36 +++ > > docs/ebpf.rst | 29 ++ > > docs/ebpf_rss.rst | 129 ++++++++ > > ebpf/EbpfElf_to_C.py | 67 ++++ > > ebpf/Makefile.ebpf | 38 +++ > > ebpf/ebpf-stub.c | 28 ++ > > ebpf/ebpf.c | 107 +++++++ > > ebpf/ebpf.h | 35 +++ > > ebpf/ebpf_rss.c | 178 +++++++++++ > > ebpf/ebpf_rss.h | 30 ++ > > ebpf/meson.build | 1 + > > ebpf/rss.bpf.c | 470 ++++++++++++++++++++++++++++ > > ebpf/trace-events | 4 + > > ebpf/trace.h | 2 + > > ebpf/tun_rss_steering.h | 556 +++++++++++++++++++++++++++++++++ > > hw/net/vhost_net.c | 2 + > > hw/net/virtio-net.c | 120 ++++++- > > include/hw/virtio/virtio-net.h | 4 + > > include/net/net.h | 2 + > > meson.build | 3 + > > net/tap-bsd.c | 5 + > > net/tap-linux.c | 19 ++ > > net/tap-solaris.c | 5 + > > net/tap-stub.c | 5 + > > net/tap.c | 9 + > > net/tap_int.h | 1 + > > net/vhost-vdpa.c | 2 + > > 28 files changed, 1889 insertions(+), 4 deletions(-) > > create mode 100644 docs/ebpf.rst > > create mode 100644 docs/ebpf_rss.rst > > create mode 100644 ebpf/EbpfElf_to_C.py > > create mode 100755 ebpf/Makefile.ebpf > > create mode 100644 ebpf/ebpf-stub.c > > create mode 100644 ebpf/ebpf.c > > create mode 100644 ebpf/ebpf.h > > create mode 100644 ebpf/ebpf_rss.c > > create mode 100644 ebpf/ebpf_rss.h > > create mode 100644 ebpf/meson.build > > create mode 100644 ebpf/rss.bpf.c > > create mode 100644 ebpf/trace-events > > create mode 100644 ebpf/trace.h > > create mode 100644 ebpf/tun_rss_steering.h > > > > <div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Nov 3, 2020 at 11:02 AM Jason Wang <<a href="mailto:jasowang@redhat.com">jasowang@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br> On 2020/11/3 上午2:51, Andrew Melnychenko wrote:<br> > Basic idea is to use eBPF to calculate and steer packets in TAP.<br> > RSS(Receive Side Scaling) is used to distribute network packets to guest virtqueues<br> > by calculating packet hash.<br> > eBPF RSS allows us to use RSS with vhost TAP.<br> ><br> > This set of patches introduces the usage of eBPF for packet steering<br> > and RSS hash calculation:<br> > * RSS(Receive Side Scaling) is used to distribute network packets to<br> > guest virtqueues by calculating packet hash<br> > * eBPF RSS suppose to be faster than already existing 'software'<br> > implementation in QEMU<br> > * Additionally adding support for the usage of RSS with vhost<br> ><br> > Supported kernels: 5.8+<br> ><br> > Implementation notes:<br> > Linux TAP TUNSETSTEERINGEBPF ioctl was used to set the eBPF program.<br> > Added eBPF support to qemu directly through a system call, see the<br> > bpf(2) for details.<br> > The eBPF program is part of the qemu and presented as an array of bpf<br> > instructions.<br> > The program can be recompiled by provided Makefile.ebpf(need to adjust<br> > 'linuxhdrs'),<br> > although it's not required to build QEMU with eBPF support.<br> > Added changes to virtio-net and vhost, primary eBPF RSS is used.<br> > 'Software' RSS used in the case of hash population and as a fallback option.<br> > For vhost, the hash population feature is not reported to the guest.<br> ><br> > Please also see the documentation in PATCH 6/6.<br> ><br> > I am sending those patches as RFC to initiate the discussions and get<br> > feedback on the following points:<br> > * Fallback when eBPF is not supported by the kernel<br> <br> <br> Yes, and it could also a lacking of CAP_BPF.<br> <br> <br> > * Live migration to the kernel that doesn't have eBPF support<br> <br> <br> Is there anything that we needs special treatment here?<br> <br></blockquote><div>Possible case: rss=on, vhost=on, source system with kernel 5.8 (everything works) -> dest. system 5.6 (bpf does not work), the adapter functions, but all the steering does not use proper queues. </div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br> > * Integration with current QEMU build<br> <br> <br> Yes, a question here:<br> <br> 1) Any reason for not using libbpf, e.g it has been shipped with some <br> distros <br></blockquote><div><br></div><div>We intentionally do not use libbpf, as it present only on some distros.</div><div>We can switch to libbpf, but this will disable bpf if libbpf is not installed </div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> 2) It would be better if we can avoid shipping bytecodes<br> </blockquote><div><br></div><div><br></div><div>This creates new dependencies: llvm + clang + ...</div><div>We would prefer byte code and ability to generate it if prerequisites are installed. <br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br> <br> > * Additional usage for eBPF for packet filtering<br> <br> <br> Another interesting topics in to implement mac/vlan filters. And in the <br> future, I plan to add mac based steering. All of these could be done via <br> eBPF.<br> <br></blockquote><div><br></div><div>No problem, we can cooperate if needed</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br> ><br> > Know issues:<br> > * hash population not supported by eBPF RSS: 'software' RSS used<br> <br> <br> Is this because there's not way to write to vnet header in STERRING BPF?<br> <br></blockquote><div>Yes. We plan to submit changes for kernel to cooperate with BPF and populate the hash, this work is in progress</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br> > as a fallback, also, hash population feature is not reported to guests<br> > with vhost.<br> > * big-endian BPF support: for now, eBPF is disabled for big-endian systems.<br> <br> <br> Are there any blocker for this?<br></blockquote><div><br></div><div>No, can be added in v2</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br> Just some quick questions after a glance of the codes. Will go through <br> them tomorrow.<br> <br> Thanks<br> <br> <br> ><br> > Andrew (6):<br> > Added SetSteeringEBPF method for NetClientState.<br> > ebpf: Added basic eBPF API.<br> > ebpf: Added eBPF RSS program.<br> > ebpf: Added eBPF RSS loader.<br> > virtio-net: Added eBPF RSS to virtio-net.<br> > docs: Added eBPF documentation.<br> ><br> > MAINTAINERS | 6 +<br> > configure | 36 +++<br> > docs/ebpf.rst | 29 ++<br> > docs/ebpf_rss.rst | 129 ++++++++<br> > ebpf/EbpfElf_to_C.py | 67 ++++<br> > ebpf/Makefile.ebpf | 38 +++<br> > ebpf/ebpf-stub.c | 28 ++<br> > ebpf/ebpf.c | 107 +++++++<br> > ebpf/ebpf.h | 35 +++<br> > ebpf/ebpf_rss.c | 178 +++++++++++<br> > ebpf/ebpf_rss.h | 30 ++<br> > ebpf/meson.build | 1 +<br> > ebpf/rss.bpf.c | 470 ++++++++++++++++++++++++++++<br> > ebpf/trace-events | 4 +<br> > ebpf/trace.h | 2 +<br> > ebpf/tun_rss_steering.h | 556 +++++++++++++++++++++++++++++++++<br> > hw/net/vhost_net.c | 2 +<br> > hw/net/virtio-net.c | 120 ++++++-<br> > include/hw/virtio/virtio-net.h | 4 +<br> > include/net/net.h | 2 +<br> > meson.build | 3 +<br> > net/tap-bsd.c | 5 +<br> > net/tap-linux.c | 19 ++<br> > net/tap-solaris.c | 5 +<br> > net/tap-stub.c | 5 +<br> > net/tap.c | 9 +<br> > net/tap_int.h | 1 +<br> > net/vhost-vdpa.c | 2 +<br> > 28 files changed, 1889 insertions(+), 4 deletions(-)<br> > create mode 100644 docs/ebpf.rst<br> > create mode 100644 docs/ebpf_rss.rst<br> > create mode 100644 ebpf/EbpfElf_to_C.py<br> > create mode 100755 ebpf/Makefile.ebpf<br> > create mode 100644 ebpf/ebpf-stub.c<br> > create mode 100644 ebpf/ebpf.c<br> > create mode 100644 ebpf/ebpf.h<br> > create mode 100644 ebpf/ebpf_rss.c<br> > create mode 100644 ebpf/ebpf_rss.h<br> > create mode 100644 ebpf/meson.build<br> > create mode 100644 ebpf/rss.bpf.c<br> > create mode 100644 ebpf/trace-events<br> > create mode 100644 ebpf/trace.h<br> > create mode 100644 ebpf/tun_rss_steering.h<br> ><br> <br> </blockquote></div></div>
On Tue, Nov 03, 2020 at 12:32:43PM +0200, Yuri Benditovich wrote: > On Tue, Nov 3, 2020 at 11:02 AM Jason Wang <jasowang@redhat.com> wrote: > > > > > On 2020/11/3 上午2:51, Andrew Melnychenko wrote: > > > Basic idea is to use eBPF to calculate and steer packets in TAP. > > > RSS(Receive Side Scaling) is used to distribute network packets to guest > > virtqueues > > > by calculating packet hash. > > > eBPF RSS allows us to use RSS with vhost TAP. > > > > > > This set of patches introduces the usage of eBPF for packet steering > > > and RSS hash calculation: > > > * RSS(Receive Side Scaling) is used to distribute network packets to > > > guest virtqueues by calculating packet hash > > > * eBPF RSS suppose to be faster than already existing 'software' > > > implementation in QEMU > > > * Additionally adding support for the usage of RSS with vhost > > > > > > Supported kernels: 5.8+ > > > > > > Implementation notes: > > > Linux TAP TUNSETSTEERINGEBPF ioctl was used to set the eBPF program. > > > Added eBPF support to qemu directly through a system call, see the > > > bpf(2) for details. > > > The eBPF program is part of the qemu and presented as an array of bpf > > > instructions. > > > The program can be recompiled by provided Makefile.ebpf(need to adjust > > > 'linuxhdrs'), > > > although it's not required to build QEMU with eBPF support. > > > Added changes to virtio-net and vhost, primary eBPF RSS is used. > > > 'Software' RSS used in the case of hash population and as a fallback > > option. > > > For vhost, the hash population feature is not reported to the guest. > > > > > > Please also see the documentation in PATCH 6/6. > > > > > > I am sending those patches as RFC to initiate the discussions and get > > > feedback on the following points: > > > * Fallback when eBPF is not supported by the kernel > > > > > > Yes, and it could also a lacking of CAP_BPF. > > > > > > > * Live migration to the kernel that doesn't have eBPF support > > > > > > Is there anything that we needs special treatment here? > > > > Possible case: rss=on, vhost=on, source system with kernel 5.8 (everything > works) -> dest. system 5.6 (bpf does not work), the adapter functions, but > all the steering does not use proper queues. > > > > > > > > > * Integration with current QEMU build > > > > > > Yes, a question here: > > > > 1) Any reason for not using libbpf, e.g it has been shipped with some > > distros > > > > We intentionally do not use libbpf, as it present only on some distros. > We can switch to libbpf, but this will disable bpf if libbpf is not > installed If we were modifying existing funtionality then introducing a dep on libbpf would be a problem as you'd be breaking existing QEMU users on distros without libbpf. This is brand new functionality though, so it is fine to place a requirement on libbpf. If distros don't ship that library and they want BPF features in QEMU, then those distros should take responsibility for adding libbpf to their package set. > > 2) It would be better if we can avoid shipping bytecodes > > > > > This creates new dependencies: llvm + clang + ... > We would prefer byte code and ability to generate it if prerequisites are > installed. I've double checked with Fedora, and generating the BPF program from source is a mandatory requirement for QEMU. Pre-generated BPF bytecode is not permitted. There was also a question raised about the kernel ABI compatibility for BPF programs ? https://lwn.net/Articles/831402/ "The basic problem is that when BPF is compiled, it uses a set of kernel headers that describe various kernel data structures for that particular version, which may be different from those on the kernel where the program is run. Until relatively recently, that was solved by distributing the BPF as C code along with the Clang compiler to build the BPF on the system where it was going to be run." Is this not an issue for QEMU's usage of BPF here ? The dependancy on llvm is unfortunate for people who build with GCC, but at least they can opt-out via a configure switch if they really want to. As that LWN article notes, GCC will gain BPF support Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On 2020/11/3 下午6:32, Yuri Benditovich wrote: > > > On Tue, Nov 3, 2020 at 11:02 AM Jason Wang <jasowang@redhat.com > <mailto:jasowang@redhat.com>> wrote: > > > On 2020/11/3 上午2:51, Andrew Melnychenko wrote: > > Basic idea is to use eBPF to calculate and steer packets in TAP. > > RSS(Receive Side Scaling) is used to distribute network packets > to guest virtqueues > > by calculating packet hash. > > eBPF RSS allows us to use RSS with vhost TAP. > > > > This set of patches introduces the usage of eBPF for packet steering > > and RSS hash calculation: > > * RSS(Receive Side Scaling) is used to distribute network packets to > > guest virtqueues by calculating packet hash > > * eBPF RSS suppose to be faster than already existing 'software' > > implementation in QEMU > > * Additionally adding support for the usage of RSS with vhost > > > > Supported kernels: 5.8+ > > > > Implementation notes: > > Linux TAP TUNSETSTEERINGEBPF ioctl was used to set the eBPF program. > > Added eBPF support to qemu directly through a system call, see the > > bpf(2) for details. > > The eBPF program is part of the qemu and presented as an array > of bpf > > instructions. > > The program can be recompiled by provided Makefile.ebpf(need to > adjust > > 'linuxhdrs'), > > although it's not required to build QEMU with eBPF support. > > Added changes to virtio-net and vhost, primary eBPF RSS is used. > > 'Software' RSS used in the case of hash population and as a > fallback option. > > For vhost, the hash population feature is not reported to the guest. > > > > Please also see the documentation in PATCH 6/6. > > > > I am sending those patches as RFC to initiate the discussions > and get > > feedback on the following points: > > * Fallback when eBPF is not supported by the kernel > > > Yes, and it could also a lacking of CAP_BPF. > > > > * Live migration to the kernel that doesn't have eBPF support > > > Is there anything that we needs special treatment here? > > Possible case: rss=on, vhost=on, source system with kernel 5.8 > (everything works) -> dest. system 5.6 (bpf does not work), the > adapter functions, but all the steering does not use proper queues. Right, I think we need to disable vhost on dest. > > > > > * Integration with current QEMU build > > > Yes, a question here: > > 1) Any reason for not using libbpf, e.g it has been shipped with some > distros > > > We intentionally do not use libbpf, as it present only on some distros. > We can switch to libbpf, but this will disable bpf if libbpf is not > installed That's better I think. > 2) It would be better if we can avoid shipping bytecodes > > > > This creates new dependencies: llvm + clang + ... > We would prefer byte code and ability to generate it if prerequisites > are installed. It's probably ok if we treat the bytecode as a kind of firmware. But in the long run, it's still worthwhile consider the qemu source is used for development and llvm/clang should be a common requirement for generating eBPF bytecode for host. > > > > * Additional usage for eBPF for packet filtering > > > Another interesting topics in to implement mac/vlan filters. And > in the > future, I plan to add mac based steering. All of these could be > done via > eBPF. > > > No problem, we can cooperate if needed > > > > > > Know issues: > > * hash population not supported by eBPF RSS: 'software' RSS used > > > Is this because there's not way to write to vnet header in > STERRING BPF? > > Yes. We plan to submit changes for kernel to cooperate with BPF and > populate the hash, this work is in progress That would require a new type of eBPF program and may need some work on verifier. Btw, macvtap is still lacking even steering ebpf program. Would you want to post a patch to support that? > > > as a fallback, also, hash population feature is not reported to > guests > > with vhost. > > * big-endian BPF support: for now, eBPF is disabled for > big-endian systems. > > > Are there any blocker for this? > > > No, can be added in v2 Cool. Thanks > > Just some quick questions after a glance of the codes. Will go > through > them tomorrow. > > Thanks > > > > > > Andrew (6): > > Added SetSteeringEBPF method for NetClientState. > > ebpf: Added basic eBPF API. > > ebpf: Added eBPF RSS program. > > ebpf: Added eBPF RSS loader. > > virtio-net: Added eBPF RSS to virtio-net. > > docs: Added eBPF documentation. > > > > MAINTAINERS | 6 + > > configure | 36 +++ > > docs/ebpf.rst | 29 ++ > > docs/ebpf_rss.rst | 129 ++++++++ > > ebpf/EbpfElf_to_C.py | 67 ++++ > > ebpf/Makefile.ebpf | 38 +++ > > ebpf/ebpf-stub.c | 28 ++ > > ebpf/ebpf.c | 107 +++++++ > > ebpf/ebpf.h | 35 +++ > > ebpf/ebpf_rss.c | 178 +++++++++++ > > ebpf/ebpf_rss.h | 30 ++ > > ebpf/meson.build | 1 + > > ebpf/rss.bpf.c | 470 ++++++++++++++++++++++++++++ > > ebpf/trace-events | 4 + > > ebpf/trace.h | 2 + > > ebpf/tun_rss_steering.h | 556 > +++++++++++++++++++++++++++++++++ > > hw/net/vhost_net.c | 2 + > > hw/net/virtio-net.c | 120 ++++++- > > include/hw/virtio/virtio-net.h | 4 + > > include/net/net.h | 2 + > > meson.build | 3 + > > net/tap-bsd.c | 5 + > > net/tap-linux.c | 19 ++ > > net/tap-solaris.c | 5 + > > net/tap-stub.c | 5 + > > net/tap.c | 9 + > > net/tap_int.h | 1 + > > net/vhost-vdpa.c | 2 + > > 28 files changed, 1889 insertions(+), 4 deletions(-) > > create mode 100644 docs/ebpf.rst > > create mode 100644 docs/ebpf_rss.rst > > create mode 100644 ebpf/EbpfElf_to_C.py > > create mode 100755 ebpf/Makefile.ebpf > > create mode 100644 ebpf/ebpf-stub.c > > create mode 100644 ebpf/ebpf.c > > create mode 100644 ebpf/ebpf.h > > create mode 100644 ebpf/ebpf_rss.c > > create mode 100644 ebpf/ebpf_rss.h > > create mode 100644 ebpf/meson.build > > create mode 100644 ebpf/rss.bpf.c > > create mode 100644 ebpf/trace-events > > create mode 100644 ebpf/trace.h > > create mode 100644 ebpf/tun_rss_steering.h > > >
On 2020/11/3 下午7:56, Daniel P. Berrangé wrote: > On Tue, Nov 03, 2020 at 12:32:43PM +0200, Yuri Benditovich wrote: >> On Tue, Nov 3, 2020 at 11:02 AM Jason Wang <jasowang@redhat.com> wrote: >> >>> On 2020/11/3 上午2:51, Andrew Melnychenko wrote: >>>> Basic idea is to use eBPF to calculate and steer packets in TAP. >>>> RSS(Receive Side Scaling) is used to distribute network packets to guest >>> virtqueues >>>> by calculating packet hash. >>>> eBPF RSS allows us to use RSS with vhost TAP. >>>> >>>> This set of patches introduces the usage of eBPF for packet steering >>>> and RSS hash calculation: >>>> * RSS(Receive Side Scaling) is used to distribute network packets to >>>> guest virtqueues by calculating packet hash >>>> * eBPF RSS suppose to be faster than already existing 'software' >>>> implementation in QEMU >>>> * Additionally adding support for the usage of RSS with vhost >>>> >>>> Supported kernels: 5.8+ >>>> >>>> Implementation notes: >>>> Linux TAP TUNSETSTEERINGEBPF ioctl was used to set the eBPF program. >>>> Added eBPF support to qemu directly through a system call, see the >>>> bpf(2) for details. >>>> The eBPF program is part of the qemu and presented as an array of bpf >>>> instructions. >>>> The program can be recompiled by provided Makefile.ebpf(need to adjust >>>> 'linuxhdrs'), >>>> although it's not required to build QEMU with eBPF support. >>>> Added changes to virtio-net and vhost, primary eBPF RSS is used. >>>> 'Software' RSS used in the case of hash population and as a fallback >>> option. >>>> For vhost, the hash population feature is not reported to the guest. >>>> >>>> Please also see the documentation in PATCH 6/6. >>>> >>>> I am sending those patches as RFC to initiate the discussions and get >>>> feedback on the following points: >>>> * Fallback when eBPF is not supported by the kernel >>> >>> Yes, and it could also a lacking of CAP_BPF. >>> >>> >>>> * Live migration to the kernel that doesn't have eBPF support >>> >>> Is there anything that we needs special treatment here? >>> >>> Possible case: rss=on, vhost=on, source system with kernel 5.8 (everything >> works) -> dest. system 5.6 (bpf does not work), the adapter functions, but >> all the steering does not use proper queues. >> >> >> >> >>>> * Integration with current QEMU build >>> >>> Yes, a question here: >>> >>> 1) Any reason for not using libbpf, e.g it has been shipped with some >>> distros >>> >> We intentionally do not use libbpf, as it present only on some distros. >> We can switch to libbpf, but this will disable bpf if libbpf is not >> installed > If we were modifying existing funtionality then introducing a dep on > libbpf would be a problem as you'd be breaking existing QEMU users > on distros without libbpf. > > This is brand new functionality though, so it is fine to place a > requirement on libbpf. If distros don't ship that library and they > want BPF features in QEMU, then those distros should take responsibility > for adding libbpf to their package set. > >>> 2) It would be better if we can avoid shipping bytecodes >>> >> >> This creates new dependencies: llvm + clang + ... >> We would prefer byte code and ability to generate it if prerequisites are >> installed. > I've double checked with Fedora, and generating the BPF program from > source is a mandatory requirement for QEMU. Pre-generated BPF bytecode > is not permitted. > > There was also a question raised about the kernel ABI compatibility > for BPF programs ? > > https://lwn.net/Articles/831402/ > > "The basic problem is that when BPF is compiled, it uses a set > of kernel headers that describe various kernel data structures > for that particular version, which may be different from those > on the kernel where the program is run. Until relatively recently, > that was solved by distributing the BPF as C code along with the > Clang compiler to build the BPF on the system where it was going > to be run." > > Is this not an issue for QEMU's usage of BPF here ? That's good point. Actually, DPDK ships RSS bytecodes but I don't know it works. But as mentioned in the link, if we generate the code with BTF that would be fine. Thanks > > The dependancy on llvm is unfortunate for people who build with GCC, > but at least they can opt-out via a configure switch if they really > want to. As that LWN article notes, GCC will gain BPF support > > > Regards, > Daniel
On 2020/11/3 上午2:51, Andrew Melnychenko wrote: > From: Andrew <andrew@daynix.com> > > For now, that method supported only by Linux TAP. > Linux TAP uses TUNSETSTEERINGEBPF ioctl. > TUNSETSTEERINGBPF was added 3 years ago. > Qemu checks if it was defined before using. > > Signed-off-by: Andrew Melnychenko <andrew@daynix.com> > --- > include/net/net.h | 2 ++ > net/tap-bsd.c | 5 +++++ > net/tap-linux.c | 19 +++++++++++++++++++ > net/tap-solaris.c | 5 +++++ > net/tap-stub.c | 5 +++++ > net/tap.c | 9 +++++++++ > net/tap_int.h | 1 + > 7 files changed, 46 insertions(+) > > diff --git a/include/net/net.h b/include/net/net.h > index 897b2d7595..d8a41fb010 100644 > --- a/include/net/net.h > +++ b/include/net/net.h > @@ -60,6 +60,7 @@ typedef int (SetVnetBE)(NetClientState *, bool); > typedef struct SocketReadState SocketReadState; > typedef void (SocketReadStateFinalize)(SocketReadState *rs); > typedef void (NetAnnounce)(NetClientState *); > +typedef bool (SetSteeringEBPF)(NetClientState *, int); > > typedef struct NetClientInfo { > NetClientDriver type; > @@ -81,6 +82,7 @@ typedef struct NetClientInfo { > SetVnetLE *set_vnet_le; > SetVnetBE *set_vnet_be; > NetAnnounce *announce; > + SetSteeringEBPF *set_steering_ebpf; > } NetClientInfo; > > struct NetClientState { > diff --git a/net/tap-bsd.c b/net/tap-bsd.c > index 77aaf674b1..4f64f31e98 100644 > --- a/net/tap-bsd.c > +++ b/net/tap-bsd.c > @@ -259,3 +259,8 @@ int tap_fd_get_ifname(int fd, char *ifname) > { > return -1; > } > + > +int tap_fd_set_steering_ebpf(int fd, int prog_fd) > +{ > + return -1; > +} > diff --git a/net/tap-linux.c b/net/tap-linux.c > index b0635e9e32..196373019f 100644 > --- a/net/tap-linux.c > +++ b/net/tap-linux.c > @@ -31,6 +31,7 @@ > > #include <net/if.h> > #include <sys/ioctl.h> > +#include <linux/if_tun.h> /* TUNSETSTEERINGEBPF */ > > #include "qapi/error.h" > #include "qemu/error-report.h" > @@ -316,3 +317,21 @@ int tap_fd_get_ifname(int fd, char *ifname) > pstrcpy(ifname, sizeof(ifr.ifr_name), ifr.ifr_name); > return 0; > } > + > +int tap_fd_set_steering_ebpf(int fd, int prog_fd) > +{ > +#ifdef TUNSETSTEERINGEBPF I'm not sure how much this can help. But looking at tap-linux.h, I wonder do we need to pull TUN/TAP uapi headers. Thanks > + if (ioctl(fd, TUNSETSTEERINGEBPF, (void *) &prog_fd) != 0) { > + error_report("Issue while setting TUNSETSTEERINGEBPF:" > + " %s with fd: %d, prog_fd: %d", > + strerror(errno), fd, prog_fd); > + > + return -1; > + } > + > + return 0; > +#else > + error_report("TUNSETSTEERINGEBPF is not supported"); > + return -1; > +#endif > +} > diff --git a/net/tap-solaris.c b/net/tap-solaris.c > index 0475a58207..d85224242b 100644 > --- a/net/tap-solaris.c > +++ b/net/tap-solaris.c > @@ -255,3 +255,8 @@ int tap_fd_get_ifname(int fd, char *ifname) > { > return -1; > } > + > +int tap_fd_set_steering_ebpf(int fd, int prog_fd) > +{ > + return -1; > +} > diff --git a/net/tap-stub.c b/net/tap-stub.c > index de525a2e69..a0fa25804b 100644 > --- a/net/tap-stub.c > +++ b/net/tap-stub.c > @@ -85,3 +85,8 @@ int tap_fd_get_ifname(int fd, char *ifname) > { > return -1; > } > + > +int tap_fd_set_steering_ebpf(int fd, int prog_fd) > +{ > + return -1; > +} > diff --git a/net/tap.c b/net/tap.c > index c46ff66184..81f50017bd 100644 > --- a/net/tap.c > +++ b/net/tap.c > @@ -337,6 +337,14 @@ static void tap_poll(NetClientState *nc, bool enable) > tap_write_poll(s, enable); > } > > +static bool tap_set_steering_ebpf(NetClientState *nc, int prog_fd) > +{ > + TAPState *s = DO_UPCAST(TAPState, nc, nc); > + assert(nc->info->type == NET_CLIENT_DRIVER_TAP); > + > + return tap_fd_set_steering_ebpf(s->fd, prog_fd) == 0; > +} > + > int tap_get_fd(NetClientState *nc) > { > TAPState *s = DO_UPCAST(TAPState, nc, nc); > @@ -362,6 +370,7 @@ static NetClientInfo net_tap_info = { > .set_vnet_hdr_len = tap_set_vnet_hdr_len, > .set_vnet_le = tap_set_vnet_le, > .set_vnet_be = tap_set_vnet_be, > + .set_steering_ebpf = tap_set_steering_ebpf, > }; > > static TAPState *net_tap_fd_init(NetClientState *peer, > diff --git a/net/tap_int.h b/net/tap_int.h > index 225a49ea48..547f8a5a28 100644 > --- a/net/tap_int.h > +++ b/net/tap_int.h > @@ -44,5 +44,6 @@ int tap_fd_set_vnet_be(int fd, int vnet_is_be); > int tap_fd_enable(int fd); > int tap_fd_disable(int fd); > int tap_fd_get_ifname(int fd, char *ifname); > +int tap_fd_set_steering_ebpf(int fd, int prog_fd); > > #endif /* NET_TAP_INT_H */
On 2020/11/3 上午2:51, Andrew Melnychenko wrote: > From: Andrew <andrew@daynix.com> > > Also, added maintainers information. > > Signed-off-by: Yuri Benditovich <yuri.benditovich@daynix.com> > Signed-off-by: Andrew Melnychenko <andrew@daynix.com> > --- > MAINTAINERS | 6 +++ > docs/ebpf.rst | 29 +++++++++++ > docs/ebpf_rss.rst | 129 ++++++++++++++++++++++++++++++++++++++++++++++ > 3 files changed, 164 insertions(+) > create mode 100644 docs/ebpf.rst > create mode 100644 docs/ebpf_rss.rst > > diff --git a/MAINTAINERS b/MAINTAINERS > index 2c22bbca5a..464b3f3c95 100644 > --- a/MAINTAINERS > +++ b/MAINTAINERS > @@ -3111,6 +3111,12 @@ S: Maintained > F: hw/semihosting/ > F: include/hw/semihosting/ > > +EBPF: > +M: Andrew Melnychenko <andrew@daynix.com> > +M: Yuri Benditovich <yuri.benditovich@daynix.com> > +S: Maintained > +F: ebpf/* > + > Build and test automation > ------------------------- > Build and test automation > diff --git a/docs/ebpf.rst b/docs/ebpf.rst > new file mode 100644 > index 0000000000..e45d085432 > --- /dev/null > +++ b/docs/ebpf.rst > @@ -0,0 +1,29 @@ > +=========================== > +eBPF qemu support > +=========================== > + > +eBPF support (CONFIG_EBPF) is enabled automatically by 'configure' script > +if 'bpf' system call is available. > +To disable eBPF support use './configure --disable-bpf' > + > +Basic eBPF functionality is located in ebpf/ebpf.c and ebpf/ebpf.h. > +There are basic functions to load the eBPF program into the kernel. > +Mostly, functions name are self-explanatory: > + > +- `bpf_create_map()`, `bpf_lookup_element()`, `bpf_update_element()`, `bpf_delete_element()` - manages eBPF maps. On error, a basic error message would be reported and returned -1. On success, 0 would be returned(`bpf_create_map()` returns map's file descriptor). > +- `bpf_prog_load()` - load the program. The program has to have proper map file descriptors if there are used. On error - the log eBPF would be reported. On success, the program file descriptor returned. > +- `bpf_fixup_mapfd()` - would place map file descriptor into the program according to 'relocate array' of 'struct fixup_mapfd_t'. The function would return how many instructions were 'fixed' aka how many relocations was occurred. > + > +Simplified workflow would look like this: > + > +.. code:: C > + > + int map1 = bpf_create_map(...); > + int map2 = bpf_create_map(...); > + > + bpf_fixup_mapfd(<fixup table>, ARRAY_SIZE(<fixup table>), <instructions pointer>, ARRAY_SIZE(<instructions pointer>), <map1 name>, map1); > + bpf_fixup_mapfd(<fixup table>, ARRAY_SIZE(<fixup table>), <instructions pointer>, ARRAY_SIZE(<instructions pointer>), <map2 name>, map2); > + > + int prog = bpf_prog_load(<program type>, <instructions pointer>, ARRAY_SIZE(<instructions pointer>), "GPL"); > + > +See the bpf(2) for details. > diff --git a/docs/ebpf_rss.rst b/docs/ebpf_rss.rst > new file mode 100644 > index 0000000000..96fee391b8 > --- /dev/null > +++ b/docs/ebpf_rss.rst > @@ -0,0 +1,129 @@ > +=========================== > +eBPF RSS virtio-net support > +=========================== > + > +RSS(Receive Side Scaling) is used to distribute network packets to guest virtqueues > +by calculating packet hash. Usually every queue is processed then by a specific guest CPU core. > + > +For now there are 2 RSS implementations in qemu: > +- 'software' RSS (functions if qemu receives network packets, i.e. vhost=off) > +- eBPF RSS (can function with also with vhost=on) > + > +If steering BPF is not set for kernel's TUN module, the TUN uses automatic selection > +of rx virtqueue based on lookup table built according to calculated symmetric hash > +of transmitted packets. > +If steering BPF is set for TUN the BPF code calculates the hash of packet header and > +returns the virtqueue number to place the packet to. > + > +Simplified decision formula: > + > +.. code:: C > + > + queue_index = indirection_table[hash(<packet data>)%<indirection_table size>] > + > + > +Not for all packets, the hash can/should be calculated. > + > +Note: currently, eBPF RSS does not support hash reporting. > + > +eBPF RSS turned on by different combinations of vhost-net, vitrio-net and tap configurations: > + > +- eBPF is used: > + > + tap,vhost=off & virtio-net-pci,rss=on,hash=off > + > +- eBPF is used: > + > + tap,vhost=on & virtio-net-pci,rss=on,hash=off > + > +- 'software' RSS is used: > + > + tap,vhost=off & virtio-net-pci,rss=on,hash=on > + > +- eBPF is used, hash population feature is not reported to the guest: > + > + tap,vhost=on & virtio-net-pci,rss=on,hash=on > + > +If CONFIG_EBPF is not set then only 'software' RSS is supported. > +Also 'software' RSS, as a fallback, is used if the eBPF program failed to load or set to TUN. > + > +RSS eBPF program > +---------------- > + > +RSS program located in ebpf/tun_rss_steering.h as an array of 'struct bpf_insn'. > +So the program is part of the qemu binary. > +Initially, the eBPF program was compiled by clang and source code located at ebpf/rss.bpf.c. > +Prerequisites to recompile the eBPF program (regenerate ebpf/tun_rss_steering.h): > + > + llvm, clang, kernel source tree, python3 + (pip3 pyelftools) > + Adjust 'linuxhdrs' in Makefile.ebpf to reflect the location of the kernel source tree > + > + $ cd ebpf > + $ make -f Makefile.ebpf > + > +Note the python script for convertation from eBPF ELF object to '.h' file - Ebpf_to_C.py: > + > + $ python EbpfElf_to_C.py rss.bpf.o tun_rss_steering > + > +The first argument of the script is ELF object, second - section name where the eBPF program located. > +The script would generate <section name>.h file with eBPF instructions and 'relocate array'. > +'relocate array' is an array of 'struct fixup_mapfd_t' with the name of the eBPF map and instruction offset where the file descriptor of the map should be placed. > + Do we still need this if we decide to use llvm/clang toolchain? (I guess not) Thanks > +Current eBPF RSS implementation uses 'bounded loops' with 'backward jump instructions' which present in the last kernels. > +Overall eBPF RSS works on kernels 5.8+. > + > +eBPF RSS implementation > +----------------------- > + > +eBPF RSS loading functionality located in ebpf/ebpf_rss.c and ebpf/ebpf_rss.h. > + > +The `struct EBPFRSSContext` structure that holds 4 file descriptors: > + > +- program_fd - file descriptor of the eBPF RSS program. > +- map_configuration - file descriptor of the 'configuration' map. This map contains one element of 'struct EBPFRSSConfig'. This configuration determines eBPF program behavior. > +- map_toeplitz_key - file descriptor of the 'Toeplitz key' map. One element of the 40byte key prepared for the hashing algorithm. > +- map_indirections_table - 128 elements of queue indexes. > + > +`struct EBPFRSSConfig` fields: > + > +- redirect - "boolean" value, should the hash be calculated, on false - `default_queue` would be used as the final decision. > +- populate_hash - for now, not used. eBPF RSS doesn't support hash reporting. > +- hash_types - binary mask of different hash types. See `VIRTIO_NET_RSS_HASH_TYPE_*` defines. If for packet hash should not be calculated - `default_queue` would be used. > +- indirections_len - length of the indirections table, maximum 128. > +- default_queue - the queue index that used for packet that shouldn't be hashed. For some packets, the hash can't be calculated(g.e ARP). > + > +Functions: > + > +- `ebpf_rss_init()` - sets program_fd to -1, which indicates that EBPFRSSContext is not loaded. > +- `ebpf_rss_load()` - creates 3 maps and loads eBPF program from tun_rss_steering.h. Returns 'true' on success. After that, program_fd can be used to set steering for TAP. > +- `ebpf_rss_set_all()` - sets values for eBPF maps. `indirections_table` length is in EBPFRSSConfig. `toeplitz_key` is VIRTIO_NET_RSS_MAX_KEY_SIZE aka 40 bytes array. > +- `ebpf_rss_unload()` - close all file descriptors and set program_fd to -1. > + > +Simplified eBPF RSS workflow: > + > +.. code:: C > + > + struct EBPFRSSConfig config; > + config.redirect = 1; > + config.hash_types = VIRTIO_NET_RSS_HASH_TYPE_UDPv4 | VIRTIO_NET_RSS_HASH_TYPE_TCPv4; > + config.indirections_len = VIRTIO_NET_RSS_MAX_TABLE_LEN; > + config.default_queue = 0; > + > + uint16_t table[VIRTIO_NET_RSS_MAX_TABLE_LEN] = {...}; > + uint8_t key[VIRTIO_NET_RSS_MAX_KEY_SIZE] = {...}; > + > + struct EBPFRSSContext ctx; > + ebpf_rss_init(&ctx); > + ebpf_rss_load(&ctx); > + ebpf_rss_set_all(&ctx, &config, table, key); > + if (net_client->info->set_steering_ebpf != NULL) { > + net_client->info->set_steering_ebpf(net_client, ctx->program_fd); > + } > + ... > + ebpf_unload(&ctx); > + > + > +NetClientState SetSteeringEBPF() > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +For now, `set_steering_ebpf()` method supported by Linux TAP NetClientState. The method requires an eBPF program file descriptor as an argument.
On Wed, Nov 04, 2020 at 10:07:52AM +0800, Jason Wang wrote: > > On 2020/11/3 下午6:32, Yuri Benditovich wrote: > > > > > > On Tue, Nov 3, 2020 at 11:02 AM Jason Wang <jasowang@redhat.com > > <mailto:jasowang@redhat.com>> wrote: > > > > > > On 2020/11/3 上午2:51, Andrew Melnychenko wrote: > > > Basic idea is to use eBPF to calculate and steer packets in TAP. > > > RSS(Receive Side Scaling) is used to distribute network packets > > to guest virtqueues > > > by calculating packet hash. > > > eBPF RSS allows us to use RSS with vhost TAP. > > > > > > This set of patches introduces the usage of eBPF for packet steering > > > and RSS hash calculation: > > > * RSS(Receive Side Scaling) is used to distribute network packets to > > > guest virtqueues by calculating packet hash > > > * eBPF RSS suppose to be faster than already existing 'software' > > > implementation in QEMU > > > * Additionally adding support for the usage of RSS with vhost > > > > > > Supported kernels: 5.8+ > > > > > > Implementation notes: > > > Linux TAP TUNSETSTEERINGEBPF ioctl was used to set the eBPF program. > > > Added eBPF support to qemu directly through a system call, see the > > > bpf(2) for details. > > > The eBPF program is part of the qemu and presented as an array > > of bpf > > > instructions. > > > The program can be recompiled by provided Makefile.ebpf(need to > > adjust > > > 'linuxhdrs'), > > > although it's not required to build QEMU with eBPF support. > > > Added changes to virtio-net and vhost, primary eBPF RSS is used. > > > 'Software' RSS used in the case of hash population and as a > > fallback option. > > > For vhost, the hash population feature is not reported to the guest. > > > > > > Please also see the documentation in PATCH 6/6. > > > > > > I am sending those patches as RFC to initiate the discussions > > and get > > > feedback on the following points: > > > * Fallback when eBPF is not supported by the kernel > > > > > > Yes, and it could also a lacking of CAP_BPF. > > > > > > > * Live migration to the kernel that doesn't have eBPF support > > > > > > Is there anything that we needs special treatment here? > > > > Possible case: rss=on, vhost=on, source system with kernel 5.8 > > (everything works) -> dest. system 5.6 (bpf does not work), the adapter > > functions, but all the steering does not use proper queues. > > > Right, I think we need to disable vhost on dest. > > > > > > > > > > > * Integration with current QEMU build > > > > > > Yes, a question here: > > > > 1) Any reason for not using libbpf, e.g it has been shipped with some > > distros > > > > > > We intentionally do not use libbpf, as it present only on some distros. > > We can switch to libbpf, but this will disable bpf if libbpf is not > > installed > > > That's better I think. > > > > 2) It would be better if we can avoid shipping bytecodes > > > > > > > > This creates new dependencies: llvm + clang + ... > > We would prefer byte code and ability to generate it if prerequisites > > are installed. > > > It's probably ok if we treat the bytecode as a kind of firmware. That is explicitly *not* OK for inclusion in Fedora. They require that BPF is compiled from source, and rejected my suggestion that it could be considered a kind of firmware and thus have an exception from building from source. > But in the long run, it's still worthwhile consider the qemu source is used > for development and llvm/clang should be a common requirement for generating > eBPF bytecode for host. So we need to do this right straight way before this merges. Regards, Daniel
On Wed, Nov 4, 2020 at 4:49 AM Jason Wang <jasowang@redhat.com> wrote: > > On 2020/11/3 上午2:51, Andrew Melnychenko wrote: > > From: Andrew <andrew@daynix.com> > > > > For now, that method supported only by Linux TAP. > > Linux TAP uses TUNSETSTEERINGEBPF ioctl. > > TUNSETSTEERINGBPF was added 3 years ago. > > Qemu checks if it was defined before using. > > > > Signed-off-by: Andrew Melnychenko <andrew@daynix.com> > > --- > > include/net/net.h | 2 ++ > > net/tap-bsd.c | 5 +++++ > > net/tap-linux.c | 19 +++++++++++++++++++ > > net/tap-solaris.c | 5 +++++ > > net/tap-stub.c | 5 +++++ > > net/tap.c | 9 +++++++++ > > net/tap_int.h | 1 + > > 7 files changed, 46 insertions(+) > > > > diff --git a/include/net/net.h b/include/net/net.h > > index 897b2d7595..d8a41fb010 100644 > > --- a/include/net/net.h > > +++ b/include/net/net.h > > @@ -60,6 +60,7 @@ typedef int (SetVnetBE)(NetClientState *, bool); > > typedef struct SocketReadState SocketReadState; > > typedef void (SocketReadStateFinalize)(SocketReadState *rs); > > typedef void (NetAnnounce)(NetClientState *); > > +typedef bool (SetSteeringEBPF)(NetClientState *, int); > > > > typedef struct NetClientInfo { > > NetClientDriver type; > > @@ -81,6 +82,7 @@ typedef struct NetClientInfo { > > SetVnetLE *set_vnet_le; > > SetVnetBE *set_vnet_be; > > NetAnnounce *announce; > > + SetSteeringEBPF *set_steering_ebpf; > > } NetClientInfo; > > > > struct NetClientState { > > diff --git a/net/tap-bsd.c b/net/tap-bsd.c > > index 77aaf674b1..4f64f31e98 100644 > > --- a/net/tap-bsd.c > > +++ b/net/tap-bsd.c > > @@ -259,3 +259,8 @@ int tap_fd_get_ifname(int fd, char *ifname) > > { > > return -1; > > } > > + > > +int tap_fd_set_steering_ebpf(int fd, int prog_fd) > > +{ > > + return -1; > > +} > > diff --git a/net/tap-linux.c b/net/tap-linux.c > > index b0635e9e32..196373019f 100644 > > --- a/net/tap-linux.c > > +++ b/net/tap-linux.c > > @@ -31,6 +31,7 @@ > > > > #include <net/if.h> > > #include <sys/ioctl.h> > > +#include <linux/if_tun.h> /* TUNSETSTEERINGEBPF */ > > > > #include "qapi/error.h" > > #include "qemu/error-report.h" > > @@ -316,3 +317,21 @@ int tap_fd_get_ifname(int fd, char *ifname) > > pstrcpy(ifname, sizeof(ifr.ifr_name), ifr.ifr_name); > > return 0; > > } > > + > > +int tap_fd_set_steering_ebpf(int fd, int prog_fd) > > +{ > > +#ifdef TUNSETSTEERINGEBPF > > > I'm not sure how much this can help. > > But looking at tap-linux.h, I wonder do we need to pull TUN/TAP uapi > headers. > > Thanks > Agree, we just need to add this define to tap-linux.h > > > > + if (ioctl(fd, TUNSETSTEERINGEBPF, (void *) &prog_fd) != 0) { > > + error_report("Issue while setting TUNSETSTEERINGEBPF:" > > + " %s with fd: %d, prog_fd: %d", > > + strerror(errno), fd, prog_fd); > > + > > + return -1; > > + } > > + > > + return 0; > > +#else > > + error_report("TUNSETSTEERINGEBPF is not supported"); > > + return -1; > > +#endif > > +} > > diff --git a/net/tap-solaris.c b/net/tap-solaris.c > > index 0475a58207..d85224242b 100644 > > --- a/net/tap-solaris.c > > +++ b/net/tap-solaris.c > > @@ -255,3 +255,8 @@ int tap_fd_get_ifname(int fd, char *ifname) > > { > > return -1; > > } > > + > > +int tap_fd_set_steering_ebpf(int fd, int prog_fd) > > +{ > > + return -1; > > +} > > diff --git a/net/tap-stub.c b/net/tap-stub.c > > index de525a2e69..a0fa25804b 100644 > > --- a/net/tap-stub.c > > +++ b/net/tap-stub.c > > @@ -85,3 +85,8 @@ int tap_fd_get_ifname(int fd, char *ifname) > > { > > return -1; > > } > > + > > +int tap_fd_set_steering_ebpf(int fd, int prog_fd) > > +{ > > + return -1; > > +} > > diff --git a/net/tap.c b/net/tap.c > > index c46ff66184..81f50017bd 100644 > > --- a/net/tap.c > > +++ b/net/tap.c > > @@ -337,6 +337,14 @@ static void tap_poll(NetClientState *nc, bool > enable) > > tap_write_poll(s, enable); > > } > > > > +static bool tap_set_steering_ebpf(NetClientState *nc, int prog_fd) > > +{ > > + TAPState *s = DO_UPCAST(TAPState, nc, nc); > > + assert(nc->info->type == NET_CLIENT_DRIVER_TAP); > > + > > + return tap_fd_set_steering_ebpf(s->fd, prog_fd) == 0; > > +} > > + > > int tap_get_fd(NetClientState *nc) > > { > > TAPState *s = DO_UPCAST(TAPState, nc, nc); > > @@ -362,6 +370,7 @@ static NetClientInfo net_tap_info = { > > .set_vnet_hdr_len = tap_set_vnet_hdr_len, > > .set_vnet_le = tap_set_vnet_le, > > .set_vnet_be = tap_set_vnet_be, > > + .set_steering_ebpf = tap_set_steering_ebpf, > > }; > > > > static TAPState *net_tap_fd_init(NetClientState *peer, > > diff --git a/net/tap_int.h b/net/tap_int.h > > index 225a49ea48..547f8a5a28 100644 > > --- a/net/tap_int.h > > +++ b/net/tap_int.h > > @@ -44,5 +44,6 @@ int tap_fd_set_vnet_be(int fd, int vnet_is_be); > > int tap_fd_enable(int fd); > > int tap_fd_disable(int fd); > > int tap_fd_get_ifname(int fd, char *ifname); > > +int tap_fd_set_steering_ebpf(int fd, int prog_fd); > > > > #endif /* NET_TAP_INT_H */ > > <div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Nov 4, 2020 at 4:49 AM Jason Wang <<a href="mailto:jasowang@redhat.com">jasowang@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br> On 2020/11/3 上午2:51, Andrew Melnychenko wrote:<br> > From: Andrew <<a href="mailto:andrew@daynix.com" target="_blank">andrew@daynix.com</a>><br> ><br> > For now, that method supported only by Linux TAP.<br> > Linux TAP uses TUNSETSTEERINGEBPF ioctl.<br> > TUNSETSTEERINGBPF was added 3 years ago.<br> > Qemu checks if it was defined before using.<br> ><br> > Signed-off-by: Andrew Melnychenko <<a href="mailto:andrew@daynix.com" target="_blank">andrew@daynix.com</a>><br> > ---<br> > include/net/net.h | 2 ++<br> > net/tap-bsd.c | 5 +++++<br> > net/tap-linux.c | 19 +++++++++++++++++++<br> > net/tap-solaris.c | 5 +++++<br> > net/tap-stub.c | 5 +++++<br> > net/tap.c | 9 +++++++++<br> > net/tap_int.h | 1 +<br> > 7 files changed, 46 insertions(+)<br> ><br> > diff --git a/include/net/net.h b/include/net/net.h<br> > index 897b2d7595..d8a41fb010 100644<br> > --- a/include/net/net.h<br> > +++ b/include/net/net.h<br> > @@ -60,6 +60,7 @@ typedef int (SetVnetBE)(NetClientState *, bool);<br> > typedef struct SocketReadState SocketReadState;<br> > typedef void (SocketReadStateFinalize)(SocketReadState *rs);<br> > typedef void (NetAnnounce)(NetClientState *);<br> > +typedef bool (SetSteeringEBPF)(NetClientState *, int);<br> > <br> > typedef struct NetClientInfo {<br> > NetClientDriver type;<br> > @@ -81,6 +82,7 @@ typedef struct NetClientInfo {<br> > SetVnetLE *set_vnet_le;<br> > SetVnetBE *set_vnet_be;<br> > NetAnnounce *announce;<br> > + SetSteeringEBPF *set_steering_ebpf;<br> > } NetClientInfo;<br> > <br> > struct NetClientState {<br> > diff --git a/net/tap-bsd.c b/net/tap-bsd.c<br> > index 77aaf674b1..4f64f31e98 100644<br> > --- a/net/tap-bsd.c<br> > +++ b/net/tap-bsd.c<br> > @@ -259,3 +259,8 @@ int tap_fd_get_ifname(int fd, char *ifname)<br> > {<br> > return -1;<br> > }<br> > +<br> > +int tap_fd_set_steering_ebpf(int fd, int prog_fd)<br> > +{<br> > + return -1;<br> > +}<br> > diff --git a/net/tap-linux.c b/net/tap-linux.c<br> > index b0635e9e32..196373019f 100644<br> > --- a/net/tap-linux.c<br> > +++ b/net/tap-linux.c<br> > @@ -31,6 +31,7 @@<br> > <br> > #include <net/if.h><br> > #include <sys/ioctl.h><br> > +#include <linux/if_tun.h> /* TUNSETSTEERINGEBPF */<br> > <br> > #include "qapi/error.h"<br> > #include "qemu/error-report.h"<br> > @@ -316,3 +317,21 @@ int tap_fd_get_ifname(int fd, char *ifname)<br> > pstrcpy(ifname, sizeof(ifr.ifr_name), ifr.ifr_name);<br> > return 0;<br> > }<br> > +<br> > +int tap_fd_set_steering_ebpf(int fd, int prog_fd)<br> > +{<br> > +#ifdef TUNSETSTEERINGEBPF<br> <br> <br> I'm not sure how much this can help.<br> <br> But looking at tap-linux.h, I wonder do we need to pull TUN/TAP uapi <br> headers.<br> <br> Thanks<br></blockquote><div><br></div><div>Agree, we just need to add this define to tap-linux.h</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br> <br> > + if (ioctl(fd, TUNSETSTEERINGEBPF, (void *) &prog_fd) != 0) {<br> > + error_report("Issue while setting TUNSETSTEERINGEBPF:"<br> > + " %s with fd: %d, prog_fd: %d",<br> > + strerror(errno), fd, prog_fd);<br> > +<br> > + return -1;<br> > + }<br> > +<br> > + return 0;<br> > +#else<br> > + error_report("TUNSETSTEERINGEBPF is not supported");<br> > + return -1;<br> > +#endif<br> > +}<br> > diff --git a/net/tap-solaris.c b/net/tap-solaris.c<br> > index 0475a58207..d85224242b 100644<br> > --- a/net/tap-solaris.c<br> > +++ b/net/tap-solaris.c<br> > @@ -255,3 +255,8 @@ int tap_fd_get_ifname(int fd, char *ifname)<br> > {<br> > return -1;<br> > }<br> > +<br> > +int tap_fd_set_steering_ebpf(int fd, int prog_fd)<br> > +{<br> > + return -1;<br> > +}<br> > diff --git a/net/tap-stub.c b/net/tap-stub.c<br> > index de525a2e69..a0fa25804b 100644<br> > --- a/net/tap-stub.c<br> > +++ b/net/tap-stub.c<br> > @@ -85,3 +85,8 @@ int tap_fd_get_ifname(int fd, char *ifname)<br> > {<br> > return -1;<br> > }<br> > +<br> > +int tap_fd_set_steering_ebpf(int fd, int prog_fd)<br> > +{<br> > + return -1;<br> > +}<br> > diff --git a/net/tap.c b/net/tap.c<br> > index c46ff66184..81f50017bd 100644<br> > --- a/net/tap.c<br> > +++ b/net/tap.c<br> > @@ -337,6 +337,14 @@ static void tap_poll(NetClientState *nc, bool enable)<br> > tap_write_poll(s, enable);<br> > }<br> > <br> > +static bool tap_set_steering_ebpf(NetClientState *nc, int prog_fd)<br> > +{<br> > + TAPState *s = DO_UPCAST(TAPState, nc, nc);<br> > + assert(nc->info->type == NET_CLIENT_DRIVER_TAP);<br> > +<br> > + return tap_fd_set_steering_ebpf(s->fd, prog_fd) == 0;<br> > +}<br> > +<br> > int tap_get_fd(NetClientState *nc)<br> > {<br> > TAPState *s = DO_UPCAST(TAPState, nc, nc);<br> > @@ -362,6 +370,7 @@ static NetClientInfo net_tap_info = {<br> > .set_vnet_hdr_len = tap_set_vnet_hdr_len,<br> > .set_vnet_le = tap_set_vnet_le,<br> > .set_vnet_be = tap_set_vnet_be,<br> > + .set_steering_ebpf = tap_set_steering_ebpf,<br> > };<br> > <br> > static TAPState *net_tap_fd_init(NetClientState *peer,<br> > diff --git a/net/tap_int.h b/net/tap_int.h<br> > index 225a49ea48..547f8a5a28 100644<br> > --- a/net/tap_int.h<br> > +++ b/net/tap_int.h<br> > @@ -44,5 +44,6 @@ int tap_fd_set_vnet_be(int fd, int vnet_is_be);<br> > int tap_fd_enable(int fd);<br> > int tap_fd_disable(int fd);<br> > int tap_fd_get_ifname(int fd, char *ifname);<br> > +int tap_fd_set_steering_ebpf(int fd, int prog_fd);<br> > <br> > #endif /* NET_TAP_INT_H */<br> <br> </blockquote></div></div>
On Wed, Nov 4, 2020 at 4:08 AM Jason Wang <jasowang@redhat.com> wrote: > > On 2020/11/3 下午6:32, Yuri Benditovich wrote: > > > > > > On Tue, Nov 3, 2020 at 11:02 AM Jason Wang <jasowang@redhat.com > > <mailto:jasowang@redhat.com>> wrote: > > > > > > On 2020/11/3 上午2:51, Andrew Melnychenko wrote: > > > Basic idea is to use eBPF to calculate and steer packets in TAP. > > > RSS(Receive Side Scaling) is used to distribute network packets > > to guest virtqueues > > > by calculating packet hash. > > > eBPF RSS allows us to use RSS with vhost TAP. > > > > > > This set of patches introduces the usage of eBPF for packet > steering > > > and RSS hash calculation: > > > * RSS(Receive Side Scaling) is used to distribute network packets > to > > > guest virtqueues by calculating packet hash > > > * eBPF RSS suppose to be faster than already existing 'software' > > > implementation in QEMU > > > * Additionally adding support for the usage of RSS with vhost > > > > > > Supported kernels: 5.8+ > > > > > > Implementation notes: > > > Linux TAP TUNSETSTEERINGEBPF ioctl was used to set the eBPF > program. > > > Added eBPF support to qemu directly through a system call, see the > > > bpf(2) for details. > > > The eBPF program is part of the qemu and presented as an array > > of bpf > > > instructions. > > > The program can be recompiled by provided Makefile.ebpf(need to > > adjust > > > 'linuxhdrs'), > > > although it's not required to build QEMU with eBPF support. > > > Added changes to virtio-net and vhost, primary eBPF RSS is used. > > > 'Software' RSS used in the case of hash population and as a > > fallback option. > > > For vhost, the hash population feature is not reported to the > guest. > > > > > > Please also see the documentation in PATCH 6/6. > > > > > > I am sending those patches as RFC to initiate the discussions > > and get > > > feedback on the following points: > > > * Fallback when eBPF is not supported by the kernel > > > > > > Yes, and it could also a lacking of CAP_BPF. > > > > > > > * Live migration to the kernel that doesn't have eBPF support > > > > > > Is there anything that we needs special treatment here? > > > > Possible case: rss=on, vhost=on, source system with kernel 5.8 > > (everything works) -> dest. system 5.6 (bpf does not work), the > > adapter functions, but all the steering does not use proper queues. > > > Right, I think we need to disable vhost on dest. > > Is this acceptable to disable vhost at time of migration? > > > > > > > > > * Integration with current QEMU build > > > > > > Yes, a question here: > > > > 1) Any reason for not using libbpf, e.g it has been shipped with some > > distros > > > > > > We intentionally do not use libbpf, as it present only on some distros. > > We can switch to libbpf, but this will disable bpf if libbpf is not > > installed > > > That's better I think. > We think the preferred way is to have an eBPF code built-in in QEMU (not distribute it as a separate file). Our initial idea was to not use the libbpf because it: 1. Does not create additional dependency during build time and during run-time 2. Gives us smaller footprint of loadable eBPF blob inside qemu 3. Do not add too much code to QEMU We can switch to libbpf, in this case: 1. Presence of dynamic library is not guaranteed on the target system 2. Static library is large 3. libbpf uses eBPF ELF which is significantly bigger than just the array or instructions (May be we succeed to reduce the ELF to some suitable size and still have it built-in) Please let us know whether you still think libbpf is better and why. Thanks > > > 2) It would be better if we can avoid shipping bytecodes > > > > > > > > This creates new dependencies: llvm + clang + ... > > We would prefer byte code and ability to generate it if prerequisites > > are installed. > > > It's probably ok if we treat the bytecode as a kind of firmware. > > But in the long run, it's still worthwhile consider the qemu source is > used for development and llvm/clang should be a common requirement for > generating eBPF bytecode for host. > > > > > > > > > * Additional usage for eBPF for packet filtering > > > > > > Another interesting topics in to implement mac/vlan filters. And > > in the > > future, I plan to add mac based steering. All of these could be > > done via > > eBPF. > > > > > > No problem, we can cooperate if needed > > > > > > > > > > Know issues: > > > * hash population not supported by eBPF RSS: 'software' RSS used > > > > > > Is this because there's not way to write to vnet header in > > STERRING BPF? > > > > Yes. We plan to submit changes for kernel to cooperate with BPF and > > populate the hash, this work is in progress > > > That would require a new type of eBPF program and may need some work on > verifier. > > May be need to allow loading of an additional type in tun.c, not only socket filter (to use bpf_set_hash) Also vhost and tun in kernel need to be aware of header extension for hash population. > Btw, macvtap is still lacking even steering ebpf program. Would you want > to post a patch to support that? > > Probably after we have full functioning BPF with TAP/TUN > > > > > > as a fallback, also, hash population feature is not reported to > > guests > > > with vhost. > > > * big-endian BPF support: for now, eBPF is disabled for > > big-endian systems. > > > > > > Are there any blocker for this? > > > > > > No, can be added in v2 > > > Cool. > > Thanks > > > > > > Just some quick questions after a glance of the codes. Will go > > through > > them tomorrow. > > > > Thanks > > > > > > > > > > Andrew (6): > > > Added SetSteeringEBPF method for NetClientState. > > > ebpf: Added basic eBPF API. > > > ebpf: Added eBPF RSS program. > > > ebpf: Added eBPF RSS loader. > > > virtio-net: Added eBPF RSS to virtio-net. > > > docs: Added eBPF documentation. > > > > > > MAINTAINERS | 6 + > > > configure | 36 +++ > > > docs/ebpf.rst | 29 ++ > > > docs/ebpf_rss.rst | 129 ++++++++ > > > ebpf/EbpfElf_to_C.py | 67 ++++ > > > ebpf/Makefile.ebpf | 38 +++ > > > ebpf/ebpf-stub.c | 28 ++ > > > ebpf/ebpf.c | 107 +++++++ > > > ebpf/ebpf.h | 35 +++ > > > ebpf/ebpf_rss.c | 178 +++++++++++ > > > ebpf/ebpf_rss.h | 30 ++ > > > ebpf/meson.build | 1 + > > > ebpf/rss.bpf.c | 470 ++++++++++++++++++++++++++++ > > > ebpf/trace-events | 4 + > > > ebpf/trace.h | 2 + > > > ebpf/tun_rss_steering.h | 556 > > +++++++++++++++++++++++++++++++++ > > > hw/net/vhost_net.c | 2 + > > > hw/net/virtio-net.c | 120 ++++++- > > > include/hw/virtio/virtio-net.h | 4 + > > > include/net/net.h | 2 + > > > meson.build | 3 + > > > net/tap-bsd.c | 5 + > > > net/tap-linux.c | 19 ++ > > > net/tap-solaris.c | 5 + > > > net/tap-stub.c | 5 + > > > net/tap.c | 9 + > > > net/tap_int.h | 1 + > > > net/vhost-vdpa.c | 2 + > > > 28 files changed, 1889 insertions(+), 4 deletions(-) > > > create mode 100644 docs/ebpf.rst > > > create mode 100644 docs/ebpf_rss.rst > > > create mode 100644 ebpf/EbpfElf_to_C.py > > > create mode 100755 ebpf/Makefile.ebpf > > > create mode 100644 ebpf/ebpf-stub.c > > > create mode 100644 ebpf/ebpf.c > > > create mode 100644 ebpf/ebpf.h > > > create mode 100644 ebpf/ebpf_rss.c > > > create mode 100644 ebpf/ebpf_rss.h > > > create mode 100644 ebpf/meson.build > > > create mode 100644 ebpf/rss.bpf.c > > > create mode 100644 ebpf/trace-events > > > create mode 100644 ebpf/trace.h > > > create mode 100644 ebpf/tun_rss_steering.h > > > > > > > <div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Nov 4, 2020 at 4:08 AM Jason Wang <<a href="mailto:jasowang@redhat.com">jasowang@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br> On 2020/11/3 下午6:32, Yuri Benditovich wrote:<br> ><br> ><br> > On Tue, Nov 3, 2020 at 11:02 AM Jason Wang <<a href="mailto:jasowang@redhat.com" target="_blank">jasowang@redhat.com</a> <br> > <mailto:<a href="mailto:jasowang@redhat.com" target="_blank">jasowang@redhat.com</a>>> wrote:<br> ><br> ><br> > On 2020/11/3 上午2:51, Andrew Melnychenko wrote:<br> > > Basic idea is to use eBPF to calculate and steer packets in TAP.<br> > > RSS(Receive Side Scaling) is used to distribute network packets<br> > to guest virtqueues<br> > > by calculating packet hash.<br> > > eBPF RSS allows us to use RSS with vhost TAP.<br> > ><br> > > This set of patches introduces the usage of eBPF for packet steering<br> > > and RSS hash calculation:<br> > > * RSS(Receive Side Scaling) is used to distribute network packets to<br> > > guest virtqueues by calculating packet hash<br> > > * eBPF RSS suppose to be faster than already existing 'software'<br> > > implementation in QEMU<br> > > * Additionally adding support for the usage of RSS with vhost<br> > ><br> > > Supported kernels: 5.8+<br> > ><br> > > Implementation notes:<br> > > Linux TAP TUNSETSTEERINGEBPF ioctl was used to set the eBPF program.<br> > > Added eBPF support to qemu directly through a system call, see the<br> > > bpf(2) for details.<br> > > The eBPF program is part of the qemu and presented as an array<br> > of bpf<br> > > instructions.<br> > > The program can be recompiled by provided Makefile.ebpf(need to<br> > adjust<br> > > 'linuxhdrs'),<br> > > although it's not required to build QEMU with eBPF support.<br> > > Added changes to virtio-net and vhost, primary eBPF RSS is used.<br> > > 'Software' RSS used in the case of hash population and as a<br> > fallback option.<br> > > For vhost, the hash population feature is not reported to the guest.<br> > ><br> > > Please also see the documentation in PATCH 6/6.<br> > ><br> > > I am sending those patches as RFC to initiate the discussions<br> > and get<br> > > feedback on the following points:<br> > > * Fallback when eBPF is not supported by the kernel<br> ><br> ><br> > Yes, and it could also a lacking of CAP_BPF.<br> ><br> ><br> > > * Live migration to the kernel that doesn't have eBPF support<br> ><br> ><br> > Is there anything that we needs special treatment here?<br> ><br> > Possible case: rss=on, vhost=on, source system with kernel 5.8 <br> > (everything works) -> dest. system 5.6 (bpf does not work), the <br> > adapter functions, but all the steering does not use proper queues.<br> <br> <br> Right, I think we need to disable vhost on dest.<br><br></blockquote><div><br></div><div>Is this acceptable to disable vhost at time of migration? </div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> ><br> ><br> ><br> > > * Integration with current QEMU build<br> ><br> ><br> > Yes, a question here:<br> ><br> > 1) Any reason for not using libbpf, e.g it has been shipped with some<br> > distros<br> ><br> ><br> > We intentionally do not use libbpf, as it present only on some distros.<br> > We can switch to libbpf, but this will disable bpf if libbpf is not <br> > installed<br> <br> <br> That's better I think.<br></blockquote><div><br></div><div>We think the preferred way is to have an eBPF code built-in in QEMU (not distribute it as a separate file).</div><div><br></div><div>Our initial idea was to not use the libbpf because it:</div><div>1. Does not create additional dependency during build time and during run-time</div><div>2. Gives us smaller footprint of loadable eBPF blob inside qemu</div><div>3. Do not add too much code to QEMU</div><div><br></div><div>We can switch to libbpf, in this case:</div><div>1. Presence of dynamic library is not guaranteed on the target system</div><div>2. Static library is large</div><div>3. libbpf uses eBPF ELF which is significantly bigger than just the array or instructions (May be we succeed to reduce the ELF to some suitable size and still have it built-in)</div><div></div><div><br></div><div>Please let us know whether you still think libbpf is better and why. </div><div><br></div><div>Thanks</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br> <br> > 2) It would be better if we can avoid shipping bytecodes<br> ><br> ><br> ><br> > This creates new dependencies: llvm + clang + ...<br> > We would prefer byte code and ability to generate it if prerequisites <br> > are installed.<br> <br> <br> It's probably ok if we treat the bytecode as a kind of firmware.<br> <br> But in the long run, it's still worthwhile consider the qemu source is <br> used for development and llvm/clang should be a common requirement for <br> generating eBPF bytecode for host.<br> <br> <br> ><br> ><br> > > * Additional usage for eBPF for packet filtering<br> ><br> ><br> > Another interesting topics in to implement mac/vlan filters. And<br> > in the<br> > future, I plan to add mac based steering. All of these could be<br> > done via<br> > eBPF.<br> ><br> ><br> > No problem, we can cooperate if needed<br> ><br> ><br> > ><br> > > Know issues:<br> > > * hash population not supported by eBPF RSS: 'software' RSS used<br> ><br> ><br> > Is this because there's not way to write to vnet header in<br> > STERRING BPF?<br> ><br> > Yes. We plan to submit changes for kernel to cooperate with BPF and <br> > populate the hash, this work is in progress<br> <br> <br> That would require a new type of eBPF program and may need some work on <br> verifier.<br> <br></blockquote><div><br></div><div>May be need to allow loading of an additional type in tun.c, not only socket filter (to use bpf_set_hash)</div><div>Also vhost and tun in kernel need to be aware of header extension for hash population.<br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Btw, macvtap is still lacking even steering ebpf program. Would you want <br> to post a patch to support that?<br> <br></blockquote><div><br></div><div>Probably after we have full functioning BPF with TAP/TUN</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br> ><br> > > as a fallback, also, hash population feature is not reported to<br> > guests<br> > > with vhost.<br> > > * big-endian BPF support: for now, eBPF is disabled for<br> > big-endian systems.<br> ><br> ><br> > Are there any blocker for this?<br> ><br> ><br> > No, can be added in v2<br> <br> <br> Cool.<br> <br> Thanks<br> <br> <br> ><br> > Just some quick questions after a glance of the codes. Will go<br> > through<br> > them tomorrow.<br> ><br> > Thanks<br> ><br> ><br> > ><br> > > Andrew (6):<br> > > Added SetSteeringEBPF method for NetClientState.<br> > > ebpf: Added basic eBPF API.<br> > > ebpf: Added eBPF RSS program.<br> > > ebpf: Added eBPF RSS loader.<br> > > virtio-net: Added eBPF RSS to virtio-net.<br> > > docs: Added eBPF documentation.<br> > ><br> > > MAINTAINERS | 6 +<br> > > configure | 36 +++<br> > > docs/ebpf.rst | 29 ++<br> > > docs/ebpf_rss.rst | 129 ++++++++<br> > > ebpf/EbpfElf_to_C.py | 67 ++++<br> > > ebpf/Makefile.ebpf | 38 +++<br> > > ebpf/ebpf-stub.c | 28 ++<br> > > ebpf/ebpf.c | 107 +++++++<br> > > ebpf/ebpf.h | 35 +++<br> > > ebpf/ebpf_rss.c | 178 +++++++++++<br> > > ebpf/ebpf_rss.h | 30 ++<br> > > ebpf/meson.build | 1 +<br> > > ebpf/rss.bpf.c | 470 ++++++++++++++++++++++++++++<br> > > ebpf/trace-events | 4 +<br> > > ebpf/trace.h | 2 +<br> > > ebpf/tun_rss_steering.h | 556<br> > +++++++++++++++++++++++++++++++++<br> > > hw/net/vhost_net.c | 2 +<br> > > hw/net/virtio-net.c | 120 ++++++-<br> > > include/hw/virtio/virtio-net.h | 4 +<br> > > include/net/net.h | 2 +<br> > > meson.build | 3 +<br> > > net/tap-bsd.c | 5 +<br> > > net/tap-linux.c | 19 ++<br> > > net/tap-solaris.c | 5 +<br> > > net/tap-stub.c | 5 +<br> > > net/tap.c | 9 +<br> > > net/tap_int.h | 1 +<br> > > net/vhost-vdpa.c | 2 +<br> > > 28 files changed, 1889 insertions(+), 4 deletions(-)<br> > > create mode 100644 docs/ebpf.rst<br> > > create mode 100644 docs/ebpf_rss.rst<br> > > create mode 100644 ebpf/EbpfElf_to_C.py<br> > > create mode 100755 ebpf/Makefile.ebpf<br> > > create mode 100644 ebpf/ebpf-stub.c<br> > > create mode 100644 ebpf/ebpf.c<br> > > create mode 100644 ebpf/ebpf.h<br> > > create mode 100644 ebpf/ebpf_rss.c<br> > > create mode 100644 ebpf/ebpf_rss.h<br> > > create mode 100644 ebpf/meson.build<br> > > create mode 100644 ebpf/rss.bpf.c<br> > > create mode 100644 ebpf/trace-events<br> > > create mode 100644 ebpf/trace.h<br> > > create mode 100644 ebpf/tun_rss_steering.h<br> > ><br> ><br> <br> </blockquote></div></div>
On Wed, Nov 04, 2020 at 01:49:05PM +0200, Yuri Benditovich wrote: > On Wed, Nov 4, 2020 at 4:08 AM Jason Wang <jasowang@redhat.com> wrote: > > > > > On 2020/11/3 下午6:32, Yuri Benditovich wrote: > > > > > > > > > On Tue, Nov 3, 2020 at 11:02 AM Jason Wang <jasowang@redhat.com > > > <mailto:jasowang@redhat.com>> wrote: > > > > > > > > > On 2020/11/3 上午2:51, Andrew Melnychenko wrote: > > > > Basic idea is to use eBPF to calculate and steer packets in TAP. > > > > RSS(Receive Side Scaling) is used to distribute network packets > > > to guest virtqueues > > > > by calculating packet hash. > > > > eBPF RSS allows us to use RSS with vhost TAP. > > > > > > > > This set of patches introduces the usage of eBPF for packet > > steering > > > > and RSS hash calculation: > > > > * RSS(Receive Side Scaling) is used to distribute network packets > > to > > > > guest virtqueues by calculating packet hash > > > > * eBPF RSS suppose to be faster than already existing 'software' > > > > implementation in QEMU > > > > * Additionally adding support for the usage of RSS with vhost > > > > > > > > Supported kernels: 5.8+ > > > > > > > > Implementation notes: > > > > Linux TAP TUNSETSTEERINGEBPF ioctl was used to set the eBPF > > program. > > > > Added eBPF support to qemu directly through a system call, see the > > > > bpf(2) for details. > > > > The eBPF program is part of the qemu and presented as an array > > > of bpf > > > > instructions. > > > > The program can be recompiled by provided Makefile.ebpf(need to > > > adjust > > > > 'linuxhdrs'), > > > > although it's not required to build QEMU with eBPF support. > > > > Added changes to virtio-net and vhost, primary eBPF RSS is used. > > > > 'Software' RSS used in the case of hash population and as a > > > fallback option. > > > > For vhost, the hash population feature is not reported to the > > guest. > > > > > > > > Please also see the documentation in PATCH 6/6. > > > > > > > > I am sending those patches as RFC to initiate the discussions > > > and get > > > > feedback on the following points: > > > > * Fallback when eBPF is not supported by the kernel > > > > > > > > > Yes, and it could also a lacking of CAP_BPF. > > > > > > > > > > * Live migration to the kernel that doesn't have eBPF support > > > > > > > > > Is there anything that we needs special treatment here? > > > > > > Possible case: rss=on, vhost=on, source system with kernel 5.8 > > > (everything works) -> dest. system 5.6 (bpf does not work), the > > > adapter functions, but all the steering does not use proper queues. > > > > > > Right, I think we need to disable vhost on dest. > > > > > Is this acceptable to disable vhost at time of migration? > > > > > > > > > > > > > > > * Integration with current QEMU build > > > > > > > > > Yes, a question here: > > > > > > 1) Any reason for not using libbpf, e.g it has been shipped with some > > > distros > > > > > > > > > We intentionally do not use libbpf, as it present only on some distros. > > > We can switch to libbpf, but this will disable bpf if libbpf is not > > > installed > > > > > > That's better I think. > > > > We think the preferred way is to have an eBPF code built-in in QEMU (not > distribute it as a separate file). > > Our initial idea was to not use the libbpf because it: > 1. Does not create additional dependency during build time and during > run-time > 2. Gives us smaller footprint of loadable eBPF blob inside qemu > 3. Do not add too much code to QEMU > > We can switch to libbpf, in this case: > 1. Presence of dynamic library is not guaranteed on the target system Again if a distro or users wants to use this feature in QEMU they should be expected build the library. > 2. Static library is large QEMU doesn't support static linking for system emulators. It may happen to work at times but there's no expectations in this respect. > 3. libbpf uses eBPF ELF which is significantly bigger than just the array > or instructions (May be we succeed to reduce the ELF to some suitable size > and still have it built-in) > > Please let us know whether you still think libbpf is better and why. It looks like both CLang and GCC compilers for BPF are moving towards a world where they use BTF to get compile once, run everywhere portability for the compiled bytecode. IIUC the libbpf is what is responsible for processing the BTF data when loading it into the running kernel. This all looks like a good thing in general. If we introduce BPF to QEMU without using libbpf, and then later decide we absolutely need libbpf features, it creates an upgrade back compat issue for existing deployments. It is better to use libbpf right from the start, so we're set up to take full advantage of what it offers long term. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On 2020/11/4 下午5:31, Daniel P. Berrangé wrote: > On Wed, Nov 04, 2020 at 10:07:52AM +0800, Jason Wang wrote: >> On 2020/11/3 下午6:32, Yuri Benditovich wrote: >>> >>> On Tue, Nov 3, 2020 at 11:02 AM Jason Wang <jasowang@redhat.com >>> <mailto:jasowang@redhat.com>> wrote: >>> >>> >>> On 2020/11/3 上午2:51, Andrew Melnychenko wrote: >>> > Basic idea is to use eBPF to calculate and steer packets in TAP. >>> > RSS(Receive Side Scaling) is used to distribute network packets >>> to guest virtqueues >>> > by calculating packet hash. >>> > eBPF RSS allows us to use RSS with vhost TAP. >>> > >>> > This set of patches introduces the usage of eBPF for packet steering >>> > and RSS hash calculation: >>> > * RSS(Receive Side Scaling) is used to distribute network packets to >>> > guest virtqueues by calculating packet hash >>> > * eBPF RSS suppose to be faster than already existing 'software' >>> > implementation in QEMU >>> > * Additionally adding support for the usage of RSS with vhost >>> > >>> > Supported kernels: 5.8+ >>> > >>> > Implementation notes: >>> > Linux TAP TUNSETSTEERINGEBPF ioctl was used to set the eBPF program. >>> > Added eBPF support to qemu directly through a system call, see the >>> > bpf(2) for details. >>> > The eBPF program is part of the qemu and presented as an array >>> of bpf >>> > instructions. >>> > The program can be recompiled by provided Makefile.ebpf(need to >>> adjust >>> > 'linuxhdrs'), >>> > although it's not required to build QEMU with eBPF support. >>> > Added changes to virtio-net and vhost, primary eBPF RSS is used. >>> > 'Software' RSS used in the case of hash population and as a >>> fallback option. >>> > For vhost, the hash population feature is not reported to the guest. >>> > >>> > Please also see the documentation in PATCH 6/6. >>> > >>> > I am sending those patches as RFC to initiate the discussions >>> and get >>> > feedback on the following points: >>> > * Fallback when eBPF is not supported by the kernel >>> >>> >>> Yes, and it could also a lacking of CAP_BPF. >>> >>> >>> > * Live migration to the kernel that doesn't have eBPF support >>> >>> >>> Is there anything that we needs special treatment here? >>> >>> Possible case: rss=on, vhost=on, source system with kernel 5.8 >>> (everything works) -> dest. system 5.6 (bpf does not work), the adapter >>> functions, but all the steering does not use proper queues. >> >> Right, I think we need to disable vhost on dest. >> >> >>> >>> >>> > * Integration with current QEMU build >>> >>> >>> Yes, a question here: >>> >>> 1) Any reason for not using libbpf, e.g it has been shipped with some >>> distros >>> >>> >>> We intentionally do not use libbpf, as it present only on some distros. >>> We can switch to libbpf, but this will disable bpf if libbpf is not >>> installed >> >> That's better I think. >> >> >>> 2) It would be better if we can avoid shipping bytecodes >>> >>> >>> >>> This creates new dependencies: llvm + clang + ... >>> We would prefer byte code and ability to generate it if prerequisites >>> are installed. >> >> It's probably ok if we treat the bytecode as a kind of firmware. > That is explicitly *not* OK for inclusion in Fedora. They require that > BPF is compiled from source, and rejected my suggestion that it could > be considered a kind of firmware and thus have an exception from building > from source. Please refer what it was done in DPDK: http://git.dpdk.org/dpdk/tree/doc/guides/nics/tap.rst#n235 I don't think what proposed here makes anything different. It's still a bytecode that lives in an array. > >> But in the long run, it's still worthwhile consider the qemu source is used >> for development and llvm/clang should be a common requirement for generating >> eBPF bytecode for host. > So we need to do this right straight way before this merges. Yes. Thanks > > Regards, > Daniel
On 2020/11/5 上午11:46, Jason Wang wrote: >> >> It's probably ok if we treat the bytecode as a kind of firmware. > That is explicitly *not* OK for inclusion in Fedora. They require that > BPF is compiled from source, and rejected my suggestion that it could > be considered a kind of firmware and thus have an exception from building > from source. Actually, there's another advantages. If we treat it as firmware, (actually it is). It allows us to upgrade it independently with qemu. Thanks
On 2020/11/3 上午2:51, Andrew Melnychenko wrote: > From: Andrew<andrew@daynix.com> > > Also, added maintainers information. > > Signed-off-by: Yuri Benditovich<yuri.benditovich@daynix.com> > Signed-off-by: Andrew Melnychenko<andrew@daynix.com> > --- > MAINTAINERS | 6 +++ > docs/ebpf.rst | 29 +++++++++++ > docs/ebpf_rss.rst | 129 ++++++++++++++++++++++++++++++++++++++++++++++ > 3 files changed, 164 insertions(+) > create mode 100644 docs/ebpf.rst > create mode 100644 docs/ebpf_rss.rst > > diff --git a/MAINTAINERS b/MAINTAINERS > index 2c22bbca5a..464b3f3c95 100644 > --- a/MAINTAINERS > +++ b/MAINTAINERS > @@ -3111,6 +3111,12 @@ S: Maintained > F: hw/semihosting/ > F: include/hw/semihosting/ > > +EBPF: > +M: Andrew Melnychenko<andrew@daynix.com> > +M: Yuri Benditovich<yuri.benditovich@daynix.com> > +S: Maintained > +F: ebpf/* > + If it's possible, I would like to be one of the maintainer or at least reviewer :) Thanks
On Thu, Nov 5, 2020 at 5:52 AM Jason Wang <jasowang@redhat.com> wrote: > > On 2020/11/5 上午11:46, Jason Wang wrote: > >> > >> It's probably ok if we treat the bytecode as a kind of firmware. > > That is explicitly *not* OK for inclusion in Fedora. They require that > > BPF is compiled from source, and rejected my suggestion that it could > > be considered a kind of firmware and thus have an exception from building > > from source. > > > Actually, there's another advantages. If we treat it as firmware, > (actually it is). It allows us to upgrade it independently with qemu. > > Hi Jason, I think this is a big disadvantage to have the BPF binary outside of QEMU. It is compiled with common structures (for example RSS configuration) defined in QEMU and if it is not built in the QEMU then nobody is responsible for the compatibility of the BPF and QEMU. Just an array of instructions (af today) is ~2k, full object file (if we use libbpf) is ~8K, so there is no big problem with the size. If we even keep the entire object in QEMU, it is for sure 100% compatible. Thanks > > <div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Nov 5, 2020 at 5:52 AM Jason Wang <<a href="mailto:jasowang@redhat.com">jasowang@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br> On 2020/11/5 上午11:46, Jason Wang wrote:<br> >><br> >> It's probably ok if we treat the bytecode as a kind of firmware.<br> > That is explicitly *not* OK for inclusion in Fedora. They require that<br> > BPF is compiled from source, and rejected my suggestion that it could<br> > be considered a kind of firmware and thus have an exception from building<br> > from source. <br> <br> <br> Actually, there's another advantages. If we treat it as firmware, <br> (actually it is). It allows us to upgrade it independently with qemu.<br> <br></blockquote><div>Hi Jason,</div><div>I think this is a big disadvantage to have the BPF binary outside of QEMU.</div><div>It is compiled with common structures (for example RSS configuration) defined in QEMU and if it is not built in the QEMU then nobody is responsible for the compatibility of the BPF and QEMU.</div><div>Just an array of instructions (af today) is ~2k, full object file (if we use libbpf) is ~8K, so there is no big problem with the size.</div><div>If we even keep the entire object in QEMU, it is for sure 100% compatible.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Thanks<br> <br> </blockquote></div></div>
On Thu, Nov 5, 2020 at 5:56 AM Jason Wang <jasowang@redhat.com> wrote: > > On 2020/11/3 上午2:51, Andrew Melnychenko wrote: > > From: Andrew<andrew@daynix.com> > > > > Also, added maintainers information. > > > > Signed-off-by: Yuri Benditovich<yuri.benditovich@daynix.com> > > Signed-off-by: Andrew Melnychenko<andrew@daynix.com> > > --- > > MAINTAINERS | 6 +++ > > docs/ebpf.rst | 29 +++++++++++ > > docs/ebpf_rss.rst | 129 ++++++++++++++++++++++++++++++++++++++++++++++ > > 3 files changed, 164 insertions(+) > > create mode 100644 docs/ebpf.rst > > create mode 100644 docs/ebpf_rss.rst > > > > diff --git a/MAINTAINERS b/MAINTAINERS > > index 2c22bbca5a..464b3f3c95 100644 > > --- a/MAINTAINERS > > +++ b/MAINTAINERS > > @@ -3111,6 +3111,12 @@ S: Maintained > > F: hw/semihosting/ > > F: include/hw/semihosting/ > > > > +EBPF: > > +M: Andrew Melnychenko<andrew@daynix.com> > > +M: Yuri Benditovich<yuri.benditovich@daynix.com> > > +S: Maintained > > +F: ebpf/* > > + > > > If it's possible, I would like to be one of the maintainer or at least > reviewer :) > > With pleasure. We did not know who would want to maintain eBPF related things, so we added ourselves as maintainers. If you agree, we'll place you as a maintainer and ourselves as reviewers to be informed about changes before they happen. Thanks > Thanks > > <div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Nov 5, 2020 at 5:56 AM Jason Wang <<a href="mailto:jasowang@redhat.com">jasowang@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br> On 2020/11/3 上午2:51, Andrew Melnychenko wrote:<br> > From: Andrew<<a href="mailto:andrew@daynix.com" target="_blank">andrew@daynix.com</a>><br> ><br> > Also, added maintainers information.<br> ><br> > Signed-off-by: Yuri Benditovich<<a href="mailto:yuri.benditovich@daynix.com" target="_blank">yuri.benditovich@daynix.com</a>><br> > Signed-off-by: Andrew Melnychenko<<a href="mailto:andrew@daynix.com" target="_blank">andrew@daynix.com</a>><br> > ---<br> > MAINTAINERS | 6 +++<br> > docs/ebpf.rst | 29 +++++++++++<br> > docs/ebpf_rss.rst | 129 ++++++++++++++++++++++++++++++++++++++++++++++<br> > 3 files changed, 164 insertions(+)<br> > create mode 100644 docs/ebpf.rst<br> > create mode 100644 docs/ebpf_rss.rst<br> ><br> > diff --git a/MAINTAINERS b/MAINTAINERS<br> > index 2c22bbca5a..464b3f3c95 100644<br> > --- a/MAINTAINERS<br> > +++ b/MAINTAINERS<br> > @@ -3111,6 +3111,12 @@ S: Maintained<br> > F: hw/semihosting/<br> > F: include/hw/semihosting/<br> > <br> > +EBPF:<br> > +M: Andrew Melnychenko<<a href="mailto:andrew@daynix.com" target="_blank">andrew@daynix.com</a>><br> > +M: Yuri Benditovich<<a href="mailto:yuri.benditovich@daynix.com" target="_blank">yuri.benditovich@daynix.com</a>><br> > +S: Maintained<br> > +F: ebpf/*<br> > +<br> <br> <br> If it's possible, I would like to be one of the maintainer or at least <br> reviewer :)<br> <br></blockquote><div>With pleasure. We did not know who would want to maintain eBPF related things, so we added ourselves as maintainers.</div><div>If you agree, we'll place you as a maintainer and ourselves as reviewers to be informed about changes before they happen.</div><div><br></div><div>Thanks</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Thanks<br> <br> </blockquote></div></div>
On Thu, Nov 05, 2020 at 11:46:18AM +0800, Jason Wang wrote: > > On 2020/11/4 下午5:31, Daniel P. Berrangé wrote: > > On Wed, Nov 04, 2020 at 10:07:52AM +0800, Jason Wang wrote: > > > On 2020/11/3 下午6:32, Yuri Benditovich wrote: > > > > > > > > On Tue, Nov 3, 2020 at 11:02 AM Jason Wang <jasowang@redhat.com > > > > <mailto:jasowang@redhat.com>> wrote: > > > > > > > > > > > > On 2020/11/3 上午2:51, Andrew Melnychenko wrote: > > > > > Basic idea is to use eBPF to calculate and steer packets in TAP. > > > > > RSS(Receive Side Scaling) is used to distribute network packets > > > > to guest virtqueues > > > > > by calculating packet hash. > > > > > eBPF RSS allows us to use RSS with vhost TAP. > > > > > > > > > > This set of patches introduces the usage of eBPF for packet steering > > > > > and RSS hash calculation: > > > > > * RSS(Receive Side Scaling) is used to distribute network packets to > > > > > guest virtqueues by calculating packet hash > > > > > * eBPF RSS suppose to be faster than already existing 'software' > > > > > implementation in QEMU > > > > > * Additionally adding support for the usage of RSS with vhost > > > > > > > > > > Supported kernels: 5.8+ > > > > > > > > > > Implementation notes: > > > > > Linux TAP TUNSETSTEERINGEBPF ioctl was used to set the eBPF program. > > > > > Added eBPF support to qemu directly through a system call, see the > > > > > bpf(2) for details. > > > > > The eBPF program is part of the qemu and presented as an array > > > > of bpf > > > > > instructions. > > > > > The program can be recompiled by provided Makefile.ebpf(need to > > > > adjust > > > > > 'linuxhdrs'), > > > > > although it's not required to build QEMU with eBPF support. > > > > > Added changes to virtio-net and vhost, primary eBPF RSS is used. > > > > > 'Software' RSS used in the case of hash population and as a > > > > fallback option. > > > > > For vhost, the hash population feature is not reported to the guest. > > > > > > > > > > Please also see the documentation in PATCH 6/6. > > > > > > > > > > I am sending those patches as RFC to initiate the discussions > > > > and get > > > > > feedback on the following points: > > > > > * Fallback when eBPF is not supported by the kernel > > > > > > > > > > > > Yes, and it could also a lacking of CAP_BPF. > > > > > > > > > > > > > * Live migration to the kernel that doesn't have eBPF support > > > > > > > > > > > > Is there anything that we needs special treatment here? > > > > > > > > Possible case: rss=on, vhost=on, source system with kernel 5.8 > > > > (everything works) -> dest. system 5.6 (bpf does not work), the adapter > > > > functions, but all the steering does not use proper queues. > > > > > > Right, I think we need to disable vhost on dest. > > > > > > > > > > > > > > > > > > > * Integration with current QEMU build > > > > > > > > > > > > Yes, a question here: > > > > > > > > 1) Any reason for not using libbpf, e.g it has been shipped with some > > > > distros > > > > > > > > > > > > We intentionally do not use libbpf, as it present only on some distros. > > > > We can switch to libbpf, but this will disable bpf if libbpf is not > > > > installed > > > > > > That's better I think. > > > > > > > > > > 2) It would be better if we can avoid shipping bytecodes > > > > > > > > > > > > > > > > This creates new dependencies: llvm + clang + ... > > > > We would prefer byte code and ability to generate it if prerequisites > > > > are installed. > > > > > > It's probably ok if we treat the bytecode as a kind of firmware. > > That is explicitly *not* OK for inclusion in Fedora. They require that > > BPF is compiled from source, and rejected my suggestion that it could > > be considered a kind of firmware and thus have an exception from building > > from source. > > > Please refer what it was done in DPDK: > > http://git.dpdk.org/dpdk/tree/doc/guides/nics/tap.rst#n235 > > I don't think what proposed here makes anything different. I'm not convinced that what DPDK does is acceptable to Fedora either based on the responses I've received when asking about BPF handling during build. I wouldn't suprise me, however, if this was simply missed by reviewers when accepting DPDK into Fedora, because it is not entirely obvious unless you are looking closely. Regards, Daniel
On Thu, Nov 05, 2020 at 10:01:09AM +0000, Daniel P. Berrangé wrote: > On Thu, Nov 05, 2020 at 11:46:18AM +0800, Jason Wang wrote: > > > > On 2020/11/4 下午5:31, Daniel P. Berrangé wrote: > > > On Wed, Nov 04, 2020 at 10:07:52AM +0800, Jason Wang wrote: > > > > On 2020/11/3 下午6:32, Yuri Benditovich wrote: > > > > > > > > > > On Tue, Nov 3, 2020 at 11:02 AM Jason Wang <jasowang@redhat.com > > > > > <mailto:jasowang@redhat.com>> wrote: > > > > > > > > > > > > > > > On 2020/11/3 上午2:51, Andrew Melnychenko wrote: > > > > > > Basic idea is to use eBPF to calculate and steer packets in TAP. > > > > > > RSS(Receive Side Scaling) is used to distribute network packets > > > > > to guest virtqueues > > > > > > by calculating packet hash. > > > > > > eBPF RSS allows us to use RSS with vhost TAP. > > > > > > > > > > > > This set of patches introduces the usage of eBPF for packet steering > > > > > > and RSS hash calculation: > > > > > > * RSS(Receive Side Scaling) is used to distribute network packets to > > > > > > guest virtqueues by calculating packet hash > > > > > > * eBPF RSS suppose to be faster than already existing 'software' > > > > > > implementation in QEMU > > > > > > * Additionally adding support for the usage of RSS with vhost > > > > > > > > > > > > Supported kernels: 5.8+ > > > > > > > > > > > > Implementation notes: > > > > > > Linux TAP TUNSETSTEERINGEBPF ioctl was used to set the eBPF program. > > > > > > Added eBPF support to qemu directly through a system call, see the > > > > > > bpf(2) for details. > > > > > > The eBPF program is part of the qemu and presented as an array > > > > > of bpf > > > > > > instructions. > > > > > > The program can be recompiled by provided Makefile.ebpf(need to > > > > > adjust > > > > > > 'linuxhdrs'), > > > > > > although it's not required to build QEMU with eBPF support. > > > > > > Added changes to virtio-net and vhost, primary eBPF RSS is used. > > > > > > 'Software' RSS used in the case of hash population and as a > > > > > fallback option. > > > > > > For vhost, the hash population feature is not reported to the guest. > > > > > > > > > > > > Please also see the documentation in PATCH 6/6. > > > > > > > > > > > > I am sending those patches as RFC to initiate the discussions > > > > > and get > > > > > > feedback on the following points: > > > > > > * Fallback when eBPF is not supported by the kernel > > > > > > > > > > > > > > > Yes, and it could also a lacking of CAP_BPF. > > > > > > > > > > > > > > > > * Live migration to the kernel that doesn't have eBPF support > > > > > > > > > > > > > > > Is there anything that we needs special treatment here? > > > > > > > > > > Possible case: rss=on, vhost=on, source system with kernel 5.8 > > > > > (everything works) -> dest. system 5.6 (bpf does not work), the adapter > > > > > functions, but all the steering does not use proper queues. > > > > > > > > Right, I think we need to disable vhost on dest. > > > > > > > > > > > > > > > > > > > > > > > > * Integration with current QEMU build > > > > > > > > > > > > > > > Yes, a question here: > > > > > > > > > > 1) Any reason for not using libbpf, e.g it has been shipped with some > > > > > distros > > > > > > > > > > > > > > > We intentionally do not use libbpf, as it present only on some distros. > > > > > We can switch to libbpf, but this will disable bpf if libbpf is not > > > > > installed > > > > > > > > That's better I think. > > > > > > > > > > > > > 2) It would be better if we can avoid shipping bytecodes > > > > > > > > > > > > > > > > > > > > This creates new dependencies: llvm + clang + ... > > > > > We would prefer byte code and ability to generate it if prerequisites > > > > > are installed. > > > > > > > > It's probably ok if we treat the bytecode as a kind of firmware. > > > That is explicitly *not* OK for inclusion in Fedora. They require that > > > BPF is compiled from source, and rejected my suggestion that it could > > > be considered a kind of firmware and thus have an exception from building > > > from source. > > > > > > Please refer what it was done in DPDK: > > > > http://git.dpdk.org/dpdk/tree/doc/guides/nics/tap.rst#n235 > > > > I don't think what proposed here makes anything different. > > I'm not convinced that what DPDK does is acceptable to Fedora either > based on the responses I've received when asking about BPF handling > during build. I wouldn't suprise me, however, if this was simply > missed by reviewers when accepting DPDK into Fedora, because it is > not entirely obvious unless you are looking closely. FWIW, I'm pushing back against the idea that we have to compile the BPF code from master source, as I think it is reasonable to have the program embedded as a static array in the source code similar to what DPDK does. It doesn't feel much different from other places where apps use generated sources, and don't build them from the original source every time. eg "configure" is never re-generated from "configure.ac" by Fedora packagers, they just use the generated "configure" script as-is. Regards, Daniel
First of all, thank you for all your feedbacks Please help me to summarize and let us understand better what we do in v2: Major questions are: 1. Building eBPF from source during qemu build vs. regenerating it on demand and keeping in the repository Solution 1a (~ as in v1): keep instructions or ELF in H file, generate it out of qemu build. In general we'll need to have BE and LE binaries. Solution 1b: build ELF or instructions during QEMU build if llvm + clang exist. Then we will have only one (BE or LE, depending on current QEMU build) We agree with any solution - I believe you know the requirements better. 2. Use libbpf or not In general we do not see any advantage of using libbpf. It works with object files (does ELF parsing at time of loading), but it does not do any magic. Solution 2a. Switch to libbpf, generate object files (LE and BE) from source, keep them inside QEMU (~8k each) or aside Solution 2b. (as in v1) Use python script to parse object -> instructions (~2k each) We'd prefer not to use libbpf at the moment. If due to some unknown reason we'll find it useful in future, we can switch to it, this does not create any incompatibility. Then this will create a dependency on libbpf.so 3. Keep instructions or ELF inside QEMU or as separate external file Solution 3a (~as in v1): Built-in array of instructions or ELF. If we generate them out of QEMU build - keep 2 arrays or instructions or ELF (BE and LE), Solution 3b: Install them as separate files (/usr/share/qemu). We'd prefer 3a: Then there is a guarantee that the eBPF is built with exactly the same config structures as QEMU (qemu creates a mapping of its structures, eBPF uses them). No need to take care on scenarios like 'file not found', 'file is not suitable' etc 4. Is there some real request to have the eBPF for big-endian? If no, we can enable eBPF only for LE builds Jason, Daniel, Michael Can you please let us know what you think and why? On Thu, Nov 5, 2020 at 3:19 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > On Thu, Nov 05, 2020 at 10:01:09AM +0000, Daniel P. Berrangé wrote: > > On Thu, Nov 05, 2020 at 11:46:18AM +0800, Jason Wang wrote: > > > > > > On 2020/11/4 下午5:31, Daniel P. Berrangé wrote: > > > > On Wed, Nov 04, 2020 at 10:07:52AM +0800, Jason Wang wrote: > > > > > On 2020/11/3 下午6:32, Yuri Benditovich wrote: > > > > > > > > > > > > On Tue, Nov 3, 2020 at 11:02 AM Jason Wang <jasowang@redhat.com > > > > > > <mailto:jasowang@redhat.com>> wrote: > > > > > > > > > > > > > > > > > > On 2020/11/3 上午2:51, Andrew Melnychenko wrote: > > > > > > > Basic idea is to use eBPF to calculate and steer packets > in TAP. > > > > > > > RSS(Receive Side Scaling) is used to distribute network > packets > > > > > > to guest virtqueues > > > > > > > by calculating packet hash. > > > > > > > eBPF RSS allows us to use RSS with vhost TAP. > > > > > > > > > > > > > > This set of patches introduces the usage of eBPF for > packet steering > > > > > > > and RSS hash calculation: > > > > > > > * RSS(Receive Side Scaling) is used to distribute network > packets to > > > > > > > guest virtqueues by calculating packet hash > > > > > > > * eBPF RSS suppose to be faster than already existing > 'software' > > > > > > > implementation in QEMU > > > > > > > * Additionally adding support for the usage of RSS with > vhost > > > > > > > > > > > > > > Supported kernels: 5.8+ > > > > > > > > > > > > > > Implementation notes: > > > > > > > Linux TAP TUNSETSTEERINGEBPF ioctl was used to set the > eBPF program. > > > > > > > Added eBPF support to qemu directly through a system > call, see the > > > > > > > bpf(2) for details. > > > > > > > The eBPF program is part of the qemu and presented as an > array > > > > > > of bpf > > > > > > > instructions. > > > > > > > The program can be recompiled by provided > Makefile.ebpf(need to > > > > > > adjust > > > > > > > 'linuxhdrs'), > > > > > > > although it's not required to build QEMU with eBPF > support. > > > > > > > Added changes to virtio-net and vhost, primary eBPF RSS > is used. > > > > > > > 'Software' RSS used in the case of hash population and as > a > > > > > > fallback option. > > > > > > > For vhost, the hash population feature is not reported to > the guest. > > > > > > > > > > > > > > Please also see the documentation in PATCH 6/6. > > > > > > > > > > > > > > I am sending those patches as RFC to initiate the > discussions > > > > > > and get > > > > > > > feedback on the following points: > > > > > > > * Fallback when eBPF is not supported by the kernel > > > > > > > > > > > > > > > > > > Yes, and it could also a lacking of CAP_BPF. > > > > > > > > > > > > > > > > > > > * Live migration to the kernel that doesn't have eBPF > support > > > > > > > > > > > > > > > > > > Is there anything that we needs special treatment here? > > > > > > > > > > > > Possible case: rss=on, vhost=on, source system with kernel 5.8 > > > > > > (everything works) -> dest. system 5.6 (bpf does not work), the > adapter > > > > > > functions, but all the steering does not use proper queues. > > > > > > > > > > Right, I think we need to disable vhost on dest. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > * Integration with current QEMU build > > > > > > > > > > > > > > > > > > Yes, a question here: > > > > > > > > > > > > 1) Any reason for not using libbpf, e.g it has been shipped > with some > > > > > > distros > > > > > > > > > > > > > > > > > > We intentionally do not use libbpf, as it present only on some > distros. > > > > > > We can switch to libbpf, but this will disable bpf if libbpf is > not > > > > > > installed > > > > > > > > > > That's better I think. > > > > > > > > > > > > > > > > 2) It would be better if we can avoid shipping bytecodes > > > > > > > > > > > > > > > > > > > > > > > > This creates new dependencies: llvm + clang + ... > > > > > > We would prefer byte code and ability to generate it if > prerequisites > > > > > > are installed. > > > > > > > > > > It's probably ok if we treat the bytecode as a kind of firmware. > > > > That is explicitly *not* OK for inclusion in Fedora. They require > that > > > > BPF is compiled from source, and rejected my suggestion that it could > > > > be considered a kind of firmware and thus have an exception from > building > > > > from source. > > > > > > > > > Please refer what it was done in DPDK: > > > > > > http://git.dpdk.org/dpdk/tree/doc/guides/nics/tap.rst#n235 > > > > > > I don't think what proposed here makes anything different. > > > > I'm not convinced that what DPDK does is acceptable to Fedora either > > based on the responses I've received when asking about BPF handling > > during build. I wouldn't suprise me, however, if this was simply > > missed by reviewers when accepting DPDK into Fedora, because it is > > not entirely obvious unless you are looking closely. > > FWIW, I'm pushing back against the idea that we have to compile the > BPF code from master source, as I think it is reasonable to have the > program embedded as a static array in the source code similar to what > DPDK does. It doesn't feel much different from other places where apps > use generated sources, and don't build them from the original source > every time. eg "configure" is never re-generated from "configure.ac" > by Fedora packagers, they just use the generated "configure" script > as-is. > > Regards, > Daniel > -- > |: https://berrange.com -o- > https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- > https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- > https://www.instagram.com/dberrange :| > > <div dir="ltr">First of all, thank you for all your feedbacks<div><br></div><div>Please help me to summarize and let us understand better what we do in v2:</div><div>Major questions are: </div><div>1. Building eBPF from source during qemu build vs. regenerating it on demand and keeping in the repository</div><div>Solution 1a (~ as in v1): keep instructions or ELF in H file, generate it out of qemu build. In general we'll need to have BE and LE binaries.</div><div>Solution 1b: build ELF or instructions during QEMU build if llvm + clang exist. Then we will have only one (BE or LE, depending on current QEMU build)</div><div>We agree with any solution - I believe you know the requirements better. </div><div><br></div><div>2. Use libbpf or not</div><div>In general we do not see any advantage of using libbpf. It works with object files (does ELF parsing at time of loading), but it does not do any magic.</div><div>Solution 2a. Switch to libbpf, generate object files (LE and BE) from source, keep them inside QEMU (~8k each) or aside<br></div><div>Solution 2b. (as in v1) Use python script to parse object -> instructions (~2k each)</div><div>We'd prefer not to use libbpf at the moment.</div><div>If due to some unknown reason we'll find it useful in future, we can switch to it, this does not create any incompatibility. Then this will create a dependency on libbpf.so </div><div><br></div><div>3. Keep instructions or ELF inside QEMU or as separate external file</div><div>Solution 3a (~as in v1): Built-in array of instructions or ELF. If we generate them out of QEMU build - keep 2 arrays or instructions or ELF (BE and LE), </div><div>Solution 3b: Install them as separate files (/usr/share/qemu).</div><div>We'd prefer 3a:</div><div> Then there is a guarantee that the eBPF is built with exactly the same config structures as QEMU (qemu creates a mapping of its structures, eBPF uses them).</div><div> No need to take care on scenarios like 'file not found', 'file is not suitable' etc</div><div><br></div><div>4. Is there some real request to have the eBPF for big-endian?<br></div><div>If no, we can enable eBPF only for LE builds</div><div><br></div><div>Jason, Daniel, Michael</div><div>Can you please let us know what you think and why?</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Nov 5, 2020 at 3:19 PM Daniel P. Berrangé <<a href="mailto:berrange@redhat.com" target="_blank">berrange@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Thu, Nov 05, 2020 at 10:01:09AM +0000, Daniel P. Berrangé wrote:<br> > On Thu, Nov 05, 2020 at 11:46:18AM +0800, Jason Wang wrote:<br> > > <br> > > On 2020/11/4 下午5:31, Daniel P. Berrangé wrote:<br> > > > On Wed, Nov 04, 2020 at 10:07:52AM +0800, Jason Wang wrote:<br> > > > > On 2020/11/3 下午6:32, Yuri Benditovich wrote:<br> > > > > > <br> > > > > > On Tue, Nov 3, 2020 at 11:02 AM Jason Wang <<a href="mailto:jasowang@redhat.com" target="_blank">jasowang@redhat.com</a><br> > > > > > <mailto:<a href="mailto:jasowang@redhat.com" target="_blank">jasowang@redhat.com</a>>> wrote:<br> > > > > > <br> > > > > > <br> > > > > > On 2020/11/3 上午2:51, Andrew Melnychenko wrote:<br> > > > > > > Basic idea is to use eBPF to calculate and steer packets in TAP.<br> > > > > > > RSS(Receive Side Scaling) is used to distribute network packets<br> > > > > > to guest virtqueues<br> > > > > > > by calculating packet hash.<br> > > > > > > eBPF RSS allows us to use RSS with vhost TAP.<br> > > > > > ><br> > > > > > > This set of patches introduces the usage of eBPF for packet steering<br> > > > > > > and RSS hash calculation:<br> > > > > > > * RSS(Receive Side Scaling) is used to distribute network packets to<br> > > > > > > guest virtqueues by calculating packet hash<br> > > > > > > * eBPF RSS suppose to be faster than already existing 'software'<br> > > > > > > implementation in QEMU<br> > > > > > > * Additionally adding support for the usage of RSS with vhost<br> > > > > > ><br> > > > > > > Supported kernels: 5.8+<br> > > > > > ><br> > > > > > > Implementation notes:<br> > > > > > > Linux TAP TUNSETSTEERINGEBPF ioctl was used to set the eBPF program.<br> > > > > > > Added eBPF support to qemu directly through a system call, see the<br> > > > > > > bpf(2) for details.<br> > > > > > > The eBPF program is part of the qemu and presented as an array<br> > > > > > of bpf<br> > > > > > > instructions.<br> > > > > > > The program can be recompiled by provided Makefile.ebpf(need to<br> > > > > > adjust<br> > > > > > > 'linuxhdrs'),<br> > > > > > > although it's not required to build QEMU with eBPF support.<br> > > > > > > Added changes to virtio-net and vhost, primary eBPF RSS is used.<br> > > > > > > 'Software' RSS used in the case of hash population and as a<br> > > > > > fallback option.<br> > > > > > > For vhost, the hash population feature is not reported to the guest.<br> > > > > > ><br> > > > > > > Please also see the documentation in PATCH 6/6.<br> > > > > > ><br> > > > > > > I am sending those patches as RFC to initiate the discussions<br> > > > > > and get<br> > > > > > > feedback on the following points:<br> > > > > > > * Fallback when eBPF is not supported by the kernel<br> > > > > > <br> > > > > > <br> > > > > > Yes, and it could also a lacking of CAP_BPF.<br> > > > > > <br> > > > > > <br> > > > > > > * Live migration to the kernel that doesn't have eBPF support<br> > > > > > <br> > > > > > <br> > > > > > Is there anything that we needs special treatment here?<br> > > > > > <br> > > > > > Possible case: rss=on, vhost=on, source system with kernel 5.8<br> > > > > > (everything works) -> dest. system 5.6 (bpf does not work), the adapter<br> > > > > > functions, but all the steering does not use proper queues.<br> > > > > <br> > > > > Right, I think we need to disable vhost on dest.<br> > > > > <br> > > > > <br> > > > > > <br> > > > > > <br> > > > > > > * Integration with current QEMU build<br> > > > > > <br> > > > > > <br> > > > > > Yes, a question here:<br> > > > > > <br> > > > > > 1) Any reason for not using libbpf, e.g it has been shipped with some<br> > > > > > distros<br> > > > > > <br> > > > > > <br> > > > > > We intentionally do not use libbpf, as it present only on some distros.<br> > > > > > We can switch to libbpf, but this will disable bpf if libbpf is not<br> > > > > > installed<br> > > > > <br> > > > > That's better I think.<br> > > > > <br> > > > > <br> > > > > > 2) It would be better if we can avoid shipping bytecodes<br> > > > > > <br> > > > > > <br> > > > > > <br> > > > > > This creates new dependencies: llvm + clang + ...<br> > > > > > We would prefer byte code and ability to generate it if prerequisites<br> > > > > > are installed.<br> > > > > <br> > > > > It's probably ok if we treat the bytecode as a kind of firmware.<br> > > > That is explicitly *not* OK for inclusion in Fedora. They require that<br> > > > BPF is compiled from source, and rejected my suggestion that it could<br> > > > be considered a kind of firmware and thus have an exception from building<br> > > > from source.<br> > > <br> > > <br> > > Please refer what it was done in DPDK:<br> > > <br> > > <a href="http://git.dpdk.org/dpdk/tree/doc/guides/nics/tap.rst#n235" rel="noreferrer" target="_blank">http://git.dpdk.org/dpdk/tree/doc/guides/nics/tap.rst#n235</a><br> > > <br> > > I don't think what proposed here makes anything different.<br> > <br> > I'm not convinced that what DPDK does is acceptable to Fedora either<br> > based on the responses I've received when asking about BPF handling<br> > during build. I wouldn't suprise me, however, if this was simply<br> > missed by reviewers when accepting DPDK into Fedora, because it is<br> > not entirely obvious unless you are looking closely.<br> <br> FWIW, I'm pushing back against the idea that we have to compile the<br> BPF code from master source, as I think it is reasonable to have the<br> program embedded as a static array in the source code similar to what<br> DPDK does. It doesn't feel much different from other places where apps<br> use generated sources, and don't build them from the original source<br> every time. eg "configure" is never re-generated from "<a href="http://configure.ac" rel="noreferrer" target="_blank">configure.ac</a>"<br> by Fedora packagers, they just use the generated "configure" script<br> as-is.<br> <br> Regards,<br> Daniel<br> -- <br> |: <a href="https://berrange.com" rel="noreferrer" target="_blank">https://berrange.com</a> -o- <a href="https://www.flickr.com/photos/dberrange" rel="noreferrer" target="_blank">https://www.flickr.com/photos/dberrange</a> :|<br> |: <a href="https://libvirt.org" rel="noreferrer" target="_blank">https://libvirt.org</a> -o- <a href="https://fstop138.berrange.com" rel="noreferrer" target="_blank">https://fstop138.berrange.com</a> :|<br> |: <a href="https://entangle-photo.org" rel="noreferrer" target="_blank">https://entangle-photo.org</a> -o- <a href="https://www.instagram.com/dberrange" rel="noreferrer" target="_blank">https://www.instagram.com/dberrange</a> :|<br> <br> </blockquote></div>
On 2020/11/5 下午11:13, Yuri Benditovich wrote: > First of all, thank you for all your feedbacks > > Please help me to summarize and let us understand better what we do in v2: > Major questions are: > 1. Building eBPF from source during qemu build vs. regenerating it on > demand and keeping in the repository > Solution 1a (~ as in v1): keep instructions or ELF in H file, generate > it out of qemu build. In general we'll need to have BE and LE binaries. > Solution 1b: build ELF or instructions during QEMU build if llvm + > clang exist. Then we will have only one (BE or LE, depending on > current QEMU build) > We agree with any solution - I believe you know the requirements better. I think we can go with 1a. (See Danial's comment) > > 2. Use libbpf or not > In general we do not see any advantage of using libbpf. It works with > object files (does ELF parsing at time of loading), but it does not do > any magic. > Solution 2a. Switch to libbpf, generate object files (LE and BE) from > source, keep them inside QEMU (~8k each) or aside Can we simply use dynamic linking here? > Solution 2b. (as in v1) Use python script to parse object -> > instructions (~2k each) > We'd prefer not to use libbpf at the moment. > If due to some unknown reason we'll find it useful in future, we can > switch to it, this does not create any incompatibility. Then this will > create a dependency on libbpf.so I think we need to care about compatibility. E.g we need to enable BTF so I don't know how hard if we add BTF support in the current design. It would be probably OK it's not a lot of effort. > > 3. Keep instructions or ELF inside QEMU or as separate external file > Solution 3a (~as in v1): Built-in array of instructions or ELF. If we > generate them out of QEMU build - keep 2 arrays or instructions or ELF > (BE and LE), > Solution 3b: Install them as separate files (/usr/share/qemu). > We'd prefer 3a: > Then there is a guarantee that the eBPF is built with exactly the > same config structures as QEMU (qemu creates a mapping of its > structures, eBPF uses them). > No need to take care on scenarios like 'file not found', 'file is not > suitable' etc Yes, let's go 3a for upstream. > > 4. Is there some real request to have the eBPF for big-endian? > If no, we can enable eBPF only for LE builds We can go with LE first. Thanks > > Jason, Daniel, Michael > Can you please let us know what you think and why? > > On Thu, Nov 5, 2020 at 3:19 PM Daniel P. Berrangé <berrange@redhat.com > <mailto:berrange@redhat.com>> wrote: > > On Thu, Nov 05, 2020 at 10:01:09AM +0000, Daniel P. Berrangé wrote: > > On Thu, Nov 05, 2020 at 11:46:18AM +0800, Jason Wang wrote: > > > > > > On 2020/11/4 下午5:31, Daniel P. Berrangé wrote: > > > > On Wed, Nov 04, 2020 at 10:07:52AM +0800, Jason Wang wrote: > > > > > On 2020/11/3 下午6:32, Yuri Benditovich wrote: > > > > > > > > > > > > On Tue, Nov 3, 2020 at 11:02 AM Jason Wang > <jasowang@redhat.com <mailto:jasowang@redhat.com> > > > > > > <mailto:jasowang@redhat.com > <mailto:jasowang@redhat.com>>> wrote: > > > > > > > > > > > > > > > > > > On 2020/11/3 上午2:51, Andrew Melnychenko wrote: > > > > > > > Basic idea is to use eBPF to calculate and steer > packets in TAP. > > > > > > > RSS(Receive Side Scaling) is used to distribute > network packets > > > > > > to guest virtqueues > > > > > > > by calculating packet hash. > > > > > > > eBPF RSS allows us to use RSS with vhost TAP. > > > > > > > > > > > > > > This set of patches introduces the usage of eBPF > for packet steering > > > > > > > and RSS hash calculation: > > > > > > > * RSS(Receive Side Scaling) is used to distribute > network packets to > > > > > > > guest virtqueues by calculating packet hash > > > > > > > * eBPF RSS suppose to be faster than already > existing 'software' > > > > > > > implementation in QEMU > > > > > > > * Additionally adding support for the usage of > RSS with vhost > > > > > > > > > > > > > > Supported kernels: 5.8+ > > > > > > > > > > > > > > Implementation notes: > > > > > > > Linux TAP TUNSETSTEERINGEBPF ioctl was used to > set the eBPF program. > > > > > > > Added eBPF support to qemu directly through a > system call, see the > > > > > > > bpf(2) for details. > > > > > > > The eBPF program is part of the qemu and > presented as an array > > > > > > of bpf > > > > > > > instructions. > > > > > > > The program can be recompiled by provided > Makefile.ebpf(need to > > > > > > adjust > > > > > > > 'linuxhdrs'), > > > > > > > although it's not required to build QEMU with > eBPF support. > > > > > > > Added changes to virtio-net and vhost, primary > eBPF RSS is used. > > > > > > > 'Software' RSS used in the case of hash > population and as a > > > > > > fallback option. > > > > > > > For vhost, the hash population feature is not > reported to the guest. > > > > > > > > > > > > > > Please also see the documentation in PATCH 6/6. > > > > > > > > > > > > > > I am sending those patches as RFC to initiate the > discussions > > > > > > and get > > > > > > > feedback on the following points: > > > > > > > * Fallback when eBPF is not supported by the kernel > > > > > > > > > > > > > > > > > > Yes, and it could also a lacking of CAP_BPF. > > > > > > > > > > > > > > > > > > > * Live migration to the kernel that doesn't have > eBPF support > > > > > > > > > > > > > > > > > > Is there anything that we needs special treatment here? > > > > > > > > > > > > Possible case: rss=on, vhost=on, source system with > kernel 5.8 > > > > > > (everything works) -> dest. system 5.6 (bpf does not > work), the adapter > > > > > > functions, but all the steering does not use proper queues. > > > > > > > > > > Right, I think we need to disable vhost on dest. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > * Integration with current QEMU build > > > > > > > > > > > > > > > > > > Yes, a question here: > > > > > > > > > > > > 1) Any reason for not using libbpf, e.g it has been > shipped with some > > > > > > distros > > > > > > > > > > > > > > > > > > We intentionally do not use libbpf, as it present only > on some distros. > > > > > > We can switch to libbpf, but this will disable bpf if > libbpf is not > > > > > > installed > > > > > > > > > > That's better I think. > > > > > > > > > > > > > > > > 2) It would be better if we can avoid shipping > bytecodes > > > > > > > > > > > > > > > > > > > > > > > > This creates new dependencies: llvm + clang + ... > > > > > > We would prefer byte code and ability to generate it if > prerequisites > > > > > > are installed. > > > > > > > > > > It's probably ok if we treat the bytecode as a kind of > firmware. > > > > That is explicitly *not* OK for inclusion in Fedora. They > require that > > > > BPF is compiled from source, and rejected my suggestion that > it could > > > > be considered a kind of firmware and thus have an exception > from building > > > > from source. > > > > > > > > > Please refer what it was done in DPDK: > > > > > > http://git.dpdk.org/dpdk/tree/doc/guides/nics/tap.rst#n235 > > > > > > I don't think what proposed here makes anything different. > > > > I'm not convinced that what DPDK does is acceptable to Fedora either > > based on the responses I've received when asking about BPF handling > > during build. I wouldn't suprise me, however, if this was simply > > missed by reviewers when accepting DPDK into Fedora, because it is > > not entirely obvious unless you are looking closely. > > FWIW, I'm pushing back against the idea that we have to compile the > BPF code from master source, as I think it is reasonable to have the > program embedded as a static array in the source code similar to what > DPDK does. It doesn't feel much different from other places where > apps > use generated sources, and don't build them from the original source > every time. eg "configure" is never re-generated from > "configure.ac <http://configure.ac>" > by Fedora packagers, they just use the generated "configure" script > as-is. > > Regards, > Daniel > -- > |: https://berrange.com -o- > https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- > https://www.instagram.com/dberrange :| >
On Mon, Nov 9, 2020 at 4:14 AM Jason Wang <jasowang@redhat.com> wrote: > > On 2020/11/5 下午11:13, Yuri Benditovich wrote: > > First of all, thank you for all your feedbacks > > > > Please help me to summarize and let us understand better what we do in > v2: > > Major questions are: > > 1. Building eBPF from source during qemu build vs. regenerating it on > > demand and keeping in the repository > > Solution 1a (~ as in v1): keep instructions or ELF in H file, generate > > it out of qemu build. In general we'll need to have BE and LE binaries. > > Solution 1b: build ELF or instructions during QEMU build if llvm + > > clang exist. Then we will have only one (BE or LE, depending on > > current QEMU build) > > We agree with any solution - I believe you know the requirements better. > > > I think we can go with 1a. (See Danial's comment) > > > > > > 2. Use libbpf or not > > In general we do not see any advantage of using libbpf. It works with > > object files (does ELF parsing at time of loading), but it does not do > > any magic. > > Solution 2a. Switch to libbpf, generate object files (LE and BE) from > > source, keep them inside QEMU (~8k each) or aside > > > Can we simply use dynamic linking here? > > Can you please explain, where exactly you suggest to use dynamic linking? > > > Solution 2b. (as in v1) Use python script to parse object -> > > instructions (~2k each) > > We'd prefer not to use libbpf at the moment. > > If due to some unknown reason we'll find it useful in future, we can > > switch to it, this does not create any incompatibility. Then this will > > create a dependency on libbpf.so > > > I think we need to care about compatibility. E.g we need to enable BTF > so I don't know how hard if we add BTF support in the current design. It > would be probably OK it's not a lot of effort. > > As far as we understand BTF helps in BPF debugging and libbpf supports it as is. Without libbpf we in v1 load the BPF instructions only. If you think the BTF is mandatory (BTW, why?) I think it is better to switch to libbpf and keep the entire ELF in the qemu data. > > > > 3. Keep instructions or ELF inside QEMU or as separate external file > > Solution 3a (~as in v1): Built-in array of instructions or ELF. If we > > generate them out of QEMU build - keep 2 arrays or instructions or ELF > > (BE and LE), > > Solution 3b: Install them as separate files (/usr/share/qemu). > > We'd prefer 3a: > > Then there is a guarantee that the eBPF is built with exactly the > > same config structures as QEMU (qemu creates a mapping of its > > structures, eBPF uses them). > > No need to take care on scenarios like 'file not found', 'file is not > > suitable' etc > > > Yes, let's go 3a for upstream. > > > > > > 4. Is there some real request to have the eBPF for big-endian? > > If no, we can enable eBPF only for LE builds > > > We can go with LE first. > > Thanks > > > > > > Jason, Daniel, Michael > > Can you please let us know what you think and why? > > > > On Thu, Nov 5, 2020 at 3:19 PM Daniel P. Berrangé <berrange@redhat.com > > <mailto:berrange@redhat.com>> wrote: > > > > On Thu, Nov 05, 2020 at 10:01:09AM +0000, Daniel P. Berrangé wrote: > > > On Thu, Nov 05, 2020 at 11:46:18AM +0800, Jason Wang wrote: > > > > > > > > On 2020/11/4 下午5:31, Daniel P. Berrangé wrote: > > > > > On Wed, Nov 04, 2020 at 10:07:52AM +0800, Jason Wang wrote: > > > > > > On 2020/11/3 下午6:32, Yuri Benditovich wrote: > > > > > > > > > > > > > > On Tue, Nov 3, 2020 at 11:02 AM Jason Wang > > <jasowang@redhat.com <mailto:jasowang@redhat.com> > > > > > > > <mailto:jasowang@redhat.com > > <mailto:jasowang@redhat.com>>> wrote: > > > > > > > > > > > > > > > > > > > > > On 2020/11/3 上午2:51, Andrew Melnychenko wrote: > > > > > > > > Basic idea is to use eBPF to calculate and steer > > packets in TAP. > > > > > > > > RSS(Receive Side Scaling) is used to distribute > > network packets > > > > > > > to guest virtqueues > > > > > > > > by calculating packet hash. > > > > > > > > eBPF RSS allows us to use RSS with vhost TAP. > > > > > > > > > > > > > > > > This set of patches introduces the usage of eBPF > > for packet steering > > > > > > > > and RSS hash calculation: > > > > > > > > * RSS(Receive Side Scaling) is used to distribute > > network packets to > > > > > > > > guest virtqueues by calculating packet hash > > > > > > > > * eBPF RSS suppose to be faster than already > > existing 'software' > > > > > > > > implementation in QEMU > > > > > > > > * Additionally adding support for the usage of > > RSS with vhost > > > > > > > > > > > > > > > > Supported kernels: 5.8+ > > > > > > > > > > > > > > > > Implementation notes: > > > > > > > > Linux TAP TUNSETSTEERINGEBPF ioctl was used to > > set the eBPF program. > > > > > > > > Added eBPF support to qemu directly through a > > system call, see the > > > > > > > > bpf(2) for details. > > > > > > > > The eBPF program is part of the qemu and > > presented as an array > > > > > > > of bpf > > > > > > > > instructions. > > > > > > > > The program can be recompiled by provided > > Makefile.ebpf(need to > > > > > > > adjust > > > > > > > > 'linuxhdrs'), > > > > > > > > although it's not required to build QEMU with > > eBPF support. > > > > > > > > Added changes to virtio-net and vhost, primary > > eBPF RSS is used. > > > > > > > > 'Software' RSS used in the case of hash > > population and as a > > > > > > > fallback option. > > > > > > > > For vhost, the hash population feature is not > > reported to the guest. > > > > > > > > > > > > > > > > Please also see the documentation in PATCH 6/6. > > > > > > > > > > > > > > > > I am sending those patches as RFC to initiate the > > discussions > > > > > > > and get > > > > > > > > feedback on the following points: > > > > > > > > * Fallback when eBPF is not supported by the kernel > > > > > > > > > > > > > > > > > > > > > Yes, and it could also a lacking of CAP_BPF. > > > > > > > > > > > > > > > > > > > > > > * Live migration to the kernel that doesn't have > > eBPF support > > > > > > > > > > > > > > > > > > > > > Is there anything that we needs special treatment > here? > > > > > > > > > > > > > > Possible case: rss=on, vhost=on, source system with > > kernel 5.8 > > > > > > > (everything works) -> dest. system 5.6 (bpf does not > > work), the adapter > > > > > > > functions, but all the steering does not use proper queues. > > > > > > > > > > > > Right, I think we need to disable vhost on dest. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > * Integration with current QEMU build > > > > > > > > > > > > > > > > > > > > > Yes, a question here: > > > > > > > > > > > > > > 1) Any reason for not using libbpf, e.g it has been > > shipped with some > > > > > > > distros > > > > > > > > > > > > > > > > > > > > > We intentionally do not use libbpf, as it present only > > on some distros. > > > > > > > We can switch to libbpf, but this will disable bpf if > > libbpf is not > > > > > > > installed > > > > > > > > > > > > That's better I think. > > > > > > > > > > > > > > > > > > > 2) It would be better if we can avoid shipping > > bytecodes > > > > > > > > > > > > > > > > > > > > > > > > > > > > This creates new dependencies: llvm + clang + ... > > > > > > > We would prefer byte code and ability to generate it if > > prerequisites > > > > > > > are installed. > > > > > > > > > > > > It's probably ok if we treat the bytecode as a kind of > > firmware. > > > > > That is explicitly *not* OK for inclusion in Fedora. They > > require that > > > > > BPF is compiled from source, and rejected my suggestion that > > it could > > > > > be considered a kind of firmware and thus have an exception > > from building > > > > > from source. > > > > > > > > > > > > Please refer what it was done in DPDK: > > > > > > > > http://git.dpdk.org/dpdk/tree/doc/guides/nics/tap.rst#n235 > > > > > > > > I don't think what proposed here makes anything different. > > > > > > I'm not convinced that what DPDK does is acceptable to Fedora > either > > > based on the responses I've received when asking about BPF handling > > > during build. I wouldn't suprise me, however, if this was simply > > > missed by reviewers when accepting DPDK into Fedora, because it is > > > not entirely obvious unless you are looking closely. > > > > FWIW, I'm pushing back against the idea that we have to compile the > > BPF code from master source, as I think it is reasonable to have the > > program embedded as a static array in the source code similar to what > > DPDK does. It doesn't feel much different from other places where > > apps > > use generated sources, and don't build them from the original source > > every time. eg "configure" is never re-generated from > > "configure.ac <http://configure.ac>" > > by Fedora packagers, they just use the generated "configure" script > > as-is. > > > > Regards, > > Daniel > > -- > > |: https://berrange.com -o- > > https://www.flickr.com/photos/dberrange :| > > |: https://libvirt.org -o- https://fstop138.berrange.com :| > > |: https://entangle-photo.org -o- > > https://www.instagram.com/dberrange :| > > > > <div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Nov 9, 2020 at 4:14 AM Jason Wang <<a href="mailto:jasowang@redhat.com">jasowang@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br> On 2020/11/5 下午11:13, Yuri Benditovich wrote:<br> > First of all, thank you for all your feedbacks<br> ><br> > Please help me to summarize and let us understand better what we do in v2:<br> > Major questions are:<br> > 1. Building eBPF from source during qemu build vs. regenerating it on <br> > demand and keeping in the repository<br> > Solution 1a (~ as in v1): keep instructions or ELF in H file, generate <br> > it out of qemu build. In general we'll need to have BE and LE binaries.<br> > Solution 1b: build ELF or instructions during QEMU build if llvm + <br> > clang exist. Then we will have only one (BE or LE, depending on <br> > current QEMU build)<br> > We agree with any solution - I believe you know the requirements better.<br> <br> <br> I think we can go with 1a. (See Danial's comment)<br> <br> <br> ><br> > 2. Use libbpf or not<br> > In general we do not see any advantage of using libbpf. It works with <br> > object files (does ELF parsing at time of loading), but it does not do <br> > any magic.<br> > Solution 2a. Switch to libbpf, generate object files (LE and BE) from <br> > source, keep them inside QEMU (~8k each) or aside<br> <br> <br> Can we simply use dynamic linking here?<br> <br></blockquote><div><br></div><div>Can you please explain, where exactly you suggest to use dynamic linking?<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br> > Solution 2b. (as in v1) Use python script to parse object -> <br> > instructions (~2k each)<br> > We'd prefer not to use libbpf at the moment.<br> > If due to some unknown reason we'll find it useful in future, we can <br> > switch to it, this does not create any incompatibility. Then this will <br> > create a dependency on libbpf.so<br> <br> <br> I think we need to care about compatibility. E.g we need to enable BTF <br> so I don't know how hard if we add BTF support in the current design. It <br> would be probably OK it's not a lot of effort.<br> <br></blockquote><div><br></div><div>As far as we understand BTF helps in BPF debugging and libbpf supports it as is.<br></div><div>Without libbpf we in v1 load the BPF instructions only.</div><div>If you think the BTF is mandatory (BTW, why?) I think it is better to switch to libbpf and keep the entire ELF in the qemu data.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br> ><br> > 3. Keep instructions or ELF inside QEMU or as separate external file<br> > Solution 3a (~as in v1): Built-in array of instructions or ELF. If we <br> > generate them out of QEMU build - keep 2 arrays or instructions or ELF <br> > (BE and LE),<br> > Solution 3b: Install them as separate files (/usr/share/qemu).<br> > We'd prefer 3a:<br> > Then there is a guarantee that the eBPF is built with exactly the <br> > same config structures as QEMU (qemu creates a mapping of its <br> > structures, eBPF uses them).<br> > No need to take care on scenarios like 'file not found', 'file is not <br> > suitable' etc<br> <br> <br> Yes, let's go 3a for upstream.<br> <br> <br> ><br> > 4. Is there some real request to have the eBPF for big-endian?<br> > If no, we can enable eBPF only for LE builds<br> <br> <br> We can go with LE first.<br> <br> Thanks<br> <br> <br> ><br> > Jason, Daniel, Michael<br> > Can you please let us know what you think and why?<br> ><br> > On Thu, Nov 5, 2020 at 3:19 PM Daniel P. Berrangé <<a href="mailto:berrange@redhat.com" target="_blank">berrange@redhat.com</a> <br> > <mailto:<a href="mailto:berrange@redhat.com" target="_blank">berrange@redhat.com</a>>> wrote:<br> ><br> > On Thu, Nov 05, 2020 at 10:01:09AM +0000, Daniel P. Berrangé wrote:<br> > > On Thu, Nov 05, 2020 at 11:46:18AM +0800, Jason Wang wrote:<br> > > ><br> > > > On 2020/11/4 下午5:31, Daniel P. Berrangé wrote:<br> > > > > On Wed, Nov 04, 2020 at 10:07:52AM +0800, Jason Wang wrote:<br> > > > > > On 2020/11/3 下午6:32, Yuri Benditovich wrote:<br> > > > > > ><br> > > > > > > On Tue, Nov 3, 2020 at 11:02 AM Jason Wang<br> > <<a href="mailto:jasowang@redhat.com" target="_blank">jasowang@redhat.com</a> <mailto:<a href="mailto:jasowang@redhat.com" target="_blank">jasowang@redhat.com</a>><br> > > > > > > <mailto:<a href="mailto:jasowang@redhat.com" target="_blank">jasowang@redhat.com</a><br> > <mailto:<a href="mailto:jasowang@redhat.com" target="_blank">jasowang@redhat.com</a>>>> wrote:<br> > > > > > ><br> > > > > > ><br> > > > > > > On 2020/11/3 上午2:51, Andrew Melnychenko wrote:<br> > > > > > > > Basic idea is to use eBPF to calculate and steer<br> > packets in TAP.<br> > > > > > > > RSS(Receive Side Scaling) is used to distribute<br> > network packets<br> > > > > > > to guest virtqueues<br> > > > > > > > by calculating packet hash.<br> > > > > > > > eBPF RSS allows us to use RSS with vhost TAP.<br> > > > > > > ><br> > > > > > > > This set of patches introduces the usage of eBPF<br> > for packet steering<br> > > > > > > > and RSS hash calculation:<br> > > > > > > > * RSS(Receive Side Scaling) is used to distribute<br> > network packets to<br> > > > > > > > guest virtqueues by calculating packet hash<br> > > > > > > > * eBPF RSS suppose to be faster than already<br> > existing 'software'<br> > > > > > > > implementation in QEMU<br> > > > > > > > * Additionally adding support for the usage of<br> > RSS with vhost<br> > > > > > > ><br> > > > > > > > Supported kernels: 5.8+<br> > > > > > > ><br> > > > > > > > Implementation notes:<br> > > > > > > > Linux TAP TUNSETSTEERINGEBPF ioctl was used to<br> > set the eBPF program.<br> > > > > > > > Added eBPF support to qemu directly through a<br> > system call, see the<br> > > > > > > > bpf(2) for details.<br> > > > > > > > The eBPF program is part of the qemu and<br> > presented as an array<br> > > > > > > of bpf<br> > > > > > > > instructions.<br> > > > > > > > The program can be recompiled by provided<br> > Makefile.ebpf(need to<br> > > > > > > adjust<br> > > > > > > > 'linuxhdrs'),<br> > > > > > > > although it's not required to build QEMU with<br> > eBPF support.<br> > > > > > > > Added changes to virtio-net and vhost, primary<br> > eBPF RSS is used.<br> > > > > > > > 'Software' RSS used in the case of hash<br> > population and as a<br> > > > > > > fallback option.<br> > > > > > > > For vhost, the hash population feature is not<br> > reported to the guest.<br> > > > > > > ><br> > > > > > > > Please also see the documentation in PATCH 6/6.<br> > > > > > > ><br> > > > > > > > I am sending those patches as RFC to initiate the<br> > discussions<br> > > > > > > and get<br> > > > > > > > feedback on the following points:<br> > > > > > > > * Fallback when eBPF is not supported by the kernel<br> > > > > > ><br> > > > > > ><br> > > > > > > Yes, and it could also a lacking of CAP_BPF.<br> > > > > > ><br> > > > > > ><br> > > > > > > > * Live migration to the kernel that doesn't have<br> > eBPF support<br> > > > > > ><br> > > > > > ><br> > > > > > > Is there anything that we needs special treatment here?<br> > > > > > ><br> > > > > > > Possible case: rss=on, vhost=on, source system with<br> > kernel 5.8<br> > > > > > > (everything works) -> dest. system 5.6 (bpf does not<br> > work), the adapter<br> > > > > > > functions, but all the steering does not use proper queues.<br> > > > > ><br> > > > > > Right, I think we need to disable vhost on dest.<br> > > > > ><br> > > > > ><br> > > > > > ><br> > > > > > ><br> > > > > > > > * Integration with current QEMU build<br> > > > > > ><br> > > > > > ><br> > > > > > > Yes, a question here:<br> > > > > > ><br> > > > > > > 1) Any reason for not using libbpf, e.g it has been<br> > shipped with some<br> > > > > > > distros<br> > > > > > ><br> > > > > > ><br> > > > > > > We intentionally do not use libbpf, as it present only<br> > on some distros.<br> > > > > > > We can switch to libbpf, but this will disable bpf if<br> > libbpf is not<br> > > > > > > installed<br> > > > > ><br> > > > > > That's better I think.<br> > > > > ><br> > > > > ><br> > > > > > > 2) It would be better if we can avoid shipping<br> > bytecodes<br> > > > > > ><br> > > > > > ><br> > > > > > ><br> > > > > > > This creates new dependencies: llvm + clang + ...<br> > > > > > > We would prefer byte code and ability to generate it if<br> > prerequisites<br> > > > > > > are installed.<br> > > > > ><br> > > > > > It's probably ok if we treat the bytecode as a kind of<br> > firmware.<br> > > > > That is explicitly *not* OK for inclusion in Fedora. They<br> > require that<br> > > > > BPF is compiled from source, and rejected my suggestion that<br> > it could<br> > > > > be considered a kind of firmware and thus have an exception<br> > from building<br> > > > > from source.<br> > > ><br> > > ><br> > > > Please refer what it was done in DPDK:<br> > > ><br> > > > <a href="http://git.dpdk.org/dpdk/tree/doc/guides/nics/tap.rst#n235" rel="noreferrer" target="_blank">http://git.dpdk.org/dpdk/tree/doc/guides/nics/tap.rst#n235</a><br> > > ><br> > > > I don't think what proposed here makes anything different.<br> > ><br> > > I'm not convinced that what DPDK does is acceptable to Fedora either<br> > > based on the responses I've received when asking about BPF handling<br> > > during build. I wouldn't suprise me, however, if this was simply<br> > > missed by reviewers when accepting DPDK into Fedora, because it is<br> > > not entirely obvious unless you are looking closely.<br> ><br> > FWIW, I'm pushing back against the idea that we have to compile the<br> > BPF code from master source, as I think it is reasonable to have the<br> > program embedded as a static array in the source code similar to what<br> > DPDK does. It doesn't feel much different from other places where<br> > apps<br> > use generated sources, and don't build them from the original source<br> > every time. eg "configure" is never re-generated from<br> > "<a href="http://configure.ac" rel="noreferrer" target="_blank">configure.ac</a> <<a href="http://configure.ac" rel="noreferrer" target="_blank">http://configure.ac</a>>"<br> > by Fedora packagers, they just use the generated "configure" script<br> > as-is.<br> ><br> > Regards,<br> > Daniel<br> > -- <br> > |: <a href="https://berrange.com" rel="noreferrer" target="_blank">https://berrange.com</a> -o-<br> > <a href="https://www.flickr.com/photos/dberrange" rel="noreferrer" target="_blank">https://www.flickr.com/photos/dberrange</a> :|<br> > |: <a href="https://libvirt.org" rel="noreferrer" target="_blank">https://libvirt.org</a> -o- <a href="https://fstop138.berrange.com" rel="noreferrer" target="_blank">https://fstop138.berrange.com</a> :|<br> > |: <a href="https://entangle-photo.org" rel="noreferrer" target="_blank">https://entangle-photo.org</a> -o-<br> > <a href="https://www.instagram.com/dberrange" rel="noreferrer" target="_blank">https://www.instagram.com/dberrange</a> :|<br> ><br> <br> </blockquote></div></div>
On 2020/11/9 下午9:33, Yuri Benditovich wrote: > > > On Mon, Nov 9, 2020 at 4:14 AM Jason Wang <jasowang@redhat.com > <mailto:jasowang@redhat.com>> wrote: > > > On 2020/11/5 下午11:13, Yuri Benditovich wrote: > > First of all, thank you for all your feedbacks > > > > Please help me to summarize and let us understand better what we > do in v2: > > Major questions are: > > 1. Building eBPF from source during qemu build vs. regenerating > it on > > demand and keeping in the repository > > Solution 1a (~ as in v1): keep instructions or ELF in H file, > generate > > it out of qemu build. In general we'll need to have BE and LE > binaries. > > Solution 1b: build ELF or instructions during QEMU build if llvm + > > clang exist. Then we will have only one (BE or LE, depending on > > current QEMU build) > > We agree with any solution - I believe you know the requirements > better. > > > I think we can go with 1a. (See Danial's comment) > > > > > > 2. Use libbpf or not > > In general we do not see any advantage of using libbpf. It works > with > > object files (does ELF parsing at time of loading), but it does > not do > > any magic. > > Solution 2a. Switch to libbpf, generate object files (LE and BE) > from > > source, keep them inside QEMU (~8k each) or aside > > > Can we simply use dynamic linking here? > > > Can you please explain, where exactly you suggest to use dynamic linking? Yes. If I understand your 2a properly, you meant static linking of libbpf. So what I want to ask is the possibility of dynamic linking of libbpf here. > > > Solution 2b. (as in v1) Use python script to parse object -> > > instructions (~2k each) > > We'd prefer not to use libbpf at the moment. > > If due to some unknown reason we'll find it useful in future, we > can > > switch to it, this does not create any incompatibility. Then > this will > > create a dependency on libbpf.so > > > I think we need to care about compatibility. E.g we need to enable > BTF > so I don't know how hard if we add BTF support in the current > design. It > would be probably OK it's not a lot of effort. > > > As far as we understand BTF helps in BPF debugging and libbpf supports > it as is. > Without libbpf we in v1 load the BPF instructions only. > If you think the BTF is mandatory (BTW, why?) I think it is better to > switch to libbpf and keep the entire ELF in the qemu data. It is used to make sure the BPF can do compile once run everywhere. This is explained in detail in here: https://facebookmicrosites.github.io/bpf/blog/2020/02/19/bpf-portability-and-co-re.html. Thanks > > > > > > 3. Keep instructions or ELF inside QEMU or as separate external file > > Solution 3a (~as in v1): Built-in array of instructions or ELF. > If we > > generate them out of QEMU build - keep 2 arrays or instructions > or ELF > > (BE and LE), > > Solution 3b: Install them as separate files (/usr/share/qemu). > > We'd prefer 3a: > > Then there is a guarantee that the eBPF is built with exactly the > > same config structures as QEMU (qemu creates a mapping of its > > structures, eBPF uses them). > > No need to take care on scenarios like 'file not found', 'file > is not > > suitable' etc > > > Yes, let's go 3a for upstream. > > > > > > 4. Is there some real request to have the eBPF for big-endian? > > If no, we can enable eBPF only for LE builds > > > We can go with LE first. > > Thanks > > > > > > Jason, Daniel, Michael > > Can you please let us know what you think and why? > > > > On Thu, Nov 5, 2020 at 3:19 PM Daniel P. Berrangé > <berrange@redhat.com <mailto:berrange@redhat.com> > > <mailto:berrange@redhat.com <mailto:berrange@redhat.com>>> wrote: > > > > On Thu, Nov 05, 2020 at 10:01:09AM +0000, Daniel P. Berrangé > wrote: > > > On Thu, Nov 05, 2020 at 11:46:18AM +0800, Jason Wang wrote: > > > > > > > > On 2020/11/4 下午5:31, Daniel P. Berrangé wrote: > > > > > On Wed, Nov 04, 2020 at 10:07:52AM +0800, Jason Wang > wrote: > > > > > > On 2020/11/3 下午6:32, Yuri Benditovich wrote: > > > > > > > > > > > > > > On Tue, Nov 3, 2020 at 11:02 AM Jason Wang > > <jasowang@redhat.com <mailto:jasowang@redhat.com> > <mailto:jasowang@redhat.com <mailto:jasowang@redhat.com>> > > > > > > > <mailto:jasowang@redhat.com > <mailto:jasowang@redhat.com> > > <mailto:jasowang@redhat.com <mailto:jasowang@redhat.com>>>> > wrote: > > > > > > > > > > > > > > > > > > > > > On 2020/11/3 上午2:51, Andrew Melnychenko wrote: > > > > > > > > Basic idea is to use eBPF to calculate and > steer > > packets in TAP. > > > > > > > > RSS(Receive Side Scaling) is used to distribute > > network packets > > > > > > > to guest virtqueues > > > > > > > > by calculating packet hash. > > > > > > > > eBPF RSS allows us to use RSS with vhost TAP. > > > > > > > > > > > > > > > > This set of patches introduces the usage of > eBPF > > for packet steering > > > > > > > > and RSS hash calculation: > > > > > > > > * RSS(Receive Side Scaling) is used to > distribute > > network packets to > > > > > > > > guest virtqueues by calculating packet hash > > > > > > > > * eBPF RSS suppose to be faster than already > > existing 'software' > > > > > > > > implementation in QEMU > > > > > > > > * Additionally adding support for the usage of > > RSS with vhost > > > > > > > > > > > > > > > > Supported kernels: 5.8+ > > > > > > > > > > > > > > > > Implementation notes: > > > > > > > > Linux TAP TUNSETSTEERINGEBPF ioctl was used to > > set the eBPF program. > > > > > > > > Added eBPF support to qemu directly through a > > system call, see the > > > > > > > > bpf(2) for details. > > > > > > > > The eBPF program is part of the qemu and > > presented as an array > > > > > > > of bpf > > > > > > > > instructions. > > > > > > > > The program can be recompiled by provided > > Makefile.ebpf(need to > > > > > > > adjust > > > > > > > > 'linuxhdrs'), > > > > > > > > although it's not required to build QEMU with > > eBPF support. > > > > > > > > Added changes to virtio-net and vhost, primary > > eBPF RSS is used. > > > > > > > > 'Software' RSS used in the case of hash > > population and as a > > > > > > > fallback option. > > > > > > > > For vhost, the hash population feature is not > > reported to the guest. > > > > > > > > > > > > > > > > Please also see the documentation in PATCH 6/6. > > > > > > > > > > > > > > > > I am sending those patches as RFC to > initiate the > > discussions > > > > > > > and get > > > > > > > > feedback on the following points: > > > > > > > > * Fallback when eBPF is not supported by > the kernel > > > > > > > > > > > > > > > > > > > > > Yes, and it could also a lacking of CAP_BPF. > > > > > > > > > > > > > > > > > > > > > > * Live migration to the kernel that doesn't > have > > eBPF support > > > > > > > > > > > > > > > > > > > > > Is there anything that we needs special > treatment here? > > > > > > > > > > > > > > Possible case: rss=on, vhost=on, source system with > > kernel 5.8 > > > > > > > (everything works) -> dest. system 5.6 (bpf does not > > work), the adapter > > > > > > > functions, but all the steering does not use > proper queues. > > > > > > > > > > > > Right, I think we need to disable vhost on dest. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > * Integration with current QEMU build > > > > > > > > > > > > > > > > > > > > > Yes, a question here: > > > > > > > > > > > > > > 1) Any reason for not using libbpf, e.g it > has been > > shipped with some > > > > > > > distros > > > > > > > > > > > > > > > > > > > > > We intentionally do not use libbpf, as it present only > > on some distros. > > > > > > > We can switch to libbpf, but this will disable bpf if > > libbpf is not > > > > > > > installed > > > > > > > > > > > > That's better I think. > > > > > > > > > > > > > > > > > > > 2) It would be better if we can avoid shipping > > bytecodes > > > > > > > > > > > > > > > > > > > > > > > > > > > > This creates new dependencies: llvm + clang + ... > > > > > > > We would prefer byte code and ability to generate > it if > > prerequisites > > > > > > > are installed. > > > > > > > > > > > > It's probably ok if we treat the bytecode as a kind of > > firmware. > > > > > That is explicitly *not* OK for inclusion in Fedora. They > > require that > > > > > BPF is compiled from source, and rejected my > suggestion that > > it could > > > > > be considered a kind of firmware and thus have an > exception > > from building > > > > > from source. > > > > > > > > > > > > Please refer what it was done in DPDK: > > > > > > > > http://git.dpdk.org/dpdk/tree/doc/guides/nics/tap.rst#n235 > > > > > > > > I don't think what proposed here makes anything different. > > > > > > I'm not convinced that what DPDK does is acceptable to > Fedora either > > > based on the responses I've received when asking about BPF > handling > > > during build. I wouldn't suprise me, however, if this was > simply > > > missed by reviewers when accepting DPDK into Fedora, > because it is > > > not entirely obvious unless you are looking closely. > > > > FWIW, I'm pushing back against the idea that we have to > compile the > > BPF code from master source, as I think it is reasonable to > have the > > program embedded as a static array in the source code > similar to what > > DPDK does. It doesn't feel much different from other places > where > > apps > > use generated sources, and don't build them from the > original source > > every time. eg "configure" is never re-generated from > > "configure.ac <http://configure.ac> <http://configure.ac>" > > by Fedora packagers, they just use the generated "configure" > script > > as-is. > > > > Regards, > > Daniel > > -- > > |: https://berrange.com -o- > > https://www.flickr.com/photos/dberrange :| > > |: https://libvirt.org -o- > https://fstop138.berrange.com :| > > |: https://entangle-photo.org -o- > > https://www.instagram.com/dberrange :| > > >
On Tue, Nov 10, 2020 at 4:23 AM Jason Wang <jasowang@redhat.com> wrote: > > On 2020/11/9 下午9:33, Yuri Benditovich wrote: > > > > > > On Mon, Nov 9, 2020 at 4:14 AM Jason Wang <jasowang@redhat.com > > <mailto:jasowang@redhat.com>> wrote: > > > > > > On 2020/11/5 下午11:13, Yuri Benditovich wrote: > > > First of all, thank you for all your feedbacks > > > > > > Please help me to summarize and let us understand better what we > > do in v2: > > > Major questions are: > > > 1. Building eBPF from source during qemu build vs. regenerating > > it on > > > demand and keeping in the repository > > > Solution 1a (~ as in v1): keep instructions or ELF in H file, > > generate > > > it out of qemu build. In general we'll need to have BE and LE > > binaries. > > > Solution 1b: build ELF or instructions during QEMU build if llvm + > > > clang exist. Then we will have only one (BE or LE, depending on > > > current QEMU build) > > > We agree with any solution - I believe you know the requirements > > better. > > > > > > I think we can go with 1a. (See Danial's comment) > > > > > > > > > > 2. Use libbpf or not > > > In general we do not see any advantage of using libbpf. It works > > with > > > object files (does ELF parsing at time of loading), but it does > > not do > > > any magic. > > > Solution 2a. Switch to libbpf, generate object files (LE and BE) > > from > > > source, keep them inside QEMU (~8k each) or aside > > > > > > Can we simply use dynamic linking here? > > > > > > Can you please explain, where exactly you suggest to use dynamic linking? > > > Yes. If I understand your 2a properly, you meant static linking of > libbpf. So what I want to ask is the possibility of dynamic linking of > libbpf here. > > As Daniel explained above, QEMU is always linked dynamically vs libraries. Also I see the libbpf package does not even contain the static library. If the build environment contains libbpf, the libbpf.so becomes runtime dependency, just as with other libs. > > > > > > Solution 2b. (as in v1) Use python script to parse object -> > > > instructions (~2k each) > > > We'd prefer not to use libbpf at the moment. > > > If due to some unknown reason we'll find it useful in future, we > > can > > > switch to it, this does not create any incompatibility. Then > > this will > > > create a dependency on libbpf.so > > > > > > I think we need to care about compatibility. E.g we need to enable > > BTF > > so I don't know how hard if we add BTF support in the current > > design. It > > would be probably OK it's not a lot of effort. > > > > > > As far as we understand BTF helps in BPF debugging and libbpf supports > > it as is. > > Without libbpf we in v1 load the BPF instructions only. > > If you think the BTF is mandatory (BTW, why?) I think it is better to > > switch to libbpf and keep the entire ELF in the qemu data. > > > It is used to make sure the BPF can do compile once run everywhere. > > This is explained in detail in here: > > https://facebookmicrosites.github.io/bpf/blog/2020/02/19/bpf-portability-and-co-re.html > . > > Thank you, then there is no question, we need to use libbpf. > Thanks > > > > > > > > > > > > 3. Keep instructions or ELF inside QEMU or as separate external > file > > > Solution 3a (~as in v1): Built-in array of instructions or ELF. > > If we > > > generate them out of QEMU build - keep 2 arrays or instructions > > or ELF > > > (BE and LE), > > > Solution 3b: Install them as separate files (/usr/share/qemu). > > > We'd prefer 3a: > > > Then there is a guarantee that the eBPF is built with exactly the > > > same config structures as QEMU (qemu creates a mapping of its > > > structures, eBPF uses them). > > > No need to take care on scenarios like 'file not found', 'file > > is not > > > suitable' etc > > > > > > Yes, let's go 3a for upstream. > > > > > > > > > > 4. Is there some real request to have the eBPF for big-endian? > > > If no, we can enable eBPF only for LE builds > > > > > > We can go with LE first. > > > > Thanks > > > > > > > > > > Jason, Daniel, Michael > > > Can you please let us know what you think and why? > > > > > > On Thu, Nov 5, 2020 at 3:19 PM Daniel P. Berrangé > > <berrange@redhat.com <mailto:berrange@redhat.com> > > > <mailto:berrange@redhat.com <mailto:berrange@redhat.com>>> wrote: > > > > > > On Thu, Nov 05, 2020 at 10:01:09AM +0000, Daniel P. Berrangé > > wrote: > > > > On Thu, Nov 05, 2020 at 11:46:18AM +0800, Jason Wang wrote: > > > > > > > > > > On 2020/11/4 下午5:31, Daniel P. Berrangé wrote: > > > > > > On Wed, Nov 04, 2020 at 10:07:52AM +0800, Jason Wang > > wrote: > > > > > > > On 2020/11/3 下午6:32, Yuri Benditovich wrote: > > > > > > > > > > > > > > > > On Tue, Nov 3, 2020 at 11:02 AM Jason Wang > > > <jasowang@redhat.com <mailto:jasowang@redhat.com> > > <mailto:jasowang@redhat.com <mailto:jasowang@redhat.com>> > > > > > > > > <mailto:jasowang@redhat.com > > <mailto:jasowang@redhat.com> > > > <mailto:jasowang@redhat.com <mailto:jasowang@redhat.com>>>> > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > On 2020/11/3 上午2:51, Andrew Melnychenko wrote: > > > > > > > > > Basic idea is to use eBPF to calculate and > > steer > > > packets in TAP. > > > > > > > > > RSS(Receive Side Scaling) is used to > distribute > > > network packets > > > > > > > > to guest virtqueues > > > > > > > > > by calculating packet hash. > > > > > > > > > eBPF RSS allows us to use RSS with vhost TAP. > > > > > > > > > > > > > > > > > > This set of patches introduces the usage of > > eBPF > > > for packet steering > > > > > > > > > and RSS hash calculation: > > > > > > > > > * RSS(Receive Side Scaling) is used to > > distribute > > > network packets to > > > > > > > > > guest virtqueues by calculating packet hash > > > > > > > > > * eBPF RSS suppose to be faster than already > > > existing 'software' > > > > > > > > > implementation in QEMU > > > > > > > > > * Additionally adding support for the usage of > > > RSS with vhost > > > > > > > > > > > > > > > > > > Supported kernels: 5.8+ > > > > > > > > > > > > > > > > > > Implementation notes: > > > > > > > > > Linux TAP TUNSETSTEERINGEBPF ioctl was used to > > > set the eBPF program. > > > > > > > > > Added eBPF support to qemu directly through a > > > system call, see the > > > > > > > > > bpf(2) for details. > > > > > > > > > The eBPF program is part of the qemu and > > > presented as an array > > > > > > > > of bpf > > > > > > > > > instructions. > > > > > > > > > The program can be recompiled by provided > > > Makefile.ebpf(need to > > > > > > > > adjust > > > > > > > > > 'linuxhdrs'), > > > > > > > > > although it's not required to build QEMU with > > > eBPF support. > > > > > > > > > Added changes to virtio-net and vhost, primary > > > eBPF RSS is used. > > > > > > > > > 'Software' RSS used in the case of hash > > > population and as a > > > > > > > > fallback option. > > > > > > > > > For vhost, the hash population feature is not > > > reported to the guest. > > > > > > > > > > > > > > > > > > Please also see the documentation in PATCH > 6/6. > > > > > > > > > > > > > > > > > > I am sending those patches as RFC to > > initiate the > > > discussions > > > > > > > > and get > > > > > > > > > feedback on the following points: > > > > > > > > > * Fallback when eBPF is not supported by > > the kernel > > > > > > > > > > > > > > > > > > > > > > > > Yes, and it could also a lacking of CAP_BPF. > > > > > > > > > > > > > > > > > > > > > > > > > * Live migration to the kernel that doesn't > > have > > > eBPF support > > > > > > > > > > > > > > > > > > > > > > > > Is there anything that we needs special > > treatment here? > > > > > > > > > > > > > > > > Possible case: rss=on, vhost=on, source system with > > > kernel 5.8 > > > > > > > > (everything works) -> dest. system 5.6 (bpf does not > > > work), the adapter > > > > > > > > functions, but all the steering does not use > > proper queues. > > > > > > > > > > > > > > Right, I think we need to disable vhost on dest. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > * Integration with current QEMU build > > > > > > > > > > > > > > > > > > > > > > > > Yes, a question here: > > > > > > > > > > > > > > > > 1) Any reason for not using libbpf, e.g it > > has been > > > shipped with some > > > > > > > > distros > > > > > > > > > > > > > > > > > > > > > > > > We intentionally do not use libbpf, as it present > only > > > on some distros. > > > > > > > > We can switch to libbpf, but this will disable bpf if > > > libbpf is not > > > > > > > > installed > > > > > > > > > > > > > > That's better I think. > > > > > > > > > > > > > > > > > > > > > > 2) It would be better if we can avoid shipping > > > bytecodes > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This creates new dependencies: llvm + clang + ... > > > > > > > > We would prefer byte code and ability to generate > > it if > > > prerequisites > > > > > > > > are installed. > > > > > > > > > > > > > > It's probably ok if we treat the bytecode as a kind of > > > firmware. > > > > > > That is explicitly *not* OK for inclusion in Fedora. They > > > require that > > > > > > BPF is compiled from source, and rejected my > > suggestion that > > > it could > > > > > > be considered a kind of firmware and thus have an > > exception > > > from building > > > > > > from source. > > > > > > > > > > > > > > > Please refer what it was done in DPDK: > > > > > > > > > > http://git.dpdk.org/dpdk/tree/doc/guides/nics/tap.rst#n235 > > > > > > > > > > I don't think what proposed here makes anything different. > > > > > > > > I'm not convinced that what DPDK does is acceptable to > > Fedora either > > > > based on the responses I've received when asking about BPF > > handling > > > > during build. I wouldn't suprise me, however, if this was > > simply > > > > missed by reviewers when accepting DPDK into Fedora, > > because it is > > > > not entirely obvious unless you are looking closely. > > > > > > FWIW, I'm pushing back against the idea that we have to > > compile the > > > BPF code from master source, as I think it is reasonable to > > have the > > > program embedded as a static array in the source code > > similar to what > > > DPDK does. It doesn't feel much different from other places > > where > > > apps > > > use generated sources, and don't build them from the > > original source > > > every time. eg "configure" is never re-generated from > > > "configure.ac <http://configure.ac> <http://configure.ac>" > > > by Fedora packagers, they just use the generated "configure" > > script > > > as-is. > > > > > > Regards, > > > Daniel > > > -- > > > |: https://berrange.com -o- > > > https://www.flickr.com/photos/dberrange :| > > > |: https://libvirt.org -o- > > https://fstop138.berrange.com :| > > > |: https://entangle-photo.org -o- > > > https://www.instagram.com/dberrange :| > > > > > > > <div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Nov 10, 2020 at 4:23 AM Jason Wang <<a href="mailto:jasowang@redhat.com">jasowang@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br> On 2020/11/9 下午9:33, Yuri Benditovich wrote:<br> ><br> ><br> > On Mon, Nov 9, 2020 at 4:14 AM Jason Wang <<a href="mailto:jasowang@redhat.com" target="_blank">jasowang@redhat.com</a> <br> > <mailto:<a href="mailto:jasowang@redhat.com" target="_blank">jasowang@redhat.com</a>>> wrote:<br> ><br> ><br> > On 2020/11/5 下午11:13, Yuri Benditovich wrote:<br> > > First of all, thank you for all your feedbacks<br> > ><br> > > Please help me to summarize and let us understand better what we<br> > do in v2:<br> > > Major questions are:<br> > > 1. Building eBPF from source during qemu build vs. regenerating<br> > it on<br> > > demand and keeping in the repository<br> > > Solution 1a (~ as in v1): keep instructions or ELF in H file,<br> > generate<br> > > it out of qemu build. In general we'll need to have BE and LE<br> > binaries.<br> > > Solution 1b: build ELF or instructions during QEMU build if llvm +<br> > > clang exist. Then we will have only one (BE or LE, depending on<br> > > current QEMU build)<br> > > We agree with any solution - I believe you know the requirements<br> > better.<br> ><br> ><br> > I think we can go with 1a. (See Danial's comment)<br> ><br> ><br> > ><br> > > 2. Use libbpf or not<br> > > In general we do not see any advantage of using libbpf. It works<br> > with<br> > > object files (does ELF parsing at time of loading), but it does<br> > not do<br> > > any magic.<br> > > Solution 2a. Switch to libbpf, generate object files (LE and BE)<br> > from<br> > > source, keep them inside QEMU (~8k each) or aside<br> ><br> ><br> > Can we simply use dynamic linking here?<br> ><br> ><br> > Can you please explain, where exactly you suggest to use dynamic linking?<br> <br> <br> Yes. If I understand your 2a properly, you meant static linking of <br> libbpf. So what I want to ask is the possibility of dynamic linking of <br> libbpf here.<br> <br></blockquote><div><br></div><div>As Daniel explained above, QEMU is always linked dynamically vs libraries.</div><div>Also I see the libbpf package does not even contain the static library.</div><div>If the build environment contains libbpf, the libbpf.so becomes runtime dependency, just as with other libs.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br> ><br> > > Solution 2b. (as in v1) Use python script to parse object -><br> > > instructions (~2k each)<br> > > We'd prefer not to use libbpf at the moment.<br> > > If due to some unknown reason we'll find it useful in future, we<br> > can<br> > > switch to it, this does not create any incompatibility. Then<br> > this will<br> > > create a dependency on libbpf.so<br> ><br> ><br> > I think we need to care about compatibility. E.g we need to enable<br> > BTF<br> > so I don't know how hard if we add BTF support in the current<br> > design. It<br> > would be probably OK it's not a lot of effort.<br> ><br> ><br> > As far as we understand BTF helps in BPF debugging and libbpf supports <br> > it as is.<br> > Without libbpf we in v1 load the BPF instructions only.<br> > If you think the BTF is mandatory (BTW, why?) I think it is better to <br> > switch to libbpf and keep the entire ELF in the qemu data.<br> <br> <br> It is used to make sure the BPF can do compile once run everywhere.<br> <br> This is explained in detail in here: <br> <a href="https://facebookmicrosites.github.io/bpf/blog/2020/02/19/bpf-portability-and-co-re.html" rel="noreferrer" target="_blank">https://facebookmicrosites.github.io/bpf/blog/2020/02/19/bpf-portability-and-co-re.html</a>.<br> <br> </blockquote><div><br></div><div>Thank you, then there is no question, we need to use libbpf.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Thanks<br> <br> <br> ><br> ><br> > ><br> > > 3. Keep instructions or ELF inside QEMU or as separate external file<br> > > Solution 3a (~as in v1): Built-in array of instructions or ELF.<br> > If we<br> > > generate them out of QEMU build - keep 2 arrays or instructions<br> > or ELF<br> > > (BE and LE),<br> > > Solution 3b: Install them as separate files (/usr/share/qemu).<br> > > We'd prefer 3a:<br> > > Then there is a guarantee that the eBPF is built with exactly the<br> > > same config structures as QEMU (qemu creates a mapping of its<br> > > structures, eBPF uses them).<br> > > No need to take care on scenarios like 'file not found', 'file<br> > is not<br> > > suitable' etc<br> ><br> ><br> > Yes, let's go 3a for upstream.<br> ><br> ><br> > ><br> > > 4. Is there some real request to have the eBPF for big-endian?<br> > > If no, we can enable eBPF only for LE builds<br> ><br> ><br> > We can go with LE first.<br> ><br> > Thanks<br> ><br> ><br> > ><br> > > Jason, Daniel, Michael<br> > > Can you please let us know what you think and why?<br> > ><br> > > On Thu, Nov 5, 2020 at 3:19 PM Daniel P. Berrangé<br> > <<a href="mailto:berrange@redhat.com" target="_blank">berrange@redhat.com</a> <mailto:<a href="mailto:berrange@redhat.com" target="_blank">berrange@redhat.com</a>><br> > > <mailto:<a href="mailto:berrange@redhat.com" target="_blank">berrange@redhat.com</a> <mailto:<a href="mailto:berrange@redhat.com" target="_blank">berrange@redhat.com</a>>>> wrote:<br> > ><br> > > On Thu, Nov 05, 2020 at 10:01:09AM +0000, Daniel P. Berrangé<br> > wrote:<br> > > > On Thu, Nov 05, 2020 at 11:46:18AM +0800, Jason Wang wrote:<br> > > > ><br> > > > > On 2020/11/4 下午5:31, Daniel P. Berrangé wrote:<br> > > > > > On Wed, Nov 04, 2020 at 10:07:52AM +0800, Jason Wang<br> > wrote:<br> > > > > > > On 2020/11/3 下午6:32, Yuri Benditovich wrote:<br> > > > > > > ><br> > > > > > > > On Tue, Nov 3, 2020 at 11:02 AM Jason Wang<br> > > <<a href="mailto:jasowang@redhat.com" target="_blank">jasowang@redhat.com</a> <mailto:<a href="mailto:jasowang@redhat.com" target="_blank">jasowang@redhat.com</a>><br> > <mailto:<a href="mailto:jasowang@redhat.com" target="_blank">jasowang@redhat.com</a> <mailto:<a href="mailto:jasowang@redhat.com" target="_blank">jasowang@redhat.com</a>>><br> > > > > > > > <mailto:<a href="mailto:jasowang@redhat.com" target="_blank">jasowang@redhat.com</a><br> > <mailto:<a href="mailto:jasowang@redhat.com" target="_blank">jasowang@redhat.com</a>><br> > > <mailto:<a href="mailto:jasowang@redhat.com" target="_blank">jasowang@redhat.com</a> <mailto:<a href="mailto:jasowang@redhat.com" target="_blank">jasowang@redhat.com</a>>>>><br> > wrote:<br> > > > > > > ><br> > > > > > > ><br> > > > > > > > On 2020/11/3 上午2:51, Andrew Melnychenko wrote:<br> > > > > > > > > Basic idea is to use eBPF to calculate and<br> > steer<br> > > packets in TAP.<br> > > > > > > > > RSS(Receive Side Scaling) is used to distribute<br> > > network packets<br> > > > > > > > to guest virtqueues<br> > > > > > > > > by calculating packet hash.<br> > > > > > > > > eBPF RSS allows us to use RSS with vhost TAP.<br> > > > > > > > ><br> > > > > > > > > This set of patches introduces the usage of<br> > eBPF<br> > > for packet steering<br> > > > > > > > > and RSS hash calculation:<br> > > > > > > > > * RSS(Receive Side Scaling) is used to<br> > distribute<br> > > network packets to<br> > > > > > > > > guest virtqueues by calculating packet hash<br> > > > > > > > > * eBPF RSS suppose to be faster than already<br> > > existing 'software'<br> > > > > > > > > implementation in QEMU<br> > > > > > > > > * Additionally adding support for the usage of<br> > > RSS with vhost<br> > > > > > > > ><br> > > > > > > > > Supported kernels: 5.8+<br> > > > > > > > ><br> > > > > > > > > Implementation notes:<br> > > > > > > > > Linux TAP TUNSETSTEERINGEBPF ioctl was used to<br> > > set the eBPF program.<br> > > > > > > > > Added eBPF support to qemu directly through a<br> > > system call, see the<br> > > > > > > > > bpf(2) for details.<br> > > > > > > > > The eBPF program is part of the qemu and<br> > > presented as an array<br> > > > > > > > of bpf<br> > > > > > > > > instructions.<br> > > > > > > > > The program can be recompiled by provided<br> > > Makefile.ebpf(need to<br> > > > > > > > adjust<br> > > > > > > > > 'linuxhdrs'),<br> > > > > > > > > although it's not required to build QEMU with<br> > > eBPF support.<br> > > > > > > > > Added changes to virtio-net and vhost, primary<br> > > eBPF RSS is used.<br> > > > > > > > > 'Software' RSS used in the case of hash<br> > > population and as a<br> > > > > > > > fallback option.<br> > > > > > > > > For vhost, the hash population feature is not<br> > > reported to the guest.<br> > > > > > > > ><br> > > > > > > > > Please also see the documentation in PATCH 6/6.<br> > > > > > > > ><br> > > > > > > > > I am sending those patches as RFC to<br> > initiate the<br> > > discussions<br> > > > > > > > and get<br> > > > > > > > > feedback on the following points:<br> > > > > > > > > * Fallback when eBPF is not supported by<br> > the kernel<br> > > > > > > ><br> > > > > > > ><br> > > > > > > > Yes, and it could also a lacking of CAP_BPF.<br> > > > > > > ><br> > > > > > > ><br> > > > > > > > > * Live migration to the kernel that doesn't<br> > have<br> > > eBPF support<br> > > > > > > ><br> > > > > > > ><br> > > > > > > > Is there anything that we needs special<br> > treatment here?<br> > > > > > > ><br> > > > > > > > Possible case: rss=on, vhost=on, source system with<br> > > kernel 5.8<br> > > > > > > > (everything works) -> dest. system 5.6 (bpf does not<br> > > work), the adapter<br> > > > > > > > functions, but all the steering does not use<br> > proper queues.<br> > > > > > ><br> > > > > > > Right, I think we need to disable vhost on dest.<br> > > > > > ><br> > > > > > ><br> > > > > > > ><br> > > > > > > ><br> > > > > > > > > * Integration with current QEMU build<br> > > > > > > ><br> > > > > > > ><br> > > > > > > > Yes, a question here:<br> > > > > > > ><br> > > > > > > > 1) Any reason for not using libbpf, e.g it<br> > has been<br> > > shipped with some<br> > > > > > > > distros<br> > > > > > > ><br> > > > > > > ><br> > > > > > > > We intentionally do not use libbpf, as it present only<br> > > on some distros.<br> > > > > > > > We can switch to libbpf, but this will disable bpf if<br> > > libbpf is not<br> > > > > > > > installed<br> > > > > > ><br> > > > > > > That's better I think.<br> > > > > > ><br> > > > > > ><br> > > > > > > > 2) It would be better if we can avoid shipping<br> > > bytecodes<br> > > > > > > ><br> > > > > > > ><br> > > > > > > ><br> > > > > > > > This creates new dependencies: llvm + clang + ...<br> > > > > > > > We would prefer byte code and ability to generate<br> > it if<br> > > prerequisites<br> > > > > > > > are installed.<br> > > > > > ><br> > > > > > > It's probably ok if we treat the bytecode as a kind of<br> > > firmware.<br> > > > > > That is explicitly *not* OK for inclusion in Fedora. They<br> > > require that<br> > > > > > BPF is compiled from source, and rejected my<br> > suggestion that<br> > > it could<br> > > > > > be considered a kind of firmware and thus have an<br> > exception<br> > > from building<br> > > > > > from source.<br> > > > ><br> > > > ><br> > > > > Please refer what it was done in DPDK:<br> > > > ><br> > > > > <a href="http://git.dpdk.org/dpdk/tree/doc/guides/nics/tap.rst#n235" rel="noreferrer" target="_blank">http://git.dpdk.org/dpdk/tree/doc/guides/nics/tap.rst#n235</a><br> > > > ><br> > > > > I don't think what proposed here makes anything different.<br> > > ><br> > > > I'm not convinced that what DPDK does is acceptable to<br> > Fedora either<br> > > > based on the responses I've received when asking about BPF<br> > handling<br> > > > during build. I wouldn't suprise me, however, if this was<br> > simply<br> > > > missed by reviewers when accepting DPDK into Fedora,<br> > because it is<br> > > > not entirely obvious unless you are looking closely.<br> > ><br> > > FWIW, I'm pushing back against the idea that we have to<br> > compile the<br> > > BPF code from master source, as I think it is reasonable to<br> > have the<br> > > program embedded as a static array in the source code<br> > similar to what<br> > > DPDK does. It doesn't feel much different from other places<br> > where<br> > > apps<br> > > use generated sources, and don't build them from the<br> > original source<br> > > every time. eg "configure" is never re-generated from<br> > > "<a href="http://configure.ac" rel="noreferrer" target="_blank">configure.ac</a> <<a href="http://configure.ac" rel="noreferrer" target="_blank">http://configure.ac</a>> <<a href="http://configure.ac" rel="noreferrer" target="_blank">http://configure.ac</a>>"<br> > > by Fedora packagers, they just use the generated "configure"<br> > script<br> > > as-is.<br> > ><br> > > Regards,<br> > > Daniel<br> > > --<br> > > |: <a href="https://berrange.com" rel="noreferrer" target="_blank">https://berrange.com</a> -o-<br> > > <a href="https://www.flickr.com/photos/dberrange" rel="noreferrer" target="_blank">https://www.flickr.com/photos/dberrange</a> :|<br> > > |: <a href="https://libvirt.org" rel="noreferrer" target="_blank">https://libvirt.org</a> -o-<br> > <a href="https://fstop138.berrange.com" rel="noreferrer" target="_blank">https://fstop138.berrange.com</a> :|<br> > > |: <a href="https://entangle-photo.org" rel="noreferrer" target="_blank">https://entangle-photo.org</a> -o-<br> > > <a href="https://www.instagram.com/dberrange" rel="noreferrer" target="_blank">https://www.instagram.com/dberrange</a> :|<br> > ><br> ><br> <br> </blockquote></div></div>