Message ID | 1500296815-10243-1-git-send-email-bhupinder.thakur@linaro.org |
---|---|
Headers | show |
Series | SBSA UART emulation support in Xen | expand |
Hi Bhupinder, I've tried this series today on an ARM64 platform. When I enable pl011 for the guest, I am not able to fully destroy the guest. It stay in zombie mode: 42sh> xl list Name ID Mem VCPUs State Time(s) Domain-0 0 3072 2 r----- 62.1 (null) 6 0 2 --p--d 1.5 It does not happen when I don't have pl011 enabled in the guest config. The step to reproduce it is: 42sh> xl create guest.cfg Parsing config from guest.cfg 42sh> xl list Name ID Mem VCPUs State Time(s) Domain-0 0 3072 2 r----- 12.9 guest 1 3265 1 r----- 0.1 42sh> xl destroy guest 42sh> xl list Name ID Mem VCPUs State Time(s) Domain-0 0 3072 2 r----- 14.9 (null) 1 0 1 --p--d 0.1 And my guest.cfg is: 42sh> cat guest.cfg kernel="/home/julien/works/guest/Image" name="guest" memory=3265 vcpus=2 vuart="sbsa_uart" I haven't dug into the problem but I would look at how you unmap the ring from Xen and xenconsole. Likely we still have a reference on it. Let me know if you need any help. Cheers, On 17/07/17 14:06, Bhupinder Thakur wrote: > SBSA UART emulation for guests in Xen > ====================================== > Linaro has published VM System specification for ARM Processors, which > provides a set of guidelines for both guest OS and hypervisor implementations, > such that building OS images according to these guidelines guarantees > that those images can also run on hypervisors compliant with this specification. > > One of the spec requirements is that the hypervisor must provide an > emulated SBSA UART as a serial console which meets the minimum requirements in > SBSA UART as defined in appendix B of the following > ARM Server Base Architecture Document: > > https://static.docs.arm.com/den0029/a/Server_Base_System_Architecture_v3_1_ARM_DEN_0029A.pdf. > > This feature allows the Xen guests to use SBSA compliant pl011 UART as > as a console. > > Note that SBSA pl011 UART is a subset of full featured ARM pl011 UART and > supports only a subset of registers as mentioned below. It does not support > rx/tx DMA. > > Currently, Xen supports paravirtualized (aka PV console) and an emulated serial > consoles. This feature will expose an emulated SBSA pl011 UART console to the > guest, which a user can access using xenconsole. > > The device tree passed to the guest VM will contain the pl011 MMIO address > range and an irq for receiving rx/tx pl011 interrupts. The device tree format > is specified in Documentation/devicetree/bindings/serial/arm_sbsa_uart.txt. > > The Xen hypervisor will expose two types of interfaces to the backend and domU. > > The interface exposed to domU will be an emulated pl011 UART by emulating the > access to the following pl011 registers by the guest. > > - Data register (DR) - RW > - Raw interrupt status register (RIS) - RO > - Masked interrupt status register (MIS)- RO > - Interrupt Mask (IMSC) - RW > - Interrupt Clear (ICR) - WO > > It will also inject the pl011 interrupts to the guest in the following > conditions: > > - incoming data in the rx buffer for the guest > - there is space in the tx buffer for the guest to write more data > > The interface exposed to the backend will be the same PV console interface, > which minimizes the changes required in xenconsole to support a new pl011 console. > > This interface has rx and tx ring buffers and an event channel for > sending/receiving events from the backend. > > So essentially Xen handles the data on behalf of domU and the backend. Any data > written by domU is captured by Xen and written to the TX (OUT) ring buffer > and a pl011 event is raised to the backend to read the TX ring buffer. > > Similarly on reciving a pl011 event, Xen injects an interrupt to guest to > indicate there is data available in the RX (IN) ring buffer. > > The pl011 UART state is completely captured in the set of registers > mentioned above and this state is updated everytime there is an event from > the backend or there is register read/write access from domU. > > For example, if domU has masked the rx interrupt in the IMSC register, then Xen > will not inject an interrupt to guest and will just update the RIS register. > Once the interrupt is unmasked by guest, the interrupt will be delivered to the > guest. > > Changes summary: > > Xen Hypervisor > =============== > > 1. Add emulation code to emulate read/write access to pl011 registers and pl011 > interrupts: > - It emulates DR read/write by reading and writing from/to the IN and > OUT ring buffers and raising an event to dom0 when there is data in > the OUT ring buffer and injecting an interrupt to the guest when there > is data in the IN ring buffer. > - Other registers are related to interrupt management and essentially > control when interrupts are delivered to the guest. > > 2. Add a new domctl API to initialize vpl011 emulation in Xen. > > 3. Enable vpl011 emulation for a domain based on a libxl option passed during > domain creation. > > Toolstack > ========== > > 1. Add a new option "vuart" in the domU configuration file to enable/disable vuart. > > 2. Create a SBSA UART DT node in the guest device tree. It uses a fixed > vpl011 SPI IRQ number and MMIO address. > > 3. Call vpl011 init DOMCTL API to enable vpl011 emulation. > > 5. Add a new vuart xenstore node, which contains: > - ring-ref > - event channel > - buffer limit > - type > > Xenconsoled > ============ > > 1. Split the domain structure to support multiple consoles. > > 2. Modify different APIs such as buffer_append() etc. to operate on the > console structure. > > 3. Add support for handling multiple consoles. > > 4. Add support for vuart console: > > The vpl011 changes available at the following repo: > > url: https://git@git.linaro.org:/people/bhupinder.thakur/xen.git > branch: vpl011_v6 > > Kindly wait for one day to checkout the code from the above URL. > > There are some TBD items which need to be looked at in the future: > > 1. Currently UEFI firmware logs the output to hvc console only. How can > UEFI firmware be made aware of pl011 console and how it can use it > as a console instead of hvc. > > There was a discussion on this and it was decided that SBSA UART should > be used as a debug port by the UEFI firmware so that all debug output > is redirected to this port. > > 2. Linux seems to have hvc console as the default console i.e. if no > console is specified then it uses hvc as the console. How can an > option be provided in Linux to select either hvc or pl011 as the > default console. > > It was suggeted to use the SPCR in ACPI and the stdout-path option in the > device tree to specify the default console. However, currently hvc console > is not describable in the ACPI/device tree. This support will have to be > added to allow the user to specify the default console. > > 3. ACPI support for pl011 device. > > CC: Ian Jackson <ian.jackson@eu.citrix.com> > CC: Wei Liu <wei.liu2@citrix.com> > CC: Stefano Stabellini <sstabellini@kernel.org> > CC: Julien Grall <julien.grall@arm.com> > CC: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> > > > Bhupinder Thakur (25): > xen/arm: vpl011: Define common ring buffer helper functions in > console.h > xen/arm: vpl011: Add SBSA UART emulation in Xen > xen/arm: vpl011: Allocate a new GFN in the toolstack for vuart > xen/arm: vpl011: Add support for vuart in libxl > xen/arm: vpl011: Rearrange xen header includes in alphabetical order > in domctl.c > xen/arm: vpl011: Add a new domctl API to initialize vpl011 > xen/arm: vpl011: Add a new vuart node in the xenstore > xen/arm: vpl011: Modify xenconsole to define and use a new console > structure > xen/arm: vpl011: Rename the console structure field conspath to xspath > xen/arm: vpl011: Modify xenconsole functions to take console structure > as input > xen/arm: vpl011: Add a new console_init function in xenconsole > xen/arm: vpl011: Add a new buffer_available function in xenconsole > xen/arm: vpl011: Add a new add_console_evtchn_fd function in > xenconsole > xen/arm: vpl011: Add a new add_console_tty_fd function in xenconsole > xen/arm: vpl011: Add a new console_evtchn_unmask function in > xenconsole > xen/arm: vpl011: Add a new handle_console_ring function in xenconsole > xen/arm: vpl011: Add a new handle_console_tty function in xenconsole > xen/arm: vpl011: Add a new console_cleanup function in xenconsole > xen/arm: vpl011: Add a new console_open_log function in xenconsole > xen/arm: vpl011: Add a new console_close_evtchn function in xenconsole > xen/arm: vpl011: Add support for multiple consoles in xenconsole > xen/arm: vpl011: Add support for vuart console in xenconsole > xen/arm: vpl011: Add a new vuart console type to xenconsole client > xen/arm: vpl011: Add a pl011 uart DT node in the guest device tree > xen/arm: vpl011: Update documentation for vuart console support > > config/arm32.mk | 1 + > config/arm64.mk | 1 + > docs/man/xl.cfg.pod.5.in | 9 + > docs/misc/console.txt | 44 ++- > tools/console/Makefile | 3 +- > tools/console/client/main.c | 13 +- > tools/console/daemon/io.c | 662 ++++++++++++++++++++++++----------- > tools/libxc/include/xc_dom.h | 2 + > tools/libxc/include/xenctrl.h | 18 + > tools/libxc/xc_dom_arm.c | 5 +- > tools/libxc/xc_dom_boot.c | 2 + > tools/libxc/xc_domain.c | 24 ++ > tools/libxl/libxl.h | 6 + > tools/libxl/libxl_arch.h | 6 + > tools/libxl/libxl_arm.c | 72 +++- > tools/libxl/libxl_console.c | 47 +++ > tools/libxl/libxl_create.c | 9 +- > tools/libxl/libxl_device.c | 9 +- > tools/libxl/libxl_dom.c | 5 + > tools/libxl/libxl_internal.h | 6 + > tools/libxl/libxl_types.idl | 7 + > tools/libxl/libxl_types_internal.idl | 1 + > tools/libxl/libxl_x86.c | 8 + > tools/xl/xl_cmdtable.c | 2 +- > tools/xl/xl_console.c | 5 +- > tools/xl/xl_parse.c | 8 + > xen/arch/arm/Kconfig | 7 + > xen/arch/arm/Makefile | 1 + > xen/arch/arm/domain.c | 5 + > xen/arch/arm/domctl.c | 43 ++- > xen/arch/arm/vpl011.c | 455 ++++++++++++++++++++++++ > xen/include/asm-arm/domain.h | 6 + > xen/include/asm-arm/pl011-uart.h | 2 + > xen/include/asm-arm/vpl011.h | 72 ++++ > xen/include/public/arch-arm.h | 6 + > xen/include/public/domctl.h | 21 ++ > xen/include/public/io/console.h | 4 + > 37 files changed, 1360 insertions(+), 237 deletions(-) > create mode 100644 xen/arch/arm/vpl011.c > create mode 100644 xen/include/asm-arm/vpl011.h >
Hi Julien, The issue seems to be that domain_vpl011_deinit() is not getting called when we destroy the domain. On further debugging, i found that domain_relinquish_resources was failing in domain_kill() function. I believe that issue could be that arch_domain_destroy() (from where domain_vpl011_deinit() is called) is called later and because domain_kill() fails, it is not getting called. Once I moved the domain_vpl011_deinit() inside domain_kill(), the domain is getting cleaned up properly. Is it ok to call domain_vpl011_deinit() inside domain_kill? Regards, Bhupinder On 19 July 2017 at 20:57, Julien Grall <julien.grall@arm.com> wrote: > Hi Bhupinder, > > I've tried this series today on an ARM64 platform. When I enable pl011 for the guest, > I am not able to fully destroy the guest. It stay in zombie mode: > > 42sh> xl list > Name ID Mem VCPUs State Time(s) > Domain-0 0 3072 2 r----- 62.1 > (null) 6 0 2 --p--d 1.5 > > It does not happen when I don't have pl011 enabled in the guest config. > > The step to reproduce it is: > > 42sh> xl create guest.cfg > Parsing config from guest.cfg > 42sh> xl list > Name ID Mem VCPUs State Time(s) > Domain-0 0 3072 2 r----- 12.9 > guest 1 3265 1 r----- 0.1 > 42sh> xl destroy guest > 42sh> xl list > Name ID Mem VCPUs State Time(s) > Domain-0 0 3072 2 r----- 14.9 > (null) 1 0 1 --p--d 0.1 > > And my guest.cfg is: > > 42sh> cat guest.cfg > kernel="/home/julien/works/guest/Image" > name="guest" > memory=3265 > vcpus=2 > vuart="sbsa_uart" > > I haven't dug into the problem but I would look at how you unmap the ring from Xen > and xenconsole. Likely we still have a reference on it. > > Let me know if you need any help. > > Cheers, > > On 17/07/17 14:06, Bhupinder Thakur wrote: >> SBSA UART emulation for guests in Xen >> ====================================== >> Linaro has published VM System specification for ARM Processors, which >> provides a set of guidelines for both guest OS and hypervisor implementations, >> such that building OS images according to these guidelines guarantees >> that those images can also run on hypervisors compliant with this specification. >> >> One of the spec requirements is that the hypervisor must provide an >> emulated SBSA UART as a serial console which meets the minimum requirements in >> SBSA UART as defined in appendix B of the following >> ARM Server Base Architecture Document: >> >> https://static.docs.arm.com/den0029/a/Server_Base_System_Architecture_v3_1_ARM_DEN_0029A.pdf. >> >> This feature allows the Xen guests to use SBSA compliant pl011 UART as >> as a console. >> >> Note that SBSA pl011 UART is a subset of full featured ARM pl011 UART and >> supports only a subset of registers as mentioned below. It does not support >> rx/tx DMA. >> >> Currently, Xen supports paravirtualized (aka PV console) and an emulated serial >> consoles. This feature will expose an emulated SBSA pl011 UART console to the >> guest, which a user can access using xenconsole. >> >> The device tree passed to the guest VM will contain the pl011 MMIO address >> range and an irq for receiving rx/tx pl011 interrupts. The device tree format >> is specified in Documentation/devicetree/bindings/serial/arm_sbsa_uart.txt. >> >> The Xen hypervisor will expose two types of interfaces to the backend and domU. >> >> The interface exposed to domU will be an emulated pl011 UART by emulating the >> access to the following pl011 registers by the guest. >> >> - Data register (DR) - RW >> - Raw interrupt status register (RIS) - RO >> - Masked interrupt status register (MIS)- RO >> - Interrupt Mask (IMSC) - RW >> - Interrupt Clear (ICR) - WO >> >> It will also inject the pl011 interrupts to the guest in the following >> conditions: >> >> - incoming data in the rx buffer for the guest >> - there is space in the tx buffer for the guest to write more data >> >> The interface exposed to the backend will be the same PV console interface, >> which minimizes the changes required in xenconsole to support a new pl011 console. >> >> This interface has rx and tx ring buffers and an event channel for >> sending/receiving events from the backend. >> >> So essentially Xen handles the data on behalf of domU and the backend. Any data >> written by domU is captured by Xen and written to the TX (OUT) ring buffer >> and a pl011 event is raised to the backend to read the TX ring buffer. >> >> Similarly on reciving a pl011 event, Xen injects an interrupt to guest to >> indicate there is data available in the RX (IN) ring buffer. >> >> The pl011 UART state is completely captured in the set of registers >> mentioned above and this state is updated everytime there is an event from >> the backend or there is register read/write access from domU. >> >> For example, if domU has masked the rx interrupt in the IMSC register, then Xen >> will not inject an interrupt to guest and will just update the RIS register. >> Once the interrupt is unmasked by guest, the interrupt will be delivered to the >> guest. >> >> Changes summary: >> >> Xen Hypervisor >> =============== >> >> 1. Add emulation code to emulate read/write access to pl011 registers and pl011 >> interrupts: >> - It emulates DR read/write by reading and writing from/to the IN and >> OUT ring buffers and raising an event to dom0 when there is data in >> the OUT ring buffer and injecting an interrupt to the guest when there >> is data in the IN ring buffer. >> - Other registers are related to interrupt management and essentially >> control when interrupts are delivered to the guest. >> >> 2. Add a new domctl API to initialize vpl011 emulation in Xen. >> >> 3. Enable vpl011 emulation for a domain based on a libxl option passed during >> domain creation. >> >> Toolstack >> ========== >> >> 1. Add a new option "vuart" in the domU configuration file to enable/disable vuart. >> >> 2. Create a SBSA UART DT node in the guest device tree. It uses a fixed >> vpl011 SPI IRQ number and MMIO address. >> >> 3. Call vpl011 init DOMCTL API to enable vpl011 emulation. >> >> 5. Add a new vuart xenstore node, which contains: >> - ring-ref >> - event channel >> - buffer limit >> - type >> >> Xenconsoled >> ============ >> >> 1. Split the domain structure to support multiple consoles. >> >> 2. Modify different APIs such as buffer_append() etc. to operate on the >> console structure. >> >> 3. Add support for handling multiple consoles. >> >> 4. Add support for vuart console: >> >> The vpl011 changes available at the following repo: >> >> url: https://git@git.linaro.org:/people/bhupinder.thakur/xen.git >> branch: vpl011_v6 >> >> Kindly wait for one day to checkout the code from the above URL. >> >> There are some TBD items which need to be looked at in the future: >> >> 1. Currently UEFI firmware logs the output to hvc console only. How can >> UEFI firmware be made aware of pl011 console and how it can use it >> as a console instead of hvc. >> >> There was a discussion on this and it was decided that SBSA UART should >> be used as a debug port by the UEFI firmware so that all debug output >> is redirected to this port. >> >> 2. Linux seems to have hvc console as the default console i.e. if no >> console is specified then it uses hvc as the console. How can an >> option be provided in Linux to select either hvc or pl011 as the >> default console. >> >> It was suggeted to use the SPCR in ACPI and the stdout-path option in the >> device tree to specify the default console. However, currently hvc console >> is not describable in the ACPI/device tree. This support will have to be >> added to allow the user to specify the default console. >> >> 3. ACPI support for pl011 device. >> >> CC: Ian Jackson <ian.jackson@eu.citrix.com> >> CC: Wei Liu <wei.liu2@citrix.com> >> CC: Stefano Stabellini <sstabellini@kernel.org> >> CC: Julien Grall <julien.grall@arm.com> >> CC: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> >> >> >> Bhupinder Thakur (25): >> xen/arm: vpl011: Define common ring buffer helper functions in >> console.h >> xen/arm: vpl011: Add SBSA UART emulation in Xen >> xen/arm: vpl011: Allocate a new GFN in the toolstack for vuart >> xen/arm: vpl011: Add support for vuart in libxl >> xen/arm: vpl011: Rearrange xen header includes in alphabetical order >> in domctl.c >> xen/arm: vpl011: Add a new domctl API to initialize vpl011 >> xen/arm: vpl011: Add a new vuart node in the xenstore >> xen/arm: vpl011: Modify xenconsole to define and use a new console >> structure >> xen/arm: vpl011: Rename the console structure field conspath to xspath >> xen/arm: vpl011: Modify xenconsole functions to take console structure >> as input >> xen/arm: vpl011: Add a new console_init function in xenconsole >> xen/arm: vpl011: Add a new buffer_available function in xenconsole >> xen/arm: vpl011: Add a new add_console_evtchn_fd function in >> xenconsole >> xen/arm: vpl011: Add a new add_console_tty_fd function in xenconsole >> xen/arm: vpl011: Add a new console_evtchn_unmask function in >> xenconsole >> xen/arm: vpl011: Add a new handle_console_ring function in xenconsole >> xen/arm: vpl011: Add a new handle_console_tty function in xenconsole >> xen/arm: vpl011: Add a new console_cleanup function in xenconsole >> xen/arm: vpl011: Add a new console_open_log function in xenconsole >> xen/arm: vpl011: Add a new console_close_evtchn function in xenconsole >> xen/arm: vpl011: Add support for multiple consoles in xenconsole >> xen/arm: vpl011: Add support for vuart console in xenconsole >> xen/arm: vpl011: Add a new vuart console type to xenconsole client >> xen/arm: vpl011: Add a pl011 uart DT node in the guest device tree >> xen/arm: vpl011: Update documentation for vuart console support >> >> config/arm32.mk | 1 + >> config/arm64.mk | 1 + >> docs/man/xl.cfg.pod.5.in | 9 + >> docs/misc/console.txt | 44 ++- >> tools/console/Makefile | 3 +- >> tools/console/client/main.c | 13 +- >> tools/console/daemon/io.c | 662 ++++++++++++++++++++++++----------- >> tools/libxc/include/xc_dom.h | 2 + >> tools/libxc/include/xenctrl.h | 18 + >> tools/libxc/xc_dom_arm.c | 5 +- >> tools/libxc/xc_dom_boot.c | 2 + >> tools/libxc/xc_domain.c | 24 ++ >> tools/libxl/libxl.h | 6 + >> tools/libxl/libxl_arch.h | 6 + >> tools/libxl/libxl_arm.c | 72 +++- >> tools/libxl/libxl_console.c | 47 +++ >> tools/libxl/libxl_create.c | 9 +- >> tools/libxl/libxl_device.c | 9 +- >> tools/libxl/libxl_dom.c | 5 + >> tools/libxl/libxl_internal.h | 6 + >> tools/libxl/libxl_types.idl | 7 + >> tools/libxl/libxl_types_internal.idl | 1 + >> tools/libxl/libxl_x86.c | 8 + >> tools/xl/xl_cmdtable.c | 2 +- >> tools/xl/xl_console.c | 5 +- >> tools/xl/xl_parse.c | 8 + >> xen/arch/arm/Kconfig | 7 + >> xen/arch/arm/Makefile | 1 + >> xen/arch/arm/domain.c | 5 + >> xen/arch/arm/domctl.c | 43 ++- >> xen/arch/arm/vpl011.c | 455 ++++++++++++++++++++++++ >> xen/include/asm-arm/domain.h | 6 + >> xen/include/asm-arm/pl011-uart.h | 2 + >> xen/include/asm-arm/vpl011.h | 72 ++++ >> xen/include/public/arch-arm.h | 6 + >> xen/include/public/domctl.h | 21 ++ >> xen/include/public/io/console.h | 4 + >> 37 files changed, 1360 insertions(+), 237 deletions(-) >> create mode 100644 xen/arch/arm/vpl011.c >> create mode 100644 xen/include/asm-arm/vpl011.h >> > > -- > Julien Grall
On 20/07/17 10:40, Bhupinder Thakur wrote: > Hi Julien, Hi Bhupinder, > The issue seems to be that domain_vpl011_deinit() is not getting > called when we destroy the domain. On further debugging, i found that > domain_relinquish_resources was failing in domain_kill() function. > > I believe that issue could be that arch_domain_destroy() (from where > domain_vpl011_deinit() is called) is called later and because > domain_kill() fails, it is not getting called. Once I moved the > domain_vpl011_deinit() inside domain_kill(), the domain is getting > cleaned up properly. What do you mean by domain_kill fails? The function may take time to execute so it may return -ERESTART for preemption and be restarted later on. The problem here is until domain_vpl011_deinit is called, there is still a reference taken on the guest memory and therefore on the domain. So the domain will not get fully destroyed (domain_destroy is called when the last reference on the domain is dropped). Because domain_vpl011_deinit is called from domain_destroy, there will always be a reference on the guest memory and domain. So no destruction. > > Is it ok to call domain_vpl011_deinit() inside domain_kill? domain_kill is a arch agnostic function. You want to call domain_vpl011_init from domain_relinquish_resources. I would do in the case RELMEM_not_started after iommu_release_dt_devices is called. Cheers,