From patchwork Fri Jul 31 18:20:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jag Raman X-Patchwork-Id: 277213 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, UNPARSEABLE_RELAY, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0D9EC433E0 for ; Fri, 31 Jul 2020 18:27:37 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AAC1B22B42 for ; Fri, 31 Jul 2020 18:27:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="Og8uFqsM" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AAC1B22B42 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:34128 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k1Zkm-0002QI-T2 for qemu-devel@archiver.kernel.org; Fri, 31 Jul 2020 14:27:36 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58138) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k1ZeW-000266-HV for qemu-devel@nongnu.org; Fri, 31 Jul 2020 14:21:08 -0400 Received: from userp2130.oracle.com ([156.151.31.86]:50936) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k1ZeT-0005Ps-VB for qemu-devel@nongnu.org; Fri, 31 Jul 2020 14:21:08 -0400 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 06VICxN7110597; Fri, 31 Jul 2020 18:20:59 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : in-reply-to : references : mime-version : content-type : content-transfer-encoding; s=corp-2020-01-29; bh=8Ome06812C8bafzlnnYF2jRWfGFZyOpxST3ApEtTdUY=; b=Og8uFqsMxLF+h4jnbfG7HUHSdG9CG+L4F50VlpJkl3rikBPs90aIbWZ6lQ/5REsnhVuh oNyLgLyW2YAxFTIAsJsmZwl8d2aMapY2aJgQwhCWYO4EpiHehv09nL/rPzsaVBGP0+k/ bZG/RMTLwCBNmkSZIsQRKclQr1kbobvuWRztBeMMCLyAyO/Zy18tzdS7G3NUiHIDGvD+ GILhve03hcYm0grSJBlM9Hnv6Ogd5oN/BpCGdwF16NFyOOD2RZxCsEfKwEK+AcBnHTY9 HF851b0Ja7mGqbcAoeDRxbHjtlHDbqzsqxkjKa9Zz2lyr579kmdrMpYcuZYc6yCZSht3 sQ== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by userp2130.oracle.com with ESMTP id 32hu1jtg0s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Fri, 31 Jul 2020 18:20:59 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 06VIIiUt080288; Fri, 31 Jul 2020 18:20:58 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userp3020.oracle.com with ESMTP id 32mf70ve40-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 31 Jul 2020 18:20:58 +0000 Received: from abhmp0005.oracle.com (abhmp0005.oracle.com [141.146.116.11]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id 06VIKv3m020219; Fri, 31 Jul 2020 18:20:57 GMT Received: from jaraman-bur-1.us.oracle.com (/10.152.33.39) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 31 Jul 2020 11:20:57 -0700 From: Jagannathan Raman To: qemu-devel@nongnu.org Subject: [PATCH v8 14/20] multi-process: Synchronize remote memory Date: Fri, 31 Jul 2020 14:20:21 -0400 Message-Id: <9e2064df688ff10c0094427dcdae58ff2c3ada34.1596217462.git.jag.raman@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9699 signatures=668679 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 suspectscore=3 spamscore=0 mlxlogscore=999 adultscore=0 malwarescore=0 mlxscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2007310137 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9699 signatures=668679 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 clxscore=1015 malwarescore=0 spamscore=0 suspectscore=3 bulkscore=0 priorityscore=1501 phishscore=0 mlxlogscore=999 lowpriorityscore=0 impostorscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2007310136 Received-SPF: pass client-ip=156.151.31.86; envelope-from=jag.raman@oracle.com; helo=userp2130.oracle.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/31 12:51:59 X-ACL-Warn: Detected OS = Linux 3.1-3.10 [fuzzy] X-Spam_score_int: -63 X-Spam_score: -6.4 X-Spam_bar: ------ X-Spam_report: (-6.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H2=-1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, UNPARSEABLE_RELAY=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, fam@euphon.net, swapnil.ingle@nutanix.com, john.g.johnson@oracle.com, kraxel@redhat.com, jag.raman@oracle.com, quintela@redhat.com, mst@redhat.com, armbru@redhat.com, kanth.ghatraju@oracle.com, felipe@nutanix.com, thuth@redhat.com, ehabkost@redhat.com, konrad.wilk@oracle.com, dgilbert@redhat.com, alex.williamson@redhat.com, stefanha@redhat.com, thanos.makatos@nutanix.com, rth@twiddle.net, kwolf@redhat.com, berrange@redhat.com, mreitz@redhat.com, ross.lagerwall@citrix.com, marcandre.lureau@gmail.com, pbonzini@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Add memory-listener object which is used to keep the view of the RAM in sync between QEMU and remote process. A MemoryListener is registered for system-memory AddressSpace. The listener sends SYNC_SYSMEM message to the remote process when memory listener commits the changes to memory, the remote process receives the message and processes it in the handler for SYNC_SYSMEM message. Signed-off-by: Jagannathan Raman Signed-off-by: John G Johnson Signed-off-by: Elena Ufimtseva --- MAINTAINERS | 2 + hw/i386/remote-msg.c | 4 + hw/pci/Makefile.objs | 1 + hw/pci/memory-sync.c | 211 +++++++++++++++++++++++++++++++++++++++++++ hw/pci/proxy.c | 6 ++ include/hw/pci/memory-sync.h | 27 ++++++ include/hw/pci/proxy.h | 3 + 7 files changed, 254 insertions(+) create mode 100644 hw/pci/memory-sync.c create mode 100644 include/hw/pci/memory-sync.h diff --git a/MAINTAINERS b/MAINTAINERS index a8f097a..bd4f4ec 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -3053,6 +3053,8 @@ F: include/hw/i386/remote-memory.h F: hw/i386/remote-memory.c F: hw/pci/proxy.c F: include/hw/pci/proxy.h +F: hw/pci/memory-sync.c +F: include/hw/pci/memory-sync.h Build and test automation ------------------------- diff --git a/hw/i386/remote-msg.c b/hw/i386/remote-msg.c index 7ccdd63..584498d 100644 --- a/hw/i386/remote-msg.c +++ b/hw/i386/remote-msg.c @@ -17,6 +17,7 @@ #include "sysemu/runstate.h" #include "hw/pci/pci.h" #include "exec/memattrs.h" +#include "hw/i386/remote-memory.h" static void process_config_write(QIOChannel *ioc, PCIDevice *dev, MPQemuMsg *msg); @@ -67,6 +68,9 @@ gboolean mpqemu_process_msg(QIOChannel *ioc, GIOCondition cond, case BAR_READ: process_bar_read(ioc, &msg, &local_err); break; + case SYNC_SYSMEM: + remote_sysmem_reconfig(&msg, &local_err); + break; default: error_setg(&local_err, "Unknown command (%d) received for device %s (pid=%d)", diff --git a/hw/pci/Makefile.objs b/hw/pci/Makefile.objs index 515dda5..c90acd5 100644 --- a/hw/pci/Makefile.objs +++ b/hw/pci/Makefile.objs @@ -13,3 +13,4 @@ common-obj-$(CONFIG_PCI_EXPRESS) += pcie_port.o pcie_host.o common-obj-$(call lnot,$(CONFIG_PCI)) += pci-stub.o common-obj-$(CONFIG_ALL) += pci-stub.o obj-$(CONFIG_MPQEMU) += proxy.o +obj-$(CONFIG_MPQEMU) += memory-sync.o diff --git a/hw/pci/memory-sync.c b/hw/pci/memory-sync.c new file mode 100644 index 0000000..1af28395 --- /dev/null +++ b/hw/pci/memory-sync.c @@ -0,0 +1,211 @@ +/* + * Copyright © 2018, 2020 Oracle and/or its affiliates. + * + * This work is licensed under the terms of the GNU GPL, version 2 or later. + * See the COPYING file in the top-level directory. + * + */ + +#include "qemu/osdep.h" +#include "qemu-common.h" + +#include "qemu/compiler.h" +#include "qemu/int128.h" +#include "qemu/range.h" +#include "exec/memory.h" +#include "exec/cpu-common.h" +#include "cpu.h" +#include "exec/ram_addr.h" +#include "exec/address-spaces.h" +#include "io/mpqemu-link.h" +#include "hw/pci/memory-sync.h" + +static void proxy_ml_begin(MemoryListener *listener) +{ + RemoteMemSync *sync = container_of(listener, RemoteMemSync, listener); + int mrs; + + for (mrs = 0; mrs < sync->n_mr_sections; mrs++) { + memory_region_unref(sync->mr_sections[mrs].mr); + } + + g_free(sync->mr_sections); + sync->mr_sections = NULL; + sync->n_mr_sections = 0; +} + +static int get_fd_from_hostaddr(uint64_t host, ram_addr_t *offset) +{ + MemoryRegion *mr; + ram_addr_t off; + + /** + * Assumes that the host address is a valid address as it's + * coming from the MemoryListener system. In the case host + * address is not valid, the following call would return + * the default subregion of "system_memory" region, and + * not NULL. So it's not possible to check for NULL here. + */ + mr = memory_region_from_host((void *)(uintptr_t)host, &off); + + if (offset) { + *offset = off; + } + + return memory_region_get_fd(mr); +} + +static bool proxy_mrs_can_merge(uint64_t host, uint64_t prev_host, size_t size) +{ + bool merge; + int fd1, fd2; + + fd1 = get_fd_from_hostaddr(host, NULL); + + fd2 = get_fd_from_hostaddr(prev_host, NULL); + + merge = (fd1 == fd2); + + merge &= ((prev_host + size) == host); + + return merge; +} + +static bool try_merge(RemoteMemSync *sync, MemoryRegionSection *section) +{ + uint64_t mrs_size, mrs_gpa, mrs_page; + MemoryRegionSection *prev_sec; + bool merged = false; + uintptr_t mrs_host; + RAMBlock *mrs_rb; + + if (!sync->n_mr_sections) { + return false; + } + + mrs_rb = section->mr->ram_block; + mrs_page = (uint64_t)qemu_ram_pagesize(mrs_rb); + mrs_size = int128_get64(section->size); + mrs_gpa = section->offset_within_address_space; + mrs_host = (uintptr_t)memory_region_get_ram_ptr(section->mr) + + section->offset_within_region; + + if (get_fd_from_hostaddr(mrs_host, NULL) < 0) { + return true; + } + + mrs_host = mrs_host & ~(mrs_page - 1); + mrs_gpa = mrs_gpa & ~(mrs_page - 1); + mrs_size = ROUND_UP(mrs_size, mrs_page); + + prev_sec = sync->mr_sections + (sync->n_mr_sections - 1); + uint64_t prev_gpa_start = prev_sec->offset_within_address_space; + uint64_t prev_size = int128_get64(prev_sec->size); + uint64_t prev_gpa_end = range_get_last(prev_gpa_start, prev_size); + uint64_t prev_host_start = + (uintptr_t)memory_region_get_ram_ptr(prev_sec->mr) + + prev_sec->offset_within_region; + uint64_t prev_host_end = range_get_last(prev_host_start, prev_size); + + if (mrs_gpa <= (prev_gpa_end + 1)) { + g_assert(mrs_gpa > prev_gpa_start); + + if ((section->mr == prev_sec->mr) && + proxy_mrs_can_merge(mrs_host, prev_host_start, + (mrs_gpa - prev_gpa_start))) { + uint64_t max_end = MAX(prev_host_end, mrs_host + mrs_size); + merged = true; + prev_sec->offset_within_address_space = + MIN(prev_gpa_start, mrs_gpa); + prev_sec->offset_within_region = + MIN(prev_host_start, mrs_host) - + (uintptr_t)memory_region_get_ram_ptr(prev_sec->mr); + prev_sec->size = int128_make64(max_end - MIN(prev_host_start, + mrs_host)); + } + } + + return merged; +} + +static void proxy_ml_region_addnop(MemoryListener *listener, + MemoryRegionSection *section) +{ + RemoteMemSync *sync = container_of(listener, RemoteMemSync, listener); + + if (!(memory_region_is_ram(section->mr) && + !memory_region_is_rom(section->mr))) { + return; + } + + if (try_merge(sync, section)) { + return; + } + + ++sync->n_mr_sections; + sync->mr_sections = g_renew(MemoryRegionSection, sync->mr_sections, + sync->n_mr_sections); + sync->mr_sections[sync->n_mr_sections - 1] = *section; + sync->mr_sections[sync->n_mr_sections - 1].fv = NULL; + memory_region_ref(section->mr); +} + +static void proxy_ml_commit(MemoryListener *listener) +{ + RemoteMemSync *sync = container_of(listener, RemoteMemSync, listener); + MPQemuMsg msg; + MemoryRegionSection *section; + ram_addr_t offset; + uintptr_t host_addr; + int region; + Error *local_err = NULL; + + memset(&msg, 0, sizeof(MPQemuMsg)); + + msg.cmd = SYNC_SYSMEM; + msg.bytestream = 0; + msg.num_fds = sync->n_mr_sections; + msg.size = sizeof(msg.data1); + if (msg.num_fds > REMOTE_MAX_FDS) { + error_report("Number of fds is more than %d", REMOTE_MAX_FDS); + return; + } + + for (region = 0; region < sync->n_mr_sections; region++) { + section = &sync->mr_sections[region]; + msg.data1.sync_sysmem.gpas[region] = + section->offset_within_address_space; + msg.data1.sync_sysmem.sizes[region] = int128_get64(section->size); + host_addr = (uintptr_t)memory_region_get_ram_ptr(section->mr) + + section->offset_within_region; + msg.fds[region] = get_fd_from_hostaddr(host_addr, &offset); + msg.data1.sync_sysmem.offsets[region] = offset; + } + mpqemu_msg_send(&msg, sync->ioc, &local_err); + if (local_err) { + error_report("Error in sending command %d", msg.cmd); + } +} + +void deconfigure_memory_sync(RemoteMemSync *sync) +{ + memory_listener_unregister(&sync->listener); + + proxy_ml_begin(&sync->listener); +} + +void configure_memory_sync(RemoteMemSync *sync, QIOChannel *ioc) +{ + sync->n_mr_sections = 0; + sync->mr_sections = NULL; + + sync->ioc = ioc; + + sync->listener.begin = proxy_ml_begin; + sync->listener.commit = proxy_ml_commit; + sync->listener.region_add = proxy_ml_region_addnop; + sync->listener.region_nop = proxy_ml_region_addnop; + sync->listener.priority = 10; + + memory_listener_register(&sync->listener, &address_space_memory); +} diff --git a/hw/pci/proxy.c b/hw/pci/proxy.c index 179f0c7..28b8c76 100644 --- a/hw/pci/proxy.c +++ b/hw/pci/proxy.c @@ -17,6 +17,8 @@ #include "monitor/monitor.h" #include "io/mpqemu-link.h" #include "qemu/error-report.h" +#include "hw/pci/memory-sync.h" +#include "qom/object.h" static void proxy_set_socket(PCIProxyDev *pdev, int fd, Error **errp) { @@ -44,6 +46,8 @@ static void pci_proxy_dev_realize(PCIDevice *device, Error **errp) error_setg(errp, "fd parameter not specified for %s", DEVICE(device)->id); } + + configure_memory_sync(&dev->sync, dev->ioc); } static void pci_proxy_dev_exit(PCIDevice *pdev) @@ -51,6 +55,8 @@ static void pci_proxy_dev_exit(PCIDevice *pdev) PCIProxyDev *dev = PCI_PROXY_DEV(pdev); qio_channel_close(dev->ioc, NULL); + + deconfigure_memory_sync(&dev->sync); } static int config_op_send(PCIProxyDev *pdev, uint32_t addr, uint32_t *val, diff --git a/include/hw/pci/memory-sync.h b/include/hw/pci/memory-sync.h new file mode 100644 index 0000000..785f76a --- /dev/null +++ b/include/hw/pci/memory-sync.h @@ -0,0 +1,27 @@ +/* + * Copyright © 2018, 2020 Oracle and/or its affiliates. + * + * This work is licensed under the terms of the GNU GPL, version 2 or later. + * See the COPYING file in the top-level directory. + * + */ + +#ifndef MEMORY_SYNC_H +#define MEMORY_SYNC_H + +#include "exec/memory.h" +#include "io/channel.h" + +typedef struct RemoteMemSync { + MemoryListener listener; + + int n_mr_sections; + MemoryRegionSection *mr_sections; + + QIOChannel *ioc; +} RemoteMemSync; + +void configure_memory_sync(RemoteMemSync *sync, QIOChannel *ioc); +void deconfigure_memory_sync(RemoteMemSync *sync); + +#endif diff --git a/include/hw/pci/proxy.h b/include/hw/pci/proxy.h index 9d57483..3ff79f4 100644 --- a/include/hw/pci/proxy.h +++ b/include/hw/pci/proxy.h @@ -11,6 +11,7 @@ #include "hw/pci/pci.h" #include "io/channel.h" +#include "hw/pci/memory-sync.h" #define TYPE_PCI_PROXY_DEV "pci-proxy-dev" @@ -32,6 +33,8 @@ struct PCIProxyDev { char *fd; QIOChannel *ioc; + RemoteMemSync sync; + ProxyMemoryRegion region[PCI_NUM_REGIONS]; };