From patchwork Thu Aug 22 06:34:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ruifeng Wang X-Patchwork-Id: 172015 Delivered-To: patch@linaro.org Received: by 2002:a92:d204:0:0:0:0:0 with SMTP id y4csp456666ily; Wed, 21 Aug 2019 23:35:43 -0700 (PDT) X-Google-Smtp-Source: APXvYqwC9Qh9ljb0Nq6s8i4r/xkQl96HY+ARzHoCp9ZVkWyg4F6E6H5RTzlLRty7cHacIl2+b4P4 X-Received: by 2002:a17:906:3b02:: with SMTP id g2mr8136971ejf.10.1566455743201; Wed, 21 Aug 2019 23:35:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1566455743; cv=none; d=google.com; s=arc-20160816; b=ijjJDzdqAFJCSmEZKA9qD8MUCfxhUaM5fYqht4HFp2ToHxI7Be1xmTxpY0cZIjU+ii +v9jyAouI8MyAKnJ3dXre2tzoVVsCuKVIclvAIW5xwT+2L/eGWlwIdlrZAp8Ys2d9MyI sPogrgFamd67pcsqsYVc3Lhq83Vt6DEhxckuCXPekoTsaDUbuv5vsbAKYglw3iesnN4O lDA8UzzaS0hmV0I00903e90xkmY5mDCk4jF9W+qp7Ua6cD80SMbgkJQ4JHUAzCYUbNxg 49YcFOuWi997hOdUxb/4zMK0ICtzJdd4ezhyAXL/C4r33O4v7K+gRGUoqa8D9oAE4ecE Lvpw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:cc:to:from; bh=Q8DiCVEBLW1grR89F0ad4c2yYK01YByvpkSCNozEL14=; b=nxVUuI//armpc/Ws0pOBn11cWZh8BdWZcNrmzYVRWbfBhmIEtBwzlym/M8JV1VLLqX 8Odm6XiF9ndn1jVYfZP9EyFzMFAZvSAJkjw53mXPiFCOPTtooXmdi+1UA64G5oAxmjn+ /lopIgfHo8gzcobpXDm+FB5lPXnwtjGtDkBoDT7jTUT0Jv4F9UFePSPS2aQCoApN0SEA 9zu8nr15Uchnu4Ktx319MLBoljw7+1gSdwWgDzsH7CXPq4VXlfeic73BKaTN+TSRPwZR u5d8hJRMgagV/52FD4X8mNme4mrI7eFkHtAgxRiwi3cZAOWzYT4mb29m0cYz68WkfB2A 7dkg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of dev-bounces@dpdk.org designates 92.243.14.124 as permitted sender) smtp.mailfrom=dev-bounces@dpdk.org Return-Path: Received: from dpdk.org (dpdk.org. [92.243.14.124]) by mx.google.com with ESMTP id c2si13895321edc.238.2019.08.21.23.35.42; Wed, 21 Aug 2019 23:35:43 -0700 (PDT) Received-SPF: pass (google.com: domain of dev-bounces@dpdk.org designates 92.243.14.124 as permitted sender) client-ip=92.243.14.124; Authentication-Results: mx.google.com; spf=pass (google.com: domain of dev-bounces@dpdk.org designates 92.243.14.124 as permitted sender) smtp.mailfrom=dev-bounces@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 8976C1BF59; Thu, 22 Aug 2019 08:35:42 +0200 (CEST) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by dpdk.org (Postfix) with ESMTP id 5F5BE1BF59 for ; Thu, 22 Aug 2019 08:35:41 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id DDF1E344; Wed, 21 Aug 2019 23:35:40 -0700 (PDT) Received: from net-arm-c2400-02.shanghai.arm.com (net-arm-c2400-02.shanghai.arm.com [10.169.40.42]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id E95BA3F706; Wed, 21 Aug 2019 23:37:56 -0700 (PDT) From: Ruifeng Wang To: bruce.richardson@intel.com, vladimir.medvedkin@intel.com, olivier.matz@6wind.com Cc: dev@dpdk.org, honnappa.nagarahalli@arm.com, dharmik.thakkar@arm.com, nd@arm.com Date: Thu, 22 Aug 2019 14:34:55 +0800 Message-Id: <20190822063457.41596-2-ruifeng.wang@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190822063457.41596-1-ruifeng.wang@arm.com> References: <20190822063457.41596-1-ruifeng.wang@arm.com> Subject: [dpdk-dev] [RFC PATCH 1/3] doc/rcu: add RCU integration design details X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Honnappa Nagarahalli Add a section to describe a design to integrate QSBR RCU library with other libraries in DPDK. Signed-off-by: Honnappa Nagarahalli Reviewed-by: Gavin Hu Reviewed-by: Ruifeng Wang --- doc/guides/prog_guide/rcu_lib.rst | 51 +++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) -- 2.17.1 diff --git a/doc/guides/prog_guide/rcu_lib.rst b/doc/guides/prog_guide/rcu_lib.rst index 8fe5b1f73..2869441ca 100644 --- a/doc/guides/prog_guide/rcu_lib.rst +++ b/doc/guides/prog_guide/rcu_lib.rst @@ -186,3 +186,54 @@ However, when ``CONFIG_RTE_LIBRTE_RCU_DEBUG`` is enabled, these APIs aid in debugging issues. One can mark the access to shared data structures on the reader side using these APIs. The ``rte_rcu_qsbr_quiescent()`` will check if all the locks are unlocked. + +Integrating QSBR RCU with other libraries +----------------------------------------- + +Lock-free algorithms place additional burden on the application to reclaim +memory. Integrating memory reclaiming mechanisms in the libraries help +remove some of the burden. Though QSBR method presents flexibility to +achieve performance, it presents challenges while integrating with libraries. + +The memory reclaiming process using QSBR can be split into 4 parts: + +#. Initialization +#. Quiescent State Reporting +#. Reclaiming Resources +#. Shutdown + +The design proposed here requires the application to handle 'Initialization' +and 'Quiescent State Reporting'. So, + +* the application has to create the RCU variable and register the reader threads to report their quiescent state. +* the application has to register the same RCU variable with the library. +* reader threads in the application have to report the quiescent state. This allows for the application to control the length of the critical section/how frequently the application wants to report the quiescent state. + +The library will handle 'Reclaiming Resources' part of the process. The +libraries will make use of the writer thread context to execute the memory +reclaiming algorithm. So, + +* library should provide an API to register a RCU variable that it will use. +* library should trigger the readers to report quiescent state status upon deleting the resources by calling ``rte_rcu_qsbr_start``. + +* library should store the token and deleted resources for later use to free them after the readers have reported their quiescent state. Since the readers will report the quiescent state status in the order of deletion, the library must store the tokens/resources in the order in which the resources were deleted. A FIFO data structure would achieve the desired results. The length of the FIFO would depend on the rate of deletion and the rate at which the readers report their quiescent state. In the worst case the length of FIFO would be equal to the maximum number of resources the data structure supports. However, in most cases, the length will be much smaller. But, the library should not take the length of FIFO as an input from the application. Instead, it should implement a data structure which should be able to grow/shrink dynamically. Overhead introduced by such a data structure on delete operations should be considered as well. + +* library should query the quiescent state and free the resources. It should make use of non-blocking ``rte_rcu_qsbr_check`` API to query the quiescent state. This allows the application to do useful work while the readers report their quiescent state. If there are tokens/resources present in the FIFO already, the delete API should peek the head of the FIFO and check the quiescent state status. If the status is success, the token/resource should be dequeued and the resource should be freed. This process can be repeated till the quiescent state status for a token returns failure indicating that subsequent tokens will also fail quiescent state status query. The same process can be incorporated while adding new entries in the data structure if the library runs out of resources. + +The 'Shutdown' process needs to be shared between the application and the +library. + +* library should check the quiescent state status of all the tokens that may be present in the FIFO and free the resources. It should make use of non-blocking ``rte_rcu_qsbr_check`` API to query the quiescent state. If any of the tokens do not pass the quiescent state check, the library should print an error and stop the memory reclaimation process. + +* the application should make sure that the reader threads are not using the shared data structure, unregister the reader threads from the QSBR variable before calling the library's shutdown function. + +Integrating the resource reclaimation with libraries removes the burden from +the application and makes it easy to use lock-free algorithms. + +This design has several advantages over currently known methods. + +#. Application does not need a dedicated thread to reclaim resources. Memory + reclaimation happens as part of the writer thread without sacrificing + a lot of performance. +#. The library has better control over the resources. For ex: the library can + attempt to reclaim when it has run out of resources.