From patchwork Wed Apr 15 10:10:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Stephan Mueller X-Patchwork-Id: 197866 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.5 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E03D5C3815B for ; Wed, 15 Apr 2020 10:37:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B3685206D9 for ; Wed, 15 Apr 2020 10:37:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=chronox.de header.i=@chronox.de header.b="MrVBETcW" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2408780AbgDOKhM (ORCPT ); Wed, 15 Apr 2020 06:37:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58462 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2408661AbgDOKcH (ORCPT ); Wed, 15 Apr 2020 06:32:07 -0400 Received: from mo6-p04-ob.smtp.rzone.de (mo6-p04-ob.smtp.rzone.de [IPv6:2a01:238:20a:202:5304::10]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 32F4FC061A0E; Wed, 15 Apr 2020 03:32:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; t=1586946725; s=strato-dkim-0002; d=chronox.de; h=References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: X-RZG-CLASS-ID:X-RZG-AUTH:From:Subject:Sender; bh=Sgr3v6dugvxga2hSTRCXcLr7eMIcwuyjEkWnP4wunHI=; b=MrVBETcW+0M3A3Us/xXfy9ElaDSRH6u9RDKaj7ry/gNZqbdhJDnOeI6TjfarwJEJyX VIwIjoaRcGsWK1kHzPsqpkv/HKwfvsto58nK/R7iS7WBcyNncW3X38hVPJZGnR2fwLiQ n+G+TVSl0HHarb2wYMF5s+shi/JW7ZwVKVXlkPtKt1Qrv+gyVrzheZ7HPui4VXF6r9Wv QlWRkdERESd7O0wjZJbfM0A7NcWwDXAh/ma9yhP8YaYz+xcf7uqZuxBdAx3PUlwQItAW TayF7bOtuKyNQvblhttU11hbh0z65s2x/doaTIXoU3VA+1oQruFHQWGwI5iEZFlOkNy/ Z7PA== X-RZG-AUTH: ":P2ERcEykfu11Y98lp/T7+hdri+uKZK8TKWEqNyiHySGSa9k9xmwdNnzGHXPZIvSaiyU=" X-RZG-CLASS-ID: mo00 Received: from positron.chronox.de by smtp.strato.de (RZmta 46.4.0 DYNA|AUTH) with ESMTPSA id 404ef0w3FAJv12C (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256 bits)) (Client did not present a certificate); Wed, 15 Apr 2020 12:19:57 +0200 (CEST) From: Stephan =?iso-8859-1?q?M=FCller?= To: Arnd Bergmann Cc: Greg Kroah-Hartman , linux-crypto@vger.kernel.org, LKML , linux-api@vger.kernel.org, "Eric W. Biederman" , "Alexander E. Patrakov" , "Ahmed S. Darwish" , "Theodore Y. Ts'o" , Willy Tarreau , Matthew Garrett , Vito Caputo , Andreas Dilger , Jan Kara , Ray Strode , William Jon McCann , zhangjs , Andy Lutomirski , Florian Weimer , Lennart Poettering , Nicolai Stange , "Peter, Matthias" , Marcelo Henrique Cerri , Roman Drahtmueller , Neil Horman , Randy Dunlap , Julia Lawall , Dan Carpenter Subject: [PATCH v30 02/12] LRNG - allocate one DRNG instance per NUMA node Date: Wed, 15 Apr 2020 12:10:46 +0200 Message-ID: <20413735.rLLoclF7Ky@positron.chronox.de> In-Reply-To: <11836144.hkEK2qVKZC@positron.chronox.de> References: <11836144.hkEK2qVKZC@positron.chronox.de> MIME-Version: 1.0 Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org In order to improve NUMA-locality when serving getrandom(2) requests, allocate one DRNG instance per node. The DRNG instance that is present right from the start of the kernel is reused as the first per-NUMA-node DRNG. For all remaining online NUMA nodes a new DRNG instance is allocated. During boot time, the multiple DRNG instances are seeded sequentially. With this, the first DRNG instance (referenced as the initial DRNG in the code) is completely seeded with 256 bits of entropy before the next DRNG instance is completely seeded. When random numbers are requested, the NUMA-node-local DRNG is checked whether it has been already fully seeded. If this is not the case, the initial DRNG is used to serve the request. CC: "Eric W. Biederman" CC: "Alexander E. Patrakov" CC: "Ahmed S. Darwish" CC: "Theodore Y. Ts'o" CC: Willy Tarreau CC: Matthew Garrett CC: Vito Caputo CC: Andreas Dilger CC: Jan Kara CC: Ray Strode CC: William Jon McCann CC: zhangjs CC: Andy Lutomirski CC: Florian Weimer CC: Lennart Poettering CC: Nicolai Stange Reviewed-by: Marcelo Henrique Cerri Reviewed-by: Roman Drahtmueller Tested-by: Roman Drahtmüller Tested-by: Marcelo Henrique Cerri Tested-by: Neil Horman Signed-off-by: Stephan Mueller --- drivers/char/lrng/Makefile | 2 + drivers/char/lrng/lrng_internal.h | 5 ++ drivers/char/lrng/lrng_numa.c | 101 ++++++++++++++++++++++++++++++ 3 files changed, 108 insertions(+) create mode 100644 drivers/char/lrng/lrng_numa.c diff --git a/drivers/char/lrng/Makefile b/drivers/char/lrng/Makefile index 1d2a0211973d..0a32f22c2c1a 100644 --- a/drivers/char/lrng/Makefile +++ b/drivers/char/lrng/Makefile @@ -7,3 +7,5 @@ obj-y += lrng_pool.o lrng_aux.o \ lrng_sw_noise.o lrng_archrandom.o \ lrng_drng.o lrng_chacha20.o \ lrng_interfaces.o \ + +obj-$(CONFIG_NUMA) += lrng_numa.o diff --git a/drivers/char/lrng/lrng_internal.h b/drivers/char/lrng/lrng_internal.h index edf38121ec65..ecb0a7ad5e7e 100644 --- a/drivers/char/lrng/lrng_internal.h +++ b/drivers/char/lrng/lrng_internal.h @@ -246,8 +246,13 @@ int lrng_drng_get_sleep(u8 *outbuf, u32 outbuflen); void lrng_drng_force_reseed(void); void lrng_drng_seed_work(struct work_struct *dummy); +#ifdef CONFIG_NUMA +struct lrng_drng **lrng_drng_instances(void); +void lrng_drngs_numa_alloc(void); +#else /* CONFIG_NUMA */ static inline struct lrng_drng **lrng_drng_instances(void) { return NULL; } static inline void lrng_drngs_numa_alloc(void) { return; } +#endif /* CONFIG_NUMA */ /************************** Health Test linking code **************************/ diff --git a/drivers/char/lrng/lrng_numa.c b/drivers/char/lrng/lrng_numa.c new file mode 100644 index 000000000000..947c5b3ed517 --- /dev/null +++ b/drivers/char/lrng/lrng_numa.c @@ -0,0 +1,101 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause +/* + * LRNG NUMA support + * + * Copyright (C) 2016 - 2020, Stephan Mueller + */ + +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt + +#include +#include + +#include "lrng_internal.h" + +static struct lrng_drng **lrng_drng __read_mostly = NULL; + +struct lrng_drng **lrng_drng_instances(void) +{ + return lrng_drng; +} + +/* Allocate the data structures for the per-NUMA node DRNGs */ +static void _lrng_drngs_numa_alloc(struct work_struct *work) +{ + struct lrng_drng **drngs; + struct lrng_drng *lrng_drng_init = lrng_drng_init_instance(); + u32 node; + bool init_drng_used = false; + + mutex_lock(&lrng_crypto_cb_update); + + /* per-NUMA-node DRNGs are already present */ + if (lrng_drng) + goto unlock; + + drngs = kcalloc(nr_node_ids, sizeof(void *), GFP_KERNEL|__GFP_NOFAIL); + for_each_online_node(node) { + struct lrng_drng *drng; + + if (!init_drng_used) { + drngs[node] = lrng_drng_init; + init_drng_used = true; + continue; + } + + drng = kmalloc_node(sizeof(struct lrng_drng), + GFP_KERNEL|__GFP_NOFAIL, node); + memset(drng, 0, sizeof(lrng_drng)); + + drng->crypto_cb = lrng_drng_init->crypto_cb; + drng->drng = drng->crypto_cb->lrng_drng_alloc( + LRNG_DRNG_SECURITY_STRENGTH_BYTES); + if (IS_ERR(drng->drng)) { + kfree(drng); + goto err; + } + + mutex_init(&drng->lock); + spin_lock_init(&drng->spin_lock); + + /* + * No reseeding of NUMA DRNGs from previous DRNGs as this + * would complicate the code. Let it simply reseed. + */ + lrng_drng_reset(drng); + drngs[node] = drng; + + lrng_pool_inc_numa_node(); + pr_info("DRNG for NUMA node %d allocated\n", node); + } + + /* Ensure that all NUMA nodes receive changed memory here. */ + mb(); + + if (!cmpxchg(&lrng_drng, NULL, drngs)) + goto unlock; + +err: + for_each_online_node(node) { + struct lrng_drng *drng = drngs[node]; + + if (drng == lrng_drng_init) + continue; + + if (drng) { + drng->crypto_cb->lrng_drng_dealloc(drng->drng); + kfree(drng); + } + } + kfree(drngs); + +unlock: + mutex_unlock(&lrng_crypto_cb_update); +} + +static DECLARE_WORK(lrng_drngs_numa_alloc_work, _lrng_drngs_numa_alloc); + +void lrng_drngs_numa_alloc(void) +{ + schedule_work(&lrng_drngs_numa_alloc_work); +}