From patchwork Tue Feb 9 20:47:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 379805 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1B2A2C4332D for ; Tue, 9 Feb 2021 21:21:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CCE3864DE7 for ; Tue, 9 Feb 2021 21:21:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234246AbhBIVUM (ORCPT ); Tue, 9 Feb 2021 16:20:12 -0500 Received: from mail-40136.protonmail.ch ([185.70.40.136]:60429 "EHLO mail-40136.protonmail.ch" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234031AbhBIUtB (ORCPT ); Tue, 9 Feb 2021 15:49:01 -0500 Date: Tue, 09 Feb 2021 20:47:35 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pm.me; s=protonmail; t=1612903667; bh=xAmMBMhVGogGcZQk9S0AyKPMPlFzoB8B7aZY5oTMpiQ=; h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References:From; b=aQ5LuT4wsTYqKxn3iSk/2Lnu9KvcFIHXtgFzQfgA3JMv2LtYTxLdZ7YbgP+OD9NbO r5h0X9tR4kiLQxbAA1Rrg23v81povpP55oFXvCOJ3fMSJtNQuWu8LSoDjR8zfqF27g CtOqCDJ42T5jSd3gSTJ0Z02QE5s68FH+4R/wDpKGi/VzfxgCjFMM/Lr/bFB5+6B+A6 Vh9U6+4qUoAYnjZDXYyJjwgeMLtH+Zc6npLNHC/I4pJyQ/HdwhTd5/6YyLIvTC/OmJ 2k+c0kqroTRsASCJj7nbUltZfNGaBSVRWmhc3zHyee6a6t+eg4ZA3eDAXDxVh5azOb z78RNxxt2IH7g== To: "David S. Miller" , Jakub Kicinski From: Alexander Lobakin Cc: Jonathan Lemon , Eric Dumazet , Dmitry Vyukov , Willem de Bruijn , Alexander Lobakin , Randy Dunlap , Kevin Hao , Pablo Neira Ayuso , Jakub Sitnicki , Marco Elver , Dexuan Cui , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Taehee Yoo , Cong Wang , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Miaohe Lin , Guillaume Nault , Yonghong Song , zhudi , Michal Kubecek , Marcelo Ricardo Leitner , Dmitry Safonov <0x7f454c46@gmail.com>, Yang Yingliang , linux-kernel@vger.kernel.org, netdev@vger.kernel.org Reply-To: Alexander Lobakin Subject: [v3 net-next 02/10] skbuff: simplify kmalloc_reserve() Message-ID: <20210209204533.327360-3-alobakin@pm.me> In-Reply-To: <20210209204533.327360-1-alobakin@pm.me> References: <20210209204533.327360-1-alobakin@pm.me> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Eversince the introduction of __kmalloc_reserve(), "ip" argument hasn't been used. _RET_IP_ is embedded inside kmalloc_node_track_caller(). Remove the redundant macro and rename the function after it. Signed-off-by: Alexander Lobakin --- net/core/skbuff.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index a0f846872d19..70289f22a6f4 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -273,11 +273,8 @@ EXPORT_SYMBOL(__netdev_alloc_frag_align); * may be used. Otherwise, the packet data may be discarded until enough * memory is free */ -#define kmalloc_reserve(size, gfp, node, pfmemalloc) \ - __kmalloc_reserve(size, gfp, node, _RET_IP_, pfmemalloc) - -static void *__kmalloc_reserve(size_t size, gfp_t flags, int node, - unsigned long ip, bool *pfmemalloc) +static void *kmalloc_reserve(size_t size, gfp_t flags, int node, + bool *pfmemalloc) { void *obj; bool ret_pfmemalloc = false; From patchwork Tue Feb 9 20:47:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 379808 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73F7EC43381 for ; Tue, 9 Feb 2021 21:01:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2065A64DB2 for ; Tue, 9 Feb 2021 21:01:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234289AbhBIU7f (ORCPT ); Tue, 9 Feb 2021 15:59:35 -0500 Received: from mail1.protonmail.ch ([185.70.40.18]:52696 "EHLO mail1.protonmail.ch" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234074AbhBIUu7 (ORCPT ); Tue, 9 Feb 2021 15:50:59 -0500 Date: Tue, 09 Feb 2021 20:47:48 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pm.me; s=protonmail; t=1612903676; bh=TNjH2OfntVMafeUExGLXFCMb3pBMh6ePd4fTgzTIZmc=; h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References:From; b=ou0BO8btB9uj5a0JyAgEuRgYLVipOvJTcyXQOi23D/fvjJyTvSB7srawpn7P7pJlv vfZCVrUUnb62n9yoVR8JzDg6AR3qMF5uQD79ypOdZ88KiJNCuJAPk1GnHBbZ9pNbhB NxyU85YPrVN+ggVUuCTPKWC0ye97KNiCiuE+KxFDBAbJjxqvOrPSUeBNNgdLEJE3/M Hfkyya8tWEIAcXtQcH1eBSjxGcU08QVWI1/NHe0E5qcggi/f10kqZfT/Hh6y8PXFQf /j/f1V5D6RtEEKUHSY+oQ3wwc/0B6SLP0Bopk7d1C3qGQ1Gi0RysAnh2Bn7C08xdFf FucsqV32sg3Iw== To: "David S. Miller" , Jakub Kicinski From: Alexander Lobakin Cc: Jonathan Lemon , Eric Dumazet , Dmitry Vyukov , Willem de Bruijn , Alexander Lobakin , Randy Dunlap , Kevin Hao , Pablo Neira Ayuso , Jakub Sitnicki , Marco Elver , Dexuan Cui , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Taehee Yoo , Cong Wang , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Miaohe Lin , Guillaume Nault , Yonghong Song , zhudi , Michal Kubecek , Marcelo Ricardo Leitner , Dmitry Safonov <0x7f454c46@gmail.com>, Yang Yingliang , linux-kernel@vger.kernel.org, netdev@vger.kernel.org Reply-To: Alexander Lobakin Subject: [v3 net-next 03/10] skbuff: make __build_skb_around() return void Message-ID: <20210209204533.327360-4-alobakin@pm.me> In-Reply-To: <20210209204533.327360-1-alobakin@pm.me> References: <20210209204533.327360-1-alobakin@pm.me> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org __build_skb_around() can never fail and always returns passed skb. Make it return void to simplify and optimize the code. Signed-off-by: Alexander Lobakin --- net/core/skbuff.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 70289f22a6f4..c7d184e11547 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -120,8 +120,8 @@ static void skb_under_panic(struct sk_buff *skb, unsigned int sz, void *addr) } /* Caller must provide SKB that is memset cleared */ -static struct sk_buff *__build_skb_around(struct sk_buff *skb, - void *data, unsigned int frag_size) +static void __build_skb_around(struct sk_buff *skb, void *data, + unsigned int frag_size) { struct skb_shared_info *shinfo; unsigned int size = frag_size ? : ksize(data); @@ -144,8 +144,6 @@ static struct sk_buff *__build_skb_around(struct sk_buff *skb, atomic_set(&shinfo->dataref, 1); skb_set_kcov_handle(skb, kcov_common_handle()); - - return skb; } /** @@ -176,8 +174,9 @@ struct sk_buff *__build_skb(void *data, unsigned int frag_size) return NULL; memset(skb, 0, offsetof(struct sk_buff, tail)); + __build_skb_around(skb, data, frag_size); - return __build_skb_around(skb, data, frag_size); + return skb; } /* build_skb() is wrapper over __build_skb(), that specifically @@ -210,9 +209,9 @@ struct sk_buff *build_skb_around(struct sk_buff *skb, if (unlikely(!skb)) return NULL; - skb = __build_skb_around(skb, data, frag_size); + __build_skb_around(skb, data, frag_size); - if (skb && frag_size) { + if (frag_size) { skb->head_frag = 1; if (page_is_pfmemalloc(virt_to_head_page(data))) skb->pfmemalloc = 1; From patchwork Tue Feb 9 20:48:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 379806 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD36EC433DB for ; Tue, 9 Feb 2021 21:16:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8811E64E74 for ; Tue, 9 Feb 2021 21:16:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233788AbhBIVPo (ORCPT ); Tue, 9 Feb 2021 16:15:44 -0500 Received: from mail-40136.protonmail.ch ([185.70.40.136]:54445 "EHLO mail-40136.protonmail.ch" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233767AbhBIUtQ (ORCPT ); Tue, 9 Feb 2021 15:49:16 -0500 Date: Tue, 09 Feb 2021 20:48:14 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pm.me; s=protonmail; t=1612903699; bh=85QQJNudKIF3Xtg97Y7g4QbqbeIMU4YL32cELilgRjM=; h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References:From; b=jhApZdmS2S6jhwoIFR1qFMrJ6Ar22V773Peax4i31bvsu8PZ9fepGhgyUjViJmVhS 7Xf5ZdSVsTMVkg8neW2VcsC8HSTrUjsSoLCvW/+KCp1HfitZdYSuYafr7U7YlatlP1 E2MO4pNbPYQ9ztadhoqkSrfN9Kbl8nROfOZmQISwoTSo979kbbfKJ1n/y83Gj7oHsn OnSLcwikPEpLi78JUssBc2v+gEoA/RFkplg4sO0iulkPxhLosp87I+ar9ypnRJLMTu p4wG38GmF9XhNW1HNCpvliQv2lyBb6uZpyPXChyhFugUsX3lZiOn5qOV8VcOBrF2lp lvfyBv34dYsXg== To: "David S. Miller" , Jakub Kicinski From: Alexander Lobakin Cc: Jonathan Lemon , Eric Dumazet , Dmitry Vyukov , Willem de Bruijn , Alexander Lobakin , Randy Dunlap , Kevin Hao , Pablo Neira Ayuso , Jakub Sitnicki , Marco Elver , Dexuan Cui , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Taehee Yoo , Cong Wang , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Miaohe Lin , Guillaume Nault , Yonghong Song , zhudi , Michal Kubecek , Marcelo Ricardo Leitner , Dmitry Safonov <0x7f454c46@gmail.com>, Yang Yingliang , linux-kernel@vger.kernel.org, netdev@vger.kernel.org Reply-To: Alexander Lobakin Subject: [v3 net-next 05/10] skbuff: use __build_skb_around() in __alloc_skb() Message-ID: <20210209204533.327360-6-alobakin@pm.me> In-Reply-To: <20210209204533.327360-1-alobakin@pm.me> References: <20210209204533.327360-1-alobakin@pm.me> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Just call __build_skb_around() instead of open-coding it. Signed-off-by: Alexander Lobakin --- net/core/skbuff.c | 18 +----------------- 1 file changed, 1 insertion(+), 17 deletions(-) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 88566de26cd1..1c6f6ef70339 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -326,7 +326,6 @@ struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mask, int flags, int node) { struct kmem_cache *cache; - struct skb_shared_info *shinfo; struct sk_buff *skb; u8 *data; bool pfmemalloc; @@ -366,21 +365,8 @@ struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mask, * the tail pointer in struct sk_buff! */ memset(skb, 0, offsetof(struct sk_buff, tail)); - /* Account for allocated memory : skb + skb->head */ - skb->truesize = SKB_TRUESIZE(size); + __build_skb_around(skb, data, 0); skb->pfmemalloc = pfmemalloc; - refcount_set(&skb->users, 1); - skb->head = data; - skb->data = data; - skb_reset_tail_pointer(skb); - skb->end = skb->tail + size; - skb->mac_header = (typeof(skb->mac_header))~0U; - skb->transport_header = (typeof(skb->transport_header))~0U; - - /* make sure we initialize shinfo sequentially */ - shinfo = skb_shinfo(skb); - memset(shinfo, 0, offsetof(struct skb_shared_info, dataref)); - atomic_set(&shinfo->dataref, 1); if (flags & SKB_ALLOC_FCLONE) { struct sk_buff_fclones *fclones; @@ -393,8 +379,6 @@ struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mask, fclones->skb2.fclone = SKB_FCLONE_CLONE; } - skb_set_kcov_handle(skb, kcov_common_handle()); - return skb; nodata: From patchwork Tue Feb 9 20:48:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 379807 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0844C4332B for ; Tue, 9 Feb 2021 21:13:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 71B2764E40 for ; Tue, 9 Feb 2021 21:13:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234354AbhBIVNE (ORCPT ); Tue, 9 Feb 2021 16:13:04 -0500 Received: from mail-40133.protonmail.ch ([185.70.40.133]:27345 "EHLO mail-40133.protonmail.ch" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234056AbhBIUty (ORCPT ); Tue, 9 Feb 2021 15:49:54 -0500 Date: Tue, 09 Feb 2021 20:48:55 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pm.me; s=protonmail; t=1612903738; bh=RvDCm6tIZoYg2bGWqrbsgh3uj/6504ovRb8wDmucExM=; h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References:From; b=KZV8LP67Z+l2Rd6WCrkmG/LpyISpqpz9CBxVx+vlfl3gPh+hnzmwKncxPno1p5KX9 13eVWCXg1Rt9eB+DAmaTHPhqAZHDSbGr2a+N0ms97bfKzSD40mKmKEIQxZDtaqBrlp +VeD4OfaDfuubIfFX/YJhZLbCpWzgPwKiwGklk+ugtp4csH/vXrb2KA0aqoiBDR6mT 92anaNSurAKxC+nJ54suOcF27lK1xlzq7EfDXvlTVkKvnCdm/5quTt+f6fH6XUbPIk hC4bPnIemYpktuVDeGjOLh6kjU+UcHqJAqgAQJVPywNomnkB2jhrVfGsBmlrI16pzb vTyide4+xG5GA== To: "David S. Miller" , Jakub Kicinski From: Alexander Lobakin Cc: Jonathan Lemon , Eric Dumazet , Dmitry Vyukov , Willem de Bruijn , Alexander Lobakin , Randy Dunlap , Kevin Hao , Pablo Neira Ayuso , Jakub Sitnicki , Marco Elver , Dexuan Cui , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Taehee Yoo , Cong Wang , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Miaohe Lin , Guillaume Nault , Yonghong Song , zhudi , Michal Kubecek , Marcelo Ricardo Leitner , Dmitry Safonov <0x7f454c46@gmail.com>, Yang Yingliang , linux-kernel@vger.kernel.org, netdev@vger.kernel.org, Edward Cree Reply-To: Alexander Lobakin Subject: [v3 net-next 08/10] skbuff: reuse NAPI skb cache on allocation path (__build_skb()) Message-ID: <20210209204533.327360-9-alobakin@pm.me> In-Reply-To: <20210209204533.327360-1-alobakin@pm.me> References: <20210209204533.327360-1-alobakin@pm.me> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Instead of just bulk-flushing skbuff_heads queued up through napi_consume_skb() or __kfree_skb_defer(), try to reuse them on allocation path. If the cache is empty on allocation, bulk-allocate the first half, which is more efficient than per-skb allocation. If the cache is full on freeing, bulk-wipe the second half. This also includes custom KASAN poisoning/unpoisoning to be double sure there are no use-after-free cases. Functions that got cache fastpath: - {,__}build_skb(); - {,__}netdev_alloc_skb(); - {,__}napi_alloc_skb(). Note on "napi_safe" argument: NAPI cache should be accessed only from BH-disabled or (better) NAPI context. To make sure access is safe, in_serving_softirq() check is used. Hovewer, there are plenty of cases when we know for sure that we're in such context. This includes: build_skb() (called only from NIC drivers in NAPI Rx context) and {,__}napi_alloc_skb() (called from the same place or from kernel network softirq functions). We can use that knowledge to avoid unnecessary checks. Suggested-by: Edward Cree # Unified cache part Suggested-by: Eric Dumazet # KASAN poisoning Suggested-by: Dmitry Vyukov # Help with KASAN Signed-off-by: Alexander Lobakin --- include/linux/skbuff.h | 2 +- net/core/skbuff.c | 61 ++++++++++++++++++++++++++++------------ net/netlink/af_netlink.c | 2 +- 3 files changed, 45 insertions(+), 20 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 0e0707296098..5bb443d37bf4 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -1082,7 +1082,7 @@ bool skb_try_coalesce(struct sk_buff *to, struct sk_buff *from, struct sk_buff *__alloc_skb(unsigned int size, gfp_t priority, int flags, int node); -struct sk_buff *__build_skb(void *data, unsigned int frag_size); +struct sk_buff *__build_skb(void *data, unsigned int frag_size, bool napi_safe); struct sk_buff *build_skb(void *data, unsigned int frag_size); struct sk_buff *build_skb_around(struct sk_buff *skb, void *data, unsigned int frag_size); diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 860a9d4f752f..8747566a8136 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -120,6 +120,7 @@ static void skb_under_panic(struct sk_buff *skb, unsigned int sz, void *addr) } #define NAPI_SKB_CACHE_SIZE 64 +#define NAPI_SKB_CACHE_HALF (NAPI_SKB_CACHE_SIZE / 2) struct napi_alloc_cache { struct page_frag_cache page; @@ -164,6 +165,30 @@ void *__netdev_alloc_frag_align(unsigned int fragsz, unsigned int align_mask) } EXPORT_SYMBOL(__netdev_alloc_frag_align); +static struct sk_buff *napi_skb_cache_get(bool napi_safe) +{ + struct napi_alloc_cache *nc; + struct sk_buff *skb; + + if (!napi_safe && unlikely(!in_serving_softirq())) + return kmem_cache_alloc(skbuff_head_cache, GFP_ATOMIC); + + nc = this_cpu_ptr(&napi_alloc_cache); + + if (unlikely(!nc->skb_count)) + nc->skb_count = kmem_cache_alloc_bulk(skbuff_head_cache, + GFP_ATOMIC, + NAPI_SKB_CACHE_HALF, + nc->skb_cache); + if (unlikely(!nc->skb_count)) + return NULL; + + skb = nc->skb_cache[--nc->skb_count]; + kasan_unpoison_object_data(skbuff_head_cache, skb); + + return skb; +} + /* Caller must provide SKB that is memset cleared */ static void __build_skb_around(struct sk_buff *skb, void *data, unsigned int frag_size) @@ -210,11 +235,11 @@ static void __build_skb_around(struct sk_buff *skb, void *data, * before giving packet to stack. * RX rings only contains data buffers, not full skbs. */ -struct sk_buff *__build_skb(void *data, unsigned int frag_size) +struct sk_buff *__build_skb(void *data, unsigned int frag_size, bool napi_safe) { struct sk_buff *skb; - skb = kmem_cache_alloc(skbuff_head_cache, GFP_ATOMIC); + skb = napi_skb_cache_get(napi_safe); if (unlikely(!skb)) return NULL; @@ -231,7 +256,7 @@ struct sk_buff *__build_skb(void *data, unsigned int frag_size) */ struct sk_buff *build_skb(void *data, unsigned int frag_size) { - struct sk_buff *skb = __build_skb(data, frag_size); + struct sk_buff *skb = __build_skb(data, frag_size, true); if (skb && frag_size) { skb->head_frag = 1; @@ -443,7 +468,7 @@ struct sk_buff *__netdev_alloc_skb(struct net_device *dev, unsigned int len, if (unlikely(!data)) return NULL; - skb = __build_skb(data, len); + skb = __build_skb(data, len, false); if (unlikely(!skb)) { skb_free_frag(data); return NULL; @@ -507,7 +532,7 @@ struct sk_buff *__napi_alloc_skb(struct napi_struct *napi, unsigned int len, if (unlikely(!data)) return NULL; - skb = __build_skb(data, len); + skb = __build_skb(data, len, true); if (unlikely(!skb)) { skb_free_frag(data); return NULL; @@ -838,31 +863,31 @@ void __consume_stateless_skb(struct sk_buff *skb) kfree_skbmem(skb); } -static inline void _kfree_skb_defer(struct sk_buff *skb) +static void napi_skb_cache_put(struct sk_buff *skb) { struct napi_alloc_cache *nc = this_cpu_ptr(&napi_alloc_cache); + u32 i; /* drop skb->head and call any destructors for packet */ skb_release_all(skb); - /* record skb to CPU local list */ + kasan_poison_object_data(skbuff_head_cache, skb); nc->skb_cache[nc->skb_count++] = skb; -#ifdef CONFIG_SLUB - /* SLUB writes into objects when freeing */ - prefetchw(skb); -#endif - - /* flush skb_cache if it is filled */ if (unlikely(nc->skb_count == NAPI_SKB_CACHE_SIZE)) { - kmem_cache_free_bulk(skbuff_head_cache, NAPI_SKB_CACHE_SIZE, - nc->skb_cache); - nc->skb_count = 0; + for (i = NAPI_SKB_CACHE_HALF; i < NAPI_SKB_CACHE_SIZE; i++) + kasan_unpoison_object_data(skbuff_head_cache, + nc->skb_cache[i]); + + kmem_cache_free_bulk(skbuff_head_cache, NAPI_SKB_CACHE_HALF, + nc->skb_cache + NAPI_SKB_CACHE_HALF); + nc->skb_count = NAPI_SKB_CACHE_HALF; } } + void __kfree_skb_defer(struct sk_buff *skb) { - _kfree_skb_defer(skb); + napi_skb_cache_put(skb); } void napi_consume_skb(struct sk_buff *skb, int budget) @@ -887,7 +912,7 @@ void napi_consume_skb(struct sk_buff *skb, int budget) return; } - _kfree_skb_defer(skb); + napi_skb_cache_put(skb); } EXPORT_SYMBOL(napi_consume_skb); diff --git a/net/netlink/af_netlink.c b/net/netlink/af_netlink.c index dd488938447f..afba4e11a526 100644 --- a/net/netlink/af_netlink.c +++ b/net/netlink/af_netlink.c @@ -1190,7 +1190,7 @@ static struct sk_buff *netlink_alloc_large_skb(unsigned int size, if (data == NULL) return NULL; - skb = __build_skb(data, size); + skb = __build_skb(data, size, false); if (skb == NULL) vfree(data); else From patchwork Tue Feb 9 20:49:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 379809 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DCEF5C433E6 for ; Tue, 9 Feb 2021 21:01:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9BA9A64E6B for ; Tue, 9 Feb 2021 21:01:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234227AbhBIU6E (ORCPT ); Tue, 9 Feb 2021 15:58:04 -0500 Received: from mail-40134.protonmail.ch ([185.70.40.134]:21330 "EHLO mail-40134.protonmail.ch" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233862AbhBIUuU (ORCPT ); Tue, 9 Feb 2021 15:50:20 -0500 Date: Tue, 09 Feb 2021 20:49:30 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pm.me; s=protonmail; t=1612903778; bh=hDcvm3I1usK8Mp4wrfq9qJ216nYvewc8ajVIpsly8xw=; h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References:From; b=Q1r9YNcKYVYQN30xsJRU37z3WfBhThWeGyroYTGTYZsFOS8Bowbel0iNEptyiJ4x9 h/jKzie6Rx0DIezVZLUdJCAc/U8ooUMMCEV6sityGhDGGNrpDMvrYqdRZyoEzxCh1h 0XMD8/XjF0RaHmEBvkzbVUNuguv7mRTQLV4qC3roT3VZg3nM62COwkqcWTB+gWOtib v8K1gyqD5VC9B10akYBpbxSOxYe3eRLiH5swFlB+yFq/hCcVq8Gr/z6PbHLR5EEyv7 0DadsEajDkH96mqsg0YOZnkqKRhTx63xworLuuPc5NAxC37Rp4EYHBlLGnZs5hgWmA /hs8LvwwTaPYA== To: "David S. Miller" , Jakub Kicinski From: Alexander Lobakin Cc: Jonathan Lemon , Eric Dumazet , Dmitry Vyukov , Willem de Bruijn , Alexander Lobakin , Randy Dunlap , Kevin Hao , Pablo Neira Ayuso , Jakub Sitnicki , Marco Elver , Dexuan Cui , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Taehee Yoo , Cong Wang , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Miaohe Lin , Guillaume Nault , Yonghong Song , zhudi , Michal Kubecek , Marcelo Ricardo Leitner , Dmitry Safonov <0x7f454c46@gmail.com>, Yang Yingliang , linux-kernel@vger.kernel.org, netdev@vger.kernel.org Reply-To: Alexander Lobakin Subject: [v3 net-next 10/10] skbuff: queue NAPI_MERGED_FREE skbs into NAPI cache instead of freeing Message-ID: <20210209204533.327360-11-alobakin@pm.me> In-Reply-To: <20210209204533.327360-1-alobakin@pm.me> References: <20210209204533.327360-1-alobakin@pm.me> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org napi_frags_finish() and napi_skb_finish() can only be called inside NAPI Rx context, so we can feed NAPI cache with skbuff_heads that got NAPI_MERGED_FREE verdict instead of immediate freeing. Replace __kfree_skb() with __kfree_skb_defer() in napi_skb_finish() and move napi_skb_free_stolen_head() to skbuff.c, so it can drop skbs to NAPI cache. As many drivers call napi_alloc_skb()/napi_get_frags() on their receive path, this becomes especially useful. Signed-off-by: Alexander Lobakin --- include/linux/skbuff.h | 1 + net/core/dev.c | 9 +-------- net/core/skbuff.c | 12 +++++++++--- 3 files changed, 11 insertions(+), 11 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 5bb443d37bf4..f8737ad91cc7 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -2919,6 +2919,7 @@ static inline struct sk_buff *napi_alloc_skb(struct napi_struct *napi, } void napi_consume_skb(struct sk_buff *skb, int budget); +void napi_skb_free_stolen_head(struct sk_buff *skb); void __kfree_skb_defer(struct sk_buff *skb); /** diff --git a/net/core/dev.c b/net/core/dev.c index 135d46c0c3c7..68ad03382f6a 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -6056,13 +6056,6 @@ struct packet_offload *gro_find_complete_by_type(__be16 type) } EXPORT_SYMBOL(gro_find_complete_by_type); -static void napi_skb_free_stolen_head(struct sk_buff *skb) -{ - skb_dst_drop(skb); - skb_ext_put(skb); - kmem_cache_free(skbuff_head_cache, skb); -} - static gro_result_t napi_skb_finish(struct napi_struct *napi, struct sk_buff *skb, gro_result_t ret) @@ -6076,7 +6069,7 @@ static gro_result_t napi_skb_finish(struct napi_struct *napi, if (NAPI_GRO_CB(skb)->free == NAPI_GRO_FREE_STOLEN_HEAD) napi_skb_free_stolen_head(skb); else - __kfree_skb(skb); + __kfree_skb_defer(skb); break; case GRO_HELD: diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 8850086f8605..6dbc486c1d68 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -872,9 +872,6 @@ static void napi_skb_cache_put(struct sk_buff *skb) struct napi_alloc_cache *nc = this_cpu_ptr(&napi_alloc_cache); u32 i; - /* drop skb->head and call any destructors for packet */ - skb_release_all(skb); - kasan_poison_object_data(skbuff_head_cache, skb); nc->skb_cache[nc->skb_count++] = skb; @@ -891,6 +888,14 @@ static void napi_skb_cache_put(struct sk_buff *skb) void __kfree_skb_defer(struct sk_buff *skb) { + skb_release_all(skb); + napi_skb_cache_put(skb); +} + +void napi_skb_free_stolen_head(struct sk_buff *skb) +{ + skb_dst_drop(skb); + skb_ext_put(skb); napi_skb_cache_put(skb); } @@ -916,6 +921,7 @@ void napi_consume_skb(struct sk_buff *skb, int budget) return; } + skb_release_all(skb); napi_skb_cache_put(skb); } EXPORT_SYMBOL(napi_consume_skb);