From patchwork Fri May 1 13:24:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Greg Kroah-Hartman X-Patchwork-Id: 226604 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B3BCC47253 for ; Fri, 1 May 2020 13:45:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0B3012051A for ; Fri, 1 May 2020 13:45:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1588340759; bh=t5G0LgOWL6P9GAICIqsd0ieuhfB9zxCfY/LCqMyZtxs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=b6Y/PGPxqSqVrajFWq1A7yNQTXt+KPTDFjUlMbJAw4CWRrcSnZzQyQbh9UoaRdYcJ PE2LZZQpjYKN1L2Om3vpug5fu/54w6UoCaxqEsy33laa7jzgDq6uxlCzYxE7jHJvaz XnvMrXXCH9f47BvVPcLmA49tz1+llabv6nMRWgOw= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731864AbgEANpW (ORCPT ); Fri, 1 May 2020 09:45:22 -0400 Received: from mail.kernel.org ([198.145.29.99]:46930 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731862AbgEANpV (ORCPT ); Fri, 1 May 2020 09:45:21 -0400 Received: from localhost (83-86-89-107.cable.dynamic.v4.ziggo.nl [83.86.89.107]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 8217B205C9; Fri, 1 May 2020 13:45:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1588340721; bh=t5G0LgOWL6P9GAICIqsd0ieuhfB9zxCfY/LCqMyZtxs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VRZWrM1CKS75OEjYVjTR+LzzNtgtcaKBG4C5sBkGGOxJkJLuHGyrktKfFQ95XqZiH w5yBliWiU3WVb900APg8niCnvHQT/dGxkCjh8woCjbv3NTmNrRd54e7OrkSHPKxf75 nWz05h8XiZa+kQw6YD58MRirLqySJuNQWB5+lzBk= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Eric Dumazet , Paolo Abeni , Willem de Bruijn , "David S. Miller" Subject: [PATCH 5.6 101/106] net: use indirect call wrappers for skb_copy_datagram_iter() Date: Fri, 1 May 2020 15:24:14 +0200 Message-Id: <20200501131555.335484038@linuxfoundation.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200501131543.421333643@linuxfoundation.org> References: <20200501131543.421333643@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Sender: stable-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org From: Eric Dumazet commit 29f3490ba9d2399d3d1b20c4aa74592d92bd4e11 upstream. TCP recvmsg() calls skb_copy_datagram_iter(), which calls an indirect function (cb pointing to simple_copy_to_iter()) for every MSS (fragment) present in the skb. CONFIG_RETPOLINE=y forces a very expensive operation that we can avoid thanks to indirect call wrappers. This patch gives a 13% increase of performance on a single flow, if the bottleneck is the thread reading the TCP socket. Fixes: 950fcaecd5cc ("datagram: consolidate datagram copy to iter helpers") Signed-off-by: Eric Dumazet Acked-by: Paolo Abeni Acked-by: Willem de Bruijn Signed-off-by: David S. Miller Signed-off-by: Greg Kroah-Hartman --- net/core/datagram.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -51,6 +51,7 @@ #include #include #include +#include #include #include @@ -414,6 +415,11 @@ int skb_kill_datagram(struct sock *sk, s } EXPORT_SYMBOL(skb_kill_datagram); +INDIRECT_CALLABLE_DECLARE(static size_t simple_copy_to_iter(const void *addr, + size_t bytes, + void *data __always_unused, + struct iov_iter *i)); + static int __skb_datagram_iter(const struct sk_buff *skb, int offset, struct iov_iter *to, int len, bool fault_short, size_t (*cb)(const void *, size_t, void *, @@ -427,7 +433,8 @@ static int __skb_datagram_iter(const str if (copy > 0) { if (copy > len) copy = len; - n = cb(skb->data + offset, copy, data, to); + n = INDIRECT_CALL_1(cb, simple_copy_to_iter, + skb->data + offset, copy, data, to); offset += n; if (n != copy) goto short_copy; @@ -449,8 +456,9 @@ static int __skb_datagram_iter(const str if (copy > len) copy = len; - n = cb(vaddr + skb_frag_off(frag) + offset - start, - copy, data, to); + n = INDIRECT_CALL_1(cb, simple_copy_to_iter, + vaddr + skb_frag_off(frag) + offset - start, + copy, data, to); kunmap(page); offset += n; if (n != copy)