From patchwork Fri Mar 12 19:47:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Lobakin X-Patchwork-Id: 399389 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D9B7C433E0 for ; Fri, 12 Mar 2021 19:48:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C7DBD64F83 for ; Fri, 12 Mar 2021 19:48:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234501AbhCLTrm (ORCPT ); Fri, 12 Mar 2021 14:47:42 -0500 Received: from mail-40131.protonmail.ch ([185.70.40.131]:49061 "EHLO mail-40131.protonmail.ch" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234486AbhCLTrN (ORCPT ); Fri, 12 Mar 2021 14:47:13 -0500 Date: Fri, 12 Mar 2021 19:47:10 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pm.me; s=protonmail; t=1615578431; bh=0WXUDRS6l3uzo7FR/B86Gske9XTm/nKNi8/gZiVufio=; h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References:From; b=W0fw0vAzGEdIe1NRTwFI0Zc+jgpfNGER5XVeToKKqjERriU/eabdNJjFcYaPzIYvR sgsHppSchZl7I1AchfFUvTTIHfHiQn6wcnBeP+CKBgsdVAPF/raBbAKhLA0Rd/x3uU TcM7ZMb+BjPva1FjJYCBHlK9NXuEl44vhYJ1GfasZIITzCKPNlW18uvDcvV5BsWGz2 R6FHupHuS4DYidzdmyScY9++8sttJFJWdffJ0skHDWhN1dgUfY7saizL7brVRZF0sP oUQCymviXI5/PNSi+Crt42y1nYhSAccdY+hbPhzVKl3UzxAydlQ8qF2tk0XFRAq85H 8/P633Zz1TQtg== To: "David S. Miller" , Jakub Kicinski From: Alexander Lobakin Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Jonathan Lemon , Alexander Lobakin , Eric Dumazet , Willem de Bruijn , Kevin Hao , Pablo Neira Ayuso , Jakub Sitnicki , Marco Elver , Dexuan Cui , Vladimir Oltean , Ariel Levkovich , Wang Qing , Davide Caratti , Guillaume Nault , Eran Ben Elisha , Mauro Carvalho Chehab , Kirill Tkhai , Bartosz Golaszewski , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org Reply-To: Alexander Lobakin Subject: [PATCH net-next 6/6] skbuff: micro-optimize {, __}skb_header_pointer() Message-ID: <20210312194538.337504-7-alobakin@pm.me> In-Reply-To: <20210312194538.337504-1-alobakin@pm.me> References: <20210312194538.337504-1-alobakin@pm.me> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org {,__}skb_header_pointer() helpers exist mainly for preventing accesses-beyond-end of the linear data. In the vast majorify of cases, they bail out on the first condition. All code going after is mostly a fallback. Mark the most common branch as 'likely' one to move it in-line. Also, skb_copy_bits() can return negative values only when the input arguments are invalid, e.g. offset is greater than skb->len. It can be safely marked as 'unlikely' branch, assuming that hotpath code provides sane input to not fail here. These two bump the throughput with a single Flow Dissector pass on every packet (e.g. with RPS or driver that uses eth_get_headlen()) on 20 Mbps per flow/core. Signed-off-by: Alexander Lobakin --- include/linux/skbuff.h | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) -- 2.30.2 diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 7873f24c0ae5..71f4d609819e 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -3680,11 +3680,10 @@ static inline void * __must_check __skb_header_pointer(const struct sk_buff *skb, int offset, int len, const void *data, int hlen, void *buffer) { - if (hlen - offset >= len) + if (likely(hlen - offset >= len)) return (void *)data + offset; - if (!skb || - skb_copy_bits(skb, offset, buffer, len) < 0) + if (!skb || unlikely(skb_copy_bits(skb, offset, buffer, len) < 0)) return NULL; return buffer;