From patchwork Tue Nov 10 11:01:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Magnus Karlsson X-Patchwork-Id: 321968 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.6 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D5774C4742C for ; Tue, 10 Nov 2020 11:02:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7DE7E2076E for ; Tue, 10 Nov 2020 11:02:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="HDo2tLvT" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730250AbgKJLCI (ORCPT ); Tue, 10 Nov 2020 06:02:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56518 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729630AbgKJLCI (ORCPT ); Tue, 10 Nov 2020 06:02:08 -0500 Received: from mail-pg1-x541.google.com (mail-pg1-x541.google.com [IPv6:2607:f8b0:4864:20::541]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EB74CC0613CF; Tue, 10 Nov 2020 03:02:07 -0800 (PST) Received: by mail-pg1-x541.google.com with SMTP id h6so9848109pgk.4; Tue, 10 Nov 2020 03:02:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=+SbxyZahfMfXnU/FChOm2quTfC2gyIuFdUfnHAtZnoI=; b=HDo2tLvTPlUvSl/ynyu+g51F9cSukXV+T7MVRGwd/ECz1C+0G0+I2wcLPsSbC18oXd inwKKlEU3sDBIhLVNZk+P/sw4/DfsG0ikuc8rcDBubiB8nStQezh2QFpA9tVWy/mmYXm 0NGQm50exAolDD2y6mImHFcw8p691Yr8NaP+eFDCmJ+Ep0mHz2HmqPLgWqxyRjBm+x/2 wrkiY0LtzeZvMJXty6GT15PD+2pZog2E2BfDbekl3yL+wTk6+IBdbl4rq/X3ialkWXa/ D5nNSkCip85Tn6/xHsVoPBXLanXtl9GKLQslz/3j3gvSFeTCMeS6SxuMRQ5eomkKYJ/f Q2Vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=+SbxyZahfMfXnU/FChOm2quTfC2gyIuFdUfnHAtZnoI=; b=oOOt8918G75V2PJdGrFK71DFQ61YEGILrQgYyK6pgRfTJQcQJrWS779t+lmJ0pe6n7 NbW8kA7dGbnpcxYuiIT/PeQY11AUeiKJOzqzsIr004OHXtbW9WPKctiSywiSgxFbwy2w dEVB6hRFYzlYgRtZw8mDiYlS4qdbHxy9Pu//bD//8Yy997XgyPLpbmSq6mxTXceIN2Ba fQpj+1lgEfRFZYPLa7Juhmp9p2AqO6bRBHP1iA3aIngDQEOFkKP/Ki3IaCKiqeR3Y73l 66d0pzex/RqMKHIa/JN1U8V3X32l/GfnTro4MjPGd8TFfw+elEQtU66t0haGCXN+FoIz Hp3Q== X-Gm-Message-State: AOAM531atR93xP7skoQ3U+S/pLVno8XLoRbFTN1sCRKsPKuaodpuI5Jn gqJOh1pFx2r2hysCkp9M0JA= X-Google-Smtp-Source: ABdhPJzxXLLj8wlpgU6Uci22CTaPUGpi5ulPRN9nH5GXepx6NeUX+461wyJD5OnMiToM2Fxo8oX5zQ== X-Received: by 2002:a63:4912:: with SMTP id w18mr17141230pga.131.1605006127544; Tue, 10 Nov 2020 03:02:07 -0800 (PST) Received: from VM.ger.corp.intel.com ([192.55.55.45]) by smtp.gmail.com with ESMTPSA id 22sm3012024pjb.40.2020.11.10.03.02.02 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 Nov 2020 03:02:06 -0800 (PST) From: Magnus Karlsson To: magnus.karlsson@intel.com, bjorn.topel@intel.com, ast@kernel.org, daniel@iogearbox.net, netdev@vger.kernel.org, jonathan.lemon@gmail.com, kuba@kernel.org, john.fastabend@gmail.com Cc: bpf@vger.kernel.org, jeffrey.t.kirsher@intel.com, anthony.l.nguyen@intel.com, maciej.fijalkowski@intel.com, maciejromanfijalkowski@gmail.com, intel-wired-lan@lists.osuosl.org Subject: [PATCH bpf-next v2 1/5] samples/bpf: increment Tx stats at sending Date: Tue, 10 Nov 2020 12:01:30 +0100 Message-Id: <1605006094-31097-2-git-send-email-magnus.karlsson@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1605006094-31097-1-git-send-email-magnus.karlsson@gmail.com> References: <1605006094-31097-1-git-send-email-magnus.karlsson@gmail.com> Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Magnus Karlsson Increment the statistics over how many Tx packets have been sent at the time of sending instead of at the time of completion. This as a completion event means that the buffer has been sent AND returned to user space. The packet always gets sent shortly after sendto() is called. The kernel might, for performance reasons, decide to not return every single buffer to user space immediately after sending, for example, only after a batch of packets have been transmitted. Incrementing the number of packets sent at completion, will in that case be confusing as if you send a single packet, the counter might show zero for a while even though the packet has been transmitted. Signed-off-by: Magnus Karlsson Acked-by: John Fastabend --- samples/bpf/xdpsock_user.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/samples/bpf/xdpsock_user.c b/samples/bpf/xdpsock_user.c index 1149e94..2567f0d 100644 --- a/samples/bpf/xdpsock_user.c +++ b/samples/bpf/xdpsock_user.c @@ -1146,7 +1146,6 @@ static inline void complete_tx_l2fwd(struct xsk_socket_info *xsk, xsk_ring_prod__submit(&xsk->umem->fq, rcvd); xsk_ring_cons__release(&xsk->umem->cq, rcvd); xsk->outstanding_tx -= rcvd; - xsk->ring_stats.tx_npkts += rcvd; } } @@ -1168,7 +1167,6 @@ static inline void complete_tx_only(struct xsk_socket_info *xsk, if (rcvd > 0) { xsk_ring_cons__release(&xsk->umem->cq, rcvd); xsk->outstanding_tx -= rcvd; - xsk->ring_stats.tx_npkts += rcvd; } } @@ -1260,6 +1258,7 @@ static void tx_only(struct xsk_socket_info *xsk, u32 *frame_nb, int batch_size) } xsk_ring_prod__submit(&xsk->tx, batch_size); + xsk->ring_stats.tx_npkts += batch_size; xsk->outstanding_tx += batch_size; *frame_nb += batch_size; *frame_nb %= NUM_FRAMES; @@ -1348,6 +1347,7 @@ static void l2fwd(struct xsk_socket_info *xsk, struct pollfd *fds) } return; } + xsk->ring_stats.rx_npkts += rcvd; ret = xsk_ring_prod__reserve(&xsk->tx, rcvd, &idx_tx); while (ret != rcvd) { @@ -1379,7 +1379,7 @@ static void l2fwd(struct xsk_socket_info *xsk, struct pollfd *fds) xsk_ring_prod__submit(&xsk->tx, rcvd); xsk_ring_cons__release(&xsk->rx, rcvd); - xsk->ring_stats.rx_npkts += rcvd; + xsk->ring_stats.tx_npkts += rcvd; xsk->outstanding_tx += rcvd; } From patchwork Tue Nov 10 11:01:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Magnus Karlsson X-Patchwork-Id: 321967 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.6 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86561C388F7 for ; Tue, 10 Nov 2020 11:02:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3035B20795 for ; Tue, 10 Nov 2020 11:02:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ukDfonde" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730397AbgKJLCT (ORCPT ); Tue, 10 Nov 2020 06:02:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56554 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730291AbgKJLCR (ORCPT ); Tue, 10 Nov 2020 06:02:17 -0500 Received: from mail-pf1-x441.google.com (mail-pf1-x441.google.com [IPv6:2607:f8b0:4864:20::441]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8AF0EC0613D1; Tue, 10 Nov 2020 03:02:17 -0800 (PST) Received: by mail-pf1-x441.google.com with SMTP id w14so8478418pfd.7; Tue, 10 Nov 2020 03:02:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=bAeSUpBLbjOCEiLqRz8yHChict8yQeBE89i4qMRCUYE=; b=ukDfondez4f1XSppfnFc+BqF8nQ4L3FdFhCthj7LN4hHv62Bc6btOnZghg75/vUhoC SgBLIjpr3dnU39LDAiTywvdqiYorGMLmYViWeiBMTc2vcSU+oKypGOfj3ctC4gOMAqv1 2BxuTVAg0FPF7IYzCWFasiov0yQN00GObJQWmVDQ4INjoONNHlP8MeV9Tsl3sXBziEHP dzVv+3U7r4iVqELVVMQkBkItHS2+aJ4nLgG4wTWhbrJ0LQNlPl1h+sUQDbUzxHmfXAVr R/zwAK00jdJuFb0ZQokoY75S4oyzlHUfRhlLxLRbi0CMB6AxlRB1g635DAAC3vop7rj8 dHDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=bAeSUpBLbjOCEiLqRz8yHChict8yQeBE89i4qMRCUYE=; b=GEFr75U2pvKv3PKRrDraw4VuA8PWPPxx/9rHXBZQjt8rV8b3xxE/hwwDojC7oDlUPL Ck+rX4TrWoDPqUuQE6rr0TdpFPJ66JMC6gAP7hS/LfE7f4bQth1MUGh7NwTa9ibmhEy6 nvohEyifYBDomW7O0EB61ebdCqK8aUSwPc3lKhxRQLFCObXoG3qdqSZj2ykzjAZcH8o2 X7togiAx3oReD4eIfU+cVoB6GAStXm0VThzGmD1/+yHbDBGMzrEp/Bu5/FrPsuMP6dj5 N/RJiK0TpfSFjNdu5WZU9pKPJ+qMtxsN9TrTKw4OOvZOrq/K7ZvQYOMelDm7qWFIEZlT En3w== X-Gm-Message-State: AOAM5307AfIDe6BX2vvvfDfh0hugdAYhyGqwTyJNEFiHqYUm3g/gKmAF VFqVesHe3e02+0+ZWUUpRRA= X-Google-Smtp-Source: ABdhPJxAFPJE7ck3w/rJbVJgGyaBFc5nGW5An7BtutpvMMirfepfVJnKzWTpZoLEvdslA1CaiziadA== X-Received: by 2002:a17:90b:395:: with SMTP id ga21mr4396814pjb.219.1605006137204; Tue, 10 Nov 2020 03:02:17 -0800 (PST) Received: from VM.ger.corp.intel.com ([192.55.55.45]) by smtp.gmail.com with ESMTPSA id 22sm3012024pjb.40.2020.11.10.03.02.12 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 Nov 2020 03:02:16 -0800 (PST) From: Magnus Karlsson To: magnus.karlsson@intel.com, bjorn.topel@intel.com, ast@kernel.org, daniel@iogearbox.net, netdev@vger.kernel.org, jonathan.lemon@gmail.com, kuba@kernel.org, john.fastabend@gmail.com Cc: bpf@vger.kernel.org, jeffrey.t.kirsher@intel.com, anthony.l.nguyen@intel.com, maciej.fijalkowski@intel.com, maciejromanfijalkowski@gmail.com, intel-wired-lan@lists.osuosl.org Subject: [PATCH bpf-next v2 3/5] xsk: introduce padding between more ring pointers Date: Tue, 10 Nov 2020 12:01:32 +0100 Message-Id: <1605006094-31097-4-git-send-email-magnus.karlsson@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1605006094-31097-1-git-send-email-magnus.karlsson@gmail.com> References: <1605006094-31097-1-git-send-email-magnus.karlsson@gmail.com> Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Magnus Karlsson Introduce one cache line worth of padding between the consumer pointer and the flags field as well as between the flags field and the start of the descriptors in all the lockless rings. This so that the x86 HW adjacency prefetcher will not prefetch the adjacent pointer/field when only one pointer/field is going to be used. This improves throughput performance for the l2fwd sample app with 1% on my machine with HW prefetching turned on in the BIOS. Signed-off-by: Magnus Karlsson Acked-by: John Fastabend --- net/xdp/xsk_queue.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/net/xdp/xsk_queue.h b/net/xdp/xsk_queue.h index cdb9cf3..74fac80 100644 --- a/net/xdp/xsk_queue.h +++ b/net/xdp/xsk_queue.h @@ -18,9 +18,11 @@ struct xdp_ring { /* Hinder the adjacent cache prefetcher to prefetch the consumer * pointer if the producer pointer is touched and vice versa. */ - u32 pad ____cacheline_aligned_in_smp; + u32 pad1 ____cacheline_aligned_in_smp; u32 consumer ____cacheline_aligned_in_smp; + u32 pad2 ____cacheline_aligned_in_smp; u32 flags; + u32 pad3 ____cacheline_aligned_in_smp; }; /* Used for the RX and TX queues for packets */ From patchwork Tue Nov 10 11:01:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Magnus Karlsson X-Patchwork-Id: 321966 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.6 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51838C55ABD for ; Tue, 10 Nov 2020 11:02:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E4A212076E for ; Tue, 10 Nov 2020 11:02:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ZzbCjSro" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730893AbgKJLC2 (ORCPT ); Tue, 10 Nov 2020 06:02:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56584 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730291AbgKJLC1 (ORCPT ); Tue, 10 Nov 2020 06:02:27 -0500 Received: from mail-pg1-x544.google.com (mail-pg1-x544.google.com [IPv6:2607:f8b0:4864:20::544]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 67CB4C0613CF; Tue, 10 Nov 2020 03:02:27 -0800 (PST) Received: by mail-pg1-x544.google.com with SMTP id r186so9861660pgr.0; Tue, 10 Nov 2020 03:02:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Np8Kwt0HgYs9CiWR284tIJBNdXajPeiuljd5LB33iOo=; b=ZzbCjSroItNvdMfepjeqAArqNYIKkDILb1A2SstxWnNgalJmtyzVsmZ5OrtCgXrGUV zJ83s9Ll2Yrng7BCz/zip8+fweAac7QBN3oYc8NqhGZQoymQDefh4nU5Ow86foqYDP8J EtW+pwdmT5/M3dkmm+JWi/g6/hGeTNnenOIu2xo+BlBEkD3vPOqgiJdI6LaXOaenlg8Z xYTt/PCkuqwTvZfu9LXVUWQb2ewNG5LxTpnURB45StJMMR89lIOl7aQuiw13pxYty+PH nfAOv0LvkI1pw0UrsEDUCakwQxoC/BRxOMDOTgp0Oi0vQz/0/usoZUpeap+G0v7KuUel wLFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Np8Kwt0HgYs9CiWR284tIJBNdXajPeiuljd5LB33iOo=; b=VRzzbh6nE8qQ9uApPwqgPPxwNwbT+vGKsWDKuTlb+cuikIg7r2jJ4y6eocFCBO2Bf+ 7wWsteLr8VJApl3iVtzfxQG7MrA6SsP1DvMbxA56yH24sWRwPyma2Dka1pZNQs0/4aIT yRFueHla4rWLRDSuRTsfQmWiWTqjlc0/0xKIjkiQrnFcoIAgPZMrDE5SYV/qs1W5jdr1 1d84bYnJbprk2gPJYnKLNbsDtNe4ivwutXHY2bQ5cMhvlmj/s75OUKsAE6qI1JSzRGHx 0lYiEEOrFtZehDGrx1DQjRxrVrupjJQYnCGt/QUQwj452DUVMxHLcrctziZpjxSLbuic O6ZQ== X-Gm-Message-State: AOAM531vwoRE/jQzEx3KrRYcK2OE/LoxU+DSbIBjOnlfhqy/zQSp/JcG g4ELcWS+vxik0tOsHZnbruE= X-Google-Smtp-Source: ABdhPJxgIiZYzHp+Qdpymo3t9JY0huD1GMCTe+j+bFgmutjAPVvpKaJSyxfvX2pRQUWqSF3+lOXsNg== X-Received: by 2002:a62:7e14:0:b029:18a:d515:dc47 with SMTP id z20-20020a627e140000b029018ad515dc47mr18316970pfc.78.1605006146962; Tue, 10 Nov 2020 03:02:26 -0800 (PST) Received: from VM.ger.corp.intel.com ([192.55.55.45]) by smtp.gmail.com with ESMTPSA id 22sm3012024pjb.40.2020.11.10.03.02.22 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 10 Nov 2020 03:02:26 -0800 (PST) From: Magnus Karlsson To: magnus.karlsson@intel.com, bjorn.topel@intel.com, ast@kernel.org, daniel@iogearbox.net, netdev@vger.kernel.org, jonathan.lemon@gmail.com, kuba@kernel.org, john.fastabend@gmail.com Cc: bpf@vger.kernel.org, jeffrey.t.kirsher@intel.com, anthony.l.nguyen@intel.com, maciej.fijalkowski@intel.com, maciejromanfijalkowski@gmail.com, intel-wired-lan@lists.osuosl.org Subject: [PATCH bpf-next v2 5/5] i40e: use batched xsk Tx interfaces to increase performance Date: Tue, 10 Nov 2020 12:01:34 +0100 Message-Id: <1605006094-31097-6-git-send-email-magnus.karlsson@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1605006094-31097-1-git-send-email-magnus.karlsson@gmail.com> References: <1605006094-31097-1-git-send-email-magnus.karlsson@gmail.com> Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Magnus Karlsson Use the new batched xsk interfaces for the Tx path in the i40e driver to improve performance. On my machine, this yields a throughput increase of 4% for the l2fwd sample app in xdpsock. If we instead just look at the Tx part, this patch set increases throughput with above 20% for Tx. Note that I had to explicitly loop unroll the inner loop to get to this performance level, by using a pragma. It is honored by both clang and gcc and should be ignored by versions that do not support it. Using the -funroll-loops compiler command line switch on the source file resulted in a loop unrolling on a higher level that lead to a performance decrease instead of an increase. Signed-off-by: Magnus Karlsson Acked-by: John Fastabend Reported-by: kernel test robot --- drivers/net/ethernet/intel/i40e/i40e_txrx.c | 11 +++ drivers/net/ethernet/intel/i40e/i40e_txrx.h | 1 + drivers/net/ethernet/intel/i40e/i40e_xsk.c | 127 ++++++++++++++++++++-------- 3 files changed, 104 insertions(+), 35 deletions(-) diff --git a/drivers/net/ethernet/intel/i40e/i40e_txrx.c b/drivers/net/ethernet/intel/i40e/i40e_txrx.c index d43ce13..c21548c 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_txrx.c +++ b/drivers/net/ethernet/intel/i40e/i40e_txrx.c @@ -676,6 +676,8 @@ void i40e_free_tx_resources(struct i40e_ring *tx_ring) i40e_clean_tx_ring(tx_ring); kfree(tx_ring->tx_bi); tx_ring->tx_bi = NULL; + kfree(tx_ring->xsk_descs); + tx_ring->xsk_descs = NULL; if (tx_ring->desc) { dma_free_coherent(tx_ring->dev, tx_ring->size, @@ -1277,6 +1279,13 @@ int i40e_setup_tx_descriptors(struct i40e_ring *tx_ring) if (!tx_ring->tx_bi) goto err; + if (ring_is_xdp(tx_ring)) { + tx_ring->xsk_descs = kcalloc(I40E_MAX_NUM_DESCRIPTORS, sizeof(*tx_ring->xsk_descs), + GFP_KERNEL); + if (!tx_ring->xsk_descs) + goto err; + } + u64_stats_init(&tx_ring->syncp); /* round up to nearest 4K */ @@ -1300,6 +1309,8 @@ int i40e_setup_tx_descriptors(struct i40e_ring *tx_ring) return 0; err: + kfree(tx_ring->xsk_descs); + tx_ring->xsk_descs = NULL; kfree(tx_ring->tx_bi); tx_ring->tx_bi = NULL; return -ENOMEM; diff --git a/drivers/net/ethernet/intel/i40e/i40e_txrx.h b/drivers/net/ethernet/intel/i40e/i40e_txrx.h index 2feed92..5f531b1 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_txrx.h +++ b/drivers/net/ethernet/intel/i40e/i40e_txrx.h @@ -389,6 +389,7 @@ struct i40e_ring { struct i40e_channel *ch; struct xdp_rxq_info xdp_rxq; struct xsk_buff_pool *xsk_pool; + struct xdp_desc *xsk_descs; /* For storing descriptors in the AF_XDP ZC path */ } ____cacheline_internodealigned_in_smp; static inline bool ring_uses_build_skb(struct i40e_ring *ring) diff --git a/drivers/net/ethernet/intel/i40e/i40e_xsk.c b/drivers/net/ethernet/intel/i40e/i40e_xsk.c index 61aa1fc..a271a02 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_xsk.c +++ b/drivers/net/ethernet/intel/i40e/i40e_xsk.c @@ -381,6 +381,78 @@ int i40e_clean_rx_irq_zc(struct i40e_ring *rx_ring, int budget) return failure ? budget : (int)total_rx_packets; } +static void i40e_xmit_pkt(struct i40e_ring *xdp_ring, struct xdp_desc *desc, + unsigned int *total_bytes) +{ + struct i40e_tx_desc *tx_desc; + dma_addr_t dma; + + dma = xsk_buff_raw_get_dma(xdp_ring->xsk_pool, desc->addr); + xsk_buff_raw_dma_sync_for_device(xdp_ring->xsk_pool, dma, desc->len); + + tx_desc = I40E_TX_DESC(xdp_ring, xdp_ring->next_to_use++); + tx_desc->buffer_addr = cpu_to_le64(dma); + tx_desc->cmd_type_offset_bsz = build_ctob(I40E_TX_DESC_CMD_ICRC | I40E_TX_DESC_CMD_EOP, + 0, desc->len, 0); + + *total_bytes += desc->len; +} + +/* This value should match the pragma below. Why 4? It is strictly + * empirical. It seems to be a good compromise between the advantage + * of having simultaneous outstanding reads to the DMA array that can + * hide each others latency and the disadvantage of having a larger + * code path. + */ +#define PKTS_PER_BATCH 4 + +static void i40e_xmit_pkt_batch(struct i40e_ring *xdp_ring, struct xdp_desc *desc, + unsigned int *total_bytes) +{ + u16 ntu = xdp_ring->next_to_use; + struct i40e_tx_desc *tx_desc; + dma_addr_t dma; + u32 i; + +#pragma GCC unroll 4 + for (i = 0; i < PKTS_PER_BATCH; i++) { + dma = xsk_buff_raw_get_dma(xdp_ring->xsk_pool, desc[i].addr); + xsk_buff_raw_dma_sync_for_device(xdp_ring->xsk_pool, dma, desc[i].len); + + tx_desc = I40E_TX_DESC(xdp_ring, ntu++); + tx_desc->buffer_addr = cpu_to_le64(dma); + tx_desc->cmd_type_offset_bsz = build_ctob(I40E_TX_DESC_CMD_ICRC | + I40E_TX_DESC_CMD_EOP, + 0, desc[i].len, 0); + + *total_bytes += desc[i].len; + } + + xdp_ring->next_to_use = ntu; +} + +static void i40e_fill_tx_hw_ring(struct i40e_ring *xdp_ring, struct xdp_desc *descs, u32 nb_pkts, + unsigned int *total_bytes) +{ + u32 batched, leftover, i; + + batched = nb_pkts & ~(PKTS_PER_BATCH - 1); + leftover = nb_pkts & (PKTS_PER_BATCH - 1); + for (i = 0; i < batched; i += PKTS_PER_BATCH) + i40e_xmit_pkt_batch(xdp_ring, &descs[i], total_bytes); + for (i = batched; i < batched + leftover; i++) + i40e_xmit_pkt(xdp_ring, &descs[i], total_bytes); +} + +static void i40e_set_rs_bit(struct i40e_ring *xdp_ring) +{ + u16 ntu = xdp_ring->next_to_use ? xdp_ring->next_to_use - 1 : xdp_ring->count - 1; + struct i40e_tx_desc *tx_desc; + + tx_desc = I40E_TX_DESC(xdp_ring, ntu); + tx_desc->cmd_type_offset_bsz |= (I40E_TX_DESC_CMD_RS << I40E_TXD_QW1_CMD_SHIFT); +} + /** * i40e_xmit_zc - Performs zero-copy Tx AF_XDP * @xdp_ring: XDP Tx ring @@ -390,45 +462,30 @@ int i40e_clean_rx_irq_zc(struct i40e_ring *rx_ring, int budget) **/ static bool i40e_xmit_zc(struct i40e_ring *xdp_ring, unsigned int budget) { - unsigned int sent_frames = 0, total_bytes = 0; - struct i40e_tx_desc *tx_desc = NULL; - struct xdp_desc desc; - dma_addr_t dma; - - while (budget-- > 0) { - if (!xsk_tx_peek_desc(xdp_ring->xsk_pool, &desc)) - break; - - dma = xsk_buff_raw_get_dma(xdp_ring->xsk_pool, desc.addr); - xsk_buff_raw_dma_sync_for_device(xdp_ring->xsk_pool, dma, - desc.len); - - tx_desc = I40E_TX_DESC(xdp_ring, xdp_ring->next_to_use); - tx_desc->buffer_addr = cpu_to_le64(dma); - tx_desc->cmd_type_offset_bsz = - build_ctob(I40E_TX_DESC_CMD_ICRC - | I40E_TX_DESC_CMD_EOP, - 0, desc.len, 0); - - sent_frames++; - total_bytes += desc.len; - - xdp_ring->next_to_use++; - if (xdp_ring->next_to_use == xdp_ring->count) - xdp_ring->next_to_use = 0; + struct xdp_desc *descs = xdp_ring->xsk_descs; + u32 nb_pkts, nb_processed = 0; + unsigned int total_bytes = 0; + + nb_pkts = xsk_tx_peek_release_desc_batch(xdp_ring->xsk_pool, descs, budget); + if (!nb_pkts) + return false; + + if (xdp_ring->next_to_use + nb_pkts >= xdp_ring->count) { + nb_processed = xdp_ring->count - xdp_ring->next_to_use; + i40e_fill_tx_hw_ring(xdp_ring, descs, nb_processed, &total_bytes); + xdp_ring->next_to_use = 0; } - if (tx_desc) { - /* Request an interrupt for the last frame and bump tail ptr. */ - tx_desc->cmd_type_offset_bsz |= (I40E_TX_DESC_CMD_RS << - I40E_TXD_QW1_CMD_SHIFT); - i40e_xdp_ring_update_tail(xdp_ring); + i40e_fill_tx_hw_ring(xdp_ring, &descs[nb_processed], nb_pkts - nb_processed, + &total_bytes); - xsk_tx_release(xdp_ring->xsk_pool); - i40e_update_tx_stats(xdp_ring, sent_frames, total_bytes); - } + /* Request an interrupt for the last frame and bump tail ptr. */ + i40e_set_rs_bit(xdp_ring); + i40e_xdp_ring_update_tail(xdp_ring); + + i40e_update_tx_stats(xdp_ring, nb_pkts, total_bytes); - return !!budget; + return true; } /**