From patchwork Wed Sep 30 19:21:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wei Wang X-Patchwork-Id: 289270 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-21.1 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C183C4727C for ; Wed, 30 Sep 2020 19:22:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DA40E20709 for ; Wed, 30 Sep 2020 19:22:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="lXwbh02y" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729996AbgI3TW0 (ORCPT ); Wed, 30 Sep 2020 15:22:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50208 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725872AbgI3TWZ (ORCPT ); Wed, 30 Sep 2020 15:22:25 -0400 Received: from mail-qv1-xf49.google.com (mail-qv1-xf49.google.com [IPv6:2607:f8b0:4864:20::f49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 34FFAC061755 for ; Wed, 30 Sep 2020 12:22:24 -0700 (PDT) Received: by mail-qv1-xf49.google.com with SMTP id t4so1599807qvr.21 for ; Wed, 30 Sep 2020 12:22:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=F5/aG9+DIoo1m4DQiZli27OAJMKSrdWwepU13qAMpPU=; b=lXwbh02y2BRUcfXkv+cQP/LvMNaclm0DGo4aZ9WWKBUa9aB7uVrnKzkExRPgQWXSlQ fRnOPWDwqeIpuPmxAPRSBw6U7QrWDtC6ADBMzrLVLOIplmpYhREYC2VLc4M8srnFo38k ddaKKubUiUYqbxNuoJRKEpClrSHxykUM9KnwhDkxRODg599ldROMsyKgg8mW+R11b0Av 4Y7VCiDzcGTYiTF27aaCscYIK/FmVDMmJSARwaXpsicpnKoEOiOwRUQ0z/PqbOpuLNoN nKS5xkyN2o5hlC/BX//nn7TZNoii4smFuzXaWsvMTXaln6Xunj3ZYe+/B0nLMYuZQ1nb DbEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=F5/aG9+DIoo1m4DQiZli27OAJMKSrdWwepU13qAMpPU=; b=l18CDWpctaQ6YRztT/XQlanVXPu79A/8q7SJT616a8NRPYQNMm87cv3U6sZTzi1+w2 haYyIFlUqBrESEHtHmKL0b7m8WiiOc1mKEHVTvKkvAfzZ/5awZecX9wMkyFJyb4DdsbX Od4ysDLmc1ESsNHCuD8FbHV6595M1jsnmhi+0PA0MN7Ucfj3YZ/AExSwVC7aCpveoS3q ZKqEFsTarbpM8tWFQuMa/qSyxtyvbbLUQarGrVfJQSgxOpWDBd1fLrsksijN0Q3zctLp FzdgnqK+Qr2mWmJkcyznDiG210lNJGllwkcdEyRYOX+oONCjbhjDp12iSS9AZ0B4Qc4q lbwg== X-Gm-Message-State: AOAM530daZBD6Gc3cqtv8SSgTDy2OIvRglafkP7Zxhxwde2GzSQtdkEj LxKa5fxs+wAkrQug2R0QCuqbnSFUw+k= X-Google-Smtp-Source: ABdhPJznqTYwPsRhd+vEjV2fBlXseRe0HLOhficuLtLhK9HggBz9UIvf/pt8n0kJv96/Xe9mDeRRcHR15f0= Sender: "weiwan via sendgmr" X-Received: from weiwan.svl.corp.google.com ([2620:15c:2c4:201:1ea0:b8ff:fe75:cf08]) (user=weiwan job=sendgmr) by 2002:a0c:f687:: with SMTP id p7mr4040712qvn.15.1601493743373; Wed, 30 Sep 2020 12:22:23 -0700 (PDT) Date: Wed, 30 Sep 2020 12:21:36 -0700 In-Reply-To: <20200930192140.4192859-1-weiwan@google.com> Message-Id: <20200930192140.4192859-2-weiwan@google.com> Mime-Version: 1.0 References: <20200930192140.4192859-1-weiwan@google.com> X-Mailer: git-send-email 2.28.0.709.gb0816b6eb0-goog Subject: [PATCH net-next 1/5] net: implement threaded-able napi poll loop support From: Wei Wang To: "David S . Miller" , netdev@vger.kernel.org Cc: Eric Dumazet , Jakub Kicinski , Hannes Frederic Sowa , Paolo Abeni , Felix Fietkau , Wei Wang Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Paolo Abeni This patch allows running each napi poll loop inside its own kernel thread. The rx mode can be enabled per napi instance via the newly addded napi_set_threaded() api; the requested kthread will be created on demand and shut down on device stop. Once that threaded mode is enabled and the kthread is started, napi_schedule() will wake-up such thread instead of scheduling the softirq. The threaded poll loop behaves quite likely the net_rx_action, but it does not have to manipulate local irqs and uses an explicit scheduling point based on netdev_budget. Signed-off-by: Paolo Abeni Signed-off-by: Hannes Frederic Sowa Signed-off-by: Wei Wang --- include/linux/netdevice.h | 5 ++ net/core/dev.c | 113 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 118 insertions(+) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 28cfa53daf72..b3516e77371e 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -348,6 +348,7 @@ struct napi_struct { struct list_head dev_list; struct hlist_node napi_hash_node; unsigned int napi_id; + struct task_struct *thread; }; enum { @@ -358,6 +359,7 @@ enum { NAPI_STATE_LISTED, /* NAPI added to system lists */ NAPI_STATE_NO_BUSY_POLL,/* Do not add in napi_hash, no busy polling */ NAPI_STATE_IN_BUSY_POLL,/* sk_busy_loop() owns this NAPI */ + NAPI_STATE_THREADED, /* The poll is performed inside its own thread*/ }; enum { @@ -368,6 +370,7 @@ enum { NAPIF_STATE_LISTED = BIT(NAPI_STATE_LISTED), NAPIF_STATE_NO_BUSY_POLL = BIT(NAPI_STATE_NO_BUSY_POLL), NAPIF_STATE_IN_BUSY_POLL = BIT(NAPI_STATE_IN_BUSY_POLL), + NAPIF_STATE_THREADED = BIT(NAPI_STATE_THREADED), }; enum gro_result { @@ -489,6 +492,8 @@ static inline bool napi_complete(struct napi_struct *n) return napi_complete_done(n, 0); } +int napi_set_threaded(struct napi_struct *n, bool threded); + /** * napi_disable - prevent NAPI from scheduling * @n: NAPI context diff --git a/net/core/dev.c b/net/core/dev.c index 9d55bf5d1a65..259cd7f3434f 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -91,6 +91,7 @@ #include #include #include +#include #include #include #include @@ -1487,9 +1488,19 @@ void netdev_notify_peers(struct net_device *dev) } EXPORT_SYMBOL(netdev_notify_peers); +static int napi_threaded_poll(void *data); + +static void napi_thread_start(struct napi_struct *n) +{ + if (test_bit(NAPI_STATE_THREADED, &n->state) && !n->thread) + n->thread = kthread_create(napi_threaded_poll, n, "%s-%d", + n->dev->name, n->napi_id); +} + static int __dev_open(struct net_device *dev, struct netlink_ext_ack *extack) { const struct net_device_ops *ops = dev->netdev_ops; + struct napi_struct *n; int ret; ASSERT_RTNL(); @@ -1521,6 +1532,9 @@ static int __dev_open(struct net_device *dev, struct netlink_ext_ack *extack) if (!ret && ops->ndo_open) ret = ops->ndo_open(dev); + list_for_each_entry(n, &dev->napi_list, dev_list) + napi_thread_start(n); + netpoll_poll_enable(dev); if (ret) @@ -1566,6 +1580,14 @@ int dev_open(struct net_device *dev, struct netlink_ext_ack *extack) } EXPORT_SYMBOL(dev_open); +static void napi_thread_stop(struct napi_struct *n) +{ + if (!n->thread) + return; + kthread_stop(n->thread); + n->thread = NULL; +} + static void __dev_close_many(struct list_head *head) { struct net_device *dev; @@ -1594,6 +1616,7 @@ static void __dev_close_many(struct list_head *head) list_for_each_entry(dev, head, close_list) { const struct net_device_ops *ops = dev->netdev_ops; + struct napi_struct *n; /* * Call the device specific close. This cannot fail. @@ -1605,6 +1628,9 @@ static void __dev_close_many(struct list_head *head) if (ops->ndo_stop) ops->ndo_stop(dev); + list_for_each_entry(n, &dev->napi_list, dev_list) + napi_thread_stop(n); + dev->flags &= ~IFF_UP; netpoll_poll_enable(dev); } @@ -4241,6 +4267,11 @@ int gro_normal_batch __read_mostly = 8; static inline void ____napi_schedule(struct softnet_data *sd, struct napi_struct *napi) { + if (napi->thread) { + wake_up_process(napi->thread); + return; + } + list_add_tail(&napi->poll_list, &sd->poll_list); __raise_softirq_irqoff(NET_RX_SOFTIRQ); } @@ -6654,6 +6685,30 @@ static void init_gro_hash(struct napi_struct *napi) napi->gro_bitmask = 0; } +int napi_set_threaded(struct napi_struct *n, bool threaded) +{ + ASSERT_RTNL(); + + if (n->dev->flags & IFF_UP) + return -EBUSY; + + if (threaded == !!test_bit(NAPI_STATE_THREADED, &n->state)) + return 0; + if (threaded) + set_bit(NAPI_STATE_THREADED, &n->state); + else + clear_bit(NAPI_STATE_THREADED, &n->state); + + /* if the device is initializing, nothing todo */ + if (test_bit(__LINK_STATE_START, &n->dev->state)) + return 0; + + napi_thread_stop(n); + napi_thread_start(n); + return 0; +} +EXPORT_SYMBOL(napi_set_threaded); + void netif_napi_add(struct net_device *dev, struct napi_struct *napi, int (*poll)(struct napi_struct *, int), int weight) { @@ -6794,6 +6849,64 @@ static int napi_poll(struct napi_struct *n, struct list_head *repoll) return work; } +static int napi_thread_wait(struct napi_struct *napi) +{ + set_current_state(TASK_INTERRUPTIBLE); + + while (!kthread_should_stop() && !napi_disable_pending(napi)) { + if (test_bit(NAPI_STATE_SCHED, &napi->state)) { + __set_current_state(TASK_RUNNING); + return 0; + } + + schedule(); + set_current_state(TASK_INTERRUPTIBLE); + } + __set_current_state(TASK_RUNNING); + return -1; +} + +static int napi_threaded_poll(void *data) +{ + struct napi_struct *napi = data; + + while (!napi_thread_wait(napi)) { + struct list_head dummy_repoll; + int budget = netdev_budget; + unsigned long time_limit; + bool again = true; + + INIT_LIST_HEAD(&dummy_repoll); + local_bh_disable(); + time_limit = jiffies + 2; + do { + /* ensure that the poll list is not empty */ + if (list_empty(&dummy_repoll)) + list_add(&napi->poll_list, &dummy_repoll); + + budget -= napi_poll(napi, &dummy_repoll); + if (unlikely(budget <= 0 || + time_after_eq(jiffies, time_limit))) { + cond_resched(); + + /* refresh the budget */ + budget = netdev_budget; + __kfree_skb_flush(); + time_limit = jiffies + 2; + } + + if (napi_disable_pending(napi)) + again = false; + else if (!test_bit(NAPI_STATE_SCHED, &napi->state)) + again = false; + } while (again); + + __kfree_skb_flush(); + local_bh_enable(); + } + return 0; +} + static __latent_entropy void net_rx_action(struct softirq_action *h) { struct softnet_data *sd = this_cpu_ptr(&softnet_data); From patchwork Wed Sep 30 19:21:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wei Wang X-Patchwork-Id: 259814 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-21.1 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 498D9C4727C for ; Wed, 30 Sep 2020 19:22:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E6B2C20709 for ; Wed, 30 Sep 2020 19:22:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="bGqrpYKx" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730056AbgI3TWa (ORCPT ); Wed, 30 Sep 2020 15:22:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50220 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725872AbgI3TW3 (ORCPT ); Wed, 30 Sep 2020 15:22:29 -0400 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B311DC061755 for ; Wed, 30 Sep 2020 12:22:29 -0700 (PDT) Received: by mail-pg1-x54a.google.com with SMTP id c26so1492764pgl.9 for ; Wed, 30 Sep 2020 12:22:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=2Fv/RrWoLcbM9P0duMxErv9yg1yz6MFdGnuq0GYXBxw=; b=bGqrpYKx4Jc85dJFg9s7r6LwTE4wK+xHdPMw5nhn6JvQkO95uy1HOLptRKtZyqNs/4 VVVMqVuXxnJBSGPUVvPKOHrLtbeutECHE52oHAsp1GJXqTc3WRF2/AFL7KWeYLPD1El6 /urin9faYJMfx5hnlqykz2ulraKaaJSCJ5u+slffb9aocxPNjBLsOOr2BwqMd/LCraFe C1eguknDpkjVlmAbux0EzKCBbPyOhaokxnbSpGERXHSXRCSpKfxZ7AaiVFTSDcRaAUTX rAI3CF2vb3328HZFRKT8buakfa54LERHre4LNHFtoD5AMBn0+xVa8e56j8RS4EdptwcE YevA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=2Fv/RrWoLcbM9P0duMxErv9yg1yz6MFdGnuq0GYXBxw=; b=AWX/eKCL/gbMSHyVld814GFy/0VVkSoCV4ijDaZIP+02QbQrbyMwM0jZ/0s/8Dc0ON yUt2Ssa94Zy8l2eLKYnoI3DoHe1bwX8ggLOgtQZoMRiMzFdSdnJp89k4mw7o9s8CE5NR nvgnatsQWIcaS8lTz2T5ypcuqKhzxvjMHkd1p62316rPGuARewN20m7yycz+PJ52tkBv HyQqTLh1TqzTcaAMkDwRM0dPBMVVsR0jSwlxd4pmjfVVQTwcqhVL/FyFzOQnrpW8nw+2 v5w5zvxUduj6V5MCWhn4BEHX05Ct6Niy29c050jxvq98oPTNYo9SOxy/jTvzsViW9V9N ro1g== X-Gm-Message-State: AOAM530coAegGQRWvsooMestmcISK0IvucnbdaikLlVdYgsgAcHyoNsL Xpnso739G/OGf4SQbukB8xoFa9W0VNM= X-Google-Smtp-Source: ABdhPJzeh5hXnILpNcKopyCOuPM/7zTUE1URK1dGFn/J+M6UOzvBzEuSjc7sdRzX/enu68NuEnYwspqfn5U= Sender: "weiwan via sendgmr" X-Received: from weiwan.svl.corp.google.com ([2620:15c:2c4:201:1ea0:b8ff:fe75:cf08]) (user=weiwan job=sendgmr) by 2002:a17:90a:940c:: with SMTP id r12mr9582pjo.1.1601493748282; Wed, 30 Sep 2020 12:22:28 -0700 (PDT) Date: Wed, 30 Sep 2020 12:21:37 -0700 In-Reply-To: <20200930192140.4192859-1-weiwan@google.com> Message-Id: <20200930192140.4192859-3-weiwan@google.com> Mime-Version: 1.0 References: <20200930192140.4192859-1-weiwan@google.com> X-Mailer: git-send-email 2.28.0.709.gb0816b6eb0-goog Subject: [PATCH net-next 2/5] net: add sysfs attribute to control napi threaded mode From: Wei Wang To: "David S . Miller" , netdev@vger.kernel.org Cc: Eric Dumazet , Jakub Kicinski , Hannes Frederic Sowa , Paolo Abeni , Felix Fietkau , Wei Wang Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Paolo Abeni this patch adds a new sysfs attribute to the network device class. Said attribute is a bitmask that allows controlling the threaded mode for all the napi instances of the given network device. The threaded mode can be switched only if related network device is down. Signed-off-by: Paolo Abeni Signed-off-by: Hannes Frederic Sowa Signed-off-by: Wei Wang --- net/core/net-sysfs.c | 103 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 103 insertions(+) diff --git a/net/core/net-sysfs.c b/net/core/net-sysfs.c index efec66fa78b7..fe81b344447d 100644 --- a/net/core/net-sysfs.c +++ b/net/core/net-sysfs.c @@ -538,6 +538,108 @@ static ssize_t phys_switch_id_show(struct device *dev, } static DEVICE_ATTR_RO(phys_switch_id); +static unsigned long *__alloc_thread_bitmap(struct net_device *netdev, + int *bits) +{ + struct napi_struct *n; + + *bits = 0; + list_for_each_entry(n, &netdev->napi_list, dev_list) + (*bits)++; + + return kmalloc_array(BITS_TO_LONGS(*bits), sizeof(unsigned long), + GFP_ATOMIC | __GFP_ZERO); +} + +static ssize_t threaded_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct net_device *netdev = to_net_dev(dev); + struct napi_struct *n; + unsigned long *bmap; + size_t count = 0; + int i, bits; + + if (!rtnl_trylock()) + return restart_syscall(); + + if (!dev_isalive(netdev)) + goto unlock; + + bmap = __alloc_thread_bitmap(netdev, &bits); + if (!bmap) { + count = -ENOMEM; + goto unlock; + } + + i = 0; + list_for_each_entry(n, &netdev->napi_list, dev_list) { + if (test_bit(NAPI_STATE_THREADED, &n->state)) + set_bit(i, bmap); + i++; + } + + count = bitmap_print_to_pagebuf(true, buf, bmap, bits); + kfree(bmap); + +unlock: + rtnl_unlock(); + + return count; +} + +static ssize_t threaded_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t len) +{ + struct net_device *netdev = to_net_dev(dev); + struct napi_struct *n; + unsigned long *bmap; + int i, bits; + size_t ret; + + if (!capable(CAP_NET_ADMIN)) + return -EPERM; + + if (!rtnl_trylock()) + return restart_syscall(); + + if (!dev_isalive(netdev)) { + ret = len; + goto unlock; + } + + if (netdev->flags & IFF_UP) { + ret = -EBUSY; + goto unlock; + } + + bmap = __alloc_thread_bitmap(netdev, &bits); + if (!bmap) { + ret = -ENOMEM; + goto unlock; + } + + ret = bitmap_parselist(buf, bmap, bits); + if (ret) + goto free_unlock; + + i = 0; + list_for_each_entry(n, &netdev->napi_list, dev_list) { + napi_set_threaded(n, test_bit(i, bmap)); + i++; + } + ret = len; + +free_unlock: + kfree(bmap); + +unlock: + rtnl_unlock(); + return ret; +} +static DEVICE_ATTR_RW(threaded); + static struct attribute *net_class_attrs[] __ro_after_init = { &dev_attr_netdev_group.attr, &dev_attr_type.attr, @@ -570,6 +672,7 @@ static struct attribute *net_class_attrs[] __ro_after_init = { &dev_attr_proto_down.attr, &dev_attr_carrier_up_count.attr, &dev_attr_carrier_down_count.attr, + &dev_attr_threaded.attr, NULL, }; ATTRIBUTE_GROUPS(net_class); From patchwork Wed Sep 30 19:21:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wei Wang X-Patchwork-Id: 289269 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-21.1 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75EE1C4727E for ; Wed, 30 Sep 2020 19:22:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2C70820719 for ; Wed, 30 Sep 2020 19:22:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="PeR3PmeL" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730074AbgI3TWe (ORCPT ); Wed, 30 Sep 2020 15:22:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50232 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725872AbgI3TWe (ORCPT ); Wed, 30 Sep 2020 15:22:34 -0400 Received: from mail-qt1-x849.google.com (mail-qt1-x849.google.com [IPv6:2607:f8b0:4864:20::849]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 59756C061755 for ; Wed, 30 Sep 2020 12:22:32 -0700 (PDT) Received: by mail-qt1-x849.google.com with SMTP id b18so1950371qto.4 for ; Wed, 30 Sep 2020 12:22:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=nxuoKptH0MeJm9Vqvc43LnSUpndmev/qZqbAMJlzA3Y=; b=PeR3PmeLSlw51GNzI58WUjBFaCLv9BpTA2Jwlgf68TB8MUv6vbRxSxgBfD/bBi5wJE rnmkDfWodkSt56r87gk6XLP82fQihaRY/dw7PNggBRAh3J0GrRqOZHlk2EbmNMjCvUXh ToR6J8kRlxFrBmtl4EJs7UiBXv6IjH4uMmWTnQ+EQzgNKFaKm33ZOmdvhQz/hrttpQN0 bcamAoHJBat2ZgM9+hTlAoEyKeoxouYKLm2+NkMxcbZ/Gm881cc6zwHduSNB1nFbEpCZ Cw/vkPmGIWZnaX27At6yorCk50tNwi0p5SJY8o0xsN/qxg9VkAuez8jW7TArfO6c06lb ltzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=nxuoKptH0MeJm9Vqvc43LnSUpndmev/qZqbAMJlzA3Y=; b=qhXhlohxbiMVhU0CKA5P6R/nCDBopY0QgH2S/Se/DbzH6ZYAEmpx7oMEwD8UggEgyi Tczr3iLNex0GS6spN6TMmvdO2EI3Fk5UAyhpTn81grDK3vCf7wcSCqiE6Gxyt5IivXgl fs0DoSL+6G/0zNuByAgqhbkBsmJG1/ItPLGh/g0rYHSzP6YLB/TZy0YIVz7dVRWBuoCU M4dTTeyGEZAUBALedQtI3cdisP4B8uKqt/rABKtNimgJZqsuaLBRBwqg0Hs6ax/JFOC8 iLumHZ9m/LhEG6iLfgxRIXe/ChGFURq8XTtHOm7R+9rXesD/HbFgyX/xZYUZElVIVlFS Em7A== X-Gm-Message-State: AOAM530hV6GrUURiC2WteIApI3ezytW9lEEUiQ/YmIqFEbQW+63ZDt1Y XSH+4gdqlihZAEvmbJJ79ApeBJEiBW0= X-Google-Smtp-Source: ABdhPJx7C/SaT8OPNRV5C9uc+cZAuZeTasU441768UWQZFd9ikIouk8Zd1UcPAIyOlIgf0MVV2JNwhizpzI= Sender: "weiwan via sendgmr" X-Received: from weiwan.svl.corp.google.com ([2620:15c:2c4:201:1ea0:b8ff:fe75:cf08]) (user=weiwan job=sendgmr) by 2002:a05:6214:a03:: with SMTP id dw3mr3722158qvb.44.1601493751484; Wed, 30 Sep 2020 12:22:31 -0700 (PDT) Date: Wed, 30 Sep 2020 12:21:38 -0700 In-Reply-To: <20200930192140.4192859-1-weiwan@google.com> Message-Id: <20200930192140.4192859-4-weiwan@google.com> Mime-Version: 1.0 References: <20200930192140.4192859-1-weiwan@google.com> X-Mailer: git-send-email 2.28.0.709.gb0816b6eb0-goog Subject: [PATCH net-next 3/5] net: extract napi poll functionality to __napi_poll() From: Wei Wang To: "David S . Miller" , netdev@vger.kernel.org Cc: Eric Dumazet , Jakub Kicinski , Hannes Frederic Sowa , Paolo Abeni , Felix Fietkau , Wei Wang Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Felix Fietkau This commit introduces a new function __napi_poll() which does the main logic of the existing napi_poll() function, and will be called by other functions in later commits. This idea and implementation is done by Felix Fietkau and is proposed as part of the patch to move napi work to work_queue context. This commit by itself is a code restructure. Signed-off-by: Felix Fietkau Signed-off-by: Wei Wang --- net/core/dev.c | 31 +++++++++++++++++++++++-------- 1 file changed, 23 insertions(+), 8 deletions(-) diff --git a/net/core/dev.c b/net/core/dev.c index 259cd7f3434f..c82522262ca8 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -6783,15 +6783,10 @@ void __netif_napi_del(struct napi_struct *napi) } EXPORT_SYMBOL(__netif_napi_del); -static int napi_poll(struct napi_struct *n, struct list_head *repoll) +static int __napi_poll(struct napi_struct *n, bool *repoll) { - void *have; int work, weight; - list_del_init(&n->poll_list); - - have = netpoll_poll_lock(n); - weight = n->weight; /* This NAPI_STATE_SCHED test is for avoiding a race @@ -6811,7 +6806,7 @@ static int napi_poll(struct napi_struct *n, struct list_head *repoll) n->poll, work, weight); if (likely(work < weight)) - goto out_unlock; + return work; /* Drivers must not modify the NAPI state if they * consume the entire weight. In such cases this code @@ -6820,7 +6815,7 @@ static int napi_poll(struct napi_struct *n, struct list_head *repoll) */ if (unlikely(napi_disable_pending(n))) { napi_complete(n); - goto out_unlock; + return work; } if (n->gro_bitmask) { @@ -6832,6 +6827,26 @@ static int napi_poll(struct napi_struct *n, struct list_head *repoll) gro_normal_list(n); + *repoll = true; + + return work; +} + +static int napi_poll(struct napi_struct *n, struct list_head *repoll) +{ + bool do_repoll = false; + void *have; + int work; + + list_del_init(&n->poll_list); + + have = netpoll_poll_lock(n); + + work = __napi_poll(n, &do_repoll); + + if (!do_repoll) + goto out_unlock; + /* Some drivers may have called napi_schedule * prior to exhausting their budget. */ From patchwork Wed Sep 30 19:21:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wei Wang X-Patchwork-Id: 259813 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-21.1 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D269C4727C for ; Wed, 30 Sep 2020 19:22:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 064BA2071E for ; Wed, 30 Sep 2020 19:22:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Y2I3C0RC" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730117AbgI3TWh (ORCPT ); Wed, 30 Sep 2020 15:22:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50244 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725872AbgI3TWh (ORCPT ); Wed, 30 Sep 2020 15:22:37 -0400 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E5569C061755 for ; Wed, 30 Sep 2020 12:22:36 -0700 (PDT) Received: by mail-pl1-x64a.google.com with SMTP id d20so1519936plr.18 for ; Wed, 30 Sep 2020 12:22:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=q0P1tSMlNAWIV7zQIDv2HdEMyoD3KRTuWxJZUM97nVU=; b=Y2I3C0RCK6Hx5PXS6nbwME5Pvk1+4meeGGrm/v21y0zr4shYRlAZOsphdLF8wNwg9l N5a/lQ6DAko9cFZTCHzfuSeD638v3BV7zSmha7G7xwyr33divE0ZyAVcZZRuWihjXpXX BpX3/+4dmLzxjjvHBVOeUPYDKPfXFzVl3jTdOxOc/muAi6YKXxsllNUXlwZD4VeW82J4 cFCSpiwpRlsl7ZW94qiaxqBDzvGUyWVwuO1oBegX0q4ur6x/Fizkux9jU990LdRtepLq Zxn71aX+PKh9+tSB4aaLUxeuhyQgKS6MKv/lev5P69NqYPW/1slpUBfBlHViXbS1Zo8X 4vBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=q0P1tSMlNAWIV7zQIDv2HdEMyoD3KRTuWxJZUM97nVU=; b=QcR6EeNWAd2vM41VPYIL8uiQmlOBgw/WjpzQoecavAj2hUb3Biz5CkSVON9nQg7sYX NltkFobB9YZgOyxIXKzTuEuLfbCGdp4fy7pUBwBmgwWS/AN9B8rRiLuomsAZRelzkBvM klVsqfxeIObIIK8SdmlUQogoa9NH6gwxpQdki+2MRgldDZU0CBRtuhp1T9LbdTXD7g7z aRyof7y2hFP4+/HkE9xc23Lp1gOCQODDFmbxrcYAoQE/PsOXOrXnMig5Ku5p0HtR9rZY bTMK+t93G22AwZ/3AvLSbqJpRljLabK9o90pWzrHXG38eS4J669jo+PUfOeqzR+3CXtK uYbw== X-Gm-Message-State: AOAM533iK7jbwlJkhfJFHXTW9Dp7Cl3CSiRVWLTAHJjR5dO87ILEpz0p thWwRY8HcFxQKgEcqnGvlSX0quJVKhg= X-Google-Smtp-Source: ABdhPJzsjnyILZ8+Xe9eHao7/1qdUQkUum8OCbGs+2flAKvowDAZdkEI876QVRQXgB0rRm24MDSwVDvfs4A= Sender: "weiwan via sendgmr" X-Received: from weiwan.svl.corp.google.com ([2620:15c:2c4:201:1ea0:b8ff:fe75:cf08]) (user=weiwan job=sendgmr) by 2002:a65:5185:: with SMTP id h5mr3325872pgq.37.1601493756340; Wed, 30 Sep 2020 12:22:36 -0700 (PDT) Date: Wed, 30 Sep 2020 12:21:39 -0700 In-Reply-To: <20200930192140.4192859-1-weiwan@google.com> Message-Id: <20200930192140.4192859-5-weiwan@google.com> Mime-Version: 1.0 References: <20200930192140.4192859-1-weiwan@google.com> X-Mailer: git-send-email 2.28.0.709.gb0816b6eb0-goog Subject: [PATCH net-next 4/5] net: modify kthread handler to use __napi_poll() From: Wei Wang To: "David S . Miller" , netdev@vger.kernel.org Cc: Eric Dumazet , Jakub Kicinski , Hannes Frederic Sowa , Paolo Abeni , Felix Fietkau , Wei Wang Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Jakub Kicinski The current kthread handler calls napi_poll() and has to pass a dummy repoll list to the function, which seems redundent. The new proposed kthread handler calls the newly proposed __napi_poll(), and respects napi->weight as before. If repoll is needed, cond_resched() is called first to give other tasks a chance to run before repolling. This change is proposed by Jakub Kicinski on top of the previous patch. Signed-off-by: Jakub Kicinski Signed-off-by: Wei Wang --- net/core/dev.c | 62 +++++++++++++++++++------------------------------- 1 file changed, 24 insertions(+), 38 deletions(-) diff --git a/net/core/dev.c b/net/core/dev.c index c82522262ca8..b4f33e442b5e 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -6827,6 +6827,15 @@ static int __napi_poll(struct napi_struct *n, bool *repoll) gro_normal_list(n); + /* Some drivers may have called napi_schedule + * prior to exhausting their budget. + */ + if (unlikely(!list_empty(&n->poll_list))) { + pr_warn_once("%s: Budget exhausted after napi rescheduled\n", + n->dev ? n->dev->name : "backlog"); + return work; + } + *repoll = true; return work; @@ -6847,15 +6856,6 @@ static int napi_poll(struct napi_struct *n, struct list_head *repoll) if (!do_repoll) goto out_unlock; - /* Some drivers may have called napi_schedule - * prior to exhausting their budget. - */ - if (unlikely(!list_empty(&n->poll_list))) { - pr_warn_once("%s: Budget exhausted after napi rescheduled\n", - n->dev ? n->dev->name : "backlog"); - goto out_unlock; - } - list_add_tail(&n->poll_list, repoll); out_unlock: @@ -6884,40 +6884,26 @@ static int napi_thread_wait(struct napi_struct *napi) static int napi_threaded_poll(void *data) { struct napi_struct *napi = data; + void *have; while (!napi_thread_wait(napi)) { - struct list_head dummy_repoll; - int budget = netdev_budget; - unsigned long time_limit; - bool again = true; + for (;;) { + bool repoll = false; - INIT_LIST_HEAD(&dummy_repoll); - local_bh_disable(); - time_limit = jiffies + 2; - do { - /* ensure that the poll list is not empty */ - if (list_empty(&dummy_repoll)) - list_add(&napi->poll_list, &dummy_repoll); - - budget -= napi_poll(napi, &dummy_repoll); - if (unlikely(budget <= 0 || - time_after_eq(jiffies, time_limit))) { - cond_resched(); - - /* refresh the budget */ - budget = netdev_budget; - __kfree_skb_flush(); - time_limit = jiffies + 2; - } + local_bh_disable(); - if (napi_disable_pending(napi)) - again = false; - else if (!test_bit(NAPI_STATE_SCHED, &napi->state)) - again = false; - } while (again); + have = netpoll_poll_lock(napi); + __napi_poll(napi, &repoll); + netpoll_poll_unlock(have); - __kfree_skb_flush(); - local_bh_enable(); + __kfree_skb_flush(); + local_bh_enable(); + + if (!repoll) + break; + + cond_resched(); + } } return 0; } From patchwork Wed Sep 30 19:21:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wei Wang X-Patchwork-Id: 289268 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-21.1 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E640C4727C for ; Wed, 30 Sep 2020 19:22:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 257D420709 for ; Wed, 30 Sep 2020 19:22:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="i7lsK89O" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730186AbgI3TWk (ORCPT ); Wed, 30 Sep 2020 15:22:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50252 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725872AbgI3TWj (ORCPT ); Wed, 30 Sep 2020 15:22:39 -0400 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BBD6BC061755 for ; Wed, 30 Sep 2020 12:22:39 -0700 (PDT) Received: by mail-pg1-x549.google.com with SMTP id c3so1497293pgj.5 for ; Wed, 30 Sep 2020 12:22:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=n4uTXHq/VXo3p/L7IiyvP0BEIy/QSB4yopeNBchJpbc=; b=i7lsK89OH5r95QLDz5rYzro50jPEcdrdYbXQvOV6qGAIEIUJ5h5ZBS19FEG3Nc34Fi a584nr63at+T5B1MkX3bsPrhT4EsRsTLfOwBYkN0XpV6x1F/67GRw9jvEs+NhEB7IEDC JzjVLg0P8pqZ4ieiz0tC8sUlNK8rQb0ZtKV3xRa7R/JK4Lh1f/CygQik1I33DlnFaNg/ OMGQy0Sg1of+MhtC77xyrZWploPpAUGXdlYtLnlGSjn3O+PhW3YfWLYFXmzRVWl/iKQ/ 6bRk9sjr4SDzZQ3m5xT2H/8yM8AwHINKt0Kxhp+HuzrGaRgZudHuP4j9E+zYWJV61HUj H6lg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=n4uTXHq/VXo3p/L7IiyvP0BEIy/QSB4yopeNBchJpbc=; b=Ml8sKSrQxsD56XZXNu+k3M83+Nf79EPDrgx8B3VF3OFgStXZsr0hcBWwvI2VH8d+av qcRXGXIz98EA0TJYkm2NOcxb7hJ2fCyAymYd/R/xz2GzRZtB5LTxGAKpQAVcWAOnRTeQ uwWtAdn6Tvu3DUC+5fRyr0ECJOcSJoEvNdXx0tT2jYlkM4erdkHgGwfDjJrJJVdPPpOl MKcRudQzh4eIrlVN+OgeTK2gbzo0q2cYqwlULUgzFrKHdqF2drVs0IJuwyf5XFcq695w iye6I/HujuWYnT7+6XcoNq5qHrBOHqJy48C9SvDqKNacFazKJp5+F0KEt3OC5DnCtUlj 6pwg== X-Gm-Message-State: AOAM531lCid9VnPhUUhdt235KoLyDI2MLzPkVRLnWe15LjM/rGV4Ho6o vLofs17tpMOLpzoIm0gEa9+SfWRycDQ= X-Google-Smtp-Source: ABdhPJw8wC8pvvboZWM66xoE8mrarejzvWPhRhJL5X3clX7INctokupYTBPu7AxCv5Fa1/DrA+gtCYmbA+w= Sender: "weiwan via sendgmr" X-Received: from weiwan.svl.corp.google.com ([2620:15c:2c4:201:1ea0:b8ff:fe75:cf08]) (user=weiwan job=sendgmr) by 2002:a17:902:ba8c:b029:d2:ab25:5425 with SMTP id k12-20020a170902ba8cb02900d2ab255425mr4174932pls.0.1601493759117; Wed, 30 Sep 2020 12:22:39 -0700 (PDT) Date: Wed, 30 Sep 2020 12:21:40 -0700 In-Reply-To: <20200930192140.4192859-1-weiwan@google.com> Message-Id: <20200930192140.4192859-6-weiwan@google.com> Mime-Version: 1.0 References: <20200930192140.4192859-1-weiwan@google.com> X-Mailer: git-send-email 2.28.0.709.gb0816b6eb0-goog Subject: [PATCH net-next 5/5] net: improve napi threaded config From: Wei Wang To: "David S . Miller" , netdev@vger.kernel.org Cc: Eric Dumazet , Jakub Kicinski , Hannes Frederic Sowa , Paolo Abeni , Felix Fietkau , Wei Wang Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org This commit mainly addresses the threaded config to make the switch between softirq based and kthread based NAPI processing not require a device down/up. It also moves the kthread_create() call to the sysfs handler when user tries to enable "threaded" on napi, and properly handles the kthread_create() failure. This is because certain drivers do not have the napi created and linked to the dev when dev_open() is called. So the previous implementation does not work properly there. Signed-off-by: Wei Wang --- Changes since RFC: changed the thread name to napi/- net/core/dev.c | 49 +++++++++++++++++++++++++------------------- net/core/net-sysfs.c | 9 +++----- 2 files changed, 31 insertions(+), 27 deletions(-) diff --git a/net/core/dev.c b/net/core/dev.c index b4f33e442b5e..bf878d3a9d89 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -1490,17 +1490,24 @@ EXPORT_SYMBOL(netdev_notify_peers); static int napi_threaded_poll(void *data); -static void napi_thread_start(struct napi_struct *n) +static int napi_kthread_create(struct napi_struct *n) { - if (test_bit(NAPI_STATE_THREADED, &n->state) && !n->thread) - n->thread = kthread_create(napi_threaded_poll, n, "%s-%d", - n->dev->name, n->napi_id); + int err = 0; + + n->thread = kthread_create(napi_threaded_poll, n, "napi/%s-%d", + n->dev->name, n->napi_id); + if (IS_ERR(n->thread)) { + err = PTR_ERR(n->thread); + pr_err("kthread_create failed with err %d\n", err); + n->thread = NULL; + } + + return err; } static int __dev_open(struct net_device *dev, struct netlink_ext_ack *extack) { const struct net_device_ops *ops = dev->netdev_ops; - struct napi_struct *n; int ret; ASSERT_RTNL(); @@ -1532,9 +1539,6 @@ static int __dev_open(struct net_device *dev, struct netlink_ext_ack *extack) if (!ret && ops->ndo_open) ret = ops->ndo_open(dev); - list_for_each_entry(n, &dev->napi_list, dev_list) - napi_thread_start(n); - netpoll_poll_enable(dev); if (ret) @@ -1585,6 +1589,7 @@ static void napi_thread_stop(struct napi_struct *n) if (!n->thread) return; kthread_stop(n->thread); + clear_bit(NAPI_STATE_THREADED, &n->state); n->thread = NULL; } @@ -4267,7 +4272,7 @@ int gro_normal_batch __read_mostly = 8; static inline void ____napi_schedule(struct softnet_data *sd, struct napi_struct *napi) { - if (napi->thread) { + if (test_bit(NAPI_STATE_THREADED, &napi->state)) { wake_up_process(napi->thread); return; } @@ -6687,25 +6692,25 @@ static void init_gro_hash(struct napi_struct *napi) int napi_set_threaded(struct napi_struct *n, bool threaded) { - ASSERT_RTNL(); + int err = 0; - if (n->dev->flags & IFF_UP) - return -EBUSY; + ASSERT_RTNL(); if (threaded == !!test_bit(NAPI_STATE_THREADED, &n->state)) return 0; - if (threaded) + if (threaded) { + if (!n->thread) { + err = napi_kthread_create(n); + if (err) + goto out; + } set_bit(NAPI_STATE_THREADED, &n->state); - else + } else { clear_bit(NAPI_STATE_THREADED, &n->state); + } - /* if the device is initializing, nothing todo */ - if (test_bit(__LINK_STATE_START, &n->dev->state)) - return 0; - - napi_thread_stop(n); - napi_thread_start(n); - return 0; +out: + return err; } EXPORT_SYMBOL(napi_set_threaded); @@ -6750,6 +6755,7 @@ void napi_disable(struct napi_struct *n) msleep(1); hrtimer_cancel(&n->timer); + napi_thread_stop(n); clear_bit(NAPI_STATE_DISABLE, &n->state); } @@ -6870,6 +6876,7 @@ static int napi_thread_wait(struct napi_struct *napi) while (!kthread_should_stop() && !napi_disable_pending(napi)) { if (test_bit(NAPI_STATE_SCHED, &napi->state)) { + WARN_ON(!list_empty(&napi->poll_list)); __set_current_state(TASK_RUNNING); return 0; } diff --git a/net/core/net-sysfs.c b/net/core/net-sysfs.c index fe81b344447d..b54dbccf00be 100644 --- a/net/core/net-sysfs.c +++ b/net/core/net-sysfs.c @@ -609,11 +609,6 @@ static ssize_t threaded_store(struct device *dev, goto unlock; } - if (netdev->flags & IFF_UP) { - ret = -EBUSY; - goto unlock; - } - bmap = __alloc_thread_bitmap(netdev, &bits); if (!bmap) { ret = -ENOMEM; @@ -626,7 +621,9 @@ static ssize_t threaded_store(struct device *dev, i = 0; list_for_each_entry(n, &netdev->napi_list, dev_list) { - napi_set_threaded(n, test_bit(i, bmap)); + ret = napi_set_threaded(n, test_bit(i, bmap)); + if (ret) + goto free_unlock; i++; } ret = len;