From patchwork Tue Sep 29 20:25:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Gleixner X-Patchwork-Id: 259108 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 775ACC47420 for ; Tue, 29 Sep 2020 20:35:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 110172074B for ; Tue, 29 Sep 2020 20:35:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="rLTnlvYV"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="wvOjnrRL" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729254AbgI2Ufm (ORCPT ); Tue, 29 Sep 2020 16:35:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37170 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728301AbgI2Ufl (ORCPT ); Tue, 29 Sep 2020 16:35:41 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3CDBAC061755; Tue, 29 Sep 2020 13:35:41 -0700 (PDT) Message-Id: <20200929203459.515312515@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1601411739; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: references:references; bh=ixX5wWeAzYgeqCbxOmX1EtJ/d8bhHMpFSLlTnl3G8oc=; b=rLTnlvYVJ44sP81i5crg2IdrMuAcGvR1WnDMWATS/f0fv0RR8XmXxSpbTNVrghfSgyLCB9 7vY05femAXzcPxxt4YIlOmS0NJE78VJjIq9GFCX/1L6H8JXaur/EmNtEhY8tGE8kBAqwsG 9ajJnRCuEPM6nBfKtFJNtfUul8IolcgLvlWkXREeiNI4DmCvKoV/DezStYuNOHxEJQcDDo U2fEfjifRbL/wigBLP6EfSWnQL0tuqOtz0Z35yDlX2QuMTQWkHyUwjNZvQ6DnJgOliMIOM u60HRnW1blplXHXswcA2Ee9vOFV2Jy4eF8wRw77i3z0uJ+KbPzDKG13g78y9Lw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1601411739; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: references:references; bh=ixX5wWeAzYgeqCbxOmX1EtJ/d8bhHMpFSLlTnl3G8oc=; b=wvOjnrRLLr5mL3R/GZv5MV5UXnvPGrg6RJ3yWooCRQWGteFyFBIq2hmIbxbv4SN0ex5MxU JFYN02SkZqGUSyCA== Date: Tue, 29 Sep 2020 22:25:10 +0200 From: Thomas Gleixner To: LKML Cc: Peter Zijlstra , Paul McKenney , Matthew Wilcox , Christian Benvenuti , Govindarajulu Varadarajan <_govind@gmx.com>, Dave Miller , Jakub Kicinski , netdev@vger.kernel.org, Jonathan Corbet , Mauro Carvalho Chehab , linux-doc@vger.kernel.org, Sebastian Andrzej Siewior , Luc Van Oostenryck , Jay Cliburn , Chris Snook , Vishal Kulkarni , Jeff Kirsher , intel-wired-lan@lists.osuosl.org, Shannon Nelson , Pensando Drivers , Andrew Lunn , Heiner Kallweit , Russell King , Thomas Bogendoerfer , Solarflare linux maintainers , Edward Cree , Martin Habets , Jon Mason , Daniel Drake , Ulrich Kunitz , Kalle Valo , linux-wireless@vger.kernel.org, linux-usb@vger.kernel.org, Greg Kroah-Hartman , Arend van Spriel , Franky Lin , Hante Meuleman , Chi-Hsien Lin , Wright Feng , brcm80211-dev-list.pdl@broadcom.com, brcm80211-dev-list@cypress.com, Stanislav Yakovlev , Stanislaw Gruszka , Johannes Berg , Emmanuel Grumbach , Luca Coelho , Intel Linux Wireless , Jouni Malinen , Amitkumar Karwar , Ganapathi Bhat , Xinming Hu , libertas-dev@lists.infradead.org, Pascal Terjan , Ping-Ke Shih Subject: [patch V2 01/36] net: enic: Cure the enic api locking trainwreck References: <20200929202509.673358734@linutronix.de> MIME-Version: 1.0 Content-transfer-encoding: 8-bit Precedence: bulk List-ID: X-Mailing-List: linux-wireless@vger.kernel.org From: Thomas Gleixner enic_dev_wait() has a BUG_ON(in_interrupt()). Chasing the callers of enic_dev_wait() revealed the gems of enic_reset() and enic_tx_hang_reset() which are both invoked through work queues in order to be able to call rtnl_lock(). So far so good. After locking rtnl both functions acquire enic::enic_api_lock which serializes against the (ab)use from infiniband. This is where the trainwreck starts. enic::enic_api_lock is a spin_lock() which implicitly disables preemption, but both functions invoke a ton of functions under that lock which can sleep. The BUG_ON(in_interrupt()) does not trigger in that case because it can't detect the preempt disabled condition. This clearly has never been tested with any of the mandatory debug options for 7+ years, which would have caught that for sure. Cure it by adding a enic_api_busy member to struct enic, which is modified and evaluated with enic::enic_api_lock held. If enic_api_devcmd_proxy_by_index() observes enic::enic_api_busy as true, it drops enic::enic_api_lock and busy waits for enic::enic_api_busy to become false. It would be smarter to wait for a completion of that busy period, but enic_api_devcmd_proxy_by_index() is called with other spin locks held which obviously can't sleep. Remove the BUG_ON(in_interrupt()) check as well because it's incomplete and with proper debugging enabled the problem would have been caught from the debug checks in schedule_timeout(). Fixes: 0b038566c0ea ("drivers/net: enic: Add an interface for USNIC to interact with firmware") Signed-off-by: Thomas Gleixner --- V2: Cure tglx's last minute rename fail --- drivers/net/ethernet/cisco/enic/enic.h | 1 + drivers/net/ethernet/cisco/enic/enic_api.c | 6 ++++++ drivers/net/ethernet/cisco/enic/enic_main.c | 27 +++++++++++++++++++++------ 3 files changed, 28 insertions(+), 6 deletions(-) --- a/drivers/net/ethernet/cisco/enic/enic.h +++ b/drivers/net/ethernet/cisco/enic/enic.h @@ -169,6 +169,7 @@ struct enic { u16 num_vfs; #endif spinlock_t enic_api_lock; + bool enic_api_busy; struct enic_port_profile *pp; /* work queue cache line section */ --- a/drivers/net/ethernet/cisco/enic/enic_api.c +++ b/drivers/net/ethernet/cisco/enic/enic_api.c @@ -34,6 +34,12 @@ int enic_api_devcmd_proxy_by_index(struc struct vnic_dev *vdev = enic->vdev; spin_lock(&enic->enic_api_lock); + while (enic->enic_api_busy) { + spin_unlock(&enic->enic_api_lock); + cpu_relax(); + spin_lock(&enic->enic_api_lock); + } + spin_lock_bh(&enic->devcmd_lock); vnic_dev_cmd_proxy_by_index_start(vdev, vf); --- a/drivers/net/ethernet/cisco/enic/enic_main.c +++ b/drivers/net/ethernet/cisco/enic/enic_main.c @@ -2107,8 +2107,6 @@ static int enic_dev_wait(struct vnic_dev int done; int err; - BUG_ON(in_interrupt()); - err = start(vdev, arg); if (err) return err; @@ -2297,6 +2295,13 @@ static int enic_set_rss_nic_cfg(struct e rss_hash_bits, rss_base_cpu, rss_enable); } +static void enic_set_api_busy(struct enic *enic, bool busy) +{ + spin_lock(&enic->enic_api_lock); + enic->enic_api_busy = busy; + spin_unlock(&enic->enic_api_lock); +} + static void enic_reset(struct work_struct *work) { struct enic *enic = container_of(work, struct enic, reset); @@ -2306,7 +2311,9 @@ static void enic_reset(struct work_struc rtnl_lock(); - spin_lock(&enic->enic_api_lock); + /* Stop any activity from infiniband */ + enic_set_api_busy(enic, true); + enic_stop(enic->netdev); enic_dev_soft_reset(enic); enic_reset_addr_lists(enic); @@ -2314,7 +2321,10 @@ static void enic_reset(struct work_struc enic_set_rss_nic_cfg(enic); enic_dev_set_ig_vlan_rewrite_mode(enic); enic_open(enic->netdev); - spin_unlock(&enic->enic_api_lock); + + /* Allow infiniband to fiddle with the device again */ + enic_set_api_busy(enic, false); + call_netdevice_notifiers(NETDEV_REBOOT, enic->netdev); rtnl_unlock(); @@ -2326,7 +2336,9 @@ static void enic_tx_hang_reset(struct wo rtnl_lock(); - spin_lock(&enic->enic_api_lock); + /* Stop any activity from infiniband */ + enic_set_api_busy(enic, true); + enic_dev_hang_notify(enic); enic_stop(enic->netdev); enic_dev_hang_reset(enic); @@ -2335,7 +2347,10 @@ static void enic_tx_hang_reset(struct wo enic_set_rss_nic_cfg(enic); enic_dev_set_ig_vlan_rewrite_mode(enic); enic_open(enic->netdev); - spin_unlock(&enic->enic_api_lock); + + /* Allow infiniband to fiddle with the device again */ + enic_set_api_busy(enic, false); + call_netdevice_notifiers(NETDEV_REBOOT, enic->netdev); rtnl_unlock();