From patchwork Tue Oct 27 13:48:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Greg KH X-Patchwork-Id: 311780 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57836C4363A for ; Tue, 27 Oct 2020 18:15:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0A679207E8 for ; Tue, 27 Oct 2020 18:15:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1603822534; bh=tMGdJPeKR2DaxUyCWy/06Bye8wISs0Lr7Y//Ta9fOnQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=iF2ftLmYKR6SHdGYYGHLY41Zeki23D5c48GhXe7FK93YPBoFe2sJiwPzEaO+zBiA3 DKFDJWssGZoG5KV1+WkaJVsw4HX1n5hyj1u/OQJsIEYK6MQwOKcbbRtCobseB3B8vv X4IsljlGHdwMSa2aQRKRrXzJVU//9p/tYDk+O4Kk= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756333AbgJ0OMl (ORCPT ); Tue, 27 Oct 2020 10:12:41 -0400 Received: from mail.kernel.org ([198.145.29.99]:33914 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2507638AbgJ0OMk (ORCPT ); Tue, 27 Oct 2020 10:12:40 -0400 Received: from localhost (83-86-74-64.cable.dynamic.v4.ziggo.nl [83.86.74.64]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id E66BA2072D; Tue, 27 Oct 2020 14:12:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1603807959; bh=tMGdJPeKR2DaxUyCWy/06Bye8wISs0Lr7Y//Ta9fOnQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RDzXBfv5l0LHNe4uGSfr+u/v7LHzYLgfsSx99oiSo1vsmkpOR+X0YHGh9i4tXjc94 UUmLZ9BZxbZaH4AeNUwOujsrArEcxxuFSzTEB6GDUQ5aFfEbUMXy+Of2X4Fd5pw8gH fiEJ9C5rrRBPKXwv/llvvKN5LaJuTUqyqc5vVZ1Y= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Thomas Gleixner , "David S. Miller" , Sasha Levin Subject: [PATCH 4.14 073/191] net: enic: Cure the enic api locking trainwreck Date: Tue, 27 Oct 2020 14:48:48 +0100 Message-Id: <20201027134913.233157361@linuxfoundation.org> X-Mailer: git-send-email 2.29.1 In-Reply-To: <20201027134909.701581493@linuxfoundation.org> References: <20201027134909.701581493@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org From: Thomas Gleixner [ Upstream commit a53b59ece86c86d16d12ccdaa1ad0c78250a9d96 ] enic_dev_wait() has a BUG_ON(in_interrupt()). Chasing the callers of enic_dev_wait() revealed the gems of enic_reset() and enic_tx_hang_reset() which are both invoked through work queues in order to be able to call rtnl_lock(). So far so good. After locking rtnl both functions acquire enic::enic_api_lock which serializes against the (ab)use from infiniband. This is where the trainwreck starts. enic::enic_api_lock is a spin_lock() which implicitly disables preemption, but both functions invoke a ton of functions under that lock which can sleep. The BUG_ON(in_interrupt()) does not trigger in that case because it can't detect the preempt disabled condition. This clearly has never been tested with any of the mandatory debug options for 7+ years, which would have caught that for sure. Cure it by adding a enic_api_busy member to struct enic, which is modified and evaluated with enic::enic_api_lock held. If enic_api_devcmd_proxy_by_index() observes enic::enic_api_busy as true, it drops enic::enic_api_lock and busy waits for enic::enic_api_busy to become false. It would be smarter to wait for a completion of that busy period, but enic_api_devcmd_proxy_by_index() is called with other spin locks held which obviously can't sleep. Remove the BUG_ON(in_interrupt()) check as well because it's incomplete and with proper debugging enabled the problem would have been caught from the debug checks in schedule_timeout(). Fixes: 0b038566c0ea ("drivers/net: enic: Add an interface for USNIC to interact with firmware") Signed-off-by: Thomas Gleixner Signed-off-by: David S. Miller Signed-off-by: Sasha Levin --- drivers/net/ethernet/cisco/enic/enic.h | 1 + drivers/net/ethernet/cisco/enic/enic_api.c | 6 +++++ drivers/net/ethernet/cisco/enic/enic_main.c | 27 ++++++++++++++++----- 3 files changed, 28 insertions(+), 6 deletions(-) diff --git a/drivers/net/ethernet/cisco/enic/enic.h b/drivers/net/ethernet/cisco/enic/enic.h index ba032ac9ae86c..893c51a94abbd 100644 --- a/drivers/net/ethernet/cisco/enic/enic.h +++ b/drivers/net/ethernet/cisco/enic/enic.h @@ -168,6 +168,7 @@ struct enic { u16 num_vfs; #endif spinlock_t enic_api_lock; + bool enic_api_busy; struct enic_port_profile *pp; /* work queue cache line section */ diff --git a/drivers/net/ethernet/cisco/enic/enic_api.c b/drivers/net/ethernet/cisco/enic/enic_api.c index b161f24522b87..b028ea2dec2b9 100644 --- a/drivers/net/ethernet/cisco/enic/enic_api.c +++ b/drivers/net/ethernet/cisco/enic/enic_api.c @@ -34,6 +34,12 @@ int enic_api_devcmd_proxy_by_index(struct net_device *netdev, int vf, struct vnic_dev *vdev = enic->vdev; spin_lock(&enic->enic_api_lock); + while (enic->enic_api_busy) { + spin_unlock(&enic->enic_api_lock); + cpu_relax(); + spin_lock(&enic->enic_api_lock); + } + spin_lock_bh(&enic->devcmd_lock); vnic_dev_cmd_proxy_by_index_start(vdev, vf); diff --git a/drivers/net/ethernet/cisco/enic/enic_main.c b/drivers/net/ethernet/cisco/enic/enic_main.c index 52a3b32390a9c..f0bbc0fdeddcb 100644 --- a/drivers/net/ethernet/cisco/enic/enic_main.c +++ b/drivers/net/ethernet/cisco/enic/enic_main.c @@ -2101,8 +2101,6 @@ static int enic_dev_wait(struct vnic_dev *vdev, int done; int err; - BUG_ON(in_interrupt()); - err = start(vdev, arg); if (err) return err; @@ -2279,6 +2277,13 @@ static int enic_set_rss_nic_cfg(struct enic *enic) rss_hash_bits, rss_base_cpu, rss_enable); } +static void enic_set_api_busy(struct enic *enic, bool busy) +{ + spin_lock(&enic->enic_api_lock); + enic->enic_api_busy = busy; + spin_unlock(&enic->enic_api_lock); +} + static void enic_reset(struct work_struct *work) { struct enic *enic = container_of(work, struct enic, reset); @@ -2288,7 +2293,9 @@ static void enic_reset(struct work_struct *work) rtnl_lock(); - spin_lock(&enic->enic_api_lock); + /* Stop any activity from infiniband */ + enic_set_api_busy(enic, true); + enic_stop(enic->netdev); enic_dev_soft_reset(enic); enic_reset_addr_lists(enic); @@ -2296,7 +2303,10 @@ static void enic_reset(struct work_struct *work) enic_set_rss_nic_cfg(enic); enic_dev_set_ig_vlan_rewrite_mode(enic); enic_open(enic->netdev); - spin_unlock(&enic->enic_api_lock); + + /* Allow infiniband to fiddle with the device again */ + enic_set_api_busy(enic, false); + call_netdevice_notifiers(NETDEV_REBOOT, enic->netdev); rtnl_unlock(); @@ -2308,7 +2318,9 @@ static void enic_tx_hang_reset(struct work_struct *work) rtnl_lock(); - spin_lock(&enic->enic_api_lock); + /* Stop any activity from infiniband */ + enic_set_api_busy(enic, true); + enic_dev_hang_notify(enic); enic_stop(enic->netdev); enic_dev_hang_reset(enic); @@ -2317,7 +2329,10 @@ static void enic_tx_hang_reset(struct work_struct *work) enic_set_rss_nic_cfg(enic); enic_dev_set_ig_vlan_rewrite_mode(enic); enic_open(enic->netdev); - spin_unlock(&enic->enic_api_lock); + + /* Allow infiniband to fiddle with the device again */ + enic_set_api_busy(enic, false); + call_netdevice_notifiers(NETDEV_REBOOT, enic->netdev); rtnl_unlock();