From patchwork Fri Nov 20 18:06:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Nguyen X-Patchwork-Id: 329713 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2387C56201 for ; Fri, 20 Nov 2020 18:07:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 84F822242B for ; Fri, 20 Nov 2020 18:07:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729630AbgKTSHG (ORCPT ); Fri, 20 Nov 2020 13:07:06 -0500 Received: from mga05.intel.com ([192.55.52.43]:16723 "EHLO mga05.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728986AbgKTSHG (ORCPT ); Fri, 20 Nov 2020 13:07:06 -0500 IronPort-SDR: Nuf2RLK0QroiCbZDaIhTDkdNGmdm3qYyzOsPC0ujb4nRJTkr1aKntXbOh1UsPjfpVcLTYPjJkq KixmNMWQ0XNQ== X-IronPort-AV: E=McAfee;i="6000,8403,9811"; a="256226259" X-IronPort-AV: E=Sophos;i="5.78,357,1599548400"; d="scan'208";a="256226259" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Nov 2020 10:07:05 -0800 IronPort-SDR: tKb8etISTp6G3sE1YGUEQowUxGgktn+s03ZWoT851lGGlX1/6L4nORi14y9l1gAiWGVfAR73xc 2IRwZlZEWX4A== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.78,357,1599548400"; d="scan'208";a="535259905" Received: from anguy11-desk2.jf.intel.com ([10.166.244.147]) by fmsmga005.fm.intel.com with ESMTP; 20 Nov 2020 10:07:04 -0800 From: Tony Nguyen To: davem@davemloft.net, kuba@kernel.org Cc: Sylwester Dziedziuch , netdev@vger.kernel.org, sassmann@redhat.com, anthony.l.nguyen@intel.com, Slawomir Laba , Brett Creeley , Konrad Jankowski Subject: [PATCH net 1/1] i40e: Fix removing driver while bare-metal VFs pass traffic Date: Fri, 20 Nov 2020 10:06:40 -0800 Message-Id: <20201120180640.3654474-1-anthony.l.nguyen@intel.com> X-Mailer: git-send-email 2.26.2 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Sylwester Dziedziuch Prevent VFs from resetting when PF driver is being unloaded: - introduce new pf state: __I40E_VF_RESETS_DISABLED; - check if pf state has __I40E_VF_RESETS_DISABLED state set, if so, disable any further VFLR event notifications; - when i40e_remove (rmmod i40e) is called, disable any resets on the VFs; Previously if there were bare-metal VFs passing traffic and PF driver was removed, there was a possibility of VFs triggering a Tx timeout right before iavf_remove. This was causing iavf_close to not be called because there is a check in the beginning of iavf_remove that bails out early if adapter->state < IAVF_DOWN_PENDING. This makes it so some resources do not get cleaned up. Fixes: 6a9ddb36eeb8 ("i40e: disable IOV before freeing resources") Signed-off-by: Slawomir Laba Signed-off-by: Brett Creeley Signed-off-by: Sylwester Dziedziuch Tested-by: Konrad Jankowski Signed-off-by: Tony Nguyen --- drivers/net/ethernet/intel/i40e/i40e.h | 1 + drivers/net/ethernet/intel/i40e/i40e_main.c | 22 +++++++++++----- .../ethernet/intel/i40e/i40e_virtchnl_pf.c | 26 +++++++++++-------- 3 files changed, 31 insertions(+), 18 deletions(-) diff --git a/drivers/net/ethernet/intel/i40e/i40e.h b/drivers/net/ethernet/intel/i40e/i40e.h index 537300e762f0..d231a2cdd98f 100644 --- a/drivers/net/ethernet/intel/i40e/i40e.h +++ b/drivers/net/ethernet/intel/i40e/i40e.h @@ -140,6 +140,7 @@ enum i40e_state_t { __I40E_CLIENT_RESET, __I40E_VIRTCHNL_OP_PENDING, __I40E_RECOVERY_MODE, + __I40E_VF_RESETS_DISABLED, /* disable resets during i40e_remove */ /* This must be last as it determines the size of the BITMAP */ __I40E_STATE_SIZE__, }; diff --git a/drivers/net/ethernet/intel/i40e/i40e_main.c b/drivers/net/ethernet/intel/i40e/i40e_main.c index 4f8a2154b93f..1337686bd099 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_main.c +++ b/drivers/net/ethernet/intel/i40e/i40e_main.c @@ -4010,8 +4010,16 @@ static irqreturn_t i40e_intr(int irq, void *data) } if (icr0 & I40E_PFINT_ICR0_VFLR_MASK) { - ena_mask &= ~I40E_PFINT_ICR0_ENA_VFLR_MASK; - set_bit(__I40E_VFLR_EVENT_PENDING, pf->state); + /* disable any further VFLR event notifications */ + if (test_bit(__I40E_VF_RESETS_DISABLED, pf->state)) { + u32 reg = rd32(hw, I40E_PFINT_ICR0_ENA); + + reg &= ~I40E_PFINT_ICR0_VFLR_MASK; + wr32(hw, I40E_PFINT_ICR0_ENA, reg); + } else { + ena_mask &= ~I40E_PFINT_ICR0_ENA_VFLR_MASK; + set_bit(__I40E_VFLR_EVENT_PENDING, pf->state); + } } if (icr0 & I40E_PFINT_ICR0_GRST_MASK) { @@ -15311,6 +15319,11 @@ static void i40e_remove(struct pci_dev *pdev) while (test_bit(__I40E_RESET_RECOVERY_PENDING, pf->state)) usleep_range(1000, 2000); + if (pf->flags & I40E_FLAG_SRIOV_ENABLED) { + set_bit(__I40E_VF_RESETS_DISABLED, pf->state); + i40e_free_vfs(pf); + pf->flags &= ~I40E_FLAG_SRIOV_ENABLED; + } /* no more scheduling of any task */ set_bit(__I40E_SUSPENDED, pf->state); set_bit(__I40E_DOWN, pf->state); @@ -15337,11 +15350,6 @@ static void i40e_remove(struct pci_dev *pdev) */ i40e_notify_client_of_netdev_close(pf->vsi[pf->lan_vsi], false); - if (pf->flags & I40E_FLAG_SRIOV_ENABLED) { - i40e_free_vfs(pf); - pf->flags &= ~I40E_FLAG_SRIOV_ENABLED; - } - i40e_fdir_teardown(pf); /* If there is a switch structure or any orphans, remove them. diff --git a/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c b/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c index 4919d22d7b6b..1b5390ec3d78 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c +++ b/drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c @@ -1403,7 +1403,8 @@ static void i40e_cleanup_reset_vf(struct i40e_vf *vf) * @vf: pointer to the VF structure * @flr: VFLR was issued or not * - * Returns true if the VF is reset, false otherwise. + * Returns true if the VF is in reset, resets successfully, or resets + * are disabled and false otherwise. **/ bool i40e_reset_vf(struct i40e_vf *vf, bool flr) { @@ -1413,11 +1414,14 @@ bool i40e_reset_vf(struct i40e_vf *vf, bool flr) u32 reg; int i; + if (test_bit(__I40E_VF_RESETS_DISABLED, pf->state)) + return true; + /* If the VFs have been disabled, this means something else is * resetting the VF, so we shouldn't continue. */ if (test_and_set_bit(__I40E_VF_DISABLE, pf->state)) - return false; + return true; i40e_trigger_vf_reset(vf, flr); @@ -1581,6 +1585,15 @@ void i40e_free_vfs(struct i40e_pf *pf) i40e_notify_client_of_vf_enable(pf, 0); + /* Disable IOV before freeing resources. This lets any VF drivers + * running in the host get themselves cleaned up before we yank + * the carpet out from underneath their feet. + */ + if (!pci_vfs_assigned(pf->pdev)) + pci_disable_sriov(pf->pdev); + else + dev_warn(&pf->pdev->dev, "VFs are assigned - not disabling SR-IOV\n"); + /* Amortize wait time by stopping all VFs at the same time */ for (i = 0; i < pf->num_alloc_vfs; i++) { if (test_bit(I40E_VF_STATE_INIT, &pf->vf[i].vf_states)) @@ -1596,15 +1609,6 @@ void i40e_free_vfs(struct i40e_pf *pf) i40e_vsi_wait_queues_disabled(pf->vsi[pf->vf[i].lan_vsi_idx]); } - /* Disable IOV before freeing resources. This lets any VF drivers - * running in the host get themselves cleaned up before we yank - * the carpet out from underneath their feet. - */ - if (!pci_vfs_assigned(pf->pdev)) - pci_disable_sriov(pf->pdev); - else - dev_warn(&pf->pdev->dev, "VFs are assigned - not disabling SR-IOV\n"); - /* free up VF resources */ tmp = pf->num_alloc_vfs; pf->num_alloc_vfs = 0;