From patchwork Thu Aug 3 13:44:46 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ding Tianhong X-Patchwork-Id: 109343 Delivered-To: patch@linaro.org Received: by 10.182.109.195 with SMTP id hu3csp15492obb; Thu, 3 Aug 2017 06:47:38 -0700 (PDT) X-Received: by 10.84.237.15 with SMTP id s15mr2140869plk.100.1501768057915; Thu, 03 Aug 2017 06:47:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1501768057; cv=none; d=google.com; s=arc-20160816; b=N6sZI25waFd+bc4Ag/Lprbf7qYfrI5gaJ1HmG5zdLklHqqKdyadZHtyrjQQe8/FV2z kPm4QXUPvasyMY4Iuwae7brFdVZtSPhe7soUhOCA6SHi1l37bGX8rRx/MJNyWgIAP95u UFVZKllwiBlZKQTSoBJy106bnwNqW9w/JnW8/dy3qkWIAv7bc6M73U/dvJ8lva+QNAF4 jpAiRa4a+vVyF+mbBceI3j8x+qI3LVpJx/qS8OQII62/Xr3TROJgpQEljyEaPJ6aYORY orQck0gTstW46KdCwSjZXGtYdYWy+aCz8QIXtwKeL1Uj5n/5DMQv8E41wCaUIlFFEFTj 3ImA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:arc-authentication-results; bh=m3FJuPpj8A+eXIbdja6kZWtiJ814O8xZhxwvR/u9Zis=; b=y0+5Mt+XHTdsjJ1sYYEY/3sx0MAmzdStp8/b9rJTxplqQFPTbsmDDGTeY5suZPm3p1 9tdO4EyqwOAgdQ5ZDTN9A+dcsWJglflnnIcvyFHGqMCkeUP8nxuPmx9K5/mFHC47MwX2 1KtZVYFQSUeXnKbl16AJJ/AOKy1xZSW2gwB6gBqci6kSPVdbMVNiR7hx0dbtEIqYVGo+ Sh1aB68dcBgB0ydrt4sCqMu4EHzw+woXndDkmK0OgOWbZXPU0sEt5lajIQRic3SeYZWa 1EZkNYXDlUEA8+dz5XmsMdEntm1yuYUMwUjEK2/5OFlzyxG1GjRGY5TZIEL4uRfSeTaO vIiw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id b65si7595866pgc.1.2017.08.03.06.47.36; Thu, 03 Aug 2017 06:47:37 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752097AbdHCNre (ORCPT + 25 others); Thu, 3 Aug 2017 09:47:34 -0400 Received: from szxga03-in.huawei.com ([45.249.212.189]:9919 "EHLO szxga03-in.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751220AbdHCNpk (ORCPT ); Thu, 3 Aug 2017 09:45:40 -0400 Received: from 172.30.72.55 (EHLO DGGEML401-HUB.china.huawei.com) ([172.30.72.55]) by dggrg03-dlp.huawei.com (MOS 4.4.6-GA FastPath queued) with ESMTP id ASU39382; Thu, 03 Aug 2017 21:45:18 +0800 (CST) Received: from localhost (10.177.23.32) by DGGEML401-HUB.china.huawei.com (10.3.17.32) with Microsoft SMTP Server id 14.3.301.0; Thu, 3 Aug 2017 21:45:07 +0800 From: Ding Tianhong To: , , , , , , , , , , , , , , , , , , , , , , , , , CC: Ding Tianhong Subject: [PATCH v8 1/4] PCI: Add new PCIe Fabric End Node flag, PCI_DEV_FLAGS_NO_RELAXED_ORDERING Date: Thu, 3 Aug 2017 21:44:46 +0800 Message-ID: <1501767889-7772-2-git-send-email-dingtianhong@huawei.com> X-Mailer: git-send-email 1.8.5.2.msysgit.0 In-Reply-To: <1501767889-7772-1-git-send-email-dingtianhong@huawei.com> References: <1501767889-7772-1-git-send-email-dingtianhong@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.177.23.32] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A0B0201.598328F1.015B, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: d9f5f711b66c40be2efb1e52a7186fff Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Casey Leedom The patch adds a new flag PCI_DEV_FLAGS_NO_RELAXED_ORDERING to indicate that Relaxed Ordering (RO) attribute should not be used for Transaction Layer Packets (TLP) targetted towards these affected root complexes. Current list of affected parts include Intel E5-26xx root complex which suffers from flow control credits that result in performance issues. On these affected parts RO can still be used for peer-2-peer traffic. AMD A1100 ARM ("SEATTLE") Root complexes don't obey PCIe 3.0 ordering rules, hence could lead to data-corruption. Signed-off-by: Casey Leedom Signed-off-by: Ding Tianhong Acked-by: Ashok Raj --- drivers/pci/quirks.c | 38 ++++++++++++++++++++++++++++++++++++++ include/linux/pci.h | 2 ++ 2 files changed, 40 insertions(+) -- 1.8.3.1 diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index 6967c6b..1e1cdbe 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -4016,6 +4016,44 @@ static void quirk_tw686x_class(struct pci_dev *pdev) quirk_tw686x_class); /* + * Some devices have problems with Transaction Layer Packets with the Relaxed + * Ordering Attribute set. Such devices should mark themselves and other + * Device Drivers should check before sending TLPs with RO set. + */ +static void quirk_relaxedordering_disable(struct pci_dev *dev) +{ + dev->dev_flags |= PCI_DEV_FLAGS_NO_RELAXED_ORDERING; +} + +/* + * Intel E5-26xx Root Complex has a Flow Control Credit issue which can + * cause performance problems with Upstream Transaction Layer Packets with + * Relaxed Ordering set. + */ +DECLARE_PCI_FIXUP_CLASS_EARLY(PCI_VENDOR_ID_INTEL, 0x6f02, PCI_CLASS_NOT_DEFINED, 8, + quirk_relaxedordering_disable); +DECLARE_PCI_FIXUP_CLASS_EARLY(PCI_VENDOR_ID_INTEL, 0x6f04, PCI_CLASS_NOT_DEFINED, 8, + quirk_relaxedordering_disable); +DECLARE_PCI_FIXUP_CLASS_EARLY(PCI_VENDOR_ID_INTEL, 0x6f08, PCI_CLASS_NOT_DEFINED, 8, + quirk_relaxedordering_disable); + +/* + * The AMD ARM A1100 (AKA "SEATTLE") SoC has a bug in its PCIe Root Complex + * where Upstream Transaction Layer Packets with the Relaxed Ordering + * Attribute clear are allowed to bypass earlier TLPs with Relaxed Ordering + * set. This is a violation of the PCIe 3.0 Transaction Ordering Rules + * outlined in Section 2.4.1 (PCI Express(r) Base Specification Revision 3.0 + * November 10, 2010). As a result, on this platform we can't use Relaxed + * Ordering for Upstream TLPs. + */ +DECLARE_PCI_FIXUP_CLASS_EARLY(PCI_VENDOR_ID_AMD, 0x1a00, PCI_CLASS_NOT_DEFINED, 8, + quirk_relaxedordering_disable); +DECLARE_PCI_FIXUP_CLASS_EARLY(PCI_VENDOR_ID_AMD, 0x1a01, PCI_CLASS_NOT_DEFINED, 8, + quirk_relaxedordering_disable); +DECLARE_PCI_FIXUP_CLASS_EARLY(PCI_VENDOR_ID_AMD, 0x1a02, PCI_CLASS_NOT_DEFINED, 8, + quirk_relaxedordering_disable); + +/* * Per PCIe r3.0, sec 2.2.9, "Completion headers must supply the same * values for the Attribute as were supplied in the header of the * corresponding Request, except as explicitly allowed when IDO is used." diff --git a/include/linux/pci.h b/include/linux/pci.h index 4869e66..412ec1c 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -188,6 +188,8 @@ enum pci_dev_flags { * the direct_complete optimization. */ PCI_DEV_FLAGS_NEEDS_RESUME = (__force pci_dev_flags_t) (1 << 11), + /* Don't use Relaxed Ordering for TLPs directed at this device */ + PCI_DEV_FLAGS_NO_RELAXED_ORDERING = (__force pci_dev_flags_t) (1 << 12), }; enum pci_irq_reroute_variant { From patchwork Thu Aug 3 13:44:47 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ding Tianhong X-Patchwork-Id: 109344 Delivered-To: patch@linaro.org Received: by 10.182.109.195 with SMTP id hu3csp16093obb; Thu, 3 Aug 2017 06:48:11 -0700 (PDT) X-Received: by 10.84.209.172 with SMTP id y41mr1982953plh.61.1501768090983; Thu, 03 Aug 2017 06:48:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1501768090; cv=none; d=google.com; s=arc-20160816; b=FGIFnU//QEGJTqQ0+hJ5v3WIu9PhoKx4eg17hVBzhj2YcyfG/jUI27Efogrw6hFc53 JIQ7CL7aKxI28NAFr1e965JJKiz+KgeOeunsPymoO2MwuaUTO8LHazWv7La4Y0s2UflG Vj1owO/9n2xk4/Qtn4R5OTNrz2SSaOa4YpN2xWVzsYlFsAJGmv6c97oN3vFVysdJxw6r TSK6RfTvPYPjsdnRDaZ2JJ9YKYOjMpLQ7V2oPo+w0+4WyHi/9AcZdaB41z8MIcfOg/IL E3hG5GhgW1LuwzEcVneCfnf4XZHdxy38KMB0srVCH+JuFETDWfFUFWzlKYehmBpLFwkB Y8mQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:arc-authentication-results; bh=iHWcaKKDUTu+3il+TyspR2KAtEmd5dnZaVR0cs/jiFs=; b=pzzqRESakvLP+Af24OpocKsM9qdM+HdU1eyULMDT0Wq6ao9h62jDd9rqBwgLrsKT7R Kk5UJ40TckQrqPSQfBhlOWS/P5qrZRSJ97FPazPIBUYm7GWpGz8nax+vb/cv/f5zxehr WKvINlNRQT8hATbSuMqUdj/15jT+C35gjrJNim8QvReIKtYhCni0+FhnsNWKhg8qGwPE r/5Pxtfs0PMHIbeVg479YkDwY0x9H3iRs2dWVBi0NAorkxZax+J4txXGx8rdlsVWY7gF VlehY5EZsYnnRAx66PlQakBYVDSp1IPlPvo3A76JPBaWm63Q1swKDy52CrTBQjD/jcbo Po5A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id k14si12525348pln.613.2017.08.03.06.48.10; Thu, 03 Aug 2017 06:48:10 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752137AbdHCNsI (ORCPT + 25 others); Thu, 3 Aug 2017 09:48:08 -0400 Received: from szxga03-in.huawei.com ([45.249.212.189]:9920 "EHLO szxga03-in.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751629AbdHCNpk (ORCPT ); Thu, 3 Aug 2017 09:45:40 -0400 Received: from 172.30.72.55 (EHLO DGGEML401-HUB.china.huawei.com) ([172.30.72.55]) by dggrg03-dlp.huawei.com (MOS 4.4.6-GA FastPath queued) with ESMTP id ASU39383; Thu, 03 Aug 2017 21:45:18 +0800 (CST) Received: from localhost (10.177.23.32) by DGGEML401-HUB.china.huawei.com (10.3.17.32) with Microsoft SMTP Server id 14.3.301.0; Thu, 3 Aug 2017 21:45:08 +0800 From: Ding Tianhong To: , , , , , , , , , , , , , , , , , , , , , , , , , CC: Ding Tianhong Subject: [PATCH v8 2/4] PCI: Disable PCIe Relaxed Ordering if unsupported Date: Thu, 3 Aug 2017 21:44:47 +0800 Message-ID: <1501767889-7772-3-git-send-email-dingtianhong@huawei.com> X-Mailer: git-send-email 1.8.5.2.msysgit.0 In-Reply-To: <1501767889-7772-1-git-send-email-dingtianhong@huawei.com> References: <1501767889-7772-1-git-send-email-dingtianhong@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.177.23.32] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A0B0205.598328F1.0102, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: b9a9a7b9619ab76640ef5944faa28941 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org When bit4 is set in the PCIe Device Control register, it indicates whether the device is permitted to use relaxed ordering. On some platforms using relaxed ordering can have performance issues or due to erratum can cause data-corruption. In such cases devices must avoid using relaxed ordering. This patch checks if there is any node in the hierarchy that indicates that using relaxed ordering is not safe. In such cases the patch turns off the relaxed ordering by clearing the eapability for this device. And if the device is probably running in a guest machine, we should do nothing. Signed-off-by: Ding Tianhong Acked-by: Alexander Duyck Acked-by: Ashok Raj --- drivers/pci/pci.c | 29 +++++++++++++++++++++++++++++ drivers/pci/probe.c | 37 +++++++++++++++++++++++++++++++++++++ include/linux/pci.h | 2 ++ 3 files changed, 68 insertions(+) -- 1.8.3.1 diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index af0cc34..4f9d7c1 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -4854,6 +4854,35 @@ int pcie_set_mps(struct pci_dev *dev, int mps) EXPORT_SYMBOL(pcie_set_mps); /** + * pcie_clear_relaxed_ordering - clear PCI Express relaxed ordering bit + * @dev: PCI device to query + * + * If possible clear relaxed ordering + */ +int pcie_clear_relaxed_ordering(struct pci_dev *dev) +{ + return pcie_capability_clear_word(dev, PCI_EXP_DEVCTL, + PCI_EXP_DEVCTL_RELAX_EN); +} +EXPORT_SYMBOL(pcie_clear_relaxed_ordering); + +/** + * pcie_relaxed_ordering_supported - Probe for PCIe relexed ordering support + * @dev: PCI device to query + * + * Returns true if the device support relaxed ordering attribute. + */ +bool pcie_relaxed_ordering_supported(struct pci_dev *dev) +{ + u16 v; + + pcie_capability_read_word(dev, PCI_EXP_DEVCTL, &v); + + return !!(v & PCI_EXP_DEVCTL_RELAX_EN); +} +EXPORT_SYMBOL(pcie_relaxed_ordering_supported); + +/** * pcie_get_minimum_link - determine minimum link settings of a PCI device * @dev: PCI device to query * @speed: storage for minimum speed diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index c31310d..48df012 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -1762,6 +1762,42 @@ static void pci_configure_extended_tags(struct pci_dev *dev) PCI_EXP_DEVCTL_EXT_TAG); } +/** + * pci_dev_should_disable_relaxed_ordering - check if the PCI device + * should disable the relaxed ordering attribute. + * @dev: PCI device + * + * Return true if any of the PCI devices above us do not support + * relaxed ordering. + */ +static bool pci_dev_should_disable_relaxed_ordering(struct pci_dev *dev) +{ + while (dev) { + if (dev->dev_flags & PCI_DEV_FLAGS_NO_RELAXED_ORDERING) + return true; + + dev = dev->bus->self; + } + + return false; +} + +static void pci_configure_relaxed_ordering(struct pci_dev *dev) +{ + /* We should not alter the relaxed ordering bit for the VF */ + if (dev->is_virtfn) + return; + + /* If the releaxed ordering enable bit is not set, do nothing. */ + if (!pcie_relaxed_ordering_supported(dev)) + return; + + if (pci_dev_should_disable_relaxed_ordering(dev)) { + pcie_clear_relaxed_ordering(dev); + dev_info(&dev->dev, "Disable Relaxed Ordering\n"); + } +} + static void pci_configure_device(struct pci_dev *dev) { struct hotplug_params hpp; @@ -1769,6 +1805,7 @@ static void pci_configure_device(struct pci_dev *dev) pci_configure_mps(dev); pci_configure_extended_tags(dev); + pci_configure_relaxed_ordering(dev); memset(&hpp, 0, sizeof(hpp)); ret = pci_get_hp_params(dev, &hpp); diff --git a/include/linux/pci.h b/include/linux/pci.h index 412ec1c..3aa23a2 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -1127,6 +1127,8 @@ int pci_add_ext_cap_save_buffer(struct pci_dev *dev, void pci_pme_wakeup_bus(struct pci_bus *bus); void pci_d3cold_enable(struct pci_dev *dev); void pci_d3cold_disable(struct pci_dev *dev); +int pcie_clear_relaxed_ordering(struct pci_dev *dev); +bool pcie_relaxed_ordering_supported(struct pci_dev *dev); /* PCI Virtual Channel */ int pci_save_vc_state(struct pci_dev *dev); From patchwork Thu Aug 3 13:44:48 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ding Tianhong X-Patchwork-Id: 109342 Delivered-To: patch@linaro.org Received: by 10.182.109.195 with SMTP id hu3csp14441obb; Thu, 3 Aug 2017 06:46:38 -0700 (PDT) X-Received: by 10.84.142.1 with SMTP id 1mr2034256plw.130.1501767998079; Thu, 03 Aug 2017 06:46:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1501767998; cv=none; d=google.com; s=arc-20160816; b=XGzrc2PKV9Ll2m15hLOqFvqZXboc63x2iUPqog67TO7vVyIDBIjDIx7mAwjpfpRlOy hi2bf9aY5aMx3nAXBYXKRLj2qkPe/+fvOeZzihQpmjWkDXEtriUDApy7gd80r5Ccvr0Y ypxRoAVxiXzk1z9ApLOQUF3oUEJr++es86882TA0ySkhHDBlgyi/55scGlacjAVrRuj5 PhH7CThiSqSjk2i0ozO6uL9er3U3RgwHHQsGyFGgFdgg0jA37GF0MR81rh3mYS6gzQWs Vxvv0owObb9VLLKJGfWR970HFE96dvztCivEUVMMdLt7s9AsHDjtfZTnN1ZHlwlULLD9 YEFw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:arc-authentication-results; bh=Sl6Tov0vUwJv5xQloB9nwwZV8NVexf/gcwQGK8aDNpQ=; b=nyAvFzkfubvlreu6+rxbUuds8Kec7e9qiwkH2O+o4B3BfhssUJS5U/sEodIe+4/VJu Ig00Z4bbnJr6g2avwehVtpc7fdhKNayFo/FGpGNYjP9JzODOqQVQ7APD6RiRIv+JVklg sGFnsSl5Vu3iw+QDig5wx61PwO5spwj1+cgprNbMsLuZAn6tpsE17kzZbZE/vbmVm4Ds Hq7II9KCarESU4ux3TUJm9QFi4ygW5YasuxKLYThstqFfi3PlryYIx/LIfh6lmzgbCrT /giIkBaPllFTyuOFKo174iK4yTJnq6Jeq50zx2joaz7IAVpRkPejHTbXrhjQsScu8zYZ H48Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id p91si22447350plb.416.2017.08.03.06.46.37; Thu, 03 Aug 2017 06:46:38 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752005AbdHCNqd (ORCPT + 25 others); Thu, 3 Aug 2017 09:46:33 -0400 Received: from szxga01-in.huawei.com ([45.249.212.187]:10762 "EHLO szxga01-in.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751865AbdHCNpo (ORCPT ); Thu, 3 Aug 2017 09:45:44 -0400 Received: from 172.30.72.53 (EHLO DGGEML401-HUB.china.huawei.com) ([172.30.72.53]) by dggrg01-dlp.huawei.com (MOS 4.4.6-GA FastPath queued) with ESMTP id ATT39301; Thu, 03 Aug 2017 21:45:21 +0800 (CST) Received: from localhost (10.177.23.32) by DGGEML401-HUB.china.huawei.com (10.3.17.32) with Microsoft SMTP Server id 14.3.301.0; Thu, 3 Aug 2017 21:45:10 +0800 From: Ding Tianhong To: , , , , , , , , , , , , , , , , , , , , , , , , , CC: Ding Tianhong Subject: [PATCH v8 3/4] net/cxgb4: Use new PCI_DEV_FLAGS_NO_RELAXED_ORDERING flag Date: Thu, 3 Aug 2017 21:44:48 +0800 Message-ID: <1501767889-7772-4-git-send-email-dingtianhong@huawei.com> X-Mailer: git-send-email 1.8.5.2.msysgit.0 In-Reply-To: <1501767889-7772-1-git-send-email-dingtianhong@huawei.com> References: <1501767889-7772-1-git-send-email-dingtianhong@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.177.23.32] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A090205.598328F4.0049, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: 64b9d86256b0747af47bab2c32a9aec3 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Casey Leedom cxgb4 Ethernet driver now queries PCIe configuration space to determine if it can send TLPs to it with the Relaxed Ordering Attribute set. Remove the enable_pcie_relaxed_ordering() to avoid enable PCIe Capability Device Control[Relaxed Ordering Enable] at probe routine, to make sure the driver will not send the Relaxed Ordering TLPs to the Root Complex which could not deal the Relaxed Ordering TLPs. Signed-off-by: Casey Leedom Signed-off-by: Ding Tianhong Reviewed-by: Casey Leedom --- drivers/net/ethernet/chelsio/cxgb4/cxgb4.h | 1 + drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c | 23 +++++++++++++++++------ drivers/net/ethernet/chelsio/cxgb4/sge.c | 5 +++-- 3 files changed, 21 insertions(+), 8 deletions(-) -- 1.8.3.1 diff --git a/drivers/net/ethernet/chelsio/cxgb4/cxgb4.h b/drivers/net/ethernet/chelsio/cxgb4/cxgb4.h index ef4be78..09ea62e 100644 --- a/drivers/net/ethernet/chelsio/cxgb4/cxgb4.h +++ b/drivers/net/ethernet/chelsio/cxgb4/cxgb4.h @@ -529,6 +529,7 @@ enum { /* adapter flags */ USING_SOFT_PARAMS = (1 << 6), MASTER_PF = (1 << 7), FW_OFLD_CONN = (1 << 9), + ROOT_NO_RELAXED_ORDERING = (1 << 10), }; enum { diff --git a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c index e403fa1..391e484 100644 --- a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c +++ b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c @@ -4654,11 +4654,6 @@ static void print_port_info(const struct net_device *dev) dev->name, adap->params.vpd.id, adap->name, buf); } -static void enable_pcie_relaxed_ordering(struct pci_dev *dev) -{ - pcie_capability_set_word(dev, PCI_EXP_DEVCTL, PCI_EXP_DEVCTL_RELAX_EN); -} - /* * Free the following resources: * - memory used for tables @@ -4908,7 +4903,6 @@ static int init_one(struct pci_dev *pdev, const struct pci_device_id *ent) } pci_enable_pcie_error_reporting(pdev); - enable_pcie_relaxed_ordering(pdev); pci_set_master(pdev); pci_save_state(pdev); @@ -4947,6 +4941,23 @@ static int init_one(struct pci_dev *pdev, const struct pci_device_id *ent) adapter->msg_enable = DFLT_MSG_ENABLE; memset(adapter->chan_map, 0xff, sizeof(adapter->chan_map)); + /* If possible, we use PCIe Relaxed Ordering Attribute to deliver + * Ingress Packet Data to Free List Buffers in order to allow for + * chipset performance optimizations between the Root Complex and + * Memory Controllers. (Messages to the associated Ingress Queue + * notifying new Packet Placement in the Free Lists Buffers will be + * send without the Relaxed Ordering Attribute thus guaranteeing that + * all preceding PCIe Transaction Layer Packets will be processed + * first.) But some Root Complexes have various issues with Upstream + * Transaction Layer Packets with the Relaxed Ordering Attribute set. + * The PCIe devices which under the Root Complexes will be cleared the + * Relaxed Ordering bit in the configuration space, So we check our + * PCIe configuration space to see if it's flagged with advice against + * using Relaxed Ordering. + */ + if (!pcie_relaxed_ordering_supported(pdev)) + adapter->flags |= ROOT_NO_RELAXED_ORDERING; + spin_lock_init(&adapter->stats_lock); spin_lock_init(&adapter->tid_release_lock); spin_lock_init(&adapter->win0_lock); diff --git a/drivers/net/ethernet/chelsio/cxgb4/sge.c b/drivers/net/ethernet/chelsio/cxgb4/sge.c index ede1220..4ef68f6 100644 --- a/drivers/net/ethernet/chelsio/cxgb4/sge.c +++ b/drivers/net/ethernet/chelsio/cxgb4/sge.c @@ -2719,6 +2719,7 @@ int t4_sge_alloc_rxq(struct adapter *adap, struct sge_rspq *iq, bool fwevtq, struct fw_iq_cmd c; struct sge *s = &adap->sge; struct port_info *pi = netdev_priv(dev); + int relaxed = !(adap->flags & ROOT_NO_RELAXED_ORDERING); /* Size needs to be multiple of 16, including status entry. */ iq->size = roundup(iq->size, 16); @@ -2772,8 +2773,8 @@ int t4_sge_alloc_rxq(struct adapter *adap, struct sge_rspq *iq, bool fwevtq, flsz = fl->size / 8 + s->stat_len / sizeof(struct tx_desc); c.iqns_to_fl0congen |= htonl(FW_IQ_CMD_FL0PACKEN_F | - FW_IQ_CMD_FL0FETCHRO_F | - FW_IQ_CMD_FL0DATARO_F | + FW_IQ_CMD_FL0FETCHRO_V(relaxed) | + FW_IQ_CMD_FL0DATARO_V(relaxed) | FW_IQ_CMD_FL0PADEN_F); if (cong >= 0) c.iqns_to_fl0congen |= From patchwork Thu Aug 3 13:44:49 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ding Tianhong X-Patchwork-Id: 109341 Delivered-To: patch@linaro.org Received: by 10.182.109.195 with SMTP id hu3csp13649obb; Thu, 3 Aug 2017 06:45:57 -0700 (PDT) X-Received: by 10.84.230.137 with SMTP id e9mr1952825plk.71.1501767957497; Thu, 03 Aug 2017 06:45:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1501767957; cv=none; d=google.com; s=arc-20160816; b=S0gRfrot0Q2/h6XNNprh6UIcItWtUcCrexoP8LiLwoOyUsN7ha2CAciXJVTm9BUQD6 Q3ifoaN9AHFnFEyW/FRwpj1OUfJoMCo9+tMaMrLLJ75T0qj9UOmtHMqqbHy5UoLQpgV8 jr2rRYycBdsJ0jFfaMOElbKvMRiR1VwqyUfwEXSez3b6JYQvnbSCorlfGXHIPBCVBzH4 ycbqWNjPnGrL6bOxP3xPqey8FfFHrz8/s+OyyD1VihWFJjgBTHUAxQDP3LWXQc+SpMQG CqA+KXjkmL9t0NE+Qaznx0goooreolsmf1yIXuB69SnZzpuvP4k9uvk6PiQXLmaDDFuR BTZg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:arc-authentication-results; bh=sGPqGQl8fNYn+qK1Zm2+RxdVKglugOBT9lJ+CBHN/A4=; b=fQN+C5oQ1pRaBqzrvYBmfz34tjvvxdPaDPYRC1rR2NGugXSdSnzDDAQcWagta5nHO3 cFQOiSEWlNK05l/GFEzqxkqvlMuiJXqtyeRmljo/aVJArUIcK29FKkvTg5rSy0ob6wnb 1d/G1VyR/fmh0oAOdmba7U2BCNnAsAGqYLm5HG6A5K8ZtT7kZGWboGmjRVlcd4Q7f7oO sXKPpmTtae1MYWEk9hF4kNWpS9B7SVJUIZdNUbqmNs1qN7nkFvpg5LNk4ovJdFmVlCdv fMBjkUPe0J01GdAm82WwAr/hnrs1+M63lZMXZU8WVNjzPj02hsbvuMNIRFOftqgOhxxq L5gw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id i12si21286196pfi.42.2017.08.03.06.45.57; Thu, 03 Aug 2017 06:45:57 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751940AbdHCNpy (ORCPT + 25 others); Thu, 3 Aug 2017 09:45:54 -0400 Received: from szxga01-in.huawei.com ([45.249.212.187]:10763 "EHLO szxga01-in.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751247AbdHCNpp (ORCPT ); Thu, 3 Aug 2017 09:45:45 -0400 Received: from 172.30.72.53 (EHLO DGGEML401-HUB.china.huawei.com) ([172.30.72.53]) by dggrg01-dlp.huawei.com (MOS 4.4.6-GA FastPath queued) with ESMTP id ATT39297; Thu, 03 Aug 2017 21:45:21 +0800 (CST) Received: from localhost (10.177.23.32) by DGGEML401-HUB.china.huawei.com (10.3.17.32) with Microsoft SMTP Server id 14.3.301.0; Thu, 3 Aug 2017 21:45:11 +0800 From: Ding Tianhong To: , , , , , , , , , , , , , , , , , , , , , , , , , CC: Ding Tianhong Subject: [PATCH v8 4/4] net/cxgb4vf: Use new PCI_DEV_FLAGS_NO_RELAXED_ORDERING flag Date: Thu, 3 Aug 2017 21:44:49 +0800 Message-ID: <1501767889-7772-5-git-send-email-dingtianhong@huawei.com> X-Mailer: git-send-email 1.8.5.2.msysgit.0 In-Reply-To: <1501767889-7772-1-git-send-email-dingtianhong@huawei.com> References: <1501767889-7772-1-git-send-email-dingtianhong@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.177.23.32] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A090203.598328F3.0009, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: dc0420fa7f89a69169b1cbe9d3976fb1 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Casey Leedom cxgb4vf Ethernet driver now queries PCIe configuration space to determine if it can send TLPs to it with the Relaxed Ordering Attribute set, just like the pf did. Signed-off-by: Casey Leedom Signed-off-by: Ding Tianhong Reviewed-by: Casey Leedom --- drivers/net/ethernet/chelsio/cxgb4vf/adapter.h | 1 + drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c | 18 ++++++++++++++++++ drivers/net/ethernet/chelsio/cxgb4vf/sge.c | 3 +++ 3 files changed, 22 insertions(+) -- 1.8.3.1 diff --git a/drivers/net/ethernet/chelsio/cxgb4vf/adapter.h b/drivers/net/ethernet/chelsio/cxgb4vf/adapter.h index 109bc63..08c6ddb 100644 --- a/drivers/net/ethernet/chelsio/cxgb4vf/adapter.h +++ b/drivers/net/ethernet/chelsio/cxgb4vf/adapter.h @@ -408,6 +408,7 @@ enum { /* adapter flags */ USING_MSI = (1UL << 1), USING_MSIX = (1UL << 2), QUEUES_BOUND = (1UL << 3), + ROOT_NO_RELAXED_ORDERING = (1UL << 4), }; /* diff --git a/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c b/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c index ac7a150..59e7639 100644 --- a/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c +++ b/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c @@ -2888,6 +2888,24 @@ static int cxgb4vf_pci_probe(struct pci_dev *pdev, */ adapter->name = pci_name(pdev); adapter->msg_enable = DFLT_MSG_ENABLE; + + /* If possible, we use PCIe Relaxed Ordering Attribute to deliver + * Ingress Packet Data to Free List Buffers in order to allow for + * chipset performance optimizations between the Root Complex and + * Memory Controllers. (Messages to the associated Ingress Queue + * notifying new Packet Placement in the Free Lists Buffers will be + * send without the Relaxed Ordering Attribute thus guaranteeing that + * all preceding PCIe Transaction Layer Packets will be processed + * first.) But some Root Complexes have various issues with Upstream + * Transaction Layer Packets with the Relaxed Ordering Attribute set. + * The PCIe devices which under the Root Complexes will be cleared the + * Relaxed Ordering bit in the configuration space, So we check our + * PCIe configuration space to see if it's flagged with advice against + * using Relaxed Ordering. + */ + if (!pcie_relaxed_ordering_supported(pdev)) + adapter->flags |= ROOT_NO_RELAXED_ORDERING; + err = adap_init0(adapter); if (err) goto err_unmap_bar; diff --git a/drivers/net/ethernet/chelsio/cxgb4vf/sge.c b/drivers/net/ethernet/chelsio/cxgb4vf/sge.c index e37dde2..05498e7 100644 --- a/drivers/net/ethernet/chelsio/cxgb4vf/sge.c +++ b/drivers/net/ethernet/chelsio/cxgb4vf/sge.c @@ -2205,6 +2205,7 @@ int t4vf_sge_alloc_rxq(struct adapter *adapter, struct sge_rspq *rspq, struct port_info *pi = netdev_priv(dev); struct fw_iq_cmd cmd, rpl; int ret, iqandst, flsz = 0; + int relaxed = !(adapter->flags & ROOT_NO_RELAXED_ORDERING); /* * If we're using MSI interrupts and we're not initializing the @@ -2300,6 +2301,8 @@ int t4vf_sge_alloc_rxq(struct adapter *adapter, struct sge_rspq *rspq, cpu_to_be32( FW_IQ_CMD_FL0HOSTFCMODE_V(SGE_HOSTFCMODE_NONE) | FW_IQ_CMD_FL0PACKEN_F | + FW_IQ_CMD_FL0FETCHRO_V(relaxed) | + FW_IQ_CMD_FL0DATARO_V(relaxed) | FW_IQ_CMD_FL0PADEN_F); /* In T6, for egress queue type FL there is internal overhead