mbox series

[net-next,0/5] Flow control for NXP ENETC

Message ID 20210416234225.3715819-1-olteanv@gmail.com
Headers show
Series Flow control for NXP ENETC | expand

Message

Vladimir Oltean April 16, 2021, 11:42 p.m. UTC
From: Vladimir Oltean <vladimir.oltean@nxp.com>

This patch series contains logic for enabling the lossless mode on the
RX rings of the ENETC, and the PAUSE thresholds on the internal FIFO
memory.

During testing it was found that, with the default FIFO configuration,
a sender which isn't persuaded by our PAUSE frames and keeps sending
will cause some MAC RX frame errors. To mitigate this, we need to ensure
that the FIFO never runs completely full, so we need to fix up a setting
that was supposed to be configured well out of reset. Unfortunately this
requires the addition of a new mini-driver.

Vladimir Oltean (5):
  net: enetc: create a common enetc_pf_to_port helper
  dt-bindings: net: fsl: enetc: add the IERB documentation
  net: enetc: add a mini driver for the Integrated Endpoint Register
    Block
  arm64: dts: ls1028a: declare the Integrated Endpoint Register Block
    node
  net: enetc: add support for flow control

 .../devicetree/bindings/net/fsl-enetc.txt     |  15 ++
 .../arm64/boot/dts/freescale/fsl-ls1028a.dtsi |   6 +
 drivers/net/ethernet/freescale/enetc/Kconfig  |   9 +
 drivers/net/ethernet/freescale/enetc/Makefile |   3 +
 drivers/net/ethernet/freescale/enetc/enetc.h  |  16 ++
 .../ethernet/freescale/enetc/enetc_ethtool.c  |  18 ++
 .../net/ethernet/freescale/enetc/enetc_hw.h   |   9 +
 .../net/ethernet/freescale/enetc/enetc_ierb.c | 155 ++++++++++++++++++
 .../net/ethernet/freescale/enetc/enetc_ierb.h |  20 +++
 .../net/ethernet/freescale/enetc/enetc_pf.c   |  95 ++++++++++-
 .../net/ethernet/freescale/enetc/enetc_qos.c  |  16 +-
 11 files changed, 349 insertions(+), 13 deletions(-)
 create mode 100644 drivers/net/ethernet/freescale/enetc/enetc_ierb.c
 create mode 100644 drivers/net/ethernet/freescale/enetc/enetc_ierb.h

Comments

Jakub Kicinski April 19, 2021, 9:04 p.m. UTC | #1
On Sat, 17 Apr 2021 02:42:20 +0300 Vladimir Oltean wrote:
> From: Vladimir Oltean <vladimir.oltean@nxp.com>
> 
> This patch series contains logic for enabling the lossless mode on the
> RX rings of the ENETC, and the PAUSE thresholds on the internal FIFO
> memory.
> 
> During testing it was found that, with the default FIFO configuration,
> a sender which isn't persuaded by our PAUSE frames and keeps sending
> will cause some MAC RX frame errors. To mitigate this, we need to ensure
> that the FIFO never runs completely full, so we need to fix up a setting
> that was supposed to be configured well out of reset. Unfortunately this
> requires the addition of a new mini-driver.

FWIW back in the day when I was working on more advanced devices than 
I deal with these days I was expecting to eventually run into this as
well and create some form of devlink umbrella. IMHO such "mini driver"
is a natural place for a devlink instance, and not the PFs/ports.
Is this your thinking as well? AFAICT enetc doesn't implement devlink
today so you start from whatever model works best without worrying
about backward compat.
Vladimir Oltean April 19, 2021, 9:49 p.m. UTC | #2
Hi Jakub,

On Mon, Apr 19, 2021 at 02:04:42PM -0700, Jakub Kicinski wrote:
> On Sat, 17 Apr 2021 02:42:20 +0300 Vladimir Oltean wrote:
> > From: Vladimir Oltean <vladimir.oltean@nxp.com>
> > 
> > This patch series contains logic for enabling the lossless mode on the
> > RX rings of the ENETC, and the PAUSE thresholds on the internal FIFO
> > memory.
> > 
> > During testing it was found that, with the default FIFO configuration,
> > a sender which isn't persuaded by our PAUSE frames and keeps sending
> > will cause some MAC RX frame errors. To mitigate this, we need to ensure
> > that the FIFO never runs completely full, so we need to fix up a setting
> > that was supposed to be configured well out of reset. Unfortunately this
> > requires the addition of a new mini-driver.
> 
> FWIW back in the day when I was working on more advanced devices than 
> I deal with these days I was expecting to eventually run into this as
> well and create some form of devlink umbrella. IMHO such "mini driver"
> is a natural place for a devlink instance, and not the PFs/ports.
> Is this your thinking as well? AFAICT enetc doesn't implement devlink
> today so you start from whatever model works best without worrying
> about backward compat.

Sorry, but I am not sure if I understood the central idea of what you
were trying to transmit. What is 'a devlink instance and not the PFs'?
I am not aware of how a single devlink instance can be exposed for a
piece of hardware presenting itself as multiple PFs with multiple driver
instances running asynchronously and potentially being assigned to AMP
software execution environments (other cores running non-Linux, and most
probably Linux is not even the privileged execution environment which
has write access to the FIFO parameters).
Are you suggesting that the FIFO size and partitioning characteristics
be exposed through the devlink subsystem? Isn't that what devlink-sb is
for? Also, that would not help with what the IERB driver is trying to
achieve. There isn't anything we want the user to view or fiddle with,
the reality is simply that the FIFO parameters were supposed to be
one-size-fits-all-and-nobody-cares-about-them (the memory usage scheme
of this NIC is smart enough to allow for that, or so I think) but
nonetheless, the hardware defaults need to be touched up. If LS1028A was
a new SoC today we would have probably done this from U-Boot, from the
same logic that already passes the MAC addresses to the PFs through the
IERB, but the ship has kind of sailed for that, bootloaders are stable,
and 'Linux needs this feature' is not a good reason to update them.
So this is all that I would like the IERB driver to do, notice how it's
all writes of predefined values but no reads. For next generation SoCs
with ENETC we'll try our best to not need an IERB driver in Linux at
all. Another option would have been to do these fixups in the arch init
code as a sort of erratum workaround, but I didn't find a place similar
to arch/arm/mach-* for arm64, so I assumed that the arm64 port just
doesn't want to go that route. So here I am with a driver for some
memory writes.
patchwork-bot+netdevbpf@kernel.org April 19, 2021, 10:40 p.m. UTC | #3
Hello:

This series was applied to netdev/net-next.git (refs/heads/master):

On Sat, 17 Apr 2021 02:42:20 +0300 you wrote:
> From: Vladimir Oltean <vladimir.oltean@nxp.com>
> 
> This patch series contains logic for enabling the lossless mode on the
> RX rings of the ENETC, and the PAUSE thresholds on the internal FIFO
> memory.
> 
> During testing it was found that, with the default FIFO configuration,
> a sender which isn't persuaded by our PAUSE frames and keeps sending
> will cause some MAC RX frame errors. To mitigate this, we need to ensure
> that the FIFO never runs completely full, so we need to fix up a setting
> that was supposed to be configured well out of reset. Unfortunately this
> requires the addition of a new mini-driver.
> 
> [...]

Here is the summary with links:
  - [net-next,1/5] net: enetc: create a common enetc_pf_to_port helper
    https://git.kernel.org/netdev/net-next/c/87614b931c24
  - [net-next,2/5] dt-bindings: net: fsl: enetc: add the IERB documentation
    https://git.kernel.org/netdev/net-next/c/4ac7acc67f29
  - [net-next,3/5] net: enetc: add a mini driver for the Integrated Endpoint Register Block
    https://git.kernel.org/netdev/net-next/c/e7d48e5fbf30
  - [net-next,4/5] arm64: dts: ls1028a: declare the Integrated Endpoint Register Block node
    https://git.kernel.org/netdev/net-next/c/b764dc6cc1ba
  - [net-next,5/5] net: enetc: add support for flow control
    https://git.kernel.org/netdev/net-next/c/a8648887880f

You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
Michael Walle April 20, 2021, 1:27 p.m. UTC | #4
Hi Vladimir,

Am 2021-04-17 01:42, schrieb Vladimir Oltean:
> From: Vladimir Oltean <vladimir.oltean@nxp.com>
> 
> This patch series contains logic for enabling the lossless mode on the
> RX rings of the ENETC, and the PAUSE thresholds on the internal FIFO
> memory.
> 
> During testing it was found that, with the default FIFO configuration,
> a sender which isn't persuaded by our PAUSE frames and keeps sending
> will cause some MAC RX frame errors. To mitigate this, we need to 
> ensure
> that the FIFO never runs completely full, so we need to fix up a 
> setting
> that was supposed to be configured well out of reset. Unfortunately 
> this
> requires the addition of a new mini-driver.

What happens if the mini driver is not enabled? Then the fixes aren't
applied and bad things happen (now with the addition of flow control),
right?

I'm asking because, if you have the arm64 defconfig its not enabled.

shouldn't it be something like:

diff --git a/drivers/net/ethernet/freescale/enetc/Kconfig 
b/drivers/net/ethernet/freescale/enetc/Kconfig
index d88f60c2bb82..cdc0ff89388a 100644
--- a/drivers/net/ethernet/freescale/enetc/Kconfig
+++ b/drivers/net/ethernet/freescale/enetc/Kconfig
@@ -2,7 +2,7 @@
  config FSL_ENETC
         tristate "ENETC PF driver"
         depends on PCI && PCI_MSI
-       depends on FSL_ENETC_IERB || FSL_ENETC_IERB=n
+       select FSL_ENETC_IERB
         select FSL_ENETC_MDIO
         select PHYLINK
         select PCS_LYNX

-michael
Vladimir Oltean April 20, 2021, 2:04 p.m. UTC | #5
Hi Michael,

On Tue, Apr 20, 2021 at 03:27:24PM +0200, Michael Walle wrote:
> Hi Vladimir,
> 
> Am 2021-04-17 01:42, schrieb Vladimir Oltean:
> > From: Vladimir Oltean <vladimir.oltean@nxp.com>
> > 
> > This patch series contains logic for enabling the lossless mode on the
> > RX rings of the ENETC, and the PAUSE thresholds on the internal FIFO
> > memory.
> > 
> > During testing it was found that, with the default FIFO configuration,
> > a sender which isn't persuaded by our PAUSE frames and keeps sending
> > will cause some MAC RX frame errors. To mitigate this, we need to ensure
> > that the FIFO never runs completely full, so we need to fix up a setting
> > that was supposed to be configured well out of reset. Unfortunately this
> > requires the addition of a new mini-driver.
> 
> What happens if the mini driver is not enabled? Then the fixes aren't
> applied and bad things happen (now with the addition of flow control),
> right?
> 
> I'm asking because, if you have the arm64 defconfig its not enabled.
> 
> shouldn't it be something like:
> 
> diff --git a/drivers/net/ethernet/freescale/enetc/Kconfig
> b/drivers/net/ethernet/freescale/enetc/Kconfig
> index d88f60c2bb82..cdc0ff89388a 100644
> --- a/drivers/net/ethernet/freescale/enetc/Kconfig
> +++ b/drivers/net/ethernet/freescale/enetc/Kconfig
> @@ -2,7 +2,7 @@
>  config FSL_ENETC
>         tristate "ENETC PF driver"
>         depends on PCI && PCI_MSI
> -       depends on FSL_ENETC_IERB || FSL_ENETC_IERB=n
> +       select FSL_ENETC_IERB
>         select FSL_ENETC_MDIO
>         select PHYLINK
>         select PCS_LYNX

Yes, ideally the IERB driver and the ENETC PF driver should be built in
the same way, or the IERB driver can be built-in and the PF driver can
be module. I don't know how to express this using Kconfig, sorry.
Michael Walle April 20, 2021, 2:10 p.m. UTC | #6
Hi Vladimir,

Am 2021-04-20 16:04, schrieb Vladimir Oltean:
> On Tue, Apr 20, 2021 at 03:27:24PM +0200, Michael Walle wrote:
>> Hi Vladimir,
>> 
>> Am 2021-04-17 01:42, schrieb Vladimir Oltean:
>> > From: Vladimir Oltean <vladimir.oltean@nxp.com>
>> >
>> > This patch series contains logic for enabling the lossless mode on the
>> > RX rings of the ENETC, and the PAUSE thresholds on the internal FIFO
>> > memory.
>> >
>> > During testing it was found that, with the default FIFO configuration,
>> > a sender which isn't persuaded by our PAUSE frames and keeps sending
>> > will cause some MAC RX frame errors. To mitigate this, we need to ensure
>> > that the FIFO never runs completely full, so we need to fix up a setting
>> > that was supposed to be configured well out of reset. Unfortunately this
>> > requires the addition of a new mini-driver.
>> 
>> What happens if the mini driver is not enabled? Then the fixes aren't
>> applied and bad things happen (now with the addition of flow control),
>> right?
>> 
>> I'm asking because, if you have the arm64 defconfig its not enabled.
>> 
>> shouldn't it be something like:
>> 
>> diff --git a/drivers/net/ethernet/freescale/enetc/Kconfig
>> b/drivers/net/ethernet/freescale/enetc/Kconfig
>> index d88f60c2bb82..cdc0ff89388a 100644
>> --- a/drivers/net/ethernet/freescale/enetc/Kconfig
>> +++ b/drivers/net/ethernet/freescale/enetc/Kconfig
>> @@ -2,7 +2,7 @@
>>  config FSL_ENETC
>>         tristate "ENETC PF driver"
>>         depends on PCI && PCI_MSI
>> -       depends on FSL_ENETC_IERB || FSL_ENETC_IERB=n
>> +       select FSL_ENETC_IERB
>>         select FSL_ENETC_MDIO
>>         select PHYLINK
>>         select PCS_LYNX
> 
> Yes, ideally the IERB driver and the ENETC PF driver should be built in
> the same way, or the IERB driver can be built-in and the PF driver can
> be module. I don't know how to express this using Kconfig, sorry.

With the small patch above it is:
  FSL_ENETC=m -> FSL_ENETC_IERB = m or y
  FSL_ENETC=y -> FSL_ENETC_IERB = y
  FSL_ENETC=n -> FSL_ENETC_IERB = m,y or n

Will you fix it? Should I prepare a patch?

-michael