From patchwork Wed Sep 12 09:02:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilias Apalodimas X-Patchwork-Id: 146525 Delivered-To: patch@linaro.org Received: by 2002:a2e:1648:0:0:0:0:0 with SMTP id 8-v6csp4574715ljw; Wed, 12 Sep 2018 02:02:55 -0700 (PDT) X-Google-Smtp-Source: ANB0VdaAVlZSaIm8pwRBqATTb51XqSdxjWMZKsHC2zL00TI/w1iEiXVHqBfK4WEYv3McpKbZGMkj X-Received: by 2002:a62:3703:: with SMTP id e3-v6mr1054084pfa.117.1536742975073; Wed, 12 Sep 2018 02:02:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536742975; cv=none; d=google.com; s=arc-20160816; b=WQc8l6wAQtjMZMxBmdaQyDLOyz374h8J49eA8I18kukPVXMttXx4cMJkLcj+YMrhMm nK8r15t1yLxdZ6eODUll/+kMVuw6RbkboqLRMQpY79GyO3Ce1UK+fNvSNw5xlvTSGTzN 2OZnffFmILOdARubAB6x8nuYvKWYEGljcxksEgay8ao13yzcL7Bk5aAzYSnoQNyQEHzY AfzfPdkJkxP8gSlM94XLZg+5S+SQi0Hdu53eHltBs/URPIc2hDlv24guUqEgsKKFBnOD 8ccJ+npV2nTc9jzA+s89LLs8WSBQqK87aBNSPwdWXAfPD9A6JSAl3ifZqA5s5RtFajm+ canw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=MEOo/Yl+JX/iyRwt+ayYOIk7joRaj6bXpX27aMmFzk4=; b=ZBzcf9uHntk6KbJ9cwkExzAq1nOI09KNMm54DwSk8+u0f2+JqA0+oL5NaPxwfyCwR9 A5zIegmHQJHUYh3NCIauw5ZaTFeQhzMxBTdXDf1yKM1oU/v/8MsnPht6dHXhMWKpC84w nAWkbhfmNiu7Xn5rRo4m9tHJd4CncEG4VhVQdkdLGtfvD3I4DDqG4pEZ+vzNGXGnPkoV KhkmT1NM1epckxV6G6DyetjkWF+OfFUno2Xo8f0H2EZh7+XwV1ltRv6NVD6K3DlVoOau nv5XSX1+VPij0wgtn4dQhDlc/bSD2K/52ZvGp/hCgyS4n3Thoa8ui4bEEc+N/cmQptuf qhbA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=WwrUvsYz; spf=pass (google.com: best guess record for domain of netdev-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=netdev-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id w14-v6si415729plp.183.2018.09.12.02.02.54; Wed, 12 Sep 2018 02:02:55 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of netdev-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=WwrUvsYz; spf=pass (google.com: best guess record for domain of netdev-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=netdev-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727876AbeILOG2 (ORCPT + 10 others); Wed, 12 Sep 2018 10:06:28 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:42678 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726919AbeILOG2 (ORCPT ); Wed, 12 Sep 2018 10:06:28 -0400 Received: by mail-wr1-f68.google.com with SMTP id v17-v6so1148012wrr.9 for ; Wed, 12 Sep 2018 02:02:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=MEOo/Yl+JX/iyRwt+ayYOIk7joRaj6bXpX27aMmFzk4=; b=WwrUvsYzRlaDupbtqguRE8EhritIVSlw0dxZQYN7zJQcZfIVYI0yxZJfH+Z/7b6cju p/B8+/EQsByiKQnkuUTfiqLAK5uSi8IkDC3I1VuWM4PGq/9060kPa3P4QDFNFMgWXcAi sWLKCFw2C7Bdu2vBhwxcDP/NDHWww7vQbssN8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=MEOo/Yl+JX/iyRwt+ayYOIk7joRaj6bXpX27aMmFzk4=; b=Enzl2zJwG/U9sCmbXlxeTJ76V4ZqPkM2qj7C42Hc3RcKIgu4aNAmL3aZKJqi5bgg/d Ke0wvo/eig/Fz73XxKgq/ZjyRG7K1kgkfBwpy8MLdpS6jT6xB0LvzIVuymtH7iIOv1+6 1RTE6gEaHa1bm2q4sNbhZSwnqws9cOnC2TWO19EIlpnya0nH+DrWvvFcryAEefava4bN ESSpikYi9JtVWbDjnq/a03dSzFcDF9Auvb+XWZWk27U3p2PELJxDQbQOwVoVcpccr4v/ vfgVh1/9BUAayTtFKP1A3EEP9jLmiyyY0W4sdD2pmaQbp20+/Fby/SSMYu/aejvTPhnq aVIg== X-Gm-Message-State: APzg51AvyOs962fsU+GoFPZsM4CylyKyuxZDhWPXY/tlsiXZQ01wNNt9 ZH0W+tTwNViB8Uh2Lk/bakhE0OPGYts= X-Received: by 2002:adf:ef51:: with SMTP id c17-v6mr733870wrp.182.1536742971102; Wed, 12 Sep 2018 02:02:51 -0700 (PDT) Received: from localhost.localdomain ([2a02:587:4607:6a00:2e56:dcff:fe9a:8f06]) by smtp.gmail.com with ESMTPSA id x15-v6sm301572wrt.53.2018.09.12.02.02.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 12 Sep 2018 02:02:50 -0700 (PDT) From: Ilias Apalodimas To: netdev@vger.kernel.org, jaswinder.singh@linaro.org Cc: ard.biesheuvel@linaro.org, masami.hiramatsu@linaro.org, arnd@arndb.de, mykyta.iziumtsev@linaro.org, bjorn.topel@intel.com, magnus.karlsson@intel.com, brouer@redhat.com, daniel@iogearbox.net, ast@kernel.org, Ilias Apalodimas Subject: [net-next, PATCH 2/2, v2] net: socionext: add XDP support Date: Wed, 12 Sep 2018 12:02:38 +0300 Message-Id: <1536742958-29887-3-git-send-email-ilias.apalodimas@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1536742958-29887-1-git-send-email-ilias.apalodimas@linaro.org> References: <1536742958-29887-1-git-send-email-ilias.apalodimas@linaro.org> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Add basic XDP support Signed-off-by: Ilias Apalodimas --- drivers/net/ethernet/socionext/netsec.c | 234 +++++++++++++++++++++++++++++--- 1 file changed, 216 insertions(+), 18 deletions(-) -- 2.7.4 diff --git a/drivers/net/ethernet/socionext/netsec.c b/drivers/net/ethernet/socionext/netsec.c index 666fee2..1f4594f 100644 --- a/drivers/net/ethernet/socionext/netsec.c +++ b/drivers/net/ethernet/socionext/netsec.c @@ -9,6 +9,9 @@ #include #include #include +#include +#include +#include #include #include @@ -238,6 +241,11 @@ #define NETSEC_F_NETSEC_VER_MAJOR_NUM(x) ((x) & 0xffff0000) +#define NETSEC_XDP_PASS 0 +#define NETSEC_XDP_CONSUMED BIT(0) +#define NETSEC_XDP_TX BIT(1) +#define NETSEC_XDP_REDIR BIT(2) + enum ring_id { NETSEC_RING_TX = 0, NETSEC_RING_RX @@ -256,11 +264,14 @@ struct netsec_desc_ring { void *vaddr; u16 pkt_cnt; u16 head, tail; + bool is_xdp; + struct xdp_rxq_info xdp_rxq; }; struct netsec_priv { struct netsec_desc_ring desc_ring[NETSEC_RING_MAX]; struct ethtool_coalesce et_coalesce; + struct bpf_prog *xdp_prog; spinlock_t reglock; /* protect reg access */ struct napi_struct napi; phy_interface_t phy_interface; @@ -297,6 +308,8 @@ struct netsec_rx_pkt_info { }; static void netsec_rx_fill(struct netsec_priv *priv, u16 from, u16 num); +static u32 netsec_run_xdp(struct netsec_desc *desc, struct netsec_priv *priv, + struct bpf_prog *prog, struct xdp_buff *xdp); static void *netsec_alloc_rx_data(struct netsec_priv *priv, dma_addr_t *dma_addr, u16 *len); @@ -613,13 +626,23 @@ static int netsec_clean_tx_dring(struct netsec_priv *priv, int budget) eop = (entry->attr >> NETSEC_TX_LAST) & 1; - dma_unmap_single(priv->dev, desc->dma_addr, desc->len, - DMA_TO_DEVICE); - if (eop) { - pkts++; + if (desc->skb) + dma_unmap_single(priv->dev, + desc->dma_addr - XDP_PACKET_HEADROOM, + desc->len, DMA_TO_DEVICE); + + if (!eop) { + *desc = (struct netsec_desc){}; + continue; + } + + if (!desc->skb) { + skb_free_frag(desc->addr); + } else { bytes += desc->skb->len; dev_kfree_skb(desc->skb); } + pkts++; *desc = (struct netsec_desc){}; } dring->pkt_cnt -= budget; @@ -659,19 +682,22 @@ static void nsetsec_adv_desc(u16 *idx) static int netsec_process_rx(struct netsec_priv *priv, int budget) { struct netsec_desc_ring *dring = &priv->desc_ring[NETSEC_RING_RX]; + struct bpf_prog *xdp_prog = READ_ONCE(priv->xdp_prog); struct net_device *ndev = priv->ndev; - struct sk_buff *skb; + struct sk_buff *skb = NULL; + u32 xdp_flush = 0; + u32 xdp_result; int done = 0; while (done < budget) { u16 idx = dring->tail; struct netsec_de *de = dring->vaddr + (DESC_SZ * idx); struct netsec_desc *desc = &dring->desc[idx]; + dma_addr_t dma_handle, dma_unmap; struct netsec_rx_pkt_info rpi; - dma_addr_t dma_handle; + u16 pkt_len, desc_len; + struct xdp_buff xdp; void *buf_addr; - u16 pkt_len; - u16 desc_len; if (de->attr & (1U << NETSEC_RX_PKT_OWN_FIELD)) break; @@ -704,10 +730,40 @@ static int netsec_process_rx(struct netsec_priv *priv, int budget) prefetch(desc->addr); buf_addr = netsec_alloc_rx_data(priv, &dma_handle, &desc_len); + if (unlikely(!buf_addr)) break; - skb = build_skb(desc->addr, desc->len); + dma_unmap = dring->is_xdp ? + desc->dma_addr - XDP_PACKET_HEADROOM : desc->dma_addr; + + xdp.data_hard_start = desc->addr; + xdp.data = desc->addr; + xdp_set_data_meta_invalid(&xdp); + xdp.data_end = xdp.data + pkt_len; + xdp.rxq = &dring->xdp_rxq; + + if (xdp_prog) { + xdp.data = desc->addr + XDP_PACKET_HEADROOM; + xdp.data_end = xdp.data + pkt_len; + xdp_result = netsec_run_xdp(desc, priv, xdp_prog, &xdp); + if (xdp_result != NETSEC_XDP_PASS) { + xdp_flush |= xdp_result & NETSEC_XDP_REDIR; + + dma_unmap_single_attrs(priv->dev, dma_unmap, + desc->len, DMA_TO_DEVICE, + DMA_ATTR_SKIP_CPU_SYNC); + + desc->len = desc_len; + desc->dma_addr = dma_handle; + desc->addr = buf_addr; + netsec_rx_fill(priv, idx, 1); + nsetsec_adv_desc(&dring->tail); + continue; + } + } + + skb = build_skb(xdp.data_hard_start, desc->len); if (unlikely(!skb)) { dma_unmap_single(priv->dev, dma_handle, desc_len, DMA_TO_DEVICE); @@ -716,7 +772,7 @@ static int netsec_process_rx(struct netsec_priv *priv, int budget) "rx failed to alloc skb\n"); break; } - dma_unmap_single_attrs(priv->dev, desc->dma_addr, desc->len, + dma_unmap_single_attrs(priv->dev, dma_unmap, desc->len, DMA_TO_DEVICE, DMA_ATTR_SKIP_CPU_SYNC); /* Update the descriptor with fresh buffers */ @@ -724,7 +780,8 @@ static int netsec_process_rx(struct netsec_priv *priv, int budget) desc->dma_addr = dma_handle; desc->addr = buf_addr; - skb_put(skb, pkt_len); + skb_reserve(skb, xdp.data - xdp.data_hard_start); + skb_put(skb, xdp.data_end - xdp.data); skb->protocol = eth_type_trans(skb, priv->ndev); if (priv->rx_cksum_offload_flag && @@ -733,13 +790,16 @@ static int netsec_process_rx(struct netsec_priv *priv, int budget) if (napi_gro_receive(&priv->napi, skb) != GRO_DROP) { ndev->stats.rx_packets++; - ndev->stats.rx_bytes += pkt_len; + ndev->stats.rx_bytes += xdp.data_end - xdp.data; } netsec_rx_fill(priv, idx, 1); nsetsec_adv_desc(&dring->tail); } + if (xdp_flush & NETSEC_XDP_REDIR) + xdp_do_flush_map(); + return done; } @@ -892,6 +952,9 @@ static void netsec_uninit_pkt_dring(struct netsec_priv *priv, int id) if (!dring->vaddr || !dring->desc) return; + if (xdp_rxq_info_is_reg(&dring->xdp_rxq)) + xdp_rxq_info_unreg(&dring->xdp_rxq); + for (idx = 0; idx < DESC_NUM; idx++) { desc = &dring->desc[idx]; if (!desc->addr) @@ -931,11 +994,14 @@ static void netsec_free_dring(struct netsec_priv *priv, int id) static void *netsec_alloc_rx_data(struct netsec_priv *priv, dma_addr_t *dma_handle, u16 *desc_len) { + struct netsec_desc_ring *dring = &priv->desc_ring[NETSEC_RING_RX]; size_t len = priv->ndev->mtu + ETH_HLEN + VLAN_HLEN * 2 + NET_SKB_PAD + NET_IP_ALIGN; dma_addr_t mapping; void *buf; + if (dring->is_xdp) + len += XDP_PACKET_HEADROOM; len = SKB_DATA_ALIGN(len); len += SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); @@ -943,11 +1009,12 @@ static void *netsec_alloc_rx_data(struct netsec_priv *priv, if (!buf) return NULL; - mapping = dma_map_single(priv->dev, buf, len, DMA_FROM_DEVICE); + mapping = dma_map_single(priv->dev, buf, len, + DMA_FROM_DEVICE); if (unlikely(dma_mapping_error(priv->dev, mapping))) goto err_out; - *dma_handle = mapping; + *dma_handle = mapping + (dring->is_xdp ? XDP_PACKET_HEADROOM : 0); *desc_len = len; return buf; @@ -994,7 +1061,13 @@ static int netsec_alloc_dring(struct netsec_priv *priv, enum ring_id id) static int netsec_setup_rx_dring(struct netsec_priv *priv) { struct netsec_desc_ring *dring = &priv->desc_ring[NETSEC_RING_RX]; - int i; + struct bpf_prog *xdp_prog = READ_ONCE(priv->xdp_prog); + int i, err; + + if (xdp_prog) + dring->is_xdp = true; + else + dring->is_xdp = false; for (i = 0; i < DESC_NUM; i++) { struct netsec_desc *desc = &dring->desc[i]; @@ -1003,20 +1076,29 @@ static int netsec_setup_rx_dring(struct netsec_priv *priv) u16 len; buf = netsec_alloc_rx_data(priv, &dma_handle, &len); - if (!buf) { - netsec_uninit_pkt_dring(priv, NETSEC_RING_RX); + if (!buf) goto err_out; - } desc->dma_addr = dma_handle; desc->addr = buf; desc->len = len; } netsec_rx_fill(priv, 0, DESC_NUM); + err = xdp_rxq_info_reg(&dring->xdp_rxq, priv->ndev, 0); + if (err) + goto err_out; + + err = xdp_rxq_info_reg_mem_model(&dring->xdp_rxq, MEM_TYPE_PAGE_SHARED, + NULL); + if (err) { + xdp_rxq_info_unreg(&dring->xdp_rxq); + goto err_out; + } return 0; err_out: + netsec_uninit_pkt_dring(priv, NETSEC_RING_RX); return -ENOMEM; } @@ -1420,6 +1502,121 @@ static int netsec_netdev_ioctl(struct net_device *ndev, struct ifreq *ifr, return phy_mii_ioctl(ndev->phydev, ifr, cmd); } +static u32 netsec_xmit_xdp(struct netsec_priv *priv, struct xdp_buff *xdp, + struct netsec_desc *rx_desc) +{ + struct netsec_desc_ring *tx_ring = &priv->desc_ring[NETSEC_RING_TX]; + struct netsec_tx_pkt_ctrl tx_ctrl = {}; + struct netsec_desc tx_desc; + int filled; + u32 len; + + len = xdp->data_end - xdp->data; + + if (tx_ring->head >= tx_ring->tail) + filled = tx_ring->head - tx_ring->tail; + else + filled = tx_ring->head + DESC_NUM - tx_ring->tail; + + if (DESC_NUM - filled <= 1) + return NETSEC_XDP_CONSUMED; + + dma_sync_single_for_device(priv->dev, rx_desc->dma_addr, len, + DMA_TO_DEVICE); + + tx_desc.dma_addr = rx_desc->dma_addr; + tx_desc.addr = xdp->data; + tx_desc.len = len; + + netsec_set_tx_de(priv, tx_ring, &tx_ctrl, &tx_desc, NULL); + netsec_write(priv, NETSEC_REG_NRM_TX_PKTCNT, 1); + + return NETSEC_XDP_TX; +} + +static u32 netsec_run_xdp(struct netsec_desc *desc, struct netsec_priv *priv, + struct bpf_prog *prog, struct xdp_buff *xdp) +{ + u32 ret = NETSEC_XDP_PASS; + int err; + u32 act; + + rcu_read_lock(); + act = bpf_prog_run_xdp(prog, xdp); + + switch (act) { + case XDP_PASS: + ret = NETSEC_XDP_PASS; + break; + case XDP_TX: + ret = netsec_xmit_xdp(priv, xdp, desc); + break; + case XDP_REDIRECT: + err = xdp_do_redirect(priv->ndev, xdp, prog); + if (!err) { + ret = NETSEC_XDP_REDIR; + } else { + ret = NETSEC_XDP_CONSUMED; + xdp_return_buff(xdp); + } + break; + default: + bpf_warn_invalid_xdp_action(act); + /* fall through */ + case XDP_ABORTED: + trace_xdp_exception(priv->ndev, prog, act); + /* fall through -- handle aborts by dropping packet */ + case XDP_DROP: + ret = NETSEC_XDP_CONSUMED; + break; + } + + rcu_read_unlock(); + + return ret; +} + +static int netsec_xdp_setup(struct netsec_priv *priv, struct bpf_prog *prog, + struct netlink_ext_ack *extack) +{ + struct net_device *dev = priv->ndev; + struct bpf_prog *old_prog; + + /* For now just support only the usual MTU sized frames */ + if (prog && dev->mtu > 1500) { + NL_SET_ERR_MSG_MOD(extack, "Jumbo frames not supported on XDP"); + return -EOPNOTSUPP; + } + + if (netif_running(dev)) + netsec_netdev_stop(dev); + + /* Detach old prog, if any */ + old_prog = xchg(&priv->xdp_prog, prog); + if (old_prog) + bpf_prog_put(old_prog); + + if (netif_running(dev)) + netsec_netdev_open(dev); + + return 0; +} + +static int netsec_xdp(struct net_device *ndev, struct netdev_bpf *xdp) +{ + struct netsec_priv *priv = netdev_priv(ndev); + + switch (xdp->command) { + case XDP_SETUP_PROG: + return netsec_xdp_setup(priv, xdp->prog, xdp->extack); + case XDP_QUERY_PROG: + xdp->prog_id = priv->xdp_prog ? priv->xdp_prog->aux->id : 0; + return 0; + default: + return -EINVAL; + } +} + static const struct net_device_ops netsec_netdev_ops = { .ndo_init = netsec_netdev_init, .ndo_uninit = netsec_netdev_uninit, @@ -1430,6 +1627,7 @@ static const struct net_device_ops netsec_netdev_ops = { .ndo_set_mac_address = eth_mac_addr, .ndo_validate_addr = eth_validate_addr, .ndo_do_ioctl = netsec_netdev_ioctl, + .ndo_bpf = netsec_xdp, }; static int netsec_of_probe(struct platform_device *pdev,