From patchwork Tue Aug 5 14:54:05 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Venkatesh Vivekanandan X-Patchwork-Id: 34954 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-oa0-f70.google.com (mail-oa0-f70.google.com [209.85.219.70]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 9E11120523 for ; Tue, 5 Aug 2014 14:54:44 +0000 (UTC) Received: by mail-oa0-f70.google.com with SMTP id eb12sf4432750oac.9 for ; Tue, 05 Aug 2014 07:54:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:from:to:date:message-id :mime-version:subject:precedence:list-id:list-unsubscribe :list-archive:list-post:list-help:list-subscribe:errors-to:sender :x-original-sender:x-original-authentication-results:mailing-list :content-type:content-transfer-encoding; bh=LTbEBLu+j4W0tVb05ERjE+nLERbA84mqszmJwyx6Vcs=; b=CfmOF6oxgxLG5rcSvC8wVszBAG+oeOO6GSusYspwon41WZtuCEIiIqZkCrhg/OJ7Fw QKIJfyKb5aAxaIhSfmGaAclmbbGNG1wYPnVjVz52sUa60CqUMRo5mu6kQs0/CRzTkQJ6 MWKpnJtDl0kGHFVP4VolfV9LGoYtcBhAl9mrUgo0dc8D1a/p3GQ5+Zc6JlWhFoaTTyEP 2RVkqvAHMEwwwxvIBZl1aNlIRblrhM4hwIMGoEYMWEA+yymHzZGZrKaUqNJxYlK/CJY8 xtlQTDFgR2Id5jwbu4urNuJkvnvRcwLxdPYHUrQAfPb1XshjG6AO6eqCtJOHoF1O5Oo0 kuHw== X-Gm-Message-State: ALoCoQlpPOQ3miJyRVDIfe4TK/bE+X6sT0vS557lf6ue10fG2MN4yZaDYnZJzu4ztr/+dLDgAgb7 X-Received: by 10.42.224.68 with SMTP id in4mr1302004icb.33.1407250484232; Tue, 05 Aug 2014 07:54:44 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.140.46.52 with SMTP id j49ls335181qga.68.gmail; Tue, 05 Aug 2014 07:54:44 -0700 (PDT) X-Received: by 10.52.185.193 with SMTP id fe1mr3536684vdc.31.1407250484068; Tue, 05 Aug 2014 07:54:44 -0700 (PDT) Received: from mail-vc0-f182.google.com (mail-vc0-f182.google.com [209.85.220.182]) by mx.google.com with ESMTPS id m8si1265095vcj.67.2014.08.05.07.54.44 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 05 Aug 2014 07:54:44 -0700 (PDT) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.220.182 as permitted sender) client-ip=209.85.220.182; Received: by mail-vc0-f182.google.com with SMTP id hy4so1708489vcb.13 for ; Tue, 05 Aug 2014 07:54:43 -0700 (PDT) X-Received: by 10.220.97.5 with SMTP id j5mr4279056vcn.16.1407250483917; Tue, 05 Aug 2014 07:54:43 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.221.37.5 with SMTP id tc5csp395452vcb; Tue, 5 Aug 2014 07:54:41 -0700 (PDT) X-Received: by 10.224.122.83 with SMTP id k19mr6602308qar.78.1407250481209; Tue, 05 Aug 2014 07:54:41 -0700 (PDT) Received: from ip-10-141-164-156.ec2.internal (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTPS id o9si3261942qgd.72.2014.08.05.07.54.40 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 05 Aug 2014 07:54:41 -0700 (PDT) Received-SPF: none (google.com: lng-odp-bounces@lists.linaro.org does not designate permitted sender hosts) client-ip=54.225.227.206; Received: from localhost ([127.0.0.1] helo=ip-10-141-164-156.ec2.internal) by ip-10-141-164-156.ec2.internal with esmtp (Exim 4.76) (envelope-from ) id 1XEg8O-00054W-31; Tue, 05 Aug 2014 14:54:40 +0000 Received: from mail-gw3-out.broadcom.com ([216.31.210.64]) by ip-10-141-164-156.ec2.internal with esmtp (Exim 4.76) (envelope-from ) id 1XEg8I-000542-9Y for lng-odp@lists.linaro.org; Tue, 05 Aug 2014 14:54:34 +0000 X-IronPort-AV: E=Sophos;i="5.01,805,1400050800"; d="scan'208";a="41386162" Received: from irvexchcas07.broadcom.com (HELO IRVEXCHCAS07.corp.ad.broadcom.com) ([10.9.208.55]) by mail-gw3-out.broadcom.com with ESMTP; 05 Aug 2014 08:06:28 -0700 Received: from IRVEXCHSMTP2.corp.ad.broadcom.com (10.9.207.52) by IRVEXCHCAS07.corp.ad.broadcom.com (10.9.208.55) with Microsoft SMTP Server (TLS) id 14.3.174.1; Tue, 5 Aug 2014 07:54:28 -0700 Received: from mail-irva-13.broadcom.com (10.10.10.20) by IRVEXCHSMTP2.corp.ad.broadcom.com (10.9.207.52) with Microsoft SMTP Server id 14.3.174.1; Tue, 5 Aug 2014 07:54:28 -0700 Received: from localhost.localdomain (unknown [10.131.61.80]) by mail-irva-13.broadcom.com (Postfix) with ESMTP id 60ABA9F9F7; Tue, 5 Aug 2014 07:54:27 -0700 (PDT) From: To: Date: Tue, 5 Aug 2014 20:24:05 +0530 Message-ID: <1407250445-8940-1-git-send-email-venkatesh.vivekanandan@linaro.org> X-Mailer: git-send-email 1.8.1.2 MIME-Version: 1.0 X-Topics: patch Subject: [lng-odp] [PATCH 1/1] ODP-DPDK multi-queue support X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: , List-Help: , List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: lng-odp-bounces@lists.linaro.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: venkatesh.vivekanandan@linaro.org X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.220.182 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 From: Venkatesh Vivekanandan - Multi queue support per interface is enabled. - odp_pktio_send with "0" packet is called in odp_pktio_recv to give the transmitted buffers back to mempool. - mbuf alloc failure during receive is fixed by giving more buffers to mempool. - mempool cache size is given equivalent to MAX_PKT_BURST. Signed-off-by: Venkatesh Vivekanandan --- platform/linux-dpdk/include/odp_packet_dpdk.h | 24 +++++ platform/linux-dpdk/odp_buffer_pool.c | 4 +- platform/linux-dpdk/odp_packet_dpdk.c | 136 +++++++++++++------------- platform/linux-dpdk/odp_packet_io.c | 2 + 4 files changed, 94 insertions(+), 72 deletions(-) diff --git a/platform/linux-dpdk/include/odp_packet_dpdk.h b/platform/linux-dpdk/include/odp_packet_dpdk.h index bcbe9e8..bcf9aa5 100644 --- a/platform/linux-dpdk/include/odp_packet_dpdk.h +++ b/platform/linux-dpdk/include/odp_packet_dpdk.h @@ -50,6 +50,30 @@ #define DPDK_BLOCKING_IO +/* + * RX and TX Prefetch, Host, and Write-back threshold values should be + * carefully set for optimal performance. Consult the network + * controller's datasheet and supporting DPDK documentation for guidance + * on how these parameters should be set. + */ +#define RX_PTHRESH 8 /**< Default values of RX prefetch threshold reg. */ +#define RX_HTHRESH 8 /**< Default values of RX host threshold reg. */ +#define RX_WTHRESH 4 /**< Default values of RX write-back threshold reg. */ + +/* + * These default values are optimized for use with the Intel(R) 82599 10 GbE + * Controller and the DPDK ixgbe PMD. Consider using other values for other + * network controllers and/or network drivers. + */ +#define TX_PTHRESH 36 /**< Default values of TX prefetch threshold reg. */ +#define TX_HTHRESH 0 /**< Default values of TX host threshold reg. */ +#define TX_WTHRESH 0 /**< Default values of TX write-back threshold reg. */ + +#define MAX_PKT_BURST 16 +#define BURST_TX_DRAIN_US 100 /* TX drain every ~100us */ +#define RTE_TEST_RX_DESC_DEFAULT 128 +#define RTE_TEST_TX_DESC_DEFAULT 512 + /** Packet socket using dpdk mmaped rings for both Rx and Tx */ typedef struct { odp_buffer_pool_t pool; diff --git a/platform/linux-dpdk/odp_buffer_pool.c b/platform/linux-dpdk/odp_buffer_pool.c index de90275..805ce68 100644 --- a/platform/linux-dpdk/odp_buffer_pool.c +++ b/platform/linux-dpdk/odp_buffer_pool.c @@ -23,7 +23,7 @@ #include #define MBUF_SIZE (2048 + sizeof(struct rte_mbuf) + RTE_PKTMBUF_HEADROOM) -#define NB_MBUF 8192 +#define NB_MBUF 32768 #ifdef POOL_USE_TICKETLOCK #include @@ -112,7 +112,7 @@ odp_buffer_pool_t odp_buffer_pool_create(const char *name, pktmbuf_pool = rte_mempool_create(name, NB_MBUF, - MBUF_SIZE, 32, + MBUF_SIZE, MAX_PKT_BURST, sizeof(struct rte_pktmbuf_pool_private), rte_pktmbuf_pool_init, NULL, rte_pktmbuf_init, NULL, diff --git a/platform/linux-dpdk/odp_packet_dpdk.c b/platform/linux-dpdk/odp_packet_dpdk.c index 31bfa30..d5c8e80 100644 --- a/platform/linux-dpdk/odp_packet_dpdk.c +++ b/platform/linux-dpdk/odp_packet_dpdk.c @@ -26,34 +26,13 @@ #include #include -/* - * RX and TX Prefetch, Host, and Write-back threshold values should be - * carefully set for optimal performance. Consult the network - * controller's datasheet and supporting DPDK documentation for guidance - * on how these parameters should be set. - */ -#define RX_PTHRESH 8 /**< Default values of RX prefetch threshold reg. */ -#define RX_HTHRESH 8 /**< Default values of RX host threshold reg. */ -#define RX_WTHRESH 4 /**< Default values of RX write-back threshold reg. */ - -/* - * These default values are optimized for use with the Intel(R) 82599 10 GbE - * Controller and the DPDK ixgbe PMD. Consider using other values for other - * network controllers and/or network drivers. - */ -#define TX_PTHRESH 36 /**< Default values of TX prefetch threshold reg. */ -#define TX_HTHRESH 0 /**< Default values of TX host threshold reg. */ -#define TX_WTHRESH 0 /**< Default values of TX write-back threshold reg. */ - -#define MAX_PKT_BURST 16 -#define BURST_TX_DRAIN_US 100 /* TX drain every ~100us */ -#define RTE_TEST_RX_DESC_DEFAULT 128 -#define RTE_TEST_TX_DESC_DEFAULT 512 static uint16_t nb_rxd = RTE_TEST_RX_DESC_DEFAULT; static uint16_t nb_txd = RTE_TEST_TX_DESC_DEFAULT; static const struct rte_eth_conf port_conf = { .rxmode = { + .mq_mode = ETH_MQ_RX_RSS, + .max_rx_pkt_len = ETHER_MAX_LEN, .split_hdr_size = 0, .header_split = 0, /**< Header Split disabled */ .hw_ip_checksum = 0, /**< IP checksum offload disabled */ @@ -61,6 +40,12 @@ static const struct rte_eth_conf port_conf = { .jumbo_frame = 0, /**< Jumbo Frame Support disabled */ .hw_strip_crc = 0, /**< CRC stripped by hardware */ }, + .rx_adv_conf = { + .rss_conf = { + .rss_key = NULL, + .rss_hf = ETH_RSS_IPV4 | ETH_RSS_IPV6, + }, + }, .txmode = { .mq_mode = ETH_MQ_TX_NONE, }, @@ -95,60 +80,71 @@ int setup_pkt_dpdk(pkt_dpdk_t * const pkt_dpdk, const char *netdev, ODP_DBG("setup_pkt_dpdk\n"); static struct ether_addr eth_addr[RTE_MAX_ETHPORTS]; - uint8_t portid = 0; - uint16_t queueid = 0; - int ret; + static int portinit[RTE_MAX_ETHPORTS]; + static int qid[RTE_MAX_ETHPORTS]; + uint8_t portid = 0, num_intf = 2; + uint16_t nbrxq = 0, nbtxq = 0; + int ret, i; + printf("dpdk netdev: %s\n", netdev); printf("dpdk pool: %lx\n", pool); - portid = atoi(netdev); pkt_dpdk->portid = portid; - pkt_dpdk->queueid = queueid; pkt_dpdk->pool = pool; printf("dpdk portid: %u\n", portid); - fflush(stdout); - ret = rte_eth_dev_configure(portid, 1, 1, &port_conf); - if (ret < 0) - ODP_ERR("Cannot configure device: err=%d, port=%u\n", - ret, (unsigned) portid); - - rte_eth_macaddr_get(portid, ð_addr[portid]); - ODP_DBG("Port %u, MAC address: %02X:%02X:%02X:%02X:%02X:%02X\n\n", - (unsigned) portid, - eth_addr[portid].addr_bytes[0], - eth_addr[portid].addr_bytes[1], - eth_addr[portid].addr_bytes[2], - eth_addr[portid].addr_bytes[3], - eth_addr[portid].addr_bytes[4], - eth_addr[portid].addr_bytes[5]); - - /* init one RX queue on each port */ - fflush(stdout); - ret = rte_eth_rx_queue_setup(portid, queueid, nb_rxd, - rte_eth_dev_socket_id(portid), &rx_conf, - (struct rte_mempool *)pool); - if (ret < 0) - ODP_ERR("rte_eth_rx_queue_setup:err=%d, port=%u\n", - ret, (unsigned) portid); - ODP_DBG("dpdk rx queue setup done\n"); - - /* init one TX queue on each port */ - fflush(stdout); - ret = rte_eth_tx_queue_setup(portid, queueid, nb_txd, - rte_eth_dev_socket_id(portid), &tx_conf); - if (ret < 0) - ODP_ERR("rte_eth_tx_queue_setup:err=%d, port=%u\n", - ret, (unsigned) portid); - ODP_DBG("dpdk tx queue setup done\n"); - - /* Start device */ - ret = rte_eth_dev_start(portid); - if (ret < 0) - ODP_ERR("rte_eth_dev_start:err=%d, port=%u\n", - ret, (unsigned) portid); - ODP_DBG("dpdk setup done\n\n"); - + nbrxq = odp_sys_core_count() / num_intf; + nbtxq = nbrxq; + if (portinit[portid] == 0) { + fflush(stdout); + ret = rte_eth_dev_configure(portid, nbrxq, nbtxq, &port_conf); + if (ret < 0) + ODP_ERR("Cannot configure device: err=%d, port=%u\n", + ret, (unsigned) portid); + + rte_eth_macaddr_get(portid, ð_addr[portid]); + ODP_DBG("Port %u, MAC address: %02X:%02X:%02X:%02X:%02X:%02X\n", + (unsigned) portid, + eth_addr[portid].addr_bytes[0], + eth_addr[portid].addr_bytes[1], + eth_addr[portid].addr_bytes[2], + eth_addr[portid].addr_bytes[3], + eth_addr[portid].addr_bytes[4], + eth_addr[portid].addr_bytes[5]); + + /* init one RX queue on each port */ + fflush(stdout); + for (i = 0; i < nbrxq; i++) { + ret = rte_eth_rx_queue_setup(portid, i, nb_rxd, + rte_eth_dev_socket_id(portid), &rx_conf, + (struct rte_mempool *)pool); + if (ret < 0) + ODP_ERR("%s rxq:err=%d, port=%u\n", + __func__, ret, (unsigned) portid); + ODP_DBG("dpdk rx queue setup done\n"); + } + + /* init one TX queue on each port */ + fflush(stdout); + for (i = 0; i < nbtxq; i++) { + ret = rte_eth_tx_queue_setup(portid, i, nb_txd, + rte_eth_dev_socket_id(portid), &tx_conf); + if (ret < 0) + ODP_ERR("%s txq:err=%d, port=%u\n", + __func__, ret, (unsigned) portid); + ODP_DBG("dpdk tx queue setup done\n"); + } + + /* Start device */ + ret = rte_eth_dev_start(portid); + if (ret < 0) + ODP_ERR("rte_eth_dev_start:err=%d, port=%u\n", + ret, (unsigned) portid); + ODP_DBG("dpdk setup done\n\n"); + + portinit[portid] = 1; + } + pkt_dpdk->queueid = qid[portid]++; return 0; } diff --git a/platform/linux-dpdk/odp_packet_io.c b/platform/linux-dpdk/odp_packet_io.c index d8d127f..3124175 100644 --- a/platform/linux-dpdk/odp_packet_io.c +++ b/platform/linux-dpdk/odp_packet_io.c @@ -230,6 +230,8 @@ int odp_pktio_recv(odp_pktio_t id, odp_packet_t pkt_table[], unsigned len) if (pktio_entry == NULL) return -1; + odp_pktio_send(id, pkt_table, 0); + lock_entry(pktio_entry); pkts = recv_pkt_dpdk(&pktio_entry->s.pkt_dpdk, pkt_table, len); unlock_entry(pktio_entry);