From patchwork Wed Oct 21 11:37:10 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stuart Haslam X-Patchwork-Id: 55377 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-wi0-f200.google.com (mail-wi0-f200.google.com [209.85.212.200]) by patches.linaro.org (Postfix) with ESMTPS id 3E1422303B for ; Wed, 21 Oct 2015 11:37:29 +0000 (UTC) Received: by wicfg8 with SMTP id fg8sf5974144wic.0 for ; Wed, 21 Oct 2015 04:37:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:delivered-to:date:from:to :message-id:references:mime-version:content-disposition:in-reply-to :user-agent:cc:subject:precedence:list-id:list-unsubscribe :list-archive:list-post:list-help:list-subscribe:content-type :content-transfer-encoding:errors-to:sender:x-original-sender :x-original-authentication-results:mailing-list; bh=Uh8RgnLqgBMQEMxisxW0nGrEuMp/lgTqEGKAIoydg6w=; b=H34s/L+kfpYpQC1QSRIx+tBcjKhkpAZHlmh+4nbEba6os08tSK9LVgpEa6I3Ss+mFE jcMVcmDNVe4T3UH86WU0+KvbdXZBnyRyMh2W1KugNkZEZ1Mn7fbvRNHJaX5OWA2g8iU9 aBL7eL0QIfTievVvfBSKwDI1ZcEu1Y0g9bz/oek0V+vSB18ucWxhIDUD4EF2Vs2hzrb2 W7a3Vzw7FTYrog4DRpTgC6t9TuQo/O3FeDfdCvD36fI6+7dhIxmqUnyoQcKJoqVe+cXP TCgMwnAKieciIx8UQIyyddYs0agRfg3zKuHuNBw9gYfPt2ka2A8vRMNhVT84LV92eJbL RyeA== X-Gm-Message-State: ALoCoQlTTeiZYhgOvHFGHPi3lVoWib2vJ0GyR0n3JFrvHuGqFVpJGN8EbufRvEOl3NtWxiY6Qtfw X-Received: by 10.112.14.4 with SMTP id l4mr2052753lbc.20.1445427448572; Wed, 21 Oct 2015 04:37:28 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.25.145.136 with SMTP id t130ls203222lfd.89.gmail; Wed, 21 Oct 2015 04:37:28 -0700 (PDT) X-Received: by 10.112.168.194 with SMTP id zy2mr5113541lbb.79.1445427448298; Wed, 21 Oct 2015 04:37:28 -0700 (PDT) Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com. [209.85.217.182]) by mx.google.com with ESMTPS id f27si6018299lfi.26.2015.10.21.04.37.28 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 21 Oct 2015 04:37:28 -0700 (PDT) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.182 as permitted sender) client-ip=209.85.217.182; Received: by lbbec13 with SMTP id ec13so33943558lbb.0 for ; Wed, 21 Oct 2015 04:37:28 -0700 (PDT) X-Received: by 10.112.17.34 with SMTP id l2mr5018224lbd.117.1445427448083; Wed, 21 Oct 2015 04:37:28 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.112.59.35 with SMTP id w3csp2639890lbq; Wed, 21 Oct 2015 04:37:26 -0700 (PDT) X-Received: by 10.140.89.47 with SMTP id u44mr10259641qgd.67.1445427446810; Wed, 21 Oct 2015 04:37:26 -0700 (PDT) Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id 27si7571921qku.100.2015.10.21.04.37.26; Wed, 21 Oct 2015 04:37:26 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Received: by lists.linaro.org (Postfix, from userid 109) id 5B30B61A41; Wed, 21 Oct 2015 11:37:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 2C82B619D2; Wed, 21 Oct 2015 11:37:18 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 565C4619DA; Wed, 21 Oct 2015 11:37:15 +0000 (UTC) Received: from mail-wi0-f177.google.com (mail-wi0-f177.google.com [209.85.212.177]) by lists.linaro.org (Postfix) with ESMTPS id 9BF75619B8 for ; Wed, 21 Oct 2015 11:37:13 +0000 (UTC) Received: by wijp11 with SMTP id p11so90380124wij.0 for ; Wed, 21 Oct 2015 04:37:12 -0700 (PDT) X-Received: by 10.194.156.195 with SMTP id wg3mr11609663wjb.2.1445427432265; Wed, 21 Oct 2015 04:37:12 -0700 (PDT) Received: from localhost ([2001:41d0:a:3cb4::abcd]) by smtp.gmail.com with ESMTPSA id lv4sm9832400wjb.43.2015.10.21.04.37.11 (version=TLSv1.2 cipher=RC4-SHA bits=128/128); Wed, 21 Oct 2015 04:37:11 -0700 (PDT) Date: Wed, 21 Oct 2015 12:37:10 +0100 From: Stuart Haslam To: "Elo, Matias (Nokia - FI/Espoo)" Message-ID: <20151021113710.GA17144@localhost> References: <1444999550-30285-1-git-send-email-matias.elo@nokia.com> <1444999550-30285-2-git-send-email-matias.elo@nokia.com> <20151020122846.GA11695@localhost> <20151020163637.GA29734@localhost> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Topics: patch Cc: "lng-odp@lists.linaro.org" Subject: Re: [lng-odp] [PATCH 2/2] linux-generic: netmap: use separate rx and tx descriptors X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: , List-Help: , List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: stuart.haslam@linaro.org X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.182 as permitted sender) smtp.mailfrom=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 On Wed, Oct 21, 2015 at 06:26:39AM +0000, Elo, Matias (Nokia - FI/Espoo) wrote: > Hi, > > I'm still unable to reproduce the problem. Have you tried reloading the netmap kernel module and driver? > I found that if I apply only this patch and not the link state patch, it works. The problem is related to the fact the ports are directly connected with an external loopback cable, so when one goes down the other does too. odp_pktio_open(eth5) [596320.186076] ixgbe 0000:04:00.0 eth1: NIC Link is Down [596320.490564] ixgbe 0000:04:00.1 eth5: detected SFP+: 65535 [596320.490604] ixgbe 0000:04:00.1 eth5: NIC Link is Up 10 Gbps, Flow Control: RX/TX odp_pktio_open(eth5) returns, test starts sending packets, but the other end of the link didn't come up until ~700ms later.. [596321.165492] ixgbe 0000:04:00.0 eth1: NIC Link is Up 10 Gbps, Flow Control: RX/TX Looking at that, I'm not entirely sure how it works without the state check, but it does work reliably. I guess it was hitting a timing window and managing to squirt out the 1-4 packets used in the test before the link bounced. Anyway, just moving the sleep so that it waits at least 1 second after (this end of) the link comes up resolves it for me. It's a bit nasty doing this sort of thing in the pktio itself rather than in the application, but the link bouncing is a particular netmap quirk and we don't have a link state check API yet, so I think this makes sense for now. Stuart. > I rebased the particular two patches to the HEAD and this is the output from the pktio validation test: > > $ sudo ODP_PKTIO_IF0=em49 ODP_PKTIO_IF1=em50 ./test/validation/pktio/pktio_main > odp_pool.c:102:odp_pool_init_global(): > Pool init global > odp_pool.c:103:odp_pool_init_global(): pool_entry_s size 320 > odp_pool.c:104:odp_pool_init_global(): pool_entry_t size 320 > odp_pool.c:105:odp_pool_init_global(): odp_buffer_hdr_t size 160 > odp_pool.c:106:odp_pool_init_global(): > odp_queue.c:109:odp_queue_init_global():Queue init ... odp_queue.c:131:odp_queue_init_global():done > odp_queue.c:132:odp_queue_init_global():Queue init global > odp_queue.c:134:odp_queue_init_global(): struct queue_entry_s size 256 > odp_queue.c:136:odp_queue_init_global(): queue_entry_t size 256 > odp_queue.c:137:odp_queue_init_global(): > odp_schedule.c:122:odp_schedule_init_global():Schedule init ... odp_schedule.c:187:odp_schedule_init_global():done > ODP API version: 1.3.0 > ODP implementation version: linux-generic 1.3.0-0 (v1.3.0) Oct 21 2015 09:14:07 'https://git.linaro.org/lng/odp.git' (v1.3.0.0-46-ge4a5493-dirty) > > > CUnit - A unit testing framework for C - Version 2.1-3 > http://cunit.sourceforge.net/ > > Using paired interfaces: em49 em50 > > Suite: Packet I/O Unsegmented > Test: pktio_test_open ...080.753804 nm_open [839] NIOCREGIF failed: No such device or address nothere > 080.753826 nm_open [839] NIOCREGIF failed: No such device or address nothere > pktio/netmap.c:145:netmap_open():nm_open(netmap:nothere) failed > pktio/socket.c:107:mac_addr_get_fd():ioctl(SIOCGIFHWADDR): No such device: "nothere". > pktio/socket.c:237:sock_setup_pkt():ioctl(SIOCGIFINDEX): No such device: "nothere". > odp_packet_io.c:240:setup_pktio_entry():Unable to init any I/O type. > passed > Test: pktio_test_lookup ...passed > Test: pktio_test_inq ...passed > Test: pktio_test_poll_queue ...passed > Test: pktio_test_poll_multi ...passed > Test: pktio_test_sched_queue ...passed > Test: pktio_test_sched_multi ...passed > Test: pktio_test_jumbo ...FAILED > 1. pktio.c:384 - CU_FAIL("failed to receive transmitted packet") > 2. pktio.c:450 - i == num_pkts > Test: pktio_test_mtu ... 1500 passed > Test: pktio_test_promisc ...passed > Test: pktio_test_mac ...testing mac for em49 > 5C:B9:1:89:88:8 passed > Test: pktio_test_inq_remdef ...passed > Test: pktio_test_start_stop ...FAILED > 1. pktio.c:775 - pkts == allocUsing paired interfaces: em49 em50 > > Suite: Packet I/O Segmented > Test: pktio_test_poll_queue ...passed > Test: pktio_test_poll_multi ...passed > Test: pktio_test_sched_queue ...passed > Test: pktio_test_sched_multi ...passed > Test: pktio_test_jumbo ...FAILED > 1. pktio.c:384 - CU_FAIL("failed to receive transmitted packet") > 2. pktio.c:450 - i == num_pkts > > Run Summary: Type Total Ran Passed Failed Inactive > suites 2 2 n/a 0 0 > tests 18 18 15 3 0 > asserts 296 296 291 5 n/a > > Elapsed time = 8.793 seconds > > -Matias > > > -----Original Message----- > > From: EXT Stuart Haslam [mailto:stuart.haslam@linaro.org] > > Sent: Tuesday, October 20, 2015 7:37 PM > > To: Elo, Matias (Nokia - FI/Espoo) > > Cc: lng-odp@lists.linaro.org > > Subject: Re: [lng-odp] [PATCH 2/2] linux-generic: netmap: use separate rx and tx > > descriptors > > > > On Tue, Oct 20, 2015 at 02:20:05PM +0000, Elo, Matias (Nokia - FI/Espoo) wrote: > > > > -----Original Message----- > > > > From: EXT Stuart Haslam [mailto:stuart.haslam@linaro.org] > > > > Sent: Tuesday, October 20, 2015 3:29 PM > > > > To: Elo, Matias (Nokia - FI/Espoo) > > > > Cc: lng-odp@lists.linaro.org > > > > Subject: Re: [lng-odp] [PATCH 2/2] linux-generic: netmap: use separate rx and > > tx > > > > descriptors > > > > > > > > On Fri, Oct 16, 2015 at 03:45:50PM +0300, Matias Elo wrote: > > > > > Using separate file descriptors enables using rx and tx > > > > > functions simultaneously from different threads. > > > > > > > > > > Previously netmap tx queues were flushed only after no free > > > > > slots were left which caused long delays on slow packet > > > > > rates. Now queues are flushed after each packet > > > > > burst. > > > > > > > > This isn't working for me. Same setup as earlier and I get similar > > > > results. I've tried adding a NIOCTXSYNC on send and a NIOCRXSYNC on recv > > > > but it didn't seem to help. I'll look further to figure out if it's the > > > > rx or tx side that's not working. > > > > > > > > > > If you use e.g. ethtool, is the link between your nic ports detected? > > > > Yes, the link state check in odp_pktio_open() is detecting the link too. > > > > > I'm wondering if the problem is related to directly connecting two netmap ports > > on the same host. > > > > > > > Possibly, although the same setup works fine on HEAD just with > > NETMAP_NO_TX_POLL removed - not suggesting this as a fix, but it > > seems to indicate my setup is alright. > > > > > -Matias > > > > > > > -- > > > > Stuart. > > > > > > > > > Signed-off-by: Matias Elo > > > > > --- > > > > > platform/linux-generic/include/odp_packet_netmap.h | 3 ++- > > > > > platform/linux-generic/pktio/netmap.c | 29 ++++++++++++++-------- > > > > > 2 files changed, 21 insertions(+), 11 deletions(-) > > > > > > > > > > diff --git a/platform/linux-generic/include/odp_packet_netmap.h > > > > b/platform/linux-generic/include/odp_packet_netmap.h > > > > > index 23aea5b..0577dfe 100644 > > > > > --- a/platform/linux-generic/include/odp_packet_netmap.h > > > > > +++ b/platform/linux-generic/include/odp_packet_netmap.h > > > > > @@ -15,7 +15,8 @@ > > > > > typedef struct { > > > > > odp_pool_t pool; /**< pool to alloc packets from */ > > > > > size_t max_frame_len; /**< buf_size - sizeof(pkt_hdr) */ > > > > > - struct nm_desc *desc; /**< netmap meta-data for the device */ > > > > > + struct nm_desc *rx_desc; /**< netmap meta-data for the device */ > > > > > + struct nm_desc *tx_desc; /**< netmap meta-data for the device */ > > > > > uint32_t if_flags; /**< interface flags */ > > > > > int sockfd; /**< control socket */ > > > > > unsigned char if_mac[ETH_ALEN]; /**< eth mac address */ > > > > > diff --git a/platform/linux-generic/pktio/netmap.c b/platform/linux- > > > > generic/pktio/netmap.c > > > > > index 0c9ad35..9ad6a22 100644 > > > > > --- a/platform/linux-generic/pktio/netmap.c > > > > > +++ b/platform/linux-generic/pktio/netmap.c > > > > > @@ -89,10 +89,13 @@ static int netmap_close(pktio_entry_t *pktio_entry) > > > > > { > > > > > pkt_netmap_t *pkt_nm = &pktio_entry->s.pkt_nm; > > > > > > > > > > - if (pkt_nm->desc != NULL) { > > > > > - nm_close(pkt_nm->desc); > > > > > + if (pkt_nm->rx_desc != NULL) { > > > > > + nm_close(pkt_nm->rx_desc); > > > > > mmap_desc.mem = NULL; > > > > > } > > > > > + if (pkt_nm->tx_desc != NULL) > > > > > + nm_close(pkt_nm->tx_desc); > > > > > + > > > > > if (pkt_nm->sockfd != -1 && close(pkt_nm->sockfd) != 0) { > > > > > __odp_errno = errno; > > > > > ODP_ERR("close(sockfd): %s\n", strerror(errno)); > > > > > @@ -131,18 +134,21 @@ static int netmap_open(odp_pktio_t id > > > > ODP_UNUSED, pktio_entry_t *pktio_entry, > > > > > snprintf(ifname, sizeof(ifname), "netmap:%s", netdev); > > > > > > > > > > if (mmap_desc.mem == NULL) > > > > > - pkt_nm->desc = nm_open(ifname, NULL, > > > > NETMAP_NO_TX_POLL, NULL); > > > > > + pkt_nm->rx_desc = nm_open(ifname, NULL, > > > > NETMAP_NO_TX_POLL, > > > > > + NULL); > > > > > else > > > > > - pkt_nm->desc = nm_open(ifname, NULL, NETMAP_NO_TX_POLL > > > > | > > > > > - NM_OPEN_NO_MMAP, &mmap_desc); > > > > > - if (pkt_nm->desc == NULL) { > > > > > + pkt_nm->rx_desc = nm_open(ifname, NULL, > > > > NETMAP_NO_TX_POLL | > > > > > + NM_OPEN_NO_MMAP, &mmap_desc); > > > > > + pkt_nm->tx_desc = nm_open(ifname, NULL, NM_OPEN_NO_MMAP, > > > > &mmap_desc); > > > > > + > > > > > + if (pkt_nm->rx_desc == NULL || pkt_nm->tx_desc == NULL) { > > > > > ODP_ERR("nm_open(%s) failed\n", ifname); > > > > > goto error; > > > > > } > > > > > > > > > > if (mmap_desc.mem == NULL) { > > > > > - mmap_desc.mem = pkt_nm->desc->mem; > > > > > - mmap_desc.memsize = pkt_nm->desc->memsize; > > > > > + mmap_desc.mem = pkt_nm->rx_desc->mem; > > > > > + mmap_desc.memsize = pkt_nm->rx_desc->memsize; > > > > > } > > > > > > > > > > sockfd = socket(AF_INET, SOCK_DGRAM, 0); > > > > > @@ -215,7 +221,7 @@ static int netmap_recv(pktio_entry_t *pktio_entry, > > > > odp_packet_t pkt_table[], > > > > > unsigned num) > > > > > { > > > > > struct dispatch_args args; > > > > > - struct nm_desc *nm_desc = pktio_entry->s.pkt_nm.desc; > > > > > + struct nm_desc *nm_desc = pktio_entry->s.pkt_nm.rx_desc; > > > > > struct pollfd polld; > > > > > > > > > > polld.fd = nm_desc->fd; > > > > > @@ -236,8 +242,8 @@ static int netmap_recv(pktio_entry_t *pktio_entry, > > > > odp_packet_t pkt_table[], > > > > > static int netmap_send(pktio_entry_t *pktio_entry, odp_packet_t > > pkt_table[], > > > > > unsigned num) > > > > > { > > > > > - struct nm_desc *nm_desc = pktio_entry->s.pkt_nm.desc; > > > > > struct pollfd polld; > > > > > + struct nm_desc *nm_desc = pktio_entry->s.pkt_nm.tx_desc; > > > > > unsigned i, nb_tx; > > > > > uint8_t *frame; > > > > > uint32_t frame_len; > > > > > @@ -259,6 +265,9 @@ static int netmap_send(pktio_entry_t *pktio_entry, > > > > odp_packet_t pkt_table[], > > > > > break; > > > > > } > > > > > } > > > > > + /* Send pending packets */ > > > > > + poll(&polld, 1, 0); > > > > > + > > > > > for (i = 0; i < nb_tx; i++) > > > > > odp_packet_free(pkt_table[i]); > > > > > > > > > > -- > > > > > 1.9.1 diff --git a/platform/linux-generic/pktio/netmap.c b/platform/linux-generic/pktio/netmap.c index 9ad6a22..b3a2ca2 100644 --- a/platform/linux-generic/pktio/netmap.c +++ b/platform/linux-generic/pktio/netmap.c @@ -171,9 +171,9 @@ static int netmap_open(odp_pktio_t id ODP_UNUSED, pktio_entry_t *pktio_entry, /* Wait for the link to come up */ for (i = 0; i < NM_OPEN_RETRIES; i++) { err = netmap_do_ioctl(pktio_entry, SIOCETHTOOL, ETHTOOL_GLINK); + sleep(1); if (err == 0) return 0; - sleep(1); } ODP_ERR("%s didn't come up\n", pktio_entry->s.name);