| Message ID | 3ff9f1d2-36d9-df56-2c29-7e3445245ee2@linaro.org |
|---|---|
| State | Superseded |
| Headers | show |
Do we want to discuss this during tomorrow's ODP public call? I'll add it to the agenda. On Mon, Aug 15, 2016 at 6:36 AM, Elo, Matias (Nokia - FI/Espoo) < matias.elo@nokia-bell-labs.com> wrote: > Hi, > > ODP_ASSERT isn’t part of the public API. After removing it the application > builds correctly. > > The argument parsing issue is now fixed. > > However, after the ODP_PKTIO_OP_MT -> ODP_PKTIO_OP_MT_UNSAFE fix the TX > side > is now broken. Syslog is full of error messages like: > > nm_txsync_prologue eth1 TX0: fail 'cur > kring->rtail && cur < > head' h 399 c 394 t 307 rh 380 rc 380 rt 307 hc 380 ht 307 > nm_txsync_prologue eth1 TX0: fail 'head < kring->rhead || head > > kring->rtail' h 53 c 53 t 508 rh 54 rc 54 rt 508 hc 54 ht 508 > netmap_ring_reinit called for eth1 TX0 > > These messages are from netmap and I would assume they are caused by using > the same > TX queues from multiple threads. > > “The rx and tx queues are linked because the qconf interface is supported, > like (0,0,0),(0,1,0) > > The thread will receive two queues (0,1) from port 0, but no txq > information is specified. > > My idea is let the txq_idx equals rxq_idx, and if the actual queue is not > enough, using share queue.” > You don’t need any additional information about the TX queues from qconf. > You can always try to > allocate as many TX queue as there are worker threads for each interface. > In this case the txq_idx can > actually be odp_thread_id() (if enough TX queues could be allocated). If > the number of worker threads > is larger than ‘capa.max_output_queues’ you have to use locking. > > -Matias > > > > From: forrest.shi [mailto:forrest.shi@linaro.org] > Sent: Monday, August 15, 2016 1:18 PM > To: Elo, Matias (Nokia - FI/Espoo) <matias.elo@nokia-bell-labs.com>; Mike > Holmes <mike.holmes@linaro.org> > Cc: lng-odp <lng-odp@lists.linaro.org> > Subject: Re: [lng-odp] [PATCHv7 1/2] example: introducing l3fwd > > > Hi Matias, > > The rx and tx queues are linked because the qconf interface is supported, > like (0,0,0),(0,1,0) > > The thread will receive two queues (0,1) from port 0, but no txq > information is specified. > > My idea is let the txq_idx equals rxq_idx, and if the actual queue is not > enough, using share queue. > > It looks like I have mis-understood ODP_PKTIO_OP_MT_UNSAFE and > ODP_PKTIO_OP_MT, and then locked txq. > > It will be corrected. > > > > About only running with the -d xxx, I checked the code, maybe caused by > odp_barrier_wait() in if(). > > Here there is some assumption that the odp-linux will reserve two threads > for OS and control thread. > > On my side, odp-nxp doesn't reserve these 2 threads, so the behavior is > not the same. Sorry for confused info. > > > > About the hash, the find_fwd_db_entry should be hot. However, it's > supposesd to return quickly from this function using a hash match. > > Have not get a clue. It is a bit slow, compared with lpm on my side. > > > > Would you please help to run the attached patch on X86? I believe the > patch should fix the first 2 issues, at least on my side. > > > > Thanks, > > Forrest > > > > 在 8/12/2016 4:28 PM, Elo, Matias (Nokia - FI/Espoo) 写道: > I tested the application on our server and the forwarding rate doesn’t > scale when core > count is increased (added l2fwd and OFP results for reference). Based on > some debugging > this seems to be caused by using locks on the TX side. > > ‘perf’ shows that the application spends most of time in odp_ticket_lock: > 49.53% odp_l3fwd odp_l3fwd [.] odp_ticketlock_lock > > In the setup_worker_qconf() function’s comments you state: “Try to have > one tx queue for > each rx queue, if not vailable, shared tx queue is used.” You should > configure one tx queue > per each worker thread for all interfaces. The number of tx and rx queues > shouldn’t be linked > in any way. This enables you to used lockless tx. You can check how this > is done in e.g. odp_l2fwd. > > When I change the ‘--style’ option to ‘hash’ forwarding rate drops to only > a few thousand packets > per second. > > Perf output in this case: > Samples: 141K of event 'cycles:pp', Event count (approx.): 90818460609 > Overhead Command Shared Object Symbol > + 99.47% odp_l3fwd odp_l3fwd [.] find_fwd_db_entry > > > -Matias > > > From: Elo, Matias (Nokia - FI/Espoo) > Sent: Friday, August 12, 2016 9:54 AM > To: 'Mike Holmes' <mike.holmes@linaro.org><mailto:mike.holmes@linaro.org>; > forrest.shi <forrest.shi@linaro.org><mailto:forrest.shi@linaro.org> > Cc: lng-odp <lng-odp@lists.linaro.org><mailto:lng-odp@lists.linaro.org> > Subject: RE: [lng-odp] [PATCHv7 1/2] example: introducing l3fwd > > Sorry for the slow review on my part. I’m currently testing the > application and > there seems to be something wrong with the argument parsing. > > With the following commands no packets are forwarded. > $ sudo ./example/l3fwd/odp_l3fwd -i 0,1 -r 1.1.1.0/24,0 -r 2.2.2.0/24,1 > -t 2 -d 0 > or > $ sudo ./example/l3fwd/odp_l3fwd -i 0,1 -r 1.1.1.0/24,0 -r 2.2.2.0/24,1 > -t 2 > > Only when some ‘-d’ value is given packets are actually forwarded > E.g. > $ sudo ./example/l3fwd/odp_l3fwd -i 0,1 -r 1.1.1.0/24,0 -r 2.2.2.0/24,1 > -t 2 -d 60 > > > -Matias > > > > From: Mike Holmes [mailto:mike.holmes@linaro.org] > Sent: Thursday, August 11, 2016 7:20 PM > To: forrest.shi <forrest.shi@linaro.org<mailto:forrest.shi@linaro.org>> > Cc: lng-odp <lng-odp@lists.linaro.org<mailto:lng-odp@lists.linaro.org>>; > Elo, Matias (Nokia - FI/Espoo) <matias.elo@nokia-bell-labs.com<mailto: > matias.elo@nokia-bell-labs.com>>; Savolainen, Petri (Nokia - FI/Espoo) < > petri.savolainen@nokia-bell-labs.com<mailto:petri. > savolainen@nokia-bell-labs.com>> > Subject: Re: [lng-odp] [PATCHv7 1/2] example: introducing l3fwd > > I CC'ed Matias, I am not sure if we can get another Member to run this on > their HW ? > > It does not have to be perfect but I think it is likely to be ready as an > example at this point. > > Mike > > On 10 August 2016 at 06:42, forrest.shi <forrest.shi@linaro.org<mailto: > forrest.shi@linaro.org>> wrote: > Ping, need review and comments. Pending a bit long. > > > -----Original Message----- > > From: forrest.shi@linaro.org<mailto:forrest.shi@linaro.org> [mailto: > forrest.shi@linaro.org<mailto:forrest.shi@linaro.org>] > > Sent: Thursday, July 28, 2016 16:52 > > To: lng-odp@lists.linaro.org<mailto:lng-odp@lists.linaro.org> > > Cc: Xuelin Shi <forrest.shi@linaro.org<mailto:forrest.shi@linaro.org>> > > Subject: [lng-odp][PATCHv7 1/2] example: introducing l3fwd > > > > From: Xuelin Shi <forrest.shi@linaro.org<mailto:forrest.shi@linaro.org>> > > > > multi-thread, multi-queues and bi-directional forwarding. > > > > support (port, queue, thread) arguments in cmdline which specify how > > the threads handle which rx queue at which port, if no this argument, > > default specification used. > > > > both hash and lpm based lookup methods are supported, default lpm. > > > > Signed-off-by: Xuelin Shi <forrest.shi@linaro.org<mailto: > forrest.shi@linaro.org>> > > --- > > example/Makefile.am | 12 +- > > example/l3fwd/.gitignore | 1 + > > example/l3fwd/Makefile.am | 12 + > > example/l3fwd/odp_l3fwd.c | 1094 > > +++++++++++++++++++++++++++++++++++++++++ > > example/l3fwd/odp_l3fwd_db.c | 432 ++++++++++++++++ > > example/l3fwd/odp_l3fwd_db.h | 136 +++++ > > example/l3fwd/odp_l3fwd_lpm.c | 224 +++++++++ > > example/l3fwd/odp_l3fwd_lpm.h | 20 + > > example/m4/configure.m4 | 9 +- > > 9 files changed, 1935 insertions(+), 5 deletions(-) > > create mode 100644 example/l3fwd/.gitignore > > create mode 100644 example/l3fwd/Makefile.am > > create mode 100644 example/l3fwd/odp_l3fwd.c > > create mode 100644 example/l3fwd/odp_l3fwd_db.c > > create mode 100644 example/l3fwd/odp_l3fwd_db.h > > create mode 100644 example/l3fwd/odp_l3fwd_lpm.c > > create mode 100644 example/l3fwd/odp_l3fwd_lpm.h > > > > diff --git a/example/Makefile.am b/example/Makefile.am > > index 37542af..dfc07b6 100644 > > --- a/example/Makefile.am > > +++ b/example/Makefile.am > > @@ -1 +1,11 @@ > > -SUBDIRS = classifier generator ipsec packet time timer traffic_mgmt > > l2fwd_simple switch hello > > +SUBDIRS = classifier \ > > + generator \ > > + hello \ > > + ipsec \ > > + l2fwd_simple \ > > + l3fwd \ > > + packet \ > > + switch \ > > + time \ > > + timer \ > > + traffic_mgmt > > diff --git a/example/l3fwd/.gitignore b/example/l3fwd/.gitignore > > new file mode 100644 > > index 0000000..74e501f > > --- /dev/null > > +++ b/example/l3fwd/.gitignore > > @@ -0,0 +1 @@ > > +odp_l3fwd > > diff --git a/example/l3fwd/Makefile.am b/example/l3fwd/Makefile.am > > new file mode 100644 > > index 0000000..f926717 > > --- /dev/null > > +++ b/example/l3fwd/Makefile.am > > @@ -0,0 +1,12 @@ > > +include $(top_srcdir)/example/Makefile.inc > > + > > +bin_PROGRAMS = odp_l3fwd$(EXEEXT) > > +odp_l3fwd_LDFLAGS = $(AM_LDFLAGS) -static > > +odp_l3fwd_CFLAGS = $(AM_CFLAGS) -I${top_srcdir}/example - > > I${top_srcdir}/test > > + > > +noinst_HEADERS = \ > > + $(top_srcdir)/example/l3fwd/odp_l3fwd_db.h \ > > + $(top_srcdir)/example/l3fwd/odp_l3fwd_lpm.h \ > > + $(top_srcdir)/example/example_debug.h > > + > > +dist_odp_l3fwd_SOURCES = odp_l3fwd.c odp_l3fwd_db.c odp_l3fwd_lpm.c > > diff --git a/example/l3fwd/odp_l3fwd.c b/example/l3fwd/odp_l3fwd.c > > new file mode 100644 > > index 0000000..b1db002 > > --- /dev/null > > +++ b/example/l3fwd/odp_l3fwd.c > > @@ -0,0 +1,1094 @@ > > +/* Copyright (c) 2016, Linaro Limited > > + * All rights reserved. > > + * > > + * SPDX-License-Identifier: BSD-3-Clause > > + */ > > + > > +#include <stdlib.h> > > +#include <stdio.h> > > +#include <errno.h> > > +#include <getopt.h> > > +#include <unistd.h> > > +#include <inttypes.h> > > + > > +#include <test_debug.h> > > + > > +#include <odp_api.h> > > +#include <odp/helper/linux.h> > > +#include <odp/helper/eth.h> > > +#include <odp/helper/ip.h> > > +#include <odp/helper/udp.h> > > +#include <odp/helper/tcp.h> > > + > > +#include "odp_l3fwd_db.h" > > +#include "odp_l3fwd_lpm.h" > > + > > +#define POOL_NUM_PKT 8192 > > +#define POOL_SEG_LEN 1856 > > +#define MAX_PKT_BURST 32 > > + > > +#define MAX_NB_WORKER 32 > > +#define MAX_NB_PKTIO 32 > > +#define MAX_NB_QUEUE 32 > > +#define MAX_NB_QCONFS 1024 > > +#define MAX_NB_ROUTE 32 > > + > > +#define INVALID_ID (-1) > > +#define PRINT_INTERVAL 10 /* interval seconds of printing > stats > */ > > + > > +/** Get rid of path in filename - only for unix-type paths using '/' */ > > +#define NO_PATH(file_name) (strrchr((file_name), '/') ? \ > > + strrchr((file_name), '/') + 1 : (file_name)) > > + > > +struct l3fwd_pktio_s { > > + odp_pktio_t pktio; > > + odph_ethaddr_t mac_addr; > > + odp_pktin_queue_t ifin[MAX_NB_QUEUE]; > > + odp_pktout_queue_t ifout[MAX_NB_QUEUE]; > > + int nb_rxq; > > + int nb_txq; > > + int rxq_idx; > > + int txq_idx; > > +}; > > + > > +struct l3fwd_qconf_s { > > + uint8_t if_idx; /* port index */ > > + uint8_t rxq_idx; /* recv queue index in a port */ > > + uint8_t core_idx; /* this core should handle traffic */ > > +}; > > + > > +struct thread_arg_s { > > + uint64_t packets; > > + uint64_t rx_drops; > > + uint64_t tx_drops; > > + struct { > > + int used; > > + int rxq[MAX_NB_QUEUE]; > > + int txq[MAX_NB_QUEUE]; > > + } pktio[MAX_NB_PKTIO]; > > + int thr_idx; > > + int nb_pktio; > > +}; > > + > > +typedef struct { > > + char *if_names[MAX_NB_PKTIO]; > > + int if_count; > > + char *route_str[MAX_NB_ROUTE]; > > + int worker_count; > > + struct l3fwd_qconf_s qconf_config[MAX_NB_QCONFS]; > > + int qconf_count; > > + uint32_t duration; /* seconds to run */ > > + uint8_t hash_mode; /* 1:hash, 0:lpm */ > > + uint8_t dest_mac_changed[MAX_NB_PKTIO]; /* 1: dest mac from > > cmdline */ > > +} app_args_t; > > + > > +struct { > > + app_args_t cmd_args; > > + struct l3fwd_pktio_s l3fwd_pktios[MAX_NB_PKTIO]; > > + odph_odpthread_t l3fwd_workers[MAX_NB_WORKER]; > > + struct thread_arg_s worker_args[MAX_NB_WORKER]; > > + odph_ethaddr_t eth_dest_mac[MAX_NB_PKTIO]; > > + > > + /* forward func, hash or lpm */ > > + int (*fwd_func)(odp_packet_t pkt, int sif); > > +} global; > > + > > +/** Global barrier to synchronize main and workers */ > > +static odp_barrier_t barrier; > > +static int exit_threads; /**< Break workers loop if set to 1 */ > > + > > +static int create_pktio(const char *name, odp_pool_t pool, > > + struct l3fwd_pktio_s *fwd_pktio) > > +{ > > + odp_pktio_param_t pktio_param; > > + odp_pktio_t pktio; > > + odp_pktio_capability_t capa; > > + int rc; > > + > > + odp_pktio_param_init(&pktio_param); > > + > > + pktio = odp_pktio_open(name, pool, &pktio_param); > > + if (pktio == ODP_PKTIO_INVALID) { > > + printf("Failed to open %s\n", name); > > + return -1; > > + } > > + fwd_pktio->pktio = pktio; > > + > > + rc = odp_pktio_capability(pktio, &capa); > > + if (rc) { > > + printf("Error: pktio %s: unable to read capabilities!\n", > > + name); > > + > > + return -1; > > + } > > + > > + fwd_pktio->nb_rxq = (int)capa.max_input_queues; > > + fwd_pktio->nb_txq = (int)capa.max_output_queues; > > + > > + if (fwd_pktio->nb_rxq > MAX_NB_QUEUE) > > + fwd_pktio->nb_rxq = MAX_NB_QUEUE; > > + > > + if (fwd_pktio->nb_txq > MAX_NB_QUEUE) > > + fwd_pktio->nb_txq = MAX_NB_QUEUE; > > + > > + return 0; > > +} > > + > > +static void setup_fwd_db(void) > > +{ > > + fwd_db_entry_t *entry; > > + int if_idx; > > + app_args_t *args; > > + > > + args = &global.cmd_args; > > + if (args->hash_mode) > > + init_fwd_hash_cache(); > > + else > > + fib_tbl_init(); > > + > > + for (entry = fwd_db->list; NULL != entry; entry = entry->next) { > > + if_idx = entry->oif_id; > > + if (!args->hash_mode) > > + fib_tbl_insert(entry->subnet.addr, if_idx, > > + entry->subnet.depth); > > + if (args->dest_mac_changed[if_idx]) > > + global.eth_dest_mac[if_idx] = entry->dst_mac; > > + else > > + entry->dst_mac = global.eth_dest_mac[if_idx]; > > + } > > +} > > + > > +static int l3fwd_pkt_hash(odp_packet_t pkt, int sif) > > +{ > > + fwd_db_entry_t *entry; > > + ipv4_tuple5_t key; > > + odph_ethhdr_t *eth; > > + odph_udphdr_t *udp; > > + odph_ipv4hdr_t *ip; > > + uint32_t len; > > + int dif; > > + > > + ip = odp_packet_l3_ptr(pkt, &len); > > + key.dst_ip = odp_be_to_cpu_32(ip->dst_addr); > > + key.src_ip = odp_be_to_cpu_32(ip->src_addr); > > + key.proto = ip->proto; > > + > > + if (odp_packet_has_udp(pkt) || > > + odp_packet_has_tcp(pkt)) { > > + /* UDP or TCP*/ > > + void *ptr = odp_packet_l4_ptr(pkt, NULL); > > + > > + udp = (odph_udphdr_t *)ptr; > > + key.src_port = odp_be_to_cpu_16(udp->src_port); > > + key.dst_port = odp_be_to_cpu_16(udp->dst_port); > > + } else { > > + key.src_port = 0; > > + key.dst_port = 0; > > + } > > + > > + entry = find_fwd_db_entry(&key); > > + ip->ttl--; > > + ip->chksum = odph_ipv4_csum_update(pkt); > > + eth = odp_packet_l2_ptr(pkt, NULL); > > + if (entry) { > > + eth->src = entry->src_mac; > > + eth->dst = entry->dst_mac; > > + dif = entry->oif_id; > > + } else { > > + /* no route, send by src port */ > > + eth->dst = eth->src; > > + dif = sif; > > + } > > + > > + return dif; > > +} > > + > > +static int l3fwd_pkt_lpm(odp_packet_t pkt, int sif) > > +{ > > + odph_ipv4hdr_t *ip; > > + odph_ethhdr_t *eth; > > + uint32_t len; > > + int dif; > > + int ret; > > + > > + ip = odp_packet_l3_ptr(pkt, &len); > > + ip->ttl--; > > + ip->chksum = odph_ipv4_csum_update(pkt); > > + eth = odp_packet_l2_ptr(pkt, NULL); > > + > > + /* network byte order maybe different from host */ > > + ret = fib_tbl_lookup(odp_be_to_cpu_32(ip->dst_addr), &dif); > > + if (ret) > > + dif = sif; > > + > > + eth->dst = global.eth_dest_mac[dif]; > > + eth->src = global.l3fwd_pktios[dif].mac_addr; > > + > > + return dif; > > +} > > + > > +/** > > + * Drop packets which input parsing marked as containing errors. > > + * > > + * Frees packets with error and modifies pkt_tbl[] to only contain > packets > with > > + * no detected errors. > > + * > > + * @param pkt_tbl Array of packets > > + * @param num Number of packets in pkt_tbl[] > > + * > > + * @return Number of packets dropped > > + */ > > +static inline int drop_err_pkts(odp_packet_t pkt_tbl[], unsigned num) > > +{ > > + odp_packet_t pkt; > > + unsigned dropped = 0; > > + unsigned i, j; > > + > > + for (i = 0, j = 0; i < num; ++i) { > > + pkt = pkt_tbl[i]; > > + > > + if (odp_unlikely(odp_packet_has_error(pkt) || > > + !odp_packet_has_ipv4(pkt))) { > > + odp_packet_free(pkt); > > + dropped++; > > + } else if (odp_unlikely(i != j++)) { > > + pkt_tbl[j - 1] = pkt; > > + } > > + } > > + > > + return dropped; > > +} > > + > > +static void l3fwd_one_queue(uint32_t sif, uint8_t rxq_id, void *thr_arg) > > +{ > > + struct l3fwd_pktio_s *port; > > + odp_packet_t *tbl; > > + odp_pktout_queue_t outq; > > + odp_packet_t pkt_tbl[MAX_PKT_BURST]; > > + struct thread_arg_s *arg; > > + uint8_t txq_id; > > + int pkts, drop, sent; > > + int dif, dst_port; > > + int i; > > + > > + arg = thr_arg; > > + port = &global.l3fwd_pktios[sif]; > > + pkts = odp_pktin_recv(port->ifin[rxq_id], pkt_tbl, MAX_PKT_BURST); > > + if (pkts <= 0) > > + return; > > + arg->packets += pkts; > > + drop = drop_err_pkts(pkt_tbl, pkts); > > + pkts -= drop; > > + arg->rx_drops += drop; > > + > > + dif = global.fwd_func(pkt_tbl[0], sif); > > + tbl = &pkt_tbl[0]; > > + while (pkts) { > > + dst_port = dif; > > + for (i = 1; i < pkts; i++) { > > + dif = global.fwd_func(tbl[i], sif); > > + if (dif != dst_port) > > + break; > > + } > > + txq_id = arg->pktio[dst_port].txq[rxq_id]; > > + outq = global.l3fwd_pktios[dst_port].ifout[txq_id]; > > + sent = odp_pktout_send(outq, tbl, i); > > + if (odp_unlikely(sent < i)) { > > + sent = sent < 0 ? 0 : sent; > > + odp_packet_free_multi(&tbl[sent], i - sent); > > + arg->tx_drops += i - sent; > > + } > > + > > + if (i < pkts) > > + tbl += i; > > + > > + pkts -= i; > > + } > > +} > > + > > +static int run_worker(void *arg) > > +{ > > + int if_idx, rxq_idx; > > + struct thread_arg_s *thr_arg = arg; > > + struct l3fwd_pktio_s *port; > > + > > + odp_barrier_wait(&barrier); > > + > > + while (!exit_threads) { > > + for (if_idx = 0; if_idx < thr_arg->nb_pktio; if_idx++) { > > + if (!thr_arg->pktio[if_idx].used || > > + thr_arg->thr_idx == INVALID_ID) > > + continue; > > + > > + port = &global.l3fwd_pktios[if_idx]; > > + for (rxq_idx = 0; rxq_idx < port->rxq_idx; > rxq_idx++) > > + l3fwd_one_queue(if_idx, rxq_idx, arg); > > + } > > + } > > + > > + /* Make sure that latest stat writes are visible to other threads > */ > > + odp_mb_full(); > > + > > + return 0; > > +} > > + > > +static int find_port_id_by_name(char *name, app_args_t *args) > > +{ > > + int i; > > + > > + if (!name) > > + return -1; > > + > > + for (i = 0; i < args->if_count; i++) { > > + if (!strcmp(name, args->if_names[i])) > > + return i; > > + } > > + > > + return -1; > > +} > > + > > +/* split string into tokens */ > > +static int split_string(char *str, int stringlen, > > + char **tokens, int maxtokens, char delim) > > +{ > > + int i, tok = 0; > > + int tokstart = 1; /* first token is right at start of string */ > > + > > + if (str == NULL || tokens == NULL) > > + goto einval_error; > > + > > + for (i = 0; i < stringlen; i++) { > > + if (str[i] == '\0' || tok >= maxtokens) > > + break; > > + if (tokstart) { > > + tokstart = 0; > > + tokens[tok++] = &str[i]; > > + } > > + if (str[i] == delim) { > > + str[i] = '\0'; > > + tokstart = 1; > > + } > > + } > > + return tok; > > + > > +einval_error: > > + errno = EINVAL; > > + return -1; > > +} > > + > > +static int parse_config(char *cfg_str, app_args_t *args) > > +{ > > + char s[256]; > > + const char *p, *p0 = cfg_str; > > + char *end; > > + enum fieldnames { > > + FLD_PORT = 0, > > + FLD_QUEUE, > > + FLD_LCORE, > > + FLD_LAST > > + }; > > + unsigned long int_fld[FLD_LAST]; > > + char *str_fld[FLD_LAST]; > > + int i; > > + unsigned size; > > + int nb_qconfs = 0; > > + struct l3fwd_qconf_s *qconf_array = &args->qconf_config[0]; > > + > > + p = strchr(p0, '('); > > + while (p != NULL) { > > + ++p; > > + p0 = strchr(p, ')'); > > + if (p0 == NULL) > > + return -1; > > + > > + size = p0 - p; > > + if (size >= sizeof(s)) > > + return -1; > > + > > + snprintf(s, sizeof(s), "%.*s", size, p); > > + i = split_string(s, sizeof(s), str_fld, FLD_LAST, ','); > > + if (i != FLD_LAST) > > + return -1; > > + for (i = 0; i < FLD_LAST; i++) { > > + errno = 0; > > + int_fld[i] = strtoul(str_fld[i], &end, 0); > > + if (errno != 0 || end == str_fld[i] || int_fld[i] > > 255) > > + return -1; > > + } > > + if (nb_qconfs >= MAX_NB_QCONFS) { > > + printf("exceeded max number of queue > > params: %hu\n", > > + nb_qconfs); > > + return -1; > > + } > > + qconf_array[nb_qconfs].if_idx = (uint8_t)int_fld[FLD_PORT]; > > + qconf_array[nb_qconfs].rxq_idx = > > (uint8_t)int_fld[FLD_QUEUE]; > > + qconf_array[nb_qconfs].core_idx = > > (uint8_t)int_fld[FLD_LCORE]; > > + ++nb_qconfs; > > + > > + p = strchr(p0, '('); > > + } > > + args->qconf_count = nb_qconfs; > > + > > + return 0; > > +} > > + > > +static void print_usage(char *progname) > > +{ > > + printf("\n" > > + "ODP L3 forwarding application.\n" > > + "\n" > > + "Usage: %s OPTIONS\n" > > + " E.g. %s -i eth0,eth1 -r 1.1.1.0/24,eth0<http://1.1.1. > 0/24,eth0> -r > 2.2.2.0/24,eth1\n<http://2.2.2.0/24,eth1/n>" > > + " In the above example,\n" > > + " eth0 will send pkts to eth1 and vice versa\n" > > + "\n" > > + "Mandatory OPTIONS:\n" > > + " -i, --interface eth interfaces (comma-separated, no > spaces)\n" > > + " -r, --route SubNet,Intf[,NextHopMAC]\n" > > + " NextHopMAC can be optional\n" > > + "\n" > > + "Optional OPTIONS:\n" > > + " -s, --style [lpm|hash], ip lookup method\n" > > + " optional, default as lpm\n" > > + " -d, --duration Seconds to run and print stats\n" > > + " optional, default as 0, run forever\n" > > + " -t, --thread Number of threads to do forwarding\n" > > + " optional, default as availbe worker cpu count\n" > > + " -q, --queue Configure rx queue(s) for port\n" > > + " optional, format: [(port, queue, thread),...]\n" > > + " for example: -q '(0, 0, 1),(1,0,2)'\n" > > + " -h, --help Display help and exit.\n\n" > > + "\n", NO_PATH(progname), NO_PATH(progname) > > + ); > > +} > > + > > +static void parse_cmdline_args(int argc, char *argv[], app_args_t *args) > > +{ > > + int opt; > > + int long_index; > > + char *token, *local; > > + size_t len, route_index = 0; > > + int i, mem_failure = 0; > > + > > + static struct option longopts[] = { > > + {"interface", required_argument, NULL, 'i'}, /* return > 'i' > */ > > + {"route", required_argument, NULL, 'r'}, /* return > 'r' > */ > > + {"style", optional_argument, NULL, 's'}, /* return > 's' > */ > > + {"duration", optional_argument, NULL, 'd'}, /* return > 'd' > */ > > + {"thread", optional_argument, NULL, 't'}, /* return > 't' > */ > > + {"queue", optional_argument, NULL, 'q'}, /* return > 'q' > */ > > + {"help", no_argument, NULL, 'h'}, /* return > 'h' > */ > > + {NULL, 0, NULL, 0} > > + }; > > + > > + while (1) { > > + opt = getopt_long(argc, argv, "+s:t:d:i:r:q:h", > > + longopts, &long_index); > > + > > + if (opt == -1) > > + break; /* No more options */ > > + > > + switch (opt) { > > + /* parse ip lookup method */ > > + case 's': > > + if (!strcmp(optarg, "hash")) > > + args->hash_mode = 1; > > + break; > > + /* parse number of worker threads to be run*/ > > + case 't': > > + i = odp_cpu_count(); > > + args->worker_count = atoi(optarg); > > + if (args->worker_count > i) { > > + printf("Too many threads," > > + "truncate to cpu count: %d\n", i); > > + args->worker_count = i; > > + } > > + > > + break; > > + > > + /* parse seconds to run */ > > + case 'd': > > + args->duration = atoi(optarg); > > + break; > > + > > + /* parse packet-io interface names */ > > + case 'i': > > + len = strlen(optarg); > > + if (len == 0) { > > + print_usage(argv[0]); > > + exit(EXIT_FAILURE); > > + } > > + len += 1; /* add room for '\0' */ > > + > > + local = malloc(len); > > + if (!local) { > > + print_usage(argv[0]); > > + exit(EXIT_FAILURE); > > + } > > + > > + /* count the number of tokens separated by ',' */ > > + strcpy(local, optarg); > > + for (token = strtok(local, ","), i = 0; > > + token != NULL; > > + token = strtok(NULL, ","), i++) > > + ; > > + > > + if (i == 0) { > > + print_usage(argv[0]); > > + free(local); > > + exit(EXIT_FAILURE); > > + } else if (i > MAX_NB_PKTIO) { > > + printf("too many ports specified, " > > + "truncated to %d", MAX_NB_PKTIO); > > + } > > + args->if_count = i; > > + > > + /* store the if names (reset names string) */ > > + strcpy(local, optarg); > > + for (token = strtok(local, ","), i = 0; > > + token != NULL; token = strtok(NULL, ","), > i++) { > > + args->if_names[i] = token; > > + } > > + break; > > + > > + /*Configure Route in forwarding database*/ > > + case 'r': > > + if (route_index >= MAX_NB_ROUTE) { > > + printf("No more routes can be added\n"); > > + break; > > + } > > + local = calloc(1, strlen(optarg) + 1); > > + if (!local) { > > + mem_failure = 1; > > + break; > > + } > > + memcpy(local, optarg, strlen(optarg)); > > + local[strlen(optarg)] = '\0'; > > + args->route_str[route_index++] = local; > > + break; > > + > > + case 'h': > > + print_usage(argv[0]); > > + exit(EXIT_SUCCESS); > > + break; > > + > > + case 'q': > > + parse_config(optarg, args); > > + break; > > + > > + default: > > + break; > > + } > > + } > > + > > + /* checking arguments */ > > + if (args->if_count == 0) { > > + printf("\nNo option -i specified.\n"); > > + goto out; > > + } > > + > > + if (args->route_str[0] == NULL) { > > + printf("\nNo option -r specified.\n"); > > + goto out; > > + } > > + > > + if (mem_failure == 1) { > > + printf("\nAllocate memory failure.\n"); > > + goto out; > > + } > > + optind = 1; /* reset 'extern optind' from the getopt > lib > */ > > + return; > > + > > +out: > > + print_usage(argv[0]); > > + exit(EXIT_FAILURE); > > +} > > + > > +static void print_info(char *progname, app_args_t *args) > > +{ > > + int i; > > + > > + printf("\n" > > + "ODP system info\n" > > + "---------------\n" > > + "ODP API version: %s\n" > > + "ODP impl name: %s\n" > > + "CPU model: %s\n" > > + "CPU freq (hz): %" PRIu64 "\n" > > + "Cache line size: %i\n" > > + "CPU count: %i\n" > > + "\n", > > + odp_version_api_str(), odp_version_impl_name(), > > + odp_cpu_model_str(), odp_cpu_hz_max(), > > + odp_sys_cache_line_size(), odp_cpu_count()); > > + > > + printf("Running ODP appl: \"%s\"\n" > > + "-----------------\n" > > + "IP Lookup: %s\n" > > + "IF Count: %i\n" > > + "Using IFs: ", > > + progname, > > + args->hash_mode ? "hash" : "lpm", > > + args->if_count); > > + > > + for (i = 0; i < args->if_count; ++i) > > + printf(" %s", args->if_names[i]); > > + > > + printf("\n\n"); > > + fflush(NULL); > > +} > > + > > +/** > > + * Setup rx and tx queues, distribute them among threads. > > + * Try to have one tx queue for each rx queue, if not vailable, > > + * shared tx queue is used. > > + * > > + * If no q argument, the queues are distribute among threads as default. > > + * The thread take one rx queue of a port one time as round-robin order. > > + */ > > +static void setup_worker_qconf(app_args_t *args) > > +{ > > + int nb_worker, if_count; > > + int i, j, rxq_idx, txq_idx; > > + struct thread_arg_s *arg; > > + struct l3fwd_pktio_s *port; > > + uint8_t queue_mask[MAX_NB_PKTIO][MAX_NB_QUEUE]; > > + > > + nb_worker = args->worker_count; > > + if_count = args->if_count; > > + > > + /* distribute queues among threads */ > > + if (!args->qconf_count) { > > + if (nb_worker > if_count) { > > + for (i = 0; i < nb_worker; i++) { > > + arg = &global.worker_args[i]; > > + arg->thr_idx = i; > > + j = i % if_count; > > + port = &global.l3fwd_pktios[j]; > > + if (port->rxq_idx < port->nb_rxq) { > > + rxq_idx = port->rxq_idx; > > + arg->pktio[j].rxq[rxq_idx] = > rxq_idx; > > + port->rxq_idx++; > > + txq_idx = port->txq_idx; > > + arg->pktio[j].txq[txq_idx] = > txq_idx; > > + port->txq_idx++; > > + arg->pktio[j].used = 1; > > + } > > + } > > + } else { > > + for (i = 0; i < if_count; i++) { > > + j = i % nb_worker; > > + arg = &global.worker_args[j]; > > + arg->thr_idx = j; > > + port = &global.l3fwd_pktios[i]; > > + if (port->rxq_idx < port->nb_rxq) { > > + rxq_idx = port->rxq_idx; > > + arg->pktio[i].rxq[rxq_idx] = > rxq_idx; > > + port->rxq_idx++; > > + txq_idx = port->txq_idx; > > + arg->pktio[i].txq[txq_idx] = > txq_idx; > > + port->txq_idx++; > > + arg->pktio[i].used = 1; > > + } > > + } > > + } > > + } > > + > > + /* specified q argument, distribute queues among threads as it > says */ > > + memset(queue_mask, 0, sizeof(queue_mask)); > > + for (i = 0; i < args->qconf_count; i++) { > > + struct l3fwd_qconf_s *q; > > + > > + q = &args->qconf_config[i]; > > + if (q->core_idx >= nb_worker || q->if_idx >= if_count) > > + LOG_ABORT("Error queue (%d, %d, %d), max port: " > > + "%d, max core: %d\n", q->if_idx, > q->rxq_idx, > > + q->core_idx, args->if_count - 1, > > + args->worker_count - 1); > > + > > + /* check if one queue is configured twice or more */ > > + if (queue_mask[q->if_idx][q->rxq_idx]) > > + LOG_ABORT("Error queue (%d, %d, %d), reconfig > > queue\n", > > + q->if_idx, q->rxq_idx, q->core_idx); > > + queue_mask[q->if_idx][q->rxq_idx] = 1; > > + > > + port = &global.l3fwd_pktios[q->if_idx]; > > + if (port->rxq_idx < q->rxq_idx) > > + LOG_ABORT("Error queue (%d, %d, %d), queue should > > be" > > + " in sequence and start from 0, queue > %d\n", > > + q->if_idx, q->rxq_idx, q->core_idx, > > + q->rxq_idx); > > + > > + if (q->rxq_idx > port->nb_rxq) { > > + LOG_ABORT("Error queue (%d, %d, %d), max > > queue %d\n", > > + q->if_idx, q->rxq_idx, q->core_idx, > > + port->nb_rxq - 1); > > + } > > + port->rxq_idx = q->rxq_idx + 1; > > + port->txq_idx = q->rxq_idx + 1; > > + > > + /* put the queue into worker_args */ > > + arg = &global.worker_args[q->core_idx]; > > + arg->pktio[q->if_idx].rxq[q->rxq_idx] = q->rxq_idx; > > + arg->pktio[q->if_idx].txq[q->rxq_idx] = q->rxq_idx; > > + arg->pktio[q->if_idx].used = 1; > > + arg->thr_idx = q->core_idx; > > + } > > + > > + /* config and initialize rx and tx queues. */ > > + for (i = 0; i < if_count; i++) { > > + odp_pktin_queue_param_t in_queue_param; > > + odp_pktout_queue_param_t out_queue_param; > > + struct odp_pktin_queue_t *inq; > > + struct odp_pktout_queue_t *outq; > > + const char *name; > > + int rc, nb_rxq, nb_txq; > > + > > + port = &global.l3fwd_pktios[i]; > > + name = args->if_names[i]; > > + odp_pktin_queue_param_init(&in_queue_param); > > + odp_pktout_queue_param_init(&out_queue_param); > > + > > + in_queue_param.op_mode = ODP_PKTIO_OP_MT; > > + out_queue_param.op_mode = ODP_PKTIO_OP_MT; > > + > > + in_queue_param.num_queues = port->rxq_idx; > > + if (port->rxq_idx > port->nb_rxq) { > > + in_queue_param.num_queues = port->nb_rxq; > > + in_queue_param.op_mode = > > ODP_PKTIO_OP_MT_UNSAFE; > > + } > > + > > + /* enable flow hashing for multi-input queues */ > > + if (in_queue_param.num_queues > 1) { > > + in_queue_param.hash_enable = 1; > > + in_queue_param.hash_proto.proto.ipv4_tcp = 1; > > + in_queue_param.hash_proto.proto.ipv4_udp = 1; > > + } > > + > > + if (odp_pktin_queue_config(port->pktio, &in_queue_param)) > > + LOG_ABORT("Fail to config input queue for %s\n", > > name); > > + > > + out_queue_param.num_queues = port->txq_idx; > > + if (port->txq_idx > port->nb_txq) { > > + out_queue_param.num_queues = port->nb_txq; > > + out_queue_param.op_mode = > > ODP_PKTIO_OP_MT_UNSAFE; > > + } > > + if (odp_pktout_queue_config(port->pktio, > > &out_queue_param)) > > + LOG_ABORT("Fail to config output queue for %s\n", > > name); > > + > > + inq = port->ifin; > > + nb_rxq = in_queue_param.num_queues; > > + if (odp_pktin_queue(port->pktio, inq, nb_rxq) != nb_rxq) > > + LOG_ABORT("Fail to set pktin queue for %s\n", > name); > > + > > + if (port->rxq_idx > port->nb_rxq) { > > + for (rc = port->nb_rxq; rc < port->rxq_idx; rc++) > > + inq[rc] = inq[rc % port->nb_rxq]; > > + } > > + > > + outq = port->ifout; > > + nb_txq = out_queue_param.num_queues; > > + if (odp_pktout_queue(port->pktio, outq, nb_txq) != nb_txq) > > + LOG_ABORT("Fail to set pktout queue for %s\n", > > name); > > + > > + if (port->txq_idx > port->nb_txq) { > > + for (rc = port->nb_txq; rc < port->txq_idx; rc++) > > + outq[rc] = outq[rc % port->nb_txq]; > > + } > > + } > > +} > > + > > +static void print_qconf_table(app_args_t *args) > > +{ > > + int i, j, k, qid; > > + char buf[32]; > > + struct thread_arg_s *thr_arg; > > + > > + printf("Rx queue table\n" > > + "-----------------\n" > > + "%-32s%-16s%-16s\n", > > + "port/id", "queue", "thread"); > > + > > + for (i = 0; i < args->worker_count; i++) { > > + thr_arg = &global.worker_args[i]; > > + for (j = 0; j < args->if_count; j++) { > > + if (!thr_arg->pktio[j].used) > > + continue; > > + > > + snprintf(buf, 32, "%s/%d", args->if_names[j], j); > > + for (k = 0; k < MAX_NB_QUEUE; k++) { > > + qid = thr_arg->pktio[j].rxq[k]; > > + if (qid != INVALID_ID) > > + printf("%-32s%-16d%-16d\n", buf, > qid, > > + thr_arg->thr_idx); > > + } > > + } > > + } > > + printf("\n"); > > + fflush(NULL); > > +} > > + > > +/** > > + * Print statistics > > + * > > + * @param num_workers Number of worker threads > > + * @param duration Number of seconds to loop in > > + * @param timeout Number of seconds for stats calculation > > + * > > + */ > > +static int print_speed_stats(int num_workers, int duration, int timeout) > > +{ > > + uint64_t pkts = 0; > > + uint64_t pkts_prev = 0; > > + uint64_t pps; > > + uint64_t rx_drops, tx_drops; > > + uint64_t maximum_pps = 0; > > + int i; > > + int elapsed = 0; > > + int stats_enabled = 1; > > + int loop_forever = (duration == 0); > > + > > + if (timeout <= 0) { > > + stats_enabled = 0; > > + timeout = 1; > > + } > > + /* Wait for all threads to be ready*/ > > + odp_barrier_wait(&barrier); > > + > > + do { > > + pkts = 0; > > + rx_drops = 0; > > + tx_drops = 0; > > + sleep(timeout); > > + > > + for (i = 0; i < num_workers; i++) { > > + pkts += global.worker_args[i].packets; > > + rx_drops += global.worker_args[i].rx_drops; > > + tx_drops += global.worker_args[i].tx_drops; > > + } > > + if (stats_enabled) { > > + pps = (pkts - pkts_prev) / timeout; > > + if (pps > maximum_pps) > > + maximum_pps = pps; > > + printf("%" PRIu64 " pps, %" PRIu64 " max pps, ", > pps, > > + maximum_pps); > > + > > + printf(" %" PRIu64 " rx drops, %" PRIu64 " tx > drops\n", > > + rx_drops, tx_drops); > > + > > + pkts_prev = pkts; > > + } > > + elapsed += timeout; > > + } while (loop_forever || (elapsed < duration)); > > + > > + if (stats_enabled) > > + printf("TEST RESULT: %" PRIu64 " maximum packets per > > second.\n", > > + maximum_pps); > > + > > + return pkts > 100 ? 0 : -1; > > +} > > + > > +int main(int argc, char **argv) > > +{ > > + odph_odpthread_t thread_tbl[MAX_NB_WORKER]; > > + odp_pool_t pool; > > + odp_pool_param_t params; > > + odp_instance_t instance; > > + odph_odpthread_params_t thr_params; > > + odp_cpumask_t cpumask; > > + int cpu, i, j, nb_worker; > > + uint8_t mac[ODPH_ETHADDR_LEN]; > > + uint8_t *dst_mac; > > + app_args_t *args; > > + struct thread_arg_s *thr_arg; > > + char *oif; > > + > > + if (odp_init_global(&instance, NULL, NULL)) { > > + printf("Error: ODP global init failed.\n"); > > + exit(1); > > + } > > + > > + if (odp_init_local(instance, ODP_THREAD_CONTROL)) { > > + printf("Error: ODP local init failed.\n"); > > + exit(1); > > + } > > + > > + /* Clear global argument and initialize the dest mac as > 2:0:0:0:0:x */ > > + memset(&global, 0, sizeof(global)); > > + mac[0] = 2; > > + for (i = 0; i < MAX_NB_PKTIO; i++) { > > + mac[ODPH_ETHADDR_LEN - 1] = (uint8_t)i; > > + memcpy(global.eth_dest_mac[i].addr, mac, > > ODPH_ETHADDR_LEN); > > + } > > + > > + /* Initialize the thread arguments */ > > + for (i = 0; i < MAX_NB_WORKER; i++) { > > + thr_arg = &global.worker_args[i]; > > + for (j = 0; j < MAX_NB_PKTIO; j++) { > > + thr_arg->thr_idx = INVALID_ID; > > + memset(thr_arg->pktio[j].rxq, INVALID_ID, > > + sizeof(thr_arg->pktio[j].rxq)); > > + } > > + } > > + > > + /* Parse cmdline arguments */ > > + args = &global.cmd_args; > > + parse_cmdline_args(argc, argv, args); > > + > > + /* Init l3fwd table */ > > + init_fwd_db(); > > + > > + /* Add route into table */ > > + for (i = 0; i < MAX_NB_ROUTE; i++) { > > + if (args->route_str[i]) { > > + create_fwd_db_entry(args->route_str[i], &oif, > > &dst_mac); > > + if (oif == NULL) { > > + printf("Error: fail to create route > entry.\n"); > > + exit(1); > > + } > > + > > + j = find_port_id_by_name(oif, args); > > + if (j == -1) { > > + printf("Error: port %s not used.\n", oif); > > + exit(1); > > + } > > + > > + if (dst_mac) > > + args->dest_mac_changed[j] = 1; > > + } > > + } > > + > > + print_info(NO_PATH(argv[0]), args); > > + > > + /* Create packet pool */ > > + odp_pool_param_init(¶ms); > > + params.pkt.seg_len = POOL_SEG_LEN; > > + params.pkt.len = POOL_SEG_LEN; > > + params.pkt.num = POOL_NUM_PKT; > > + params.type = ODP_POOL_PACKET; > > + > > + pool = odp_pool_create("packet pool", ¶ms); > > + > > + if (pool == ODP_POOL_INVALID) { > > + printf("Error: packet pool create failed.\n"); > > + exit(1); > > + } > > + > > + /* Resolve fwd db*/ > > + for (i = 0; i < args->if_count; i++) { > > + struct l3fwd_pktio_s *port; > > + char *if_name; > > + > > + if_name = args->if_names[i]; > > + port = &global.l3fwd_pktios[i]; > > + if (create_pktio(if_name, pool, port)) { > > + printf("Error: create pktio %s\n", if_name); > > + exit(1); > > + } > > + odp_pktio_mac_addr(port->pktio, mac, ODPH_ETHADDR_LEN); > > + resolve_fwd_db(if_name, i, mac); > > + memcpy(port->mac_addr.addr, mac, ODPH_ETHADDR_LEN); > > + } > > + > > + setup_fwd_db(); > > + dump_fwd_db(); > > + > > + /* Dicide available workers */ > > + nb_worker = MAX_NB_WORKER; > > + if (args->worker_count) > > + nb_worker = args->worker_count; > > + nb_worker = odp_cpumask_default_worker(&cpumask, nb_worker); > > + args->worker_count = nb_worker; > > + > > + /* Setup rx and tx queues for each port */ > > + setup_worker_qconf(args); > > + print_qconf_table(args); > > + > > + /* Decide ip lookup method */ > > + if (args->hash_mode) > > + global.fwd_func = l3fwd_pkt_hash; > > + else > > + global.fwd_func = l3fwd_pkt_lpm; > > + > > + /* Start all the available ports */ > > + for (i = 0; i < args->if_count; i++) { > > + struct l3fwd_pktio_s *port; > > + char *if_name; > > + char buf[32]; > > + > > + if_name = args->if_names[i]; > > + port = &global.l3fwd_pktios[i]; > > + /* start pktio */ > > + if (odp_pktio_start(port->pktio)) { > > + printf("unable to start pktio: %s\n", if_name); > > + exit(1); > > + } > > + > > + sprintf(buf, "%02x:%02x:%02x:%02x:%02x:%02x", > > + port->mac_addr.addr[0], > > + port->mac_addr.addr[1], > > + port->mac_addr.addr[2], > > + port->mac_addr.addr[3], > > + port->mac_addr.addr[4], > > + port->mac_addr.addr[5]); > > + printf("start pktio: %s, mac %s\n", if_name, buf); > > + } > > + > > + odp_barrier_init(&barrier, nb_worker + 1); > > + > > + memset(&thr_params, 0, sizeof(thr_params)); > > + thr_params.start = run_worker; > > + thr_params.thr_type = ODP_THREAD_WORKER; > > + thr_params.instance = instance; > > + > > + memset(thread_tbl, 0, sizeof(thread_tbl)); > > + cpu = odp_cpumask_first(&cpumask); > > + for (i = 0; i < nb_worker; i++) { > > + struct thread_arg_s *arg; > > + odp_cpumask_t thr_mask; > > + > > + arg = &global.worker_args[i]; > > + arg->nb_pktio = args->if_count; > > + odp_cpumask_zero(&thr_mask); > > + odp_cpumask_set(&thr_mask, cpu); > > + thr_params.arg = arg; > > + odph_odpthreads_create(&thread_tbl[i], &thr_mask, > > + &thr_params); > > + cpu = odp_cpumask_next(&cpumask, cpu); > > + } > > + > > + if (args->duration) { > > + print_speed_stats(nb_worker, args->duration, > > PRINT_INTERVAL); > > + exit_threads = 1; > > + } > > + > > + /* wait for other threads to join */ > > + for (i = 0; i < nb_worker; i++) > > + odph_odpthreads_join(&thread_tbl[i]); > > + > > + /* release resource on exit */ > > + destroy_fwd_db(); > > + > > + /* if_names share a single buffer, so only one free */ > > + free(args->if_names[0]); > > + > > + for (i = 0; i < MAX_NB_ROUTE; i++) > > + free(args->route_str[i]); > > + > > + if (odp_pool_destroy(pool)) { > > + printf("Error: pool destroy\n"); > > + exit(EXIT_FAILURE); > > + } > > + > > + if (odp_term_local()) { > > + printf("Error: term local\n"); > > + exit(EXIT_FAILURE); > > + } > > + > > + if (odp_term_global(instance)) { > > + printf("Error: term global\n"); > > + exit(EXIT_FAILURE); > > + } > > + > > + return 0; > > +} > > diff --git a/example/l3fwd/odp_l3fwd_db.c b/example/l3fwd/odp_l3fwd_db.c > > new file mode 100644 > > index 0000000..b3a237f > > --- /dev/null > > +++ b/example/l3fwd/odp_l3fwd_db.c > > @@ -0,0 +1,432 @@ > > +/* Copyright (c) 2016, Linaro Limited > > + * All rights reserved. > > + * > > + * SPDX-License-Identifier: BSD-3-Clause > > + */ > > + > > +#ifndef _GNU_SOURCE > > +#define _GNU_SOURCE > > +#endif > > + > > +#include <stdlib.h> > > +#include <string.h> > > + > > +#include <example_debug.h> > > +#include <odp_api.h> > > +#include <odp_l3fwd_db.h> > > + > > +/** Jenkins hash support. > > + * > > + * Copyright (C) 2006 Bob Jenkins (bob_jenkins@burtleburtle.net< > mailto:bob_jenkins@burtleburtle.net>) > > + * > > + * http://burtleburtle.net/bob/hash/ > > + * > > + * These are the credits from Bob's sources: > > + * > > + * lookup3.c, by Bob Jenkins, May 2006, Public Domain. > > + * > > + * These are functions for producing 32-bit hashes for hash table > lookup. > > + * hashword(), hashlittle(), hashlittle2(), hashbig(), mix(), and > final() > > + * are externally useful functions. Routines to test the hash are > included > > + * if SELF_TEST is defined. You can use this free for any purpose. > It's > in > > + * the public domain. It has no warranty. > > + * > > + * $FreeBSD$ > > + */ > > +#define JHASH_GOLDEN_RATIO 0x9e3779b9 > > +#define rot(x, k) (((x) << (k)) | ((x) >> (32 - (k)))) > > +#define FWD_BJ3_MIX(a, b, c) \ > > +{ \ > > + a -= c; a ^= rot(c, 4); c += b; \ > > + b -= a; b ^= rot(a, 6); a += c; \ > > + c -= b; c ^= rot(b, 8); b += a; \ > > + a -= c; a ^= rot(c, 16); c += b; \ > > + b -= a; b ^= rot(a, 19); a += c; \ > > + c -= b; c ^= rot(b, 4); b += a; \ > > +} > > + > > +/** > > + * Compute hash value from a flow > > + */ > > +static inline > > +uint64_t l3fwd_calc_hash(ipv4_tuple5_t *key) > > +{ > > + uint64_t l4_ports = 0; > > + uint32_t dst_ip, src_ip; > > + > > + src_ip = key->src_ip; > > + dst_ip = key->dst_ip + JHASH_GOLDEN_RATIO; > > + FWD_BJ3_MIX(src_ip, dst_ip, l4_ports); > > + > > + return l4_ports; > > +} > > + > > +/** > > + * Parse text string representing an IPv4 address or subnet > > + * > > + * String is of the format "XXX.XXX.XXX.XXX(/W)" where > > + * "XXX" is decimal value and "/W" is optional subnet length > > + * > > + * @param ipaddress Pointer to IP address/subnet string to convert > > + * @param addr Pointer to return IPv4 address, host endianness > > + * @param depth Pointer to subnet bit width > > + * @return 0 if successful else -1 > > + */ > > +static inline > > +int parse_ipv4_string(char *ipaddress, uint32_t *addr, uint32_t *depth) > > +{ > > + int b[4]; > > + int qualifier = 32; > > + int converted; > > + uint32_t addr_le; > > + > > + if (strchr(ipaddress, '/')) { > > + converted = sscanf(ipaddress, "%d.%d.%d.%d/%d", > > + &b[3], &b[2], &b[1], &b[0], > > + &qualifier); > > + if (5 != converted) > > + return -1; > > + } else { > > + converted = sscanf(ipaddress, "%d.%d.%d.%d", > > + &b[3], &b[2], &b[1], &b[0]); > > + if (4 != converted) > > + return -1; > > + } > > + > > + if ((b[0] > 255) || (b[1] > 255) || (b[2] > 255) || (b[3] > 255)) > > + return -1; > > + if (!qualifier || (qualifier > 32)) > > + return -1; > > + > > + addr_le = b[0] | b[1] << 8 | b[2] << 16 | b[3] << 24; > > + *addr = odp_le_to_cpu_32(addr_le); > > + *depth = qualifier; > > + > > + return 0; > > +} > > + > > +/** > > + * Generate text string representing IPv4 range/subnet, output > > + * in "XXX.XXX.XXX.XXX/W" format > > + * > > + * @param b Pointer to buffer to store string > > + * @param range Pointer to IPv4 address range > > + * > > + * @return Pointer to supplied buffer > > + */ > > +static inline > > +char *ipv4_subnet_str(char *b, ip_addr_range_t *range) > > +{ > > + sprintf(b, "%d.%d.%d.%d/%d", > > + 0xFF & ((range->addr) >> 24), > > + 0xFF & ((range->addr) >> 16), > > + 0xFF & ((range->addr) >> 8), > > + 0xFF & ((range->addr) >> 0), > > + range->depth); > > + return b; > > +} > > + > > +/** > > + * Generate text string representing MAC address > > + * > > + * @param b Pointer to buffer to store string > > + * @param mac Pointer to MAC address > > + * > > + * @return Pointer to supplied buffer > > + */ > > +static inline > > +char *mac_addr_str(char *b, odph_ethaddr_t *mac) > > +{ > > + uint8_t *byte; > > + > > + byte = mac->addr; > > + sprintf(b, "%02X:%02X:%02X:%02X:%02X:%02X", > > + byte[0], byte[1], byte[2], byte[3], byte[4], byte[5]); > > + return b; > > +} > > + > > +/** > > + * Flow cache table entry > > + */ > > +typedef struct flow_entry_s { > > + ipv4_tuple5_t key; /**< match key */ > > + struct flow_entry_s *next; /**< next entry on the list */ > > + fwd_db_entry_t *fwd_entry; /**< entry info in db */ > > +} flow_entry_t; > > + > > +/** > > + * Flow cache table bucket > > + */ > > +typedef struct flow_bucket_s { > > + odp_spinlock_t lock; /**< Bucket lock*/ > > + flow_entry_t *next; /**< Pointer to first flow entry in > > bucket*/ > > +} flow_bucket_t; > > + > > +/** > > + * Flow hash table, fast lookup cache > > + */ > > +typedef struct flow_table_s { > > + flow_bucket_t *bucket; > > + uint32_t count; > > +} flow_table_t; > > + > > +static flow_table_t fwd_lookup_cache; > > + > > +void init_fwd_hash_cache(void) > > +{ > > + odp_shm_t hash_shm; > > + flow_bucket_t *bucket; > > + uint32_t bucket_count; > > + uint32_t i; > > + > > + bucket_count = FWD_DEF_BUCKET_COUNT; > > + > > + /*Reserve memory for Routing hash table*/ > > + hash_shm = odp_shm_reserve("route_table", > > + sizeof(flow_bucket_t) * bucket_count, > > + ODP_CACHE_LINE_SIZE, 0); > > + > > + bucket = odp_shm_addr(hash_shm); > > + if (!bucket) { > > + EXAMPLE_ERR("Error: shared mem alloc failed.\n"); > > + exit(-1); > > + } > > + > > + fwd_lookup_cache.bucket = bucket; > > + fwd_lookup_cache.count = bucket_count; > > + > > + /*Initialize Locks*/ > > + for (i = 0; i < bucket_count; i++) { > > + bucket = &fwd_lookup_cache.bucket[i]; > > + odp_spinlock_init(&bucket->lock); > > + bucket->next = NULL; > > + } > > +} > > + > > +static inline > > +int match_key_flow(ipv4_tuple5_t *key, flow_entry_t *flow) > > +{ > > + if (key->src_ip == flow->key.src_ip && > > + key->dst_ip == flow->key.dst_ip && > > + key->src_port == flow->key.src_port && > > + key->dst_port == flow->key.dst_port && > > + key->proto == flow->key.proto) > > + return 1; > > + > > + return 0; > > +} > > + > > +static inline > > +flow_entry_t *lookup_fwd_cache(ipv4_tuple5_t *key, flow_bucket_t > > *bucket) > > +{ > > + flow_entry_t *rst; > > + > > + for (rst = bucket->next; rst != NULL; rst = rst->next) { > > + if (match_key_flow(key, rst)) > > + break; > > + } > > + > > + return rst; > > +} > > + > > +static inline > > +flow_entry_t *insert_fwd_cache(ipv4_tuple5_t *key, > > + flow_bucket_t *bucket, > > + fwd_db_entry_t *entry) > > +{ > > + flow_entry_t *flow; > > + > > + if (!entry) > > + return NULL; > > + > > + flow = malloc(sizeof(flow_entry_t)); > > + flow->key = *key; > > + flow->fwd_entry = entry; > > + > > + odp_spinlock_lock(&bucket->lock); > > + if (!bucket->next) { > > + bucket->next = flow; > > + } else { > > + flow->next = bucket->next; > > + bucket->next = flow; > > + } > > + odp_spinlock_unlock(&bucket->lock); > > + > > + return flow; > > +} > > + > > +/** Global pointer to fwd db */ > > +fwd_db_t *fwd_db; > > + > > +void init_fwd_db(void) > > +{ > > + odp_shm_t shm; > > + > > + shm = odp_shm_reserve("shm_fwd_db", > > + sizeof(fwd_db_t), > > + ODP_CACHE_LINE_SIZE, > > + 0); > > + > > + fwd_db = odp_shm_addr(shm); > > + > > + if (fwd_db == NULL) { > > + EXAMPLE_ERR("Error: shared mem alloc failed.\n"); > > + exit(EXIT_FAILURE); > > + } > > + memset(fwd_db, 0, sizeof(*fwd_db)); > > +} > > + > > +int create_fwd_db_entry(char *input, char **oif, uint8_t **dst_mac) > > +{ > > + int pos = 0; > > + char *local; > > + char *str; > > + char *save; > > + char *token; > > + fwd_db_entry_t *entry = &fwd_db->array[fwd_db->index]; > > + > > + *oif = NULL; > > + *dst_mac = NULL; > > + > > + /* Verify we haven't run out of space */ > > + if (MAX_DB <= fwd_db->index) > > + return -1; > > + > > + /* Make a local copy */ > > + local = malloc(strlen(input) + 1); > > + if (NULL == local) > > + return -1; > > + strcpy(local, input); > > + > > + /* Setup for using "strtok_r" to search input string */ > > + str = local; > > + save = NULL; > > + > > + /* Parse tokens separated by ',' */ > > + while (NULL != (token = strtok_r(str, ",", &save))) { > > + str = NULL; /* reset str for subsequent strtok_r calls */ > > + > > + /* Parse token based on its position */ > > + switch (pos) { > > + case 0: > > + parse_ipv4_string(token, > > + &entry->subnet.addr, > > + &entry->subnet.depth); > > + break; > > + case 1: > > + strncpy(entry->oif, token, OIF_LEN - 1); > > + entry->oif[OIF_LEN - 1] = 0; > > + *oif = entry->oif; > > + break; > > + case 2: > > + odph_eth_addr_parse(&entry->dst_mac, token); > > + *dst_mac = entry->dst_mac.addr; > > + break; > > + > > + default: > > + printf("ERROR: extra token \"%s\" at position > %d\n", > > + token, pos); > > + break; > > + } > > + > > + /* Advance to next position */ > > + pos++; > > + } > > + > > + /* Add route to the list */ > > + fwd_db->index++; > > + entry->next = fwd_db->list; > > + fwd_db->list = entry; > > + > > + free(local); > > + return 0; > > +} > > + > > +void resolve_fwd_db(char *intf, int portid, uint8_t *mac) > > +{ > > + fwd_db_entry_t *entry; > > + > > + /* Walk the list and attempt to set output and MAC */ > > + for (entry = fwd_db->list; NULL != entry; entry = entry->next) { > > + if (strcmp(intf, entry->oif)) > > + continue; > > + > > + entry->oif_id = portid; > > + memcpy(entry->src_mac.addr, mac, ODPH_ETHADDR_LEN); > > + } > > +} > > + > > +void dump_fwd_db_entry(fwd_db_entry_t *entry) > > +{ > > + char subnet_str[MAX_STRING]; > > + char mac_str[MAX_STRING]; > > + > > + mac_addr_str(mac_str, &entry->dst_mac); > > + printf("%-32s%-32s%-16s\n", > > + ipv4_subnet_str(subnet_str, &entry->subnet), > > + entry->oif, mac_str); > > +} > > + > > +void dump_fwd_db(void) > > +{ > > + fwd_db_entry_t *entry; > > + > > + printf("Routing table\n" > > + "-----------------\n" > > + "%-32s%-32s%-16s\n", > > + "subnet", "next_hop", "dest_mac"); > > + > > + for (entry = fwd_db->list; NULL != entry; entry = entry->next) > > + dump_fwd_db_entry(entry); > > + > > + printf("\n"); > > +} > > + > > +void destroy_fwd_db(void) > > +{ > > + flow_bucket_t *bucket; > > + flow_entry_t *flow, *tmp; > > + uint32_t i; > > + > > + for (i = 0; i < fwd_lookup_cache.count; i++) { > > + bucket = &fwd_lookup_cache.bucket[i]; > > + flow = bucket->next; > > + odp_spinlock_lock(&bucket->lock); > > + while (flow) { > > + tmp = flow->next; > > + free(flow); > > + flow = tmp; > > + } > > + odp_spinlock_unlock(&bucket->lock); > > + } > > +} > > + > > +fwd_db_entry_t *find_fwd_db_entry(ipv4_tuple5_t *key) > > +{ > > + fwd_db_entry_t *entry; > > + flow_entry_t *flow; > > + flow_bucket_t *bucket; > > + uint64_t hash; > > + > > + /* first find in cache */ > > + hash = l3fwd_calc_hash(key); > > + hash &= fwd_lookup_cache.count - 1; > > + bucket = &fwd_lookup_cache.bucket[hash]; > > + flow = lookup_fwd_cache(key, bucket); > > + if (flow) > > + return flow->fwd_entry; > > + > > + for (entry = fwd_db->list; NULL != entry; entry = entry->next) { > > + uint32_t mask; > > + > > + mask = ((1u << entry->subnet.depth) - 1) << > > + (32 - entry->subnet.depth); > > + > > + if (entry->subnet.addr == (key->dst_ip & mask)) > > + break; > > + } > > + > > + insert_fwd_cache(key, bucket, entry); > > + > > + return entry; > > +} > > diff --git a/example/l3fwd/odp_l3fwd_db.h > > b/example/l3fwd/odp_l3fwd_db.h > > new file mode 100644 > > index 0000000..d9f6d61 > > --- /dev/null > > +++ b/example/l3fwd/odp_l3fwd_db.h > > @@ -0,0 +1,136 @@ > > +/* Copyright (c) 2016, Linaro Limited > > + * All rights reserved. > > + * > > + * SPDX-License-Identifier: BSD-3-Clause > > + */ > > + > > +#ifndef _ODP_L3FWD_DB_H_ > > +#define _ODP_L3FWD_DB_H_ > > + > > +#ifdef __cplusplus > > +extern "C" { > > +#endif > > + > > +#include <odp_api.h> > > +#include <odp/helper/eth.h> > > + > > +#define OIF_LEN 32 > > +#define MAX_DB 32 > > +#define MAX_STRING 32 > > + > > +/** > > + * Default number of flows > > + */ > > +#define FWD_DEF_FLOW_COUNT 100000 > > + > > +/** > > + * Default Hash bucket number > > + */ > > +#define FWD_DEF_BUCKET_COUNT (FWD_DEF_FLOW_COUNT / 8) > > + > > +/** > > + * IP address range (subnet) > > + */ > > +typedef struct ip_addr_range_s { > > + uint32_t addr; /**< IP address, host endianness */ > > + uint32_t depth; /**< subnet bit width */ > > +} ip_addr_range_t; > > + > > +/** > > + * TCP/UDP flow > > + */ > > +typedef struct ipv4_tuple5_s { > > + uint32_t src_ip; > > + uint32_t dst_ip; > > + uint16_t src_port; > > + uint16_t dst_port; > > + uint8_t proto; > > +} ipv4_tuple5_t; > > + > > +/** > > + * Forwarding data base entry > > + */ > > +typedef struct fwd_db_entry_s { > > + struct fwd_db_entry_s *next; /**< Next entry on list */ > > + char oif[OIF_LEN]; /**< Output interface name */ > > + int oif_id; /**< Output interface idx */ > > + odph_ethaddr_t src_mac; /**< Output source MAC */ > > + odph_ethaddr_t dst_mac; /**< Output destination > > MAC */ > > + ip_addr_range_t subnet; /**< Subnet for this router > > */ > > +} fwd_db_entry_t; > > + > > +/** > > + * Forwarding data base > > + */ > > +typedef struct fwd_db_s { > > + uint32_t index; /**< Next available entry */ > > + fwd_db_entry_t *list; /**< List of active routes */ > > + fwd_db_entry_t array[MAX_DB]; /**< Entry storage */ > > +} fwd_db_t; > > + > > +/** Global pointer to fwd db */ > > +extern fwd_db_t *fwd_db; > > + > > +/** > > + * Initialize FWD DB > > + */ > > +void init_fwd_db(void); > > + > > +/** > > + * Initialize forward lookup cache based on hash > > + */ > > +void init_fwd_hash_cache(void); > > + > > +/** > > + * Create a forwarding database entry > > + * > > + * String is of the format "SubNet,Intf,NextHopMAC" > > + * > > + * @param input Pointer to string describing route > > + * @param oif Pointer to out interface name, as a return value > > + * @param dst_mac Pointer to dest mac for output packet, as a return > value > > + * > > + * @return 0 if successful else -1 > > + */ > > +int create_fwd_db_entry(char *input, char **oif, uint8_t **dst_mac); > > + > > +/** > > + * Scan FWD DB entries and resolve output queue and source MAC address > > + * > > + * @param intf Interface name string > > + * @param portid Output queue for packet transmit > > + * @param mac MAC address of this interface > > + */ > > +void resolve_fwd_db(char *intf, int portid, uint8_t *mac); > > + > > +/** > > + * Display one forwarding database entry > > + * > > + * @param entry Pointer to entry to display > > + */ > > +void dump_fwd_db_entry(fwd_db_entry_t *entry); > > + > > +/** > > + * Display the forwarding database > > + */ > > +void dump_fwd_db(void); > > + > > +/** > > + * Destroy the forwarding database > > + */ > > +void destroy_fwd_db(void); > > + > > +/** > > + * Find a matching forwarding database entry > > + * > > + * @param key ipv4 tuple > > + * > > + * @return pointer to forwarding DB entry else NULL > > + */ > > +fwd_db_entry_t *find_fwd_db_entry(ipv4_tuple5_t *key); > > + > > +#ifdef __cplusplus > > +} > > +#endif > > + > > +#endif > > diff --git a/example/l3fwd/odp_l3fwd_lpm.c > > b/example/l3fwd/odp_l3fwd_lpm.c > > new file mode 100644 > > index 0000000..1b3bfcf > > --- /dev/null > > +++ b/example/l3fwd/odp_l3fwd_lpm.c > > @@ -0,0 +1,224 @@ > > +/* Copyright (c) 2016, Linaro Limited > > + * All rights reserved. > > + * > > + * SPDX-License-Identifier: BSD-3-Clause > > + */ > > +#ifndef _GNU_SOURCE > > +#define _GNU_SOURCE > > +#endif > > + > > +#include <stdio.h> > > +#include <stdlib.h> > > + > > +#include <example_debug.h> > > +#include <odp_api.h> > > + > > +#include <odp_l3fwd_lpm.h> > > + > > +/** > > + * This is a simple implementation of lpm based on patricia tree. > > + * > > + * Tradeoff exists between memory consumption and lookup time. > > + * Currently it prefers 5 levels: {16, 4, 4, 4, 4}, could be 3 > > + * levels: {16, 8, 8} by defining FIB_NEXT_STRIDE as 8. Other > > + * levels are also possible. > > + * > > + * the ip here is host endian, when doing init or lookup, the > > + * caller should do endianness conversion if needed. > > + */ > > + > > +#define FIB_IP_WIDTH 32 > > +#define FIB_FIRST_STRIDE 16 > > +#define FIB_NEXT_STRIDE 4 > > +#define FIB_NEXT_SIZE (1 << FIB_NEXT_STRIDE) > > +#define FIB_SUB_COUNT 16384 > > +#define DEPTH_TO_MASK(depth) ((1 << (depth)) - 1) > > + > > +typedef struct fib_node_s { > > + union { > > + uint32_t next_hop; > > + struct fib_node_s *next; /* next level table */ > > + }; > > + uint8_t valid :1; /* 1, this node has a valid next hop */ > > + uint8_t end :1; /* 0, next points to the extended table */ > > + uint8_t depth :6; /* bit length of subnet mask */ > > +} fib_node_t; > > + > > +typedef struct fib_sub_tbl_s { > > + fib_node_t *fib_nodes; > > + uint32_t fib_count; > > + uint32_t fib_idx; > > +} fib_sub_tbl_t; > > + > > +static fib_node_t fib_rt_tbl[1 << FIB_FIRST_STRIDE]; > > +static fib_sub_tbl_t fib_lpm_cache; > > + > > +static inline fib_node_t *fib_alloc_sub(void) > > +{ > > + fib_node_t *sub_tbl = NULL; > > + uint32_t i, nb_entry; > > + > > + /* extend to next level */ > > + if (fib_lpm_cache.fib_idx < fib_lpm_cache.fib_count) { > > + nb_entry = (fib_lpm_cache.fib_idx + 1) * FIB_NEXT_SIZE; > > + sub_tbl = &fib_lpm_cache.fib_nodes[nb_entry]; > > + fib_lpm_cache.fib_idx++; > > + for (i = 0; i < nb_entry; i++) { > > + sub_tbl[i].valid = 0; > > + sub_tbl[i].end = 1; > > + } > > + } > > + > > + return sub_tbl; > > +} > > + > > +static void fib_update_node(fib_node_t *fe, int port, int depth) > > +{ > > + fib_node_t *p; > > + int i; > > + > > + if (fe->end) { > > + if (!fe->valid) { > > + fe->depth = depth; > > + fe->next_hop = port; > > + fe->valid = 1; > > + } else if (fe->depth <= depth) { > > + fe->next_hop = port; > > + fe->depth = depth; > > + } > > + > > + return; > > + } > > + > > + for (i = 0; i < FIB_NEXT_SIZE; i++) { > > + p = &fe->next[i]; > > + if (p->end) > > + fib_update_node(p, port, depth); > > + } > > +} > > + > > +static void fib_insert_node(fib_node_t *fe, uint32_t ip, uint32_t > next_hop, > > + int ip_width, int eat_bits, int depth) > > +{ > > + int i; > > + uint32_t idx, port; > > + fib_node_t *p; > > + > > + if (fe->end) { > > + port = fe->next_hop; > > + p = fib_alloc_sub(); > > + if (!p) > > + return; > > + > > + fe->next = p; > > + fe->end = 0; > > + if (fe->valid) { > > + for (i = 0; i < FIB_NEXT_SIZE; i++) { > > + p = &fe->next[i]; > > + p->next_hop = port; > > + p->depth = fe->depth; > > + } > > + } > > + } > > + if (depth - eat_bits <= FIB_NEXT_STRIDE) { > > + ip_width -= depth - eat_bits; > > + idx = ip >> ip_width; > > + ip &= DEPTH_TO_MASK(ip_width); > > + p = &fe->next[idx]; > > + fib_update_node(p, next_hop, depth); > > + } else { > > + ip_width -= FIB_NEXT_STRIDE; > > + idx = ip >> ip_width; > > + p = &fe->next[idx]; > > + ip &= DEPTH_TO_MASK(ip_width); > > + eat_bits += FIB_NEXT_STRIDE; > > + fib_insert_node(p, ip, next_hop, ip_width, eat_bits, > depth); > > + } > > +} > > + > > +void fib_tbl_init(void) > > +{ > > + int i; > > + fib_node_t *fe; > > + uint32_t size; > > + odp_shm_t lpm_shm; > > + > > + for (i = 0; i < (1 << FIB_FIRST_STRIDE); i++) { > > + fe = &fib_rt_tbl[i]; > > + fe->valid = 0; > > + fe->end = 1; > > + fe->depth = 0; > > + fe->next_hop = 0; > > + } > > + > > + size = FIB_NEXT_SIZE * FIB_SUB_COUNT; > > + /*Reserve memory for Routing hash table*/ > > + lpm_shm = odp_shm_reserve("fib_lpm_sub", size, > > ODP_CACHE_LINE_SIZE, 0); > > + fe = odp_shm_addr(lpm_shm); > > + if (!fe) { > > + EXAMPLE_ERR("Error: shared mem alloc failed for lpm > > cache.\n"); > > + exit(-1); > > + } > > + > > + fib_lpm_cache.fib_nodes = fe; > > + fib_lpm_cache.fib_count = FIB_SUB_COUNT; > > + fib_lpm_cache.fib_idx = 0; > > +} > > + > > +void fib_tbl_insert(uint32_t ip, int port, int depth) > > +{ > > + fib_node_t *fe, *p; > > + uint32_t idx; > > + int i, j; > > + int nb_bits; > > + > > + nb_bits = FIB_FIRST_STRIDE; > > + idx = ip >> nb_bits; > > + fe = &fib_rt_tbl[idx]; > > + if (depth <= nb_bits) { > > + if (fe->end) { > > + fe->next_hop = port; > > + fe->depth = depth; > > + fe->valid = 1; > > + return; > > + } > > + > > + for (i = 0; i < FIB_NEXT_SIZE; i++) { > > + p = &fe->next[i]; > > + if (p->end) > > + fib_update_node(p, port, depth); > > + else > > + for (j = 0; j < FIB_NEXT_SIZE; j++) > > + fib_update_node(&p->next[j], port, > > + depth); > > + } > > + > > + return; > > + } > > + > > + /* need to check sub table */ > > + ip &= DEPTH_TO_MASK(FIB_IP_WIDTH - nb_bits); > > + fib_insert_node(fe, ip, port, FIB_IP_WIDTH - nb_bits, nb_bits, > depth); > > +} > > + > > +int fib_tbl_lookup(uint32_t ip, int *port) > > +{ > > + fib_node_t *fe; > > + uint32_t idx; > > + int nb_bits; > > + > > + nb_bits = FIB_IP_WIDTH - FIB_FIRST_STRIDE; > > + idx = ip >> nb_bits; > > + fe = &fib_rt_tbl[idx]; > > + > > + ip &= DEPTH_TO_MASK(nb_bits); > > + while (!fe->end) { > > + nb_bits -= FIB_NEXT_STRIDE; > > + idx = ip >> nb_bits; > > + fe = &fe->next[idx]; > > + ip &= DEPTH_TO_MASK(nb_bits); > > + } > > + *port = fe->next_hop; > > + > > + return fe->valid ? 0 : -1; > > +} > > diff --git a/example/l3fwd/odp_l3fwd_lpm.h > > b/example/l3fwd/odp_l3fwd_lpm.h > > new file mode 100644 > > index 0000000..9256642 > > --- /dev/null > > +++ b/example/l3fwd/odp_l3fwd_lpm.h > > @@ -0,0 +1,20 @@ > > +/* Copyright (c) 2016, Linaro Limited > > + * All rights reserved. > > + * > > + * SPDX-License-Identifier: BSD-3-Clause > > + */ > > + > > +#ifndef _ODP_L3FWD_LPM_H_ > > +#define _ODP_L3FWD_LPM_H_ > > + > > +#ifdef __cplusplus > > +extern "C" { > > +#endif > > +void fib_tbl_init(void); > > +void fib_tbl_insert(uint32_t ip, int port, int depth); > > +int fib_tbl_lookup(uint32_t ip, int *port); > > +#ifdef __cplusplus > > +} > > +#endif > > + > > +#endif > > diff --git a/example/m4/configure.m4 b/example/m4/configure.m4 > > index bbda38f..620db04 100644 > > --- a/example/m4/configure.m4 > > +++ b/example/m4/configure.m4 > > @@ -12,12 +12,13 @@ AC_ARG_ENABLE([test-example], > > > > AC_CONFIG_FILES([example/classifier/Makefile > > example/generator/Makefile > > + example/hello/Makefile > > example/ipsec/Makefile > > - example/Makefile > > + example/l2fwd_simple/Makefile > > + example/l3fwd/Makefile > > example/packet/Makefile > > + example/switch/Makefile > > example/time/Makefile > > example/timer/Makefile > > example/traffic_mgmt/Makefile > > - example/l2fwd_simple/Makefile > > - example/switch/Makefile > > - example/hello/Makefile]) > > + example/Makefile]) > > -- > > 2.1.4 > > > > -- > Mike Holmes > Technical Manager - Linaro Networking Group > Linaro.org<http://www.linaro.org/> │ Open source software for ARM SoCs > "Work should be fun and collaborative, the rest follows" > > >
diff --git a/example/Makefile.am b/example/Makefile.am index 37542af..dfc07b6 100644 --- a/example/Makefile.am +++ b/example/Makefile.am @@ -1 +1,11 @@ -SUBDIRS = classifier generator ipsec packet time timer traffic_mgmt l2fwd_simple switch hello +SUBDIRS = classifier \ + generator \ + hello \ + ipsec \ + l2fwd_simple \ + l3fwd \ + packet \ + switch \ + time \ + timer \ + traffic_mgmt diff --git a/example/l3fwd/.gitignore b/example/l3fwd/.gitignore new file mode 100644 index 0000000..74e501f --- /dev/null +++ b/example/l3fwd/.gitignore @@ -0,0 +1 @@ +odp_l3fwd diff --git a/example/l3fwd/Makefile.am b/example/l3fwd/Makefile.am new file mode 100644 index 0000000..f926717 --- /dev/null +++ b/example/l3fwd/Makefile.am @@ -0,0 +1,12 @@ +include $(top_srcdir)/example/Makefile.inc + +bin_PROGRAMS = odp_l3fwd$(EXEEXT) +odp_l3fwd_LDFLAGS = $(AM_LDFLAGS) -static +odp_l3fwd_CFLAGS = $(AM_CFLAGS) -I${top_srcdir}/example -I${top_srcdir}/test + +noinst_HEADERS = \ + $(top_srcdir)/example/l3fwd/odp_l3fwd_db.h \ + $(top_srcdir)/example/l3fwd/odp_l3fwd_lpm.h \ + $(top_srcdir)/example/example_debug.h + +dist_odp_l3fwd_SOURCES = odp_l3fwd.c odp_l3fwd_db.c odp_l3fwd_lpm.c diff --git a/example/l3fwd/odp_l3fwd.c b/example/l3fwd/odp_l3fwd.c new file mode 100644 index 0000000..930e432 --- /dev/null +++ b/example/l3fwd/odp_l3fwd.c @@ -0,0 +1,1078 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#include <stdlib.h> +#include <stdio.h> +#include <errno.h> +#include <getopt.h> +#include <unistd.h> +#include <inttypes.h> + +#include <test_debug.h> + +#include <odp_api.h> +#include <odp/helper/linux.h> +#include <odp/helper/eth.h> +#include <odp/helper/ip.h> +#include <odp/helper/udp.h> +#include <odp/helper/tcp.h> + +#include "odp_l3fwd_db.h" +#include "odp_l3fwd_lpm.h" + +#define POOL_NUM_PKT 8192 +#define POOL_SEG_LEN 1856 +#define MAX_PKT_BURST 32 + +#define MAX_NB_WORKER 32 +#define MAX_NB_PKTIO 32 +#define MAX_NB_QUEUE 32 +#define MAX_NB_QCONFS 1024 +#define MAX_NB_ROUTE 32 + +#define INVALID_ID (-1) +#define PRINT_INTERVAL 10 /* interval seconds of printing stats */ + +/** Get rid of path in filename - only for unix-type paths using '/' */ +#define NO_PATH(file_name) (strrchr((file_name), '/') ? \ + strrchr((file_name), '/') + 1 : (file_name)) + +struct l3fwd_pktio_s { + odp_pktio_t pktio; + odph_ethaddr_t mac_addr; + odp_pktin_queue_t ifin[MAX_NB_QUEUE]; + odp_pktout_queue_t ifout[MAX_NB_QUEUE]; + int nb_rxq; + int nb_txq; + int rxq_idx; + int txq_idx; +}; + +struct l3fwd_qconf_s { + uint8_t if_idx; /* port index */ + uint8_t rxq_idx; /* recv queue index in a port */ + uint8_t core_idx; /* this core should handle traffic */ +}; + +struct thread_arg_s { + uint64_t packets; + uint64_t rx_drops; + uint64_t tx_drops; + struct { + int used; + int rxq[MAX_NB_QUEUE]; + int txq[MAX_NB_QUEUE]; + } pktio[MAX_NB_PKTIO]; + int nb_pktio; + int thr_idx; +}; + +typedef struct { + char *if_names[MAX_NB_PKTIO]; + int if_count; + char *route_str[MAX_NB_ROUTE]; + int worker_count; + struct l3fwd_qconf_s qconf_config[MAX_NB_QCONFS]; + int qconf_count; + uint32_t duration; /* seconds to run */ + uint8_t hash_mode; /* 1:hash, 0:lpm */ + uint8_t dest_mac_changed[MAX_NB_PKTIO]; /* 1: dest mac from cmdline */ +} app_args_t; + +struct { + app_args_t cmd_args; + struct l3fwd_pktio_s l3fwd_pktios[MAX_NB_PKTIO]; + odph_odpthread_t l3fwd_workers[MAX_NB_WORKER]; + struct thread_arg_s worker_args[MAX_NB_WORKER]; + odph_ethaddr_t eth_dest_mac[MAX_NB_PKTIO]; + + /* forward func, hash or lpm */ + int (*fwd_func)(odp_packet_t pkt, int sif); +} global; + +/** Global barrier to synchronize main and workers */ +static odp_barrier_t barrier; +static int exit_threads; /**< Break workers loop if set to 1 */ + +static int create_pktio(const char *name, odp_pool_t pool, + struct l3fwd_pktio_s *fwd_pktio) +{ + odp_pktio_param_t pktio_param; + odp_pktio_t pktio; + odp_pktio_capability_t capa; + int rc; + + odp_pktio_param_init(&pktio_param); + + pktio = odp_pktio_open(name, pool, &pktio_param); + if (pktio == ODP_PKTIO_INVALID) { + printf("Failed to open %s\n", name); + return -1; + } + fwd_pktio->pktio = pktio; + + rc = odp_pktio_capability(pktio, &capa); + if (rc) { + printf("Error: pktio %s: unable to read capabilities!\n", + name); + + return -1; + } + + fwd_pktio->nb_rxq = (int)capa.max_input_queues; + fwd_pktio->nb_txq = (int)capa.max_output_queues; + + if (fwd_pktio->nb_rxq > MAX_NB_QUEUE) + fwd_pktio->nb_rxq = MAX_NB_QUEUE; + + if (fwd_pktio->nb_txq > MAX_NB_QUEUE) + fwd_pktio->nb_txq = MAX_NB_QUEUE; + + return 0; +} + +static void setup_fwd_db(void) +{ + fwd_db_entry_t *entry; + int if_idx; + app_args_t *args; + + args = &global.cmd_args; + if (args->hash_mode) + init_fwd_hash_cache(); + else + fib_tbl_init(); + + for (entry = fwd_db->list; NULL != entry; entry = entry->next) { + if_idx = entry->oif_id; + if (!args->hash_mode) + fib_tbl_insert(entry->subnet.addr, if_idx, + entry->subnet.depth); + if (args->dest_mac_changed[if_idx]) + global.eth_dest_mac[if_idx] = entry->dst_mac; + else + entry->dst_mac = global.eth_dest_mac[if_idx]; + } +} + +static int l3fwd_pkt_hash(odp_packet_t pkt, int sif) +{ + fwd_db_entry_t *entry; + ipv4_tuple5_t key; + odph_ethhdr_t *eth; + odph_udphdr_t *udp; + odph_ipv4hdr_t *ip; + uint32_t len; + int dif; + + ip = odp_packet_l3_ptr(pkt, &len); + key.dst_ip = odp_be_to_cpu_32(ip->dst_addr); + key.src_ip = odp_be_to_cpu_32(ip->src_addr); + key.proto = ip->proto; + + if (odp_packet_has_udp(pkt) || + odp_packet_has_tcp(pkt)) { + /* UDP or TCP*/ + void *ptr = odp_packet_l4_ptr(pkt, NULL); + + udp = (odph_udphdr_t *)ptr; + key.src_port = odp_be_to_cpu_16(udp->src_port); + key.dst_port = odp_be_to_cpu_16(udp->dst_port); + } else { + key.src_port = 0; + key.dst_port = 0; + } + + entry = find_fwd_db_entry(&key); + ip->ttl--; + ip->chksum = odph_ipv4_csum_update(pkt); + eth = odp_packet_l2_ptr(pkt, NULL); + if (entry) { + eth->src = entry->src_mac; + eth->dst = entry->dst_mac; + dif = entry->oif_id; + } else { + /* no route, send by src port */ + eth->dst = eth->src; + dif = sif; + } + + return dif; +} + +static int l3fwd_pkt_lpm(odp_packet_t pkt, int sif) +{ + odph_ipv4hdr_t *ip; + odph_ethhdr_t *eth; + uint32_t len; + int dif; + int ret; + + ip = odp_packet_l3_ptr(pkt, &len); + ip->ttl--; + ip->chksum = odph_ipv4_csum_update(pkt); + eth = odp_packet_l2_ptr(pkt, NULL); + + /* network byte order maybe different from host */ + ret = fib_tbl_lookup(odp_be_to_cpu_32(ip->dst_addr), &dif); + if (ret) + dif = sif; + + eth->dst = global.eth_dest_mac[dif]; + eth->src = global.l3fwd_pktios[dif].mac_addr; + + return dif; +} + +/** + * Drop packets which input parsing marked as containing errors. + * + * Frees packets with error and modifies pkt_tbl[] to only contain packets with + * no detected errors. + * + * @param pkt_tbl Array of packets + * @param num Number of packets in pkt_tbl[] + * + * @return Number of packets dropped + */ +static inline int drop_err_pkts(odp_packet_t pkt_tbl[], unsigned num) +{ + odp_packet_t pkt; + unsigned dropped = 0; + unsigned i, j; + + for (i = 0, j = 0; i < num; ++i) { + pkt = pkt_tbl[i]; + + if (odp_unlikely(odp_packet_has_error(pkt) || + !odp_packet_has_ipv4(pkt))) { + odp_packet_free(pkt); + dropped++; + } else if (odp_unlikely(i != j++)) { + pkt_tbl[j - 1] = pkt; + } + } + + return dropped; +} + +static void l3fwd_one_queue(uint32_t sif, uint8_t rxq_id, void *thr_arg) +{ + struct l3fwd_pktio_s *port; + odp_packet_t *tbl; + odp_pktout_queue_t outq; + odp_packet_t pkt_tbl[MAX_PKT_BURST]; + struct thread_arg_s *arg; + uint8_t txq_id; + int pkts, drop, sent; + int dif, dst_port; + int i; + + arg = thr_arg; + port = &global.l3fwd_pktios[sif]; + pkts = odp_pktin_recv(port->ifin[rxq_id], pkt_tbl, MAX_PKT_BURST); + if (pkts <= 0) + return; + arg->packets += pkts; + drop = drop_err_pkts(pkt_tbl, pkts); + pkts -= drop; + arg->rx_drops += drop; + + dif = global.fwd_func(pkt_tbl[0], sif); + tbl = &pkt_tbl[0]; + while (pkts) { + dst_port = dif; + for (i = 1; i < pkts; i++) { + dif = global.fwd_func(tbl[i], sif); + if (dif != dst_port) + break; + } + txq_id = arg->pktio[dst_port].txq[rxq_id]; + outq = global.l3fwd_pktios[dst_port].ifout[txq_id]; + sent = odp_pktout_send(outq, tbl, i); + if (odp_unlikely(sent < i)) { + sent = sent < 0 ? 0 : sent; + odp_packet_free_multi(&tbl[sent], i - sent); + arg->tx_drops += i - sent; + } + + if (i < pkts) + tbl += i; + + pkts -= i; + } +} + +static int run_worker(void *arg) +{ + int if_idx, rxq_idx; + struct thread_arg_s *thr_arg = arg; + struct l3fwd_pktio_s *port; + + odp_barrier_wait(&barrier); + + while (!exit_threads) { + for (if_idx = 0; if_idx < thr_arg->nb_pktio; if_idx++) { + if (!thr_arg->pktio[if_idx].used || + thr_arg->thr_idx == INVALID_ID) + continue; + + port = &global.l3fwd_pktios[if_idx]; + for (rxq_idx = 0; rxq_idx < port->rxq_idx; rxq_idx++) + l3fwd_one_queue(if_idx, rxq_idx, arg); + } + } + + /* Make sure that latest stat writes are visible to other threads */ + odp_mb_full(); + + return 0; +} + +static int find_port_id_by_name(char *name, app_args_t *args) +{ + int i; + + if (!name) + return -1; + + for (i = 0; i < args->if_count; i++) { + if (!strcmp(name, args->if_names[i])) + return i; + } + + return -1; +} + +/* split string into tokens */ +static int split_string(char *str, int stringlen, + char **tokens, int maxtokens, char delim) +{ + int i, tok = 0; + int tokstart = 1; /* first token is right at start of string */ + + if (str == NULL || tokens == NULL) + goto einval_error; + + for (i = 0; i < stringlen; i++) { + if (str[i] == '\0' || tok >= maxtokens) + break; + if (tokstart) { + tokstart = 0; + tokens[tok++] = &str[i]; + } + if (str[i] == delim) { + str[i] = '\0'; + tokstart = 1; + } + } + return tok; + +einval_error: + errno = EINVAL; + return -1; +} + +static int parse_config(char *cfg_str, app_args_t *args) +{ + char s[256]; + const char *p, *p0 = cfg_str; + char *end; + enum fieldnames { + FLD_PORT = 0, + FLD_QUEUE, + FLD_LCORE, + FLD_LAST + }; + unsigned long int_fld[FLD_LAST]; + char *str_fld[FLD_LAST]; + int i; + unsigned size; + int nb_qconfs = 0; + struct l3fwd_qconf_s *qconf_array = &args->qconf_config[0]; + + p = strchr(p0, '('); + while (p != NULL) { + ++p; + p0 = strchr(p, ')'); + if (p0 == NULL) + return -1; + + size = p0 - p; + if (size >= sizeof(s)) + return -1; + + snprintf(s, sizeof(s), "%.*s", size, p); + i = split_string(s, sizeof(s), str_fld, FLD_LAST, ','); + if (i != FLD_LAST) + return -1; + for (i = 0; i < FLD_LAST; i++) { + errno = 0; + int_fld[i] = strtoul(str_fld[i], &end, 0); + if (errno != 0 || end == str_fld[i] || int_fld[i] > 255) + return -1; + } + if (nb_qconfs >= MAX_NB_QCONFS) { + printf("exceeded max number of queue params: %hu\n", + nb_qconfs); + return -1; + } + qconf_array[nb_qconfs].if_idx = (uint8_t)int_fld[FLD_PORT]; + qconf_array[nb_qconfs].rxq_idx = (uint8_t)int_fld[FLD_QUEUE]; + qconf_array[nb_qconfs].core_idx = (uint8_t)int_fld[FLD_LCORE]; + ++nb_qconfs; + + p = strchr(p0, '('); + } + args->qconf_count = nb_qconfs; + + return 0; +} + +static void print_usage(char *progname) +{ + printf("\n" + "ODP L3 forwarding application.\n" + "\n" + "Usage: %s OPTIONS\n" + " E.g. %s -i eth0,eth1 -r 1.1.1.0/24,eth0 -r 2.2.2.0/24,eth1\n" + " In the above example,\n" + " eth0 will send pkts to eth1 and vice versa\n" + "\n" + "Mandatory OPTIONS:\n" + " -i, --interface eth interfaces (comma-separated, no spaces)\n" + " -r, --route SubNet,Intf[,NextHopMAC]\n" + " NextHopMAC can be optional\n" + "\n" + "Optional OPTIONS:\n" + " -s, --style [lpm|hash], ip lookup method\n" + " optional, default as lpm\n" + " -d, --duration Seconds to run and print stats\n" + " optional, default as 0, run forever\n" + " -t, --thread Number of threads to do forwarding\n" + " optional, default as availbe worker cpu count\n" + " -q, --queue Configure rx queue(s) for port\n" + " optional, format: [(port, queue, thread),...]\n" + " for example: -q '(0, 0, 1),(1,0,2)'\n" + " -h, --help Display help and exit.\n\n" + "\n", NO_PATH(progname), NO_PATH(progname) + ); +} + +static void parse_cmdline_args(int argc, char *argv[], app_args_t *args) +{ + int opt; + int long_index; + char *token, *local; + size_t len, route_index = 0; + int i, mem_failure = 0; + + static struct option longopts[] = { + {"interface", required_argument, NULL, 'i'}, /* return 'i' */ + {"route", required_argument, NULL, 'r'}, /* return 'r' */ + {"style", optional_argument, NULL, 's'}, /* return 's' */ + {"duration", optional_argument, NULL, 'd'}, /* return 'd' */ + {"thread", optional_argument, NULL, 't'}, /* return 't' */ + {"queue", optional_argument, NULL, 'q'}, /* return 'q' */ + {"help", no_argument, NULL, 'h'}, /* return 'h' */ + {NULL, 0, NULL, 0} + }; + + while (1) { + opt = getopt_long(argc, argv, "+s:t:d:i:r:q:h", + longopts, &long_index); + + if (opt == -1) + break; /* No more options */ + + switch (opt) { + /* parse ip lookup method */ + case 's': + if (!strcmp(optarg, "hash")) + args->hash_mode = 1; + break; + /* parse number of worker threads to be run*/ + case 't': + i = odp_cpu_count(); + args->worker_count = atoi(optarg); + if (args->worker_count > i) { + printf("Too many threads," + "truncate to cpu count: %d\n", i); + args->worker_count = i; + } + + break; + + /* parse seconds to run */ + case 'd': + args->duration = atoi(optarg); + break; + + /* parse packet-io interface names */ + case 'i': + len = strlen(optarg); + if (len == 0) { + print_usage(argv[0]); + exit(EXIT_FAILURE); + } + len += 1; /* add room for '\0' */ + + local = malloc(len); + if (!local) { + print_usage(argv[0]); + exit(EXIT_FAILURE); + } + + /* count the number of tokens separated by ',' */ + strcpy(local, optarg); + for (token = strtok(local, ","), i = 0; + token != NULL; + token = strtok(NULL, ","), i++) + ; + + if (i == 0) { + print_usage(argv[0]); + free(local); + exit(EXIT_FAILURE); + } else if (i > MAX_NB_PKTIO) { + printf("too many ports specified, " + "truncated to %d", MAX_NB_PKTIO); + } + args->if_count = i; + + /* store the if names (reset names string) */ + strcpy(local, optarg); + for (token = strtok(local, ","), i = 0; + token != NULL; token = strtok(NULL, ","), i++) { + args->if_names[i] = token; + } + break; + + /*Configure Route in forwarding database*/ + case 'r': + if (route_index >= MAX_NB_ROUTE) { + printf("No more routes can be added\n"); + break; + } + local = calloc(1, strlen(optarg) + 1); + if (!local) { + mem_failure = 1; + break; + } + memcpy(local, optarg, strlen(optarg)); + local[strlen(optarg)] = '\0'; + args->route_str[route_index++] = local; + break; + + case 'h': + print_usage(argv[0]); + exit(EXIT_SUCCESS); + break; + + case 'q': + parse_config(optarg, args); + break; + + default: + break; + } + } + + /* checking arguments */ + if (args->if_count == 0) { + printf("\nNo option -i specified.\n"); + goto out; + } + + if (args->route_str[0] == NULL) { + printf("\nNo option -r specified.\n"); + goto out; + } + + if (mem_failure == 1) { + printf("\nAllocate memory failure.\n"); + goto out; + } + optind = 1; /* reset 'extern optind' from the getopt lib */ + return; + +out: + print_usage(argv[0]); + exit(EXIT_FAILURE); +} + +static void print_info(char *progname, app_args_t *args) +{ + int i; + + printf("\n" + "ODP system info\n" + "---------------\n" + "ODP API version: %s\n" + "ODP impl name: %s\n" + "CPU model: %s\n" + "CPU freq (hz): %" PRIu64 "\n" + "Cache line size: %i\n" + "CPU count: %i\n" + "\n", + odp_version_api_str(), odp_version_impl_name(), + odp_cpu_model_str(), odp_cpu_hz_max(), + odp_sys_cache_line_size(), odp_cpu_count()); + + printf("Running ODP appl: \"%s\"\n" + "-----------------\n" + "IP Lookup: %s\n" + "IF Count: %i\n" + "Using IFs: ", + progname, + args->hash_mode ? "hash" : "lpm", + args->if_count); + + for (i = 0; i < args->if_count; ++i) + printf(" %s", args->if_names[i]); + + printf("\n\n"); + fflush(NULL); +} + +/** + * Setup rx and tx queues, distribute them among threads. + * Try to have one tx queue for each rx queue, if not available, + * shared tx queue is used. + * + * If no q argument, the queues are distribute among threads as default. + * The thread take one rx queue of a port one time as round-robin order. + */ +static void setup_worker_qconf(app_args_t *args) +{ + int nb_worker, if_count; + int i, j, rxq_idx, txq_idx; + struct thread_arg_s *arg; + struct l3fwd_pktio_s *port; + uint8_t queue_mask[MAX_NB_PKTIO][MAX_NB_QUEUE]; + + nb_worker = args->worker_count; + if_count = args->if_count; + + /* distribute queues among threads */ + if (!args->qconf_count) { + if (nb_worker > if_count) { + for (i = 0; i < nb_worker; i++) { + arg = &global.worker_args[i]; + arg->thr_idx = i; + j = i % if_count; + port = &global.l3fwd_pktios[j]; + rxq_idx = port->rxq_idx % port->nb_rxq; + arg->pktio[j].rxq[port->rxq_idx] = rxq_idx; + port->rxq_idx++; + txq_idx = port->txq_idx % port->nb_txq; + arg->pktio[j].txq[port->txq_idx] = txq_idx; + port->txq_idx++; + arg->pktio[j].used = 1; + } + } else { + for (i = 0; i < if_count; i++) { + j = i % nb_worker; + arg = &global.worker_args[j]; + arg->thr_idx = j; + port = &global.l3fwd_pktios[i]; + rxq_idx = port->rxq_idx % port->nb_rxq; + arg->pktio[i].rxq[port->rxq_idx] = rxq_idx; + port->rxq_idx++; + txq_idx = port->txq_idx % port->nb_txq; + arg->pktio[i].txq[port->txq_idx] = txq_idx; + port->txq_idx++; + arg->pktio[i].used = 1; + } + } + } + + /* specified q argument, distribute queues among threads as it says */ + memset(queue_mask, 0, sizeof(queue_mask)); + for (i = 0; i < args->qconf_count; i++) { + struct l3fwd_qconf_s *q; + + q = &args->qconf_config[i]; + if (q->core_idx >= nb_worker || q->if_idx >= if_count) + LOG_ABORT("Error queue (%d, %d, %d), max port: " + "%d, max core: %d\n", q->if_idx, q->rxq_idx, + q->core_idx, args->if_count - 1, + args->worker_count - 1); + + /* check if one queue is configured twice or more */ + if (queue_mask[q->if_idx][q->rxq_idx]) + LOG_ABORT("Error queue (%d, %d, %d), reconfig queue\n", + q->if_idx, q->rxq_idx, q->core_idx); + queue_mask[q->if_idx][q->rxq_idx] = 1; + + port = &global.l3fwd_pktios[q->if_idx]; + if (port->rxq_idx < q->rxq_idx) + LOG_ABORT("Error queue (%d, %d, %d), queue should be" + " in sequence and start from 0, queue %d\n", + q->if_idx, q->rxq_idx, q->core_idx, + q->rxq_idx); + + if (q->rxq_idx > port->nb_rxq) { + LOG_ABORT("Error queue (%d, %d, %d), max queue %d\n", + q->if_idx, q->rxq_idx, q->core_idx, + port->nb_rxq - 1); + } + port->rxq_idx = q->rxq_idx + 1; + port->txq_idx = q->rxq_idx + 1; + + /* put the queue into worker_args */ + arg = &global.worker_args[q->core_idx]; + arg->pktio[q->if_idx].rxq[q->rxq_idx] = q->rxq_idx; + arg->pktio[q->if_idx].txq[q->rxq_idx] = q->rxq_idx; + arg->pktio[q->if_idx].used = 1; + arg->thr_idx = q->core_idx; + } + + /* config and initialize rx and tx queues. */ + for (i = 0; i < if_count; i++) { + odp_pktin_queue_param_t in_queue_param; + odp_pktout_queue_param_t out_queue_param; + odp_pktin_queue_t *inq; + odp_pktout_queue_t *outq; + const char *name; + int nb_rxq, nb_txq; + + port = &global.l3fwd_pktios[i]; + name = args->if_names[i]; + odp_pktin_queue_param_init(&in_queue_param); + odp_pktout_queue_param_init(&out_queue_param); + + in_queue_param.op_mode = ODP_PKTIO_OP_MT_UNSAFE; + out_queue_param.op_mode = ODP_PKTIO_OP_MT_UNSAFE; + + ODP_ASSERT(port->rxq_idx == port->txq_idx); + + in_queue_param.num_queues = port->rxq_idx; + if (port->rxq_idx > port->nb_rxq) { + in_queue_param.num_queues = port->nb_rxq; + in_queue_param.op_mode = ODP_PKTIO_OP_MT; + } + + /* enable flow hashing for multi-input queues */ + in_queue_param.hash_enable = 1; + in_queue_param.hash_proto.proto.ipv4_tcp = 1; + in_queue_param.hash_proto.proto.ipv4_udp = 1; + + if (odp_pktin_queue_config(port->pktio, &in_queue_param)) + LOG_ABORT("Fail to config input queue for %s\n", name); + + out_queue_param.num_queues = port->txq_idx; + if (port->txq_idx > port->nb_txq) { + out_queue_param.num_queues = port->nb_txq; + out_queue_param.op_mode = ODP_PKTIO_OP_MT; + } + if (odp_pktout_queue_config(port->pktio, &out_queue_param)) + LOG_ABORT("Fail to config output queue for %s\n", name); + + inq = port->ifin; + nb_rxq = in_queue_param.num_queues; + if (odp_pktin_queue(port->pktio, inq, nb_rxq) != nb_rxq) + LOG_ABORT("Fail to set pktin queue for %s\n", name); + + outq = port->ifout; + nb_txq = out_queue_param.num_queues; + if (odp_pktout_queue(port->pktio, outq, nb_txq) != nb_txq) + LOG_ABORT("Fail to set pktout queue for %s\n", name); + } +} + +static void print_qconf_table(app_args_t *args) +{ + int i, j, k, qid; + char buf[32]; + struct thread_arg_s *thr_arg; + + printf("Rx queue table\n" + "-----------------\n" + "%-32s%-16s%-16s\n", + "port/id", "queue", "thread"); + + for (i = 0; i < args->worker_count; i++) { + thr_arg = &global.worker_args[i]; + for (j = 0; j < args->if_count; j++) { + if (!thr_arg->pktio[j].used) + continue; + + snprintf(buf, 32, "%s/%d", args->if_names[j], j); + for (k = 0; k < MAX_NB_QUEUE; k++) { + qid = thr_arg->pktio[j].rxq[k]; + if (qid != INVALID_ID) + printf("%-32s%-16d%-16d\n", buf, qid, + thr_arg->thr_idx); + } + } + } + printf("\n"); + fflush(NULL); +} + +/** + * Print statistics + * + * @param num_workers Number of worker threads + * @param duration Number of seconds to loop in + * @param timeout Number of seconds for stats calculation + * + */ +static int print_speed_stats(int num_workers, int duration, int timeout) +{ + uint64_t pkts = 0; + uint64_t pkts_prev = 0; + uint64_t pps; + uint64_t rx_drops, tx_drops; + uint64_t maximum_pps = 0; + int i; + int elapsed = 0; + int stats_enabled = 1; + int loop_forever = (duration == 0); + + if (timeout <= 0) { + stats_enabled = 0; + timeout = 1; + } + /* Wait for all threads to be ready*/ + odp_barrier_wait(&barrier); + + do { + pkts = 0; + rx_drops = 0; + tx_drops = 0; + sleep(timeout); + + for (i = 0; i < num_workers; i++) { + pkts += global.worker_args[i].packets; + rx_drops += global.worker_args[i].rx_drops; + tx_drops += global.worker_args[i].tx_drops; + } + if (stats_enabled) { + pps = (pkts - pkts_prev) / timeout; + if (pps > maximum_pps) + maximum_pps = pps; + printf("%" PRIu64 " pps, %" PRIu64 " max pps, ", pps, + maximum_pps); + + printf(" %" PRIu64 " rx drops, %" PRIu64 " tx drops\n", + rx_drops, tx_drops); + + pkts_prev = pkts; + } + elapsed += timeout; + } while (loop_forever || (elapsed < duration)); + + if (stats_enabled) + printf("TEST RESULT: %" PRIu64 " maximum packets per second.\n", + maximum_pps); + + return pkts > 100 ? 0 : -1; +} + +int main(int argc, char **argv) +{ + odph_odpthread_t thread_tbl[MAX_NB_WORKER]; + odp_pool_t pool; + odp_pool_param_t params; + odp_instance_t instance; + odph_odpthread_params_t thr_params; + odp_cpumask_t cpumask; + int cpu, i, j, nb_worker; + uint8_t mac[ODPH_ETHADDR_LEN]; + uint8_t *dst_mac; + app_args_t *args; + struct thread_arg_s *thr_arg; + char *oif; + + if (odp_init_global(&instance, NULL, NULL)) { + printf("Error: ODP global init failed.\n"); + exit(1); + } + + if (odp_init_local(instance, ODP_THREAD_CONTROL)) { + printf("Error: ODP local init failed.\n"); + exit(1); + } + + /* Clear global argument and initialize the dest mac as 2:0:0:0:0:x */ + memset(&global, 0, sizeof(global)); + mac[0] = 2; + for (i = 0; i < MAX_NB_PKTIO; i++) { + mac[ODPH_ETHADDR_LEN - 1] = (uint8_t)i; + memcpy(global.eth_dest_mac[i].addr, mac, ODPH_ETHADDR_LEN); + } + + /* Initialize the thread arguments */ + for (i = 0; i < MAX_NB_WORKER; i++) { + thr_arg = &global.worker_args[i]; + for (j = 0; j < MAX_NB_PKTIO; j++) { + thr_arg->thr_idx = INVALID_ID; + memset(thr_arg->pktio[j].rxq, INVALID_ID, + sizeof(thr_arg->pktio[j].rxq)); + } + } + + /* Parse cmdline arguments */ + args = &global.cmd_args; + parse_cmdline_args(argc, argv, args); + + /* Init l3fwd table */ + init_fwd_db(); + + /* Add route into table */ + for (i = 0; i < MAX_NB_ROUTE; i++) { + if (args->route_str[i]) { + create_fwd_db_entry(args->route_str[i], &oif, &dst_mac); + if (oif == NULL) { + printf("Error: fail to create route entry.\n"); + exit(1); + } + + j = find_port_id_by_name(oif, args); + if (j == -1) { + printf("Error: port %s not used.\n", oif); + exit(1); + } + + if (dst_mac) + args->dest_mac_changed[j] = 1; + } + } + + print_info(NO_PATH(argv[0]), args); + + /* Create packet pool */ + odp_pool_param_init(¶ms); + params.pkt.seg_len = POOL_SEG_LEN; + params.pkt.len = POOL_SEG_LEN; + params.pkt.num = POOL_NUM_PKT; + params.type = ODP_POOL_PACKET; + + pool = odp_pool_create("packet pool", ¶ms); + + if (pool == ODP_POOL_INVALID) { + printf("Error: packet pool create failed.\n"); + exit(1); + } + + /* Resolve fwd db*/ + for (i = 0; i < args->if_count; i++) { + struct l3fwd_pktio_s *port; + char *if_name; + + if_name = args->if_names[i]; + port = &global.l3fwd_pktios[i]; + if (create_pktio(if_name, pool, port)) { + printf("Error: create pktio %s\n", if_name); + exit(1); + } + odp_pktio_mac_addr(port->pktio, mac, ODPH_ETHADDR_LEN); + resolve_fwd_db(if_name, i, mac); + memcpy(port->mac_addr.addr, mac, ODPH_ETHADDR_LEN); + } + + setup_fwd_db(); + dump_fwd_db(); + + /* Dicide available workers */ + nb_worker = MAX_NB_WORKER; + if (args->worker_count) + nb_worker = args->worker_count; + nb_worker = odp_cpumask_default_worker(&cpumask, nb_worker); + args->worker_count = nb_worker; + + /* Setup rx and tx queues for each port */ + setup_worker_qconf(args); + print_qconf_table(args); + + /* Decide ip lookup method */ + if (args->hash_mode) + global.fwd_func = l3fwd_pkt_hash; + else + global.fwd_func = l3fwd_pkt_lpm; + + /* Start all the available ports */ + for (i = 0; i < args->if_count; i++) { + struct l3fwd_pktio_s *port; + char *if_name; + char buf[32]; + + if_name = args->if_names[i]; + port = &global.l3fwd_pktios[i]; + /* start pktio */ + if (odp_pktio_start(port->pktio)) { + printf("unable to start pktio: %s\n", if_name); + exit(1); + } + + sprintf(buf, "%02x:%02x:%02x:%02x:%02x:%02x", + port->mac_addr.addr[0], + port->mac_addr.addr[1], + port->mac_addr.addr[2], + port->mac_addr.addr[3], + port->mac_addr.addr[4], + port->mac_addr.addr[5]); + printf("start pktio: %s, mac %s\n", if_name, buf); + } + + odp_barrier_init(&barrier, nb_worker + 1); + + memset(&thr_params, 0, sizeof(thr_params)); + thr_params.start = run_worker; + thr_params.thr_type = ODP_THREAD_WORKER; + thr_params.instance = instance; + + memset(thread_tbl, 0, sizeof(thread_tbl)); + cpu = odp_cpumask_first(&cpumask); + for (i = 0; i < nb_worker; i++) { + struct thread_arg_s *arg; + odp_cpumask_t thr_mask; + + arg = &global.worker_args[i]; + arg->nb_pktio = args->if_count; + odp_cpumask_zero(&thr_mask); + odp_cpumask_set(&thr_mask, cpu); + thr_params.arg = arg; + odph_odpthreads_create(&thread_tbl[i], &thr_mask, + &thr_params); + cpu = odp_cpumask_next(&cpumask, cpu); + } + + print_speed_stats(nb_worker, args->duration, PRINT_INTERVAL); + exit_threads = 1; + + /* wait for other threads to join */ + for (i = 0; i < nb_worker; i++) + odph_odpthreads_join(&thread_tbl[i]); + + /* release resource on exit */ + destroy_fwd_db(); + + /* if_names share a single buffer, so only one free */ + free(args->if_names[0]); + + for (i = 0; i < MAX_NB_ROUTE; i++) + free(args->route_str[i]); + + if (odp_pool_destroy(pool)) { + printf("Error: pool destroy\n"); + exit(EXIT_FAILURE); + } + + if (odp_term_local()) { + printf("Error: term local\n"); + exit(EXIT_FAILURE); + } + + if (odp_term_global(instance)) { + printf("Error: term global\n"); + exit(EXIT_FAILURE); + } + + return 0; +} diff --git a/example/l3fwd/odp_l3fwd_db.c b/example/l3fwd/odp_l3fwd_db.c new file mode 100644 index 0000000..b3a237f --- /dev/null +++ b/example/l3fwd/odp_l3fwd_db.c @@ -0,0 +1,432 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#ifndef _GNU_SOURCE +#define _GNU_SOURCE +#endif + +#include <stdlib.h> +#include <string.h> + +#include <example_debug.h> +#include <odp_api.h> +#include <odp_l3fwd_db.h> + +/** Jenkins hash support. + * + * Copyright (C) 2006 Bob Jenkins (bob_jenkins@burtleburtle.net) + * + * http://burtleburtle.net/bob/hash/ + * + * These are the credits from Bob's sources: + * + * lookup3.c, by Bob Jenkins, May 2006, Public Domain. + * + * These are functions for producing 32-bit hashes for hash table lookup. + * hashword(), hashlittle(), hashlittle2(), hashbig(), mix(), and final() + * are externally useful functions. Routines to test the hash are included + * if SELF_TEST is defined. You can use this free for any purpose. It's in + * the public domain. It has no warranty. + * + * $FreeBSD$ + */ +#define JHASH_GOLDEN_RATIO 0x9e3779b9 +#define rot(x, k) (((x) << (k)) | ((x) >> (32 - (k)))) +#define FWD_BJ3_MIX(a, b, c) \ +{ \ + a -= c; a ^= rot(c, 4); c += b; \ + b -= a; b ^= rot(a, 6); a += c; \ + c -= b; c ^= rot(b, 8); b += a; \ + a -= c; a ^= rot(c, 16); c += b; \ + b -= a; b ^= rot(a, 19); a += c; \ + c -= b; c ^= rot(b, 4); b += a; \ +} + +/** + * Compute hash value from a flow + */ +static inline +uint64_t l3fwd_calc_hash(ipv4_tuple5_t *key) +{ + uint64_t l4_ports = 0; + uint32_t dst_ip, src_ip; + + src_ip = key->src_ip; + dst_ip = key->dst_ip + JHASH_GOLDEN_RATIO; + FWD_BJ3_MIX(src_ip, dst_ip, l4_ports); + + return l4_ports; +} + +/** + * Parse text string representing an IPv4 address or subnet + * + * String is of the format "XXX.XXX.XXX.XXX(/W)" where + * "XXX" is decimal value and "/W" is optional subnet length + * + * @param ipaddress Pointer to IP address/subnet string to convert + * @param addr Pointer to return IPv4 address, host endianness + * @param depth Pointer to subnet bit width + * @return 0 if successful else -1 + */ +static inline +int parse_ipv4_string(char *ipaddress, uint32_t *addr, uint32_t *depth) +{ + int b[4]; + int qualifier = 32; + int converted; + uint32_t addr_le; + + if (strchr(ipaddress, '/')) { + converted = sscanf(ipaddress, "%d.%d.%d.%d/%d", + &b[3], &b[2], &b[1], &b[0], + &qualifier); + if (5 != converted) + return -1; + } else { + converted = sscanf(ipaddress, "%d.%d.%d.%d", + &b[3], &b[2], &b[1], &b[0]); + if (4 != converted) + return -1; + } + + if ((b[0] > 255) || (b[1] > 255) || (b[2] > 255) || (b[3] > 255)) + return -1; + if (!qualifier || (qualifier > 32)) + return -1; + + addr_le = b[0] | b[1] << 8 | b[2] << 16 | b[3] << 24; + *addr = odp_le_to_cpu_32(addr_le); + *depth = qualifier; + + return 0; +} + +/** + * Generate text string representing IPv4 range/subnet, output + * in "XXX.XXX.XXX.XXX/W" format + * + * @param b Pointer to buffer to store string + * @param range Pointer to IPv4 address range + * + * @return Pointer to supplied buffer + */ +static inline +char *ipv4_subnet_str(char *b, ip_addr_range_t *range) +{ + sprintf(b, "%d.%d.%d.%d/%d", + 0xFF & ((range->addr) >> 24), + 0xFF & ((range->addr) >> 16), + 0xFF & ((range->addr) >> 8), + 0xFF & ((range->addr) >> 0), + range->depth); + return b; +} + +/** + * Generate text string representing MAC address + * + * @param b Pointer to buffer to store string + * @param mac Pointer to MAC address + * + * @return Pointer to supplied buffer + */ +static inline +char *mac_addr_str(char *b, odph_ethaddr_t *mac) +{ + uint8_t *byte; + + byte = mac->addr; + sprintf(b, "%02X:%02X:%02X:%02X:%02X:%02X", + byte[0], byte[1], byte[2], byte[3], byte[4], byte[5]); + return b; +} + +/** + * Flow cache table entry + */ +typedef struct flow_entry_s { + ipv4_tuple5_t key; /**< match key */ + struct flow_entry_s *next; /**< next entry on the list */ + fwd_db_entry_t *fwd_entry; /**< entry info in db */ +} flow_entry_t; + +/** + * Flow cache table bucket + */ +typedef struct flow_bucket_s { + odp_spinlock_t lock; /**< Bucket lock*/ + flow_entry_t *next; /**< Pointer to first flow entry in bucket*/ +} flow_bucket_t; + +/** + * Flow hash table, fast lookup cache + */ +typedef struct flow_table_s { + flow_bucket_t *bucket; + uint32_t count; +} flow_table_t; + +static flow_table_t fwd_lookup_cache; + +void init_fwd_hash_cache(void) +{ + odp_shm_t hash_shm; + flow_bucket_t *bucket; + uint32_t bucket_count; + uint32_t i; + + bucket_count = FWD_DEF_BUCKET_COUNT; + + /*Reserve memory for Routing hash table*/ + hash_shm = odp_shm_reserve("route_table", + sizeof(flow_bucket_t) * bucket_count, + ODP_CACHE_LINE_SIZE, 0); + + bucket = odp_shm_addr(hash_shm); + if (!bucket) { + EXAMPLE_ERR("Error: shared mem alloc failed.\n"); + exit(-1); + } + + fwd_lookup_cache.bucket = bucket; + fwd_lookup_cache.count = bucket_count; + + /*Initialize Locks*/ + for (i = 0; i < bucket_count; i++) { + bucket = &fwd_lookup_cache.bucket[i]; + odp_spinlock_init(&bucket->lock); + bucket->next = NULL; + } +} + +static inline +int match_key_flow(ipv4_tuple5_t *key, flow_entry_t *flow) +{ + if (key->src_ip == flow->key.src_ip && + key->dst_ip == flow->key.dst_ip && + key->src_port == flow->key.src_port && + key->dst_port == flow->key.dst_port && + key->proto == flow->key.proto) + return 1; + + return 0; +} + +static inline +flow_entry_t *lookup_fwd_cache(ipv4_tuple5_t *key, flow_bucket_t *bucket) +{ + flow_entry_t *rst; + + for (rst = bucket->next; rst != NULL; rst = rst->next) { + if (match_key_flow(key, rst)) + break; + } + + return rst; +} + +static inline +flow_entry_t *insert_fwd_cache(ipv4_tuple5_t *key, + flow_bucket_t *bucket, + fwd_db_entry_t *entry) +{ + flow_entry_t *flow; + + if (!entry) + return NULL; + + flow = malloc(sizeof(flow_entry_t)); + flow->key = *key; + flow->fwd_entry = entry; + + odp_spinlock_lock(&bucket->lock); + if (!bucket->next) { + bucket->next = flow; + } else { + flow->next = bucket->next; + bucket->next = flow; + } + odp_spinlock_unlock(&bucket->lock); + + return flow; +} + +/** Global pointer to fwd db */ +fwd_db_t *fwd_db; + +void init_fwd_db(void) +{ + odp_shm_t shm; + + shm = odp_shm_reserve("shm_fwd_db", + sizeof(fwd_db_t), + ODP_CACHE_LINE_SIZE, + 0); + + fwd_db = odp_shm_addr(shm); + + if (fwd_db == NULL) { + EXAMPLE_ERR("Error: shared mem alloc failed.\n"); + exit(EXIT_FAILURE); + } + memset(fwd_db, 0, sizeof(*fwd_db)); +} + +int create_fwd_db_entry(char *input, char **oif, uint8_t **dst_mac) +{ + int pos = 0; + char *local; + char *str; + char *save; + char *token; + fwd_db_entry_t *entry = &fwd_db->array[fwd_db->index]; + + *oif = NULL; + *dst_mac = NULL; + + /* Verify we haven't run out of space */ + if (MAX_DB <= fwd_db->index) + return -1; + + /* Make a local copy */ + local = malloc(strlen(input) + 1); + if (NULL == local) + return -1; + strcpy(local, input); + + /* Setup for using "strtok_r" to search input string */ + str = local; + save = NULL; + + /* Parse tokens separated by ',' */ + while (NULL != (token = strtok_r(str, ",", &save))) { + str = NULL; /* reset str for subsequent strtok_r calls */ + + /* Parse token based on its position */ + switch (pos) { + case 0: + parse_ipv4_string(token, + &entry->subnet.addr, + &entry->subnet.depth); + break; + case 1: + strncpy(entry->oif, token, OIF_LEN - 1); + entry->oif[OIF_LEN - 1] = 0; + *oif = entry->oif; + break; + case 2: + odph_eth_addr_parse(&entry->dst_mac, token); + *dst_mac = entry->dst_mac.addr; + break; + + default: + printf("ERROR: extra token \"%s\" at position %d\n", + token, pos); + break; + } + + /* Advance to next position */ + pos++; + } + + /* Add route to the list */ + fwd_db->index++; + entry->next = fwd_db->list; + fwd_db->list = entry; + + free(local); + return 0; +} + +void resolve_fwd_db(char *intf, int portid, uint8_t *mac) +{ + fwd_db_entry_t *entry; + + /* Walk the list and attempt to set output and MAC */ + for (entry = fwd_db->list; NULL != entry; entry = entry->next) { + if (strcmp(intf, entry->oif)) + continue; + + entry->oif_id = portid; + memcpy(entry->src_mac.addr, mac, ODPH_ETHADDR_LEN); + } +} + +void dump_fwd_db_entry(fwd_db_entry_t *entry) +{ + char subnet_str[MAX_STRING]; + char mac_str[MAX_STRING]; + + mac_addr_str(mac_str, &entry->dst_mac); + printf("%-32s%-32s%-16s\n", + ipv4_subnet_str(subnet_str, &entry->subnet), + entry->oif, mac_str); +} + +void dump_fwd_db(void) +{ + fwd_db_entry_t *entry; + + printf("Routing table\n" + "-----------------\n" + "%-32s%-32s%-16s\n", + "subnet", "next_hop", "dest_mac"); + + for (entry = fwd_db->list; NULL != entry; entry = entry->next) + dump_fwd_db_entry(entry); + + printf("\n"); +} + +void destroy_fwd_db(void) +{ + flow_bucket_t *bucket; + flow_entry_t *flow, *tmp; + uint32_t i; + + for (i = 0; i < fwd_lookup_cache.count; i++) { + bucket = &fwd_lookup_cache.bucket[i]; + flow = bucket->next; + odp_spinlock_lock(&bucket->lock); + while (flow) { + tmp = flow->next; + free(flow); + flow = tmp; + } + odp_spinlock_unlock(&bucket->lock); + } +} + +fwd_db_entry_t *find_fwd_db_entry(ipv4_tuple5_t *key) +{ + fwd_db_entry_t *entry; + flow_entry_t *flow; + flow_bucket_t *bucket; + uint64_t hash; + + /* first find in cache */ + hash = l3fwd_calc_hash(key); + hash &= fwd_lookup_cache.count - 1; + bucket = &fwd_lookup_cache.bucket[hash]; + flow = lookup_fwd_cache(key, bucket); + if (flow) + return flow->fwd_entry; + + for (entry = fwd_db->list; NULL != entry; entry = entry->next) { + uint32_t mask; + + mask = ((1u << entry->subnet.depth) - 1) << + (32 - entry->subnet.depth); + + if (entry->subnet.addr == (key->dst_ip & mask)) + break; + } + + insert_fwd_cache(key, bucket, entry); + + return entry; +} diff --git a/example/l3fwd/odp_l3fwd_db.h b/example/l3fwd/odp_l3fwd_db.h new file mode 100644 index 0000000..d9f6d61 --- /dev/null +++ b/example/l3fwd/odp_l3fwd_db.h @@ -0,0 +1,136 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#ifndef _ODP_L3FWD_DB_H_ +#define _ODP_L3FWD_DB_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +#include <odp_api.h> +#include <odp/helper/eth.h> + +#define OIF_LEN 32 +#define MAX_DB 32 +#define MAX_STRING 32 + +/** + * Default number of flows + */ +#define FWD_DEF_FLOW_COUNT 100000 + +/** + * Default Hash bucket number + */ +#define FWD_DEF_BUCKET_COUNT (FWD_DEF_FLOW_COUNT / 8) + +/** + * IP address range (subnet) + */ +typedef struct ip_addr_range_s { + uint32_t addr; /**< IP address, host endianness */ + uint32_t depth; /**< subnet bit width */ +} ip_addr_range_t; + +/** + * TCP/UDP flow + */ +typedef struct ipv4_tuple5_s { + uint32_t src_ip; + uint32_t dst_ip; + uint16_t src_port; + uint16_t dst_port; + uint8_t proto; +} ipv4_tuple5_t; + +/** + * Forwarding data base entry + */ +typedef struct fwd_db_entry_s { + struct fwd_db_entry_s *next; /**< Next entry on list */ + char oif[OIF_LEN]; /**< Output interface name */ + int oif_id; /**< Output interface idx */ + odph_ethaddr_t src_mac; /**< Output source MAC */ + odph_ethaddr_t dst_mac; /**< Output destination MAC */ + ip_addr_range_t subnet; /**< Subnet for this router */ +} fwd_db_entry_t; + +/** + * Forwarding data base + */ +typedef struct fwd_db_s { + uint32_t index; /**< Next available entry */ + fwd_db_entry_t *list; /**< List of active routes */ + fwd_db_entry_t array[MAX_DB]; /**< Entry storage */ +} fwd_db_t; + +/** Global pointer to fwd db */ +extern fwd_db_t *fwd_db; + +/** + * Initialize FWD DB + */ +void init_fwd_db(void); + +/** + * Initialize forward lookup cache based on hash + */ +void init_fwd_hash_cache(void); + +/** + * Create a forwarding database entry + * + * String is of the format "SubNet,Intf,NextHopMAC" + * + * @param input Pointer to string describing route + * @param oif Pointer to out interface name, as a return value + * @param dst_mac Pointer to dest mac for output packet, as a return value + * + * @return 0 if successful else -1 + */ +int create_fwd_db_entry(char *input, char **oif, uint8_t **dst_mac); + +/** + * Scan FWD DB entries and resolve output queue and source MAC address + * + * @param intf Interface name string + * @param portid Output queue for packet transmit + * @param mac MAC address of this interface + */ +void resolve_fwd_db(char *intf, int portid, uint8_t *mac); + +/** + * Display one forwarding database entry + * + * @param entry Pointer to entry to display + */ +void dump_fwd_db_entry(fwd_db_entry_t *entry); + +/** + * Display the forwarding database + */ +void dump_fwd_db(void); + +/** + * Destroy the forwarding database + */ +void destroy_fwd_db(void); + +/** + * Find a matching forwarding database entry + * + * @param key ipv4 tuple + * + * @return pointer to forwarding DB entry else NULL + */ +fwd_db_entry_t *find_fwd_db_entry(ipv4_tuple5_t *key); + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/example/l3fwd/odp_l3fwd_lpm.c b/example/l3fwd/odp_l3fwd_lpm.c new file mode 100644 index 0000000..1b3bfcf --- /dev/null +++ b/example/l3fwd/odp_l3fwd_lpm.c @@ -0,0 +1,224 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ +#ifndef _GNU_SOURCE +#define _GNU_SOURCE +#endif + +#include <stdio.h> +#include <stdlib.h> + +#include <example_debug.h> +#include <odp_api.h> + +#include <odp_l3fwd_lpm.h> + +/** + * This is a simple implementation of lpm based on patricia tree. + * + * Tradeoff exists between memory consumption and lookup time. + * Currently it prefers 5 levels: {16, 4, 4, 4, 4}, could be 3 + * levels: {16, 8, 8} by defining FIB_NEXT_STRIDE as 8. Other + * levels are also possible. + * + * the ip here is host endian, when doing init or lookup, the + * caller should do endianness conversion if needed. + */ + +#define FIB_IP_WIDTH 32 +#define FIB_FIRST_STRIDE 16 +#define FIB_NEXT_STRIDE 4 +#define FIB_NEXT_SIZE (1 << FIB_NEXT_STRIDE) +#define FIB_SUB_COUNT 16384 +#define DEPTH_TO_MASK(depth) ((1 << (depth)) - 1) + +typedef struct fib_node_s { + union { + uint32_t next_hop; + struct fib_node_s *next; /* next level table */ + }; + uint8_t valid :1; /* 1, this node has a valid next hop */ + uint8_t end :1; /* 0, next points to the extended table */ + uint8_t depth :6; /* bit length of subnet mask */ +} fib_node_t; + +typedef struct fib_sub_tbl_s { + fib_node_t *fib_nodes; + uint32_t fib_count; + uint32_t fib_idx; +} fib_sub_tbl_t; + +static fib_node_t fib_rt_tbl[1 << FIB_FIRST_STRIDE]; +static fib_sub_tbl_t fib_lpm_cache; + +static inline fib_node_t *fib_alloc_sub(void) +{ + fib_node_t *sub_tbl = NULL; + uint32_t i, nb_entry; + + /* extend to next level */ + if (fib_lpm_cache.fib_idx < fib_lpm_cache.fib_count) { + nb_entry = (fib_lpm_cache.fib_idx + 1) * FIB_NEXT_SIZE; + sub_tbl = &fib_lpm_cache.fib_nodes[nb_entry]; + fib_lpm_cache.fib_idx++; + for (i = 0; i < nb_entry; i++) { + sub_tbl[i].valid = 0; + sub_tbl[i].end = 1; + } + } + + return sub_tbl; +} + +static void fib_update_node(fib_node_t *fe, int port, int depth) +{ + fib_node_t *p; + int i; + + if (fe->end) { + if (!fe->valid) { + fe->depth = depth; + fe->next_hop = port; + fe->valid = 1; + } else if (fe->depth <= depth) { + fe->next_hop = port; + fe->depth = depth; + } + + return; + } + + for (i = 0; i < FIB_NEXT_SIZE; i++) { + p = &fe->next[i]; + if (p->end) + fib_update_node(p, port, depth); + } +} + +static void fib_insert_node(fib_node_t *fe, uint32_t ip, uint32_t next_hop, + int ip_width, int eat_bits, int depth) +{ + int i; + uint32_t idx, port; + fib_node_t *p; + + if (fe->end) { + port = fe->next_hop; + p = fib_alloc_sub(); + if (!p) + return; + + fe->next = p; + fe->end = 0; + if (fe->valid) { + for (i = 0; i < FIB_NEXT_SIZE; i++) { + p = &fe->next[i]; + p->next_hop = port; + p->depth = fe->depth; + } + } + } + if (depth - eat_bits <= FIB_NEXT_STRIDE) { + ip_width -= depth - eat_bits; + idx = ip >> ip_width; + ip &= DEPTH_TO_MASK(ip_width); + p = &fe->next[idx]; + fib_update_node(p, next_hop, depth); + } else { + ip_width -= FIB_NEXT_STRIDE; + idx = ip >> ip_width; + p = &fe->next[idx]; + ip &= DEPTH_TO_MASK(ip_width); + eat_bits += FIB_NEXT_STRIDE; + fib_insert_node(p, ip, next_hop, ip_width, eat_bits, depth); + } +} + +void fib_tbl_init(void) +{ + int i; + fib_node_t *fe; + uint32_t size; + odp_shm_t lpm_shm; + + for (i = 0; i < (1 << FIB_FIRST_STRIDE); i++) { + fe = &fib_rt_tbl[i]; + fe->valid = 0; + fe->end = 1; + fe->depth = 0; + fe->next_hop = 0; + } + + size = FIB_NEXT_SIZE * FIB_SUB_COUNT; + /*Reserve memory for Routing hash table*/ + lpm_shm = odp_shm_reserve("fib_lpm_sub", size, ODP_CACHE_LINE_SIZE, 0); + fe = odp_shm_addr(lpm_shm); + if (!fe) { + EXAMPLE_ERR("Error: shared mem alloc failed for lpm cache.\n"); + exit(-1); + } + + fib_lpm_cache.fib_nodes = fe; + fib_lpm_cache.fib_count = FIB_SUB_COUNT; + fib_lpm_cache.fib_idx = 0; +} + +void fib_tbl_insert(uint32_t ip, int port, int depth) +{ + fib_node_t *fe, *p; + uint32_t idx; + int i, j; + int nb_bits; + + nb_bits = FIB_FIRST_STRIDE; + idx = ip >> nb_bits; + fe = &fib_rt_tbl[idx]; + if (depth <= nb_bits) { + if (fe->end) { + fe->next_hop = port; + fe->depth = depth; + fe->valid = 1; + return; + } + + for (i = 0; i < FIB_NEXT_SIZE; i++) { + p = &fe->next[i]; + if (p->end) + fib_update_node(p, port, depth); + else + for (j = 0; j < FIB_NEXT_SIZE; j++) + fib_update_node(&p->next[j], port, + depth); + } + + return; + } + + /* need to check sub table */ + ip &= DEPTH_TO_MASK(FIB_IP_WIDTH - nb_bits); + fib_insert_node(fe, ip, port, FIB_IP_WIDTH - nb_bits, nb_bits, depth); +} + +int fib_tbl_lookup(uint32_t ip, int *port) +{ + fib_node_t *fe; + uint32_t idx; + int nb_bits; + + nb_bits = FIB_IP_WIDTH - FIB_FIRST_STRIDE; + idx = ip >> nb_bits; + fe = &fib_rt_tbl[idx]; + + ip &= DEPTH_TO_MASK(nb_bits); + while (!fe->end) { + nb_bits -= FIB_NEXT_STRIDE; + idx = ip >> nb_bits; + fe = &fe->next[idx]; + ip &= DEPTH_TO_MASK(nb_bits); + } + *port = fe->next_hop; + + return fe->valid ? 0 : -1; +} diff --git a/example/l3fwd/odp_l3fwd_lpm.h b/example/l3fwd/odp_l3fwd_lpm.h new file mode 100644 index 0000000..9256642 --- /dev/null +++ b/example/l3fwd/odp_l3fwd_lpm.h @@ -0,0 +1,20 @@ +/* Copyright (c) 2016, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#ifndef _ODP_L3FWD_LPM_H_ +#define _ODP_L3FWD_LPM_H_ + +#ifdef __cplusplus +extern "C" { +#endif +void fib_tbl_init(void); +void fib_tbl_insert(uint32_t ip, int port, int depth); +int fib_tbl_lookup(uint32_t ip, int *port); +#ifdef __cplusplus +} +#endif + +#endif diff --git a/example/m4/configure.m4 b/example/m4/configure.m4 index bbda38f..620db04 100644 --- a/example/m4/configure.m4 +++ b/example/m4/configure.m4 @@ -12,12 +12,13 @@ AC_ARG_ENABLE([test-example], AC_CONFIG_FILES([example/classifier/Makefile example/generator/Makefile + example/hello/Makefile example/ipsec/Makefile - example/Makefile + example/l2fwd_simple/Makefile + example/l3fwd/Makefile example/packet/Makefile + example/switch/Makefile example/time/Makefile example/timer/Makefile example/traffic_mgmt/Makefile - example/l2fwd_simple/Makefile - example/switch/Makefile - example/hello/Makefile]) + example/Makefile])