From patchwork Thu Feb 19 11:27:29 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Venkatesh Vivekanandan X-Patchwork-Id: 44815 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-wg0-f69.google.com (mail-wg0-f69.google.com [74.125.82.69]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id A35EF21554 for ; Thu, 19 Feb 2015 11:31:50 +0000 (UTC) Received: by mail-wg0-f69.google.com with SMTP id k14sf4630024wgh.0 for ; Thu, 19 Feb 2015 03:31:49 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:from:to:date:message-id :mime-version:subject:precedence:list-id:list-unsubscribe :list-archive:list-post:list-help:list-subscribe:content-type :content-transfer-encoding:errors-to:sender:x-original-sender :x-original-authentication-results:mailing-list; bh=5qgxXYTARDYST6LT34EW+lftSdTZchHuVop/RDiF1u8=; b=FZRAdSMOrpYBTJVMj0Zp7B9a+PkZ7Fo9Um2hZcS3sBbS8kVT4bp5IsM3CQyWweyao3 qK1GtnJL1rrY6mx3Y+Wy7X/A/FTy+hWf/w/ri5t9gKINQ+vB4D+nwQrkLmyGcyg0VBof 59aQtOyo/1NK0pIUjT9D1Zah0KELwLpiQle53BOFrv750+xt5FTAZcyiRAoJMv5iuEAk fS8W55KSiiEeJH2ZU2PCSdf1iEpJvQNUGNoOXEztPiZ57KyQZ8Qq6MJdTzqcLMNPbJDR 5YOPs2k8DHp7hmopuU2y6huxC39NrsaXR/dzFxS5EYL3tGTUGVo9WJsoQtcc0LNy5kCa 3+hQ== X-Gm-Message-State: ALoCoQkB/H1ZKRUpJeiKceuzed8BjZYe/prDJBhoW9eCBgI9aFcTVow1lAi7b14frG9szamLtQCx X-Received: by 10.152.5.3 with SMTP id o3mr515877lao.6.1424345509731; Thu, 19 Feb 2015 03:31:49 -0800 (PST) X-BeenThere: patchwork-forward@linaro.org Received: by 10.152.5.137 with SMTP id s9ls140692las.15.gmail; Thu, 19 Feb 2015 03:31:49 -0800 (PST) X-Received: by 10.112.212.42 with SMTP id nh10mr3672583lbc.102.1424345509537; Thu, 19 Feb 2015 03:31:49 -0800 (PST) Received: from mail-la0-f52.google.com (mail-la0-f52.google.com. [209.85.215.52]) by mx.google.com with ESMTPS id sm5si14615578lbb.39.2015.02.19.03.31.49 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 19 Feb 2015 03:31:49 -0800 (PST) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.215.52 as permitted sender) client-ip=209.85.215.52; Received: by lams18 with SMTP id s18so6999444lam.11 for ; Thu, 19 Feb 2015 03:31:49 -0800 (PST) X-Received: by 10.112.42.225 with SMTP id r1mr79638lbl.72.1424345509330; Thu, 19 Feb 2015 03:31:49 -0800 (PST) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.112.35.133 with SMTP id h5csp430568lbj; Thu, 19 Feb 2015 03:31:47 -0800 (PST) X-Received: by 10.140.91.74 with SMTP id y68mr10569146qgd.90.1424345506765; Thu, 19 Feb 2015 03:31:46 -0800 (PST) Received: from ip-10-35-177-41.ec2.internal (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTPS id 63si16132428qhw.118.2015.02.19.03.31.45 (version=TLSv1 cipher=RC4-SHA bits=128/128); Thu, 19 Feb 2015 03:31:46 -0800 (PST) Received-SPF: none (google.com: lng-odp-bounces@lists.linaro.org does not designate permitted sender hosts) client-ip=54.225.227.206; Received: from localhost ([127.0.0.1] helo=ip-10-35-177-41.ec2.internal) by ip-10-35-177-41.ec2.internal with esmtp (Exim 4.76) (envelope-from ) id 1YOPKZ-0005Zf-8l; Thu, 19 Feb 2015 11:31:43 +0000 Received: from mail-gw2-out.broadcom.com ([216.31.210.63]) by ip-10-35-177-41.ec2.internal with esmtp (Exim 4.76) (envelope-from ) id 1YOPKO-0005Yt-WB for lng-odp@lists.linaro.org; Thu, 19 Feb 2015 11:31:33 +0000 X-IronPort-AV: E=Sophos;i="5.09,512,1418112000"; d="scan'208";a="57486157" Received: from irvexchcas08.broadcom.com (HELO IRVEXCHCAS08.corp.ad.broadcom.com) ([10.9.208.57]) by mail-gw2-out.broadcom.com with ESMTP; 19 Feb 2015 04:12:32 -0800 Received: from IRVEXCHSMTP2.corp.ad.broadcom.com (10.9.207.52) by IRVEXCHCAS08.corp.ad.broadcom.com (10.9.208.57) with Microsoft SMTP Server (TLS) id 14.3.174.1; Thu, 19 Feb 2015 03:31:27 -0800 Received: from mail-irva-13.broadcom.com (10.10.10.20) by IRVEXCHSMTP2.corp.ad.broadcom.com (10.9.207.52) with Microsoft SMTP Server id 14.3.174.1; Thu, 19 Feb 2015 03:31:27 -0800 Received: from localhost.localdomain (unknown [10.131.60.56]) by mail-irva-13.broadcom.com (Postfix) with ESMTP id BBFBE40FE8; Thu, 19 Feb 2015 03:30:19 -0800 (PST) From: To: Date: Thu, 19 Feb 2015 16:57:29 +0530 Message-ID: <1424345249-9026-1-git-send-email-venkatesh.vivekanandan@linaro.org> X-Mailer: git-send-email 1.9.1 MIME-Version: 1.0 X-Topics: patch Subject: [lng-odp] [PATCHv4 DPDK 6/6] Packet related changes X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: , List-Help: , List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: lng-odp-bounces@lists.linaro.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: venkatesh.vivekanandan@linaro.org X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.215.52 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 From: Venkatesh Vivekanandan - Packet API changes to be in sync with linux-generic - API's that are not implemented in odp-dpdk is marked ODP_UNIMPLEMENTED Signed-off-by: Venkatesh Vivekanandan --- platform/linux-dpdk/include/api/odp_packet.h | 772 ++++++++++++++++-- platform/linux-dpdk/include/odp_packet_internal.h | 135 ++- platform/linux-dpdk/odp_packet.c | 947 ++++++++++++++++------ 3 files changed, 1493 insertions(+), 361 deletions(-) diff --git a/platform/linux-dpdk/include/api/odp_packet.h b/platform/linux-dpdk/include/api/odp_packet.h index 64a92e8..97c2cb6 100644 --- a/platform/linux-dpdk/include/api/odp_packet.h +++ b/platform/linux-dpdk/include/api/odp_packet.h @@ -1,7 +1,7 @@ /* Copyright (c) 2013, Linaro Limited * All rights reserved. * - * SPDX-License-Identifier: BSD-3-Clause + * SPDX-License-Identifier: BSD-3-Clause */ @@ -19,33 +19,84 @@ extern "C" { #endif #include +#include +/** @defgroup odp_packet ODP PACKET + * Operations on a packet. + * @{ + */ -/** - * ODP packet descriptor + +/* + * Packet API v0.5 notes + * - Push/pull operations only on packet level + * - Push/pull within limits of segment headroom/tailroom/data lengths + * - Segment data length must be always at least one byte (i.e. there are no + * empty segments) + * - Head/tailroom content belong to packet content (in addition to data + * and meta-data) and thus is preserved over packet ownership changes. + * - _addr refer to a fixed address, which operations do not modify + * - _ptr refer to pointer to data, which may be modified by operations */ -typedef odp_buffer_t odp_packet_t; -/** Invalid packet */ -#define ODP_PACKET_INVALID ODP_BUFFER_INVALID +/* + * + * Alloc and free + * ******************************************************** + * + */ -/** Invalid offset */ -#define ODP_PACKET_OFFSET_INVALID ((size_t)-1) +/** + * Allocate a packet from a buffer pool + * + * Allocates a packet of the requested length from the specified buffer pool. + * Pool must have been created with buffer type ODP_BUFFER_TYPE_PACKET. The + * packet is initialized with data pointers and lengths set according to the + * specified len, and the default headroom and tailroom length settings. All + * other packet metadata are set to their default values. + * + * @param pool Pool handle + * @param len Packet data length + * + * @return Handle of allocated packet + * @retval ODP_PACKET_INVALID Packet could not be allocated + * + * @note The default headroom and tailroom used for packets is specified by + * the ODP_CONFIG_PACKET_HEADROOM and ODP_CONFIG_PACKET_TAILROOM defines in + * odp_config.h. + */ +odp_packet_t odp_packet_alloc(odp_buffer_pool_t pool, uint32_t len); +/** + * Free packet + * + * Frees the packet into the buffer pool it was allocated from. + * + * @param pkt Packet handle + */ +void odp_packet_free(odp_packet_t pkt); /** - * Initialize the packet + * Reset packet * - * Needs to be called if the user allocates a packet buffer, i.e. the packet - * has not been received from I/O through ODP. + * Resets all packet meta-data to their default values. Packet length is used + * to initialize pointers and lengths. It must be less than the total buffer + * length of the packet minus the default headroom length. Packet is not + * modified on failure. * - * @param pkt Packet handle + * @param pkt Packet handle + * @param len Packet data length + * + * @retval 0 Success + * @retval Non-zero Failure + * + * @see odp_packet_buf_len() */ -void odp_packet_init(odp_packet_t pkt); +int odp_packet_reset(odp_packet_t pkt, uint32_t len); /** - * Convert from packet handle to buffer handle + * Convert a buffer handle to a packet handle * * @param buf Buffer handle * @@ -54,190 +105,737 @@ void odp_packet_init(odp_packet_t pkt); odp_packet_t odp_packet_from_buffer(odp_buffer_t buf); /** - * Convert from buffer handle to packet handle + * Convert a packet handle to a buffer handle * * @param pkt Packet handle * * @return Buffer handle */ -odp_buffer_t odp_buffer_from_packet(odp_packet_t pkt); +odp_buffer_t odp_packet_to_buffer(odp_packet_t pkt); + + +/* + * + * Pointers and lengths + * ******************************************************** + * + */ + +/** + * Packet head address + * + * Returns start address of the first segment. Packet level headroom starts + * from here. Use odp_packet_data() or odp_packet_l2_ptr() to return the + * packet data start address. + * + * @param pkt Packet handle + * + * @return Pointer to the start address of the first packet segment + * + * @see odp_packet_data(), odp_packet_l2_ptr(), odp_packet_headroom() + */ +void *odp_packet_head(odp_packet_t pkt); + +/** + * Total packet buffer length + * + * Returns sum of buffer lengths over all packet segments. + * + * @param pkt Packet handle + * + * @return Total packet buffer length in bytes + * + * @see odp_packet_reset() + */ +uint32_t odp_packet_buf_len(odp_packet_t pkt); /** - * Set the packet length + * Packet data pointer + * + * Returns the current packet data pointer. When a packet is received + * from packet input, this points to the first byte of the received + * packet. Packet level offsets are calculated relative to this position. + * + * User can adjust the data pointer with head_push/head_pull (does not modify + * segmentation) and add_data/rem_data calls (may modify segmentation). + * + * @param pkt Packet handle + * + * @return Pointer to the packet data + * + * @see odp_packet_l2_ptr(), odp_packet_seg_len() + */ +void *odp_packet_data(odp_packet_t pkt); + +/** + * Packet segment data length + * + * Returns number of data bytes following the current data pointer + * (odp_packet_data()) location in the segment. * * @param pkt Packet handle - * @param len Length of packet in bytes + * + * @return Segment data length in bytes (pointed by odp_packet_data()) + * + * @see odp_packet_data() */ -void odp_packet_set_len(odp_packet_t pkt, size_t len); +uint32_t odp_packet_seg_len(odp_packet_t pkt); /** - * Get the packet length + * Packet data length + * + * Returns sum of data lengths over all packet segments. * * @param pkt Packet handle * - * @return Packet length in bytes + * @return Packet data length */ -size_t odp_packet_get_len(odp_packet_t pkt); +uint32_t odp_packet_len(odp_packet_t pkt); /** - * Set packet user context + * Packet headroom length * - * @param buf Packet handle - * @param ctx User context + * Returns the current packet level headroom length. + * + * @param pkt Packet handle * + * @return Headroom length */ -void odp_packet_set_ctx(odp_packet_t buf, const void *ctx); +uint32_t odp_packet_headroom(odp_packet_t pkt); /** - * Get packet user context + * Packet tailroom length * - * @param buf Packet handle + * Returns the current packet level tailroom length. * - * @return User context + * @param pkt Packet handle + * + * @return Tailroom length */ -void *odp_packet_get_ctx(odp_packet_t buf); +uint32_t odp_packet_tailroom(odp_packet_t pkt); /** - * Get address to the start of the packet buffer + * Packet tailroom pointer * - * The address of the packet buffer is not necessarily the same as the start - * address of the received frame, e.g. an eth frame may be offset by 2 or 6 - * bytes to ensure 32 or 64-bit alignment of the IP header. - * Use odp_packet_l2(pkt) to get the start address of a received valid frame - * or odp_packet_start(pkt) to get the start address even if no valid L2 header - * could be found. + * Returns pointer to the start of the current packet level tailroom. + * + * User can adjust the tail pointer with tail_push/tail_pull (does not modify + * segmentation) and add_data/rem_data calls (may modify segmentation). * * @param pkt Packet handle * - * @return Pointer to the start of the packet buffer + * @return Tailroom pointer * - * @see odp_packet_l2(), odp_packet_start() + * @see odp_packet_tailroom() */ -uint8_t *odp_packet_buf_addr(odp_packet_t pkt); +void *odp_packet_tail(odp_packet_t pkt); /** - * Get pointer to the start of the received frame + * Push out packet head + * + * Increase packet data length by moving packet head into packet headroom. + * Packet headroom is decreased with the same amount. The packet head may be + * pushed out up to 'headroom' bytes. Packet is not modified if there's not + * enough headroom space. * - * The address of the packet buffer is not necessarily the same as the start - * address of the received frame, e.g. an eth frame may be offset by 2 or 6 - * bytes to ensure 32 or 64-bit alignment of the IP header. - * Use odp_packet_l2(pkt) to get the start address of a received valid eth frame + * odp_packet_xxx: + * seg_len += len + * len += len + * headroom -= len + * data -= len * - * odp_packet_start() will always return a pointer to the start of the frame, - * even if the frame is unrecognized and no valid L2 header could be found. + * Operation does not modify packet segmentation or move data. Handles and + * pointers remain valid. User is responsible to update packet meta-data + * offsets when needed. * * @param pkt Packet handle + * @param len Number of bytes to push the head (0 ... headroom) * - * @return Pointer to the start of the received frame + * @return The new data pointer + * @retval NULL Requested offset exceeds available headroom * - * @see odp_packet_l2(), odp_packet_buf_addr() + * @see odp_packet_headroom(), odp_packet_pull_head() */ -uint8_t *odp_packet_start(odp_packet_t pkt); +void *odp_packet_push_head(odp_packet_t pkt, uint32_t len); /** - * Get pointer to the start of the L2 frame + * Pull in packet head * - * The L2 frame header address is not necessarily the same as the address of the - * packet buffer, see odp_packet_buf_addr() + * Decrease packet data length by removing data from the head of the packet. + * Packet headroom is increased with the same amount. Packet head may be pulled + * in up to seg_len - 1 bytes (i.e. packet data pointer must stay in the + * first segment). Packet is not modified if there's not enough data. + * + * odp_packet_xxx: + * seg_len -= len + * len -= len + * headroom += len + * data += len + * + * Operation does not modify packet segmentation or move data. Handles and + * pointers remain valid. User is responsible to update packet meta-data + * offsets when needed. * * @param pkt Packet handle + * @param len Number of bytes to pull the head (0 ... seg_len - 1) * - * @return Pointer to L2 header or NULL if not found + * @return The new data pointer, or NULL in case of an error. + * @retval NULL Requested offset exceeds packet segment length * - * @see odp_packet_buf_addr(), odp_packet_start() + * @see odp_packet_seg_len(), odp_packet_push_head() */ -uint8_t *odp_packet_l2(odp_packet_t pkt); +void *odp_packet_pull_head(odp_packet_t pkt, uint32_t len); /** - * Return the byte offset from the packet buffer to the L2 frame + * Push out packet tail + * + * Increase packet data length by moving packet tail into packet tailroom. + * Packet tailroom is decreased with the same amount. The packet tail may be + * pushed out up to 'tailroom' bytes. Packet is not modified if there's not + * enough tailroom. + * + * last_seg: + * data_len += len + * + * odp_packet_xxx: + * len += len + * tail += len + * tailroom -= len + * + * Operation does not modify packet segmentation or move data. Handles, + * pointers and offsets remain valid. * * @param pkt Packet handle + * @param len Number of bytes to push the tail (0 ... tailroom) * - * @return L2 byte offset or ODP_PACKET_OFFSET_INVALID if not found + * @return The old tail pointer + * @retval NULL Requested offset exceeds available tailroom + * + * @see odp_packet_tailroom(), odp_packet_pull_tail() */ -size_t odp_packet_l2_offset(odp_packet_t pkt); +void *odp_packet_push_tail(odp_packet_t pkt, uint32_t len); /** - * Set the byte offset to the L2 frame + * Pull in packet tail + * + * Decrease packet data length by removing data from the tail of the packet. + * Packet tailroom is increased with the same amount. Packet tail may be pulled + * in up to last segment data_len - 1 bytes. (i.e. packet tail must stay in the + * last segment). Packet is not modified if there's not enough data. + * + * last_seg: + * data_len -= len + * + * odp_packet_xxx: + * len -= len + * tail -= len + * tailroom += len + * + * Operation does not modify packet segmentation or move data. Handles and + * pointers remain valid. User is responsible to update packet meta-data + * offsets when needed. + * + * @param pkt Packet handle + * @param len Number of bytes to pull the tail (0 ... last_seg:data_len - 1) + * + * @return The new tail pointer + * @retval NULL The specified offset exceeds allowable data length + */ +void *odp_packet_pull_tail(odp_packet_t pkt, uint32_t len); + +/** + * Packet offset pointer + * + * Returns pointer to data in the packet offset. The packet level byte offset is + * calculated from the current odp_packet_data() position. Optionally outputs + * handle to the segment and number of data bytes in the segment following the + * pointer. + * + * @param pkt Packet handle + * @param offset Byte offset into the packet + * @param[out] len Number of data bytes remaining in the segment (output). + * Ignored when NULL. + * @param[out] seg Handle to the segment containing the address (output). + * Ignored when NULL. + * + * @return Pointer to the offset + * @retval NULL Requested offset exceeds packet length + */ +void *odp_packet_offset(odp_packet_t pkt, uint32_t offset, uint32_t *len, + odp_packet_seg_t *seg); + +/* + * + * Meta-data + * ******************************************************** + * + */ + +/** + * Packet pool + * + * Returns handle to the buffer pool where the packet was allocated from. + * + * @param pkt Packet handle + * + * @return Buffer pool handle + */ +odp_buffer_pool_t odp_packet_pool(odp_packet_t pkt); + +/** + * Packet input interface + * + * Returns handle to the packet IO interface which received the packet or + * ODP_PKTIO_INVALID when the packet was allocated/reset by the application. + * + * @param pkt Packet handle + * + * @return Packet interface handle + * @retval ODP_PKTIO_INVALID Packet was not received + */ +odp_pktio_t odp_packet_input(odp_packet_t pkt); + +/** + * User context pointer + * + * Return previously stored user context pointer. + * + * @param pkt Packet handle + * + * @return User context pointer + */ +void *odp_packet_user_ptr(odp_packet_t pkt); + +/** + * Set user context pointer + * + * Each packet has room for a user defined context. The context can be stored + * either as a pointer OR as a uint64_t value, but not both at the same time. + * The latest context set operation determines which one has been stored. + * + * @param pkt Packet handle + * @param ctx User context pointer + */ +void odp_packet_user_ptr_set(odp_packet_t pkt, const void *ctx); + +/** + * User context data (uint64_t) + * + * Return previously stored user context uint64_t value. + * + * @param pkt Packet handle + * + * @return User context data + */ +uint64_t odp_packet_user_u64(odp_packet_t pkt); + +/** + * Set user context data (uint64_t) + * + * Each packet has room for a user defined context. The context can be stored + * either as a pointer OR as a uint64_t value, but not both at the same time. + * The latest context set operation determines which one has been stored. + * + * @param pkt Packet handle + * @param ctx User context data + */ +void odp_packet_user_u64_set(odp_packet_t pkt, uint64_t ctx); + +/** + * Layer 2 start pointer + * + * Returns pointer to the start of the layer 2 header. Optionally, outputs + * number of data bytes in the segment following the pointer. + * + * @param pkt Packet handle + * @param[out] len Number of data bytes remaining in the segment (output). + * Ignored when NULL. + * + * @return Layer 2 start pointer, or offset 0 by default + * + * @see odp_packet_l2_offset(), odp_packet_l2_offset_set() + */ +void *odp_packet_l2_ptr(odp_packet_t pkt, uint32_t *len); + +/** + * Layer 2 start offset + * + * Returns offset to the start of the layer 2 header. The offset is calculated + * from the current odp_packet_data() position in bytes. + * + * User is responsible to update the offset when modifying the packet data + * pointer position. + * + * @param pkt Packet handle + * + * @return Layer 2 start offset + */ +uint32_t odp_packet_l2_offset(odp_packet_t pkt); + +/** + * Set layer 2 start offset + * + * Set offset to the start of the layer 2 header. The offset is calculated from + * the current odp_packet_data() position in bytes. Offset must not exceed + * packet data length. Packet is not modified on an error. * * @param pkt Packet handle - * @param offset L2 byte offset + * @param offset Layer 2 start offset (0 ... odp_packet_len()-1) + * + * @retval 0 Success + * @retval Non-zero Failure */ -void odp_packet_set_l2_offset(odp_packet_t pkt, size_t offset); +int odp_packet_l2_offset_set(odp_packet_t pkt, uint32_t offset); +/** + * Layer 3 start pointer + * + * Returns pointer to the start of the layer 3 header. Optionally, outputs + * number of data bytes in the segment following the pointer. + * + * @param pkt Packet handle + * @param[out] len Number of data bytes remaining in the segment (output). + * Ignored when NULL. + * + * @return Layer 3 start pointer, or NULL + * + * @see odp_packet_l3_offset(), odp_packet_l3_offset_set() + */ +void *odp_packet_l3_ptr(odp_packet_t pkt, uint32_t *len); /** - * Get pointer to the start of the L3 packet + * Layer 3 start offset + * + * Returns offset to the start of the layer 3 header. The offset is calculated + * from the current odp_packet_data() position in bytes. + * + * User is responsible to update the offset when modifying the packet data + * pointer position. * * @param pkt Packet handle * - * @return Pointer to L3 packet or NULL if not found + * @return Layer 3 start offset or ODP_PACKET_OFFSET_INVALID if not found + */ +uint32_t odp_packet_l3_offset(odp_packet_t pkt); + +/** + * Set layer 3 start offset + * + * Set offset to the start of the layer 3 header. The offset is calculated from + * the current odp_packet_data() position in bytes. Offset must not exceed + * packet data length. Packet is not modified on an error. + * + * @param pkt Packet handle + * @param offset Layer 3 start offset (0 ... odp_packet_len()-1) * + * @retval 0 Success + * @retval Non-zero Failure */ -uint8_t *odp_packet_l3(odp_packet_t pkt); +int odp_packet_l3_offset_set(odp_packet_t pkt, uint32_t offset); /** - * Return the byte offset from the packet buffer to the L3 packet + * Layer 4 start pointer + * + * Returns pointer to the start of the layer 4 header. Optionally, outputs + * number of data bytes in the segment following the pointer. + * + * @param pkt Packet handle + * @param[out] len Number of data bytes remaining in the segment (output). + * Ignored when NULL. + * + * @return Layer 4 start pointer, or NULL + * + * @see odp_packet_l4_offset(), odp_packet_l4_offset_set() + */ +void *odp_packet_l4_ptr(odp_packet_t pkt, uint32_t *len); + +/** + * Layer 4 start offset + * + * Returns offset to the start of the layer 4 header. The offset is calculated + * from the current odp_packet_data() position in bytes. + * + * User is responsible to update the offset when modifying the packet data + * pointer position. * * @param pkt Packet handle * - * @return L3 byte offset or ODP_PACKET_OFFSET_INVALID if not found + * @return Layer 4 start offset or ODP_PACKET_OFFSET_INVALID if not found */ -size_t odp_packet_l3_offset(odp_packet_t pkt); +uint32_t odp_packet_l4_offset(odp_packet_t pkt); /** - * Set the byte offset to the L3 packet + * Set layer 4 start offset + * + * Set offset to the start of the layer 4 header. The offset is calculated from + * the current odp_packet_data() position in bytes. Offset must not exceed + * packet data length. Packet is not modified on an error. * * @param pkt Packet handle - * @param offset L3 byte offset + * @param offset Layer 4 start offset (0 ... odp_packet_len()-1) + * + * @retval 0 Success + * @retval Non-zero Failure + */ +int odp_packet_l4_offset_set(odp_packet_t pkt, uint32_t offset); + +/** + * Tests if packet is segmented + * + * @param pkt Packet handle + * + * @retval 0 Packet is not segmented + * @retval 1 Packet is segmented + */ +int odp_packet_is_segmented(odp_packet_t pkt); + +/** + * Number of segments + * + * Returns number of segments in the packet. A packet has always at least one + * segment. + * + * @param pkt Packet handle + * + * @return Number of segments (>0) + */ +int odp_packet_num_segs(odp_packet_t pkt); + +/** + * First segment in packet + * + * A packet has always the first segment (has at least one segment). + * + * @param pkt Packet handle + * + * @return Handle to the first segment + */ +odp_packet_seg_t odp_packet_first_seg(odp_packet_t pkt); + +/** + * Last segment in packet + * + * A packet has always the last segment (has at least one segment). + * + * @param pkt Packet handle + * + * @return Handle to the last segment + */ +odp_packet_seg_t odp_packet_last_seg(odp_packet_t pkt); + +/** + * Next segment in packet + * + * Returns handle to the next segment after the current segment, or + * ODP_PACKET_SEG_INVALID if there are no more segments. Use + * odp_packet_first_seg() to get handle to the first segment. + * + * @param pkt Packet handle + * @param seg Current segment handle + * + * @return Handle to the next segment, or ODP_PACKET_SEG_INVALID + */ +odp_packet_seg_t odp_packet_next_seg(odp_packet_t pkt, odp_packet_seg_t seg); + + +/* + * + * Segment level + * ******************************************************** + * */ -void odp_packet_set_l3_offset(odp_packet_t pkt, size_t offset); +/** + * Segment buffer address + * + * Returns start address of the segment. + * + * @param pkt Packet handle + * @param seg Segment handle + * + * @return Start address of the segment, or NULL on an error + * + * @see odp_packet_seg_buf_len() + */ +void *odp_packet_seg_buf_addr(odp_packet_t pkt, odp_packet_seg_t seg); /** - * Get pointer to the start of the L4 packet + * Segment buffer length + * + * Returns segment buffer length in bytes. * * @param pkt Packet handle + * @param seg Segment handle * - * @return Pointer to L4 packet or NULL if not found + * @return Segment buffer length in bytes * + * @see odp_packet_seg_buf_addr() */ -uint8_t *odp_packet_l4(odp_packet_t pkt); +uint32_t odp_packet_seg_buf_len(odp_packet_t pkt, odp_packet_seg_t seg); /** - * Return the byte offset from the packet buffer to the L4 packet + * Segment data pointer + * + * Returns pointer to the first byte of data in the segment. + * + * @param pkt Packet handle + * @param seg Segment handle + * + * @return Pointer to the segment data, or NULL on an error + * + * @see odp_packet_seg_data_len() + */ +void *odp_packet_seg_data(odp_packet_t pkt, odp_packet_seg_t seg); + +/** + * Segment data length + * + * Returns segment data length in bytes. * * @param pkt Packet handle + * @param seg Segment handle * - * @return L4 byte offset or ODP_PACKET_OFFSET_INVALID if not found + * @return Segment data length in bytes + * + * @see odp_packet_seg_data() */ -size_t odp_packet_l4_offset(odp_packet_t pkt); +uint32_t odp_packet_seg_data_len(odp_packet_t pkt, odp_packet_seg_t seg); + + +/* + * + * Manipulation + * ******************************************************** + * + */ + /** - * Set the byte offset to the L4 packet + * Add data into an offset + * + * Increases packet data length by adding new data area into the specified + * offset. The operation returns a new packet handle on success. It may modify + * packet segmentation and move data. Handles and pointers must be updated + * after the operation. User is responsible to update packet meta-data offsets + * when needed. The packet is not modified on an error. * * @param pkt Packet handle - * @param offset L4 byte offset + * @param offset Byte offset into the packet + * @param len Number of bytes to add into the offset + * + * @return New packet handle, or ODP_PACKET_INVALID in case of an error. */ -void odp_packet_set_l4_offset(odp_packet_t pkt, size_t offset); +odp_packet_t odp_packet_add_data(odp_packet_t pkt, uint32_t offset, + uint32_t len); /** - * Print (debug) information about the packet + * Remove data from an offset + * + * Decreases packet data length by removing data from the specified offset. + * The operation returns a new packet handle on success, and may modify + * packet segmentation and move data. Handles and pointers must be updated + * after the operation. User is responsible to update packet meta-data offsets + * when needed. The packet is not modified on an error. + * + * @param pkt Packet handle + * @param offset Byte offset into the packet + * @param len Number of bytes to remove from the offset + * + * @return New packet handle, or ODP_PACKET_INVALID in case of an error. + */ +odp_packet_t odp_packet_rem_data(odp_packet_t pkt, uint32_t offset, + uint32_t len); + + +/* + * + * Copy + * ******************************************************** + * + */ + +/** + * Copy packet + * + * Create a new copy of the packet. The new packet is exact copy of the source + * packet (incl. data and meta-data). The pool must have been created with + * buffer type ODP_BUFFER_TYPE_PACKET. + * + * @param pkt Packet handle + * @param pool Buffer pool for allocation of the new packet. + * + * @return Handle to the copy of the packet, or ODP_PACKET_INVALID + */ +odp_packet_t odp_packet_copy(odp_packet_t pkt, odp_buffer_pool_t pool); + +/** + * Copy data from packet + * + * Copy 'len' bytes of data from the packet level offset to the destination + * address. + * + * @param pkt Packet handle + * @param offset Byte offset into the packet + * @param len Number of bytes to copy + * @param dst Destination address + * + * @retval 0 Success + * @retval Non-zero Failure + */ +int odp_packet_copydata_out(odp_packet_t pkt, uint32_t offset, + uint32_t len, void *dst); + +/** + * Copy data into packet + * + * Copy 'len' bytes of data from the source address into the packet level + * offset. Maximum number of bytes to copy is packet data length minus the + * offset. Packet is not modified on an error. + * + * @param pkt Packet handle + * @param offset Byte offset into the packet + * @param len Number of bytes to copy + * @param src Source address + * + * @retval 0 Success + * @retval Non-zero Failure + */ +int odp_packet_copydata_in(odp_packet_t pkt, uint32_t offset, + uint32_t len, const void *src); + +/* + * + * Debugging + * ******************************************************** + * + */ + +/** + * Print packet to the console + * + * Print all packet debug information to the console. * * @param pkt Packet handle */ void odp_packet_print(odp_packet_t pkt); /** - * Copy contents and metadata from pkt_src to pkt_dst - * Useful when creating copies of packets + * Perform full packet validity check * - * @param pkt_dst Destination packet - * @param pkt_src Source packet + * The operation may consume considerable number of cpu cycles depending on + * the check level. + * + * @param pkt Packet handle * - * @return 0 if successful + * @retval 0 Packet is not valid + * @retval 1 Packet is valid + */ +int odp_packet_is_valid(odp_packet_t pkt); + + +/** + * @} */ -int odp_packet_copy(odp_packet_t pkt_dst, odp_packet_t pkt_src); #ifdef __cplusplus } diff --git a/platform/linux-dpdk/include/odp_packet_internal.h b/platform/linux-dpdk/include/odp_packet_internal.h index 1cd44b1..24de33f 100644 --- a/platform/linux-dpdk/include/odp_packet_internal.h +++ b/platform/linux-dpdk/include/odp_packet_internal.h @@ -22,6 +22,7 @@ extern "C" { #include #include #include +#include #include #include @@ -43,6 +44,7 @@ typedef union { uint32_t vlan:1; /**< VLAN hdr found */ uint32_t vlan_qinq:1; /**< Stacked VLAN found, QinQ */ + uint32_t snap:1; /**< SNAP */ uint32_t arp:1; /**< ARP */ uint32_t ipv4:1; /**< IPv4 */ @@ -53,13 +55,14 @@ typedef union { uint32_t udp:1; /**< UDP */ uint32_t tcp:1; /**< TCP */ + uint32_t tcpopt:1; /**< TCP options present */ uint32_t sctp:1; /**< SCTP */ uint32_t icmp:1; /**< ICMP */ }; } input_flags_t; -ODP_STATIC_ASSERT(sizeof(input_flags_t) == sizeof(uint32_t), - "INPUT_FLAGS_SIZE_ERROR"); +_ODP_STATIC_ASSERT(sizeof(input_flags_t) == sizeof(uint32_t), + "INPUT_FLAGS_SIZE_ERROR"); /** * Packet error flags @@ -70,7 +73,9 @@ typedef union { struct { /* Bitfield flags for each detected error */ + uint32_t app_error:1; /**< Error bit for application use */ uint32_t frame_len:1; /**< Frame length error */ + uint32_t snap_len:1; /**< Snap length error */ uint32_t l2_chksum:1; /**< L2 checksum error, checks TBD */ uint32_t ip_err:1; /**< IP error, checks TBD */ uint32_t tcp_err:1; /**< TCP error, checks TBD */ @@ -78,8 +83,8 @@ typedef union { }; } error_flags_t; -ODP_STATIC_ASSERT(sizeof(error_flags_t) == sizeof(uint32_t), - "ERROR_FLAGS_SIZE_ERROR"); +_ODP_STATIC_ASSERT(sizeof(error_flags_t) == sizeof(uint32_t), + "ERROR_FLAGS_SIZE_ERROR"); /** * Packet output flags @@ -90,12 +95,15 @@ typedef union { struct { /* Bitfield flags for each output option */ - uint32_t l4_chksum:1; /**< Request L4 checksum calculation */ + uint32_t l3_chksum_set:1; /**< L3 chksum bit is valid */ + uint32_t l3_chksum:1; /**< L3 chksum override */ + uint32_t l4_chksum_set:1; /**< L3 chksum bit is valid */ + uint32_t l4_chksum:1; /**< L4 chksum override */ }; } output_flags_t; -ODP_STATIC_ASSERT(sizeof(output_flags_t) == sizeof(uint32_t), - "OUTPUT_FLAGS_SIZE_ERROR"); +_ODP_STATIC_ASSERT(sizeof(output_flags_t) == sizeof(uint32_t), + "OUTPUT_FLAGS_SIZE_ERROR"); /** * Internal Packet header @@ -108,28 +116,131 @@ typedef struct { error_flags_t error_flags; output_flags_t output_flags; - uint32_t frame_offset; /**< offset to start of frame, even on error */ uint32_t l2_offset; /**< offset to L2 hdr, e.g. Eth */ uint32_t l3_offset; /**< offset to L3 hdr, e.g. IPv4, IPv6 */ uint32_t l4_offset; /**< offset to L4 hdr (TCP, UDP, SCTP, also ICMP) */ + uint32_t payload_offset; /**< offset to payload */ - uint64_t user_ctx; /**< user context */ + uint32_t vlan_s_tag; /**< Parsed 1st VLAN header (S-TAG) */ + uint32_t vlan_c_tag; /**< Parsed 2nd VLAN header (C-TAG) */ + uint32_t l3_protocol; /**< Parsed L3 protocol */ + uint32_t l3_len; /**< Layer 3 length */ + uint32_t l4_protocol; /**< Parsed L4 protocol */ + uint32_t l4_len; /**< Layer 4 length */ + + uint32_t frame_len; + uint32_t headroom; + uint32_t tailroom; odp_pktio_t input; } odp_packet_hdr_t; +typedef struct odp_packet_hdr_stride { + uint8_t pad[ODP_CACHE_LINE_SIZE_ROUNDUP(sizeof(odp_packet_hdr_t))]; +} odp_packet_hdr_stride; + +_ODP_STATIC_ASSERT(sizeof(odp_packet_hdr_t) % sizeof(uint64_t) == 0, + "ODP_PACKET_HDR_T__SIZE_ERR2"); + /** * Return the packet header */ static inline odp_packet_hdr_t *odp_packet_hdr(odp_packet_t pkt) { - return (odp_packet_hdr_t *)pkt; + return (odp_packet_hdr_t *)odp_buf_to_hdr((odp_buffer_t)pkt); } /** - * Parse packet and set internal metadata + * Initialize packet buffer */ -void odp_packet_parse(odp_packet_t pkt, size_t len, size_t l2_offset); +static inline void packet_init(pool_entry_t *pool, + odp_packet_hdr_t *pkt_hdr, + size_t size) +{ + /* + * Reset parser metadata. Note that we clear via memset to make + * this routine indepenent of any additional adds to packet metadata. + */ + struct rte_mbuf *mb; + uint8_t *start; + + mb = (struct rte_mbuf *)&pkt_hdr->buf_hdr; + mb->buf_len = size; + start = (uint8_t *)mb->buf_addr; + memset(start, 0, mb->buf_len); + + /* + * Packet headroom is set from the pool's headroom + * Packet tailroom is rounded up to fill the last + * segment occupied by the allocated length. + */ + pkt_hdr->frame_len = size; + pkt_hdr->headroom = pool->s.headroom; + pkt_hdr->tailroom = + (pool->s.seg_size * pkt_hdr->buf_hdr.segcount) - + (pool->s.headroom + size); +} + +static inline void *packet_map(odp_packet_hdr_t *pkt_hdr, + uint32_t offset, uint32_t *seglen) +{ + struct rte_mbuf *mbuf = (struct rte_mbuf *)&pkt_hdr->buf_hdr; + if (offset > pkt_hdr->frame_len) + return NULL; + + if (seglen != NULL) { + *seglen = mbuf->pkt.data_len; + pkt_hdr->frame_len = mbuf->pkt.data_len; + } + return (void *)((uint8_t *)mbuf->pkt.data + offset); +} + +#define pull_offset(x, len) (x = x < len ? 0 : x - len) + +static inline void push_head(odp_packet_hdr_t *pkt_hdr, size_t len) +{ + pkt_hdr->headroom -= len; + pkt_hdr->frame_len += len; + pkt_hdr->l2_offset += len; + pkt_hdr->l3_offset += len; + pkt_hdr->l4_offset += len; +} + +static inline void pull_head(odp_packet_hdr_t *pkt_hdr, size_t len) +{ + pkt_hdr->headroom += len; + pkt_hdr->frame_len -= len; + pull_offset(pkt_hdr->l2_offset, len); + pull_offset(pkt_hdr->l3_offset, len); + pull_offset(pkt_hdr->l4_offset, len); +} + +static inline void push_tail(odp_packet_hdr_t *pkt_hdr, size_t len) +{ + pkt_hdr->tailroom -= len; + pkt_hdr->frame_len += len; +} + + +static inline void pull_tail(odp_packet_hdr_t *pkt_hdr, size_t len) +{ + pkt_hdr->tailroom += len; + pkt_hdr->frame_len -= len; +} + +static inline void packet_set_len(odp_packet_t pkt, uint32_t len) +{ + odp_packet_hdr(pkt)->frame_len = len; +} + +/* Forward declarations */ +int _odp_packet_copy_to_packet(odp_packet_t srcpkt, uint32_t srcoffset, + odp_packet_t dstpkt, uint32_t dstoffset, + uint32_t len); + +odp_packet_t _odp_packet_alloc(odp_buffer_pool_t pool_hdl); + +int _odp_packet_parse(odp_packet_t pkt); #ifdef __cplusplus } diff --git a/platform/linux-dpdk/odp_packet.c b/platform/linux-dpdk/odp_packet.c index beb69b2..0668be8 100644 --- a/platform/linux-dpdk/odp_packet.c +++ b/platform/linux-dpdk/odp_packet.c @@ -6,34 +6,63 @@ #include #include +#include #include #include #include #include +#include +#include #include #include -static inline uint8_t parse_ipv4(odp_packet_hdr_t *pkt_hdr, - odph_ipv4hdr_t *ipv4, size_t *offset_out); -static inline uint8_t parse_ipv6(odp_packet_hdr_t *pkt_hdr, - odph_ipv6hdr_t *ipv6, size_t *offset_out); +/* + * + * Alloc and free + * ******************************************************** + * + */ -void odp_packet_init(odp_packet_t pkt) +odp_packet_t odp_packet_alloc(odp_buffer_pool_t pool_hdl, uint32_t len) { - odp_packet_hdr_t *const pkt_hdr = odp_packet_hdr(pkt); - struct rte_mbuf *mb; - void *start; + pool_entry_t *pool = odp_pool_to_entry(pool_hdl); + + if (pool->s.params.buf_type != ODP_BUFFER_TYPE_PACKET) + return ODP_PACKET_INVALID; + + /* Handle special case for zero-length packets */ + if (len == 0) { + odp_packet_t pkt = + (odp_packet_t)buffer_alloc(pool_hdl, + pool->s.params.buf_size); + if (pkt != ODP_PACKET_INVALID) + pull_tail(odp_packet_hdr(pkt), + pool->s.params.buf_size); + + return pkt; + } + + return (odp_packet_t)buffer_alloc(pool_hdl, len); +} - mb = &pkt_hdr->buf_hdr.mb; +void odp_packet_free(odp_packet_t pkt) +{ + odp_buffer_free((odp_buffer_t)pkt); +} + +int odp_packet_reset(odp_packet_t pkt, uint32_t len) +{ + odp_packet_hdr_t *const pkt_hdr = odp_packet_hdr(pkt); + pool_entry_t *pool = odp_buf_to_pool(&pkt_hdr->buf_hdr); + uint32_t totsize = pool->s.headroom + len + pool->s.tailroom; - start = mb->buf_addr; - memset(start, 0, mb->buf_len); + if (totsize > pkt_hdr->buf_hdr.size) + return -1; - pkt_hdr->l2_offset = (uint32_t) ODP_PACKET_OFFSET_INVALID; - pkt_hdr->l3_offset = (uint32_t) ODP_PACKET_OFFSET_INVALID; - pkt_hdr->l4_offset = (uint32_t) ODP_PACKET_OFFSET_INVALID; + packet_init(pool, pkt_hdr, len); + return 0; } odp_packet_t odp_packet_from_buffer(odp_buffer_t buf) @@ -41,305 +70,418 @@ odp_packet_t odp_packet_from_buffer(odp_buffer_t buf) return (odp_packet_t)buf; } -odp_buffer_t odp_buffer_from_packet(odp_packet_t pkt) +odp_buffer_t odp_packet_to_buffer(odp_packet_t pkt) { return (odp_buffer_t)pkt; } -/* Advance the pkt data pointer and set len in one call */ -static int odp_packet_set_offset_len(odp_packet_t pkt, size_t frame_offset, - size_t len) +/* + * + * Pointers and lengths + * ******************************************************** + * + */ + +void *odp_packet_head(odp_packet_t pkt) { - struct rte_mbuf *mb = &(odp_packet_hdr(pkt)->buf_hdr.mb); - uint16_t offset; - uint16_t data_len; + odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt); + return buffer_map(&pkt_hdr->buf_hdr, 0, NULL, 0); +} - /* The pkt buf may have been pulled back into the headroom - * so we cannot rely on finding the data right after the - * ODP header and HEADROOM */ - offset = (uint16_t)((unsigned long)mb->pkt.data - - (unsigned long)mb->buf_addr); - ODP_ASSERT(mb->buf_len >= offset, "Corrupted mbuf"); - data_len = mb->buf_len - offset; +uint32_t odp_packet_buf_len(odp_packet_t pkt) +{ + return odp_packet_hdr(pkt)->buf_hdr.size; +} - if (data_len < frame_offset) { - ODP_ERR("Frame offset too big"); - return -1; - } - mb->pkt.data = (void *)((char *)mb->pkt.data + frame_offset); - data_len -= frame_offset; +void *odp_packet_data(odp_packet_t pkt) +{ + odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt); + return packet_map(pkt_hdr, 0, NULL); +} - if (data_len < len) { - ODP_ERR("Packet len too big"); - return -1; - } - mb->pkt.pkt_len = len; - mb->pkt.data_len = len; +uint32_t odp_packet_seg_len(odp_packet_t pkt) +{ + odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt); + uint32_t seglen; - return 0; + /* Call returns length of 1st data segment */ + packet_map(pkt_hdr, 0, &seglen); + return seglen; } -void odp_packet_set_len(odp_packet_t pkt, size_t len) +uint32_t odp_packet_len(odp_packet_t pkt) { - (void)odp_packet_set_offset_len(pkt, 0, len); + return odp_packet_hdr(pkt)->frame_len; } -size_t odp_packet_get_len(odp_packet_t pkt) +uint32_t odp_packet_headroom(odp_packet_t pkt) { - struct rte_mbuf *mb = &(odp_packet_hdr(pkt)->buf_hdr.mb); - return mb->pkt.pkt_len; + return odp_packet_hdr(pkt)->headroom; } -uint8_t *odp_packet_buf_addr(odp_packet_t pkt) +uint32_t odp_packet_tailroom(odp_packet_t pkt) { - return odp_buffer_addr(odp_buffer_from_packet(pkt)); + return odp_packet_hdr(pkt)->tailroom; } -uint8_t *odp_packet_start(odp_packet_t pkt) +void *odp_packet_tail(odp_packet_t pkt) { - struct rte_mbuf *mb = &(odp_packet_hdr(pkt)->buf_hdr.mb); - return mb->pkt.data; + odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt); + return packet_map(pkt_hdr, pkt_hdr->frame_len, NULL); } - -uint8_t *odp_packet_l2(odp_packet_t pkt) +void *odp_packet_push_head(odp_packet_t pkt, uint32_t len) { - const size_t offset = odp_packet_l2_offset(pkt); + odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt); - if (odp_unlikely(offset == ODP_PACKET_OFFSET_INVALID)) + if (len > pkt_hdr->headroom) return NULL; - return odp_packet_start(pkt) + offset; + push_head(pkt_hdr, len); + return packet_map(pkt_hdr, 0, NULL); } -size_t odp_packet_l2_offset(odp_packet_t pkt) +void *odp_packet_pull_head(odp_packet_t pkt, uint32_t len) { - return odp_packet_hdr(pkt)->l2_offset; + odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt); + + if (len > pkt_hdr->frame_len) + return NULL; + + pull_head(pkt_hdr, len); + return packet_map(pkt_hdr, 0, NULL); } -void odp_packet_set_l2_offset(odp_packet_t pkt, size_t offset) +void *odp_packet_push_tail(odp_packet_t pkt, uint32_t len) { - odp_packet_hdr(pkt)->l2_offset = offset; + odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt); + struct rte_mbuf *mbuf = (struct rte_mbuf *)&pkt_hdr->buf_hdr; + uint32_t origin = pkt_hdr->frame_len; + + mbuf->pkt.pkt_len += len; + mbuf->pkt.data_len += len; + + if (len > pkt_hdr->tailroom) + return NULL; + + push_tail(pkt_hdr, len); + return packet_map(pkt_hdr, origin, NULL); } -uint8_t *odp_packet_l3(odp_packet_t pkt) +void *odp_packet_pull_tail(odp_packet_t pkt, uint32_t len) { - const size_t offset = odp_packet_l3_offset(pkt); + odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt); - if (odp_unlikely(offset == ODP_PACKET_OFFSET_INVALID)) + if (len > pkt_hdr->frame_len) return NULL; - return odp_packet_start(pkt) + offset; + pull_tail(pkt_hdr, len); + return packet_map(pkt_hdr, pkt_hdr->frame_len, NULL); } -size_t odp_packet_l3_offset(odp_packet_t pkt) +void *odp_packet_offset(odp_packet_t pkt, uint32_t offset, uint32_t *len, + odp_packet_seg_t *seg) { - return odp_packet_hdr(pkt)->l3_offset; + odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt); + void *addr = packet_map(pkt_hdr, offset, len); + + if (addr != NULL && seg != NULL) { + odp_buffer_bits_t seghandle; + seghandle.u32 = (uint32_t)pkt; + seghandle.seg = (pkt_hdr->headroom + offset) / + pkt_hdr->buf_hdr.segsize; + *seg = seghandle.handle; + } + + return addr; } -void odp_packet_set_l3_offset(odp_packet_t pkt, size_t offset) +/* + * + * Meta-data + * ******************************************************** + * + */ + +odp_buffer_pool_t odp_packet_pool(odp_packet_t pkt) { - odp_packet_hdr(pkt)->l3_offset = offset; + return odp_packet_hdr(pkt)->buf_hdr.pool_hdl; } -uint8_t *odp_packet_l4(odp_packet_t pkt) +odp_pktio_t odp_packet_input(odp_packet_t pkt) { - const size_t offset = odp_packet_l4_offset(pkt); + return odp_packet_hdr(pkt)->input; +} - if (odp_unlikely(offset == ODP_PACKET_OFFSET_INVALID)) - return NULL; +void *odp_packet_user_ptr(odp_packet_t pkt) +{ + return odp_packet_hdr(pkt)->buf_hdr.buf_ctx; +} + +void odp_packet_user_ptr_set(odp_packet_t pkt, const void *ctx) +{ + odp_packet_hdr(pkt)->buf_hdr.buf_cctx = ctx; +} + +uint64_t odp_packet_user_u64(odp_packet_t pkt) +{ + return odp_packet_hdr(pkt)->buf_hdr.buf_u64; +} + +void odp_packet_user_u64_set(odp_packet_t pkt, uint64_t ctx) +{ + odp_packet_hdr(pkt)->buf_hdr.buf_u64 = ctx; +} + +void *odp_packet_l2_ptr(odp_packet_t pkt, uint32_t *len) +{ + odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt); + return packet_map(pkt_hdr, pkt_hdr->l2_offset, len); +} + +uint32_t odp_packet_l2_offset(odp_packet_t pkt) +{ + return odp_packet_hdr(pkt)->l2_offset; +} - return odp_packet_start(pkt) + offset; +int odp_packet_l2_offset_set(odp_packet_t pkt, uint32_t offset) +{ + odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt); + + if (offset >= pkt_hdr->frame_len) + return -1; + + pkt_hdr->l2_offset = offset; + return 0; +} + +void *odp_packet_l3_ptr(odp_packet_t pkt, uint32_t *len) +{ + odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt); + return packet_map(pkt_hdr, pkt_hdr->l3_offset, len); +} + +uint32_t odp_packet_l3_offset(odp_packet_t pkt) +{ + return odp_packet_hdr(pkt)->l3_offset; } -size_t odp_packet_l4_offset(odp_packet_t pkt) +int odp_packet_l3_offset_set(odp_packet_t pkt, uint32_t offset) +{ + odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt); + + if (offset >= pkt_hdr->frame_len) + return -1; + + pkt_hdr->l3_offset = offset; + return 0; +} + +void *odp_packet_l4_ptr(odp_packet_t pkt, uint32_t *len) +{ + odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt); + return packet_map(pkt_hdr, pkt_hdr->l4_offset, len); +} + +uint32_t odp_packet_l4_offset(odp_packet_t pkt) { return odp_packet_hdr(pkt)->l4_offset; } -void odp_packet_set_l4_offset(odp_packet_t pkt, size_t offset) +int odp_packet_l4_offset_set(odp_packet_t pkt, uint32_t offset) +{ + odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt); + + if (offset >= pkt_hdr->frame_len) + return -1; + + pkt_hdr->l4_offset = offset; + return 0; +} + +int odp_packet_is_segmented(odp_packet_t pkt ODP_UNUSED) { - odp_packet_hdr(pkt)->l4_offset = offset; + ODP_UNIMPLEMENTED(); + return -1; } -/** - * Simple packet parser: eth, VLAN, IP, TCP/UDP/ICMP +int odp_packet_num_segs(odp_packet_t pkt ODP_UNUSED) +{ + ODP_UNIMPLEMENTED(); + return -1; +} + +odp_packet_seg_t odp_packet_first_seg(odp_packet_t pkt ODP_UNUSED) +{ + ODP_UNIMPLEMENTED(); + return -1; +} + +odp_packet_seg_t odp_packet_last_seg(odp_packet_t pkt ODP_UNUSED) +{ + ODP_UNIMPLEMENTED(); + return -1; +} + +odp_packet_seg_t odp_packet_next_seg(odp_packet_t pkt ODP_UNUSED, + odp_packet_seg_t seg ODP_UNUSED) +{ + ODP_UNIMPLEMENTED(); + return -1; +} + +/* * - * Internal function: caller is resposible for passing only valid packet handles - * , lengths and offsets (usually done&called in packet input). + * Segment level + * ******************************************************** * - * @param pkt Packet handle - * @param len Packet length in bytes - * @param frame_offset Byte offset to L2 header */ -void odp_packet_parse(odp_packet_t pkt, size_t len, size_t frame_offset) + +void *odp_packet_seg_buf_addr(odp_packet_t pkt ODP_UNUSED, + odp_packet_seg_t seg ODP_UNUSED) { - odp_packet_hdr_t *const pkt_hdr = odp_packet_hdr(pkt); - odph_ethhdr_t *eth; - odph_vlanhdr_t *vlan; - odph_ipv4hdr_t *ipv4; - odph_ipv6hdr_t *ipv6; - uint16_t ethtype; - size_t offset = 0; - uint8_t ip_proto = 0; + ODP_UNIMPLEMENTED(); + return (void *)-1; +} - /* The frame_offset is not relevant for frames from DPDK */ - pkt_hdr->input_flags.eth = 1; - (void) frame_offset; - pkt_hdr->frame_offset = 0; - if (odp_packet_set_offset_len(pkt, 0, len)) { - return; - } +uint32_t odp_packet_seg_buf_len(odp_packet_t pkt ODP_UNUSED, + odp_packet_seg_t seg ODP_UNUSED) +{ + ODP_UNIMPLEMENTED(); + return -1; +} - if (odp_unlikely(len < ODPH_ETH_LEN_MIN)) { - pkt_hdr->error_flags.frame_len = 1; - return; - } else if (len > ODPH_ETH_LEN_MAX) { - pkt_hdr->input_flags.jumbo = 1; - } +void *odp_packet_seg_data(odp_packet_t pkt ODP_UNUSED, + odp_packet_seg_t seg ODP_UNUSED) +{ + ODP_UNIMPLEMENTED(); + return (void *)-1; +} - /* Assume valid L2 header, no CRC/FCS check in SW */ - pkt_hdr->input_flags.l2 = 1; - pkt_hdr->l2_offset = 0; +uint32_t odp_packet_seg_data_len(odp_packet_t pkt ODP_UNUSED, + odp_packet_seg_t seg ODP_UNUSED) +{ + ODP_UNIMPLEMENTED(); + return -1; +} - eth = (odph_ethhdr_t *)odp_packet_start(pkt); - ethtype = odp_be_to_cpu_16(eth->type); - vlan = (odph_vlanhdr_t *)ð->type; +/* + * + * Manipulation + * ******************************************************** + * + */ - if (ethtype == ODPH_ETHTYPE_VLAN_OUTER) { - pkt_hdr->input_flags.vlan_qinq = 1; - ethtype = odp_be_to_cpu_16(vlan->tpid); - offset += sizeof(odph_vlanhdr_t); - vlan = &vlan[1]; - } +odp_packet_t odp_packet_add_data(odp_packet_t pkt ODP_UNUSED, + uint32_t offset ODP_UNUSED, + uint32_t len ODP_UNUSED) +{ + ODP_UNIMPLEMENTED(); + return -1; +} - if (ethtype == ODPH_ETHTYPE_VLAN) { - pkt_hdr->input_flags.vlan = 1; - ethtype = odp_be_to_cpu_16(vlan->tpid); - offset += sizeof(odph_vlanhdr_t); - } +odp_packet_t odp_packet_rem_data(odp_packet_t pkt ODP_UNUSED, + uint32_t offset ODP_UNUSED, + uint32_t len ODP_UNUSED) +{ + ODP_UNIMPLEMENTED(); + return -1; +} - /* Set l3_offset+flag only for known ethtypes */ - switch (ethtype) { - case ODPH_ETHTYPE_IPV4: - pkt_hdr->input_flags.ipv4 = 1; - pkt_hdr->input_flags.l3 = 1; - pkt_hdr->l3_offset = ODPH_ETHHDR_LEN + offset; - ipv4 = (odph_ipv4hdr_t *)odp_packet_l3(pkt); - ip_proto = parse_ipv4(pkt_hdr, ipv4, &offset); - break; - case ODPH_ETHTYPE_IPV6: - pkt_hdr->input_flags.ipv6 = 1; - pkt_hdr->input_flags.l3 = 1; - pkt_hdr->l3_offset = frame_offset + ODPH_ETHHDR_LEN + offset; - ipv6 = (odph_ipv6hdr_t *)odp_packet_l3(pkt); - ip_proto = parse_ipv6(pkt_hdr, ipv6, &offset); - break; - case ODPH_ETHTYPE_ARP: - pkt_hdr->input_flags.arp = 1; - /* fall through */ - default: - ip_proto = 0; - break; - } +/* + * + * Copy + * ******************************************************** + * + */ - switch (ip_proto) { - case ODPH_IPPROTO_UDP: - pkt_hdr->input_flags.udp = 1; - pkt_hdr->input_flags.l4 = 1; - pkt_hdr->l4_offset = pkt_hdr->l3_offset + offset; - break; - case ODPH_IPPROTO_TCP: - pkt_hdr->input_flags.tcp = 1; - pkt_hdr->input_flags.l4 = 1; - pkt_hdr->l4_offset = pkt_hdr->l3_offset + offset; - break; - case ODPH_IPPROTO_SCTP: - pkt_hdr->input_flags.sctp = 1; - pkt_hdr->input_flags.l4 = 1; - pkt_hdr->l4_offset = pkt_hdr->l3_offset + offset; - break; - case ODPH_IPPROTO_ICMP: - pkt_hdr->input_flags.icmp = 1; - pkt_hdr->input_flags.l4 = 1; - pkt_hdr->l4_offset = pkt_hdr->l3_offset + offset; - break; - default: - /* 0 or unhandled IP protocols, don't set L4 flag+offset */ - if (pkt_hdr->input_flags.ipv6) { - /* IPv6 next_hdr is not L4, mark as IP-option instead */ - pkt_hdr->input_flags.ipopt = 1; +odp_packet_t odp_packet_copy(odp_packet_t pkt, odp_buffer_pool_t pool) +{ + odp_packet_hdr_t *srchdr = odp_packet_hdr(pkt); + uint32_t pktlen = srchdr->frame_len; + uint32_t meta_offset = ODP_FIELD_SIZEOF(odp_packet_hdr_t, buf_hdr); + odp_packet_t newpkt = odp_packet_alloc(pool, pktlen); + + if (newpkt != ODP_PACKET_INVALID) { + odp_packet_hdr_t *newhdr = odp_packet_hdr(newpkt); + uint8_t *newstart, *srcstart; + + /* Must copy meta data first, followed by packet data */ + newstart = (uint8_t *)newhdr + meta_offset; + srcstart = (uint8_t *)srchdr + meta_offset; + + memcpy(newstart, srcstart, + sizeof(odp_packet_hdr_t) - meta_offset); + + if (_odp_packet_copy_to_packet(pkt, 0, + newpkt, 0, pktlen) != 0) { + odp_packet_free(newpkt); + newpkt = ODP_PACKET_INVALID; } - break; } + + return newpkt; } -static inline uint8_t parse_ipv4(odp_packet_hdr_t *pkt_hdr, - odph_ipv4hdr_t *ipv4, size_t *offset_out) +int odp_packet_copydata_out(odp_packet_t pkt, uint32_t offset, + uint32_t len, void *dst) { - uint8_t ihl; - uint16_t frag_offset; - - ihl = ODPH_IPV4HDR_IHL(ipv4->ver_ihl); - if (odp_unlikely(ihl < ODPH_IPV4HDR_IHL_MIN)) { - pkt_hdr->error_flags.ip_err = 1; - return 0; - } - - if (odp_unlikely(ihl > ODPH_IPV4HDR_IHL_MIN)) { - pkt_hdr->input_flags.ipopt = 1; - return 0; - } + void *mapaddr; + uint32_t seglen = 0; /* GCC */ + uint32_t cpylen; + uint8_t *dstaddr = (uint8_t *)dst; + odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt); - /* A packet is a fragment if: - * "more fragments" flag is set (all fragments except the last) - * OR - * "fragment offset" field is nonzero (all fragments except the first) - */ - frag_offset = odp_be_to_cpu_16(ipv4->frag_offset); - if (odp_unlikely(ODPH_IPV4HDR_IS_FRAGMENT(frag_offset))) { - pkt_hdr->input_flags.ipfrag = 1; - return 0; - } + if (offset + len > pkt_hdr->frame_len) + return -1; - if (ipv4->proto == ODPH_IPPROTO_ESP || - ipv4->proto == ODPH_IPPROTO_AH) { - pkt_hdr->input_flags.ipsec = 1; - return 0; + while (len > 0) { + mapaddr = packet_map(pkt_hdr, offset, &seglen); + cpylen = len > seglen ? seglen : len; + memcpy(dstaddr, mapaddr, cpylen); + offset += cpylen; + dstaddr += cpylen; + len -= cpylen; } - /* Set pkt_hdr->input_flags.ipopt when checking L4 hdrs after return */ - - *offset_out = sizeof(uint32_t) * ihl; - return ipv4->proto; + return 0; } -static inline uint8_t parse_ipv6(odp_packet_hdr_t *pkt_hdr, - odph_ipv6hdr_t *ipv6, size_t *offset_out) +int odp_packet_copydata_in(odp_packet_t pkt, uint32_t offset, + uint32_t len, const void *src) { - if (ipv6->next_hdr == ODPH_IPPROTO_ESP || - ipv6->next_hdr == ODPH_IPPROTO_AH) { - pkt_hdr->input_flags.ipopt = 1; - pkt_hdr->input_flags.ipsec = 1; - return 0; - } + void *mapaddr; + uint32_t seglen = 0; /* GCC */ + uint32_t cpylen; + const uint8_t *srcaddr = (const uint8_t *)src; + odp_packet_hdr_t *pkt_hdr = odp_packet_hdr(pkt); - if (odp_unlikely(ipv6->next_hdr == ODPH_IPPROTO_FRAG)) { - pkt_hdr->input_flags.ipopt = 1; - pkt_hdr->input_flags.ipfrag = 1; - return 0; + if (offset + len > pkt_hdr->frame_len) + return -1; + + while (len > 0) { + mapaddr = packet_map(pkt_hdr, offset, &seglen); + cpylen = len > seglen ? seglen : len; + memcpy(mapaddr, srcaddr, cpylen); + offset += cpylen; + srcaddr += cpylen; + len -= cpylen; } - /* Don't step through more extensions */ - *offset_out = ODPH_IPV6HDR_LEN; - return ipv6->next_hdr; + return 0; } +/* + * + * Debugging + * ******************************************************** + * + */ + void odp_packet_print(odp_packet_t pkt) { int max_len = 512; char str[max_len]; - uint8_t *p; int len = 0; int n = max_len-1; odp_packet_hdr_t *hdr = odp_packet_hdr(pkt); @@ -353,77 +495,358 @@ void odp_packet_print(odp_packet_t pkt) len += snprintf(&str[len], n-len, " output_flags 0x%x\n", hdr->output_flags.all); len += snprintf(&str[len], n-len, - " frame_offset %u\n", hdr->frame_offset); - len += snprintf(&str[len], n-len, " l2_offset %u\n", hdr->l2_offset); len += snprintf(&str[len], n-len, " l3_offset %u\n", hdr->l3_offset); len += snprintf(&str[len], n-len, " l4_offset %u\n", hdr->l4_offset); len += snprintf(&str[len], n-len, - " frame_len %u\n", hdr->buf_hdr.mb.pkt.pkt_len); + " frame_len %u\n", hdr->frame_len); len += snprintf(&str[len], n-len, " input %u\n", hdr->input); str[len] = '\0'; - printf("\n%s\n", str); - rte_pktmbuf_dump(stdout, &hdr->buf_hdr.mb, 32); + ODP_PRINT("\n%s\n", str); +} - p = odp_packet_start(pkt); - printf("00000000: %02X %02X %02X %02X %02X %02X %02X %02X\n", - p[0], p[1], p[2], p[3], p[4], p[5], p[6], p[7]); - printf("00000008: %02X %02X %02X %02X %02X %02X %02X %02X\n", - p[8], p[9], p[10], p[11], p[12], p[13], p[14], p[15]); +int odp_packet_is_valid(odp_packet_t pkt) +{ + odp_buffer_hdr_t *buf = validate_buf((odp_buffer_t)pkt); + return (buf != NULL && buf->type == ODP_BUFFER_TYPE_PACKET); } -/* For now we can only copy between packets of the same segment size - * We should probably refine this API, maybe introduce a clone API */ -int odp_packet_copy(odp_packet_t pkt_dst, odp_packet_t pkt_src) +/* + * + * Internal Use Routines + * ******************************************************** + * + */ + +int _odp_packet_copy_to_packet(odp_packet_t srcpkt, uint32_t srcoffset, + odp_packet_t dstpkt, uint32_t dstoffset, + uint32_t len) { - struct rte_mbuf *mb_dst, *mb_src; - uint8_t nb_segs, i; + odp_packet_hdr_t *srchdr = odp_packet_hdr(srcpkt); + odp_packet_hdr_t *dsthdr = odp_packet_hdr(dstpkt); + void *srcmap; + void *dstmap; + uint32_t cpylen, minseg; + uint32_t srcseglen = 0; /* GCC */ + uint32_t dstseglen = 0; /* GCC */ + + if (srcoffset + len > srchdr->frame_len || + dstoffset + len > dsthdr->frame_len) + return -1; - ODP_ASSERT(odp_buffer_type(pkt_dst) == ODP_BUFFER_TYPE_PACKET && - odp_buffer_type(pkt_src) == ODP_BUFFER_TYPE_PACKET, - "dst_pkt or src_pkt not of type ODP_BUFFER_TYPE_PACKET"); + while (len > 0) { + srcmap = packet_map(srchdr, srcoffset, &srcseglen); + dstmap = packet_map(dsthdr, dstoffset, &dstseglen); - if (pkt_dst == ODP_PACKET_INVALID || pkt_src == ODP_PACKET_INVALID) - return -1; + minseg = dstseglen > srcseglen ? srcseglen : dstseglen; + cpylen = len > minseg ? minseg : len; + memcpy(dstmap, srcmap, cpylen); - mb_dst = &(odp_packet_hdr(pkt_dst)->buf_hdr.mb); - mb_src = &(odp_packet_hdr(pkt_src)->buf_hdr.mb); + srcoffset += cpylen; + dstoffset += cpylen; + len -= cpylen; + } - if (mb_dst->pkt.nb_segs != mb_src->pkt.nb_segs) { - ODP_ERR("Different nb_segs in pkt_dst and pkt_src"); - return -1; + return 0; +} + +odp_packet_t _odp_packet_alloc(odp_buffer_pool_t pool_hdl) +{ + pool_entry_t *pool = odp_pool_to_entry(pool_hdl); + + if (pool->s.params.buf_type != ODP_BUFFER_TYPE_PACKET) + return ODP_PACKET_INVALID; + + return (odp_packet_t)buffer_alloc(pool_hdl, + pool->s.params.buf_size); +} + +/** + * Parser helper function for IPv4 + */ +static inline uint8_t parse_ipv4(odp_packet_hdr_t *pkt_hdr, + uint8_t **parseptr, uint32_t *offset) +{ + odph_ipv4hdr_t *ipv4 = (odph_ipv4hdr_t *)*parseptr; + uint8_t ver = ODPH_IPV4HDR_VER(ipv4->ver_ihl); + uint8_t ihl = ODPH_IPV4HDR_IHL(ipv4->ver_ihl); + uint16_t frag_offset; + + pkt_hdr->l3_len = odp_be_to_cpu_16(ipv4->tot_len); + + if (odp_unlikely(ihl < ODPH_IPV4HDR_IHL_MIN) || + odp_unlikely(ver != 4) || + (pkt_hdr->l3_len > pkt_hdr->frame_len - *offset)) { + pkt_hdr->error_flags.ip_err = 1; + return 0; } - nb_segs = mb_src->pkt.nb_segs; + *offset += ihl * 4; + *parseptr += ihl * 4; - if (mb_dst->buf_len < mb_src->buf_len) { - ODP_ERR("dst_pkt smaller than src_pkt"); - return -1; + if (odp_unlikely(ihl > ODPH_IPV4HDR_IHL_MIN)) + pkt_hdr->input_flags.ipopt = 1; + + /* A packet is a fragment if: + * "more fragments" flag is set (all fragments except the last) + * OR + * "fragment offset" field is nonzero (all fragments except the first) + */ + frag_offset = odp_be_to_cpu_16(ipv4->frag_offset); + if (odp_unlikely(ODPH_IPV4HDR_IS_FRAGMENT(frag_offset))) + pkt_hdr->input_flags.ipfrag = 1; + + return ipv4->proto; +} + +/** + * Parser helper function for IPv6 + */ +static inline uint8_t parse_ipv6(odp_packet_hdr_t *pkt_hdr, + uint8_t **parseptr, uint32_t *offset) +{ + odph_ipv6hdr_t *ipv6 = (odph_ipv6hdr_t *)*parseptr; + odph_ipv6hdr_ext_t *ipv6ext; + + pkt_hdr->l3_len = odp_be_to_cpu_16(ipv6->payload_len); + + /* Basic sanity checks on IPv6 header */ + if ((ipv6->ver_tc_flow >> 28) != 6 || + pkt_hdr->l3_len > pkt_hdr->frame_len - *offset) { + pkt_hdr->error_flags.ip_err = 1; + return 0; } - for (i = 0; i < nb_segs; i++) { - if (mb_src == NULL || mb_dst == NULL) { - ODP_ERR("Corrupted packets"); - return -1; + /* Skip past IPv6 header */ + *offset += sizeof(odph_ipv6hdr_t); + *parseptr += sizeof(odph_ipv6hdr_t); + + + /* Skip past any IPv6 extension headers */ + if (ipv6->next_hdr == ODPH_IPPROTO_HOPOPTS || + ipv6->next_hdr == ODPH_IPPROTO_ROUTE) { + pkt_hdr->input_flags.ipopt = 1; + + do { + ipv6ext = (odph_ipv6hdr_ext_t *)*parseptr; + uint16_t extlen = 8 + ipv6ext->ext_len * 8; + + *offset += extlen; + *parseptr += extlen; + } while ((ipv6ext->next_hdr == ODPH_IPPROTO_HOPOPTS || + ipv6ext->next_hdr == ODPH_IPPROTO_ROUTE) && + *offset < pkt_hdr->frame_len); + + if (*offset >= pkt_hdr->l3_offset + ipv6->payload_len) { + pkt_hdr->error_flags.ip_err = 1; + return 0; } - memcpy(mb_dst->buf_addr, mb_src->buf_addr, mb_src->buf_len); - mb_dst = mb_dst->pkt.next; - mb_src = mb_src->pkt.next; + + if (ipv6ext->next_hdr == ODPH_IPPROTO_FRAG) + pkt_hdr->input_flags.ipfrag = 1; + + return ipv6ext->next_hdr; } - return 0; + + if (odp_unlikely(ipv6->next_hdr == ODPH_IPPROTO_FRAG)) { + pkt_hdr->input_flags.ipopt = 1; + pkt_hdr->input_flags.ipfrag = 1; + } + + return ipv6->next_hdr; +} + +/** + * Parser helper function for TCP + */ +static inline void parse_tcp(odp_packet_hdr_t *pkt_hdr, + uint8_t **parseptr, uint32_t *offset) +{ + odph_tcphdr_t *tcp = (odph_tcphdr_t *)*parseptr; + + if (tcp->hl < sizeof(odph_tcphdr_t)/sizeof(uint32_t)) + pkt_hdr->error_flags.tcp_err = 1; + else if ((uint32_t)tcp->hl * 4 > sizeof(odph_tcphdr_t)) + pkt_hdr->input_flags.tcpopt = 1; + + pkt_hdr->l4_len = pkt_hdr->l3_len + + pkt_hdr->l3_offset - pkt_hdr->l4_offset; + + *offset += (uint32_t)tcp->hl * 4; + *parseptr += (uint32_t)tcp->hl * 4; } -void odp_packet_set_ctx(odp_packet_t pkt, const void *ctx) +/** + * Parser helper function for UDP + */ +static inline void parse_udp(odp_packet_hdr_t *pkt_hdr, + uint8_t **parseptr, uint32_t *offset) { - odp_packet_hdr(pkt)->user_ctx = (intptr_t)ctx; + odph_udphdr_t *udp = (odph_udphdr_t *)*parseptr; + uint32_t udplen = odp_be_to_cpu_16(udp->length); + + if (udplen < sizeof(odph_udphdr_t) || + udplen > (pkt_hdr->l3_len + + pkt_hdr->l3_offset - pkt_hdr->l4_offset)) { + pkt_hdr->error_flags.udp_err = 1; + } + + pkt_hdr->l4_len = udplen; + + *offset += sizeof(odph_udphdr_t); + *parseptr += sizeof(odph_udphdr_t); } -void *odp_packet_get_ctx(odp_packet_t pkt) +/** + * Simple packet parser + */ + +int _odp_packet_parse(odp_packet_t pkt) { - return (void *)(intptr_t)odp_packet_hdr(pkt)->user_ctx; + odp_packet_hdr_t *const pkt_hdr = odp_packet_hdr(pkt); + odph_ethhdr_t *eth; + odph_vlanhdr_t *vlan; + uint16_t ethtype; + uint8_t *parseptr; + uint32_t offset, seglen; + uint8_t ip_proto = 0; + + /* Reset parser metadata for new parse */ + pkt_hdr->error_flags.all = 0; + pkt_hdr->input_flags.all = 0; + pkt_hdr->output_flags.all = 0; + pkt_hdr->l2_offset = 0; + pkt_hdr->l3_offset = 0; + pkt_hdr->l4_offset = 0; + pkt_hdr->payload_offset = 0; + pkt_hdr->vlan_s_tag = 0; + pkt_hdr->vlan_c_tag = 0; + pkt_hdr->l3_protocol = 0; + pkt_hdr->l4_protocol = 0; + + /* We only support Ethernet for now */ + pkt_hdr->input_flags.eth = 1; + + /* Detect jumbo frames */ + if (pkt_hdr->frame_len > ODPH_ETH_LEN_MAX) + pkt_hdr->input_flags.jumbo = 1; + + /* Assume valid L2 header, no CRC/FCS check in SW */ + pkt_hdr->input_flags.l2 = 1; + + eth = (odph_ethhdr_t *)packet_map(pkt_hdr, 0, &seglen); + offset = sizeof(odph_ethhdr_t); + parseptr = (uint8_t *)ð->type; + ethtype = odp_be_to_cpu_16(*((uint16_t *)(void *)parseptr)); + + /* Parse the VLAN header(s), if present */ + if (ethtype == ODPH_ETHTYPE_VLAN_OUTER) { + pkt_hdr->input_flags.vlan_qinq = 1; + pkt_hdr->input_flags.vlan = 1; + vlan = (odph_vlanhdr_t *)(void *)parseptr; + pkt_hdr->vlan_s_tag = ((ethtype << 16) | + odp_be_to_cpu_16(vlan->tci)); + offset += sizeof(odph_vlanhdr_t); + parseptr += sizeof(odph_vlanhdr_t); + ethtype = odp_be_to_cpu_16(*((uint16_t *)(void *)parseptr)); + } + + if (ethtype == ODPH_ETHTYPE_VLAN) { + pkt_hdr->input_flags.vlan = 1; + vlan = (odph_vlanhdr_t *)(void *)parseptr; + pkt_hdr->vlan_c_tag = ((ethtype << 16) | + odp_be_to_cpu_16(vlan->tci)); + offset += sizeof(odph_vlanhdr_t); + parseptr += sizeof(odph_vlanhdr_t); + ethtype = odp_be_to_cpu_16(*((uint16_t *)(void *)parseptr)); + } + + /* Check for SNAP vs. DIX */ + if (ethtype < ODPH_ETH_LEN_MAX) { + pkt_hdr->input_flags.snap = 1; + if (ethtype > pkt_hdr->frame_len - offset) { + pkt_hdr->error_flags.snap_len = 1; + goto parse_exit; + } + offset += 8; + parseptr += 8; + ethtype = odp_be_to_cpu_16(*((uint16_t *)(void *)parseptr)); + } + + /* Consume Ethertype for Layer 3 parse */ + parseptr += 2; + + /* Set l3_offset+flag only for known ethtypes */ + pkt_hdr->input_flags.l3 = 1; + pkt_hdr->l3_offset = offset; + pkt_hdr->l3_protocol = ethtype; + + /* Parse Layer 3 headers */ + switch (ethtype) { + case ODPH_ETHTYPE_IPV4: + pkt_hdr->input_flags.ipv4 = 1; + ip_proto = parse_ipv4(pkt_hdr, &parseptr, &offset); + break; + + case ODPH_ETHTYPE_IPV6: + pkt_hdr->input_flags.ipv6 = 1; + ip_proto = parse_ipv6(pkt_hdr, &parseptr, &offset); + break; + + case ODPH_ETHTYPE_ARP: + pkt_hdr->input_flags.arp = 1; + ip_proto = 255; /* Reserved invalid by IANA */ + break; + + default: + pkt_hdr->input_flags.l3 = 0; + ip_proto = 255; /* Reserved invalid by IANA */ + } + + /* Set l4_offset+flag only for known ip_proto */ + pkt_hdr->input_flags.l4 = 1; + pkt_hdr->l4_offset = offset; + pkt_hdr->l4_protocol = ip_proto; + + /* Parse Layer 4 headers */ + switch (ip_proto) { + case ODPH_IPPROTO_ICMP: + pkt_hdr->input_flags.icmp = 1; + break; + + case ODPH_IPPROTO_TCP: + pkt_hdr->input_flags.tcp = 1; + parse_tcp(pkt_hdr, &parseptr, &offset); + break; + + case ODPH_IPPROTO_UDP: + pkt_hdr->input_flags.udp = 1; + parse_udp(pkt_hdr, &parseptr, &offset); + break; + + case ODPH_IPPROTO_AH: + case ODPH_IPPROTO_ESP: + pkt_hdr->input_flags.ipsec = 1; + break; + + default: + pkt_hdr->input_flags.l4 = 0; + break; + } + + /* + * Anything beyond what we parse here is considered payload. + * Note: Payload is really only relevant for TCP and UDP. For + * all other protocols, the payload offset will point to the + * final header (ARP, ICMP, AH, ESP, or IP Fragment). + */ + pkt_hdr->payload_offset = offset; + +parse_exit: + return pkt_hdr->error_flags.all != 0; }