From patchwork Thu Apr 27 11:51:50 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dmitry Eremin-Solenikov X-Patchwork-Id: 98284 Delivered-To: patch@linaro.org Received: by 10.140.109.52 with SMTP id k49csp55844qgf; Thu, 27 Apr 2017 04:54:52 -0700 (PDT) X-Received: by 10.55.0.65 with SMTP id 62mr4186921qka.158.1493294092297; Thu, 27 Apr 2017 04:54:52 -0700 (PDT) Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id 44si2712331qtx.185.2017.04.27.04.54.52; Thu, 27 Apr 2017 04:54:52 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 0265361627; Thu, 27 Apr 2017 11:54:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, RCVD_IN_SORBS_SPAM, URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 29BF561FD1; Thu, 27 Apr 2017 11:52:25 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id EB42A60D2C; Thu, 27 Apr 2017 11:52:07 +0000 (UTC) Received: from mail-lf0-f54.google.com (mail-lf0-f54.google.com [209.85.215.54]) by lists.linaro.org (Postfix) with ESMTPS id A209560820 for ; Thu, 27 Apr 2017 11:51:58 +0000 (UTC) Received: by mail-lf0-f54.google.com with SMTP id 88so15982933lfr.0 for ; Thu, 27 Apr 2017 04:51:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=bk3o8W9cwUMaJnx954DbUuFh35fet9IkS2Y14czIOL8=; b=l4vxsmKL8+R8PWFCqqFaEU33PiwAsxNBwRkin7sV/9kyRZ9uX2LULjymwKPvESi5Ut GAVXcsiK84AM5DToihKH5ZaIZ4HB0JDRgEhQgjSqVJoIkvqzKhxM+wKfaAdo8agAYFei cRc8SEiIJggL19HWMIjGeeDTsugQi5o5aA5dOPTeb48IbLPRURErbDSOkjglwlqAmYFz p3LshbhzezSZx7LSEi7R56Jtw2G41dHnec2cd8K1xL4Ww+8AuB/G7NXftb1ZVT7zaUUp pC8mZUfEPLtfCgQmnUoQ6J9/Ru9DXQLnHe9iUDi9jQPx0bv+Ji8+A6x8ZguLuSGn243Z Mogg== X-Gm-Message-State: AN3rC/4rOGgcyqfHmAvtLAno9pLrLZriw7HUrNp2vt6E/Fn5Y36RSZGu k8w8ocZRWpkXFcRro7P5FNpQ X-Received: by 10.25.26.193 with SMTP id a184mr1843488lfa.90.1493293916756; Thu, 27 Apr 2017 04:51:56 -0700 (PDT) Received: from forlindon.lumag.auriga.ru ([94.25.229.39]) by smtp.gmail.com with ESMTPSA id 30sm425282lju.0.2017.04.27.04.51.55 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 27 Apr 2017 04:51:56 -0700 (PDT) From: Dmitry Eremin-Solenikov To: lng-odp@lists.linaro.org Date: Thu, 27 Apr 2017 14:51:50 +0300 Message-Id: <20170427115150.19452-4-dmitry.ereminsolenikov@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20170427115150.19452-1-dmitry.ereminsolenikov@linaro.org> References: <20170427115150.19452-1-dmitry.ereminsolenikov@linaro.org> Subject: [lng-odp] [[RFCv2] 4/4] linux-gen: ipsec: draft IPsec implementation X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" For now it's only a preview with the following limitation: - No inline processing support - No SA lookups - Only IPv4 support - No zeroing of mutable IPv4 options for AH ICV calculation - No replay protection - No ESN support Signed-off-by: Dmitry Eremin-Solenikov --- .../include/odp/api/plat/event_types.h | 3 +- platform/linux-generic/include/odp_internal.h | 4 + .../linux-generic/include/odp_ipsec_internal.h | 107 ++ platform/linux-generic/include/protocols/ip.h | 52 + platform/linux-generic/odp_event.c | 8 + platform/linux-generic/odp_init.c | 13 + platform/linux-generic/odp_ipsec.c | 1450 +++++++++++++++++++- 7 files changed, 1602 insertions(+), 35 deletions(-) create mode 100644 platform/linux-generic/include/odp_ipsec_internal.h -- 2.11.0 diff --git a/platform/linux-generic/include/odp/api/plat/event_types.h b/platform/linux-generic/include/odp/api/plat/event_types.h index 0f517834..cb3a1f89 100644 --- a/platform/linux-generic/include/odp/api/plat/event_types.h +++ b/platform/linux-generic/include/odp/api/plat/event_types.h @@ -39,7 +39,8 @@ typedef enum odp_event_type_t { ODP_EVENT_PACKET = 2, ODP_EVENT_TIMEOUT = 3, ODP_EVENT_CRYPTO_COMPL = 4, - ODP_EVENT_IPSEC_RESULT = 5 + ODP_EVENT_IPSEC_RESULT = 5, + ODP_EVENT_IPSEC_STATUS = 6 } odp_event_type_t; /** diff --git a/platform/linux-generic/include/odp_internal.h b/platform/linux-generic/include/odp_internal.h index acfc3012..5bbd0f13 100644 --- a/platform/linux-generic/include/odp_internal.h +++ b/platform/linux-generic/include/odp_internal.h @@ -70,6 +70,7 @@ enum init_stage { CLASSIFICATION_INIT, TRAFFIC_MNGR_INIT, NAME_TABLE_INIT, + IPSEC_INIT, MODULES_INIT, ALL_INIT /* All init stages completed */ }; @@ -129,6 +130,9 @@ int _odp_ishm_init_local(void); int _odp_ishm_term_global(void); int _odp_ishm_term_local(void); +int odp_ipsec_init_global(void); +int odp_ipsec_term_global(void); + int _odp_modules_init_global(void); int cpuinfo_parser(FILE *file, system_info_t *sysinfo); diff --git a/platform/linux-generic/include/odp_ipsec_internal.h b/platform/linux-generic/include/odp_ipsec_internal.h new file mode 100644 index 00000000..21026569 --- /dev/null +++ b/platform/linux-generic/include/odp_ipsec_internal.h @@ -0,0 +1,107 @@ +/* Copyright (c) 2017, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +/** + * @file + * + * ODP internal IPsec routines + */ + +#ifndef ODP_IPSEC_INTERNAL_H_ +#define ODP_IPSEC_INTERNAL_H_ + +#ifdef __cplusplus +extern "C" { +#endif + +#include +#include + +/** @ingroup odp_ipsec + * @{ + */ + +typedef ODP_HANDLE_T(odp_ipsec_result_t); + +#define ODP_IPSEC_RESULT_INVALID \ + _odp_cast_scalar(odp_ipsec_result_t, 0xffffffff) + +typedef ODP_HANDLE_T(odp_ipsec_status_event_t); + +#define ODP_IPSEC_STATUS_EVENT_INVALID \ + _odp_cast_scalar(odp_ipsec_status_event_t, 0xffffffff) + +/** + * Get ipsec_result handle from event + * + * Converts an ODP_EVENT_IPSEC_RESULT type event to an IPsec result event. + * + * @param ev Event handle + * + * @return IPsec result handle + * + * @see odp_event_type() + */ +odp_ipsec_result_t odp_ipsec_result_from_event(odp_event_t ev); + +/** + * Convert IPsec result event handle to event + * + * @param res IPsec result handle + * + * @return Event handle + */ +odp_event_t odp_ipsec_result_to_event(odp_ipsec_result_t res); + +/** + * Free IPsec result event + * + * Frees the ipsec_result into the ipsec_result pool it was allocated from. + * + * @param res IPsec result handle + */ +void odp_ipsec_result_free(odp_ipsec_result_t res); + +/** + * Get ipsec_status handle from event + * + * Converts an ODP_EVENT_IPSEC_STATUS type event to an IPsec status event. + * + * @param ev Event handle + * + * @return IPsec status handle + * + * @see odp_event_type() + */ +odp_ipsec_status_event_t odp_ipsec_status_from_event(odp_event_t ev); + +/** + * Convert IPsec status event handle to event + * + * @param res IPsec status handle + * + * @return Event handle + */ +odp_event_t odp_ipsec_status_to_event(odp_ipsec_status_event_t res); + +/** + * Free IPsec status event + * + * Frees the ipsec_status into the ipsec_status pool it was allocated from. + * + * @param res IPsec status handle + */ +void odp_ipsec_status_free(odp_ipsec_status_event_t res); + +/** + * @} + */ + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/platform/linux-generic/include/protocols/ip.h b/platform/linux-generic/include/protocols/ip.h index 2b34a753..9f3e1616 100644 --- a/platform/linux-generic/include/protocols/ip.h +++ b/platform/linux-generic/include/protocols/ip.h @@ -89,6 +89,58 @@ typedef struct ODP_PACKED { ODP_STATIC_ASSERT(sizeof(_odp_ipv4hdr_t) == _ODP_IPV4HDR_LEN, "_ODP_IPV4HDR_T__SIZE_ERROR"); +/** + * Checksum + * + * @param buffer calculate chksum for buffer + * @param len buffer length + * + * @return checksum value in host cpu order + */ +static inline odp_u16sum_t _odp_chksum(void *buffer, int len) +{ + uint16_t *buf = (uint16_t *)buffer; + uint32_t sum = 0; + uint16_t result; + + for (sum = 0; len > 1; len -= 2) + sum += *buf++; + + if (len == 1) + sum += *(unsigned char *)buf; + + sum = (sum >> 16) + (sum & 0xFFFF); + sum += (sum >> 16); + result = ~sum; + + return (__odp_force odp_u16sum_t) result; +} + +/** + * Calculate and fill in IPv4 checksum + * + * @note when using this api to populate data destined for the wire + * odp_cpu_to_be_16() can be used to remove sparse warnings + * + * @param pkt ODP packet + * + * @return IPv4 checksum in host cpu order, or 0 on failure + */ +static inline odp_u16sum_t _odp_ipv4_csum_update(odp_packet_t pkt) +{ + uint16_t *w; + _odp_ipv4hdr_t *ip; + int nleft = sizeof(_odp_ipv4hdr_t); + + ip = (_odp_ipv4hdr_t *)odp_packet_l3_ptr(pkt, NULL); + if (ip == NULL) + return 0; + + w = (uint16_t *)(void *)ip; + ip->chksum = _odp_chksum(w, nleft); + return ip->chksum; +} + /** IPv6 version */ #define _ODP_IPV6 6 diff --git a/platform/linux-generic/odp_event.c b/platform/linux-generic/odp_event.c index d71f4464..16bd35cc 100644 --- a/platform/linux-generic/odp_event.c +++ b/platform/linux-generic/odp_event.c @@ -7,10 +7,12 @@ #include #include #include +#include #include #include #include #include +#include #include #include @@ -34,6 +36,12 @@ void odp_event_free(odp_event_t event) case ODP_EVENT_CRYPTO_COMPL: odp_crypto_compl_free(odp_crypto_compl_from_event(event)); break; + case ODP_EVENT_IPSEC_RESULT: + odp_ipsec_result_free(odp_ipsec_result_from_event(event)); + break; + case ODP_EVENT_IPSEC_STATUS: + odp_ipsec_status_free(odp_ipsec_status_from_event(event)); + break; default: ODP_ABORT("Invalid event type: %d\n", odp_event_type(event)); } diff --git a/platform/linux-generic/odp_init.c b/platform/linux-generic/odp_init.c index 685e02fa..bebcc62e 100644 --- a/platform/linux-generic/odp_init.c +++ b/platform/linux-generic/odp_init.c @@ -266,6 +266,12 @@ int odp_init_global(odp_instance_t *instance, } stage = NAME_TABLE_INIT; + if (odp_ipsec_init_global()) { + ODP_ERR("ODP IPsec init failed.\n"); + goto init_failed; + } + stage = IPSEC_INIT; + if (_odp_modules_init_global()) { ODP_ERR("ODP modules init failed\n"); goto init_failed; @@ -296,6 +302,13 @@ int _odp_term_global(enum init_stage stage) switch (stage) { case ALL_INIT: case MODULES_INIT: + case IPSEC_INIT: + if (odp_ipsec_term_global()) { + ODP_ERR("ODP IPsec term failed.\n"); + rc = -1; + } + /* Fall through */ + case NAME_TABLE_INIT: if (_odp_int_name_tbl_term_global()) { ODP_ERR("Name table term failed.\n"); diff --git a/platform/linux-generic/odp_ipsec.c b/platform/linux-generic/odp_ipsec.c index 10918dfb..f15cd8d0 100644 --- a/platform/linux-generic/odp_ipsec.c +++ b/platform/linux-generic/odp_ipsec.c @@ -4,105 +4,1442 @@ * SPDX-License-Identifier: BSD-3-Clause */ +#include #include +#include +#include +#include + +#include +#include +#include +#include +#include + +#include + +#include +#include #include +#include + +#define ODP_CONFIG_IPSEC_SAS 8 + +#define MAX_IV_LEN 32 /**< Maximum IV length in bytes */ + +#define IPSEC_USERS_DISABLING 0x80000000 + +typedef struct ipsec_sa_t { + odp_ticketlock_t lock ODP_ALIGNED_CACHE; + unsigned is_reserved : 1; + odp_atomic_u32_t users; + odp_ipsec_sa_t ipsec_sa_hdl; + uint32_t ipsec_sa_idx; + + odp_crypto_session_t session; + odp_bool_t in_place; + void *context; + odp_queue_t queue; + + odp_ipsec_mode_t mode; + odp_u32be_t tun_src_ip; + odp_u32be_t tun_dst_ip; + uint8_t tun_ttl; + /* 32-bit from which low 16 are used */ + odp_atomic_u32_t tun_hdr_id; + + odp_ipsec_protocol_t proto; + uint32_t icv_len; + uint32_t esp_iv_len; + uint32_t esp_block_len; + uint32_t spi; + odp_atomic_u32_t seq; + uint8_t iv[MAX_IV_LEN]; /**< ESP IV storage */ +} ipsec_sa_t; + +typedef struct ipsec_sa_table_t { + ipsec_sa_t ipsec_sa[ODP_CONFIG_IPSEC_SAS]; + odp_shm_t shm; +} ipsec_sa_table_t; + +static ipsec_sa_table_t *ipsec_sa_tbl; + +typedef struct ipsec_ctx_s ipsec_ctx_t; + +typedef void (*ipsec_postprocess_t)(ipsec_ctx_t *ctx); + +/** + * Per packet IPsec processing context + */ +struct ipsec_ctx_s { + odp_buffer_t buffer; /**< Buffer for context */ + ipsec_ctx_t *next; /**< Next context in event */ + + uint8_t ip_tos; /**< Saved IP TOS value */ + uint16_t ip_frag_offset; /**< Saved IP flags value */ + uint8_t ip_ttl; /**< Saved IP TTL value */ + unsigned hdr_len; /**< Length of IPsec headers */ + unsigned trl_len; /**< Length of IPsec trailers */ + uint16_t ipsec_offset; /**< Offset of IPsec header from + buffer start */ + + ipsec_postprocess_t postprocess; + odp_ipsec_sa_t sa; + odp_crypto_op_result_t crypto; + odp_ipsec_op_status_t status; + odp_packet_t pkt; + + uint32_t src_ip; /**< SA source IP address */ + uint32_t dst_ip; /**< SA dest IP address */ + uint8_t iv[MAX_IV_LEN]; /**< ESP IV storage */ +}; + +typedef struct { + /* common buffer header */ + odp_buffer_hdr_t buf_hdr; + ipsec_ctx_t *ctx; +} ipsec_result_hdr_t; + +typedef struct { + /* common buffer header */ + odp_buffer_hdr_t buf_hdr; + + odp_ipsec_status_t status; +} ipsec_status_hdr_t; + +static +int odp_ipsec_status_send(odp_ipsec_sa_t sa, + odp_ipsec_status_id_t id, + int ret); + +#define IPSEC_CTX_POOL_BUF_COUNT 1024 + +static odp_pool_t ipsec_ctx_pool = ODP_POOL_INVALID; +static odp_pool_t ipsec_result_pool = ODP_POOL_INVALID; +static odp_pool_t ipsec_status_pool = ODP_POOL_INVALID; + +static inline ipsec_sa_t *ipsec_sa_entry(uint32_t ipsec_sa_idx) +{ + return &ipsec_sa_tbl->ipsec_sa[ipsec_sa_idx]; +} + +static inline ipsec_sa_t *ipsec_sa_entry_from_hdl(odp_ipsec_sa_t ipsec_sa_hdl) +{ + return ipsec_sa_entry(_odp_typeval(ipsec_sa_hdl)); +} + +static inline odp_ipsec_sa_t ipsec_sa_index_to_handle(uint32_t ipsec_sa_idx) +{ + return _odp_cast_scalar(odp_ipsec_sa_t, ipsec_sa_idx); +} + +int odp_ipsec_init_global(void) +{ + uint32_t i; + odp_shm_t shm; + odp_pool_param_t param; + + /* Create context buffer pool */ + param.buf.size = sizeof(ipsec_ctx_t); + param.buf.align = 0; + param.buf.num = IPSEC_CTX_POOL_BUF_COUNT; + param.type = ODP_POOL_BUFFER; + + ipsec_ctx_pool = odp_pool_create("ipsec_ctx_pool", ¶m); + if (ODP_POOL_INVALID == ipsec_ctx_pool) { + ODP_ERR("Error: context pool create failed.\n"); + return -1; + } + + param.buf.size = sizeof(ipsec_result_hdr_t); + param.buf.align = 0; + param.buf.num = IPSEC_CTX_POOL_BUF_COUNT; + param.type = ODP_POOL_BUFFER; + + ipsec_result_pool = odp_pool_create("ipsec_result_pool", ¶m); + if (ODP_POOL_INVALID == ipsec_result_pool) { + ODP_ERR("Error: result pool create failed.\n"); + (void)odp_pool_destroy(ipsec_ctx_pool); + return -1; + } + + param.buf.size = sizeof(ipsec_status_hdr_t); + param.buf.align = 0; + param.buf.num = IPSEC_CTX_POOL_BUF_COUNT; + param.type = ODP_POOL_BUFFER; + + ipsec_status_pool = odp_pool_create("ipsec_status_pool", ¶m); + if (ODP_POOL_INVALID == ipsec_status_pool) { + ODP_ERR("Error: status pool create failed.\n"); + (void)odp_pool_destroy(ipsec_ctx_pool); + return -1; + } + + shm = odp_shm_reserve("ipsec_sa_table", + sizeof(ipsec_sa_table_t), + ODP_CACHE_LINE_SIZE, 0); + + ipsec_sa_tbl = odp_shm_addr(shm); + if (ipsec_sa_tbl == NULL) { + (void)odp_pool_destroy(ipsec_result_pool); + (void)odp_pool_destroy(ipsec_ctx_pool); + return -1; + } + + memset(ipsec_sa_tbl, 0, sizeof(ipsec_sa_table_t)); + ipsec_sa_tbl->shm = shm; + + for (i = 0; i < ODP_CONFIG_IPSEC_SAS; i++) { + ipsec_sa_t *ipsec_sa = ipsec_sa_entry(i); + + odp_ticketlock_init(&ipsec_sa->lock); + ipsec_sa->ipsec_sa_hdl = ipsec_sa_index_to_handle(i); + ipsec_sa->ipsec_sa_idx = i; + odp_atomic_store_u32(&ipsec_sa->users, 0); + } + + return 0; +} + +int odp_ipsec_term_global(void) +{ + int i; + ipsec_sa_t *ipsec_sa; + int ret = 0; + int rc = 0; + + for (i = 0; i < ODP_CONFIG_IPSEC_SAS; i++) { + ipsec_sa = ipsec_sa_entry(i); + + odp_ticketlock_lock(&ipsec_sa->lock); + if (ipsec_sa->is_reserved) { + ODP_ERR("Not destroyed ipsec_sa: %u\n", ipsec_sa->ipsec_sa_idx); + rc = -1; + } + ipsec_sa->is_reserved = 1; + odp_ticketlock_unlock(&ipsec_sa->lock); + } + + ret = odp_shm_free(ipsec_sa_tbl->shm); + if (ret < 0) { + ODP_ERR("shm free failed"); + rc = -1; + } + + ret = odp_pool_destroy(ipsec_status_pool); + if (ret < 0) { + ODP_ERR("status pool destroy failed"); + rc = -1; + } + + ret = odp_pool_destroy(ipsec_result_pool); + if (ret < 0) { + ODP_ERR("result pool destroy failed"); + rc = -1; + } + + ret = odp_pool_destroy(ipsec_ctx_pool); + if (ret < 0) { + ODP_ERR("ctx pool destroy failed"); + rc = -1; + } + + return rc; +} + +static ipsec_sa_t *ipsec_sa_reserve(void) +{ + int i; + ipsec_sa_t *ipsec_sa; + + for (i = 0; i < ODP_CONFIG_IPSEC_SAS; i++) { + ipsec_sa = ipsec_sa_entry(i); + + odp_ticketlock_lock(&ipsec_sa->lock); + if (ipsec_sa->is_reserved == 0) { + ipsec_sa->is_reserved = 1; + odp_atomic_store_u32(&ipsec_sa->users, 0); + odp_ticketlock_unlock(&ipsec_sa->lock); + + return ipsec_sa; + } + odp_ticketlock_unlock(&ipsec_sa->lock); + } + + return NULL; +} + +static +int ipsec_sa_release(ipsec_sa_t *ipsec_sa) +{ + int rc = 0; + + odp_ticketlock_lock(&ipsec_sa->lock); + if (ipsec_sa->is_reserved) { + ODP_ERR("Releasing unallocated ipsec_sa: %u\n", ipsec_sa->ipsec_sa_idx); + rc = -1; + } else { + ipsec_sa->is_reserved = 0; + } + odp_ticketlock_unlock(&ipsec_sa->lock); + + return rc; +} int odp_ipsec_capability(odp_ipsec_capability_t *capa) { + int rc; + odp_crypto_capability_t crypto_capa; + memset(capa, 0, sizeof(odp_ipsec_capability_t)); + rc = odp_crypto_capability(&crypto_capa); + if (rc < 0) + return rc; + + capa->max_num_sa = ODP_CONFIG_IPSEC_SAS; + capa->op_mode_sync = ODP_SUPPORT_PREFERRED; + capa->op_mode_async = ODP_SUPPORT_YES; + capa->ciphers = crypto_capa.ciphers; + capa->auths = crypto_capa.auths; + return 0; } int odp_ipsec_cipher_capability(odp_cipher_alg_t cipher, odp_crypto_cipher_capability_t capa[], int num) { - (void)cipher; - (void)capa; - (void)num; - - return -1; + return odp_crypto_cipher_capability(cipher, capa, num); } int odp_ipsec_auth_capability(odp_auth_alg_t auth, odp_crypto_auth_capability_t capa[], int num) { - (void)auth; - (void)capa; - (void)num; - - return -1; + return odp_crypto_auth_capability(auth, capa, num); } void odp_ipsec_config_init(odp_ipsec_config_t *config) { memset(config, 0, sizeof(odp_ipsec_config_t)); + config->inbound_mode = ODP_IPSEC_OP_MODE_SYNC; + config->outbound_mode = ODP_IPSEC_OP_MODE_SYNC; + config->max_num_sa = ODP_CONFIG_IPSEC_SAS; + config->inbound.default_queue = ODP_QUEUE_INVALID; + config->inbound.lookup.min_spi = 0; + config->inbound.lookup.max_spi = UINT32_MAX; + config->outbound.default_queue = ODP_QUEUE_INVALID; } +static odp_ipsec_config_t ipsec_config; + int odp_ipsec_config(const odp_ipsec_config_t *config) { - (void)config; + /* FIXME: unsupported for now */ + if (ODP_IPSEC_OP_MODE_INLINE == config->outbound_mode) + return -1; - return -1; + /* FIXME: unsupported for now */ + if (ODP_IPSEC_OP_MODE_INLINE == config->inbound_mode) + return -1; + + ipsec_config = *config; + + return 0; } void odp_ipsec_sa_param_init(odp_ipsec_sa_param_t *param) { memset(param, 0, sizeof(odp_ipsec_sa_param_t)); + param->dest_queue = ODP_QUEUE_INVALID; } odp_ipsec_sa_t odp_ipsec_sa_create(const odp_ipsec_sa_param_t *param) { - (void)param; + ipsec_sa_t *ipsec_sa; + odp_crypto_session_param_t crypto_param; + odp_crypto_ses_create_err_t ses_create_rc; + + ipsec_sa = ipsec_sa_reserve(); + if (NULL == ipsec_sa) { + ODP_ERR("No more free SA\n"); + return ODP_IPSEC_SA_INVALID; + } + +#if 1 + ipsec_sa->in_place = false; +#else + ipsec_sa->in_place = true; +#endif + ipsec_sa->proto = param->proto; + ipsec_sa->spi = param->spi; + odp_atomic_init_u32(&ipsec_sa->seq, param->seq); + ipsec_sa->context = param->context; + ipsec_sa->queue = param->dest_queue; + ipsec_sa->mode = param->mode; + + if (ODP_IPSEC_MODE_TUNNEL == ipsec_sa->mode) { + if (param->tunnel.type != ODP_IPSEC_TUNNEL_IPV4) { + ipsec_sa_release(ipsec_sa); + + return ODP_IPSEC_SA_INVALID; + } + memcpy(&ipsec_sa->tun_src_ip, param->tunnel.ipv4.src_addr, sizeof(ipsec_sa->tun_src_ip)); + memcpy(&ipsec_sa->tun_dst_ip, param->tunnel.ipv4.dst_addr, sizeof(ipsec_sa->tun_dst_ip)); + odp_atomic_init_u32(&ipsec_sa->tun_hdr_id, 0); + ipsec_sa->tun_ttl = param->tunnel.ipv4.ttl; + } + + odp_crypto_session_param_init(&crypto_param); + + /* Setup parameters and call crypto library to create session */ + crypto_param.op = (ODP_IPSEC_DIR_INBOUND == param->dir) ? + ODP_CRYPTO_OP_DECODE : + ODP_CRYPTO_OP_ENCODE; + crypto_param.auth_cipher_text = 1; + + // FIXME: is it possible to use ASYNC crypto to implement ASYNC IPsec? + crypto_param.pref_mode = ODP_CRYPTO_SYNC; + crypto_param.compl_queue = ODP_QUEUE_INVALID; + crypto_param.output_pool = ODP_POOL_INVALID; + + crypto_param.cipher_alg = param->crypto.cipher_alg; + crypto_param.cipher_key = param->crypto.cipher_key; + crypto_param.auth_alg = param->crypto.auth_alg; + crypto_param.auth_key = param->crypto.auth_key; + + switch (crypto_param.auth_alg) { + case ODP_AUTH_ALG_NULL: + ipsec_sa->icv_len = 0; + break; + case ODP_AUTH_ALG_MD5_HMAC: + case ODP_AUTH_ALG_MD5_96: + ipsec_sa->icv_len = 12; + break; + case ODP_AUTH_ALG_SHA1_HMAC: + ipsec_sa->icv_len = 12; + break; + case ODP_AUTH_ALG_SHA256_HMAC: + case ODP_AUTH_ALG_SHA256_128: + ipsec_sa->icv_len = 16; + break; + case ODP_AUTH_ALG_SHA512_HMAC: + ipsec_sa->icv_len = 32; + break; + default: + return ODP_IPSEC_SA_INVALID; + } + + switch (crypto_param.cipher_alg) { + case ODP_CIPHER_ALG_NULL: + ipsec_sa->esp_iv_len = 0; + ipsec_sa->esp_block_len = 1; + break; + case ODP_CIPHER_ALG_DES: + case ODP_CIPHER_ALG_3DES_CBC: + ipsec_sa->esp_iv_len = 8; + ipsec_sa->esp_block_len = 8; + break; + case ODP_CIPHER_ALG_AES_CBC: + case ODP_CIPHER_ALG_AES128_CBC: + case ODP_CIPHER_ALG_AES_GCM: + case ODP_CIPHER_ALG_AES128_GCM: + ipsec_sa->esp_iv_len = 16; + ipsec_sa->esp_block_len = 16; + break; + } + + /* Generate an IV */ + if (ipsec_sa->esp_iv_len) { + crypto_param.iv.data = ipsec_sa->iv; + crypto_param.iv.length = odp_random_data(crypto_param.iv.data, ipsec_sa->esp_iv_len, ODP_RANDOM_CRYPTO); + if (crypto_param.iv.length != ipsec_sa->esp_iv_len) + goto error; + } + + if (odp_crypto_session_create(&crypto_param, &ipsec_sa->session, &ses_create_rc)) + goto error; + + return ipsec_sa->ipsec_sa_hdl; + +error: + ipsec_sa_release(ipsec_sa); return ODP_IPSEC_SA_INVALID; } int odp_ipsec_sa_disable(odp_ipsec_sa_t sa) { - (void)sa; + ipsec_sa_t *ipsec_sa = ipsec_sa_entry_from_hdl(sa); + uint32_t users; + int cas = 0; - return -1; + /* This is a custom rwlock implementation. It is not possible to use + * original rwlock, because there is no way to test if current code is + * the last reader when disable operation is pending. */ + users = odp_atomic_load_u32(&ipsec_sa->users); + + while (0 == cas) { + if (users & IPSEC_USERS_DISABLING) + return -1; + + cas = odp_atomic_cas_acq_u32(&ipsec_sa->users, &users, + users | IPSEC_USERS_DISABLING); + } + + if (ODP_QUEUE_INVALID != ipsec_sa->queue) { + /* + * If there were not active users when we disabled SA, + * send the event. + */ + if (IPSEC_USERS_DISABLING == users) + odp_ipsec_status_send(sa, + ODP_IPSEC_STATUS_SA_DISABLE, + 0); + + return 0; + } + + while (IPSEC_USERS_DISABLING != users) { + odp_cpu_pause(); + users = odp_atomic_load_u32(&ipsec_sa->users); + } + + return 0; } int odp_ipsec_sa_destroy(odp_ipsec_sa_t sa) { - (void)sa; + ipsec_sa_t *ipsec_sa = ipsec_sa_entry_from_hdl(sa); + int rc = 0; + uint32_t users = odp_atomic_load_u32(&ipsec_sa->users); - return -1; + if (IPSEC_USERS_DISABLING != users) { + ODP_ERR("Distroying not disabled ipsec_sa: %u\n", ipsec_sa->ipsec_sa_idx); + return -1; + } + + if (odp_crypto_session_destroy(ipsec_sa->session) < 0) { + ODP_ERR("Error destroying crypto session for ipsec_sa: %u\n", ipsec_sa->ipsec_sa_idx); + rc = -1; + } + + if (ipsec_sa_release(ipsec_sa) < 0) + return -1; + else + return rc; +} + +static +ipsec_sa_t *ipsec_sa_use(odp_ipsec_sa_t sa) +{ + ipsec_sa_t *ipsec_sa; + uint32_t users; + int cas = 0; + + if (ODP_IPSEC_SA_INVALID == sa) + return NULL; + + ipsec_sa = ipsec_sa_entry_from_hdl(sa); + users = odp_atomic_load_u32(&ipsec_sa->users); + while (0 == cas) { + if (users & IPSEC_USERS_DISABLING) + return NULL; + + cas = odp_atomic_cas_acq_u32(&ipsec_sa->users, &users, + users + 1); + } + + return ipsec_sa; +} + +static +void ipsec_sa_unuse(ipsec_sa_t *ipsec_sa) +{ + uint32_t users; + int cas = 0; + + users = odp_atomic_load_u32(&ipsec_sa->users); + while (0 == cas) + cas = odp_atomic_cas_acq_u32(&ipsec_sa->users, &users, + users - 1); + + if (users == IPSEC_USERS_DISABLING) + odp_ipsec_status_send(ipsec_sa->ipsec_sa_hdl, + ODP_IPSEC_STATUS_SA_DISABLE, + 0); +} + +#define ipv4_hdr_len(ip) (_ODP_IPV4HDR_IHL(ip->ver_ihl) * 4) +static inline +void ipv4_adjust_len(_odp_ipv4hdr_t *ip, int adj) +{ + ip->tot_len = odp_cpu_to_be_16(odp_be_to_cpu_16(ip->tot_len) + adj); +} + +/** + * Allocate per packet processing context and associate it with + * packet buffer + * + * @param pkt Packet + * + * @return pointer to context area + */ +static +ipsec_ctx_t *ipsec_ctx_alloc(void) +{ + odp_buffer_t ctx_buf = odp_buffer_alloc(ipsec_ctx_pool); + ipsec_ctx_t *ctx; + + if (odp_unlikely(ODP_BUFFER_INVALID == ctx_buf)) + return NULL; + + ctx = odp_buffer_addr(ctx_buf); + memset(ctx, 0, sizeof(*ctx)); + ctx->buffer = ctx_buf; + + return ctx; +} + +/** + * Release per packet resources + * + * @param ctx Packet context + */ +static +void ipsec_ctx_free(ipsec_ctx_t *ctx) +{ + if (ODP_PACKET_INVALID != ctx->crypto.pkt) + odp_packet_free(ctx->crypto.pkt); + + if (ODP_PACKET_INVALID != ctx->pkt) + odp_packet_free(ctx->pkt); + + odp_buffer_free(ctx->buffer); +} + +odp_ipsec_result_t odp_ipsec_result_from_event(odp_event_t ev) +{ + if (odp_unlikely(ODP_EVENT_INVALID == ev)) + return ODP_IPSEC_RESULT_INVALID; + + if (odp_event_type(ev) != ODP_EVENT_IPSEC_RESULT) + ODP_ABORT("Event not an IPsec result"); + + return (odp_ipsec_result_t)ev; +} + +static ipsec_result_hdr_t *ipsec_result_hdr_from_buf(odp_buffer_t buf) +{ + return (ipsec_result_hdr_t *)(void *)buf_hdl_to_hdr(buf); +} + +static ipsec_result_hdr_t *ipsec_result_hdr(odp_ipsec_result_t res) +{ + odp_buffer_t buf = odp_buffer_from_event(odp_ipsec_result_to_event(res)); + + return ipsec_result_hdr_from_buf(buf); +} + +odp_event_t odp_ipsec_result_to_event(odp_ipsec_result_t res) +{ + if (odp_unlikely(res == ODP_IPSEC_RESULT_INVALID)) + return ODP_EVENT_INVALID; + + return (odp_event_t)res; +} + +static +odp_ipsec_result_t odp_ipsec_result_alloc(void) +{ + odp_buffer_t buf = odp_buffer_alloc(ipsec_result_pool); + + if (odp_unlikely(buf == ODP_BUFFER_INVALID)) + return ODP_IPSEC_RESULT_INVALID; + + _odp_buffer_event_type_set(buf, ODP_EVENT_IPSEC_RESULT); + + return odp_ipsec_result_from_event(odp_buffer_to_event(buf)); +} + +void odp_ipsec_result_free(odp_ipsec_result_t res) +{ + odp_event_t ev = odp_ipsec_result_to_event(res); + ipsec_result_hdr_t *res_hdr; + + res_hdr = ipsec_result_hdr(res); + while (NULL != res_hdr->ctx) { + ipsec_ctx_t *ctx = res_hdr->ctx; + + res_hdr->ctx = ctx->next; + ipsec_ctx_free(ctx); + } + + odp_buffer_free(odp_buffer_from_event(ev)); +} + +odp_ipsec_status_event_t odp_ipsec_status_from_event(odp_event_t ev) +{ + if (odp_unlikely(ODP_EVENT_INVALID == ev)) + return ODP_IPSEC_STATUS_EVENT_INVALID; + + if (odp_event_type(ev) != ODP_EVENT_IPSEC_STATUS) + ODP_ABORT("Event not an IPsec status"); + + return (odp_ipsec_status_event_t)ev; +} + +static ipsec_status_hdr_t *ipsec_status_hdr_from_buf(odp_buffer_t buf) +{ + return (ipsec_status_hdr_t *)(void *)buf_hdl_to_hdr(buf); +} + +static ipsec_status_hdr_t *ipsec_status_hdr(odp_ipsec_status_event_t res) +{ + odp_buffer_t buf = odp_buffer_from_event(odp_ipsec_status_to_event(res)); + + return ipsec_status_hdr_from_buf(buf); +} + +odp_event_t odp_ipsec_status_to_event(odp_ipsec_status_event_t res) +{ + if (odp_unlikely(res == ODP_IPSEC_STATUS_EVENT_INVALID)) + return ODP_EVENT_INVALID; + + return (odp_event_t)res; +} + +static +odp_ipsec_status_event_t odp_ipsec_status_alloc(void) +{ + odp_buffer_t buf = odp_buffer_alloc(ipsec_status_pool); + + if (odp_unlikely(buf == ODP_BUFFER_INVALID)) + return ODP_IPSEC_STATUS_EVENT_INVALID; + + _odp_buffer_event_type_set(buf, ODP_EVENT_IPSEC_STATUS); + + return odp_ipsec_status_from_event(odp_buffer_to_event(buf)); +} + +void odp_ipsec_status_free(odp_ipsec_status_event_t res) +{ + odp_event_t ev = odp_ipsec_status_to_event(res); + + odp_buffer_free(odp_buffer_from_event(ev)); +} + +static +int odp_ipsec_status_send(odp_ipsec_sa_t sa, + odp_ipsec_status_id_t id, + int ret) +{ + odp_ipsec_status_event_t ipsec_ev = odp_ipsec_status_alloc(); + odp_event_t ev; + odp_queue_t queue; + ipsec_status_hdr_t *status_hdr; + + if (ODP_IPSEC_STATUS_EVENT_INVALID == ipsec_ev) + return -1; + + status_hdr = ipsec_status_hdr(ipsec_ev); + + status_hdr->status.id = id; + status_hdr->status.ret = ret; + status_hdr->status.sa = sa; + + if (ODP_IPSEC_SA_INVALID == sa) + queue = ipsec_config.inbound.default_queue; + else + queue = ipsec_sa_entry_from_hdl(sa)->queue; + + ev = odp_ipsec_status_to_event(ipsec_ev); + + if (odp_queue_enq(queue, ev)) { + odp_event_free(ev); + return -1; + } + + return 0; +} + +static +void ipsec_finish(ipsec_ctx_t *ctx, + odp_ipsec_packet_result_t *res, + odp_packet_t *pkt) +{ + odp_crypto_op_result_t *result = &ctx->crypto; + + res->status = ctx->status; + + /* Check crypto result */ + if (!result->ok) { + if (result->cipher_status.alg_err == ODP_CRYPTO_ALG_ERR_NONE || + result->cipher_status.hw_err == ODP_CRYPTO_HW_ERR_NONE) + res->status.error.alg = 1; + + if (result->auth_status.alg_err == ODP_CRYPTO_ALG_ERR_NONE || + result->auth_status.hw_err == ODP_CRYPTO_HW_ERR_NONE) + res->status.error.auth = 1; + } else { + ctx->pkt = result->pkt; + result->pkt = ODP_PACKET_INVALID; + + if (ctx->postprocess) + ctx->postprocess(ctx); + } + + *pkt = ctx->pkt; + ctx->pkt = ODP_PACKET_INVALID; + + res->sa = ctx->sa; +} + +static +void ipsec_in_postprocess(ipsec_ctx_t *ctx); + +static +int ipsec_in_single(ipsec_ctx_t *ctx) +{ + odp_packet_t pkt = ctx->pkt; + ipsec_sa_t *ipsec_sa; + uint32_t ip_offset = odp_packet_l3_offset(pkt); + _odp_ipv4hdr_t *ip = odp_packet_l3_ptr(pkt, NULL); + uint16_t ip_hdr_len = ipv4_hdr_len(ip); + odp_crypto_op_param_t param; + odp_bool_t posted = 0; + int rc = -1; + + ctx->crypto.pkt = ODP_PACKET_INVALID; + + ipsec_sa = ipsec_sa_use(ctx->sa); + if (odp_unlikely(NULL == ipsec_sa)) { + ctx->status.error.sa_lookup = 1; + return -1; + } + + /* Initialize parameters block */ + memset(¶m, 0, sizeof(param)); + param.session = ipsec_sa->session; + param.ctx = ctx; + + /* Save everything to context */ + ctx->ip_tos = ip->tos; + ctx->ip_frag_offset = odp_be_to_cpu_16(ip->frag_offset); + ctx->ip_ttl = ip->ttl; + + ctx->postprocess = ipsec_in_postprocess; + ctx->ipsec_offset = ip_offset + ip_hdr_len; + + /* Check IP header for IPSec protocols and look it up */ + if (_ODP_IPPROTO_AH == ip->proto) { + _odp_ahhdr_t ah; + + if (ODP_IPSEC_AH != ipsec_sa->proto) { + ctx->status.error.proto = 1; + goto out; + } + + if (odp_packet_copy_to_mem(pkt, ctx->ipsec_offset, sizeof(ah), &ah) < 0) { + ctx->status.error.alg = 1; + goto out; + } + + ctx->hdr_len = (ah.ah_len + 2) * 4; + ctx->trl_len = 0; + + /* If authenticating, zero the mutable fields build the request */ + ip->chksum = 0; + ip->tos = 0; + ip->frag_offset = 0; + ip->ttl = 0; + + param.auth_range.offset = ip_offset; + param.auth_range.length = odp_be_to_cpu_16(ip->tot_len); + param.hash_result_offset = ctx->ipsec_offset + _ODP_AHHDR_LEN; + } else if (_ODP_IPPROTO_ESP == ip->proto) { + _odp_esphdr_t esp; + + if (ODP_IPSEC_ESP != ipsec_sa->proto) { + ctx->status.error.proto = 1; + goto out; + } + + if (odp_packet_copy_to_mem(pkt, ctx->ipsec_offset, sizeof(esp), &esp) < 0) { + ctx->status.error.alg = 1; + goto out; + } + + if (odp_packet_copy_to_mem(pkt, ctx->ipsec_offset + _ODP_ESPHDR_LEN, ipsec_sa->esp_iv_len, ctx->iv) < 0) { + ctx->status.error.alg = 1; + goto out; + } + + ctx->hdr_len = _ODP_ESPHDR_LEN + ipsec_sa->esp_iv_len; + ctx->trl_len = _ODP_ESPTRL_LEN + ipsec_sa->icv_len; + + param.cipher_range.offset = ctx->ipsec_offset + ctx->hdr_len; + param.cipher_range.length = odp_be_to_cpu_16(ip->tot_len) - ip_hdr_len - ctx->hdr_len - ipsec_sa->icv_len; + param.override_iv_ptr = ctx->iv; + + param.auth_range.offset = ctx->ipsec_offset; + param.auth_range.length = odp_be_to_cpu_16(ip->tot_len) - ip_hdr_len; + param.hash_result_offset = ip_offset + odp_be_to_cpu_16(ip->tot_len) - ipsec_sa->icv_len; + } else { + ctx->status.error.proto = 1; + goto out; + } + + param.pkt = pkt; + /* Create new packet after all length extensions */ + if (ipsec_sa->in_place) { + param.out_pkt = pkt; + } else { + param.out_pkt = odp_packet_alloc(odp_packet_pool(pkt), + odp_packet_len(pkt)); + /* uarea will be copied by odp_crypto_operation */ + odp_packet_user_ptr_set(param.out_pkt, + odp_packet_user_ptr(param.pkt)); + } + + rc = odp_crypto_operation(¶m, &posted, &ctx->crypto); + if (rc < 0) { + ODP_DBG("Crypto failed\n"); + ctx->status.error.alg = 1; + ipsec_sa_unuse(ipsec_sa); + + return rc; + } + + ODP_ASSERT(!posted); + +out: + ipsec_sa_unuse(ipsec_sa); + + return rc; +} + +static +void ipsec_in_postprocess(ipsec_ctx_t *ctx) +{ + odp_packet_t pkt = ctx->pkt; + uint32_t ip_offset = odp_packet_l3_offset(pkt); + _odp_ipv4hdr_t *ip = odp_packet_l3_ptr(pkt, NULL); + uint16_t ip_hdr_len = ipv4_hdr_len(ip); + + if (_ODP_IPPROTO_AH == ip->proto) { + /* + * Finish auth + */ + _odp_ahhdr_t ah; + + if (odp_packet_copy_to_mem(pkt, ctx->ipsec_offset, sizeof(ah), &ah) < 0) { + ctx->status.error.alg = 1; + goto out; + } + + ip->proto = ah.next_header; + + /* Restore mutable fields */ + ip->ttl = ctx->ip_ttl; + ip->tos = ctx->ip_tos; + ip->frag_offset = odp_cpu_to_be_16(ctx->ip_frag_offset); + } else if (_ODP_IPPROTO_ESP == ip->proto) { + /* + * Finish cipher by finding ESP trailer and processing + */ + _odp_esptrl_t esptrl; + uint32_t esptrl_offset = ip_offset + odp_be_to_cpu_16(ip->tot_len) - ctx->trl_len; + + if (odp_packet_copy_to_mem(pkt, esptrl_offset, sizeof(esptrl), &esptrl) < 0) { + ctx->status.error.alg = 1; + goto out; + } + + ip->proto = esptrl.next_header; + ctx->trl_len += esptrl.pad_len; + } else { + ctx->status.error.proto = 1; + goto out; + } + + if (ip->proto == _ODP_IPV4) { + ip->ttl -= 1; + ip->chksum = 0; + _odp_ipv4_csum_update(pkt); + + /* We have a tunneled IPv4 packet, strip outer and IPsec headers */ + odp_packet_move_data(pkt, ip_hdr_len + ctx->hdr_len, 0, ip_offset); + if (odp_packet_trunc_head(&pkt, ip_hdr_len + ctx->hdr_len, NULL, NULL) < 0) { + ctx->status.error.alg = 1; + goto out; + } + + } else { + /* Finalize the IPv4 header */ + ipv4_adjust_len(ip, -(ctx->hdr_len + ctx->trl_len)); + + ip->chksum = 0; + _odp_ipv4_csum_update(pkt); + + odp_packet_move_data(pkt, ctx->hdr_len, 0, ip_offset + ip_hdr_len); + if (odp_packet_trunc_head(&pkt, ctx->hdr_len, NULL, NULL) < 0) { + ctx->status.error.alg = 1; + goto out; + } + } + + if (odp_packet_trunc_tail(&pkt, ctx->trl_len, NULL, NULL) < 0) + ctx->status.error.alg = 1; + +out: + ctx->pkt = pkt; +} + +/** Helper for calculating encode length using data length and block size */ +#define ESP_ENCODE_LEN(x, b) ((((x) + (b - 1)) / b) * b) + +static +void ipsec_out_postprocess(ipsec_ctx_t *ctx); + +static +int ipsec_out_single(ipsec_ctx_t *ctx) +{ + odp_packet_t pkt = ctx->pkt; + ipsec_sa_t *ipsec_sa; + uint32_t ip_offset = odp_packet_l3_offset(pkt); + _odp_ipv4hdr_t *ip = odp_packet_l3_ptr(pkt, NULL); + uint16_t ip_hdr_len = ipv4_hdr_len(ip); + odp_crypto_op_param_t param; + odp_bool_t posted = 0; + int rc = -1; + + ctx->crypto.pkt = ODP_PACKET_INVALID; + + ipsec_sa = ipsec_sa_use(ctx->sa); + if (odp_unlikely(NULL == ipsec_sa)) { + ctx->status.error.sa_lookup = 1; + return -1; + } + + /* Initialize parameters block */ + memset(¶m, 0, sizeof(param)); + param.session = ipsec_sa->session; + param.pkt = pkt; + param.ctx = ctx; + + if (ipsec_sa->mode == ODP_IPSEC_MODE_TUNNEL) { + _odp_ipv4hdr_t out_ip; + _odp_ipv4hdr_t *inner_ip; + uint16_t tun_hdr_offset = ip_offset + ip_hdr_len; + + if (odp_packet_extend_head(&pkt, _ODP_IPV4HDR_LEN, NULL, NULL) < 0) { + ctx->status.error.alg = 1; + goto out; + } + + odp_packet_move_data(pkt, 0, _ODP_IPV4HDR_LEN, ip_offset); + + inner_ip = odp_packet_offset(pkt, tun_hdr_offset, NULL, NULL); + + out_ip.ver_ihl = 0x45; + out_ip.tos = inner_ip->tos; // FIXME + out_ip.tot_len = odp_cpu_to_be_16(odp_be_to_cpu_16(inner_ip->tot_len) + _ODP_IPV4HDR_LEN); + /* No need to convert to BE: ID just should not be duplicated */ + out_ip.id = (odp_atomic_fetch_add_u32(&ipsec_sa->tun_hdr_id, 1) + 1) & 0xffff; + out_ip.frag_offset = 0; + out_ip.ttl = ipsec_sa->tun_ttl; + out_ip.proto = _ODP_IPV4; + out_ip.chksum = 0; + out_ip.src_addr = ipsec_sa->tun_src_ip; + out_ip.dst_addr = ipsec_sa->tun_dst_ip; + + odp_packet_copy_from_mem(pkt, ip_offset, _ODP_IPV4HDR_LEN, &out_ip); + + odp_packet_l4_offset_set(pkt, ip_offset + _ODP_IPV4HDR_LEN); + + ip = odp_packet_l3_ptr(pkt, NULL); + ip_hdr_len = _ODP_IPV4HDR_LEN; + } + + /* Save IPv4 stuff */ + ctx->ip_tos = ip->tos; + ctx->ip_frag_offset = odp_be_to_cpu_16(ip->frag_offset); + ctx->ip_ttl = ip->ttl; + + ctx->postprocess = ipsec_out_postprocess; + + ctx->ipsec_offset = ip_offset + ip_hdr_len; + + if (ipsec_sa->proto == ODP_IPSEC_AH) { + ctx->hdr_len = _ODP_AHHDR_LEN + ipsec_sa->icv_len; + ctx->trl_len = 0; + } else if (ipsec_sa->proto == ODP_IPSEC_ESP) { + uint32_t encrypt_len; + uint16_t ip_next_len = odp_be_to_cpu_16(ip->tot_len) - ip_hdr_len; + + ctx->hdr_len += _ODP_ESPHDR_LEN + ipsec_sa->esp_iv_len; + + encrypt_len = ESP_ENCODE_LEN(ip_next_len + _ODP_ESPTRL_LEN, + ipsec_sa->esp_block_len); + ctx->trl_len = encrypt_len - ip_next_len + ipsec_sa->icv_len; + } else { + ctx->status.error.proto = 1; + goto out; + } + + if (odp_packet_extend_tail(&pkt, ctx->trl_len, NULL, NULL) < 0) { + ctx->status.error.alg = 1; + goto out; + } + + if (odp_packet_extend_head(&pkt, ctx->hdr_len, NULL, NULL) < 0) { + ctx->status.error.alg = 1; + goto out; + } + + odp_packet_move_data(pkt, 0, ctx->hdr_len, ctx->ipsec_offset); + + ip = odp_packet_l3_ptr(pkt, NULL); + + /* Set IPv4 length before authentication */ + ipv4_adjust_len(ip, ctx->hdr_len + ctx->trl_len); + + /* For authentication, build header clear mutables and build request */ + if (ipsec_sa->proto == ODP_IPSEC_AH) { + _odp_ahhdr_t ah = {}; + uint8_t icv[ipsec_sa->icv_len]; + + ah.spi = odp_cpu_to_be_32(ipsec_sa->spi); + ah.ah_len = 1 + (ipsec_sa->icv_len / 4); + ah.seq_no = odp_cpu_to_be_32(odp_atomic_fetch_add_u32(&ipsec_sa->seq, 1) + 1); + ah.next_header = ip->proto; + ip->proto = _ODP_IPPROTO_AH; + + odp_packet_copy_from_mem(pkt, ctx->ipsec_offset, _ODP_AHHDR_LEN, &ah); + memset(icv, 0, ipsec_sa->icv_len); + odp_packet_copy_from_mem(pkt, ctx->ipsec_offset + _ODP_AHHDR_LEN, ipsec_sa->icv_len, icv); + + ip->chksum = 0; + ip->tos = 0; + ip->frag_offset = 0; + ip->ttl = 0; + + param.auth_range.offset = ip_offset; + param.auth_range.length = odp_be_to_cpu_16(ip->tot_len); + param.hash_result_offset = ctx->ipsec_offset + _ODP_AHHDR_LEN; + } + + if (ipsec_sa->proto == ODP_IPSEC_ESP) { + _odp_esphdr_t esp = {}; + _odp_esptrl_t esptrl = {}; + uint32_t esptrl_offset = ip_offset + odp_be_to_cpu_16(ip->tot_len) - ipsec_sa->icv_len - _ODP_ESPTRL_LEN; + + esp.spi = odp_cpu_to_be_32(ipsec_sa->spi); + esp.seq_no = odp_cpu_to_be_32(odp_atomic_fetch_add_u32(&ipsec_sa->seq, 1) + 1); + + esptrl.pad_len = ctx->trl_len - _ODP_ESPTRL_LEN - ipsec_sa->icv_len; + esptrl.next_header = ip->proto; + ip->proto = _ODP_IPPROTO_ESP; + + odp_packet_copy_from_mem(pkt, ctx->ipsec_offset, _ODP_ESPHDR_LEN, &esp); + odp_packet_copy_from_mem(pkt, ctx->ipsec_offset + _ODP_ESPHDR_LEN, ipsec_sa->esp_iv_len, ipsec_sa->iv); + odp_packet_copy_from_mem(pkt, esptrl_offset, _ODP_ESPTRL_LEN, &esptrl); + + param.cipher_range.offset = ctx->ipsec_offset + ctx->hdr_len; + param.cipher_range.length = odp_be_to_cpu_16(ip->tot_len) - ip_hdr_len - ctx->hdr_len - ipsec_sa->icv_len; + + param.auth_range.offset = ctx->ipsec_offset; + param.auth_range.length = odp_be_to_cpu_16(ip->tot_len) - ip_hdr_len; + param.hash_result_offset = ip_offset + odp_be_to_cpu_16(ip->tot_len) - ipsec_sa->icv_len; + } + + param.pkt = pkt; + /* Create new packet after all length extensions */ + if (ipsec_sa->in_place) { + param.out_pkt = pkt; + } else { + param.out_pkt = odp_packet_alloc(odp_packet_pool(pkt), + odp_packet_len(pkt)); + odp_packet_user_ptr_set(param.out_pkt, + odp_packet_user_ptr(param.pkt)); + } + + rc = odp_crypto_operation(¶m, &posted, &ctx->crypto); + if (rc < 0) { + ODP_DBG("Crypto failed\n"); + ctx->status.error.alg = 1; + ipsec_sa_unuse(ipsec_sa); + + return rc; + } + + ODP_ASSERT(!posted); + +out: + ctx->pkt = pkt; + ipsec_sa_unuse(ipsec_sa); + + return rc; +} + +static +void ipsec_out_postprocess(ipsec_ctx_t *ctx) +{ + odp_packet_t pkt = ctx->pkt; + _odp_ipv4hdr_t *ip = odp_packet_l3_ptr(pkt, NULL); + + /* Finalize the IPv4 header */ + if (ip->proto == _ODP_IPPROTO_AH) { + ip->ttl = ctx->ip_ttl; + ip->tos = ctx->ip_tos; + ip->frag_offset = odp_cpu_to_be_16(ctx->ip_frag_offset); + } + + ip->chksum = 0; + _odp_ipv4_csum_update(pkt); } +#if 0 +static odp_ipsec_op_opt_t default_opt = { + .mode = ODP_IPSEC_FRAG_DISABLED, +}; +#endif + int odp_ipsec_in(const odp_ipsec_op_param_t *input, odp_ipsec_op_result_t *output) { - (void)input; - (void)output; + unsigned in_pkt = 0; + unsigned out_pkt = 0; + unsigned sa_idx = 0; + unsigned opt_idx = 0; + unsigned sa_inc = (input->num_sa > 1) ? 1 : 0; + unsigned opt_inc = (input->num_opt > 1) ? 1 : 0; - return -1; + while (in_pkt < input->num_pkt && out_pkt < output->num_pkt) { + odp_ipsec_sa_t sa; + ipsec_ctx_t ctx; + + memset(&ctx, 0, sizeof(ctx)); + ctx.buffer = ODP_BUFFER_INVALID; + + if (0 == input->num_sa) + sa = ODP_IPSEC_SA_INVALID; + else + sa = input->sa[sa_idx]; + +#if 0 + odp_ipsec_op_opt_t *opt; + + if (0 == input->num_opt) + opt = &default_opt; + else + opt = &input->opt[opt_idx]; +#endif + + ctx.pkt = input->pkt[in_pkt]; + ctx.sa = sa; + + if (ipsec_in_single(&ctx) < 0) + ctx.status.error.alg = 1; + + ipsec_finish(&ctx, &output->res[out_pkt], &output->pkt[out_pkt]); + + in_pkt++; + out_pkt++; + sa_idx += sa_inc; + opt_idx += opt_inc; + } + + return in_pkt; } int odp_ipsec_out(const odp_ipsec_op_param_t *input, - odp_ipsec_op_result_t *output) + odp_ipsec_op_result_t *output) { - (void)input; - (void)output; + unsigned in_pkt = 0; + unsigned out_pkt = 0; + unsigned sa_idx = 0; + unsigned opt_idx = 0; + unsigned sa_inc = (input->num_sa > 1) ? 1 : 0; + unsigned opt_inc = (input->num_opt > 1) ? 1 : 0; - return -1; + while (in_pkt < input->num_pkt && out_pkt < output->num_pkt) { + odp_ipsec_sa_t sa; + ipsec_ctx_t ctx; + + memset(&ctx, 0, sizeof(ctx)); + ctx.buffer = ODP_BUFFER_INVALID; + + if (0 == input->num_sa) + sa = ODP_IPSEC_SA_INVALID; + else + sa = input->sa[sa_idx]; + +#if 0 + odp_ipsec_op_opt_t *opt; + + if (0 == input->num_opt) + opt = &default_opt; + else + opt = &input->opt[opt_idx]; +#endif + + ctx.pkt = input->pkt[in_pkt]; + ctx.sa = sa; + + if (ipsec_out_single(&ctx) < 0) + ctx.status.error.alg = 1; + + ipsec_finish(&ctx, &output->res[out_pkt], &output->pkt[out_pkt]); + + in_pkt++; + out_pkt++; + sa_idx += sa_inc; + opt_idx += opt_inc; + } + + return in_pkt; } int odp_ipsec_in_enq(const odp_ipsec_op_param_t *input) { - (void)input; + unsigned in_pkt = 0; + unsigned sa_idx = 0; + unsigned opt_idx = 0; + unsigned sa_inc = (input->num_sa > 1) ? 1 : 0; + unsigned opt_inc = (input->num_opt > 1) ? 1 : 0; - return -1; + while (in_pkt < input->num_pkt) { + odp_ipsec_result_t ipsec_ev; + odp_event_t ev; + ipsec_result_hdr_t *res_hdr; + odp_ipsec_sa_t sa; + ipsec_ctx_t *ctx; + odp_queue_t queue; + + ipsec_ev = odp_ipsec_result_alloc(); + if (ODP_IPSEC_RESULT_INVALID == ipsec_ev) + break; + + ev = odp_ipsec_result_to_event(ipsec_ev); + + res_hdr = ipsec_result_hdr(ipsec_ev); + + ctx = ipsec_ctx_alloc(); + if (NULL == ctx) { + odp_event_free(ev); + break; + } + + res_hdr->ctx = ctx; + + if (0 == input->num_sa) + sa = ODP_IPSEC_SA_INVALID; + else + sa = input->sa[sa_idx]; + +#if 0 + odp_ipsec_op_opt_t *opt; + + if (0 == input->num_opt) + opt = &default_opt; + else + opt = &input->opt[opt_idx]; +#endif + + ctx->pkt = input->pkt[in_pkt]; + ctx->sa = sa; + + if (ipsec_in_single(ctx) < 0) + ctx->status.error.alg = 1; + + in_pkt++; + sa_idx += sa_inc; + opt_idx += opt_inc; + + if (ODP_IPSEC_SA_INVALID == sa) + queue = ipsec_config.inbound.default_queue; + else + queue = ipsec_sa_entry_from_hdl(sa)->queue; + + if (odp_queue_enq(queue, ev)) { + odp_event_free(ev); + return -1; + } + } + + return in_pkt; } int odp_ipsec_out_enq(const odp_ipsec_op_param_t *input) { - (void)input; + unsigned in_pkt = 0; + unsigned sa_idx = 0; + unsigned opt_idx = 0; + unsigned sa_inc = (input->num_sa > 1) ? 1 : 0; + unsigned opt_inc = (input->num_opt > 1) ? 1 : 0; - return -1; + while (in_pkt < input->num_pkt) { + odp_ipsec_result_t ipsec_ev; + odp_event_t ev; + ipsec_result_hdr_t *res_hdr; + odp_ipsec_sa_t sa; + ipsec_ctx_t *ctx; + odp_queue_t queue; + + ipsec_ev = odp_ipsec_result_alloc(); + if (ODP_IPSEC_RESULT_INVALID == ipsec_ev) + break; + + ev = odp_ipsec_result_to_event(ipsec_ev); + + res_hdr = ipsec_result_hdr(ipsec_ev); + + ctx = ipsec_ctx_alloc(); + if (NULL == ctx) { + odp_event_free(ev); + break; + } + + res_hdr->ctx = ctx; + + if (0 == input->num_sa) + sa = ODP_IPSEC_SA_INVALID; + else + sa = input->sa[sa_idx]; + +#if 0 + odp_ipsec_op_opt_t *opt; + + if (0 == input->num_opt) + opt = &default_opt; + else + opt = &input->opt[opt_idx]; +#endif + + ctx->pkt = input->pkt[in_pkt]; + ctx->sa = sa; + + if (ipsec_out_single(ctx) < 0) + ctx->status.error.alg = 1; + + in_pkt++; + sa_idx += sa_inc; + opt_idx += opt_inc; + + if (ODP_IPSEC_SA_INVALID == sa) + queue = ipsec_config.outbound.default_queue; + else + queue = ipsec_sa_entry_from_hdl(sa)->queue; + + if (odp_queue_enq(queue, ev)) { + odp_event_free(ev); + return -1; + } + } + + return in_pkt; } int odp_ipsec_out_inline(const odp_ipsec_op_param_t *op_param, @@ -116,18 +1453,63 @@ int odp_ipsec_out_inline(const odp_ipsec_op_param_t *op_param, int odp_ipsec_result(odp_ipsec_op_result_t *result, odp_event_t event) { - (void)result; - (void)event; + odp_ipsec_result_t ipsec_ev; + ipsec_result_hdr_t *res_hdr; + unsigned out_pkt; + ipsec_ctx_t *ctx; - return -1; + if (odp_unlikely(ODP_EVENT_INVALID == event)) + return -1; + + ipsec_ev = odp_ipsec_result_from_event(event); + if (odp_unlikely(ODP_IPSEC_RESULT_INVALID == ipsec_ev)) + return -1; + + res_hdr = ipsec_result_hdr(ipsec_ev); + + for (out_pkt = 0; out_pkt < result->num_pkt; out_pkt++) { + ipsec_ctx_t *ctx = res_hdr->ctx; + + res_hdr->ctx = ctx->next; + ipsec_finish(ctx, &result->res[out_pkt], &result->pkt[out_pkt]); + + ipsec_ctx_free(ctx); + } + + result->num_pkt = out_pkt; + + /* FIXME: this is place with semantics change */ + if (!res_hdr->ctx) { + odp_ipsec_result_free(ipsec_ev); + return 0; + } + out_pkt = 0; + + for (ctx = res_hdr->ctx; NULL != ctx; ctx = ctx->next) + out_pkt++; + + return out_pkt; } int odp_ipsec_status(odp_ipsec_status_t *status, odp_event_t event) { (void)status; (void)event; + odp_ipsec_status_event_t ipsec_ev; + ipsec_status_hdr_t *status_hdr; - return -1; + if (odp_unlikely(ODP_EVENT_INVALID == event)) + return -1; + + ipsec_ev = odp_ipsec_status_from_event(event); + if (odp_unlikely(ODP_IPSEC_STATUS_EVENT_INVALID == ipsec_ev)) + return -1; + + status_hdr = ipsec_status_hdr(ipsec_ev); + + *status = status_hdr->status; + + return 0; } int odp_ipsec_mtu_update(odp_ipsec_sa_t sa, uint32_t mtu) @@ -140,9 +1522,9 @@ int odp_ipsec_mtu_update(odp_ipsec_sa_t sa, uint32_t mtu) void *odp_ipsec_sa_context(odp_ipsec_sa_t sa) { - (void)sa; + ipsec_sa_t *ipsec_sa = ipsec_sa_entry_from_hdl(sa); - return NULL; + return ipsec_sa->context; } uint64_t odp_ipsec_sa_to_u64(odp_ipsec_sa_t sa)