From patchwork Tue Nov 25 22:37:45 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ola Liljedahl X-Patchwork-Id: 41509 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-wi0-f197.google.com (mail-wi0-f197.google.com [209.85.212.197]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id AFF3525E72 for ; Tue, 25 Nov 2014 22:38:22 +0000 (UTC) Received: by mail-wi0-f197.google.com with SMTP id l15sf1203143wiw.8 for ; Tue, 25 Nov 2014 14:38:21 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:from:to:date:message-id:in-reply-to :references:subject:precedence:list-id:list-unsubscribe:list-archive :list-post:list-help:list-subscribe:mime-version:content-type :content-transfer-encoding:errors-to:sender:x-original-sender :x-original-authentication-results:mailing-list; bh=/PcWtFY7zlB1if6GlN5FZcMrqBitB7RcsaEjBN+ugKI=; b=IQDt0V1Hc8Mm2tAR3co7hXEhQg92AxZh0p2nkyujYLjH0wPvumQtSHDl3Y82MeuLqR R5Gf8f2FxtATnZIWIqSknRaiXOzJd0GCnhM7IH6EJkox4FmchqnI6nw66o8DMbIw96cs ulFbgNxjH95pDq1MvIaGEs974rXmD4pB/VuENW2AJOsRIyurkz69U/qnZ5eV09TdrGqt /Z9fdpyznx/CxQrL61IQnlAbPTi2eB0+iciE6UG87pbfeVa5DBgCKN4MO/trtGunosH3 r2BXXqD/3XLtMXeGmUHX9BB0Of8VFSdkUo3Qv3QdPABdTwWlJMeYXVlWm7JuNVD718Hb ghqg== X-Gm-Message-State: ALoCoQkez4iWggd3fcS60M0TG89pKK+5PvcjqiYyuhb2cVmgML9zYG6ODaCzwugMRC/2WW6dsH+G X-Received: by 10.194.134.193 with SMTP id pm1mr5901wjb.4.1416955101959; Tue, 25 Nov 2014 14:38:21 -0800 (PST) X-BeenThere: patchwork-forward@linaro.org Received: by 10.152.23.130 with SMTP id m2ls647743laf.63.gmail; Tue, 25 Nov 2014 14:38:21 -0800 (PST) X-Received: by 10.112.139.196 with SMTP id ra4mr30042349lbb.95.1416955101653; Tue, 25 Nov 2014 14:38:21 -0800 (PST) Received: from mail-lb0-f175.google.com (mail-lb0-f175.google.com. [209.85.217.175]) by mx.google.com with ESMTPS id xe3si2591909lbb.114.2014.11.25.14.38.21 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 25 Nov 2014 14:38:21 -0800 (PST) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.175 as permitted sender) client-ip=209.85.217.175; Received: by mail-lb0-f175.google.com with SMTP id u10so1434240lbd.6 for ; Tue, 25 Nov 2014 14:38:21 -0800 (PST) X-Received: by 10.112.189.10 with SMTP id ge10mr30408115lbc.23.1416955101533; Tue, 25 Nov 2014 14:38:21 -0800 (PST) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.112.184.201 with SMTP id ew9csp531998lbc; Tue, 25 Nov 2014 14:38:20 -0800 (PST) X-Received: by 10.52.135.68 with SMTP id pq4mr13700710vdb.35.1416955099821; Tue, 25 Nov 2014 14:38:19 -0800 (PST) Received: from ip-10-35-177-41.ec2.internal (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTPS id qr16si940369vdb.90.2014.11.25.14.38.19 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 25 Nov 2014 14:38:19 -0800 (PST) Received-SPF: none (google.com: lng-odp-bounces@lists.linaro.org does not designate permitted sender hosts) client-ip=54.225.227.206; Received: from localhost ([127.0.0.1] helo=ip-10-35-177-41.ec2.internal) by ip-10-35-177-41.ec2.internal with esmtp (Exim 4.76) (envelope-from ) id 1XtOkT-00041z-Ht; Tue, 25 Nov 2014 22:38:17 +0000 Received: from mail-lb0-f180.google.com ([209.85.217.180]) by ip-10-35-177-41.ec2.internal with esmtp (Exim 4.76) (envelope-from ) id 1XtOkH-00041N-7r for lng-odp@lists.linaro.org; Tue, 25 Nov 2014 22:38:05 +0000 Received: by mail-lb0-f180.google.com with SMTP id l4so1461883lbv.11 for ; Tue, 25 Nov 2014 14:37:59 -0800 (PST) X-Received: by 10.152.8.82 with SMTP id p18mr31217739laa.25.1416955079442; Tue, 25 Nov 2014 14:37:59 -0800 (PST) Received: from macmini.lan (84-217-193-77.tn.glocalnet.net. [84.217.193.77]) by mx.google.com with ESMTPSA id vr7sm696196lbb.21.2014.11.25.14.37.58 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 25 Nov 2014 14:37:58 -0800 (PST) From: Ola Liljedahl To: lng-odp@lists.linaro.org Date: Tue, 25 Nov 2014 23:37:45 +0100 Message-Id: <1416955068-6580-2-git-send-email-ola.liljedahl@linaro.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1416955068-6580-1-git-send-email-ola.liljedahl@linaro.org> References: <1416955068-6580-1-git-send-email-ola.liljedahl@linaro.org> X-Topics: patch Subject: [lng-odp] [PATCH 1/4] linux-generic: odp_atomic_internal.h: C11-style 32/64-bit/pointer/flag atomics X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: , List-Help: , List-Subscribe: , MIME-Version: 1.0 Errors-To: lng-odp-bounces@lists.linaro.org Sender: lng-odp-bounces@lists.linaro.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: ola.liljedahl@linaro.org X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.175 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 Signed-off-by: Ola Liljedahl --- Operations (including exchange and compare-and-exchange) with specified C11-based memory models on odp_atomic_u32_t and odp_atomic_u64_t. odp_atomic_ptr_t definition and operations (init, load, store, exchange). odp_atomic_flag_t definitions and operations (init, load, tas, clear). .../linux-generic/include/odp_atomic_internal.h | 603 +++++++++++++++++++++ 1 file changed, 603 insertions(+) create mode 100644 platform/linux-generic/include/odp_atomic_internal.h diff --git a/platform/linux-generic/include/odp_atomic_internal.h b/platform/linux-generic/include/odp_atomic_internal.h new file mode 100644 index 0000000..b330e8f --- /dev/null +++ b/platform/linux-generic/include/odp_atomic_internal.h @@ -0,0 +1,603 @@ +/* Copyright (c) 2013, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +/** + * @file + * + * ODP atomic types and operations, semantically a subset of C11 atomics. + * Reuse the 32-bit and 64-bit type definitions from odp_atomic.h. Introduces + * new atomic pointer and flag types. + * Atomic functions must be used to operate on atomic variables! + */ + +#ifndef ODP_ATOMIC_INTERNAL_H_ +#define ODP_ATOMIC_INTERNAL_H_ + +#include +#include +#include +#include +#include + +#ifdef __cplusplus +extern "C" { +#endif + +/** @addtogroup odp_synchronizers + * Atomic operations. + * @{ + */ + +/** + * Pointer atomic type + */ +typedef struct { + void *v; /**< Actual storage for the atomic variable */ +} odp_atomic_ptr_t +ODP_ALIGNED(sizeof(void *)); /* Enforce alignement! */ + +/** + * Atomic flag (boolean) type + */ +typedef char odp_atomic_flag_t; + +/** + * Memory consistency models supported by ODP. + */ +typedef enum { +/** Relaxed memory ordering, no ordering of other accesses enforced. + * Atomic operations with relaxed memory model cannot be used for + * synchronization */ + ODP_MEMMODEL_RLX = __ATOMIC_RELAXED, +/** Acquire memory ordering, synchronize with release stores from another + * thread (later accesses cannot move before acquire operation). + * Use acquire and release memory models for Release Consistency */ + ODP_MEMMODEL_ACQ = __ATOMIC_ACQUIRE, +/** Release memory ordering, synchronize with acquire loads from another + * thread (earlier accesses cannot move after release operation). + * Use acquire and release memory models for Release Consistency */ + ODP_MEMMODEL_RLS = __ATOMIC_RELEASE, +/** Acquire&release memory ordering, synchronize with acquire loads and release + * stores in another (one other) thread */ + ODP_MEMMODEL_ACQ_RLS = __ATOMIC_ACQ_REL, +/** Sequential consistent memory ordering, synchronize with acquire loads and + * release stores in all threads */ + ODP_MEMMODEL_SC = __ATOMIC_SEQ_CST +} odp_memmodel_t; + +/** + * Insert a full memory barrier (fence) in the compiler and instruction + * sequence. + */ +#define ODP_FULL_BARRIER() __atomic_thread_fence(__ATOMIC_SEQ_CST) + +/***************************************************************************** + * Operations on 32-bit atomics + * odp_atomic_u32_load_mm - return current value + * odp_atomic_u32_store_mm - no return value + * odp_atomic_u32_xchg_mm - return old value + * odp_atomic_u32_cmp_xchg_strong_mm - return bool + * odp_atomic_u32_fetch_add_mm - return old value + * odp_atomic_u32_add_mm - no return value + * odp_atomic_u32_fetch_sub_mm - return old value + * odp_atomic_u32_sub_mm - no return value + *****************************************************************************/ + +/** + * Atomic load of 32-bit atomic variable + * + * @param ptr Pointer to a 32-bit atomic variable + * @param mmodel Memory model associated with the load operation + * + * @return Value of the variable + */ +static inline uint32_t odp_atomic_u32_load_mm(const odp_atomic_u32_t *ptr, + odp_memmodel_t mmodel) +{ + return __atomic_load_n(&ptr->v, mmodel); +} + +/** + * Atomic store to 32-bit atomic variable + * + * @param ptr Pointer to a 32-bit atomic variable + * @param val Value to store in the atomic variable + * @param mmodel Memory model associated with the store operation + */ +static inline void odp_atomic_u32_store_mm(odp_atomic_u32_t *ptr, + uint32_t val, + odp_memmodel_t mmodel) +{ + __atomic_store_n(&ptr->v, val, mmodel); +} + +/** + * Atomic exchange (swap) of 32-bit atomic variable + * + * @param ptr Pointer to a 32-bit atomic variable + * @param val New value to store in the atomic variable + * @param mmodel Memory model associated with the exchange operation + * + * @return Old value of the variable + */ +static inline uint32_t odp_atomic_u32_xchg_mm(odp_atomic_u32_t *ptr, + uint32_t val, + odp_memmodel_t mmodel) + +{ + return __atomic_exchange_n(&ptr->v, val, mmodel); +} + +/** + * Atomic compare and exchange (swap) of 32-bit atomic variable + * "Strong" semantics, will not fail spuriously. + * + * @param ptr Pointer to a 32-bit atomic variable + * @param exp_p Pointer to expected value (updated on failure) + * @param val New value to write + * @param succ Memory model associated with a successful compare-and-swap + * operation + * @param fail Memory model associated with a failed compare-and-swap + * operation + * + * @return 1 (true) if exchange successful, 0 (false) if not successful (and + * '*exp_p' updated with current value) + */ +static inline int odp_atomic_u32_cmp_xchg_strong_mm(odp_atomic_u32_t *ptr, + uint32_t *exp_p, + uint32_t val, + odp_memmodel_t succ, + odp_memmodel_t fail) +{ + return __atomic_compare_exchange_n(&ptr->v, exp_p, val, + false/*strong*/, succ, fail); +} + +/** + * Atomic fetch and add of 32-bit atomic variable + * + * @param ptr Pointer to a 32-bit atomic variable + * @param val Value to add to the atomic variable + * @param mmodel Memory model associated with the add operation + * + * @return Value of the atomic variable before the addition + */ +static inline uint32_t odp_atomic_u32_fetch_add_mm(odp_atomic_u32_t *ptr, + uint32_t val, + odp_memmodel_t mmodel) +{ + return __atomic_fetch_add(&ptr->v, val, mmodel); +} + +/** + * Atomic add of 32-bit atomic variable + * + * @param ptr Pointer to a 32-bit atomic variable + * @param val Value to add to the atomic variable + * @param mmodel Memory model associated with the add operation + */ +static inline void odp_atomic_u32_add_mm(odp_atomic_u32_t *ptr, + uint32_t val, + odp_memmodel_t mmodel) + +{ + (void)__atomic_fetch_add(&ptr->v, val, mmodel); +} + +/** + * Atomic fetch and subtract of 32-bit atomic variable + * + * @param ptr Pointer to a 32-bit atomic variable + * @param val Value to subtract from the atomic variable + * @param mmodel Memory model associated with the subtract operation + * + * @return Value of the atomic variable before the subtraction + */ +static inline uint32_t odp_atomic_u32_fetch_sub_mm(odp_atomic_u32_t *ptr, + uint32_t val, + odp_memmodel_t mmodel) +{ + return __atomic_fetch_sub(&ptr->v, val, mmodel); +} + +/** + * Atomic subtract of 32-bit atomic variable + * + * @param ptr Pointer to a 32-bit atomic variable + * @param val Value to subtract from the atomic variable + * @param mmodel Memory model associated with the subtract operation + */ +static inline void odp_atomic_u32_sub_mm(odp_atomic_u32_t *ptr, + uint32_t val, + odp_memmodel_t mmodel) + +{ + (void)__atomic_fetch_sub(&ptr->v, val, mmodel); +} + +/***************************************************************************** + * Operations on 64-bit atomics + * odp_atomic_u64_load_mm - return current value + * odp_atomic_u64_store_mm - no return value + * odp_atomic_u64_xchg_mm - return old value + * odp_atomic_u64_cmp_xchg_strong_mm - return bool + * odp_atomic_u64_fetch_add_mm - return old value + * odp_atomic_u64_add_mm - no return value + * odp_atomic_u64_fetch_sub_mm - return old value + * odp_atomic_u64_sub_mm - no return value + *****************************************************************************/ + +/* Check if the compiler support lock-less atomic operations on 64-bit types */ +#if defined __GCC_ATOMIC_LLONG_LOCK_FREE && __GCC_ATOMIC_LLONG_LOCK_FREE < 2 +/** + * @internal + * Helper macro for lock-based atomic operations on 64-bit integers + * @param ptr Pointer to the 64-bit atomic variable + * @param expr Expression used update the variable. + * @param mm Memory model to use. + * @return The old value of the variable. + */ +#define ATOMIC_OP_MM(ptr, expr, mm) \ +({ \ + uint64_t old_val; \ + /* Loop while lock is already taken, stop when lock becomes clear */ \ + while (__atomic_test_and_set(&(ptr)->lock, \ + (mm) == ODP_MEMMODEL_SC ? \ + __ATOMIC_SEQ_CST : __ATOMIC_ACQUIRE)) \ + (void)0; \ + old_val = (ptr)->v; \ + (expr); /* Perform whatever update is desired */ \ + __atomic_clear(&(ptr)->lock, \ + (mm) == ODP_MEMMODEL_SC ? \ + __ATOMIC_SEQ_CST : __ATOMIC_RELEASE); \ + old_val; /* Return old value */ \ +}) +#endif + +/** + * Atomic load of 64-bit atomic variable + * + * @param ptr Pointer to a 64-bit atomic variable + * @param mmodel Memory model associated with the load operation + * + * @return Value of the variable + */ +static inline uint64_t odp_atomic_u64_load_mm(odp_atomic_u64_t *ptr, + odp_memmodel_t mmodel) +{ +#if defined __GCC_ATOMIC_LLONG_LOCK_FREE && __GCC_ATOMIC_LLONG_LOCK_FREE < 2 + return ATOMIC_OP_MM(ptr, (void)0, mmodel); +#else + return __atomic_load_n(&ptr->v, mmodel); +#endif +} + +/** + * Atomic store to 64-bit atomic variable + * + * @param ptr Pointer to a 64-bit atomic variable + * @param val Value to write to the atomic variable + * @param mmodel Memory model associated with the store operation + */ +static inline void odp_atomic_u64_store_mm(odp_atomic_u64_t *ptr, + uint64_t val, + odp_memmodel_t mmodel) +{ +#if defined __GCC_ATOMIC_LLONG_LOCK_FREE && __GCC_ATOMIC_LLONG_LOCK_FREE < 2 + (void)ATOMIC_OP_MM(ptr, ptr->v = val, mmodel); +#else + __atomic_store_n(&ptr->v, val, mmodel); +#endif +} + +/** + * Atomic exchange (swap) of 64-bit atomic variable + * + * @param ptr Pointer to a 64-bit atomic variable + * @param val New value to write to the atomic variable + * @param mmodel Memory model associated with the exchange operation + * + * @return Old value of variable + */ +static inline uint64_t odp_atomic_u64_xchg_mm(odp_atomic_u64_t *ptr, + uint64_t val, + odp_memmodel_t mmodel) + +{ +#if defined __GCC_ATOMIC_LLONG_LOCK_FREE && __GCC_ATOMIC_LLONG_LOCK_FREE < 2 + return ATOMIC_OP_MM(ptr, ptr->v = val, mmodel); +#else + return __atomic_exchange_n(&ptr->v, val, mmodel); +#endif +} + +/** + * Atomic compare and exchange (swap) of 64-bit atomic variable + * "Strong" semantics, will not fail spuriously. + * + * @param ptr Pointer to a 64-bit atomic variable + * @param exp_p Pointer to expected value (updated on failure) + * @param val New value to write + * @param succ Memory model associated with a successful compare-and-swap + * operation + * @param fail Memory model associated with a failed compare-and-swap operation + * + * @return 1 (true) if exchange successful, 0 (false) if not successful (and + * '*exp_p' updated with current value) + */ +static inline int odp_atomic_u64_cmp_xchg_strong_mm(odp_atomic_u64_t *ptr, + uint64_t *exp_p, + uint64_t val, + odp_memmodel_t succ, + odp_memmodel_t fail) +{ +#if defined __GCC_ATOMIC_LLONG_LOCK_FREE && __GCC_ATOMIC_LLONG_LOCK_FREE < 2 + /* Possibly we are a bit pessimistic with the memory models */ + int success; + /* Loop while lock is already taken, stop when lock becomes clear */ + while (__atomic_test_and_set(&(ptr)->lock, + (succ) == ODP_MEMMODEL_SC ? + __ATOMIC_SEQ_CST : __ATOMIC_ACQUIRE)) + (void)0; + if (ptr->v == *exp_p) { + ptr->v = val; + success = 1; + } else { + *exp_p = ptr->v; + success = 0; + succ = fail; + } + __atomic_clear(&(ptr)->lock, + (succ) == ODP_MEMMODEL_SC ? + __ATOMIC_SEQ_CST : __ATOMIC_RELEASE); + return success; +#else + return __atomic_compare_exchange_n(&ptr->v, exp_p, val, + false, succ, fail); +#endif +} + +/** + * Atomic fetch and add of 64-bit atomic variable + * + * @param ptr Pointer to a 64-bit atomic variable + * @param val Value to add to the atomic variable + * @param mmodel Memory model associated with the add operation + * + * @return Value of the atomic variable before the addition + */ +static inline uint64_t odp_atomic_u64_fetch_add_mm(odp_atomic_u64_t *ptr, + uint64_t val, + odp_memmodel_t mmodel) +{ +#if defined __GCC_ATOMIC_LLONG_LOCK_FREE && __GCC_ATOMIC_LLONG_LOCK_FREE < 2 + return ATOMIC_OP_MM(ptr, ptr->v += val, mmodel); +#else + return __atomic_fetch_add(&ptr->v, val, mmodel); +#endif +} + +/** + * Atomic add of 64-bit atomic variable + * + * @param ptr Pointer to a 64-bit atomic variable + * @param val Value to add to the atomic variable + * @param mmodel Memory model associated with the add operation. + */ +static inline void odp_atomic_u64_add_mm(odp_atomic_u64_t *ptr, + uint64_t val, + odp_memmodel_t mmodel) + +{ +#if defined __GCC_ATOMIC_LLONG_LOCK_FREE && __GCC_ATOMIC_LLONG_LOCK_FREE < 2 + (void)ATOMIC_OP_MM(ptr, ptr->v += val, mmodel); +#else + (void)__atomic_fetch_add(&ptr->v, val, mmodel); +#endif +} + +/** + * Atomic fetch and subtract of 64-bit atomic variable + * + * @param ptr Pointer to a 64-bit atomic variable + * @param val Value to subtract from the atomic variable + * @param mmodel Memory model associated with the subtract operation + * + * @return Value of the atomic variable before the subtraction + */ +static inline uint64_t odp_atomic_u64_fetch_sub_mm(odp_atomic_u64_t *ptr, + uint64_t val, + odp_memmodel_t mmodel) +{ +#if defined __GCC_ATOMIC_LLONG_LOCK_FREE && __GCC_ATOMIC_LLONG_LOCK_FREE < 2 + return ATOMIC_OP_MM(ptr, ptr->v -= val, mmodel); +#else + return __atomic_fetch_sub(&ptr->v, val, mmodel); +#endif +} + +/** + * Atomic subtract of 64-bit atomic variable + * + * @param ptr Pointer to a 64-bit atomic variable + * @param val Value to subtract from the atomic variable + * @param mmodel Memory model associated with the subtract operation + */ +static inline void odp_atomic_u64_sub_mm(odp_atomic_u64_t *ptr, + uint64_t val, + odp_memmodel_t mmodel) + +{ +#if defined __GCC_ATOMIC_LLONG_LOCK_FREE && __GCC_ATOMIC_LLONG_LOCK_FREE < 2 + (void)ATOMIC_OP_MM(ptr, ptr->v -= val, mmodel); +#else + (void)__atomic_fetch_sub(&ptr->v, val, mmodel); +#endif +} + +#if defined __GCC_ATOMIC_LLONG_LOCK_FREE && __GCC_ATOMIC_LLONG_LOCK_FREE < 2 +#undef ATOMIC_OP_MM +#endif + +/***************************************************************************** + * Operations on pointer atomics + * odp_atomic_ptr_init - no return value + * odp_atomic_ptr_load - return current value + * odp_atomic_ptr_store - no return value + * odp_atomic_ptr_xchg - return old value + *****************************************************************************/ + +/** + * Initialization of pointer atomic variable + * + * @param ptr Pointer to a pointer atomic variable + * @param val Value to initialize the variable with + */ +static inline void odp_atomic_ptr_init(odp_atomic_ptr_t *ptr, void *val) +{ + __atomic_store_n(&ptr->v, val, __ATOMIC_RELAXED); +} + +/** + * Atomic load of pointer atomic variable + * + * @param ptr Pointer to a pointer atomic variable + * @param mmodel Memory model associated with the load operation + * + * @return Value of the variable + */ +static inline void *odp_atomic_ptr_load(const odp_atomic_ptr_t *ptr, + odp_memmodel_t mmodel) +{ + return __atomic_load_n(&ptr->v, mmodel); +} + +/** + * Atomic store to pointer atomic variable + * + * @param ptr Pointer to a pointer atomic variable + * @param val Value to write to the atomic variable + * @param mmodel Memory model associated with the store operation + */ +static inline void odp_atomic_ptr_store(odp_atomic_ptr_t *ptr, + void *val, + odp_memmodel_t mmodel) +{ + __atomic_store_n(&ptr->v, val, mmodel); +} + +/** + * Atomic exchange (swap) of pointer atomic variable + * + * @param ptr Pointer to a pointer atomic variable + * @param val New value to write + * @param mmodel Memory model associated with the exchange operation + * + * @return Old value of variable + */ +static inline void *odp_atomic_ptr_xchg(odp_atomic_ptr_t *ptr, + void *val, + odp_memmodel_t mmodel) + +{ + return __atomic_exchange_n(&ptr->v, val, mmodel); +} + +/** + * Atomic compare and exchange (swap) of pointer atomic variable + * "Strong" semantics, will not fail spuriously. + * + * @param ptr Pointer to a pointer atomic variable + * @param exp_p Pointer to expected value (updated on failure) + * @param val New value to write + * @param succ Memory model associated with a successful compare-and-swap + * operation + * @param fail Memory model associated with a failed compare-and-swap + * operation + * + * @return 1 (true) if exchange successful, 0 (false) if not successful (and + * '*exp_p' updated with current value) + */ +static inline int odp_atomic_ptr_cmp_xchg_strong(odp_atomic_ptr_t *ptr, + void **exp_p, + void *val, + odp_memmodel_t succ, + odp_memmodel_t fail) +{ + return __atomic_compare_exchange_n(&ptr->v, exp_p, val, + false/*strong*/, succ, fail); +} + +/***************************************************************************** + * Operations on flag atomics + * odp_atomic_flag_init - no return value + * odp_atomic_flag_load - return current value + * odp_atomic_flag_tas - return old value + * odp_atomic_flag_clear - no return value + * + * Flag atomics use Release Consistency model. + *****************************************************************************/ + +/** + * Initialize a flag atomic variable + * + * @param ptr Pointer to a flag atomic variable + * @param val The initial value (true or false) of the variable + */ +static inline void odp_atomic_flag_init(odp_atomic_flag_t *ptr, int val) +{ + __atomic_clear(ptr, __ATOMIC_RELAXED); + if (val) + __atomic_test_and_set(ptr, __ATOMIC_RELAXED); +} + +/** + * Load atomic flag variable + * @Note Operation has relaxed memory ordering. + * + * @param ptr Pointer to a flag atomic variable + * @return The current value (true or false) of the variable + */ +static inline int odp_atomic_flag_load(odp_atomic_flag_t *ptr) +{ + return __atomic_load_n(ptr, __ATOMIC_RELAXED); +} + +/** + * Test-and-set of atomic flag variable + * @Note Operation has acquire memory ordering. It pairs with a later + * release operation in some thread. + * + * @param ptr Pointer to a flag atomic variable + * @return The old value (true or false) of the variable + */ +static inline int odp_atomic_flag_tas(odp_atomic_flag_t *ptr) +{ + return __atomic_test_and_set(ptr, __ATOMIC_ACQUIRE); +} + +/** + * Clear atomic flag variable + * The flag variable is cleared (set to the false value). + * @Note Operation has release memory ordering. It pairs with an earlier + * acquire operation in some thread. + * + * @param ptr Pointer to a flag atomic variable + */ +static inline void odp_atomic_flag_clear(odp_atomic_flag_t *ptr) +{ + __atomic_clear(ptr, __ATOMIC_RELEASE); +} + +/** + * @} + */ + +#ifdef __cplusplus +} +#endif + +#endif