From patchwork Mon Nov 24 22:38:49 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ola Liljedahl X-Patchwork-Id: 41443 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-wg0-f69.google.com (mail-wg0-f69.google.com [74.125.82.69]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 2A1E525E18 for ; Mon, 24 Nov 2014 22:40:02 +0000 (UTC) Received: by mail-wg0-f69.google.com with SMTP id k14sf5839259wgh.0 for ; Mon, 24 Nov 2014 14:40:01 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:from:to:date:message-id:in-reply-to :references:subject:precedence:list-id:list-unsubscribe:list-archive :list-post:list-help:list-subscribe:mime-version:content-type :content-transfer-encoding:errors-to:sender:x-original-sender :x-original-authentication-results:mailing-list; bh=utkqPHhbOHemZE20Kyw0H1zSHnVKjIBXgnIykmnHtEs=; b=dg1fzP2TvuVow0QipFQRbYhxXLvP1VBayxUZmWEgCtlf31/diid7Qo/RUsf9zsxpti RQEW2x6GwuFMLL7BhBdyR8AhlrGRR/quvXyzFFfqTPuOylJsP9YRNs6cTLmTcEtjkA3J K26cdYES1e9A/UCiI5LEV5HMpAl2Jf5z/WGXPJq8vS6cxGaIM+meS6ppFNXq/0cxk3Lf 7N/DGeY3VHpIAcBUlNecA8qTQ75bbfPlKC3WReW83aGnUv2ulKxJSKG/sqWmv4s6g8YJ BP8+y7gwHLUO/8kFisMNZCAeSobIKg7OxZHuw78D5Vo7tXh4TYB+t/SNetQ6f7t5w/P5 Ibiw== X-Gm-Message-State: ALoCoQmIcWt5GPuLgWRZpXOG9KvmGroi928mnkDUpDPHDgrf11zTjAGFOFjTdNChgY2uv6RkpPot X-Received: by 10.180.19.226 with SMTP id i2mr4639620wie.5.1416868801392; Mon, 24 Nov 2014 14:40:01 -0800 (PST) X-BeenThere: patchwork-forward@linaro.org Received: by 10.152.22.72 with SMTP id b8ls536402laf.2.gmail; Mon, 24 Nov 2014 14:40:01 -0800 (PST) X-Received: by 10.112.235.196 with SMTP id uo4mr22559994lbc.66.1416868801087; Mon, 24 Nov 2014 14:40:01 -0800 (PST) Received: from mail-lb0-f169.google.com (mail-lb0-f169.google.com. [209.85.217.169]) by mx.google.com with ESMTPS id b4si16546047lbd.70.2014.11.24.14.40.01 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 24 Nov 2014 14:40:01 -0800 (PST) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.169 as permitted sender) client-ip=209.85.217.169; Received: by mail-lb0-f169.google.com with SMTP id p9so7225240lbv.14 for ; Mon, 24 Nov 2014 14:40:01 -0800 (PST) X-Received: by 10.112.52.37 with SMTP id q5mr22590238lbo.32.1416868800992; Mon, 24 Nov 2014 14:40:00 -0800 (PST) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.112.184.201 with SMTP id ew9csp348219lbc; Mon, 24 Nov 2014 14:39:59 -0800 (PST) X-Received: by 10.140.29.9 with SMTP id a9mr31832554qga.41.1416868798814; Mon, 24 Nov 2014 14:39:58 -0800 (PST) Received: from ip-10-35-177-41.ec2.internal (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTPS id w2si17439440qab.9.2014.11.24.14.39.55 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Mon, 24 Nov 2014 14:39:58 -0800 (PST) Received-SPF: none (google.com: lng-odp-bounces@lists.linaro.org does not designate permitted sender hosts) client-ip=54.225.227.206; Received: from localhost ([127.0.0.1] helo=ip-10-35-177-41.ec2.internal) by ip-10-35-177-41.ec2.internal with esmtp (Exim 4.76) (envelope-from ) id 1Xt2IU-0007tW-Cv; Mon, 24 Nov 2014 22:39:54 +0000 Received: from mail-lb0-f178.google.com ([209.85.217.178]) by ip-10-35-177-41.ec2.internal with esmtp (Exim 4.76) (envelope-from ) id 1Xt2Hn-0007mH-Np for lng-odp@lists.linaro.org; Mon, 24 Nov 2014 22:39:12 +0000 Received: by mail-lb0-f178.google.com with SMTP id f15so7447915lbj.37 for ; Mon, 24 Nov 2014 14:39:06 -0800 (PST) X-Received: by 10.152.5.6 with SMTP id o6mr23154967lao.8.1416868746212; Mon, 24 Nov 2014 14:39:06 -0800 (PST) Received: from macmini.lan (84-217-193-77.tn.glocalnet.net. [84.217.193.77]) by mx.google.com with ESMTPSA id pr9sm2952957lbc.12.2014.11.24.14.39.04 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 24 Nov 2014 14:39:05 -0800 (PST) From: Ola Liljedahl To: lng-odp@lists.linaro.org Date: Mon, 24 Nov 2014 23:38:49 +0100 Message-Id: <1416868730-26140-9-git-send-email-ola.liljedahl@linaro.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1416868730-26140-1-git-send-email-ola.liljedahl@linaro.org> References: <1416868730-26140-1-git-send-email-ola.liljedahl@linaro.org> X-Topics: patch Subject: [lng-odp] [PATCH v3 8/9] api: odp_atomic.h: struct type, relaxed mm, missing funcs, use __atomic X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: , List-Help: , List-Subscribe: , MIME-Version: 1.0 Errors-To: lng-odp-bounces@lists.linaro.org Sender: lng-odp-bounces@lists.linaro.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: ola.liljedahl@linaro.org X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.169 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 Signed-off-by: Ola Liljedahl --- (Tried to break up this patch but "git add -p" couldn't separate the different changes from each other and my git-fu isn't strong enough yet). odp_atomic.h: Use struct for 32- and 64-bit atomic types, this prevents non-atomic access to atomic variables. Emulated 64-bit atomic support for functionally challenged 32-bit architectures (e.g. PPC32). Relaxed memory model for atomic operations, this enhances performance on weakly ordered architectures (e.g. most RISC). Use GCC __atomic builtins (previously __sync builtins). Added missing atomic_add_u32, atomic_sub_u32 and atomic_sub_64 operations. Removed 32- and 64-bit cmpset operations (will reappear in odp_atomic_internal.h). Made some 64-bit operations (e.g. load, store) atomic on 32-bit architectures. odp_atomic_init_u32() and odp_atomic_init_u64() now take initialization value as parameter. odp_barrier.c: Use new odp_atomic_init_xxx() prototype. Use odp_atomic.h functions to access atomic variables. Fixed non-atomic wraparound (using new odp_atomic_sub_u32() function). odp_generator.c: Use new odp_atomic_init_xxx() prototype. odp_ticketlock.c: Use new odp_atomic_init_xxx() prototype. odp_atomic_text.c: Use new odp_atomic_init_xxx() prototype. example/generator/odp_generator.c | 8 +- platform/linux-generic/include/api/odp_atomic.h | 271 +++++++++++++++--------- platform/linux-generic/odp_barrier.c | 13 +- platform/linux-generic/odp_ticketlock.c | 3 +- test/api_test/odp_atomic_test.c | 4 +- 5 files changed, 181 insertions(+), 118 deletions(-) diff --git a/example/generator/odp_generator.c b/example/generator/odp_generator.c index 7d1c237..73b0369 100644 --- a/example/generator/odp_generator.c +++ b/example/generator/odp_generator.c @@ -540,10 +540,10 @@ int main(int argc, char *argv[]) } /* init counters */ - odp_atomic_init_u64(&counters.seq); - odp_atomic_init_u64(&counters.ip); - odp_atomic_init_u64(&counters.udp); - odp_atomic_init_u64(&counters.icmp); + odp_atomic_init_u64(&counters.seq, 0); + odp_atomic_init_u64(&counters.ip, 0); + odp_atomic_init_u64(&counters.udp, 0); + odp_atomic_init_u64(&counters.icmp, 0); /* Reserve memory for args from shared mem */ shm = odp_shm_reserve("shm_args", sizeof(args_t), diff --git a/platform/linux-generic/include/api/odp_atomic.h b/platform/linux-generic/include/api/odp_atomic.h index 5c83b39..de9d91c 100644 --- a/platform/linux-generic/include/api/odp_atomic.h +++ b/platform/linux-generic/include/api/odp_atomic.h @@ -18,11 +18,12 @@ extern "C" { #endif - -#include +#include +#include /** @addtogroup odp_synchronizers - * Atomic operations. + * Atomic types and relaxed operations. These operations cannot be used for + * synchronization. * @{ */ @@ -30,56 +31,67 @@ extern "C" { /** * Atomic unsigned integer 64 bits */ -typedef volatile uint64_t odp_atomic_u64_t; +typedef struct { + uint64_t v; /**< Actual storage for the atomic variable */ +#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 + /* Some architectures do not support lock-free operations on 64-bit + * data types. We use a spin lock to ensure atomicity. */ + char lock; +#endif +} odp_atomic_u64_t +ODP_ALIGNED(sizeof(uint64_t)); /* Enforce alignement! */ + /** * Atomic unsigned integer 32 bits */ -typedef volatile uint32_t odp_atomic_u32_t; +typedef struct { + uint32_t v; /**< Actual storage for the atomic variable */ +} odp_atomic_u32_t +ODP_ALIGNED(sizeof(uint32_t)); /* Enforce alignement! */ /** * Initialize atomic uint32 + * @note Relaxed memory order, cannot be used for synchronization * * @param ptr An atomic variable - * - * @note The operation is not synchronized with other threads + * @param val Value to initialize the variable with */ -static inline void odp_atomic_init_u32(odp_atomic_u32_t *ptr) +static inline void odp_atomic_init_u32(odp_atomic_u32_t *ptr, uint32_t val) { - *ptr = 0; + __atomic_store_n(&ptr->v, val, __ATOMIC_RELAXED); } /** * Load value of atomic uint32 + * @note Relaxed memory order, cannot be used for synchronization * * @param ptr An atomic variable * - * @return atomic uint32 value - * - * @note The operation is not synchronized with other threads + * @return Value of the variable */ static inline uint32_t odp_atomic_load_u32(odp_atomic_u32_t *ptr) { - return *ptr; + return __atomic_load_n(&ptr->v, __ATOMIC_RELAXED); } /** * Store value to atomic uint32 + * @note Relaxed memory order, cannot be used for synchronization * * @param ptr An atomic variable * @param new_value Store new_value to a variable - * - * @note The operation is not synchronized with other threads */ static inline void odp_atomic_store_u32(odp_atomic_u32_t *ptr, uint32_t new_value) { - *ptr = new_value; + __atomic_store_n(&ptr->v, new_value, __ATOMIC_RELAXED); } /** * Fetch and add atomic uint32 + * @note Relaxed memory order, cannot be used for synchronization * * @param ptr An atomic variable * @param value A value to be added to the variable @@ -89,11 +101,25 @@ static inline void odp_atomic_store_u32(odp_atomic_u32_t *ptr, static inline uint32_t odp_atomic_fetch_add_u32(odp_atomic_u32_t *ptr, uint32_t value) { - return __sync_fetch_and_add(ptr, value); + return __atomic_fetch_add(&ptr->v, value, __ATOMIC_RELAXED); +} + +/** + * Add atomic uint32 + * @note Relaxed memory order, cannot be used for synchronization + * + * @param ptr An atomic variable + * @param value A value to be added to the variable + */ +static inline void odp_atomic_add_u32(odp_atomic_u32_t *ptr, + uint32_t value) +{ + (void)__atomic_fetch_add(&ptr->v, value, __ATOMIC_RELAXED); } /** * Fetch and subtract uint32 + * @note Relaxed memory order, cannot be used for synchronization * * @param ptr An atomic variable * @param value A value to be sub to the variable @@ -103,51 +129,59 @@ static inline uint32_t odp_atomic_fetch_add_u32(odp_atomic_u32_t *ptr, static inline uint32_t odp_atomic_fetch_sub_u32(odp_atomic_u32_t *ptr, uint32_t value) { - return __sync_fetch_and_sub(ptr, value); + return __atomic_fetch_sub(&ptr->v, value, __ATOMIC_RELAXED); +} + +/** + * Subtract uint32 + * @note Relaxed memory order, cannot be used for synchronization + * + * @param ptr An atomic variable + * @param value A value to be subtract from the variable + */ +static inline void odp_atomic_sub_u32(odp_atomic_u32_t *ptr, + uint32_t value) +{ + (void)__atomic_fetch_sub(&ptr->v, value, __ATOMIC_RELAXED); } /** * Fetch and increment atomic uint32 by 1 + * @note Relaxed memory order, cannot be used for synchronization * * @param ptr An atomic variable * * @return Value of the variable before the operation */ -#if defined __OCTEON__ static inline uint32_t odp_atomic_fetch_inc_u32(odp_atomic_u32_t *ptr) { +#if defined __OCTEON__ uint32_t ret; - __asm__ __volatile__ ("syncws"); __asm__ __volatile__ ("lai %0,(%2)" : "=r" (ret), "+m" (ptr) : "r" (ptr)); - return ret; -} - #else - -static inline uint32_t odp_atomic_fetch_inc_u32(odp_atomic_u32_t *ptr) -{ - return odp_atomic_fetch_add_u32(ptr, 1); -} - + return __atomic_fetch_add(&ptr->v, 1, __ATOMIC_RELAXED); #endif +} /** * Increment atomic uint32 by 1 + * @note Relaxed memory order, cannot be used for synchronization * * @param ptr An atomic variable * */ static inline void odp_atomic_inc_u32(odp_atomic_u32_t *ptr) { - odp_atomic_fetch_add_u32(ptr, 1); + (void)__atomic_fetch_add(&ptr->v, 1, __ATOMIC_RELAXED); } /** * Fetch and decrement uint32 by 1 + * @note Relaxed memory order, cannot be used for synchronization * * @param ptr An atomic variable * @@ -155,62 +189,77 @@ static inline void odp_atomic_inc_u32(odp_atomic_u32_t *ptr) */ static inline uint32_t odp_atomic_fetch_dec_u32(odp_atomic_u32_t *ptr) { - return odp_atomic_fetch_sub_u32(ptr, 1); + return __atomic_fetch_sub(&ptr->v, 1, __ATOMIC_RELAXED); } /** * Decrement atomic uint32 by 1 + * @note Relaxed memory order, cannot be used for synchronization * * @param ptr An atomic variable * */ static inline void odp_atomic_dec_u32(odp_atomic_u32_t *ptr) { - odp_atomic_fetch_sub_u32(ptr, 1); + (void)__atomic_fetch_sub(&ptr->v, 1, __ATOMIC_RELAXED); } /** - * Atomic compare and set for 32bit + * Initialize atomic uint64 + * @note Relaxed memory order, cannot be used for synchronization * - * @param dst destination location into which the value will be written. - * @param exp expected value. - * @param src new value. - * @return Non-zero on success; 0 on failure. + * @param ptr An atomic variable + * @param val Value to initialize the variable with */ -static inline int -odp_atomic_cmpset_u32(odp_atomic_u32_t *dst, uint32_t exp, uint32_t src) +static inline void odp_atomic_init_u64(odp_atomic_u64_t *ptr, uint64_t val) { - return __sync_bool_compare_and_swap(dst, exp, src); + ptr->v = val; +#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 + __atomic_clear(&ptr->lock, __ATOMIC_RELAXED); +#endif } +#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 /** - * Initialize atomic uint64 - * - * @param ptr An atomic variable - * - * @note The operation is not synchronized with other threads + * @internal + * Helper macro for lock-based atomic operations on 64-bit integers + * @param ptr Pointer to the 64-bit atomic variable + * @param expr Expression used update the variable. + * @return The old value of the variable. */ -static inline void odp_atomic_init_u64(odp_atomic_u64_t *ptr) -{ - *ptr = 0; -} +#define ATOMIC_OP(ptr, expr) \ +({ \ + uint64_t old_val; \ + /* Loop while lock is already taken, stop when lock becomes clear */ \ + while (__atomic_test_and_set(&(ptr)->lock, __ATOMIC_ACQUIRE)) \ + (void)0; \ + old_val = (ptr)->v; \ + (expr); /* Perform whatever update is desired */ \ + __atomic_clear(&(ptr)->lock, __ATOMIC_RELEASE); \ + old_val; /* Return old value */ \ +}) +#endif /** * Load value of atomic uint64 + * @note Relaxed memory order, cannot be used for synchronization * * @param ptr An atomic variable * * @return atomic uint64 value - * - * @note The operation is not synchronized with other threads */ static inline uint64_t odp_atomic_load_u64(odp_atomic_u64_t *ptr) { - return *ptr; +#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 + return ATOMIC_OP(ptr, (void)0); +#else + return __atomic_load_n(&ptr->v, __ATOMIC_RELAXED); +#endif } /** * Store value to atomic uint64 + * @note Relaxed memory order, cannot be used for synchronization * * @param ptr An atomic variable * @param new_value Store new_value to a variable @@ -220,23 +269,16 @@ static inline uint64_t odp_atomic_load_u64(odp_atomic_u64_t *ptr) static inline void odp_atomic_store_u64(odp_atomic_u64_t *ptr, uint64_t new_value) { - *ptr = new_value; -} - -/** - * Add atomic uint64 - * - * @param ptr An atomic variable - * @param value A value to be added to the variable - * - */ -static inline void odp_atomic_add_u64(odp_atomic_u64_t *ptr, uint64_t value) -{ - __sync_fetch_and_add(ptr, value); +#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 + (void)ATOMIC_OP(ptr, ptr->v = new_value); +#else + __atomic_store_n(&ptr->v, new_value, __ATOMIC_RELAXED); +#endif } /** * Fetch and add atomic uint64 + * @note Relaxed memory order, cannot be used for synchronization * * @param ptr An atomic variable * @param value A value to be added to the variable @@ -244,56 +286,72 @@ static inline void odp_atomic_add_u64(odp_atomic_u64_t *ptr, uint64_t value) * @return Value of the variable before the operation */ -#if defined __powerpc__ && !defined __powerpc64__ static inline uint64_t odp_atomic_fetch_add_u64(odp_atomic_u64_t *ptr, uint64_t value) { - return __sync_fetch_and_add((odp_atomic_u32_t *)ptr, - (uint32_t)value); -} +#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 + return ATOMIC_OP(ptr, ptr->v += value); #else -static inline uint64_t odp_atomic_fetch_add_u64(odp_atomic_u64_t *ptr, - uint64_t value) -{ - return __sync_fetch_and_add(ptr, value); -} + return __atomic_fetch_add(&ptr->v, value, __ATOMIC_RELAXED); #endif +} + /** - * Subtract atomic uint64 + * Add atomic uint64 + * @note Relaxed memory order, cannot be used for synchronization * * @param ptr An atomic variable - * @param value A value to be subtracted from the variable + * @param value A value to be added to the variable * */ -static inline void odp_atomic_sub_u64(odp_atomic_u64_t *ptr, uint64_t value) +static inline void odp_atomic_add_u64(odp_atomic_u64_t *ptr, uint64_t value) { - __sync_fetch_and_sub(ptr, value); +#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 + (void)ATOMIC_OP(ptr, ptr->v += value); +#else + (void)__atomic_fetch_add(&ptr->v, value, __ATOMIC_RELAXED); +#endif } /** * Fetch and subtract atomic uint64 + * @note Relaxed memory order, cannot be used for synchronization * * @param ptr An atomic variable * @param value A value to be subtracted from the variable * * @return Value of the variable before the operation */ -#if defined __powerpc__ && !defined __powerpc64__ static inline uint64_t odp_atomic_fetch_sub_u64(odp_atomic_u64_t *ptr, uint64_t value) { - return __sync_fetch_and_sub((odp_atomic_u32_t *)ptr, - (uint32_t)value); -} +#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 + return ATOMIC_OP(ptr, ptr->v -= value); #else -static inline uint64_t odp_atomic_fetch_sub_u64(odp_atomic_u64_t *ptr, - uint64_t value) -{ - return __sync_fetch_and_sub(ptr, value); + return __atomic_fetch_sub(&ptr->v, value, __ATOMIC_RELAXED); +#endif } + +/** + * Subtract atomic uint64 + * @note Relaxed memory order, cannot be used for synchronization + * + * @param ptr An atomic variable + * @param value A value to be subtracted from the variable + * + */ +static inline void odp_atomic_sub_u64(odp_atomic_u64_t *ptr, uint64_t value) +{ +#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 + (void)ATOMIC_OP(ptr, ptr->v -= value); +#else + (void)__atomic_fetch_sub(&ptr->v, value, __ATOMIC_RELAXED); #endif +} + /** * Fetch and increment atomic uint64 by 1 + * @note Relaxed memory order, cannot be used for synchronization * * @param ptr An atomic variable * @@ -301,22 +359,32 @@ static inline uint64_t odp_atomic_fetch_sub_u64(odp_atomic_u64_t *ptr, */ static inline uint64_t odp_atomic_fetch_inc_u64(odp_atomic_u64_t *ptr) { - return odp_atomic_fetch_add_u64(ptr, 1); +#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 + return ATOMIC_OP(ptr, ptr->v++); +#else + return __atomic_fetch_add(&ptr->v, 1, __ATOMIC_RELAXED); +#endif } /** * Increment atomic uint64 by 1 + * @note Relaxed memory order, cannot be used for synchronization * * @param ptr An atomic variable * */ static inline void odp_atomic_inc_u64(odp_atomic_u64_t *ptr) { - odp_atomic_fetch_add_u64(ptr, 1); +#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 + (void)ATOMIC_OP(ptr, ptr->v++); +#else + (void)__atomic_fetch_add(&ptr->v, 1, __ATOMIC_RELAXED); +#endif } /** * Fetch and decrement atomic uint64 by 1 + * @note Relaxed memory order, cannot be used for synchronization * * @param ptr An atomic variable * @@ -324,32 +392,27 @@ static inline void odp_atomic_inc_u64(odp_atomic_u64_t *ptr) */ static inline uint64_t odp_atomic_fetch_dec_u64(odp_atomic_u64_t *ptr) { - return odp_atomic_fetch_sub_u64(ptr, 1); +#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 + return ATOMIC_OP(ptr, ptr->v--); +#else + return __atomic_fetch_sub(&ptr->v, 1, __ATOMIC_RELAXED); +#endif } /** * Decrement atomic uint64 by 1 + * @note Relaxed memory order, cannot be used for synchronization * * @param ptr An atomic variable * */ static inline void odp_atomic_dec_u64(odp_atomic_u64_t *ptr) { - odp_atomic_fetch_sub_u64(ptr, 1); -} - -/** - * Atomic compare and set for 64bit - * - * @param dst destination location into which the value will be written. - * @param exp expected value. - * @param src new value. - * @return Non-zero on success; 0 on failure. - */ -static inline int -odp_atomic_cmpset_u64(odp_atomic_u64_t *dst, uint64_t exp, uint64_t src) -{ - return __sync_bool_compare_and_swap(dst, exp, src); +#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 + (void)ATOMIC_OP(ptr, ptr->v--); +#else + (void)__atomic_fetch_sub(&ptr->v, 1, __ATOMIC_RELAXED); +#endif } /** diff --git a/platform/linux-generic/odp_barrier.c b/platform/linux-generic/odp_barrier.c index de1bf60..3a511b2 100644 --- a/platform/linux-generic/odp_barrier.c +++ b/platform/linux-generic/odp_barrier.c @@ -10,9 +10,8 @@ void odp_barrier_init(odp_barrier_t *barrier, int count) { - barrier->count = count; - barrier->bar = 0; - odp_sync_stores(); + barrier->count = (uint32_t)count; + odp_atomic_init_u32(&barrier->bar, 0); } /* @@ -33,14 +32,16 @@ void odp_barrier_wait(odp_barrier_t *barrier) uint32_t count; int wasless; - wasless = barrier->bar < barrier->count; __atomic_thread_fence(__ATOMIC_SEQ_CST); count = odp_atomic_fetch_inc_u32(&barrier->bar); + wasless = count < barrier->count; if (count == 2*barrier->count-1) { - barrier->bar = 0; + /* Wrap around *atomically* */ + odp_atomic_sub_u32(&barrier->bar, 2 * barrier->count); } else { - while ((barrier->bar < barrier->count) == wasless) + while ((odp_atomic_load_u32(&barrier->bar) < barrier->count) + == wasless) odp_spin(); } diff --git a/platform/linux-generic/odp_ticketlock.c b/platform/linux-generic/odp_ticketlock.c index 096bdc0..e60f526 100644 --- a/platform/linux-generic/odp_ticketlock.c +++ b/platform/linux-generic/odp_ticketlock.c @@ -12,9 +12,8 @@ void odp_ticketlock_init(odp_ticketlock_t *ticketlock) { - ticketlock->next_ticket = 0; + odp_atomic_init_u32(&ticketlock->next_ticket, 0); ticketlock->cur_ticket = 0; - odp_sync_stores(); } diff --git a/test/api_test/odp_atomic_test.c b/test/api_test/odp_atomic_test.c index 553c447..dcf2c59 100644 --- a/test/api_test/odp_atomic_test.c +++ b/test/api_test/odp_atomic_test.c @@ -152,8 +152,8 @@ void test_atomic_basic(void) void test_atomic_init(void) { - odp_atomic_init_u32(&a32u); - odp_atomic_init_u64(&a64u); + odp_atomic_init_u32(&a32u, 0); + odp_atomic_init_u64(&a64u, 0); } void test_atomic_store(void)