From patchwork Tue Oct 25 00:27:52 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brian Brooks X-Patchwork-Id: 80283 Delivered-To: patch@linaro.org Received: by 10.140.97.247 with SMTP id m110csp333790qge; Mon, 31 Oct 2016 14:51:59 -0700 (PDT) X-Received: by 10.237.56.40 with SMTP id j37mr16905850qte.134.1477950719710; Mon, 31 Oct 2016 14:51:59 -0700 (PDT) Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id w69si18889161qkw.146.2016.10.31.14.51.59; Mon, 31 Oct 2016 14:51:59 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=pass (p=NONE dis=NONE) header.from=linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 67323617D6; Mon, 31 Oct 2016 21:51:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: * X-Spam-Status: No, score=1.5 required=5.0 tests=BAYES_00, DATE_IN_PAST_96_XX, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id B8C6B617D9; Mon, 31 Oct 2016 21:51:02 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id DD6FF617D6; Mon, 31 Oct 2016 21:50:54 +0000 (UTC) Received: from mail-oi0-f42.google.com (mail-oi0-f42.google.com [209.85.218.42]) by lists.linaro.org (Postfix) with ESMTPS id BAB7861787 for ; Mon, 31 Oct 2016 21:50:49 +0000 (UTC) Received: by mail-oi0-f42.google.com with SMTP id 62so133302623oif.1 for ; Mon, 31 Oct 2016 14:50:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=qIxaKOEHaI4BCmRe0rYO12oQQcXws6a6zYBamjfhgkc=; b=hLoJJ+t5c7/vxkeRMOHLZn0k7EYzHrY5orEtdBcUM3gyvh7DFvBBwJHUdJznFL2Dpm Z3IT1DlpRSjf3wR1hcv4DFLpkgvrkRuD+x8OqSWBfdTjq7DMP0smYaqnxApbqVeMxEGS MVzWpFz6xESwoM5X5pmp+E134TovXQBMRNpLcQwjJPfOf03Jf8n0OaRYCntX+bNdz/K0 A6bOJzEDCHspOZWcfguNFZ4BZUbtedm5Z2HucA8SlPtfykHAd7mmX6qZCjqnWCj6K216 4pzfDGUENp8zDOOamYU1+zhCTWhbTSbciaGRgeYiOkazPpvsin9gnl5lwN/kC+ZRT5Jv qtUg== X-Gm-Message-State: ABUngvfZs+lBKfTwVcT4dneUoGEzxFVMJyVM0oQVHA3jirhHS8NXp/PIoHWXrimryuqsXxd4x8U= X-Received: by 10.202.92.197 with SMTP id q188mr29422066oib.31.1477950648883; Mon, 31 Oct 2016 14:50:48 -0700 (PDT) Received: from vm-arch-01.localdomain (68-248-140-212.lightspeed.austtx.sbcglobal.net. [68.248.140.212]) by smtp.gmail.com with ESMTPSA id w22sm177107otw.31.2016.10.31.14.50.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 31 Oct 2016 14:50:48 -0700 (PDT) From: Brian Brooks To: lng-odp@lists.linaro.org Date: Mon, 24 Oct 2016 19:27:52 -0500 Message-Id: <20161025002752.11939-4-brian.brooks@linaro.org> X-Mailer: git-send-email 2.10.1 In-Reply-To: <20161025002752.11939-1-brian.brooks@linaro.org> References: <20161025002752.11939-1-brian.brooks@linaro.org> X-Topics: timers Subject: [lng-odp] [RFC 4/4] timers: poll-mode timers X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" Remove the use of POSIX timers and one background thread per timer pool by running timer expiration processing directly inside odp_schedule(). odp_timers stress test shows improvements for test cases with higher resolution and lower latency timers. For further details please see: https://docs.google.com/a/linaro.org/document/d/1AI0TFlIb3QJFAd3mJz74kPzMrLmiQgwmuOx31LlffZA/edit?usp=sharing Signed-off-by: Brian Brooks --- .../linux-generic/include/odp_config_internal.h | 3 + platform/linux-generic/include/odp_time_internal.h | 2 + .../linux-generic/include/odp_timer_internal.h | 2 + platform/linux-generic/odp_schedule.c | 35 +++-- platform/linux-generic/odp_time.c | 57 ++++++- platform/linux-generic/odp_timer.c | 173 ++++----------------- 6 files changed, 109 insertions(+), 163 deletions(-) -- 2.10.1 diff --git a/platform/linux-generic/include/odp_config_internal.h b/platform/linux-generic/include/odp_config_internal.h index b7ff610..eec9dac 100644 --- a/platform/linux-generic/include/odp_config_internal.h +++ b/platform/linux-generic/include/odp_config_internal.h @@ -118,6 +118,9 @@ extern "C" { */ #define CONFIG_BURST_SIZE 16 +/* Value used to rate limit timer pool expiration processing. */ +#define ODP_CONFIG_TIMER_RUN_NSEC (250) + #ifdef __cplusplus } #endif diff --git a/platform/linux-generic/include/odp_time_internal.h b/platform/linux-generic/include/odp_time_internal.h index 5a0bc75..1185f58 100644 --- a/platform/linux-generic/include/odp_time_internal.h +++ b/platform/linux-generic/include/odp_time_internal.h @@ -24,4 +24,6 @@ static inline uint64_t core_tick(void) #endif } +uint64_t core_tick_diff_ns(uint64_t before, uint64_t after); + #endif diff --git a/platform/linux-generic/include/odp_timer_internal.h b/platform/linux-generic/include/odp_timer_internal.h index b1cd73f..51959c0 100644 --- a/platform/linux-generic/include/odp_timer_internal.h +++ b/platform/linux-generic/include/odp_timer_internal.h @@ -39,4 +39,6 @@ typedef struct odp_timeout_hdr_stride { uint8_t pad[ODP_CACHE_LINE_SIZE_ROUNDUP(sizeof(odp_timeout_hdr_t))]; } odp_timeout_hdr_stride; +int timer_run(void); + #endif diff --git a/platform/linux-generic/odp_schedule.c b/platform/linux-generic/odp_schedule.c index 81e79c9..17cfaea 100644 --- a/platform/linux-generic/odp_schedule.c +++ b/platform/linux-generic/odp_schedule.c @@ -22,6 +22,8 @@ #include #include #include +#include +#include #include /* Number of priority levels */ @@ -788,19 +790,28 @@ static int do_schedule(odp_queue_t *out_queue, odp_event_t out_ev[], return 0; } +static __thread uint64_t last_timer_run; static int schedule_loop(odp_queue_t *out_queue, uint64_t wait, odp_event_t out_ev[], unsigned int max_num) { - odp_time_t next, wtime; - int first = 1; - int ret; + uint64_t start, now; + int nr_events; + + start = core_tick(); + + for (;;) { + now = core_tick(); + if (ODP_CONFIG_TIMER_RUN_NSEC <= + core_tick_diff_ns(last_timer_run, now)) { + last_timer_run = now; + (void)timer_run(); + } - while (1) { - ret = do_schedule(out_queue, out_ev, max_num); + nr_events = do_schedule(out_queue, out_ev, max_num); - if (ret) + if (nr_events) break; if (wait == ODP_SCHED_WAIT) @@ -809,18 +820,12 @@ static int schedule_loop(odp_queue_t *out_queue, uint64_t wait, if (wait == ODP_SCHED_NO_WAIT) break; - if (first) { - wtime = odp_time_local_from_ns(wait); - next = odp_time_sum(odp_time_local(), wtime); - first = 0; - continue; - } - - if (odp_time_cmp(next, odp_time_local()) < 0) + now = core_tick(); + if (core_tick_diff_ns(start, now) >= wait) break; } - return ret; + return nr_events; } static odp_event_t schedule(odp_queue_t *out_queue, uint64_t wait) diff --git a/platform/linux-generic/odp_time.c b/platform/linux-generic/odp_time.c index 81e0522..cd5c2a8 100644 --- a/platform/linux-generic/odp_time.c +++ b/platform/linux-generic/odp_time.c @@ -6,13 +6,65 @@ #include -#include +#include #include #include #include +#include + +#include +#include +#include + +/* Number of nanoseconds per second. */ +#define ODP_NSEC_PER_SEC (ODP_TIME_SEC_IN_NS) + +static uint64_t core_tick_hz; static odp_time_t start_time; +uint64_t core_tick_diff_ns(uint64_t before, uint64_t after) +{ + uint64_t diff = after - before; + + return llround(((double)diff / (double)core_tick_hz) * + ODP_NSEC_PER_SEC); +} + +static bool core_tick_init(void) +{ +#if defined(__aarch64__) + uint64_t hz; + __asm__ volatile("mrs %0, cntfrq_el0" : "=r"(hz)); + core_tick_hz = hz; + return true; +#elif defined(__x86_64__) + uint32_t cpu_info[4]; + + /* Check for Invariant TSC. */ + __asm__ volatile("cpuid" : + "=a"(cpu_info[0]), + "=b"(cpu_info[1]), + "=c"(cpu_info[2]), + "=d"(cpu_info[3]) : + "a"(0x80000000)); + if (cpu_info[0] >= 0x80000007) { + __asm__ volatile("cpuid" : + "=a"(cpu_info[0]), + "=b"(cpu_info[1]), + "=c"(cpu_info[2]), + "=d"(cpu_info[3]) : + "a"(0x80000007)); + if (!(cpu_info[3] & (1 << 8))) + return false; + } + + core_tick_hz = odp_cpu_hz(); + return true; +#endif + return false; +} + static inline uint64_t time_to_ns(odp_time_t time) { @@ -204,6 +256,9 @@ int odp_time_init_global(void) start_time.tv_nsec = time.tv_nsec; } + if (!core_tick_init()) + ODP_ABORT("core_tick_init failed\n"); + return ret; } diff --git a/platform/linux-generic/odp_timer.c b/platform/linux-generic/odp_timer.c index 6abb532..da569be 100644 --- a/platform/linux-generic/odp_timer.c +++ b/platform/linux-generic/odp_timer.c @@ -166,6 +166,7 @@ static inline void set_next_free(odp_timer *tim, uint32_t nf) typedef struct odp_timer_pool_s { /* Put frequently accessed fields in the first cache line */ + uint64_t cur_ts; odp_atomic_u64_t cur_tick;/* Current tick value */ uint64_t min_rel_tck; uint64_t max_rel_tck; @@ -179,11 +180,6 @@ typedef struct odp_timer_pool_s { odp_timer_pool_param_t param; char name[ODP_TIMER_POOL_NAME_LEN]; odp_shm_t shm; - timer_t timerid; - int notify_overrun; - pthread_t timer_thread; /* pthread_t of timer thread */ - pid_t timer_thread_id; /* gettid() for timer thread */ - int timer_thread_exit; /* request to exit for timer thread */ } odp_timer_pool; #define MAX_TIMER_POOLS 255 /* Leave one for ODP_TIMER_INVALID */ @@ -219,10 +215,6 @@ static inline odp_timer_t tp_idx_to_handle(struct odp_timer_pool_s *tp, return _odp_cast_scalar(odp_timer_t, (tp->tp_idx << INDEX_BITS) | idx); } -/* Forward declarations */ -static void itimer_init(odp_timer_pool *tp); -static void itimer_fini(odp_timer_pool *tp); - static odp_timer_pool_t odp_timer_pool_new(const char *_name, const odp_timer_pool_param_t *param) { @@ -245,6 +237,7 @@ static odp_timer_pool_t odp_timer_pool_new(const char *_name, ODP_ABORT("%s: timer pool shm-alloc(%zuKB) failed\n", _name, (sz0 + sz1 + sz2) / 1024); odp_timer_pool *tp = (odp_timer_pool *)odp_shm_addr(shm); + __atomic_store_n(&tp->cur_ts, core_tick(), __ATOMIC_RELAXED); odp_atomic_init_u64(&tp->cur_tick, 0); snprintf(tp->name, sizeof(tp->name), "%s", _name); tp->shm = shm; @@ -254,7 +247,6 @@ static odp_timer_pool_t odp_timer_pool_new(const char *_name, tp->num_alloc = 0; odp_atomic_init_u32(&tp->high_wm, 0); tp->first_free = 0; - tp->notify_overrun = 1; tp->tick_buf = (void *)((char *)odp_shm_addr(shm) + sz0); tp->timers = (void *)((char *)odp_shm_addr(shm) + sz0 + sz1); /* Initialize all odp_timer entries */ @@ -273,42 +265,14 @@ static odp_timer_pool_t odp_timer_pool_new(const char *_name, tp->tp_idx = tp_idx; odp_spinlock_init(&tp->lock); timer_pool[tp_idx] = tp; - if (tp->param.clk_src == ODP_CLOCK_CPU) - itimer_init(tp); return tp; } -static void block_sigalarm(void) -{ - sigset_t sigset; - - sigemptyset(&sigset); - sigaddset(&sigset, SIGALRM); - sigprocmask(SIG_BLOCK, &sigset, NULL); -} - -static void stop_timer_thread(odp_timer_pool *tp) -{ - int ret; - - ODP_DBG("stop\n"); - tp->timer_thread_exit = 1; - ret = pthread_join(tp->timer_thread, NULL); - if (ret != 0) - ODP_ABORT("unable to join thread, err %d\n", ret); -} - static void odp_timer_pool_del(odp_timer_pool *tp) { odp_spinlock_lock(&tp->lock); timer_pool[tp->tp_idx] = NULL; - /* Stop timer triggering */ - if (tp->param.clk_src == ODP_CLOCK_CPU) - itimer_fini(tp); - - stop_timer_thread(tp); - if (tp->num_alloc != 0) { /* It's a programming error to attempt to destroy a */ /* timer pool which is still in use */ @@ -673,121 +637,38 @@ static unsigned odp_timer_pool_expire(odp_timer_pool_t tpid, uint64_t tick) return nexp; } -/****************************************************************************** - * POSIX timer support - * Functions that use Linux/POSIX per-process timers and related facilities - *****************************************************************************/ - -static void timer_notify(odp_timer_pool *tp) +int timer_run(void) { - int overrun; - int64_t prev_tick; - - if (tp->notify_overrun) { - overrun = timer_getoverrun(tp->timerid); - if (overrun) { - ODP_ERR("\n\t%d ticks overrun on timer pool \"%s\", timer resolution too high\n", - overrun, tp->name); - tp->notify_overrun = 0; - } - } + odp_timer_pool *tp; + size_t i; + int nexp = 0; - odp_timer *array = &tp->timers[0]; - uint32_t i; - /* Prefetch initial cache lines (match 32 above) */ - for (i = 0; i < 32; i += ODP_CACHE_LINE_SIZE / sizeof(array[0])) - __builtin_prefetch(&array[i], 0, 0); - prev_tick = odp_atomic_fetch_inc_u64(&tp->cur_tick); + for (i = 0; i < MAX_TIMER_POOLS; i++) { + tp = timer_pool[i]; - /* Scan timer array, looking for timers to expire */ - (void)odp_timer_pool_expire(tp, prev_tick + 1); + if (tp == NULL) break; - /* Else skip scan of timers. cur_tick was updated and next itimer - * invocation will process older expiration ticks as well */ -} - -static void *timer_thread(void *arg) -{ - odp_timer_pool *tp = (odp_timer_pool *)arg; - sigset_t sigset; - int ret; - struct timespec tmo; - siginfo_t si; - - tp->timer_thread_id = (pid_t)syscall(SYS_gettid); + /* + * Check the last time this timer pool was expired. If one or + * more periods have passed, attempt to expire it. + */ + uint64_t tp_ts = + __atomic_load_n(&tp->cur_ts, __ATOMIC_RELAXED); + uint64_t now = core_tick(); - tmo.tv_sec = 0; - tmo.tv_nsec = ODP_TIME_MSEC_IN_NS * 100; - - sigemptyset(&sigset); - /* unblock sigalarm in this thread */ - sigprocmask(SIG_BLOCK, &sigset, NULL); - - sigaddset(&sigset, SIGALRM); - - while (1) { - ret = sigtimedwait(&sigset, &si, &tmo); - if (tp->timer_thread_exit) { - tp->timer_thread_id = 0; - return NULL; + if (core_tick_diff_ns(tp_ts, now) >= odp_timer_pool_res(tp)) { + if (__atomic_compare_exchange_n( + &tp->cur_ts, &tp_ts, now, false, + __ATOMIC_RELEASE, __ATOMIC_RELAXED)) { + uint64_t tp_tick; + tp_tick = odp_atomic_fetch_inc_u64( + &tp->cur_tick); + nexp += odp_timer_pool_expire(tp, tp_tick + 1); + } } - if (ret > 0) - timer_notify(tp); } - return NULL; -} - -static void itimer_init(odp_timer_pool *tp) -{ - struct sigevent sigev; - struct itimerspec ispec; - uint64_t res, sec, nsec; - int ret; - - ODP_DBG("Creating POSIX timer for timer pool %s, period %" - PRIu64" ns\n", tp->name, tp->param.res_ns); - - tp->timer_thread_id = 0; - ret = pthread_create(&tp->timer_thread, NULL, timer_thread, tp); - if (ret) - ODP_ABORT("unable to create timer thread\n"); - - /* wait thread set tp->timer_thread_id */ - do { - sched_yield(); - } while (tp->timer_thread_id == 0); - - memset(&sigev, 0, sizeof(sigev)); - sigev.sigev_notify = SIGEV_THREAD_ID; - sigev.sigev_value.sival_ptr = tp; - sigev._sigev_un._tid = tp->timer_thread_id; - sigev.sigev_signo = SIGALRM; - - if (timer_create(CLOCK_MONOTONIC, &sigev, &tp->timerid)) - ODP_ABORT("timer_create() returned error %s\n", - strerror(errno)); - - res = tp->param.res_ns; - sec = res / ODP_TIME_SEC_IN_NS; - nsec = res - sec * ODP_TIME_SEC_IN_NS; - - memset(&ispec, 0, sizeof(ispec)); - ispec.it_interval.tv_sec = (time_t)sec; - ispec.it_interval.tv_nsec = (long)nsec; - ispec.it_value.tv_sec = (time_t)sec; - ispec.it_value.tv_nsec = (long)nsec; - - if (timer_settime(tp->timerid, 0, &ispec, NULL)) - ODP_ABORT("timer_settime() returned error %s\n", - strerror(errno)); -} - -static void itimer_fini(odp_timer_pool *tp) -{ - if (timer_delete(tp->timerid) != 0) - ODP_ABORT("timer_delete() returned error %s\n", - strerror(errno)); + return nexp; } /****************************************************************************** @@ -1013,8 +894,6 @@ int odp_timer_init_global(void) #endif odp_atomic_init_u32(&num_timer_pools, 0); - block_sigalarm(); - return 0; }