From patchwork Tue Jul 17 21:49:30 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: john stultz X-Patchwork-Id: 10087 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id 8E82D23E3D for ; Tue, 17 Jul 2012 21:50:47 +0000 (UTC) Received: from mail-yw0-f52.google.com (mail-yw0-f52.google.com [209.85.213.52]) by fiordland.canonical.com (Postfix) with ESMTP id 47D05A18704 for ; Tue, 17 Jul 2012 21:50:47 +0000 (UTC) Received: by yhpp61 with SMTP id p61so1041047yhp.11 for ; Tue, 17 Jul 2012 14:50:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-forwarded-to:x-forwarded-for:delivered-to:received-spf:from:to:cc :subject:date:message-id:x-mailer:in-reply-to:references :x-content-scanned:x-cbid:x-ibm-iss-spamdetectors :x-ibm-iss-detailinfo:x-gm-message-state; bh=BIWUsbbWtMRlDZ3DH9dZEbMTke5bk1Z/YnQIxPHc87s=; b=kHciO+6AlFfw116rsbzEN8r+v92YyGyjBSO8YjZ6dslvkM5dXWxXwk2dEr5FCd21Gg Donsf3FbXyaA6lbzDffaKoPoQoMqeehINkhSQznGh02a+teIWPICOTHKYqPh6y0Fh0EH +jQLpn0QTqRbM/UpbpTnxAAP1QeDisyjMUQjgsb1XwJ9PQ2wSfy8CmsMipUF+p2uxbvL YiIe18QYyw583rVAgr1uMDT8lxuDrK3OTkmSeVhF4EDu65VkYTHFG0PgnqU9Lpf1wI2u OjrnsUmAuR721CSYIUgX7GQyxWKMzwvp6VLIo+ej4U8d0sapIXzNE5DZrs5gBdEWlXIE GMgA== Received: by 10.50.87.227 with SMTP id bb3mr163012igb.57.1342561846618; Tue, 17 Jul 2012 14:50:46 -0700 (PDT) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.231.241.2 with SMTP id lc2csp24887ibb; Tue, 17 Jul 2012 14:50:46 -0700 (PDT) Received: by 10.50.156.196 with SMTP id wg4mr171690igb.54.1342561846188; Tue, 17 Jul 2012 14:50:46 -0700 (PDT) Received: from e34.co.us.ibm.com (e34.co.us.ibm.com. [32.97.110.152]) by mx.google.com with ESMTPS id dg6si26545551icc.104.2012.07.17.14.50.45 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 17 Jul 2012 14:50:46 -0700 (PDT) Received-SPF: pass (google.com: domain of johnstul@us.ibm.com designates 32.97.110.152 as permitted sender) client-ip=32.97.110.152; Authentication-Results: mx.google.com; spf=pass (google.com: domain of johnstul@us.ibm.com designates 32.97.110.152 as permitted sender) smtp.mail=johnstul@us.ibm.com Received: from /spool/local by e34.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 17 Jul 2012 15:50:45 -0600 Received: from d03dlp02.boulder.ibm.com (9.17.202.178) by e34.co.us.ibm.com (192.168.1.134) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Tue, 17 Jul 2012 15:50:21 -0600 Received: from d03relay05.boulder.ibm.com (d03relay05.boulder.ibm.com [9.17.195.107]) by d03dlp02.boulder.ibm.com (Postfix) with ESMTP id 210A43E4004E for ; Tue, 17 Jul 2012 21:50:18 +0000 (WET) Received: from d03av05.boulder.ibm.com (d03av05.boulder.ibm.com [9.17.195.85]) by d03relay05.boulder.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id q6HLnsDw095942 for ; Tue, 17 Jul 2012 15:49:55 -0600 Received: from d03av05.boulder.ibm.com (loopback [127.0.0.1]) by d03av05.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id q6HLnjeE026928 for ; Tue, 17 Jul 2012 15:49:45 -0600 Received: from kernel.stglabs.ibm.com (kernel.stglabs.ibm.com [9.114.214.19]) by d03av05.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id q6HLnZI8026401; Tue, 17 Jul 2012 15:49:44 -0600 From: John Stultz To: stable@vger.kernel.org Cc: John Stultz , Thomas Gleixner , Prarit Bhargava , Linux Kernel Subject: [PATCH 10/11] 2.6.34.x: hrtimer: Update hrtimer base offsets each hrtimer_interrupt Date: Tue, 17 Jul 2012 17:49:30 -0400 Message-Id: <1342561771-55678-11-git-send-email-johnstul@us.ibm.com> X-Mailer: git-send-email 1.7.9.5 In-Reply-To: <1342561771-55678-1-git-send-email-johnstul@us.ibm.com> References: <1342561771-55678-1-git-send-email-johnstul@us.ibm.com> X-Content-Scanned: Fidelis XPS MAILER x-cbid: 12071721-1780-0000-0000-0000077E4116 X-IBM-ISS-SpamDetectors: X-IBM-ISS-DetailInfo: BY=3.00000287; HX=3.00000193; KW=3.00000007; PH=3.00000001; SC=3.00000004; SDB=6.00157490; UDB=6.00035526; UTC=2012-07-17 21:50:38 X-Gm-Message-State: ALoCoQmA41MoKaiySWav3QX1BHNmfyzJ83U+gegWMixhV90xmpKyOvWVwB8zJAh2HeZ04aiQMqFE This is a backport of 5baefd6d84163443215f4a99f6a20f054ef11236 The update of the hrtimer base offsets on all cpus cannot be made atomically from the timekeeper.lock held and interrupt disabled region as smp function calls are not allowed there. clock_was_set(), which enforces the update on all cpus, is called either from preemptible process context in case of do_settimeofday() or from the softirq context when the offset modification happened in the timer interrupt itself due to a leap second. In both cases there is a race window for an hrtimer interrupt between dropping timekeeper lock, enabling interrupts and clock_was_set() issuing the updates. Any interrupt which arrives in that window will see the new time but operate on stale offsets. So we need to make sure that an hrtimer interrupt always sees a consistent state of time and offsets. ktime_get_update_offsets() allows us to get the current monotonic time and update the per cpu hrtimer base offsets from hrtimer_interrupt() to capture a consistent state of monotonic time and the offsets. The function replaces the existing ktime_get() calls in hrtimer_interrupt(). The overhead of the new function vs. ktime_get() is minimal as it just adds two store operations. This ensures that any changes to realtime or boottime offsets are noticed and stored into the per-cpu hrtimer base structures, prior to any hrtimer expiration and guarantees that timers are not expired early. Signed-off-by: John Stultz Reviewed-by: Ingo Molnar Acked-by: Peter Zijlstra Acked-by: Prarit Bhargava Cc: stable@vger.kernel.org Link: http://lkml.kernel.org/r/1341960205-56738-8-git-send-email-johnstul@us.ibm.com Signed-off-by: Thomas Gleixner Cc: Prarit Bhargava Cc: Thomas Gleixner Cc: Linux Kernel Signed-off-by: John Stultz --- kernel/hrtimer.c | 27 ++++++++++++--------------- 1 file changed, 12 insertions(+), 15 deletions(-) diff --git a/kernel/hrtimer.c b/kernel/hrtimer.c index 039db35..6b6835f 100644 --- a/kernel/hrtimer.c +++ b/kernel/hrtimer.c @@ -603,6 +603,12 @@ static int hrtimer_reprogram(struct hrtimer *timer, return res; } +static inline ktime_t hrtimer_update_base(struct hrtimer_cpu_base *base) +{ + ktime_t *offs_real = &base->clock_base[CLOCK_REALTIME].offset; + + return ktime_get_update_offsets(offs_real); +} /* * Retrigger next event is called after clock was set @@ -612,26 +618,15 @@ static int hrtimer_reprogram(struct hrtimer *timer, static void retrigger_next_event(void *arg) { struct hrtimer_cpu_base *base; - struct timespec realtime_offset; - unsigned long seq; if (!hrtimer_hres_active()) return; - do { - seq = read_seqbegin(&xtime_lock); - set_normalized_timespec(&realtime_offset, - -wall_to_monotonic.tv_sec, - -wall_to_monotonic.tv_nsec); - } while (read_seqretry(&xtime_lock, seq)); - base = &__get_cpu_var(hrtimer_bases); /* Adjust CLOCK_REALTIME offset */ raw_spin_lock(&base->lock); - base->clock_base[CLOCK_REALTIME].offset = - timespec_to_ktime(realtime_offset); - + hrtimer_update_base(base); hrtimer_force_reprogram(base, 0); raw_spin_unlock(&base->lock); } @@ -731,7 +726,6 @@ static int hrtimer_switch_to_hres(void) base->clock_base[CLOCK_MONOTONIC].resolution = KTIME_HIGH_RES; tick_setup_sched_timer(); - /* "Retrigger" the interrupt to get things going */ retrigger_next_event(NULL); local_irq_restore(flags); @@ -1280,7 +1274,7 @@ void hrtimer_interrupt(struct clock_event_device *dev) dev->next_event.tv64 = KTIME_MAX; raw_spin_lock(&cpu_base->lock); - entry_time = now = ktime_get(); + entry_time = now = hrtimer_update_base(cpu_base); retry: expires_next.tv64 = KTIME_MAX; /* @@ -1356,9 +1350,12 @@ retry: * We need to prevent that we loop forever in the hrtimer * interrupt routine. We give it 3 attempts to avoid * overreacting on some spurious event. + * + * Acquire base lock for updating the offsets and retrieving + * the current time. */ raw_spin_lock(&cpu_base->lock); - now = ktime_get(); + now = hrtimer_update_base(cpu_base); cpu_base->nr_retries++; if (++retries < 3) goto retry;