From patchwork Thu Jun 1 03:07:56 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 100811 Delivered-To: patches@linaro.org Received: by 10.140.96.100 with SMTP id j91csp612358qge; Wed, 31 May 2017 20:08:03 -0700 (PDT) X-Received: by 10.98.29.79 with SMTP id d76mr34500109pfd.141.1496286483375; Wed, 31 May 2017 20:08:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1496286483; cv=none; d=google.com; s=arc-20160816; b=1DWHvtcaVqer4C8YrWTwm5Xdwzh44/nkFyu6eNA130cJ3GwWhWjdXfkt8hkosPeWY9 RDoj/uUGvSdjr2H/U0V8PUFKSwcwd1kLY8fztCemGwc+Erf6aiRx8uzWVf+oh3tLPmbc IU+vRSGWXA3juob2NAF5IVvodSOYV+p0HOXkSLEHY5kB7yCXmvpfIZKJ98UJWtj6GLu+ eIKbPIlyQjVnvFmBDWUINvlrt6qa9+1CVHdUD6jTFPuqsEYur0wcR24DhjNmtEkChD4X 7GpaAAp+vW6c5cxLzqVbcjMvuwxMAts88cb/0nHH4P+iGxYCXUl3x0P+t/4qrgJyAgkb 4f8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=fk9no4IRZh5X+9A6LInx7ePel4XeZQjOafNUpWg8ttk=; b=Sz6BAeRTfWosC+LkYKLfowXhrqEvASXQ0rkmGXnLyyl0GNS4f0uEwWVEIhyAH8V2s4 qyc9d1fmo8vaTS/uPcr2FaxzzQljsl/dPkd8234heIlO/pd4d1MX2Qh7B6bGGyLge8UM TRfPrRf0VR/bPOoooz6Wy2xYpZPrPwBMO+k6xBPrMzaCG1b2MDnx8RiMteQ6QqrUC063 dAYQFiKNmpiJcreFAW1yWLN5kRulzlXzutT3a1InXtHrv9/j/EHCaQ4l6EB+9fkDgihH 1R41o5J/YNEmj9GuX3u1/PS//P2oUeH5J7OJ3v6ukpeFTTAqgIHXjrnF1Tl9/yoOIbXF IOeg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org; spf=pass (google.com: domain of john.stultz@linaro.org designates 2607:f8b0:400e:c00::229 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from mail-pf0-x229.google.com (mail-pf0-x229.google.com. [2607:f8b0:400e:c00::229]) by mx.google.com with ESMTPS id w124si18904103pgb.147.2017.05.31.20.08.03 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 31 May 2017 20:08:03 -0700 (PDT) Received-SPF: pass (google.com: domain of john.stultz@linaro.org designates 2607:f8b0:400e:c00::229 as permitted sender) client-ip=2607:f8b0:400e:c00::229; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org; spf=pass (google.com: domain of john.stultz@linaro.org designates 2607:f8b0:400e:c00::229 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: by mail-pf0-x229.google.com with SMTP id n23so24595442pfb.2 for ; Wed, 31 May 2017 20:08:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=fk9no4IRZh5X+9A6LInx7ePel4XeZQjOafNUpWg8ttk=; b=OTb0vG1a27miV01NCAdML5i91duhfYeXGCwINjWNm7xx9UHjQunN2/5uqJ+XxZf/Sd NkNIIC2T7DlKxngC8tmMq+nphVrCVVPWuMQHosct+iUjnA1yBQNV6zKnSKx8r03On1N5 T3EP+KxuoKBQ6PWF4m8U+7AHeXi93Kd4QA9CI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=fk9no4IRZh5X+9A6LInx7ePel4XeZQjOafNUpWg8ttk=; b=MnEFAo8OjqGkOJHBvPZgEafdxTWlTxPdmJ+O76h8GrVUD7N8IR8FzN6vgEJMSf5jeH RXjdxXvhbEXHCmU6y1nSAlmtqLxzW++b1FU1WxKFiHncggUldXlT3seremN9uTAhm/Cf VTjLa0hTI4B/XLBlDNqdeXPo7JMwCaBNXvRCRmXTzs2l6snC93GxaQboNe4binO2Pe2p 1MqoxulONi+fcuEvp0qJogK7QxoF5kaW+1shAwdNb97OfQUu6afSe3sm6GsR2sz3zbBQ ogzN0+fFIsSVrrdGCBLl33ZcE0wYkK465vVR0evKkvqg8PLr6FpX+BEZJRv3cR4yV+aN ax1Q== X-Gm-Message-State: AODbwcAmWqtcqIO3y/mBc02B/edUuRnlKmlN8Y7pkvYvQUcDm+oQeqiF AFkxW1e7jZ5qp60qAyo= X-Received: by 10.98.7.149 with SMTP id 21mr33576891pfh.54.1496286483024; Wed, 31 May 2017 20:08:03 -0700 (PDT) Return-Path: Received: from localhost.localdomain ([2601:1c2:1002:83f0:4e72:b9ff:fe99:466a]) by smtp.gmail.com with ESMTPSA id s68sm29137756pgc.5.2017.05.31.20.08.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 31 May 2017 20:08:02 -0700 (PDT) From: John Stultz To: lkml Cc: John Stultz , Thomas Gleixner , Ingo Molnar , Miroslav Lichvar , Richard Cochran , Prarit Bhargava , Stephen Boyd , Daniel Mentz , stable Subject: [PATCH 1/3 v2] time: Fix clock->read(clock) race around clocksource changes Date: Wed, 31 May 2017 20:07:56 -0700 Message-Id: <1496286478-13584-2-git-send-email-john.stultz@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1496286478-13584-1-git-send-email-john.stultz@linaro.org> References: <1496286478-13584-1-git-send-email-john.stultz@linaro.org> In some testing on arm64 platforms, I was seeing null ptr crashes in the kselftest/timers clocksource-switch test. This was happening in a read function like: u64 clocksource_mmio_readl_down(struct clocksource *c) { return ~(u64)readl_relaxed(to_mmio_clksrc(c)->reg) & c->mask; } Where the callers enter the seqlock, and then call something like: cycle_now = tkr->read(tkr->clock); The problem seeming to be that since the ->read() and ->clock pointer references are happening separately, its possible the clocksource change happens in between and we end up calling the old ->read() function with the new clocksource, (or vice-versa) which causes the to_mmio_clksrc() in the read function to run off into space. This patch tries to address the issue by providing a helper function that atomically reads the clock value and then calls the clock->read(clock) function so that we always call the read funciton with the appropriate clocksource and don't accidentally mix them. The one exception where this helper isn't necessary is for the fast-timekepers which use their own locking and update logic to the tkr structures. Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Miroslav Lichvar Cc: Richard Cochran Cc: Prarit Bhargava Cc: Stephen Boyd Cc: Daniel Mentz Cc: stable Acked-by: Ingo Molnar Signed-off-by: John Stultz --- v2: Addressed Ingo's feedback on wording --- kernel/time/timekeeping.c | 40 +++++++++++++++++++++++++++++----------- 1 file changed, 29 insertions(+), 11 deletions(-) -- 2.7.4 diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index 9652bc5..797c73e 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -118,6 +118,26 @@ static inline void tk_update_sleep_time(struct timekeeper *tk, ktime_t delta) tk->offs_boot = ktime_add(tk->offs_boot, delta); } +/* + * tk_clock_read - atomic clocksource read() helper + * + * This helper is necessary to use in the read paths because, while the + * seqlock ensures we don't return a bad value while structures are updated, + * it doesn't protect from potential crashes. There is the possibility that + * the tkr's clocksource may change between the read reference, and the + * clock reference passed to the read function. This can cause crashes if + * the wrong clocksource is passed to the wrong read function. + * This isn't necessary to use when holding the timekeeper_lock or doing + * a read of the fast-timekeeper tkrs (which is protected by its own locking + * and update logic). + */ +static inline u64 tk_clock_read(struct tk_read_base *tkr) +{ + struct clocksource *clock = READ_ONCE(tkr->clock); + + return clock->read(clock); +} + #ifdef CONFIG_DEBUG_TIMEKEEPING #define WARNING_FREQ (HZ*300) /* 5 minute rate-limiting */ @@ -175,7 +195,7 @@ static inline u64 timekeeping_get_delta(struct tk_read_base *tkr) */ do { seq = read_seqcount_begin(&tk_core.seq); - now = tkr->read(tkr->clock); + now = tk_clock_read(tkr); last = tkr->cycle_last; mask = tkr->mask; max = tkr->clock->max_cycles; @@ -209,7 +229,7 @@ static inline u64 timekeeping_get_delta(struct tk_read_base *tkr) u64 cycle_now, delta; /* read clocksource */ - cycle_now = tkr->read(tkr->clock); + cycle_now = tk_clock_read(tkr); /* calculate the delta since the last update_wall_time */ delta = clocksource_delta(cycle_now, tkr->cycle_last, tkr->mask); @@ -240,7 +260,7 @@ static void tk_setup_internals(struct timekeeper *tk, struct clocksource *clock) tk->tkr_mono.clock = clock; tk->tkr_mono.read = clock->read; tk->tkr_mono.mask = clock->mask; - tk->tkr_mono.cycle_last = tk->tkr_mono.read(clock); + tk->tkr_mono.cycle_last = tk_clock_read(&tk->tkr_mono); tk->tkr_raw.clock = clock; tk->tkr_raw.read = clock->read; @@ -477,7 +497,7 @@ static void halt_fast_timekeeper(struct timekeeper *tk) struct tk_read_base *tkr = &tk->tkr_mono; memcpy(&tkr_dummy, tkr, sizeof(tkr_dummy)); - cycles_at_suspend = tkr->read(tkr->clock); + cycles_at_suspend = tk_clock_read(tkr); tkr_dummy.read = dummy_clock_read; update_fast_timekeeper(&tkr_dummy, &tk_fast_mono); @@ -649,11 +669,10 @@ static void timekeeping_update(struct timekeeper *tk, unsigned int action) */ static void timekeeping_forward_now(struct timekeeper *tk) { - struct clocksource *clock = tk->tkr_mono.clock; u64 cycle_now, delta; u64 nsec; - cycle_now = tk->tkr_mono.read(clock); + cycle_now = tk_clock_read(&tk->tkr_mono); delta = clocksource_delta(cycle_now, tk->tkr_mono.cycle_last, tk->tkr_mono.mask); tk->tkr_mono.cycle_last = cycle_now; tk->tkr_raw.cycle_last = cycle_now; @@ -929,8 +948,7 @@ void ktime_get_snapshot(struct system_time_snapshot *systime_snapshot) do { seq = read_seqcount_begin(&tk_core.seq); - - now = tk->tkr_mono.read(tk->tkr_mono.clock); + now = tk_clock_read(&tk->tkr_mono); systime_snapshot->cs_was_changed_seq = tk->cs_was_changed_seq; systime_snapshot->clock_was_set_seq = tk->clock_was_set_seq; base_real = ktime_add(tk->tkr_mono.base, @@ -1108,7 +1126,7 @@ int get_device_system_crosststamp(int (*get_time_fn) * Check whether the system counter value provided by the * device driver is on the current timekeeping interval. */ - now = tk->tkr_mono.read(tk->tkr_mono.clock); + now = tk_clock_read(&tk->tkr_mono); interval_start = tk->tkr_mono.cycle_last; if (!cycle_between(interval_start, cycles, now)) { clock_was_set_seq = tk->clock_was_set_seq; @@ -1629,7 +1647,7 @@ void timekeeping_resume(void) * The less preferred source will only be tried if there is no better * usable source. The rtc part is handled separately in rtc core code. */ - cycle_now = tk->tkr_mono.read(clock); + cycle_now = tk_clock_read(&tk->tkr_mono); if ((clock->flags & CLOCK_SOURCE_SUSPEND_NONSTOP) && cycle_now > tk->tkr_mono.cycle_last) { u64 nsec, cyc_delta; @@ -2030,7 +2048,7 @@ void update_wall_time(void) #ifdef CONFIG_ARCH_USES_GETTIMEOFFSET offset = real_tk->cycle_interval; #else - offset = clocksource_delta(tk->tkr_mono.read(tk->tkr_mono.clock), + offset = clocksource_delta(tk_clock_read(&tk->tkr_mono), tk->tkr_mono.cycle_last, tk->tkr_mono.mask); #endif From patchwork Thu Jun 1 03:07:57 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 100812 Delivered-To: patches@linaro.org Received: by 10.140.96.100 with SMTP id j91csp612367qge; Wed, 31 May 2017 20:08:05 -0700 (PDT) X-Received: by 10.98.59.212 with SMTP id w81mr845625pfj.107.1496286484986; Wed, 31 May 2017 20:08:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1496286484; cv=none; d=google.com; s=arc-20160816; b=Q5H9eEu8BzrGYIoklkWhlEHmNpumPKJPlNdq5/mxtk4nCo8pQftfCZLTjfjgRt+28+ fxilYpsQJupmJh8Enr6Ai7/9YL+mjQNq1b9sKmvX4tDxJ+8UyhvgxKJBmz7jSVKsLpEI m0+UjYlzJDH9zTx75ljOnlgQ9Fk1ot1M+c7/g6piGyyQo/Unl0rQQ7HM87dkYDI311dC Zx32Ya6a4J28NbRS6Eu2tchKWE8lM8+n0RHmWITBqYUxFfHbcCDrxP/fIRkZujm8pvAt bNUhu9XC1pLC4N7qfMi1li5j95fVEhmcHCXjY1mHZIgOwGc1xhSVjmUTHg1AL0QsrgGL dPdg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=Ab6ods9YiYrr2j7MU9kZNIWIg9LpUKHaxEwPdZtMW9I=; b=TBbkLgbQIPKPv+EEXnkiMvfV41F9DtPSuiqUq8e1MBpC19nAaBNkj/snNbJGNjo3bX V+KlkMdbrdDtJQNq4oQvVQrWX/hEWsnhapJ+raUbwlDpjy2zKLvK29nNWeaIng+mR2a8 CacgELevQs13FHCOqdyYXxXdi/U7XZq2Wi7lwVqY8RXG1krNJTBvEpq00wQaum/PJf6R VEb24x1Pv/prki/dTf6C2iHLsbLazQhq0RLUbKJUHykjQfjVexaY7s5zluABdoZBuApp rUSAaZTOkq+VZ0raO1YpGTR0CQI3yQkp9qUTtgiOnVuRsXIZGhFm8Oej5lanY9zm3R94 LbBQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org; spf=pass (google.com: domain of john.stultz@linaro.org designates 2607:f8b0:400e:c00::232 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from mail-pf0-x232.google.com (mail-pf0-x232.google.com. [2607:f8b0:400e:c00::232]) by mx.google.com with ESMTPS id v5si49216652plg.227.2017.05.31.20.08.04 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 31 May 2017 20:08:04 -0700 (PDT) Received-SPF: pass (google.com: domain of john.stultz@linaro.org designates 2607:f8b0:400e:c00::232 as permitted sender) client-ip=2607:f8b0:400e:c00::232; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org; spf=pass (google.com: domain of john.stultz@linaro.org designates 2607:f8b0:400e:c00::232 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: by mail-pf0-x232.google.com with SMTP id m17so24585507pfg.3 for ; Wed, 31 May 2017 20:08:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Ab6ods9YiYrr2j7MU9kZNIWIg9LpUKHaxEwPdZtMW9I=; b=bCAo6V6iVFFKD7IkKQ9us4A9yR0CTP9lHM91offzDVsZxcD32eWsA6UJgCNjCDpmm1 o1gJUP3wXbs0l8gXax4t7m2B3xDjFVDXXHhyg42EJ1YxxR/+LfUF3/UH7y+U9lL2BTu1 aDjq6IWX5taD4GVmRPG0X1m6NE7SBD+dFKYPw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Ab6ods9YiYrr2j7MU9kZNIWIg9LpUKHaxEwPdZtMW9I=; b=j8WSfUo141neZ7KgOcDmUvucuCnfIXtSV4CWZH9WXhR3uwgc9E7rDcAIa2DXFtrKo3 gE2FfDd79mxpPlaWUll3L88jMX+Hg3fyFGyYzZcnG/IQqbF13dcfxSe/t9HtAs/pWoAN 3lX9TJ1/9xIibIzlWcOKkY6cmouYKqGx+r6ANtbFN4t6yTdUiBtNZ8jHr6HETknWX0qn opyAEAbH7CWJlMMOxE2G5QD6kSj2tzw+YGZT0UgEIING0Ih+V/Qu9jT1bxBQsBowhWjh 4lIPRXp9fRTKXVmZNPtwy0dO8fHcg2jcRbLik+Zx5Gy3colp7cxjxHMx1/FYKJ4cnOlk WxXw== X-Gm-Message-State: AODbwcCU6YPbUxGwk6p7YfpowzfjrySGp6zKF0unbHdE8fAE5ASSuobu fT5cjo+sMQgatfNNcds= X-Received: by 10.84.130.7 with SMTP id 7mr92751269plc.35.1496286484656; Wed, 31 May 2017 20:08:04 -0700 (PDT) Return-Path: Received: from localhost.localdomain ([2601:1c2:1002:83f0:4e72:b9ff:fe99:466a]) by smtp.gmail.com with ESMTPSA id s68sm29137756pgc.5.2017.05.31.20.08.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 31 May 2017 20:08:03 -0700 (PDT) From: John Stultz To: lkml Cc: John Stultz , Thomas Gleixner , Ingo Molnar , Miroslav Lichvar , Richard Cochran , Prarit Bhargava , Stephen Boyd , Kevin Brodsky , Will Deacon , Daniel Mentz , "stable #4 . 8+" Subject: [PATCH 2/3 v2] time: Fix CLOCK_MONOTONIC_RAW sub-nanosecond accounting Date: Wed, 31 May 2017 20:07:57 -0700 Message-Id: <1496286478-13584-3-git-send-email-john.stultz@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1496286478-13584-1-git-send-email-john.stultz@linaro.org> References: <1496286478-13584-1-git-send-email-john.stultz@linaro.org> Due to how the MONOTONIC_RAW accumulation logic was handled, there is the potential for a 1ns discontinuity when we do accumulations. This small discontinuity has for the most part gone un-noticed, but since ARM64 enabled CLOCK_MONOTONIC_RAW in their vDSO clock_gettime implementation, we've seen failures with the inconsistency-check test in kselftest. This patch addresses the issue by using the same sub-ns accumulation handling that CLOCK_MONOTONIC uses, which avoids the issue for in-kernel users. Since the ARM64 vDSO implementation has its own clock_gettime calculation logic, this patch reduces the frequency of errors, but failures are still seen. The ARM64 vDSO will need to be updated to include the sub-nanosecond xtime_nsec values in its calculation for this issue to be completely fixed. Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Miroslav Lichvar Cc: Richard Cochran Cc: Prarit Bhargava Cc: Stephen Boyd Cc: Kevin Brodsky Cc: Will Deacon Cc: Daniel Mentz Cc: stable #4.8+ Tested-by: Daniel Mentz Signed-off-by: John Stultz --- v2: Address Ingo's style feedback --- include/linux/timekeeper_internal.h | 4 ++-- kernel/time/timekeeping.c | 19 ++++++++++--------- 2 files changed, 12 insertions(+), 11 deletions(-) -- 2.7.4 diff --git a/include/linux/timekeeper_internal.h b/include/linux/timekeeper_internal.h index 110f453..528cc86 100644 --- a/include/linux/timekeeper_internal.h +++ b/include/linux/timekeeper_internal.h @@ -58,7 +58,7 @@ struct tk_read_base { * interval. * @xtime_remainder: Shifted nano seconds left over when rounding * @cycle_interval - * @raw_interval: Raw nano seconds accumulated per NTP interval. + * @raw_interval: Shifted raw nano seconds accumulated per NTP interval. * @ntp_error: Difference between accumulated time and NTP time in ntp * shifted nano seconds. * @ntp_error_shift: Shift conversion between clock shifted nano seconds and @@ -100,7 +100,7 @@ struct timekeeper { u64 cycle_interval; u64 xtime_interval; s64 xtime_remainder; - u32 raw_interval; + u64 raw_interval; /* The ntp_tick_length() value currently being used. * This cached copy ensures we consistently apply the tick * length for an entire tick, as ntp_tick_length may change diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index 797c73e..8eaa95c 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -282,7 +282,7 @@ static void tk_setup_internals(struct timekeeper *tk, struct clocksource *clock) /* Go back from cycles -> shifted ns */ tk->xtime_interval = interval * clock->mult; tk->xtime_remainder = ntpinterval - tk->xtime_interval; - tk->raw_interval = (interval * clock->mult) >> clock->shift; + tk->raw_interval = interval * clock->mult; /* if changing clocks, convert xtime_nsec shift units */ if (old_clock) { @@ -1994,7 +1994,7 @@ static u64 logarithmic_accumulation(struct timekeeper *tk, u64 offset, u32 shift, unsigned int *clock_set) { u64 interval = tk->cycle_interval << shift; - u64 raw_nsecs; + u64 snsec_per_sec; /* If the offset is smaller than a shifted interval, do nothing */ if (offset < interval) @@ -2009,14 +2009,15 @@ static u64 logarithmic_accumulation(struct timekeeper *tk, u64 offset, *clock_set |= accumulate_nsecs_to_secs(tk); /* Accumulate raw time */ - raw_nsecs = (u64)tk->raw_interval << shift; - raw_nsecs += tk->raw_time.tv_nsec; - if (raw_nsecs >= NSEC_PER_SEC) { - u64 raw_secs = raw_nsecs; - raw_nsecs = do_div(raw_secs, NSEC_PER_SEC); - tk->raw_time.tv_sec += raw_secs; + tk->tkr_raw.xtime_nsec += (u64)tk->raw_time.tv_nsec << tk->tkr_raw.shift; + tk->tkr_raw.xtime_nsec += tk->raw_interval << shift; + snsec_per_sec = (u64)NSEC_PER_SEC << tk->tkr_raw.shift; + while (tk->tkr_raw.xtime_nsec >= snsec_per_sec) { + tk->tkr_raw.xtime_nsec -= snsec_per_sec; + tk->raw_time.tv_sec++; } - tk->raw_time.tv_nsec = raw_nsecs; + tk->raw_time.tv_nsec = tk->tkr_raw.xtime_nsec >> tk->tkr_raw.shift; + tk->tkr_raw.xtime_nsec -= (u64)tk->raw_time.tv_nsec << tk->tkr_raw.shift; /* Accumulate error between NTP and clock interval */ tk->ntp_error += tk->ntp_tick << shift; From patchwork Thu Jun 1 03:07:58 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 100813 Delivered-To: patches@linaro.org Received: by 10.140.96.100 with SMTP id j91csp612373qge; Wed, 31 May 2017 20:08:06 -0700 (PDT) X-Received: by 10.84.169.3 with SMTP id g3mr38258064plb.37.1496286486489; Wed, 31 May 2017 20:08:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1496286486; cv=none; d=google.com; s=arc-20160816; b=jZBJplq/VyRUP6Zu7PiUw4/VbPzo0hZSpZWExURtTQd0YzjMRmWZzszx8sN46SI6UJ CwJwoeD8HWjFk12pqc+4xGwwZzw0O3LjMG9ZatfS9arUGbjFEN45ouetnJVcv6Gz/wPl iTeSL6cLBlH5XqwBvofu/uId3WaLV5xlpfv8qO7OYC7qXlJef6+c/MtFK+53IsgPCH40 Mycj8Ea7A1msa6UUQYgwBhfZ3lZZ59KQC6+7lbqrbKqgIG/ODjJpsL8B4a1KiMhUmz+D kyWYBZlJDMxldNVPJTmFaQo99WQAJA2QOFbRWtuPFLy4S7smbwjNE5uNikcBruK5ZICo QnPg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=P6Z1iOyxQBLRyPZe09EyceQVC4iWS7ZuJ35jqxv5oWs=; b=VsiJDF9iiwpO5ltGB7OuRF1s9Ozmx5dFK7JSjE8hZfW2ww5ENiJWyH6ygavFPxMxmX ikCX2rcGLGJpOhc4BMmARZUvH+vVFvLlfwYWLJ/mMuzU5lL/Vd64IKxxeHRjeaRS+7/A c2u7b/0+rsWo2dl4Op5eUd9YS/h1HHE1xdjz9IxbDSW8KqLLv8TAGhuIRKl9L5zUxoa7 HE9lBbXKiejq0LfStG4KRLKDHIdqtD2mCWlYj3JQ9G5+XLAXrp+7lzLCs8ve5s7MyONx IS3RyLeG51cKSWZyQf00F83Zm+QL8r3TPsOkFjFVaXiE0+nnNNFKKGjGvZagRm0+x2/X ZMhA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org; spf=pass (google.com: domain of john.stultz@linaro.org designates 2607:f8b0:400e:c00::233 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from mail-pf0-x233.google.com (mail-pf0-x233.google.com. [2607:f8b0:400e:c00::233]) by mx.google.com with ESMTPS id v130si18096946pgb.188.2017.05.31.20.08.06 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 31 May 2017 20:08:06 -0700 (PDT) Received-SPF: pass (google.com: domain of john.stultz@linaro.org designates 2607:f8b0:400e:c00::233 as permitted sender) client-ip=2607:f8b0:400e:c00::233; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org; spf=pass (google.com: domain of john.stultz@linaro.org designates 2607:f8b0:400e:c00::233 as permitted sender) smtp.mailfrom=john.stultz@linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: by mail-pf0-x233.google.com with SMTP id n23so24596515pfb.2 for ; Wed, 31 May 2017 20:08:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=P6Z1iOyxQBLRyPZe09EyceQVC4iWS7ZuJ35jqxv5oWs=; b=OA3Jqoo8GeGqdbe7l95fzOuQvvvSH0B/6ZTJ64fMTCaJZNBDOvpD7FGINQpQkloCcd DC7Ux5d3l4Xe3XWCgGnQ+V9kg5l0vAamwD7GXyQ1vgnz2CHzaPMD9w90HSSJt/ZioxPj sXXC0f1zPafIjANycq4yNx6GGdan11BosVTHY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=P6Z1iOyxQBLRyPZe09EyceQVC4iWS7ZuJ35jqxv5oWs=; b=gtK1tmnZ5Er70sF/TxxSOhzH+cja9fgDjXqybc9Y96j3B/pQY/Aiq3v3I6y8xPafGK 2cQfQqWwJl42Us2tQHsRnxiIyTIBv48nvoPmkEuIR6ez0uo/FGYkZkz3taNiauPdPmhd NHuGQJM/A5fYjqn7Ts4cZQWlHI1/FifMw+fukuUQNqv2HIBpl6vUqOrY3D5B1j1ez9Go lLdgTJlQELdr49Bx5HlbiC2bwRieWAtElfgItiBgrML3nOG7L2RU9RaMjpVmXmPiCTIW e8t8ZL3nEpN4zjMCEopIlSqylYSKz0ZvjGe4Rj3luduG0Zvntj3mnk/XF4Hk1njuSQKn Rq+w== X-Gm-Message-State: AODbwcChx4PH7HBbZgAjt7xbQZI6xIus+QvhIBgZpYPIns1eqoOXEz9G cVXTZp2VPPl2L0oTeTI= X-Received: by 10.98.214.14 with SMTP id r14mr34019730pfg.156.1496286486149; Wed, 31 May 2017 20:08:06 -0700 (PDT) Return-Path: Received: from localhost.localdomain ([2601:1c2:1002:83f0:4e72:b9ff:fe99:466a]) by smtp.gmail.com with ESMTPSA id s68sm29137756pgc.5.2017.05.31.20.08.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 31 May 2017 20:08:05 -0700 (PDT) From: John Stultz To: lkml Cc: Will Deacon , Thomas Gleixner , Ingo Molnar , Miroslav Lichvar , Richard Cochran , Prarit Bhargava , Stephen Boyd , Kevin Brodsky , Daniel Mentz , "stable #4 . 8+" , John Stultz Subject: [PATCH 3/3 v2] arm64: vdso: Fix nsec handling for CLOCK_MONOTONIC_RAW Date: Wed, 31 May 2017 20:07:58 -0700 Message-Id: <1496286478-13584-4-git-send-email-john.stultz@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1496286478-13584-1-git-send-email-john.stultz@linaro.org> References: <1496286478-13584-1-git-send-email-john.stultz@linaro.org> From: Will Deacon Recently vDSO support for CLOCK_MONOTONIC_RAW was added in 49eea433b326 ("arm64: Add support for CLOCK_MONOTONIC_RAW in clock_gettime() vDSO"). Noticing that the core timekeeping code never set tkr_raw.xtime_nsec, the vDSO implementation didn't bother exposing it via the data page and instead took the unshifted tk->raw_time.tv_nsec value which was then immediately shifted left in the vDSO code. Unfortunately, by accellerating the MONOTONIC_RAW clockid, it uncovered potential 1ns time inconsistencies caused by the timekeeping core not handing sub-ns resolution. Now that the core code has been fixed and is actually setting tkr_raw.xtime_nsec, we need to take that into account in the vDSO by adding it to the shifted raw_time value, in order to fix the user-visible inconsistency. Rather than do that at each use (and expand the data page in the process), instead perform the shift/addition operation when populating the data page and remove the shift from the vDSO code entirely. Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Miroslav Lichvar Cc: Richard Cochran Cc: Prarit Bhargava Cc: Stephen Boyd Cc: Kevin Brodsky Cc: Will Deacon Cc: Daniel Mentz Cc: stable #4.8+ Reported-by: John Stultz Acked-by: Kevin Brodsky Tested-by: Daniel Mentz Signed-off-by: Will Deacon [jstultz: minor whitespace tweak, tried to improve commit message to make it more clear this fixes a regression] Signed-off-by: John Stultz --- v2: Tweak commit message to address Ingo's feedback --- arch/arm64/kernel/vdso.c | 5 +++-- arch/arm64/kernel/vdso/gettimeofday.S | 1 - 2 files changed, 3 insertions(+), 3 deletions(-) -- 2.7.4 diff --git a/arch/arm64/kernel/vdso.c b/arch/arm64/kernel/vdso.c index 41b6e31..d0cb007 100644 --- a/arch/arm64/kernel/vdso.c +++ b/arch/arm64/kernel/vdso.c @@ -221,10 +221,11 @@ void update_vsyscall(struct timekeeper *tk) /* tkr_mono.cycle_last == tkr_raw.cycle_last */ vdso_data->cs_cycle_last = tk->tkr_mono.cycle_last; vdso_data->raw_time_sec = tk->raw_time.tv_sec; - vdso_data->raw_time_nsec = tk->raw_time.tv_nsec; + vdso_data->raw_time_nsec = (tk->raw_time.tv_nsec << + tk->tkr_raw.shift) + + tk->tkr_raw.xtime_nsec; vdso_data->xtime_clock_sec = tk->xtime_sec; vdso_data->xtime_clock_nsec = tk->tkr_mono.xtime_nsec; - /* tkr_raw.xtime_nsec == 0 */ vdso_data->cs_mono_mult = tk->tkr_mono.mult; vdso_data->cs_raw_mult = tk->tkr_raw.mult; /* tkr_mono.shift == tkr_raw.shift */ diff --git a/arch/arm64/kernel/vdso/gettimeofday.S b/arch/arm64/kernel/vdso/gettimeofday.S index e00b467..76320e9 100644 --- a/arch/arm64/kernel/vdso/gettimeofday.S +++ b/arch/arm64/kernel/vdso/gettimeofday.S @@ -256,7 +256,6 @@ monotonic_raw: seqcnt_check fail=monotonic_raw /* All computations are done with left-shifted nsecs. */ - lsl x14, x14, x12 get_nsec_per_sec res=x9 lsl x9, x9, x12