From patchwork Mon Feb 17 18:45:36 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 24787 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-ie0-f198.google.com (mail-ie0-f198.google.com [209.85.223.198]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 329BB20143 for ; Mon, 17 Feb 2014 18:45:45 +0000 (UTC) Received: by mail-ie0-f198.google.com with SMTP id at1sf25498608iec.1 for ; Mon, 17 Feb 2014 10:45:44 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:delivered-to:from:to:cc:subject :date:message-id:in-reply-to:references:x-original-sender :x-original-authentication-results:precedence:mailing-list:list-id :list-post:list-help:list-archive:list-unsubscribe; bh=Fd1B2Y6zUuL1etigGPEkn9LSRxCIseBr9Qs9hpnWOrM=; b=eC1uvrts47fTosLuu89Cia6Q0HQjBnKH+wK1xC6f4SP7aYJqaNUWIz108pRhKPDLtu lLYWVM+vnTCvgY2VfkJs/+c2u2qz2jmF6CvS85GlYYUK/m09YgwmhirRgaPY5WBeqj3J 7q9Z4ZLXhcU2KW0gxB+YKIhb7wldQmWh5C1dQftt7WZAVlRFSqy+rfe2FX8MOz6sx66C 32LMUJv+bguhnzIDa4oCM8J8NWynB+tcJ5K3mpK3+lj6fC+nkhoTwoYWcPuOkunjAMr9 +kr2oAMazI2YJvCuX2cf7A2M6QScfgXELgQK8I4zyUhxh9cuzuCkLpIHI2y0i31h1ad7 YCng== X-Gm-Message-State: ALoCoQlGJ4esZEYcL9HhDu0QH+CDUwRRh2IIA0Tp2o3LhBF4NOWYwh7J/oQe1xmwEIEih2aYq1m+ X-Received: by 10.50.127.148 with SMTP id ng20mr696522igb.4.1392662744427; Mon, 17 Feb 2014 10:45:44 -0800 (PST) MIME-Version: 1.0 X-BeenThere: patchwork-forward@linaro.org Received: by 10.140.39.37 with SMTP id u34ls1003430qgu.37.gmail; Mon, 17 Feb 2014 10:45:44 -0800 (PST) X-Received: by 10.220.159.4 with SMTP id h4mr18245711vcx.1.1392662744346; Mon, 17 Feb 2014 10:45:44 -0800 (PST) Received: from mail-vc0-f173.google.com (mail-vc0-f173.google.com [209.85.220.173]) by mx.google.com with ESMTPS id be7si4683454vcb.35.2014.02.17.10.45.44 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 17 Feb 2014 10:45:44 -0800 (PST) Received-SPF: neutral (google.com: 209.85.220.173 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) client-ip=209.85.220.173; Received: by mail-vc0-f173.google.com with SMTP id ld13so12026873vcb.18 for ; Mon, 17 Feb 2014 10:45:44 -0800 (PST) X-Received: by 10.52.84.102 with SMTP id x6mr1481491vdy.49.1392662744252; Mon, 17 Feb 2014 10:45:44 -0800 (PST) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patches@linaro.org Received: by 10.220.174.196 with SMTP id u4csp159808vcz; Mon, 17 Feb 2014 10:45:43 -0800 (PST) X-Received: by 10.68.226.200 with SMTP id ru8mr28154388pbc.77.1392662743240; Mon, 17 Feb 2014 10:45:43 -0800 (PST) Received: from mail-pd0-f169.google.com (mail-pd0-f169.google.com [209.85.192.169]) by mx.google.com with ESMTPS id cf2si15687463pad.24.2014.02.17.10.45.42 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 17 Feb 2014 10:45:43 -0800 (PST) Received-SPF: neutral (google.com: 209.85.192.169 is neither permitted nor denied by best guess record for domain of john.stultz@linaro.org) client-ip=209.85.192.169; Received: by mail-pd0-f169.google.com with SMTP id v10so15271895pde.0 for ; Mon, 17 Feb 2014 10:45:42 -0800 (PST) X-Received: by 10.68.241.134 with SMTP id wi6mr27879702pbc.44.1392662742845; Mon, 17 Feb 2014 10:45:42 -0800 (PST) Received: from localhost.localdomain (c-67-170-153-23.hsd1.or.comcast.net. [67.170.153.23]) by mx.google.com with ESMTPSA id y9sm123298660pas.10.2014.02.17.10.45.41 for (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 17 Feb 2014 10:45:42 -0800 (PST) From: John Stultz To: LKML Cc: Stephen Boyd , Peter Zijlstra , Thomas Gleixner , Ingo Molnar , Will Deacon , Josh Cartwright , John Stultz Subject: [PATCH] sched_clock: Prevent callers from seeing half-updated data Date: Mon, 17 Feb 2014 10:45:36 -0800 Message-Id: <1392662736-7803-2-git-send-email-john.stultz@linaro.org> X-Mailer: git-send-email 1.8.3.2 In-Reply-To: <1392662736-7803-1-git-send-email-john.stultz@linaro.org> References: <1392662736-7803-1-git-send-email-john.stultz@linaro.org> X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: john.stultz@linaro.org X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: 209.85.220.173 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Precedence: list Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org List-ID: X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , From: Stephen Boyd The generic sched_clock registration function was previously done lockless, due to the fact that it was expected to be called only once. However, now there are systems that may register multiple sched_clock sources, for which the lack of locking has casued problems: If two sched_clock sources are registered we may end up in a situation where a call to sched_clock() may be accessing the epoch cycle count for the old counter and the cycle count for the new counter. This can lead to confusing results where sched_clock() values jump and then are reset to 0 (due to the way the registration function forces the epoch_ns to be 0). Fix this by reorganizing the registration function to hold the seqlock for as short a time as possible while we update the clock_data structure for a new counter. We also put any accumulated time into epoch_ns instead of resetting the time to 0 so that the clock doesn't reset after each successful registration. Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Stephen Boyd Cc: Will Deacon Cc: Josh Cartwright Reported-by: Will Deacon Signed-off-by: Stephen Boyd [jstultz: Added extra context to the commit message] Signed-off-by: John Stultz --- kernel/time/sched_clock.c | 46 +++++++++++++++++++++++++++++----------------- 1 file changed, 29 insertions(+), 17 deletions(-) diff --git a/kernel/time/sched_clock.c b/kernel/time/sched_clock.c index 0abb364..4d23dc4 100644 --- a/kernel/time/sched_clock.c +++ b/kernel/time/sched_clock.c @@ -116,20 +116,42 @@ static enum hrtimer_restart sched_clock_poll(struct hrtimer *hrt) void __init sched_clock_register(u64 (*read)(void), int bits, unsigned long rate) { + u64 res, wrap, new_mask, new_epoch, cyc, ns; + u32 new_mult, new_shift; + ktime_t new_wrap_kt; unsigned long r; - u64 res, wrap; char r_unit; if (cd.rate > rate) return; WARN_ON(!irqs_disabled()); - read_sched_clock = read; - sched_clock_mask = CLOCKSOURCE_MASK(bits); - cd.rate = rate; /* calculate the mult/shift to convert counter ticks to ns. */ - clocks_calc_mult_shift(&cd.mult, &cd.shift, rate, NSEC_PER_SEC, 3600); + clocks_calc_mult_shift(&new_mult, &new_shift, rate, NSEC_PER_SEC, 3600); + + new_mask = CLOCKSOURCE_MASK(bits); + + /* calculate how many ns until we wrap */ + wrap = clocks_calc_max_nsecs(new_mult, new_shift, 0, new_mask); + new_wrap_kt = ns_to_ktime(wrap - (wrap >> 3)); + + /* update epoch for new counter and update epoch_ns from old counter*/ + new_epoch = read(); + cyc = read_sched_clock(); + ns = cd.epoch_ns + cyc_to_ns((cyc - cd.epoch_cyc) & sched_clock_mask, + cd.mult, cd.shift); + + raw_write_seqcount_begin(&cd.seq); + read_sched_clock = read; + sched_clock_mask = new_mask; + cd.rate = rate; + cd.wrap_kt = new_wrap_kt; + cd.mult = new_mult; + cd.shift = new_shift; + cd.epoch_cyc = new_epoch; + cd.epoch_ns = ns; + raw_write_seqcount_end(&cd.seq); r = rate; if (r >= 4000000) { @@ -141,22 +163,12 @@ void __init sched_clock_register(u64 (*read)(void), int bits, } else r_unit = ' '; - /* calculate how many ns until we wrap */ - wrap = clocks_calc_max_nsecs(cd.mult, cd.shift, 0, sched_clock_mask); - cd.wrap_kt = ns_to_ktime(wrap - (wrap >> 3)); - /* calculate the ns resolution of this counter */ - res = cyc_to_ns(1ULL, cd.mult, cd.shift); + res = cyc_to_ns(1ULL, new_mult, new_shift); + pr_info("sched_clock: %u bits at %lu%cHz, resolution %lluns, wraps every %lluns\n", bits, r, r_unit, res, wrap); - update_sched_clock(); - - /* - * Ensure that sched_clock() starts off at 0ns - */ - cd.epoch_ns = 0; - /* Enable IRQ time accounting if we have a fast enough sched_clock */ if (irqtime > 0 || (irqtime == -1 && rate >= 1000000)) enable_sched_clock_irqtime();