From patchwork Fri Mar 2 07:12:42 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 7054 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id A40D823E4A for ; Fri, 2 Mar 2012 07:13:08 +0000 (UTC) Received: from mail-iy0-f180.google.com (mail-iy0-f180.google.com [209.85.210.180]) by fiordland.canonical.com (Postfix) with ESMTP id 5325AA184A6 for ; Fri, 2 Mar 2012 07:13:08 +0000 (UTC) Received: by mail-iy0-f180.google.com with SMTP id e36so2587473iag.11 for ; Thu, 01 Mar 2012 23:13:08 -0800 (PST) Received: from mr.google.com ([10.50.89.201]) by 10.50.89.201 with SMTP id bq9mr734447igb.55.1330672388179 (num_hops = 1); Thu, 01 Mar 2012 23:13:08 -0800 (PST) MIME-Version: 1.0 Received: by 10.50.89.201 with SMTP id bq9mr611074igb.55.1330672388110; Thu, 01 Mar 2012 23:13:08 -0800 (PST) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.231.53.18 with SMTP id k18csp3346ibg; Thu, 1 Mar 2012 23:13:07 -0800 (PST) Received: by 10.101.176.6 with SMTP id d6mr2880189anp.27.1330672386987; Thu, 01 Mar 2012 23:13:06 -0800 (PST) Received: from e7.ny.us.ibm.com (e7.ny.us.ibm.com. [32.97.182.137]) by mx.google.com with ESMTPS id d68si5464052yhe.7.2012.03.01.23.13.06 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 01 Mar 2012 23:13:06 -0800 (PST) Received-SPF: pass (google.com: domain of jstultz@us.ibm.com designates 32.97.182.137 as permitted sender) client-ip=32.97.182.137; Authentication-Results: mx.google.com; spf=pass (google.com: domain of jstultz@us.ibm.com designates 32.97.182.137 as permitted sender) smtp.mail=jstultz@us.ibm.com Received: from /spool/local by e7.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 2 Mar 2012 02:13:06 -0500 Received: from d01dlp02.pok.ibm.com (9.56.224.85) by e7.ny.us.ibm.com (192.168.1.107) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Fri, 2 Mar 2012 02:13:03 -0500 Received: from d01relay04.pok.ibm.com (d01relay04.pok.ibm.com [9.56.227.236]) by d01dlp02.pok.ibm.com (Postfix) with ESMTP id EACAF6E804F; Fri, 2 Mar 2012 02:13:02 -0500 (EST) Received: from d03av04.boulder.ibm.com (d03av04.boulder.ibm.com [9.17.195.170]) by d01relay04.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id q227D0Yo323746; Fri, 2 Mar 2012 02:13:02 -0500 Received: from d03av04.boulder.ibm.com (loopback [127.0.0.1]) by d03av04.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id q227D00h004223; Fri, 2 Mar 2012 00:13:00 -0700 Received: from kernel.beaverton.ibm.com (kernel.beaverton.ibm.com [9.47.67.96]) by d03av04.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id q227Cx53004209; Fri, 2 Mar 2012 00:12:59 -0700 Received: by kernel.beaverton.ibm.com (Postfix, from userid 1056) id CC48CC03E4; Thu, 1 Mar 2012 23:12:58 -0800 (PST) From: John Stultz To: lkml Cc: John Stultz , Ingo Molnar , Thomas Gleixner , Eric Dumazet , Richard Cochran Subject: [PATCH 3/9] time: Split timekeeper lock into separate reader/writer locks Date: Thu, 1 Mar 2012 23:12:42 -0800 Message-Id: <1330672368-32290-4-git-send-email-john.stultz@linaro.org> X-Mailer: git-send-email 1.7.3.2.146.gca209 In-Reply-To: <1330672368-32290-1-git-send-email-john.stultz@linaro.org> References: <1330672368-32290-1-git-send-email-john.stultz@linaro.org> X-Content-Scanned: Fidelis XPS MAILER x-cbid: 12030207-5806-0000-0000-00001314C091 X-Gm-Message-State: ALoCoQlDHNnnVcEDfi3TWvBh50DCuLKd5Zgjoa2ympqV6bJoJJJqAuND5ONX+RN1NLr70MvY5Qgk In order to reduce the lock hold time, split the timekeeper lock into a writer lock, which serializes updates to the timekeeper structure, and a reader sequence counter, which ensures readers see a consistent version of the timekeeper. This will allow us to reduce the lock wait time for readers, by doing updates on a shadow copy of the timekeeper. This patch also has been reworked to move the split locks out of the timekeeper structure, so that we don't break lockdep when updating from the shadow copy CC: Ingo Molnar CC: Thomas Gleixner CC: Eric Dumazet CC: Richard Cochran Signed-off-by: John Stultz --- kernel/time/timekeeping.c | 116 +++++++++++++++++++++++++++----------------- 1 files changed, 71 insertions(+), 45 deletions(-) diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index 74568ca..f9ee96c 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -71,12 +71,16 @@ struct timekeeper { /* The raw monotonic time for the CLOCK_MONOTONIC_RAW posix clock. */ struct timespec raw_time; - /* Seqlock for all timekeeper values */ - seqlock_t lock; }; static struct timekeeper timekeeper; +/* Locks for timekeeper variable: */ +/* This seqcount serializes readers from updates */ +static seqcount_t timekeeper_rlock; +/* This spinlock serializes updaters */ +static spinlock_t timekeeper_wlock; + /* * This read-write spinlock protects us from races in SMP while * playing with xtime. @@ -212,7 +216,7 @@ static inline s64 timekeeping_get_ns_raw(struct timekeeper *tk) return clocksource_cyc2ns(cycle_delta, clock->mult, clock->shift); } -/* must hold write on timekeeper.lock */ +/* must hold write on timekeeper_wlock */ static void timekeeping_update(struct timekeeper *tk, bool clearntp) { struct timespec xt; @@ -229,11 +233,13 @@ void timekeeping_leap_insert(int leapsecond) { unsigned long flags; - write_seqlock_irqsave(&timekeeper.lock, flags); + spin_lock_irqsave(&timekeeper_wlock, flags); + write_seqcount_begin(&timekeeper_rlock); timekeeper.xtime_sec += leapsecond; timekeeper.wall_to_monotonic.tv_sec -= leapsecond; timekeeping_update(&timekeeper, false); - write_sequnlock_irqrestore(&timekeeper.lock, flags); + write_seqcount_end(&timekeeper_rlock); + spin_unlock_irqrestore(&timekeeper_wlock, flags); } @@ -280,7 +286,7 @@ void getnstimeofday(struct timespec *ts) WARN_ON(timekeeping_suspended); do { - seq = read_seqbegin(&timekeeper.lock); + seq = read_seqcount_begin(&timekeeper_rlock); ts->tv_sec = timekeeper.xtime_sec; ts->tv_nsec = timekeeping_get_ns(&timekeeper); @@ -288,7 +294,7 @@ void getnstimeofday(struct timespec *ts) /* If arch requires, add in gettimeoffset() */ nsecs += arch_gettimeoffset(); - } while (read_seqretry(&timekeeper.lock, seq)); + } while (read_seqcount_retry(&timekeeper_rlock, seq)); timespec_add_ns(ts, nsecs); } @@ -303,7 +309,7 @@ ktime_t ktime_get(void) WARN_ON(timekeeping_suspended); do { - seq = read_seqbegin(&timekeeper.lock); + seq = read_seqcount_begin(&timekeeper_rlock); secs = timekeeper.xtime_sec + timekeeper.wall_to_monotonic.tv_sec; nsecs = timekeeping_get_ns(&timekeeper) + @@ -311,7 +317,7 @@ ktime_t ktime_get(void) /* If arch requires, add in gettimeoffset() */ nsecs += arch_gettimeoffset(); - } while (read_seqretry(&timekeeper.lock, seq)); + } while (read_seqcount_retry(&timekeeper_rlock, seq)); /* * Use ktime_set/ktime_add_ns to create a proper ktime on * 32-bit architectures without CONFIG_KTIME_SCALAR. @@ -336,14 +342,14 @@ void ktime_get_ts(struct timespec *ts) WARN_ON(timekeeping_suspended); do { - seq = read_seqbegin(&timekeeper.lock); + seq = read_seqcount_begin(&timekeeper_rlock); ts->tv_sec = timekeeper.xtime_sec; ts->tv_nsec = timekeeping_get_ns(&timekeeper); tomono = timekeeper.wall_to_monotonic; /* If arch requires, add in gettimeoffset() */ ts->tv_nsec += arch_gettimeoffset(); - } while (read_seqretry(&timekeeper.lock, seq)); + } while (read_seqcount_retry(&timekeeper_rlock, seq)); set_normalized_timespec(ts, ts->tv_sec + tomono.tv_sec, ts->tv_nsec + tomono.tv_nsec); @@ -371,7 +377,7 @@ void getnstime_raw_and_real(struct timespec *ts_raw, struct timespec *ts_real) do { u32 arch_offset; - seq = read_seqbegin(&timekeeper.lock); + seq = read_seqcount_begin(&timekeeper_rlock); *ts_raw = timekeeper.raw_time; ts_real->tv_sec = timekeeper.xtime_sec; @@ -385,7 +391,7 @@ void getnstime_raw_and_real(struct timespec *ts_raw, struct timespec *ts_real) nsecs_raw += arch_offset; nsecs_real += arch_offset; - } while (read_seqretry(&timekeeper.lock, seq)); + } while (read_seqcount_retry(&timekeeper_rlock, seq)); timespec_add_ns(ts_raw, nsecs_raw); timespec_add_ns(ts_real, nsecs_real); @@ -424,7 +430,8 @@ int do_settimeofday(const struct timespec *tv) if ((unsigned long)tv->tv_nsec >= NSEC_PER_SEC) return -EINVAL; - write_seqlock_irqsave(&timekeeper.lock, flags); + spin_lock_irqsave(&timekeeper_wlock, flags); + write_seqcount_begin(&timekeeper_rlock); timekeeping_forward_now(&timekeeper); @@ -439,7 +446,8 @@ int do_settimeofday(const struct timespec *tv) timekeeping_update(&timekeeper, true); - write_sequnlock_irqrestore(&timekeeper.lock, flags); + write_seqcount_end(&timekeeper_rlock); + spin_unlock_irqrestore(&timekeeper_wlock, flags); /* signal hrtimers about time change */ clock_was_set(); @@ -463,7 +471,8 @@ int timekeeping_inject_offset(struct timespec *ts) if ((unsigned long)ts->tv_nsec >= NSEC_PER_SEC) return -EINVAL; - write_seqlock_irqsave(&timekeeper.lock, flags); + spin_lock_irqsave(&timekeeper_wlock, flags); + write_seqcount_begin(&timekeeper_rlock); timekeeping_forward_now(&timekeeper); @@ -474,7 +483,8 @@ int timekeeping_inject_offset(struct timespec *ts) timekeeping_update(&timekeeper, true); - write_sequnlock_irqrestore(&timekeeper.lock, flags); + write_seqcount_end(&timekeeper_rlock); + spin_unlock_irqrestore(&timekeeper_wlock, flags); /* signal hrtimers about time change */ clock_was_set(); @@ -546,11 +556,11 @@ void getrawmonotonic(struct timespec *ts) s64 nsecs; do { - seq = read_seqbegin(&timekeeper.lock); + seq = read_seqcount_begin(&timekeeper_rlock); nsecs = timekeeping_get_ns_raw(&timekeeper); *ts = timekeeper.raw_time; - } while (read_seqretry(&timekeeper.lock, seq)); + } while (read_seqcount_retry(&timekeeper_rlock, seq)); timespec_add_ns(ts, nsecs); } @@ -566,11 +576,11 @@ int timekeeping_valid_for_hres(void) int ret; do { - seq = read_seqbegin(&timekeeper.lock); + seq = read_seqcount_begin(&timekeeper_rlock); ret = timekeeper.clock->flags & CLOCK_SOURCE_VALID_FOR_HRES; - } while (read_seqretry(&timekeeper.lock, seq)); + } while (read_seqcount_retry(&timekeeper_rlock, seq)); return ret; } @@ -583,11 +593,11 @@ u64 timekeeping_max_deferment(void) unsigned long seq; u64 ret; do { - seq = read_seqbegin(&timekeeper.lock); + seq = read_seqcount_begin(&timekeeper_rlock); ret = timekeeper.clock->max_idle_ns; - } while (read_seqretry(&timekeeper.lock, seq)); + } while (read_seqcount_retry(&timekeeper_rlock, seq)); return ret; } @@ -634,11 +644,13 @@ void __init timekeeping_init(void) read_persistent_clock(&now); read_boot_clock(&boot); - seqlock_init(&timekeeper.lock); - + seqcount_init(&timekeeper_rlock); + spin_lock_init(&timekeeper_wlock); ntp_init(); - write_seqlock_irqsave(&timekeeper.lock, flags); + spin_lock_irqsave(&timekeeper_wlock, flags); + write_seqcount_begin(&timekeeper_rlock); + clock = clocksource_default_clock(); if (clock->enable) clock->enable(clock); @@ -654,7 +666,10 @@ void __init timekeeping_init(void) -boot.tv_sec, -boot.tv_nsec); timekeeper.total_sleep_time.tv_sec = 0; timekeeper.total_sleep_time.tv_nsec = 0; - write_sequnlock_irqrestore(&timekeeper.lock, flags); + + write_seqcount_end(&timekeeper_rlock); + spin_unlock_irqrestore(&timekeeper_wlock, flags); + } /* time in seconds when suspend began */ @@ -703,7 +718,8 @@ void timekeeping_inject_sleeptime(struct timespec *delta) if (!(ts.tv_sec == 0 && ts.tv_nsec == 0)) return; - write_seqlock_irqsave(&timekeeper.lock, flags); + spin_lock_irqsave(&timekeeper_wlock, flags); + write_seqcount_begin(&timekeeper_rlock); timekeeping_forward_now(&timekeeper); @@ -711,7 +727,8 @@ void timekeeping_inject_sleeptime(struct timespec *delta) timekeeping_update(&timekeeper, true); - write_sequnlock_irqrestore(&timekeeper.lock, flags); + write_seqcount_end(&timekeeper_rlock); + spin_unlock_irqrestore(&timekeeper_wlock, flags); /* signal hrtimers about time change */ clock_was_set(); @@ -734,7 +751,8 @@ static void timekeeping_resume(void) clocksource_resume(); - write_seqlock_irqsave(&timekeeper.lock, flags); + spin_lock_irqsave(&timekeeper_wlock, flags); + write_seqcount_begin(&timekeeper_rlock); if (timespec_compare(&ts, &timekeeping_suspend_time) > 0) { ts = timespec_sub(ts, timekeeping_suspend_time); @@ -744,7 +762,9 @@ static void timekeeping_resume(void) timekeeper.clock->cycle_last = timekeeper.clock->read(timekeeper.clock); timekeeper.ntp_error = 0; timekeeping_suspended = 0; - write_sequnlock_irqrestore(&timekeeper.lock, flags); + + write_seqcount_end(&timekeeper_rlock); + spin_unlock_irqrestore(&timekeeper_wlock, flags); touch_softlockup_watchdog(); @@ -762,7 +782,9 @@ static int timekeeping_suspend(void) read_persistent_clock(&timekeeping_suspend_time); - write_seqlock_irqsave(&timekeeper.lock, flags); + spin_lock_irqsave(&timekeeper_wlock, flags); + write_seqcount_begin(&timekeeper_rlock); + timekeeping_forward_now(&timekeeper); timekeeping_suspended = 1; @@ -785,7 +807,9 @@ static int timekeeping_suspend(void) timekeeping_suspend_time = timespec_add(timekeeping_suspend_time, delta_delta); } - write_sequnlock_irqrestore(&timekeeper.lock, flags); + + write_seqcount_end(&timekeeper_rlock); + spin_unlock_irqrestore(&timekeeper_wlock, flags); clockevents_notify(CLOCK_EVT_NOTIFY_SUSPEND, NULL); clocksource_suspend(); @@ -1041,7 +1065,8 @@ static void update_wall_time(void) unsigned long flags; s64 remainder; - write_seqlock_irqsave(&timekeeper.lock, flags); + spin_lock_irqsave(&timekeeper_wlock, flags); + write_seqcount_begin(&timekeeper_rlock); /* Make sure we're fully resumed: */ if (unlikely(timekeeping_suspended)) @@ -1128,7 +1153,8 @@ static void update_wall_time(void) timekeeping_update(&timekeeper, false); out: - write_sequnlock_irqrestore(&timekeeper.lock, flags); + write_seqcount_end(&timekeeper_rlock); + spin_unlock_irqrestore(&timekeeper_wlock, flags); } @@ -1174,13 +1200,13 @@ void get_monotonic_boottime(struct timespec *ts) WARN_ON(timekeeping_suspended); do { - seq = read_seqbegin(&timekeeper.lock); + seq = read_seqcount_begin(&timekeeper_rlock); ts->tv_sec = timekeeper.xtime_sec; ts->tv_nsec = timekeeping_get_ns(&timekeeper); tomono = timekeeper.wall_to_monotonic; sleep = timekeeper.total_sleep_time; - } while (read_seqretry(&timekeeper.lock, seq)); + } while (read_seqcount_retry(&timekeeper_rlock, seq)); set_normalized_timespec(ts, ts->tv_sec + tomono.tv_sec + sleep.tv_sec, ts->tv_nsec + tomono.tv_nsec + sleep.tv_nsec); @@ -1231,10 +1257,10 @@ struct timespec current_kernel_time(void) unsigned long seq; do { - seq = read_seqbegin(&timekeeper.lock); + seq = read_seqcount_begin(&timekeeper_rlock); now = tk_xtime(&timekeeper); - } while (read_seqretry(&timekeeper.lock, seq)); + } while (read_seqcount_retry(&timekeeper_rlock, seq)); return now; } @@ -1246,11 +1272,11 @@ struct timespec get_monotonic_coarse(void) unsigned long seq; do { - seq = read_seqbegin(&timekeeper.lock); + seq = read_seqcount_begin(&timekeeper_rlock); now = tk_xtime(&timekeeper); mono = timekeeper.wall_to_monotonic; - } while (read_seqretry(&timekeeper.lock, seq)); + } while (read_seqcount_retry(&timekeeper_rlock, seq)); set_normalized_timespec(&now, now.tv_sec + mono.tv_sec, now.tv_nsec + mono.tv_nsec); @@ -1282,11 +1308,11 @@ void get_xtime_and_monotonic_and_sleep_offset(struct timespec *xtim, unsigned long seq; do { - seq = read_seqbegin(&timekeeper.lock); + seq = read_seqcount_begin(&timekeeper_rlock); *xtim = tk_xtime(&timekeeper); *wtom = timekeeper.wall_to_monotonic; *sleep = timekeeper.total_sleep_time; - } while (read_seqretry(&timekeeper.lock, seq)); + } while (read_seqcount_retry(&timekeeper_rlock, seq)); } /** @@ -1298,9 +1324,9 @@ ktime_t ktime_get_monotonic_offset(void) struct timespec wtom; do { - seq = read_seqbegin(&timekeeper.lock); + seq = read_seqcount_begin(&timekeeper_rlock); wtom = timekeeper.wall_to_monotonic; - } while (read_seqretry(&timekeeper.lock, seq)); + } while (read_seqcount_retry(&timekeeper_rlock, seq)); return timespec_to_ktime(wtom); }