From patchwork Sat Apr 5 12:03:35 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Viresh Kumar X-Patchwork-Id: 27874 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-ig0-f199.google.com (mail-ig0-f199.google.com [209.85.213.199]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id E685120553 for ; Sat, 5 Apr 2014 12:03:58 +0000 (UTC) Received: by mail-ig0-f199.google.com with SMTP id c1sf5065791igq.2 for ; Sat, 05 Apr 2014 05:03:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:delivered-to:from:to:cc:subject :date:message-id:sender:precedence:list-id:x-original-sender :x-original-authentication-results:mailing-list:list-post:list-help :list-archive:list-unsubscribe; bh=ByU00EWf5mO7Dot465tpydbc2tZfRGpYkLWYJO0a2e8=; b=TFftu9bMjzy/vz4QBD7EyVqPEyjQKHR0LYgC/RxLnhNmbUlmf1mDoMBWoMNJBbUmDq mukz2Ht1W6ZezmZ/QfuPWrU/aD3xtik5QIFCcuohoJTSIEuSE/DhkFX3AhXbp8h0Nq4G JmfyldGE2EV4PCcog4mXfr8VMcX66N+0GD4WVO5VxL/q/f/Pi6N2jcubwsguCNM9N8s9 zU2ThvKvLaQ33V7mjqgsJju+QCnuHD7eG/dW5j/j9IVtQWNTSBB3SvZ80hJ/rw7vpQ+J AHZMln+qUIJy0vSBuL+RXxxQTzZTFEOc8HZf79qnihAFg5mYwQCxdEUYfQKQvNM6Hfi2 5w6w== X-Gm-Message-State: ALoCoQk/6KNHEKcjmSv/DSW0vaURKwKZs6ZpBXahCUr5no8yu7C+yh1ShvSLwYCw50LCyUQdvC8C X-Received: by 10.182.51.200 with SMTP id m8mr2504551obo.16.1396699438206; Sat, 05 Apr 2014 05:03:58 -0700 (PDT) MIME-Version: 1.0 X-BeenThere: patchwork-forward@linaro.org Received: by 10.140.23.232 with SMTP id 95ls135442qgp.68.gmail; Sat, 05 Apr 2014 05:03:58 -0700 (PDT) X-Received: by 10.220.12.66 with SMTP id w2mr1851706vcw.15.1396699438098; Sat, 05 Apr 2014 05:03:58 -0700 (PDT) Received: from mail-ve0-f176.google.com (mail-ve0-f176.google.com [209.85.128.176]) by mx.google.com with ESMTPS id at8si1128074vec.91.2014.04.05.05.03.58 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 05 Apr 2014 05:03:58 -0700 (PDT) Received-SPF: neutral (google.com: 209.85.128.176 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) client-ip=209.85.128.176; Received: by mail-ve0-f176.google.com with SMTP id db11so2297089veb.35 for ; Sat, 05 Apr 2014 05:03:58 -0700 (PDT) X-Received: by 10.58.31.136 with SMTP id a8mr717302vei.20.1396699437994; Sat, 05 Apr 2014 05:03:57 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.220.12.8 with SMTP id v8csp21196vcv; Sat, 5 Apr 2014 05:03:57 -0700 (PDT) X-Received: by 10.66.66.66 with SMTP id d2mr20234138pat.24.1396699437173; Sat, 05 Apr 2014 05:03:57 -0700 (PDT) Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id ug9si5807242pab.171.2014.04.05.05.03.56; Sat, 05 Apr 2014 05:03:56 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752741AbaDEMDp (ORCPT + 27 others); Sat, 5 Apr 2014 08:03:45 -0400 Received: from mail-pd0-f171.google.com ([209.85.192.171]:40581 "EHLO mail-pd0-f171.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752578AbaDEMDn (ORCPT ); Sat, 5 Apr 2014 08:03:43 -0400 Received: by mail-pd0-f171.google.com with SMTP id r10so4485998pdi.2 for ; Sat, 05 Apr 2014 05:03:43 -0700 (PDT) X-Received: by 10.66.136.17 with SMTP id pw17mr20507753pab.86.1396699423402; Sat, 05 Apr 2014 05:03:43 -0700 (PDT) Received: from localhost ([122.167.72.32]) by mx.google.com with ESMTPSA id c7sm24028064pbt.0.2014.04.05.05.03.39 for (version=TLSv1.2 cipher=RC4-SHA bits=128/128); Sat, 05 Apr 2014 05:03:42 -0700 (PDT) From: Viresh Kumar To: tglx@linutronix.de, john.stultz@linaro.org Cc: linaro-kernel@lists.linaro.org, fengguang.wu@intel.com, jet.chen@intel.com, linux-kernel@vger.kernel.org, linaro-networking@linaro.org, Viresh Kumar Subject: [PATCH] clocksource: register cpu notifier to remove timer from dying CPU Date: Sat, 5 Apr 2014 17:33:35 +0530 Message-Id: <227e5e019c175010b593d41cf1154e2e6505c22e.1396699157.git.viresh.kumar@linaro.org> X-Mailer: git-send-email 1.7.12.rc2.18.g61b472e Sender: linux-kernel-owner@vger.kernel.org Precedence: list List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: viresh.kumar@linaro.org X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: 209.85.128.176 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , clocksource core is using add_timer_on() to run clocksource_watchdog() on all CPUs one by one. But when a core is brought down, clocksource core doesn't remove this timer from the dying CPU. And in this case timer core gives this (Gives this only with unmerged code, anyway in the current code as well timer core is migrating a pinned timer to other CPUs, which is also wrong: http://www.gossamer-threads.com/lists/linux/kernel/1898117) migrate_timer_list: can't migrate pinned timer: ffffffff81f06a60, timer->function: ffffffff810d7010,deactivating it Modules linked in: CPU: 0 PID: 1932 Comm: 01-cpu-hotplug Not tainted 3.14.0-rc1-00088-gab3c4fd #4 Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 0000000000000009 ffff88001d407c38 ffffffff817237bd ffff88001d407c80 ffff88001d407c70 ffffffff8106a1dd 0000000000000010 ffffffff81f06a60 ffff88001e04d040 ffffffff81e3d4c0 ffff88001e04d030 ffff88001d407cd0 Call Trace: [] dump_stack+0x4d/0x66 [] warn_slowpath_common+0x7d/0xa0 [] warn_slowpath_fmt+0x4c/0x50 [] ? __internal_add_timer+0x113/0x130 [] ? clocksource_watchdog_kthread+0x40/0x40 [] migrate_timer_list+0xdb/0xf0 [] timer_cpu_notify+0xfc/0x1f0 [] notifier_call_chain+0x4c/0x70 [] __raw_notifier_call_chain+0xe/0x10 [] cpu_notify+0x23/0x50 [] cpu_notify_nofail+0xe/0x20 [] _cpu_down+0x1ad/0x2e0 [] cpu_down+0x34/0x50 [] cpu_subsys_offline+0x14/0x20 [] device_offline+0x95/0xc0 [] online_store+0x40/0x90 [] dev_attr_store+0x18/0x30 [] sysfs_kf_write+0x3d/0x50 This patch tries to fix this by registering cpu notifiers from clocksource core, only when we start clocksource-watchdog. And if on the CPU_DEAD notification it is found that dying CPU was the CPU on which this timer is queued on, then it is removed from that CPU and queued to next CPU. Reported-by: Fengguang Wu Reported-and-tested-by: Jet Chen Signed-off-by: Viresh Kumar --- kernel/time/clocksource.c | 64 +++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 53 insertions(+), 11 deletions(-) diff --git a/kernel/time/clocksource.c b/kernel/time/clocksource.c index ba3e502..8054fb1 100644 --- a/kernel/time/clocksource.c +++ b/kernel/time/clocksource.c @@ -23,16 +23,21 @@ * o Allow clocksource drivers to be unregistered */ +#include #include #include #include #include +#include #include /* for spin_unlock_irq() using preempt_count() m68k */ #include #include #include "tick-internal.h" +/* Tracks current CPU to queue watchdog timer on */ +static int timer_cpu; + void timecounter_init(struct timecounter *tc, const struct cyclecounter *cc, u64 start_tstamp) @@ -246,12 +251,25 @@ void clocksource_mark_unstable(struct clocksource *cs) spin_unlock_irqrestore(&watchdog_lock, flags); } +static void queue_timer_on_next_cpu(void) +{ + /* + * Cycle through CPUs to check if the CPUs stay synchronized to each + * other. + */ + timer_cpu = cpumask_next(timer_cpu, cpu_online_mask); + if (timer_cpu >= nr_cpu_ids) + timer_cpu = cpumask_first(cpu_online_mask); + watchdog_timer.expires = jiffies + WATCHDOG_INTERVAL; + add_timer_on(&watchdog_timer, timer_cpu); +} + static void clocksource_watchdog(unsigned long data) { struct clocksource *cs; cycle_t csnow, wdnow; int64_t wd_nsec, cs_nsec; - int next_cpu, reset_pending; + int reset_pending; spin_lock(&watchdog_lock); if (!watchdog_running) @@ -336,27 +354,50 @@ static void clocksource_watchdog(unsigned long data) if (reset_pending) atomic_dec(&watchdog_reset_pending); - /* - * Cycle through CPUs to check if the CPUs stay synchronized - * to each other. - */ - next_cpu = cpumask_next(raw_smp_processor_id(), cpu_online_mask); - if (next_cpu >= nr_cpu_ids) - next_cpu = cpumask_first(cpu_online_mask); - watchdog_timer.expires += WATCHDOG_INTERVAL; - add_timer_on(&watchdog_timer, next_cpu); + queue_timer_on_next_cpu(); out: spin_unlock(&watchdog_lock); } +static int clocksource_cpu_notify(struct notifier_block *self, + unsigned long action, void *hcpu) +{ + long cpu = (long)hcpu; + + spin_lock(&watchdog_lock); + if (!watchdog_running) + goto notify_out; + + switch (action) { + case CPU_DEAD: + case CPU_DEAD_FROZEN: + if (cpu != timer_cpu) + break; + del_timer(&watchdog_timer); + queue_timer_on_next_cpu(); + break; + } + +notify_out: + spin_unlock(&watchdog_lock); + return NOTIFY_OK; +} + +static struct notifier_block clocksource_nb = { + .notifier_call = clocksource_cpu_notify, + .priority = 1, +}; + static inline void clocksource_start_watchdog(void) { if (watchdog_running || !watchdog || list_empty(&watchdog_list)) return; + timer_cpu = cpumask_first(cpu_online_mask); + register_cpu_notifier(&clocksource_nb); init_timer(&watchdog_timer); watchdog_timer.function = clocksource_watchdog; watchdog_timer.expires = jiffies + WATCHDOG_INTERVAL; - add_timer_on(&watchdog_timer, cpumask_first(cpu_online_mask)); + add_timer_on(&watchdog_timer, timer_cpu); watchdog_running = 1; } @@ -365,6 +406,7 @@ static inline void clocksource_stop_watchdog(void) if (!watchdog_running || (watchdog && !list_empty(&watchdog_list))) return; del_timer(&watchdog_timer); + unregister_cpu_notifier(&clocksource_nb); watchdog_running = 0; }