From patchwork Thu Feb 20 08:19:39 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juri Lelli X-Patchwork-Id: 24974 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-ob0-f198.google.com (mail-ob0-f198.google.com [209.85.214.198]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id ACBC9203C6 for ; Thu, 20 Feb 2014 08:19:58 +0000 (UTC) Received: by mail-ob0-f198.google.com with SMTP id vb8sf4542940obc.9 for ; Thu, 20 Feb 2014 00:19:58 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:delivered-to:from:to:cc:subject :date:message-id:sender:precedence:list-id:x-original-sender :x-original-authentication-results:mailing-list:list-post:list-help :list-archive:list-unsubscribe; bh=QeCysrFMvGYQBk7SCRtGFdbthSGtahbfN4fLZKLjCdU=; b=RY8oEft2NU45Ny+zBn/PzUkBZS0DsrGvhY79kXOPBm6nO03nO9s77s7MBx+CayGHiQ M5fSzp8NSWPc3kHga5ybQg6Q/ssVfb6ZZWzFJkjLgIoGGNbdrQhbblRZfNRbgK4n3RZM dnzO0WD0yul1BnziLUfWKYyc8RDpDsHXCrLRgPaG1QxGElPdb8B1J/DSa/gZDssiQ4Mn AqfuzMMUOdmIcWimDBMxITGsYtIYsjv2VDFCaoyio/Ofd3c7rFVixs5J9vruiRRWRYxX ryqLBy1BM1G6VByuJpL6Ejv5PdRBSVGVoiTFsT08ZjKTbtMXKzgMIr6TMaVGD3KW/XXS 8QgA== X-Gm-Message-State: ALoCoQmGcRfsHSQNqKBIKdQrT8IHJlj/WUBOH13Djgcmppqr/wxcPhYbjTSpB//BId+SbsNflNv1 X-Received: by 10.42.51.141 with SMTP id e13mr191161icg.28.1392884398205; Thu, 20 Feb 2014 00:19:58 -0800 (PST) MIME-Version: 1.0 X-BeenThere: patchwork-forward@linaro.org Received: by 10.140.20.100 with SMTP id 91ls421595qgi.18.gmail; Thu, 20 Feb 2014 00:19:58 -0800 (PST) X-Received: by 10.220.98.143 with SMTP id q15mr282648vcn.38.1392884398060; Thu, 20 Feb 2014 00:19:58 -0800 (PST) Received: from mail-ve0-x236.google.com (mail-ve0-x236.google.com [2607:f8b0:400c:c01::236]) by mx.google.com with ESMTPS id t20si1158620vek.117.2014.02.20.00.19.58 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 20 Feb 2014 00:19:58 -0800 (PST) Received-SPF: neutral (google.com: 2607:f8b0:400c:c01::236 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) client-ip=2607:f8b0:400c:c01::236; Received: by mail-ve0-f182.google.com with SMTP id jy13so1565084veb.27 for ; Thu, 20 Feb 2014 00:19:58 -0800 (PST) X-Received: by 10.52.106.166 with SMTP id gv6mr205662vdb.86.1392884397950; Thu, 20 Feb 2014 00:19:57 -0800 (PST) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.220.174.196 with SMTP id u4csp38229vcz; Thu, 20 Feb 2014 00:19:57 -0800 (PST) X-Received: by 10.68.226.9 with SMTP id ro9mr604557pbc.72.1392884396920; Thu, 20 Feb 2014 00:19:56 -0800 (PST) Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id zj6si2542436pac.1.2014.02.20.00.19.55; Thu, 20 Feb 2014 00:19:55 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753561AbaBTITt (ORCPT + 26 others); Thu, 20 Feb 2014 03:19:49 -0500 Received: from mail-ea0-f179.google.com ([209.85.215.179]:34210 "EHLO mail-ea0-f179.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752522AbaBTITs (ORCPT ); Thu, 20 Feb 2014 03:19:48 -0500 Received: by mail-ea0-f179.google.com with SMTP id q10so726344ead.38 for ; Thu, 20 Feb 2014 00:19:46 -0800 (PST) X-Received: by 10.14.174.193 with SMTP id x41mr385454eel.87.1392884386619; Thu, 20 Feb 2014 00:19:46 -0800 (PST) Received: from neville.retis (nat-cataldo.sssup.it. [193.205.81.5]) by mx.google.com with ESMTPSA id d9sm10794838eei.9.2014.02.20.00.19.44 for (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 20 Feb 2014 00:19:45 -0800 (PST) From: Juri Lelli To: peterz@infradead.org, rostedt@goodmis.org Cc: mingo@redhat.com, tglx@linutronix.de, linux-kernel@vger.kernel.org, torvalds@linux-foundation.org, akpm@linux-foundation.org, Juri Lelli Subject: [PATCH v4] sched/deadline: Fix bad accounting of nr_running Date: Thu, 20 Feb 2014 09:19:39 +0100 Message-Id: <1392884379-13744-1-git-send-email-juri.lelli@gmail.com> X-Mailer: git-send-email 1.7.9.5 Sender: linux-kernel-owner@vger.kernel.org Precedence: list List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Original-Sender: juri.lelli@gmail.com X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: 2607:f8b0:400c:c01::236 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org; dkim=neutral (bad format) header.i=@gmail.com; dmarc=fail (p=NONE dis=NONE) header.from=gmail.com Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , >From Rostedt's v3: https://lkml.org/lkml/2014/2/18/882 My test suite was locking up hard when enabling mmiotracer. This was due to the mmiotracer placing all but one CPU offline. I found this out when I was able to reproduce the bug with just my stress-cpu-hotplug test. This bug baffled me because it would not always trigger, and would only trigger on the first run after boot up. The stress-cpu-hotplug test would crash hard the first run, or never crash at all. But a new reboot may cause it to crash on the first run again. I spent all week bisecting this, as I couldn't find a consistent reproducer. I finally narrowed it down to the sched deadline patches, and even more peculiar, to the commit that added the sched deadline boot up self test to the latency tracer. Then it dawned on me to what the bug was. All it took was to run a task under sched deadline to screw up the CPU hot plugging. This explained why it would lock up only on the first run of the stress-cpu-hotplug test. The bug happened when the boot up self test of the schedule latency tracer would test a deadline task. The deadline task would corrupt something that would cause CPU hotplug to fail. If it didn't corrupt it, the stress test would always work (there's no other sched deadline tasks that would run to cause problems). If it did corrupt on boot up, the first test would lockup hard. I proved this theory by running my deadline test program on another box, and then run the stress-cpu-hotplug test, and it would now consistently lock up. I could run stress-cpu-hotplug over and over with no problem, but once I ran the deadline test, the next run of the stress-cpu-hotplug would lock hard. After adding lots of tracing to the code, I found the cause. The function tracer showed that migrate_tasks() was stuck in an infinite loop, where rq->nr_running never equaled 1 to break out of it. When I added a trace_printk() to see what that number was, it was 335 and never decrementing! Looking at the deadline code I found: static void __dequeue_task_dl(struct rq *rq, struct task_struct *p, int flags) { dequeue_dl_entity(&p->dl); dequeue_pushable_dl_task(rq, p); } static void dequeue_task_dl(struct rq *rq, struct task_struct *p, int flags) { update_curr_dl(rq); __dequeue_task_dl(rq, p, flags); dec_nr_running(rq); } And this: if (dl_runtime_exceeded(rq, dl_se)) { __dequeue_task_dl(rq, curr, 0); if (likely(start_dl_timer(dl_se, curr->dl.dl_boosted))) dl_se->dl_throttled = 1; else enqueue_task_dl(rq, curr, ENQUEUE_REPLENISH); if (!is_leftmost(curr, &rq->dl)) resched_task(curr); } Notice how we call __dequeue_task_dl() and in the else case we call enqueue_task_dl()? Also notice that dequeue_task_dl() has underscores where enqueue_task_dl() does not. The enqueue_task_dl() calls inc_nr_running(rq), but __dequeue_task_dl() does not. This is where we get nr_running out of sync. [snip] Another point where nr_running can get out of sync is when the dl_timer fires: dl_se->dl_throttled = 0; if (p->on_rq) { enqueue_task_dl(rq, p, ENQUEUE_REPLENISH); if (task_has_dl_policy(rq->curr)) check_preempt_curr_dl(rq, p, 0); else resched_task(rq->curr); This patch does two things: - correctly accounts for throttled tasks (that are now considered !running); - fixes the bug, updating nr_running from {inc,dec}_dl_tasks(), since we risk to update it twice in some situations (e.g., a task is dequeued while it has exceeded its budget). Reported-by: Steven Rostedt Reviewed-by: Steven Rostedt Tested-by: Steven Rostedt Signed-off-by: Juri Lelli --- kernel/sched/deadline.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index 0dd5e09..b819577 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -717,6 +717,7 @@ void inc_dl_tasks(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq) WARN_ON(!dl_prio(prio)); dl_rq->dl_nr_running++; + inc_nr_running(rq_of_dl_rq(dl_rq)); inc_dl_deadline(dl_rq, deadline); inc_dl_migration(dl_se, dl_rq); @@ -730,6 +731,7 @@ void dec_dl_tasks(struct sched_dl_entity *dl_se, struct dl_rq *dl_rq) WARN_ON(!dl_prio(prio)); WARN_ON(!dl_rq->dl_nr_running); dl_rq->dl_nr_running--; + dec_nr_running(rq_of_dl_rq(dl_rq)); dec_dl_deadline(dl_rq, dl_se->deadline); dec_dl_migration(dl_se, dl_rq); @@ -836,8 +838,6 @@ static void enqueue_task_dl(struct rq *rq, struct task_struct *p, int flags) if (!task_current(rq, p) && p->nr_cpus_allowed > 1) enqueue_pushable_dl_task(rq, p); - - inc_nr_running(rq); } static void __dequeue_task_dl(struct rq *rq, struct task_struct *p, int flags) @@ -850,8 +850,6 @@ static void dequeue_task_dl(struct rq *rq, struct task_struct *p, int flags) { update_curr_dl(rq); __dequeue_task_dl(rq, p, flags); - - dec_nr_running(rq); } /*