From patchwork Thu Feb 1 16:51:09 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathieu Poirier X-Patchwork-Id: 126558 Delivered-To: patch@linaro.org Received: by 10.46.124.24 with SMTP id x24csp1878299ljc; Thu, 1 Feb 2018 08:51:43 -0800 (PST) X-Google-Smtp-Source: AH8x226nLeAXBxasYg+Cs+EztUPW9guPPGDHDw+6/NWus6XzqsPik423W/USOJNTDPlhMM4tzJlg X-Received: by 10.101.92.129 with SMTP id a1mr29875810pgt.198.1517503903283; Thu, 01 Feb 2018 08:51:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1517503903; cv=none; d=google.com; s=arc-20160816; b=MGX9Fz8ijeYImr9Y6EVEO9OPfgJKm3r1jBCLZl5EiQUBtvoi0ss82l9CNRdn3Tdsh5 wB/ur2oYaN95jQXFl/kCIT7pVTcD+1KsZceABtOihf/qiUhbb5AHbyjAtyJ2+oWa5nIf TS1TGAcZrS/Z8XjTptV6wojStYtBkJ16crL9eeoAB4iw3DCtmSpR3VfZLEHmzK+zvOXI qh1hAyM+oehWgiykz3+xBY601ncBmWj6wZZbBJQ2qNwUlJIs8GoPDpT8TnzYv3ok8uuq HK0lhnISSWUbl2eg9pbhPhq0uTlStedhw+6+6ZDuRJ4nics9vRza+Vnks+O2x9E3bUJ0 Q/ww== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=cw439ri7YDuH2p/A5GKIcVjp/mn2EguSohu8jazS960=; b=KneSbSqVcIh7F7mWXIF0CZ6PmiMs+dE1R/xzePS4aJml+ZSoDEceSnj60JCBoPCoLO YvtNnFf8msQdFO6G3lr8R5kD+Ex7GjmyheUuK9T62fakpxSMvDlqFEwcd2Qo9vSARAtQ EVo9xYmG26k9Dzo6lTgyPk26pVR2RzDvqNO1ysZ8j0Lrn4OnI5IpoBX6UmUQhekWeJUO 26IWbqatM5WXSlxGTX4808NKGaT0d4WNbqvm4VGFTlzya7fJuPNq6ucXBqv7MQxF/MWX a+qjA0Si/cVHk5vFKoT9f4akxWmsJPPX6R3zSceeHhmvErMN2qMpxeqAJNNiQ73otajP sVsQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=A3HJH55D; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id x5si1719499pgo.709.2018.02.01.08.51.43; Thu, 01 Feb 2018 08:51:43 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=A3HJH55D; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752847AbeBAQvl (ORCPT + 28 others); Thu, 1 Feb 2018 11:51:41 -0500 Received: from mail-it0-f66.google.com ([209.85.214.66]:39810 "EHLO mail-it0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752808AbeBAQvZ (ORCPT ); Thu, 1 Feb 2018 11:51:25 -0500 Received: by mail-it0-f66.google.com with SMTP id 68so4558015ite.4 for ; Thu, 01 Feb 2018 08:51:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=cw439ri7YDuH2p/A5GKIcVjp/mn2EguSohu8jazS960=; b=A3HJH55DrPpe7Fo7pb800pDW1wZFahzH0/pTqj91Df2ufY5FR2ENxgQmbGEJGEVBxX VAVfeUXgJNQ1+ufWdgQKOcKUOab7eqcPFEpe0M6tu6XfsH8mGtrGSELCagFjouTnHb9/ f9KIc8ycLuG1M0ChobTFfdjdlns6eiy/f06io= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=cw439ri7YDuH2p/A5GKIcVjp/mn2EguSohu8jazS960=; b=LuFbK0EEv2XawfGpODzik2syYBJ0rQ4Cu1pb9ryckDKaZ2yIl6AJ+swi87C7PAbLMp gLwtAn7BG5hVbM1FclXzc0jFk1/caUo47vgtA9cV3o4Uv3UknH3hajXKzFYwUZL5RCUs NuL6xixYMtdt8GCpND3oyUS4GITYxVWadz5jq7CgpmExjweL8TzOi1ifwfsy+CL1XUWO ilURY7QFjb9r4m3HxSr4U46Ca0J5HPQ9xAgEETzzFwCCnCI9VQnJfIM9rptxEVKjU6et Lsmp9YmSAD+XzXIkqjWwDCUyU34Y5RNJl9FyI9EbbkrkMhjttK1G9RE+abLp/idycEwV YcMA== X-Gm-Message-State: AKwxytd25uSEuXmCdJOM8PjX5lXKesjEVRVbCjkhftL9TC0WkwuQFXkW jQ1Lcos5pukybpbqY4jFItnujg== X-Received: by 10.36.63.73 with SMTP id d70mr8859732ita.96.1517503884855; Thu, 01 Feb 2018 08:51:24 -0800 (PST) Received: from xps15.cg.shawcable.net (S0106002369de4dac.cg.shawcable.net. [68.147.8.254]) by smtp.gmail.com with ESMTPSA id e83sm9270773iof.71.2018.02.01.08.51.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 01 Feb 2018 08:51:23 -0800 (PST) From: Mathieu Poirier To: peterz@infradead.org Cc: lizefan@huawei.com, mingo@redhat.com, rostedt@goodmis.org, claudio@evidence.eu.com, bristot@redhat.com, tommaso.cucinotta@santannapisa.it, juri.lelli@redhat.com, luca.abeni@santannapisa.it, linux-kernel@vger.kernel.org Subject: [PATCH V2 7/7] sched/deadline: Prevent CPU hotplug operation if DL task on CPU Date: Thu, 1 Feb 2018 09:51:09 -0700 Message-Id: <1517503869-3179-8-git-send-email-mathieu.poirier@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1517503869-3179-1-git-send-email-mathieu.poirier@linaro.org> References: <1517503869-3179-1-git-send-email-mathieu.poirier@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org When a DL task is assigned a CPU the "utilisation" (this_bw) and the "active utilisation" (running_bw) fields of rq->dl are incremented accordingly. If the CPU is hotplugged out the DL task is transferred to another CPU but the task's contribution to this_bw and running_bw isn't substracted from the outgoing CPU's rq nor added to the newly appointed CPU. In this example (where a kernel has been instrumented to output the relevant information) we have a 4 CPU system with one 6:10 DL task that has been assigned to CPU 2: root@dragon:/home/linaro/# cat /proc/rq_debug dl_rq[0]: .online : yes .dl_nr_running : 0 .running_bw : 0 .this_bw : 0 .rd->span : 0-3 .dl_nr_migratory : 0 .rd->dl_bw->bw : 3984588 .rd->dl_bw->total_bw : 629145 dl_rq[1]: .online : yes .dl_nr_running : 0 .running_bw : 0 .this_bw : 0 .rd->span: : 0-3 .dl_nr_migratory : 0 .rd->dl_bw->bw : 3984588 <-- RD capacity for 4 CPUs .rd->dl_bw->total_bw : 629145 dl_rq[2]: .online : yes .dl_nr_running : 1 <-- One task running .running_bw : 629145 <-- Normal behavior .this_bw : 629145 <-- Normal behavior .rd->span : 0-3 .dl_nr_migratory : 1 .rd->dl_bw->bw : 3984588 .rd->dl_bw->total_bw : 629145 dl_rq[3]: .online : yes .dl_nr_running : 0 .running_bw : 0 .this_bw : 0 .rd->span : 0-3 .dl_nr_migratory : 0 .rd->dl_bw->bw : 3984588 .rd->dl_bw->total_bw : 629145 At this point we hotplug out CPU2 and list the status again: root@dragon:/home/linaro/# echo 0 > /sys/devices/system/cpu/cpu2/online root@dragon:/home/linaro/# cat /proc/rq_debug dl_rq[0]: .online : yes .dl_nr_running : 1 <-- DL task was moved here .running_bw : 0 <-- Contribution not added .this_bw : 0 <-- Contribution not added .rd->span : 0-1,3 .dl_nr_migratory : 1 .rd->dl_bw->bw : 2988441 <-- RD capacity updated .rd->dl_bw->total_bw : 629145 dl_rq[1]: .online : yes .dl_nr_running : 0 .running_bw : 0 .this_bw : 0 .rd->span : 0-1,3 .dl_nr_migratory : 0 .rd->dl_bw->bw : 2988441 .rd->dl_bw->total_bw : 629145 dl_rq[2]: .online : no <-- runqueue no longer online .dl_nr_running : 0 <-- DL task was moved .running_bw : 629145 <-- Contribution not substracted .this_bw : 629145 <-- Contribution not substracted .rd->span : 2 .dl_nr_migratory : 0 .rd->dl_bw->bw : 996147 .rd->dl_bw->total_bw : 0 dl_rq[3]: .online : yes .dl_nr_running : 0 .running_bw : 0 .this_bw : 0 .rd->span : 0-1,3 .dl_nr_migratory : 0 .rd->dl_bw->bw : 2988441 .rd->dl_bw->total_bw: 629145 Upon rebooting the system a splat is also produced: [ 578.184789] ------------[ cut here ]------------ [ 578.184813] dl_rq->running_bw > old [ 578.184838] WARNING: CPU: 0 PID: 4076 at /home/mpoirier/work/linaro/deadline/kernel/kernel/sched/deadline.c:98 dequeue_task_dl+0x128/0x168 [ 578.191693] Modules linked in: [ 578.191705] CPU: 0 PID: 4076 Comm: burn Not tainted 4.15.0-00009-gf597fc1e5764-dirty #259 [ 578.191708] Hardware name: Qualcomm Technologies, Inc. APQ 8016 SBC (DT) [ 578.191713] pstate: 60000085 (nZCv daIf -PAN -UAO) [ 578.191718] pc : dequeue_task_dl+0x128/0x168 [ 578.191722] lr : dequeue_task_dl+0x128/0x168 [ 578.191724] sp : ffff8000383ebbf0 [ 578.191727] x29: ffff8000383ebbf0 x28: ffff800038288000 [ 578.191733] x27: 0000000000000009 x26: ffff800038890000 [ 578.191739] x25: ffff800038994e60 x24: ffff800038994e00 [ 578.191744] x23: 0000000000000000 x22: 0000000000000000 [ 578.191749] x21: 000000000000000e x20: ffff800038288000 [ 578.191755] x19: ffff80003d950aa8 x18: 0000000000000010 [ 578.191761] x17: 0000000000000001 x16: 0000000000002710 [ 578.191766] x15: 0000000000000006 x14: ffff0000892ed37f [ 578.191772] x13: ffff0000092ed38d x12: 0000000000000000 [ 578.191778] x11: ffff8000383eb840 x10: 0000000005f5e0ff [ 578.191784] x9 : 0000000000000034 x8 : 625f676e696e6e75 [ 578.191794] x7 : 723e2d71725f6c64 x6 : 000000000000016c [ 578.191800] x5 : 0000000000000000 x4 : 0000000000000000 [ 578.191806] x3 : ffffffffffffffff x2 : 000080003480f000 [ 578.191812] x1 : ffff800038288000 x0 : 0000000000000017 [ 578.191818] Call trace: [ 578.191824] dequeue_task_dl+0x128/0x168 [ 578.191830] sched_move_task+0xa8/0x150 [ 578.191837] sched_autogroup_exit_task+0x20/0x30 [ 578.191843] do_exit+0x2c4/0x9f8 [ 578.191847] do_group_exit+0x3c/0xa0 [ 578.191853] get_signal+0x2a4/0x568 [ 578.191860] do_signal+0x70/0x210 [ 578.191866] do_notify_resume+0xe0/0x138 [ 578.191870] work_pending+0x8/0x10 [ 578.191874] ---[ end trace 345388d10dc698fe ]--- As a stop-gap measure before the real solution is available this patch prevents users from carrying out a CPU hotplug operation if a DL task is running (or suspended) on said CPU. Signed-off-by: Mathieu Poirier --- kernel/sched/deadline.c | 5 +++++ 1 file changed, 5 insertions(+) -- 2.7.4 diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index 1a5c42e7b076..9fa9421bb23f 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -2672,11 +2672,16 @@ bool dl_cpu_busy(unsigned int cpu) int cpus; rcu_read_lock_sched(); + overflow = !!(cpu_rq(cpu)->dl.this_bw); + if (overflow) + goto out; + dl_b = dl_bw_of(cpu); raw_spin_lock_irqsave(&dl_b->lock, flags); cpus = dl_bw_cpus(cpu); overflow = __dl_overflow(dl_b, cpus, 0, 0); raw_spin_unlock_irqrestore(&dl_b->lock, flags); +out: rcu_read_unlock_sched(); return overflow; }