From patchwork Tue Nov 4 11:13:03 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "pang.xunlei" X-Patchwork-Id: 40129 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-lb0-f197.google.com (mail-lb0-f197.google.com [209.85.217.197]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id AB67C240A6 for ; Tue, 4 Nov 2014 11:15:58 +0000 (UTC) Received: by mail-lb0-f197.google.com with SMTP id w7sf2033320lbi.8 for ; Tue, 04 Nov 2014 03:15:57 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:delivered-to:from:to:cc:subject :date:message-id:in-reply-to:references:sender:precedence:list-id :x-original-sender:x-original-authentication-results:mailing-list :list-post:list-help:list-archive:list-unsubscribe; bh=WRUAJb566EzC7wYdTXbuNpcOt00lTsI+vDv3bRB6OLc=; b=BsHoy+1kFM06YHh3g/Kn0fALCFRNYPeQSHKzd5VuDLEBVwK6WQOrRjGmR1T5JwJ77j VEcRGo1n1Z4W0bZvqkwyen1shROKQuKhAhGEJxdbnOeRwmYwBdiQSB/W1ZuU8MVfpc72 KmPQ1pziLyoNwFKws9mtcc3obsXSpDYuGkmkndtc1zvqiymyeojk/PSG6g6zg85IWxaJ rHkfS608ILb2T2wTVHwjFzuz0xXJ52kf2YVWV6/IE3Jmiuvu6VqJ7vSCNHwh9nH9NesJ xAr11XJHqUTNhXocb2QXfFnDNjpswWC7GZsflCVudwO5Z9rJhQQCADhZdRgF7yTsKZnx SdYQ== X-Gm-Message-State: ALoCoQm/NoaQWa25jaqsBkRN14RtOfDBFtPDIewOOM+AIBGBjzMESZ3q9xaiJ1gbMIgoSXpF99YS X-Received: by 10.180.90.115 with SMTP id bv19mr4051357wib.1.1415099757650; Tue, 04 Nov 2014 03:15:57 -0800 (PST) MIME-Version: 1.0 X-BeenThere: patchwork-forward@linaro.org Received: by 10.153.11.163 with SMTP id ej3ls808115lad.95.gmail; Tue, 04 Nov 2014 03:15:57 -0800 (PST) X-Received: by 10.152.116.47 with SMTP id jt15mr19459151lab.13.1415099757462; Tue, 04 Nov 2014 03:15:57 -0800 (PST) Received: from mail-lb0-f170.google.com (mail-lb0-f170.google.com. [209.85.217.170]) by mx.google.com with ESMTPS id zm10si30168lbb.125.2014.11.04.03.15.57 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 04 Nov 2014 03:15:57 -0800 (PST) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.170 as permitted sender) client-ip=209.85.217.170; Received: by mail-lb0-f170.google.com with SMTP id z12so2533530lbi.1 for ; Tue, 04 Nov 2014 03:15:57 -0800 (PST) X-Received: by 10.112.12.35 with SMTP id v3mr58328476lbb.80.1415099757305; Tue, 04 Nov 2014 03:15:57 -0800 (PST) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.112.184.201 with SMTP id ew9csp98520lbc; Tue, 4 Nov 2014 03:15:56 -0800 (PST) X-Received: by 10.67.30.34 with SMTP id kb2mr48477547pad.97.1415099753022; Tue, 04 Nov 2014 03:15:53 -0800 (PST) Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id dt17si17699478pdb.181.2014.11.04.03.15.52 for ; Tue, 04 Nov 2014 03:15:53 -0800 (PST) Received-SPF: none (google.com: linux-kernel-owner@vger.kernel.org does not designate permitted sender hosts) client-ip=209.132.180.67; Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753166AbaKDLPm (ORCPT + 25 others); Tue, 4 Nov 2014 06:15:42 -0500 Received: from mail-pa0-f52.google.com ([209.85.220.52]:39278 "EHLO mail-pa0-f52.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752676AbaKDLPi (ORCPT ); Tue, 4 Nov 2014 06:15:38 -0500 Received: by mail-pa0-f52.google.com with SMTP id fa1so14266750pad.39 for ; Tue, 04 Nov 2014 03:15:37 -0800 (PST) X-Received: by 10.66.102.2 with SMTP id fk2mr7585922pab.143.1415099737453; Tue, 04 Nov 2014 03:15:37 -0800 (PST) Received: from wangdeqiang.com ([210.21.223.3]) by mx.google.com with ESMTPSA id ol3sm158568pbb.17.2014.11.04.03.15.34 for (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 04 Nov 2014 03:15:36 -0800 (PST) From: "pang.xunlei" To: linux-kernel@vger.kernel.org Cc: Ingo Molnar , Peter Zijlstra , Steven Rostedt , Juri Lelli , "pang.xunlei" Subject: [PATCH v2 4/6] sched/dl: Modify cpudl_find() for more cases of electing best_cpu Date: Tue, 4 Nov 2014 19:13:03 +0800 Message-Id: <1415099585-31174-4-git-send-email-pang.xunlei@linaro.org> X-Mailer: git-send-email 1.7.9.5 In-Reply-To: <1415099585-31174-1-git-send-email-pang.xunlei@linaro.org> References: <1415099585-31174-1-git-send-email-pang.xunlei@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: list List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: pang.xunlei@linaro.org X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.217.170 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , When a runqueue runs out of DL tasks, it may have RT tasks or non-RT tasks or just idle. It'd be better to put the DL task to an idle cpu or non-RT cpu if there is any. Adds idle_enter_dl()/idle_exit_dl() to detect idle cases. Adds rt_enter_dl()/rt_exit_dl() to detect non-RT cases. Use the same thought as tackling RT in the former patch. Signed-off-by: pang.xunlei --- kernel/sched/cpudeadline.c | 79 +++++++++++++++++++++++++++++++++++--------- kernel/sched/cpudeadline.h | 13 ++++++-- kernel/sched/deadline.c | 32 +++++++++++++++--- kernel/sched/idle_task.c | 2 ++ kernel/sched/rt.c | 7 ++++ kernel/sched/sched.h | 11 ++++++ 6 files changed, 121 insertions(+), 23 deletions(-) diff --git a/kernel/sched/cpudeadline.c b/kernel/sched/cpudeadline.c index 539ca3c..d5ebc34 100644 --- a/kernel/sched/cpudeadline.c +++ b/kernel/sched/cpudeadline.c @@ -106,10 +106,24 @@ int cpudl_find(struct cpudl *cp, struct task_struct *p, { int best_cpu = -1; const struct sched_dl_entity *dl_se = &p->dl; + struct cpumask tmp_mask; - if (later_mask && cpumask_and(later_mask, later_mask, cp->free_cpus)) { - best_cpu = cpumask_any(later_mask); - goto out; + if (later_mask) { + cpumask_and(&tmp_mask, &p->cpus_allowed, cpu_active_mask); + if (cpumask_and(later_mask, &tmp_mask, cp->idle_cpus)) { + best_cpu = cpumask_any(later_mask); + goto out; + } + + if (cpumask_and(later_mask, &tmp_mask, cp->freert_cpus)) { + best_cpu = cpumask_any(later_mask); + goto out; + } + + if (cpumask_and(later_mask, &tmp_mask, cp->freedl_cpus)) { + best_cpu = cpumask_any(later_mask); + goto out; + } } else if (cpumask_test_cpu(cpudl_maximum(cp), &p->cpus_allowed) && dl_time_before(dl_se->deadline, cp->elements[0].dl)) { best_cpu = cpudl_maximum(cp); @@ -128,12 +142,12 @@ out: * @cp: the cpudl max-heap context * @cpu: the target cpu * @dl: the new earliest deadline for this cpu - * + * @set_flags: CPUDL_SET_XXX, CPUDL_CLEAR_XXX * Notes: assumes cpu_rq(cpu)->lock is locked * * Returns: (void) */ -void cpudl_set(struct cpudl *cp, int cpu, u64 dl, int is_valid) +void cpudl_set(struct cpudl *cp, int cpu, u64 dl, int set_flags) { int old_idx, new_cpu; unsigned long flags; @@ -141,8 +155,25 @@ void cpudl_set(struct cpudl *cp, int cpu, u64 dl, int is_valid) WARN_ON(!cpu_present(cpu)); raw_spin_lock_irqsave(&cp->lock, flags); + switch (set_flags) { + case CPUDL_SET_IDLE: + cpumask_set_cpu(cpu, cp->idle_cpus); + goto out; + case CPUDL_CLEAR_IDLE: + cpumask_clear_cpu(cpu, cp->idle_cpus); + goto out; + case CPUDL_SET_FREERT: + cpumask_set_cpu(cpu, cp->freert_cpus); + goto out; + case CPUDL_CLEAR_FREERT: + cpumask_set_cpu(cpu, cp->freert_cpus); + goto out; + default: + break; + } + old_idx = cp->elements[cpu].idx; - if (!is_valid) { + if (set_flags == CPUDL_SET_FREEDL) { /* remove item */ if (old_idx == IDX_INVALID) { /* @@ -164,8 +195,8 @@ void cpudl_set(struct cpudl *cp, int cpu, u64 dl, int is_valid) cpudl_exchange(cp, old_idx, parent(old_idx)); old_idx = parent(old_idx); } - cpumask_set_cpu(cpu, cp->free_cpus); - cpudl_heapify(cp, old_idx); + cpumask_set_cpu(cpu, cp->freedl_cpus); + cpudl_heapify(cp, old_idx); goto out; } @@ -176,7 +207,7 @@ void cpudl_set(struct cpudl *cp, int cpu, u64 dl, int is_valid) cp->elements[cp->size - 1].cpu = cpu; cp->elements[cpu].idx = cp->size - 1; cpudl_change_key(cp, cp->size - 1, dl); - cpumask_clear_cpu(cpu, cp->free_cpus); + cpumask_clear_cpu(cpu, cp->freedl_cpus); } else { cpudl_change_key(cp, old_idx, dl); } @@ -201,19 +232,33 @@ int cpudl_init(struct cpudl *cp) sizeof(struct cpudl_item), GFP_KERNEL); if (!cp->elements) - return -ENOMEM; + goto out; + + if (!alloc_cpumask_var(&cp->freedl_cpus, GFP_KERNEL)) + goto free_elements; + + if (!zalloc_cpumask_var(&cp->freert_cpus, GFP_KERNEL)) + goto free_freedl_cpus; + + if (!zalloc_cpumask_var(&cp->idle_cpus, GFP_KERNEL)) + goto free_freert_cpus; - if (!alloc_cpumask_var(&cp->free_cpus, GFP_KERNEL)) { - kfree(cp->elements); - return -ENOMEM; - } for_each_possible_cpu(i) cp->elements[i].idx = IDX_INVALID; - cpumask_setall(cp->free_cpus); + cpumask_setall(cp->freedl_cpus); return 0; + +free_freert_cpus: + kfree(cp->freert_cpus); +free_freedl_cpus: + kfree(cp->freedl_cpus); +free_elements: + kfree(cp->elements); +out: + return -ENOMEM; } /* @@ -222,6 +267,8 @@ int cpudl_init(struct cpudl *cp) */ void cpudl_cleanup(struct cpudl *cp) { - free_cpumask_var(cp->free_cpus); + free_cpumask_var(cp->freedl_cpus); + free_cpumask_var(cp->freert_cpus); + free_cpumask_var(cp->idle_cpus); kfree(cp->elements); } diff --git a/kernel/sched/cpudeadline.h b/kernel/sched/cpudeadline.h index 538c979..d79e4d8 100644 --- a/kernel/sched/cpudeadline.h +++ b/kernel/sched/cpudeadline.h @@ -5,6 +5,13 @@ #define IDX_INVALID -1 +#define CPUDL_SET_DL 1 /* set deadline value, clear freedl_cpus */ +#define CPUDL_SET_FREEDL 2 /* set freedl_cpus */ +#define CPUDL_SET_FREERT 3 /* set freert_cpus */ +#define CPUDL_CLEAR_FREERT 4 /* clear freert_cpus */ +#define CPUDL_SET_IDLE 5 /* set idle_cpus */ +#define CPUDL_CLEAR_IDLE 6 /* clear idle_cpus */ + struct cpudl_item { u64 dl; int cpu; @@ -14,7 +21,9 @@ struct cpudl_item { struct cpudl { raw_spinlock_t lock; int size; - cpumask_var_t free_cpus; + cpumask_var_t idle_cpus; + cpumask_var_t freert_cpus; + cpumask_var_t freedl_cpus; struct cpudl_item *elements; }; @@ -22,7 +31,7 @@ struct cpudl { #ifdef CONFIG_SMP int cpudl_find(struct cpudl *cp, struct task_struct *p, struct cpumask *later_mask); -void cpudl_set(struct cpudl *cp, int cpu, u64 dl, int is_valid); +void cpudl_set(struct cpudl *cp, int cpu, u64 dl, int set_flags); int cpudl_init(struct cpudl *cp); void cpudl_cleanup(struct cpudl *cp); #else diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index 5285332..7b0b2d2 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -673,6 +673,26 @@ static void update_curr_dl(struct rq *rq) #ifdef CONFIG_SMP +void idle_enter_dl(struct rq *this_rq) +{ + cpudl_set(&this_rq->rd->cpudl, this_rq->cpu, 0, CPUDL_SET_IDLE); +} + +void idle_exit_dl(struct rq *this_rq) +{ + cpudl_set(&this_rq->rd->cpudl, this_rq->cpu, 0, CPUDL_CLEAR_IDLE); +} + +void rt_enter_dl(struct rq *this_rq) +{ + cpudl_set(&this_rq->rd->cpudl, this_rq->cpu, 0, CPUDL_CLEAR_FREERT); +} + +void rt_exit_dl(struct rq *this_rq) +{ + cpudl_set(&this_rq->rd->cpudl, this_rq->cpu, 0, CPUDL_SET_FREERT); +} + static struct task_struct *pick_next_earliest_dl_task(struct rq *rq, int cpu); static inline u64 next_deadline(struct rq *rq) @@ -699,7 +719,7 @@ static void inc_dl_deadline(struct dl_rq *dl_rq, u64 deadline) */ dl_rq->earliest_dl.next = dl_rq->earliest_dl.curr; dl_rq->earliest_dl.curr = deadline; - cpudl_set(&rq->rd->cpudl, rq->cpu, deadline, 1); + cpudl_set(&rq->rd->cpudl, rq->cpu, deadline, CPUDL_SET_DL); } else if (dl_rq->earliest_dl.next == 0 || dl_time_before(deadline, dl_rq->earliest_dl.next)) { /* @@ -723,7 +743,7 @@ static void dec_dl_deadline(struct dl_rq *dl_rq, u64 deadline) if (!dl_rq->dl_nr_running) { dl_rq->earliest_dl.curr = 0; dl_rq->earliest_dl.next = 0; - cpudl_set(&rq->rd->cpudl, rq->cpu, 0, 0); + cpudl_set(&rq->rd->cpudl, rq->cpu, 0, CPUDL_SET_FREEDL); } else { struct rb_node *leftmost = dl_rq->rb_leftmost; struct sched_dl_entity *entry; @@ -731,7 +751,8 @@ static void dec_dl_deadline(struct dl_rq *dl_rq, u64 deadline) entry = rb_entry(leftmost, struct sched_dl_entity, rb_node); dl_rq->earliest_dl.curr = entry->deadline; dl_rq->earliest_dl.next = next_deadline(rq); - cpudl_set(&rq->rd->cpudl, rq->cpu, entry->deadline, 1); + cpudl_set(&rq->rd->cpudl, rq->cpu, + entry->deadline, CPUDL_SET_DL); } } @@ -1563,7 +1584,8 @@ static void rq_online_dl(struct rq *rq) dl_set_overload(rq); if (rq->dl.dl_nr_running > 0) - cpudl_set(&rq->rd->cpudl, rq->cpu, rq->dl.earliest_dl.curr, 1); + cpudl_set(&rq->rd->cpudl, rq->cpu, + rq->dl.earliest_dl.curr, CPUDL_SET_DL); } /* Assumes rq->lock is held */ @@ -1572,7 +1594,7 @@ static void rq_offline_dl(struct rq *rq) if (rq->dl.overloaded) dl_clear_overload(rq); - cpudl_set(&rq->rd->cpudl, rq->cpu, 0, 0); + cpudl_set(&rq->rd->cpudl, rq->cpu, 0, CPUDL_SET_FREEDL); } void init_sched_dl_class(void) diff --git a/kernel/sched/idle_task.c b/kernel/sched/idle_task.c index e053347..7838e56 100644 --- a/kernel/sched/idle_task.c +++ b/kernel/sched/idle_task.c @@ -26,6 +26,7 @@ static void check_preempt_curr_idle(struct rq *rq, struct task_struct *p, int fl static struct task_struct * pick_next_task_idle(struct rq *rq, struct task_struct *prev) { + idle_enter_dl(rq); idle_enter_rt(rq); put_prev_task(rq, prev); @@ -49,6 +50,7 @@ dequeue_task_idle(struct rq *rq, struct task_struct *p, int flags) static void put_prev_task_idle(struct rq *rq, struct task_struct *prev) { + idle_exit_dl(rq); idle_exit_rt(rq); idle_exit_fair(rq); rq_last_tick_reset(rq); diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c index dc1f7f0..a5bcded 100644 --- a/kernel/sched/rt.c +++ b/kernel/sched/rt.c @@ -1488,6 +1488,9 @@ pick_next_task_rt(struct rq *rq, struct task_struct *prev) if (!rt_rq->rt_queued) return NULL; + if (prev->sched_class != &rt_sched_class) + rt_enter_dl(rq); + put_prev_task(rq, prev); p = _pick_next_task_rt(rq); @@ -1502,6 +1505,10 @@ pick_next_task_rt(struct rq *rq, struct task_struct *prev) static void put_prev_task_rt(struct rq *rq, struct task_struct *p) { + /* Neglect stop preempt. As for dl preempt, doesn't matter */ + if (rq->curr->sched_class != &rt_sched_class) + rt_exit_dl(rq); + update_curr_rt(rq); /* diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index cc603fa..b76dfef 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1162,6 +1162,12 @@ extern void update_group_capacity(struct sched_domain *sd, int cpu); extern void trigger_load_balance(struct rq *rq); +extern void rt_enter_dl(struct rq *this_rq); +extern void rt_exit_dl(struct rq *this_rq); + +extern void idle_enter_dl(struct rq *this_rq); +extern void idle_exit_dl(struct rq *this_rq); + extern void idle_enter_rt(struct rq *this_rq); extern void idle_exit_rt(struct rq *this_rq); @@ -1169,6 +1175,11 @@ extern void idle_enter_fair(struct rq *this_rq); extern void idle_exit_fair(struct rq *this_rq); #else +static inline void rt_enter_dl(struct rq *rq) { } +static inline void rt_exit_dl(struct rq *rq) { } + +static inline void idle_enter_dl(struct rq *rq) { } +static inline void idle_exit_dl(struct rq *rq) { } static inline void idle_enter_rt(struct rq *rq) { } static inline void idle_exit_rt(struct rq *rq) { }