From patchwork Thu Mar 10 23:46:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rob Clark X-Patchwork-Id: 550183 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D0D6C433FE for ; Thu, 10 Mar 2022 23:45:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344799AbiCJXqi (ORCPT ); Thu, 10 Mar 2022 18:46:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44590 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344551AbiCJXqh (ORCPT ); Thu, 10 Mar 2022 18:46:37 -0500 Received: from mail-pj1-x1034.google.com (mail-pj1-x1034.google.com [IPv6:2607:f8b0:4864:20::1034]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 06E6119E00A; Thu, 10 Mar 2022 15:45:36 -0800 (PST) Received: by mail-pj1-x1034.google.com with SMTP id z12-20020a17090ad78c00b001bf022b69d6so6660434pju.2; Thu, 10 Mar 2022 15:45:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=OXegA51qD+CyYO9M7fSmY5tcIBfLmUdoHlbWl+RWLYA=; b=Uya5BbF/Dea3p/slOxGOh0T5z35dl1DZwLwD64KB0b7uIeODxa63MztEUsLQQTBK5d D+RjR/B/DCsuUXe5VTqlL/sVZ3iPPAX+wmHNi4zmP0hvbQTRdsGTuBd7ORxEdJMxQFeg f1xCP90/4jcWi0d1L6liTOs8x1pwQLwnFecjcadNtFhxQA3ZKwLSWrUu/BOxU9K6a3IA iIADELN9Zu4tKQm0yfuEuHJKcQTydAKZJJMSbstWPtbiAgwGHFuqDWEF4jWAYDsCX00/ F58vxhlnFriX7g9PPn/gcLj609Pg+h+n3ssPGc2FC26qCmURAEBWNMbfrkTooHiy+UVs /neA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OXegA51qD+CyYO9M7fSmY5tcIBfLmUdoHlbWl+RWLYA=; b=hZmhq0fztMAJugpCtlZupoZ78a6pJxz7fehiHl4rlw5rr5fonC3p3GkqqHIXDlhDON 4pRrnBNQzzVitoDDgnQ4HAyOX+B+tlp7KKKM9QXpTP41mmf/Ps+P9MoYS0bpReSE7B21 RSP/Tgrvm3d61JZd79Df3hbUoIuGhTn6+uTWBeYJi8NUWKl8VPxUWn28eKeVPz85KJfW KODOqfxqipNq/+chRIfsN0J1VALzeijRSeN0uyfKfOi5wl3wEuokICnfa9xEy76znqd7 gjcy+YQY30FUTj6Xg+b5i089bSa2lga7nVw7Lhg4ixtM3AlBaz4AItyWQENHSNTYf30o FFbQ== X-Gm-Message-State: AOAM533ofemZxKV7376MlXF879vIDdIosM3YHYTo/C2Qv7pRzxR592NT shMYPWfp152tsr9Y+ElkSxM= X-Google-Smtp-Source: ABdhPJyMokPnWrLeEdPTbchYVoLn7v3pddlxvswdSoMvVNd5x9sSPY6XLBlteAA4HXr/Z8aINByJyA== X-Received: by 2002:a17:90b:4b47:b0:1be:fccf:d1a8 with SMTP id mi7-20020a17090b4b4700b001befccfd1a8mr7586614pjb.33.1646955935191; Thu, 10 Mar 2022 15:45:35 -0800 (PST) Received: from localhost ([2601:1c0:5200:a6:307:a401:7b76:c6e5]) by smtp.gmail.com with ESMTPSA id p125-20020a622983000000b004f6c5d58225sm8252049pfp.90.2022.03.10.15.45.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Mar 2022 15:45:34 -0800 (PST) From: Rob Clark To: dri-devel@lists.freedesktop.org Cc: freedreno@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, Rob Clark , Rob Clark , Sean Paul , Abhinav Kumar , David Airlie , Daniel Vetter , Akhil P Oommen , AngeloGioacchino Del Regno , Bjorn Andersson , Jonathan Marek , Vladimir Lypak , linux-kernel@vger.kernel.org (open list) Subject: [PATCH 1/3] drm/msm/gpu: Rename runtime suspend/resume functions Date: Thu, 10 Mar 2022 15:46:04 -0800 Message-Id: <20220310234611.424743-2-robdclark@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220310234611.424743-1-robdclark@gmail.com> References: <20220310234611.424743-1-robdclark@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org From: Rob Clark Signed-off-by: Rob Clark --- drivers/gpu/drm/msm/adreno/adreno_device.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/gpu/drm/msm/adreno/adreno_device.c b/drivers/gpu/drm/msm/adreno/adreno_device.c index 89cfd84760d7..8859834b51b8 100644 --- a/drivers/gpu/drm/msm/adreno/adreno_device.c +++ b/drivers/gpu/drm/msm/adreno/adreno_device.c @@ -600,7 +600,7 @@ static const struct of_device_id dt_match[] = { }; #ifdef CONFIG_PM -static int adreno_resume(struct device *dev) +static int adreno_runtime_resume(struct device *dev) { struct msm_gpu *gpu = dev_to_gpu(dev); @@ -616,7 +616,7 @@ static int active_submits(struct msm_gpu *gpu) return active_submits; } -static int adreno_suspend(struct device *dev) +static int adreno_runtime_suspend(struct device *dev) { struct msm_gpu *gpu = dev_to_gpu(dev); int remaining; @@ -635,7 +635,7 @@ static int adreno_suspend(struct device *dev) static const struct dev_pm_ops adreno_pm_ops = { SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend, pm_runtime_force_resume) - SET_RUNTIME_PM_OPS(adreno_suspend, adreno_resume, NULL) + SET_RUNTIME_PM_OPS(adreno_runtime_suspend, adreno_runtime_resume, NULL) }; static struct platform_driver adreno_driver = { From patchwork Thu Mar 10 23:46:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rob Clark X-Patchwork-Id: 550796 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB048C433EF for ; Thu, 10 Mar 2022 23:45:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344845AbiCJXqp (ORCPT ); Thu, 10 Mar 2022 18:46:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44722 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344840AbiCJXqm (ORCPT ); Thu, 10 Mar 2022 18:46:42 -0500 Received: from mail-pf1-x431.google.com (mail-pf1-x431.google.com [IPv6:2607:f8b0:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5A9A319E022; Thu, 10 Mar 2022 15:45:40 -0800 (PST) Received: by mail-pf1-x431.google.com with SMTP id a5so6476659pfv.2; Thu, 10 Mar 2022 15:45:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=84Raz5JYKVeX/pUi+12Qa70UMUs/NibyW3b6HrH/7lY=; b=RIWCIMYLv6fAl3X5yNuGJ/ZhuuVr/0l9TPnu448skU2Fk/3Dd5CxT4a38n+h8IJrI0 mUL+urptgd7ItaeuQrx16YN3HcVwrGQ5V/P3uZvlhW/2MX2PNw2KgZ9dqjC6ewvNi9Gy ssv1VBfyC56RqCVv5L90TaEfFpRctyF+Vwm+Y74O4CCV02XokyiQbjoJF7mAwUUj/yfN d/SSBZ5edruXH+IARzrbune3t+cTVvPOSofeG4QygAfOAmyOTZSkrIDOokc8zIVHzR4M KdKwn5BULrZW408bbHXjNVZOhh0xJoP4mf5iaZEBbEYnSX6EatPviXdw0oKJMGRyVAC5 BgRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=84Raz5JYKVeX/pUi+12Qa70UMUs/NibyW3b6HrH/7lY=; b=21TRZO6tAG6D3XNUqdd+g5yzjxcL/Rr2z7oPuOG0B3Gy6btYzYnCHtIfKbPBjeSaJ/ mqVqQVaXnvnvDpbJHAd8ojgLPb8Eqm4YmAQtqUZ0JYANjIxIQUfgDs6j4VhnBWmyTD2Q HfYSCDEgXr8MyxjX7UtLWnl8ZhsMO/TWmSxxWDtb75hPN+jVXk8enDSfSB/AIFLxujpL d0xfRx3b5jG5ZVkvb6g7XK5Aea681CeJkTMqOQyPhZR5VOcnfRNttvYewxT9O5A/DobR FOtDcY/HoCo9qoZDuQY4bxmgVRjD4vdlGyxmDxiOYxxpXAIAWnKVYiSFvrKx4Hxk1Tf/ Nqmg== X-Gm-Message-State: AOAM532mGQd47eEC8vZpeLWMEWzWE8Ku1UB1QwKSm6UFUiKj1Mc03GcP ZHQOYNa0eLg+SifRG6/L36o= X-Google-Smtp-Source: ABdhPJxmw99IW2XBwcSNPKsasjmkkdChoRp9s2QpSC2oczo3xnYQMDiMHz4ROeXE5pNO8L79IYoaTQ== X-Received: by 2002:a63:28d:0:b0:365:8e16:5c19 with SMTP id 135-20020a63028d000000b003658e165c19mr6158756pgc.579.1646955939174; Thu, 10 Mar 2022 15:45:39 -0800 (PST) Received: from localhost ([2601:1c0:5200:a6:307:a401:7b76:c6e5]) by smtp.gmail.com with ESMTPSA id y7-20020aa78047000000b004f7299bf685sm7917827pfm.91.2022.03.10.15.45.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Mar 2022 15:45:38 -0800 (PST) From: Rob Clark To: dri-devel@lists.freedesktop.org Cc: freedreno@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, Rob Clark , Rob Clark , Sean Paul , Abhinav Kumar , David Airlie , Daniel Vetter , Akhil P Oommen , Jonathan Marek , AngeloGioacchino Del Regno , Bjorn Andersson , Vladimir Lypak , linux-kernel@vger.kernel.org (open list) Subject: [PATCH 2/3] drm/msm/gpu: Park scheduler threads for system suspend Date: Thu, 10 Mar 2022 15:46:05 -0800 Message-Id: <20220310234611.424743-3-robdclark@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220310234611.424743-1-robdclark@gmail.com> References: <20220310234611.424743-1-robdclark@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org From: Rob Clark In the system suspend path, we don't want to be racing with the scheduler kthreads pushing additional queued up jobs to the hw queue (ringbuffer). So park them first. While we are at it, move the wait for active jobs to complete into the new system- suspend path. Signed-off-by: Rob Clark --- drivers/gpu/drm/msm/adreno/adreno_device.c | 68 ++++++++++++++++++++-- 1 file changed, 64 insertions(+), 4 deletions(-) diff --git a/drivers/gpu/drm/msm/adreno/adreno_device.c b/drivers/gpu/drm/msm/adreno/adreno_device.c index 8859834b51b8..0440a98988fc 100644 --- a/drivers/gpu/drm/msm/adreno/adreno_device.c +++ b/drivers/gpu/drm/msm/adreno/adreno_device.c @@ -619,22 +619,82 @@ static int active_submits(struct msm_gpu *gpu) static int adreno_runtime_suspend(struct device *dev) { struct msm_gpu *gpu = dev_to_gpu(dev); - int remaining; + + /* + * We should be holding a runpm ref, which will prevent + * runtime suspend. In the system suspend path, we've + * already waited for active jobs to complete. + */ + WARN_ON_ONCE(gpu->active_submits); + + return gpu->funcs->pm_suspend(gpu); +} + +static void suspend_scheduler(struct msm_gpu *gpu) +{ + int i; + + /* + * Shut down the scheduler before we force suspend, so that + * suspend isn't racing with scheduler kthread feeding us + * more work. + * + * Note, we just want to park the thread, and let any jobs + * that are already on the hw queue complete normally, as + * opposed to the drm_sched_stop() path used for handling + * faulting/timed-out jobs. We can't really cancel any jobs + * already on the hw queue without racing with the GPU. + */ + for (i = 0; i < gpu->nr_rings; i++) { + struct drm_gpu_scheduler *sched = &gpu->rb[i]->sched; + kthread_park(sched->thread); + } +} + +static void resume_scheduler(struct msm_gpu *gpu) +{ + int i; + + for (i = 0; i < gpu->nr_rings; i++) { + struct drm_gpu_scheduler *sched = &gpu->rb[i]->sched; + kthread_unpark(sched->thread); + } +} + +static int adreno_system_suspend(struct device *dev) +{ + struct msm_gpu *gpu = dev_to_gpu(dev); + int remaining, ret; + + suspend_scheduler(gpu); remaining = wait_event_timeout(gpu->retire_event, active_submits(gpu) == 0, msecs_to_jiffies(1000)); if (remaining == 0) { dev_err(dev, "Timeout waiting for GPU to suspend\n"); - return -EBUSY; + ret = -EBUSY; + goto out; } - return gpu->funcs->pm_suspend(gpu); + ret = pm_runtime_force_suspend(dev); +out: + if (ret) + resume_scheduler(gpu); + + return ret; } + +static int adreno_system_resume(struct device *dev) +{ + resume_scheduler(dev_to_gpu(dev)); + return pm_runtime_force_resume(dev); +} + #endif static const struct dev_pm_ops adreno_pm_ops = { - SET_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend, pm_runtime_force_resume) + SET_SYSTEM_SLEEP_PM_OPS(adreno_system_suspend, adreno_system_resume) SET_RUNTIME_PM_OPS(adreno_runtime_suspend, adreno_runtime_resume, NULL) }; From patchwork Thu Mar 10 23:46:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rob Clark X-Patchwork-Id: 550182 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8130C433EF for ; Thu, 10 Mar 2022 23:45:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344240AbiCJXqr (ORCPT ); Thu, 10 Mar 2022 18:46:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44732 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344854AbiCJXqp (ORCPT ); Thu, 10 Mar 2022 18:46:45 -0500 Received: from mail-pf1-x436.google.com (mail-pf1-x436.google.com [IPv6:2607:f8b0:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 09B0819E03F; Thu, 10 Mar 2022 15:45:43 -0800 (PST) Received: by mail-pf1-x436.google.com with SMTP id s8so6431524pfk.12; Thu, 10 Mar 2022 15:45:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Sc+BLeBhOeunpwNV1naAxApoyho4RIVwZsup23KixGA=; b=i/kCIR2M9+f6iR0IxrdfD6uzdUTkXYHeSmmSe0xEfqNmiGuvFwvEa9Rhc7SjtoyDRm iSWmf2YUwJTqDGe5yr62zNAIiPpSnKRmbpMcjNLvMsY5+aM8+89n6AjSPTudJ35s0cJ/ cDBM6keO22ne0gjnhQGSRsVh3eyxL68tm3z7MsqZgKk9hEN2wBW+GYS5nQ81KHZBpB0U C4Oq5c3OhQiC3wPgyLyYCR5OQ8I894o2dkYOtOIBXyh4UkGY+Ko+6mEPpwpaiHban9ou FuN7FOqQk45kn4IvAWIWVJ7ZQs1Nstlv+EjKVvMQ2F7RPEnvRR0zDpw9qgXZPs/dM8Cw Jx7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Sc+BLeBhOeunpwNV1naAxApoyho4RIVwZsup23KixGA=; b=4cmVGO+nAFvQyL8eJHwg2DLyVBNztombRvxryklM8YPxM1cWZRPsd7/4mKI8l+KJYZ l+sSYs3kG9P3xobH++4Rj/lbJc05DbCy1I/y140xT/Ffl94BQA60kusUghtCxBHZRzmT zJ+UHj5ckB2C7P1qBfGof6pmBAOS0AVB2gfXt61S+o2/v8Yv5OENVqMA8HxFQTEXGlUQ 9RBTsudRRBOSxprCfUQqzHCyhHgmZLxhi2gwk4mfcMk6mMP2+/hJ17n3ny8/1WDphyOx QIQcJe20ZcRlTw3xsRqbkDv9nBY3ggwxzM3ET0dyoQoNShD/cVrpQrC0CGU2JvWmaLuQ 3qqw== X-Gm-Message-State: AOAM533JbAKBwaSC4UtPoNQTNrZ9LkidgnZ9K9Q2uMKDv7oS6UPxVji9 Rj6Um1nhlgZqap60s4gZTxCct/M1b5E= X-Google-Smtp-Source: ABdhPJyrhezjAT6CU0w7i5q6gN4DLMtI4haYY25Kw92sxxE/XVe1+QHeusf/BWUEj++yK+OMlHBWTw== X-Received: by 2002:a63:89:0:b0:37c:54f9:25b6 with SMTP id 131-20020a630089000000b0037c54f925b6mr6190197pga.494.1646955942569; Thu, 10 Mar 2022 15:45:42 -0800 (PST) Received: from localhost ([2601:1c0:5200:a6:307:a401:7b76:c6e5]) by smtp.gmail.com with ESMTPSA id q14-20020a056a00150e00b004f741b5c071sm8767891pfu.86.2022.03.10.15.45.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Mar 2022 15:45:41 -0800 (PST) From: Rob Clark To: dri-devel@lists.freedesktop.org Cc: freedreno@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, Rob Clark , Rob Clark , Sean Paul , Abhinav Kumar , David Airlie , Daniel Vetter , Akhil P Oommen , Jonathan Marek , AngeloGioacchino Del Regno , Bjorn Andersson , Vladimir Lypak , linux-kernel@vger.kernel.org (open list) Subject: [PATCH 3/3] drm/msm/gpu: Remove mutex from wait_event condition Date: Thu, 10 Mar 2022 15:46:06 -0800 Message-Id: <20220310234611.424743-4-robdclark@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220310234611.424743-1-robdclark@gmail.com> References: <20220310234611.424743-1-robdclark@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-arm-msm@vger.kernel.org From: Rob Clark The mutex wasn't really protecting anything before. Before the previous patch we could still be racing with the scheduler's kthread, as that is not necessarily frozen yet. Now that we've parked the sched threads, the only race is with jobs retiring, and that is harmless, ie. Signed-off-by: Rob Clark --- drivers/gpu/drm/msm/adreno/adreno_device.c | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-) diff --git a/drivers/gpu/drm/msm/adreno/adreno_device.c b/drivers/gpu/drm/msm/adreno/adreno_device.c index 0440a98988fc..661dfa7681fb 100644 --- a/drivers/gpu/drm/msm/adreno/adreno_device.c +++ b/drivers/gpu/drm/msm/adreno/adreno_device.c @@ -607,15 +607,6 @@ static int adreno_runtime_resume(struct device *dev) return gpu->funcs->pm_resume(gpu); } -static int active_submits(struct msm_gpu *gpu) -{ - int active_submits; - mutex_lock(&gpu->active_lock); - active_submits = gpu->active_submits; - mutex_unlock(&gpu->active_lock); - return active_submits; -} - static int adreno_runtime_suspend(struct device *dev) { struct msm_gpu *gpu = dev_to_gpu(dev); @@ -669,7 +660,7 @@ static int adreno_system_suspend(struct device *dev) suspend_scheduler(gpu); remaining = wait_event_timeout(gpu->retire_event, - active_submits(gpu) == 0, + gpu->active_submits == 0, msecs_to_jiffies(1000)); if (remaining == 0) { dev_err(dev, "Timeout waiting for GPU to suspend\n");