From patchwork Sun Sep 1 13:54:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Lypak X-Patchwork-Id: 824621 Received: from mail-lf1-f47.google.com (mail-lf1-f47.google.com [209.85.167.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DD1C323776; Sun, 1 Sep 2024 13:58:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725199115; cv=none; b=d8ga+ASSdjfDt0Nv5ZhunJfXyOU7Vbm7URguEVSdDlw09jR7EPoGpNGnMjm8AjLVX9EsjHIa7/X0Fu3RZsuzAdDzYgxwYAtlRB+JifAfUAOka/+xivFC7C65PyfK0ijeTqGTJnAo2GkUyMWE8ZkpLJTHKFUZ9AbxXDksbAk0sJY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725199115; c=relaxed/simple; bh=snn4m/zasEEhcc7exBGazglhiLuUySVTjszGGgcH1SY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=G0P4LvGIwZiaz4GlEkeqTXpQEA6OfV+5xTlqvo4hHUrv50cYxhfqdmrAx4xfKti/47iLeuSf0hXBnCUOOd0+iVsUhx0ufM6HI1ahy3hBj/JYO8ADNasDlprfAz+3OKMyBeMgPeshpbKRQghjAotZBM1DOU8EcpfBS/sawdw0WlY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=POd3zqZH; arc=none smtp.client-ip=209.85.167.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="POd3zqZH" Received: by mail-lf1-f47.google.com with SMTP id 2adb3069b0e04-5333b2fbedaso5787648e87.0; Sun, 01 Sep 2024 06:58:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725199112; x=1725803912; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+eCXOU2o0WyaxUOWiQryFOZ2WSRKqjV/d0DGCFko1Bo=; b=POd3zqZH7Unzaq21QxJkMyOzFqwv6QqYF8W6qkilFqyjnk23Q9Pe4gU0pTPumLm2Ow AKn/LxATcBHLROEYmxsT3M8yG+BrWIXLc2gtvqGFbmVz2xcvDnti74CtY7Xo6PwqKwJi nCrTn4L9SppZPkcpEEjTOOLxloka2KPP4WzKgbQlsQX5foYFL6iO8Q5ISakQ7GsivQiJ u8YWmC+bSmIGCovfA2T0ZQ39shrLUDVGlYs0nurHA5D9/uB6ZNgn3skz3ng4Suc6bOkb p+aXmYu7ZH6X+DdEHcJC6FCahdNhVj5D4m/f7c9aEZeMVGOt8UJf1QR1hc6nmew5w/8J 2ADw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725199112; x=1725803912; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+eCXOU2o0WyaxUOWiQryFOZ2WSRKqjV/d0DGCFko1Bo=; b=VtaK7VVBPflU4yS0HwsqCvRkxA8e/y09NDc6AAdSnMqov6O25AilV3IhPqO1koo0zJ ehKt9+tna53R/9KQRAsF0tOW6IvrBfApRIgratjpLL/bpmI4TRkbwxy0OlyX+w4jBBrP W1wLr2/xo71No5PH39FrM7XAs/Snv0fTbtL1XMy6A6O2WTw3sGcKepGF69heV/D9JMCL x6CC8x3XnUVRgNwYwofCZhi9ZSXQeMU129eJtsnLP0gNRDzHzM6xmIE8rCfoPL9ma6W9 hOe2BDvB0PzbqeruwjQ8hmwHxhCRGM/cZAOO8EnNfLZOr1s2VQYiPjyEEboMSwBGgsI/ Wv2A== X-Forwarded-Encrypted: i=1; AJvYcCVnQATY/DlwN/H4OGTZbzD8Fye+Xej0aDiIqvg2BxJsodSGR8l70UPORiwOSMsrxnxrL488cQCrfZdT+cwM@vger.kernel.org, AJvYcCX8DE9lJOU1NFjnkRWSjtFefTUgqBNhvtwTDo5MWW3qwLIajfPC8iWuLznsUZkeSwRuU6sl3Bj6FkYC0CVp@vger.kernel.org X-Gm-Message-State: AOJu0Yw3OlbkNy6Mux/TsJgn6EXou6ZZSFaBXWJ/ewxaprQKO8Kwy1zK QMEhsVY705WKuiuyV5cd+w9gcho6NMO0bh9IgpUrG5cPQpDL1jt2 X-Google-Smtp-Source: AGHT+IEhJhR+EGTz9SVuSnRPn3GWsNoA7zhQ8yY5bQFJ0WhnpllSB30EbmSSaMGVYnNQTqlr8qSSfg== X-Received: by 2002:a2e:851:0:b0:2ee:7a71:6e3b with SMTP id 38308e7fff4ca-2f61e05d76fmr44960571fa.27.1725199111104; Sun, 01 Sep 2024 06:58:31 -0700 (PDT) Received: from localhost.localdomain (public-nat-01.vpngate.v4.open.ad.jp. [219.100.37.233]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5c226ccff17sm4051295a12.73.2024.09.01.06.58.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 01 Sep 2024 06:58:30 -0700 (PDT) From: Vladimir Lypak To: Vladimir Lypak Cc: Rob Clark , Sean Paul , Konrad Dybcio , Abhinav Kumar , Dmitry Baryshkov , Marijn Suijten , David Airlie , Daniel Vetter , Jordan Crouse , linux-arm-msm@vger.kernel.org, dri-devel@lists.freedesktop.org, freedreno@lists.freedesktop.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 1/4] drm/msm/a5xx: disable preemption in submits by default Date: Sun, 1 Sep 2024 13:54:00 +0000 Message-ID: <20240901135419.1075412-2-vladimir.lypak@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20240901135419.1075412-1-vladimir.lypak@gmail.com> References: <20240901135419.1075412-1-vladimir.lypak@gmail.com> Precedence: bulk X-Mailing-List: linux-arm-msm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Fine grain preemption (switching from/to points within submits) requires extra handling in command stream of those submits, especially when rendering with tiling (using GMEM). However this handling is missing at this point in mesa (and always was). For this reason we get random GPU faults and hangs if more than one priority level is used because local preemption is enabled prior to executing command stream from submit. With that said it was ahead of time to enable local preemption by default considering the fact that even on downstream kernel it is only enabled if requested via UAPI. Fixes: a7a4c19c36de ("drm/msm/a5xx: fix setting of the CP_PREEMPT_ENABLE_LOCAL register") Signed-off-by: Vladimir Lypak --- drivers/gpu/drm/msm/adreno/a5xx_gpu.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/msm/adreno/a5xx_gpu.c b/drivers/gpu/drm/msm/adreno/a5xx_gpu.c index c0b5373e90d7..6c80d3003966 100644 --- a/drivers/gpu/drm/msm/adreno/a5xx_gpu.c +++ b/drivers/gpu/drm/msm/adreno/a5xx_gpu.c @@ -150,9 +150,13 @@ static void a5xx_submit(struct msm_gpu *gpu, struct msm_gem_submit *submit) OUT_PKT7(ring, CP_SET_PROTECTED_MODE, 1); OUT_RING(ring, 1); - /* Enable local preemption for finegrain preemption */ + /* + * Disable local preemption by default because it requires + * user-space to be aware of it and provide additional handling + * to restore rendering state or do various flushes on switch. + */ OUT_PKT7(ring, CP_PREEMPT_ENABLE_LOCAL, 1); - OUT_RING(ring, 0x1); + OUT_RING(ring, 0x0); /* Allow CP_CONTEXT_SWITCH_YIELD packets in the IB2 */ OUT_PKT7(ring, CP_YIELD_ENABLE, 1); From patchwork Sun Sep 1 13:54:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Lypak X-Patchwork-Id: 825018 Received: from mail-ed1-f47.google.com (mail-ed1-f47.google.com [209.85.208.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9214F17E009; Sun, 1 Sep 2024 13:58:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725199136; cv=none; b=jCVKZs9x104fR7ohs4m5zQogdKCev4B+CPQg0VdlN44fPRO5fpiURwuB6mRhG6Ht2nb0nowO+vqxR47nOHOM8MzDUHVNwQwODDWWJoOCfTtbY95DkzQUWEK1VOIhGveXcSh7+xqxKc0MLG3K5wWjRH75luXGLadcuxPsLbEqRUI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725199136; c=relaxed/simple; bh=s0i5ADl0K7DntgQCTfBvEdaBgPw83fZVlWFAqfcd7oI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Y7+jImiOTkvEVd85DHDNCbDDgLcfiMWnxy+CMG8gsZgR4egYVeNKkVPkOiin9sGWIpFZ5P5h33pPykpF4vVm3M2V9NkdiyQkBpwN0Gs5Ysh7VC5zfxuLELVWXJjsKTKp040PQtTDpYh6a0VMz+DE7+aLNKH9RSUUVC/+0o4yr6Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=jooq4BOn; arc=none smtp.client-ip=209.85.208.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="jooq4BOn" Received: by mail-ed1-f47.google.com with SMTP id 4fb4d7f45d1cf-5c245c62362so938472a12.0; Sun, 01 Sep 2024 06:58:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725199133; x=1725803933; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=B+CwOEtqu3IBPCzfQKTkK2Lp868q6E74YpTPp0btJmU=; b=jooq4BOnX1S5+ZKjqukZiCzpvpitfeHy8bCBBmf9yo5LPZuJrleFXGCpQHWBJZ1lJ1 lsBsF2JjzqNkio1AQMk5/07MhZEXkwSKoYH4knMfbF2nIxmzx14luBapthp0LbJbH0VM 0XZEw85rDzQNVdb77a4qIr1X431c1aJXQh9vWT0xqxC4uHiVB4OPe6blgz2eToU9YMBc i5LOLSaHLF39TpNTPcIHWzlCYvOjcn+R+8Xp+tp7Az8tLWZP9XiWm8qifCA2YBifID0w MR6G1xcIqGU5MuFE3v0h7xsI6KnOIg0g3puayvZjOqjlOVuev/xNlpXLh0jdPWBMdCK7 0oGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725199133; x=1725803933; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=B+CwOEtqu3IBPCzfQKTkK2Lp868q6E74YpTPp0btJmU=; b=XI/A4c6jmeje6tDB7AFo2pY7kIoeIzDjqS4+v3gK5oYhY1r09kZPHMIkLXUofavu4P V1yHKio/KIbuufU5mczX8GqFczy5X6U5LLrF7qWHJSXpz/5ia11ybtdpg7tBs8UzN1C2 SfDYTNYmGaFTifOTd4rT6WJRMtnp5Vrb/kKRpjCRz2Qfy7g4RHinR1FAI7AJI17MDE8l 6fRganlls7jpUMEMsxksUAl0fexOmlylFZc55+eewTUkq6PtW6mtaGkoqmkUNljBPY2e XLb40dbXR3M+NS6oSloUxGbgBnRahyqii1Nfp+NieZTs1zJV76HQeboPjqudiBzRqGw0 YpvA== X-Forwarded-Encrypted: i=1; AJvYcCUdhz2JTMbSbVEyadrspafvAWJg2DasoIs27K+hIsrkyh56YYKTll3s0a3BxB4Pyf4Zvlf6KRDOlUtacylt@vger.kernel.org, AJvYcCWDkz8pXfdi00JUwOPXzT2F+o8zR5Qchy8HOBPLT5+VvLjGozbtjk+B9KHsvq3lkkhukJts7Xvf3d04z/E5@vger.kernel.org X-Gm-Message-State: AOJu0YwMRjSXNuvBu8AZiuMJRA1d6b1HE3LVbWkgNquZ+1HTcKYPCmoA dEeR1SXX+laiVnuS477+Lj+3pnnY4njqyJxwcSaJT/RPrhJudkcx X-Google-Smtp-Source: AGHT+IGB9/p9xuAKpPD5hLbzPgo/k1PC7HI4vXcDLu/C9nNRamu9OWu9KjLrtObSAMysisCiqYCWgA== X-Received: by 2002:a05:6402:254d:b0:5c2:5cc8:353f with SMTP id 4fb4d7f45d1cf-5c25cc837bcmr436347a12.22.1725199132019; Sun, 01 Sep 2024 06:58:52 -0700 (PDT) Received: from localhost.localdomain (public-nat-01.vpngate.v4.open.ad.jp. [219.100.37.233]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5c226ccff17sm4051295a12.73.2024.09.01.06.58.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 01 Sep 2024 06:58:51 -0700 (PDT) From: Vladimir Lypak To: Vladimir Lypak Cc: Konrad Dybcio , Rob Clark , Sean Paul , Konrad Dybcio , Abhinav Kumar , Dmitry Baryshkov , Marijn Suijten , David Airlie , Daniel Vetter , Jordan Crouse , linux-arm-msm@vger.kernel.org, dri-devel@lists.freedesktop.org, freedreno@lists.freedesktop.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 2/4] drm/msm/a5xx: properly clear preemption records on resume Date: Sun, 1 Sep 2024 13:54:01 +0000 Message-ID: <20240901135419.1075412-3-vladimir.lypak@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20240901135419.1075412-1-vladimir.lypak@gmail.com> References: <20240901135419.1075412-1-vladimir.lypak@gmail.com> Precedence: bulk X-Mailing-List: linux-arm-msm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Two fields of preempt_record which are used by CP aren't reset on resume: "data" and "info". This is the reason behind faults which happen when we try to switch to the ring that was active last before suspend. In addition those faults can't be recovered from because we use suspend and resume to do so (keeping values of those fields again). Fixes: b1fc2839d2f9 ("drm/msm: Implement preemption for A5XX targets") Signed-off-by: Vladimir Lypak Reviewed-by: Konrad Dybcio --- drivers/gpu/drm/msm/adreno/a5xx_preempt.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/gpu/drm/msm/adreno/a5xx_preempt.c b/drivers/gpu/drm/msm/adreno/a5xx_preempt.c index f58dd564d122..67a8ef4adf6b 100644 --- a/drivers/gpu/drm/msm/adreno/a5xx_preempt.c +++ b/drivers/gpu/drm/msm/adreno/a5xx_preempt.c @@ -204,6 +204,8 @@ void a5xx_preempt_hw_init(struct msm_gpu *gpu) return; for (i = 0; i < gpu->nr_rings; i++) { + a5xx_gpu->preempt[i]->data = 0; + a5xx_gpu->preempt[i]->info = 0; a5xx_gpu->preempt[i]->wptr = 0; a5xx_gpu->preempt[i]->rptr = 0; a5xx_gpu->preempt[i]->rbase = gpu->rb[i]->iova; From patchwork Sun Sep 1 13:54:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Lypak X-Patchwork-Id: 824620 Received: from mail-lj1-f176.google.com (mail-lj1-f176.google.com [209.85.208.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9A7B017DFE4; Sun, 1 Sep 2024 13:59:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725199158; cv=none; b=d5dYs8CXJeKqY6TVjeXS6Ukw20HwgCT43ZybOSi3oFvYbij75Zh8WTcRzlRhXTRWraZAOLNyQJjg/Ft4AbVdbbdVT5KQwhSseHKrVmWiqh0RPXcL5wKMVdZ6OyvTa7F0Q/+Dls2MF3EMBRLthDww92ZD5jXY2IAd+KWHvFpbkrk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725199158; c=relaxed/simple; bh=eyhl/34IFIo+EgeNhe/OGrVE6CHvdGyi0/Ejw77Ja98=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hf+4x3IMWmXyZouzVtlv6Hn/21pZrUpIPFJ13AXWHQW8ugBeI5thG2rnB3T/cD3EGTwUk0UL0iwbhMG45XlTFV6nZ/JxIUoKSJY24dpeozrxCW254n3aeGRq7rVGyj43OE2e3kZP808lfsJK+58N02Zqo9HDT41qAd7OlmBRUgc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=WOIaHIyO; arc=none smtp.client-ip=209.85.208.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="WOIaHIyO" Received: by mail-lj1-f176.google.com with SMTP id 38308e7fff4ca-2f50966c478so34825861fa.1; Sun, 01 Sep 2024 06:59:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725199155; x=1725803955; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=T92lHW8RUNhDHAQvibb0adZUC3Zfr3PwSsvkl+HJFgA=; b=WOIaHIyOeSPrPEnl4wd4mBygIr4FQH5hVgDtehRg9/T/sEolSnScuK7vBtUoSPibXI +xp8E/hzt0U1qrRw+A9q/iD9xTZxpFE/87XfYwehayDrHtK1Hzp4WEGgaqGMMRLEgWys gFypWz/a8syJCH1zYvrG5HLx6tsmHe+DpooQ/h9x/E6eDfZAy62YoRXEQEGeNR30ebn6 Bwc+olSeYumVnKB1e0AplJMARV/+VadPxmP8STmtnM2pWl13CyRNsGBB6MRbi9AIeaw3 Vi7j6lMs8L9ETVI/iz9tMbdDQmI+HkRotG1Xb5sBQtkuXhq99AfY3k+LLBi+HJjjnpkV 7CKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725199155; x=1725803955; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=T92lHW8RUNhDHAQvibb0adZUC3Zfr3PwSsvkl+HJFgA=; b=ITMOLybdyVSA/9XXZdOS1e6dPSeSRjSyO3uR3i5PfW/vdNMEyoKrEVdxUt5eVoqQCE 4MtwcmEtEhMGPsG+8zXFUX+6GCxNxouvCUryoPppzVDAIKRXqzYuEpcDnC6uGm0gGpG+ 30bzZUIZ9m23fBfQOPT1G+kEDYV6z+8T0z0pYf53nULKl+nqP62DT/4Ax8tljfLD5yUE s4Ag/kM9VwcKKYyVyy+GGE0oJ83MELjCKqudZ3FCqW9vkeF5rRzAPM0w7R3y1JJHHLiZ WQAnN1NE6f+n/z8hLIUPF8lY+mTmRQwRejJQ5vzCwDZO10IwxydBCl7mJinhLuKh1ci7 OlyQ== X-Forwarded-Encrypted: i=1; AJvYcCVIc8dg7la5sLWHo5cJq6NfQ0l176g+4vKlZE9WlFa36n25eZS66ZgeYKu6e3PALWgxIgI0IpSM5nCYVJhP@vger.kernel.org, AJvYcCVzuHX/ALXUDNWRQ5UxcclVdgG+A/RlezrLKYjeijsjwvBLhouXxZ8mL+0fjSU+SdLwmj97/DWh44gmdxwY@vger.kernel.org X-Gm-Message-State: AOJu0YwZkpJYzPS3gcaeK3K6sk1dgpz2AKj1uf0pviSOqAuD7xQKq+UY oH/4HaIaeNAOuXofXJcLhWYplabi2TgXwTp0KcJfjdM3eLKNj4PT X-Google-Smtp-Source: AGHT+IFwUQ0fX0i6XjltYeo1Dom59dNY1fFMpQnob6UlvVpTnE3Q+SJIj64awZOB1p5Wn1LEwZqoVA== X-Received: by 2002:a05:651c:198c:b0:2f3:d560:ed9f with SMTP id 38308e7fff4ca-2f61e025823mr55094181fa.5.1725199153752; Sun, 01 Sep 2024 06:59:13 -0700 (PDT) Received: from localhost.localdomain (public-nat-01.vpngate.v4.open.ad.jp. [219.100.37.233]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5c226ccff17sm4051295a12.73.2024.09.01.06.58.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 01 Sep 2024 06:59:13 -0700 (PDT) From: Vladimir Lypak To: Vladimir Lypak Cc: Rob Clark , Sean Paul , Konrad Dybcio , Abhinav Kumar , Dmitry Baryshkov , Marijn Suijten , David Airlie , Daniel Vetter , Jordan Crouse , linux-arm-msm@vger.kernel.org, dri-devel@lists.freedesktop.org, freedreno@lists.freedesktop.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 3/4] drm/msm/a5xx: fix races in preemption evaluation stage Date: Sun, 1 Sep 2024 13:54:02 +0000 Message-ID: <20240901135419.1075412-4-vladimir.lypak@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20240901135419.1075412-1-vladimir.lypak@gmail.com> References: <20240901135419.1075412-1-vladimir.lypak@gmail.com> Precedence: bulk X-Mailing-List: linux-arm-msm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 On A5XX GPUs when preemption is used it's invietable to enter a soft lock-up state in which GPU is stuck at empty ring-buffer doing nothing. This appears as full UI lockup and not detected as GPU hang (because it's not). This happens due to not triggering preemption when it was needed. Sometimes this state can be recovered by some new submit but generally it won't happen because applications are waiting for old submits to retire. One of the reasons why this happens is a race between a5xx_submit and a5xx_preempt_trigger called from IRQ during submit retire. Former thread updates ring->cur of previously empty and not current ring right after latter checks it for emptiness. Then both threads can just exit because for first one preempt_state wasn't NONE yet and for second one all rings appeared to be empty. To prevent such situations from happening we need to establish guarantee for preempt_trigger to make decision after each submit or retire. To implement this we serialize preemption initiation using spinlock. If switch is already in progress we need to re-trigger preemption when it finishes. Fixes: b1fc2839d2f9 ("drm/msm: Implement preemption for A5XX targets") Signed-off-by: Vladimir Lypak --- drivers/gpu/drm/msm/adreno/a5xx_gpu.h | 1 + drivers/gpu/drm/msm/adreno/a5xx_preempt.c | 24 +++++++++++++++++++++-- 2 files changed, 23 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/msm/adreno/a5xx_gpu.h b/drivers/gpu/drm/msm/adreno/a5xx_gpu.h index c7187bcc5e90..b4d06ca3e499 100644 --- a/drivers/gpu/drm/msm/adreno/a5xx_gpu.h +++ b/drivers/gpu/drm/msm/adreno/a5xx_gpu.h @@ -36,6 +36,7 @@ struct a5xx_gpu { uint64_t preempt_iova[MSM_GPU_MAX_RINGS]; atomic_t preempt_state; + spinlock_t preempt_start_lock; struct timer_list preempt_timer; struct drm_gem_object *shadow_bo; diff --git a/drivers/gpu/drm/msm/adreno/a5xx_preempt.c b/drivers/gpu/drm/msm/adreno/a5xx_preempt.c index 67a8ef4adf6b..c65b34a4a8cc 100644 --- a/drivers/gpu/drm/msm/adreno/a5xx_preempt.c +++ b/drivers/gpu/drm/msm/adreno/a5xx_preempt.c @@ -97,12 +97,19 @@ void a5xx_preempt_trigger(struct msm_gpu *gpu) if (gpu->nr_rings == 1) return; + /* + * Serialize preemption start to ensure that we always make + * decision on latest state. Otherwise we can get stuck in + * lower priority or empty ring. + */ + spin_lock_irqsave(&a5xx_gpu->preempt_start_lock, flags); + /* * Try to start preemption by moving from NONE to START. If * unsuccessful, a preemption is already in flight */ if (!try_preempt_state(a5xx_gpu, PREEMPT_NONE, PREEMPT_START)) - return; + goto out; /* Get the next ring to preempt to */ ring = get_next_ring(gpu); @@ -127,9 +134,11 @@ void a5xx_preempt_trigger(struct msm_gpu *gpu) set_preempt_state(a5xx_gpu, PREEMPT_ABORT); update_wptr(gpu, a5xx_gpu->cur_ring); set_preempt_state(a5xx_gpu, PREEMPT_NONE); - return; + goto out; } + spin_unlock_irqrestore(&a5xx_gpu->preempt_start_lock, flags); + /* Make sure the wptr doesn't update while we're in motion */ spin_lock_irqsave(&ring->preempt_lock, flags); a5xx_gpu->preempt[ring->id]->wptr = get_wptr(ring); @@ -152,6 +161,10 @@ void a5xx_preempt_trigger(struct msm_gpu *gpu) /* And actually start the preemption */ gpu_write(gpu, REG_A5XX_CP_CONTEXT_SWITCH_CNTL, 1); + return; + +out: + spin_unlock_irqrestore(&a5xx_gpu->preempt_start_lock, flags); } void a5xx_preempt_irq(struct msm_gpu *gpu) @@ -188,6 +201,12 @@ void a5xx_preempt_irq(struct msm_gpu *gpu) update_wptr(gpu, a5xx_gpu->cur_ring); set_preempt_state(a5xx_gpu, PREEMPT_NONE); + + /* + * Try to trigger preemption again in case there was a submit or + * retire during ring switch + */ + a5xx_preempt_trigger(gpu); } void a5xx_preempt_hw_init(struct msm_gpu *gpu) @@ -300,5 +319,6 @@ void a5xx_preempt_init(struct msm_gpu *gpu) } } + spin_lock_init(&a5xx_gpu->preempt_start_lock); timer_setup(&a5xx_gpu->preempt_timer, a5xx_preempt_timer, 0); } From patchwork Sun Sep 1 13:54:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Lypak X-Patchwork-Id: 825017 Received: from mail-ed1-f49.google.com (mail-ed1-f49.google.com [209.85.208.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E0C0317CA1B; Sun, 1 Sep 2024 13:59:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725199174; cv=none; b=KJoQyeY/SKtIgrLPQLAk8DTJaEhU53p9hlb/7re4znOvo46JudXj2ewHVmsQGntrt69xryaG3PU/QwEHFrD7w6RBLsMCEDKed3AYeRtpN/BVdK1ixgn2htNHtnbtlaV9gb1MmR9f2lG+RujoKKNRSVfaw9akZV6KiNHUElBPjqI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725199174; c=relaxed/simple; bh=ppuXfwFLkVoCcl1RlKr5sWejOfYJd+xSscE5E5DdCl0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=n9hpnCR2O173/x5eLE+x+e79RisQXBOlCR4vSTi/LqO3U8r2zDSQFm/GTHViPVaLpnMt++Cfl1ZLX8twHvR50rPR6+OaOs88NvQ4zOgcdH2KETZ5pzX7fE4S3EvZK6Ky5dCo1mzHVvroIcyFBaZ77Zae/QGnl/LX8np2j1njRF4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Hk2KR2z2; arc=none smtp.client-ip=209.85.208.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Hk2KR2z2" Received: by mail-ed1-f49.google.com with SMTP id 4fb4d7f45d1cf-5c24c92f699so891221a12.2; Sun, 01 Sep 2024 06:59:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725199171; x=1725803971; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=p8JB0+Bs8ErmsZEQvGhO4vH+N+VrdVzHLUwxPZWQG6E=; b=Hk2KR2z2EiqWskXbupr+j7wXLUp/kMXKEb8NSUafwz1jdg70MCQRSrBXMIvziWy4f8 rY40EIeXNYi1c8iHnnR0gEMQ+D6Zs/mUWE4zWt8eTeg0hj6IwBG5IlaEw00fLSzKSXRY LOfkaxzgeBaoab8bOSpJnqgPaHQqfG695wOInzRZOPVIKkuVs0WObF7RGeZCQBQgob1w +3nXpAi76I82kZzs9pwqjS3a05Zij6Zl+uA33teaWv0VRaYGosPKRraxHSiZHQKwXT82 Xo8ZwVAXXA0ec/nAs8rUkxhG/kkYCXPc76ZLPvBAUAmoWYmGjZRQYo6pMThzlG7KEeSc yJ2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725199171; x=1725803971; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=p8JB0+Bs8ErmsZEQvGhO4vH+N+VrdVzHLUwxPZWQG6E=; b=K5D4t+fe3tz527qJ+9FyA7p35qHmvkda9sZAhIbIvfCIXr0cdqxp1lNWvj0W8HwqYI DtbOVDZqnBpqqQjqnTnxMXYol+X3SC6kctZa09/dZ8HQrpTG7QMiXhQ8I6i7ISBShjGm eV/xb1JdBzQ6sdxDmt5NS8+Qc8I6OYpFvOH8NHt/JypLAL8XQtJnrmZm9pUX+PUvtTQ0 4aaW4HisjYB0s8Pwu0tPH00PPwAX94qt85ptU2h/gfRWZAbSksJV4k1bf8MKPoS5R9NC NWZjW0rWsyqeGdsT+qSCrmCvnwS5oBKwodZTBIfbivwiMEVyprq3iVmfqqdM8THMsA9c yUkg== X-Forwarded-Encrypted: i=1; AJvYcCU34jrfOAfuG67m6oEw9VcrtZHDKuHxN6+uTx+TUqaFcjyucgi3W8ny2Lm428SB2y14lcat9TBLGUz5zpaw@vger.kernel.org, AJvYcCXyX/v/xLIQTmKUollmzgapvqM35l6MyPknV9NSU4lOZ+W3zBX1GbvSKVQLMSOv+uxvazJDJL4fxx048XDZ@vger.kernel.org X-Gm-Message-State: AOJu0YzGgQl/jjM5L6QzveFaAwOtgvwKnHwFB7IDCwwNlrN/JinUvtmR bv04RlyPIzYY7NZsZvBPRTGIvCM58U2mArZkOWx34Kp50VNz8nhn X-Google-Smtp-Source: AGHT+IGJ7S2u5swfQvFXUIz2GMCV89hMMGXtojuyCo4a1PrgHeRFYoBgj84CD4toi9BdIGtUsaJ9yA== X-Received: by 2002:a05:6402:1d48:b0:5b8:34a9:7fd9 with SMTP id 4fb4d7f45d1cf-5c21ed86b5amr8442101a12.27.1725199170609; Sun, 01 Sep 2024 06:59:30 -0700 (PDT) Received: from localhost.localdomain (public-nat-01.vpngate.v4.open.ad.jp. [219.100.37.233]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5c226ccff17sm4051295a12.73.2024.09.01.06.59.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 01 Sep 2024 06:59:30 -0700 (PDT) From: Vladimir Lypak To: Vladimir Lypak Cc: Rob Clark , Sean Paul , Konrad Dybcio , Abhinav Kumar , Dmitry Baryshkov , Marijn Suijten , David Airlie , Daniel Vetter , Jordan Crouse , linux-arm-msm@vger.kernel.org, dri-devel@lists.freedesktop.org, freedreno@lists.freedesktop.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 4/4] drm/msm/a5xx: workaround early ring-buffer emptiness check Date: Sun, 1 Sep 2024 13:54:03 +0000 Message-ID: <20240901135419.1075412-5-vladimir.lypak@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20240901135419.1075412-1-vladimir.lypak@gmail.com> References: <20240901135419.1075412-1-vladimir.lypak@gmail.com> Precedence: bulk X-Mailing-List: linux-arm-msm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 There is another cause for soft lock-up of GPU in empty ring-buffer: race between GPU executing last commands and CPU checking ring for emptiness. On GPU side IRQ for retire is triggered by CACHE_FLUSH_TS event and RPTR shadow (which is used to check ring emptiness) is updated a bit later from CP_CONTEXT_SWITCH_YIELD. Thus if GPU is executing its last commands slow enough or we check that ring too fast we will miss a chance to trigger switch to lower priority ring because current ring isn't empty just yet. This can escalate to lock-up situation described in previous patch. To work-around this issue we keep track of last submit sequence number for each ring and compare it with one written to memptrs from GPU during execution of CACHE_FLUSH_TS event. Fixes: b1fc2839d2f9 ("drm/msm: Implement preemption for A5XX targets") Signed-off-by: Vladimir Lypak --- drivers/gpu/drm/msm/adreno/a5xx_gpu.c | 4 ++++ drivers/gpu/drm/msm/adreno/a5xx_gpu.h | 1 + drivers/gpu/drm/msm/adreno/a5xx_preempt.c | 4 ++++ 3 files changed, 9 insertions(+) diff --git a/drivers/gpu/drm/msm/adreno/a5xx_gpu.c b/drivers/gpu/drm/msm/adreno/a5xx_gpu.c index 6c80d3003966..7cfefb5e6221 100644 --- a/drivers/gpu/drm/msm/adreno/a5xx_gpu.c +++ b/drivers/gpu/drm/msm/adreno/a5xx_gpu.c @@ -65,6 +65,8 @@ void a5xx_flush(struct msm_gpu *gpu, struct msm_ringbuffer *ring, static void a5xx_submit_in_rb(struct msm_gpu *gpu, struct msm_gem_submit *submit) { + struct adreno_gpu *adreno_gpu = to_adreno_gpu(gpu); + struct a5xx_gpu *a5xx_gpu = to_a5xx_gpu(adreno_gpu); struct msm_ringbuffer *ring = submit->ring; struct drm_gem_object *obj; uint32_t *ptr, dwords; @@ -109,6 +111,7 @@ static void a5xx_submit_in_rb(struct msm_gpu *gpu, struct msm_gem_submit *submit } } + a5xx_gpu->last_seqno[ring->id] = submit->seqno; a5xx_flush(gpu, ring, true); a5xx_preempt_trigger(gpu); @@ -210,6 +213,7 @@ static void a5xx_submit(struct msm_gpu *gpu, struct msm_gem_submit *submit) /* Write the fence to the scratch register */ OUT_PKT4(ring, REG_A5XX_CP_SCRATCH_REG(2), 1); OUT_RING(ring, submit->seqno); + a5xx_gpu->last_seqno[ring->id] = submit->seqno; /* * Execute a CACHE_FLUSH_TS event. This will ensure that the diff --git a/drivers/gpu/drm/msm/adreno/a5xx_gpu.h b/drivers/gpu/drm/msm/adreno/a5xx_gpu.h index b4d06ca3e499..9c0d701fe4b8 100644 --- a/drivers/gpu/drm/msm/adreno/a5xx_gpu.h +++ b/drivers/gpu/drm/msm/adreno/a5xx_gpu.h @@ -34,6 +34,7 @@ struct a5xx_gpu { struct drm_gem_object *preempt_counters_bo[MSM_GPU_MAX_RINGS]; struct a5xx_preempt_record *preempt[MSM_GPU_MAX_RINGS]; uint64_t preempt_iova[MSM_GPU_MAX_RINGS]; + uint32_t last_seqno[MSM_GPU_MAX_RINGS]; atomic_t preempt_state; spinlock_t preempt_start_lock; diff --git a/drivers/gpu/drm/msm/adreno/a5xx_preempt.c b/drivers/gpu/drm/msm/adreno/a5xx_preempt.c index c65b34a4a8cc..0469fea55010 100644 --- a/drivers/gpu/drm/msm/adreno/a5xx_preempt.c +++ b/drivers/gpu/drm/msm/adreno/a5xx_preempt.c @@ -55,6 +55,8 @@ static inline void update_wptr(struct msm_gpu *gpu, struct msm_ringbuffer *ring) /* Return the highest priority ringbuffer with something in it */ static struct msm_ringbuffer *get_next_ring(struct msm_gpu *gpu) { + struct adreno_gpu *adreno_gpu = to_adreno_gpu(gpu); + struct a5xx_gpu *a5xx_gpu = to_a5xx_gpu(adreno_gpu); unsigned long flags; int i; @@ -64,6 +66,8 @@ static struct msm_ringbuffer *get_next_ring(struct msm_gpu *gpu) spin_lock_irqsave(&ring->preempt_lock, flags); empty = (get_wptr(ring) == gpu->funcs->get_rptr(gpu, ring)); + if (!empty && ring == a5xx_gpu->cur_ring) + empty = ring->memptrs->fence == a5xx_gpu->last_seqno[i]; spin_unlock_irqrestore(&ring->preempt_lock, flags); if (!empty)