From patchwork Tue Apr 19 10:29:33 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 66084 Delivered-To: patch@linaro.org Received: by 10.140.93.198 with SMTP id d64csp1765456qge; Tue, 19 Apr 2016 03:29:45 -0700 (PDT) X-Received: by 10.66.66.10 with SMTP id b10mr2956523pat.12.1461061785042; Tue, 19 Apr 2016 03:29:45 -0700 (PDT) Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id yt2si9476764pab.188.2016.04.19.03.29.44; Tue, 19 Apr 2016 03:29:45 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752573AbcDSK3n (ORCPT + 29 others); Tue, 19 Apr 2016 06:29:43 -0400 Received: from mail-wm0-f49.google.com ([74.125.82.49]:38079 "EHLO mail-wm0-f49.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751336AbcDSK3l (ORCPT ); Tue, 19 Apr 2016 06:29:41 -0400 Received: by mail-wm0-f49.google.com with SMTP id u206so21612923wme.1 for ; Tue, 19 Apr 2016 03:29:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id; bh=cQzBML+mb/+mOwCCPywh5W3fkKsWQrinyVHktmWyEuw=; b=K5gVzy74U6E+EBpv14DcFI4iWfJWL4i1eI0sSunEuhIA0fTSfXqCF2iIYaHokGKlQO v2hPYaXrgY9VuhiT0yPTyXMm6fCRin3XQD3LwVUKXGwxXgjLhJko0GhHQRA/TewjZGsr XgeNNXUiMZBjNS0t6E9U8EMUIznvU7GVy2qj4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=cQzBML+mb/+mOwCCPywh5W3fkKsWQrinyVHktmWyEuw=; b=fU4FSc+XjCz6en7AKiABS0qVx9gtvp9i2T8pE9MHvG6bTq/MVpPaeXy78QNCExD2zG zMANIrnQFuq8E92i7OzJaYceG1CHzOgJgbRmrB67isqpSHha9oa26AivYnPtZ+85QiA5 gR09RQk2A70darUBvf5H2iHh2VIF/CRiWFHza3WAXtQDiCycB/bmc1By39bDEawhC+xS jSTHqxYIXTT0QNKdo79h6/RFR7yhLR5B824KKakP8Un5j3yOsXbAmLTzFkYEA38xlbl5 4BSH3IFK4bhuuKa4TvlCKOFAlCmhZayyGd5mUEPoC0qnfvk9UyjVwLiWe79QKpLx6hop LDLQ== X-Gm-Message-State: AOPr4FX+SXVbfnTaHlqjtmVHs7kCxt8ZXMBvVWfdbBCaOvQXiKfucwLkqiMQgWiShHUaLSjw X-Received: by 10.194.216.40 with SMTP id on8mr2542867wjc.40.1461061780377; Tue, 19 Apr 2016 03:29:40 -0700 (PDT) Received: from localhost.localdomain ([195.55.142.58]) by smtp.gmail.com with ESMTPSA id p67sm3539773wmp.17.2016.04.19.03.29.37 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 19 Apr 2016 03:29:39 -0700 (PDT) From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, catalin.marinas@arm.com, will.deacon@arm.com, mark.rutland@arm.com, james.morse@arm.com Cc: Ard Biesheuvel Subject: [PATCH] arm64: mm: take CWG into account in __inval_cache_range() Date: Tue, 19 Apr 2016 12:29:33 +0200 Message-Id: <1461061773-19571-1-git-send-email-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.5.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Currently, the arm64 implementation of __inval_cache_range() [aka __dma_inv_range()] takes CTR_EL0.Dminline into account for two purposes: - the stride to use for doing by-VA cache maintenance, - to check whether the start and end arguments are unaligned with respect to the cache line size, in which case the unaligned extremes need to be cleaned before being invalidated, to avoid corrupting adjacent unrelated memory contents. In the second case, the use of Dminline is incorrect, and should use the CWG field instead, since an invalidate operation could result in cache lines that are larger than Dminline to be evicted at any level of the cache hierarchy. So introduce a macro cache_cwg_size to retrieve the CWG value, and use it to clean as many cachelines as required on either end of the [start, end) interval. Signed-off-by: Ard Biesheuvel --- arch/arm64/mm/cache.S | 34 ++++++++++++++++++++++------------ arch/arm64/mm/proc-macros.S | 13 +++++++++++++ 2 files changed, 35 insertions(+), 12 deletions(-) -- 2.5.0 diff --git a/arch/arm64/mm/cache.S b/arch/arm64/mm/cache.S index 6df07069a025..e5067e87e1b5 100644 --- a/arch/arm64/mm/cache.S +++ b/arch/arm64/mm/cache.S @@ -120,19 +120,29 @@ ENTRY(__inval_cache_range) * - end - virtual end address of region */ __dma_inv_range: - dcache_line_size x2, x3 - sub x3, x2, #1 - tst x1, x3 // end cache line aligned? - bic x1, x1, x3 - b.eq 1f - dc civac, x1 // clean & invalidate D / U line -1: tst x0, x3 // start cache line aligned? - bic x0, x0, x3 + dcache_line_size x2, x3 // get Dminline in x2 + sub x3, x2, #1 // Dminline mask in x3 + bic x0, x0, x3 // align start down to line size + + cache_cwg_size x4, x3 // get CWG + sub x3, x4, #1 // CWG mask in x3 + + tst x1, x3 // end CWG aligned? b.eq 2f - dc civac, x0 // clean & invalidate D / U line - b 3f -2: dc ivac, x0 // invalidate D / U line -3: add x0, x0, x2 + bic x5, x1, x3 +0: dc civac, x5 // clean & invalidate D / U line + add x5, x5, x2 + tst x5, x3 + b.ne 0b + b 2f + +1: dc civac, x0 // clean & invalidate D / U line + add x0, x0, x2 +2: tst x0, x3 // start CWG aligned? + b.ne 1b + + dc ivac, x0 // invalidate D / U line + add x0, x0, x2 cmp x0, x1 b.lo 2b dsb sy diff --git a/arch/arm64/mm/proc-macros.S b/arch/arm64/mm/proc-macros.S index e6a30e1268a8..872299ce3081 100644 --- a/arch/arm64/mm/proc-macros.S +++ b/arch/arm64/mm/proc-macros.S @@ -54,6 +54,19 @@ .endm /* + * cache_cwg_size - get the maximum cache line size from the CTR register + */ + .macro cache_cwg_size, reg, tmp + mrs \tmp, ctr_el0 // read CTR + ubfm \tmp, \tmp, #24, #27 // CTR_EL0.CWG [27:24] + mov \reg, #9 // use architectural default of + cmp \tmp, xzr // 2 KB (2^9 words) if CWG is + csel \tmp, \tmp, \reg, ne // not provided + mov \reg, #4 // bytes per word + lsl \reg, \reg, \tmp // actual cache line size + .endm + +/* * tcr_set_idmap_t0sz - update TCR.T0SZ so that we can load the ID map */ .macro tcr_set_idmap_t0sz, valreg, tmpreg