From patchwork Fri Jan 12 00:46:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 124242 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp1432778qgn; Thu, 11 Jan 2018 16:54:49 -0800 (PST) X-Google-Smtp-Source: ACJfBovZDeVv3GoCTiJ7lgh4D+GU/GhECXy5K3SxeaI3VAbFYepnbZ7qdA8LCxaQSe6YM/CQzbDh X-Received: by 10.98.247.19 with SMTP id h19mr1451685pfi.77.1515718489865; Thu, 11 Jan 2018 16:54:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515718489; cv=none; d=google.com; s=arc-20160816; b=AYmyX54V6rCg20Jvp9SK1/oqw1KOGSx13na7fNXYo53ktroSu0+a3YjersaBmvo24l lZq9qIMf+zm4aU06EymENjZxUeUIeCjuWaPpAwUYCiS/bSRItvV94NNTsVCfpGxbrLWM aRoj5ITT9qKjOUrdERZUcU6JY954WRah+u3UGRQG1f11nizbmzmo3a6WygQwNJo0PfZm E6X1znk7XEmCa5RGLi+p0AeejUyBZfIGlj3Tjs+HwKMCqET4rTGxqAw/zle70QLXb3+f pIsT5dhW8Z0l2yqFU9gh4JSO863bluW/gYxvpoSIHUwTOIpvmOw2qTKQy7ULuVZr1JwO JQVQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:cc:to:from :subject:arc-authentication-results; bh=EsM9gikBqpCAHJ9y6fAKmrOx2BpVdiLWEi7F4mlucXo=; b=i/NqbsnlWZ0Got52HY84MYnVdKuFUFfnVVSPQoC6TDugjAdMKA1PQDcGhUVXvkAhRo 5liR7BWBTeQESs0EYU29NbWu161xKJp/PylW6zSSQey45widSYyFcbZg3bqd9I0Pn0BP Eexg11rAOA0QASfSkCiQkIjsRXBOfdThkTn6vobYcUbBLA0IE0TswHP1phdqDzDul3os WMb6ftx7lo3fFnqpAgDEfxNVpnBJgJzG6W5YS/xKvB/YAU5PtDK/IcE2N4UMra7oly0a qPB7USb8950RnNAnCOqYSulW1hj+kAeqoFSl6f2VQ88FpGDhFwRbyzgURxNqC4m/8iFX pvUw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e5si12753727pgv.415.2018.01.11.16.54.49; Thu, 11 Jan 2018 16:54:49 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932586AbeALAys (ORCPT + 28 others); Thu, 11 Jan 2018 19:54:48 -0500 Received: from mga07.intel.com ([134.134.136.100]:16015 "EHLO mga07.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932148AbeALAyp (ORCPT ); Thu, 11 Jan 2018 19:54:45 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Jan 2018 16:54:44 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.46,346,1511856000"; d="scan'208";a="9502958" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.16]) by fmsmga002.fm.intel.com with ESMTP; 11 Jan 2018 16:54:43 -0800 Subject: [PATCH v2 01/19] Documentation: document array_ptr From: Dan Williams To: linux-kernel@vger.kernel.org Cc: Mark Rutland , linux-arch@vger.kernel.org, kernel-hardening@lists.openwall.com, Peter Zijlstra , Jonathan Corbet , Will Deacon , tglx@linutronix.de, torvalds@linux-foundation.org, akpm@linux-foundation.org, alan@linux.intel.com Date: Thu, 11 Jan 2018 16:46:30 -0800 Message-ID: <151571799008.27429.12325141216769795517.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <151571798296.27429.7166552848688034184.stgit@dwillia2-desk3.amr.corp.intel.com> References: <151571798296.27429.7166552848688034184.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.17.1-9-g687f MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Mark Rutland Document the rationale and usage of the new array_ptr() helper. Signed-off-by: Mark Rutland Signed-off-by: Will Deacon Cc: Dan Williams Cc: Jonathan Corbet Cc: Peter Zijlstra Signed-off-by: Dan Williams --- Documentation/speculation.txt | 142 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 142 insertions(+) create mode 100644 Documentation/speculation.txt Reviewed-by: Kees Cook diff --git a/Documentation/speculation.txt b/Documentation/speculation.txt new file mode 100644 index 000000000000..a4d465fd42cd --- /dev/null +++ b/Documentation/speculation.txt @@ -0,0 +1,142 @@ +This document explains potential effects of speculation, and how undesirable +effects can be mitigated portably using common APIs. + +=========== +Speculation +=========== + +To improve performance and minimize average latencies, many contemporary CPUs +employ speculative execution techniques such as branch prediction, performing +work which may be discarded at a later stage. + +Typically speculative execution cannot be observed from architectural state, +such as the contents of registers. However, in some cases it is possible to +observe its impact on microarchitectural state, such as the presence or +absence of data in caches. Such state may form side-channels which can be +observed to extract secret information. + +For example, in the presence of branch prediction, it is possible for bounds +checks to be ignored by code which is speculatively executed. Consider the +following code: + + int load_array(int *array, unsigned int idx) { + if (idx >= MAX_ARRAY_ELEMS) + return 0; + else + return array[idx]; + } + +Which, on arm64, may be compiled to an assembly sequence such as: + + CMP , #MAX_ARRAY_ELEMS + B.LT less + MOV , #0 + RET + less: + LDR , [, ] + RET + +It is possible that a CPU mis-predicts the conditional branch, and +speculatively loads array[idx], even if idx >= MAX_ARRAY_ELEMS. This value +will subsequently be discarded, but the speculated load may affect +microarchitectural state which can be subsequently measured. + +More complex sequences involving multiple dependent memory accesses may result +in sensitive information being leaked. Consider the following code, building +on the prior example: + + int load_dependent_arrays(int *arr1, int *arr2, int idx) + { + int val1, val2, + + val1 = load_array(arr1, idx); + val2 = load_array(arr2, val1); + + return val2; + } + +Under speculation, the first call to load_array() may return the value of an +out-of-bounds address, while the second call will influence microarchitectural +state dependent on this value. This may provide an arbitrary read primitive. + +==================================== +Mitigating speculation side-channels +==================================== + +The kernel provides a generic API to ensure that bounds checks are respected +even under speculation. Architectures which are affected by speculation-based +side-channels are expected to implement these primitives. + +The array_ptr() helper in can be used to prevent +information from being leaked via side-channels. + +A call to array_ptr(arr, idx, sz) returns a sanitized pointer to +arr[idx] only if idx falls in the [0, sz) interval. When idx < 0 or idx > sz, +NULL is returned. Additionally, array_ptr() an out-of-bounds poitner is +not propagated to code which is speculatively executed. + +This can be used to protect the earlier load_array() example: + + int load_array(int *array, unsigned int idx) + { + int *elem; + + elem = array_ptr(array, idx, MAX_ARRAY_ELEMS); + if (elem) + return *elem; + else + return 0; + } + +This can also be used in situations where multiple fields on a structure are +accessed: + + struct foo array[SIZE]; + int a, b; + + void do_thing(int idx) + { + struct foo *elem; + + elem = array_ptr(array, idx, SIZE); + if (elem) { + a = elem->field_a; + b = elem->field_b; + } + } + +It is imperative that the returned pointer is used. Pointers which are +generated separately are subject to a number of potential CPU and compiler +optimizations, and may still be used speculatively. For example, this means +that the following sequence is unsafe: + + struct foo array[SIZE]; + int a, b; + + void do_thing(int idx) + { + if (array_ptr(array, idx, SIZE) != NULL) { + // unsafe as wrong pointer is used + a = array[idx].field_a; + b = array[idx].field_b; + } + } + +Similarly, it is unsafe to compare the returned pointer with other pointers, +as this may permit the compiler to substitute one pointer with another, +permitting speculation. For example, the following sequence is unsafe: + + struct foo array[SIZE]; + int a, b; + + void do_thing(int idx) + { + struct foo *elem = array_ptr(array, idx, size); + + // unsafe due to pointer substitution + if (elem == &array[idx]) { + a = elem->field_a; + b = elem->field_b; + } + } + From patchwork Fri Jan 12 00:46:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 124243 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp1432860qgn; Thu, 11 Jan 2018 16:54:57 -0800 (PST) X-Google-Smtp-Source: ACJfBosbCsmbKmEUaT8W4bEFv3jmfWSl1Sz3EetxBq19Yp4Vmh2C8G3Ofct1MWTtmjp9Z+BEBteY X-Received: by 10.98.60.132 with SMTP id b4mr17178798pfk.120.1515718497274; Thu, 11 Jan 2018 16:54:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515718497; cv=none; d=google.com; s=arc-20160816; b=sqwi//QdLMG4KRYFLz1zmuLCFrYw6u2w6h0AGuQavf0ajSM+hwVCGGyPxTiLaLiwSU /iOBZAtrKvD/bB/ZwCTEUMIhl7OJ8g0RFfCj3TfKIV4hWfX0wZjhibJHGCwphsnVxUza 4Z1mkBiaUS8Ygo2bKUv6wICbRBrYAgRsAzzUW0fFP2qpAymuXWno7CabsS8xFH5nRl6i QzGo0b83r6G8Rnn09Pzrf4EzGlVre0etWUu2Ub4xYLuLVxfVd27eZWaBYFDGD72k4jBu Qet3VM3JvuT0e8C1QusfZPxf/A5nLvWvnnnjTwpUi+GtKFx6wyblOvqg35p91NaRWwya tNwQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:cc:to:from :subject:arc-authentication-results; bh=CAzMmmBJMECbvTfGzAhTC3lRFGGz5Wt2cghVThC75dA=; b=PttKhes/QS7PCuoxHk9viECmLNDN/vjYDRseEmpYdY/IQ/ZN/PUvOvlQ+DducMquBj 7Yz807kMotNvusSGrvL/jcN9YKmK76rorS9EmFePJBPHoiUTmGpvjGIKweLeXmm3jIpw s4TPnnEilE7R28zxz4V3aqGJ6vYaGoCW7X4mRHJzSJ2YWAOUD9nkmA+Prn9ji6/Mj5SX x7SVKTaT7Z/pMCcz9RouvdCVW1FsCL5aygYHr2mB45cfi3hpOJ8qJXOTxmR9nXtjiC3l ZrHdari+ID6VKDhjuOeQDyOj9tSWhJoaZ4Zx8FiFJxAdbxDd0GfA4kTw7t8tSdDZAeKR rlHw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id n8si2706537pgr.519.2018.01.11.16.54.57; Thu, 11 Jan 2018 16:54:57 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932783AbeALAyz (ORCPT + 28 others); Thu, 11 Jan 2018 19:54:55 -0500 Received: from mga01.intel.com ([192.55.52.88]:23328 "EHLO mga01.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932148AbeALAyv (ORCPT ); Thu, 11 Jan 2018 19:54:51 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Jan 2018 16:54:49 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.46,346,1511856000"; d="scan'208";a="10599391" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.16]) by orsmga006.jf.intel.com with ESMTP; 11 Jan 2018 16:54:49 -0800 Subject: [PATCH v2 02/19] arm64: implement ifence_array_ptr() From: Dan Williams To: linux-kernel@vger.kernel.org Cc: Mark Rutland , linux-arch@vger.kernel.org, kernel-hardening@lists.openwall.com, Peter Zijlstra , Will Deacon , tglx@linutronix.de, torvalds@linux-foundation.org, akpm@linux-foundation.org, alan@linux.intel.com Date: Thu, 11 Jan 2018 16:46:35 -0800 Message-ID: <151571799562.27429.4130452417520177349.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <151571798296.27429.7166552848688034184.stgit@dwillia2-desk3.amr.corp.intel.com> References: <151571798296.27429.7166552848688034184.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.17.1-9-g687f MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Mark Rutland This patch implements ifence_array_ptr() for arm64, using an LDR+CSEL+CSDB sequence to inhibit speculative use of the returned value. Signed-off-by: Mark Rutland Signed-off-by: Will Deacon Cc: Peter Zijlstra Signed-off-by: Dan Williams --- arch/arm64/include/asm/barrier.h | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/arch/arm64/include/asm/barrier.h b/arch/arm64/include/asm/barrier.h index 77651c49ef44..74ffcddb26e6 100644 --- a/arch/arm64/include/asm/barrier.h +++ b/arch/arm64/include/asm/barrier.h @@ -40,6 +40,30 @@ #define dma_rmb() dmb(oshld) #define dma_wmb() dmb(oshst) +#define ifence_array_ptr(arr, idx, sz) \ +({ \ + typeof(&(arr)[0]) __nap_arr = (arr); \ + typeof(idx) __nap_idx = (idx); \ + typeof(sz) __nap_sz = (sz); \ + \ + unsigned long __nap_ptr = (unsigned long)__nap_arr + \ + sizeof(__nap_arr[0]) * idx; \ + \ + asm volatile( \ + " cmp %[i], %[s]\n" \ + " b.cs 1f\n" \ + " ldr %[p], %[pp]\n" \ + "1: csel %[p], %[p], xzr, cc\n" \ + " hint #0x14 // CSDB\n" \ + : [p] "=&r" (__nap_ptr) \ + : [pp] "m" (__nap_ptr), \ + [i] "r" ((unsigned long)__nap_idx), \ + [s] "r" ((unsigned long)__nap_sz) \ + : "cc"); \ + \ + (typeof(&(__nap_arr)[0]))__nap_ptr; \ +}) + #define __smp_mb() dmb(ish) #define __smp_rmb() dmb(ishld) #define __smp_wmb() dmb(ishst) From patchwork Fri Jan 12 00:46:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 124255 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp1437568qgn; Thu, 11 Jan 2018 17:00:40 -0800 (PST) X-Google-Smtp-Source: ACJfBoscM5FKiHvMkVK6RL3xUQrYylTBSWcDAUpGD1t38SRs7QNDX93QbPA9W/a800G/ECsL1y32 X-Received: by 10.98.61.208 with SMTP id x77mr16973729pfj.2.1515718840488; Thu, 11 Jan 2018 17:00:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515718840; cv=none; d=google.com; s=arc-20160816; b=trLm3cLRrgXJF0q+4DtTLc7rnB7Cz2cFW0+2v6Aq3rTuq2TxnSaAGiZEg/IpT2agfg Sjk0AUF2m8QulODUC0mJgP/6U/DdjlhshKWyENS8NZ+riXIh6bH1gsZMmLvhu8OpYpr9 jTpG4MB6cpRUMfcOeM7hHpK7oOyPWdxwkhi6zhKtL5GsUVytf/0vrLTrD5x9lpOScMwz jDqCtL27jOHNR1AtNCWqFb7wp8GerHlQm10SWFFHkpZ5tPhXIr7xJqCHdN8FW1qmdqsT HQjL8DOjPjkl+30rPFuO1TtYsntj3yWRi5b6WCNG8kYReBJpW0EhRB32E4nf3fB+LT3K lC/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:cc:to:from :subject:arc-authentication-results; bh=BkLGLtbQW9bkVucVMp86LgsUzo6h0NgEkcJYYjlhOM4=; b=ZAmM70/QRuZNlB/Rup+sqxt5nL5tCie6OVvGp6jJx/3kvzOplsEBOn6kmO9iQF+rrx i8D8Q0qFnBv0XnjS7m8YCxh+tsku9A4GsFfcOqh1VbB7/pyz841E3MIP+e7y4sGXDeFo uB5UdnRZZpEgD5QgB9T872Pc5yBOfPU9UEsolaMZQmk6L1K6MbvlcowirArKGwtKXlDb SLYsLYsx6mIkx+ESKM6k9ramMsklG4dSERrYKkhgFw0+f2vs6CLwEv1j8Lr5yGpVkkNy MuFVV2APh/vl1PHo28RanuvohhxWGvzQcuFZ3fosx/+qbixAij7yPfrAHXbtl237jc/l 8G6Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id y20si12800228pgv.215.2018.01.11.17.00.40; Thu, 11 Jan 2018 17:00:40 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933432AbeALBAi (ORCPT + 28 others); Thu, 11 Jan 2018 20:00:38 -0500 Received: from mga01.intel.com ([192.55.52.88]:23328 "EHLO mga01.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932685AbeALAyz (ORCPT ); Thu, 11 Jan 2018 19:54:55 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Jan 2018 16:54:54 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.46,346,1511856000"; d="scan'208";a="193156924" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.16]) by orsmga005.jf.intel.com with ESMTP; 11 Jan 2018 16:54:54 -0800 Subject: [PATCH v2 03/19] arm: implement ifence_array_ptr() From: Dan Williams To: linux-kernel@vger.kernel.org Cc: Mark Rutland , linux-arch@vger.kernel.org, kernel-hardening@lists.openwall.com, tglx@linutronix.de, torvalds@linux-foundation.org, akpm@linux-foundation.org, alan@linux.intel.com Date: Thu, 11 Jan 2018 16:46:40 -0800 Message-ID: <151571800075.27429.5000616446434113763.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <151571798296.27429.7166552848688034184.stgit@dwillia2-desk3.amr.corp.intel.com> References: <151571798296.27429.7166552848688034184.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.17.1-9-g687f MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Mark Rutland This patch implements ifence_array_ptr() for arm, using an LDR+MOVCS+CSDB sequence to inhibit speculative use of the returned value. Signed-off-by: Mark Rutland Signed-off-by: Dan Williams --- arch/arm/include/asm/barrier.h | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/arch/arm/include/asm/barrier.h b/arch/arm/include/asm/barrier.h index 40f5c410fd8c..919235ed6e68 100644 --- a/arch/arm/include/asm/barrier.h +++ b/arch/arm/include/asm/barrier.h @@ -59,6 +59,30 @@ extern void arm_heavy_mb(void); #define dma_wmb() barrier() #endif +#define ifence_array_ptr(arr, idx, sz) \ +({ \ + typeof(&(arr)[0]) __nap_arr = (arr); \ + typeof(idx) __nap_idx = (idx); \ + typeof(sz) __nap_sz = (sz); \ + \ + unsigned long __nap_ptr = (unsigned long)__nap_arr + \ + sizeof(__nap_arr[0]) * idx; \ + \ + asm volatile( \ + " cmp %[i], %[s]\n" \ + " bcs 1f\n" \ + " ldr %[p], %[pp]\n" \ + "1: movcs %[p], #0\n" \ + " .inst 0xe320f018 @ CSDB\n" \ + : [p] "=&r" (__nap_ptr) \ + : [pp] "m" (__nap_ptr), \ + [i] "r" ((unsigned long)__nap_idx), \ + [s] "r" ((unsigned long)__nap_sz) \ + : "cc"); \ + \ + (typeof(&(__nap_arr)[0]))__nap_ptr; \ +}) + #define __smp_mb() dmb(ish) #define __smp_rmb() __smp_mb() #define __smp_wmb() dmb(ishst)