From patchwork Thu Mar 1 12:53:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Shi X-Patchwork-Id: 130194 Delivered-To: patch@linaro.org Received: by 10.80.172.228 with SMTP id x91csp2761883edc; Thu, 1 Mar 2018 04:57:17 -0800 (PST) X-Google-Smtp-Source: AG47ELsYiW9Y28mSr/9HXv4J/Z4jy7bAr8onTa+P4yXCP+ti7SKD8SIwtiRu5rUx2QtF0L/7ie/q X-Received: by 10.98.224.208 with SMTP id d77mr1885490pfm.194.1519909036980; Thu, 01 Mar 2018 04:57:16 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1519909036; cv=none; d=google.com; s=arc-20160816; b=NABXXPmCHISWdAnUEoW8KOS5wf4eC4IGRPr/e5CXNmHCKaiZ2slsDtsc7Qqx+71uCq AnOK22bwHHeckC3w101UdyyGIL+XOb1Ig54QBM6NycHPq5lWZ7Jaonb5VrFOLFdzFVqj rVEU9l2lT2b2kdJEM7ehBHDya9Lt5alnZQSOdkzRf3RpVCTnB+gwa5m7XFZ4Dx3HlGP5 CW6GkpDn0R74dlwrYZ0RBhxyMLtvxvh9TTmGKYYXcdQc32/obcfRXWuJnx97Sf3RT1q1 byhAoVBH25PEWwf7mflSuYW77Sb3mplDc+uMXXEM1KADOmb1+jKdiA8MyLTWoRk/jVXn AIRg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=B06UrOpFOhDhk8CXgd0QbaeXaNIGUHvFPN317fRlFQ4=; b=LUR/mVVPlqdIzfOdYZTMDCbAr3L1ntIQu3hJ/sY28hq9aCtdPPZjBoEoW/tehegUci gpsY1T6pUkpbeZ2f49ZSaZDJKXSF015LE9/E1wxEtD2COV2zwOwCQ9l9HG6iT+X2wlSi 6OhrC918IwkpIvvfN7gO/vvpSWEgjkv1/3z7lDPdw+rMuq/k2bYnWXjAtHfE/HgGEYif m+18aqAqurJIf0ooWwrlRi8I1AUXlcqSOcK0VzeZDPh5LexxGxrVqF83yxJbVtroKetq gdZPeIX3oaAsTRw1/3WTX2MFJBQUm/+cU8tbPqRCAzVrMhUCzpZJbewSrWCjuJQTFG5a KfJA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=D/+FhXt5; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id b185si2439066pga.370.2018.03.01.04.57.16; Thu, 01 Mar 2018 04:57:16 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=D/+FhXt5; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1030821AbeCAM5M (ORCPT + 28 others); Thu, 1 Mar 2018 07:57:12 -0500 Received: from mail-pg0-f66.google.com ([74.125.83.66]:36481 "EHLO mail-pg0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1030790AbeCAM5C (ORCPT ); Thu, 1 Mar 2018 07:57:02 -0500 Received: by mail-pg0-f66.google.com with SMTP id i14so2276325pgv.3 for ; Thu, 01 Mar 2018 04:57:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=B06UrOpFOhDhk8CXgd0QbaeXaNIGUHvFPN317fRlFQ4=; b=D/+FhXt5iKbRmF6usXiWGMZw6TVW3dhAcUiL4cOj1ExLjn8byPNEftelFZB6NgiL6m hDnmkPfXNAEK64ROsIPjHe3v4lDxFzNZOUU+ZAQLu/W69DNNMJx3CFpu5CyaGi6vQ0qG NeiU3VeNURVbqPLyJIEQ/wfKw/7LULtU41Qlk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=B06UrOpFOhDhk8CXgd0QbaeXaNIGUHvFPN317fRlFQ4=; b=THpjgMk0jWa5d302WTUCeFs4xM8dnbjOhZyj51Dbf+rMDytmgIc2CehahsFbqRKek/ 2Yqm2KuEZPm1MigJisHKzXWVcUJ1nv2a0JdH3JQJbrjCR779KCb4tjTVxEWo8HS9X3CP C4MlyDDcujD2fYmCCB/RauE841E96cPiWMxxjVFgr/ZCxRy5fGs4u78sI4c0JN1e0h1E LQHaxRqlCu16Vw1mra1bOYoKmcrDgZvozULXYyDAkMcCRa1QAzFoa5taNVv0kh9fi5Yk HjE9K8rIbsU4iCykgEtiJO3o8WG6Hvhv+8q4N/WHKeKKxaSA+J7pjDS2GE3tzgARoCQ/ xx2A== X-Gm-Message-State: APf1xPCKQmhE+B5qJLGNkrdUKWyA7PQ7yv+7dVesMZ2IAY7bOWF44wX8 nN1Y2mhZ4wrk8pnBT+LT+KUNFg== X-Received: by 10.101.69.4 with SMTP id n4mr1510244pgq.184.1519909022161; Thu, 01 Mar 2018 04:57:02 -0800 (PST) Received: from localhost.localdomain (176.122.172.82.16clouds.com. [176.122.172.82]) by smtp.gmail.com with ESMTPSA id x4sm2289655pfb.46.2018.03.01.04.56.54 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 01 Mar 2018 04:57:01 -0800 (PST) From: Alex Shi To: Marc Zyngier , Will Deacon , Ard Biesheuvel , Catalin Marinas , stable@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Cc: Alex Shi Subject: [PATCH 14/45] arm64: uaccess: Don't bother eliding access_ok checks in __{get, put}_user Date: Thu, 1 Mar 2018 20:53:51 +0800 Message-Id: <1519908862-11425-15-git-send-email-alex.shi@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1519908862-11425-1-git-send-email-alex.shi@linaro.org> References: <1519908862-11425-1-git-send-email-alex.shi@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Will Deacon commit 84624087dd7e upstream. access_ok isn't an expensive operation once the addr_limit for the current thread has been loaded into the cache. Given that the initial access_ok check preceding a sequence of __{get,put}_user operations will take the brunt of the miss, we can make the __* variants identical to the full-fat versions, which brings with it the benefits of address masking. The likely cost in these sequences will be from toggling PAN/UAO, which we can address later by implementing the *_unsafe versions. Reviewed-by: Robin Murphy Signed-off-by: Will Deacon Signed-off-by: Catalin Marinas Signed-off-by: Alex Shi --- arch/arm64/include/asm/uaccess.h | 54 ++++++++++++++++++++++++---------------- 1 file changed, 32 insertions(+), 22 deletions(-) -- 2.7.4 diff --git a/arch/arm64/include/asm/uaccess.h b/arch/arm64/include/asm/uaccess.h index 00025c5..ffa4e39 100644 --- a/arch/arm64/include/asm/uaccess.h +++ b/arch/arm64/include/asm/uaccess.h @@ -248,30 +248,35 @@ do { \ (x) = (__force __typeof__(*(ptr)))__gu_val; \ } while (0) -#define __get_user(x, ptr) \ +#define __get_user_check(x, ptr, err) \ ({ \ - int __gu_err = 0; \ - __get_user_err((x), (ptr), __gu_err); \ - __gu_err; \ + __typeof__(*(ptr)) __user *__p = (ptr); \ + might_fault(); \ + if (access_ok(VERIFY_READ, __p, sizeof(*__p))) { \ + __p = uaccess_mask_ptr(__p); \ + __get_user_err((x), __p, (err)); \ + } else { \ + (x) = 0; (err) = -EFAULT; \ + } \ }) #define __get_user_error(x, ptr, err) \ ({ \ - __get_user_err((x), (ptr), (err)); \ + __get_user_check((x), (ptr), (err)); \ (void)0; \ }) #define __get_user_unaligned __get_user -#define get_user(x, ptr) \ +#define __get_user(x, ptr) \ ({ \ - __typeof__(*(ptr)) __user *__p = (ptr); \ - might_fault(); \ - access_ok(VERIFY_READ, __p, sizeof(*__p)) ? \ - __p = uaccess_mask_ptr(__p), __get_user((x), __p) : \ - ((x) = 0, -EFAULT); \ + int __gu_err = 0; \ + __get_user_check((x), (ptr), __gu_err); \ + __gu_err; \ }) +#define get_user __get_user + #define __put_user_asm(instr, alt_instr, reg, x, addr, err, feature) \ asm volatile( \ "1:"ALTERNATIVE(instr " " reg "1, [%2]\n", \ @@ -314,30 +319,35 @@ do { \ uaccess_disable_not_uao(); \ } while (0) -#define __put_user(x, ptr) \ +#define __put_user_check(x, ptr, err) \ ({ \ - int __pu_err = 0; \ - __put_user_err((x), (ptr), __pu_err); \ - __pu_err; \ + __typeof__(*(ptr)) __user *__p = (ptr); \ + might_fault(); \ + if (access_ok(VERIFY_WRITE, __p, sizeof(*__p))) { \ + __p = uaccess_mask_ptr(__p); \ + __put_user_err((x), __p, (err)); \ + } else { \ + (err) = -EFAULT; \ + } \ }) #define __put_user_error(x, ptr, err) \ ({ \ - __put_user_err((x), (ptr), (err)); \ + __put_user_check((x), (ptr), (err)); \ (void)0; \ }) #define __put_user_unaligned __put_user -#define put_user(x, ptr) \ +#define __put_user(x, ptr) \ ({ \ - __typeof__(*(ptr)) __user *__p = (ptr); \ - might_fault(); \ - access_ok(VERIFY_WRITE, __p, sizeof(*__p)) ? \ - __p = uaccess_mask_ptr(__p), __put_user((x), __p) : \ - -EFAULT; \ + int __pu_err = 0; \ + __put_user_check((x), (ptr), __pu_err); \ + __pu_err; \ }) +#define put_user __put_user + extern unsigned long __must_check __arch_copy_from_user(void *to, const void __user *from, unsigned long n); extern unsigned long __must_check __arch_copy_to_user(void __user *to, const void *from, unsigned long n); extern unsigned long __must_check __copy_in_user(void __user *to, const void __user *from, unsigned long n);