From patchwork Fri Jun 14 03:07:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Viresh Kumar X-Patchwork-Id: 166747 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp1505437ilk; Thu, 13 Jun 2019 20:12:19 -0700 (PDT) X-Google-Smtp-Source: APXvYqyu1S+xNQVqMVvqLmBJ6Hr1VxVMWzNp4subNmrE3iOX9pG1/ftdZt2OzKm4UOMrLbIZeTIs X-Received: by 2002:aa7:8007:: with SMTP id j7mr32204796pfi.154.1560481939669; Thu, 13 Jun 2019 20:12:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560481939; cv=none; d=google.com; s=arc-20160816; b=wLUtrsA0OoslslTP9mS6AcbkW+9bZj9E4jTfLN2LFOhixoSu+Fmt2NYJRH6+0RVcMj knSPVYuFEy+qufqc3ghF+WGMrhyI47yt9j1RoXG1Y/O6vOLyKNa2WPu0CxAEEp+4cdH8 d+4tkEkjeyAkcLVpI5lS5FB4U1KjpO6J6uw7oesHeQvCtWEA03i9bQpLVDfHvv/++no9 Z/EUWHxt6LcDetfCFu0tzVOvzQ6bM0RYk2y5PEKVUSIMNVkISpBsR8rQg3ubWE0k40zK Mn+n67PW3GYGQq8oAebkPCiqFJDxDUBxLYTJUtVspC1PJLocdHb5G7IsRz3TwwVjjZnt X1/Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Lt3w372XAiP2T5A3AE8p2nBAFVuCKsgxbI6J0UJSdP8=; b=ktWLnpoMs7zuIQ7waI66H+k5KEeBtbaIiub4PGfHv04rhbzPPNZFOir2R2sww23SGW uMdQhOWs9ipr90+AW2uimWEc2zogZY4wxSRu9n4pMjgaOz0tS6Vt7udWreQka9B0gWnh Mph6mtZI2qg0IQ11kY58c9e8mPQf+wK8DXazLBsMTOKG2tkqGHDBYA/6spS1o6q3PH3i aU459aOzy2aRvL4F0gWFuNK4clcaAmGO5I6AtA6ULqnQvQ0czzLcK6xfrsQZ9VhqnJKc +eVj/PiFF3/DGK7HmPstrUYMlTHIHt/qT8zueP6mPPLLHsiaj4g/ILzPk8Y+UlqXy3ei EE8A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ARF6EuIO; spf=pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=stable-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e8si1073936plb.420.2019.06.13.20.12.19; Thu, 13 Jun 2019 20:12:19 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ARF6EuIO; spf=pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=stable-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726030AbfFNDMT (ORCPT + 14 others); Thu, 13 Jun 2019 23:12:19 -0400 Received: from mail-pl1-f195.google.com ([209.85.214.195]:36175 "EHLO mail-pl1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725819AbfFNDMS (ORCPT ); Thu, 13 Jun 2019 23:12:18 -0400 Received: by mail-pl1-f195.google.com with SMTP id w10so185551plz.3 for ; Thu, 13 Jun 2019 20:12:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Lt3w372XAiP2T5A3AE8p2nBAFVuCKsgxbI6J0UJSdP8=; b=ARF6EuIOrYoCPrmP9r2NO8meY7IeSPACptb2qiSC3sZkkfZa+YnowaYv5BXUGvg14G C6EkcuPeGocTfXv3Ut4XUoAz74Qf4c/opqiS12y2PM49RuCirdTuRfuexd7PCUPelcWM seJw+buAjTHBvaLeXrb//UjwS3FIMYl+J0ppj3mNRrBFUjiV76zkc/cIeqUYJrMLNiQK Y+o4x5bUuJEAKR+I26nOky6nQqJVT7l01L7J+YkEL/W+Nn8ihvvgxos25+rssBLI28Qz pGM5oTBZTZ7Amfmd7RlRVOjJZNYjzjF09r/6nTXVZKIOiFQM2FgaLmhhA/lsVMGV+uFf 3vEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Lt3w372XAiP2T5A3AE8p2nBAFVuCKsgxbI6J0UJSdP8=; b=bBdbalL0NtOS05raNvpNZa184AJB4qztYb8lnl4hpxvcu4/5+qdh5/zfoqU2OQ92ZD 7XUTRvK/v/qGs5axN+Gcp8SGFkCW0DIt8r3t246wy/Jzlp36g8Is7sNmuJzoTGz2RR39 W9Pi7oDW7SfjAGJciaSFiECoOKNv6HQydBDe//d9Qpmh5/uaUIRY3bP23Ix/U9yllD6x p8xVn05Phnx4jy83HzetLKfKJ5lIbS146q9NS6ArkHuGqRo8AbrtuFnqvUyowuKBNizM 8hqqD4N8eO3UatJUJqs/ZbXSfM07H4v3pUjaaa/PwLW6SaoSwibcW/5DyJtrL3u/F/Q2 QOSg== X-Gm-Message-State: APjAAAWsEI3jS0A62bkJLyR5+XDqqUQkJx1JJhflTsOD6BuTaI/3G13c wUEpZiMd1Tfq6SfioZVbR6oFrw== X-Received: by 2002:a17:902:121:: with SMTP id 30mr86998560plb.314.1560481937851; Thu, 13 Jun 2019 20:12:17 -0700 (PDT) Received: from localhost ([122.172.66.84]) by smtp.gmail.com with ESMTPSA id k20sm1051646pgh.31.2019.06.13.20.12.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 13 Jun 2019 20:12:17 -0700 (PDT) From: Viresh Kumar To: linux-arm-kernel@lists.infradead.org, Julien Thierry Cc: Viresh Kumar , stable@vger.kernel.org, Catalin Marinas , Marc Zyngier , Mark Rutland , Will Deacon , Russell King , Vincent Guittot , mark.brown@arm.com Subject: [PATCH v4.4 09/45] arm64: uaccess: Don't bother eliding access_ok checks in __{get, put}_user Date: Fri, 14 Jun 2019 08:37:52 +0530 Message-Id: X-Mailer: git-send-email 2.21.0.rc0.269.g1a574e7a288b In-Reply-To: References: MIME-Version: 1.0 Sender: stable-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org From: Will Deacon commit 84624087dd7e3b482b7b11c170ebc1f329b3a218 upstream. access_ok isn't an expensive operation once the addr_limit for the current thread has been loaded into the cache. Given that the initial access_ok check preceding a sequence of __{get,put}_user operations will take the brunt of the miss, we can make the __* variants identical to the full-fat versions, which brings with it the benefits of address masking. The likely cost in these sequences will be from toggling PAN/UAO, which we can address later by implementing the *_unsafe versions. Reviewed-by: Robin Murphy Signed-off-by: Will Deacon Signed-off-by: Catalin Marinas [ v4.4: Fixed conflicts around {__get_user|__put_user}_unaligned macros ] Signed-off-by: Viresh Kumar --- arch/arm64/include/asm/uaccess.h | 62 ++++++++++++++++++-------------- 1 file changed, 36 insertions(+), 26 deletions(-) -- 2.21.0.rc0.269.g1a574e7a288b diff --git a/arch/arm64/include/asm/uaccess.h b/arch/arm64/include/asm/uaccess.h index fc11c50af558..a34324436ce1 100644 --- a/arch/arm64/include/asm/uaccess.h +++ b/arch/arm64/include/asm/uaccess.h @@ -200,30 +200,35 @@ do { \ CONFIG_ARM64_PAN)); \ } while (0) -#define __get_user(x, ptr) \ +#define __get_user_check(x, ptr, err) \ ({ \ - int __gu_err = 0; \ - __get_user_err((x), (ptr), __gu_err); \ - __gu_err; \ + __typeof__(*(ptr)) __user *__p = (ptr); \ + might_fault(); \ + if (access_ok(VERIFY_READ, __p, sizeof(*__p))) { \ + __p = uaccess_mask_ptr(__p); \ + __get_user_err((x), __p, (err)); \ + } else { \ + (x) = 0; (err) = -EFAULT; \ + } \ }) #define __get_user_error(x, ptr, err) \ ({ \ - __get_user_err((x), (ptr), (err)); \ + __get_user_check((x), (ptr), (err)); \ (void)0; \ }) -#define __get_user_unaligned __get_user - -#define get_user(x, ptr) \ +#define __get_user(x, ptr) \ ({ \ - __typeof__(*(ptr)) __user *__p = (ptr); \ - might_fault(); \ - access_ok(VERIFY_READ, __p, sizeof(*__p)) ? \ - __p = uaccess_mask_ptr(__p), __get_user((x), __p) : \ - ((x) = 0, -EFAULT); \ + int __gu_err = 0; \ + __get_user_check((x), (ptr), __gu_err); \ + __gu_err; \ }) +#define __get_user_unaligned __get_user + +#define get_user __get_user + #define __put_user_asm(instr, reg, x, addr, err) \ asm volatile( \ "1: " instr " " reg "1, [%2]\n" \ @@ -266,30 +271,35 @@ do { \ CONFIG_ARM64_PAN)); \ } while (0) -#define __put_user(x, ptr) \ +#define __put_user_check(x, ptr, err) \ ({ \ - int __pu_err = 0; \ - __put_user_err((x), (ptr), __pu_err); \ - __pu_err; \ + __typeof__(*(ptr)) __user *__p = (ptr); \ + might_fault(); \ + if (access_ok(VERIFY_WRITE, __p, sizeof(*__p))) { \ + __p = uaccess_mask_ptr(__p); \ + __put_user_err((x), __p, (err)); \ + } else { \ + (err) = -EFAULT; \ + } \ }) #define __put_user_error(x, ptr, err) \ ({ \ - __put_user_err((x), (ptr), (err)); \ + __put_user_check((x), (ptr), (err)); \ (void)0; \ }) -#define __put_user_unaligned __put_user - -#define put_user(x, ptr) \ +#define __put_user(x, ptr) \ ({ \ - __typeof__(*(ptr)) __user *__p = (ptr); \ - might_fault(); \ - access_ok(VERIFY_WRITE, __p, sizeof(*__p)) ? \ - __p = uaccess_mask_ptr(__p), __put_user((x), __p) : \ - -EFAULT; \ + int __pu_err = 0; \ + __put_user_check((x), (ptr), __pu_err); \ + __pu_err; \ }) +#define __put_user_unaligned __put_user + +#define put_user __put_user + extern unsigned long __must_check __copy_from_user(void *to, const void __user *from, unsigned long n); extern unsigned long __must_check __copy_to_user(void __user *to, const void *from, unsigned long n); extern unsigned long __must_check __copy_in_user(void __user *to, const void __user *from, unsigned long n);