From patchwork Thu Nov 19 23:52:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Axtens X-Patchwork-Id: 329917 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09DD6C63798 for ; Thu, 19 Nov 2020 23:53:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A5B3322226 for ; Thu, 19 Nov 2020 23:53:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=axtens.net header.i=@axtens.net header.b="HoCMf06t" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726563AbgKSXwy (ORCPT ); Thu, 19 Nov 2020 18:52:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37990 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726475AbgKSXwy (ORCPT ); Thu, 19 Nov 2020 18:52:54 -0500 Received: from mail-pl1-x643.google.com (mail-pl1-x643.google.com [IPv6:2607:f8b0:4864:20::643]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1B87EC0613CF for ; Thu, 19 Nov 2020 15:52:53 -0800 (PST) Received: by mail-pl1-x643.google.com with SMTP id 5so3868693plj.8 for ; Thu, 19 Nov 2020 15:52:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axtens.net; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fd8+abPY3FXnBnlA7/77pqVtnD1+7oKgxfhh0N16GrU=; b=HoCMf06tYnEolaltkVBGTqUWoANCb9kbx5nAO/St2g6b9n6znznvOsj3E7MpOAtBks M+QJfeI9Pt6HpF68rfrlWfihpdDUm5A5HCF45Qd7GqbHHA1l7Z87j+R5xMqpN3WamAe9 mDQ4G+1I2q7BlmIGK5OzS936cDxjQbXfkKn70= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fd8+abPY3FXnBnlA7/77pqVtnD1+7oKgxfhh0N16GrU=; b=MVoE1Ppmro1XbDSqpZG/8jw2ygcCYmt7jsIBdb7Mog5SC2IRjGp64Lexavh8Fnw7Qp 9jrGVrz6t3M3dJZwoKB1+bJFGMDnkQ6rl6Q+kVzjD0WpOD+DuZWqrlAlVCOqbi6VUfh+ LTEgQKvFi2S3u1bYhjBDFiAlC11iL6R1ha6dYAWe5slj3Po/m/Fn5k+qY+JpaNUG4Wzx 5JO2zh2gO5t7hjG+OggvRcRo8Fc79wqtDB/J5sIWQzmgriEHTx35KcnXqYOp8ZDgqtvG 893AIXTyzSiMCZ/UFm1aMij80q0ugme+D+KmQyeGevxa8Fj88cg8mdGDYbmdAbxIknxi xEqg== X-Gm-Message-State: AOAM530E8yWuylDy51YvJdZZqD4mp1iZzPYFsug5UzXm/dMpwmgoihHV VGGR8mBauK/nr7DrV1td1cYzfcUmukw7jw== X-Google-Smtp-Source: ABdhPJzXooM/icahlKrm9NAdG0IYcuYcG/Qk6Ds7MmugGnCDhVKQsmcxRP5kv8JZ9pInFjDz4E+uSg== X-Received: by 2002:a17:902:6943:b029:d6:bd35:c84b with SMTP id k3-20020a1709026943b02900d6bd35c84bmr11151377plt.53.1605829972516; Thu, 19 Nov 2020 15:52:52 -0800 (PST) Received: from localhost (2001-44b8-1113-6700-4d44-522c-3789-8f33.static.ipv6.internode.on.net. [2001:44b8:1113:6700:4d44:522c:3789:8f33]) by smtp.gmail.com with ESMTPSA id k6sm1110224pfd.169.2020.11.19.15.52.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 Nov 2020 15:52:52 -0800 (PST) From: Daniel Axtens To: stable@vger.kernel.org Cc: dja@axtens.net Subject: [PATCH 4.14 1/8] powerpc/64s: Define MASKABLE_RELON_EXCEPTION_PSERIES_OOL Date: Fri, 20 Nov 2020 10:52:37 +1100 Message-Id: <20201119235244.373127-2-dja@axtens.net> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201119235244.373127-1-dja@axtens.net> References: <20201119235244.373127-1-dja@axtens.net> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org Commit da2bc4644c75 ("powerpc/64s: Add new exception vector macros") adds: +#define __TRAMP_REAL_VIRT_OOL_MASKABLE(name, realvec) \ + TRAMP_REAL_BEGIN(tramp_virt_##name); \ + MASKABLE_RELON_EXCEPTION_PSERIES_OOL(realvec, name##_common); \ However there's no reference there or anywhere else to MASKABLE_RELON_EXCEPTION_PSERIES_OOL and an attempt to use it unsurprisingly doesn't work. Add a definition provided by mpe. Fixes: da2bc4644c75 ("powerpc/64s: Add new exception vector macros") Signed-off-by: Daniel Axtens --- arch/powerpc/include/asm/exception-64s.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/arch/powerpc/include/asm/exception-64s.h b/arch/powerpc/include/asm/exception-64s.h index c3bdd2d8ec90..a99a455bc267 100644 --- a/arch/powerpc/include/asm/exception-64s.h +++ b/arch/powerpc/include/asm/exception-64s.h @@ -645,6 +645,10 @@ END_FTR_SECTION_NESTED(ftr,ftr,943) EXCEPTION_PROLOG_1(PACA_EXGEN, SOFTEN_TEST_HV, vec); \ EXCEPTION_RELON_PROLOG_PSERIES_1(label, EXC_HV) +#define MASKABLE_RELON_EXCEPTION_PSERIES_OOL(vec, label) \ + EXCEPTION_PROLOG_1(PACA_EXGEN, SOFTEN_NOTEST_PR, vec); \ + EXCEPTION_PROLOG_PSERIES_1(label, EXC_STD) + /* * Our exception common code can be passed various "additions" * to specify the behaviour of interrupts, whether to kick the From patchwork Thu Nov 19 23:52:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Axtens X-Patchwork-Id: 329916 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3866C64E69 for ; Thu, 19 Nov 2020 23:53:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6E39F221FC for ; Thu, 19 Nov 2020 23:53:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=axtens.net header.i=@axtens.net header.b="mfszcpAf" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726654AbgKSXxF (ORCPT ); Thu, 19 Nov 2020 18:53:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38026 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726629AbgKSXxE (ORCPT ); Thu, 19 Nov 2020 18:53:04 -0500 Received: from mail-pf1-x443.google.com (mail-pf1-x443.google.com [IPv6:2607:f8b0:4864:20::443]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A2C6DC0613CF for ; Thu, 19 Nov 2020 15:53:04 -0800 (PST) Received: by mail-pf1-x443.google.com with SMTP id b63so6060820pfg.12 for ; Thu, 19 Nov 2020 15:53:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axtens.net; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NAVGBVBne1H/frJQYxiNaO9h8BpIPF5Kruh+CggtOp8=; b=mfszcpAf8CTKT9FfP6AyUnEkj/cMkVHnimbU2hblnFE2wYdr4jc9Pe8nVnQrRA213Y 5fJO3sWD+yJ9y7cibKrNQafOW+pJeIkqj4HMghVQfWs6tsDYOXRvgiJsAxrf7JOpmR/y 1jmMuM2XO+AUWjx5+8MNWgDZXTXFNW9gW2ZOo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NAVGBVBne1H/frJQYxiNaO9h8BpIPF5Kruh+CggtOp8=; b=Bg93kkGnodvU0NkISM/oPsjdgfkbdNiiSjs40zIp2uOGHlDQvrFegG1K6CTZtSCAyA sekf9mkWxBLSGijfIeSJg+9dAVbU9DC15tcnPuwvz7RdRCUgOM3ZvcP7aUPWwdK0vq2T 9D6p7K8j8KF2uNcrApIvcbaa+zh9pezKlvYOWfB0TS0PPzVIU1+n2wgW5avdePNNVwtl bgu1Acx5uVZ/3cnarsD85BH8kb3L3p2OslNF0wHHp6cTb9hARvr+3iPG7dfwY9yKNMU8 Z3gllnzWF0dTg6ZM4gzFEWmXrk2X0GQnLXZAaUpwc3f/+/Gr/ANwSztrT0R2ng1OBSn2 IUoA== X-Gm-Message-State: AOAM530wANgU8V7X5gtZaPtILII0QOMO8lS3wZFzYqS+SFRN0aJnsXJb LlhwR89phl0J1ECEhu1qI5H9QjdFLmWzqQ== X-Google-Smtp-Source: ABdhPJxp3RcRxv+ODZ33OIIg/Mg6JVS0zRDODmR7tCwNa5Y2uq2vTxyvvR4zkAsWPLjES0YlhIuBjQ== X-Received: by 2002:a17:90b:a02:: with SMTP id gg2mr2560350pjb.225.1605829983991; Thu, 19 Nov 2020 15:53:03 -0800 (PST) Received: from localhost (2001-44b8-1113-6700-4d44-522c-3789-8f33.static.ipv6.internode.on.net. [2001:44b8:1113:6700:4d44:522c:3789:8f33]) by smtp.gmail.com with ESMTPSA id r8sm1114782pfq.197.2020.11.19.15.53.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 Nov 2020 15:53:03 -0800 (PST) From: Daniel Axtens To: stable@vger.kernel.org Cc: dja@axtens.net Subject: [PATCH 4.14 4/8] powerpc: Add a framework for user access tracking Date: Fri, 20 Nov 2020 10:52:40 +1100 Message-Id: <20201119235244.373127-5-dja@axtens.net> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201119235244.373127-1-dja@axtens.net> References: <20201119235244.373127-1-dja@axtens.net> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org From: Christophe Leroy Backported from commit de78a9c42a79 ("powerpc: Add a framework for Kernel Userspace Access Protection"). Here we don't try to add the KUAP framework, we just want the helper functions because we want to put uaccess flush helpers in them. In terms of fixes, we don't need commit 1d8f739b07bd ("powerpc/kuap: Fix set direction in allow/prevent_user_access()") as we don't have real KUAP. Likewise as all our allows are noops and all our prevents are just flushes, we don't need commit 9dc086f1e9ef ("powerpc/futex: Fix incorrect user access blocking") The other 2 fixes we do need. The original description is: This patch implements a framework for Kernel Userspace Access Protection. Then subarches will have the possibility to provide their own implementation by providing setup_kuap() and allow/prevent_user_access(). Some platforms will need to know the area accessed and whether it is accessed from read, write or both. Therefore source, destination and size and handed over to the two functions. mpe: Rename to allow/prevent rather than unlock/lock, and add read/write wrappers. Drop the 32-bit code for now until we have an implementation for it. Add kuap to pt_regs for 64-bit as well as 32-bit. Don't split strings, use pr_crit_ratelimited(). Signed-off-by: Christophe Leroy Signed-off-by: Russell Currey Signed-off-by: Michael Ellerman Signed-off-by: Daniel Axtens --- arch/powerpc/include/asm/futex.h | 4 +++ arch/powerpc/include/asm/kup.h | 36 ++++++++++++++++++++++++++ arch/powerpc/include/asm/uaccess.h | 38 ++++++++++++++++++++++------ arch/powerpc/lib/checksum_wrappers.c | 4 +++ 4 files changed, 74 insertions(+), 8 deletions(-) create mode 100644 arch/powerpc/include/asm/kup.h diff --git a/arch/powerpc/include/asm/futex.h b/arch/powerpc/include/asm/futex.h index 3c7d85945229..cbcb97c43d82 100644 --- a/arch/powerpc/include/asm/futex.h +++ b/arch/powerpc/include/asm/futex.h @@ -35,6 +35,7 @@ static inline int arch_futex_atomic_op_inuser(int op, int oparg, int *oval, { int oldval = 0, ret; + allow_write_to_user(uaddr, sizeof(*uaddr)); pagefault_disable(); switch (op) { @@ -61,6 +62,7 @@ static inline int arch_futex_atomic_op_inuser(int op, int oparg, int *oval, *oval = oldval; + prevent_write_to_user(uaddr, sizeof(*uaddr)); return ret; } @@ -74,6 +76,7 @@ futex_atomic_cmpxchg_inatomic(u32 *uval, u32 __user *uaddr, if (!access_ok(VERIFY_WRITE, uaddr, sizeof(u32))) return -EFAULT; + allow_write_to_user(uaddr, sizeof(*uaddr)); __asm__ __volatile__ ( PPC_ATOMIC_ENTRY_BARRIER "1: lwarx %1,0,%3 # futex_atomic_cmpxchg_inatomic\n\ @@ -94,6 +97,7 @@ futex_atomic_cmpxchg_inatomic(u32 *uval, u32 __user *uaddr, : "cc", "memory"); *uval = prev; + prevent_write_to_user(uaddr, sizeof(*uaddr)); return ret; } diff --git a/arch/powerpc/include/asm/kup.h b/arch/powerpc/include/asm/kup.h new file mode 100644 index 000000000000..7895d5eeaf21 --- /dev/null +++ b/arch/powerpc/include/asm/kup.h @@ -0,0 +1,36 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_POWERPC_KUP_H_ +#define _ASM_POWERPC_KUP_H_ + +#ifndef __ASSEMBLY__ + +#include + +static inline void allow_user_access(void __user *to, const void __user *from, + unsigned long size) { } +static inline void prevent_user_access(void __user *to, const void __user *from, + unsigned long size) { } + +static inline void allow_read_from_user(const void __user *from, unsigned long size) +{ + allow_user_access(NULL, from, size); +} + +static inline void allow_write_to_user(void __user *to, unsigned long size) +{ + allow_user_access(to, NULL, size); +} + +static inline void prevent_read_from_user(const void __user *from, unsigned long size) +{ + prevent_user_access(NULL, from, size); +} + +static inline void prevent_write_to_user(void __user *to, unsigned long size) +{ + prevent_user_access(to, NULL, size); +} + +#endif /* !__ASSEMBLY__ */ + +#endif /* _ASM_POWERPC_KUP_H_ */ diff --git a/arch/powerpc/include/asm/uaccess.h b/arch/powerpc/include/asm/uaccess.h index 3865d1d23597..f07528ba5b52 100644 --- a/arch/powerpc/include/asm/uaccess.h +++ b/arch/powerpc/include/asm/uaccess.h @@ -7,6 +7,7 @@ #include #include #include +#include /* * The fs value determines whether argument validity checking should be @@ -132,6 +133,7 @@ extern long __put_user_bad(void); #define __put_user_size(x, ptr, size, retval) \ do { \ retval = 0; \ + allow_write_to_user(ptr, size); \ switch (size) { \ case 1: __put_user_asm(x, ptr, retval, "stb"); break; \ case 2: __put_user_asm(x, ptr, retval, "sth"); break; \ @@ -139,6 +141,7 @@ do { \ case 8: __put_user_asm2(x, ptr, retval); break; \ default: __put_user_bad(); \ } \ + prevent_write_to_user(ptr, size); \ } while (0) #define __put_user_nocheck(x, ptr, size) \ @@ -214,6 +217,7 @@ do { \ __chk_user_ptr(ptr); \ if (size > sizeof(x)) \ (x) = __get_user_bad(); \ + allow_read_from_user(ptr, size); \ switch (size) { \ case 1: __get_user_asm(x, ptr, retval, "lbz"); break; \ case 2: __get_user_asm(x, ptr, retval, "lhz"); break; \ @@ -221,6 +225,7 @@ do { \ case 8: __get_user_asm2(x, ptr, retval); break; \ default: (x) = __get_user_bad(); \ } \ + prevent_read_from_user(ptr, size); \ } while (0) /* @@ -280,16 +285,22 @@ extern unsigned long __copy_tofrom_user(void __user *to, static inline unsigned long raw_copy_in_user(void __user *to, const void __user *from, unsigned long n) { + unsigned long ret; + barrier_nospec(); - return __copy_tofrom_user(to, from, n); + allow_user_access(to, from, n); + ret = __copy_tofrom_user(to, from, n); + prevent_user_access(to, from, n); + return ret; } #endif /* __powerpc64__ */ static inline unsigned long raw_copy_from_user(void *to, const void __user *from, unsigned long n) { + unsigned long ret; if (__builtin_constant_p(n) && (n <= 8)) { - unsigned long ret = 1; + ret = 1; switch (n) { case 1: @@ -314,14 +325,18 @@ static inline unsigned long raw_copy_from_user(void *to, } barrier_nospec(); - return __copy_tofrom_user((__force void __user *)to, from, n); + allow_read_from_user(from, n); + ret = __copy_tofrom_user((__force void __user *)to, from, n); + prevent_read_from_user(from, n); + return ret; } static inline unsigned long raw_copy_to_user(void __user *to, const void *from, unsigned long n) { + unsigned long ret; if (__builtin_constant_p(n) && (n <= 8)) { - unsigned long ret = 1; + ret = 1; switch (n) { case 1: @@ -341,17 +356,24 @@ static inline unsigned long raw_copy_to_user(void __user *to, return 0; } - return __copy_tofrom_user(to, (__force const void __user *)from, n); + allow_write_to_user(to, n); + ret = __copy_tofrom_user(to, (__force const void __user *)from, n); + prevent_write_to_user(to, n); + return ret; } extern unsigned long __clear_user(void __user *addr, unsigned long size); static inline unsigned long clear_user(void __user *addr, unsigned long size) { + unsigned long ret = size; might_fault(); - if (likely(access_ok(VERIFY_WRITE, addr, size))) - return __clear_user(addr, size); - return size; + if (likely(access_ok(VERIFY_WRITE, addr, size))) { + allow_write_to_user(addr, size); + ret = __clear_user(addr, size); + prevent_write_to_user(addr, size); + } + return ret; } extern long strncpy_from_user(char *dst, const char __user *src, long count); diff --git a/arch/powerpc/lib/checksum_wrappers.c b/arch/powerpc/lib/checksum_wrappers.c index a0cb63fb76a1..8d83c39be7e4 100644 --- a/arch/powerpc/lib/checksum_wrappers.c +++ b/arch/powerpc/lib/checksum_wrappers.c @@ -29,6 +29,7 @@ __wsum csum_and_copy_from_user(const void __user *src, void *dst, unsigned int csum; might_sleep(); + allow_read_from_user(src, len); *err_ptr = 0; @@ -60,6 +61,7 @@ __wsum csum_and_copy_from_user(const void __user *src, void *dst, } out: + prevent_read_from_user(src, len); return (__force __wsum)csum; } EXPORT_SYMBOL(csum_and_copy_from_user); @@ -70,6 +72,7 @@ __wsum csum_and_copy_to_user(const void *src, void __user *dst, int len, unsigned int csum; might_sleep(); + allow_write_to_user(dst, len); *err_ptr = 0; @@ -97,6 +100,7 @@ __wsum csum_and_copy_to_user(const void *src, void __user *dst, int len, } out: + prevent_write_to_user(dst, len); return (__force __wsum)csum; } EXPORT_SYMBOL(csum_and_copy_to_user); From patchwork Thu Nov 19 23:52:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Axtens X-Patchwork-Id: 329915 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DA97C6379F for ; Thu, 19 Nov 2020 23:53:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id ACFD622226 for ; Thu, 19 Nov 2020 23:53:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=axtens.net header.i=@axtens.net header.b="Exy2EIqG" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726662AbgKSXxJ (ORCPT ); Thu, 19 Nov 2020 18:53:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38038 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726629AbgKSXxJ (ORCPT ); Thu, 19 Nov 2020 18:53:09 -0500 Received: from mail-pf1-x442.google.com (mail-pf1-x442.google.com [IPv6:2607:f8b0:4864:20::442]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E4BEDC0613CF for ; Thu, 19 Nov 2020 15:53:08 -0800 (PST) Received: by mail-pf1-x442.google.com with SMTP id v5so2035792pff.10 for ; Thu, 19 Nov 2020 15:53:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axtens.net; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1KKTCa+MnnuFlnnES4VgVAqb5G8ahj/pCQWAXIcfb+8=; b=Exy2EIqGXsqFt54gIq3a59LDkjtkHTRtztiuR0c4UDnRPnTMo+Q1627zKzkpbunla5 dza1x6JajDIIqYa8mF8yJVl875PTY0Ws7gHYLXMIa1DsDuZS+GxQ/oKP+q+ailOYboJL qNUTYQbwUAz89t4GkZWGSEgmbd90SsE0L/JGA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1KKTCa+MnnuFlnnES4VgVAqb5G8ahj/pCQWAXIcfb+8=; b=MCAO8f+RW69YLStEiwIB/dObQKafwnK8K3/D3tkP4R+4SdGPRrXDrUFXxX/HOlYOhA 0spssUoqkt7+FVNBL+YAjW///F47I9UChGcoH2MRyh6r2wsUBP6Wry8EMA75F+upt7sl BM4RXPrMokV9uPQ4g9FoZzYcoXpjBJU4nze76cFNbTI8e0y4LIa1N8Zx6ppY5D+BNRAB 1aJmoH/QmjDGMeHJeGdDmF66dFxls48GrWICuL9kZT1MJKlq6QdFv/wbhzzRqXCLVM5n 9WMXduEZRf4UYefx+Wg0m4n/wYFx/CR38MvBGlWb7bNWhcwVcNY3fG3gvGF0RdVI6T9D QGmg== X-Gm-Message-State: AOAM530kSd0EWDdGtzuolDuQoUNNQwaRHguPdOfmOM1oS2AUwvf4o/YZ cm0tXd4hTJkYiphq3BYYxHEqSoBdMZx5CA== X-Google-Smtp-Source: ABdhPJwLXII5ZNUcTu/ncr49ZlI1Q3xytiY3bWiq7wC+pS/aaM8ywuLh3DM1vSYzU2VKEOaCueq3wQ== X-Received: by 2002:a62:293:0:b029:197:96c2:bef6 with SMTP id 141-20020a6202930000b029019796c2bef6mr8131141pfc.62.1605829988235; Thu, 19 Nov 2020 15:53:08 -0800 (PST) Received: from localhost (2001-44b8-1113-6700-4d44-522c-3789-8f33.static.ipv6.internode.on.net. [2001:44b8:1113:6700:4d44:522c:3789:8f33]) by smtp.gmail.com with ESMTPSA id c8sm1000908pjd.13.2020.11.19.15.53.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 Nov 2020 15:53:07 -0800 (PST) From: Daniel Axtens To: stable@vger.kernel.org Cc: dja@axtens.net Subject: [PATCH 4.14 5/8] powerpc: Implement user_access_begin and friends Date: Fri, 20 Nov 2020 10:52:41 +1100 Message-Id: <20201119235244.373127-6-dja@axtens.net> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201119235244.373127-1-dja@axtens.net> References: <20201119235244.373127-1-dja@axtens.net> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org From: Christophe Leroy commit 5cd623333e7cf4e3a334c70529268b65f2a6c2c7 upstream. Today, when a function like strncpy_from_user() is called, the userspace access protection is de-activated and re-activated for every word read. By implementing user_access_begin and friends, the protection is de-activated at the beginning of the copy and re-activated at the end. Implement user_access_begin(), user_access_end() and unsafe_get_user(), unsafe_put_user() and unsafe_copy_to_user() For the time being, we keep user_access_save() and user_access_restore() as nops. Signed-off-by: Christophe Leroy Signed-off-by: Michael Ellerman Link: https://lore.kernel.org/r/36d4fbf9e56a75994aca4ee2214c77b26a5a8d35.1579866752.git.christophe.leroy@c-s.fr Signed-off-by: Daniel Axtens --- arch/powerpc/include/asm/uaccess.h | 76 ++++++++++++++++++++++-------- 1 file changed, 57 insertions(+), 19 deletions(-) diff --git a/arch/powerpc/include/asm/uaccess.h b/arch/powerpc/include/asm/uaccess.h index f07528ba5b52..f5a56f103197 100644 --- a/arch/powerpc/include/asm/uaccess.h +++ b/arch/powerpc/include/asm/uaccess.h @@ -83,9 +83,14 @@ __put_user_check((__typeof__(*(ptr)))(x), (ptr), sizeof(*(ptr))) #define __get_user(x, ptr) \ - __get_user_nocheck((x), (ptr), sizeof(*(ptr))) + __get_user_nocheck((x), (ptr), sizeof(*(ptr)), true) #define __put_user(x, ptr) \ - __put_user_nocheck((__typeof__(*(ptr)))(x), (ptr), sizeof(*(ptr))) + __put_user_nocheck((__typeof__(*(ptr)))(x), (ptr), sizeof(*(ptr)), true) + +#define __get_user_allowed(x, ptr) \ + __get_user_nocheck((x), (ptr), sizeof(*(ptr)), false) +#define __put_user_allowed(x, ptr) \ + __put_user_nocheck((__typeof__(*(ptr)))(x), (ptr), sizeof(*(ptr)), false) #define __get_user_inatomic(x, ptr) \ __get_user_nosleep((x), (ptr), sizeof(*(ptr))) @@ -130,10 +135,9 @@ extern long __put_user_bad(void); : "r" (x), "b" (addr), "i" (-EFAULT), "0" (err)) #endif /* __powerpc64__ */ -#define __put_user_size(x, ptr, size, retval) \ +#define __put_user_size_allowed(x, ptr, size, retval) \ do { \ retval = 0; \ - allow_write_to_user(ptr, size); \ switch (size) { \ case 1: __put_user_asm(x, ptr, retval, "stb"); break; \ case 2: __put_user_asm(x, ptr, retval, "sth"); break; \ @@ -141,17 +145,26 @@ do { \ case 8: __put_user_asm2(x, ptr, retval); break; \ default: __put_user_bad(); \ } \ +} while (0) + +#define __put_user_size(x, ptr, size, retval) \ +do { \ + allow_write_to_user(ptr, size); \ + __put_user_size_allowed(x, ptr, size, retval); \ prevent_write_to_user(ptr, size); \ } while (0) -#define __put_user_nocheck(x, ptr, size) \ +#define __put_user_nocheck(x, ptr, size, do_allow) \ ({ \ long __pu_err; \ __typeof__(*(ptr)) __user *__pu_addr = (ptr); \ if (!is_kernel_addr((unsigned long)__pu_addr)) \ might_fault(); \ __chk_user_ptr(ptr); \ - __put_user_size((x), __pu_addr, (size), __pu_err); \ + if (do_allow) \ + __put_user_size((x), __pu_addr, (size), __pu_err); \ + else \ + __put_user_size_allowed((x), __pu_addr, (size), __pu_err); \ __pu_err; \ }) @@ -211,13 +224,12 @@ extern long __get_user_bad(void); : "b" (addr), "i" (-EFAULT), "0" (err)) #endif /* __powerpc64__ */ -#define __get_user_size(x, ptr, size, retval) \ +#define __get_user_size_allowed(x, ptr, size, retval) \ do { \ retval = 0; \ __chk_user_ptr(ptr); \ if (size > sizeof(x)) \ (x) = __get_user_bad(); \ - allow_read_from_user(ptr, size); \ switch (size) { \ case 1: __get_user_asm(x, ptr, retval, "lbz"); break; \ case 2: __get_user_asm(x, ptr, retval, "lhz"); break; \ @@ -225,6 +237,12 @@ do { \ case 8: __get_user_asm2(x, ptr, retval); break; \ default: (x) = __get_user_bad(); \ } \ +} while (0) + +#define __get_user_size(x, ptr, size, retval) \ +do { \ + allow_read_from_user(ptr, size); \ + __get_user_size_allowed(x, ptr, size, retval); \ prevent_read_from_user(ptr, size); \ } while (0) @@ -235,7 +253,7 @@ do { \ #define __long_type(x) \ __typeof__(__builtin_choose_expr(sizeof(x) > sizeof(0UL), 0ULL, 0UL)) -#define __get_user_nocheck(x, ptr, size) \ +#define __get_user_nocheck(x, ptr, size, do_allow) \ ({ \ long __gu_err; \ __long_type(*(ptr)) __gu_val; \ @@ -244,7 +262,10 @@ do { \ if (!is_kernel_addr((unsigned long)__gu_addr)) \ might_fault(); \ barrier_nospec(); \ - __get_user_size(__gu_val, __gu_addr, (size), __gu_err); \ + if (do_allow) \ + __get_user_size(__gu_val, __gu_addr, (size), __gu_err); \ + else \ + __get_user_size_allowed(__gu_val, __gu_addr, (size), __gu_err); \ (x) = (__typeof__(*(ptr)))__gu_val; \ __gu_err; \ }) @@ -331,33 +352,40 @@ static inline unsigned long raw_copy_from_user(void *to, return ret; } -static inline unsigned long raw_copy_to_user(void __user *to, - const void *from, unsigned long n) +static inline unsigned long +raw_copy_to_user_allowed(void __user *to, const void *from, unsigned long n) { - unsigned long ret; if (__builtin_constant_p(n) && (n <= 8)) { - ret = 1; + unsigned long ret = 1; switch (n) { case 1: - __put_user_size(*(u8 *)from, (u8 __user *)to, 1, ret); + __put_user_size_allowed(*(u8 *)from, (u8 __user *)to, 1, ret); break; case 2: - __put_user_size(*(u16 *)from, (u16 __user *)to, 2, ret); + __put_user_size_allowed(*(u16 *)from, (u16 __user *)to, 2, ret); break; case 4: - __put_user_size(*(u32 *)from, (u32 __user *)to, 4, ret); + __put_user_size_allowed(*(u32 *)from, (u32 __user *)to, 4, ret); break; case 8: - __put_user_size(*(u64 *)from, (u64 __user *)to, 8, ret); + __put_user_size_allowed(*(u64 *)from, (u64 __user *)to, 8, ret); break; } if (ret == 0) return 0; } + return __copy_tofrom_user(to, (__force const void __user *)from, n); +} + +static inline unsigned long +raw_copy_to_user(void __user *to, const void *from, unsigned long n) +{ + unsigned long ret; + allow_write_to_user(to, n); - ret = __copy_tofrom_user(to, (__force const void __user *)from, n); + ret = raw_copy_to_user_allowed(to, from, n); prevent_write_to_user(to, n); return ret; } @@ -379,4 +407,14 @@ static inline unsigned long clear_user(void __user *addr, unsigned long size) extern long strncpy_from_user(char *dst, const char __user *src, long count); extern __must_check long strnlen_user(const char __user *str, long n); + +#define user_access_begin(type, ptr, len) access_ok(type, ptr, len) +#define user_access_end() prevent_user_access(NULL, NULL, ~0ul) + +#define unsafe_op_wrap(op, err) do { if (unlikely(op)) goto err; } while (0) +#define unsafe_get_user(x, p, e) unsafe_op_wrap(__get_user_allowed(x, p), e) +#define unsafe_put_user(x, p, e) unsafe_op_wrap(__put_user_allowed(x, p), e) +#define unsafe_copy_to_user(d, s, l, e) \ + unsafe_op_wrap(raw_copy_to_user_allowed(d, s, l), e) + #endif /* _ARCH_POWERPC_UACCESS_H */ From patchwork Thu Nov 19 23:52:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Axtens X-Patchwork-Id: 329914 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6FA10C63798 for ; Thu, 19 Nov 2020 23:53:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2126D221FC for ; Thu, 19 Nov 2020 23:53:19 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=axtens.net header.i=@axtens.net header.b="kf+KcFEP" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726449AbgKSXxS (ORCPT ); Thu, 19 Nov 2020 18:53:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38062 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726826AbgKSXxS (ORCPT ); Thu, 19 Nov 2020 18:53:18 -0500 Received: from mail-pg1-x544.google.com (mail-pg1-x544.google.com [IPv6:2607:f8b0:4864:20::544]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E251AC0613CF for ; Thu, 19 Nov 2020 15:53:16 -0800 (PST) Received: by mail-pg1-x544.google.com with SMTP id 81so5725268pgf.0 for ; Thu, 19 Nov 2020 15:53:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axtens.net; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BtcdQWJw61GyDMXr+wB/i7S4fSXo+ErJxOeMAtGLl5s=; b=kf+KcFEP7K6KGzPECN9BVsr8xT/VKhX20xYtmX21RTdwuwMI5fjIMnanMb5hQcIk9D ffdc3twN+srXvTFmtAH4i0Pjt7iuPYF1R52sDj9hh0ERwNXaX8Lrs6LxcnKlkDMUcgGi nQziIbwp0HjiLuN8mzRjBSNjPPSR2I8I02Nw8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BtcdQWJw61GyDMXr+wB/i7S4fSXo+ErJxOeMAtGLl5s=; b=qO1zcMPNiB8j54/ArP3NSmuFmcma6Z1pPMDRFxA4m4LxWZCiQZ7T2+cn6q7DxJTz9M 3tbtdJ5mxzjsWWNBH4lC3BvvTHAPE+WNuJGA06CSzPsStyDt/Uq0Rw/srhdFCt6z1hOC 0G/82o2s4Xa6A+hm2GTTuwouUuStwlByydEzv2Rc1uaPAayzkpIyjOhqIsU6hJMtUxAR O92anH8/86orfp1z8nvrlURSibJk7re1j407Xx3R0yCwebrZ1e13krTMtu0VmRTdlGd5 g7I6UautR+hmVBXoDFKS4cD3+2UKD8cO9cdBvGuY2l7SRMgDRXBZU83DOaxdxhgVvmta 4cpw== X-Gm-Message-State: AOAM532UR4w3utvZJomjnqM99pvvKnTD6d4GsAGQ53IbGstW5ulV668q mPF4wVCObmJehOGdFf+D7sUaXN+gmcRJ0g== X-Google-Smtp-Source: ABdhPJzCsKSx0ftthyD3agvczz7avGg95LYHnxjxOtl0u3LA1e/jDX/RJgxALCJMLmvHXioL8K77Tg== X-Received: by 2002:a63:4d07:: with SMTP id a7mr15320328pgb.274.1605829996273; Thu, 19 Nov 2020 15:53:16 -0800 (PST) Received: from localhost (2001-44b8-1113-6700-4d44-522c-3789-8f33.static.ipv6.internode.on.net. [2001:44b8:1113:6700:4d44:522c:3789:8f33]) by smtp.gmail.com with ESMTPSA id 23sm898117pgs.19.2020.11.19.15.53.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 Nov 2020 15:53:15 -0800 (PST) From: Daniel Axtens To: stable@vger.kernel.org Cc: dja@axtens.net Subject: [PATCH 4.14 7/8] powerpc/uaccess: Evaluate macro arguments once, before user access is allowed Date: Fri, 20 Nov 2020 10:52:43 +1100 Message-Id: <20201119235244.373127-8-dja@axtens.net> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201119235244.373127-1-dja@axtens.net> References: <20201119235244.373127-1-dja@axtens.net> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org From: Nicholas Piggin commit d02f6b7dab8228487268298ea1f21081c0b4b3eb upstream. get/put_user() can be called with nontrivial arguments. fs/proc/page.c has a good example: if (put_user(stable_page_flags(ppage), out)) { stable_page_flags() is quite a lot of code, including spin locks in the page allocator. Ensure these arguments are evaluated before user access is allowed. This improves security by reducing code with access to userspace, but it also fixes a PREEMPT bug with KUAP on powerpc/64s: stable_page_flags() is currently called with AMR set to allow writes, it ends up calling spin_unlock(), which can call preempt_schedule. But the task switch code can not be called with AMR set (it relies on interrupts saving the register), so this blows up. It's fine if the code inside allow_user_access() is preemptible, because a timer or IPI will save the AMR, but it's not okay to explicitly cause a reschedule. Fixes: de78a9c42a79 ("powerpc: Add a framework for Kernel Userspace Access Protection") Signed-off-by: Nicholas Piggin Signed-off-by: Michael Ellerman Link: https://lore.kernel.org/r/20200407041245.600651-1-npiggin@gmail.com Signed-off-by: Daniel Axtens --- arch/powerpc/include/asm/uaccess.h | 49 +++++++++++++++++++++--------- 1 file changed, 35 insertions(+), 14 deletions(-) diff --git a/arch/powerpc/include/asm/uaccess.h b/arch/powerpc/include/asm/uaccess.h index 6cba56b911d9..95f060cb7a09 100644 --- a/arch/powerpc/include/asm/uaccess.h +++ b/arch/powerpc/include/asm/uaccess.h @@ -158,13 +158,17 @@ do { \ ({ \ long __pu_err; \ __typeof__(*(ptr)) __user *__pu_addr = (ptr); \ + __typeof__(*(ptr)) __pu_val = (x); \ + __typeof__(size) __pu_size = (size); \ + \ if (!is_kernel_addr((unsigned long)__pu_addr)) \ might_fault(); \ - __chk_user_ptr(ptr); \ + __chk_user_ptr(__pu_addr); \ if (do_allow) \ - __put_user_size((x), __pu_addr, (size), __pu_err); \ + __put_user_size(__pu_val, __pu_addr, __pu_size, __pu_err); \ else \ - __put_user_size_allowed((x), __pu_addr, (size), __pu_err); \ + __put_user_size_allowed(__pu_val, __pu_addr, __pu_size, __pu_err); \ + \ __pu_err; \ }) @@ -172,9 +176,13 @@ do { \ ({ \ long __pu_err = -EFAULT; \ __typeof__(*(ptr)) __user *__pu_addr = (ptr); \ + __typeof__(*(ptr)) __pu_val = (x); \ + __typeof__(size) __pu_size = (size); \ + \ might_fault(); \ - if (access_ok(VERIFY_WRITE, __pu_addr, size)) \ - __put_user_size((x), __pu_addr, (size), __pu_err); \ + if (access_ok(VERIFY_WRITE, __pu_addr, __pu_size)) \ + __put_user_size(__pu_val, __pu_addr, __pu_size, __pu_err); \ + \ __pu_err; \ }) @@ -182,8 +190,12 @@ do { \ ({ \ long __pu_err; \ __typeof__(*(ptr)) __user *__pu_addr = (ptr); \ - __chk_user_ptr(ptr); \ - __put_user_size((x), __pu_addr, (size), __pu_err); \ + __typeof__(*(ptr)) __pu_val = (x); \ + __typeof__(size) __pu_size = (size); \ + \ + __chk_user_ptr(__pu_addr); \ + __put_user_size(__pu_val, __pu_addr, __pu_size, __pu_err); \ + \ __pu_err; \ }) @@ -258,15 +270,18 @@ do { \ long __gu_err; \ __long_type(*(ptr)) __gu_val; \ __typeof__(*(ptr)) __user *__gu_addr = (ptr); \ - __chk_user_ptr(ptr); \ + __typeof__(size) __gu_size = (size); \ + \ + __chk_user_ptr(__gu_addr); \ if (!is_kernel_addr((unsigned long)__gu_addr)) \ might_fault(); \ barrier_nospec(); \ if (do_allow) \ - __get_user_size(__gu_val, __gu_addr, (size), __gu_err); \ + __get_user_size(__gu_val, __gu_addr, __gu_size, __gu_err); \ else \ - __get_user_size_allowed(__gu_val, __gu_addr, (size), __gu_err); \ + __get_user_size_allowed(__gu_val, __gu_addr, __gu_size, __gu_err); \ (x) = (__typeof__(*(ptr)))__gu_val; \ + \ __gu_err; \ }) @@ -275,12 +290,15 @@ do { \ long __gu_err = -EFAULT; \ __long_type(*(ptr)) __gu_val = 0; \ __typeof__(*(ptr)) __user *__gu_addr = (ptr); \ + __typeof__(size) __gu_size = (size); \ + \ might_fault(); \ - if (access_ok(VERIFY_READ, __gu_addr, (size))) { \ + if (access_ok(VERIFY_READ, __gu_addr, __gu_size)) { \ barrier_nospec(); \ - __get_user_size(__gu_val, __gu_addr, (size), __gu_err); \ + __get_user_size(__gu_val, __gu_addr, __gu_size, __gu_err); \ } \ (x) = (__force __typeof__(*(ptr)))__gu_val; \ + \ __gu_err; \ }) @@ -289,10 +307,13 @@ do { \ long __gu_err; \ __long_type(*(ptr)) __gu_val; \ __typeof__(*(ptr)) __user *__gu_addr = (ptr); \ - __chk_user_ptr(ptr); \ + __typeof__(size) __gu_size = (size); \ + \ + __chk_user_ptr(__gu_addr); \ barrier_nospec(); \ - __get_user_size(__gu_val, __gu_addr, (size), __gu_err); \ + __get_user_size(__gu_val, __gu_addr, __gu_size, __gu_err); \ (x) = (__force __typeof__(*(ptr)))__gu_val; \ + \ __gu_err; \ })