From patchwork Tue Sep 7 13:16:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ovidiu Panait X-Patchwork-Id: 508325 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, MSGID_FROM_MTA_HEADER, SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01E63C4332F for ; Tue, 7 Sep 2021 13:19:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D1E176112D for ; Tue, 7 Sep 2021 13:19:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344644AbhIGNUV (ORCPT ); Tue, 7 Sep 2021 09:20:21 -0400 Received: from mx0b-0064b401.pphosted.com ([205.220.178.238]:3984 "EHLO mx0b-0064b401.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234054AbhIGNUA (ORCPT ); Tue, 7 Sep 2021 09:20:00 -0400 Received: from pps.filterd (m0250811.ppops.net [127.0.0.1]) by mx0a-0064b401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 187CCRLB018662; Tue, 7 Sep 2021 13:18:38 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=windriver.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=PPS06212021; bh=/Vmeb52B+9ixGp87OLYDUIKdF9M3v6IdqQ6CpmYcPv0=; b=X+8jnZlHw8+uUNVfa0EyQ7kkw/bO4Nhn4qofTsZ5ERqMaItSo+Ps7f0WR624aBjfkdSi iTTea7NuAMqGYzUismEzsGGDspL2VDc8SOZN7qpXpOpwdgfa4F+Z7MLOSkp6SBb2jV2q ssK0M1YT3YbHpgbJL3HQSdn4V5jXZiyf6BYegrwMUhaCIkFGt0IJBiTrfTRp17am+Nc6 NSKUiraUttLN5+kmfq+vb626MLbbhOvHIVQTYnhD+0RU16uDGukhMUDcP1Ewurbmluyq ouZgFQPhnSnsP7eDp4hr5CGspiCqgxSRGHxq3YcKTOlcJLkvT5J/OuJ8RJbgnpLJsFqN vg== Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2103.outbound.protection.outlook.com [104.47.70.103]) by mx0a-0064b401.pphosted.com with ESMTP id 3awjhygq68-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Sep 2021 13:18:38 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gU8TwJH8wu3iX2hQHkubXx/MmhzSe2+sE5tw2PlDRv9hCS4ABNtg7eT8IJCjN9mjX7oBQicPu3G9RW1JNFAYl+hdcH1cBs4dsL1mgi/ZDOo21bSsPgwiNvH5ABBSo5BqqGostZHrkkexzc6vf7CbLoaA/6GTk4XCP/h2arcauWF+ZGA8vPj/DzL6bqCSD+jLKO8XxgmC+h63GSs7oAieMIj/0Tsx3fqPfrXPJqS00MLAPzEv4t+Lun7cuYkwiQZdZz3D9HXETo2l7N05/YOIYaSQCy3H1FXrLlW9C8nUJTcX+1JXkmTQKh92RKsslnTmVGtm4UeiWtVV/zc36OYP3g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=/Vmeb52B+9ixGp87OLYDUIKdF9M3v6IdqQ6CpmYcPv0=; b=b8dJbu56fl+wp0hshe39hIK/3sHXzvpuiO0YsDYaJG/X0gRD8pObjpkU8A6rZWENCob8EmZy60gwEgbiGgywUR39o7cfYthiR9iVk+us4cPWG0JOw8GXWtpqWGpGM96vic4Mb9UK1CEkiy0KUcdckxLdWkG15JHsbgurcKzSP7vx6rP54VJcm6JPWwvZfC2TcrCi8LqTvrz23yQWj0unrVljaG9Pwic34gWKdTsTfXe4kkCHIN4WyATicsiAxM1slm9i+aTkHErxTks++65+G1jJiAc9CNR2fRxGPg/t36fC1JxX3iXj1NwfqbXrW7eullPih+9XFDiOqKhy6H+OOQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=windriver.com; dmarc=pass action=none header.from=windriver.com; dkim=pass header.d=windriver.com; arc=none Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=windriver.com; Received: from DM4PR11MB5327.namprd11.prod.outlook.com (2603:10b6:5:392::22) by DM6PR11MB3114.namprd11.prod.outlook.com (2603:10b6:5:6d::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4478.19; Tue, 7 Sep 2021 13:18:37 +0000 Received: from DM4PR11MB5327.namprd11.prod.outlook.com ([fe80::952a:bdc6:20c3:94cb]) by DM4PR11MB5327.namprd11.prod.outlook.com ([fe80::952a:bdc6:20c3:94cb%6]) with mapi id 15.20.4478.025; Tue, 7 Sep 2021 13:18:37 +0000 From: Ovidiu Panait To: stable@vger.kernel.org Cc: bpf@vger.kernel.org, daniel@iogearbox.net Subject: [PATCH 5.4 1/4] bpf: Introduce BPF nospec instruction for mitigating Spectre v4 Date: Tue, 7 Sep 2021 16:16:58 +0300 Message-Id: <20210907131701.1910024-2-ovidiu.panait@windriver.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210907131701.1910024-1-ovidiu.panait@windriver.com> References: <20210907131701.1910024-1-ovidiu.panait@windriver.com> X-ClientProxiedBy: VI1PR07CA0181.eurprd07.prod.outlook.com (2603:10a6:802:3e::29) To DM4PR11MB5327.namprd11.prod.outlook.com (2603:10b6:5:392::22) MIME-Version: 1.0 Received: from otp-linux03.wrs.com (46.97.150.20) by VI1PR07CA0181.eurprd07.prod.outlook.com (2603:10a6:802:3e::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4500.6 via Frontend Transport; Tue, 7 Sep 2021 13:18:36 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c5de3aa6-faee-4490-f2ea-08d972020050 X-MS-TrafficTypeDiagnostic: DM6PR11MB3114: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:229; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 1yt0N+QXaKOq9br7PSK4jrJkKqRqAojiRvKpcjhum6uOHQpLjPmDkMnzvyNsn5nT8lRhutLRNmvPvqR/lHgHZuoszzgw2/x3d4yXYm49EsWv2dm+Hpq4ma3SYPgYBuRysvzLl7qI7CUG3lTORo4Ftex6R0ShP4VzfuhJryNWjKmWxzq/R+wSO/QuVzcJ8SLi1EmPA1PCraXpiUgMm4JEKTPPVMw2e/xTkFpxoe4Nk/5GWGz3yXWOrNlUiIn5N513sMDLtZNYsvvLTLuXafAxkh5PbPxcgzWtKC5CLmgA35YpXhmRHZ25u1nMnieXrc5NV4WcmB/AJuFre4D9Aw2NQv9mHnsW20uzyNx1uJu1TWN8+xg1YJxEToDrI1p3QTL2kwotJJ7k0IrrSu4dwfFpnYrfNcS85klMRprH5Rfy1etv6mzdP8zVKvf0P88omvUcP6ojoKH1VIJpW+ES46TRkTew6vtNKEl5vuHxCEzTnMzvQ47XhJrKY1gJ8+CFUC4pUXRn9C1KbUUBu19fV4WgEOUHwaolHM7ei1DxeVVDkfuP/pw6Cn8B8cVWLmnj6pdZWEqkgHChDZs350WZUbBGNC/SS59DEwWp6XUlWZY7Lzl5LuXQBeZZ2hCj7eAzQLcGwNAipurNh6+DCUjlmR+T2blNVvUa4zNp/vHpXbJAhhLiviv8Ew0Cb5VMAl+LUwEfufmiFmf5PVDONnA46iEcfg== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM4PR11MB5327.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(346002)(366004)(136003)(376002)(39850400004)(8936002)(83380400001)(38100700002)(6916009)(186003)(4326008)(36756003)(38350700002)(86362001)(2906002)(956004)(26005)(6666004)(2616005)(8676002)(66476007)(1076003)(66556008)(44832011)(6506007)(6512007)(52116002)(6486002)(30864003)(478600001)(66946007)(316002)(5660300002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: mqeLYXQ3I05hwe1i+EPQRbqzPgLMpNCLz0glDLksi8fdW22jH+SR92YSSVqAGfzx0/SNMX05F+QxLt6zmneKLyps7dnC3I0coPOJzWUjXx/i1sWD5EDAPmy25SzG6WCxvJFZ3ueIacQWCrmijWo9V62IWMQmTGaxE41jODYI/r8BztjFLInoeV/vVx4QPpLtFpAnrGmkVIdKpp4G84ITrzfZzaoYVLNbb+JcqOuIygWAcUQyUmFEKQv0G4Buw6fskOD84XNd+6/tepIbuX2YQQe1ldqF2Asi+DucYWG4cNb8F4b0AQmK5RSbW0YZc0dDov3wfK752x0327wu5XMeMoVJgnhj1RrA5xL5kidZRde04oWfj0wM/LCE7NACO5ODNsxBQWLNGV7z37c9nCDkNDgxn/hQbGpIfQIkAvEZC1M/DhSKSDKVGzNSHN8N6vBoqxsn/plLmR8MWaSc8vrjEnlTxpiC9Ryd7tYveOqQ3U2aHzce8aN5KDOnIbtCbnN7ozJc2xJO481nMX2Rhe9BlpMRqhE/F8yx3m12LlJJXOrtLKPUf7saBLMdMTiVEbwiF/6TN0uNNQCP3mpWhwuCrxHXCiAUG3SLKacNEvyeu87MGZJAZtDnFoDFb9zrU3vEf2E0M8Gp7PvbD0sndWpoFt0icwI1R5ZHu7fVHtXiUmKIzwG8ssMK62HVtyHzrelyEfmSG3RHWqYn+HSPyIuJWCHWP299+pIUcb1XF+Qd61tCOJeynU/8kAnA/h47RvwYr72xENKWwzXLdDJpZj988kSewH0I9KSYu3oY6Fs5aFatrPwaxXCwe+BsJpq6AatTseniKraLMWKT6qaGSA1oVqqM607c4Q9XUwW6fABUYGleobKw/FDgMH6tzkABJs/9vRDDvx0+AxhEDnnwPxPtVBeJGn3kHjTGELHgIcBwos6yLMAPyLtMMOtaZUz9ENjikGSqlzcpKTsZKGhLYhdvI7shR7cDX0EYvyv2ovFAeWuWQ/XCACEV+hHqbhC4MFehFpEktQ26EwaGZct1iPZoAq8ijbT1+sJYDm9mghgzde8L3VXFMfWpptnhWArnEHrwf+cyfgp19BB8nCyEHUy74zPOkwn0piObdInE3xekdTSeJvJro2qAL+HIDkIrv2pJGlbFqR9vzO7/oiDsrDrpFOGYLj+jUPKKashmtDt2NfOQ5MySqjG6XCyuj3c6kje3DpZauV75/JVNdROMdL7WHKCDhd3q9nlrxGTAacjr6ASnM/wng5NW897M4jQzrGxywHEDbQvSL348YVKgooc4a+Md+wqfrbEAUxvnGuO9NPhXQUQ39U8OBih3tM2HYMYM X-OriginatorOrg: windriver.com X-MS-Exchange-CrossTenant-Network-Message-Id: c5de3aa6-faee-4490-f2ea-08d972020050 X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB5327.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Sep 2021 13:18:37.3259 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 8ddb2873-a1ad-4a18-ae4e-4644631433be X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 11SxoCi7jJjoF7VnnW1mY4Tyn5g6FROQfFLRJct87gDC7/SKMTJT+nGPyJsEMnTgEcvP1tINodulKMotdVbKcto1gKGrFQ49vFrUxI7Nv4o= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR11MB3114 X-Proofpoint-GUID: 0juY_PYzOZq9wOdzEiD6blK9t5QKCqie X-Proofpoint-ORIG-GUID: 0juY_PYzOZq9wOdzEiD6blK9t5QKCqie X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1, Aquarius:18.0.790, Hydra:6.0.391, FMLib:17.0.607.475 definitions=2021-09-07_04,2021-09-07_01,2020-04-07_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 malwarescore=0 lowpriorityscore=0 bulkscore=0 suspectscore=0 mlxscore=0 adultscore=0 impostorscore=0 priorityscore=1501 phishscore=0 mlxlogscore=999 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2108310000 definitions=main-2109070088 Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org From: Daniel Borkmann commit f5e81d1117501546b7be050c5fbafa6efd2c722c upstream. In case of JITs, each of the JIT backends compiles the BPF nospec instruction /either/ to a machine instruction which emits a speculation barrier /or/ to /no/ machine instruction in case the underlying architecture is not affected by Speculative Store Bypass or has different mitigations in place already. This covers both x86 and (implicitly) arm64: In case of x86, we use 'lfence' instruction for mitigation. In case of arm64, we rely on the firmware mitigation as controlled via the ssbd kernel parameter. Whenever the mitigation is enabled, it works for all of the kernel code with no need to provide any additional instructions here (hence only comment in arm64 JIT). Other archs can follow as needed. The BPF nospec instruction is specifically targeting Spectre v4 since i) we don't use a serialization barrier for the Spectre v1 case, and ii) mitigation instructions for v1 and v4 might be different on some archs. The BPF nospec is required for a future commit, where the BPF verifier does annotate intermediate BPF programs with speculation barriers. Co-developed-by: Piotr Krysiuk Co-developed-by: Benedict Schlueter Signed-off-by: Daniel Borkmann Signed-off-by: Piotr Krysiuk Signed-off-by: Benedict Schlueter Acked-by: Alexei Starovoitov Signed-off-by: Sasha Levin [OP: - adjusted context for 5.4 - apply riscv changes to /arch/riscv/net/bpf_jit_comp.c] Signed-off-by: Ovidiu Panait --- arch/arm/net/bpf_jit_32.c | 3 +++ arch/arm64/net/bpf_jit_comp.c | 13 +++++++++++++ arch/mips/net/ebpf_jit.c | 3 +++ arch/powerpc/net/bpf_jit_comp64.c | 6 ++++++ arch/riscv/net/bpf_jit_comp.c | 4 ++++ arch/s390/net/bpf_jit_comp.c | 5 +++++ arch/sparc/net/bpf_jit_comp_64.c | 3 +++ arch/x86/net/bpf_jit_comp.c | 7 +++++++ arch/x86/net/bpf_jit_comp32.c | 6 ++++++ include/linux/filter.h | 15 +++++++++++++++ kernel/bpf/core.c | 18 +++++++++++++++++- kernel/bpf/disasm.c | 16 +++++++++------- 12 files changed, 91 insertions(+), 8 deletions(-) diff --git a/arch/arm/net/bpf_jit_32.c b/arch/arm/net/bpf_jit_32.c index 7216653424fd..b51a8c7b0111 100644 --- a/arch/arm/net/bpf_jit_32.c +++ b/arch/arm/net/bpf_jit_32.c @@ -1602,6 +1602,9 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx) rn = arm_bpf_get_reg32(src_lo, tmp2[1], ctx); emit_ldx_r(dst, rn, off, ctx, BPF_SIZE(code)); break; + /* speculation barrier */ + case BPF_ST | BPF_NOSPEC: + break; /* ST: *(size *)(dst + off) = imm */ case BPF_ST | BPF_MEM | BPF_W: case BPF_ST | BPF_MEM | BPF_H: diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index 945e5f690ede..afc7d41347f7 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -701,6 +701,19 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, } break; + /* speculation barrier */ + case BPF_ST | BPF_NOSPEC: + /* + * Nothing required here. + * + * In case of arm64, we rely on the firmware mitigation of + * Speculative Store Bypass as controlled via the ssbd kernel + * parameter. Whenever the mitigation is enabled, it works + * for all of the kernel code with no need to provide any + * additional instructions. + */ + break; + /* ST: *(size *)(dst + off) = imm */ case BPF_ST | BPF_MEM | BPF_W: case BPF_ST | BPF_MEM | BPF_H: diff --git a/arch/mips/net/ebpf_jit.c b/arch/mips/net/ebpf_jit.c index 561154cbcc40..b31b91e57c34 100644 --- a/arch/mips/net/ebpf_jit.c +++ b/arch/mips/net/ebpf_jit.c @@ -1355,6 +1355,9 @@ static int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, } break; + case BPF_ST | BPF_NOSPEC: /* speculation barrier */ + break; + case BPF_ST | BPF_B | BPF_MEM: case BPF_ST | BPF_H | BPF_MEM: case BPF_ST | BPF_W | BPF_MEM: diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c index be3517ef0574..20bfd753bcba 100644 --- a/arch/powerpc/net/bpf_jit_comp64.c +++ b/arch/powerpc/net/bpf_jit_comp64.c @@ -644,6 +644,12 @@ static int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, } break; + /* + * BPF_ST NOSPEC (speculation barrier) + */ + case BPF_ST | BPF_NOSPEC: + break; + /* * BPF_ST(X) */ diff --git a/arch/riscv/net/bpf_jit_comp.c b/arch/riscv/net/bpf_jit_comp.c index e2279fed8f56..0eefe6193253 100644 --- a/arch/riscv/net/bpf_jit_comp.c +++ b/arch/riscv/net/bpf_jit_comp.c @@ -1313,6 +1313,10 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, emit(rv_ld(rd, 0, RV_REG_T1), ctx); break; + /* speculation barrier */ + case BPF_ST | BPF_NOSPEC: + break; + /* ST: *(size *)(dst + off) = imm */ case BPF_ST | BPF_MEM | BPF_B: emit_imm(RV_REG_T1, imm, ctx); diff --git a/arch/s390/net/bpf_jit_comp.c b/arch/s390/net/bpf_jit_comp.c index e160f4650f8e..3e6612d8b921 100644 --- a/arch/s390/net/bpf_jit_comp.c +++ b/arch/s390/net/bpf_jit_comp.c @@ -913,6 +913,11 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp, break; } break; + /* + * BPF_NOSPEC (speculation barrier) + */ + case BPF_ST | BPF_NOSPEC: + break; /* * BPF_ST(X) */ diff --git a/arch/sparc/net/bpf_jit_comp_64.c b/arch/sparc/net/bpf_jit_comp_64.c index 3364e2a00989..fef734473c0f 100644 --- a/arch/sparc/net/bpf_jit_comp_64.c +++ b/arch/sparc/net/bpf_jit_comp_64.c @@ -1287,6 +1287,9 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx) return 1; break; } + /* speculation barrier */ + case BPF_ST | BPF_NOSPEC: + break; /* ST: *(size *)(dst + off) = imm */ case BPF_ST | BPF_MEM | BPF_W: case BPF_ST | BPF_MEM | BPF_H: diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 6e884f17634f..55f62dca28aa 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -728,6 +728,13 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, } break; + /* speculation barrier */ + case BPF_ST | BPF_NOSPEC: + if (boot_cpu_has(X86_FEATURE_XMM2)) + /* Emit 'lfence' */ + EMIT3(0x0F, 0xAE, 0xE8); + break; + /* ST: *(u8*)(dst_reg + off) = imm */ case BPF_ST | BPF_MEM | BPF_B: if (is_ereg(dst_reg)) diff --git a/arch/x86/net/bpf_jit_comp32.c b/arch/x86/net/bpf_jit_comp32.c index 0fcba32077c8..2914f900034e 100644 --- a/arch/x86/net/bpf_jit_comp32.c +++ b/arch/x86/net/bpf_jit_comp32.c @@ -1705,6 +1705,12 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, i++; break; } + /* speculation barrier */ + case BPF_ST | BPF_NOSPEC: + if (boot_cpu_has(X86_FEATURE_XMM2)) + /* Emit 'lfence' */ + EMIT3(0x0F, 0xAE, 0xE8); + break; /* ST: *(u8*)(dst_reg + off) = imm */ case BPF_ST | BPF_MEM | BPF_H: case BPF_ST | BPF_MEM | BPF_B: diff --git a/include/linux/filter.h b/include/linux/filter.h index c53e2fe3c8f7..c4f89340f498 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -68,6 +68,11 @@ struct ctl_table_header; /* unused opcode to mark call to interpreter with arguments */ #define BPF_CALL_ARGS 0xe0 +/* unused opcode to mark speculation barrier for mitigating + * Speculative Store Bypass + */ +#define BPF_NOSPEC 0xc0 + /* As per nm, we expose JITed images as text (code) section for * kallsyms. That way, tools like perf can find it to match * addresses. @@ -368,6 +373,16 @@ static inline bool insn_is_zext(const struct bpf_insn *insn) .off = 0, \ .imm = 0 }) +/* Speculation barrier */ + +#define BPF_ST_NOSPEC() \ + ((struct bpf_insn) { \ + .code = BPF_ST | BPF_NOSPEC, \ + .dst_reg = 0, \ + .src_reg = 0, \ + .off = 0, \ + .imm = 0 }) + /* Internal classic blocks for direct assignment */ #define __BPF_STMT(CODE, K) \ diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 323913ba13b3..d9a3d995bd96 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -31,6 +31,7 @@ #include #include +#include #include /* Registers */ @@ -1310,6 +1311,7 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn, u64 *stack) /* Non-UAPI available opcodes. */ [BPF_JMP | BPF_CALL_ARGS] = &&JMP_CALL_ARGS, [BPF_JMP | BPF_TAIL_CALL] = &&JMP_TAIL_CALL, + [BPF_ST | BPF_NOSPEC] = &&ST_NOSPEC, }; #undef BPF_INSN_3_LBL #undef BPF_INSN_2_LBL @@ -1550,7 +1552,21 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn, u64 *stack) COND_JMP(s, JSGE, >=) COND_JMP(s, JSLE, <=) #undef COND_JMP - /* STX and ST and LDX*/ + /* ST, STX and LDX*/ + ST_NOSPEC: + /* Speculation barrier for mitigating Speculative Store Bypass. + * In case of arm64, we rely on the firmware mitigation as + * controlled via the ssbd kernel parameter. Whenever the + * mitigation is enabled, it works for all of the kernel code + * with no need to provide any additional instructions here. + * In case of x86, we use 'lfence' insn for mitigation. We + * reuse preexisting logic from Spectre v1 mitigation that + * happens to produce the required code on x86 for v4 as well. + */ +#ifdef CONFIG_X86 + barrier_nospec(); +#endif + CONT; #define LDST(SIZEOP, SIZE) \ STX_MEM_##SIZEOP: \ *(SIZE *)(unsigned long) (DST + insn->off) = SRC; \ diff --git a/kernel/bpf/disasm.c b/kernel/bpf/disasm.c index b44d8c447afd..ff1dd7d45b58 100644 --- a/kernel/bpf/disasm.c +++ b/kernel/bpf/disasm.c @@ -162,15 +162,17 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs, else verbose(cbs->private_data, "BUG_%02x\n", insn->code); } else if (class == BPF_ST) { - if (BPF_MODE(insn->code) != BPF_MEM) { + if (BPF_MODE(insn->code) == BPF_MEM) { + verbose(cbs->private_data, "(%02x) *(%s *)(r%d %+d) = %d\n", + insn->code, + bpf_ldst_string[BPF_SIZE(insn->code) >> 3], + insn->dst_reg, + insn->off, insn->imm); + } else if (BPF_MODE(insn->code) == 0xc0 /* BPF_NOSPEC, no UAPI */) { + verbose(cbs->private_data, "(%02x) nospec\n", insn->code); + } else { verbose(cbs->private_data, "BUG_st_%02x\n", insn->code); - return; } - verbose(cbs->private_data, "(%02x) *(%s *)(r%d %+d) = %d\n", - insn->code, - bpf_ldst_string[BPF_SIZE(insn->code) >> 3], - insn->dst_reg, - insn->off, insn->imm); } else if (class == BPF_LDX) { if (BPF_MODE(insn->code) != BPF_MEM) { verbose(cbs->private_data, "BUG_ldx_%02x\n", insn->code); From patchwork Tue Sep 7 13:17:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ovidiu Panait X-Patchwork-Id: 507853 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, MSGID_FROM_MTA_HEADER, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2489C433F5 for ; Tue, 7 Sep 2021 13:19:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C0D8D61103 for ; Tue, 7 Sep 2021 13:19:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344624AbhIGNUD (ORCPT ); Tue, 7 Sep 2021 09:20:03 -0400 Received: from mx0b-0064b401.pphosted.com ([205.220.178.238]:4562 "EHLO mx0b-0064b401.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344309AbhIGNUA (ORCPT ); Tue, 7 Sep 2021 09:20:00 -0400 Received: from pps.filterd (m0250811.ppops.net [127.0.0.1]) by mx0a-0064b401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 187Bn3Fa002074; Tue, 7 Sep 2021 13:18:41 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=windriver.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=PPS06212021; bh=jNYWWnJ9wnoALQ8FQb6eFsu85yvUs7HLqs2EKmzgW44=; b=WQJ5FfQYIR9/Egd4uubqsrOqXU3XHeVSDWaUKYDopgMJGPPjht4KGDkMIE2wvY/dXH+p F6F82UA0jLVClPllsjf6lwyhv4C14G06m4QhWdN4F0BdFvbaNq60L4tjAMNGW9S0mhjj xrthgFdzRimqBltGz8J+FfPi81jZRj9vRx+tH2JEloq385DtCpmoXsHfgTune6Dm3qBV HJC55lVA4ZLlC6AWJY1NfaY+PhB7DqGyMpRIZCUldYuIS10LZ8MF6usiLiDrgS2kykUs RflyENsVr3VuEEzXzwjxfXN1o/LW0SCTKD2jZxU12NKaKRfhvUElHpwv9J0TegzTj60Y xg== Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2105.outbound.protection.outlook.com [104.47.70.105]) by mx0a-0064b401.pphosted.com with ESMTP id 3awjhygq69-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Sep 2021 13:18:40 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=a/HpKgsMPvnAz/pETMVL85/LnTYh6u2KMeYuq83VCVajRUlUX4pH2gzL0CIiVdqjAj466jlwfl2kXsxxFSvlZTUgyt8AluxrQtxxG7KGvhx9If4XdAZ7WEMCLRaOVm67ZgQMcPH2J2kamX1CYf0TAFrnzp4r2xoKLi/ez3yJDwk4b0lDvZLKfLbs7krRQ6dUYXY2PGfkqA/HcMwVKds8l6whc4k5kS39rArKtE+xi8I9VIjp1ZYlh5zmIQ23vX6bYEmmqD2lf3/jCokYkySrfTgKPkEuaUWnq3fB590FdNSke/nvYDtClHroAt904urIyQvwVN1orUPcy2JofH4rcQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=jNYWWnJ9wnoALQ8FQb6eFsu85yvUs7HLqs2EKmzgW44=; b=XBf19zyLtFZNuC5ImWU0z7LHIRHA6GfT/qPbsvcKWxgQnxso3swfjN/wRkkglJArvVJZsUQ5vTgBiyOf/g8rMO7eGy8fbBgaGXJzo92tUcuKvSYFFMyEvzCSPCkKgRfpfg6/EkyhjCGCfSPzcfhdKafQbpL4/W8RbZs6Q6+4DpyNC0TdmBvPS0Zsi0GxgT4agltK4lZq0iZnakolHyvQJx/+ES6KAaamTUwLN+FsGLvigHEHMi1GgjNF4sYKryUhr31sCWIKbCnXUTS5UVFqU/dxZrJZ4wNsBb8+nW6YCUf2mkwUf27a5yvsGoDAG1asl/G/Y2TQwl1Lmu7OmJ2O1A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=windriver.com; dmarc=pass action=none header.from=windriver.com; dkim=pass header.d=windriver.com; arc=none Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=windriver.com; Received: from DM4PR11MB5327.namprd11.prod.outlook.com (2603:10b6:5:392::22) by DM6PR11MB3114.namprd11.prod.outlook.com (2603:10b6:5:6d::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4478.19; Tue, 7 Sep 2021 13:18:40 +0000 Received: from DM4PR11MB5327.namprd11.prod.outlook.com ([fe80::952a:bdc6:20c3:94cb]) by DM4PR11MB5327.namprd11.prod.outlook.com ([fe80::952a:bdc6:20c3:94cb%6]) with mapi id 15.20.4478.025; Tue, 7 Sep 2021 13:18:40 +0000 From: Ovidiu Panait To: stable@vger.kernel.org Cc: bpf@vger.kernel.org, daniel@iogearbox.net Subject: [PATCH 5.4 3/4] bpf: verifier: Allocate idmap scratch in verifier env Date: Tue, 7 Sep 2021 16:17:00 +0300 Message-Id: <20210907131701.1910024-4-ovidiu.panait@windriver.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210907131701.1910024-1-ovidiu.panait@windriver.com> References: <20210907131701.1910024-1-ovidiu.panait@windriver.com> X-ClientProxiedBy: VI1PR07CA0181.eurprd07.prod.outlook.com (2603:10a6:802:3e::29) To DM4PR11MB5327.namprd11.prod.outlook.com (2603:10b6:5:392::22) MIME-Version: 1.0 Received: from otp-linux03.wrs.com (46.97.150.20) by VI1PR07CA0181.eurprd07.prod.outlook.com (2603:10a6:802:3e::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4500.6 via Frontend Transport; Tue, 7 Sep 2021 13:18:38 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 94a45f4c-f1db-498c-1f30-08d9720201f2 X-MS-TrafficTypeDiagnostic: DM6PR11MB3114: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3173; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 0DG34BEKHkgUEZGii/9B2dvsYAe/vAFPf5HVOhFxfRVVHizIoeuuTV0B2XuXMt/M3UwMNysgZAqPIkTlj6STlVCxmIYVJY/xsmURSUhaMVIdZlbmyObQJECB9eCIociNDYbFQ5C/+vC6kKRHhFWgtDOdT5nWaxRysO50F1UlFzfELCBHn0Lvb/0oihjnjPUPSLN64gcTjmd6nBCppsYVfh3rtAypCZ59htzDxmaB6nM+xUWIIac7rtD6zxQxqTf/fKR/YjcFNsmcWZJTWLOiytwCOOGk6klITQsT3sw6TTMkJEnGSWXB67LIhe29dJ5jg908db5HJyyS1e+oejp/kJz1OocCU2SAJQ6aNpsMHWZTFW68LfbLsVhk7g0Lhy6j/6mO+tvEu8//N0ESEWho0iVXCGpsw7+tvhg2sz8osXcytYLFRZYIgelBy1OMkAN4E7Vg+XrWRSJe1uwVUoJruAZv3vu0R6Ts0ypaOz+/YvafAgTKN1jyQ5pFrcFvpv8gvupqfFbQNd/fwz9Ex/UVCw1XiHVnjte8VE49lN6pWePq4ikQR8KOWJeGw1vmE12D76TSb4294NO3fzg1jbCtVnJgwOWhUO34KpaaqP85cWCKbRUvhGW6b7QHUilAuBHdHoFqPSn+Klt/noYTkEEx1gF8EkYzOPZ+fps0xSU4CwLlv5KdnrbTRnsRTOduTRodmcYTSHCDdbEuaPKzEArrtJeXZOhUo1BYFrGGuF8yCnFKqGcbeAjmGi9WVFx4GDUj9pfxisTkUbfDxxCtvo8R7lNpstEKtX01rEkuyIriAnA= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM4PR11MB5327.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(346002)(366004)(136003)(376002)(39850400004)(8936002)(83380400001)(38100700002)(6916009)(186003)(4326008)(36756003)(38350700002)(86362001)(2906002)(956004)(26005)(6666004)(2616005)(8676002)(66476007)(1076003)(66556008)(44832011)(6506007)(6512007)(52116002)(966005)(6486002)(478600001)(66946007)(316002)(5660300002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: ui76tonfjPcbtMTm71YmUn3MKSBEKiekqxQ2tdBFW0QC0r4d4BZH7hDCwm8Ivp46yfuhLq4dIxHy79oVQh0SW4ceVqPFw5JvETqO1KEchEHMMQQuMaLBDuPpT2Ni4s6rhHgwjEUxIwNvX0jmvnFqtYN//9rvkWd5pvV/9rvQHO7wpSQF3WbpLJlHWyssKW1YS6tsoM54+M0aO6DtprbQsxPeDH/ji2lp10gWkIXx89HvD/aErV0svaDnj3CKVCqGaRV8smb7O0XQ6WsUQWSDN+WViXHgdgevo6ef0HFrxumpXBuXqCUI8YZna35n3lpjG4q16hxR2NtIQP7EkxzOglw14y/bDOOVGuCB+sRcG5wVqaEyB6fOy6/p+ZFJPg09cb7JJH8LNa/vsd0xiPqlzqH3sjgzbiDGl+u+m59Dq4D4FCGh+3oph0ouejTxVrBJeMeVkmNrQLx6KWH+aNCiM+b1xWDxbjjdcMkYrKZMexRRPtCrZDyUI4k7ck93zOyZ+SeHGl1whIDi6v1iwkjDQzOm5Y3KfmA+Tx2HQsObV6XDFUuIXXiXgF+PHPyyMBAbGDhYq3Ul7u4KIPAtihlH7ZLFXWz8RQ2ixOuc+OrwDi4N8oVrfQVB/shsQbdcfhARuACmcYilbCdPo1F5D62hlCLALxIVTJOYrNp+n1jnciEhyKzBjz4liNEznWwAKkxVBY9hHQpcnsSS42B3DAv2ccWY8j/4HY/AomdEszGB85/hp7/1RMSiDGHGBIs6s+Etctv9gFn/B7IQeQZvfyfaH14iSW68Tn1WfSU5UD3bbbdCMKVMie85CWP4oplGwU63YvpDd3uc4/4Oeymsi0mdWRtozgMtenWROUIxaiCSF9cteQ1OxkCiSiueSx0H0H4ZxkGETlteSXzwFFvVO5bC5Ox45TF/fl21yWWToKltMWzgzSYKsxPJ+ZZaARKKNFxPjMwvtlBfEjjtQCE/50+uqol97EXJvYyhltWg7vuawUd3mSsy3/8MFlEGEX9w7lt2m0TSzCjkw4kn0ILhlu7cXPZMMjGeDXvj+aiWdlGu/21kq9KieTt6Rn2DOSARvb0CeqOBKkJXuNrFS9otsS4aRiIxN/xlxpYgY10tqav6BjRBH+IlRg3OrZMu33k+dlcn/0fK7S5oDlBZEXGU8DswNwjdYC2HQ35PmXv+qovcQxOnmDm2hksJrcMDzcT39EBIDY1iwupzjbidfC32HnCxpOpotNNBopjNKHwuTbIFnabhMIw9XgwdMAjTUbDuGLkJAnY8W20+0iJHlnihBoKu84SI2kEKg00yK3JjcEu381ANFSjgR1qQxZh2+m6f01Yn X-OriginatorOrg: windriver.com X-MS-Exchange-CrossTenant-Network-Message-Id: 94a45f4c-f1db-498c-1f30-08d9720201f2 X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB5327.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Sep 2021 13:18:39.9315 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 8ddb2873-a1ad-4a18-ae4e-4644631433be X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: KB0PbABujV8SX8NJ6ew/YJBEokTqvk5tnfbjF0ZARIeXMqa3f+iEKjDooZrbom+ucHU7VdYKRDtcqBT040rIHs3CVnsYUwaCw+iypi3UtYo= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR11MB3114 X-Proofpoint-GUID: hOnyf-4YR8z0dR_NoCm9eD4-3SeDVDYM X-Proofpoint-ORIG-GUID: hOnyf-4YR8z0dR_NoCm9eD4-3SeDVDYM X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1, Aquarius:18.0.790, Hydra:6.0.391, FMLib:17.0.607.475 definitions=2021-09-07_04,2021-09-07_01,2020-04-07_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 malwarescore=0 lowpriorityscore=0 bulkscore=0 suspectscore=0 mlxscore=0 adultscore=0 impostorscore=0 priorityscore=1501 phishscore=0 mlxlogscore=825 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2108310000 definitions=main-2109070088 Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org From: Lorenz Bauer commit c9e73e3d2b1eb1ea7ff068e05007eec3bd8ef1c9 upstream. func_states_equal makes a very short lived allocation for idmap, probably because it's too large to fit on the stack. However the function is called quite often, leading to a lot of alloc / free churn. Replace the temporary allocation with dedicated scratch space in struct bpf_verifier_env. Signed-off-by: Lorenz Bauer Signed-off-by: Alexei Starovoitov Acked-by: Edward Cree Link: https://lore.kernel.org/bpf/20210429134656.122225-4-lmb@cloudflare.com Signed-off-by: Greg Kroah-Hartman [OP: adjusted context for 5.4] Signed-off-by: Ovidiu Panait --- include/linux/bpf_verifier.h | 8 +++++++ kernel/bpf/verifier.c | 46 ++++++++++++------------------------ 2 files changed, 23 insertions(+), 31 deletions(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 4292e8e42c12..d5a7798a4cbf 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -194,6 +194,13 @@ struct bpf_idx_pair { u32 idx; }; +struct bpf_id_pair { + u32 old; + u32 cur; +}; + +/* Maximum number of register states that can exist at once */ +#define BPF_ID_MAP_SIZE (MAX_BPF_REG + MAX_BPF_STACK / BPF_REG_SIZE) #define MAX_CALL_FRAMES 8 struct bpf_verifier_state { /* call stack tracking */ @@ -370,6 +377,7 @@ struct bpf_verifier_env { const struct bpf_line_info *prev_linfo; struct bpf_verifier_log log; struct bpf_subprog_info subprog_info[BPF_MAX_SUBPROGS + 1]; + struct bpf_id_pair idmap_scratch[BPF_ID_MAP_SIZE]; struct { int *insn_state; int *insn_stack; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index ec06c4f0402e..1ec44872c8e4 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -6976,13 +6976,6 @@ static bool range_within(struct bpf_reg_state *old, old->smax_value >= cur->smax_value; } -/* Maximum number of register states that can exist at once */ -#define ID_MAP_SIZE (MAX_BPF_REG + MAX_BPF_STACK / BPF_REG_SIZE) -struct idpair { - u32 old; - u32 cur; -}; - /* If in the old state two registers had the same id, then they need to have * the same id in the new state as well. But that id could be different from * the old state, so we need to track the mapping from old to new ids. @@ -6993,11 +6986,11 @@ struct idpair { * So we look through our idmap to see if this old id has been seen before. If * so, we require the new id to match; otherwise, we add the id pair to the map. */ -static bool check_ids(u32 old_id, u32 cur_id, struct idpair *idmap) +static bool check_ids(u32 old_id, u32 cur_id, struct bpf_id_pair *idmap) { unsigned int i; - for (i = 0; i < ID_MAP_SIZE; i++) { + for (i = 0; i < BPF_ID_MAP_SIZE; i++) { if (!idmap[i].old) { /* Reached an empty slot; haven't seen this id before */ idmap[i].old = old_id; @@ -7110,7 +7103,7 @@ static void clean_live_states(struct bpf_verifier_env *env, int insn, /* Returns true if (rold safe implies rcur safe) */ static bool regsafe(struct bpf_reg_state *rold, struct bpf_reg_state *rcur, - struct idpair *idmap) + struct bpf_id_pair *idmap) { bool equal; @@ -7227,7 +7220,7 @@ static bool regsafe(struct bpf_reg_state *rold, struct bpf_reg_state *rcur, static bool stacksafe(struct bpf_func_state *old, struct bpf_func_state *cur, - struct idpair *idmap) + struct bpf_id_pair *idmap) { int i, spi; @@ -7324,32 +7317,23 @@ static bool refsafe(struct bpf_func_state *old, struct bpf_func_state *cur) * whereas register type in current state is meaningful, it means that * the current state will reach 'bpf_exit' instruction safely */ -static bool func_states_equal(struct bpf_func_state *old, +static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_state *old, struct bpf_func_state *cur) { - struct idpair *idmap; - bool ret = false; int i; - idmap = kcalloc(ID_MAP_SIZE, sizeof(struct idpair), GFP_KERNEL); - /* If we failed to allocate the idmap, just say it's not safe */ - if (!idmap) - return false; - - for (i = 0; i < MAX_BPF_REG; i++) { - if (!regsafe(&old->regs[i], &cur->regs[i], idmap)) - goto out_free; - } + memset(env->idmap_scratch, 0, sizeof(env->idmap_scratch)); + for (i = 0; i < MAX_BPF_REG; i++) + if (!regsafe(&old->regs[i], &cur->regs[i], env->idmap_scratch)) + return false; - if (!stacksafe(old, cur, idmap)) - goto out_free; + if (!stacksafe(old, cur, env->idmap_scratch)) + return false; if (!refsafe(old, cur)) - goto out_free; - ret = true; -out_free: - kfree(idmap); - return ret; + return false; + + return true; } static bool states_equal(struct bpf_verifier_env *env, @@ -7376,7 +7360,7 @@ static bool states_equal(struct bpf_verifier_env *env, for (i = 0; i <= old->curframe; i++) { if (old->frame[i]->callsite != cur->frame[i]->callsite) return false; - if (!func_states_equal(old->frame[i], cur->frame[i])) + if (!func_states_equal(env, old->frame[i], cur->frame[i])) return false; } return true; From patchwork Tue Sep 7 13:17:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ovidiu Panait X-Patchwork-Id: 508324 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER, INCLUDES_PATCH, MAILING_LIST_MULTI, MSGID_FROM_MTA_HEADER, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5990C433FE for ; Tue, 7 Sep 2021 13:19:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B022A610F8 for ; Tue, 7 Sep 2021 13:19:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344691AbhIGNU1 (ORCPT ); Tue, 7 Sep 2021 09:20:27 -0400 Received: from mx0a-0064b401.pphosted.com ([205.220.166.238]:4624 "EHLO mx0a-0064b401.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344600AbhIGNUD (ORCPT ); Tue, 7 Sep 2021 09:20:03 -0400 Received: from pps.filterd (m0250809.ppops.net [127.0.0.1]) by mx0a-0064b401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 187CwixK009946; Tue, 7 Sep 2021 06:18:43 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=windriver.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=PPS06212021; bh=PktO9w0nxKKES3p2webYAane285B8fZmgK9l+7T459M=; b=dD/a2k1Z3umc/bAVT9//7sCx9+X17DEPdKIE26A395766Con0qEKbA2uynfnZy/t316C pyPBAlu80qPiVrWH6OJpc9xxDLKA9njROC+zs3w1sIAP+3Q0UkFWkS9ucDY3VB6x2z0n RBE19h7yssSgXBv/oCCXYnU+yJDzdN+F6egoSwGPlGNlLqryLSHSCMUM9sbLq8Gv8D5y YvUWPmwG+k15+Io/li094enQ/sacG0Ll/mWefgA6PB5MWmnwnkcxi9g5r5j/H4mcHafE iSLG5eK7solXwWDS9jJvd9/BOgyJm8HjvuCs9MZl3S69tcrwvntXCyyFdTJpFGm4jPBT Lw== Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2103.outbound.protection.outlook.com [104.47.70.103]) by mx0a-0064b401.pphosted.com with ESMTP id 3ax4nj856n-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Sep 2021 06:18:42 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=VVbGFCCul4rBlH9YY68+zh9vzk1yFCBPR61pSar5rfhqtBZ/V4GmWdL9HDzqew67Ii2geJ/CRJWW3cms4ZSVwMrRIaQts0wauR/6N4dHWWfALO1wAZ4470pHKaRQQO73Kdk8dN++QGW7XH2cTRr6Lxg9xMnIYAXHMOio4msyVU8jpBqpOfeQ8xPG+h4j/uhi5a9uu42KjNOf2NgntUAXfnih3YDTiWkw8jalwZ2hxonEZejZd7OwivU4TFkTSzXrK8z27n+yV3VVen/grzfmUEDJIwg/nXsK+k4WwqnigJhqsiAyYSHuLdK/NMCzrEMok/fWA1qDXKFOaNQUiT3TnQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=PktO9w0nxKKES3p2webYAane285B8fZmgK9l+7T459M=; b=k+uAbLu2ufJPxTPr6JjqrbTcM6r4FPM1VkHjihOLUV6JpXn3Xz7cDXINstz/j7WDqJUF5Mc+ipKeaDBF2FYNDrQuGDZQPpn/9S95yV37+H0GMjW2Rr91rm2WRvOMBsaUOldyfOt5L7KzMaw6r3k57tLhWWvUqXFkxQuE1+UoEoTOsociLzkbS4cjt+s3JSgospmuWPgKf8EnzQYDe478qDYUMw/CWbbeLSYYQEBgrM9je0MIRKnb+G/AmpXbj5/dn6BdXjRtKLS3S9a+he/vko1mzzix2crOtrKV8WqUH2Zz4SEZb5gTrapLkeGNI2dm4ZQqrj0r/etH/PjYk1E48w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=windriver.com; dmarc=pass action=none header.from=windriver.com; dkim=pass header.d=windriver.com; arc=none Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=windriver.com; Received: from DM4PR11MB5327.namprd11.prod.outlook.com (2603:10b6:5:392::22) by DM6PR11MB3114.namprd11.prod.outlook.com (2603:10b6:5:6d::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4478.19; Tue, 7 Sep 2021 13:18:41 +0000 Received: from DM4PR11MB5327.namprd11.prod.outlook.com ([fe80::952a:bdc6:20c3:94cb]) by DM4PR11MB5327.namprd11.prod.outlook.com ([fe80::952a:bdc6:20c3:94cb%6]) with mapi id 15.20.4478.025; Tue, 7 Sep 2021 13:18:41 +0000 From: Ovidiu Panait To: stable@vger.kernel.org Cc: bpf@vger.kernel.org, daniel@iogearbox.net Subject: [PATCH 5.4 4/4] bpf: Fix pointer arithmetic mask tightening under state pruning Date: Tue, 7 Sep 2021 16:17:01 +0300 Message-Id: <20210907131701.1910024-5-ovidiu.panait@windriver.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210907131701.1910024-1-ovidiu.panait@windriver.com> References: <20210907131701.1910024-1-ovidiu.panait@windriver.com> X-ClientProxiedBy: VI1PR07CA0181.eurprd07.prod.outlook.com (2603:10a6:802:3e::29) To DM4PR11MB5327.namprd11.prod.outlook.com (2603:10b6:5:392::22) MIME-Version: 1.0 Received: from otp-linux03.wrs.com (46.97.150.20) by VI1PR07CA0181.eurprd07.prod.outlook.com (2603:10a6:802:3e::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4500.6 via Frontend Transport; Tue, 7 Sep 2021 13:18:40 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 3764d177-65b6-4498-02aa-08d9720202b7 X-MS-TrafficTypeDiagnostic: DM6PR11MB3114: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8882; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 7rOBrr3e4iWLIbvfutvW68FDfZSZ1OJ3HM/rpJqcTvv20hFSFUn6/3FYxtbgI4504d0+bN5/3iAbZ6t/kwdVJ1Zome2vYRPjyt4St//HQtv8aptcYC7MFeEPgxDh7uQ/mHbwqxChr/qZqPng3FyFuHevID2A6KTH4hJ4FL6rcUr1oi1onyraRIoVCGUUaY2euLOPDjkPop8ghqVheeASxam+p37n8QL9U3jYFqmJpJU+K++57an8zHuVygVVyRp/gAIe2Qc92yDtiIHubxB3Ovf99Dv6kKaTawVezkrYL/UMBQjVUw+EeJfMC6kaoC6DNqELwqHiO6uIJno7cnrPzt95BTzDGHgsLRCn0rtEplc2RaR8T0cZn9xurCOtFy40u/UtdK0yn0sT/KjYLzc9UNgC1J3JwdXzIqVLY7Z3KZbcqcy0o/pKO37YVOrXBv6CFg0pQjue568vCs5LxS6Rb6auZM+ieWMRezqLt4lPueJmXnW7i4ZnRGK5u2EUWh56yqzIvoIbUq7LiVWx2k4qwUhgNqwRXWpzoQzGBL86oEcndb8TGo045cMFTnfU3QH4GLWZyJM5+wC2pGJiaCMxcrg0OG/60Xj3DFfik6wD1QB7eer9boZBntkXApjuEzfRiWMsAKXlkrLw9xKllD8AFvKGU1KBgPN819mVn7rJI5Cxus9TK9O4nloGADc9Fs0O X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM4PR11MB5327.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(346002)(366004)(136003)(376002)(39850400004)(8936002)(83380400001)(38100700002)(6916009)(186003)(4326008)(36756003)(38350700002)(86362001)(2906002)(956004)(26005)(6666004)(2616005)(8676002)(66476007)(1076003)(66556008)(44832011)(6506007)(6512007)(52116002)(6486002)(478600001)(66946007)(316002)(5660300002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: DIWO9WdCzsmYxupXjBWKaMNSLaYZ9Be5zPN0uiDI3I7aCbEAzwqHvuF1gtU54z+GrxBbmHtoNuLRhFzANMlbdBz2SIQXnP3/wuhuFS1zIIVLWgBYWDcvTIHfeOaorFr7B/oDjC0I4elS3ioft8ZOj4APr78vztYn27Ofw/7IUNG407rvvBxyO/bFBhxdXfDym3VrhAt7ctT2mzQMF5BAkoJpZULkNE0Vcmn3JyWO78JiPX+Yg5UutfjDLXOP3vBInwTY6GU40TnhbvZl5ZHmws14FEyQcDVb4vGkT7cPx6r3VpoaULj3VcRo4gjgYEn3JZcFiNa6rLag/6oY8NFgSypXWib27tGBRKAidPTi3cccst2Dh1eIdaxuDoPjIdCkGhaTUpIFHGg9JydxFIQdn5ljDg+fz1e1yI0RJE+K4rihr1pX4qz8Df/X3WxEbUwhp3RqnjdKpB0J7ngD/ExwtQF2u7BUYow7X22QCm4OIeS7WvfABXu7ADIhTf8Jz64GMBEqMhUjqGsybsBeMKXXPSbzQC6fQWk0CIU5UPM8/Zvv+spneVpJhLfU2g1a011pL70K8B6V9QMu//Pwe9sYgFaUsxmX/ZU/ZL/jOQw25rjqlm+UnJRD2ewXGCB8W6+lYq81oM7G9beONe0SrTp5SktUGaHbCuEoMWDSwV8bhEAfZNPwxRIGFiRSWdu7+crRC6qAOIC0rHaMvsaUk9Ir7ITKzb+mDqPONwIgoTxwhq9kh56Swg+zqBuz3KIOUP8Brpb86aalxbYwj0K3wKo1WGdOyOzkMPlROWX5hwycW2fA2YEiUU+JGsb484ecRGzPFDItbfN0Gals2hlf0mi0pj08PrhdKSq6z1fbbMoU+FzIMmTH3oi9vlUMT4Rv55q0dKZStcbUG/Bbr6T8sHr5tmZDSsVIUSTlfSGsS1pmkRMX2Qn68dZZRvL2njTcQGn86klTPJ93FX7suC588Lbtv0X9ww9hn+AnRDdW9ShULlNZOeltS9NnYuf9gVLF+Z/TCr12gjisfDaiYGKJlL95Hdtsy8QCWbVgz5WtTI7Q/zjODQ2bEH9ncE7Pl2v4Ro6eXQKWAh/eOQMhP3gpx6c+zZ2wafdghdxTScuZf1j2rW/GIwXJvBVrabwbCSKN8aBJgRsqsK8Za5Jcue0ou9y7vQItdVH07nCZYvVTVKcCpq2tscyn+n3HC6OlZpVa1GkQkypsnwi6zNbrCwtYcXkeYgrMSU/AmROnPA03qX69J13/etEa/efQgR3EIdsznEce4WnzkjuvjnYdOwCY2nkE+67pqqZ2icH814oaPRNWzyGAUElIYV4imTJ/OZKHy94e X-OriginatorOrg: windriver.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3764d177-65b6-4498-02aa-08d9720202b7 X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB5327.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Sep 2021 13:18:41.2059 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 8ddb2873-a1ad-4a18-ae4e-4644631433be X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Hx8OWv6oe+2gYEV7qiohwZ+ZEQy0cZz/h6l0AsJfZGpqsaCJQ7U9A/t9fWyifNpqaTF8MWC95hVVpe9DR+EgKgLDEyvU6Gx+8Z0fTLNgUxc= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR11MB3114 X-Proofpoint-GUID: UJQGwKvlKlcirG8UBc_KC1jAW8xQ3rzS X-Proofpoint-ORIG-GUID: UJQGwKvlKlcirG8UBc_KC1jAW8xQ3rzS X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1, Aquarius:18.0.790, Hydra:6.0.391, FMLib:17.0.607.475 definitions=2021-09-07_04,2021-09-07_01,2020-04-07_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 priorityscore=1501 adultscore=0 mlxlogscore=797 phishscore=0 suspectscore=0 impostorscore=0 bulkscore=0 spamscore=0 lowpriorityscore=0 mlxscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2108310000 definitions=main-2109070088 Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org From: Daniel Borkmann commit e042aa532c84d18ff13291d00620502ce7a38dda upstream. In 7fedb63a8307 ("bpf: Tighten speculative pointer arithmetic mask") we narrowed the offset mask for unprivileged pointer arithmetic in order to mitigate a corner case where in the speculative domain it is possible to advance, for example, the map value pointer by up to value_size-1 out-of- bounds in order to leak kernel memory via side-channel to user space. The verifier's state pruning for scalars leaves one corner case open where in the first verification path R_x holds an unknown scalar with an aux->alu_limit of e.g. 7, and in a second verification path that same register R_x, here denoted as R_x', holds an unknown scalar which has tighter bounds and would thus satisfy range_within(R_x, R_x') as well as tnum_in(R_x, R_x') for state pruning, yielding an aux->alu_limit of 3: Given the second path fits the register constraints for pruning, the final generated mask from aux->alu_limit will remain at 7. While technically not wrong for the non-speculative domain, it would however be possible to craft similar cases where the mask would be too wide as in 7fedb63a8307. One way to fix it is to detect the presence of unknown scalar map pointer arithmetic and force a deeper search on unknown scalars to ensure that we do not run into a masking mismatch. Signed-off-by: Daniel Borkmann Acked-by: Alexei Starovoitov Signed-off-by: Greg Kroah-Hartman [OP: adjusted context in include/linux/bpf_verifier.h for 5.4] Signed-off-by: Ovidiu Panait --- include/linux/bpf_verifier.h | 1 + kernel/bpf/verifier.c | 27 +++++++++++++++++---------- 2 files changed, 18 insertions(+), 10 deletions(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index d5a7798a4cbf..ee10a9f06b97 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -371,6 +371,7 @@ struct bpf_verifier_env { struct bpf_map *used_maps[MAX_USED_MAPS]; /* array of map's used by eBPF program */ u32 used_map_cnt; /* number of used maps */ u32 id_gen; /* used to generate unique reg IDs */ + bool explore_alu_limits; bool allow_ptr_leaks; bool seen_direct_write; struct bpf_insn_aux_data *insn_aux_data; /* array of per-insn state */ diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 1ec44872c8e4..66b27d9f2569 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -4449,6 +4449,12 @@ static int sanitize_ptr_alu(struct bpf_verifier_env *env, alu_state |= off_is_imm ? BPF_ALU_IMMEDIATE : 0; alu_state |= ptr_is_dst_reg ? BPF_ALU_SANITIZE_SRC : BPF_ALU_SANITIZE_DST; + + /* Limit pruning on unknown scalars to enable deep search for + * potential masking differences from other program paths. + */ + if (!off_is_imm) + env->explore_alu_limits = true; } err = update_alu_sanitation_state(aux, alu_state, alu_limit); @@ -7102,8 +7108,8 @@ static void clean_live_states(struct bpf_verifier_env *env, int insn, } /* Returns true if (rold safe implies rcur safe) */ -static bool regsafe(struct bpf_reg_state *rold, struct bpf_reg_state *rcur, - struct bpf_id_pair *idmap) +static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold, + struct bpf_reg_state *rcur, struct bpf_id_pair *idmap) { bool equal; @@ -7129,6 +7135,8 @@ static bool regsafe(struct bpf_reg_state *rold, struct bpf_reg_state *rcur, return false; switch (rold->type) { case SCALAR_VALUE: + if (env->explore_alu_limits) + return false; if (rcur->type == SCALAR_VALUE) { if (!rold->precise && !rcur->precise) return true; @@ -7218,9 +7226,8 @@ static bool regsafe(struct bpf_reg_state *rold, struct bpf_reg_state *rcur, return false; } -static bool stacksafe(struct bpf_func_state *old, - struct bpf_func_state *cur, - struct bpf_id_pair *idmap) +static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old, + struct bpf_func_state *cur, struct bpf_id_pair *idmap) { int i, spi; @@ -7265,9 +7272,8 @@ static bool stacksafe(struct bpf_func_state *old, continue; if (old->stack[spi].slot_type[0] != STACK_SPILL) continue; - if (!regsafe(&old->stack[spi].spilled_ptr, - &cur->stack[spi].spilled_ptr, - idmap)) + if (!regsafe(env, &old->stack[spi].spilled_ptr, + &cur->stack[spi].spilled_ptr, idmap)) /* when explored and current stack slot are both storing * spilled registers, check that stored pointers types * are the same as well. @@ -7324,10 +7330,11 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat memset(env->idmap_scratch, 0, sizeof(env->idmap_scratch)); for (i = 0; i < MAX_BPF_REG; i++) - if (!regsafe(&old->regs[i], &cur->regs[i], env->idmap_scratch)) + if (!regsafe(env, &old->regs[i], &cur->regs[i], + env->idmap_scratch)) return false; - if (!stacksafe(old, cur, env->idmap_scratch)) + if (!stacksafe(env, old, cur, env->idmap_scratch)) return false; if (!refsafe(old, cur))