From patchwork Fri Nov 23 17:29:02 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 151916 Delivered-To: patch@linaro.org Received: by 2002:a2e:299d:0:0:0:0:0 with SMTP id p29-v6csp2408786ljp; Fri, 23 Nov 2018 09:29:18 -0800 (PST) X-Google-Smtp-Source: AFSGD/VDPlp7lu0pcUkF2v5TGaK4V6x2qe1FLUhiF6l8KLbQbT8SeTa/tDO0QZxbMl7roTNFLJp9 X-Received: by 2002:a63:fa46:: with SMTP id g6mr14937948pgk.18.1542994158594; Fri, 23 Nov 2018 09:29:18 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542994158; cv=none; d=google.com; s=arc-20160816; b=VpSe9KYjOBlqRmBqpPgNudgNevetZ2L38Jecnnjr0SCCnAaROxdEdU9hJD03zYMwmI AL7BT/Tj/0gKdOexaUA9a/6iAa94v0i0g6cgpyJL+1jwyMtrZ1WjFcfxobqgAFuAgZfE qsdrRS6bWMNXZ3pg3KLhfrCwdkFZAst3w9XzYLQ1z8mneiBukwoG86kzrW3Y2tcTaWoK u4h+XoCuoBBqbAZUpAIlw65kJ8Xhy8norC9TBoQiBnZpd9+rG28n5y318eJUjX61m1sv /Nfbw9X05csVyhNqjBO3kvrHQlN+9qrNTAqxQmrHL4wktH9BXmx29SfjWjwSEH6z3NmW DScg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=vsz8kXb6CVVsBzyFCRZkaXVWSBUyNLZRbPvH2/3m1zI=; b=Lf7fmJUge4v2J1L8Xl+K5tWOn2/Z1fPwII3dMWhyy7evWZj38dlFxKNxLm6tr6LZHg GTMc6W2qJspsWv/PJ06J0WByha6BmBGLmyVHG/IPtBmTH+zfCuErLjkmPsidPp+ZP1my QwRKsBDny3ticzQQXztpmKzr8Ae/bq1hcEUmKHasxfiqlHCAGbAh8cyB0GY4VkGiL1qU hn5HyArVGem+fUCABl5XWTDeyR9Lo5i/FVrj3QjKdQVrawJ4x5YFsr2dDX+R0uLLO2aa Dp20fGYetkv99/QPJ77SyxKZfoKqg1FaCtuUS423czh0yrFugjraPlOuv2OTZ7JnPdsZ 5EXQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Cii11l+5; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id k25si30329838pfe.10.2018.11.23.09.29.18; Fri, 23 Nov 2018 09:29:18 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Cii11l+5; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2440877AbeKXEOZ (ORCPT + 32 others); Fri, 23 Nov 2018 23:14:25 -0500 Received: from mail-wr1-f67.google.com ([209.85.221.67]:45832 "EHLO mail-wr1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2440864AbeKXEOZ (ORCPT ); Fri, 23 Nov 2018 23:14:25 -0500 Received: by mail-wr1-f67.google.com with SMTP id v6so13037241wrr.12 for ; Fri, 23 Nov 2018 09:29:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=vsz8kXb6CVVsBzyFCRZkaXVWSBUyNLZRbPvH2/3m1zI=; b=Cii11l+5xtXHlR9rXHCV63qwX5vwMHzHo8g/av47h2GbA8ZB0n8xnNUVYoZix5gzSL EzhMannNO+RLhXEirMVjjyRDZGZlauqrsA+zseqpgT/NgjAXE7IsHm8rOLtC0hJ5mLLI r94bw82iV4vBldk9UgMQWQN583nbS/ZvAkQCU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=vsz8kXb6CVVsBzyFCRZkaXVWSBUyNLZRbPvH2/3m1zI=; b=s//TnxzlFnhh2dTomw0s3HpN8A0MGHNXZvVMbTm9BErcHa4qWBAaJV5FqZBdaDaQIV /L0CjflYxZLZgkqeamNlyR3wstXpDjkyFYjRkWI2BJ6POf8cDRykyl15zsxqH1s+gyVa 6ULKlp/MxYPpFCrNxSvAnnu43wfek0wU0V1r2gYg6qtukz30uc5jZpXh1fM8vdUaRY3/ 5ynmDXVVUw6XQ/fvBGLP0+vEg+50IViEjL9zGqcSwbIG6l/hjFoYwzZsSzIMHe8KMlIE I7uSPfT0iFUOJPng80W+oDMSKq2fBwicvefXi7XbkqXwmFJP6RGIYOK1bVXyU++kXTCP 61Mw== X-Gm-Message-State: AA+aEWbMqp17uwtXPbXMWXq+5pyBhjiKZH9krTBvLzzkNLQ9Itc4LQnE oPwKn2/OKxicsp8EUxni/Jwp5Zfxjvg= X-Received: by 2002:adf:c6c3:: with SMTP id c3-v6mr14341030wrh.243.1542994153948; Fri, 23 Nov 2018 09:29:13 -0800 (PST) Received: from harold.home ([2a01:cb1d:112:6f00:6913:f64b:5e59:5ba5]) by smtp.gmail.com with ESMTPSA id 125sm5041606wml.35.2018.11.23.09.29.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 23 Nov 2018 09:29:12 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, will.deacon@arm.com, catalin.marinas@arm.com, ast@fb.com, daniel@iogearbox.net, Ard Biesheuvel Subject: [PATCH] arm64/bpf: use movn/movk/movk sequence to generate kernel addresses Date: Fri, 23 Nov 2018 18:29:02 +0100 Message-Id: <20181123172902.21480-1-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.19.1 MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On arm64, all executable code is guaranteed to reside in the vmalloc space (or the module space), and so jump targets will only use 48 bits at most, and the remaining bits are guaranteed to be 0x1. This means we can generate an immediate jump address using a sequence of one MOVN (move wide negated) and two MOVK instructions, where the first one sets the lower 16 bits but also sets all top bits to 0x1. Signed-off-by: Ard Biesheuvel --- I looked into using ADRP/ADD pairs, but this is very fiddly, since it requires knowledge about where the ADRP instruction ends up in memory. (ADRP produces a PC-relative address with bits [11:0] cleared, and so in addition to the distance between the instruction and the target, we also need to know their offsets modulo 4096 and I wasn't sure whether the offsets are guaranteed to be relative to the start of a page or not) arch/arm64/net/bpf_jit_comp.c | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) -- 2.19.1 Acked-by: Will Deacon diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index a6fdaea07c63..3b4d2c6fc133 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -134,10 +134,9 @@ static inline void emit_a64_mov_i64(const int reg, const u64 val, } /* - * This is an unoptimized 64 immediate emission used for BPF to BPF call - * addresses. It will always do a full 64 bit decomposition as otherwise - * more complexity in the last extra pass is required since we previously - * reserved 4 instructions for the address. + * Kernel addresses in the vmalloc space use at most 48 bits, and the + * remaining bits are guaranteed to be 0x1. So we can compose the address + * with a fixed length movn/movk/movk sequence. */ static inline void emit_addr_mov_i64(const int reg, const u64 val, struct jit_ctx *ctx) @@ -145,8 +144,8 @@ static inline void emit_addr_mov_i64(const int reg, const u64 val, u64 tmp = val; int shift = 0; - emit(A64_MOVZ(1, reg, tmp & 0xffff, shift), ctx); - for (;shift < 48;) { + emit(A64_MOVN(1, reg, ~tmp & 0xffff, shift), ctx); + while (shift < 32) { tmp >>= 16; shift += 16; emit(A64_MOVK(1, reg, tmp & 0xffff, shift), ctx); @@ -627,10 +626,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx) const u8 r0 = bpf2a64[BPF_REG_0]; const u64 func = (u64)__bpf_call_base + imm; - if (ctx->prog->is_func) - emit_addr_mov_i64(tmp, func, ctx); - else - emit_a64_mov_i64(tmp, func, ctx); + emit_addr_mov_i64(tmp, func, ctx); emit(A64_BLR(tmp), ctx); emit(A64_MOV(1, r0, A64_R(0)), ctx); break;