From patchwork Thu Sep 3 22:33:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Luo X-Patchwork-Id: 261545 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_INVALID, DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCC5FC43461 for ; Thu, 3 Sep 2020 22:34:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 87ADA20786 for ; Thu, 3 Sep 2020 22:34:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="jK/PKpzx" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729566AbgICWel (ORCPT ); Thu, 3 Sep 2020 18:34:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54414 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729189AbgICWdn (ORCPT ); Thu, 3 Sep 2020 18:33:43 -0400 Received: from mail-qv1-xf49.google.com (mail-qv1-xf49.google.com [IPv6:2607:f8b0:4864:20::f49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 70AB9C061258 for ; Thu, 3 Sep 2020 15:33:40 -0700 (PDT) Received: by mail-qv1-xf49.google.com with SMTP id a13so2682379qvl.6 for ; Thu, 03 Sep 2020 15:33:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=XDM1OhuX3TLQpzqxRlLR5VVQ3tUt6GSeykQfLg51+H8=; b=jK/PKpzxGHZ47H23t4exek9Nw1koW4AIm1uvnHG5RB9gO2CmUG8C6EYw3I+9JcchLP X/IqMQP8EkKkUV8iLZnKiUNnPVrJvkT0iVFG+5ISzglDaAiolBxdI20OVQ+fkOYIzFE4 smgyW8N55saZG3+oHeh7XqTuyhcjL5JWs7iyxMRol/9/sf4cbZonDJbEKfAfL4CyTSIB w9um+Me1S7Lf5JL8FQSacXKi4VEUmzKGaKhwYWwbfGw+fYWYq5PQg1PmD8fopBh7ifTD Par7AIQFn4xr+VJbe4qeuglfq5Z3cOvWTUApkIrxXKOY3C6Bw0enQfubmV3GNllrgQqA 8wXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=XDM1OhuX3TLQpzqxRlLR5VVQ3tUt6GSeykQfLg51+H8=; b=HVStnE/lvEy2NrCut2LVjFkQ2e0KHGKyfVEOzoex056JM3LpLuBXa97l5gL2Au7XiN 06ZgU3+Ugw1xOt9U05SAmMf3Nuf59bgvxddYvxjrNZdGTIxhcr0dCwhBzl4s3DpyjORJ tVNXz6y1SBsoLoZZAFIJR6jDeUFGclzUtTAKIw2ZZx54YcIj82BFn65pRrCg90IfsyCD h/CLs37IRseyIhfD/w+8yrilvPzl+1R8Zx/ha6rEBf2GlvCNAR8L282eDkoaDJWEtPWy 2SlvzD5cdefJwlPON/+EactixXyZ8J8NBiJy3sq9q6XY2Zt/mHRGOcLjOH+XzH9Uv/oP 4KPg== X-Gm-Message-State: AOAM530qVHlqS7V4S3hQLXMHA4l6Xb0TcuTfPpNdO7DtYPfdluJwMoNw g882R7fHaWd1S2GQpw9dige8Xy4+kELNJEU72x8weLgm9nBVSH7ad63KZgoL4LbDnBxj72liAN3 yjFK+2vwgY3TNdV/tbVIPAUEfhb5HTtPPmiknBDEodaAnFOR2JLa/uhcIABD/lw== X-Google-Smtp-Source: ABdhPJxzitrzFma4gOCFnUIKGFV8IGlR9DohDGBkBru2fjjJ598mh2bQr3Sx+oYjTj3Ks+w63XlG3yZoaxw= X-Received: from haoluo.svl.corp.google.com ([2620:15c:2cd:202:f693:9fff:fef4:e444]) (user=haoluo job=sendgmr) by 2002:a0c:f704:: with SMTP id w4mr5094197qvn.79.1599172419477; Thu, 03 Sep 2020 15:33:39 -0700 (PDT) Date: Thu, 3 Sep 2020 15:33:28 -0700 In-Reply-To: <20200903223332.881541-1-haoluo@google.com> Message-Id: <20200903223332.881541-3-haoluo@google.com> Mime-Version: 1.0 References: <20200903223332.881541-1-haoluo@google.com> X-Mailer: git-send-email 2.28.0.526.ge36021eeef-goog Subject: [PATCH bpf-next v2 2/6] bpf/libbpf: BTF support for typed ksyms From: Hao Luo To: netdev@vger.kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Shuah Khan , Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Quentin Monnet , Hao Luo , Steven Rostedt , Ingo Molnar , Andrey Ignatov , Jakub Sitnicki Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org If a ksym is defined with a type, libbpf will try to find the ksym's btf information from kernel btf. If a valid btf entry for the ksym is found, libbpf can pass in the found btf id to the verifier, which validates the ksym's type and value. Typeless ksyms (i.e. those defined as 'void') will not have such btf_id, but it has the symbol's address (read from kallsyms) and its value is treated as a raw pointer. Signed-off-by: Hao Luo --- tools/lib/bpf/libbpf.c | 116 ++++++++++++++++++++++++++++++++++++----- 1 file changed, 102 insertions(+), 14 deletions(-) diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index b688aadf09c5..c7a8d7d72b46 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -361,6 +361,12 @@ struct extern_desc { } kcfg; struct { unsigned long long addr; + + /* target btf_id of the corresponding kernel var. */ + int vmlinux_btf_id; + + /* local btf_id of the ksym extern's type. */ + __u32 type_id; } ksym; }; }; @@ -2479,12 +2485,23 @@ static int bpf_object__load_vmlinux_btf(struct bpf_object *obj) { bool need_vmlinux_btf = false; struct bpf_program *prog; - int err; + int i, err; /* CO-RE relocations need kernel BTF */ if (obj->btf_ext && obj->btf_ext->core_relo_info.len) need_vmlinux_btf = true; + /* Support for typed ksyms needs kernel BTF */ + for (i = 0; i < obj->nr_extern; i++) { + const struct extern_desc *ext; + + ext = &obj->externs[i]; + if (ext->type == EXT_KSYM && ext->ksym.type_id) { + need_vmlinux_btf = true; + break; + } + } + bpf_object__for_each_program(prog, obj) { if (!prog->load) continue; @@ -3060,16 +3077,10 @@ static int bpf_object__collect_externs(struct bpf_object *obj) return -ENOTSUP; } } else if (strcmp(sec_name, KSYMS_SEC) == 0) { - const struct btf_type *vt; - ksym_sec = sec; ext->type = EXT_KSYM; - - vt = skip_mods_and_typedefs(obj->btf, t->type, NULL); - if (!btf_is_void(vt)) { - pr_warn("extern (ksym) '%s' is not typeless (void)\n", ext_name); - return -ENOTSUP; - } + skip_mods_and_typedefs(obj->btf, t->type, + &ext->ksym.type_id); } else { pr_warn("unrecognized extern section '%s'\n", sec_name); return -ENOTSUP; @@ -3119,6 +3130,8 @@ static int bpf_object__collect_externs(struct bpf_object *obj) vt->type = int_btf_id; vs->offset = off; vs->size = sizeof(int); + pr_debug("ksym var_secinfo: var '%s', type #%d, size %d, offset %d\n", + ext->name, vt->type, vs->size, vs->offset); } sec->size = off; } @@ -5724,8 +5737,13 @@ bpf_program__relocate(struct bpf_program *prog, struct bpf_object *obj) insn[0].imm = obj->maps[obj->kconfig_map_idx].fd; insn[1].imm = ext->kcfg.data_off; } else /* EXT_KSYM */ { - insn[0].imm = (__u32)ext->ksym.addr; - insn[1].imm = ext->ksym.addr >> 32; + if (ext->ksym.type_id) { /* typed ksyms */ + insn[0].src_reg = BPF_PSEUDO_BTF_ID; + insn[0].imm = ext->ksym.vmlinux_btf_id; + } else { /* typeless ksyms */ + insn[0].imm = (__u32)ext->ksym.addr; + insn[1].imm = ext->ksym.addr >> 32; + } } break; case RELO_CALL: @@ -6462,10 +6480,72 @@ static int bpf_object__read_kallsyms_file(struct bpf_object *obj) return err; } +static int bpf_object__resolve_ksyms_btf_id(struct bpf_object *obj) +{ + struct extern_desc *ext; + int i, id; + + if (!obj->btf_vmlinux) { + pr_warn("support of typed ksyms needs kernel btf.\n"); + return -ENOENT; + } + + for (i = 0; i < obj->nr_extern; i++) { + const struct btf_type *targ_var, *targ_type; + __u32 targ_type_id, local_type_id; + int ret; + + ext = &obj->externs[i]; + if (ext->type != EXT_KSYM || !ext->ksym.type_id) + continue; + + id = btf__find_by_name_kind(obj->btf_vmlinux, ext->name, + BTF_KIND_VAR); + if (id <= 0) { + pr_warn("no btf entry for ksym '%s' in vmlinux.\n", + ext->name); + return -ESRCH; + } + + /* find target type_id */ + targ_var = btf__type_by_id(obj->btf_vmlinux, id); + targ_type = skip_mods_and_typedefs(obj->btf_vmlinux, + targ_var->type, + &targ_type_id); + + /* find local type_id */ + local_type_id = ext->ksym.type_id; + + ret = bpf_core_types_are_compat(obj->btf_vmlinux, targ_type_id, + obj->btf, local_type_id); + if (ret <= 0) { + const struct btf_type *local_type; + const char *targ_name, *local_name; + + local_type = btf__type_by_id(obj->btf, local_type_id); + targ_name = btf__name_by_offset(obj->btf_vmlinux, + targ_type->name_off); + local_name = btf__name_by_offset(obj->btf, + local_type->name_off); + + pr_warn("ksym '%s' expects type '%s' (vmlinux_btf_id: #%d), " + "but got '%s' (btf_id: #%d)\n", ext->name, + targ_name, targ_type_id, local_name, local_type_id); + return -EINVAL; + } + + ext->is_set = true; + ext->ksym.vmlinux_btf_id = id; + pr_debug("extern (ksym) %s=vmlinux_btf_id(#%d)\n", ext->name, id); + } + return 0; +} + static int bpf_object__resolve_externs(struct bpf_object *obj, const char *extra_kconfig) { - bool need_config = false, need_kallsyms = false; + bool need_kallsyms = false, need_vmlinux_btf = false; + bool need_config = false; struct extern_desc *ext; void *kcfg_data = NULL; int err, i; @@ -6496,7 +6576,10 @@ static int bpf_object__resolve_externs(struct bpf_object *obj, strncmp(ext->name, "CONFIG_", 7) == 0) { need_config = true; } else if (ext->type == EXT_KSYM) { - need_kallsyms = true; + if (ext->ksym.type_id) + need_vmlinux_btf = true; + else + need_kallsyms = true; } else { pr_warn("unrecognized extern '%s'\n", ext->name); return -EINVAL; @@ -6525,6 +6608,11 @@ static int bpf_object__resolve_externs(struct bpf_object *obj, if (err) return -EINVAL; } + if (need_vmlinux_btf) { + err = bpf_object__resolve_ksyms_btf_id(obj); + if (err) + return -EINVAL; + } for (i = 0; i < obj->nr_extern; i++) { ext = &obj->externs[i]; @@ -6557,10 +6645,10 @@ int bpf_object__load_xattr(struct bpf_object_load_attr *attr) } err = bpf_object__probe_loading(obj); + err = err ? : bpf_object__load_vmlinux_btf(obj); err = err ? : bpf_object__resolve_externs(obj, obj->kconfig); err = err ? : bpf_object__sanitize_and_load_btf(obj); err = err ? : bpf_object__sanitize_maps(obj); - err = err ? : bpf_object__load_vmlinux_btf(obj); err = err ? : bpf_object__init_kern_struct_ops_maps(obj); err = err ? : bpf_object__create_maps(obj); err = err ? : bpf_object__relocate(obj, attr->target_btf_path); From patchwork Thu Sep 3 22:33:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Luo X-Patchwork-Id: 261546 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_INVALID, DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE091C43461 for ; Thu, 3 Sep 2020 22:34:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 70DD520716 for ; Thu, 3 Sep 2020 22:34:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="FpdspfuA" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729226AbgICWea (ORCPT ); Thu, 3 Sep 2020 18:34:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54434 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729356AbgICWdo (ORCPT ); Thu, 3 Sep 2020 18:33:44 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 40EE3C061247 for ; Thu, 3 Sep 2020 15:33:44 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id w38so2350336ybi.20 for ; Thu, 03 Sep 2020 15:33:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=0Glc7TccRxwMcDc5UApAFzlQy+c+Mw+LnskMm2Kovo0=; b=FpdspfuAH314geZIlOd3qlPzV3nN9QTeYoDjMnU1QD/RUke65/GPYnBLJRIcOpFa9k b19cD3Hn1wYHVxY5gplrDGBrnyFbyFZzq9SAYoCdzrQ71A+kfr3ARqs+3IZLLh0Y+JD+ BuiJRugqUumB9rBf3FkqRzxb94GJhkAGRpH5ZQqf3zgLYvaz5Wf5NIfMi8c/GtWcsSe8 q/mrgBlAA7B+k6gnr76pFCIRaSKQAgoW+EpFR7c23JSlli40dXBQfVqi1TQABFu1OSDQ VVxHB68geQxFZWotnipPNzK8sgeR7rLv5R1RBYoD7Cb66uxSuScCyO5jQcILuAJzzCG9 hayg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=0Glc7TccRxwMcDc5UApAFzlQy+c+Mw+LnskMm2Kovo0=; b=n5E0zDInZ5rSxARs6ZiLa4pGeOD10O7+wdw2GylCG7VAXWThztx0YgwDwzFjS8cVsY XQzYKKknTGjAPbUZ1nbX/XPECq6DoUj2nWTeRbK7koEIDqcnpJmrpsOEOCkCafhtOUEV Tot7+rCZZxflUDgUMEC5WrUwT0fk0/mbMPn6omDL6vUKVWDvcqJtdrWgFAglc9M+QCgk 8F1HKnWiWUQ3KURCeMV5DCH7HpPUMul+WfRHK6SQzhYwbgC4o6jaJA5E7T/H5vvVOz8i 0lfNvX6d5EsLb7sQ7gjogilKssARGA7NOyUbeXwYpMO8LqWA/Y+EeFrYV2i2xijhMaLT ZvGQ== X-Gm-Message-State: AOAM533ItbDY4ZC57vFBHjtHUZN0ut0OZaVF0wmWo4WhoPhjrr3xgKzF RsACyIdLxQt7QYIyJJ5AJcM153lV6703008Q3V5KMnf/XMNVzLE/Zvr2K805y+yB4fI3t36VrXT A/nKevZsOJMrMOyCW2Zl3nDQja8mNo4zZu0RcQfBD3IUPcA8TBgUBPGxzF3PLFw== X-Google-Smtp-Source: ABdhPJxdVm8RZQIjKc4D9/LZQWvBhazfT9+snZf0zkFJSyvFXVQ4xk4Y10+51qlTBzQ0GU1x1cOHlCzjc8I= X-Received: from haoluo.svl.corp.google.com ([2620:15c:2cd:202:f693:9fff:fef4:e444]) (user=haoluo job=sendgmr) by 2002:a25:e03:: with SMTP id 3mr6232976ybo.334.1599172423347; Thu, 03 Sep 2020 15:33:43 -0700 (PDT) Date: Thu, 3 Sep 2020 15:33:30 -0700 In-Reply-To: <20200903223332.881541-1-haoluo@google.com> Message-Id: <20200903223332.881541-5-haoluo@google.com> Mime-Version: 1.0 References: <20200903223332.881541-1-haoluo@google.com> X-Mailer: git-send-email 2.28.0.526.ge36021eeef-goog Subject: [PATCH bpf-next v2 4/6] bpf: Introduce bpf_per_cpu_ptr() From: Hao Luo To: netdev@vger.kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Shuah Khan , Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Quentin Monnet , Hao Luo , Steven Rostedt , Ingo Molnar , Andrey Ignatov , Jakub Sitnicki Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Add bpf_per_cpu_ptr() to help bpf programs access percpu vars. bpf_per_cpu_ptr() has the same semantic as per_cpu_ptr() in the kernel except that it may return NULL. This happens when the cpu parameter is out of range. So the caller must check the returned value. Acked-by: Andrii Nakryiko Signed-off-by: Hao Luo --- include/linux/bpf.h | 3 ++ include/linux/btf.h | 11 ++++++ include/uapi/linux/bpf.h | 17 +++++++++ kernel/bpf/btf.c | 10 ------ kernel/bpf/verifier.c | 66 +++++++++++++++++++++++++++++++--- kernel/trace/bpf_trace.c | 18 ++++++++++ tools/include/uapi/linux/bpf.h | 17 +++++++++ 7 files changed, 128 insertions(+), 14 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index c6d9f2c444f4..6b2034f7665e 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -292,6 +292,7 @@ enum bpf_arg_type { ARG_PTR_TO_ALLOC_MEM, /* pointer to dynamically allocated memory */ ARG_PTR_TO_ALLOC_MEM_OR_NULL, /* pointer to dynamically allocated memory or NULL */ ARG_CONST_ALLOC_SIZE_OR_ZERO, /* number of allocated bytes requested */ + ARG_PTR_TO_PERCPU_BTF_ID, /* pointer to in-kernel percpu type */ }; /* type of values returned from helper functions */ @@ -305,6 +306,7 @@ enum bpf_return_type { RET_PTR_TO_SOCK_COMMON_OR_NULL, /* returns a pointer to a sock_common or NULL */ RET_PTR_TO_ALLOC_MEM_OR_NULL, /* returns a pointer to dynamically allocated memory or NULL */ RET_PTR_TO_BTF_ID_OR_NULL, /* returns a pointer to a btf_id or NULL */ + RET_PTR_TO_MEM_OR_BTF_ID_OR_NULL, /* returns a pointer to a valid memory or a btf_id or NULL */ }; /* eBPF function prototype used by verifier to allow BPF_CALLs from eBPF programs @@ -385,6 +387,7 @@ enum bpf_reg_type { PTR_TO_RDONLY_BUF_OR_NULL, /* reg points to a readonly buffer or NULL */ PTR_TO_RDWR_BUF, /* reg points to a read/write buffer */ PTR_TO_RDWR_BUF_OR_NULL, /* reg points to a read/write buffer or NULL */ + PTR_TO_PERCPU_BTF_ID, /* reg points to percpu kernel type */ }; /* The information passed from prog-specific *_is_valid_access diff --git a/include/linux/btf.h b/include/linux/btf.h index 592373d359b9..07b7de1c05b0 100644 --- a/include/linux/btf.h +++ b/include/linux/btf.h @@ -71,6 +71,11 @@ btf_resolve_size(const struct btf *btf, const struct btf_type *type, i < btf_type_vlen(struct_type); \ i++, member++) +#define for_each_vsi(i, struct_type, member) \ + for (i = 0, member = btf_type_var_secinfo(struct_type); \ + i < btf_type_vlen(struct_type); \ + i++, member++) + static inline bool btf_type_is_ptr(const struct btf_type *t) { return BTF_INFO_KIND(t->info) == BTF_KIND_PTR; @@ -155,6 +160,12 @@ static inline const struct btf_member *btf_type_member(const struct btf_type *t) return (const struct btf_member *)(t + 1); } +static inline const struct btf_var_secinfo *btf_type_var_secinfo( + const struct btf_type *t) +{ + return (const struct btf_var_secinfo *)(t + 1); +} + #ifdef CONFIG_BPF_SYSCALL const struct btf_type *btf_type_by_id(const struct btf *btf, u32 type_id); const char *btf_name_by_offset(const struct btf *btf, u32 offset); diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index ab00ad9b32e5..d0ec94d5bdbf 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -3596,6 +3596,22 @@ union bpf_attr { * the data in *dst*. This is a wrapper of copy_from_user(). * Return * 0 on success, or a negative error in case of failure. + * + * void *bpf_per_cpu_ptr(const void *percpu_ptr, u32 cpu) + * Description + * Take a pointer to a percpu ksym, *percpu_ptr*, and return a + * pointer to the percpu kernel variable on *cpu*. A ksym is an + * extern variable decorated with '__ksym'. For ksym, there is a + * global var (either static or global) defined of the same name + * in the kernel. The ksym is percpu if the global var is percpu. + * The returned pointer points to the global percpu var on *cpu*. + * + * bpf_per_cpu_ptr() has the same semantic as per_cpu_ptr() in the + * kernel, except that bpf_per_cpu_ptr() may return NULL. This + * happens if *cpu* is larger than nr_cpu_ids. The caller of + * bpf_per_cpu_ptr() must check the returned value. + * Return + * A generic pointer pointing to the kernel percpu variable on *cpu*. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -3747,6 +3763,7 @@ union bpf_attr { FN(inode_storage_delete), \ FN(d_path), \ FN(copy_from_user), \ + FN(bpf_per_cpu_ptr), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 5831d9f3f3c5..4aa22e31774c 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -188,11 +188,6 @@ i < btf_type_vlen(struct_type); \ i++, member++) -#define for_each_vsi(i, struct_type, member) \ - for (i = 0, member = btf_type_var_secinfo(struct_type); \ - i < btf_type_vlen(struct_type); \ - i++, member++) - #define for_each_vsi_from(i, from, struct_type, member) \ for (i = from, member = btf_type_var_secinfo(struct_type) + from; \ i < btf_type_vlen(struct_type); \ @@ -513,11 +508,6 @@ static const struct btf_var *btf_type_var(const struct btf_type *t) return (const struct btf_var *)(t + 1); } -static const struct btf_var_secinfo *btf_type_var_secinfo(const struct btf_type *t) -{ - return (const struct btf_var_secinfo *)(t + 1); -} - static const struct btf_kind_operations *btf_type_ops(const struct btf_type *t) { return kind_ops[BTF_INFO_KIND(t->info)]; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 3b382c080cfd..a702600ff581 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -239,6 +239,7 @@ struct bpf_call_arg_meta { int ref_obj_id; int func_id; u32 btf_id; + u32 ret_btf_id; }; struct btf *btf_vmlinux; @@ -504,6 +505,7 @@ static const char * const reg_type_str[] = { [PTR_TO_XDP_SOCK] = "xdp_sock", [PTR_TO_BTF_ID] = "ptr_", [PTR_TO_BTF_ID_OR_NULL] = "ptr_or_null_", + [PTR_TO_PERCPU_BTF_ID] = "percpu_ptr_", [PTR_TO_MEM] = "mem", [PTR_TO_MEM_OR_NULL] = "mem_or_null", [PTR_TO_RDONLY_BUF] = "rdonly_buf", @@ -570,7 +572,9 @@ static void print_verifier_state(struct bpf_verifier_env *env, /* reg->off should be 0 for SCALAR_VALUE */ verbose(env, "%lld", reg->var_off.value + reg->off); } else { - if (t == PTR_TO_BTF_ID || t == PTR_TO_BTF_ID_OR_NULL) + if (t == PTR_TO_BTF_ID || + t == PTR_TO_BTF_ID_OR_NULL || + t == PTR_TO_PERCPU_BTF_ID) verbose(env, "%s", kernel_type_name(reg->btf_id)); verbose(env, "(id=%d", reg->id); if (reg_type_may_be_refcounted_or_null(t)) @@ -2184,6 +2188,7 @@ static bool is_spillable_regtype(enum bpf_reg_type type) case PTR_TO_RDONLY_BUF_OR_NULL: case PTR_TO_RDWR_BUF: case PTR_TO_RDWR_BUF_OR_NULL: + case PTR_TO_PERCPU_BTF_ID: return true; default: return false; @@ -4003,6 +4008,15 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, if (type != expected_type) goto err_type; } + } else if (arg_type == ARG_PTR_TO_PERCPU_BTF_ID) { + expected_type = PTR_TO_PERCPU_BTF_ID; + if (type != expected_type) + goto err_type; + if (!reg->btf_id) { + verbose(env, "Helper has invalid btf_id in R%d\n", regno); + return -EACCES; + } + meta->ret_btf_id = reg->btf_id; } else if (arg_type == ARG_PTR_TO_BTF_ID) { bool ids_match = false; @@ -5002,6 +5016,30 @@ static int check_helper_call(struct bpf_verifier_env *env, int func_id, int insn regs[BPF_REG_0].type = PTR_TO_MEM_OR_NULL; regs[BPF_REG_0].id = ++env->id_gen; regs[BPF_REG_0].mem_size = meta.mem_size; + } else if (fn->ret_type == RET_PTR_TO_MEM_OR_BTF_ID_OR_NULL) { + const struct btf_type *t; + + mark_reg_known_zero(env, regs, BPF_REG_0); + t = btf_type_skip_modifiers(btf_vmlinux, meta.ret_btf_id, NULL); + if (!btf_type_is_struct(t)) { + u32 tsize; + const struct btf_type *ret; + const char *tname; + + /* resolve the type size of ksym. */ + ret = btf_resolve_size(btf_vmlinux, t, &tsize); + if (IS_ERR(ret)) { + tname = btf_name_by_offset(btf_vmlinux, t->name_off); + verbose(env, "unable to resolve the size of type '%s': %ld\n", + tname, PTR_ERR(ret)); + return -EINVAL; + } + regs[BPF_REG_0].type = PTR_TO_MEM_OR_NULL; + regs[BPF_REG_0].mem_size = tsize; + } else { + regs[BPF_REG_0].type = PTR_TO_BTF_ID_OR_NULL; + regs[BPF_REG_0].btf_id = meta.ret_btf_id; + } } else if (fn->ret_type == RET_PTR_TO_BTF_ID_OR_NULL) { int ret_btf_id; @@ -7407,7 +7445,8 @@ static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn) regs[insn->dst_reg].type = type; if (type == PTR_TO_MEM) { regs[insn->dst_reg].mem_size = meta; - } else if (type == PTR_TO_BTF_ID) { + } else if (type == PTR_TO_BTF_ID || + type == PTR_TO_PERCPU_BTF_ID) { regs[insn->dst_reg].btf_id = meta; } else { verbose(env, "bpf verifier is misconfigured\n"); @@ -9307,10 +9346,14 @@ static int check_pseudo_btf_id(struct bpf_verifier_env *env, struct bpf_insn *insn, struct bpf_insn_aux_data *aux) { - u32 type, id = insn->imm; + u32 datasec_id, type, id = insn->imm; + const struct btf_var_secinfo *vsi; + const struct btf_type *datasec; const struct btf_type *t; const char *sym_name; + bool percpu = false; u64 addr; + int i; if (!btf_vmlinux) { verbose(env, "%s: btf not available. verifier misconfigured.\n", @@ -9343,12 +9386,27 @@ static int check_pseudo_btf_id(struct bpf_verifier_env *env, return -ENOENT; } + datasec_id = btf_find_by_name_kind(btf_vmlinux, ".data..percpu", + BTF_KIND_DATASEC); + if (datasec_id > 0) { + datasec = btf_type_by_id(btf_vmlinux, datasec_id); + for_each_vsi(i, datasec, vsi) { + if (vsi->type == id) { + percpu = true; + break; + } + } + } + insn[0].imm = (u32)addr; insn[1].imm = addr >> 32; type = t->type; t = btf_type_skip_modifiers(btf_vmlinux, type, NULL); - if (!btf_type_is_struct(t)) { + if (percpu) { + aux->pseudo_btf_id_type = PTR_TO_PERCPU_BTF_ID; + aux->pseudo_btf_id_meta = type; + } else if (!btf_type_is_struct(t)) { const struct btf_type *ret; const char *tname; u32 tsize; diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index b2a5380eb187..d474c1530f87 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -1144,6 +1144,22 @@ static const struct bpf_func_proto bpf_d_path_proto = { .allowed = bpf_d_path_allowed, }; +BPF_CALL_2(bpf_per_cpu_ptr, const void *, ptr, u32, cpu) +{ + if (cpu >= nr_cpu_ids) + return 0; + + return (u64)per_cpu_ptr(ptr, cpu); +} + +static const struct bpf_func_proto bpf_per_cpu_ptr_proto = { + .func = bpf_per_cpu_ptr, + .gpl_only = false, + .ret_type = RET_PTR_TO_MEM_OR_BTF_ID_OR_NULL, + .arg1_type = ARG_PTR_TO_PERCPU_BTF_ID, + .arg2_type = ARG_ANYTHING, +}; + const struct bpf_func_proto * bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) { @@ -1230,6 +1246,8 @@ bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_get_task_stack_proto; case BPF_FUNC_copy_from_user: return prog->aux->sleepable ? &bpf_copy_from_user_proto : NULL; + case BPF_FUNC_bpf_per_cpu_ptr: + return &bpf_per_cpu_ptr_proto; default: return NULL; } diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index ab00ad9b32e5..d0ec94d5bdbf 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -3596,6 +3596,22 @@ union bpf_attr { * the data in *dst*. This is a wrapper of copy_from_user(). * Return * 0 on success, or a negative error in case of failure. + * + * void *bpf_per_cpu_ptr(const void *percpu_ptr, u32 cpu) + * Description + * Take a pointer to a percpu ksym, *percpu_ptr*, and return a + * pointer to the percpu kernel variable on *cpu*. A ksym is an + * extern variable decorated with '__ksym'. For ksym, there is a + * global var (either static or global) defined of the same name + * in the kernel. The ksym is percpu if the global var is percpu. + * The returned pointer points to the global percpu var on *cpu*. + * + * bpf_per_cpu_ptr() has the same semantic as per_cpu_ptr() in the + * kernel, except that bpf_per_cpu_ptr() may return NULL. This + * happens if *cpu* is larger than nr_cpu_ids. The caller of + * bpf_per_cpu_ptr() must check the returned value. + * Return + * A generic pointer pointing to the kernel percpu variable on *cpu*. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -3747,6 +3763,7 @@ union bpf_attr { FN(inode_storage_delete), \ FN(d_path), \ FN(copy_from_user), \ + FN(bpf_per_cpu_ptr), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper From patchwork Thu Sep 3 22:33:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Luo X-Patchwork-Id: 261547 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_INVALID, DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3B2CC10DAA for ; Thu, 3 Sep 2020 22:34:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7950A20716 for ; Thu, 3 Sep 2020 22:34:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="XY11zWkK" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729486AbgICWeI (ORCPT ); Thu, 3 Sep 2020 18:34:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54446 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729421AbgICWds (ORCPT ); Thu, 3 Sep 2020 18:33:48 -0400 Received: from mail-qv1-xf49.google.com (mail-qv1-xf49.google.com [IPv6:2607:f8b0:4864:20::f49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D840AC061262 for ; Thu, 3 Sep 2020 15:33:47 -0700 (PDT) Received: by mail-qv1-xf49.google.com with SMTP id i15so2676602qvr.15 for ; Thu, 03 Sep 2020 15:33:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=UW/0PGzjXFNLpgvsGBdXGPzkiQxfa78I9ft59r2N/eQ=; b=XY11zWkKXAZxLJ9vRo4yHPqTSberWaYsHouzkHd91K67vZInZlS3w/HGq0e6ogCBTk aWV/sMHfo1eQcJyerLI7GnKlBBw6dHNIXyb9+HBOVcwN4Eu6Gh7ZeoqqjsVoIQ2cqiA/ l9HJMYtuh5JT7LSWyWGQ78IAh6eOablm9ZpDUtYYfF0qFpYPnFP9sy8f3LYaHENDesw8 RXTn1dRHAmB893UlWGp5fxDBkKWRTOHqCRg/bm8MagpTOlA2yQbjin2MthfK9TUctbPA 03frbeRjM5BqgZSvjOTX8CFPwOmR8n5DJUoMvzjb+ymNh9bJR14WAChjo1380b2tzIr7 oCtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=UW/0PGzjXFNLpgvsGBdXGPzkiQxfa78I9ft59r2N/eQ=; b=DbYkX3IUXh4aL8A7JOlSVZjQjTf27it0xUwMVtGzMMFdjoWlFPDpYZ7hXJ1D2trx9x nBWnztsIatvuTsxjiGH+WD7dM3ECP2DtMiJVfkG1rOmf45v1ue6rJrspOtu5xRm3ALZQ WiQNgIV48CaWh2QfX5g8640VM9ks1QBTYxRCCYNaRwMGyDCJqJbbpAcEvs+KSwZa2iex IqQpNfq6+NDhoUslFYE213Wn/1pCnTLH/hE2jNl0DTdt5oEJRa0Y+p4IJ+W+EMx0tVeM vo9akLDAGT446Q+A4wwB3wQmcdq0hAnM7MAq6eE6+YR7JfMNDIBvHaCJmwoZ+7tcpmx5 GHww== X-Gm-Message-State: AOAM531eDKsgaltPILF/aBiBt3DdT1Up8P3n/SP12wXusIEz5ztYvh2i F+AfufoFYNqfaAB/iNcZu0Cd+/QTQRkBsDVx2lIqRGHgmNA/Fg+UDHqCU2F8btSPeE4CB/6e/QR UvrhFra1bpGJZrFC68jXRe7ZcZWOep2eaktS1jud5ULQ9z60GP0AokVY39udFsg== X-Google-Smtp-Source: ABdhPJwMqcx89txh5tXQBtUtjcZyiaptSRj7S1RflllH58s5G4zJUG+EC8RwDGjZKEdQQ1HYDhpzgYctmw0= X-Received: from haoluo.svl.corp.google.com ([2620:15c:2cd:202:f693:9fff:fef4:e444]) (user=haoluo job=sendgmr) by 2002:a0c:c712:: with SMTP id w18mr4106712qvi.7.1599172426915; Thu, 03 Sep 2020 15:33:46 -0700 (PDT) Date: Thu, 3 Sep 2020 15:33:32 -0700 In-Reply-To: <20200903223332.881541-1-haoluo@google.com> Message-Id: <20200903223332.881541-7-haoluo@google.com> Mime-Version: 1.0 References: <20200903223332.881541-1-haoluo@google.com> X-Mailer: git-send-email 2.28.0.526.ge36021eeef-goog Subject: [PATCH bpf-next v2 6/6] bpf/selftests: Test for bpf_per_cpu_ptr() and bpf_this_cpu_ptr() From: Hao Luo To: netdev@vger.kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Shuah Khan , Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Quentin Monnet , Hao Luo , Steven Rostedt , Ingo Molnar , Andrey Ignatov , Jakub Sitnicki Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Test bpf_per_cpu_ptr() and bpf_this_cpu_ptr(). Test two paths in the kernel. If the base pointer points to a struct, the returned reg is of type PTR_TO_BTF_ID. Direct pointer dereference can be applied on the returned variable. If the base pointer isn't a struct, the returned reg is of type PTR_TO_MEM, which also supports direct pointer dereference. Acked-by: Andrii Nakryiko Signed-off-by: Hao Luo --- .../selftests/bpf/prog_tests/ksyms_btf.c | 10 +++++++ .../selftests/bpf/progs/test_ksyms_btf.c | 26 +++++++++++++++++++ 2 files changed, 36 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/ksyms_btf.c b/tools/testing/selftests/bpf/prog_tests/ksyms_btf.c index 7b6846342449..22cc642dbc0e 100644 --- a/tools/testing/selftests/bpf/prog_tests/ksyms_btf.c +++ b/tools/testing/selftests/bpf/prog_tests/ksyms_btf.c @@ -58,6 +58,16 @@ void test_ksyms_btf(void) CHECK(data->out__bpf_prog_active != bpf_prog_active_addr, "bpf_prog_active", "got %llu, exp %llu\n", data->out__bpf_prog_active, bpf_prog_active_addr); + CHECK(data->out__rq_cpu == -1, "rq_cpu", + "got %u, exp != -1\n", data->out__rq_cpu); + CHECK(data->out__percpu_bpf_prog_active == -1, "percpu_bpf_prog_active", + "got %d, exp != -1\n", data->out__percpu_bpf_prog_active); + + CHECK(data->out__this_rq_cpu == -1, "this_rq_cpu", + "got %u, exp != -1\n", data->out__this_rq_cpu); + CHECK(data->out__this_bpf_prog_active == -1, "this_bpf_prog_active", + "got %d, exp != -1\n", data->out__this_bpf_prog_active); + cleanup: test_ksyms_btf__destroy(skel); } diff --git a/tools/testing/selftests/bpf/progs/test_ksyms_btf.c b/tools/testing/selftests/bpf/progs/test_ksyms_btf.c index e04e31117f84..02d564349892 100644 --- a/tools/testing/selftests/bpf/progs/test_ksyms_btf.c +++ b/tools/testing/selftests/bpf/progs/test_ksyms_btf.c @@ -8,15 +8,41 @@ __u64 out__runqueues = -1; __u64 out__bpf_prog_active = -1; +__u32 out__rq_cpu = -1; /* percpu struct fields */ +int out__percpu_bpf_prog_active = -1; /* percpu int */ + +__u32 out__this_rq_cpu = -1; +int out__this_bpf_prog_active = -1; + extern const struct rq runqueues __ksym; /* struct type global var. */ extern const int bpf_prog_active __ksym; /* int type global var. */ SEC("raw_tp/sys_enter") int handler(const void *ctx) { + struct rq *rq; + int *active; + __u32 cpu; + out__runqueues = (__u64)&runqueues; out__bpf_prog_active = (__u64)&bpf_prog_active; + cpu = bpf_get_smp_processor_id(); + + /* test bpf_per_cpu_ptr() */ + rq = (struct rq *)bpf_per_cpu_ptr(&runqueues, cpu); + if (rq) + out__rq_cpu = rq->cpu; + active = (int *)bpf_per_cpu_ptr(&bpf_prog_active, cpu); + if (active) + out__percpu_bpf_prog_active = *active; + + /* test bpf_this_cpu_ptr */ + rq = (struct rq *)bpf_this_cpu_ptr(&runqueues); + out__this_rq_cpu = rq->cpu; + active = (int *)bpf_this_cpu_ptr(&bpf_prog_active); + out__this_bpf_prog_active = *active; + return 0; }