From patchwork Wed Sep 15 05:09:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 512808 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A083C433F5 for ; Wed, 15 Sep 2021 05:09:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 21C2761211 for ; Wed, 15 Sep 2021 05:09:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234248AbhIOFLO (ORCPT ); Wed, 15 Sep 2021 01:11:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44312 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230292AbhIOFLL (ORCPT ); Wed, 15 Sep 2021 01:11:11 -0400 Received: from mail-pl1-x641.google.com (mail-pl1-x641.google.com [IPv6:2607:f8b0:4864:20::641]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6E965C061574; Tue, 14 Sep 2021 22:09:52 -0700 (PDT) Received: by mail-pl1-x641.google.com with SMTP id n4so881008plh.9; Tue, 14 Sep 2021 22:09:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JixwwQvoswotlhIZ0ugZlwceSQMyoImG0zUo+jrOdB8=; b=Y9DZURWJ097NBbiNNG/fMRFvaL3FYpysuZ0pWhDAWrPHNE4PUi73gM9QZlZgxzqaDW O15Hi0QDRuSv7QlElHsiMp8u79Fx7D06ug0g5H6vnw6q1GuCnSVzNUG/qV+TU/TD+XtC LTVgrFE9y7LoAU2a9kSv0xnLZULu66jWVFx6ySFkFX04WoBfhSJMUhiTFIqCT20PfuKQ K0TNmFXArPAI1mqRIWvm1ZxbfZYN4/S9X7xqe1VxL4cGxgGoLUAHSJDhObh4xtoxSyY8 MgYGq3dXO7oAEDjrQ2UFZmOKaXtYOkMJb48nY2ykTt5XO4L8EF3XGIoCaja5+8Xsj+48 R2MQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JixwwQvoswotlhIZ0ugZlwceSQMyoImG0zUo+jrOdB8=; b=U/RwJpvnyFNJd4tlYxy/mUFPQ+WkaRR025OkgdCzqHusy4m3UbTnrIfobjk4X2jGK1 ugN4a4D6EpywGRgXKeQNXfUi/+55pK2Y0687qMrnGiXYXB2ZWb1EtdQpqACIGiS/chF8 ji4mfP23D9nyUiYKbsknFWXPXUWoIXDADwhRe5YN/HSyobY6kgmaJGgLae+P1NllKYBX TsKfHEokw912Jz7BkQv/nwPjcETO1CtX32NNzB9LuOwONsNfvtx23+n/VTJbsRoSgxUs gVCaFwVdQVM1hbNbUIuNpX13ZkkAURaApRvZ6KoCB6Qj/z3rMd8xnnb9cUEVwK5+jPiV /3hg== X-Gm-Message-State: AOAM530iH/35NUmXE9mTSrF+Vdkk+Tar9KQi4YA/Y7ctY8blea0qIx5h Nig3zsbM1I576MKlclBFib/sH4TL0qbb7g== X-Google-Smtp-Source: ABdhPJz0+NXgNttHloULJqTdDBlTb8cIpgKS65ciHWjmJUKaDDBkYFjE7wvGRg5niw4CpVVABO+kAQ== X-Received: by 2002:a17:90a:55cb:: with SMTP id o11mr6443690pjm.244.1631682591650; Tue, 14 Sep 2021 22:09:51 -0700 (PDT) Received: from localhost ([2405:201:6014:d058:a28d:3909:6ed5:29e7]) by smtp.gmail.com with ESMTPSA id x1sm3890779pfc.53.2021.09.14.22.09.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 22:09:51 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Kumar Kartikeya Dwivedi , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , Jesper Dangaard Brouer , =?utf-8?q?Toke_H=C3=B8iland-J?= =?utf-8?b?w7hyZ2Vuc2Vu?= , netdev@vger.kernel.org Subject: [PATCH bpf-next v3 01/10] bpf: Introduce BPF support for kernel module function calls Date: Wed, 15 Sep 2021 10:39:34 +0530 Message-Id: <20210915050943.679062-2-memxor@gmail.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20210915050943.679062-1-memxor@gmail.com> References: <20210915050943.679062-1-memxor@gmail.com> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=17733; h=from:subject; bh=td3XGiBoeHD9GfzaZbwtH05Uj7oNV4MW9qKtMmxWn3E=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBhQX4b0sSBgtnGsio+RkS/tJaXg4740cUZt4uyUYv+ hrhO2OCJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCYUF+GwAKCRBM4MiGSL8RyjoSD/ 4ji/VqBBeBFI2O7Yg7uTpx4og1yQi+sRqBI1eN11wF6ikuDjkzG35QocJLNjX3Ns7hHujAPdyjEkoe KWFLbWaLYo0QIPL9qr0Hc4N/nwAOdPMVtHtMatRy0G1bNNQ5nn5Y3e3OKmRzYqmxPA48sMEgluhjSw 1fp9vjH4OfPUSgXOVJggYqKI7VbYpRqsBhAjvNxdkvHW83rUubd1KHZDLpTNqxZqLXq4LfjbqIV7Gr DvSyxwa4HY+CWUf1cB3CASZKekZBSiBhgg/YebUs+PAwQAD5DuYQcG5k6BNhuuVrtqJOlupdTVDUEX 1hKMkIUrUijnduanJY+An8j+Ngq+QVaROjQeY7q2zR3iDRYGxhevrXVzXEAWtQCbE66H/iYx5bWsKg vXYCduCATKvVeUaVTWJbMKB7XcEpny5TzsNh4dSkrC4bE8wH8uaCCZz3tTKwVdY5+ixTKoQgB3IoNf S/s80wwFDasWAjW7INVe6HlCHmEoC1lAp1CVsa0nrSPsMSdkLTMKBhJDezHKEENQxxSKECZfuOIMAq 6Sp7I6GexgCVSRGwDRvbjPg3Xz8cqZ1JoATDkGpIzmnkxbxt1BLDjOP6SOREvmQXBBDHbD6Z3eAzzH FSSkZaRWqov4jWJdeTlq/M9ORaQKYLpJqZdHyOeluE0qLYU5sCkP4VJ0o/vA== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org This change adds support on the kernel side to allow for BPF programs to call kernel module functions. Userspace will prepare an array of module BTF fds that is passed in during BPF_PROG_LOAD using fd_array parameter. In the kernel, the module BTFs are placed in the auxilliary struct for bpf_prog, and loaded as needed. The verifier then uses insn->off to index into the fd_array. insn->off is used by subtracting one from it, as userspace has to set the index of array in insn->off incremented by 1. This lets us denote vmlinux btf by insn->off == 0, and the module kfunc using insn->off > 0. They are sorted based on offset in an array, and each offset corresponds to one descriptor, with a max limit up to 256 such module BTFs. Another change is to check_kfunc_call callback, which now include a struct module * pointer, this is to be used in later patch such that the kfunc_id and module pointer are matched for dynamically registered BTF sets from loadable modules, so that same kfunc_id in two modules doesn't lead to check_kfunc_call succeeding. For the duration of the check_kfunc_call, the reference to struct module exists, as it returns the pointer stored in kfunc_btf_tab. Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf.h | 8 +- include/linux/bpf_verifier.h | 2 + kernel/bpf/core.c | 4 + kernel/bpf/verifier.c | 199 ++++++++++++++++++++++++++++++----- net/bpf/test_run.c | 2 +- net/ipv4/bpf_tcp_ca.c | 2 +- 6 files changed, 185 insertions(+), 32 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index f4c16f19f83e..148bd899411e 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -511,7 +511,7 @@ struct bpf_verifier_ops { const struct btf_type *t, int off, int size, enum bpf_access_type atype, u32 *next_btf_id); - bool (*check_kfunc_call)(u32 kfunc_btf_id); + bool (*check_kfunc_call)(u32 kfunc_btf_id, struct module *owner); }; struct bpf_prog_offload_ops { @@ -874,6 +874,7 @@ struct bpf_prog_aux { void *jit_data; /* JIT specific data. arch dependent */ struct bpf_jit_poke_descriptor *poke_tab; struct bpf_kfunc_desc_tab *kfunc_tab; + struct bpf_kfunc_btf_tab *kfunc_btf_tab; u32 size_poke_tab; struct bpf_ksym ksym; const struct bpf_prog_ops *ops; @@ -1635,7 +1636,7 @@ int bpf_prog_test_run_raw_tp(struct bpf_prog *prog, int bpf_prog_test_run_sk_lookup(struct bpf_prog *prog, const union bpf_attr *kattr, union bpf_attr __user *uattr); -bool bpf_prog_test_check_kfunc_call(u32 kfunc_id); +bool bpf_prog_test_check_kfunc_call(u32 kfunc_id, struct module *owner); bool btf_ctx_access(int off, int size, enum bpf_access_type type, const struct bpf_prog *prog, struct bpf_insn_access_aux *info); @@ -1856,7 +1857,8 @@ static inline int bpf_prog_test_run_sk_lookup(struct bpf_prog *prog, return -ENOTSUPP; } -static inline bool bpf_prog_test_check_kfunc_call(u32 kfunc_id) +static inline bool bpf_prog_test_check_kfunc_call(u32 kfunc_id, + struct module *owner) { return false; } diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 5424124dbe36..c8a78e830fca 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -527,5 +527,7 @@ int bpf_check_attach_target(struct bpf_verifier_log *log, const struct bpf_prog *tgt_prog, u32 btf_id, struct bpf_attach_target_info *tgt_info); +void bpf_free_kfunc_btf_tab(struct bpf_kfunc_btf_tab *tab); + #endif /* _LINUX_BPF_VERIFIER_H */ diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 9f4636d021b1..8fe14942e6b6 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -32,6 +32,7 @@ #include #include #include +#include #include #include @@ -2255,6 +2256,9 @@ static void bpf_prog_free_deferred(struct work_struct *work) int i; aux = container_of(work, struct bpf_prog_aux, work); +#ifdef CONFIG_BPF_SYSCALL + bpf_free_kfunc_btf_tab(aux->kfunc_btf_tab); +#endif bpf_free_used_maps(aux); bpf_free_used_btfs(aux); if (bpf_prog_is_dev_bound(aux)) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 047ac4b4703b..3a35af7d1180 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -1626,52 +1626,170 @@ static int add_subprog(struct bpf_verifier_env *env, int off) return env->subprog_cnt - 1; } +#define MAX_KFUNC_DESCS 256 +#define MAX_KFUNC_BTFS 256 + struct bpf_kfunc_desc { struct btf_func_model func_model; u32 func_id; s32 imm; + u16 offset; +}; + +struct bpf_kfunc_btf { + struct btf *btf; + struct module *module; + u16 offset; }; -#define MAX_KFUNC_DESCS 256 struct bpf_kfunc_desc_tab { struct bpf_kfunc_desc descs[MAX_KFUNC_DESCS]; u32 nr_descs; }; -static int kfunc_desc_cmp_by_id(const void *a, const void *b) +struct bpf_kfunc_btf_tab { + struct bpf_kfunc_btf descs[MAX_KFUNC_BTFS]; + u32 nr_descs; +}; + +static int kfunc_desc_cmp_by_id_off(const void *a, const void *b) { const struct bpf_kfunc_desc *d0 = a; const struct bpf_kfunc_desc *d1 = b; /* func_id is not greater than BTF_MAX_TYPE */ - return d0->func_id - d1->func_id; + return d0->func_id - d1->func_id ?: d0->offset - d1->offset; +} + +static int kfunc_btf_cmp_by_off(const void *a, const void *b) +{ + const struct bpf_kfunc_btf *d0 = a; + const struct bpf_kfunc_btf *d1 = b; + + return d0->offset - d1->offset; } static const struct bpf_kfunc_desc * -find_kfunc_desc(const struct bpf_prog *prog, u32 func_id) +find_kfunc_desc(const struct bpf_prog *prog, u32 func_id, u16 offset) { struct bpf_kfunc_desc desc = { .func_id = func_id, + .offset = offset, }; struct bpf_kfunc_desc_tab *tab; tab = prog->aux->kfunc_tab; return bsearch(&desc, tab->descs, tab->nr_descs, - sizeof(tab->descs[0]), kfunc_desc_cmp_by_id); + sizeof(tab->descs[0]), kfunc_desc_cmp_by_id_off); +} + +/* Expects offset always greater than 0 */ +static struct btf *__find_kfunc_desc_btf(struct bpf_verifier_env *env, + s16 offset, struct module **btf_modp) +{ + struct bpf_kfunc_btf kf_btf = { .offset = offset - 1 }; + struct bpf_kfunc_btf_tab *tab; + struct module *mod = NULL; + struct bpf_kfunc_btf *b; + struct btf *btf; + int btf_fd; + + tab = env->prog->aux->kfunc_btf_tab; + b = bsearch(&kf_btf, tab->descs, tab->nr_descs, + sizeof(tab->descs[0]), kfunc_btf_cmp_by_off); + if (!b) { + if (tab->nr_descs == MAX_KFUNC_BTFS) { + verbose(env, "too many different module BTFs\n"); + return ERR_PTR(-E2BIG); + } + + offset -= 1; + if (copy_from_bpfptr_offset(&btf_fd, env->fd_array, + offset * sizeof(btf_fd), + sizeof(btf_fd))) + return ERR_PTR(-EFAULT); + + btf = btf_get_by_fd(btf_fd); + if (IS_ERR(btf)) + return btf; + + if (!btf_is_module(btf)) { + verbose(env, "BTF fd for kfunc is not a module BTF\n"); + btf_put(btf); + return ERR_PTR(-EINVAL); + } + + mod = btf_try_get_module(btf); + if (!mod) { + btf_put(btf); + return ERR_PTR(-ENXIO); + } + + b = &tab->descs[tab->nr_descs++]; + b->btf = btf; + b->module = mod; + b->offset = offset; + + sort(tab->descs, tab->nr_descs, sizeof(tab->descs[0]), + kfunc_btf_cmp_by_off, NULL); + } + if (btf_modp) + *btf_modp = b->module; + return b->btf; +} + +void bpf_free_kfunc_btf_tab(struct bpf_kfunc_btf_tab *tab) +{ + if (!tab) + return; + + while (tab->nr_descs--) { + module_put(tab->descs[tab->nr_descs].module); + btf_put(tab->descs[tab->nr_descs].btf); + } + kfree(tab); +} + +static struct btf *find_kfunc_desc_btf(struct bpf_verifier_env *env, + u32 func_id, s16 offset, + struct module **btf_modp) +{ + struct btf *kfunc_btf; + + if (offset) { + if (offset < 0) { + /* In the future, this can be allowed to increase limit + * of fd index into fd_array, interpreted as unsigned u16. + */ + verbose(env, "negative offset disallowed for kernel module function call\n"); + return ERR_PTR(-EINVAL); + } + + kfunc_btf = __find_kfunc_desc_btf(env, offset, btf_modp); + if (IS_ERR_OR_NULL(kfunc_btf)) { + verbose(env, "cannot find module BTF for func_id %u\n", func_id); + return kfunc_btf ?: ERR_PTR(-ENOENT); + } + return kfunc_btf; + } + return btf_vmlinux ?: ERR_PTR(-ENOENT); } -static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id) +static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset) { const struct btf_type *func, *func_proto; + struct bpf_kfunc_btf_tab *btf_tab; struct bpf_kfunc_desc_tab *tab; struct bpf_prog_aux *prog_aux; struct bpf_kfunc_desc *desc; const char *func_name; + struct btf *desc_btf; unsigned long addr; int err; prog_aux = env->prog->aux; tab = prog_aux->kfunc_tab; + btf_tab = prog_aux->kfunc_btf_tab; if (!tab) { if (!btf_vmlinux) { verbose(env, "calling kernel function is not supported without CONFIG_DEBUG_INFO_BTF\n"); @@ -1699,7 +1817,20 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id) prog_aux->kfunc_tab = tab; } - if (find_kfunc_desc(env->prog, func_id)) + if (!btf_tab && offset) { + btf_tab = kzalloc(sizeof(*btf_tab), GFP_KERNEL); + if (!btf_tab) + return -ENOMEM; + prog_aux->kfunc_btf_tab = btf_tab; + } + + desc_btf = find_kfunc_desc_btf(env, func_id, offset, NULL); + if (IS_ERR(desc_btf)) { + verbose(env, "failed to find BTF for kernel function\n"); + return PTR_ERR(desc_btf); + } + + if (find_kfunc_desc(env->prog, func_id, offset)) return 0; if (tab->nr_descs == MAX_KFUNC_DESCS) { @@ -1707,20 +1838,20 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id) return -E2BIG; } - func = btf_type_by_id(btf_vmlinux, func_id); + func = btf_type_by_id(desc_btf, func_id); if (!func || !btf_type_is_func(func)) { verbose(env, "kernel btf_id %u is not a function\n", func_id); return -EINVAL; } - func_proto = btf_type_by_id(btf_vmlinux, func->type); + func_proto = btf_type_by_id(desc_btf, func->type); if (!func_proto || !btf_type_is_func_proto(func_proto)) { verbose(env, "kernel function btf_id %u does not have a valid func_proto\n", func_id); return -EINVAL; } - func_name = btf_name_by_offset(btf_vmlinux, func->name_off); + func_name = btf_name_by_offset(desc_btf, func->name_off); addr = kallsyms_lookup_name(func_name); if (!addr) { verbose(env, "cannot find address for kernel function %s\n", @@ -1730,13 +1861,14 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id) desc = &tab->descs[tab->nr_descs++]; desc->func_id = func_id; + desc->offset = offset; desc->imm = BPF_CAST_CALL(addr) - __bpf_call_base; - err = btf_distill_func_proto(&env->log, btf_vmlinux, + err = btf_distill_func_proto(&env->log, desc_btf, func_proto, func_name, &desc->func_model); if (!err) sort(tab->descs, tab->nr_descs, sizeof(tab->descs[0]), - kfunc_desc_cmp_by_id, NULL); + kfunc_desc_cmp_by_id_off, NULL); return err; } @@ -1815,7 +1947,7 @@ static int add_subprog_and_kfunc(struct bpf_verifier_env *env) } else if (bpf_pseudo_call(insn)) { ret = add_subprog(env, i + insn->imm + 1); } else { - ret = add_kfunc_call(env, insn->imm); + ret = add_kfunc_call(env, insn->imm, insn->off); } if (ret < 0) @@ -2152,12 +2284,17 @@ static int get_prev_insn_idx(struct bpf_verifier_state *st, int i, static const char *disasm_kfunc_name(void *data, const struct bpf_insn *insn) { const struct btf_type *func; + struct btf *desc_btf; if (insn->src_reg != BPF_PSEUDO_KFUNC_CALL) return NULL; - func = btf_type_by_id(btf_vmlinux, insn->imm); - return btf_name_by_offset(btf_vmlinux, func->name_off); + desc_btf = find_kfunc_desc_btf(data, insn->imm, insn->off, NULL); + if (IS_ERR(desc_btf)) + return ""; + + func = btf_type_by_id(desc_btf, insn->imm); + return btf_name_by_offset(desc_btf, func->name_off); } /* For given verifier state backtrack_insn() is called from the last insn to @@ -6485,23 +6622,29 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn) struct bpf_reg_state *regs = cur_regs(env); const char *func_name, *ptr_type_name; u32 i, nargs, func_id, ptr_type_id; + struct module *btf_mod = NULL; const struct btf_param *args; + struct btf *desc_btf; int err; + desc_btf = find_kfunc_desc_btf(env, insn->imm, insn->off, &btf_mod); + if (IS_ERR(desc_btf)) + return PTR_ERR(desc_btf); + func_id = insn->imm; - func = btf_type_by_id(btf_vmlinux, func_id); - func_name = btf_name_by_offset(btf_vmlinux, func->name_off); - func_proto = btf_type_by_id(btf_vmlinux, func->type); + func = btf_type_by_id(desc_btf, func_id); + func_name = btf_name_by_offset(desc_btf, func->name_off); + func_proto = btf_type_by_id(desc_btf, func->type); if (!env->ops->check_kfunc_call || - !env->ops->check_kfunc_call(func_id)) { + !env->ops->check_kfunc_call(func_id, btf_mod)) { verbose(env, "calling kernel function %s is not allowed\n", func_name); return -EACCES; } /* Check the arguments */ - err = btf_check_kfunc_arg_match(env, btf_vmlinux, func_id, regs); + err = btf_check_kfunc_arg_match(env, desc_btf, func_id, regs); if (err) return err; @@ -6509,15 +6652,15 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn) mark_reg_not_init(env, regs, caller_saved[i]); /* Check return type */ - t = btf_type_skip_modifiers(btf_vmlinux, func_proto->type, NULL); + t = btf_type_skip_modifiers(desc_btf, func_proto->type, NULL); if (btf_type_is_scalar(t)) { mark_reg_unknown(env, regs, BPF_REG_0); mark_btf_func_reg_size(env, BPF_REG_0, t->size); } else if (btf_type_is_ptr(t)) { - ptr_type = btf_type_skip_modifiers(btf_vmlinux, t->type, + ptr_type = btf_type_skip_modifiers(desc_btf, t->type, &ptr_type_id); if (!btf_type_is_struct(ptr_type)) { - ptr_type_name = btf_name_by_offset(btf_vmlinux, + ptr_type_name = btf_name_by_offset(desc_btf, ptr_type->name_off); verbose(env, "kernel function %s returns pointer type %s %s is not supported\n", func_name, btf_type_str(ptr_type), @@ -6525,7 +6668,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn) return -EINVAL; } mark_reg_known_zero(env, regs, BPF_REG_0); - regs[BPF_REG_0].btf = btf_vmlinux; + regs[BPF_REG_0].btf = desc_btf; regs[BPF_REG_0].type = PTR_TO_BTF_ID; regs[BPF_REG_0].btf_id = ptr_type_id; mark_btf_func_reg_size(env, BPF_REG_0, sizeof(void *)); @@ -6536,7 +6679,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn) for (i = 0; i < nargs; i++) { u32 regno = i + 1; - t = btf_type_skip_modifiers(btf_vmlinux, args[i].type, NULL); + t = btf_type_skip_modifiers(desc_btf, args[i].type, NULL); if (btf_type_is_ptr(t)) mark_btf_func_reg_size(env, regno, sizeof(void *)); else @@ -11074,7 +11217,8 @@ static int do_check(struct bpf_verifier_env *env) env->jmps_processed++; if (opcode == BPF_CALL) { if (BPF_SRC(insn->code) != BPF_K || - insn->off != 0 || + (insn->src_reg != BPF_PSEUDO_KFUNC_CALL + && insn->off != 0) || (insn->src_reg != BPF_REG_0 && insn->src_reg != BPF_PSEUDO_CALL && insn->src_reg != BPF_PSEUDO_KFUNC_CALL) || @@ -12430,6 +12574,7 @@ static int jit_subprogs(struct bpf_verifier_env *env) func[i]->aux->stack_depth = env->subprog_info[i].stack_depth; func[i]->jit_requested = 1; func[i]->aux->kfunc_tab = prog->aux->kfunc_tab; + func[i]->aux->kfunc_btf_tab = prog->aux->kfunc_btf_tab; func[i]->aux->linfo = prog->aux->linfo; func[i]->aux->nr_linfo = prog->aux->nr_linfo; func[i]->aux->jited_linfo = prog->aux->jited_linfo; @@ -12619,7 +12764,7 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, /* insn->imm has the btf func_id. Replace it with * an address (relative to __bpf_base_call). */ - desc = find_kfunc_desc(env->prog, insn->imm); + desc = find_kfunc_desc(env->prog, insn->imm, insn->off); if (!desc) { verbose(env, "verifier internal error: kernel function descriptor not found for func_id %u\n", insn->imm); diff --git a/net/bpf/test_run.c b/net/bpf/test_run.c index fcb2f493f710..fe5c34f414a2 100644 --- a/net/bpf/test_run.c +++ b/net/bpf/test_run.c @@ -241,7 +241,7 @@ BTF_ID(func, bpf_kfunc_call_test2) BTF_ID(func, bpf_kfunc_call_test3) BTF_SET_END(test_sk_kfunc_ids) -bool bpf_prog_test_check_kfunc_call(u32 kfunc_id) +bool bpf_prog_test_check_kfunc_call(u32 kfunc_id, struct module *owner) { return btf_id_set_contains(&test_sk_kfunc_ids, kfunc_id); } diff --git a/net/ipv4/bpf_tcp_ca.c b/net/ipv4/bpf_tcp_ca.c index 0dcee9df1326..b3afd3361f34 100644 --- a/net/ipv4/bpf_tcp_ca.c +++ b/net/ipv4/bpf_tcp_ca.c @@ -255,7 +255,7 @@ BTF_ID(func, bbr_set_state) #endif /* CONFIG_X86 */ BTF_SET_END(bpf_tcp_ca_kfunc_ids) -static bool bpf_tcp_ca_check_kfunc_call(u32 kfunc_btf_id) +static bool bpf_tcp_ca_check_kfunc_call(u32 kfunc_btf_id, struct module *owner) { return btf_id_set_contains(&bpf_tcp_ca_kfunc_ids, kfunc_btf_id); } From patchwork Wed Sep 15 05:09:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 512807 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F887C433F5 for ; Wed, 15 Sep 2021 05:10:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 207F161211 for ; Wed, 15 Sep 2021 05:10:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236228AbhIOFLT (ORCPT ); Wed, 15 Sep 2021 01:11:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44334 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235806AbhIOFLQ (ORCPT ); Wed, 15 Sep 2021 01:11:16 -0400 Received: from mail-pj1-x1041.google.com (mail-pj1-x1041.google.com [IPv6:2607:f8b0:4864:20::1041]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1605AC061764; Tue, 14 Sep 2021 22:09:58 -0700 (PDT) Received: by mail-pj1-x1041.google.com with SMTP id t20so1300732pju.5; Tue, 14 Sep 2021 22:09:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=hFzji/ML/N9X7kNxTQiBpHNfDWFkWIIw8KLpiz32rk8=; b=e34w/aHHbi2jcnvwKIFDFkA6Ex/XypyClH99JcK8DgI7HS5BlWqN2JGSp9oezqGZls UCHvZ2KkHzhg4CXX9ixT6zJbUeheKsQ+koESe/ImujzG0SvTerJ3BfmyvUwMOB4GSTbx RYXtRkkhDvZczZoNIVaZkd7e187PttTXBTljB/+ox/3lDJ7GT78oQkaUgbIB+dnbcCKg JZMAPhIGynbWGMjYa+8FPutBvmVevzgJ4+LXI2wMKjGa7Cnjbst2KUGIXrFqePB0rT7k qtKmqT/bbvZC2VpldEKc4j8uwGyrfimCGSWD5GiIEza9izn1oniNJZKnerD2gWbOaTee PncQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=hFzji/ML/N9X7kNxTQiBpHNfDWFkWIIw8KLpiz32rk8=; b=cu1BzTcctAAgv4hJUwl5EfSuQGmJX0FudGBEI5Ol1gQYoFJCC5vcBmaphRUZm67KNk YG1B4K1DEwOMsfXPI+lRCclIESu1oJ+762QxmSVjePKZ6+kOkPFHgwcHXp7XK9qq/oWq Hnm4T8GzYe+2km1SJJ2RqVO7ve3g6QoxPHHvQP6jiVDnh84pYzykxvptZ7Y1U/3OFMKx jkKm1IWrZaf6nXC6kylfQuC2NwKX/mhABjsHXzpbubecajjNSURX3SyxmIGe09r/3pOc TRuuJ4rD/xY6tKErnhCZZH/Wc7BnxOnNEd1+ytcbcpr8MxHv7XnunAESC3TPTu/WR23q qd3A== X-Gm-Message-State: AOAM530muYjgJV8BvE2k+rhBjzJmtHZ71PxV4PK1hClut7IqZ8A+zvEv ivXA7r86t0sRmOxqkGIx6oCfbWzFw8y4nw== X-Google-Smtp-Source: ABdhPJzw0xYjPBfNApyZ2eL4wGAU9wSKeHhjLUviaFwr2CX6tlN94m2HzLg3nv3CdCIkG4PxwBFVuw== X-Received: by 2002:a17:90b:383:: with SMTP id ga3mr6294477pjb.72.1631682597479; Tue, 14 Sep 2021 22:09:57 -0700 (PDT) Received: from localhost ([2405:201:6014:d058:a28d:3909:6ed5:29e7]) by smtp.gmail.com with ESMTPSA id b20sm7572292pfp.26.2021.09.14.22.09.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 22:09:57 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Kumar Kartikeya Dwivedi , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , Jesper Dangaard Brouer , =?utf-8?q?Toke_H=C3=B8iland-J?= =?utf-8?b?w7hyZ2Vuc2Vu?= , netdev@vger.kernel.org Subject: [PATCH bpf-next v3 03/10] bpf: btf: Introduce helpers for dynamic BTF set registration Date: Wed, 15 Sep 2021 10:39:36 +0530 Message-Id: <20210915050943.679062-4-memxor@gmail.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20210915050943.679062-1-memxor@gmail.com> References: <20210915050943.679062-1-memxor@gmail.com> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5061; h=from:subject; bh=m8fXSGEm0/XSIbkC12+3PP6uKRh1g+Ip6wRNtdVKH7g=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBhQX4bKtLF9C/vBW3tp+ns5ExsjPWzaQ06bPFUtoxh r1W8J3aJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCYUF+GwAKCRBM4MiGSL8Ryg27D/ 9PcJ+gtbxuCZcjVgqQqCzVfS20iicP+Omr7PIbePMWq1kPFBUGvfgY2EHqGommShDTQwmgYdWv8qoZ W0b5UplycSu2D2qV/0e3JaH/8hZzBpWpbok2GSBId82RbzJwAD/m+F6hyU5Lpz5Ehv5hac5djtalXm kNyUwrd80eMkLxRoaKVy1upAYFYR4kT184YVkzUM9gKCkeNdO496Fa+ULDnqBvkSuGfkNBx3EnFoPI dKXSF77+A/OrHU3xbnqOJjRiAXvX4OUAS0S/amlLIUa0qxll3+tXe80vgfKftSCBLad37quutVpoW6 H8iIxFrhhhVhLr7sad14NFLfmoJyAayusMMD2HrulqaDvldhzAXVs6o4lHEy6dO3oh7rwJyFhoWgVh DKWLB5uOc4lfAV+KTEf5hmTu0dbwaaFAqTbz+TIc8/YPkci7ajw/aSfcxfvFJKBTL17QM2p+XfravN 91q29umf0GBMFMbKj5+/gkLzRkO2MQOPfAmM3jseKJ640edZXnTH17KtarThEyedzdDtxla4/Eq2Hl Pw8aEvAoCZgax7Bxx5ZBrnUaSj5DLSW3xs38KxaEJuWcEqfojvu+FpQnJQx8NpQMA5uSIOnkon8tBX craTGG/PWbN/aM5wboRhqMtknoS6XgrTerSXvmvZZQsP7txuPr5Aw50rLkGw== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org This adds helpers for registering btf_id_set from modules and the check_kfunc_call callback that can be used to look them up. With in kernel sets, the way this is supposed to work is, in kernel callback looks up within the in-kernel kfunc whitelist, and then defers to the dynamic BTF set lookup if it doesn't find the BTF id. If there is no in-kernel BTF id set, this callback can be used directly. Also fix includes for btf.h and bpfptr.h so that they can included in isolation. This is in preparation for their usage in tcp_bbr, tcp_cubic and tcp_dctcp modules in the next patch. Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpfptr.h | 1 + include/linux/btf.h | 32 ++++++++++++++++++++++++++ kernel/bpf/btf.c | 51 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 84 insertions(+) diff --git a/include/linux/bpfptr.h b/include/linux/bpfptr.h index 546e27fc6d46..46e1757d06a3 100644 --- a/include/linux/bpfptr.h +++ b/include/linux/bpfptr.h @@ -3,6 +3,7 @@ #ifndef _LINUX_BPFPTR_H #define _LINUX_BPFPTR_H +#include #include typedef sockptr_t bpfptr_t; diff --git a/include/linux/btf.h b/include/linux/btf.h index 214fde93214b..e29a486d09d4 100644 --- a/include/linux/btf.h +++ b/include/linux/btf.h @@ -5,8 +5,10 @@ #define _LINUX_BTF_H 1 #include +#include #include #include +#include #define BTF_TYPE_EMIT(type) ((void)(type *)0) #define BTF_TYPE_EMIT_ENUM(enum_val) ((void)enum_val) @@ -238,4 +240,34 @@ static inline const char *btf_name_by_offset(const struct btf *btf, } #endif +struct kfunc_btf_id_set { + struct list_head list; + struct btf_id_set *set; + struct module *owner; +}; + +struct kfunc_btf_id_list; + +#ifdef CONFIG_DEBUG_INFO_BTF_MODULES +void register_kfunc_btf_id_set(struct kfunc_btf_id_list *l, + struct kfunc_btf_id_set *s); +void unregister_kfunc_btf_id_set(struct kfunc_btf_id_list *l, + struct kfunc_btf_id_set *s); +#else +static inline void register_kfunc_btf_id_set(struct kfunc_btf_id_list *l, + struct kfunc_btf_id_set *s) +{ +} +static inline void unregister_kfunc_btf_id_set(struct kfunc_btf_id_list *l, + struct kfunc_btf_id_set *s) +{ +} +#endif + +#define DECLARE_CHECK_KFUNC_CALLBACK(type) \ + bool __bpf_check_##type##_kfunc_call(u32 kfunc_id, struct module *owner) +#define DEFINE_KFUNC_BTF_ID_SET(set, name) \ + struct kfunc_btf_id_set name = { LIST_HEAD_INIT(name.list), (set), \ + THIS_MODULE } + #endif diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index c3d605b22473..d17f45b163f5 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -6343,3 +6343,54 @@ const struct bpf_func_proto bpf_btf_find_by_name_kind_proto = { }; BTF_ID_LIST_GLOBAL_SINGLE(btf_task_struct_ids, struct, task_struct) + +struct kfunc_btf_id_list { + struct list_head list; + struct mutex mutex; +}; + +#ifdef CONFIG_DEBUG_INFO_BTF_MODULES + +void register_kfunc_btf_id_set(struct kfunc_btf_id_list *l, + struct kfunc_btf_id_set *s) +{ + mutex_lock(&l->mutex); + list_add(&s->list, &l->list); + mutex_unlock(&l->mutex); +} +EXPORT_SYMBOL_GPL(register_kfunc_btf_id_set); + +void unregister_kfunc_btf_id_set(struct kfunc_btf_id_list *l, + struct kfunc_btf_id_set *s) +{ + mutex_lock(&l->mutex); + list_del_init(&s->list); + mutex_unlock(&l->mutex); +} +EXPORT_SYMBOL_GPL(unregister_kfunc_btf_id_set); + +#endif + +#define DEFINE_KFUNC_BTF_ID_LIST(name) \ + struct kfunc_btf_id_list name = { LIST_HEAD_INIT(name.list), \ + __MUTEX_INITIALIZER(name.mutex) }; \ + EXPORT_SYMBOL_GPL(name) + +#define DEFINE_CHECK_KFUNC_CALLBACK(type, list_name) \ + bool __bpf_check_##type##_kfunc_call(u32 kfunc_id, \ + struct module *owner) \ + { \ + struct kfunc_btf_id_set *s; \ + if (!owner || !IS_ENABLED(CONFIG_DEBUG_INFO_BTF_MODULES)) \ + return false; \ + mutex_lock(&list_name.mutex); \ + list_for_each_entry(s, &list_name.list, list) { \ + if (s->owner == owner && \ + btf_id_set_contains(s->set, kfunc_id)) { \ + mutex_unlock(&list_name.mutex); \ + return true; \ + } \ + } \ + mutex_unlock(&list_name.mutex); \ + return false; \ + } From patchwork Wed Sep 15 05:09:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 512806 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43268C433EF for ; Wed, 15 Sep 2021 05:10:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2ACFA6121E for ; Wed, 15 Sep 2021 05:10:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236344AbhIOFLc (ORCPT ); Wed, 15 Sep 2021 01:11:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44374 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236292AbhIOFLW (ORCPT ); Wed, 15 Sep 2021 01:11:22 -0400 Received: from mail-pg1-x542.google.com (mail-pg1-x542.google.com [IPv6:2607:f8b0:4864:20::542]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4210EC0613D8; Tue, 14 Sep 2021 22:10:04 -0700 (PDT) Received: by mail-pg1-x542.google.com with SMTP id g184so1541040pgc.6; Tue, 14 Sep 2021 22:10:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yGr9iACuV6BoX7TnMjNVhLVmAfdKWT67ERnimSMzQJU=; b=Ozv2Y5fzYZgdIGcpfERY6WJdVgMmuXZ7g+xnpHKn1YLd/yUrD+mGYsKEIUSsCzTe67 srukjThS64/e7j6pVKz3BqKDRzcJEonFSB+Xo1nKn1Aa56TA+CRErXs6GwAFPYf2pivB HLEz099ID4X0+77Z0F4exxKUT1ypm9NL55UGxF227CmTxXsDvLBg12/5drvdq3kNwlsx v4FUeCoIQgeKPos2IGEIlhuXn5TMpfQL4dOdIeL6rTso9UjwVohFnIOM/eVhnxVAd/GU RaMXIX6wn0dzEAtZsok9CwK/j7SnYBuRzAWLjx8sk+vqjF0O77KXZfcctWE4KmGUeN4/ 9kXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yGr9iACuV6BoX7TnMjNVhLVmAfdKWT67ERnimSMzQJU=; b=U/4uCPiw3iMB5RwaXS2SzNjb7XrfRMEkE9MUIf1rc4IGz09+osPUhUvxnojf3lqBRD DC/sshpzhhgFFXSzN2jIwMPg25GcAuVY0szVxVagJwo8MFtAFEDIjv06AUZUvrlQEUxb f74A9a9f8/FPSqXKcQpTmHr5lVtj3PAQvZwcJty0WhNS8+50MafRcAqR/cA0//cFJ/H9 fABa5bUutt8w6a43lR4RHFkaEGqDJMymlwGre98seFKZxtTVT2IVOv+jwhzTkiYmagpO VNtHQKIb+5tijOBHW2vqOjThXbL5wqlaiNI3B6MjhHJsLIWaFIBHvsdiSeujNQXNvJzX 6dxw== X-Gm-Message-State: AOAM531XVADEI1rrafdVL4WnyC4K4c8eLbYLNeFrV0GC3oV4GeUmW0lp Xh1RMlQYjBa0rsRlkyYSEBEIeGRckm6HZg== X-Google-Smtp-Source: ABdhPJxAnUGtrD3wPIooUxYyMXP6vXIdPNFc2+ssd2gvZfgMBsrMPfQ6+8NpGl3Z24mVVfZDQ8rVGg== X-Received: by 2002:a05:6a00:a23:b0:43d:e856:a3d4 with SMTP id p35-20020a056a000a2300b0043de856a3d4mr8530023pfh.17.1631682603571; Tue, 14 Sep 2021 22:10:03 -0700 (PDT) Received: from localhost ([2405:201:6014:d058:a28d:3909:6ed5:29e7]) by smtp.gmail.com with ESMTPSA id z17sm3627633pfh.110.2021.09.14.22.10.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 22:10:03 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Kumar Kartikeya Dwivedi , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , Jesper Dangaard Brouer , =?utf-8?q?Toke_H=C3=B8iland-J?= =?utf-8?b?w7hyZ2Vuc2Vu?= , netdev@vger.kernel.org Subject: [PATCH bpf-next v3 05/10] bpf: Enable TCP congestion control kfunc from modules Date: Wed, 15 Sep 2021 10:39:38 +0530 Message-Id: <20210915050943.679062-6-memxor@gmail.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20210915050943.679062-1-memxor@gmail.com> References: <20210915050943.679062-1-memxor@gmail.com> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=9235; h=from:subject; bh=DKtrsf0TcthrgG/VRKnYyF97AuWzH3768PxQ2CQccBU=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBhQX4bYQ953eMkGd8AAWtVAvZlgF+ZL5LsDP3U9Sn7 9pFqXuOJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCYUF+GwAKCRBM4MiGSL8RyjE/D/ 90GmI0Ze83yFMpQ9D65HQYlsHtfjbC6UrzGeqqYmsSsrs0f71jxUFi9v3EmUlXmdOYMwK+g6c/7Jsk IxNsp1uT9OqVqdhrnoorivSHUMk0S2uKDaz3jDlZGoGh/TjFmxLLM80MF+3mhwB03F3g+rKjWVIuIR t+k5GMVvMwx0s0Mq2tv8GtQTRkdDDyggVS7N1P/CjRr32CT92z9NqV/JEubmpvtLzrUwOHG9wWISWp u8yweEWITwERxRrLuEfx+g0NWW5d0ueWcFoHDFJKBxe9vjxqgVoBRL4l7FljMlaZMDzTA9AJJACglr IpCg6RRShgbfvODT4Tqtk+ZMVHrruze5GCOO0AEX8qTS6///j7upMaRF3uqLD7tNctp92kQX1Gfbro vDpj0WsWyMD9iPcU770hD+gKb0D7bN9cDd5LA9zQf1BAs8UDViKI9M8TvUBGDc63GyKQPC7PguMW4y FL+mpi/Yrbr7j0bYjaH0TCnpibbo2lD9KVeWeDYAliq8jJBB76+OWx4UBTuO9Mn44MKm/fJw863MrS /fecENvmECwKA7XT0/LuS/U1xLiqC5bf/GtL00WcOJnx8teKAz0aLmK4Nj8STQ3SzHLbcGHuXxserr xDD+MTWKi2VBPOJaNk/qaNskWjmAJT4I7Zk7B7ZWjhfTjiZ4AabIpfvAwOug== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org This commit moves BTF ID lookup into the newly added registration helper, in a way that the bbr, cubic, and dctcp implementation set up their sets in the bpf_tcp_ca kfunc_btf_set list, while the ones not dependent on modules are looked up from the wrapper function. This lifts the restriction for them to be compiled as built in objects, and can be loaded as modules if required. Also modify Makefile.modfinal to resolve_btfids in TCP congestion control modules if the config option is set, using the base BTF support added in the previous commit. See following commits for background on use of: CONFIG_X86 ifdef: 569c484f9995 (bpf: Limit static tcp-cc functions in the .BTF_ids list to x86) CONFIG_DYNAMIC_FTRACE ifdef: 7aae231ac93b (bpf: tcp: Limit calling some tcp cc functions to CONFIG_DYNAMIC_FTRACE) [ resolve_btfids uses --no-fail because some crypto kernel modules under arch/x86/crypto generated from ASM do not have the .BTF sections ] Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/btf.h | 4 ++++ kernel/bpf/btf.c | 3 +++ net/ipv4/bpf_tcp_ca.c | 34 +++------------------------------- net/ipv4/tcp_bbr.c | 28 +++++++++++++++++++++++++++- net/ipv4/tcp_cubic.c | 26 +++++++++++++++++++++++++- net/ipv4/tcp_dctcp.c | 26 +++++++++++++++++++++++++- scripts/Makefile.modfinal | 1 + 7 files changed, 88 insertions(+), 34 deletions(-) diff --git a/include/linux/btf.h b/include/linux/btf.h index e29a486d09d4..c7b6382123e1 100644 --- a/include/linux/btf.h +++ b/include/linux/btf.h @@ -270,4 +270,8 @@ static inline void unregister_kfunc_btf_id_set(struct kfunc_btf_id_list *l, struct kfunc_btf_id_set name = { LIST_HEAD_INIT(name.list), (set), \ THIS_MODULE } +extern struct kfunc_btf_id_list bpf_tcp_ca_kfunc_list; + +DECLARE_CHECK_KFUNC_CALLBACK(bpf_tcp_ca); + #endif diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index d17f45b163f5..671b4f713a51 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -6394,3 +6394,6 @@ EXPORT_SYMBOL_GPL(unregister_kfunc_btf_id_set); mutex_unlock(&list_name.mutex); \ return false; \ } + +DEFINE_KFUNC_BTF_ID_LIST(bpf_tcp_ca_kfunc_list); +DEFINE_CHECK_KFUNC_CALLBACK(bpf_tcp_ca, bpf_tcp_ca_kfunc_list); diff --git a/net/ipv4/bpf_tcp_ca.c b/net/ipv4/bpf_tcp_ca.c index b3afd3361f34..c9f1d2dcf67b 100644 --- a/net/ipv4/bpf_tcp_ca.c +++ b/net/ipv4/bpf_tcp_ca.c @@ -223,41 +223,13 @@ BTF_ID(func, tcp_reno_cong_avoid) BTF_ID(func, tcp_reno_undo_cwnd) BTF_ID(func, tcp_slow_start) BTF_ID(func, tcp_cong_avoid_ai) -#ifdef CONFIG_X86 -#ifdef CONFIG_DYNAMIC_FTRACE -#if IS_BUILTIN(CONFIG_TCP_CONG_CUBIC) -BTF_ID(func, cubictcp_init) -BTF_ID(func, cubictcp_recalc_ssthresh) -BTF_ID(func, cubictcp_cong_avoid) -BTF_ID(func, cubictcp_state) -BTF_ID(func, cubictcp_cwnd_event) -BTF_ID(func, cubictcp_acked) -#endif -#if IS_BUILTIN(CONFIG_TCP_CONG_DCTCP) -BTF_ID(func, dctcp_init) -BTF_ID(func, dctcp_update_alpha) -BTF_ID(func, dctcp_cwnd_event) -BTF_ID(func, dctcp_ssthresh) -BTF_ID(func, dctcp_cwnd_undo) -BTF_ID(func, dctcp_state) -#endif -#if IS_BUILTIN(CONFIG_TCP_CONG_BBR) -BTF_ID(func, bbr_init) -BTF_ID(func, bbr_main) -BTF_ID(func, bbr_sndbuf_expand) -BTF_ID(func, bbr_undo_cwnd) -BTF_ID(func, bbr_cwnd_event) -BTF_ID(func, bbr_ssthresh) -BTF_ID(func, bbr_min_tso_segs) -BTF_ID(func, bbr_set_state) -#endif -#endif /* CONFIG_DYNAMIC_FTRACE */ -#endif /* CONFIG_X86 */ BTF_SET_END(bpf_tcp_ca_kfunc_ids) static bool bpf_tcp_ca_check_kfunc_call(u32 kfunc_btf_id, struct module *owner) { - return btf_id_set_contains(&bpf_tcp_ca_kfunc_ids, kfunc_btf_id); + if (btf_id_set_contains(&bpf_tcp_ca_kfunc_ids, kfunc_btf_id)) + return true; + return __bpf_check_bpf_tcp_ca_kfunc_call(kfunc_btf_id, owner); } static const struct bpf_verifier_ops bpf_tcp_ca_verifier_ops = { diff --git a/net/ipv4/tcp_bbr.c b/net/ipv4/tcp_bbr.c index 6274462b86b4..ec5550089b4d 100644 --- a/net/ipv4/tcp_bbr.c +++ b/net/ipv4/tcp_bbr.c @@ -56,6 +56,8 @@ * otherwise TCP stack falls back to an internal pacing using one high * resolution timer per TCP socket and may use more resources. */ +#include +#include #include #include #include @@ -1152,14 +1154,38 @@ static struct tcp_congestion_ops tcp_bbr_cong_ops __read_mostly = { .set_state = bbr_set_state, }; +BTF_SET_START(tcp_bbr_kfunc_ids) +#ifdef CONFIG_X86 +#ifdef CONFIG_DYNAMIC_FTRACE +BTF_ID(func, bbr_init) +BTF_ID(func, bbr_main) +BTF_ID(func, bbr_sndbuf_expand) +BTF_ID(func, bbr_undo_cwnd) +BTF_ID(func, bbr_cwnd_event) +BTF_ID(func, bbr_ssthresh) +BTF_ID(func, bbr_min_tso_segs) +BTF_ID(func, bbr_set_state) +#endif +#endif +BTF_SET_END(tcp_bbr_kfunc_ids) + +static DEFINE_KFUNC_BTF_ID_SET(&tcp_bbr_kfunc_ids, tcp_bbr_kfunc_btf_set); + static int __init bbr_register(void) { + int ret; + BUILD_BUG_ON(sizeof(struct bbr) > ICSK_CA_PRIV_SIZE); - return tcp_register_congestion_control(&tcp_bbr_cong_ops); + ret = tcp_register_congestion_control(&tcp_bbr_cong_ops); + if (ret) + return ret; + register_kfunc_btf_id_set(&bpf_tcp_ca_kfunc_list, &tcp_bbr_kfunc_btf_set); + return 0; } static void __exit bbr_unregister(void) { + unregister_kfunc_btf_id_set(&bpf_tcp_ca_kfunc_list, &tcp_bbr_kfunc_btf_set); tcp_unregister_congestion_control(&tcp_bbr_cong_ops); } diff --git a/net/ipv4/tcp_cubic.c b/net/ipv4/tcp_cubic.c index 4a30deaa9a37..5e9d9c51164c 100644 --- a/net/ipv4/tcp_cubic.c +++ b/net/ipv4/tcp_cubic.c @@ -25,6 +25,8 @@ */ #include +#include +#include #include #include #include @@ -482,8 +484,25 @@ static struct tcp_congestion_ops cubictcp __read_mostly = { .name = "cubic", }; +BTF_SET_START(tcp_cubic_kfunc_ids) +#ifdef CONFIG_X86 +#ifdef CONFIG_DYNAMIC_FTRACE +BTF_ID(func, cubictcp_init) +BTF_ID(func, cubictcp_recalc_ssthresh) +BTF_ID(func, cubictcp_cong_avoid) +BTF_ID(func, cubictcp_state) +BTF_ID(func, cubictcp_cwnd_event) +BTF_ID(func, cubictcp_acked) +#endif +#endif +BTF_SET_END(tcp_cubic_kfunc_ids) + +static DEFINE_KFUNC_BTF_ID_SET(&tcp_cubic_kfunc_ids, tcp_cubic_kfunc_btf_set); + static int __init cubictcp_register(void) { + int ret; + BUILD_BUG_ON(sizeof(struct bictcp) > ICSK_CA_PRIV_SIZE); /* Precompute a bunch of the scaling factors that are used per-packet @@ -514,11 +533,16 @@ static int __init cubictcp_register(void) /* divide by bic_scale and by constant Srtt (100ms) */ do_div(cube_factor, bic_scale * 10); - return tcp_register_congestion_control(&cubictcp); + ret = tcp_register_congestion_control(&cubictcp); + if (ret) + return ret; + register_kfunc_btf_id_set(&bpf_tcp_ca_kfunc_list, &tcp_cubic_kfunc_btf_set); + return 0; } static void __exit cubictcp_unregister(void) { + unregister_kfunc_btf_id_set(&bpf_tcp_ca_kfunc_list, &tcp_cubic_kfunc_btf_set); tcp_unregister_congestion_control(&cubictcp); } diff --git a/net/ipv4/tcp_dctcp.c b/net/ipv4/tcp_dctcp.c index 79f705450c16..0d7ab3cc7b61 100644 --- a/net/ipv4/tcp_dctcp.c +++ b/net/ipv4/tcp_dctcp.c @@ -36,6 +36,8 @@ * Glenn Judd */ +#include +#include #include #include #include @@ -236,14 +238,36 @@ static struct tcp_congestion_ops dctcp_reno __read_mostly = { .name = "dctcp-reno", }; +BTF_SET_START(tcp_dctcp_kfunc_ids) +#ifdef CONFIG_X86 +#ifdef CONFIG_DYNAMIC_FTRACE +BTF_ID(func, dctcp_init) +BTF_ID(func, dctcp_update_alpha) +BTF_ID(func, dctcp_cwnd_event) +BTF_ID(func, dctcp_ssthresh) +BTF_ID(func, dctcp_cwnd_undo) +BTF_ID(func, dctcp_state) +#endif +#endif +BTF_SET_END(tcp_dctcp_kfunc_ids) + +static DEFINE_KFUNC_BTF_ID_SET(&tcp_dctcp_kfunc_ids, tcp_dctcp_kfunc_btf_set); + static int __init dctcp_register(void) { + int ret; + BUILD_BUG_ON(sizeof(struct dctcp) > ICSK_CA_PRIV_SIZE); - return tcp_register_congestion_control(&dctcp); + ret = tcp_register_congestion_control(&dctcp); + if (ret) + return ret; + register_kfunc_btf_id_set(&bpf_tcp_ca_kfunc_list, &tcp_dctcp_kfunc_btf_set); + return 0; } static void __exit dctcp_unregister(void) { + unregister_kfunc_btf_id_set(&bpf_tcp_ca_kfunc_list, &tcp_dctcp_kfunc_btf_set); tcp_unregister_congestion_control(&dctcp); } diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal index ff805777431c..b4f83533eda6 100644 --- a/scripts/Makefile.modfinal +++ b/scripts/Makefile.modfinal @@ -41,6 +41,7 @@ quiet_cmd_btf_ko = BTF [M] $@ cmd_btf_ko = \ if [ -f vmlinux ]; then \ LLVM_OBJCOPY="$(OBJCOPY)" $(PAHOLE) -J --btf_base vmlinux $@; \ + $(RESOLVE_BTFIDS) --no-fail -s vmlinux $@; \ else \ printf "Skipping BTF generation for %s due to unavailability of vmlinux\n" $@ 1>&2; \ fi; From patchwork Wed Sep 15 05:09:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 512805 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF28BC433EF for ; Wed, 15 Sep 2021 05:10:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C452E61216 for ; Wed, 15 Sep 2021 05:10:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236367AbhIOFLi (ORCPT ); Wed, 15 Sep 2021 01:11:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44408 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230225AbhIOFL2 (ORCPT ); Wed, 15 Sep 2021 01:11:28 -0400 Received: from mail-pj1-x1043.google.com (mail-pj1-x1043.google.com [IPv6:2607:f8b0:4864:20::1043]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 12591C0613C1; Tue, 14 Sep 2021 22:10:10 -0700 (PDT) Received: by mail-pj1-x1043.google.com with SMTP id u13-20020a17090abb0db0290177e1d9b3f7so4057779pjr.1; Tue, 14 Sep 2021 22:10:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=unBtJ4BgEaQ1q3x2SN/FO8ndnAfXWb2rghCo8mQpwXY=; b=D6P6oDA86AfKIfKnehKa+l5GHGTK6+EDuk0ntifX18se3+S0iBsrCvk0ZH2CmZcFC2 UA/kSanaiPgw1PG9TSKaMiBrdEbDn6BqrA3VHJRdg65INrASSPE89guCGDJVuEhwBIOX nRrwhFyRHrIDakiTcsbjdSFmHeZKarsBmP1FofWlBDgKgugIrYBmGrC5PIgBdnZeTiay +5EGCe8hhQBrNt6JuYuODCHSYvkhh+0Pf8AYTR3cHZjXWQ+kaXE+vwK5zdhy8lQpFrih OzltjJxKwlo2QB8CwoKIg5DE5LKHuV7CQbTKQnlMkpWJrXlPFEEuwEU/h9fJwTSQFHXM gvyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=unBtJ4BgEaQ1q3x2SN/FO8ndnAfXWb2rghCo8mQpwXY=; b=wqXTkg0MkG8/EXqwTlMdNxiH3qc27zvgv1HubLsLj8cYy9aDAXsshgbFC+SJQwbEh1 q1MVa7BwIVhmHvbEw1eQyvTtbY0yN51lsP/pEqgpKbXXtVqWArPMELPuzu0Vi6lbBAEd 34eDuHCG3qusp/gYclx761TRiPgUKgn2I56N3qtkWM2ehmS0mA25XCjTTgjef+aQKdzu AzZaaf1K2W3lJsBscumYPlsD3PJDnqM3x0/oqa/pXuVj9bI0ivJQWbunoSIF/iRZ6OPo T0DHCX5B7OwVFkJKEpkEYu14Owk4WNDTSkg+EcsLb2WljpIXsH+188ULlvFyFaEKFy1r 2B4w== X-Gm-Message-State: AOAM533mwAhv9Cm6tOFw9dIHSQzwHyCdgPVkLBclgFmEooSi0o4TiNF/ 1wU6ZB1+K5E0sMx1GSUlvPmnb71LNXGJkw== X-Google-Smtp-Source: ABdhPJwsSwzOPidEO306IfobM6wvCAiBPGySaLKkPpjKlXuptz73EgZWBVDgHyczJI7shGLTleg3gg== X-Received: by 2002:a17:90a:de16:: with SMTP id m22mr6382525pjv.54.1631682609454; Tue, 14 Sep 2021 22:10:09 -0700 (PDT) Received: from localhost ([2405:201:6014:d058:a28d:3909:6ed5:29e7]) by smtp.gmail.com with ESMTPSA id m10sm2333891pjs.21.2021.09.14.22.10.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 22:10:09 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Kumar Kartikeya Dwivedi , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , Jesper Dangaard Brouer , =?utf-8?q?Toke_H=C3=B8iland-J?= =?utf-8?b?w7hyZ2Vuc2Vu?= , netdev@vger.kernel.org Subject: [PATCH bpf-next v3 07/10] libbpf: Support kernel module function calls Date: Wed, 15 Sep 2021 10:39:40 +0530 Message-Id: <20210915050943.679062-8-memxor@gmail.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20210915050943.679062-1-memxor@gmail.com> References: <20210915050943.679062-1-memxor@gmail.com> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4871; h=from:subject; bh=1xbQtnet8ONbSdxejO5SOUckglIx2nhjKxRa61HQUGg=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBhQX4c2Y0kuFgftjf+pdnO7/VUz/pKeQlitvNrGk+d 8NpVdoKJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCYUF+HAAKCRBM4MiGSL8RyjD5EA CUQ/KD+pRbfgnc5hqIp562Rk07O3uyGI8RvH5fy8pA4B65k6O3YCVWuhmh5REQ7f+HVLYt+uIC0Xgg UAGpUST1GJUpxC//uNKaz/w6VJuQOtimge/EhWriaITb6S7yVfFs7EFouQ9ZuGzF/sX+RXo++IJ3Ew caWHD6vahNguziY3HHyc6478QInDyWt2cjurPz05sf7jSjp8Ar4dObIDS36bG/d2I2W3/F5pWZ/j5n QmL3yjmMwydQkNj7aKjeoqElTKW409C5y+X5mh+oKnN49uYYRadFoxsnHqs8LYpAE/dNxiONb+NGc1 dMqyEPGPSSG6qJ7yh8hm9DWNKGltBZkxga82/4sRaDyXCpKMvms7UyjeghVOhzeuRziTOXx0L+yEiR K4bWnseLMeiTod8dsiCJPStwZvWVC6sQDgpLOownXBYgWPg2u+8Tcl1KG9POdzMKiL+hLQ4Wj1Xx2y h3Qr0qYN7Jdge+1flOYwnb7K42AZ6/aesbPGD7GSSp6f9iGaZND1kZNwGqF4IKodX+CzWfV226jj35 ne8W3mJCt7hnSjZgV5ND6DOLcxm/oo60x3b7RY2iizaZV3+ouIfgKPTKcdqHlsxYYxLDNhIvNkwq6Y eavHVDtFdfC0s15xXDeASKl8I21Yn1NRo/ujNjV3auUn08i6N4wY7BGpcJ2A== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org This patch adds libbpf support for kernel module function call support. The fd_array parameter is used during BPF program load is used to pass module BTFs referenced by the program. insn->off is set to index into this array + 1, because insn->off as 0 is reserved for btf_vmlinux. The kernel subtracts 1 from insn->off when indexing into env->fd_array. We try to use existing insn->off for a module, since the kernel limits the maximum distinct module BTFs for kfuncs to 256, and also because index must never exceed the maximum allowed value that can fit in insn->off (INT16_MAX). In the future, if kernel interprets signed offset as unsigned for kfunc calls, this limit can be increased to UINT16_MAX. Signed-off-by: Kumar Kartikeya Dwivedi --- tools/lib/bpf/bpf.c | 1 + tools/lib/bpf/libbpf.c | 56 +++++++++++++++++++++++++++++++-- tools/lib/bpf/libbpf_internal.h | 1 + 3 files changed, 55 insertions(+), 3 deletions(-) diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c index 2401fad090c5..7d1741ceaa32 100644 --- a/tools/lib/bpf/bpf.c +++ b/tools/lib/bpf/bpf.c @@ -264,6 +264,7 @@ int libbpf__bpf_prog_load(const struct bpf_prog_load_params *load_attr) attr.line_info_rec_size = load_attr->line_info_rec_size; attr.line_info_cnt = load_attr->line_info_cnt; attr.line_info = ptr_to_u64(load_attr->line_info); + attr.fd_array = ptr_to_u64(load_attr->fd_array); if (load_attr->name) memcpy(attr.prog_name, load_attr->name, diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 62a43c408d73..accf2586fa76 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -420,6 +420,12 @@ struct extern_desc { /* local btf_id of the ksym extern's type. */ __u32 type_id; + /* offset to be patched in for insn->off, + * this is 0 for btf_vmlinux, and index + 1 + * for module BTF, where index is BTF index in + * obj->fd_array + */ + __s16 offset; } ksym; }; }; @@ -516,6 +522,10 @@ struct bpf_object { void *priv; bpf_object_clear_priv_t clear_priv; + int *fd_array; + size_t fd_cap_cnt; + int nr_fds; + char path[]; }; #define obj_elf_valid(o) ((o)->efile.elf) @@ -5357,6 +5367,7 @@ bpf_object__relocate_data(struct bpf_object *obj, struct bpf_program *prog) ext = &obj->externs[relo->sym_off]; insn[0].src_reg = BPF_PSEUDO_KFUNC_CALL; insn[0].imm = ext->ksym.kernel_btf_id; + insn[0].off = ext->ksym.offset; break; case RELO_SUBPROG_ADDR: if (insn[0].src_reg != BPF_PSEUDO_FUNC) { @@ -6151,6 +6162,7 @@ load_program(struct bpf_program *prog, struct bpf_insn *insns, int insns_cnt, } load_attr.log_level = prog->log_level; load_attr.prog_flags = prog->prog_flags; + load_attr.fd_array = prog->obj->fd_array; if (prog->obj->gen_loader) { bpf_gen__prog_load(prog->obj->gen_loader, &load_attr, @@ -6760,9 +6772,44 @@ static int bpf_object__resolve_ksym_func_btf_id(struct bpf_object *obj, } if (kern_btf != obj->btf_vmlinux) { - pr_warn("extern (func ksym) '%s': function in kernel module is not supported\n", - ext->name); - return -ENOTSUP; + int index = -1; + + if (!obj->fd_array) { + obj->fd_array = calloc(8, sizeof(*obj->fd_array)); + if (!obj->fd_array) + return -ENOMEM; + obj->fd_cap_cnt = 8; + } + + for (int i = 0; i < obj->nr_fds; i++) { + if (obj->fd_array[i] == kern_btf_fd) { + index = i; + break; + } + } + + if (index == -1) { + if (obj->nr_fds == obj->fd_cap_cnt) { + ret = libbpf_ensure_mem((void **)&obj->fd_array, + &obj->fd_cap_cnt, sizeof(int), + obj->fd_cap_cnt + 1); + if (ret) + return ret; + } + + index = obj->nr_fds; + obj->fd_array[obj->nr_fds++] = kern_btf_fd; + } + + if (index >= INT16_MAX) { + /* insn->off is s16 */ + pr_warn("extern (func ksym) '%s': module btf fd index too big\n", + ext->name); + return -E2BIG; + } + ext->ksym.offset = index + 1; + } else { + ext->ksym.offset = 0; } kern_func = btf__type_by_id(kern_btf, kfunc_id); @@ -6938,6 +6985,9 @@ int bpf_object__load_xattr(struct bpf_object_load_attr *attr) err = bpf_gen__finish(obj->gen_loader); } + /* clean up fd_array */ + zfree(&obj->fd_array); + /* clean up module BTFs */ for (i = 0; i < obj->btf_module_cnt; i++) { close(obj->btf_modules[i].fd); diff --git a/tools/lib/bpf/libbpf_internal.h b/tools/lib/bpf/libbpf_internal.h index ceb0c98979bc..44b8f381b035 100644 --- a/tools/lib/bpf/libbpf_internal.h +++ b/tools/lib/bpf/libbpf_internal.h @@ -291,6 +291,7 @@ struct bpf_prog_load_params { __u32 log_level; char *log_buf; size_t log_buf_sz; + int *fd_array; }; int libbpf__bpf_prog_load(const struct bpf_prog_load_params *load_attr); From patchwork Wed Sep 15 05:09:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 512804 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96838C433EF for ; Wed, 15 Sep 2021 05:10:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7D25E6121E for ; Wed, 15 Sep 2021 05:10:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236438AbhIOFLr (ORCPT ); Wed, 15 Sep 2021 01:11:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44368 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236298AbhIOFLe (ORCPT ); Wed, 15 Sep 2021 01:11:34 -0400 Received: from mail-pg1-x543.google.com (mail-pg1-x543.google.com [IPv6:2607:f8b0:4864:20::543]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 06899C061574; Tue, 14 Sep 2021 22:10:16 -0700 (PDT) Received: by mail-pg1-x543.google.com with SMTP id s11so1522082pgr.11; Tue, 14 Sep 2021 22:10:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8OMtHYl/0NsOkE5QU+j8BFHyewhBeadRVhqSpUjbZrY=; b=GtHVK0NZATyW7TyxSlhpKPtOS84g6g8ldNlPIsF8JhY59zLMem8QYUDOgyEyJ5TcjA x1iZP5IfcPuKst5oOTG44DCTHuc3hxxwPJQkrL9kn+BcO2efuArUiZh9z0gB6692k0uz v8TRrLSfuN40tznOX6iFC0r2K086E1qj2cqq3YJFolFtdigIkLzlX4MhdtNosfHdio1S JbCjj9PmV364xUlgNlzvWBJaT7NbQYZGTrYKkirXfXCak9m4LwqBnzVk0oANgdlUa+uE 13dSXzTLF6FZDOGjyQ2f3Gz/zr2Q37bhAJZIZJ4mOHQGeKvbzNShxW/bp6i5vHAA33YI nltw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8OMtHYl/0NsOkE5QU+j8BFHyewhBeadRVhqSpUjbZrY=; b=iY7Z7YwtT+0qxG7eTlzMUBzI0L4oVzfd4PAp3UIFa5OdacErQifqAgnZ8cqLy78KA0 l9sZgvmK53Yoh4hKSquQBhK4tNH05VMCwR1OLS7J6LebKMvGxYw+kIXstTfvIam91taH GDRC4Ws5RvxfcOBZFOiHC/wia+kgaVHsTV1qFoUojavTYdjbWJDvYo7FOGyLIQ2YujGI wD6z5V5Xa5s7IAFt6/LvTmey/377GpxetUMV+GNe7VyhjIaUhDKPT/Ej6CIFbKYSxAfT SwrujN1MGhdP1h4ftr/RDbarqBzWKMh3fGvqla2lrGpUWOidQNZ8tie/lepQ9FlIBDok v1pw== X-Gm-Message-State: AOAM533a2yXRByIDC0DyT1oPJwuqKPRQnLBh5IfyL0zXwQOdpWv1uB/M 3yRzmYxs0Ssd3Xu26ZTpn2w9fJJ5YQAJ8w== X-Google-Smtp-Source: ABdhPJxa0189ETnkQsICa+fJsVzlLLIuDDK8dX0xhlhNXCfwLPIIiAbdGqW3dvoDqwe4FvzW3/Y/tw== X-Received: by 2002:a65:4486:: with SMTP id l6mr19074482pgq.145.1631682615356; Tue, 14 Sep 2021 22:10:15 -0700 (PDT) Received: from localhost ([2405:201:6014:d058:a28d:3909:6ed5:29e7]) by smtp.gmail.com with ESMTPSA id j17sm12302106pfn.148.2021.09.14.22.10.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 22:10:15 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Kumar Kartikeya Dwivedi , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , Jesper Dangaard Brouer , =?utf-8?q?Toke_H=C3=B8iland-J?= =?utf-8?b?w7hyZ2Vuc2Vu?= , netdev@vger.kernel.org Subject: [PATCH bpf-next v3 09/10] libbpf: Update gen_loader to emit BTF_KIND_FUNC relocations Date: Wed, 15 Sep 2021 10:39:42 +0530 Message-Id: <20210915050943.679062-10-memxor@gmail.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20210915050943.679062-1-memxor@gmail.com> References: <20210915050943.679062-1-memxor@gmail.com> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=8801; h=from:subject; bh=hayAcysmBL7vuknb5yER44ADm9rIVaUmu2XHK8D79mA=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBhQX4cxuKAJvKTBZUj77WPtC5De8S05cwgBHQ6re1j NCYR1LqJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCYUF+HAAKCRBM4MiGSL8RyhcfD/ 9WBM0KOO/g1NuesU64U1bTxU66p/3OwmkMs3/DdOh/G9Ix2ESkCaEAUS5YEQ9U0dpl2ubLyYc6opaZ ySNgBgkGe7AtuyY7QcgJVmYerI4u6Rnr17P1oePYzZH4l2esbEfxGQARg2eKXcoPPFhudYm5FrAsQZ OeDyoiTtSUyPVBO0F8lxsfGTulMnLQvD22vXf296I8ZNfnt4crK+7vixCRQ6OoWd0cLyRdkp/entLE MQ7gDSmw0zhJyNZDu5rEkjzOrk4igi+DmjeXvN3Tast+qY0QLFVSgaPUhVvo/r+FA9oxHXcDrzjeRk BBVW+3S6Gedv7sp1nKFW8TngciWrLbbHuo7WXhX1B/6ftmgrd/vzUtubkR6pU71C8XrqUjr0XFXDlM ww71qIn2vJv32u2lCLRnvpNow2r6qv69UtBheY5GK2hfPhZvVUTMi8S9Eb8zxnI2rCnV6XCGuSsaOk lQ2tqUbUZarCYSt8QNXUHNXxRtZNccG6oh5ZRzkwhZSpDXe827jDJx4RrNn3UcCFgaMAacdAFIOyX+ QvskV3jaPUrzFkWWptb77MM9fd9zeo9DOjUHiEE4IRWc5IodM5GwunFIBpFtddk3R/1gAspdzE9FKG dmzoZWU07lPXKI3pPUQMDmTXv6MO5a14hPLYnX+juQQcgQ6BwCCyEjI86VcQ== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org This change updates the BPF syscall loader to relocate BTF_KIND_FUNC relocations, with support for weak kfunc relocations. One of the disadvantages of gen_loader is that due to stack size limitation, BTF fd array size is clamped to a smaller limit than what the kernel allows. Also, finding an existing BTF fd's slot is not trivial, because that would require to open all module BTFs and match on the open fds (like we do for libbpf), so we do the next best thing: deduplicate slots for the same symbol. Signed-off-by: Kumar Kartikeya Dwivedi --- tools/lib/bpf/bpf_gen_internal.h | 12 ++++- tools/lib/bpf/gen_loader.c | 93 ++++++++++++++++++++++++++++---- tools/lib/bpf/libbpf.c | 8 +-- 3 files changed, 99 insertions(+), 14 deletions(-) diff --git a/tools/lib/bpf/bpf_gen_internal.h b/tools/lib/bpf/bpf_gen_internal.h index 615400391e57..4826adf18d7b 100644 --- a/tools/lib/bpf/bpf_gen_internal.h +++ b/tools/lib/bpf/bpf_gen_internal.h @@ -7,8 +7,15 @@ struct ksym_relo_desc { const char *name; int kind; int insn_idx; + bool is_weak; }; +struct kfunc_desc { + const char *name; + int index; +}; + +#define MAX_KFUNC_DESCS 94 struct bpf_gen { struct gen_loader_opts *opts; void *data_start; @@ -24,6 +31,8 @@ struct bpf_gen { int relo_cnt; char attach_target[128]; int attach_kind; + struct kfunc_desc kfunc_descs[MAX_KFUNC_DESCS]; + __u32 nr_kfuncs; }; void bpf_gen__init(struct bpf_gen *gen, int log_level); @@ -36,6 +45,7 @@ void bpf_gen__prog_load(struct bpf_gen *gen, struct bpf_prog_load_params *load_a void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *value, __u32 value_size); void bpf_gen__map_freeze(struct bpf_gen *gen, int map_idx); void bpf_gen__record_attach_target(struct bpf_gen *gen, const char *name, enum bpf_attach_type type); -void bpf_gen__record_extern(struct bpf_gen *gen, const char *name, int kind, int insn_idx); +void bpf_gen__record_extern(struct bpf_gen *gen, const char *name, bool is_weak, int kind, + int insn_idx); #endif diff --git a/tools/lib/bpf/gen_loader.c b/tools/lib/bpf/gen_loader.c index 8df718a6b142..5e8c15e36c46 100644 --- a/tools/lib/bpf/gen_loader.c +++ b/tools/lib/bpf/gen_loader.c @@ -5,6 +5,7 @@ #include #include #include +#include #include "btf.h" #include "bpf.h" #include "libbpf.h" @@ -13,6 +14,7 @@ #include "bpf_gen_internal.h" #include "skel_internal.h" +/* MAX_BPF_STACK is 768 bytes, so (64 + 32 + 94 (MAX_KFUNC_DESCS) + 2) * 4 */ #define MAX_USED_MAPS 64 #define MAX_USED_PROGS 32 @@ -31,6 +33,8 @@ struct loader_stack { __u32 btf_fd; __u32 map_fd[MAX_USED_MAPS]; __u32 prog_fd[MAX_USED_PROGS]; + /* Update insn->off store when reordering kfunc_btf_fd */ + __u32 kfunc_btf_fd[MAX_KFUNC_DESCS]; __u32 inner_map_fd; }; @@ -506,8 +510,8 @@ static void emit_find_attach_target(struct bpf_gen *gen) */ } -void bpf_gen__record_extern(struct bpf_gen *gen, const char *name, int kind, - int insn_idx) +void bpf_gen__record_extern(struct bpf_gen *gen, const char *name, bool is_weak, + int kind, int insn_idx) { struct ksym_relo_desc *relo; @@ -519,14 +523,39 @@ void bpf_gen__record_extern(struct bpf_gen *gen, const char *name, int kind, gen->relos = relo; relo += gen->relo_cnt; relo->name = name; + relo->is_weak = is_weak; relo->kind = kind; relo->insn_idx = insn_idx; gen->relo_cnt++; } +static struct kfunc_desc *find_kfunc_desc(struct bpf_gen *gen, const char *name) +{ + /* Try to reuse BTF fd index for repeating symbol */ + for (int i = 0; i < gen->nr_kfuncs; i++) { + if (!strcmp(gen->kfunc_descs[i].name, name)) + return &gen->kfunc_descs[i]; + } + return NULL; +} + +static struct kfunc_desc *add_kfunc_desc(struct bpf_gen *gen, const char *name) +{ + struct kfunc_desc *kdesc; + + if (gen->nr_kfuncs == ARRAY_SIZE(gen->kfunc_descs)) + return NULL; + kdesc = &gen->kfunc_descs[gen->nr_kfuncs]; + kdesc->name = name; + kdesc->index = gen->nr_kfuncs++; + return kdesc; +} + static void emit_relo(struct bpf_gen *gen, struct ksym_relo_desc *relo, int insns) { int name, insn, len = strlen(relo->name) + 1; + int off = MAX_USED_MAPS + MAX_USED_PROGS; + struct kfunc_desc *kdesc; pr_debug("gen: emit_relo: %s at %d\n", relo->name, relo->insn_idx); name = add_data(gen, relo->name, len); @@ -539,18 +568,64 @@ static void emit_relo(struct bpf_gen *gen, struct ksym_relo_desc *relo, int insn emit(gen, BPF_EMIT_CALL(BPF_FUNC_btf_find_by_name_kind)); emit(gen, BPF_MOV64_REG(BPF_REG_7, BPF_REG_0)); debug_ret(gen, "find_by_name_kind(%s,%d)", relo->name, relo->kind); - emit_check_err(gen); + /* if not weak kfunc, emit err check */ + if (relo->kind != BTF_KIND_FUNC || !relo->is_weak) + emit_check_err(gen); + insn = insns + sizeof(struct bpf_insn) * relo->insn_idx; + emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_0, BPF_PSEUDO_MAP_IDX_VALUE, 0, 0, 0, insn)); + /* set a default value */ + emit(gen, BPF_ST_MEM(BPF_W, BPF_REG_0, offsetof(struct bpf_insn, imm), 0)); + /* skip success case store if ret < 0 */ + emit(gen, BPF_JMP_IMM(BPF_JSLT, BPF_REG_7, 0, 1)); /* store btf_id into insn[insn_idx].imm */ - insn = insns + sizeof(struct bpf_insn) * relo->insn_idx + - offsetof(struct bpf_insn, imm); - emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_0, BPF_PSEUDO_MAP_IDX_VALUE, - 0, 0, 0, insn)); - emit(gen, BPF_STX_MEM(BPF_W, BPF_REG_0, BPF_REG_7, 0)); + emit(gen, BPF_STX_MEM(BPF_W, BPF_REG_0, BPF_REG_7, offsetof(struct bpf_insn, imm))); if (relo->kind == BTF_KIND_VAR) { /* store btf_obj_fd into insn[insn_idx + 1].imm */ emit(gen, BPF_ALU64_IMM(BPF_RSH, BPF_REG_7, 32)); emit(gen, BPF_STX_MEM(BPF_W, BPF_REG_0, BPF_REG_7, - sizeof(struct bpf_insn))); + sizeof(struct bpf_insn) + offsetof(struct bpf_insn, imm))); + } else if (relo->kind == BTF_KIND_FUNC) { + kdesc = find_kfunc_desc(gen, relo->name); + if (!kdesc) + kdesc = add_kfunc_desc(gen, relo->name); + if (kdesc) { + /* store btf_obj_fd in index in kfunc_btf_fd array + * but skip storing fd if ret < 0 + */ + emit(gen, BPF_ST_MEM(BPF_W, BPF_REG_10, + stack_off(kfunc_btf_fd[kdesc->index]), 0)); + emit(gen, BPF_MOV64_REG(BPF_REG_8, BPF_REG_7)); + emit(gen, BPF_JMP_IMM(BPF_JSLT, BPF_REG_7, 0, 4)); + emit(gen, BPF_ALU64_IMM(BPF_RSH, BPF_REG_7, 32)); + /* if vmlinux BTF, skip store */ + emit(gen, BPF_JMP_IMM(BPF_JEQ, BPF_REG_7, 0, 1)); + emit(gen, BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_7, + stack_off(kfunc_btf_fd[kdesc->index]))); + emit(gen, BPF_MOV64_REG(BPF_REG_7, BPF_REG_8)); + /* remember BTF obj fd */ + emit(gen, BPF_ALU64_IMM(BPF_RSH, BPF_REG_8, 32)); + } else { + pr_warn("Out of BTF fd slots (total: %u), skipping for %s\n", + gen->nr_kfuncs, relo->name); + emit(gen, BPF_MOV64_REG(BPF_REG_1, BPF_REG_7)); + emit(gen, BPF_ALU64_IMM(BPF_RSH, BPF_REG_1, 32)); + __emit_sys_close(gen); + } + /* store index + 1 into insn[insn_idx].off */ + off = kdesc ? off + kdesc->index + 1 : 0; + off = off > INT16_MAX ? 0 : off; + /* set a default value */ + emit(gen, BPF_ST_MEM(BPF_H, BPF_REG_0, offsetof(struct bpf_insn, off), 0)); + /* skip success case store if ret < 0 */ + emit(gen, BPF_JMP_IMM(BPF_JSLT, BPF_REG_7, 0, 2)); + /* skip if vmlinux BTF */ + emit(gen, BPF_JMP_IMM(BPF_JEQ, BPF_REG_8, 0, 1)); + /* store offset */ + emit(gen, BPF_ST_MEM(BPF_H, BPF_REG_0, offsetof(struct bpf_insn, off), off)); + /* log relocation */ + emit(gen, BPF_LDX_MEM(BPF_W, BPF_REG_7, BPF_REG_0, offsetof(struct bpf_insn, imm))); + emit(gen, BPF_LDX_MEM(BPF_H, BPF_REG_8, BPF_REG_0, offsetof(struct bpf_insn, off))); + debug_regs(gen, BPF_REG_7, BPF_REG_8, "sym (%s): imm: %%d, off: %%d", relo->name); } } diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 50a7c995979a..3ac26dcb60b0 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -6264,12 +6264,12 @@ static int bpf_program__record_externs(struct bpf_program *prog) ext->name); return -ENOTSUP; } - bpf_gen__record_extern(obj->gen_loader, ext->name, BTF_KIND_VAR, - relo->insn_idx); + bpf_gen__record_extern(obj->gen_loader, ext->name, ext->is_weak, + BTF_KIND_VAR, relo->insn_idx); break; case RELO_EXTERN_FUNC: - bpf_gen__record_extern(obj->gen_loader, ext->name, BTF_KIND_FUNC, - relo->insn_idx); + bpf_gen__record_extern(obj->gen_loader, ext->name, ext->is_weak, + BTF_KIND_FUNC, relo->insn_idx); break; default: continue;