From patchwork Mon Sep 14 16:13:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 260911 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.6 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AAB2CC43461 for ; Mon, 14 Sep 2020 16:33:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 66D96214F1 for ; Mon, 14 Sep 2020 16:33:13 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="J0R8Cf8W" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726348AbgINQdK (ORCPT ); Mon, 14 Sep 2020 12:33:10 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:21021 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726485AbgINQNc (ORCPT ); Mon, 14 Sep 2020 12:13:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1600100009; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hFXKiAVJpU4aGkR9h2ePr4YYyxSQIIc/XozWY5DegDw=; b=J0R8Cf8WuGI4zAUw3MnDcrmnPN2NtSY9+FBvnMX6nW++E0qlY7pQcoDtwM0E4q4IGVCr03 a8HXxbiLVQnjNlHiEzXfL0Oiu8ih8A4s6KusSW0ShVcFx7Uvt52ciz3ot3R333K43VPcoy YF3SmU1dKGbo5AQe6n/tg/Ev3hTA2vs= Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-562-RjLmOlO3OdecMYv_rx5now-1; Mon, 14 Sep 2020 12:13:28 -0400 X-MC-Unique: RjLmOlO3OdecMYv_rx5now-1 Received: by mail-wr1-f72.google.com with SMTP id v12so60199wrm.9 for ; Mon, 14 Sep 2020 09:13:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:date:message-id:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=hFXKiAVJpU4aGkR9h2ePr4YYyxSQIIc/XozWY5DegDw=; b=P9A8ZWIO5bHjWYZ6E+8RnD0OWSO1hRq7GyuRg4d1TlvgxLVsbFj2+JHwTv9KWILYWI ELPEyd3zsqKegKgH/CmCSDplLt2MnxYAhE+H4HMqXjo4aeIN29VXeOfX+krT0qWPDx9A BuF9VLLbO2cDYxYpYN70JE/jOjKe6RGd5LKAjE5KTzpJr7TNpVfz2pJuV1HavrdIQb8p VCIiTjQaF0yJ45MzROmNJ61TVssbuBu0SrxnPlIHzSTmIHmhRix9enBHQcvWpJAT5Gwi V3pjKWo5+7dMmOF4nrM6rXxs6Y4bG3k1HMFKVKOaz4sRVtTatURUzlYICeUSywVG45Gu ZH0Q== X-Gm-Message-State: AOAM531M1J1kQddatVa1GsyijVL4gzu5iHySadkqi+ZaHmQjvTIprjvr HPlJ4/ZI/sFq/p5WyIaTVnJUm5HzM70EA2lxNZj2SbcCAri/LiMP24eyZ9IhQ2XEEhKCbRXKy2r xIxFpQpBS31c49CRA X-Received: by 2002:adf:f5c7:: with SMTP id k7mr17275282wrp.246.1600100006620; Mon, 14 Sep 2020 09:13:26 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzbpVBr4sOuqS2Kgq+3AfSLi689O4b3xGyIHYe5qGIZhYZckfJJONIUsgbYIx6pTb3CDg9ojw== X-Received: by 2002:adf:f5c7:: with SMTP id k7mr17275262wrp.246.1600100006320; Mon, 14 Sep 2020 09:13:26 -0700 (PDT) Received: from alrua-x1.borgediget.toke.dk ([45.145.92.2]) by smtp.gmail.com with ESMTPSA id h204sm20364840wmf.35.2020.09.14.09.13.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Sep 2020 09:13:25 -0700 (PDT) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id 103A11829CB; Mon, 14 Sep 2020 18:13:25 +0200 (CEST) Subject: [PATCH bpf-next v4 2/8] bpf: verifier: refactor check_attach_btf_id() From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= To: Alexei Starovoitov Cc: Daniel Borkmann , Martin KaFai Lau , Song Liu , Yonghong Song , Andrii Nakryiko , John Fastabend , Jiri Olsa , Eelco Chaudron , KP Singh , netdev@vger.kernel.org, bpf@vger.kernel.org Date: Mon, 14 Sep 2020 18:13:25 +0200 Message-ID: <160010000496.80898.8023973720943514259.stgit@toke.dk> In-Reply-To: <160010000272.80898.13117015273092905112.stgit@toke.dk> References: <160010000272.80898.13117015273092905112.stgit@toke.dk> User-Agent: StGit/0.23 MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Toke Høiland-Jørgensen The check_attach_btf_id() function really does three things: 1. It performs a bunch of checks on the program to ensure that the attachment is valid. 2. It stores a bunch of state about the attachment being requested in the verifier environment and struct bpf_prog objects. 3. It allocates a trampoline for the attachment. This patch splits out (1.) and (3.) into separate functions in preparation for reusing them when the actual attachment is happening (in the raw_tracepoint_open syscall operation), which will allow tracing programs to have multiple (compatible) attachments. No functional change is intended with this patch. Signed-off-by: Toke Høiland-Jørgensen --- include/linux/bpf.h | 7 + include/linux/bpf_verifier.h | 9 ++ kernel/bpf/trampoline.c | 20 ++++ kernel/bpf/verifier.c | 197 ++++++++++++++++++++++++------------------ 4 files changed, 149 insertions(+), 84 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 5ad4a935a24e..b3aefbdca1a3 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -616,6 +616,8 @@ static __always_inline unsigned int bpf_dispatcher_nop_func( struct bpf_trampoline *bpf_trampoline_lookup(u64 key); int bpf_trampoline_link_prog(struct bpf_prog *prog); int bpf_trampoline_unlink_prog(struct bpf_prog *prog); +struct bpf_trampoline *bpf_trampoline_get(u64 key, void *addr, + struct btf_func_model *fmodel); void bpf_trampoline_put(struct bpf_trampoline *tr); #define BPF_DISPATCHER_INIT(_name) { \ .mutex = __MUTEX_INITIALIZER(_name.mutex), \ @@ -672,6 +674,11 @@ static inline int bpf_trampoline_unlink_prog(struct bpf_prog *prog) { return -ENOTSUPP; } +static inline bpf_trampoline *bpf_trampoline_get(u64 key, void *addr, + struct btf_func_model *fmodel) +{ + return ERR_PTR(-EOPNOTSUPP); +} static inline void bpf_trampoline_put(struct bpf_trampoline *tr) {} #define DEFINE_BPF_DISPATCHER(name) #define DECLARE_BPF_DISPATCHER(name) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 20009e766805..db3db0b69aad 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -447,4 +447,13 @@ bpf_prog_offload_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt); int check_ctx_reg(struct bpf_verifier_env *env, const struct bpf_reg_state *reg, int regno); +int bpf_check_attach_target(struct bpf_verifier_log *log, + const struct bpf_prog *prog, + const struct bpf_prog *tgt_prog, + u32 btf_id, + struct btf_func_model *fmodel, + long *tgt_addr, + const char **tgt_name, + const struct btf_type **tgt_type); + #endif /* _LINUX_BPF_VERIFIER_H */ diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 7dd523a7e32d..7845913e7e41 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -336,6 +336,26 @@ int bpf_trampoline_unlink_prog(struct bpf_prog *prog) return err; } +struct bpf_trampoline *bpf_trampoline_get(u64 key, void *addr, + struct btf_func_model *fmodel) +{ + struct bpf_trampoline *tr; + + tr = bpf_trampoline_lookup(key); + if (!tr) + return ERR_PTR(-ENOMEM); + + mutex_lock(&tr->mutex); + if (tr->func.addr) + goto out; + + memcpy(&tr->func.model, fmodel, sizeof(*fmodel)); + tr->func.addr = addr; +out: + mutex_unlock(&tr->mutex); + return tr; +} + void bpf_trampoline_put(struct bpf_trampoline *tr) { if (!tr) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 0be7a187fb7f..d38678319ca4 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -10997,11 +10997,11 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env) } #define SECURITY_PREFIX "security_" -static int check_attach_modify_return(struct bpf_prog *prog, unsigned long addr) +static int check_attach_modify_return(const struct bpf_prog *prog, unsigned long addr, + const char *func_name) { if (within_error_injection_list(addr) || - !strncmp(SECURITY_PREFIX, prog->aux->attach_func_name, - sizeof(SECURITY_PREFIX) - 1)) + !strncmp(SECURITY_PREFIX, func_name, sizeof(SECURITY_PREFIX) - 1)) return 0; return -EINVAL; @@ -11038,43 +11038,29 @@ static int check_non_sleepable_error_inject(u32 btf_id) return btf_id_set_contains(&btf_non_sleepable_error_inject, btf_id); } -static int check_attach_btf_id(struct bpf_verifier_env *env) +int bpf_check_attach_target(struct bpf_verifier_log *log, + const struct bpf_prog *prog, + const struct bpf_prog *tgt_prog, + u32 btf_id, + struct btf_func_model *fmodel, + long *tgt_addr, + const char **tgt_name, + const struct btf_type **tgt_type) { - struct bpf_prog *prog = env->prog; bool prog_extension = prog->type == BPF_PROG_TYPE_EXT; - struct bpf_prog *tgt_prog = prog->aux->linked_prog; - struct bpf_verifier_log *log = &env->log; - u32 btf_id = prog->aux->attach_btf_id; const char prefix[] = "btf_trace_"; - struct btf_func_model fmodel; int ret = 0, subprog = -1, i; - struct bpf_trampoline *tr; const struct btf_type *t; bool conservative = true; const char *tname; struct btf *btf; - long addr; - u64 key; - - if (prog->aux->sleepable && prog->type != BPF_PROG_TYPE_TRACING && - prog->type != BPF_PROG_TYPE_LSM) { - verbose(env, "Only fentry/fexit/fmod_ret and lsm programs can be sleepable\n"); - return -EINVAL; - } - - if (prog->type == BPF_PROG_TYPE_STRUCT_OPS) - return check_struct_ops_btf_id(env); - - if (prog->type != BPF_PROG_TYPE_TRACING && - prog->type != BPF_PROG_TYPE_LSM && - !prog_extension) - return 0; + long addr = 0; if (!btf_id) { bpf_log(log, "Tracing programs must provide btf_id\n"); return -EINVAL; } - btf = bpf_prog_get_target_btf(prog); + btf = tgt_prog ? tgt_prog->aux->btf : btf_vmlinux; if (!btf) { bpf_log(log, "FENTRY/FEXIT program can only be attached to another program annotated with BTF\n"); @@ -11114,8 +11100,6 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) "Extension programs should be JITed\n"); return -EINVAL; } - env->ops = bpf_verifier_ops[tgt_prog->type]; - prog->expected_attach_type = tgt_prog->expected_attach_type; } if (!tgt_prog->jited) { bpf_log(log, "Can attach to only JITed progs\n"); @@ -11151,13 +11135,11 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) bpf_log(log, "Cannot extend fentry/fexit\n"); return -EINVAL; } - key = ((u64)aux->id) << 32 | btf_id; } else { if (prog_extension) { bpf_log(log, "Cannot replace kernel functions\n"); return -EINVAL; } - key = btf_id; } switch (prog->expected_attach_type) { @@ -11187,13 +11169,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) /* should never happen in valid vmlinux build */ return -EINVAL; - /* remember two read only pointers that are valid for - * the life time of the kernel - */ - prog->aux->attach_func_name = tname; - prog->aux->attach_func_proto = t; - prog->aux->attach_btf_trace = true; - return 0; + break; case BPF_TRACE_ITER: if (!btf_type_is_func(t)) { bpf_log(log, "attach_btf_id %u is not a function\n", @@ -11203,12 +11179,10 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) t = btf_type_by_id(btf, t->type); if (!btf_type_is_func_proto(t)) return -EINVAL; - prog->aux->attach_func_name = tname; - prog->aux->attach_func_proto = t; - if (!bpf_iter_prog_supported(prog)) - return -EINVAL; - ret = btf_distill_func_proto(log, btf, t, tname, &fmodel); - return ret; + ret = btf_distill_func_proto(log, btf, t, tname, fmodel); + if (ret) + return ret; + break; default: if (!prog_extension) return -EINVAL; @@ -11217,13 +11191,6 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) case BPF_LSM_MAC: case BPF_TRACE_FENTRY: case BPF_TRACE_FEXIT: - prog->aux->attach_func_name = tname; - if (prog->type == BPF_PROG_TYPE_LSM) { - ret = bpf_lsm_verify_prog(log, prog); - if (ret < 0) - return ret; - } - if (!btf_type_is_func(t)) { bpf_log(log, "attach_btf_id %u is not a function\n", btf_id); @@ -11235,24 +11202,14 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) t = btf_type_by_id(btf, t->type); if (!btf_type_is_func_proto(t)) return -EINVAL; - tr = bpf_trampoline_lookup(key); - if (!tr) - return -ENOMEM; - /* t is either vmlinux type or another program's type */ - prog->aux->attach_func_proto = t; - mutex_lock(&tr->mutex); - if (tr->func.addr) { - prog->aux->trampoline = tr; - goto out; - } - if (tgt_prog && conservative) { - prog->aux->attach_func_proto = NULL; + + if (tgt_prog && conservative) t = NULL; - } - ret = btf_distill_func_proto(log, btf, t, - tname, &tr->func.model); + + ret = btf_distill_func_proto(log, btf, t, tname, fmodel); if (ret < 0) - goto out; + return ret; + if (tgt_prog) { if (subprog == 0) addr = (long) tgt_prog->bpf_func; @@ -11264,8 +11221,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) bpf_log(log, "The address of function %s cannot be found\n", tname); - ret = -ENOENT; - goto out; + return -ENOENT; } } @@ -11290,25 +11246,98 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) default: break; } - if (ret) - bpf_log(log, "%s is not sleepable\n", - prog->aux->attach_func_name); + if (ret) { + bpf_log(log, "%s is not sleepable\n", tname); + return ret; + } } else if (prog->expected_attach_type == BPF_MODIFY_RETURN) { - ret = check_attach_modify_return(prog, addr); - if (ret) - bpf_log(log, "%s() is not modifiable\n", - prog->aux->attach_func_name); + ret = check_attach_modify_return(prog, addr, tname); + if (ret) { + bpf_log(log, "%s() is not modifiable\n", tname); + return ret; + } } - if (ret) - goto out; - tr->func.addr = (void *)addr; - prog->aux->trampoline = tr; -out: - mutex_unlock(&tr->mutex); - if (ret) - bpf_trampoline_put(tr); + + break; + } + *tgt_addr = addr; + if (tgt_name) + *tgt_name = tname; + if (tgt_type) + *tgt_type = t; + return 0; +} + +static int check_attach_btf_id(struct bpf_verifier_env *env) +{ + struct bpf_prog *prog = env->prog; + struct bpf_prog *tgt_prog = prog->aux->linked_prog; + u32 btf_id = prog->aux->attach_btf_id; + struct btf_func_model fmodel; + struct bpf_trampoline *tr; + const struct btf_type *t; + const char *tname; + long addr; + int ret; + u64 key; + + if (prog->aux->sleepable && prog->type != BPF_PROG_TYPE_TRACING && + prog->type != BPF_PROG_TYPE_LSM) { + verbose(env, "Only fentry/fexit/fmod_ret and lsm programs can be sleepable\n"); + return -EINVAL; + } + + if (prog->type == BPF_PROG_TYPE_STRUCT_OPS) + return check_struct_ops_btf_id(env); + + if (prog->type != BPF_PROG_TYPE_TRACING && + prog->type != BPF_PROG_TYPE_LSM && + prog->type != BPF_PROG_TYPE_EXT) + return 0; + + ret = bpf_check_attach_target(&env->log, prog, tgt_prog, btf_id, + &fmodel, &addr, &tname, &t); + if (ret) return ret; + + if (tgt_prog) { + if (prog->type == BPF_PROG_TYPE_EXT) { + env->ops = bpf_verifier_ops[tgt_prog->type]; + prog->expected_attach_type = + tgt_prog->expected_attach_type; + } + key = ((u64)tgt_prog->aux->id) << 32 | btf_id; + } else { + key = btf_id; } + + /* remember two read only pointers that are valid for + * the life time of the kernel + */ + prog->aux->attach_func_proto = t; + prog->aux->attach_func_name = tname; + + if (prog->expected_attach_type == BPF_TRACE_RAW_TP) { + prog->aux->attach_btf_trace = true; + return 0; + } else if (prog->expected_attach_type == BPF_TRACE_ITER) { + if (!bpf_iter_prog_supported(prog)) + return -EINVAL; + return 0; + } + + if (prog->type == BPF_PROG_TYPE_LSM) { + ret = bpf_lsm_verify_prog(&env->log, prog); + if (ret < 0) + return ret; + } + + tr = bpf_trampoline_get(key, (void *)addr, &fmodel); + if (IS_ERR(tr)) + return PTR_ERR(tr); + + prog->aux->trampoline = tr; + return 0; } int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, From patchwork Mon Sep 14 16:13:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 260912 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.6 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 08DCCC433E2 for ; Mon, 14 Sep 2020 16:32:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9F5E8214F1 for ; Mon, 14 Sep 2020 16:32:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="enQul9hW" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726475AbgINQcs (ORCPT ); Mon, 14 Sep 2020 12:32:48 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:23889 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726513AbgINQNd (ORCPT ); Mon, 14 Sep 2020 12:13:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1600100011; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+Rg3ftGEq9iymCxm6c0JVOkkajyz9sUOBcjMZh1ItDU=; b=enQul9hWf0Q7GOfZo81D7RTdK0NBPN2Q3Zfd1hq3yOwf7wXl5mzYepbNRQFvFjCowciNMB rwQmcjJH5xD7cm+TXRoax6jl3clPL77tvjzgVV2SIGQ7mZpb4mg8QXqrRJX0FmMaoBhYKA PdyCeFr14Nhu4aKfL2EPO9z+fhBm5tU= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-139-HwBkmuR5NwSLjJZsD96Ghg-1; Mon, 14 Sep 2020 12:13:29 -0400 X-MC-Unique: HwBkmuR5NwSLjJZsD96Ghg-1 Received: by mail-wm1-f71.google.com with SMTP id l26so175040wmg.7 for ; Mon, 14 Sep 2020 09:13:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:date:message-id:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=+Rg3ftGEq9iymCxm6c0JVOkkajyz9sUOBcjMZh1ItDU=; b=e8MeU+bS+Kp5YCgC4Q7HUpNNxvuHgj3g4HlV1UeILrMfaqzeX86Tja2YB5NUZ2awj2 36NOwmn1xLZb2LR72LlImClC/wsN3yDyY0Ro9MtCrO6dLRiqLEqLMDmeppfFyRQbFfon 8+sqHAGT1tLzVdPOBkhw8JmnBBfJYDJbPe78ScTphTQ10oCAsMSTP4gju5FjzZ4m7lFM 0/PQ3asK/sM6bUpx6tXHZFAXnUcfJdXhQG8TOvBk8cV6VrUfGIpFU3ZVLu2pgqc2sZsU t/0uMKwNYpj8uLGngPnuOYsNUiy1p5Czq6lr25MIz6npH4p9WSR3EeyTWpPlkxT+K5FG LLMw== X-Gm-Message-State: AOAM533mM5QK0v7VRR3VjbH8+Ed8AkW8g8/tPLRtwpOKnkYrLtWnpnq4 GyVnKvoln14ER33Xa3xa40bXBvC2XVySGuZUq86jHZaBn7BEMtGMBEbLl4vHIbPugehoqCBZJs6 BWUl9Rud8689R5r3r X-Received: by 2002:adf:e8ce:: with SMTP id k14mr17932523wrn.394.1600100007832; Mon, 14 Sep 2020 09:13:27 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxZXDwpP2Y1hfZjxK//42r+hJ1FiJNabfsbunISqDsdjMNaIUqHX6IL59GkK8nKtGplPWQkyA== X-Received: by 2002:adf:e8ce:: with SMTP id k14mr17932484wrn.394.1600100007296; Mon, 14 Sep 2020 09:13:27 -0700 (PDT) Received: from alrua-x1.borgediget.toke.dk ([2a0c:4d80:42:443::2]) by smtp.gmail.com with ESMTPSA id v204sm20556717wmg.20.2020.09.14.09.13.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Sep 2020 09:13:26 -0700 (PDT) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id 2725D1829CC; Mon, 14 Sep 2020 18:13:26 +0200 (CEST) Subject: [PATCH bpf-next v4 3/8] bpf: wrap prog->aux->linked_prog in a bpf_tracing_link From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= To: Alexei Starovoitov Cc: Daniel Borkmann , Martin KaFai Lau , Song Liu , Yonghong Song , Andrii Nakryiko , John Fastabend , Jiri Olsa , Eelco Chaudron , KP Singh , netdev@vger.kernel.org, bpf@vger.kernel.org Date: Mon, 14 Sep 2020 18:13:26 +0200 Message-ID: <160010000607.80898.8871493983916568282.stgit@toke.dk> In-Reply-To: <160010000272.80898.13117015273092905112.stgit@toke.dk> References: <160010000272.80898.13117015273092905112.stgit@toke.dk> User-Agent: StGit/0.23 MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Toke Høiland-Jørgensen The bpf_tracing_link structure is a convenient data structure to contain the reference to a linked program; in preparation for supporting multiple attachments for the same freplace program, move the linked_prog in prog->aux into a bpf_tracing_link wrapper. With this change, it is no longer possible to attach the same tracing program multiple times (detaching in-between), since the reference from the tracing program to the target disappears on the first attach. However, since the next patch will let the caller supply an attach target, that will also make it possible to attach to the same place multiple times. Signed-off-by: Toke Høiland-Jørgensen --- include/linux/bpf.h | 23 +++++++++---- kernel/bpf/btf.c | 13 ++++--- kernel/bpf/core.c | 4 +- kernel/bpf/syscall.c | 86 ++++++++++++++++++++++++++++++++++++----------- kernel/bpf/trampoline.c | 12 ++----- kernel/bpf/verifier.c | 13 +++++-- 6 files changed, 105 insertions(+), 46 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index b3aefbdca1a3..8949e9794dc9 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -26,6 +26,7 @@ struct bpf_verifier_log; struct perf_event; struct bpf_prog; struct bpf_prog_aux; +struct bpf_tracing_link; struct bpf_map; struct sock; struct seq_file; @@ -614,8 +615,8 @@ static __always_inline unsigned int bpf_dispatcher_nop_func( } #ifdef CONFIG_BPF_JIT struct bpf_trampoline *bpf_trampoline_lookup(u64 key); -int bpf_trampoline_link_prog(struct bpf_prog *prog); -int bpf_trampoline_unlink_prog(struct bpf_prog *prog); +int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr); +int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline *tr); struct bpf_trampoline *bpf_trampoline_get(u64 key, void *addr, struct btf_func_model *fmodel); void bpf_trampoline_put(struct bpf_trampoline *tr); @@ -666,11 +667,13 @@ static inline struct bpf_trampoline *bpf_trampoline_lookup(u64 key) { return NULL; } -static inline int bpf_trampoline_link_prog(struct bpf_prog *prog) +static inline int bpf_trampoline_link_prog(struct bpf_prog *prog, + struct bpf_trampoline *tr) { return -ENOTSUPP; } -static inline int bpf_trampoline_unlink_prog(struct bpf_prog *prog) +static inline int bpf_trampoline_unlink_prog(struct bpf_prog *prog, + struct bpf_trampoline *tr) { return -ENOTSUPP; } @@ -738,14 +741,13 @@ struct bpf_prog_aux { u32 max_rdonly_access; u32 max_rdwr_access; const struct bpf_ctx_arg_aux *ctx_arg_info; - struct bpf_prog *linked_prog; + struct bpf_tracing_link *tgt_link; bool verifier_zext; /* Zero extensions has been inserted by verifier. */ bool offload_requested; bool attach_btf_trace; /* true if attaching to BTF-enabled raw tp */ bool func_proto_unreliable; bool sleepable; enum bpf_tramp_prog_type trampoline_prog_type; - struct bpf_trampoline *trampoline; struct hlist_node tramp_hlist; /* BTF_KIND_FUNC_PROTO for valid attach_btf_id */ const struct btf_type *attach_func_proto; @@ -825,6 +827,15 @@ struct bpf_link { struct work_struct work; }; +struct bpf_tracing_link { + struct bpf_link link; + enum bpf_attach_type attach_type; + struct bpf_trampoline *trampoline; + struct bpf_prog *tgt_prog; +}; + +void bpf_tracing_link_free(struct bpf_tracing_link *link); + struct bpf_link_ops { void (*release)(struct bpf_link *link); void (*dealloc)(struct bpf_link *link); diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 2ace56c99c36..e10f13f8251c 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -3706,10 +3706,10 @@ struct btf *btf_parse_vmlinux(void) struct btf *bpf_prog_get_target_btf(const struct bpf_prog *prog) { - struct bpf_prog *tgt_prog = prog->aux->linked_prog; + struct bpf_tracing_link *tgt_link = prog->aux->tgt_link; - if (tgt_prog) { - return tgt_prog->aux->btf; + if (tgt_link && tgt_link->tgt_prog) { + return tgt_link->tgt_prog->aux->btf; } else { return btf_vmlinux; } @@ -3733,14 +3733,17 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, struct bpf_insn_access_aux *info) { const struct btf_type *t = prog->aux->attach_func_proto; - struct bpf_prog *tgt_prog = prog->aux->linked_prog; struct btf *btf = bpf_prog_get_target_btf(prog); const char *tname = prog->aux->attach_func_name; struct bpf_verifier_log *log = info->log; + struct bpf_prog *tgt_prog = NULL; const struct btf_param *args; u32 nr_args, arg; int i, ret; + if (prog->aux->tgt_link) + tgt_prog = prog->aux->tgt_link->tgt_prog; + if (off % 8) { bpf_log(log, "func '%s' offset %d is not multiple of 8\n", tname, off); @@ -4572,7 +4575,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog, return -EFAULT; } if (prog_type == BPF_PROG_TYPE_EXT) - prog_type = prog->aux->linked_prog->type; + prog_type = prog->aux->tgt_link->tgt_prog->type; t = btf_type_by_id(btf, t->type); if (!t || !btf_type_is_func_proto(t)) { diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index ed0b3578867c..11696e849366 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -2130,7 +2130,7 @@ static void bpf_prog_free_deferred(struct work_struct *work) if (aux->prog->has_callchain_buf) put_callchain_buffers(); #endif - bpf_trampoline_put(aux->trampoline); + bpf_tracing_link_free(aux->tgt_link); for (i = 0; i < aux->func_cnt; i++) bpf_jit_free(aux->func[i]); if (aux->func_cnt) { @@ -2146,8 +2146,6 @@ void bpf_prog_free(struct bpf_prog *fp) { struct bpf_prog_aux *aux = fp->aux; - if (aux->linked_prog) - bpf_prog_put(aux->linked_prog); INIT_WORK(&aux->work, bpf_prog_free_deferred); schedule_work(&aux->work); } diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 4108ef3b828b..266ddd695914 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -2095,10 +2095,13 @@ static bool is_perfmon_prog_type(enum bpf_prog_type prog_type) /* last field in 'union bpf_attr' used by this command */ #define BPF_PROG_LOAD_LAST_FIELD attach_prog_fd +static struct bpf_tracing_link *bpf_tracing_link_create(struct bpf_prog *prog, + struct bpf_prog *tgt_prog); + static int bpf_prog_load(union bpf_attr *attr, union bpf_attr __user *uattr) { enum bpf_prog_type type = attr->prog_type; - struct bpf_prog *prog; + struct bpf_prog *prog, *tgt_prog = NULL; int err; char license[128]; bool is_gpl; @@ -2154,14 +2157,24 @@ static int bpf_prog_load(union bpf_attr *attr, union bpf_attr __user *uattr) prog->expected_attach_type = attr->expected_attach_type; prog->aux->attach_btf_id = attr->attach_btf_id; if (attr->attach_prog_fd) { - struct bpf_prog *tgt_prog; - tgt_prog = bpf_prog_get(attr->attach_prog_fd); if (IS_ERR(tgt_prog)) { err = PTR_ERR(tgt_prog); goto free_prog_nouncharge; } - prog->aux->linked_prog = tgt_prog; + } + + if (tgt_prog || prog->aux->attach_btf_id) { + struct bpf_tracing_link *link; + + link = bpf_tracing_link_create(prog, tgt_prog); + if (IS_ERR(link)) { + err = PTR_ERR(link); + if (tgt_prog) + bpf_prog_put(tgt_prog); + goto free_prog_nouncharge; + } + prog->aux->tgt_link = link; } prog->aux->offload_requested = !!attr->prog_ifindex; @@ -2495,14 +2508,20 @@ struct bpf_link *bpf_link_get_from_fd(u32 ufd) return link; } -struct bpf_tracing_link { - struct bpf_link link; - enum bpf_attach_type attach_type; -}; - static void bpf_tracing_link_release(struct bpf_link *link) { - WARN_ON_ONCE(bpf_trampoline_unlink_prog(link->prog)); + struct bpf_tracing_link *tr_link = + container_of(link, struct bpf_tracing_link, link); + + if (tr_link->trampoline) { + WARN_ON_ONCE(bpf_trampoline_unlink_prog(link->prog, + tr_link->trampoline)); + + bpf_trampoline_put(tr_link->trampoline); + } + + if (tr_link->tgt_prog) + bpf_prog_put(tr_link->tgt_prog); } static void bpf_tracing_link_dealloc(struct bpf_link *link) @@ -2542,6 +2561,34 @@ static const struct bpf_link_ops bpf_tracing_link_lops = { .fill_link_info = bpf_tracing_link_fill_link_info, }; +static struct bpf_tracing_link *bpf_tracing_link_create(struct bpf_prog *prog, + struct bpf_prog *tgt_prog) +{ + struct bpf_tracing_link *link; + + link = kzalloc(sizeof(*link), GFP_USER); + if (!link) + return ERR_PTR(-ENOMEM); + + bpf_link_init(&link->link, BPF_LINK_TYPE_TRACING, + &bpf_tracing_link_lops, prog); + link->attach_type = prog->expected_attach_type; + link->tgt_prog = tgt_prog; + + return link; +} + +void bpf_tracing_link_free(struct bpf_tracing_link *link) +{ + if (!link) + return; + if (link->trampoline) + bpf_trampoline_put(link->trampoline); + if (link->tgt_prog) + bpf_prog_put(link->tgt_prog); + kfree(link); +} + static int bpf_tracing_prog_attach(struct bpf_prog *prog) { struct bpf_link_primer link_primer; @@ -2574,28 +2621,27 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog) goto out_put_prog; } - link = kzalloc(sizeof(*link), GFP_USER); + link = xchg(&prog->aux->tgt_link, NULL); if (!link) { - err = -ENOMEM; + err = -ENOENT; goto out_put_prog; } - bpf_link_init(&link->link, BPF_LINK_TYPE_TRACING, - &bpf_tracing_link_lops, prog); - link->attach_type = prog->expected_attach_type; err = bpf_link_prime(&link->link, &link_primer); - if (err) { - kfree(link); - goto out_put_prog; - } + if (err) + goto out_restore_link; - err = bpf_trampoline_link_prog(prog); + err = bpf_trampoline_link_prog(prog, link->trampoline); if (err) { + bpf_trampoline_put(link->trampoline); + link->trampoline = NULL; bpf_link_cleanup(&link_primer); goto out_put_prog; } return bpf_link_settle(&link_primer); +out_restore_link: + WARN_ON_ONCE(cmpxchg(&prog->aux->tgt_link, NULL, link) != NULL); out_put_prog: bpf_prog_put(prog); return err; diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 7845913e7e41..e010a0641e99 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -261,14 +261,12 @@ static enum bpf_tramp_prog_type bpf_attach_type_to_tramp(struct bpf_prog *prog) } } -int bpf_trampoline_link_prog(struct bpf_prog *prog) +int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr) { enum bpf_tramp_prog_type kind; - struct bpf_trampoline *tr; int err = 0; int cnt; - tr = prog->aux->trampoline; kind = bpf_attach_type_to_tramp(prog); mutex_lock(&tr->mutex); if (tr->extension_prog) { @@ -301,7 +299,7 @@ int bpf_trampoline_link_prog(struct bpf_prog *prog) } hlist_add_head(&prog->aux->tramp_hlist, &tr->progs_hlist[kind]); tr->progs_cnt[kind]++; - err = bpf_trampoline_update(prog->aux->trampoline); + err = bpf_trampoline_update(tr); if (err) { hlist_del(&prog->aux->tramp_hlist); tr->progs_cnt[kind]--; @@ -312,13 +310,11 @@ int bpf_trampoline_link_prog(struct bpf_prog *prog) } /* bpf_trampoline_unlink_prog() should never fail. */ -int bpf_trampoline_unlink_prog(struct bpf_prog *prog) +int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline *tr) { enum bpf_tramp_prog_type kind; - struct bpf_trampoline *tr; int err; - tr = prog->aux->trampoline; kind = bpf_attach_type_to_tramp(prog); mutex_lock(&tr->mutex); if (kind == BPF_TRAMP_REPLACE) { @@ -330,7 +326,7 @@ int bpf_trampoline_unlink_prog(struct bpf_prog *prog) } hlist_del(&prog->aux->tramp_hlist); tr->progs_cnt[kind]--; - err = bpf_trampoline_update(prog->aux->trampoline); + err = bpf_trampoline_update(tr); out: mutex_unlock(&tr->mutex); return err; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index d38678319ca4..f0697a3619d7 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -2628,8 +2628,10 @@ static int check_map_access(struct bpf_verifier_env *env, u32 regno, static enum bpf_prog_type resolve_prog_type(struct bpf_prog *prog) { - return prog->aux->linked_prog ? prog->aux->linked_prog->type - : prog->type; + if (prog->aux->tgt_link && prog->aux->tgt_link->tgt_prog) + return prog->aux->tgt_link->tgt_prog->type; + + return prog->type; } static bool may_access_direct_pkt_data(struct bpf_verifier_env *env, @@ -11271,8 +11273,8 @@ int bpf_check_attach_target(struct bpf_verifier_log *log, static int check_attach_btf_id(struct bpf_verifier_env *env) { struct bpf_prog *prog = env->prog; - struct bpf_prog *tgt_prog = prog->aux->linked_prog; u32 btf_id = prog->aux->attach_btf_id; + struct bpf_prog *tgt_prog = NULL; struct btf_func_model fmodel; struct bpf_trampoline *tr; const struct btf_type *t; @@ -11281,6 +11283,9 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) int ret; u64 key; + if (prog->aux->tgt_link) + tgt_prog = prog->aux->tgt_link->tgt_prog; + if (prog->aux->sleepable && prog->type != BPF_PROG_TYPE_TRACING && prog->type != BPF_PROG_TYPE_LSM) { verbose(env, "Only fentry/fexit/fmod_ret and lsm programs can be sleepable\n"); @@ -11336,7 +11341,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) if (IS_ERR(tr)) return PTR_ERR(tr); - prog->aux->trampoline = tr; + prog->aux->tgt_link->trampoline = tr; return 0; } From patchwork Mon Sep 14 16:13:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 260914 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.6 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD77EC43461 for ; Mon, 14 Sep 2020 16:28:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 825762193E for ; Mon, 14 Sep 2020 16:28:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="iFEPrEm0" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726131AbgINQ2r (ORCPT ); Mon, 14 Sep 2020 12:28:47 -0400 Received: from us-smtp-2.mimecast.com ([205.139.110.61]:52243 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726519AbgINQNu (ORCPT ); Mon, 14 Sep 2020 12:13:50 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1600100011; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X5oPubw54gBEhkU8LHQCpNd8SkaDcVURsMVh7sKYhu8=; b=iFEPrEm0Ka2OlQ3cmNLYxEasZOd16UT1hihkMaDJmr07uXb6W8Mlp7qveKFKPM3qMUC583 wLBkMH1AW2ALnT14Rimc1Qkx3IoiFxWMkiT1Bfqa5gKyPH9SQHvKz80NnczPbOaWHKfISL p/yLHXpFg7g+bOVha7ZcKb9bU0mliQ8= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-568-LotBjAI5MOaKpaQWHMX8Tw-1; Mon, 14 Sep 2020 12:13:29 -0400 X-MC-Unique: LotBjAI5MOaKpaQWHMX8Tw-1 Received: by mail-wm1-f69.google.com with SMTP id m125so160129wmm.7 for ; Mon, 14 Sep 2020 09:13:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:date:message-id:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=X5oPubw54gBEhkU8LHQCpNd8SkaDcVURsMVh7sKYhu8=; b=OBB2ezGxdcmTL7mMJWvT4/s4skQ/fPK31MXJ7YtZ9hqIphRAQdL7wAgOhSxtOd1UYb xrD7CRs8YjA8iNZ0nMyxpxELaGdpjlHNRadncfRFLLAzvZ8loaQJTUKd3LfO9zXNoLY5 9bCUctkDf+a+l/b7TvtKaIc6dTNAVB5XWNuuja+2qWGQu3kg+V865zojfWwyP7MTa41G zyTeszNWkoG1k1WJ6l2y40mcNA9q3R8nC9Pj8+jaun8b7ED9XUEqPUhlFcRHWrdZJlBg VHUSIQelBxLzS0hdL9jFhzMRqUX8eqtDgCyT0lpkycBnrMlMvXrfIbmqC7LEa596ZQ1x K5VQ== X-Gm-Message-State: AOAM531Br9H1ZcumjPvYG4j00fX9x1UpTNIVyjcodnBnHOhsT4zz4rOM AsGWElm6PZM+1XVRyz+Vxl2YlyfgGN9r0zBGmbvAPMVwx01gNlWmwBcGEQgXu3ujnvtWvC7/3Ok tloZMrQAQ+CkqlFqM X-Received: by 2002:a5d:6547:: with SMTP id z7mr16555476wrv.322.1600100008335; Mon, 14 Sep 2020 09:13:28 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwR2iQXnO1m3G0oGnUQMs1QJGSGqLFOnY0v9cFsD5UucpzWZ/euusQ9YeZFnWsR7eyfjRpU1Q== X-Received: by 2002:a5d:6547:: with SMTP id z7mr16555452wrv.322.1600100008057; Mon, 14 Sep 2020 09:13:28 -0700 (PDT) Received: from alrua-x1.borgediget.toke.dk ([45.145.92.2]) by smtp.gmail.com with ESMTPSA id o15sm3004903wmh.29.2020.09.14.09.13.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Sep 2020 09:13:27 -0700 (PDT) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id 3CED81829CB; Mon, 14 Sep 2020 18:13:27 +0200 (CEST) Subject: [PATCH bpf-next v4 4/8] bpf: support attaching freplace programs to multiple attach points From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= To: Alexei Starovoitov Cc: Daniel Borkmann , Martin KaFai Lau , Song Liu , Yonghong Song , Andrii Nakryiko , John Fastabend , Jiri Olsa , Eelco Chaudron , KP Singh , netdev@vger.kernel.org, bpf@vger.kernel.org Date: Mon, 14 Sep 2020 18:13:27 +0200 Message-ID: <160010000716.80898.17666778529772660646.stgit@toke.dk> In-Reply-To: <160010000272.80898.13117015273092905112.stgit@toke.dk> References: <160010000272.80898.13117015273092905112.stgit@toke.dk> User-Agent: StGit/0.23 MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Toke Høiland-Jørgensen This enables support for attaching freplace programs to multiple attach points. It does this by amending the UAPI for bpf_link_Create with a target btf ID that can be used to supply the new attachment point along with the target program fd. The target must be compatible with the target that was supplied at program load time. The implementation reuses the checks that were factored out of check_attach_btf_id() to ensure compatibility between the BTF types of the old and new attachment. If these match, a new bpf_tracing_link will be created for the new attach target, allowing multiple attachments to co-exist simultaneously. The code could theoretically support multiple-attach of other types of tracing programs as well, but since I don't have a use case for any of those, there is no API support for doing so. Signed-off-by: Toke Høiland-Jørgensen --- include/linux/bpf.h | 3 + include/uapi/linux/bpf.h | 2 + kernel/bpf/syscall.c | 102 +++++++++++++++++++++++++++++++++++++--- kernel/bpf/verifier.c | 9 ++++ tools/include/uapi/linux/bpf.h | 2 + 5 files changed, 110 insertions(+), 8 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 8949e9794dc9..6834e6af9e09 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -751,6 +751,9 @@ struct bpf_prog_aux { struct hlist_node tramp_hlist; /* BTF_KIND_FUNC_PROTO for valid attach_btf_id */ const struct btf_type *attach_func_proto; + /* target BPF prog types for trace programs */ + enum bpf_prog_type tgt_prog_type; + enum bpf_attach_type tgt_attach_type; /* function name for valid attach_btf_id */ const char *attach_func_name; struct bpf_prog **func; diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 7dd314176df7..46eaa3024dc3 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -239,6 +239,7 @@ enum bpf_attach_type { BPF_XDP_CPUMAP, BPF_SK_LOOKUP, BPF_XDP, + BPF_TRACE_FREPLACE, __MAX_BPF_ATTACH_TYPE }; @@ -633,6 +634,7 @@ union bpf_attr { __u32 flags; /* extra flags */ __aligned_u64 iter_info; /* extra bpf_iter_link_info */ __u32 iter_info_len; /* iter_info length */ + __u32 target_btf_id; /* btf_id of target to attach to */ } link_create; struct { /* struct used by BPF_LINK_UPDATE command */ diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 266ddd695914..db0a391cd77f 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -4,6 +4,7 @@ #include #include #include +#include #include #include #include @@ -2578,6 +2579,7 @@ static struct bpf_tracing_link *bpf_tracing_link_create(struct bpf_prog *prog, return link; } + void bpf_tracing_link_free(struct bpf_tracing_link *link) { if (!link) @@ -2589,10 +2591,17 @@ void bpf_tracing_link_free(struct bpf_tracing_link *link) kfree(link); } -static int bpf_tracing_prog_attach(struct bpf_prog *prog) +static int bpf_tracing_prog_attach(struct bpf_prog *prog, + int tgt_prog_fd, + u32 btf_id) { struct bpf_link_primer link_primer; + struct bpf_prog *tgt_prog = NULL; struct bpf_tracing_link *link; + struct btf_func_model fmodel; + bool restore_link = false; + long addr; + u64 key; int err; switch (prog->type) { @@ -2620,16 +2629,81 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog) err = -EINVAL; goto out_put_prog; } + if (tgt_prog_fd) { + /* For now we only allow new targets for BPF_PROG_TYPE_EXT */ + if (prog->type != BPF_PROG_TYPE_EXT || !btf_id) { + err = -EINVAL; + goto out_put_prog; + } + + tgt_prog = bpf_prog_get(tgt_prog_fd); + if (IS_ERR(tgt_prog)) { + err = PTR_ERR(tgt_prog); + goto out_put_prog; + } + + key = ((u64)tgt_prog->aux->id) << 32 | btf_id; + } else if (btf_id) { + err = -EINVAL; + goto out_put_prog; + } link = xchg(&prog->aux->tgt_link, NULL); + if (link && tgt_prog) { + if (link->trampoline->key != key) { + /* supplying a tgt_prog is always destructive of the + * target ref from the extension prog, which means that + * mixing old and new API is not supported. + */ + bpf_tracing_link_free(link); + link = NULL; + } else { + /* re-using link that already has ref on tgt_prog, don't + * take another + */ + bpf_prog_put(tgt_prog); + tgt_prog = NULL; + } + } else if (link) { + /* called without a target fd, so restore link on failure for + * compatibility + */ + restore_link = true; + } + if (!link) { - err = -ENOENT; - goto out_put_prog; + struct bpf_trampoline *tr; + + if (!tgt_prog) { + err = -ENOENT; + goto out_put_prog; + } + + err = bpf_check_attach_target(NULL, prog, tgt_prog, btf_id, + &fmodel, &addr, NULL, NULL); + if (err) { + bpf_prog_put(tgt_prog); + goto out_put_prog; + } + + link = bpf_tracing_link_create(prog, tgt_prog); + if (IS_ERR(link)) { + bpf_prog_put(tgt_prog); + err = PTR_ERR(link); + goto out_put_prog; + } + + tr = bpf_trampoline_get(key, (void *)addr, &fmodel); + if (IS_ERR(tr)) { + err = PTR_ERR(tr); + goto out_put_link; + } + link->trampoline = tr; } err = bpf_link_prime(&link->link, &link_primer); if (err) - goto out_restore_link; + goto out_put_link; err = bpf_trampoline_link_prog(prog, link->trampoline); if (err) { @@ -2640,8 +2714,11 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog) } return bpf_link_settle(&link_primer); -out_restore_link: - WARN_ON_ONCE(cmpxchg(&prog->aux->tgt_link, NULL, link) != NULL); +out_put_link: + if (restore_link) + WARN_ON_ONCE(cmpxchg(&prog->aux->tgt_link, NULL, link) != NULL); + else + bpf_tracing_link_free(link); out_put_prog: bpf_prog_put(prog); return err; @@ -2756,7 +2833,7 @@ static int bpf_raw_tracepoint_open(const union bpf_attr *attr) tp_name = prog->aux->attach_func_name; break; } - return bpf_tracing_prog_attach(prog); + return bpf_tracing_prog_attach(prog, 0, 0); case BPF_PROG_TYPE_RAW_TRACEPOINT: case BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE: if (strncpy_from_user(buf, @@ -2876,6 +2953,8 @@ attach_type_to_prog_type(enum bpf_attach_type attach_type) case BPF_CGROUP_GETSOCKOPT: case BPF_CGROUP_SETSOCKOPT: return BPF_PROG_TYPE_CGROUP_SOCKOPT; + case BPF_TRACE_FREPLACE: + return BPF_PROG_TYPE_EXT; case BPF_TRACE_ITER: return BPF_PROG_TYPE_TRACING; case BPF_SK_LOOKUP: @@ -3936,10 +4015,16 @@ static int tracing_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog * prog->expected_attach_type == BPF_TRACE_ITER) return bpf_iter_link_attach(attr, prog); + if (attr->link_create.attach_type == BPF_TRACE_FREPLACE && + !prog->expected_attach_type) + return bpf_tracing_prog_attach(prog, + attr->link_create.target_fd, + attr->link_create.target_btf_id); + return -EINVAL; } -#define BPF_LINK_CREATE_LAST_FIELD link_create.iter_info_len +#define BPF_LINK_CREATE_LAST_FIELD link_create.target_btf_id static int link_create(union bpf_attr *attr) { enum bpf_prog_type ptype; @@ -3973,6 +4058,7 @@ static int link_create(union bpf_attr *attr) ret = cgroup_bpf_link_attach(attr, prog); break; case BPF_PROG_TYPE_TRACING: + case BPF_PROG_TYPE_EXT: ret = tracing_bpf_link_attach(attr, prog); break; case BPF_PROG_TYPE_FLOW_DISSECTOR: diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index f0697a3619d7..bf8b16721c82 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -11205,6 +11205,12 @@ int bpf_check_attach_target(struct bpf_verifier_log *log, if (!btf_type_is_func_proto(t)) return -EINVAL; + if ((prog->aux->tgt_prog_type && + prog->aux->tgt_prog_type != tgt_prog->type) || + (prog->aux->tgt_attach_type && + prog->aux->tgt_attach_type != tgt_prog->expected_attach_type)) + return -EINVAL; + if (tgt_prog && conservative) t = NULL; @@ -11306,6 +11312,9 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) return ret; if (tgt_prog) { + prog->aux->tgt_prog_type = tgt_prog->type; + prog->aux->tgt_attach_type = tgt_prog->expected_attach_type; + if (prog->type == BPF_PROG_TYPE_EXT) { env->ops = bpf_verifier_ops[tgt_prog->type]; prog->expected_attach_type = diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 7dd314176df7..46eaa3024dc3 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -239,6 +239,7 @@ enum bpf_attach_type { BPF_XDP_CPUMAP, BPF_SK_LOOKUP, BPF_XDP, + BPF_TRACE_FREPLACE, __MAX_BPF_ATTACH_TYPE }; @@ -633,6 +634,7 @@ union bpf_attr { __u32 flags; /* extra flags */ __aligned_u64 iter_info; /* extra bpf_iter_link_info */ __u32 iter_info_len; /* iter_info length */ + __u32 target_btf_id; /* btf_id of target to attach to */ } link_create; struct { /* struct used by BPF_LINK_UPDATE command */ From patchwork Mon Sep 14 16:13:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Patchwork-Id: 260913 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.6 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05F4EC43461 for ; Mon, 14 Sep 2020 16:31:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B30FB214F1 for ; Mon, 14 Sep 2020 16:31:49 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="OA6nBLGa" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726019AbgINQbk (ORCPT ); Mon, 14 Sep 2020 12:31:40 -0400 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:36645 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726524AbgINQNt (ORCPT ); Mon, 14 Sep 2020 12:13:49 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1600100013; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KIRSjC4TU6Nemo+ZjQS/Vw28P2tbe5/t6vRROo8Kshk=; b=OA6nBLGatZd8uV/i2yyfyz4tPuwCoRSk0IZkjB3ubOlNYr3sRgvb2yozjkKtkdfxjezB2U sdRUzi4FZTD8zujPPPhTsUiV8FNCGC5+GkmZ5kCfsAvAnAdHcYGuGKWCgpAjdY/c/61pTe EtB+SYsFaKTwznzaZRjgq+9/AS9LVzs= Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-302-wSnjEZOPMiqhjme2EzwRMA-1; Mon, 14 Sep 2020 12:13:32 -0400 X-MC-Unique: wSnjEZOPMiqhjme2EzwRMA-1 Received: by mail-wm1-f70.google.com with SMTP id q205so1709094wme.0 for ; Mon, 14 Sep 2020 09:13:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:date:message-id:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=KIRSjC4TU6Nemo+ZjQS/Vw28P2tbe5/t6vRROo8Kshk=; b=ZN/bA/08q7itcLfxP0mvKshu0mMr1TFR4q/Wic1wrjUiGksBStojytS6xs7srVgidv EKztLL9yFR6nMPSz2JVZRNz39GhvCmh3aMBjDs1AoAMKPRTi9+eYdZVhy1OxXYXDLLNE 6VM4wkH/Jt+PpgAuHkgrwWyapqsExqQPGkfjGA+B/3InQf3A2wxAeBVXDX9ypGDpW4Hc q48G4WDRln42+TKKI736y55F6goLvkuyrR9y3kEtpT951SLoilEsRm5fui0MQWFCe1Cm sPUvpA9E4uNAO/wPFNn64/aMOJIWg55WVDGU3ErF3oOnPZxMVZjDLfwBP8jh1+h+fGIy T9Rw== X-Gm-Message-State: AOAM531LmGlLbpbj6vwCAEAe35UqxdsZrHTucyGNQP1h2YnLDceJmWZH l2Ef5FQCT2vflKmVlZmNrKxHNafSyirhYUkX14jWG/nUrkxlsTgTUWoc2jIHjQpHYFXr4gvx0xt Ug6gifooahv6SO7ho X-Received: by 2002:a05:600c:230f:: with SMTP id 15mr135649wmo.186.1600100010613; Mon, 14 Sep 2020 09:13:30 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwsLyvw0+rZnKrTQr3mhXbwl+qrAkBTO3UgsCwKeuhgTjF+f7ZNzvDn+bYHnniFDCGscZqQkw== X-Received: by 2002:a05:600c:230f:: with SMTP id 15mr135617wmo.186.1600100010312; Mon, 14 Sep 2020 09:13:30 -0700 (PDT) Received: from alrua-x1.borgediget.toke.dk ([2a0c:4d80:42:443::2]) by smtp.gmail.com with ESMTPSA id v6sm21237616wrt.90.2020.09.14.09.13.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Sep 2020 09:13:29 -0700 (PDT) Received: by alrua-x1.borgediget.toke.dk (Postfix, from userid 1000) id 568281829CB; Mon, 14 Sep 2020 18:13:28 +0200 (CEST) Subject: [PATCH bpf-next v4 5/8] bpf: Fix context type resolving for extension programs From: =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rgensen?= To: Alexei Starovoitov Cc: Daniel Borkmann , Martin KaFai Lau , Song Liu , Yonghong Song , Andrii Nakryiko , John Fastabend , Jiri Olsa , Eelco Chaudron , KP Singh , netdev@vger.kernel.org, bpf@vger.kernel.org Date: Mon, 14 Sep 2020 18:13:28 +0200 Message-ID: <160010000825.80898.12912186494003019860.stgit@toke.dk> In-Reply-To: <160010000272.80898.13117015273092905112.stgit@toke.dk> References: <160010000272.80898.13117015273092905112.stgit@toke.dk> User-Agent: StGit/0.23 MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Toke Høiland-Jørgensen Eelco reported we can't properly access arguments if the tracing program is attached to extension program. Having following program: SEC("classifier/test_pkt_md_access") int test_pkt_md_access(struct __sk_buff *skb) with its extension: SEC("freplace/test_pkt_md_access") int test_pkt_md_access_new(struct __sk_buff *skb) and tracing that extension with: SEC("fentry/test_pkt_md_access_new") int BPF_PROG(fentry, struct sk_buff *skb) It's not possible to access skb argument in the fentry program, with following error from verifier: ; int BPF_PROG(fentry, struct sk_buff *skb) 0: (79) r1 = *(u64 *)(r1 +0) invalid bpf_context access off=0 size=8 The problem is that btf_ctx_access gets the context type for the traced program, which is in this case the extension. But when we trace extension program, we want to get the context type of the program that the extension is attached to, so we can access the argument properly in the trace program. This version of the patch is tweaked slightly from Jiri's original one, since the refactoring in the previous patches means we have to get the target prog type from the new variable in prog->aux instead of directly from the target prog. Reported-by: Eelco Chaudron Suggested-by: Jiri Olsa Signed-off-by: Toke Høiland-Jørgensen --- kernel/bpf/btf.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index e10f13f8251c..1a48253ba168 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -3863,7 +3863,14 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, info->reg_type = PTR_TO_BTF_ID; if (tgt_prog) { - ret = btf_translate_to_vmlinux(log, btf, t, tgt_prog->type, arg); + enum bpf_prog_type tgt_type; + + if (tgt_prog->type == BPF_PROG_TYPE_EXT) + tgt_type = tgt_prog->aux->tgt_prog_type; + else + tgt_type = tgt_prog->type; + + ret = btf_translate_to_vmlinux(log, btf, t, tgt_type, arg); if (ret > 0) { info->btf_id = ret; return true;