Message ID | 20210629095543.391ac606@oasis.local.home |
---|---|
State | New |
Headers | show |
Series | tracepoint: Add tracepoint_probe_register_may_exist() for BPF tracing | expand |
On Tue, Jun 29, 2021 at 6:55 AM Steven Rostedt <rostedt@goodmis.org> wrote: > > From: "Steven Rostedt (VMware)" <rostedt@goodmis.org> > > All internal use cases for tracepoint_probe_register() is set to not ever > be called with the same function and data. If it is, it is considered a > bug, as that means the accounting of handling tracepoints is corrupted. > If the function and data for a tracepoint is already registered when > tracepoint_probe_register() is called, it will call WARN_ON_ONCE() and > return with EEXISTS. > > The BPF system call can end up calling tracepoint_probe_register() with > the same data, which now means that this can trigger the warning because > of a user space process. As WARN_ON_ONCE() should not be called because > user space called a system call with bad data, there needs to be a way to > register a tracepoint without triggering a warning. > > Enter tracepoint_probe_register_may_exist(), which can be called, but will > not cause a WARN_ON() if the probe already exists. It will still error out > with EEXIST, which will then be sent to the user space that performed the > BPF system call. > > This keeps the previous testing for issues with other users of the > tracepoint code, while letting BPF call it with duplicated data and not > warn about it. There doesn't seem to be anything conceptually wrong with attaching the same BPF program twice to the same tracepoint. Is it a hard requirement to have a unique tp+callback combination, or was it done mostly to detect an API misuse? How hard is it to support such use cases? I was surprised to discover this is not supported (though I never had a use for this, had to construct a test to see the warning). > > Link: https://lore.kernel.org/lkml/20210626135845.4080-1-penguin-kernel@I-love.SAKURA.ne.jp/ > Link: https://syzkaller.appspot.com/bug?id=41f4318cf01762389f4d1c1c459da4f542fe5153 [1]` > > Cc: stable@vger.kernel.org > Fixes: c4f6699dfcb85 ("bpf: introduce BPF_RAW_TRACEPOINT") > Reported-by: syzbot <syzbot+721aa903751db87aa244@syzkaller.appspotmail.com> > Reported-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp> > Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org> > --- > > #syz test: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master > > include/linux/tracepoint.h | 10 ++++++++++ > kernel/trace/bpf_trace.c | 3 ++- > kernel/tracepoint.c | 33 ++++++++++++++++++++++++++++++--- > 3 files changed, 42 insertions(+), 4 deletions(-) > [...]
On Wed, 7 Jul 2021 15:12:28 -0700 Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote: > There doesn't seem to be anything conceptually wrong with attaching > the same BPF program twice to the same tracepoint. Is it a hard > requirement to have a unique tp+callback combination, or was it done > mostly to detect an API misuse? How hard is it to support such use > cases? > > I was surprised to discover this is not supported (though I never had > a use for this, had to construct a test to see the warning). The callback is identified by the function and its data combination. If there's two callbacks calling the same function with the same data on the same tracepoint, one question is, why? And the second is how do you differentiate the two? -- Steve
On Wed, Jul 7, 2021 at 3:45 PM Steven Rostedt <rostedt@goodmis.org> wrote: > > On Wed, 7 Jul 2021 15:12:28 -0700 > Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote: > > > There doesn't seem to be anything conceptually wrong with attaching > > the same BPF program twice to the same tracepoint. Is it a hard > > requirement to have a unique tp+callback combination, or was it done > > mostly to detect an API misuse? How hard is it to support such use > > cases? > > > > I was surprised to discover this is not supported (though I never had > > a use for this, had to construct a test to see the warning). > > The callback is identified by the function and its data combination. If > there's two callbacks calling the same function with the same data on > the same tracepoint, one question is, why? And the second is how do you > differentiate the two? For places where multiple BPF programs can be attached (kprobes, cgroup programs, etc), we don't put a restriction that all programs have to be unique. It's totally legal to have the same program attached multiple times. So having this for tracepoints will be a one-off behavior. As for why the user might need that, it's up to the user and I don't want to speculate because it will always sound contrived without a specific production use case. But people are very creative and we try not to dictate how and what can be done if it doesn't break any fundamental assumption and safety. > > -- Steve
On Wed, 7 Jul 2021 16:49:26 -0700 Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote: > As for why the user might need that, it's up to the user and I don't > want to speculate because it will always sound contrived without a > specific production use case. But people are very creative and we try > not to dictate how and what can be done if it doesn't break any > fundamental assumption and safety. I guess it doesn't matter, because if they try to do it, the second attachment will simply fail to attach. -- Steve
On Wed, Jul 7, 2021 at 5:05 PM Steven Rostedt <rostedt@goodmis.org> wrote: > > On Wed, 7 Jul 2021 16:49:26 -0700 > Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote: > > > As for why the user might need that, it's up to the user and I don't > > want to speculate because it will always sound contrived without a > > specific production use case. But people are very creative and we try > > not to dictate how and what can be done if it doesn't break any > > fundamental assumption and safety. > > I guess it doesn't matter, because if they try to do it, the second > attachment will simply fail to attach. > But not for the kprobe case. And it might not always be possible to know that the same BPF program is being attached. It could be attached by different processes that re-use pinned program (without being aware of each other). Or it could be done from some generic library that just accepts prog_fd and doesn't really know the exact BPF program and whether it was already attached. Not sure why it doesn't matter that attachment will fail where it is expected to succeed. The question is rather why such restriction? > -- Steve
On Wed, 7 Jul 2021 17:23:54 -0700 Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote: > On Wed, Jul 7, 2021 at 5:05 PM Steven Rostedt <rostedt@goodmis.org> wrote: > > > > On Wed, 7 Jul 2021 16:49:26 -0700 > > Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote: > > > > > As for why the user might need that, it's up to the user and I don't > > > want to speculate because it will always sound contrived without a > > > specific production use case. But people are very creative and we try > > > not to dictate how and what can be done if it doesn't break any > > > fundamental assumption and safety. > > > > I guess it doesn't matter, because if they try to do it, the second > > attachment will simply fail to attach. > > > > But not for the kprobe case. What do you mean "not for the kprobe case"? What kprobe case? You attach the same program twice to the same kprobe? Or do you create two kprobes at the same location? > > And it might not always be possible to know that the same BPF program > is being attached. It could be attached by different processes that > re-use pinned program (without being aware of each other). Or it could > be done from some generic library that just accepts prog_fd and > doesn't really know the exact BPF program and whether it was already > attached. > > Not sure why it doesn't matter that attachment will fail where it is > expected to succeed. The question is rather why such restriction? Why is it expected to succeed? It never did. And why such a restriction? Because it complicates the code, and there's no good use case to do so. Why complicate something for little reward? -- Steve
----- On Jul 7, 2021, at 8:23 PM, Andrii Nakryiko andrii.nakryiko@gmail.com wrote: > On Wed, Jul 7, 2021 at 5:05 PM Steven Rostedt <rostedt@goodmis.org> wrote: >> >> On Wed, 7 Jul 2021 16:49:26 -0700 >> Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote: >> >> > As for why the user might need that, it's up to the user and I don't >> > want to speculate because it will always sound contrived without a >> > specific production use case. But people are very creative and we try >> > not to dictate how and what can be done if it doesn't break any >> > fundamental assumption and safety. >> >> I guess it doesn't matter, because if they try to do it, the second >> attachment will simply fail to attach. >> > > But not for the kprobe case. > > And it might not always be possible to know that the same BPF program > is being attached. It could be attached by different processes that > re-use pinned program (without being aware of each other). Or it could > be done from some generic library that just accepts prog_fd and > doesn't really know the exact BPF program and whether it was already > attached. > > Not sure why it doesn't matter that attachment will fail where it is > expected to succeed. The question is rather why such restriction? Before eBPF came to exist, all in-kernel users of the tracepoint API never required multiple registrations for a given (tracepoint, probe, data) tuple. This allowed us to expose an API which can consider that the (tracepoint, probe, data) tuple is unique for each registration/unregistration pair, and therefore use that same tuple for unregistration. Refusing multiple registrations for a given tuple allows us to forgo the complexity of reference counting for duplicate registrations, and provide immediate feedback to misbehaving tracers which have duplicate registration or unbalanced registration/unregistration pairs. From the perspective of a ring buffer tracer, the notion of multiple instances of a given (tracepoint, probe, data) tuple is rather silly: it would mean that a given tracepoint hit would generate many instances of the exact same event into the same trace buffer. AFAIR, having the WARN_ON_ONCE() within the tracepoint code to highlight this kind of misuse allowed Steven to find a few unbalanced registration/unregistration issues while developing ftrace in the past. I vaguely recall that it triggered for blktrace at some point as well. Considering that allowing duplicates would add complexity to the tracepoint code, what is the use-case justifying allowing many instances of the exact same callback and data for a given tracepoint ? One key difference I notice here between eBPF and ring buffer tracers is what eBPF considers a "program". AFAIU (please let me know if I'm mistaken), the "callback" argument provided by eBPF to the tracepoint API is a limited set of trampoline routines. The bulk of the eBPF "program" is provided in the "data" argument. So this means the "program" is both the eBPF code and some context. So I understand that a given eBPF code could be loaded more than once for a given tracepoint, but I would expect that each registration on a given tracepoint be provided with its own "context", otherwise we end up in a similar situation as the ring buffer's duplicated events scenario I explained above. Also, we should discuss whether kprobes might benefit from being more strict by rejecting duplicated (instrumentation site, probe, data) tuples. Thanks, Mathieu -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
On Wed, Jul 7, 2021 at 5:43 PM Steven Rostedt <rostedt@goodmis.org> wrote: > > On Wed, 7 Jul 2021 17:23:54 -0700 > Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote: > > > On Wed, Jul 7, 2021 at 5:05 PM Steven Rostedt <rostedt@goodmis.org> wrote: > > > > > > On Wed, 7 Jul 2021 16:49:26 -0700 > > > Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote: > > > > > > > As for why the user might need that, it's up to the user and I don't > > > > want to speculate because it will always sound contrived without a > > > > specific production use case. But people are very creative and we try > > > > not to dictate how and what can be done if it doesn't break any > > > > fundamental assumption and safety. > > > > > > I guess it doesn't matter, because if they try to do it, the second > > > attachment will simply fail to attach. > > > > > > > But not for the kprobe case. > > What do you mean "not for the kprobe case"? What kprobe case? > > You attach the same program twice to the same kprobe? Or do you create > two kprobes at the same location? > I meant attaching the same BPF program twice to the same kernel function through the kprobe mechanism (through perf_event_open() syscall). From user perspective it's one BPF program attached twice to the same kprobe. Not entirely sure if two perf_event_open() calls will create two kprobes or re-use one internally. > > > > And it might not always be possible to know that the same BPF program > > is being attached. It could be attached by different processes that > > re-use pinned program (without being aware of each other). Or it could > > be done from some generic library that just accepts prog_fd and > > doesn't really know the exact BPF program and whether it was already > > attached. > > > > Not sure why it doesn't matter that attachment will fail where it is > > expected to succeed. The question is rather why such restriction? > > Why is it expected to succeed? It never did. And why such a > restriction? Because it complicates the code, and there's no good use > case to do so. Why complicate something for little reward? See above about kprobe for why it was my expectation. But it was my original question whether this causes some complications or it's just an attempt to detect API mis-use. Seems like it's the former, alright. > > -- Steve
On Thu, Jul 8, 2021 at 10:30 AM Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote: > > ----- On Jul 7, 2021, at 8:23 PM, Andrii Nakryiko andrii.nakryiko@gmail.com wrote: > > > On Wed, Jul 7, 2021 at 5:05 PM Steven Rostedt <rostedt@goodmis.org> wrote: > >> > >> On Wed, 7 Jul 2021 16:49:26 -0700 > >> Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote: > >> > >> > As for why the user might need that, it's up to the user and I don't > >> > want to speculate because it will always sound contrived without a > >> > specific production use case. But people are very creative and we try > >> > not to dictate how and what can be done if it doesn't break any > >> > fundamental assumption and safety. > >> > >> I guess it doesn't matter, because if they try to do it, the second > >> attachment will simply fail to attach. > >> > > > > But not for the kprobe case. > > > > And it might not always be possible to know that the same BPF program > > is being attached. It could be attached by different processes that > > re-use pinned program (without being aware of each other). Or it could > > be done from some generic library that just accepts prog_fd and > > doesn't really know the exact BPF program and whether it was already > > attached. > > > > Not sure why it doesn't matter that attachment will fail where it is > > expected to succeed. The question is rather why such restriction? > > Before eBPF came to exist, all in-kernel users of the tracepoint API never > required multiple registrations for a given (tracepoint, probe, data) tuple. > > This allowed us to expose an API which can consider that the (tracepoint, probe, data) > tuple is unique for each registration/unregistration pair, and therefore use that same > tuple for unregistration. Refusing multiple registrations for a given tuple allows us to > forgo the complexity of reference counting for duplicate registrations, and provide > immediate feedback to misbehaving tracers which have duplicate registration or > unbalanced registration/unregistration pairs. > > From the perspective of a ring buffer tracer, the notion of multiple instances of > a given (tracepoint, probe, data) tuple is rather silly: it would mean that a given > tracepoint hit would generate many instances of the exact same event into the > same trace buffer. > > AFAIR, having the WARN_ON_ONCE() within the tracepoint code to highlight this kind of misuse > allowed Steven to find a few unbalanced registration/unregistration issues while developing > ftrace in the past. I vaguely recall that it triggered for blktrace at some point as well. > > Considering that allowing duplicates would add complexity to the tracepoint code, > what is the use-case justifying allowing many instances of the exact same callback > and data for a given tracepoint ? It wasn't clear to me if supporting this would cause any added complexity, which is why I asked. > > One key difference I notice here between eBPF and ring buffer tracers is what eBPF > considers a "program". AFAIU (please let me know if I'm mistaken), the "callback" > argument provided by eBPF to the tracepoint API is a limited set of trampoline routines. > The bulk of the eBPF "program" is provided in the "data" argument. So this means the > "program" is both the eBPF code and some context. > > So I understand that a given eBPF code could be loaded more than once for a given No, it turns out it can't, I was just surprised to learn that. Surprised, because AFAIK we don't have such restrictions on uniqueness of attached BPF programs anywhere else where multiple BPF programs are allowed. > tracepoint, but I would expect that each registration on a given tracepoint be > provided with its own "context", otherwise we end up in a similar situation as the > ring buffer's duplicated events scenario I explained above. > > Also, we should discuss whether kprobes might benefit from being more strict by > rejecting duplicated (instrumentation site, probe, data) tuples. > > Thanks, > > Mathieu > > -- > Mathieu Desnoyers > EfficiOS Inc. > http://www.efficios.com
diff --git a/include/linux/tracepoint.h b/include/linux/tracepoint.h index 13f65420f188..ab58696d0ddd 100644 --- a/include/linux/tracepoint.h +++ b/include/linux/tracepoint.h @@ -41,7 +41,17 @@ extern int tracepoint_probe_register_prio(struct tracepoint *tp, void *probe, void *data, int prio); extern int +tracepoint_probe_register_prio_may_exist(struct tracepoint *tp, void *probe, void *data, + int prio); +extern int tracepoint_probe_unregister(struct tracepoint *tp, void *probe, void *data); +static inline int +tracepoint_probe_register_may_exist(struct tracepoint *tp, void *probe, + void *data) +{ + return tracepoint_probe_register_prio_may_exist(tp, probe, data, + TRACEPOINT_DEFAULT_PRIO); +} extern void for_each_kernel_tracepoint(void (*fct)(struct tracepoint *tp, void *priv), void *priv); diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index 7a52bc172841..f0568b3d6bd1 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -1840,7 +1840,8 @@ static int __bpf_probe_register(struct bpf_raw_event_map *btp, struct bpf_prog * if (prog->aux->max_tp_access > btp->writable_size) return -EINVAL; - return tracepoint_probe_register(tp, (void *)btp->bpf_func, prog); + return tracepoint_probe_register_may_exist(tp, (void *)btp->bpf_func, + prog); } int bpf_probe_register(struct bpf_raw_event_map *btp, struct bpf_prog *prog) diff --git a/kernel/tracepoint.c b/kernel/tracepoint.c index 9f478d29b926..976bf8ce8039 100644 --- a/kernel/tracepoint.c +++ b/kernel/tracepoint.c @@ -273,7 +273,8 @@ static void tracepoint_update_call(struct tracepoint *tp, struct tracepoint_func * Add the probe function to a tracepoint. */ static int tracepoint_add_func(struct tracepoint *tp, - struct tracepoint_func *func, int prio) + struct tracepoint_func *func, int prio, + bool warn) { struct tracepoint_func *old, *tp_funcs; int ret; @@ -288,7 +289,7 @@ static int tracepoint_add_func(struct tracepoint *tp, lockdep_is_held(&tracepoints_mutex)); old = func_add(&tp_funcs, func, prio); if (IS_ERR(old)) { - WARN_ON_ONCE(PTR_ERR(old) != -ENOMEM); + WARN_ON_ONCE(warn && PTR_ERR(old) != -ENOMEM); return PTR_ERR(old); } @@ -343,6 +344,32 @@ static int tracepoint_remove_func(struct tracepoint *tp, return 0; } +/** + * tracepoint_probe_register_prio_may_exist - Connect a probe to a tracepoint with priority + * @tp: tracepoint + * @probe: probe handler + * @data: tracepoint data + * @prio: priority of this function over other registered functions + * + * Same as tracepoint_probe_register_prio() except that it will not warn + * if the tracepoint is already registered. + */ +int tracepoint_probe_register_prio_may_exist(struct tracepoint *tp, void *probe, + void *data, int prio) +{ + struct tracepoint_func tp_func; + int ret; + + mutex_lock(&tracepoints_mutex); + tp_func.func = probe; + tp_func.data = data; + tp_func.prio = prio; + ret = tracepoint_add_func(tp, &tp_func, prio, false); + mutex_unlock(&tracepoints_mutex); + return ret; +} +EXPORT_SYMBOL_GPL(tracepoint_probe_register_prio_may_exist); + /** * tracepoint_probe_register_prio - Connect a probe to a tracepoint with priority * @tp: tracepoint @@ -366,7 +393,7 @@ int tracepoint_probe_register_prio(struct tracepoint *tp, void *probe, tp_func.func = probe; tp_func.data = data; tp_func.prio = prio; - ret = tracepoint_add_func(tp, &tp_func, prio); + ret = tracepoint_add_func(tp, &tp_func, prio, true); mutex_unlock(&tracepoints_mutex); return ret; }