From patchwork Tue Jul 25 21:25:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rae Moar X-Patchwork-Id: 707323 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A66D0C001DF for ; Tue, 25 Jul 2023 21:25:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230472AbjGYVZh (ORCPT ); Tue, 25 Jul 2023 17:25:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56774 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229844AbjGYVZd (ORCPT ); Tue, 25 Jul 2023 17:25:33 -0400 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D30021FEB for ; Tue, 25 Jul 2023 14:25:30 -0700 (PDT) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-56cf9a86277so77741947b3.3 for ; Tue, 25 Jul 2023 14:25:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690320330; x=1690925130; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+60Tc7YqYxSgNOnPP6t+KCX+Q0mMLWkCi2vWrGv2WOA=; b=O33O6dcN7np5BUSga9TXzG1LDLwt1AFRB7D6mNqrgd/2N/f1TN/5Y3FUbAvMtzj/DT MVrwmIidggfWqd6fg/DphlOTAUG0R++Z5hulPSPviX/CbDDO4FhMeKDEUCYLxFEqbTmc ocoDPrRY+S8a0tv6XMniRNePrDcDfuhGp6KVWJhANqbuzk6PGWajjjCLf2c+VtOPsZFj c0PNsStVMCJTSTqeGLIsErlBHJswA3CCjizm1MMZ1lSjcltKvwIgmXoA8jTUYtmISxx5 3Da1n/xYXYDFBlGJA/FABAh52gCAMmNLUaMkf2VsOXXYPrh2fwVGDd+kmXkMISglBU42 yKCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690320330; x=1690925130; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+60Tc7YqYxSgNOnPP6t+KCX+Q0mMLWkCi2vWrGv2WOA=; b=XJOfl7jMbskPogy9A7zqraVbBjJCc62uGV3M3xMfKBeEGgOpFARIVyAjH3oo4MdtVl fvp85iN8Mqz9zsBsiP3OJ1uhoJbGk35maH6J3mTex+0g6LU5FkjWTazduVgrJMbHsglh Myv39ESgNz7IEfxVF/COpVIuW9HiaY9Qpx7G23ltY/2gFJdQTVyRGgBwUCuYMHHLrHb9 +OgPGeRglKSZZsI25jylAj58J741pLZ8RylYsgHPydIJZrZ28BV6oq8KbigNi/Ewy52z QZdrwyEslxs34qIc+zDSql4lz6HZ8n4/hnr5zl+wPzr3B38MB+qqkNLUI1f3OzJBVljG jdIg== X-Gm-Message-State: ABy/qLbFGL/9j+UBF+tpvRCxgOlzB5hgY0Yx71dtDWpxAgIq0uzEj8u1 icGiIdogZVB1QontxMoHBQqQQFkBgw== X-Google-Smtp-Source: APBJJlHo3rJD2idTeke3bCU5Z6xeu0pVQ5E+a6aP1e2DoWT50t3ZppaKXRbMhaMU72TjL7utTtbRDtgWLw== X-Received: from rmoar-specialist.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:45d3]) (user=rmoar job=sendgmr) by 2002:a81:b647:0:b0:583:4551:858d with SMTP id h7-20020a81b647000000b005834551858dmr4146ywk.9.1690320330168; Tue, 25 Jul 2023 14:25:30 -0700 (PDT) Date: Tue, 25 Jul 2023 21:25:12 +0000 In-Reply-To: <20230725212522.1622716-1-rmoar@google.com> Mime-Version: 1.0 References: <20230725212522.1622716-1-rmoar@google.com> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog Message-ID: <20230725212522.1622716-2-rmoar@google.com> Subject: [PATCH v3 1/9] kunit: Add test attributes API structure From: Rae Moar To: shuah@kernel.org, davidgow@google.com, dlatypov@google.com, brendan.higgins@linux.dev Cc: linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, keescook@chromium.org, linux-hardening@vger.kernel.org, jstultz@google.com, tglx@linutronix.de, sboyd@kernel.org, Rae Moar Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Add the basic structure of the test attribute API to KUnit, which can be used to save and access test associated data. Add attributes.c and attributes.h to hold associated structs and functions for the API. Create a struct that holds a variety of associated helper functions for each test attribute. These helper functions will be used to get the attribute value, convert the value to a string, and filter based on the value. This struct is flexible by design to allow for attributes of numerous types and contexts. Add a method to print test attributes in the format of "# [.]: ". Example for a suite: "# speed: slow" Example for a test case: "# test_case.speed: very_slow" Use this method to report attributes in the KTAP output (KTAP spec: https://docs.kernel.org/dev-tools/ktap.html) and _list_tests output when kernel's new kunit.action=list_attr option is used. Note this is derivative of the kunit.action=list option. In test.h, add fields and associated helper functions to test cases and suites to hold user-inputted test attributes. Reviewed-by: David Gow Signed-off-by: Rae Moar --- Changes since v2: - No changes. Changes since v1: - No changes. Changes since RFC v2: - No major changes. Changes since RFC v1: - Add list_attr option to only include attribute in the _list_tests output when this module param is set - Add printing options for attributes to print always, print only for suites, or print never. include/kunit/attributes.h | 19 +++++++++ include/kunit/test.h | 33 ++++++++++++++++ lib/kunit/Makefile | 3 +- lib/kunit/attributes.c | 80 ++++++++++++++++++++++++++++++++++++++ lib/kunit/executor.c | 21 ++++++++-- lib/kunit/test.c | 17 ++++---- 6 files changed, 161 insertions(+), 12 deletions(-) create mode 100644 include/kunit/attributes.h create mode 100644 lib/kunit/attributes.c diff --git a/include/kunit/attributes.h b/include/kunit/attributes.h new file mode 100644 index 000000000000..9fcd184cce36 --- /dev/null +++ b/include/kunit/attributes.h @@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * KUnit API to save and access test attributes + * + * Copyright (C) 2023, Google LLC. + * Author: Rae Moar + */ + +#ifndef _KUNIT_ATTRIBUTES_H +#define _KUNIT_ATTRIBUTES_H + +/* + * Print all test attributes for a test case or suite. + * Output format for test cases: "# .: " + * Output format for test suites: "# : " + */ +void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level); + +#endif /* _KUNIT_ATTRIBUTES_H */ diff --git a/include/kunit/test.h b/include/kunit/test.h index 23120d50499e..1fc9155988e9 100644 --- a/include/kunit/test.h +++ b/include/kunit/test.h @@ -63,12 +63,16 @@ enum kunit_status { KUNIT_SKIPPED, }; +/* Holds attributes for each test case and suite */ +struct kunit_attributes {}; + /** * struct kunit_case - represents an individual test case. * * @run_case: the function representing the actual test case. * @name: the name of the test case. * @generate_params: the generator function for parameterized tests. + * @attr: the attributes associated with the test * * A test case is a function with the signature, * ``void (*)(struct kunit *)`` @@ -104,6 +108,7 @@ struct kunit_case { void (*run_case)(struct kunit *test); const char *name; const void* (*generate_params)(const void *prev, char *desc); + struct kunit_attributes attr; /* private: internal use only. */ enum kunit_status status; @@ -133,6 +138,18 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status) */ #define KUNIT_CASE(test_name) { .run_case = test_name, .name = #test_name } +/** + * KUNIT_CASE_ATTR - A helper for creating a &struct kunit_case + * with attributes + * + * @test_name: a reference to a test case function. + * @attributes: a reference to a struct kunit_attributes object containing + * test attributes + */ +#define KUNIT_CASE_ATTR(test_name, attributes) \ + { .run_case = test_name, .name = #test_name, \ + .attr = attributes } + /** * KUNIT_CASE_PARAM - A helper for creation a parameterized &struct kunit_case * @@ -154,6 +171,20 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status) { .run_case = test_name, .name = #test_name, \ .generate_params = gen_params } +/** + * KUNIT_CASE_PARAM_ATTR - A helper for creating a parameterized &struct + * kunit_case with attributes + * + * @test_name: a reference to a test case function. + * @gen_params: a reference to a parameter generator function. + * @attributes: a reference to a struct kunit_attributes object containing + * test attributes + */ +#define KUNIT_CASE_PARAM_ATTR(test_name, gen_params, attributes) \ + { .run_case = test_name, .name = #test_name, \ + .generate_params = gen_params, \ + .attr = attributes } + /** * struct kunit_suite - describes a related collection of &struct kunit_case * @@ -163,6 +194,7 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status) * @init: called before every test case. * @exit: called after every test case. * @test_cases: a null terminated array of test cases. + * @attr: the attributes associated with the test suite * * A kunit_suite is a collection of related &struct kunit_case s, such that * @init is called before every test case and @exit is called after every @@ -182,6 +214,7 @@ struct kunit_suite { int (*init)(struct kunit *test); void (*exit)(struct kunit *test); struct kunit_case *test_cases; + struct kunit_attributes attr; /* private: internal use only */ char status_comment[KUNIT_STATUS_COMMENT_SIZE]; diff --git a/lib/kunit/Makefile b/lib/kunit/Makefile index cb417f504996..46f75f23dfe4 100644 --- a/lib/kunit/Makefile +++ b/lib/kunit/Makefile @@ -6,7 +6,8 @@ kunit-objs += test.o \ string-stream.o \ assert.o \ try-catch.o \ - executor.o + executor.o \ + attributes.o ifeq ($(CONFIG_KUNIT_DEBUGFS),y) kunit-objs += debugfs.o diff --git a/lib/kunit/attributes.c b/lib/kunit/attributes.c new file mode 100644 index 000000000000..9bda5a5f4030 --- /dev/null +++ b/lib/kunit/attributes.c @@ -0,0 +1,80 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * KUnit API to save and access test attributes + * + * Copyright (C) 2023, Google LLC. + * Author: Rae Moar + */ + +#include +#include + +/* Options for printing attributes: + * PRINT_ALWAYS - attribute is printed for every test case and suite if set + * PRINT_SUITE - attribute is printed for every suite if set but not for test cases + * PRINT_NEVER - attribute is never printed + */ +enum print_ops { + PRINT_ALWAYS, + PRINT_SUITE, + PRINT_NEVER, +}; + +/** + * struct kunit_attr - represents a test attribute and holds flexible + * helper functions to interact with attribute. + * + * @name: name of test attribute, eg. speed + * @get_attr: function to return attribute value given a test + * @to_string: function to return string representation of given + * attribute value + * @filter: function to indicate whether a given attribute value passes a + * filter + */ +struct kunit_attr { + const char *name; + void *(*get_attr)(void *test_or_suite, bool is_test); + const char *(*to_string)(void *attr, bool *to_free); + int (*filter)(void *attr, const char *input, int *err); + void *attr_default; + enum print_ops print; +}; + +/* List of all Test Attributes */ + +static struct kunit_attr kunit_attr_list[] = {}; + +/* Helper Functions to Access Attributes */ + +void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level) +{ + int i; + bool to_free; + void *attr; + const char *attr_name, *attr_str; + struct kunit_suite *suite = is_test ? NULL : test_or_suite; + struct kunit_case *test = is_test ? test_or_suite : NULL; + + for (i = 0; i < ARRAY_SIZE(kunit_attr_list); i++) { + if (kunit_attr_list[i].print == PRINT_NEVER || + (test && kunit_attr_list[i].print == PRINT_SUITE)) + continue; + attr = kunit_attr_list[i].get_attr(test_or_suite, is_test); + if (attr) { + attr_name = kunit_attr_list[i].name; + attr_str = kunit_attr_list[i].to_string(attr, &to_free); + if (test) { + kunit_log(KERN_INFO, test, "%*s# %s.%s: %s", + KUNIT_INDENT_LEN * test_level, "", test->name, + attr_name, attr_str); + } else { + kunit_log(KERN_INFO, suite, "%*s# %s: %s", + KUNIT_INDENT_LEN * test_level, "", attr_name, attr_str); + } + + /* Free to_string of attribute if needed */ + if (to_free) + kfree(attr_str); + } + } +} diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c index 74982b83707c..12e38a48a5cc 100644 --- a/lib/kunit/executor.c +++ b/lib/kunit/executor.c @@ -2,6 +2,7 @@ #include #include +#include #include #include @@ -24,7 +25,8 @@ module_param_named(action, action_param, charp, 0); MODULE_PARM_DESC(action, "Changes KUnit executor behavior, valid values are:\n" ": run the tests like normal\n" - "'list' to list test names instead of running them.\n"); + "'list' to list test names instead of running them.\n" + "'list_attr' to list test names and attributes instead of running them.\n"); /* glob_match() needs NULL terminated strings, so we need a copy of filter_glob_param. */ struct kunit_test_filter { @@ -172,7 +174,7 @@ static void kunit_exec_run_tests(struct suite_set *suite_set) __kunit_test_suites_init(suite_set->start, num_suites); } -static void kunit_exec_list_tests(struct suite_set *suite_set) +static void kunit_exec_list_tests(struct suite_set *suite_set, bool include_attr) { struct kunit_suite * const *suites; struct kunit_case *test_case; @@ -180,10 +182,19 @@ static void kunit_exec_list_tests(struct suite_set *suite_set) /* Hack: print a ktap header so kunit.py can find the start of KUnit output. */ pr_info("KTAP version 1\n"); - for (suites = suite_set->start; suites < suite_set->end; suites++) + for (suites = suite_set->start; suites < suite_set->end; suites++) { + /* Print suite name and suite attributes */ + pr_info("%s\n", (*suites)->name); + if (include_attr) + kunit_print_attr((void *)(*suites), false, 0); + + /* Print test case name and attributes in suite */ kunit_suite_for_each_test_case((*suites), test_case) { pr_info("%s.%s\n", (*suites)->name, test_case->name); + if (include_attr) + kunit_print_attr((void *)test_case, true, 0); } + } } int kunit_run_all_tests(void) @@ -206,7 +217,9 @@ int kunit_run_all_tests(void) if (!action_param) kunit_exec_run_tests(&suite_set); else if (strcmp(action_param, "list") == 0) - kunit_exec_list_tests(&suite_set); + kunit_exec_list_tests(&suite_set, false); + else if (strcmp(action_param, "list_attr") == 0) + kunit_exec_list_tests(&suite_set, true); else pr_err("kunit executor: unknown action '%s'\n", action_param); diff --git a/lib/kunit/test.c b/lib/kunit/test.c index 84e4666555c9..9ee55139ecd1 100644 --- a/lib/kunit/test.c +++ b/lib/kunit/test.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -168,6 +169,13 @@ size_t kunit_suite_num_test_cases(struct kunit_suite *suite) } EXPORT_SYMBOL_GPL(kunit_suite_num_test_cases); +/* Currently supported test levels */ +enum { + KUNIT_LEVEL_SUITE = 0, + KUNIT_LEVEL_CASE, + KUNIT_LEVEL_CASE_PARAM, +}; + static void kunit_print_suite_start(struct kunit_suite *suite) { /* @@ -181,17 +189,11 @@ static void kunit_print_suite_start(struct kunit_suite *suite) pr_info(KUNIT_SUBTEST_INDENT "KTAP version 1\n"); pr_info(KUNIT_SUBTEST_INDENT "# Subtest: %s\n", suite->name); + kunit_print_attr((void *)suite, false, KUNIT_LEVEL_CASE); pr_info(KUNIT_SUBTEST_INDENT "1..%zd\n", kunit_suite_num_test_cases(suite)); } -/* Currently supported test levels */ -enum { - KUNIT_LEVEL_SUITE = 0, - KUNIT_LEVEL_CASE, - KUNIT_LEVEL_CASE_PARAM, -}; - static void kunit_print_ok_not_ok(struct kunit *test, unsigned int test_level, enum kunit_status status, @@ -651,6 +653,7 @@ int kunit_run_tests(struct kunit_suite *suite) } } + kunit_print_attr((void *)test_case, true, KUNIT_LEVEL_CASE); kunit_print_test_stats(&test, param_stats); From patchwork Tue Jul 25 21:25:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rae Moar X-Patchwork-Id: 707322 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD47EC00528 for ; Tue, 25 Jul 2023 21:25:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230415AbjGYVZr (ORCPT ); Tue, 25 Jul 2023 17:25:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56822 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230433AbjGYVZf (ORCPT ); Tue, 25 Jul 2023 17:25:35 -0400 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7B89A1FE6 for ; Tue, 25 Jul 2023 14:25:34 -0700 (PDT) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-564fb1018bcso72150587b3.0 for ; Tue, 25 Jul 2023 14:25:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690320333; x=1690925133; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=cQABQp9/fgq5do2PMDLxADuXEdETdAo+fTGFrKhT3xM=; b=jflwaAyedN/44AlTS/1LBQZ2iOctYb54voSmLUC7KqyJwqTDzemtXJi2RUu0W8uF0B Zn+c1hipTe94C17FmEdWXbpbkMZ0V1gPdMeobJE/K55o8vVB7nvNd/nsXkEBknQUwuRo NjP/M1egHlpX69ETuSa/AYPlS18wxOxne37oLti2YRj1roxxT8YsJQC57oCJ1hb5rBPG nGsMO3FE26jfPLXiCtnb4LCz2vWcYgkUysEGZNtRn1uQiGIacB4U1akaMInllMJeoCNA fl3uBVC4v/xCr6Myn3tv5jLaCk9sN5VJumemnf9MXeofRTPYb5ylW7USeqPG0jc9OC5e PeSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690320333; x=1690925133; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=cQABQp9/fgq5do2PMDLxADuXEdETdAo+fTGFrKhT3xM=; b=Jy90a26IjYIQfJZvuVxzaYTNiSMGwp1IL04ItS+171nHDrv4DtOS64PxRcLJCTQEVI L+T0vIychZGcaOV6Z8Z/MSXn6Trw5P/nwGPSCWOg2K3m6HdRkDb/H2iweOyRQ3KquyVh kCdCvaQ4k0Xt9XuSw962HYCiqfUcvyK0sWYkIwrXLz5dnCgoTSbHHKY+yTRPqHFDJCBh iyr+P2sOStZCzQtgwyT/t/pHOUVfd+E31ynVSTF/PTtfMDw0uDD1k+Tsslk+7vzgNP1a 7zSXE0DBnMC2C+/iDOhuUm/u0vU5ESe6/xA0fEsp6OjltP4Nawhh4JiEhn21FIFgXNbE JKAg== X-Gm-Message-State: ABy/qLatjgQjgj2gEPBQZSYvx1+t1o0mYxl4ETJNukdpLUxx4Zt26LmI Ivf+RyTcFj0Wans63F8sTjNJKfHJWg== X-Google-Smtp-Source: APBJJlHlirou98NBt4AkCpdrgz6nHsMm8KPUoNreSfSI1YkyEZ/9+IhgTHHhe1LAkpBK0VKXMvnLxQR1JA== X-Received: from rmoar-specialist.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:45d3]) (user=rmoar job=sendgmr) by 2002:a25:db84:0:b0:d0a:86fc:536c with SMTP id g126-20020a25db84000000b00d0a86fc536cmr1183ybf.2.1690320333783; Tue, 25 Jul 2023 14:25:33 -0700 (PDT) Date: Tue, 25 Jul 2023 21:25:14 +0000 In-Reply-To: <20230725212522.1622716-1-rmoar@google.com> Mime-Version: 1.0 References: <20230725212522.1622716-1-rmoar@google.com> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog Message-ID: <20230725212522.1622716-4-rmoar@google.com> Subject: [PATCH v3 3/9] kunit: Add module attribute From: Rae Moar To: shuah@kernel.org, davidgow@google.com, dlatypov@google.com, brendan.higgins@linux.dev Cc: linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, keescook@chromium.org, linux-hardening@vger.kernel.org, jstultz@google.com, tglx@linutronix.de, sboyd@kernel.org, Rae Moar Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Add module attribute to the test attribute API. This attribute stores the module name associated with the test using KBUILD_MODNAME. The name of a test suite and the module name often do not match. A reference to the module name associated with the suite could be extremely helpful in running tests as modules without needing to check the codebase. This attribute will be printed for each suite. Reviewed-by: David Gow Signed-off-by: Rae Moar --- Changes since v2: - No changes. Changes since v1: - Change kunit_attr_list definition to fix compile error Changes since RFC v2: - No changes. Changes: since RFC v1: - This is a new patch. include/kunit/test.h | 13 ++++++++----- lib/kunit/attributes.c | 25 +++++++++++++++++++++++++ 2 files changed, 33 insertions(+), 5 deletions(-) diff --git a/include/kunit/test.h b/include/kunit/test.h index ed5f5000a095..011e0d6bb506 100644 --- a/include/kunit/test.h +++ b/include/kunit/test.h @@ -131,6 +131,7 @@ struct kunit_case { /* private: internal use only. */ enum kunit_status status; + char *module_name; char *log; }; @@ -155,7 +156,9 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status) * &struct kunit_case object from it. See the documentation for * &struct kunit_case for an example on how to use it. */ -#define KUNIT_CASE(test_name) { .run_case = test_name, .name = #test_name } +#define KUNIT_CASE(test_name) \ + { .run_case = test_name, .name = #test_name, \ + .module_name = KBUILD_MODNAME} /** * KUNIT_CASE_ATTR - A helper for creating a &struct kunit_case @@ -167,7 +170,7 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status) */ #define KUNIT_CASE_ATTR(test_name, attributes) \ { .run_case = test_name, .name = #test_name, \ - .attr = attributes } + .attr = attributes, .module_name = KBUILD_MODNAME} /** * KUNIT_CASE_SLOW - A helper for creating a &struct kunit_case @@ -178,7 +181,7 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status) #define KUNIT_CASE_SLOW(test_name) \ { .run_case = test_name, .name = #test_name, \ - .attr.speed = KUNIT_SPEED_SLOW } + .attr.speed = KUNIT_SPEED_SLOW, .module_name = KBUILD_MODNAME} /** * KUNIT_CASE_PARAM - A helper for creation a parameterized &struct kunit_case @@ -199,7 +202,7 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status) */ #define KUNIT_CASE_PARAM(test_name, gen_params) \ { .run_case = test_name, .name = #test_name, \ - .generate_params = gen_params } + .generate_params = gen_params, .module_name = KBUILD_MODNAME} /** * KUNIT_CASE_PARAM_ATTR - A helper for creating a parameterized &struct @@ -213,7 +216,7 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status) #define KUNIT_CASE_PARAM_ATTR(test_name, gen_params, attributes) \ { .run_case = test_name, .name = #test_name, \ .generate_params = gen_params, \ - .attr = attributes } + .attr = attributes, .module_name = KBUILD_MODNAME} /** * struct kunit_suite - describes a related collection of &struct kunit_case diff --git a/lib/kunit/attributes.c b/lib/kunit/attributes.c index ffd0d692b334..9dce4f4d726c 100644 --- a/lib/kunit/attributes.c +++ b/lib/kunit/attributes.c @@ -61,6 +61,12 @@ static const char *attr_speed_to_string(void *attr, bool *to_free) return attr_enum_to_string(attr, speed_str_list, to_free); } +static const char *attr_string_to_string(void *attr, bool *to_free) +{ + *to_free = false; + return (char *) attr; +} + /* Get Attribute Methods */ static void *attr_speed_get(void *test_or_suite, bool is_test) @@ -74,6 +80,18 @@ static void *attr_speed_get(void *test_or_suite, bool is_test) return ((void *) suite->attr.speed); } +static void *attr_module_get(void *test_or_suite, bool is_test) +{ + struct kunit_suite *suite = is_test ? NULL : test_or_suite; + struct kunit_case *test = is_test ? test_or_suite : NULL; + + // Suites get their module attribute from their first test_case + if (test) + return ((void *) test->module_name); + else + return ((void *) suite->test_cases[0].module_name); +} + /* List of all Test Attributes */ static struct kunit_attr kunit_attr_list[] = { @@ -84,6 +102,13 @@ static struct kunit_attr kunit_attr_list[] = { .attr_default = (void *)KUNIT_SPEED_NORMAL, .print = PRINT_ALWAYS, }, + { + .name = "module", + .get_attr = attr_module_get, + .to_string = attr_string_to_string, + .attr_default = (void *)"", + .print = PRINT_SUITE, + } }; /* Helper Functions to Access Attributes */ From patchwork Tue Jul 25 21:25:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Rae Moar X-Patchwork-Id: 707321 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3A9EEB64DD for ; Tue, 25 Jul 2023 21:25:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231299AbjGYVZv (ORCPT ); Tue, 25 Jul 2023 17:25:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56792 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231183AbjGYVZk (ORCPT ); Tue, 25 Jul 2023 17:25:40 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 478A01FFA for ; Tue, 25 Jul 2023 14:25:36 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-c8f360a07a2so5523642276.2 for ; Tue, 25 Jul 2023 14:25:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690320335; x=1690925135; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=7QRX0uJ1Dawaqn3oTc7CEfaqw28ncjMa2otOk0jIQRs=; b=qQs9SujaitBZk7sy3aSEOU2gNA6h45AH8WWgH0Cpo7Q8m/XIug887gIYEOLjTRdzQf ndR+9fJ00Qxc/c9PChNlXt7cQH/dJfYbS91B3UHLHyazVFvxREyqPCmjpU2U8TbaCUQd BK/VXGSSXJADw93rJ11L4N4Qfqp0LYkhm22O4U42o8mBS8kEUlAjXlcL1FSbxQ/oFUOq chRd+JNZbKL0RDG2Hg/L00ok+HfhtXD+okCyEap0xiD7ConPC4smMaBh4TZVuvwc+x1m dFt2OptAIOUzkse7svFUvFLYbJf/hD6Vj/mpS4NLK6HBHZHF2IgcCSnEffWcmqFMYAL1 4AiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690320335; x=1690925135; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=7QRX0uJ1Dawaqn3oTc7CEfaqw28ncjMa2otOk0jIQRs=; b=kfPcoHt5C1vC3OOYLlvkQdtaBf8u8znHkkBkVFFjFS3WXhaHN1UySLsefijRsBCn7r 0aMmVilu1/v63L4jeOPrXzWSxHVIlPV1vS2BqfSaaB/U7l8wLbXgufVH7IHf5aCgshjP Ix3HuntKR674SrxMxYVITvjFXMZiLS1qgQ+taM8jGbRbCo8eEQN5/Zdt+bcx7ztO+YGG QZ9eEixDebc1ZGrzMksKYN32GFYIcAliXuNfA5IDXKDosOUecB3SXuKVx72WN8OdOOM7 2DW9qMuzWIWsgx3Dv1FUq8RUWQ/+KcdZw6c9i/mGzr2Dls7en0VEDsnYdHMQW8CsOHJY DZ1g== X-Gm-Message-State: ABy/qLZYN49U4O/9lwyOB8c3ZbmS6t6NRC3xjnijbmN/41epzBaEKHcR /H2K4gEbdQ2vaZFDSGgL1ld10yYgtA== X-Google-Smtp-Source: APBJJlGxgaInCvz5noCMntvxT84vYGMu04TD8UwuQnOU3ZzzW6XOcjhrnmStxKTHb+TtH/S1A6+s+ASdbA== X-Received: from rmoar-specialist.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:45d3]) (user=rmoar job=sendgmr) by 2002:a25:3251:0:b0:c61:7151:6727 with SMTP id y78-20020a253251000000b00c6171516727mr1118yby.10.1690320335593; Tue, 25 Jul 2023 14:25:35 -0700 (PDT) Date: Tue, 25 Jul 2023 21:25:15 +0000 In-Reply-To: <20230725212522.1622716-1-rmoar@google.com> Mime-Version: 1.0 References: <20230725212522.1622716-1-rmoar@google.com> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog Message-ID: <20230725212522.1622716-5-rmoar@google.com> Subject: [PATCH v3 4/9] kunit: Add ability to filter attributes From: Rae Moar To: shuah@kernel.org, davidgow@google.com, dlatypov@google.com, brendan.higgins@linux.dev Cc: linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, keescook@chromium.org, linux-hardening@vger.kernel.org, jstultz@google.com, tglx@linutronix.de, sboyd@kernel.org, Rae Moar Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Add filtering of test attributes. Users can filter tests using the module_param called "filter". Filters are imputed in the format: Example: kunit.filter="speed>slow" Operations include: >, <, >=, <=, !=, and =. These operations will act the same for attributes of the same type but may not between types. Note multiple filters can be inputted by separating them with a comma. Example: kunit.filter="speed=slow, module!=example" Since both suites and test cases can have attributes, there may be conflicts. The process of filtering follows these rules: - Filtering always operates at a per-test level. - If a test has an attribute set, then the test's value is filtered on. - Otherwise, the value falls back to the suite's value. - If neither are set, the attribute has a global "default" value, which is used. Filtered tests will not be run or show in output. The tests can instead be skipped using the configurable option "kunit.filter_action=skip". Note the default settings for running tests remains unfiltered. Finally, add "filter" methods for the speed and module attributes to parse and compare attribute values. Note this filtering functionality will be added to kunit.py in the next patch. Reviewed-by: David Gow Signed-off-by: Rae Moar --- Changes since v2: - Remove extra line. Changes since v1: - Fix compile warning of use of uninitialized variable Changes since RFC v2: - Change to output only one error before exiting. Changes since RFC v1: - Change method for inputting filters to allow for spaces in filtering values - Add option to skip filtered tests instead of not run or show them with the- –filter_skip flag include/kunit/attributes.h | 31 +++++ lib/kunit/attributes.c | 271 +++++++++++++++++++++++++++++++++++++ lib/kunit/executor.c | 93 ++++++++++--- lib/kunit/executor_test.c | 12 +- lib/kunit/test.c | 10 +- 5 files changed, 389 insertions(+), 28 deletions(-) diff --git a/include/kunit/attributes.h b/include/kunit/attributes.h index 9fcd184cce36..bc76a0b786d2 100644 --- a/include/kunit/attributes.h +++ b/include/kunit/attributes.h @@ -9,6 +9,20 @@ #ifndef _KUNIT_ATTRIBUTES_H #define _KUNIT_ATTRIBUTES_H +/* + * struct kunit_attr_filter - representation of attributes filter with the + * attribute object and string input + */ +struct kunit_attr_filter { + struct kunit_attr *attr; + char *input; +}; + +/* + * Returns the name of the filter's attribute. + */ +const char *kunit_attr_filter_name(struct kunit_attr_filter filter); + /* * Print all test attributes for a test case or suite. * Output format for test cases: "# .: " @@ -16,4 +30,21 @@ */ void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level); +/* + * Returns the number of fitlers in input. + */ +int kunit_get_filter_count(char *input); + +/* + * Parse attributes filter input and return an objects containing the + * attribute object and the string input of the next filter. + */ +struct kunit_attr_filter kunit_next_attr_filter(char **filters, int *err); + +/* + * Returns a copy of the suite containing only tests that pass the filter. + */ +struct kunit_suite *kunit_filter_attr_tests(const struct kunit_suite *const suite, + struct kunit_attr_filter filter, char *action, int *err); + #endif /* _KUNIT_ATTRIBUTES_H */ diff --git a/lib/kunit/attributes.c b/lib/kunit/attributes.c index 9dce4f4d726c..d37c40c0ce4f 100644 --- a/lib/kunit/attributes.c +++ b/lib/kunit/attributes.c @@ -67,6 +67,104 @@ static const char *attr_string_to_string(void *attr, bool *to_free) return (char *) attr; } +/* Filter Methods */ + +static const char op_list[] = "<>!="; + +/* + * Returns whether the inputted integer value matches the filter given + * by the operation string and inputted integer. + */ +static int int_filter(long val, const char *op, int input, int *err) +{ + if (!strncmp(op, "<=", 2)) + return (val <= input); + else if (!strncmp(op, ">=", 2)) + return (val >= input); + else if (!strncmp(op, "!=", 2)) + return (val != input); + else if (!strncmp(op, ">", 1)) + return (val > input); + else if (!strncmp(op, "<", 1)) + return (val < input); + else if (!strncmp(op, "=", 1)) + return (val == input); + *err = -EINVAL; + pr_err("kunit executor: invalid filter operation: %s\n", op); + return false; +} + +/* + * Returns whether the inputted enum value "attr" matches the filter given + * by the input string. Note: the str_list includes the corresponding string + * list to the enum values. + */ +static int attr_enum_filter(void *attr, const char *input, int *err, + const char * const str_list[], int max) +{ + int i, j, input_int; + long test_val = (long)attr; + const char *input_val = NULL; + + for (i = 0; input[i]; i++) { + if (!strchr(op_list, input[i])) { + input_val = input + i; + break; + } + } + + if (!input_val) { + *err = -EINVAL; + pr_err("kunit executor: filter value not found: %s\n", input); + return false; + } + + for (j = 0; j <= max; j++) { + if (!strcmp(input_val, str_list[j])) + input_int = j; + } + + if (!input_int) { + *err = -EINVAL; + pr_err("kunit executor: invalid filter input: %s\n", input); + return false; + } + + return int_filter(test_val, input, input_int, err); +} + +static int attr_speed_filter(void *attr, const char *input, int *err) +{ + return attr_enum_filter(attr, input, err, speed_str_list, KUNIT_SPEED_MAX); +} + +/* + * Returns whether the inputted string value (attr) matches the filter given + * by the input string. + */ +static int attr_string_filter(void *attr, const char *input, int *err) +{ + char *str = attr; + + if (!strncmp(input, "<", 1)) { + *err = -EINVAL; + pr_err("kunit executor: invalid filter input: %s\n", input); + return false; + } else if (!strncmp(input, ">", 1)) { + *err = -EINVAL; + pr_err("kunit executor: invalid filter input: %s\n", input); + return false; + } else if (!strncmp(input, "!=", 2)) { + return (strcmp(input + 2, str) != 0); + } else if (!strncmp(input, "=", 1)) { + return (strcmp(input + 1, str) == 0); + } + *err = -EINVAL; + pr_err("kunit executor: invalid filter operation: %s\n", input); + return false; +} + + /* Get Attribute Methods */ static void *attr_speed_get(void *test_or_suite, bool is_test) @@ -99,6 +197,7 @@ static struct kunit_attr kunit_attr_list[] = { .name = "speed", .get_attr = attr_speed_get, .to_string = attr_speed_to_string, + .filter = attr_speed_filter, .attr_default = (void *)KUNIT_SPEED_NORMAL, .print = PRINT_ALWAYS, }, @@ -106,6 +205,7 @@ static struct kunit_attr kunit_attr_list[] = { .name = "module", .get_attr = attr_module_get, .to_string = attr_string_to_string, + .filter = attr_string_filter, .attr_default = (void *)"", .print = PRINT_SUITE, } @@ -113,6 +213,11 @@ static struct kunit_attr kunit_attr_list[] = { /* Helper Functions to Access Attributes */ +const char *kunit_attr_filter_name(struct kunit_attr_filter filter) +{ + return filter.attr->name; +} + void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level) { int i; @@ -145,3 +250,169 @@ void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level } } } + +/* Helper Functions to Filter Attributes */ + +int kunit_get_filter_count(char *input) +{ + int i, comma_index, count = 0; + + for (i = 0; input[i]; i++) { + if (input[i] == ',') { + if ((i - comma_index) > 1) + count++; + comma_index = i; + } + } + if ((i - comma_index) > 0) + count++; + return count; +} + +struct kunit_attr_filter kunit_next_attr_filter(char **filters, int *err) +{ + struct kunit_attr_filter filter = {}; + int i, j, comma_index, new_start_index; + int op_index = -1, attr_index = -1; + char op; + char *input = *filters; + + /* Parse input until operation */ + for (i = 0; input[i]; i++) { + if (op_index < 0 && strchr(op_list, input[i])) { + op_index = i; + } else if (!comma_index && input[i] == ',') { + comma_index = i; + } else if (comma_index && input[i] != ' ') { + new_start_index = i; + break; + } + } + + if (op_index <= 0) { + *err = -EINVAL; + pr_err("kunit executor: filter operation not found: %s\n", input); + return filter; + } + + /* Temporarily set operator to \0 character. */ + op = input[op_index]; + input[op_index] = '\0'; + + /* Find associated kunit_attr object */ + for (j = 0; j < ARRAY_SIZE(kunit_attr_list); j++) { + if (!strcmp(input, kunit_attr_list[j].name)) { + attr_index = j; + break; + } + } + + input[op_index] = op; + + if (attr_index < 0) { + *err = -EINVAL; + pr_err("kunit executor: attribute not found: %s\n", input); + } else { + filter.attr = &kunit_attr_list[attr_index]; + } + + if (comma_index) { + input[comma_index] = '\0'; + filter.input = input + op_index; + input = input + new_start_index; + } else { + filter.input = input + op_index; + input = NULL; + } + + *filters = input; + + return filter; +} + +struct kunit_suite *kunit_filter_attr_tests(const struct kunit_suite *const suite, + struct kunit_attr_filter filter, char *action, int *err) +{ + int n = 0; + struct kunit_case *filtered, *test_case; + struct kunit_suite *copy; + void *suite_val, *test_val; + bool suite_result, test_result, default_result, result; + + /* Allocate memory for new copy of suite and list of test cases */ + copy = kmemdup(suite, sizeof(*copy), GFP_KERNEL); + if (!copy) + return ERR_PTR(-ENOMEM); + + kunit_suite_for_each_test_case(suite, test_case) { n++; } + + filtered = kcalloc(n + 1, sizeof(*filtered), GFP_KERNEL); + if (!filtered) { + kfree(copy); + return ERR_PTR(-ENOMEM); + } + + n = 0; + + /* Save filtering result on default value */ + default_result = filter.attr->filter(filter.attr->attr_default, filter.input, err); + if (*err) { + kfree(copy); + kfree(filtered); + return NULL; + } + + /* Save suite attribute value and filtering result on that value */ + suite_val = filter.attr->get_attr((void *)suite, false); + suite_result = filter.attr->filter(suite_val, filter.input, err); + if (*err) { + kfree(copy); + kfree(filtered); + return NULL; + } + + /* For each test case, save test case if passes filtering. */ + kunit_suite_for_each_test_case(suite, test_case) { + test_val = filter.attr->get_attr((void *) test_case, true); + test_result = filter.attr->filter(filter.attr->get_attr(test_case, true), + filter.input, err); + if (*err) { + kfree(copy); + kfree(filtered); + return NULL; + } + + /* + * If attribute value of test case is set, filter on that value. + * If not, filter on suite value if set. If not, filter on + * default value. + */ + result = false; + if (test_val) { + if (test_result) + result = true; + } else if (suite_val) { + if (suite_result) + result = true; + } else if (default_result) { + result = true; + } + + if (result) { + filtered[n++] = *test_case; + } else if (action && strcmp(action, "skip") == 0) { + test_case->status = KUNIT_SKIPPED; + filtered[n++] = *test_case; + } + } + + if (n == 0) { + kfree(copy); + kfree(filtered); + return NULL; + } + + copy->test_cases = filtered; + + return copy; +} diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c index 12e38a48a5cc..483f7b7873a7 100644 --- a/lib/kunit/executor.c +++ b/lib/kunit/executor.c @@ -17,6 +17,8 @@ extern struct kunit_suite * const __kunit_suites_end[]; static char *filter_glob_param; static char *action_param; +static char *filter_param; +static char *filter_action_param; module_param_named(filter_glob, filter_glob_param, charp, 0); MODULE_PARM_DESC(filter_glob, @@ -27,15 +29,23 @@ MODULE_PARM_DESC(action, ": run the tests like normal\n" "'list' to list test names instead of running them.\n" "'list_attr' to list test names and attributes instead of running them.\n"); +module_param_named(filter, filter_param, charp, 0); +MODULE_PARM_DESC(filter, + "Filter which KUnit test suites/tests run at boot-time using attributes, e.g. speed>slow"); +module_param_named(filter_action, filter_action_param, charp, 0); +MODULE_PARM_DESC(filter_action, + "Changes behavior of filtered tests using attributes, valid values are:\n" + ": do not run filtered tests as normal\n" + "'skip': skip all filtered tests instead so tests will appear in output\n"); /* glob_match() needs NULL terminated strings, so we need a copy of filter_glob_param. */ -struct kunit_test_filter { +struct kunit_glob_filter { char *suite_glob; char *test_glob; }; /* Split "suite_glob.test_glob" into two. Assumes filter_glob is not empty. */ -static void kunit_parse_filter_glob(struct kunit_test_filter *parsed, +static void kunit_parse_glob_filter(struct kunit_glob_filter *parsed, const char *filter_glob) { const int len = strlen(filter_glob); @@ -57,7 +67,7 @@ static void kunit_parse_filter_glob(struct kunit_test_filter *parsed, /* Create a copy of suite with only tests that match test_glob. */ static struct kunit_suite * -kunit_filter_tests(const struct kunit_suite *const suite, const char *test_glob) +kunit_filter_glob_tests(const struct kunit_suite *const suite, const char *test_glob) { int n = 0; struct kunit_case *filtered, *test_case; @@ -111,12 +121,15 @@ static void kunit_free_suite_set(struct suite_set suite_set) static struct suite_set kunit_filter_suites(const struct suite_set *suite_set, const char *filter_glob, + char *filters, + char *filter_action, int *err) { - int i; - struct kunit_suite **copy, *filtered_suite; + int i, j, k, filter_count; + struct kunit_suite **copy, *filtered_suite, *new_filtered_suite; struct suite_set filtered; - struct kunit_test_filter filter; + struct kunit_glob_filter parsed_glob; + struct kunit_attr_filter *parsed_filters; const size_t max = suite_set->end - suite_set->start; @@ -127,17 +140,52 @@ static struct suite_set kunit_filter_suites(const struct suite_set *suite_set, return filtered; } - kunit_parse_filter_glob(&filter, filter_glob); + if (filter_glob) + kunit_parse_glob_filter(&parsed_glob, filter_glob); - for (i = 0; &suite_set->start[i] != suite_set->end; i++) { - if (!glob_match(filter.suite_glob, suite_set->start[i]->name)) - continue; - - filtered_suite = kunit_filter_tests(suite_set->start[i], filter.test_glob); - if (IS_ERR(filtered_suite)) { - *err = PTR_ERR(filtered_suite); + /* Parse attribute filters */ + if (filters) { + filter_count = kunit_get_filter_count(filters); + parsed_filters = kcalloc(filter_count + 1, sizeof(*parsed_filters), GFP_KERNEL); + for (j = 0; j < filter_count; j++) + parsed_filters[j] = kunit_next_attr_filter(&filters, err); + if (*err) return filtered; + } + + for (i = 0; &suite_set->start[i] != suite_set->end; i++) { + filtered_suite = suite_set->start[i]; + if (filter_glob) { + if (!glob_match(parsed_glob.suite_glob, filtered_suite->name)) + continue; + filtered_suite = kunit_filter_glob_tests(filtered_suite, + parsed_glob.test_glob); + if (IS_ERR(filtered_suite)) { + *err = PTR_ERR(filtered_suite); + return filtered; + } } + if (filter_count) { + for (k = 0; k < filter_count; k++) { + new_filtered_suite = kunit_filter_attr_tests(filtered_suite, + parsed_filters[k], filter_action, err); + + /* Free previous copy of suite */ + if (k > 0 || filter_glob) + kfree(filtered_suite); + filtered_suite = new_filtered_suite; + + if (*err) + return filtered; + if (IS_ERR(filtered_suite)) { + *err = PTR_ERR(filtered_suite); + return filtered; + } + if (!filtered_suite) + break; + } + } + if (!filtered_suite) continue; @@ -145,8 +193,14 @@ static struct suite_set kunit_filter_suites(const struct suite_set *suite_set, } filtered.end = copy; - kfree(filter.suite_glob); - kfree(filter.test_glob); + if (filter_glob) { + kfree(parsed_glob.suite_glob); + kfree(parsed_glob.test_glob); + } + + if (filter_count) + kfree(parsed_filters); + return filtered; } @@ -206,8 +260,9 @@ int kunit_run_all_tests(void) goto out; } - if (filter_glob_param) { - suite_set = kunit_filter_suites(&suite_set, filter_glob_param, &err); + if (filter_glob_param || filter_param) { + suite_set = kunit_filter_suites(&suite_set, filter_glob_param, + filter_param, filter_action_param, &err); if (err) { pr_err("kunit executor: error filtering suites: %d\n", err); goto out; @@ -223,7 +278,7 @@ int kunit_run_all_tests(void) else pr_err("kunit executor: unknown action '%s'\n", action_param); - if (filter_glob_param) { /* a copy was made of each suite */ + if (filter_glob_param || filter_param) { /* a copy was made of each suite */ kunit_free_suite_set(suite_set); } diff --git a/lib/kunit/executor_test.c b/lib/kunit/executor_test.c index ce6749af374d..d7ab069324b5 100644 --- a/lib/kunit/executor_test.c +++ b/lib/kunit/executor_test.c @@ -24,15 +24,15 @@ static struct kunit_case dummy_test_cases[] = { static void parse_filter_test(struct kunit *test) { - struct kunit_test_filter filter = {NULL, NULL}; + struct kunit_glob_filter filter = {NULL, NULL}; - kunit_parse_filter_glob(&filter, "suite"); + kunit_parse_glob_filter(&filter, "suite"); KUNIT_EXPECT_STREQ(test, filter.suite_glob, "suite"); KUNIT_EXPECT_FALSE(test, filter.test_glob); kfree(filter.suite_glob); kfree(filter.test_glob); - kunit_parse_filter_glob(&filter, "suite.test"); + kunit_parse_glob_filter(&filter, "suite.test"); KUNIT_EXPECT_STREQ(test, filter.suite_glob, "suite"); KUNIT_EXPECT_STREQ(test, filter.test_glob, "test"); kfree(filter.suite_glob); @@ -50,7 +50,7 @@ static void filter_suites_test(struct kunit *test) subsuite[1] = alloc_fake_suite(test, "suite2", dummy_test_cases); /* Want: suite1, suite2, NULL -> suite2, NULL */ - got = kunit_filter_suites(&suite_set, "suite2", &err); + got = kunit_filter_suites(&suite_set, "suite2", NULL, NULL, &err); KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start); KUNIT_ASSERT_EQ(test, err, 0); kfree_at_end(test, got.start); @@ -74,7 +74,7 @@ static void filter_suites_test_glob_test(struct kunit *test) subsuite[1] = alloc_fake_suite(test, "suite2", dummy_test_cases); /* Want: suite1, suite2, NULL -> suite2 (just test1), NULL */ - got = kunit_filter_suites(&suite_set, "suite2.test2", &err); + got = kunit_filter_suites(&suite_set, "suite2.test2", NULL, NULL, &err); KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start); KUNIT_ASSERT_EQ(test, err, 0); kfree_at_end(test, got.start); @@ -100,7 +100,7 @@ static void filter_suites_to_empty_test(struct kunit *test) subsuite[0] = alloc_fake_suite(test, "suite1", dummy_test_cases); subsuite[1] = alloc_fake_suite(test, "suite2", dummy_test_cases); - got = kunit_filter_suites(&suite_set, "not_found", &err); + got = kunit_filter_suites(&suite_set, "not_found", NULL, NULL, &err); KUNIT_ASSERT_EQ(test, err, 0); kfree_at_end(test, got.start); /* just in case */ diff --git a/lib/kunit/test.c b/lib/kunit/test.c index 9ee55139ecd1..cb9797fa6303 100644 --- a/lib/kunit/test.c +++ b/lib/kunit/test.c @@ -613,18 +613,22 @@ int kunit_run_tests(struct kunit_suite *suite) kunit_suite_for_each_test_case(suite, test_case) { struct kunit test = { .param_value = NULL, .param_index = 0 }; struct kunit_result_stats param_stats = { 0 }; - test_case->status = KUNIT_SKIPPED; kunit_init_test(&test, test_case->name, test_case->log); - - if (!test_case->generate_params) { + if (test_case->status == KUNIT_SKIPPED) { + /* Test marked as skip */ + test.status = KUNIT_SKIPPED; + kunit_update_stats(¶m_stats, test.status); + } else if (!test_case->generate_params) { /* Non-parameterised test. */ + test_case->status = KUNIT_SKIPPED; kunit_run_case_catch_errors(suite, test_case, &test); kunit_update_stats(¶m_stats, test.status); } else { /* Get initial param. */ param_desc[0] = '\0'; test.param_value = test_case->generate_params(NULL, param_desc); + test_case->status = KUNIT_SKIPPED; kunit_log(KERN_INFO, &test, KUNIT_SUBTEST_INDENT KUNIT_SUBTEST_INDENT "KTAP version 1\n"); kunit_log(KERN_INFO, &test, KUNIT_SUBTEST_INDENT KUNIT_SUBTEST_INDENT From patchwork Tue Jul 25 21:25:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rae Moar X-Patchwork-Id: 707320 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE00FEB64DD for ; Tue, 25 Jul 2023 21:26:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231617AbjGYV0O (ORCPT ); Tue, 25 Jul 2023 17:26:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56890 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231422AbjGYVZr (ORCPT ); Tue, 25 Jul 2023 17:25:47 -0400 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DEB6A1FD2 for ; Tue, 25 Jul 2023 14:25:41 -0700 (PDT) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5840614b107so26443087b3.1 for ; Tue, 25 Jul 2023 14:25:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690320341; x=1690925141; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=iO/u2ma4F5rc1TFFHk9kuvKaOVRKNU+kB++I9CdxlGI=; b=xkKTkt65gZtUNFOilfKIRn9yKyfXwjSLm6qRtv/uZyV7CnWl1JKQ3ByQRR02Ft1aMJ vgpwGBYuba/cJSX+kxOgOUbfmB91n0vkIqvhatZhuXb9hH16mIQesZ8wQifx/8z5Sro7 9Qa9Yv9Q5iv2w03f2E+liANoKkAnG0Vi8RzOcUP5/37qwluMmtsIxXOQnbppprw8Ccq5 AZOlkAEqlrTClr0NSEIMY33eFJaVAwDrqEIOxS5ek7B/zTVPp44dbV5lawzYGyid459x g8wDO1As1SqiIlOCKHuYIy/B/mtAmxYMBh7UShG/0L5JKkdRpibZxFWYsFHEB3jfm/Pr RRqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690320341; x=1690925141; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=iO/u2ma4F5rc1TFFHk9kuvKaOVRKNU+kB++I9CdxlGI=; b=VIYAFD1/3GQtXxhx04lD7IdkG+FgplclFzrTldXqrb7XuTGscopReE9TGxyJ3It6gk 0d3OYvwVP2FUijMNYr7Q28ybO+c+gG2MgR9bkdk1DT37MsGZJdbfhlseYMOHPxaeTd9j BU00ykmhoUnTeVABk1bSL3AX4+dSLnqJC8iEnociKcO/W9nzHPcdlAVeT3FGvEtpmbJE yNTFR+CFA3uOy9Rxhp6wkBvj4dlCq03X764/CQ1rPYvMEiRZLnZBJUjat7vnj543ZkjX tPmMyz3Afj9W0WyFO9k1MFogEoIpfThRzG6IiF+X4g8bQzwFX4mrfbA/mKL3k2p9+taw AZSg== X-Gm-Message-State: ABy/qLbsR1jDES2rsXXwbiH7FJ2nrlvllq9HDtYwuFxl53cddZ0KwtTC g/a4QgTVm9LvYgxaBg9t6PGj6xHiKQ== X-Google-Smtp-Source: APBJJlF4feLrAMojCZfRgMcZi+J1UYJUvASL9ku7rbSLt7CTEIIOkB7UL3EDEDfBxKBDV1H+5hcAoLMEqQ== X-Received: from rmoar-specialist.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:45d3]) (user=rmoar job=sendgmr) by 2002:a81:b647:0:b0:576:b244:5a4e with SMTP id h7-20020a81b647000000b00576b2445a4emr3811ywk.10.1690320340980; Tue, 25 Jul 2023 14:25:40 -0700 (PDT) Date: Tue, 25 Jul 2023 21:25:18 +0000 In-Reply-To: <20230725212522.1622716-1-rmoar@google.com> Mime-Version: 1.0 References: <20230725212522.1622716-1-rmoar@google.com> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog Message-ID: <20230725212522.1622716-8-rmoar@google.com> Subject: [PATCH v3 7/9] kunit: time: Mark test as slow using test attributes From: Rae Moar To: shuah@kernel.org, davidgow@google.com, dlatypov@google.com, brendan.higgins@linux.dev Cc: linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, keescook@chromium.org, linux-hardening@vger.kernel.org, jstultz@google.com, tglx@linutronix.de, sboyd@kernel.org, Rae Moar Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Mark the time KUnit test, time64_to_tm_test_date_range, as slow using test attributes. This test ran relatively much slower than most other KUnit tests. By marking this test as slow, the test can now be filtered using the KUnit test attribute filtering feature. Example: --filter "speed>slow". This will run only the tests that have speeds faster than slow. The slow attribute will also be outputted in KTAP. Reviewed-by: David Gow Signed-off-by: Rae Moar --- Changes since v2: - No changes. Changes since v1: - No changes. Changes since RFC v2: - No changes. Changes since RFC v1: - No changes. kernel/time/time_test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/time/time_test.c b/kernel/time/time_test.c index 831e8e779ace..ca058c8af6ba 100644 --- a/kernel/time/time_test.c +++ b/kernel/time/time_test.c @@ -86,7 +86,7 @@ static void time64_to_tm_test_date_range(struct kunit *test) } static struct kunit_case time_test_cases[] = { - KUNIT_CASE(time64_to_tm_test_date_range), + KUNIT_CASE_SLOW(time64_to_tm_test_date_range), {} }; From patchwork Tue Jul 25 21:25:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rae Moar X-Patchwork-Id: 707319 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0077C001DE for ; Tue, 25 Jul 2023 21:26:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231655AbjGYV02 (ORCPT ); Tue, 25 Jul 2023 17:26:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56788 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231439AbjGYVZx (ORCPT ); Tue, 25 Jul 2023 17:25:53 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3C9B5269E for ; Tue, 25 Jul 2023 14:25:45 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-d1851c52f3dso1290230276.1 for ; Tue, 25 Jul 2023 14:25:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690320344; x=1690925144; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=T7K0SwNjZ7cDQq8m8sEtyq9n1FsAQbDDAlgc1iAoacs=; b=iJrFDpYz4pwDiFgDJo+9rsVRNzdUOrxcF1CqXyiEoiOKzdBrkEa5Gq/0vWr+2ylO1N esp5RTVx824mkg6IEdpt1TcZraHPjGrN9UyUwuOzp/pv5Z4DAj/2+pJPhRwUDPqsLfQ+ RJp1LEvUt7z0GPxhHLlaq0I1yHGCZ98bwRyR6Y0KlJLSR+hW5f5zJ8pHjx5rjn6812ER AIl2PdRWbba+31bCD5VvcI/bTtC+QQJscdZT14xFKv8LbZEcw5WXL+7QnsNu5GJLaREN rLwcPYufJ98JEQyxwW3o7Kxo42nDCdY3OHlBwNT4kbVjiH95rHSYu/1UdBx+WQBrvQPq K52w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690320344; x=1690925144; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=T7K0SwNjZ7cDQq8m8sEtyq9n1FsAQbDDAlgc1iAoacs=; b=YLTjqD32z8NEwvIWgbjKl8Kmyt4nAeBilV+4ARQVe9nC344IduRNd/0P6a7mSsiGNo HuwtWR+T6QYLxw3F2x2k+U8n/g8M2zuTfuvNSFrnLLuvMoEoHLU3nxkhIHymsTEKkSlJ ZmsBgAtqUuBiJBzyB2TuunTt6maz8Rit/tlTzT/myoihGtpLIgidhjDrQ5x8Xx4otLm6 5j1LxrGsuwiTsacHI3QvfGTy/ywx0YhvmOALkvYN/Lt88TjhmMT6tL8NxEe+atjDS7uP U9JEB8FiBa3ErkB9+yu0aTO8IrYxbE3Mc3qr+B0874EM8Y5nwqpvaES/fnL7OH8hdGrD Fm8g== X-Gm-Message-State: ABy/qLYICAvMxn4isYGggp/sRVKOs2leB/CEGc7tycSsVG1GHdrm4yB1 SfURv+7aVqsjGUzD8wg+lCeVHvKQgA== X-Google-Smtp-Source: APBJJlF7vOYdQoZqQUtjSKqvLzB5C1brzb7bzbe9ArLJuVqGxMMjibdGEneFOuoF5O434q4d6+UtTgRMew== X-Received: from rmoar-specialist.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:45d3]) (user=rmoar job=sendgmr) by 2002:a25:abd2:0:b0:d05:e080:63c6 with SMTP id v76-20020a25abd2000000b00d05e08063c6mr1313ybi.9.1690320344550; Tue, 25 Jul 2023 14:25:44 -0700 (PDT) Date: Tue, 25 Jul 2023 21:25:20 +0000 In-Reply-To: <20230725212522.1622716-1-rmoar@google.com> Mime-Version: 1.0 References: <20230725212522.1622716-1-rmoar@google.com> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog Message-ID: <20230725212522.1622716-10-rmoar@google.com> Subject: [PATCH v3 9/9] kunit: Add documentation of KUnit test attributes From: Rae Moar To: shuah@kernel.org, davidgow@google.com, dlatypov@google.com, brendan.higgins@linux.dev Cc: linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, keescook@chromium.org, linux-hardening@vger.kernel.org, jstultz@google.com, tglx@linutronix.de, sboyd@kernel.org, Rae Moar Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Add documentation on the use of test attributes under the section "Tips for Running KUnit Tests" in the KUnit docs. Documentation includes three sections on how to mark tests with attributes, how attributes are reported, and how the user can filter tests using test attributes. Add descriptions of new flags to list of command-line arguments. Reviewed-by: David Gow Signed-off-by: Rae Moar --- Changes since v2: - Add to list of command-line arguments in run_wrapper.rst. Changes since v1: - No changes. Changes since RFC v2: - Add comment on KUNIT_CASE_SLOW() to documentation. - Add comment on how to expose raw kernel output. - Remove an extra line at the end of file. Changes since RFC v1: - This is a new patch Documentation/dev-tools/kunit/run_wrapper.rst | 12 ++ .../dev-tools/kunit/running_tips.rst | 166 ++++++++++++++++++ 2 files changed, 178 insertions(+) diff --git a/Documentation/dev-tools/kunit/run_wrapper.rst b/Documentation/dev-tools/kunit/run_wrapper.rst index dafe8eb28d30..19ddf5e07013 100644 --- a/Documentation/dev-tools/kunit/run_wrapper.rst +++ b/Documentation/dev-tools/kunit/run_wrapper.rst @@ -321,3 +321,15 @@ command line arguments: - ``--json``: If set, stores the test results in a JSON format and prints to `stdout` or saves to a file if a filename is specified. + +- ``--filter``: Specifies filters on test attributes, for example, ``speed!=slow``. + Multiple filters can be used by wrapping input in quotes and separating filters + by commas. Example: ``--filter "speed>slow, module=example"``. + +- ``--filter_action``: If set to ``skip``, filtered tests will be shown as skipped + in the output rather than showing no output. + +- ``--list_tests``: If set, lists all tests that will be run. + +- ``--list_tests_attr``: If set, lists all tests that will be run and all of their + attributes. diff --git a/Documentation/dev-tools/kunit/running_tips.rst b/Documentation/dev-tools/kunit/running_tips.rst index 8e8c493f17d1..766f9cdea0fa 100644 --- a/Documentation/dev-tools/kunit/running_tips.rst +++ b/Documentation/dev-tools/kunit/running_tips.rst @@ -262,3 +262,169 @@ other code executed during boot, e.g. # Reset coverage counters before running the test. $ echo 0 > /sys/kernel/debug/gcov/reset $ modprobe kunit-example-test + + +Test Attributes and Filtering +============================= + +Test suites and cases can be marked with test attributes, such as speed of +test. These attributes will later be printed in test output and can be used to +filter test execution. + +Marking Test Attributes +----------------------- + +Tests are marked with an attribute by including a ``kunit_attributes`` object +in the test definition. + +Test cases can be marked using the ``KUNIT_CASE_ATTR(test_name, attributes)`` +macro to define the test case instead of ``KUNIT_CASE(test_name)``. + +.. code-block:: c + + static const struct kunit_attributes example_attr = { + .speed = KUNIT_VERY_SLOW, + }; + + static struct kunit_case example_test_cases[] = { + KUNIT_CASE_ATTR(example_test, example_attr), + }; + +.. note:: + To mark a test case as slow, you can also use ``KUNIT_CASE_SLOW(test_name)``. + This is a helpful macro as the slow attribute is the most commonly used. + +Test suites can be marked with an attribute by setting the "attr" field in the +suite definition. + +.. code-block:: c + + static const struct kunit_attributes example_attr = { + .speed = KUNIT_VERY_SLOW, + }; + + static struct kunit_suite example_test_suite = { + ..., + .attr = example_attr, + }; + +.. note:: + Not all attributes need to be set in a ``kunit_attributes`` object. Unset + attributes will remain uninitialized and act as though the attribute is set + to 0 or NULL. Thus, if an attribute is set to 0, it is treated as unset. + These unset attributes will not be reported and may act as a default value + for filtering purposes. + +Reporting Attributes +-------------------- + +When a user runs tests, attributes will be present in the raw kernel output (in +KTAP format). Note that attributes will be hidden by default in kunit.py output +for all passing tests but the raw kernel output can be accessed using the +``--raw_output`` flag. This is an example of how test attributes for test cases +will be formatted in kernel output: + +.. code-block:: none + + # example_test.speed: slow + ok 1 example_test + +This is an example of how test attributes for test suites will be formatted in +kernel output: + +.. code-block:: none + + KTAP version 2 + # Subtest: example_suite + # module: kunit_example_test + 1..3 + ... + ok 1 example_suite + +Additionally, users can output a full attribute report of tests with their +attributes, using the command line flag ``--list_tests_attr``: + +.. code-block:: bash + + kunit.py run "example" --list_tests_attr + +.. note:: + This report can be accessed when running KUnit manually by passing in the + module_param ``kunit.action=list_attr``. + +Filtering +--------- + +Users can filter tests using the ``--filter`` command line flag when running +tests. As an example: + +.. code-block:: bash + + kunit.py run --filter speed=slow + + +You can also use the following operations on filters: "<", ">", "<=", ">=", +"!=", and "=". Example: + +.. code-block:: bash + + kunit.py run --filter "speed>slow" + +This example will run all tests with speeds faster than slow. Note that the +characters < and > are often interpreted by the shell, so they may need to be +quoted or escaped, as above. + +Additionally, you can use multiple filters at once. Simply separate filters +using commas. Example: + +.. code-block:: bash + + kunit.py run --filter "speed>slow, module=kunit_example_test" + +.. note:: + You can use this filtering feature when running KUnit manually by passing + the filter as a module param: ``kunit.filter="speed>slow, speed<=normal"``. + +Filtered tests will not run or show up in the test output. You can use the +``--filter_action=skip`` flag to skip filtered tests instead. These tests will be +shown in the test output in the test but will not run. To use this feature when +running KUnit manually, use the module param ``kunit.filter_action=skip``. + +Rules of Filtering Procedure +---------------------------- + +Since both suites and test cases can have attributes, there may be conflicts +between attributes during filtering. The process of filtering follows these +rules: + +- Filtering always operates at a per-test level. + +- If a test has an attribute set, then the test's value is filtered on. + +- Otherwise, the value falls back to the suite's value. + +- If neither are set, the attribute has a global "default" value, which is used. + +List of Current Attributes +-------------------------- + +``speed`` + +This attribute indicates the speed of a test's execution (how slow or fast the +test is). + +This attribute is saved as an enum with the following categories: "normal", +"slow", or "very_slow". The assumed default speed for tests is "normal". This +indicates that the test takes a relatively trivial amount of time (less than +1 second), regardless of the machine it is running on. Any test slower than +this could be marked as "slow" or "very_slow". + +The macro ``KUNIT_CASE_SLOW(test_name)`` can be easily used to set the speed +of a test case to "slow". + +``module`` + +This attribute indicates the name of the module associated with the test. + +This attribute is automatically saved as a string and is printed for each suite. +Tests can also be filtered using this attribute.