From patchwork Tue Mar 13 09:12:03 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Masahiro Yamada X-Patchwork-Id: 131422 Delivered-To: patch@linaro.org Received: by 10.46.84.17 with SMTP id i17csp485631ljb; Tue, 13 Mar 2018 02:17:29 -0700 (PDT) X-Google-Smtp-Source: AG47ELuqlLiZTf1ETFfkGQZZnjp+Ch685MtRT9jAj0QvUlUGWFet0D2LycL/i4FCOh5GuUJg5enN X-Received: by 10.99.180.2 with SMTP id s2mr3551135pgf.26.1520932649064; Tue, 13 Mar 2018 02:17:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1520932649; cv=none; d=google.com; s=arc-20160816; b=WAAbHxnI3cyLbXwKYEl++fP/GPELdkl5yNYO8Q1yHA4+33x54YHUwgOV3mM6r8lVMK wV5e7riCYg0QDMhlMi75CQd1UcDGVhQjjH7TTUeTXeI6oqz2o2CxevDrGMusb5b4Yobk homSHKsdHh7SIkaQldjiN25WKclgDFgSLWpDkQmaSsAvX7GaeCHX28gkP+Tv6RnTMJFt bLHt8ivdycRu/JrWVrU12ioms0d4eKF3touFGi2urVHURBR4eo+dUSFcuuRUxCF7tx5d 3sJm0/slcMxAtnrk3nPYnxel0SNJfMEgMxMkEGpnak8q7BbPpvARbFrd0+TfJXC4qDft EVdQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:dkim-filter :arc-authentication-results; bh=B5HkizdL7kl7WCc/yuhv27TUnC72pluNFomvD03QqUY=; b=Ap+wj/L+WmRmyO+Tg+qsEwzmVgSU3B4Vs+ryNdWrUkQTKbS9NssSr98v7hdb2Hhmku gAFAes9tdEg01t9OH0FB14iQ+VHkj5//uPOr6IsP7jT1U+4zJ8b2tqQy6J4C41aFhKbK 01yN1c+ZRBCCAT/SCXb1VwQv7zbLSJ+YJnBlfkfYv2g1kA0lDz9uaMu3NQlGLWMsqa9y 8FkjtaoSXfjK6dRDYK4kWawsPiTPfxH82ioqpaJPp1w/5rOzP6zowQWA6eLNB67+dY3+ bJNBh3nwaz2X1IomkfNgLdObVPvqhFF2bIEIaotqE1vX+04KCm3gsW8xrmde8A4bCjRc UuGA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@nifty.com header.s=dec2015msa header.b=l6uMZlnH; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 31-v6si2886364plg.227.2018.03.13.02.17.28; Tue, 13 Mar 2018 02:17:29 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@nifty.com header.s=dec2015msa header.b=l6uMZlnH; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752605AbeCMJR0 (ORCPT + 28 others); Tue, 13 Mar 2018 05:17:26 -0400 Received: from conuserg-07.nifty.com ([210.131.2.74]:41654 "EHLO conuserg-07.nifty.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752218AbeCMJN4 (ORCPT ); Tue, 13 Mar 2018 05:13:56 -0400 Received: from pug.e01.socionext.com (p14092-ipngnfx01kyoto.kyoto.ocn.ne.jp [153.142.97.92]) (authenticated) by conuserg-07.nifty.com with ESMTP id w2D9CIo3016505; Tue, 13 Mar 2018 18:12:20 +0900 DKIM-Filter: OpenDKIM Filter v2.10.3 conuserg-07.nifty.com w2D9CIo3016505 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nifty.com; s=dec2015msa; t=1520932340; bh=B5HkizdL7kl7WCc/yuhv27TUnC72pluNFomvD03QqUY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=l6uMZlnHUwyVts7prmmE+F1kuEW1REDRLxf0nzqFxt10b6ZcoLNXFED94tyhClO8F se8bG6BIRar9MPPlayrKOPKZKNEMFrmL/Zgw7nYEMSDFPtLHAXOlrtgJ23UyieRYB0 SIaePvej+/yw9yaCmCXq8x39U5hWR0f0YXaw94bKjRfYRNli4mwyw/P5G6wG0nkKRh ph8FGk6+vGn/OuTAoxlaC1UURED+3m0vBR6pfqqSUCWNac9De6q/wDtt7Eo2X7jI/P +OX+SkRAYD4GRFhqMERiofdEaVS6ZYvEAn7SPXErutYvWlFYuNkUJ4Wvea1FRYjKM4 m4jsNxOiBP0jw== X-Nifty-SrcIP: [153.142.97.92] From: Masahiro Yamada To: linux-kbuild@vger.kernel.org Cc: Sam Ravnborg , Ulf Magnusson , Randy Dunlap , "Luis R . Rodriguez" , Tony Luck , Masahiro Yamada , linux-kernel@vger.kernel.org Subject: [PATCH v3 02/11] kconfig: tests: add framework for Kconfig unit testing Date: Tue, 13 Mar 2018 18:12:03 +0900 Message-Id: <1520932332-2449-3-git-send-email-yamada.masahiro@socionext.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1520932332-2449-1-git-send-email-yamada.masahiro@socionext.com> References: <1520932332-2449-1-git-send-email-yamada.masahiro@socionext.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Many parts in Kconfig are so cryptic and need refactoring. However, its complexity prevents us from moving forward. There are several naive corner cases where it is difficult to notice breakage. If those are covered by unit tests, we will be able to touch the code with more confidence. Here is a simple test framework based on pytest. The conftest.py provides a fixture useful to run commands such as 'oldaskconfig' etc. and to compare the resulted .config, stdout, stderr with expectations. How to add test cases? ---------------------- For each test case, you should create a subdirectory under scripts/kconfig/tests/ (so test cases are separated from each other). Every test case directory should contain the following files: - __init__.py: describes test functions - Kconfig: the top level Kconfig file for the test To do a useful job, test cases generally need additional data like input .config and information about expected results. How to run tests? ----------------- You need python3 and pytest. Then, run "make testconfig". O= option is supported. If V=1 is given, detailed logs captured during tests are displayed. Signed-off-by: Masahiro Yamada Reviewed-by: Ulf Magnusson --- Changes in v3: None Changes in v2: - Add 'randconfig' support - Add docstring - Fix various style errors detected by 'pep8' and 'pep257' scripts/kconfig/Makefile | 8 ++ scripts/kconfig/tests/conftest.py | 291 ++++++++++++++++++++++++++++++++++++++ scripts/kconfig/tests/pytest.ini | 7 + 3 files changed, 306 insertions(+) create mode 100644 scripts/kconfig/tests/conftest.py create mode 100644 scripts/kconfig/tests/pytest.ini -- 2.7.4 diff --git a/scripts/kconfig/Makefile b/scripts/kconfig/Makefile index cb3ec53..c5d1d1a 100644 --- a/scripts/kconfig/Makefile +++ b/scripts/kconfig/Makefile @@ -135,6 +135,14 @@ PHONY += tinyconfig tinyconfig: $(Q)$(MAKE) -f $(srctree)/Makefile allnoconfig tiny.config +# CHECK: -o cache_dir= working? +PHONY += testconfig +testconfig: $(obj)/conf + $(PYTHON3) -B -m pytest $(srctree)/$(src)/tests \ + -o cache_dir=$(abspath $(obj)/tests/.cache) \ + $(if $(findstring 1,$(KBUILD_VERBOSE)),--capture=no) +clean-dirs += tests/.cache + # Help text used by make help help: @echo ' config - Update current config utilising a line-oriented program' diff --git a/scripts/kconfig/tests/conftest.py b/scripts/kconfig/tests/conftest.py new file mode 100644 index 0000000..0345ef6 --- /dev/null +++ b/scripts/kconfig/tests/conftest.py @@ -0,0 +1,291 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright (C) 2018 Masahiro Yamada +# + +""" +Kconfig unit testing framework. + +This provides fixture functions commonly used from test files. +""" + +import os +import pytest +import shutil +import subprocess +import tempfile + +CONF_PATH = os.path.abspath(os.path.join('scripts', 'kconfig', 'conf')) + + +class Conf: + """Kconfig runner and result checker. + + This class provides methods to run text-based interface of Kconfig + (scripts/kconfig/conf) and retrieve the resulted configuration, + stdout, and stderr. It also provides methods to compare those + results with expectations. + """ + + def __init__(self, request): + """Create a new Conf instance. + + request: object to introspect the requesting test module + """ + # the directory of the test being run + self._test_dir = os.path.dirname(str(request.fspath)) + + # runners + def _run_conf(self, mode, dot_config=None, out_file='.config', + interactive=False, in_keys=None, extra_env={}): + """Run text-based Kconfig executable and save the result. + + mode: input mode option (--oldaskconfig, --defconfig= etc.) + dot_config: .config file to use for configuration base + out_file: file name to contain the output config data + interactive: flag to specify the interactive mode + in_keys: key inputs for interactive modes + extra_env: additional environments + returncode: exit status of the Kconfig executable + """ + command = [CONF_PATH, mode, 'Kconfig'] + + # Override 'srctree' environment to make the test as the top directory + extra_env['srctree'] = self._test_dir + + # Run Kconfig in a temporary directory. + # This directory is automatically removed when done. + with tempfile.TemporaryDirectory() as temp_dir: + + # if .config is given, copy it to the working directory + if dot_config: + shutil.copyfile(os.path.join(self._test_dir, dot_config), + os.path.join(temp_dir, '.config')) + + ps = subprocess.Popen(command, + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + cwd=temp_dir, + env=dict(os.environ, **extra_env)) + + # If input key sequence is given, feed it to stdin. + if in_keys: + ps.stdin.write(in_keys.encode('utf-8')) + + while ps.poll() is None: + # For interactive modes such as oldaskconfig, oldconfig, + # send 'Enter' key until the program finishes. + if interactive: + ps.stdin.write(b'\n') + + self.retcode = ps.returncode + self.stdout = ps.stdout.read().decode() + self.stderr = ps.stderr.read().decode() + + # Retrieve the resulted config data only when .config is supposed + # to exist. If the command fails, the .config does not exist. + # 'listnewconfig' does not produce .config in the first place. + if self.retcode == 0 and out_file: + with open(os.path.join(temp_dir, out_file)) as f: + self.config = f.read() + else: + self.config = None + + # Logging: + # Pytest captures the following information by default. In failure + # of tests, the captured log will be displayed. This will be useful to + # figure out what has happened. + + print("[command]\n{}\n".format(' '.join(command))) + + print("[retcode]\n{}\n".format(self.retcode)) + + print("[stdout]") + print(self.stdout) + + print("[stderr]") + print(self.stderr) + + if self.config is not None: + print("[output for '{}']".format(out_file)) + print(self.config) + + return self.retcode + + def oldaskconfig(self, dot_config=None, in_keys=None): + """Run oldaskconfig. + + dot_config: .config file to use for configuration base (optional) + in_key: key inputs (optional) + returncode: exit status of the Kconfig executable + """ + return self._run_conf('--oldaskconfig', dot_config=dot_config, + interactive=True, in_keys=in_keys) + + def oldconfig(self, dot_config=None, in_keys=None): + """Run oldconfig. + + dot_config: .config file to use for configuration base (optional) + in_key: key inputs (optional) + returncode: exit status of the Kconfig executable + """ + return self._run_conf('--oldconfig', dot_config=dot_config, + interactive=True, in_keys=in_keys) + + def olddefconfig(self, dot_config=None): + """Run olddefconfig. + + dot_config: .config file to use for configuration base (optional) + returncode: exit status of the Kconfig executable + """ + return self._run_conf('--olddefconfig', dot_config=dot_config) + + def defconfig(self, defconfig): + """Run defconfig. + + defconfig: defconfig file for input + returncode: exit status of the Kconfig executable + """ + defconfig_path = os.path.join(self._test_dir, defconfig) + return self._run_conf('--defconfig={}'.format(defconfig_path)) + + def _allconfig(self, mode, all_config): + if all_config: + all_config_path = os.path.join(self._test_dir, all_config) + extra_env = {'KCONFIG_ALLCONFIG': all_config_path} + else: + extra_env = {} + + return self._run_conf('--{}config'.format(mode), extra_env=extra_env) + + def allyesconfig(self, all_config=None): + """Run allyesconfig. + + all_config: fragment config file for KCONFIG_ALLCONFIG (optional) + returncode: exit status of the Kconfig executable + """ + return self._allconfig('allyes', all_config) + + def allmodconfig(self, all_config=None): + """Run allmodconfig. + + all_config: fragment config file for KCONFIG_ALLCONFIG (optional) + returncode: exit status of the Kconfig executable + """ + return self._allconfig('allmod', all_config) + + def allnoconfig(self, all_config=None): + """Run allnoconfig. + + all_config: fragment config file for KCONFIG_ALLCONFIG (optional) + returncode: exit status of the Kconfig executable + """ + return self._allconfig('allno', all_config) + + def alldefconfig(self, all_config=None): + """Run alldefconfig. + + all_config: fragment config file for KCONFIG_ALLCONFIG (optional) + returncode: exit status of the Kconfig executable + """ + return self._allconfig('alldef', all_config) + + def randconfig(self, all_config=None): + """Run randconfig. + + all_config: fragment config file for KCONFIG_ALLCONFIG (optional) + returncode: exit status of the Kconfig executable + """ + return self._allconfig('rand', all_config) + + def savedefconfig(self, dot_config): + """Run savedefconfig. + + dot_config: .config file for input + returncode: exit status of the Kconfig executable + """ + return self._run_conf('--savedefconfig', out_file='defconfig') + + def listnewconfig(self, dot_config=None): + """Run listnewconfig. + + dot_config: .config file to use for configuration base (optional) + returncode: exit status of the Kconfig executable + """ + return self._run_conf('--listnewconfig', dot_config=dot_config, + out_file=None) + + # checkers + def _read_and_compare(self, compare, expected): + """Compare the result with expectation. + + compare: function to compare the result with expectation + expected: file that contains the expected data + """ + with open(os.path.join(self._test_dir, expected)) as f: + expected_data = f.read() + return compare(self, expected_data) + + def _contains(self, attr, expected): + return self._read_and_compare( + lambda s, e: getattr(s, attr).find(e) >= 0, + expected) + + def _matches(self, attr, expected): + return self._read_and_compare(lambda s, e: getattr(s, attr) == e, + expected) + + def config_contains(self, expected): + """Check if resulted configuration contains expected data. + + expected: file that contains the expected data + returncode: True if result contains the expected data, False otherwise + """ + return self._contains('config', expected) + + def config_matches(self, expected): + """Check if resulted configuration exactly matches expected data. + + expected: file that contains the expected data + returncode: True if result matches the expected data, False otherwise + """ + return self._matches('config', expected) + + def stdout_contains(self, expected): + """Check if resulted stdout contains expected data. + + expected: file that contains the expected data + returncode: True if result contains the expected data, False otherwise + """ + return self._contains('stdout', expected) + + def stdout_matches(self, expected): + """Check if resulted stdout exactly matches expected data. + + expected: file that contains the expected data + returncode: True if result matches the expected data, False otherwise + """ + return self._matches('stdout', expected) + + def stderr_contains(self, expected): + """Check if resulted stderr contains expected data. + + expected: file that contains the expected data + returncode: True if result contains the expected data, False otherwise + """ + return self._contains('stderr', expected) + + def stderr_matches(self, expected): + """Check if resulted stderr exactly matches expected data. + + expected: file that contains the expected data + returncode: True if result matches the expected data, False otherwise + """ + return self._matches('stderr', expected) + + +@pytest.fixture(scope="module") +def conf(request): + """Create a Conf instance and provide it to test functions.""" + return Conf(request) diff --git a/scripts/kconfig/tests/pytest.ini b/scripts/kconfig/tests/pytest.ini new file mode 100644 index 0000000..85d7ce8 --- /dev/null +++ b/scripts/kconfig/tests/pytest.ini @@ -0,0 +1,7 @@ +[pytest] +addopts = --verbose + +# Pytest requires that test files have unique names, because pytest imports +# them as top-level modules. It is silly to prefix or suffix a test file with +# the directory name that contains it. Use __init__.py for all test files. +python_files = __init__.py