From patchwork Tue Feb 6 00:34:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Masahiro Yamada X-Patchwork-Id: 126934 Delivered-To: patch@linaro.org Received: by 10.46.124.24 with SMTP id x24csp2441528ljc; Mon, 5 Feb 2018 16:39:20 -0800 (PST) X-Google-Smtp-Source: AH8x225BkFUA8E9210OIvNUIbFw1m97oyGy1MMRzTEKk4WWo79zptAjj0HtunGTyWRMi89lEaGVu X-Received: by 10.98.31.131 with SMTP id l3mr602926pfj.116.1517877560330; Mon, 05 Feb 2018 16:39:20 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1517877560; cv=none; d=google.com; s=arc-20160816; b=XaHj3i30qR19aDyyZ0I+BTshst8AAhsCrNAmGtKMt9XPv8EReTNpk0FcpXj49pUZsj mZ4qqOiYtLtdaombXBU9isAq8nY0qfi1eWB2Ayam2OavCcowDIyiHEH2p6q9Zc5qWOpj sGbmpjPylmEqMZg4l4fbr8Gj2ROW/NudlhdgSa66MUChq/bpRAtDo1vZ7XT3B6sGtXgZ j5Q8nbnLyVcgJV3aYEMFrlAQHUlFcK3eoR4jVdvQyyT3PKaY2sfFhHcd7aqEh322WWB+ E/oKO5kxaCbwUA1JBquqRQYA95Shn+tGnTSlk1Wy0MBVnmw5iUGPnZAbaM2MjGaKEE96 TDuw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:message-id:date:subject:cc:to:from :dkim-signature:dkim-filter:arc-authentication-results; bh=5AGHZtUo/Sjif9bJgV7IUvBHuRjMd+Y/nEux/J0xoFk=; b=SSiGC42t9QqvU5+c9B8wD5diLl6o4ySNFcYt2JcH3rArPWJ5XuttKiTOi0RuhUxnOc KWD9xv0voXBfn+rSIZwOXT6/LqaVVYEwCTCmz15kmeIClY/oKlqwyS618fjoezNL9a1x O55VQ3BBsEhGpzqd6hiq0PpeGyVwXKwMNdG8yYL8A+2K6+SV5fi/BRSve9udQsj2FO/W /LT3e3P3R6dTBZhuK4mPauLLlyrSLCPSYFcIkqX2o2Qtkdp+NP+yxg6jlz6mDN2WyFYt WR7cwlD+XIKliOl4lDBb/HTzc1vhJRNKjeVZLSCXngPYbecSKku2qLq3DpGDKRWVrGfi 3sCw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@nifty.com header.s=dec2015msa header.b=T3xDrVt/; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id s2si5108957pfj.303.2018.02.05.16.39.20; Mon, 05 Feb 2018 16:39:20 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@nifty.com header.s=dec2015msa header.b=T3xDrVt/; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752608AbeBFAjS (ORCPT + 21 others); Mon, 5 Feb 2018 19:39:18 -0500 Received: from conuserg-07.nifty.com ([210.131.2.74]:41230 "EHLO conuserg-07.nifty.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752523AbeBFAij (ORCPT ); Mon, 5 Feb 2018 19:38:39 -0500 Received: from grover.sesame (FL1-125-199-20-195.osk.mesh.ad.jp [125.199.20.195]) (authenticated) by conuserg-07.nifty.com with ESMTP id w160ZHAf011351; Tue, 6 Feb 2018 09:35:17 +0900 DKIM-Filter: OpenDKIM Filter v2.10.3 conuserg-07.nifty.com w160ZHAf011351 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nifty.com; s=dec2015msa; t=1517877318; bh=5AGHZtUo/Sjif9bJgV7IUvBHuRjMd+Y/nEux/J0xoFk=; h=From:To:Cc:Subject:Date:From; b=T3xDrVt/XQmHaHqQQamJzIowpU4S0ZxadqPROSAY1UBoqUsAjgxfm7KQ5nUnUFOZL HqjoQ2IM4eGyyYdZLMm5xRo3i/JefkffN80j+nPUgSKrgTKqVY2n2WcvsDrXcId2CV TTRt4wYBQKjhhOaWra/BiWsvt5TxGI7hua7cuDND8FeDJB8pJcYqJG5DTcnReUusa1 7g/iHwdBdD/kHBavTaDEGTP0Eg+in3vNeUUmDimp6qaLZcFhNzaLb4aVxgBFhNAN4Z k3UUiT/cIiwSyWOvDMYygL611vFHyaAhzWxn5dvq48Fhbjd7fLwH9bwhgHumOs2sNb FQota7K1LlGzA== X-Nifty-SrcIP: [125.199.20.195] From: Masahiro Yamada To: linux-kbuild@vger.kernel.org Cc: Greg Kroah-Hartman , Andrew Morton , Nicolas Pitre , "Luis R . Rodriguez" , Randy Dunlap , Ulf Magnusson , Sam Ravnborg , Michal Marek , Linus Torvalds , Masahiro Yamada , "Luis R. Rodriguez" , Tony Luck , Borislav Petkov , linux-kernel@vger.kernel.org, Thomas Gleixner , Heinrich Schuchardt , Yaakov Selkowitz , linux-ia64@vger.kernel.org, Fenghua Yu , Marc Herbert Subject: [PATCH 00/14] Add Kconfig unit tests Date: Tue, 6 Feb 2018 09:34:40 +0900 Message-Id: <1517877294-4826-1-git-send-email-yamada.masahiro@socionext.com> X-Mailer: git-send-email 2.7.4 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org I am applying various patches to Kconfig these days. However, I fear regressions. I have been thinking of unit-tests. There are various cryptic parts in Kconfig and corner cases where it is difficult to notice breakage. If unit-tests cover those, I will be able to apply changes more confidently. So, here is the trial. After fixing some problems, I will add a basic test framework. This is based on pytest. Also, this is written in Python 3. Python 2 will return in 2020. So, I believe new python tools should be written in Python 3. This is my Python 3 and pytest versions. $ python3 --version Python 3.5.2 $ python3 -m pytest --version This is pytest version 3.4.0, imported from /home/masahiro/.local/lib/python3.5/site-packages/pytest.py If I use old pytest version, some parts did not work as expected. If this does not work for you, please consider using newer pytest. I will brush up the code more and add more test cases to do a better job. Before proceeding more, I'd like to get consensus for this approach. If you have an idea for better implementation, comments are appreciated. How to use? ----------- Please make sure Python3 and pytest for Python3 is installed on your system. Then, run "make testconfig" The result looks like as follows: masahiro@grover:~/workspace/linux-yamada$ make testconfig HOSTCC scripts/kconfig/conf.o HOSTCC scripts/kconfig/zconf.tab.o HOSTLD scripts/kconfig/conf python3 -B -m pytest ./scripts/kconfig/tests \ -o cache_dir=/home/masahiro/workspace/linux-yamada/scripts/kconfig/tests/.cache \ ==================================================== test session starts ===================================================== platform linux -- Python 3.5.2, pytest-3.3.2, py-1.5.2, pluggy-0.6.0 -- /usr/bin/python3 cachedir: scripts/kconfig/tests/.cache rootdir: /home/masahiro/workspace/linux-yamada/scripts/kconfig/tests, inifile: pytest.ini collected 12 items scripts/kconfig/tests/auto_submenu_creation/__init__.py::test PASSED [ 8%] scripts/kconfig/tests/choice/__init__.py::test_oldask0 PASSED [ 16%] scripts/kconfig/tests/choice/__init__.py::test_oldask1 PASSED [ 25%] scripts/kconfig/tests/choice/__init__.py::test_allyes PASSED [ 33%] scripts/kconfig/tests/choice/__init__.py::test_allmod PASSED [ 41%] scripts/kconfig/tests/choice/__init__.py::test_allno PASSED [ 50%] scripts/kconfig/tests/choice/__init__.py::test_alldef PASSED [ 58%] scripts/kconfig/tests/choice_value_with_m_dep/__init__.py::test PASSED [ 66%] scripts/kconfig/tests/err_recursive_inc/__init__.py::test PASSED [ 75%] scripts/kconfig/tests/new_choice_with_dep/__init__.py::test PASSED [ 83%] scripts/kconfig/tests/no_write_if_dep_unmet/__init__.py::test PASSED [ 91%] scripts/kconfig/tests/warn_recursive_dep/__init__.py::test PASSED [100%] ================================================= 12 passed in 0.08 seconds ================================================== Masahiro Yamada (14): kconfig: send error messages to stderr kconfig: do not write choice values when their dependency becomes n kconfig: show '?' prompt even if no help text is available kconfig: print additional new line for choice for redirection kconfig: remove 'config*' pattern from .gitignnore kbuild: define PYTHON2 and PYTHON3 variables instead of PYTHON kconfig: test: add framework for Kconfig unit-tests kconfig: test: add basic 'choice' tests kconfig: test: test automatic submenu creation kconfig: test: check if new symbols in choice are asked kconfig: test: check .config sanity for choice values with unmet dep kconfig: test: check visibility of tristate choice values in y choice kconfig: test: check if recursive dependencies are detected kconfig: test: check if recursive inclusion is detected Makefile | 5 +- arch/ia64/Makefile | 2 +- scripts/kconfig/.gitignore | 1 - scripts/kconfig/Makefile | 8 + scripts/kconfig/conf.c | 29 +-- scripts/kconfig/symbol.c | 4 +- .../kconfig/tests/auto_submenu_creation/Kconfig | 50 ++++ .../tests/auto_submenu_creation/__init__.py | 12 + .../tests/auto_submenu_creation/expected_stdout | 10 + scripts/kconfig/tests/choice/Kconfig | 54 +++++ scripts/kconfig/tests/choice/__init__.py | 37 +++ .../kconfig/tests/choice/alldef_expected_config | 5 + .../kconfig/tests/choice/allmod_expected_config | 9 + scripts/kconfig/tests/choice/allno_expected_config | 5 + .../kconfig/tests/choice/allyes_expected_config | 9 + .../kconfig/tests/choice/oldask0_expected_stdout | 10 + scripts/kconfig/tests/choice/oldask1_config | 2 + .../kconfig/tests/choice/oldask1_expected_stdout | 15 ++ .../kconfig/tests/choice_value_with_m_dep/Kconfig | 20 ++ .../tests/choice_value_with_m_dep/__init__.py | 15 ++ .../kconfig/tests/choice_value_with_m_dep/config | 2 + .../tests/choice_value_with_m_dep/expected_config | 3 + .../tests/choice_value_with_m_dep/expected_stdout | 4 + scripts/kconfig/tests/conftest.py | 255 +++++++++++++++++++++ scripts/kconfig/tests/err_recursive_inc/Kconfig | 1 + .../kconfig/tests/err_recursive_inc/Kconfig.inc | 1 + .../kconfig/tests/err_recursive_inc/__init__.py | 10 + .../tests/err_recursive_inc/expected_stderr | 4 + scripts/kconfig/tests/new_choice_with_dep/Kconfig | 37 +++ .../kconfig/tests/new_choice_with_dep/__init__.py | 14 ++ scripts/kconfig/tests/new_choice_with_dep/config | 3 + .../tests/new_choice_with_dep/expected_stdout | 10 + .../kconfig/tests/no_write_if_dep_unmet/Kconfig | 14 ++ .../tests/no_write_if_dep_unmet/__init__.py | 17 ++ scripts/kconfig/tests/no_write_if_dep_unmet/config | 1 + .../tests/no_write_if_dep_unmet/expected_config | 5 + scripts/kconfig/tests/pytest.ini | 6 + scripts/kconfig/tests/warn_recursive_dep/Kconfig | 62 +++++ .../kconfig/tests/warn_recursive_dep/__init__.py | 10 + .../tests/warn_recursive_dep/expected_stderr | 33 +++ scripts/kconfig/zconf.l | 27 ++- 41 files changed, 789 insertions(+), 32 deletions(-) create mode 100644 scripts/kconfig/tests/auto_submenu_creation/Kconfig create mode 100644 scripts/kconfig/tests/auto_submenu_creation/__init__.py create mode 100644 scripts/kconfig/tests/auto_submenu_creation/expected_stdout create mode 100644 scripts/kconfig/tests/choice/Kconfig create mode 100644 scripts/kconfig/tests/choice/__init__.py create mode 100644 scripts/kconfig/tests/choice/alldef_expected_config create mode 100644 scripts/kconfig/tests/choice/allmod_expected_config create mode 100644 scripts/kconfig/tests/choice/allno_expected_config create mode 100644 scripts/kconfig/tests/choice/allyes_expected_config create mode 100644 scripts/kconfig/tests/choice/oldask0_expected_stdout create mode 100644 scripts/kconfig/tests/choice/oldask1_config create mode 100644 scripts/kconfig/tests/choice/oldask1_expected_stdout create mode 100644 scripts/kconfig/tests/choice_value_with_m_dep/Kconfig create mode 100644 scripts/kconfig/tests/choice_value_with_m_dep/__init__.py create mode 100644 scripts/kconfig/tests/choice_value_with_m_dep/config create mode 100644 scripts/kconfig/tests/choice_value_with_m_dep/expected_config create mode 100644 scripts/kconfig/tests/choice_value_with_m_dep/expected_stdout create mode 100644 scripts/kconfig/tests/conftest.py create mode 100644 scripts/kconfig/tests/err_recursive_inc/Kconfig create mode 100644 scripts/kconfig/tests/err_recursive_inc/Kconfig.inc create mode 100644 scripts/kconfig/tests/err_recursive_inc/__init__.py create mode 100644 scripts/kconfig/tests/err_recursive_inc/expected_stderr create mode 100644 scripts/kconfig/tests/new_choice_with_dep/Kconfig create mode 100644 scripts/kconfig/tests/new_choice_with_dep/__init__.py create mode 100644 scripts/kconfig/tests/new_choice_with_dep/config create mode 100644 scripts/kconfig/tests/new_choice_with_dep/expected_stdout create mode 100644 scripts/kconfig/tests/no_write_if_dep_unmet/Kconfig create mode 100644 scripts/kconfig/tests/no_write_if_dep_unmet/__init__.py create mode 100644 scripts/kconfig/tests/no_write_if_dep_unmet/config create mode 100644 scripts/kconfig/tests/no_write_if_dep_unmet/expected_config create mode 100644 scripts/kconfig/tests/pytest.ini create mode 100644 scripts/kconfig/tests/warn_recursive_dep/Kconfig create mode 100644 scripts/kconfig/tests/warn_recursive_dep/__init__.py create mode 100644 scripts/kconfig/tests/warn_recursive_dep/expected_stderr -- 2.7.4