From patchwork Tue Jul 31 10:39:10 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Tunnicliffe X-Patchwork-Id: 10385 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id 2994B24039 for ; Tue, 31 Jul 2012 10:39:14 +0000 (UTC) Received: from mail-gh0-f180.google.com (mail-gh0-f180.google.com [209.85.160.180]) by fiordland.canonical.com (Postfix) with ESMTP id D2A18A1885B for ; Tue, 31 Jul 2012 10:39:13 +0000 (UTC) Received: by ghbz12 with SMTP id z12so5716055ghb.11 for ; Tue, 31 Jul 2012 03:39:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-forwarded-to:x-forwarded-for:delivered-to:received-spf :content-type:mime-version:x-launchpad-project:x-launchpad-branch :x-launchpad-message-rationale:x-launchpad-branch-revision-number :x-launchpad-notification-type:to:from:subject:message-id:date :reply-to:sender:errors-to:precedence:x-generated-by :x-launchpad-hash:x-gm-message-state; bh=amBB5gaLWhzmf5nZ4rxyrnV919onn/Q2zamD/oBA6Hg=; b=nL4z/mpY1DoM0dYPf8/7nmon8PmeM9OImFJpCqDLYqFIOSRudUmsNL9vOKw62npkq9 A4FJhyY58onJxcv4InQD5sOUVQrKjdWuyCod0hKYr2VKgUyJVXsf9uXuRHLSih1+QIKg +n6wooBiC/KJdZ8X9HtLJN4lgJ9eT9JH50s6SofFv0i7tttjgjiqrcbEO4Uor1NUE62U RsmSLDShpGSAaUGCLvEhE/IkxsQHhlS5kp7DqEDonbwcPdN/i5gClzkAq6fi9vOwSLGt dWtH54ydtcEKFQ+CTuslwlPspOPxdmOocxnALdhdq56L7KcYzqEA5nvoYWTa+eys9Lj6 UoUg== Received: by 10.50.213.39 with SMTP id np7mr1512869igc.51.1343731152974; Tue, 31 Jul 2012 03:39:12 -0700 (PDT) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.50.87.40 with SMTP id u8csp135343igz; Tue, 31 Jul 2012 03:39:12 -0700 (PDT) Received: by 10.216.134.101 with SMTP id r79mr7151885wei.60.1343731151541; Tue, 31 Jul 2012 03:39:11 -0700 (PDT) Received: from indium.canonical.com (indium.canonical.com. [91.189.90.7]) by mx.google.com with ESMTPS id bu8si21906605wib.2.2012.07.31.03.39.11 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 31 Jul 2012 03:39:11 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) client-ip=91.189.90.7; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) smtp.mail=bounces@canonical.com Received: from ackee.canonical.com ([91.189.89.26]) by indium.canonical.com with esmtp (Exim 4.71 #1 (Debian)) id 1Sw9r4-0007Sr-Ut for ; Tue, 31 Jul 2012 10:39:10 +0000 Received: from ackee.canonical.com (localhost [127.0.0.1]) by ackee.canonical.com (Postfix) with ESMTP id D6A43E027C for ; Tue, 31 Jul 2012 10:39:10 +0000 (UTC) MIME-Version: 1.0 X-Launchpad-Project: linaro-image-tools X-Launchpad-Branch: ~linaro-image-tools/linaro-image-tools/trunk X-Launchpad-Message-Rationale: Subscriber X-Launchpad-Branch-Revision-Number: 544 X-Launchpad-Notification-Type: branch-revision To: Linaro Patch Tracker From: noreply@launchpad.net Subject: [Branch ~linaro-image-tools/linaro-image-tools/trunk] Rev 544: Add support for checking V3 metadata has fields in the right palce, and only the expected fields Message-Id: <20120731103910.26005.19806.launchpad@ackee.canonical.com> Date: Tue, 31 Jul 2012 10:39:10 -0000 Reply-To: noreply@launchpad.net Sender: bounces@canonical.com Errors-To: bounces@canonical.com Precedence: bulk X-Generated-By: Launchpad (canonical.com); Revision="15702"; Instance="launchpad-lazr.conf" X-Launchpad-Hash: f64dd0c51baa23d08b39cf2e9a336d20e3761d93 X-Gm-Message-State: ALoCoQk/1+EyYu6ToWCOfG63SdY5ax6OvwwUVydsfxRfyXS8fhxWnwsi4cs52hoKNtzMTiF5Cixt Merge authors: James Tunnicliffe (dooferlad) Related merge proposals: https://code.launchpad.net/~dooferlad/linaro-image-tools/yaml-checker/+merge/117298 proposed by: James Tunnicliffe (dooferlad) review: Approve - Milo Casagrande (milo) ------------------------------------------------------------ revno: 544 [merge] committer: James Tunnicliffe branch nick: linaro-image-tools timestamp: Tue 2012-07-31 11:38:05 +0100 message: Add support for checking V3 metadata has fields in the right palce, and only the expected fields modified: linaro_image_tools/hwpack/config.py linaro_image_tools/hwpack/hwpack_fields.py linaro_image_tools/hwpack/tests/test_config_v3.py --- lp:linaro-image-tools https://code.launchpad.net/~linaro-image-tools/linaro-image-tools/trunk You are subscribed to branch lp:linaro-image-tools. To unsubscribe from this branch go to https://code.launchpad.net/~linaro-image-tools/linaro-image-tools/trunk/+edit-subscription === modified file 'linaro_image_tools/hwpack/config.py' --- linaro_image_tools/hwpack/config.py 2012-07-26 15:15:51 +0000 +++ linaro_image_tools/hwpack/config.py 2012-07-31 10:36:49 +0000 @@ -80,6 +80,7 @@ WIRELESS_INTERFACES_FIELD, DEFINED_PARTITION_LAYOUTS, VERSION_FIELD, + hwpack_v3_layout, ) @@ -184,6 +185,7 @@ if isinstance(self.parser, ConfigParser.RawConfigParser): if not self.parser.has_section(self.MAIN_SECTION): raise HwpackConfigError("No [%s] section" % self.MAIN_SECTION) + self._validate_keys() self._validate_format() self._validate_name() self._validate_include_debs() @@ -1198,3 +1200,61 @@ if not found: raise HwpackConfigError( "No sections other than [%s]" % self.MAIN_SECTION) + + def _validate_keys(self): + """Check the dictionary created by the YAML parser for unknown keys""" + + if not self._is_v3: + # We don't check V1 or V2 configurations in this way + return + + self._validate_keys_layout = hwpack_v3_layout + self._do_validate_keys_prefix = [] + self._do_validate_keys(self._validate_keys_layout, self.parser) + + def _do_validate_keys_push_prefix(self, prefix): + self._do_validate_keys_prefix.append(prefix) + prefix = ": ".join(self._do_validate_keys_prefix)[2:] + if prefix: + prefix += ": " + return prefix + + def _do_validate_keys(self, expected, config, prefix=""): + prefix = self._do_validate_keys_push_prefix(prefix) + + if not isinstance(config, dict): + raise HwpackConfigError("Invalid structure in metadata. Expected " + "key: value pairs, found: '%s'" % + (prefix + str(config))) + + for key in config.keys(): + # If expected == {"*": {...}} then we can accept any key + if("*" in expected and expected.keys() == ["*"] and + isinstance(expected["*"], dict)): + # Have found a sub-dictionary to check. Recurse. + self._do_validate_keys(expected["*"], config[key], key) + continue + + # Check to see if the key is valid + if key not in expected: + raise HwpackConfigError("Unknown key in metadata: '%s'" % + (prefix + str(key))) + + # Have a valid key. If it should point to a dictionary, recurse + if expected[key]: + if isinstance(expected[key], dict): + # Have found a sub-dictionary to check. Recurse. + self._do_validate_keys(expected[key], config[key], key) + continue + + if expected[key] == "root": + config = config[key] + prefix = self._do_validate_keys_push_prefix(key) + + for key in config.keys(): + self._do_validate_keys(self._validate_keys_layout, + config[key], key) + + self._do_validate_keys_prefix.pop() + + self._do_validate_keys_prefix.pop() === modified file 'linaro_image_tools/hwpack/hwpack_fields.py' --- linaro_image_tools/hwpack/hwpack_fields.py 2012-07-23 14:10:35 +0000 +++ linaro_image_tools/hwpack/hwpack_fields.py 2012-07-31 10:36:49 +0000 @@ -89,3 +89,63 @@ 'bootfs16_rootfs', 'bootfs_rootfs', 'reserved_bootfs_rootfs', ] + +# Define where fields are valid, so we can test them. +# If a key has a value None, this indicates there is either a value or +# list of values that can be associated with it. +# If a key contains a dictionary, this means that the key can +# contain a dictionary. +# The string "root" indicates that the key can contain the root +# structure. This is used for the boards section, where each +# board can contain the full or partial layout, overwriting the global +# settings. +hwpack_v3_layout = { + FORMAT_FIELD: None, + NAME_FIELD: None, + ARCHITECTURES_FIELD: None, + ORIGIN_FIELD: None, + MAINTAINER_FIELD: None, + SUPPORT_FIELD: None, + ASSUME_INSTALLED_FIELD: None, + INCLUDE_DEBS_FIELD: None, + DTB_FILE_FIELD: None, + DTB_ADDR_FIELD: None, + SERIAL_TTY_FIELD: None, + EXTRA_SERIAL_OPTIONS_FIELD: None, + MMC_ID_FIELD: None, + PACKAGES_FIELD: None, + PARTITION_LAYOUT_FIELD: None, + KERNEL_FILE_FIELD: None, + KERNEL_ADDR_FIELD: None, + INITRD_FILE_FIELD: None, + INITRD_ADDR_FIELD: None, + LOAD_ADDR_FIELD: None, + BOOT_SCRIPT_FIELD: None, + LOADER_START_FIELD: None, + WIRED_INTERFACES_FIELD: None, + WIRELESS_INTERFACES_FIELD: None, + BOOT_MIN_SIZE_FIELD: None, + ROOT_MIN_SIZE_FIELD: None, + LOADER_MIN_SIZE_FIELD: None, + SAMSUNG_BL1_LEN_FIELD: None, + SAMSUNG_BL1_LEN_FIELD: None, + SAMSUNG_ENV_LEN_FIELD: None, + SAMSUNG_BL2_LEN_FIELD: None, + SNOWBALL_STARTUP_FILES_CONFIG_FIELD: None, + SOURCES_FIELD: None, + BOOTLOADERS_FIELD: { + "*": { + PACKAGE_FIELD: None, + FILE_FIELD: None, + IN_BOOT_PART_FIELD: None, + DD_FIELD: None, + EXTRA_BOOT_OPTIONS_FIELD: None, + SPL_PACKAGE_FIELD: None, + SPL_FILE_FIELD: None, + SPL_IN_BOOT_PART_FIELD: None, + SPL_DD_FIELD: None, + ENV_DD_FIELD: None, + } + }, + BOARDS_FIELD: "root", +} === modified file 'linaro_image_tools/hwpack/tests/test_config_v3.py' --- linaro_image_tools/hwpack/tests/test_config_v3.py 2012-07-26 14:46:31 +0000 +++ linaro_image_tools/hwpack/tests/test_config_v3.py 2012-07-30 16:56:12 +0000 @@ -694,7 +694,6 @@ def test_architectures(self): config = self.get_config( - "hello: there\n" "name: ahwpack\n" "packages: foo\n" "architectures:\n" @@ -736,3 +735,47 @@ " - bar\n" " - foo\n") self.assertEqual(["foo", "bar"], config.assume_installed) + + def test_invalid_key_in_root(self): + config = self.get_config("foo: bar") + self.assertValidationError("Unknown key in metadata: 'foo'", + config._validate_keys) + + def test_invalid_key_value_root(self): + config = self.get_config("bootloaders: bar") + self.assertValidationError("Invalid structure in metadata. Expected " + "key: value pairs, found: 'bootloaders: " + "bar'", + config._validate_keys) + + def test_invalid_key_value_bootloaders(self): + config = self.get_config("\n".join([ + "bootloaders:", + " u_boot:", + " foo: bar" + ])) + self.assertValidationError("Unknown key in metadata: 'bootloaders: " + "u_boot: foo'", + config._validate_keys) + + def test_invalid_key_in_board(self): + config = self.get_config("\n".join([ + "boards:", + " pandaboard:", + " foo: bar" + ])) + self.assertValidationError("Unknown key in metadata: " + "'boards: pandaboard: foo'", + config._validate_keys) + + def test_invalid_key_in_board_2(self): + config = self.get_config("\n".join([ + "boards:", + " pandaboard:", + " name: bar", + " snowball:", + " foo: bar", + ])) + self.assertValidationError("Unknown key in metadata: " + "'boards: snowball: foo'", + config._validate_keys)