From patchwork Thu May 19 06:51:55 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Masahiro Yamada X-Patchwork-Id: 68097 Delivered-To: patch@linaro.org Received: by 10.140.92.199 with SMTP id b65csp3033098qge; Wed, 18 May 2016 23:54:05 -0700 (PDT) X-Received: by 10.194.121.8 with SMTP id lg8mr11664159wjb.16.1463640845884; Wed, 18 May 2016 23:54:05 -0700 (PDT) Return-Path: Received: from theia.denx.de (theia.denx.de. [85.214.87.163]) by mx.google.com with ESMTP id m189si16360688wme.113.2016.05.18.23.54.05; Wed, 18 May 2016 23:54:05 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of u-boot-bounces@lists.denx.de designates 85.214.87.163 as permitted sender) client-ip=85.214.87.163; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@nifty.com; spf=pass (google.com: best guess record for domain of u-boot-bounces@lists.denx.de designates 85.214.87.163 as permitted sender) smtp.mailfrom=u-boot-bounces@lists.denx.de Received: from localhost (localhost [127.0.0.1]) by theia.denx.de (Postfix) with ESMTP id A7CE6A7690; Thu, 19 May 2016 08:53:21 +0200 (CEST) Received: from theia.denx.de ([127.0.0.1]) by localhost (theia.denx.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id F0RNtGdu5LER; Thu, 19 May 2016 08:53:21 +0200 (CEST) Received: from theia.denx.de (localhost [127.0.0.1]) by theia.denx.de (Postfix) with ESMTP id C46F0A76AD; Thu, 19 May 2016 08:52:30 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by theia.denx.de (Postfix) with ESMTP id CF3B7A7517 for ; Thu, 19 May 2016 08:51:31 +0200 (CEST) Received: from theia.denx.de ([127.0.0.1]) by localhost (theia.denx.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WOMs0bpkQgKw for ; Thu, 19 May 2016 08:51:31 +0200 (CEST) X-policyd-weight: NOT_IN_SBL_XBL_SPAMHAUS=-1.5 NOT_IN_SPAMCOP=-1.5 NOT_IN_BL_NJABL=-1.5 (only DNSBL check requested) Received: from conuserg-12.nifty.com (conuserg-12.nifty.com [210.131.2.79]) by theia.denx.de (Postfix) with ESMTPS id 213E6A74E3 for ; Thu, 19 May 2016 08:51:26 +0200 (CEST) Received: from beagle.diag.org (p14092-ipngnfx01kyoto.kyoto.ocn.ne.jp [153.142.97.92]) (authenticated) by conuserg-12.nifty.com with ESMTP id u4J6p2Zk001067; Thu, 19 May 2016 15:51:07 +0900 DKIM-Filter: OpenDKIM Filter v2.10.3 conuserg-12.nifty.com u4J6p2Zk001067 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nifty.com; s=dec2015msa; t=1463640668; bh=Kd9nLewo23XZSnMDg/+agt9GhEbk8o7oVBgREIL6wQg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=aTCcWIQRhgk7H3SP2oTnND6tP1axAAlfUSG3x5kbq24ztxvxQFYiV/YtYN079NvDb eH4XYrso4jGKHgFgvPDgKaO9zDIjkC/yDvUKU8dYtrwaHK2lDuKWB443HrwjOJy4ll qoc+OqfGLBa5psLalWaBrFKB28Y8oWFt605fky15OC1atawbbK3hK8K/AUgyyN5ArI 7p1dBPciqOwMZ6vzsLWyS0YZ7FAnJRMa89GpPhfg5HVfWGeVKvR28rzeQma43MS+GA keuKi9v/OK51AkWJ691zLonLYRZXQGuj1WSM2jr3r9RRQzS+4neY+/t3IydQWkwzhQ 8VJLNd7810/Fg== X-Nifty-SrcIP: [153.142.97.92] From: Masahiro Yamada To: u-boot@lists.denx.de Date: Thu, 19 May 2016 15:51:55 +0900 Message-Id: <1463640729-25666-8-git-send-email-yamada.masahiro@socionext.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1463640729-25666-1-git-send-email-yamada.masahiro@socionext.com> References: <1463640729-25666-1-git-send-email-yamada.masahiro@socionext.com> Cc: Joe Hershberger Subject: [U-Boot] [PATCH 07/21] tools: moveconfig: increment number of processed files monotonically X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.15 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" Currently, the progress " * defconfigs out of 1133" does not increase monotonically. Moreover, the number of processed defconfigs does not match the total number of defconfigs when this tool finishes, like: 1132 defconfigs out of 1133 Clean up headers? [y/n]: It looks like the task was not completed, and some users might feel upset about it. Signed-off-by: Masahiro Yamada --- tools/moveconfig.py | 65 ++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 49 insertions(+), 16 deletions(-) -- 1.9.1 _______________________________________________ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 3da4726..5e78266 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -388,6 +388,29 @@ def cleanup_headers(config_attrs, dry_run): patterns, dry_run) ### classes ### +class Progress: + + """Progress Indicator""" + + def __init__(self, total): + """Create a new progress indicator. + + Arguments: + total: A number of defconfig files to process. + """ + self.current = 0 + self.total = total + + def inc(self): + """Increment the number of processed defconfig files.""" + + self.current += 1 + + def show(self): + """Display the progress.""" + print ' %d defconfigs out of %d\r' % (self.current, self.total), + sys.stdout.flush() + class KconfigParser: """A parser of .config and include/autoconf.mk.""" @@ -395,17 +418,19 @@ class KconfigParser: re_arch = re.compile(r'CONFIG_SYS_ARCH="(.*)"') re_cpu = re.compile(r'CONFIG_SYS_CPU="(.*)"') - def __init__(self, config_attrs, options, build_dir): + def __init__(self, config_attrs, options, progress, build_dir): """Create a new parser. Arguments: config_attrs: A list of dictionaris, each of them includes the name, the type, and the default value of the target config. options: option flags. + progress: A progress indicator build_dir: Build directory. """ self.config_attrs = config_attrs self.options = options + self.progress = progress self.build_dir = build_dir def get_cross_compile(self): @@ -541,6 +566,7 @@ class KconfigParser: # Some threads are running in parallel. # Print log in one shot to not mix up logs from different threads. print log, + self.progress.show() with open(dotconfig_path, 'a') as f: for (action, value) in results: @@ -559,21 +585,24 @@ class Slot: for faster processing. """ - def __init__(self, config_attrs, options, devnull, make_cmd): + def __init__(self, config_attrs, options, progress, devnull, make_cmd): """Create a new process slot. Arguments: config_attrs: A list of dictionaris, each of them includes the name, the type, and the default value of the target config. options: option flags. + progress: A progress indicator. devnull: A file object of '/dev/null'. make_cmd: command name of GNU Make. """ self.options = options + self.progress = progress self.build_dir = tempfile.mkdtemp() self.devnull = devnull self.make_cmd = (make_cmd, 'O=' + self.build_dir) - self.parser = KconfigParser(config_attrs, options, self.build_dir) + self.parser = KconfigParser(config_attrs, options, progress, + self.build_dir) self.state = STATE_IDLE self.failed_boards = [] @@ -592,7 +621,7 @@ class Slot: pass shutil.rmtree(self.build_dir) - def add(self, defconfig, num, total): + def add(self, defconfig): """Assign a new subprocess for defconfig and add it to the slot. If the slot is vacant, create a new subprocess for processing the @@ -613,8 +642,6 @@ class Slot: stderr=subprocess.PIPE) self.defconfig = defconfig self.state = STATE_DEFCONFIG - self.num = num - self.total = total return True def poll(self): @@ -643,6 +670,8 @@ class Slot: print >> sys.stderr, color_text(self.options.color, COLOR_LIGHT_CYAN, self.ps.stderr.read()) + self.progress.inc() + self.progress.show() if self.options.exit_on_error: sys.exit("Exit on error.") # If --exit-on-error flag is not set, skip this board and continue. @@ -654,9 +683,6 @@ class Slot: if self.state == STATE_AUTOCONF: self.parser.update_dotconfig(self.defconfig) - print ' %d defconfigs out of %d\r' % (self.num + 1, self.total), - sys.stdout.flush() - """Save off the defconfig in a consistent way""" cmd = list(self.make_cmd) cmd.append('savedefconfig') @@ -669,6 +695,7 @@ class Slot: if not self.options.dry_run: shutil.move(os.path.join(self.build_dir, 'defconfig'), os.path.join('configs', self.defconfig)) + self.progress.inc() self.state = STATE_IDLE return True @@ -677,6 +704,8 @@ class Slot: print >> sys.stderr, log_msg(self.options.color, COLOR_YELLOW, self.defconfig, "Compiler is missing. Do nothing."), + self.progress.inc() + self.progress.show() if self.options.exit_on_error: sys.exit("Exit on error.") # If --exit-on-error flag is not set, skip this board and continue. @@ -704,22 +733,24 @@ class Slots: """Controller of the array of subprocess slots.""" - def __init__(self, config_attrs, options): + def __init__(self, config_attrs, options, progress): """Create a new slots controller. Arguments: config_attrs: A list of dictionaris containing the name, the type, and the default value of the target CONFIG. options: option flags. + progress: A progress indicator. """ self.options = options self.slots = [] devnull = get_devnull() make_cmd = get_make_cmd() for i in range(options.jobs): - self.slots.append(Slot(config_attrs, options, devnull, make_cmd)) + self.slots.append(Slot(config_attrs, options, progress, devnull, + make_cmd)) - def add(self, defconfig, num, total): + def add(self, defconfig): """Add a new subprocess if a vacant slot is found. Arguments: @@ -729,7 +760,7 @@ class Slots: Return True on success or False on failure """ for slot in self.slots: - if slot.add(defconfig, num, total): + if slot.add(defconfig): return True return False @@ -808,13 +839,14 @@ def move_config(config_attrs, options): for filename in fnmatch.filter(filenames, '*_defconfig'): defconfigs.append(os.path.join(dirpath, filename)) - slots = Slots(config_attrs, options) + progress = Progress(len(defconfigs)) + slots = Slots(config_attrs, options, progress) # Main loop to process defconfig files: # Add a new subprocess into a vacant slot. # Sleep if there is no available slot. - for i, defconfig in enumerate(defconfigs): - while not slots.add(defconfig, i, len(defconfigs)): + for defconfig in defconfigs: + while not slots.add(defconfig): while not slots.available(): # No available slot: sleep for a while time.sleep(SLEEP_TIME) @@ -823,6 +855,7 @@ def move_config(config_attrs, options): while not slots.empty(): time.sleep(SLEEP_TIME) + progress.show() print '' slots.show_failed_boards()