From patchwork Fri Aug 28 10:40:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ahmed Karaman X-Patchwork-Id: 275302 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.6 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 722C2C433E2 for ; Fri, 28 Aug 2020 10:42:46 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2E685208CA for ; Fri, 28 Aug 2020 10:42:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ULG82S0L" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2E685208CA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:56658 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kBbqH-0004wk-DI for qemu-devel@archiver.kernel.org; Fri, 28 Aug 2020 06:42:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34384) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kBbpB-0003MZ-Ex for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:37 -0400 Received: from mail-wr1-x442.google.com ([2a00:1450:4864:20::442]:42827) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kBbp8-0005fA-Ql for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:37 -0400 Received: by mail-wr1-x442.google.com with SMTP id c18so833999wrm.9 for ; Fri, 28 Aug 2020 03:41:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=GGrLy3IbvYhI9UUc6EFtGpu2pNno9f5HSPArLmtNmFM=; b=ULG82S0L3kCib5VT4GaouqmZivM+bay/mzr4qks3za+D7dtBqR++RqUjn5NKpkuVLE /3WjJcnTTdwSFiaGo8qwVLUJahcQ7YdGddB3xqtaldRO0TWnk8RyT7Cau2kciF5uc7V7 Nk/GzfRznwhn6IjcEL5I8C3h1O59R/HBQrVnu7SUJ3+UrmTDzW1TF7XsuaY1bR65VcNx sZ3AFsVXCOcMZP1jtQhLb8qc/1cQPTORJ/AxlxH/uybNPNQO95i10kyWpJ8XPe87hLJA AQLF6MfAcC6rwAgym1SY74AMz2zNBol60n2pAN+xDpkD9mAz2NmUuitE7eVfTahJz+cX 4BPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=GGrLy3IbvYhI9UUc6EFtGpu2pNno9f5HSPArLmtNmFM=; b=edhvkGgaW9FNDljG3BIyWtLg3vd0nXEB+/VdohZOpFWeWNs3yQU+HeWPI6aFOXHqip w9iA0fpMObgbfEg/q9ms4kU1Td1b/+f5x4Ujjxb4tbeSLEfHVuBXTCrmeu+dzRBVRdOd KTaTcdGN+h854DLMezFJJJE+lqYJyY7gep7kYzCj5BzIqTS9cqSIIVPBBcFB1hC+3U4d 7RX1jMrvlxe7E5Qh1aZIcQ2ntgdnxUDhsbZhfIezgHAVJt9rbhqI/58mbqcy9rwhmJLr FXKvpSlFTuI4+ereHs7Wv3rBgAizzUm+s/azfzQAc6n7LkmlymlVZCZkfCZ4KHaN585s GQig== X-Gm-Message-State: AOAM532cer+BEXV1Y4xxukMb6m+oXF80a4vYTa7iq27DCeVULpPw8SUI B7Snuz/ED3TY/vhWtxrk80YWy3tKpaP0UA== X-Google-Smtp-Source: ABdhPJy+UxZw0jQnURExU+I0JXJh37Lpqu5CD/VUPQVnG0uk7ICLiSSbKv/D/w+rB/FxGK0ERvMoDA== X-Received: by 2002:adf:a1d6:: with SMTP id v22mr954931wrv.185.1598611292937; Fri, 28 Aug 2020 03:41:32 -0700 (PDT) Received: from localhost.localdomain ([197.58.77.158]) by smtp.gmail.com with ESMTPSA id e18sm1307453wrx.50.2020.08.28.03.41.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Aug 2020 03:41:32 -0700 (PDT) From: Ahmed Karaman To: qemu-devel@nongnu.org, aleksandar.qemu.devel@gmail.com, philmd@redhat.com, alex.bennee@linaro.org, eblake@redhat.com, ldoktor@redhat.com, jsnow@redhat.com, rth@twiddle.net, ehabkost@redhat.com, crosa@redhat.com Subject: [PATCH 1/9] scripts/performance: Refactor topN_perf.py Date: Fri, 28 Aug 2020 12:40:54 +0200 Message-Id: <20200828104102.4490-2-ahmedkhaledkaraman@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> References: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> Received-SPF: pass client-ip=2a00:1450:4864:20::442; envelope-from=ahmedkhaledkaraman@gmail.com; helo=mail-wr1-x442.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_PDS_OTHER_BAD_TLD=0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ahmed Karaman Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" - Apply pylint and flake8 formatting rules to the script. - Use 'tempfile' instead of '/tmp' for creating temporary files. Signed-off-by: Ahmed Karaman Reviewed-by: Aleksandar Markovic Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
--- scripts/performance/topN_perf.py | 174 +++++++++++++++---------------- 1 file changed, 87 insertions(+), 87 deletions(-) diff --git a/scripts/performance/topN_perf.py b/scripts/performance/topN_perf.py index 07be195fc8..56b100da87 100755 --- a/scripts/performance/topN_perf.py +++ b/scripts/performance/topN_perf.py @@ -1,72 +1,77 @@ #!/usr/bin/env python3 -# Print the top N most executed functions in QEMU using perf. -# Syntax: -# topN_perf.py [-h] [-n] -- \ -# [] \ -# [] -# -# [-h] - Print the script arguments help message. -# [-n] - Specify the number of top functions to print. -# - If this flag is not specified, the tool defaults to 25. -# -# Example of usage: -# topN_perf.py -n 20 -- qemu-arm coulomb_double-arm -# -# This file is a part of the project "TCG Continuous Benchmarking". -# -# Copyright (C) 2020 Ahmed Karaman -# Copyright (C) 2020 Aleksandar Markovic -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . +""" +Print the top N most executed functions in QEMU using perf. + +Syntax: +topN_perf.py [-h] [-n ] -- \ + [] \ + [] + +[-h] - Print the script arguments help message. +[-n] - Specify the number of top functions to print. + - If this flag is not specified, the tool defaults to 25. + +Example of usage: +topN_perf.py -n 20 -- qemu-arm coulomb_double-arm + +This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman +Copyright (C) 2020 Aleksandar Markovic + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see . +""" import argparse import os import subprocess import sys +import tempfile # Parse the command line arguments -parser = argparse.ArgumentParser( - usage='topN_perf.py [-h] [-n] -- ' +PARSER = argparse.ArgumentParser( + usage='topN_perf.py [-h] [-n ] -- ' ' [] ' ' []') -parser.add_argument('-n', dest='top', type=int, default=25, +PARSER.add_argument('-n', dest='top', type=int, default=25, help='Specify the number of top functions to print.') -parser.add_argument('command', type=str, nargs='+', help=argparse.SUPPRESS) +PARSER.add_argument('command', type=str, nargs='+', help=argparse.SUPPRESS) -args = parser.parse_args() +ARGS = PARSER.parse_args() # Extract the needed variables from the args -command = args.command -top = args.top +COMMAND = ARGS.command +TOP = ARGS.top # Insure that perf is installed -check_perf_presence = subprocess.run(["which", "perf"], - stdout=subprocess.DEVNULL) -if check_perf_presence.returncode: +CHECK_PERF_PRESENCE = subprocess.run(["which", "perf"], + stdout=subprocess.DEVNULL, + check=False) +if CHECK_PERF_PRESENCE.returncode: sys.exit("Please install perf before running the script!") # Insure user has previllage to run perf -check_perf_executability = subprocess.run(["perf", "stat", "ls", "/"], +CHECK_PERF_EXECUTABILITY = subprocess.run(["perf", "stat", "ls", "/"], stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL) -if check_perf_executability.returncode: - sys.exit( -""" + stderr=subprocess.DEVNULL, + check=False) +if CHECK_PERF_EXECUTABILITY.returncode: + sys.exit(""" Error: You may not have permission to collect stats. @@ -85,43 +90,42 @@ To make this setting permanent, edit /etc/sysctl.conf too, e.g.: kernel.perf_event_paranoid = -1 * Alternatively, you can run this script under sudo privileges. -""" -) - -# Run perf record -perf_record = subprocess.run((["perf", "record", "--output=/tmp/perf.data"] + - command), - stdout=subprocess.DEVNULL, - stderr=subprocess.PIPE) -if perf_record.returncode: - os.unlink('/tmp/perf.data') - sys.exit(perf_record.stderr.decode("utf-8")) - -# Save perf report output to /tmp/perf_report.out -with open("/tmp/perf_report.out", "w") as output: - perf_report = subprocess.run( - ["perf", "report", "--input=/tmp/perf.data", "--stdio"], - stdout=output, - stderr=subprocess.PIPE) - if perf_report.returncode: - os.unlink('/tmp/perf.data') - output.close() - os.unlink('/tmp/perf_report.out') - sys.exit(perf_report.stderr.decode("utf-8")) - -# Read the reported data to functions[] -functions = [] -with open("/tmp/perf_report.out", "r") as data: - # Only read lines that are not comments (comments start with #) - # Only read lines that are not empty - functions = [line for line in data.readlines() if line and line[0] - != '#' and line[0] != "\n"] - -# Limit the number of top functions to "top" -number_of_top_functions = top if len(functions) > top else len(functions) - -# Store the data of the top functions in top_functions[] -top_functions = functions[:number_of_top_functions] +""") + +# Run perf and save all intermediate files in a temporary directory +with tempfile.TemporaryDirectory() as tmpdir: + RECORD_PATH = os.path.join(tmpdir, "record.data") + REPORT_PATH = os.path.join(tmpdir, "report.txt") + + PERF_RECORD = subprocess.run((["perf", "record", "--output="+RECORD_PATH] + + COMMAND), + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + check=False) + if PERF_RECORD.returncode: + sys.exit(PERF_RECORD.stderr.decode("utf-8")) + + with open(REPORT_PATH, "w") as output: + PERF_REPORT = subprocess.run( + ["perf", "report", "--input="+RECORD_PATH, "--stdio"], + stdout=output, + stderr=subprocess.PIPE, + check=False) + if PERF_REPORT.returncode: + sys.exit(PERF_REPORT.stderr.decode("utf-8")) + + # Save the reported data to FUNCTIONS[] + with open(REPORT_PATH, "r") as data: + # Only read lines that are not comments (comments start with #) + # Only read lines that are not empty + FUNCTIONS = [line for line in data.readlines() if line and + line[0] != '#' and line[0] != "\n"] + +# Limit the number of top functions to "TOP" +NO_TOP_FUNCTIONS = TOP if len(FUNCTIONS) > TOP else len(FUNCTIONS) + +# Store the data of the top functions in TOP_FUNCTIONS[] +TOP_FUNCTIONS = FUNCTIONS[:NO_TOP_FUNCTIONS] # Print table header print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.format('No.', @@ -134,7 +138,7 @@ print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.format('No.', '-' * 25)) # Print top N functions -for (index, function) in enumerate(top_functions, start=1): +for (index, function) in enumerate(TOP_FUNCTIONS, start=1): function_data = function.split() function_percentage = function_data[0] function_name = function_data[-1] @@ -143,7 +147,3 @@ for (index, function) in enumerate(top_functions, start=1): function_percentage, function_name, function_invoker)) - -# Remove intermediate files -os.unlink('/tmp/perf.data') -os.unlink('/tmp/perf_report.out') From patchwork Fri Aug 28 10:40:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ahmed Karaman X-Patchwork-Id: 275300 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.6 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 346EAC433E7 for ; Fri, 28 Aug 2020 10:44:29 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D45DA208CA for ; Fri, 28 Aug 2020 10:44:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="HK/+cklW" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D45DA208CA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:36646 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kBbrw-0008Gm-5K for qemu-devel@archiver.kernel.org; Fri, 28 Aug 2020 06:44:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34414) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kBbpD-0003Pe-Ia for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:39 -0400 Received: from mail-wr1-x443.google.com ([2a00:1450:4864:20::443]:45770) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kBbpA-0005gS-RI for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:39 -0400 Received: by mail-wr1-x443.google.com with SMTP id h15so830654wrt.12 for ; Fri, 28 Aug 2020 03:41:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=6VnUOaw47q1sBFWEPDkKi8Rl+L6HJXn0J6GhoPyLYWM=; b=HK/+cklW0orMXNl4Is5lsX4XkmBhx776gGavdwJf8IBs89t4vZJzXPAaCtxc6yKIvc RzQB1Qus6S9Bn6Mzp/bHtBKVuU99Vcl4/E54OUhOshOmMFSRImyfLeEu3yKgjaZNuKZr Q/R91V2p8iDPXFPeLzOrwTJzcHoK+Rwdtb5Izj+iATpzclxbu7oMxWbdaKr5AbmjWolH WQEPE/DkQ7BvxFwG2cz0hL0+6qnx3otbnGYt7t0mxUeUiEXYIxZeDOGK0qs9WxCdaBSo ZNaz31UqFtszJ4YvYS+YHjq3NuI1ipWTa24bHx8MYl6pALyPRYiokQMv3shLjDT4F9bH NkiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=6VnUOaw47q1sBFWEPDkKi8Rl+L6HJXn0J6GhoPyLYWM=; b=NriiN+nz4yZkfD74HZulD9tIomLmngDWDA8gazf8+8/WJHHvFU5zcT5gGhzY5GBOQQ E+WlbiNH+y2SDEIjCdO7wq/KE26t8X/PZaUepCvMWKRVxyxifYmXvJrfQMn89RuDPkIJ sYzFCFV+adVzh8kg4R26iZWSpisLfzwYbeoiBsUvP3DdYwcYKzywaia/4Tu+yLTK7a+/ 11BbMJiuu9+c8iIjVEX4LUBEhEg8bL/fSEfxWtOJtIVqfld2ausPS8moJRsmx93PLtNC n7liJJuATdjssXl9WRFwM+zup86OT3AXSAmeKCOe59p1sI5+ACWWMErExjyzkKAKXZRY iTdA== X-Gm-Message-State: AOAM533QXLo3R0Uz/AxpcfakWt4+/QsIVEYutCI00pdz4sFd8rW4P2Rl rZ3xit9u9SaskK8XOYJn6gdLKIUiLPu4/A== X-Google-Smtp-Source: ABdhPJzJsAjybpcBhmfxWDs+uG3yuDlTJWs5Nnd98OPlyvZdKNUSixe42FeYCDweCVGzHmXJt3aKPA== X-Received: by 2002:adf:9d44:: with SMTP id o4mr864841wre.99.1598611294899; Fri, 28 Aug 2020 03:41:34 -0700 (PDT) Received: from localhost.localdomain ([197.58.77.158]) by smtp.gmail.com with ESMTPSA id e18sm1307453wrx.50.2020.08.28.03.41.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Aug 2020 03:41:34 -0700 (PDT) From: Ahmed Karaman To: qemu-devel@nongnu.org, aleksandar.qemu.devel@gmail.com, philmd@redhat.com, alex.bennee@linaro.org, eblake@redhat.com, ldoktor@redhat.com, jsnow@redhat.com, rth@twiddle.net, ehabkost@redhat.com, crosa@redhat.com Subject: [PATCH 2/9] scripts/performance: Refactor topN_callgrind.py Date: Fri, 28 Aug 2020 12:40:55 +0200 Message-Id: <20200828104102.4490-3-ahmedkhaledkaraman@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> References: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> Received-SPF: pass client-ip=2a00:1450:4864:20::443; envelope-from=ahmedkhaledkaraman@gmail.com; helo=mail-wr1-x443.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_PDS_OTHER_BAD_TLD=0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ahmed Karaman Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" - Apply pylint and flake8 formatting rules to the script. - Use 'tempfile' instead of '/tmp' for creating temporary files. Signed-off-by: Ahmed Karaman --- scripts/performance/topN_callgrind.py | 169 +++++++++++++------------- 1 file changed, 87 insertions(+), 82 deletions(-) diff --git a/scripts/performance/topN_callgrind.py b/scripts/performance/topN_callgrind.py index 67c59197af..f8a554f393 100755 --- a/scripts/performance/topN_callgrind.py +++ b/scripts/performance/topN_callgrind.py @@ -1,113 +1,122 @@ #!/usr/bin/env python3 -# Print the top N most executed functions in QEMU using callgrind. -# Syntax: -# topN_callgrind.py [-h] [-n] -- \ -# [] \ -# [] -# -# [-h] - Print the script arguments help message. -# [-n] - Specify the number of top functions to print. -# - If this flag is not specified, the tool defaults to 25. -# -# Example of usage: -# topN_callgrind.py -n 20 -- qemu-arm coulomb_double-arm -# -# This file is a part of the project "TCG Continuous Benchmarking". -# -# Copyright (C) 2020 Ahmed Karaman -# Copyright (C) 2020 Aleksandar Markovic -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . +""" +Print the top N most executed functions in QEMU using callgrind. + +Syntax: +topN_callgrind.py [-h] [-n ] -- \ + [] \ + [] + +[-h] - Print the script arguments help message. +[-n] - Specify the number of top functions to print. + - If this flag is not specified, the tool defaults to 25. + +Example of usage: +topN_callgrind.py -n 20 -- qemu-arm coulomb_double-arm + +This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman +Copyright (C) 2020 Aleksandar Markovic + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see . +""" import argparse import os import subprocess import sys +import tempfile # Parse the command line arguments -parser = argparse.ArgumentParser( - usage='topN_callgrind.py [-h] [-n] -- ' +PARSER = argparse.ArgumentParser( + usage='topN_callgrind.py [-h] [-n] -- ' ' [] ' ' []') -parser.add_argument('-n', dest='top', type=int, default=25, +PARSER.add_argument('-n', dest='top', type=int, default=25, help='Specify the number of top functions to print.') -parser.add_argument('command', type=str, nargs='+', help=argparse.SUPPRESS) +PARSER.add_argument('command', type=str, nargs='+', help=argparse.SUPPRESS) -args = parser.parse_args() +ARGS = PARSER.parse_args() # Extract the needed variables from the args -command = args.command -top = args.top +COMMAND = ARGS.command +TOP = ARGS.top # Insure that valgrind is installed -check_valgrind_presence = subprocess.run(["which", "valgrind"], - stdout=subprocess.DEVNULL) -if check_valgrind_presence.returncode: +CHECK_VALGRIND_PRESENCE = subprocess.run(["which", "valgrind"], + stdout=subprocess.DEVNULL, + check=False) +if CHECK_VALGRIND_PRESENCE.returncode: sys.exit("Please install valgrind before running the script!") -# Run callgrind -callgrind = subprocess.run(( - ["valgrind", "--tool=callgrind", "--callgrind-out-file=/tmp/callgrind.data"] - + command), - stdout=subprocess.DEVNULL, - stderr=subprocess.PIPE) -if callgrind.returncode: - sys.exit(callgrind.stderr.decode("utf-8")) - -# Save callgrind_annotate output to /tmp/callgrind_annotate.out -with open("/tmp/callgrind_annotate.out", "w") as output: - callgrind_annotate = subprocess.run(["callgrind_annotate", - "/tmp/callgrind.data"], - stdout=output, - stderr=subprocess.PIPE) - if callgrind_annotate.returncode: - os.unlink('/tmp/callgrind.data') - output.close() - os.unlink('/tmp/callgrind_annotate.out') - sys.exit(callgrind_annotate.stderr.decode("utf-8")) - -# Read the callgrind_annotate output to callgrind_data[] -callgrind_data = [] -with open('/tmp/callgrind_annotate.out', 'r') as data: - callgrind_data = data.readlines() +# Run callgrind and save all intermediate files in a temporary directory +with tempfile.TemporaryDirectory() as tmpdir: + CALLGRIND_DATA_PATH = os.path.join(tmpdir, "callgrind.data") + ANNOTATE_OUT_PATH = os.path.join(tmpdir, "callgrind_annotate.out") + + # Run callgrind + CALLGRIND = subprocess.run((["valgrind", + "--tool=callgrind", + "--callgrind-out-file="+CALLGRIND_DATA_PATH] + + COMMAND), + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + check=False) + if CALLGRIND.returncode: + sys.exit(CALLGRIND.stderr.decode("utf-8")) + + with open(ANNOTATE_OUT_PATH, "w") as output: + CALLGRIND_ANNOTATE = subprocess.run(["callgrind_annotate", + CALLGRIND_DATA_PATH], + stdout=output, + stderr=subprocess.PIPE, + check=False) + if CALLGRIND_ANNOTATE.returncode: + sys.exit(CALLGRIND_ANNOTATE.stderr.decode("utf-8")) + + # Read the callgrind_annotate output to CALLGRIND_DATA[] + CALLGRIND_DATA = [] + with open(ANNOTATE_OUT_PATH, 'r') as data: + CALLGRIND_DATA = data.readlines() # Line number with the total number of instructions -total_instructions_line_number = 20 +TOTAL_INSTRUCTIONS_LINE_NO = 20 # Get the total number of instructions -total_instructions_line_data = callgrind_data[total_instructions_line_number] -total_number_of_instructions = total_instructions_line_data.split(' ')[0] -total_number_of_instructions = int( - total_number_of_instructions.replace(',', '')) +TOTAL_INSTRUCTIONS_LINE_DATA = CALLGRIND_DATA[TOTAL_INSTRUCTIONS_LINE_NO] +TOTAL_NUMBER_OF_INSTRUCTIONS = TOTAL_INSTRUCTIONS_LINE_DATA.split(' ')[0] +TOTAL_NUMBER_OF_INSTRUCTIONS = int( + TOTAL_NUMBER_OF_INSTRUCTIONS.replace(',', '')) # Line number with the top function -first_func_line = 25 +FIRST_FUNC_LINE = 25 # Number of functions recorded by callgrind, last two lines are always empty -number_of_functions = len(callgrind_data) - first_func_line - 2 +NUMBER_OF_FUNCTIONS = len(CALLGRIND_DATA) - FIRST_FUNC_LINE - 2 # Limit the number of top functions to "top" -number_of_top_functions = (top if number_of_functions > - top else number_of_functions) +NUMBER_OF_TOP_FUNCTIONS = (TOP if NUMBER_OF_FUNCTIONS > + TOP else NUMBER_OF_FUNCTIONS) # Store the data of the top functions in top_functions[] -top_functions = callgrind_data[first_func_line: - first_func_line + number_of_top_functions] +TOP_FUNCTIONS = CALLGRIND_DATA[FIRST_FUNC_LINE: + FIRST_FUNC_LINE + NUMBER_OF_TOP_FUNCTIONS] # Print table header print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.format('No.', @@ -121,12 +130,12 @@ print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.format('No.', )) # Print top N functions -for (index, function) in enumerate(top_functions, start=1): +for (index, function) in enumerate(TOP_FUNCTIONS, start=1): function_data = function.split() # Calculate function percentage function_instructions = float(function_data[0].replace(',', '')) function_percentage = (function_instructions / - total_number_of_instructions)*100 + TOTAL_NUMBER_OF_INSTRUCTIONS)*100 # Get function name and source files path function_source_file, function_name = function_data[1].split(':') # Print extracted data @@ -134,7 +143,3 @@ for (index, function) in enumerate(top_functions, start=1): round(function_percentage, 3), function_name, function_source_file)) - -# Remove intermediate files -os.unlink('/tmp/callgrind.data') -os.unlink('/tmp/callgrind_annotate.out') From patchwork Fri Aug 28 10:40:57 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ahmed Karaman X-Patchwork-Id: 275301 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.6 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A201AC43461 for ; Fri, 28 Aug 2020 10:43:07 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 60A68208CA for ; Fri, 28 Aug 2020 10:43:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Ee1yLB37" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 60A68208CA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:57154 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kBbqb-00058u-9I for qemu-devel@archiver.kernel.org; Fri, 28 Aug 2020 06:43:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34440) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kBbpH-0003To-FF for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:43 -0400 Received: from mail-wm1-x329.google.com ([2a00:1450:4864:20::329]:39586) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kBbpE-0005gz-JQ for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:43 -0400 Received: by mail-wm1-x329.google.com with SMTP id b79so461882wmb.4 for ; Fri, 28 Aug 2020 03:41:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=BLJGQzaC58i7Jzawy2mcPTfB5Z7zUVTcLSsOZpQQw9o=; b=Ee1yLB37j//pyLubHmHwGE8x2jNKvKKRnrZaLj1/xd5pSgwuowx6nfn0DOwOpSopLR lGrY3IzEbRA9n29sUMWiAFi5Nj2kl0YaJ/mhdAW/uodYb0B7NNvBctZw+DZXFQ3yidzC EP9i1sM/0hQ2O7xlHkMT6RC679fmLATZv4VoQMCxTWt+zhdBTs6I14b7TXYDDG/KkRCZ 86si0/dF9yibzmS08RLtZXmlOAvo88BSy/Ly0dVn0/ALObIPfXIAO08mTXce39zOOXZf kv9JgEYhlYyp+y5XTcjCCx0XcGHyghgbGVBw4WAQTtaQYY+5HAiIs0lfR5QrPs6MPB7N CXvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=BLJGQzaC58i7Jzawy2mcPTfB5Z7zUVTcLSsOZpQQw9o=; b=A7mJaKGoZVAedVWUOv2sMnSRbe+WRYr5r+gHJs3hCyDC0+PItsbn+MH7zYaac3qPgU oXzUp0EByUSSp383wDWES+vsiNquyZMwj0qQBErrwFtHis8nKRZYZOg6VplliMw+rey8 1v8zDcljO6rTiM10nM2HomWd2ry3rSz6hVcMbj5B9LoUQDaYIT9yHbzPJknThkaoSjuN CezMwJ3yhn2icmsk54U0xDTLCx2zXMnIff2WNrfSTxSEklbr8D8it5TrXi2O9GdckmS1 muKWR3W4ToEFsnOhLcpQyVO+E8Yk1MooiftFUKQVwo5U+YgZBt4UOSE7eIKtR55iZ7cV yu1Q== X-Gm-Message-State: AOAM5333ZetjEBn2uL10Mll/K7q/nQzpX04uhpuT9XH4KXcjM0STbCF0 gsvUkaXDkT+NMIoPkYA2qeJoP0JO5oQqHg== X-Google-Smtp-Source: ABdhPJxzVzFgK/pQN+NIm9cfQdPnRVmfEJwuo9hH0KmgVb+A2Odn35Uv2iFuBuHTUbF0FlSOjwCBsQ== X-Received: by 2002:a1c:e919:: with SMTP id q25mr953381wmc.123.1598611298903; Fri, 28 Aug 2020 03:41:38 -0700 (PDT) Received: from localhost.localdomain ([197.58.77.158]) by smtp.gmail.com with ESMTPSA id e18sm1307453wrx.50.2020.08.28.03.41.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Aug 2020 03:41:38 -0700 (PDT) From: Ahmed Karaman To: qemu-devel@nongnu.org, aleksandar.qemu.devel@gmail.com, philmd@redhat.com, alex.bennee@linaro.org, eblake@redhat.com, ldoktor@redhat.com, jsnow@redhat.com, rth@twiddle.net, ehabkost@redhat.com, crosa@redhat.com Subject: [PATCH 4/9] scripts/performance: Add list_fn_callees.py script Date: Fri, 28 Aug 2020 12:40:57 +0200 Message-Id: <20200828104102.4490-5-ahmedkhaledkaraman@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> References: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> Received-SPF: pass client-ip=2a00:1450:4864:20::329; envelope-from=ahmedkhaledkaraman@gmail.com; helo=mail-wm1-x329.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ahmed Karaman Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Python script that prints the callees of a given list of QEMU functions. Syntax: list_fn_callees.py [-h] -f FUNCTION [FUNCTION ...] -- \ [] \ [] [-h] - Print the script arguments help message. -f FUNCTION [FUNCTION ...] - List of function names Example of usage: list_fn_callees.py -f helper_float_sub_d helper_float_mul_d -- \ qemu-mips coulomb_double-mips -n10 Example output: Total number of instructions: 108,952,851 Callees of helper_float_sub_d: No. Instructions Percentage Calls Ins/Call Function Name Source File --- ------------ ---------- ------ -------- ------------- --------------- 1 153,160 0.141% 1,305 117 float64_sub /fpu/softfloat.c Callees of helper_float_mul_d: No. Instructions Percentage Calls Ins/Call Function Name Source File --- ------------ ---------- ------ -------- ------------- --------------- 1 131,137 0.120% 1,014 129 float64_mul /fpu/softfloat.c Signed-off-by: Ahmed Karaman --- scripts/performance/list_fn_callees.py | 245 +++++++++++++++++++++++++ 1 file changed, 245 insertions(+) create mode 100755 scripts/performance/list_fn_callees.py diff --git a/scripts/performance/list_fn_callees.py b/scripts/performance/list_fn_callees.py new file mode 100755 index 0000000000..6aa8f6b6ca --- /dev/null +++ b/scripts/performance/list_fn_callees.py @@ -0,0 +1,245 @@ +#!/usr/bin/env python3 + +""" +Print the callees of a given list of QEMU functions. + +This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman +Copyright (C) 2020 Aleksandar Markovic + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see . +""" + +import argparse +import os +import subprocess +import sys +import tempfile + +from typing import List, Union + + +def find_function_lines(function_name: str, + callgrind_data: List[str]) -> List[int]: + """ + Search for the line with the function name in the + callgrind_annotate output when ran using --tre=calling. + All the function callees should be listed after that line. + + Parameters: + function_name (string): The desired function name to print its callees + callgrind_data (List[str]): callgrind_annotate output + + Returns: + (List[int]): List of function line numbers + """ + lines = [] + for (i, callgrind_datum) in enumerate(callgrind_data): + split_line = callgrind_datum.split() + if len(split_line) > 2 and \ + split_line[1] == "*" and \ + split_line[2].split(":")[-1] == function_name: + # Function might be in the callgrind_annotate output more than + # once, so don't break after finding an instance + if callgrind_data[i + 1] != "\n": + # Only append the line number if the found instance has + # callees + lines.append(i) + return lines + + +def get_function_calles( + function_lines: List[int], + callgrind_data: List[str]) -> List[List[Union[str, int]]]: + """ + Get all callees data for a function given its list of line numbers in + callgrind_annotate output. + + Parameters: + function_lines (List[int]): Line numbers of the function to get its callees + callgrind_data (List[str]): callgrind_annotate output + + Returns: + (List[List[Union[str, int]]]):[[number_of_instructions(int), + callee_name(str), + number_of_calls(int), + source_file(str)], + ...] + """ + callees: List[List[Union[str, int]]] = [] + for function_line in function_lines: + next_callee = function_line + 1 + while callgrind_data[next_callee] != "\n": + split_line = callgrind_data[next_callee].split() + number_of_instructions = int(split_line[0].replace(",", "")) + source_file = split_line[2].split(":")[0] + callee_name = split_line[2].split(":")[1] + number_of_calls = int(split_line[3][1:-2]) + callees.append([number_of_instructions, callee_name, + number_of_calls, source_file]) + next_callee += 1 + return sorted(callees, reverse=True) + + +def main(): + """ + Parse the command line arguments then start execution. + + Syntax: + list_fn_callees.py [-h] -f FUNCTION [FUNCTION ...] -- \ + [] \ + [] + + [-h] - Print the script arguments help message. + -f FUNCTION [FUNCTION ...] - List of function names + + Example of usage: + list_fn_callees.py -f helper_float_sub_d helper_float_mul_d -- \ + qemu-mips coulomb_double-mips + """ + + # Parse the command line arguments + parser = argparse.ArgumentParser( + usage="list_fn_callees.py [-h] -f FUNCTION [FUNCTION ...] -- " + " [] " + " []") + + parser.add_argument("-f", dest="function", type=str, + nargs="+", required=True, + help="list of function names to print their callees") + + parser.add_argument("command", type=str, nargs="+", help=argparse.SUPPRESS) + + args = parser.parse_args() + + # Extract the needed variables from the args + command = args.command + function_names = args.function + + # Insure that valgrind is installed + check_valgrind = subprocess.run( + ["which", "valgrind"], stdout=subprocess.DEVNULL, check=False) + if check_valgrind.returncode: + sys.exit("Please install valgrind before running the script.") + + # Save all intermediate files in a temporary directory + with tempfile.TemporaryDirectory() as tmpdirname: + # callgrind output file path + data_path = os.path.join(tmpdirname, "callgrind.data") + # callgrind_annotate output file path + annotate_out_path = os.path.join(tmpdirname, "callgrind_annotate.out") + + # Run callgrind + callgrind = subprocess.run((["valgrind", + "--tool=callgrind", + "--callgrind-out-file=" + data_path] + + command), + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + check=False) + if callgrind.returncode: + sys.exit(callgrind.stderr.decode("utf-8")) + + # Save callgrind_annotate output + with open(annotate_out_path, "w") as output: + callgrind_annotate = subprocess.run(["callgrind_annotate", + data_path, + "--threshold=100", + "--tree=calling"], + stdout=output, + stderr=subprocess.PIPE, + check=False) + if callgrind_annotate.returncode: + sys.exit(callgrind_annotate.stderr.decode("utf-8")) + + # Read the callgrind_annotate output to callgrind_data[] + callgrind_data = [] + with open(annotate_out_path, "r") as data: + callgrind_data = data.readlines() + + # Line number with the total number of instructions + total_instructions_line_number = 20 + # Get the total number of instructions + total_instructions_line_data = \ + callgrind_data[total_instructions_line_number] + total_instructions = total_instructions_line_data.split()[0] + + print("Total number of instructions: {}\n".format(total_instructions)) + + # Remove commas and convert to int + total_instructions = int(total_instructions.replace(",", "")) + + for function_name in function_names: + # Line numbers with the desired function + function_lines = find_function_lines(function_name, callgrind_data) + + if len(function_lines) == 0: + print("Couldn't locate function: {}.\n".format( + function_name)) + continue + + # Get function callees + function_callees = get_function_calles( + function_lines, callgrind_data) + + print("Callees of {}:\n".format(function_name)) + + # Print table header + print("{:>4} {:>15} {:>10} {:>15} {:>10} {:<25} {}". + format( + "No.", + "Instructions", + "Percentage", + "Calls", + "Ins/Call", + "Function Name", + "Source File") + ) + + print("{:>4} {:>15} {:>10} {:>15} {:>10} {:<25} {}". + format( + "-" * 4, + "-" * 15, + "-" * 10, + "-" * 15, + "-" * 10, + "-" * 25, + "-" * 30) + ) + + for (index, callee) in enumerate(function_callees, start=1): + instructions = callee[0] + percentage = (callee[0] / total_instructions) * 100 + calls = callee[2] + instruction_per_call = int(callee[0] / callee[2]) + function_name = callee[1] + source_file = callee[3] + # Print extracted data + print("{:>4} {:>15} {:>9.3f}% {:>15} {:>10} {:<25} {}". + format( + index, + format(instructions, ","), + round(percentage, 3), + format(calls, ","), + format(instruction_per_call, ","), + function_name, + source_file) + ) + + print("\n") + + +if __name__ == "__main__": + main() From patchwork Fri Aug 28 10:40:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ahmed Karaman X-Patchwork-Id: 275299 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.6 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19285C433E6 for ; Fri, 28 Aug 2020 10:47:02 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C245C208CA for ; Fri, 28 Aug 2020 10:47:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="d22AIMjK" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C245C208CA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:42302 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kBbuP-0002Fi-3d for qemu-devel@archiver.kernel.org; Fri, 28 Aug 2020 06:47:01 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34486) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kBbpM-0003Yn-4j for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:48 -0400 Received: from mail-wr1-x441.google.com ([2a00:1450:4864:20::441]:40951) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kBbpJ-0005hg-3a for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:47 -0400 Received: by mail-wr1-x441.google.com with SMTP id b18so843147wrs.7 for ; Fri, 28 Aug 2020 03:41:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=LIf0cxQtW4hzKm0O7L8caZ594n9XcAZSf6i1rCeNjZk=; b=d22AIMjKeRUgBAq6Me+AslRt0Z7/E/+lfAYk3Vhyj8k1DRf8LQV5OXxTbpE/8tCkRC frlF68m70idHOn5t4oyqLtwzxdjs7EiR0UzvYpV4K+XqBu0PKE6PHCzb2jhS7CDPuLFr e6l82FARRMrbNCzVWFcNX8xoYPvXz7IV8nrAr3gN+zTMuytZyPFx4RtHP0uIevCpmbCn d0C2HPig5+nf8mvn+Gq8UER1gcI3Vi7YAbNXUfQkj4v5mP40brC6JnPd/dO51+ZfXlqP KKel+EL8jPYe0ojuwXc/Qyj3AGzSWETumaaUC7xLZ+91uV1sYj4smmidCJfTClMEunx5 /eIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=LIf0cxQtW4hzKm0O7L8caZ594n9XcAZSf6i1rCeNjZk=; b=feEeLct9rkHjZ88mvTPUA4l7AWPH55VmgjDlav8Grcx26XOoq2RlMTk0buX2lCXkeg x03GsHgfixSS59dkFF+/2nle12X8jwBSvsFvpB87A4LoIlMs8EIKEwbnbZ1yNodWAPo2 d2vn4PJnBZOucu+GUl65aOMYUmE4fXi9mAUTMiCsYU+fyEwezbp+PNohHVXvftwqVtLZ DktxSXdJnE90NuV6PrZQW3w2VA0kB0r1uIC5Xlr+/0074aqDGEHOwnlL8VTdiUP28s5F 3cPk8OzFs+1H1dvz6R9FyhxsF8YgFH/tQlbnT1msTIhrxK/WrJA9w39fpNlnjkYnT9kt XlqA== X-Gm-Message-State: AOAM532DQqWgeRH7B5EkZJz1RXs6MuN8PCxUHB3U7wvnmQLDRfFpfYpE VjAsOWrAltggJMxmA+zqq90nsyKBD7J7Rw== X-Google-Smtp-Source: ABdhPJzLgdt+VUg1kK5s5dAGQlueSyKj9ZHTrjhfUnKU3dUnC08oX7yEINk3+0r1h3AlB5U2mS4WfA== X-Received: by 2002:a5d:6087:: with SMTP id w7mr916406wrt.290.1598611303029; Fri, 28 Aug 2020 03:41:43 -0700 (PDT) Received: from localhost.localdomain ([197.58.77.158]) by smtp.gmail.com with ESMTPSA id e18sm1307453wrx.50.2020.08.28.03.41.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Aug 2020 03:41:42 -0700 (PDT) From: Ahmed Karaman To: qemu-devel@nongnu.org, aleksandar.qemu.devel@gmail.com, philmd@redhat.com, alex.bennee@linaro.org, eblake@redhat.com, ldoktor@redhat.com, jsnow@redhat.com, rth@twiddle.net, ehabkost@redhat.com, crosa@redhat.com Subject: [PATCH 6/9] scripts/performance: Add bisect.py script Date: Fri, 28 Aug 2020 12:40:59 +0200 Message-Id: <20200828104102.4490-7-ahmedkhaledkaraman@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> References: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> Received-SPF: pass client-ip=2a00:1450:4864:20::441; envelope-from=ahmedkhaledkaraman@gmail.com; helo=mail-wr1-x441.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ahmed Karaman Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Python script that locates the commit that caused a performance degradation or improvement in QEMU using the git bisect command (binary search). Syntax: bisect.py [-h] -s,--start START [-e,--end END] [-q,--qemu QEMU] \ --target TARGET --tool {perf,callgrind} -- \ [] [-h] - Print the script arguments help message -s,--start START - First commit hash in the search range [-e,--end END] - Last commit hash in the search range (default: Latest commit) [-q,--qemu QEMU] - QEMU path. (default: Path to a GitHub QEMU clone) --target TARGET - QEMU target name --tool {perf,callgrind} - Underlying tool used for measurements Example of usage: bisect.py --start=fdd76fecdd --qemu=/path/to/qemu --target=ppc \ --tool=perf -- coulomb_double-ppc -n 1000 Example output: Start Commit Instructions: 12,710,790,060 End Commit Instructions: 13,031,083,512 Performance Change: -2.458% Estimated Number of Steps: 10 *****************BISECT STEP 1***************** Instructions: 13,031,097,790 Status: slow commit *****************BISECT STEP 2***************** Instructions: 12,710,805,265 Status: fast commit *****************BISECT STEP 3***************** Instructions: 13,031,028,053 Status: slow commit *****************BISECT STEP 4***************** Instructions: 12,711,763,211 Status: fast commit *****************BISECT STEP 5***************** Instructions: 13,031,027,292 Status: slow commit *****************BISECT STEP 6***************** Instructions: 12,711,748,738 Status: fast commit *****************BISECT STEP 7***************** Instructions: 12,711,748,788 Status: fast commit *****************BISECT STEP 8***************** Instructions: 13,031,100,493 Status: slow commit *****************BISECT STEP 9***************** Instructions: 12,714,472,954 Status: fast commit ****************BISECT STEP 10***************** Instructions: 12,715,409,153 Status: fast commit ****************BISECT STEP 11***************** Instructions: 12,715,394,739 Status: fast commit *****************BISECT RESULT***************** commit 0673ecdf6cb2b1445a85283db8cbacb251c46516 Author: Richard Henderson Date: Tue May 5 10:40:23 2020 -0700 softfloat: Inline float64 compare specializations Replace the float64 compare specializations with inline functions that call the standard float64_compare{,_quiet} functions. Use bool as the return type. *********************************************** Signed-off-by: Ahmed Karaman --- scripts/performance/bisect.py | 425 ++++++++++++++++++++++++++++++++++ 1 file changed, 425 insertions(+) create mode 100755 scripts/performance/bisect.py diff --git a/scripts/performance/bisect.py b/scripts/performance/bisect.py new file mode 100755 index 0000000000..0c60be22ce --- /dev/null +++ b/scripts/performance/bisect.py @@ -0,0 +1,425 @@ +#!/usr/bin/env python3 + +""" +Locate the commit that caused a performance degradation or improvement in +QEMU using the git bisect command (binary search). + +This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman +Copyright (C) 2020 Aleksandar Markovic + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see . +""" + +import argparse +import multiprocessing +import tempfile +import os +import shutil +import subprocess +import sys + +from typing import List + + +# --------------------------- GIT WRAPPERS -------------------------- +def git_bisect(qemu_path: str, qemu_build_path: str, command: str, + args: List[str] = None) -> str: + """ + Wrapper function for running git bisect. + + Parameters: + qemu_path (str): QEMU path + qemu_build_path (str): Path to the build directory with configuration files + command (str): bisect command (start|fast|slow|reset) + args (list): Optional arguments + + Returns: + (str): git bisect stdout. + """ + process = ["git", "bisect", command] + if args: + process += args + bisect = subprocess.run(process, + cwd=qemu_path, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + check=False) + if bisect.returncode: + clean_exit(qemu_build_path, bisect.stderr.decode("utf-8")) + + return bisect.stdout.decode("utf-8") + + +def git_checkout(commit: str, qemu_path: str, qemu_build_path: str) -> None: + """ + Wrapper function for checking out a given git commit. + + Parameters: + commit (str): Commit hash of a git commit + qemu_path (str): QEMU path + qemu_build_path (str): Path to the build directory with configuration files + """ + checkout_commit = subprocess.run(["git", + "checkout", + commit], + cwd=qemu_path, + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + check=False) + if checkout_commit.returncode: + clean_exit(qemu_build_path, checkout_commit.stderr.decode("utf-8")) + + +def git_clone(qemu_path: str) -> None: + """ + Wrapper function for cloning QEMU git repo from GitHub. + + Parameters: + qemu_path (str): Path to clone the QEMU repo to + """ + clone_qemu = subprocess.run(["git", + "clone", + "https://github.com/qemu/qemu.git", + qemu_path], + stderr=subprocess.STDOUT, + check=False) + if clone_qemu.returncode: + sys.exit("Failed to clone QEMU!") +# ------------------------------------------------------------------- + + +def check_requirements(tool: str) -> None: + """ + Verify that all script requirements are installed (perf|callgrind & git). + + Parameters: + tool (str): Tool used for the measurement (perf or callgrind) + """ + if tool == "perf": + check_perf_installation = subprocess.run(["which", "perf"], + stdout=subprocess.DEVNULL, + check=False) + if check_perf_installation.returncode: + sys.exit("Please install perf before running the script.") + + # Insure user has previllage to run perf + check_perf_executability = subprocess.run(["perf", "stat", "ls", "/"], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + check=False) + if check_perf_executability.returncode: + sys.exit(""" + Error: + You may not have permission to collect stats. + Consider tweaking /proc/sys/kernel/perf_event_paranoid, + which controls use of the performance events system by + unprivileged users (without CAP_SYS_ADMIN). + -1: Allow use of (almost) all events by all users + Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK + 0: Disallow ftrace function tracepoint by users without CAP_SYS_ADMIN + Disallow raw tracepoint access by users without CAP_SYS_ADMIN + 1: Disallow CPU event access by users without CAP_SYS_ADMIN + 2: Disallow kernel profiling by users without CAP_SYS_ADMIN + To make this setting permanent, edit /etc/sysctl.conf too, e.g.: + kernel.perf_event_paranoid = -1 + + *Alternatively, you can run this script under sudo privileges. + """) + elif tool == "callgrind": + check_valgrind_installation = subprocess.run(["which", "valgrind"], + stdout=subprocess.DEVNULL, + check=False) + if check_valgrind_installation.returncode: + sys.exit("Please install valgrind before running the script.") + + # Insure that git is installed + check_git_installation = subprocess.run(["which", "git"], + stdout=subprocess.DEVNULL, + check=False) + if check_git_installation.returncode: + sys.exit("Please install git before running the script.") + + +def clean_exit(qemu_build_path: str, error_message: str) -> None: + """ + Clean up intermediate files and exit. + + Parameters: + qemu_build_path (str): Path to the build directory with configuration files + error_message (str): Error message to display after exiting + """ + shutil.rmtree(qemu_build_path) + sys.exit(error_message) + + +def make(qemu_build_path: str) -> None: + """ + Build QEMU by running the Makefile. + + Parameters: + qemu_build_path (str): Path to the build directory with configuration files + """ + run_make = subprocess.run(["make", + "-j", + str(multiprocessing.cpu_count())], + cwd=qemu_build_path, + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + check=False) + if run_make.returncode: + clean_exit(qemu_build_path, run_make.stderr.decode("utf-8")) + + +def measure_instructions(tool: str, qemu_build_path: str, target: str, + command: List[str]) -> int: + """ + Measure the number of instructions when running an program with QEMU. + + Parameters: + tool (str): Tool used for the measurement (perf|callgrind) + qemu_build_path (str): Path to the build directory with configuration files + target (str): QEMU target + command (list): Program path and arguments + + Returns: + (int): Number of instructions. + """ + qemu_exe_path = os.path.join(qemu_build_path, + "{}-linux-user".format(target), + "qemu-{}".format(target)) + instructions = 0 + if tool == "perf": + run_perf = subprocess.run((["perf", + "stat", + "-x", + " ", + "-e", + "instructions", + qemu_exe_path] + + command), + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + check=False) + if run_perf.returncode: + clean_exit(qemu_build_path, run_perf.stderr.decode("utf-8")) + + else: + perf_output = run_perf.stderr.decode("utf-8").split(" ") + instructions = int(perf_output[0]) + + elif tool == "callgrind": + with tempfile.NamedTemporaryFile() as tmpfile: + run_callgrind = subprocess.run((["valgrind", + "--tool=callgrind", + "--callgrind-out-file={}".format( + tmpfile.name), + qemu_exe_path] + + command), + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + check=False) + if run_callgrind.returncode: + clean_exit(qemu_build_path, run_callgrind.stderr.decode("utf-8")) + + else: + callgrind_output = run_callgrind.stderr.decode("utf-8").split("\n") + instructions = int(callgrind_output[8].split(" ")[-1]) + + return instructions + + +def main(): + """ + Parse the command line arguments then start the execution. + + Syntax: + bisect.py [-h] -s,--start START [-e,--end END] [-q,--qemu QEMU] \ + --target TARGET --tool {perf,callgrind} -- \ + [] + + Arguments: + [-h] - Print the script arguments help message + -s,--start START - First commit hash in the search range + [-e,--end END] - Last commit hash in the search range + (default: Latest commit) + [-q,--qemu QEMU] - QEMU path. + (default: Path to a GitHub QEMU clone) + --target TARGET - QEMU target name + --tool {perf,callgrind} - Underlying tool used for measurements + + Example of usage: + bisect.py --start=fdd76fecdd --qemu=/path/to/qemu --target=ppc \ + --tool=perf coulomb_double-ppc -n 1000 + """ + + # Parse the command line arguments + parser = argparse.ArgumentParser( + usage="bisect.py [-h] -s,--start START [-e,--end END] [-q,--qemu QEMU]" + " --target TARGET --tool {perf,callgrind} -- " + " []") + + parser.add_argument("-s", "--start", dest="start", type=str, required=True, + help="First commit hash in the search range") + parser.add_argument("-e", "--end", dest="end", type=str, default="master", + help="Last commit hash in the search range") + parser.add_argument("-q", "--qemu", dest="qemu", type=str, default="", + help="QEMU path") + parser.add_argument("--target", dest="target", type=str, required=True, + help="QEMU target") + parser.add_argument("--tool", dest="tool", choices=["perf", "callgrind"], + required=True, help="Tool used for measurements") + + parser.add_argument("command", type=str, nargs="+", help=argparse.SUPPRESS) + + args = parser.parse_args() + + # Extract the needed variables from the args + start_commit = args.start + end_commit = args.end + qemu = args.qemu + target = args.target + tool = args.tool + command = args.command + + # Set QEMU path + if qemu == "": + # Create a temp directory for cloning QEMU + tmpdir = tempfile.TemporaryDirectory() + qemu_path = os.path.join(tmpdir.name, "qemu") + + # Clone QEMU into the temporary directory + print("Fetching QEMU: ", end="", flush=True) + git_clone(qemu_path) + print() + else: + qemu_path = qemu + + # Create the build directory + qemu_build_path = os.path.join(qemu_path, "tmp-build-gcc") + + if not os.path.exists(qemu_build_path): + os.mkdir(qemu_build_path) + else: + sys.exit("A build directory with the same name (tmp-build-gcc) used in" + " the script is already in the provided QEMU path.") + + # Configure QEMU + configure = subprocess.run(["../configure", + "--target-list={}-linux-user".format(target)], + cwd=qemu_build_path, + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + check=False) + if configure.returncode: + clean_exit(qemu_build_path, configure.stderr.decode("utf-8")) + + # Do performance measurements for the start commit + git_checkout(start_commit, qemu_path, qemu_build_path) + make(qemu_build_path) + start_commit_instructions = measure_instructions(tool, + qemu_build_path, + target, + command) + print("{:<30} {}".format("Start Commit Instructions:", + format(start_commit_instructions, ","))) + + # Do performance measurements for the end commit + git_checkout(end_commit, qemu_path, qemu_build_path) + make(qemu_build_path) + end_commit_instructions = measure_instructions(tool, + qemu_build_path, + target, + command) + print("{:<30} {}".format("End Commit Instructions:", + format(end_commit_instructions, ","))) + + # Calculate performance difference between start and end commits + performance_difference = \ + (start_commit_instructions - end_commit_instructions) / \ + max(end_commit_instructions, start_commit_instructions) * 100 + performance_change = "+" if performance_difference > 0 else "-" + print("{:<30} {}".format("Performance Change:", + performance_change + + str(round(abs(performance_difference), 3))+"%")) + + # Set the custom terms used for progressing in "git bisect" + term_old = "fast" if performance_difference < 0 else "slow" + term_new = "slow" if term_old == "fast" else "fast" + + # Start git bisect + git_bisect(qemu_path, qemu_build_path, "start", + ["--term-old", term_old, "--term-new", term_new]) + # Set start commit state + git_bisect(qemu_path, qemu_build_path, term_old, [start_commit]) + # Set end commit state + bisect_output = git_bisect( + qemu_path, qemu_build_path, term_new, [end_commit]) + # Print estimated bisect steps + print("\n{:<30} {}\n".format( + "Estimated Number of Steps:", bisect_output.split()[9])) + + # Initialize bisect_count to track the number of performed + bisect_count = 1 + + while True: + print("**************BISECT STEP {}**************". + format(bisect_count)) + + make(qemu_build_path) + + instructions = measure_instructions(tool, + qemu_build_path, + target, + command) + # Find the difference between the current instructions and start/end + # instructions. + diff_end = abs(instructions - end_commit_instructions) + diff_start = abs(instructions - start_commit_instructions) + + # If current number of insructions is closer to that of start, + # set current commit as term_old. + # Else, set current commit as term_new. + if diff_end > diff_start: + bisect_command = term_old + else: + bisect_command = term_new + + print("{:<20} {}".format("Instructions:", format(instructions, ","))) + print("{:<20} {}".format("Status:", "{} commit". + format(bisect_command))) + + bisect_output = git_bisect(qemu_path, qemu_build_path, bisect_command) + + # Continue if still bisecting, + # else, print result and break. + if not bisect_output.split(" ")[0] == "Bisecting:": + print("\n*****************BISECT RESULT*****************") + commit_message_start = bisect_output.find("commit\n") + 7 + commit_message_end = bisect_output.find(":040000") - 1 + print(bisect_output[commit_message_start:commit_message_end]) + break + + bisect_count += 1 + + # Reset git bisect + git_bisect(qemu_path, qemu_build_path, "reset") + + # Delete temp build directory + shutil.rmtree(qemu_build_path) + + +if __name__ == "__main__": + main() From patchwork Fri Aug 28 10:41:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Ahmed Karaman X-Patchwork-Id: 275298 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.6 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98CD3C433E6 for ; Fri, 28 Aug 2020 10:48:15 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2FE812086A for ; Fri, 28 Aug 2020 10:48:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="dX41ymz9" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2FE812086A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:46104 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kBbva-0003qb-Gj for qemu-devel@archiver.kernel.org; Fri, 28 Aug 2020 06:48:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34590) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kBbpU-0003rt-Tz for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:56 -0400 Received: from mail-wr1-x433.google.com ([2a00:1450:4864:20::433]:34674) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kBbpP-0005jN-Rs for qemu-devel@nongnu.org; Fri, 28 Aug 2020 06:41:56 -0400 Received: by mail-wr1-x433.google.com with SMTP id f7so857616wrw.1 for ; Fri, 28 Aug 2020 03:41:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ztdp06tHpUTupGG5kZnKkBKNJQ2GBi6jOrEePLCl0Kw=; b=dX41ymz9tc93PMf0ZRirpDHXn9NI60cKLfOFjuDhBM1gUH6SuQItoBy6JDX/CJOvk6 bDz6/aq3ZOE1IxebLaizUmTQkWn7Mz5zHGHn+HrNOhHXjjGoeVdPGn+5PoPp6HRUwEfp FTV+fQ+S9yrf7NDm0zHDjoGsJACo53WQDRRCEIDKIhvrv/aWtZl+w1PJinQyqz1z+gh9 M79aCKCCrSWxwJA6/g2LHsSvIXFrDCu3Tnu0TlZ1ieQV3oyVWhj0HmDcOh7sqeqlGpCk fKYMHuC2qqtaF4h4gvEzINeRW9s6JlztrNEID7Gjo8L/Uw84Mc3tfOZCi0zDSdCUd0rp ABAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ztdp06tHpUTupGG5kZnKkBKNJQ2GBi6jOrEePLCl0Kw=; b=mb1l57QQfTdbklIqsDkSWNR77EcX+iAQSb3xsZd+/RZUBx4PY0mxygyGjRHAUYIZKG wY0fJbM065LqBCBI8VGSAn7g6Pk83rxyhdi75dAbgfvPthAzh4KUovl7B2OC9934c1N2 pCvnP5GlX+jfeljiqgKlmownBCdrJ0CmIJorDl1OOif0JRfSWnmHOUAwkOObP5dSsw2Z BfXn2MYH+gUUVIZmJ1xB5LaCgejcJuMzgUssWN1GQK4pax4EGLhsO3sMF2NSW+x7k532 HUjy3XViWHJfNTsSoAQ4d5EuZN+3o6jDAb1KL5ztdSy60T5r2LEAOEyT7yK4GVlRMfZr 6Ofg== X-Gm-Message-State: AOAM531X5tgLPxIq1FUHahh1HI+Ptk03nI1WDbyHLPkJLviYBMkh1W8j Xc4C27V5KgC0IjvJwaFFHYnnNjQeIFtW8g== X-Google-Smtp-Source: ABdhPJxZOldAOxbmDbmBbcG+/cHy/Z3PNSWRU55JhPE+ZYjn/P9C2pkVeNhEcVnBDOtvT5yTaUrhXA== X-Received: by 2002:a5d:4145:: with SMTP id c5mr986131wrq.18.1598611306982; Fri, 28 Aug 2020 03:41:46 -0700 (PDT) Received: from localhost.localdomain ([197.58.77.158]) by smtp.gmail.com with ESMTPSA id e18sm1307453wrx.50.2020.08.28.03.41.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Aug 2020 03:41:46 -0700 (PDT) From: Ahmed Karaman To: qemu-devel@nongnu.org, aleksandar.qemu.devel@gmail.com, philmd@redhat.com, alex.bennee@linaro.org, eblake@redhat.com, ldoktor@redhat.com, jsnow@redhat.com, rth@twiddle.net, ehabkost@redhat.com, crosa@redhat.com Subject: [PATCH 7/9] tests/performance: Add nightly tests Date: Fri, 28 Aug 2020 12:41:00 +0200 Message-Id: <20200828104102.4490-8-ahmedkhaledkaraman@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> References: <20200828104102.4490-1-ahmedkhaledkaraman@gmail.com> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::433; envelope-from=ahmedkhaledkaraman@gmail.com; helo=mail-wr1-x433.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ahmed Karaman Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" A nightly performance testing system to monitor any change in QEMU performance across seventeen different targets. The system includes eight different benchmarks to provide a variety of testing workloads. dijkstra_double: Find the shortest path between the source node and all other nodes using Dijkstra’s algorithm. The graph contains n nodes where all nxn distances are double values. The value of n can be specified using the -n flag. The default value is 2000. dijkstra_int32: Find the shortest path between the source node and all other nodes using Dijkstra’s algorithm. The graph contains n nodes where all nxn distances are int32 values. The value of n can be specified using the -n flag. The default value is 2000. matmult_double: Standard matrix multiplication of an n*n matrix of randomly generated double numbers from 0 to 100. The value of n is passed as an argument with the -n flag. The default value is 200. matmult_int32: Standard matrix multiplication of an n*n matrix of randomly generated integer numbers from 0 to 100. The value of n is passed as an argument with the -n flag. The default value is 200. qsort_double: Quick sort of an array of n randomly generated double numbers from 0 to 1000. The value of n is passed as an argument with the -n flag. The default value is 300000. qsort_int32: Quick sort of an array of n randomly generated integer numbers from 0 to 50000000. The value of n is passed as an argument with the -n flag.The default value is 300000. qsort_string: Quick sort of an array of 10000 randomly generated strings of size 8 (including null terminating character). The sort process is repeated n number of times. The value of n is passed as an argument with the -n flag. The default value is 20. search_string: Search for the occurrence of a small string in a much larger random string (“needle in a hay”). The search process is repeated n number of times and each time, a different large random string (“hay”) is generated. The value of n can be specified using the -n flag. The default value is 20. Syntax: nightly_tests_core.py [-h] [-r REF] Optional arguments: -h, --help Show this help message and exit -r REF, --reference REF Reference QEMU version - Default is v5.1.0 Example of usage: nightly_tests_core.py -r v5.1.0 2>log.txt The following report includes detailed setup and execution details of the system: https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/QEMU-Nightly-Performance-Tests/ Signed-off-by: Ahmed Karaman --- tests/performance/nightly-tests/README.md | 243 +++++ .../source/dijkstra_double/dijkstra_double.c | 194 ++++ .../source/dijkstra_int32/dijkstra_int32.c | 192 ++++ .../source/matmult_double/matmult_double.c | 123 +++ .../source/matmult_int32/matmult_int32.c | 121 +++ .../source/qsort_double/qsort_double.c | 104 ++ .../source/qsort_int32/qsort_int32.c | 103 ++ .../source/qsort_string/qsort_string.c | 122 +++ .../source/search_string/search_string.c | 110 +++ .../scripts/nightly_tests_core.py | 920 ++++++++++++++++++ .../scripts/run_nightly_tests.py | 135 +++ .../nightly-tests/scripts/send_email.py | 56 ++ 12 files changed, 2423 insertions(+) create mode 100644 tests/performance/nightly-tests/README.md create mode 100644 tests/performance/nightly-tests/benchmarks/source/dijkstra_double/dijkstra_double.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/dijkstra_int32/dijkstra_int32.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/matmult_double/matmult_double.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/matmult_int32/matmult_int32.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_double/qsort_double.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_int32/qsort_int32.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_string/qsort_string.c create mode 100644 tests/performance/nightly-tests/benchmarks/source/search_string/search_string.c create mode 100755 tests/performance/nightly-tests/scripts/nightly_tests_core.py create mode 100755 tests/performance/nightly-tests/scripts/run_nightly_tests.py create mode 100644 tests/performance/nightly-tests/scripts/send_email.py diff --git a/tests/performance/nightly-tests/README.md b/tests/performance/nightly-tests/README.md new file mode 100644 index 0000000000..6db3b351b3 --- /dev/null +++ b/tests/performance/nightly-tests/README.md @@ -0,0 +1,243 @@ +### QEMU Nightly Tests + +**Required settings:** + +Update the `GMAIL_USER` object in `send_email.py` with your credentials. + +For more details on how the system works, please check the [eighth report](https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/QEMU-Nightly-Performance-Tests/) of the "TCG Continuos Benchmarking" series. + +**Running the System:** + +The default reference version is v5.1.0. To specify a custom version, please use the `-r, --reference` flag. + +```bash +./run_nightly_tests.py +``` + +**Output:** + +``` +Host CPU : Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz +Host Memory : 15.49 GB + +Start Time (UTC) : 2020-08-25 21:30:01 +End Time (UTC) : 2020-08-25 22:02:37 +Execution Time : 0:32:35.896990 + +Status : SUCCESS + +Note: +Changes denoted by '-----' are less than 0.01%. + +-------------------------------------------------------- + SUMMARY REPORT - COMMIT d1a2b51f +-------------------------------------------------------- + AVERAGE RESULTS +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 2 158 355 274 ----- +1.693% +alpha 1 914 967 171 ----- +3.524% +arm 8 076 402 940 ----- +2.304% +hppa 4 261 685 987 -0.182% +3.164% +m68k 2 690 273 044 ----- +7.131% +mips 1 862 033 667 ----- +2.494% +mipsel 2 008 211 069 ----- +2.674% +mips64 1 918 635 565 ----- +2.818% +mips64el 2 051 565 677 ----- +3.026% +ppc 2 480 141 217 ----- +3.107% +ppc64 2 576 713 959 ----- +3.143% +ppc64le 2 558 853 539 ----- +3.173% +riscv64 1 406 704 050 ----- +2.65% +s390x 3 158 140 046 ----- +3.118% +sh4 2 364 449 748 ----- +3.33% +sparc64 3 318 544 783 ----- +3.851% +x86_64 1 775 844 158 ----- +2.156% +-------------------------------------------------------- + + DETAILED RESULTS +-------------------------------------------------------- +Test Program: dijkstra_double +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 3 062 583 464 ----- +1.424% +alpha 3 191 864 698 ----- +3.696% +arm 16 357 157 526 ----- +2.347% +hppa 7 228 376 315 -0.139% +3.086% +m68k 4 294 016 587 ----- +9.692% +mips 3 051 419 166 ----- +2.427% +mipsel 3 231 509 618 ----- +2.869% +mips64 3 245 837 754 ----- +2.596% +mips64el 3 414 195 796 ----- +3.021% +ppc 4 914 520 972 -0.041% +4.74% +ppc64 5 098 154 311 ----- +4.565% +ppc64le 5 082 419 054 ----- +4.58% +riscv64 2 192 294 915 ----- +1.955% +s390x 4 584 503 977 ----- +2.896% +sh4 3 949 036 447 ----- +3.464% +sparc64 4 586 203 546 ----- +4.237% +x86_64 2 484 092 105 ----- +1.75% +-------------------------------------------------------- +-------------------------------------------------------- +Test Program: dijkstra_int32 +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 2 210 194 577 ----- +1.493% +alpha 1 494 133 274 ----- +2.15% +arm 8 262 935 967 ----- +2.665% +hppa 5 207 318 306 ----- +3.047% +m68k 1 725 856 962 ----- +2.527% +mips 1 495 227 032 ----- +1.492% +mipsel 1 497 147 869 ----- +1.479% +mips64 1 715 388 570 ----- +1.892% +mips64el 1 695 276 864 ----- +1.913% +ppc 2 014 557 389 ----- +1.819% +ppc64 2 206 267 901 ----- +2.139% +ppc64le 2 197 998 781 ----- +2.146% +riscv64 1 354 912 745 ----- +2.396% +s390x 2 916 247 062 ----- +1.241% +sh4 1 990 532 533 ----- +2.669% +sparc64 2 872 231 051 ----- +3.758% +x86_64 1 553 981 241 ----- +2.12% +-------------------------------------------------------- +-------------------------------------------------------- +Test Program: matmult_double +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 1 412 273 223 ----- +0.302% +alpha 3 233 991 649 ----- +7.473% +arm 8 545 173 979 ----- +1.088% +hppa 3 483 597 802 -1.267% +4.468% +m68k 3 919 065 529 ----- +18.431% +mips 2 344 774 894 ----- +4.091% +mipsel 3 329 886 464 ----- +5.177% +mips64 2 359 046 988 ----- +4.076% +mips64el 3 343 664 785 ----- +5.167% +ppc 3 209 457 051 ----- +3.246% +ppc64 3 287 503 981 ----- +3.173% +ppc64le 3 287 189 065 ----- +3.173% +riscv64 1 221 603 682 ----- +0.277% +s390x 2 874 199 923 ----- +5.827% +sh4 3 543 943 634 ----- +6.416% +sparc64 3 426 150 004 ----- +7.139% +x86_64 1 248 917 276 ----- +0.322% +-------------------------------------------------------- +-------------------------------------------------------- +Test Program: matmult_int32 +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 598 681 621 ----- +0.585% +alpha 372 437 418 ----- +0.677% +arm 746 583 193 ----- +1.462% +hppa 674 278 359 ----- +1.183% +m68k 410 495 553 ----- +0.9% +mips 499 698 837 ----- +0.531% +mipsel 499 500 429 ----- +0.497% +mips64 481 554 664 ----- +0.599% +mips64el 465 057 054 ----- +0.619% +ppc 341 334 603 ----- +0.944% +ppc64 393 796 203 ----- +0.966% +ppc64le 393 977 298 ----- +0.965% +riscv64 351 709 769 ----- +0.785% +s390x 494 427 384 ----- +0.599% +sh4 402 668 444 ----- +0.899% +sparc64 495 952 959 ----- +1.192% +x86_64 402 928 461 ----- +0.833% +-------------------------------------------------------- +-------------------------------------------------------- +Test Program: qsort_double +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 2 709 683 624 ----- +2.417% +alpha 1 969 460 172 ----- +3.68% +arm 8 322 998 390 ----- +2.587% +hppa 3 188 301 995 -0.047% +2.9% +m68k 4 953 930 065 ----- +15.153% +mips 2 123 919 587 ----- +3.055% +mipsel 2 124 212 187 ----- +3.048% +mips64 1 999 047 826 ----- +3.405% +mips64el 1 996 426 772 ----- +3.409% +ppc 2 819 267 902 -0.021% +5.435% +ppc64 2 768 186 548 ----- +5.513% +ppc64le 2 724 803 772 -0.011% +5.603% +riscv64 1 638 328 937 ----- +4.021% +s390x 2 519 081 708 ----- +3.362% +sh4 2 595 545 154 ----- +2.994% +sparc64 3 988 986 587 ----- +2.747% +x86_64 2 033 468 588 ----- +3.234% +-------------------------------------------------------- +-------------------------------------------------------- +Test Program: qsort_int32 +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 2 193 392 565 ----- +2.916% +alpha 1 521 291 933 ----- +4.193% +arm 3 465 445 043 ----- +2.756% +hppa 2 280 034 340 ----- +3.821% +m68k 1 843 189 041 ----- +3.583% +mips 1 558 024 873 ----- +3.863% +mipsel 1 560 583 980 ----- +3.846% +mips64 1 563 415 749 ----- +4.412% +mips64el 1 542 677 320 ----- +4.474% +ppc 1 728 698 880 ----- +3.665% +ppc64 1 842 444 545 ----- +3.555% +ppc64le 1 791 822 067 ----- +3.661% +riscv64 1 348 866 430 ----- +4.654% +s390x 2 184 073 151 ----- +3.319% +sh4 1 946 492 539 ----- +3.624% +sparc64 3 452 215 585 ----- +2.937% +x86_64 1 813 544 414 ----- +3.537% +-------------------------------------------------------- +-------------------------------------------------------- +Test Program: qsort_string +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 2 592 218 418 ----- +2.468% +alpha 1 855 834 626 ----- +3.487% +arm 7 347 721 165 ----- +2.682% +hppa 4 758 753 926 ----- +3.543% +m68k 2 376 811 462 ----- +3.567% +mips 2 166 608 045 ----- +2.532% +mipsel 2 163 392 541 ----- +2.528% +mips64 2 029 251 969 ----- +3.117% +mips64el 2 011 628 621 ----- +3.145% +ppc 2 492 942 463 ----- +2.673% +ppc64 2 464 702 554 ----- +2.488% +ppc64le 2 445 253 307 ----- +2.505% +riscv64 1 625 053 328 ----- +3.953% +s390x 4 194 608 798 ----- +6.623% +sh4 2 164 142 539 ----- +3.166% +sparc64 4 299 516 539 ----- +4.065% +x86_64 2 940 456 780 ----- +2.649% +-------------------------------------------------------- +-------------------------------------------------------- +Test Program: search_string +-------------------------------------------------------- +Target Instructions Latest v5.1.0 +---------- -------------------- ---------- ---------- +aarch64 2 487 814 704 ----- +1.94% +alpha 1 680 723 605 ----- +2.835% +arm 11 563 208 260 ----- +2.848% +hppa 7 272 826 858 ----- +3.263% +m68k 1 998 819 159 ----- +3.198% +mips 1 656 596 909 ----- +1.959% +mipsel 1 659 455 464 ----- +1.947% +mips64 1 955 541 001 ----- +2.447% +mips64el 1 943 598 207 ----- +2.462% +ppc 2 320 350 477 ----- +2.332% +ppc64 2 552 655 634 ----- +2.742% +ppc64le 2 547 364 971 ----- +2.748% +riscv64 1 520 862 601 ----- +3.159% +s390x 5 497 978 370 ----- +1.078% +sh4 2 323 236 696 ----- +3.41% +sparc64 3 427 101 999 ----- +4.73% +x86_64 1 729 364 402 ----- +2.806% +-------------------------------------------------------- +``` diff --git a/tests/performance/nightly-tests/benchmarks/source/dijkstra_double/dijkstra_double.c b/tests/performance/nightly-tests/benchmarks/source/dijkstra_double/dijkstra_double.c new file mode 100644 index 0000000000..9c0bb804ac --- /dev/null +++ b/tests/performance/nightly-tests/benchmarks/source/dijkstra_double/dijkstra_double.c @@ -0,0 +1,194 @@ +/* + * Source file of a benchmark program involving calculations of the + * shortest distances between a source node and all other nodes in a + * graph of n nodes in which all nxn distances are defined as "double". + * The number n can be given via command line, and the default is 2000. + * The algorithm used is Dijsktra's. + * + * This file is a part of the project "TCG Continuous Benchmarking". + * + * Copyright (C) 2020 Ahmed Karaman + * Copyright (C) 2020 Aleksandar Markovic + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + * + */ + +#include +#include +#include +#include +#include +#include + +/* Number of columns and rows in all matrixes*/ +#define DEFAULT_NODE_COUNT 2000 +#define MIN_NODE_COUNT 3 +#define MAX_NODE_COUNT 10000 + + +int32_t closest_index(int32_t count, double *distances, bool *flags) +{ + int32_t closest; + double minimum = DBL_MAX; + + for (size_t i = 0; i < count; i++) { + if (flags[i] == false && distances[i] <= minimum) { + closest = i; + minimum = distances[i]; + } + } + + return closest; +} + +/** + * Calculate the shortest distances from the source node using Dijkstra method. + * @param (out) distances An array of shortest distances from the source node. + * @param (out) via An array of nodes needed to be taken as the the last + * before destination, for each destination. + * @param (out) eccent Eccentricity of the source node. + * @param (in) count The number of nodes. + * @param (in) source Source node. + * @param (in) matrix Distance matrix. + */ +void find_shortest_distances(double *distances, int32_t *via, double *eccent, + int32_t count, int32_t source, double **matrix) +{ + bool *flags; + + flags = (bool *)malloc(count * sizeof(bool)); + + for (size_t i = 0; i < count; i++) { + distances[i] = DBL_MAX; + flags[i] = false; + } + + distances[source] = 0.0; + via[source] = source; + + for (size_t i = 0; i < count - 1; i++) { + int32_t closest = closest_index(count, distances, flags); + flags[closest] = true; + for (size_t j = 0; j < count; j++) { + if ((!flags[j]) && + (matrix[closest][j]) && + (distances[closest] != DBL_MAX) && + (distances[j] > distances[closest] + matrix[closest][j])) { + distances[j] = distances[closest] + matrix[closest][j]; + via[j] = closest; + } + } + } + + *eccent = 0; + for (size_t i = 0; i < count; i++) { + if (*eccent < distances[i]) { + *eccent = distances[i]; + } + } + + free(flags); +} + + +void main(int argc, char *argv[]) +{ + double **distance_matrix; + double *shortest_distances; + int32_t *via_node; + int32_t node_count = DEFAULT_NODE_COUNT; + int32_t source_node = 0; + double node_eccentricity = 0.0; + double range_factor = 999.0 / (double)(RAND_MAX); + int32_t option; + + /* Parse command line options */ + while ((option = getopt(argc, argv, "n:")) != -1) { + if (option == 'n') { + int32_t user_node_count = atoi(optarg); + + /* Check if the value is a string or zero */ + if (user_node_count == 0) { + fprintf(stderr, "Error ... Invalid value for option '-n'.\n"); + exit(EXIT_FAILURE); + } + /* Check if the value is a negative number */ + if (user_node_count < MIN_NODE_COUNT) { + fprintf(stderr, "Error ... Value for option '-n' cannot be a " + "number less than %d.\n", MIN_NODE_COUNT); + exit(EXIT_FAILURE); + } + /* Check if the value is too large */ + if (user_node_count > MAX_NODE_COUNT) { + fprintf(stderr, "Error ... Value for option '-n' cannot be " + "more than %d.\n", MAX_NODE_COUNT); + exit(EXIT_FAILURE); + } + node_count = user_node_count; + } else { + exit(EXIT_FAILURE); + } + } + + /* Allocate the memory space for all matrixes */ + distance_matrix = (double **)malloc(node_count * sizeof(double *)); + for (size_t i = 0; i < node_count; i++) { + distance_matrix[i] = (double *)malloc(node_count * sizeof(double)); + } + shortest_distances = (double *)malloc(node_count * sizeof(double)); + via_node = (int32_t *)malloc(node_count * sizeof(int32_t)); + + /* Initialize helper arrays and populate distance_matrix */ + srand(1); + for (size_t i = 0; i < node_count; i++) { + shortest_distances[i] = 0.0; + via_node[i] = -1; + distance_matrix[i][i] = 0.0; + } + for (size_t i = 0; i < node_count; i++) { + for (size_t j = i + 1; j < node_count; j++) { + distance_matrix[i][j] = 1.0 + range_factor * (double)rand(); + distance_matrix[j][i] = distance_matrix[i][j]; + } + } + + find_shortest_distances(shortest_distances, via_node, &node_eccentricity, + node_count, source_node, distance_matrix); + + /* Control printing */ + printf("CONTROL RESULT:\n"); + printf(" Distance matrix (top left part):\n"); + for (size_t i = 0; i < 3; i++) { + for (size_t j = 0; j < 3; j++) { + printf(" %7.2f", distance_matrix[i][j]); + } + printf("\n"); + } + printf(" Source: %d (eccentricity: %f)\n", + source_node, node_eccentricity); + printf(" Destination Distance Via Node\n"); + for (size_t i = 0; i < 3; i++) { + printf(" %5d %7.2f %4d\n", + i, shortest_distances[i], via_node[i]); + } + + /* Free all previously allocated space */ + for (size_t i = 0; i < node_count; i++) { + free(distance_matrix[i]); + } + free(distance_matrix); + free(shortest_distances); + free(via_node); +} diff --git a/tests/performance/nightly-tests/benchmarks/source/dijkstra_int32/dijkstra_int32.c b/tests/performance/nightly-tests/benchmarks/source/dijkstra_int32/dijkstra_int32.c new file mode 100644 index 0000000000..2663cde943 --- /dev/null +++ b/tests/performance/nightly-tests/benchmarks/source/dijkstra_int32/dijkstra_int32.c @@ -0,0 +1,192 @@ +/* + * Source file of a benchmark program involving calculations of the + * shortest distances between a source node and all other nodes in a + * graph of n nodes in which all nxn distances are defined as "int32". + * The number n can be given via command line, and the default is 2000. + * The algorithm used is Dijsktra's. + * + * This file is a part of the project "TCG Continuous Benchmarking". + * + * Copyright (C) 2020 Ahmed Karaman + * Copyright (C) 2020 Aleksandar Markovic + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + * + */ + +#include +#include +#include +#include +#include + +/* Number of columns and rows in all matrixes*/ +#define DEFAULT_NODE_COUNT 2000 +#define MIN_NODE_COUNT 3 +#define MAX_NODE_COUNT 10000 + + +int32_t closest_index(int32_t count, int32_t *distances, bool *flags) +{ + int32_t closest; + int32_t minimum = INT_MAX; + + for (size_t i = 0; i < count; i++) { + if (flags[i] == false && distances[i] <= minimum) { + closest = i; + minimum = distances[i]; + } + } + + return closest; +} + +/** + * Calculate the shortest distances from the source node using Dijkstra method. + * @param (out) distances An array of shortest distances from the source node. + * @param (out) via An array of nodes needed to be taken as the the last + * before destination, for each destination. + * @param (out) eccent Eccentricity of the source node. + * @param (in) count The number of nodes. + * @param (in) source Source node. + * @param (in) matrix Distance matrix. + */ +void find_shortest_distances(int32_t *distances, int32_t *via, int32_t *eccent, + int32_t count, int32_t source, int32_t **matrix) +{ + bool *flags; + + flags = (bool *)malloc(count * sizeof(bool)); + + for (size_t i = 0; i < count; i++) { + distances[i] = INT_MAX; + flags[i] = false; + } + + distances[source] = 0; + via[source] = source; + + for (size_t i = 0; i < count - 1; i++) { + int32_t closest = closest_index(count, distances, flags); + flags[closest] = true; + for (size_t j = 0; j < count; j++) { + if ((!flags[j]) && + (matrix[closest][j]) && + (distances[closest] != INT_MAX) && + (distances[j] > distances[closest] + matrix[closest][j])) { + distances[j] = distances[closest] + matrix[closest][j]; + via[j] = closest; + } + } + } + + *eccent = 0; + for (size_t i = 0; i < count; i++) { + if (*eccent < distances[i]) { + *eccent = distances[i]; + } + } + + free(flags); +} + + +void main(int argc, char *argv[]) +{ + int32_t **distance_matrix; + int32_t *shortest_distances; + int32_t *via_node; + int32_t node_count = DEFAULT_NODE_COUNT; + int32_t source_node = 0; + int32_t node_eccentricity = 0; + int32_t option; + + /* Parse command line options */ + while ((option = getopt(argc, argv, "n:")) != -1) { + if (option == 'n') { + int32_t user_node_count = atoi(optarg); + + /* Check if the value is a string or zero */ + if (user_node_count == 0) { + fprintf(stderr, "Error ... Invalid value for option '-n'.\n"); + exit(EXIT_FAILURE); + } + /* Check if the value is a negative number */ + if (user_node_count < MIN_NODE_COUNT) { + fprintf(stderr, "Error ... Value for option '-n' cannot be a " + "number less than %d.\n", MIN_NODE_COUNT); + exit(EXIT_FAILURE); + } + /* Check if the value is too large */ + if (user_node_count > MAX_NODE_COUNT) { + fprintf(stderr, "Error ... Value for option '-n' cannot be " + "more than %d.\n", MAX_NODE_COUNT); + exit(EXIT_FAILURE); + } + node_count = user_node_count; + } else { + exit(EXIT_FAILURE); + } + } + + /* Allocate the memory space for all matrixes */ + distance_matrix = (int32_t **)malloc(node_count * sizeof(int32_t *)); + for (size_t i = 0; i < node_count; i++) { + distance_matrix[i] = (int32_t *)malloc(node_count * sizeof(int32_t)); + } + shortest_distances = (int32_t *)malloc(node_count * sizeof(int32_t)); + via_node = (int32_t *)malloc(node_count * sizeof(int32_t)); + + /* Initialize helper arrays and populate distance_matrix */ + srand(1); + for (size_t i = 0; i < node_count; i++) { + shortest_distances[i] = 0; + via_node[i] = -1; + distance_matrix[i][i] = 0; + } + for (size_t i = 0; i < node_count; i++) { + for (size_t j = i + 1; j < node_count; j++) { + distance_matrix[i][j] = 1 + (rand()) / (RAND_MAX / 999); + distance_matrix[j][i] = distance_matrix[i][j]; + } + } + + find_shortest_distances(shortest_distances, via_node, &node_eccentricity, + node_count, source_node, distance_matrix); + + /* Control printing */ + printf("CONTROL RESULT:\n"); + printf(" Distance matrix (top left part):\n"); + for (size_t i = 0; i < 3; i++) { + for (size_t j = 0; j < 3; j++) { + printf(" %6d", distance_matrix[i][j]); + } + printf("\n"); + } + printf(" Source: %d (eccentricity: %d)\n", + source_node, node_eccentricity); + printf(" Destination Distance Via Node\n"); + for (size_t i = 0; i < 3; i++) { + printf(" %5d %3d %4d\n", + i, shortest_distances[i], via_node[i]); + } + + /* Free all previously allocated space */ + for (size_t i = 0; i < node_count; i++) { + free(distance_matrix[i]); + } + free(distance_matrix); + free(shortest_distances); + free(via_node); +} diff --git a/tests/performance/nightly-tests/benchmarks/source/matmult_double/matmult_double.c b/tests/performance/nightly-tests/benchmarks/source/matmult_double/matmult_double.c new file mode 100644 index 0000000000..42bbb4717a --- /dev/null +++ b/tests/performance/nightly-tests/benchmarks/source/matmult_double/matmult_double.c @@ -0,0 +1,123 @@ +/* + * Source file of a benchmark program involving calculations of + * a product of two matrixes nxn whose elements are "double". The + * number n can be given via command line, and the default is 200. + * + * This file is a part of the project "TCG Continuous Benchmarking". + * + * Copyright (C) 2020 Ahmed Karaman + * Copyright (C) 2020 Aleksandar Markovic + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + * + */ + +#include +#include +#include + +/* Number of columns and rows in all matrixes*/ +#define DEFAULT_MATRIX_SIZE 200 +#define MIN_MATRIX_SIZE 2 +#define MAX_MATRIX_SIZE 200000 + +void main(int argc, char *argv[]) +{ + double **matrix_a; + double **matrix_b; + double **matrix_res; + size_t i; + size_t j; + size_t k; + int32_t matrix_size = DEFAULT_MATRIX_SIZE; + int32_t option; + double range_factor = 100.0 / (double)(RAND_MAX); + + + /* Parse command line options */ + while ((option = getopt(argc, argv, "n:")) != -1) { + if (option == 'n') { + int32_t user_matrix_size = atoi(optarg); + + /* Check if the value is a string or zero */ + if (user_matrix_size == 0) { + fprintf(stderr, "Error ... Invalid value for option '-n'.\n"); + exit(EXIT_FAILURE); + } + /* Check if the value is a negative number */ + if (user_matrix_size < MIN_MATRIX_SIZE) { + fprintf(stderr, "Error ... Value for option '-n' cannot be a " + "number less than %d.\n", MIN_MATRIX_SIZE); + exit(EXIT_FAILURE); + } + /* Check if the value is too large */ + if (user_matrix_size > MAX_MATRIX_SIZE) { + fprintf(stderr, "Error ... Value for option '-n' cannot be " + "more than %d.\n", MAX_MATRIX_SIZE); + exit(EXIT_FAILURE); + } + matrix_size = user_matrix_size; + } else { + exit(EXIT_FAILURE); + } + } + + /* Allocate the memory space for all matrixes */ + matrix_a = (double **)malloc(matrix_size * sizeof(double *)); + for (i = 0; i < matrix_size; i++) { + matrix_a[i] = (double *)malloc(matrix_size * sizeof(double)); + } + matrix_b = (double **)malloc(matrix_size * sizeof(double *)); + for (i = 0; i < matrix_size; i++) { + matrix_b[i] = (double *)malloc(matrix_size * sizeof(double)); + } + matrix_res = (double **)malloc(matrix_size * sizeof(double *)); + for (i = 0; i < matrix_size; i++) { + matrix_res[i] = (double *)malloc(matrix_size * sizeof(double)); + } + + /* Populate matrix_a and matrix_b with random numbers */ + srand(1); + for (i = 0; i < matrix_size; i++) { + for (j = 0; j < matrix_size; j++) { + matrix_a[i][j] = range_factor * (double)rand(); + matrix_b[i][j] = range_factor * (double)rand(); + } + } + + /* Calculate the product of two matrixes */ + for (i = 0; i < matrix_size; i++) { + for (j = 0; j < matrix_size; j++) { + matrix_res[i][j] = 0.0; + for (k = 0; k < matrix_size; k++) { + matrix_res[i][j] += matrix_a[i][k] * matrix_b[k][j]; + } + } + } + + /* Control printing */ + printf("CONTROL RESULT:\n"); + printf(" %f %f\n", matrix_res[0][0], matrix_res[0][1]); + printf(" %f %f\n", matrix_res[1][0], matrix_res[1][1]); + + /* Free all previously allocated space */ + for (i = 0; i < matrix_size; i++) { + free(matrix_a[i]); + free(matrix_b[i]); + free(matrix_res[i]); + } + free(matrix_a); + free(matrix_b); + free(matrix_res); +} diff --git a/tests/performance/nightly-tests/benchmarks/source/matmult_int32/matmult_int32.c b/tests/performance/nightly-tests/benchmarks/source/matmult_int32/matmult_int32.c new file mode 100644 index 0000000000..29a6eb000d --- /dev/null +++ b/tests/performance/nightly-tests/benchmarks/source/matmult_int32/matmult_int32.c @@ -0,0 +1,121 @@ +/* + * Source file of a benchmark program involving calculations of + * a product of two matrixes nxn whose elements are "int32_t". The + * number n can be given via command line, and the default is 200. + * + * This file is a part of the project "TCG Continuous Benchmarking". + * + * Copyright (C) 2020 Ahmed Karaman + * Copyright (C) 2020 Aleksandar Markovic + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + * + */ + +#include +#include +#include + +/* Number of columns and rows in all matrixes*/ +#define DEFAULT_MATRIX_SIZE 200 +#define MIN_MATRIX_SIZE 2 +#define MAX_MATRIX_SIZE 200000 + +void main(int argc, char *argv[]) +{ + int32_t **matrix_a; + int32_t **matrix_b; + int32_t **matrix_res; + size_t i; + size_t j; + size_t k; + int32_t matrix_size = DEFAULT_MATRIX_SIZE; + int32_t option; + + /* Parse command line options */ + while ((option = getopt(argc, argv, "n:")) != -1) { + if (option == 'n') { + int32_t user_matrix_size = atoi(optarg); + + /* Check if the value is a string or zero */ + if (user_matrix_size == 0) { + fprintf(stderr, "Error ... Invalid value for option '-n'.\n"); + exit(EXIT_FAILURE); + } + /* Check if the value is a negative number */ + if (user_matrix_size < MIN_MATRIX_SIZE) { + fprintf(stderr, "Error ... Value for option '-n' cannot be a " + "number less than %d.\n", MIN_MATRIX_SIZE); + exit(EXIT_FAILURE); + } + /* Check if the value is too large */ + if (user_matrix_size > MAX_MATRIX_SIZE) { + fprintf(stderr, "Error ... Value for option '-n' cannot be " + "more than %d.\n", MAX_MATRIX_SIZE); + exit(EXIT_FAILURE); + } + matrix_size = user_matrix_size; + } else { + exit(EXIT_FAILURE); + } + } + + /* Allocate the memory space for all matrixes */ + matrix_a = (int32_t **)malloc(matrix_size * sizeof(int32_t *)); + for (i = 0; i < matrix_size; i++) { + matrix_a[i] = (int32_t *)malloc(matrix_size * sizeof(int32_t)); + } + matrix_b = (int32_t **)malloc(matrix_size * sizeof(int32_t *)); + for (i = 0; i < matrix_size; i++) { + matrix_b[i] = (int32_t *)malloc(matrix_size * sizeof(int32_t)); + } + matrix_res = (int32_t **)malloc(matrix_size * sizeof(int32_t *)); + for (i = 0; i < matrix_size; i++) { + matrix_res[i] = (int32_t *)malloc(matrix_size * sizeof(int32_t)); + } + + /* Populate matrix_a and matrix_b with random numbers */ + srand(1); + for (i = 0; i < matrix_size; i++) { + for (j = 0; j < matrix_size; j++) { + matrix_a[i][j] = (rand()) / (RAND_MAX / 100); + matrix_b[i][j] = (rand()) / (RAND_MAX / 100); + } + } + + /* Calculate the product of two matrixes */ + for (i = 0; i < matrix_size; i++) { + for (j = 0; j < matrix_size; j++) { + matrix_res[i][j] = 0; + for (k = 0; k < matrix_size; k++) { + matrix_res[i][j] += matrix_a[i][k] * matrix_b[k][j]; + } + } + } + + /* Control printing */ + printf("CONTROL RESULT:\n"); + printf(" %d %d\n", matrix_res[0][0], matrix_res[0][1]); + printf(" %d %d\n", matrix_res[1][0], matrix_res[1][1]); + + /* Free all previously allocated space */ + for (i = 0; i < matrix_size; i++) { + free(matrix_a[i]); + free(matrix_b[i]); + free(matrix_res[i]); + } + free(matrix_a); + free(matrix_b); + free(matrix_res); +} diff --git a/tests/performance/nightly-tests/benchmarks/source/qsort_double/qsort_double.c b/tests/performance/nightly-tests/benchmarks/source/qsort_double/qsort_double.c new file mode 100644 index 0000000000..efc1b2eee1 --- /dev/null +++ b/tests/performance/nightly-tests/benchmarks/source/qsort_double/qsort_double.c @@ -0,0 +1,104 @@ +/* + * Source file of a benchmark program involving sorting of an array + * of length n whose elements are "double". The default value for n + * is 300000, and it can be set via command line as well. + * + * This file is a part of the project "TCG Continuous Benchmarking". + * + * Copyright (C) 2020 Ahmed Karaman + * Copyright (C) 2020 Aleksandar Markovic + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + * + */ + +#include +#include +#include + +/* Number of elements in the array to be sorted */ +#define DEFAULT_ARRAY_LEN 300000 +#define MIN_ARRAY_LEN 3 +#define MAX_ARRAY_LEN 30000000 + +/* Upper limit used for generation of random numbers */ +#define UPPER_LIMIT 1000.0 + +/* Comparison function passed to qsort() */ +static int compare(const void *a, const void *b) +{ + if (*(const double *)a > *(const double *)b) { + return 1; + } else if (*(const double *)a < *(const double *)b) { + return -1; + } + return 0; +} + +void main(int argc, char *argv[]) +{ + double *array_to_be_sorted; + int32_t array_len = DEFAULT_ARRAY_LEN; + int32_t option; + double range_factor = UPPER_LIMIT / (double)(RAND_MAX); + + /* Parse command line options */ + while ((option = getopt(argc, argv, "n:")) != -1) { + if (option == 'n') { + int32_t user_array_len = atoi(optarg); + + /* Check if the value is a string or zero */ + if (user_array_len == 0) { + fprintf(stderr, "Error ... Invalid value for option '-n'.\n"); + exit(EXIT_FAILURE); + } + /* Check if the value is a negative number */ + if (user_array_len < MIN_ARRAY_LEN) { + fprintf(stderr, "Error ... Value for option '-n' cannot be a " + "number less than %d.\n", MIN_ARRAY_LEN); + exit(EXIT_FAILURE); + } + /* Check if the value is too large */ + if (user_array_len > MAX_ARRAY_LEN) { + fprintf(stderr, "Error ... Value for option '-n' cannot be " + "more than %d.\n", MAX_ARRAY_LEN); + exit(EXIT_FAILURE); + } + array_len = user_array_len; + } else { + exit(EXIT_FAILURE); + } + } + + /* Allocate the memory space for the array */ + array_to_be_sorted = (double *) malloc(array_len * sizeof(double)); + + /* Populate the_array with random numbers */ + srand(1); + for (size_t i = 0; i < array_len; i++) { + array_to_be_sorted[i] = range_factor * (double)rand(); + } + + /* Sort the_array using qsort() */ + qsort(array_to_be_sorted, array_len, sizeof(array_to_be_sorted[0]), + compare); + + /* Control printing */ + printf("CONTROL RESULT:\n"); + printf("%14.10f %14.10f %14.10f\n", + array_to_be_sorted[0], array_to_be_sorted[1], array_to_be_sorted[2]); + + /* Free all previously allocated space */ + free(array_to_be_sorted); +} diff --git a/tests/performance/nightly-tests/benchmarks/source/qsort_int32/qsort_int32.c b/tests/performance/nightly-tests/benchmarks/source/qsort_int32/qsort_int32.c new file mode 100644 index 0000000000..76ca9c3490 --- /dev/null +++ b/tests/performance/nightly-tests/benchmarks/source/qsort_int32/qsort_int32.c @@ -0,0 +1,103 @@ +/* + * Source file of a benchmark program involving sorting of an array + * of length n whose elements are "int32_t". The default value for n + * is 300000, and it can be set via command line as well. + * + * This file is a part of the project "TCG Continuous Benchmarking". + * + * Copyright (C) 2020 Ahmed Karaman + * Copyright (C) 2020 Aleksandar Markovic + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + * + */ + +#include +#include +#include + +/* Number of elements in the array to be sorted */ +#define DEFAULT_ARRAY_LEN 300000 +#define MIN_ARRAY_LEN 3 +#define MAX_ARRAY_LEN 30000000 + +/* Upper limit used for generation of random numbers */ +#define UPPER_LIMIT 50000000 + +/* Comparison function passed to qsort() */ +static int compare(const void *a, const void *b) +{ + if (*(const int32_t *)a > *(const int32_t *)b) { + return 1; + } else if (*(const int32_t *)a < *(const int32_t *)b) { + return -1; + } + return 0; +} + +void main(int argc, char *argv[]) +{ + int32_t *array_to_be_sorted; + int32_t array_len = DEFAULT_ARRAY_LEN; + int32_t option; + + /* Parse command line options */ + while ((option = getopt(argc, argv, "n:")) != -1) { + if (option == 'n') { + int32_t user_array_len = atoi(optarg); + + /* Check if the value is a string or zero */ + if (user_array_len == 0) { + fprintf(stderr, "Error ... Invalid value for option '-n'.\n"); + exit(EXIT_FAILURE); + } + /* Check if the value is a negative number */ + if (user_array_len < MIN_ARRAY_LEN) { + fprintf(stderr, "Error ... Value for option '-n' cannot be a " + "number less than %d.\n", MIN_ARRAY_LEN); + exit(EXIT_FAILURE); + } + /* Check if the value is too large */ + if (user_array_len > MAX_ARRAY_LEN) { + fprintf(stderr, "Error ... Value for option '-n' cannot be " + "more than %d.\n", MAX_ARRAY_LEN); + exit(EXIT_FAILURE); + } + array_len = user_array_len; + } else { + exit(EXIT_FAILURE); + } + } + + /* Allocate the memory space for the array */ + array_to_be_sorted = (int32_t *) malloc(array_len * sizeof(int32_t)); + + /* Populate the_array with random numbers */ + srand(1); + for (size_t i = 0; i < array_len; i++) { + array_to_be_sorted[i] = (rand()) / (RAND_MAX / UPPER_LIMIT); + } + + /* Sort the_array using qsort() */ + qsort(array_to_be_sorted, array_len, sizeof(array_to_be_sorted[0]), + compare); + + /* Control printing */ + printf("CONTROL RESULT:\n"); + printf("%d %d %d\n", + array_to_be_sorted[0], array_to_be_sorted[1], array_to_be_sorted[2]); + + /* Free all previously allocated space */ + free(array_to_be_sorted); +} diff --git a/tests/performance/nightly-tests/benchmarks/source/qsort_string/qsort_string.c b/tests/performance/nightly-tests/benchmarks/source/qsort_string/qsort_string.c new file mode 100644 index 0000000000..7d582b2dd0 --- /dev/null +++ b/tests/performance/nightly-tests/benchmarks/source/qsort_string/qsort_string.c @@ -0,0 +1,122 @@ +/* + * Source file of a benchmark program involving sorting of an array + * of 10000 random strings of length 8 (including terminating zero). + * That sorting is repeated a number of times (default is 20 times), + * and each time a different array of random strings is generated. + * The number of repetitions can be set via command line. + * + * This file is a part of the project "TCG Continuous Benchmarking". + * + * Copyright (C) 2020 Ahmed Karaman + * Copyright (C) 2020 Aleksandar Markovic + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + * + */ + +#include +#include +#include +#include + +/* Length of an individual random string (including terminating zero) */ +#define RANDOM_STRING_LEN 8 +/* Number of elements of the array of random strings */ +#define NUMBER_OF_RANDOM_STRINGS 10000 + +/* Number of repetitions to be performed each with different input */ +#define DEFAULT_REPETITION_COUNT 20 +#define MIN_REPETITION_COUNT 1 +#define MAX_REPETITION_COUNT 1000 + +/* Structure that keeps an array of random strings to be sorted */ +struct StringStruct { + char chars[RANDOM_STRING_LEN]; +}; + +/* Comparison function passed to qsort() */ +int compare_strings(const void *element1, const void *element2) +{ + int result; + + result = strcmp((*((struct StringStruct *)element1)).chars, + (*((struct StringStruct *)element2)).chars); + + return (result < 0) ? -1 : ((result == 0) ? 0 : 1); +} + +/* Generate a random string of given length and containing only small letters */ +static void gen_random_string(char *s, const int len) +{ + static const char letters[] = "abcdefghijklmnopqrstuvwxyz"; + + for (size_t i = 0; i < (len - 1); i++) { + s[i] = letters[rand() % (sizeof(letters) - 1)]; + } + + s[len - 1] = 0; +} + +void main(int argc, char *argv[]) +{ + struct StringStruct strings_to_be_sorted[NUMBER_OF_RANDOM_STRINGS]; + int32_t repetition_count = DEFAULT_REPETITION_COUNT; + int32_t option; + + /* Parse command line options */ + while ((option = getopt(argc, argv, "n:")) != -1) { + if (option == 'n') { + int32_t user_repetition_count = atoi(optarg); + + /* Check if the value is a string or zero */ + if (user_repetition_count == 0) { + fprintf(stderr, "Error ... Invalid value for option '-n'.\n"); + exit(EXIT_FAILURE); + } + /* Check if the value is a negative number */ + if (user_repetition_count < MIN_REPETITION_COUNT) { + fprintf(stderr, "Error ... Value for option '-n' cannot be a " + "number less than %d.\n", MIN_REPETITION_COUNT); + exit(EXIT_FAILURE); + } + /* Check if the value is too large */ + if (user_repetition_count > MAX_REPETITION_COUNT) { + fprintf(stderr, "Error ... Value for option '-n' cannot be " + "more than %d.\n", MAX_REPETITION_COUNT); + exit(EXIT_FAILURE); + } + repetition_count = user_repetition_count; + } else { + exit(EXIT_FAILURE); + } + } + + srand(1); + + for (size_t i = 0; i < repetition_count; ++i) { + /* Generate random strings, and, in turn, sort them */ + for (size_t i = 0; i < NUMBER_OF_RANDOM_STRINGS; ++i) { + gen_random_string(strings_to_be_sorted[i].chars, RANDOM_STRING_LEN); + } + qsort(strings_to_be_sorted, NUMBER_OF_RANDOM_STRINGS, + sizeof(struct StringStruct), compare_strings); + } + + /* Control printing */ + printf("CONTROL RESULT:\n"); + for (size_t i = 0; i < 2; ++i) { + printf(" %s", strings_to_be_sorted[i].chars); + } + printf("\n"); +} diff --git a/tests/performance/nightly-tests/benchmarks/source/search_string/search_string.c b/tests/performance/nightly-tests/benchmarks/source/search_string/search_string.c new file mode 100644 index 0000000000..2827ea032e --- /dev/null +++ b/tests/performance/nightly-tests/benchmarks/source/search_string/search_string.c @@ -0,0 +1,110 @@ +/* + * Source file of a benchmark program that searches for the occurrence + * of a small string in a much larger random string ("needle in a hay"). + * That searching is repeated a number of times (default is 20 times), + * and each time a different large random string ("hay") is generated. + * The number of repetitions can be set via command line. + * + * This file is a part of the project "TCG Continuous Benchmarking". + * + * Copyright (C) 2020 Ahmed Karaman + * Copyright (C) 2020 Aleksandar Markovic + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + * + */ + +#include +#include +#include +#include + +/* Length of a long string to be searched (including terminating zero) */ +#define HAYSTACK_LEN 30000 + +/* Number of repetitions to be performed each with different input */ +#define DEFAULT_REPETITION_COUNT 100 +#define MIN_REPETITION_COUNT 1 +#define MAX_REPETITION_COUNT 10000 + + +/* Generate a random string of given length and containing only small letters */ +static void gen_random_string(char *s, const int len) +{ + static const char letters[] = "abcdefghijklmnopqrstuvwxyz"; + + for (size_t i = 0; i < (len - 1); i++) { + s[i] = letters[rand() % (sizeof(letters) - 1)]; + } + + s[len - 1] = 0; +} + +void main(int argc, char *argv[]) +{ + char haystack[HAYSTACK_LEN]; + const char needle[] = "aaa "; + char *found_needle; + int32_t found_cnt = 0; + int32_t not_found_cnt = 0; + int32_t repetition_count = DEFAULT_REPETITION_COUNT; + int32_t option; + + printf("needle is %s, size %d\n", needle, sizeof(needle)); + + /* Parse command line options */ + while ((option = getopt(argc, argv, "n:")) != -1) { + if (option == 'n') { + int32_t user_repetition_count = atoi(optarg); + + /* Check if the value is a string or zero */ + if (user_repetition_count == 0) { + fprintf(stderr, "Error ... Invalid value for option '-n'.\n"); + exit(EXIT_FAILURE); + } + /* Check if the value is a negative number */ + if (user_repetition_count < MIN_REPETITION_COUNT) { + fprintf(stderr, "Error ... Value for option '-n' cannot be a " + "number less than %d.\n", MIN_REPETITION_COUNT); + exit(EXIT_FAILURE); + } + /* Check if the value is too large */ + if (user_repetition_count > MAX_REPETITION_COUNT) { + fprintf(stderr, "Error ... Value for option '-n' cannot be " + "more than %d.\n", MAX_REPETITION_COUNT); + exit(EXIT_FAILURE); + } + repetition_count = user_repetition_count; + } else { + exit(EXIT_FAILURE); + } + } + + srand(1); + + for (size_t i = 0; i < repetition_count; ++i) { + /* Generate random hay, and, in turn, find a needle */ + gen_random_string(haystack, HAYSTACK_LEN); + found_needle = strstr(haystack, needle); + if (found_needle != NULL) { + found_cnt++; + } else { + not_found_cnt++; + } + } + + /* Control printing */ + printf("CONTROL RESULT:\n"); + printf(" Found %d times. Not found %d times.\n", found_cnt, not_found_cnt); +} diff --git a/tests/performance/nightly-tests/scripts/nightly_tests_core.py b/tests/performance/nightly-tests/scripts/nightly_tests_core.py new file mode 100755 index 0000000000..da192c704a --- /dev/null +++ b/tests/performance/nightly-tests/scripts/nightly_tests_core.py @@ -0,0 +1,920 @@ +#!/usr/bin/env python3 + +""" +Core script for performing nightly performance tests on QEMU. + +This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman +Copyright (C) 2020 Aleksandar Markovic + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see . +""" + +import argparse +import csv +import datetime +import glob +import multiprocessing +import os +import pathlib +import shutil +import subprocess +import sys +import tempfile +import time +from typing import Dict, List, Optional, Union + + +def get_benchmark_name(benchmark_path: str) -> str: + """ + Return the benchmark name given its path. + + Parameters: + benchmarks_path (str): Absolute path to benchmark + + Return: + (str): Benchmark name + """ + benchmark_source_file = os.path.split(benchmark_path)[1] + return os.path.splitext(benchmark_source_file)[0] + + +def get_benchmark_parent_dir(benchmark_path: str) -> str: + """ + Return the benchmark parent directory name given the benchmark path. + + Parameters: + benchmarks_path (str): Absolute path to benchmark + + Return: + (str): Benchmark parent directory name + """ + benchmark_parent_dir_path = os.path.split(benchmark_path)[0] + benchmark_parent_dir = os.path.split(benchmark_parent_dir_path)[1] + + return benchmark_parent_dir + + +def get_executable_parent_dir_path( + benchmark_path: str, benchmarks_executables_dir_path: str) -> str: + """ + Return the executables parent directory of a benchmark given its path. + This is the directory that includes all compiled executables for the + benchmark. + + Parameters: + benchmarks_path (str): Absolute path to benchmark + benchmarks_executables_dir_path (str): Absolute path to the executables + + Return: + (str): Executables parent directory path + """ + benchmark_parent_dir_path = os.path.split(benchmark_path)[0] + benchmark_parent_dir = os.path.split(benchmark_parent_dir_path)[1] + executable_parent_dir_path = os.path.join(benchmarks_executables_dir_path, + benchmark_parent_dir) + + return executable_parent_dir_path + + +def get_commit_hash(commit_tag: str, qemu_path: str) -> str: + """ + Find commit hash given the Git commit tag. + + Parameters: + commit_tag (str): Commit tag + qemu_path (str): Absolute path to QEMU + + Returns: + (str): 8 digit commit hash + """ + + commit_hash = subprocess.run(["git", + "rev-parse", + commit_tag], + cwd=qemu_path, + stdout=subprocess.PIPE, + check=False) + if commit_hash.returncode: + clean_exit(qemu_path, + "Failed to find the commit hash of {}.".format(commit_tag)) + + return commit_hash.stdout.decode("utf-8")[:8] + + +def git_checkout(commit: str, qemu_path: str) -> None: + """ + Checkout a given Git commit. + Also pull the latest changes from origin/master if the commit is "master". + + Parameters: + commit (str): Commit hash or tag + qemu_path (str): Absolute path to QEMU + """ + print(datetime.datetime.utcnow().isoformat(), + "- Checking out {}".format(commit), file=sys.stderr, flush=True) + + checkout_commit = subprocess.run(["git", + "checkout", + commit], + cwd=qemu_path, + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + check=False) + if checkout_commit.returncode: + clean_exit(qemu_path, checkout_commit.stderr.decode("utf-8")) + + if commit == "master": + print(datetime.datetime.utcnow().isoformat(), + "- Pulling the latest changes from QEMU master", + file=sys.stderr, flush=True) + # Try pulling the latest changes. + # Limit the number of failed trials to 10. + failure_count, failure_limit = 0, 10 + while True: + pull_latest = subprocess.run(["git", + "pull", + "origin", + "master"], + cwd=qemu_path, + stdout=subprocess.DEVNULL, + check=False) + if pull_latest.returncode: + failure_count += 1 + if failure_count == failure_limit: + print(datetime.datetime.utcnow().isoformat(), + "- Trial {}/{}: Failed to pull QEMU".format( + failure_count, failure_limit), + file=sys.stderr, flush=True) + clean_exit(qemu_path, "") + else: + print(datetime.datetime.utcnow().isoformat(), + "- Trial {}/{}: Failed to pull QEMU" + " ... retrying again in a minute!".format( + failure_count, failure_limit), + file=sys.stderr, flush=True) + time.sleep(60) + else: + break + + +def git_clone(qemu_path: str) -> None: + """ + Clone QEMU from Git. + + Parameters: + qemu_path (str): Absolute path to clone the QEMU repo to + """ + # Try cloning QEMU. + # Limit the number of failed trials to 10. + failure_count, failure_limit = 0, 10 + while True: + clone_qemu = subprocess.run(["git", + "clone", + "https://git.qemu.org/git/qemu.git", + qemu_path], + check=False) + if clone_qemu.returncode: + failure_count += 1 + if failure_count == failure_limit: + print(datetime.datetime.utcnow().isoformat(), + "- Trial {}/{}: Failed to clone QEMU".format( + failure_count, failure_limit), + file=sys.stderr, flush=True) + clean_exit(qemu_path, "") + else: + print(datetime.datetime.utcnow().isoformat(), + "- Trial {}/{}: Failed to clone QEMU" + " ... retrying again in a minute!".format( + failure_count, failure_limit), + file=sys.stderr, flush=True) + time.sleep(60) + else: + break + + +def build_qemu(qemu_path: str, git_tag: str, targets: List[str]) -> None: + """ + Checkout the Git tag then configure and build QEMU. + + Parameters: + qemu_path (str): Absolute path to QEMU + git_tag (str): Git tag to checkout before building + targets (List[str]): List of targets to configure QEMU for + """ + + # Clean the QEMU build path + qemu_build_path = os.path.join(qemu_path, "build-gcc") + if os.path.isdir(qemu_build_path): + shutil.rmtree(qemu_build_path) + os.mkdir(qemu_build_path) + + git_checkout(git_tag, qemu_path) + + # Specify target list for configuring QEMU + target_list = ["{}-linux-user".format(target) for target in targets] + + # Configure QEMU + print(datetime.datetime.utcnow().isoformat(), + "- Running 'configure' for {}".format(git_tag), + file=sys.stderr, flush=True) + configure = subprocess.run(["../configure", + "--disable-system", + "--disable-tools", + "--target-list={}". + format(",".join(target_list))], + cwd=qemu_build_path, + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + check=False) + if configure.returncode: + clean_exit(qemu_path, configure.stderr.decode("utf-8")) + + # Run "make -j$(nproc)" + print(datetime.datetime.utcnow().isoformat(), + "- Running 'make' for {}".format(git_tag), file=sys.stderr, + flush=True) + make = subprocess.run(["make", + "-j", + str(multiprocessing.cpu_count())], + cwd=qemu_build_path, + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + check=False) + if make.returncode: + clean_exit(qemu_path, make.stderr.decode("utf-8")) + + +def compile_target(benchmark_path: str, compiled_benchmark_path: str, + target_compiler: str) -> None: + """ + Compile a benchmark using the provided cross compiler. + + Parameters: + benchmarks_path (str): Absolute path to benchmark + compiled_benchmark_path (str): Path to the output executable + target_compiler (str): Cross compiler + """ + compile_benchmark = subprocess.run([target_compiler, + "-O2", + "-static", + "-w", + benchmark_path, + "-o", + compiled_benchmark_path], + check=False) + if compile_benchmark.returncode: + sys.exit("Compilation of {} failed".format( + os.path.split(compiled_benchmark_path)[1])) + + +def measure_instructions( + benchmark_path: str, benchmarks_executables_dir_path: str, + qemu_path: str, targets: List[str]) -> List[List[Union[str, int]]]: + """ + Measure the number of instructions when running an program with QEMU. + + Parameters: + benchmarks_path (str): Absolute path to benchmark + benchmarks_executables_dir_path (str): Absolute path to the executables + qemu_path (str): Absolute path to QEMU + targets (List[str]): List of QEMU targets + + Returns: + (List[List[Union[str, int]]]): [[target_name, instructions],[...],...] + """ + + benchmark_name = get_benchmark_name(benchmark_path) + executable_parent_dir_path = get_executable_parent_dir_path( + benchmark_path, benchmarks_executables_dir_path) + qemu_build_path = os.path.join(qemu_path, "build-gcc") + + instructions: List[List[Union[str, int]]] = [] + + for target in targets: + executable_path = os.path.join( + executable_parent_dir_path, "{}-{}".format(benchmark_name, target)) + + qemu_exe_path = os.path.join(qemu_build_path, + "{}-linux-user".format(target), + "qemu-{}".format(target)) + + with tempfile.NamedTemporaryFile() as tmpfile: + run_callgrind = subprocess.run(["valgrind", + "--tool=callgrind", + "--callgrind-out-file={}".format( + tmpfile.name), + qemu_exe_path, + executable_path], + stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, + check=False) + if run_callgrind.returncode == 1: + clean_exit(qemu_path, run_callgrind.stderr.decode("utf-8")) + + callgrind_output = run_callgrind.stderr.decode("utf-8").split("\n") + instructions.append([target, int(callgrind_output[8].split(" ")[-1])]) + + return instructions + + +def measure_master_instructions( + reference_version_path: str, reference_commit_hash: str, + latest_version_path: str, benchmark_path: str, + benchmarks_executables_dir_path: str, qemu_path: str, + targets: List[str]) -> List[List[Union[str, int]]]: + """ + Measure the latest QEMU "master" instructions and also append the latest + instructions and reference version instructions to the instructions list. + + Parameters: + reference_version_path (str): Absolute path to reference version results + reference_commit_hash (str): Git hash of the reference version + latest_version_path (str): Absolute path to the latest version results + benchmark_path (str): Absolute path to benchmark + benchmarks_executables_dir_path (str): + Absolute path to the executables of the benchmark + qemu_path (str): Absolute path to QEMU + targets (List[str]): List of QEMU targets + + + Return: + (List[List[Union[str, int]]]): + [[target_name, instructions, comparison_instructions],[...],...] + comparsion_instructions: *[latest, reference] + If latest is not available, + then comparsion_instructions = reference + """ + benchmark_name = get_benchmark_name(benchmark_path) + + print(datetime.datetime.utcnow().isoformat(), + "- Measuring instructions for master - {}".format(benchmark_name), + file=sys.stderr, flush=True) + + instructions = measure_instructions( + benchmark_path, benchmarks_executables_dir_path, qemu_path, targets) + + reference_result = "{}-{}-results.csv".format( + reference_commit_hash, benchmark_name) + reference_result_path = os.path.join( + reference_version_path, reference_result) + + # Find if this benchmark has a record in the latest results + latest_result = "" + latest_results = os.listdir(latest_version_path) + for result in latest_results: + if result.split("-")[1] == benchmark_name: + latest_result = result + break + + # Append instructions from latest version if available + if latest_result != "": + latest_result_path = os.path.join(latest_version_path, latest_result) + with open(latest_result_path, "r") as file: + file.readline() + for target_instructions in instructions: + target_instructions.append( + int(file.readline().split('"')[1].replace(",", ""))) + # Delete the latest results. The directory will contain the new latest + # when the new "master" results are stored later. + os.unlink(latest_result_path) + + # Append instructions from reference version + with open(reference_result_path, "r") as file: + file.readline() + for target_instructions in instructions: + target_instructions.append( + int(file.readline().split('"')[1].replace(",", ""))) + + return instructions + + +def calculate_percentage(old_instructions: int, new_instructions: int) -> str: + """ + Calculate the change in percentage between two instruction counts + + Parameters: + old_instructions (int): Old number + new_instructions (int): New number + + Return: + (str): [+|-][change][%] or "-----" in case of 0.01% change + """ + percentage = round(((new_instructions - old_instructions) / + old_instructions) * 100, 3) + return format_percentage(percentage) + + +def format_percentage(percentage: float) -> str: + """ + Format the percentage value to add +|- and %. + + Parameters: + percentage (float): Percentage + + Returns: + (str): Formatted percentage string + """ + if abs(percentage) <= 0.01: + return "-----" + return "+" + str(percentage) + "%" if percentage > 0 \ + else str(percentage) + "%" + + +def calculate_change(instructions: List[List[Union[str, int]]]) -> None: + """ + Calculate the change in the recorded instructions for master compared to + latest results and reference version results. + + Parameters: + instructions (List[List[Union[str, int]]]): + [[target_name, instructions, comparison_instructions],[...],...] + comparsion_instructions: *[latest, reference] + If latest is not available, + then comparsion_instructions = reference + """ + for target_instructions in instructions: + target_instructions[-1] = calculate_percentage( + int(target_instructions[-1]), int(target_instructions[1])) + # If latest instructions exists + if len(target_instructions) == 4: + target_instructions[-2] = calculate_percentage( + int(target_instructions[-2]), int(target_instructions[1])) + + +def calculate_average(results: List[List[List[Union[str, int]]]], + targets: List[str], + num_benchmarks: int) -> List[List[Union[str, int]]]: + """ + Calculate the average results for each target for all benchmarks. + + Parameters: + results (List[List[List[Union[str, int]]]]): + [[target_name, instructions, comparison_instructions],[...],...] + comparsion_instructions: *[latest, reference] + If latest is not available, + then comparsion_instructions = reference + targets (List[str]): List of target names + num_benchmarks (int): Number of benchmarks + + Return: + (List[List[Union[str, int]]]): + [[target_name, average_instructions, \ + comparison_instructions],[...],...] + comparsion_instructions: *[average_latest, average_reference] + If latest is not available, + then comparsion_instructions = reference + """ + average_instructions: List[List[Union[str, int]]] = [] + + for i, target in enumerate(targets): + average_instructions.append([target]) + + total_instructions = 0 + total_latest_percentages: Optional[float] = 0.0 + total_reference_percentages = 0.0 + + for instructions in results: + total_instructions += int(instructions[i][1]) + if instructions[i][3] != "-----": + total_reference_percentages += float( + str(instructions[i][3])[:-1]) + if total_latest_percentages is not None: + if instructions[i][2] != "N/A": + if instructions[i][2] != "-----": + total_latest_percentages += float( + str(instructions[i][2])[:-1]) + else: + total_latest_percentages = None + + avg_instructions = total_instructions // num_benchmarks + avg_reference_percentages = format_percentage( + round(total_reference_percentages / num_benchmarks, 3)) + avg_latest_percentages = format_percentage( + round(total_latest_percentages / num_benchmarks, 3)) \ + if total_latest_percentages is not None else "N/A" + + average_instructions[-1].extend([avg_instructions, + avg_latest_percentages, + avg_reference_percentages]) + + return average_instructions + + +def write_to_csv(instructions: List[List[Union[str, int]]], + output_csv_path: str, percentages: bool = False, + reference_version: str = "") -> None: + """ + Write the [Target, Instructions] for each target in a CSV file. + comparison_instructions are ignored. + + Parameters: + instructions (List[List[Union[str, int]]]): + [[target_name, instructions, comparison_instructions],[...],...] + comparsion_instructions: *[latest, reference] + If latest is not available, + then comparsion_instructions = reference + output_csv_path (str): Absolute path to output CSV file + percentages (bool): Add percentages to the output CSV file + """ + with open(output_csv_path, "w") as file: + writer = csv.writer(file) + header = ["Target", "Instructions"] + if percentages: + header.extend(["Latest", reference_version]) + writer.writerow(header) + for target_instructions in instructions: + row = [] + row.extend([target_instructions[0], format( + target_instructions[1], ",")]) + if percentages: + row.extend(target_instructions[2:]) + writer.writerow(row) + + +def print_table(instructions: str, text: str, reference_version: str) -> None: + """ + Print the results in a tabular form + + Parameters: + instructions (List[List[Union[str, int]]]): + [[target_name, instructions, comparison_instructions],[...],...] + comparsion_instructions: *[latest, reference] + If latest is not available, + then comparsion_instructions = reference + text (str): Text be added to the table header + reference_version (str): Reference version used in these results + """ + print("{}\n{}\n{}". + format("-" * 56, text, "-" * 56)) + + print('{:<10} {:>20} {:>10} {:>10}\n{} {} {} {}'. + format('Target', + 'Instructions', + 'Latest', + reference_version, + '-' * 10, + '-' * 20, + '-' * 10, + '-' * 10)) + + for target_change in instructions: + # Replace commas with spaces in instruction count + # for easier readability. + formatted_instructions = format( + target_change[1], ",").replace(",", " ") + print('{:<10} {:>20} {:>10} {:>10}'.format( + target_change[0], formatted_instructions, *target_change[2:])) + + print("-" * 56) + + +def clean_exit(qemu_path: str, error_message: str) -> None: + """ + Clean up intermediate files and exit. + + Parameters: + qemu_path (str): Absolute path to QEMU + error_message (str): Error message to display after exiting + """ + # Clean the QEMU build path + qemu_build_path = os.path.join(qemu_path, "build-gcc") + if os.path.isdir(qemu_build_path): + shutil.rmtree(qemu_build_path) + sys.exit(error_message) + + +def verify_executables(benchmark_paths: List[str], targets: Dict[str, str], + benchmarks: List[Dict[str, str]], + benchmarks_executables_dir_path: str) -> None: + """ + Verify that all executables exist for each benchmark. + + Parameters: + benchmark_paths (List[str]): List of all paths to benchmarks + targets (Dict[str, str]): Dictionary the contains for each target, + target_name: target_compiler + benchmarks (List[Dict[str, str]]): Benchmarks data (name, parent_dir, path) + benchmarks_executables_dir_path (str): Absolute path to the executables dir + """ + print(datetime.datetime.utcnow().isoformat(), + "- Verifying executables of {} benchmarks for {} targets". + format(len(benchmark_paths), len(targets)), + file=sys.stderr, flush=True) + + for benchmark in benchmarks: + executable_parent_dir_path = get_executable_parent_dir_path( + benchmark["path"], benchmarks_executables_dir_path) + + # Verify that the exists for this benchmark executables, if not, + # create it + if not os.path.isdir(executable_parent_dir_path): + os.mkdir(executable_parent_dir_path) + + for target_name, target_compiler in targets.items(): + compiled_benchmark = "{}-{}".format( + benchmark["name"], target_name) + compiled_benchmark_path = os.path.join( + executable_parent_dir_path, compiled_benchmark) + # Verify that the the executable for this target is available, + # if not, compile it + if not os.path.isfile(compiled_benchmark_path): + compile_target(benchmark["path"], + compiled_benchmark_path, + target_compiler) + + +def verify_reference_results(reference_version: str, qemu_path: str, + benchmarks: List[Dict[str, str]], + reference_version_results_dir_path: str, + targets: List[str], + benchmarks_executables_dir_path: str) -> None: + """ + Verify that results are available for reference version. + If results are missing, build QEMU for the reference version then perform + the measurements. + + Paramters: + reference_version (str): Reference QEMU version + qemu_path (str): Absolute path to QEMU + benchmark_paths (List[str]): List of all paths to benchmarks + reference_version_results_dir_path (str): Absolute path to the reference + version results dir + targets (List[str]): Target names + benchmarks (List[Dict[str, str]]): Benchmarks data (name, parent_dir, path) + benchmarks_executables_dir_path (str): Path to the root executables dir + """ + print(datetime.datetime.utcnow().isoformat(), + "- Verifying results of reference version {}". + format(reference_version), file=sys.stderr, flush=True) + + # Set flag to know if QEMU was built for reference version before + did_build_reference = False + + latest_commit_hash = get_commit_hash(reference_version, qemu_path) + + for benchmark in benchmarks: + benchmark_results_dir_path = os.path.join( + reference_version_results_dir_path, benchmark["parent_dir"]) + + # Verify that the results directory for the benchmark exists, if not, + # create it + if not os.path.isdir(benchmark_results_dir_path): + os.mkdir(benchmark_results_dir_path) + + # Verify that the the results.csv file for the benchmark exits, if not, + # create it + results_path = os.path.join(benchmark_results_dir_path, + "{}-{}-results.csv". + format(latest_commit_hash, + benchmark["name"])) + if not os.path.isfile(results_path): + # Only build qemu if reference version wasn't built before + if not did_build_reference: + build_qemu(qemu_path, reference_version, targets) + did_build_reference = True + print(datetime.datetime.utcnow().isoformat(), + "- Measuring instructions for reference version {} - {}". + format(reference_version, benchmark["name"]), + file=sys.stderr, flush=True) + instructions = measure_instructions( + benchmark["path"], + benchmarks_executables_dir_path, + qemu_path, + targets) + write_to_csv(instructions, results_path) + + +def verify_requirements() -> None: + """ + Verify that all script requirements are installed (valgrind & git). + """ + # Insure that valgrind is installed + check_valgrind_installation = subprocess.run(["which", "valgrind"], + stdout=subprocess.DEVNULL, + check=False) + if check_valgrind_installation.returncode: + sys.exit("Please install valgrind before running the script.") + + # Insure that git is installed + check_git_installation = subprocess.run(["which", "git"], + stdout=subprocess.DEVNULL, + check=False) + if check_git_installation.returncode: + sys.exit("Please install git before running the script.") + + +def main(): + """ + Parse the command line arguments then start the execution. + Output on STDOUT represents the nightly test results. + Output on STDERR represents the execution log and errors if any. + Output on STDERR must be redirected to either /dev/null or to a log file. + + Syntax: + nightly_tests_core.py [-h] [-r REF] + Optional arguments: + -h, --help Show this help message and exit + -r REF, --reference REF + Reference QEMU version - Default is v5.1.0 + Example of usage: + nightly_tests_core.py -r v5.1.0 2>log.txt + """ + parser = argparse.ArgumentParser() + parser.add_argument("-r", "--reference", dest="ref", + default="v5.1.0", + help="Reference QEMU version - Default is v5.1.0") + reference_version = parser.parse_args().ref + + targets = { + "aarch64": "aarch64-linux-gnu-gcc", + "alpha": "alpha-linux-gnu-gcc", + "arm": "arm-linux-gnueabi-gcc", + "hppa": "hppa-linux-gnu-gcc", + "m68k": "m68k-linux-gnu-gcc", + "mips": "mips-linux-gnu-gcc", + "mipsel": "mipsel-linux-gnu-gcc", + "mips64": "mips64-linux-gnuabi64-gcc", + "mips64el": "mips64el-linux-gnuabi64-gcc", + "ppc": "powerpc-linux-gnu-gcc", + "ppc64": "powerpc64-linux-gnu-gcc", + "ppc64le": "powerpc64le-linux-gnu-gcc", + "riscv64": "riscv64-linux-gnu-gcc", + "s390x": "s390x-linux-gnu-gcc", + "sh4": "sh4-linux-gnu-gcc", + "sparc64": "sparc64-linux-gnu-gcc", + "x86_64": "gcc" + } + + # Verify that the script requirements are installed + verify_requirements() + + # Get required paths + nightly_tests_dir_path = pathlib.Path(__file__).parent.parent.absolute() + benchmarks_dir_path = os.path.join(nightly_tests_dir_path, "benchmarks") + benchmarks_source_dir_path = os.path.join(benchmarks_dir_path, "source") + benchmarks_executables_dir_path = os.path.join( + benchmarks_dir_path, "executables") + + # Verify that If the executables directory exists, if not, create it + if not os.path.isdir(benchmarks_executables_dir_path): + os.mkdir(benchmarks_executables_dir_path) + + # Get absolute path to all available benchmarks + benchmark_paths = sorted([y for x in os.walk(benchmarks_source_dir_path) + for y in glob.glob(os.path.join(x[0], '*.c'))]) + + benchmarks = [{ + "name": get_benchmark_name(benchmark_path), + "parent_dir": get_benchmark_parent_dir(benchmark_path), + "path": benchmark_path} for benchmark_path in benchmark_paths] + + # Verify that all executables exist for each benchmark + verify_executables(benchmark_paths, targets, benchmarks, + benchmarks_executables_dir_path) + + # Set QEMU path and clone from Git if the path doesn't exist + qemu_path = os.path.join(nightly_tests_dir_path, "qemu-nightly") + if not os.path.isdir(qemu_path): + # Clone QEMU into the temporary directory + print(datetime.datetime.utcnow().isoformat(), + "- Fetching QEMU: ", end="", flush=True, file=sys.stderr) + git_clone(qemu_path) + print("\n", file=sys.stderr, flush=True) + + # Verify that the results directory exists, if not, create it + results_dir_path = os.path.join(nightly_tests_dir_path, "results") + if not os.path.isdir(results_dir_path): + os.mkdir(results_dir_path) + + # Verify that the reference version results directory exists, if not, + # create it + reference_version_results_dir_path = os.path.join( + results_dir_path, reference_version) + if not os.path.isdir(reference_version_results_dir_path): + os.mkdir(reference_version_results_dir_path) + + # Verify that previous results are available for reference version + verify_reference_results(reference_version, qemu_path, benchmarks, + reference_version_results_dir_path, + targets.keys(), benchmarks_executables_dir_path) + + # Compare results with the latest QEMU master + # ------------------------------------------------------------------------- + # Verify that the "latest" results directory exists, if not, + # create it + latest_version_results_dir_path = os.path.join(results_dir_path, "latest") + if not os.path.isdir(latest_version_results_dir_path): + os.mkdir(latest_version_results_dir_path) + + # Verify that the "history" results directory exists, if not, create it + history_results_dir_path = os.path.join(results_dir_path, "history") + if not os.path.isdir(history_results_dir_path): + os.mkdir(history_results_dir_path) + + # Build QEMU for master + build_qemu(qemu_path, "master", targets.keys()) + + # Get the commit hash for the top commit at master + master_commit_hash = get_commit_hash("master", qemu_path) + + # Print report summary header + print("{}\n{}".format("-" * 56, "{} SUMMARY REPORT - COMMIT {}". + format(" "*11, master_commit_hash))) + + # For each benchmark, compare the current master results with + # latest and reference version + results = [] + for benchmark in benchmarks: + reference_version_benchmark_results_dir_path = os.path.join( + reference_version_results_dir_path, benchmark["parent_dir"]) + latest_version_benchmark_results_dir_path = os.path.join( + latest_version_results_dir_path, benchmark["parent_dir"]) + history_benchmark_results_dir_path = os.path.join( + history_results_dir_path, benchmark["parent_dir"]) + + # Verify that the the benchmark directory exists in the "latest" + # directory, if not, create it + if not os.path.isdir(latest_version_benchmark_results_dir_path): + os.mkdir(latest_version_benchmark_results_dir_path) + + # Verify that the the benchmark directory exists in the "history" + # directory, if not, create it + if not os.path.isdir(history_benchmark_results_dir_path): + os.mkdir(history_benchmark_results_dir_path) + + # Obtain the instructions array which will contain for each target, + # the target name, the number of "master" instructions, + # "latest" instructions if available, + # and "reference version" instructions + instructions = measure_master_instructions( + reference_version_benchmark_results_dir_path, + get_commit_hash(reference_version, qemu_path), + latest_version_benchmark_results_dir_path, + benchmark["path"], + benchmarks_executables_dir_path, + qemu_path, + targets.keys()) + + # Update the "latest" directory with the new results form master + updated_latest_version_benchmark_results = os.path.join( + latest_version_benchmark_results_dir_path, + "{}-{}-results.csv".format(master_commit_hash, benchmark["name"])) + write_to_csv(instructions, updated_latest_version_benchmark_results) + + calculate_change(instructions) + + # Insert "N/A" for targets that don't have "latest" instructions + for target_instructions in instructions: + if len(target_instructions) == 3: + target_instructions.insert(2, "N/A") + + history_benchmark_results = os.path.join( + history_benchmark_results_dir_path, + "{}-{}-results.csv".format(master_commit_hash, benchmark["name"])) + write_to_csv(instructions, history_benchmark_results, + True, reference_version) + + # Store the results + results.append([benchmark["name"], instructions]) + + # Calculate the average instructions for each target + # Only send the instructions as results without the benchmark names + average_instructions = calculate_average( + [result[1] for result in results], targets.keys(), len(benchmarks)) + + # Save average results to results/history directory + average_results = os.path.join( + history_results_dir_path, + "{}-average-results.csv".format(master_commit_hash)) + write_to_csv(average_instructions, average_results, + True, reference_version) + + # Print results + print_table(average_instructions, " "*20 + + "AVERAGE RESULTS", reference_version) + print("\n", " "*17, "DETAILED RESULTS") + for [benchmark_name, instructions] in results: + print_table(instructions, + "Test Program: " + benchmark_name, + reference_version) + + # Cleanup (delete the build directory) + shutil.rmtree(os.path.join(qemu_path, "build-gcc")) + # ------------------------------------------------------------------------- + + +if __name__ == "__main__": + main() diff --git a/tests/performance/nightly-tests/scripts/run_nightly_tests.py b/tests/performance/nightly-tests/scripts/run_nightly_tests.py new file mode 100755 index 0000000000..ea65be60dc --- /dev/null +++ b/tests/performance/nightly-tests/scripts/run_nightly_tests.py @@ -0,0 +1,135 @@ +#!/usr/bin/env python3 + +""" +Entry point script for running nightly performance tests on QEMU. + +This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman +Copyright (C) 2020 Aleksandar Markovic + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see . +""" + + +import datetime +import io +import os +import subprocess +import time +import sys +from send_email import send_email + + +# Record system hardware information +with open('/proc/cpuinfo', 'r') as cpuinfo: + for line in cpuinfo: + if line.startswith('model name'): + HOST_CPU = line.rstrip('\n').split(':')[1].strip() + break +with open('/proc/meminfo', 'r') as meminfo: + for line in meminfo: + if line.startswith('MemTotal'): + HOST_MEMORY_KB = int(line.rstrip('\n').split(':')[1]. + strip().split(' ')[0]) + HOST_MEMORY = str(round(HOST_MEMORY_KB / (1024 * 1024), 2)) + " GB" + break + +# Find path for the "nightly_tests_core.py" script +NIGHTLY_TESTS_CORE_PATH = os.path.join( + os.path.dirname(os.path.abspath(__file__)), + "nightly_tests_core.py") + +NIGHTLY_TESTS_ARGUMENTS = sys.argv[1:] + +# Start the nightly test +START_EPOCH = time.time() +RUN_NIGHTLY_TESTS = subprocess.run([NIGHTLY_TESTS_CORE_PATH, + *NIGHTLY_TESTS_ARGUMENTS], + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + check=False) +END_EPOCH = time.time() + +# Perform time calculations +EXECUTION_TIME = datetime.timedelta(seconds=END_EPOCH - START_EPOCH) +TEST_DATE = datetime.datetime.utcfromtimestamp( + START_EPOCH).strftime('%A, %B %-d, %Y') +START_TIME = datetime.datetime.utcfromtimestamp( + START_EPOCH).strftime('%Y-%m-%d %H:%M:%S') +END_TIME = datetime.datetime.utcfromtimestamp( + END_EPOCH).strftime('%Y-%m-%d %H:%M:%S') + +# Get nightly test status +if RUN_NIGHTLY_TESTS.returncode: + STATUS = "FAILURE" +else: + STATUS = "SUCCESS" + +# Initialize a StringIO to print all the output into +OUTPUT = io.StringIO() + +# Print the nightly test statistical information +print("{:<17}: {}\n{:<17}: {}\n". + format("Host CPU", + HOST_CPU, + "Host Memory", + HOST_MEMORY), file=OUTPUT) +print("{:<17}: {}\n{:<17}: {}\n{:<17}: {}\n". + format("Start Time (UTC)", + START_TIME, + "End Time (UTC)", + END_TIME, + "Execution Time", + EXECUTION_TIME), file=OUTPUT) +print("{:<17}: {}\n".format("Status", STATUS), file=OUTPUT) + +if STATUS == "SUCCESS": + print("Note:\nChanges denoted by '-----' are less than 0.01%.\n", + file=OUTPUT) + +# Print the nightly test stdout (main output) +print(RUN_NIGHTLY_TESTS.stdout.decode("utf-8"), file=OUTPUT) + +# If the nightly test failed, print the stderr (error logs) +if STATUS == "FAILURE": + print("{}\n{}\n{}".format("-" * 56, + " " * 18 + "ERROR LOGS", + "-" * 56), file=OUTPUT) + print(RUN_NIGHTLY_TESTS.stderr.decode("utf-8"), file=OUTPUT) + + +# Temp file to store the output in case sending the email failed +# with open("temp.txt", "w") as file: +# file.write(OUTPUT.getvalue()) + +# Use an HTML message to preserve monospace formatting +HTML_MESSAGE = """\ +
+{body}
+
+""".format(body=OUTPUT.getvalue()) +OUTPUT.close() + +# Send the nightly test results email to the QEMU mailing list +while True: + try: + send_email("[REPORT] Nightly Performance Tests - {}".format(TEST_DATE), + ["qemu-devel@nongnu.org"], HTML_MESSAGE) + except Exception: # pylint: disable=W0703 + # Wait for a minute then retry sending + time.sleep(60) + continue + else: + break diff --git a/tests/performance/nightly-tests/scripts/send_email.py b/tests/performance/nightly-tests/scripts/send_email.py new file mode 100644 index 0000000000..e24e244f51 --- /dev/null +++ b/tests/performance/nightly-tests/scripts/send_email.py @@ -0,0 +1,56 @@ +""" +Helper script for sending emails with the nightly performance test results. + +This file is a part of the project "TCG Continuous Benchmarking". + +Copyright (C) 2020 Ahmed Karaman +Copyright (C) 2020 Aleksandar Markovic + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 2 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see . +""" + + +import smtplib +from typing import List +from email.mime.multipart import MIMEMultipart +from email.mime.text import MIMEText +from email.utils import COMMASPACE, formatdate + + +GMAIL_USER = {"name": "", + "email": "", + "pass": ""} + + +def send_email(subject: str, send_to: List[str], html: str) -> None: + """ + Send an HTML email. + + Parameters: + subject (str): Email subject + send_to (List(str)): List of recipients + html (str): HTML message + """ + msg = MIMEMultipart('alternative') + msg['From'] = "{} <{}>".format(GMAIL_USER["name"], GMAIL_USER["email"]) + msg['To'] = COMMASPACE.join(send_to) + msg['Date'] = formatdate(localtime=True) + msg['Subject'] = subject + + msg.attach(MIMEText(html, 'html')) + + server = smtplib.SMTP_SSL('smtp.gmail.com', 465) + server.login(GMAIL_USER["email"], GMAIL_USER["pass"]) + server.sendmail(msg['From'], send_to, msg.as_string()) + server.quit()