From patchwork Fri Jun 15 19:46:57 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alex_Benn=C3=A9e?= X-Patchwork-Id: 138820 Delivered-To: patch@linaro.org Received: by 2002:a2e:970d:0:0:0:0:0 with SMTP id r13-v6csp1297891lji; Fri, 15 Jun 2018 13:44:41 -0700 (PDT) X-Google-Smtp-Source: ADUXVKIaYCkBcLnl7iFycLoHLzoswQW5zo8yIdhlD/KxdxnyEouV1/JOavR6qgCu94i8tlqRBnsm X-Received: by 2002:a37:b943:: with SMTP id j64-v6mr2700077qkf.291.1529095481449; Fri, 15 Jun 2018 13:44:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529095481; cv=none; d=google.com; s=arc-20160816; b=a75sFOq788znvMFrhbEwqJFGlGQnWymho+vwRmvAKx5lt7RD2tqFuK/zgcaqqjoEp8 bFu99zLKba3RqL1B+f4RB1S+iocwNADniGUe7o4rByWE+rpQ6/bcsjHfJrX9J85Z/WsN Hnhlow8q53dZsYHlJw3VO78IW4HEibH4jVeGjfIwZc1gwqxl8DXDeY6tb7itz2vBHCSu UnCqHLh9kZD4qUd12Vo5waDMavOpcA9LruwOig6P6f4un6X3HwB160UE6Tb+GxHiotgf G7Wp3Tf3AW2OFX/QDyNmweo1CNDv62a202SeefhfcMr1vxDWvCQd9YNIRfuSwUbt9i5q 52hQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject :content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=Wh2wk02MK4iYtp/RWzY2WpOp3D+0lt8WtErhgVyAemI=; b=aSNXENV+v+OXorORDX5WA7tqVd9idV+ZRuxhpYPmGQO8/bQQYUBfu08oHtlF14mshb vwQB2QvfB+RQu57Dn4fTISCDgDvuVTb8g3ESDtUk/r4JAOQOLOs8g/fPxDZFip61bpjG WMVPXjQaAR/No74EgGpEPHOEoMwTlLKNpCEHGXpbaEl5KMwWDOtMu1R0x+aeJaF1OI1E 0CJ4fin89eU9qpiTnrIWQym8kPEeZUySnU1WtT25TapG6hIEwTjVYrDd9uay4n2AS9Dd w95aTRV+irEc548/ldwjR7BYFrSWVVRHchE8t54uPSV27+3mssLFVEvulUekBcvyUM+Y hwZg== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=Gj2GKi1S; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id i188-v6si328574qka.144.2018.06.15.13.44.41 for (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 15 Jun 2018 13:44:41 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=Gj2GKi1S; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:49314 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTvaK-00054c-UD for patch@linaro.org; Fri, 15 Jun 2018 16:44:40 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56215) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTuyL-0000e2-JV for qemu-devel@nongnu.org; Fri, 15 Jun 2018 16:05:27 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fTuyI-0007p1-1I for qemu-devel@nongnu.org; Fri, 15 Jun 2018 16:05:25 -0400 Received: from mail-wm0-x235.google.com ([2a00:1450:400c:c09::235]:40368) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fTuyH-0007oA-LI for qemu-devel@nongnu.org; Fri, 15 Jun 2018 16:05:21 -0400 Received: by mail-wm0-x235.google.com with SMTP id n5-v6so5756228wmc.5 for ; Fri, 15 Jun 2018 13:05:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Wh2wk02MK4iYtp/RWzY2WpOp3D+0lt8WtErhgVyAemI=; b=Gj2GKi1SSKjaz5r1E4FA/UmmLgHyR+JG7emN6y/OetdGylfoPLgaH3BfycI3na4X+Y ZqGVtSiACXwJwL4SZS2mmmOlBmY8f0zoSaNf4J6SSrCuOcPTdCXCTWWXcCUO62kAR5ds erT0X0ap05cpMcdb7ZgaFIoB0xyrkiQiImBp4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Wh2wk02MK4iYtp/RWzY2WpOp3D+0lt8WtErhgVyAemI=; b=tHyfLfvssOTYzXNJKCgo+os6N++6pwfrtRS02yMYJV1NCCwCOuoWjlWPWoORTW2ceJ MR0QDc7lYZlicC2a38Rt6KIXw7dPWXhC7hKfvavEfkif0/o01LvqNfyQmsUNBafVQ4LD YkxAvPy92VfFjoqT1El4jdtsIMKYAmLKGOAQJrfrnDuIlnIGxX+TZuE5IvU58c3jlz3a Q/7731A/wV1nkYnG0EhOtuOWcHW/bMBZpxMLOxv3/urhh7dFmU0MxAYOC3ZX5DFeztD9 0X6wqA7fy1WwETWFFzEy1JP0jI/VRCh+dGgROOEFBy4VY6O1KLDxN5EfMIISd/8lITPt 1m3Q== X-Gm-Message-State: APt69E10cfQ/ibXTz9qMr7KHODnIhOIgdP4eFog4PECPO3BoYk5hIwqf XCoThHkD/+SWCgHoFgBfap5iPI3vI/8= X-Received: by 2002:a1c:4189:: with SMTP id o131-v6mr2115705wma.7.1529093120482; Fri, 15 Jun 2018 13:05:20 -0700 (PDT) Received: from zen.linaro.local ([81.128.185.34]) by smtp.gmail.com with ESMTPSA id t124-v6sm1806263wmt.29.2018.06.15.13.05.11 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 15 Jun 2018 13:05:14 -0700 (PDT) Received: from zen.linaroharston (localhost [127.0.0.1]) by zen.linaro.local (Postfix) with ESMTP id 76AB23E0FA9; Fri, 15 Jun 2018 20:47:09 +0100 (BST) From: =?utf-8?q?Alex_Benn=C3=A9e?= To: cota@braap.org, famz@redhat.com, berrange@redhat.com, f4bug@amsat.org, richard.henderson@linaro.org, balrogg@gmail.com, aurelien@aurel32.net, agraf@suse.de, peter.maydell@linaro.org Date: Fri, 15 Jun 2018 20:46:57 +0100 Message-Id: <20180615194705.28019-47-alex.bennee@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180615194705.28019-1-alex.bennee@linaro.org> References: <20180615194705.28019-1-alex.bennee@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c09::235 Subject: [Qemu-devel] [PATCH v7 46/54] tests/tcg: add run, diff, and skip helper macros X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , qemu-devel@nongnu.org, "open list:ARM" , Paolo Bonzini , =?utf-8?q?Alex_Benn=C3=A9e?= , Richard Henderson Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" As we aren't using the default runners for all the test cases it is easy to miss out things like timeouts. To help with this we add some helpers and use them so we only need to make core changes in one place. Signed-off-by: Alex Bennée Reviewed-by: Philippe Mathieu-Daudé Tested-by: Philippe Mathieu-Daudé --- v5 - new for v5 v5.1 - diff-test->diff-out v7 - mention $(TARGET_NAME) when skipping --- tests/tcg/Makefile | 13 ++++++++++--- tests/tcg/aarch64/Makefile.target | 6 ++---- tests/tcg/arm/Makefile.target | 6 ++---- tests/tcg/i386/Makefile.target | 10 ++++------ tests/tcg/multiarch/Makefile.target | 4 ++-- 5 files changed, 20 insertions(+), 19 deletions(-) -- 2.17.1 diff --git a/tests/tcg/Makefile b/tests/tcg/Makefile index e7dbcdb5bf..bf06415390 100644 --- a/tests/tcg/Makefile +++ b/tests/tcg/Makefile @@ -34,6 +34,15 @@ quiet-command = $(if $(V),$1,$(if $(2),@printf " %-7s %s\n" $2 $3 && $1, @$1)) +# $1 = test name, $2 = cmd, $3 = desc +run-test = $(call quiet-command, timeout $(TIMEOUT) $2 > $1.out,"TEST",$3) + +# $1 = test name, $2 = reference +diff-out = $(call quiet-command, diff -u $1.out $2 | head -n 10,"DIFF","$1.out with $2") + +# $1 = test name, $2 = reason +skip-test = @printf " SKIPPED %s on $(TARGET_NAME) because %s\n" $1 $2 + # Tests we are building TESTS= @@ -84,9 +93,7 @@ RUN_TESTS=$(patsubst %,run-%, $(TESTS)) RUN_TESTS+=$(EXTRA_RUNS) run-%: % - $(call quiet-command, \ - timeout $(TIMEOUT) $(QEMU) $< > $<.out, \ - "TEST", "$< on $(TARGET_NAME)") + $(call run-test, $<, $(QEMU) $<, "$< on $(TARGET_NAME)") .PHONY: run run: $(RUN_TESTS) diff --git a/tests/tcg/aarch64/Makefile.target b/tests/tcg/aarch64/Makefile.target index 7dba32138d..08c45b8470 100644 --- a/tests/tcg/aarch64/Makefile.target +++ b/tests/tcg/aarch64/Makefile.target @@ -13,7 +13,5 @@ TESTS:=$(AARCH64_TESTS) fcvt: LDFLAGS+=-lm run-fcvt: fcvt - $(call quiet-command, \ - $(QEMU) $< > fcvt.out && \ - diff -u $(AARCH64_SRC)/fcvt.ref fcvt.out, \ - "TEST", "$< (default) on $(TARGET_NAME)") + $(call run-test,$<,$(QEMU) $<, "$< on $(TARGET_NAME)") + $(call diff-out,$<,$(AARCH64_SRC)/fcvt.ref) diff --git a/tests/tcg/arm/Makefile.target b/tests/tcg/arm/Makefile.target index 14b5435fc6..aa4e4e3782 100644 --- a/tests/tcg/arm/Makefile.target +++ b/tests/tcg/arm/Makefile.target @@ -24,10 +24,8 @@ fcvt: LDFLAGS+=-lm # fcvt: CFLAGS+=-march=armv8.2-a+fp16 -mfpu=neon-fp-armv8 run-fcvt: fcvt - $(call quiet-command, \ - $(QEMU) $< > fcvt.out && \ - diff -u $(ARM_SRC)/fcvt.ref fcvt.out, \ - "TEST", "$< (default) on $(TARGET_NAME)") + $(call run-test,fcvt,$(QEMU) $<,"$< on $(TARGET_NAME)") + $(call diff-out,fcvt,$(ARM_SRC)/fcvt.ref) endif # On ARM Linux only supports 4k pages diff --git a/tests/tcg/i386/Makefile.target b/tests/tcg/i386/Makefile.target index cd173363ee..97b7c23cf1 100644 --- a/tests/tcg/i386/Makefile.target +++ b/tests/tcg/i386/Makefile.target @@ -31,7 +31,7 @@ test-i386: test-i386.c test-i386-code16.S test-i386-vm86.S test-i386.h test-i386 # Specialist test runners run-runcom: runcom pi_10.com - $(call quiet-command, $(QEMU) ./runcom $(I386_SRC)/pi_10.com > runcom.out, "TEST", "$< on $(TARGET_NAME)") + $(call run-test,$<,$(QEMU) ./runcom $(I386_SRC)/pi_10.com,"$< on $(TARGET_NAME)") ifeq ($(SPEED), slow) @@ -40,13 +40,11 @@ test-i386-fprem.ref: test-i386-fprem run-test-i386-fprem: TIMEOUT=60 run-test-i386-fprem: test-i386-fprem - $(call quiet-command, \ - $(QEMU) $< > $<.out && \ - diff -u $(I386_SRC)/$<.ref $<.out, \ - "TEST", "$< (default) on $(TARGET_NAME)") + $(call run-test,test-i386-fprem, $(QEMU) $<,"$< on $(TARGET_NAME)") + $(call diff-out,test-i386-fprem, $(I386_SRC)/$<.ref) else run-test-i386-fprem: test-i386-fprem - $(call quiet-command, /bin/true, "SLOW TEST", "$< SKIPPED on $(TARGET_NAME)") + $(call skip-test, $<, "SLOW") endif # On i386 and x86_64 Linux only supports 4k pages (large pages are a different hack) diff --git a/tests/tcg/multiarch/Makefile.target b/tests/tcg/multiarch/Makefile.target index 90e45a881e..b77084c146 100644 --- a/tests/tcg/multiarch/Makefile.target +++ b/tests/tcg/multiarch/Makefile.target @@ -27,10 +27,10 @@ testthread: LDFLAGS+=-lpthread # default case (host page size) run-test-mmap: test-mmap - $(call quiet-command, $(QEMU) $< > test-mmap.out, "TEST", \ + $(call run-test, test-mmap, $(QEMU) $<, \ "$< (default) on $(TARGET_NAME)") # additional page sizes (defined by each architecture adding to EXTRA_RUNS) run-test-mmap-%: test-mmap - $(call quiet-command, $(QEMU) -p $* $< > test-mmap-$*.out, "TEST", \ + $(call run-test, test-mmap-$*, $(QEMU) -p $* $<,\ "$< ($* byte pages) on $(TARGET_NAME)")