From patchwork Mon Jan 20 13:20:10 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Campbell X-Patchwork-Id: 23391 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-yh0-f70.google.com (mail-yh0-f70.google.com [209.85.213.70]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 0199E2107B for ; Mon, 20 Jan 2014 13:22:17 +0000 (UTC) Received: by mail-yh0-f70.google.com with SMTP id l109sf13405368yhq.1 for ; Mon, 20 Jan 2014 05:22:17 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:from:to:date:message-id :mime-version:cc:subject:precedence:list-id:list-unsubscribe :list-post:list-help:list-subscribe:sender:errors-to :x-original-sender:x-original-authentication-results:mailing-list :list-archive:content-type:content-transfer-encoding; bh=pjtQ+i+6rqRwL1mkLJ2SW+RHXwrqufRKc6Ru6j6Z8+4=; b=T4Xlw1C/JDqT83OQDXn0adzgrBhEeSY3vt0Sc4ouRRqsqPYyqYKClUTRWHcQH1siMC 8S5pchzv73QipsBsCNO1FNzJClKcby3TcAjVS9j615QlBE2o6u1OSapNBn86Seo82U+4 aS7vLICHiruD6GsisRFbxPhdIYFZR6bfqbcI0hiEDcaPNuNBbhCGDDbESLYxZZW4nP1c UNexiOTKVpQaS/tv3CHcXT7iUCGI5/nj9YeErjiK+X0FIySjY2Fo8eHvElbef6zSaVOT ldh9HkactVbjL/4qOUKjBthwUnmaOYpy6q3aenthy7DkUwdHqzX6kdq2HbhHqXgw+ZYf ggRA== X-Gm-Message-State: ALoCoQk+tBluToOo7bNDsshDEfnbc/f6JL3I3NciRQYKTT34st/zNbNTmnD35eooN1pzXKw6hB8Z X-Received: by 10.52.159.99 with SMTP id xb3mr5732767vdb.4.1390224136964; Mon, 20 Jan 2014 05:22:16 -0800 (PST) X-BeenThere: patchwork-forward@linaro.org Received: by 10.140.21.136 with SMTP id 8ls1007131qgl.27.gmail; Mon, 20 Jan 2014 05:22:16 -0800 (PST) X-Received: by 10.220.99.72 with SMTP id t8mr10662503vcn.10.1390224136695; Mon, 20 Jan 2014 05:22:16 -0800 (PST) Received: from mail-vb0-f52.google.com (mail-vb0-f52.google.com [209.85.212.52]) by mx.google.com with ESMTPS id vc3si326222vdc.122.2014.01.20.05.22.16 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 20 Jan 2014 05:22:16 -0800 (PST) Received-SPF: neutral (google.com: 209.85.212.52 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) client-ip=209.85.212.52; Received: by mail-vb0-f52.google.com with SMTP id p14so2839247vbm.11 for ; Mon, 20 Jan 2014 05:22:16 -0800 (PST) X-Received: by 10.220.106.84 with SMTP id w20mr10627213vco.18.1390224136595; Mon, 20 Jan 2014 05:22:16 -0800 (PST) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.220.174.196 with SMTP id u4csp57614vcz; Mon, 20 Jan 2014 05:22:16 -0800 (PST) X-Received: by 10.140.97.73 with SMTP id l67mr26345357qge.56.1390224135806; Mon, 20 Jan 2014 05:22:15 -0800 (PST) Received: from lists.xen.org (lists.xen.org. [50.57.142.19]) by mx.google.com with ESMTPS id q3si642173qef.74.2014.01.20.05.22.15 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Mon, 20 Jan 2014 05:22:15 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of xen-devel-bounces@lists.xen.org designates 50.57.142.19 as permitted sender) client-ip=50.57.142.19; Received: from localhost ([127.0.0.1] helo=lists.xen.org) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1W5Em2-00080X-Av; Mon, 20 Jan 2014 13:20:18 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1W5Em0-00080R-Iw for xen-devel@lists.xen.org; Mon, 20 Jan 2014 13:20:16 +0000 Received: from [85.158.137.68:55326] by server-5.bemta-3.messagelabs.com id D0/EF-25188-F822DD25; Mon, 20 Jan 2014 13:20:15 +0000 X-Env-Sender: Ian.Campbell@citrix.com X-Msg-Ref: server-5.tower-31.messagelabs.com!1390224013!10161388!1 X-Originating-IP: [66.165.176.89] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogNjYuMTY1LjE3Ni44OSA9PiAyMDMwMDc=\n X-StarScan-Received: X-StarScan-Version: 6.9.16; banners=-,-,- X-VirusChecked: Checked Received: (qmail 26794 invoked from network); 20 Jan 2014 13:20:14 -0000 Received: from smtp.citrix.com (HELO SMTP.CITRIX.COM) (66.165.176.89) by server-5.tower-31.messagelabs.com with RC4-SHA encrypted SMTP; 20 Jan 2014 13:20:14 -0000 X-IronPort-AV: E=Sophos;i="4.95,690,1384300800"; d="scan'208";a="94475441" Received: from accessns.citrite.net (HELO FTLPEX01CL02.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 20 Jan 2014 13:20:12 +0000 Received: from norwich.cam.xci-test.com (10.80.248.129) by smtprelay.citrix.com (10.13.107.79) with Microsoft SMTP Server id 14.2.342.4; Mon, 20 Jan 2014 08:20:12 -0500 Received: from spare.cam.xci-test.com ([10.80.2.80] helo=kazak.uk.xensource.com.) by norwich.cam.xci-test.com with esmtp (Exim 4.72) (envelope-from ) id 1W5Elv-0000XR-AR; Mon, 20 Jan 2014 13:20:11 +0000 From: Ian Campbell To: Date: Mon, 20 Jan 2014 13:20:10 +0000 Message-ID: <1390224010-25510-1-git-send-email-ian.campbell@citrix.com> X-Mailer: git-send-email 1.8.5.2 MIME-Version: 1.0 X-DLP: MIA1 Cc: Ian Campbell , xen-devel@lists.xen.org Subject: [Xen-devel] [PATCH OSSTEST] README: Add some core concepts and terminology X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Post: , List-Help: , List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: ian.campbell@citrix.com X-Original-Authentication-Results: mx.google.com; spf=neutral (google.com: 209.85.212.52 is neither permitted nor denied by best guess record for domain of patch+caf_=patchwork-forward=linaro.org@linaro.org) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 List-Archive: --- README | 202 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 201 insertions(+), 1 deletion(-) Signed-off-by: Ian Campbell diff --git a/README b/README index 29c9d45..60379c4 100644 --- a/README +++ b/README @@ -1,3 +1,197 @@ +Introduction +============ + +OSStest is the Xen Project automated test infrastructure. + +Terminology +=========== + +"flight": + + Each run of osstest is referred to as a "flight". Each flight is + given a unique ID (a number or name). + +"job": + + Each flight consists of one or more "jobs". These are a sequence + of test steps run in order and correspond to a column in the test + report grid. They have names like "build-amd64" or + "test-amd64-amd64-pv". A job can depend on the output of another + job in the flight -- e.g. most test-* jobs depend on one or more + build-* jobs. + +"step": + + Each job consists of multiple "steps" which is an individual test + operation, such as "build the hypervisor", "install a guest", + "start a guest", "migrate a guest", etc. A step corresponds to a + cell in the results grid. A given step can be reused in multiple + different jobs, e.g. the "xen build" step is used in several + different build-* jobs. This reuse can be seen in the rows of the + results grid. + +"runvar": + + A runvar is a named textual variable associated with each job in a + given flight. They serve as both the inputs and outputs to the + job. + + For example a Xen build job may have input runvars "tree_xen" (the + tree to clone) and "revision_xen" (the version to test). As output + it sets "path_xendist" which is the path to the tarball of the + resulting binary. + + As a further example a test job may have an input runvar + "xenbuildjob" which specifies which build job produced the binary + to be tested. The "xen install" step can then read this runvar in + order to find the binary to install. + + Other runvars also exist covering things such as: + + * constraints on which machines in the test pool a job can be + run on (e.g. the architecure, the need for a particular + processor class, the presence of SR-IOV etc). + + * the parameters of the guest to test (e.g. distro, PV vs HVM + etc). + +Operation +========= + +A flight is constructed by the "make-flight" script. + +"make-flight" will allocate a new flight number, create a set of jobs +with input runvars depending on the configuration (e.g. branch/version +to test). + +A flight is run by the "mg-execute-flight" script, which in turn calls +"sg-execute-flight". "sg-execute-flight" then spawns an instance of +"sg-run-job" for each job in the flight. + +"sg-run-job" encodes various recipes (sequences of steps) which are +referenced by each job's configuration. It then runs each of these in +turn, taking into account the prerequisites etc, by calling the +relevant "ts-*" scripts. + +When running in standalone mode it is possible to run any of these +steps by hand, ("mg-execute-flight", "sg-run-job", "ts-*") although +you will need to find the correct inputs (some of which are documented +below) and perhaps take care of prerequisites yourself (e.g. running +"./sg-run-job test-armhf-armhf-xl" means you must have done +"./sg-runjob build-armhf" and "build-armhf-pvops" first. + +Results +======= + +For flights run automatically by the infrastructure an email report is +produced. For most normal flights this is mailed to the xen-devel +mailing list. The report for flight 24438 can be seen at + + http://lists.xen.org/archives/html/xen-devel/2014-01/msg01614.html + +The report will link to a set of complete logs. Since these are +regularly expired due to space constraints the logs for flight 24438 +have been archived to + + http://xenbits.xen.org/docs/osstest-output-example/24438/ + +NB: to save space any files larger than 100K have been replaced with a +placeholder. + +The results grid contains an overview of the flight's execution. + +The results for each job are reached by clicking the header of the +column in the results grid which will lead to reports such as: + + http://xenbits.xen.org/docs/osstest-output-example/24438/test-amd64-amd64-xl/info.html + http://xenbits.xen.org/docs/osstest-output-example/24438/build-amd64/info.html + +The job report contains all of the logs and build outputs associated +with this job. + +The logs for a step are reached by clicking the individual cells of +the results grid, or by clicking the list of steps on the job +report. In either case this will lead to a report such as + + http://xenbits.xen.org/docs/osstest-output-example/24438/test-amd64-amd64-xl/4.ts-xen-install.log + +Additional details (e.g. serial logs, guest cfg files, etc) will be +available in the complete logs associated with the containing job. + +The runvars are listed in the results page for the job as "Test +control variables". e.g. See the end of: + + http://xenbits.xen.org/docs/osstest-output-example/24438/test-amd64-amd64-xl/info.html + +In order to find the binaries which went into a test job you should +consult the results page for that job and find the relevant build +job. e.g. + + http://www.chiark.greenend.org.uk/~xensrcts/logs/24438/test-amd64-amd64-xl/info.html + +lists "xenbuildjob" as "build-amd64". Therefore the input binaries are +found at + + http://www.chiark.greenend.org.uk/~xensrcts/logs/24438/build-amd64/info.html + +which is linked from the top of the relevant column in the overview +grid. + +Script Naming Conventions +========================= + +Most of the scripts follow a naming convention: + +ap-*: Adhoc push scripts + +cr-*: Cron scripts + +cri-*: Cron scripts (internal) + +cs-*: Control Scripts + +mg-*: Management scripts + +ms-*: Management Services + +sg-*: ? + +ts-*: Test Step scripts. + +Jobs +==== + +The names of jobs follow some common patterns: + + build-$ARCH + + Build Xen for $ARCH + + build-$ARCH-xend + + Build Xen for $ARCH, with xend enabled + + build-$ARCH-pvops + + Build an upstream ("pvops") kernel for $ARCH + +build-$ARCH-oldkern + + Build the old "linux-2.6.18-xen" tree for $ARCH + +test-$XENARCH-$DOM0ARCH- + + A test running a $XENARCH hypervisor with a $DOM0ARCH + dom0. + + Some tests also have a -$DOMUARCH suffix indicating the + obvious thing. + +NB: $ARCH (and $XENARCH etc) are Debian arch names, i386, amd64, armhf. + +Standalone Mode +=============== + To run osstest in standalone mode: - You need to install @@ -18,7 +212,7 @@ To run osstest in standalone mode: gives you the "branch" consisting of tests run for the xen-unstable push gate. You need to select a job. The list of available jobs is that shown in the publicly emailed test reports on xen-devel, eg - http://lists.xen.org/archives/html/xen-devel/2013-08/msg02529.html + http://lists.xen.org/archives/html/xen-devel/2014-01/msg01614.html If you don't want to repro one of those and don't know how to choose a job, choose one of @@ -26,6 +220,12 @@ To run osstest in standalone mode: - Run ./standalone-reset + This will call "make-flight" for you to create a flight targetting + xen-unstable (this can be adjusted by passing parameters to + standalone-reset). By default the flight identifier is + "standalone". standalone-reset will also make sure that certain + bits of static data are available (e.g. Debian installer images) + - Then you can run ./sg-run-job to run that job on the default host. NB in most cases this will