From patchwork Sat Jul 11 17:07:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alex_Benn=C3=A9e?= X-Patchwork-Id: 235294 Delivered-To: patch@linaro.org Received: by 2002:a92:d244:0:0:0:0:0 with SMTP id v4csp1464582ilg; Sat, 11 Jul 2020 10:07:51 -0700 (PDT) X-Google-Smtp-Source: ABdhPJw2RWMuy1KqBjmAHBUhXebiBlLYClopxRbsaAf0CSFfWWt8xRkXWYWYhDneq6W6fE+1Fa3N X-Received: by 2002:a25:73d7:: with SMTP id o206mr109744936ybc.99.1594487270953; Sat, 11 Jul 2020 10:07:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1594487270; cv=none; d=google.com; s=arc-20160816; b=Nq2qv6U36hr4uYEZH0fC7yuJYrIxrQ95uUAfODAUQ04CFgFsGh3AAT0rPqmcQIp17J jhrSbfbY54W1qkoSH552FJ4ieuubrNQJBGe7Q7Y4pU1S8nH1yfJtJsKXw82O/IZvIbfh 3IDJNV4Kf0RKGDBNK23Rbb5uRqxN4hwgBkSAiUJvS6HFGniEFVClDJeb+hwqaW9gbVnN +q+RCFjZWac6WrwOeymCtq8Fr2okEYXGtOw54mQvVKzcP/Hna/VTP9ljDaVb6srDeXnq bOWWNwRT6+qCg6YysIKcHS9pjyu/HW66vKZWxyFQoa9g/KgA9sWGTO7GKgj83/l7bPcr ETFw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=geUpkH2ThEs8zqbxtFpdjObRhrd+cez0VcjEpzmTBhk=; b=o+7pyj+iQaz0+TUVd3EdGV+Eaido9DZaReBY6/zTGIMQsaHOdUJ+70bIJY7FgGK9TU 5cUQfHtC6PXLegZqXcUbkjvBlziyJNpUIv1tsTVc0E1fl4COJ8Kr+CL8BDQFRSC7eN7D dDMZBFSZSO/HCVH8wpsVm+sfhT4Ng2QIe6pdY7oM/UMEU+OnsQdyblQXyCSJEiTOqZ1D sKyTT+sRB3cSGsW7LrlrMzPVsjPsW6eMUtT0Ygzzjld58KFGjnzcwoyxDit6CGne2kdr FJlLy2Ox/oM0PvciUclwO7Fgmlg2FPyXVPfR5ktXxyu1PUEp2JwIxEEbOQJbNT47kd8v 4+Sw== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=WJmgAS3s; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id g16si9396919ybk.54.2020.07.11.10.07.50 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Sat, 11 Jul 2020 10:07:50 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=WJmgAS3s; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:40464 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1juIyc-0003NN-CG for patch@linaro.org; Sat, 11 Jul 2020 13:07:50 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33198) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1juIyR-0003NF-9V for qemu-devel@nongnu.org; Sat, 11 Jul 2020 13:07:39 -0400 Received: from mail-wr1-x42a.google.com ([2a00:1450:4864:20::42a]:44530) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1juIyP-0006hA-1k for qemu-devel@nongnu.org; Sat, 11 Jul 2020 13:07:39 -0400 Received: by mail-wr1-x42a.google.com with SMTP id b6so8968061wrs.11 for ; Sat, 11 Jul 2020 10:07:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=geUpkH2ThEs8zqbxtFpdjObRhrd+cez0VcjEpzmTBhk=; b=WJmgAS3sH3yZuIgGp3tBDoIbCOpxo7v65CqGxrZO0R0XTrQbvE+bx9mTUoMqqUXeYZ epShWfQeSvDp/UyOrD7Ca8vwJovvRdnWvN/v/ddG4501EjeMwu3ZuBL0bndqd0abYGqN DsWKx4vCxmLT8G7BoBIsEWUQ8rQfnH70w0Kn0LXORFy9Qq2AnuLpyudgX2DxykYdv3/K 1rar7SEY+RRu1Eq2gnyuqNxzSF9zvfef5UKkxZ4AbSdL957uGp0OXBTz9kzTraJp1PyO 3BPqAuiMlQf2UnjT4jGX0BTOgi9qZQVLA5Nqsnk+mSCwQw89KUo4/tgDRChzwrF4hslS IkZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=geUpkH2ThEs8zqbxtFpdjObRhrd+cez0VcjEpzmTBhk=; b=Mqo+qKGW2qZZsdm6LFXhSb3U3R2z1p1Qt6B8/iHe+TulMTWwcc1I9HS+HOek6LBLcq lGrLESZzRv33Ntnfa8aol1AsBZPXNFUesdeNmiTWGAX+1HNtQY4zQn5wFqXW3FS6Hai7 GpsE28d48rIyK7ZHgu9xCJSxfzoLE7RvPG6xFxDIadBnoROekkNJ6NYnVi357ztQxljr qPlFdHV++WJuSj1BT0cOn0/xDWgfFdEwhySsY7WgDCjZ/auIsA4idQL4XDCG5uhSSbBv wKfln+hrUKqwa1JTI6Fwdfv7KvfXQ202ujtaEEShJ5c0z3MjHgbN8z1z2Qe96tLltYjE mRQA== X-Gm-Message-State: AOAM5304T3vD/PAW5bjRcjtrlqBIMI63zr5dOYVg030dQyx7nuQ5xOFi kPs0UM4qa2RyeWtgzGjQ6oWN0w== X-Received: by 2002:adf:b312:: with SMTP id j18mr68408528wrd.195.1594487255446; Sat, 11 Jul 2020 10:07:35 -0700 (PDT) Received: from zen.linaroharston ([51.148.130.216]) by smtp.gmail.com with ESMTPSA id h84sm15694997wme.22.2020.07.11.10.07.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 11 Jul 2020 10:07:32 -0700 (PDT) Received: from zen.lan (localhost [127.0.0.1]) by zen.linaroharston (Postfix) with ESMTP id 3A7FC1FF87; Sat, 11 Jul 2020 18:07:32 +0100 (BST) From: =?utf-8?q?Alex_Benn=C3=A9e?= To: peter.maydell@linaro.org Subject: [PULL v2 04/50] tests/vm: Add configuration to basevm.py Date: Sat, 11 Jul 2020 18:07:23 +0100 Message-Id: <20200711170732.26213-2-alex.bennee@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200711170732.26213-1-alex.bennee@linaro.org> References: <20200711170732.26213-1-alex.bennee@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::42a; envelope-from=alex.bennee@linaro.org; helo=mail-wr1-x42a.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Robert Foley , =?utf-8?q?Alex_Benn=C3=A9e?= , qemu-devel@nongnu.org, Peter Puhov , =?utf-8?q?Philippe_Mathieu-Daud?= =?utf-8?b?w6k=?= Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" From: Robert Foley Added use of a configuration to tests/vm/basevm.py. The configuration provides parameters used to configure a VM. This allows for providing alternate configurations to the VM being created/launched. cpu, machine, memory, and NUMA configuration are all examples of configuration which we might want to vary on the VM being created or launched. This will for example allow for creating an aarch64 vm. Signed-off-by: Robert Foley Reviewed-by: Peter Puhov Reviewed-by: Alex Bennée Signed-off-by: Alex Bennée Message-Id: <20200601211421.1277-3-robert.foley@linaro.org> Message-Id: <20200701135652.1366-7-alex.bennee@linaro.org> -- 2.20.1 diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py index 5a58e6c393..5ae39ad113 100644 --- a/tests/vm/basevm.py +++ b/tests/vm/basevm.py @@ -29,16 +29,41 @@ import tempfile import shutil import multiprocessing import traceback - -SSH_KEY = open(os.path.join(os.path.dirname(__file__), - "..", "keys", "id_rsa")).read() -SSH_PUB_KEY = open(os.path.join(os.path.dirname(__file__), - "..", "keys", "id_rsa.pub")).read() - +import shlex + +SSH_KEY_FILE = os.path.join(os.path.dirname(__file__), + "..", "keys", "id_rsa") +SSH_PUB_KEY_FILE = os.path.join(os.path.dirname(__file__), + "..", "keys", "id_rsa.pub") + +# This is the standard configuration. +# Any or all of these can be overridden by +# passing in a config argument to the VM constructor. +DEFAULT_CONFIG = { + 'cpu' : "max", + 'machine' : 'pc', + 'guest_user' : "qemu", + 'guest_pass' : "qemupass", + 'root_pass' : "qemupass", + 'ssh_key_file' : SSH_KEY_FILE, + 'ssh_pub_key_file': SSH_PUB_KEY_FILE, + 'memory' : "4G", + 'extra_args' : [], + 'qemu_args' : "", + 'dns' : "", + 'ssh_port' : 0, + 'install_cmds' : "", + 'boot_dev_type' : "block", + 'ssh_timeout' : 1, +} +BOOT_DEVICE = { + 'block' : "-drive file={},if=none,id=drive0,cache=writeback "\ + "-device virtio-blk,drive=drive0,bootindex=0", + 'scsi' : "-device virtio-scsi-device,id=scsi "\ + "-drive file={},format=raw,if=none,id=hd0 "\ + "-device scsi-hd,drive=hd0,bootindex=0", +} class BaseVM(object): - GUEST_USER = "qemu" - GUEST_PASS = "qemupass" - ROOT_PASS = "qemupass" envvars = [ "https_proxy", @@ -57,25 +82,38 @@ class BaseVM(object): poweroff = "poweroff" # enable IPv6 networking ipv6 = True + # This is the timeout on the wait for console bytes. + socket_timeout = 120 # Scale up some timeouts under TCG. # 4 is arbitrary, but greater than 2, # since we found we need to wait more than twice as long. tcg_ssh_timeout_multiplier = 4 - def __init__(self, args): + def __init__(self, args, config=None): self._guest = None self._genisoimage = args.genisoimage self._build_path = args.build_path + # Allow input config to override defaults. + self._config = DEFAULT_CONFIG.copy() + if config != None: + self._config.update(config) + self.validate_ssh_keys() self._tmpdir = os.path.realpath(tempfile.mkdtemp(prefix="vm-test-", suffix=".tmp", dir=".")) atexit.register(shutil.rmtree, self._tmpdir) - - self._ssh_key_file = os.path.join(self._tmpdir, "id_rsa") - open(self._ssh_key_file, "w").write(SSH_KEY) - subprocess.check_call(["chmod", "600", self._ssh_key_file]) - - self._ssh_pub_key_file = os.path.join(self._tmpdir, "id_rsa.pub") - open(self._ssh_pub_key_file, "w").write(SSH_PUB_KEY) + # Copy the key files to a temporary directory. + # Also chmod the key file to agree with ssh requirements. + self._config['ssh_key'] = \ + open(self._config['ssh_key_file']).read().rstrip() + self._config['ssh_pub_key'] = \ + open(self._config['ssh_pub_key_file']).read().rstrip() + self._ssh_tmp_key_file = os.path.join(self._tmpdir, "id_rsa") + open(self._ssh_tmp_key_file, "w").write(self._config['ssh_key']) + subprocess.check_call(["chmod", "600", self._ssh_tmp_key_file]) + + self._ssh_tmp_pub_key_file = os.path.join(self._tmpdir, "id_rsa.pub") + open(self._ssh_tmp_pub_key_file, + "w").write(self._config['ssh_pub_key']) self.debug = args.debug self._stderr = sys.stderr @@ -84,11 +122,14 @@ class BaseVM(object): self._stdout = sys.stdout else: self._stdout = self._devnull + netdev = "user,id=vnet,hostfwd=:127.0.0.1:{}-:22" self._args = [ \ - "-nodefaults", "-m", "4G", - "-cpu", "max", - "-netdev", "user,id=vnet,hostfwd=:127.0.0.1:0-:22" + - (",ipv6=no" if not self.ipv6 else ""), + "-nodefaults", "-m", self._config['memory'], + "-cpu", self._config['cpu'], + "-netdev", + netdev.format(self._config['ssh_port']) + + (",ipv6=no" if not self.ipv6 else "") + + (",dns=" + self._config['dns'] if self._config['dns'] else ""), "-device", "virtio-net-pci,netdev=vnet", "-vnc", "127.0.0.1:0,to=20"] if args.jobs and args.jobs > 1: @@ -99,6 +140,55 @@ class BaseVM(object): logging.info("KVM not available, not using -enable-kvm") self._data_args = [] + if self._config['qemu_args'] != None: + qemu_args = self._config['qemu_args'] + qemu_args = qemu_args.replace('\n',' ').replace('\r','') + # shlex groups quoted arguments together + # we need this to keep the quoted args together for when + # the QEMU command is issued later. + args = shlex.split(qemu_args) + self._config['extra_args'] = [] + for arg in args: + if arg: + # Preserve quotes around arguments. + # shlex above takes them out, so add them in. + if " " in arg: + arg = '"{}"'.format(arg) + self._config['extra_args'].append(arg) + + def validate_ssh_keys(self): + """Check to see if the ssh key files exist.""" + if 'ssh_key_file' not in self._config or\ + not os.path.exists(self._config['ssh_key_file']): + raise Exception("ssh key file not found.") + if 'ssh_pub_key_file' not in self._config or\ + not os.path.exists(self._config['ssh_pub_key_file']): + raise Exception("ssh pub key file not found.") + + def wait_boot(self, wait_string=None): + """Wait for the standard string we expect + on completion of a normal boot. + The user can also choose to override with an + alternate string to wait for.""" + if wait_string is None: + if self.login_prompt is None: + raise Exception("self.login_prompt not defined") + wait_string = self.login_prompt + # Intentionally bump up the default timeout under TCG, + # since the console wait below takes longer. + timeout = self.socket_timeout + if not kvm_available(self.arch): + timeout *= 8 + self.console_init(timeout=timeout) + self.console_wait(wait_string) + + def __getattr__(self, name): + # Support direct access to config by key. + # for example, access self._config['cpu'] by self.cpu + if name.lower() in self._config.keys(): + return self._config[name.lower()] + return object.__getattribute__(self, name) + def _download_with_cache(self, url, sha256sum=None, sha512sum=None): def check_sha256sum(fname): if not sha256sum: @@ -130,8 +220,9 @@ class BaseVM(object): "-t", "-o", "StrictHostKeyChecking=no", "-o", "UserKnownHostsFile=" + os.devnull, - "-o", "ConnectTimeout=1", - "-p", self.ssh_port, "-i", self._ssh_key_file] + "-o", + "ConnectTimeout={}".format(self._config["ssh_timeout"]), + "-p", self.ssh_port, "-i", self._ssh_tmp_key_file] # If not in debug mode, set ssh to quiet mode to # avoid printing the results of commands. if not self.debug: @@ -180,14 +271,14 @@ class BaseVM(object): "virtio-blk,drive=%s,serial=%s,bootindex=1" % (name, name)] def boot(self, img, extra_args=[]): - args = self._args + [ - "-drive", "file=%s,if=none,id=drive0,cache=writeback" % img, - "-device", "virtio-blk,drive=drive0,bootindex=0"] - args += self._data_args + extra_args + boot_dev = BOOT_DEVICE[self._config['boot_dev_type']] + boot_params = boot_dev.format(img) + args = self._args + boot_params.split(' ') + args += self._data_args + extra_args + self._config['extra_args'] logging.debug("QEMU args: %s", " ".join(args)) qemu_path = get_qemu_path(self.arch, self._build_path) guest = QEMUMachine(binary=qemu_path, args=args) - guest.set_machine('pc') + guest.set_machine(self._config['machine']) guest.set_console() try: guest.launch() @@ -301,7 +392,8 @@ class BaseVM(object): self.console_send(command) def console_ssh_init(self, prompt, user, pw): - sshkey_cmd = "echo '%s' > .ssh/authorized_keys\n" % SSH_PUB_KEY.rstrip() + sshkey_cmd = "echo '%s' > .ssh/authorized_keys\n" \ + % self._config['ssh_pub_key'].rstrip() self.console_wait_send("login:", "%s\n" % user) self.console_wait_send("Password:", "%s\n" % pw) self.console_wait_send(prompt, "mkdir .ssh\n") @@ -360,23 +452,23 @@ class BaseVM(object): "local-hostname: {}-guest\n".format(name)]) mdata.close() udata = open(os.path.join(cidir, "user-data"), "w") - print("guest user:pw {}:{}".format(self.GUEST_USER, - self.GUEST_PASS)) + print("guest user:pw {}:{}".format(self._config['guest_user'], + self._config['guest_pass'])) udata.writelines(["#cloud-config\n", "chpasswd:\n", " list: |\n", - " root:%s\n" % self.ROOT_PASS, - " %s:%s\n" % (self.GUEST_USER, - self.GUEST_PASS), + " root:%s\n" % self._config['root_pass'], + " %s:%s\n" % (self._config['guest_user'], + self._config['guest_pass']), " expire: False\n", "users:\n", - " - name: %s\n" % self.GUEST_USER, + " - name: %s\n" % self._config['guest_user'], " sudo: ALL=(ALL) NOPASSWD:ALL\n", " ssh-authorized-keys:\n", - " - %s\n" % SSH_PUB_KEY, + " - %s\n" % self._config['ssh_pub_key'], " - name: root\n", " ssh-authorized-keys:\n", - " - %s\n" % SSH_PUB_KEY, + " - %s\n" % self._config['ssh_pub_key'], "locale: en_US.UTF-8\n"]) proxy = os.environ.get("http_proxy") if not proxy is None: @@ -447,15 +539,17 @@ def parse_args(vmcls): parser.disable_interspersed_args() return parser.parse_args() -def main(vmcls): +def main(vmcls, config=None): try: + if config == None: + config = DEFAULT_CONFIG args, argv = parse_args(vmcls) if not argv and not args.build_qemu and not args.build_image: print("Nothing to do?") return 1 logging.basicConfig(level=(logging.DEBUG if args.debug else logging.WARN)) - vm = vmcls(args) + vm = vmcls(args, config=config) if args.build_image: if os.path.exists(args.image) and not args.force: sys.stderr.writelines(["Image file exists: %s\n" % args.image,