From patchwork Fri Jun 12 18:03:37 2015
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Patchwork-Submitter: Peter Maydell <peter.maydell@linaro.org>
X-Patchwork-Id: 49834
Return-Path: <patchwork-forward+bncBC6Z756YVMIBBN565SVQKGQEQWCA7HY@linaro.org>
X-Original-To: linaro@patches.linaro.org
Delivered-To: linaro@patches.linaro.org
Received: from mail-lb0-f198.google.com (mail-lb0-f198.google.com
 [209.85.217.198])
 by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id B0849211FD
 for <linaro@patches.linaro.org>; Fri, 12 Jun 2015 18:04:40 +0000 (UTC)
Received: by lbcak1 with SMTP id ak1sf11544854lbc.2
 for <linaro@patches.linaro.org>; Fri, 12 Jun 2015 11:04:39 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20130820;
 h=x-gm-message-state:delivered-to:mime-version:in-reply-to:references
 :from:date:message-id:to:content-type:content-transfer-encoding:cc
 :subject:precedence:list-id:list-unsubscribe:list-archive:list-post
 :list-help:list-subscribe:errors-to:sender:x-original-sender
 :x-original-authentication-results:mailing-list;
 bh=t2CCbNQCSSCwkPvn7EuBW/z6sRuTAHuFrgOu/vJVtWs=;
 b=mBC9f4qL9JSZiKuUbuShHS/0R2r/70UFbyYhiSlZQbIFGtWBLM2YvRMLE6usq1zTUR
 Xgua38JM/jfuu+Xvl3wtjFFgOTsEpAJYDS8zXKah5EXZU5+Sih14HxbdJPwiDm3yzlvh
 u16g9iLcWL+PeykdL4UPF8rXKyyIRpxq3DnLjJttWMLGz8zbLpOxWvK2fRv5oS/ogHQu
 inQGtHwmxmG6wRjysmz82fvcMGETu4m0dRBuMH8rjbFP7jyXWew9SoEvWVAmaCV5MZSb
 3hpWVgVf2ZGfI8Eqs5iF74tGDJ+kokLuCuO+9Qbsj+t5QaxNYV4aaHAEg1MSgZJRNb8T
 XSlA==
X-Gm-Message-State: ALoCoQkoVsKyavoMC93KXgesb7wi9/WgMIVOH9QqyBWCKsc43+VjbPy4msa8t/QS+tF3tgg8dz3v
X-Received: by 10.194.236.230 with SMTP id ux6mr15458211wjc.5.1434132279715; 
 Fri, 12 Jun 2015 11:04:39 -0700 (PDT)
X-BeenThere: patchwork-forward@linaro.org
Received: by 10.152.1.73 with SMTP id 9ls498181lak.17.gmail; Fri, 12 Jun 2015
 11:04:39 -0700 (PDT)
X-Received: by 10.152.5.227 with SMTP id v3mr16580118lav.122.1434132279477; 
 Fri, 12 Jun 2015 11:04:39 -0700 (PDT)
Received: from mail-lb0-f174.google.com (mail-lb0-f174.google.com.
 [209.85.217.174]) by mx.google.com with ESMTPS id
 y5si3953613lay.123.2015.06.12.11.04.39
 for <patchwork-forward@linaro.org>
 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
 Fri, 12 Jun 2015 11:04:39 -0700 (PDT)
Received-SPF: pass (google.com: domain of
 patch+caf_=patchwork-forward=linaro.org@linaro.org designates
 209.85.217.174 as permitted sender) client-ip=209.85.217.174; 
Received: by lbbqq2 with SMTP id qq2so23809750lbb.3
 for <patchwork-forward@linaro.org>;
 Fri, 12 Jun 2015 11:04:39 -0700 (PDT)
X-Received: by 10.152.7.7 with SMTP id f7mr16470026laa.106.1434132279374;
 Fri, 12 Jun 2015 11:04:39 -0700 (PDT)
X-Forwarded-To: patchwork-forward@linaro.org
X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org
Delivered-To: patch@linaro.org
Received: by 10.112.108.230 with SMTP id hn6csp728684lbb;
 Fri, 12 Jun 2015 11:04:38 -0700 (PDT)
X-Received: by 10.140.150.78 with SMTP id 75mr21299745qhw.10.1434132277973; 
 Fri, 12 Jun 2015 11:04:37 -0700 (PDT)
Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11])
 by mx.google.com with ESMTPS id
 f132si4514189qhc.118.2015.06.12.11.04.37 for <patch@linaro.org>
 (version=TLSv1 cipher=RC4-SHA bits=128/128);
 Fri, 12 Jun 2015 11:04:37 -0700 (PDT)
Received-SPF: pass (google.com: domain of
 qemu-devel-bounces+patch=linaro.org@nongnu.org designates
 2001:4830:134:3::11 as permitted sender)
 client-ip=2001:4830:134:3::11; 
Received: from localhost ([::1]:53155 helo=lists.gnu.org)
 by lists.gnu.org with esmtp (Exim 4.71)
 (envelope-from <qemu-devel-bounces+patch=linaro.org@nongnu.org>)
 id 1Z3TJl-0002Zm-0B
 for patch@linaro.org; Fri, 12 Jun 2015 14:04:37 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:33222)
 by lists.gnu.org with esmtp (Exim 4.71)
 (envelope-from <peter.maydell@linaro.org>) id 1Z3TJC-0002ET-4d
 for qemu-devel@nongnu.org; Fri, 12 Jun 2015 14:04:06 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
 (envelope-from <peter.maydell@linaro.org>) id 1Z3TJ8-0006dV-NB
 for qemu-devel@nongnu.org; Fri, 12 Jun 2015 14:04:02 -0400
Received: from mail-vn0-f51.google.com ([209.85.216.51]:35464)
 by eggs.gnu.org with esmtp (Exim 4.71)
 (envelope-from <peter.maydell@linaro.org>) id 1Z3TJ8-0006dR-IC
 for qemu-devel@nongnu.org; Fri, 12 Jun 2015 14:03:58 -0400
Received: by vnbf129 with SMTP id f129so7255055vnb.2
 for <qemu-devel@nongnu.org>; Fri, 12 Jun 2015 11:03:58 -0700 (PDT)
X-Received: by 10.52.114.42 with SMTP id jd10mr27701383vdb.90.1434132238033; 
 Fri, 12 Jun 2015 11:03:58 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.31.153.135 with HTTP; Fri, 12 Jun 2015 11:03:37 -0700 (PDT)
In-Reply-To: <557B0B21.2030009@hs-rm.de>
References: <557B0B21.2030009@hs-rm.de>
From: Peter Maydell <peter.maydell@linaro.org>
Date: Fri, 12 Jun 2015 19:03:37 +0100
Message-ID: <CAFEAcA8iCmgwgmhTkRryTyVBSxS+Cbh6bGSLqt7EPM028F1tUg@mail.gmail.com>
To: =?UTF-8?Q?Alex_Z=C3=BCpke?= <alexander.zuepke@hs-rm.de>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
 [fuzzy]
X-Received-From: 209.85.216.51
Cc: QEMU Developers <qemu-devel@nongnu.org>
Subject: Re: [Qemu-devel] QEMU ARM SMP: IPI delivery delayed until next main
 loop event // how to improve IPI latency?
X-BeenThere: qemu-devel@nongnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: <patchwork-forward.linaro.org>
List-Unsubscribe: <mailto:googlegroups-manage+836684582541+unsubscribe@googlegroups.com>, 
 <http://groups.google.com/a/linaro.org/group/patchwork-forward/subscribe>
List-Archive: <http://groups.google.com/a/linaro.org/group/patchwork-forward/>
List-Post: <http://groups.google.com/a/linaro.org/group/patchwork-forward/post>, 
 <mailto:patchwork-forward@linaro.org>
List-Help: <http://support.google.com/a/linaro.org/bin/topic.py?topic=25838>, 
 <mailto:patchwork-forward+help@linaro.org>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
 <mailto:qemu-devel-request@nongnu.org?subject=subscribe>
Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org
Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org
X-Removed-Original-Auth: Dkim didn't pass.
X-Original-Sender: peter.maydell@linaro.org
X-Original-Authentication-Results: mx.google.com; spf=pass (google.com:
 domain of
 patch+caf_=patchwork-forward=linaro.org@linaro.org designates
 209.85.217.174 as permitted sender)
 smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org
Mailing-list: list patchwork-forward@linaro.org;
 contact patchwork-forward+owners@linaro.org
X-Google-Group-Id: 836684582541

On 12 June 2015 at 17:38, Alex Züpke <alexander.zuepke@hs-rm.de> wrote:
> Hi,
>
> I'm benchmarking some IPI (== inter-processor-interrupt) synchronization stuff of my custom kernel on QEMU ARM (qemu-system-arm -M vexpress-a15 -smp 2) and ran into the following problem: pending IPIs are delayed until the QEMU main loop receives an event (for example the timer interrupt expires or I press a key on the console).
>
> The following timing diagram tries to show this:
>
>   CPU #0                       CPU #1
>   ======                       ======
>   ... other stuff ...          WFI (wait for interrupt, like x86 "HLT")
>   send SGI in MPCore
>   polls for completeness
>                  <time passes ...>
>   polls ...
>                  <... and passes ...>
>   still polls ...
>                  <... and passes ...>
>   still polls ...
>                  <... and passes ...>
>
>
>                  <timer interrupt expires>
>                  <now QEMU switches to CPU #1>
>                                receives IPI
>                                signals completeness
>                                WFI
>                  <QEMU switches to CPU #0>
>   polling done
>   process timer interrupt
>   ...

Right. The problem is that we don't have any way of telling
that CPU 0 is just sat busy waiting for CPU 1.

> It works as expects (I get thousands of IPIs per second now), but
> it does not "feel right", so is there a better way to improve the
> responsiveness of IPI handling in QEMU?

Probably the best approach would be to have something in
arm_cpu_set_irq() which says "if we are CPU X and we've
just caused an interrupt to be set for CPU Y, then we
should ourselves yield back to the main loop".

Something like this, maybe, though I have done no more testing
than checking it doesn't actively break kernel booting :-)

 static void arm_cpu_kvm_set_irq(void *opaque, int irq, int level)


-- PMM

--- a/target-arm/cpu.c
+++ b/target-arm/cpu.c
@@ -325,6 +325,18 @@ static void arm_cpu_set_irq(void *opaque, int
irq, int level)
     default:
         hw_error("arm_cpu_set_irq: Bad interrupt line %d\n", irq);
     }
+
+    /* If we are currently executing code for CPU X, and this
+     * CPU we've just triggered an interrupt on is CPU Y, then
+     * make CPU X yield control back to the main loop at the
+     * end of the TB it's currently executing.
+     * This avoids problems where the interrupt was an IPI
+     * and CPU X would otherwise sit busy looping for the rest
+     * of its timeslice because Y hasn't had a chance to run.
+     */
+    if (current_cpu && current_cpu != cs) {
+        cpu_exit(current_cpu);
+    }
 }