From patchwork Mon Jul 21 14:13:40 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefano Stabellini X-Patchwork-Id: 33966 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-pa0-f70.google.com (mail-pa0-f70.google.com [209.85.220.70]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id A8AE420492 for ; Mon, 21 Jul 2014 14:15:07 +0000 (UTC) Received: by mail-pa0-f70.google.com with SMTP id lf10sf55897847pab.1 for ; Mon, 21 Jul 2014 07:15:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:date:from:to:cc:subject:in-reply-to :message-id:references:user-agent:mime-version:sender:precedence :list-id:x-original-sender:x-original-authentication-results :mailing-list:list-post:list-help:list-archive:list-unsubscribe :content-type; bh=/R2osWYUXfNFhh8W3LLl2i7L2RUfeOILqlfK8ON7SgE=; b=GuQapI9WNPWQyk/P3nUuQ3WwSkTOh+CkJ/9iF4ZmH9ofX06AgwxQaT9zV9VGKrxaeD wQNXqDxXR0pw0BsrHKMHa1GRkNkm0QR6v9Ew8JS2ohfn13Ls3D3e/Jz98sKVTXNu24NP 8shrtKL5DGxR8pAr2lnzfhDTtn9Ub5jWZFuVONle0J4EDywo86KzH2LA8PrHxWsh3sQk 8QRxSSY+rfroZG/At9UVcXfHZyVoCeqkawwDhxLDxhX59KR2IGyRDlvAu/XiIjjW2Jz0 Re66X4NUzxostkjT9UtpU+9lXZdySyawMFNNDkc6B31m5zdvrQeclsLeeYQZExO5aXuL s90w== X-Gm-Message-State: ALoCoQn8NTNDy9Pev37cBinjTQaLXn5sOaMbgGvAR22BQUzuJ/1WswZzdTGrIJakW+55ZDlQ/nyG X-Received: by 10.66.144.228 with SMTP id sp4mr12543467pab.5.1405952107009; Mon, 21 Jul 2014 07:15:07 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.140.83.233 with SMTP id j96ls1103504qgd.80.gmail; Mon, 21 Jul 2014 07:15:06 -0700 (PDT) X-Received: by 10.52.185.72 with SMTP id fa8mr25020822vdc.4.1405952106795; Mon, 21 Jul 2014 07:15:06 -0700 (PDT) Received: from mail-vc0-f174.google.com (mail-vc0-f174.google.com [209.85.220.174]) by mx.google.com with ESMTPS id s10si11416281vcx.51.2014.07.21.07.15.06 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 21 Jul 2014 07:15:06 -0700 (PDT) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.220.174 as permitted sender) client-ip=209.85.220.174; Received: by mail-vc0-f174.google.com with SMTP id la4so12238017vcb.19 for ; Mon, 21 Jul 2014 07:15:06 -0700 (PDT) X-Received: by 10.52.248.146 with SMTP id ym18mr25208716vdc.8.1405952106621; Mon, 21 Jul 2014 07:15:06 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.221.37.5 with SMTP id tc5csp115663vcb; Mon, 21 Jul 2014 07:15:05 -0700 (PDT) X-Received: by 10.66.122.135 with SMTP id ls7mr25805378pab.84.1405952105053; Mon, 21 Jul 2014 07:15:05 -0700 (PDT) Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id zm3si14389755pac.97.2014.07.21.07.14.55 for ; Mon, 21 Jul 2014 07:14:55 -0700 (PDT) Received-SPF: none (google.com: linux-kernel-owner@vger.kernel.org does not designate permitted sender hosts) client-ip=209.132.180.67; Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933015AbaGUOOo (ORCPT + 12 others); Mon, 21 Jul 2014 10:14:44 -0400 Received: from smtp.citrix.com ([66.165.176.89]:28727 "EHLO SMTP.CITRIX.COM" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932601AbaGUOOk (ORCPT ); Mon, 21 Jul 2014 10:14:40 -0400 X-IronPort-AV: E=Sophos;i="5.01,701,1400025600"; d="scan'208";a="154219790" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 21 Jul 2014 14:14:38 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.3.181.6; Mon, 21 Jul 2014 10:14:37 -0400 Received: from kaball.uk.xensource.com ([10.80.2.59]) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1X9EMO-0005Or-UG; Mon, 21 Jul 2014 15:14:36 +0100 Date: Mon, 21 Jul 2014 15:13:40 +0100 From: Stefano Stabellini X-X-Sender: sstabellini@kaball.uk.xensource.com To: Konrad Rzeszutek Wilk CC: Vitaly Kuznetsov , , , Boris Ostrovsky , David Vrabel , Andrew Jones , Subject: Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec In-Reply-To: <20140716134050.GH19585@laptop.dumpdata.com> Message-ID: References: <1405431640-649-1-git-send-email-vkuznets@redhat.com> <1405431640-649-5-git-send-email-vkuznets@redhat.com> <20140715152105.GP3403@laptop.dumpdata.com> <87fvi1u16k.fsf@vitty.brq.redhat.com> <20140716134050.GH19585@laptop.dumpdata.com> User-Agent: Alpine 2.02 (DEB 1266 2009-07-14) MIME-Version: 1.0 X-DLP: MIA1 Sender: linux-kernel-owner@vger.kernel.org Precedence: list List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: stefano.stabellini@eu.citrix.com X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.220.174 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , On Wed, 16 Jul 2014, Konrad Rzeszutek Wilk wrote: > On Wed, Jul 16, 2014 at 11:01:55AM +0200, Vitaly Kuznetsov wrote: > > Konrad Rzeszutek Wilk writes: > > > > > On Tue, Jul 15, 2014 at 03:40:40PM +0200, Vitaly Kuznetsov wrote: > > >> When kexec was peformed MSI IRQs for passthrough-ed devices were already > > >> mapped and we see non-zero pirq extracted from MSI msg. xen_irq_from_pirq() > > >> fails as we have no IRQ mapping information for that. Requesting for new > > >> mapping with __write_msi_msg() does not result in MSI IRQ being remapped so > > >> we don't recieve these IRQs. > > > > > > receive > > > > > > > Thanks for your comments! > > Thank you for quick turnaround with the answers! > > > > > How come '__write_msi_msg' does not result in new MSI IRQs? > > > > > > > Actually that was the hidden question in my RFC :-) > > > > Let me describe what I see. When normal boot is performed we have the > > following in xen_hvm_setup_msi_irqs(): > > > > __read_msi_msg() > > pirq -> 0 > > > > then we allocate new pirq with > > pirq = xen_allocate_pirq_msi() > > pirq -> 54 > > > > and we have the following mapping: > > xen: msi --> pirq=54 --> irq=72 > > > > in 'xl debug-keys i': > > (XEN) IRQ: 29 affinity:04 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(----), > > > > After kexec we see the following: > > __read_msi_msg() > > pirq -> 54 > > > > but as xen_irq_from_pirq() fails we follow the same path allocating new pirq: > > pirq = xen_allocate_pirq_msi() > > pirq -> 55 > > > > and we have the following mapping: > > xen: msi --> pirq=55 --> irq=75 > > > > However (afaict) mapping in xen wasn't updated: > > > > in 'xl debug-keys i': > > (XEN) IRQ: 29 affinity:02 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(--M-), > > I am wondering if that is related to in QEMU traditional: > > qemu-xen-trad: free all the pirqs for msi/msix when driver unloads > > (which in the upstream QEMU is 1d4fd4f0e2fc5dcae0c60e00cc9af95f52988050) > > If you have that patch in, is the PIRQ value correctly updated? > > > > > > Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) > > > from the device and updating the internal PIRQ<->IRQ code to match > > > with the reality? > > > > > > > Yea, 'always trust the device'. > > > > >> > > >> RFC: I wasn't able to understand why commit af42b8d1 which introduced > > >> xen_irq_from_pirq() check in xen_hvm_setup_msi_irqs() is checking that instead > > >> of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) > > >> we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also > > >> not sure when __write_msi_msg() with new PIRQ will result in new mapping. > > > > > > We don't request a new pirq. We end up returning before we call xen_allocate_pirq_msi. > > > At least that is how the commit you mentioned worked. > > > > > > > I meant to say that in case we have pirq > 0 from __read_msi_msg() but > > xen_irq_from_pirq(pirq) fails (kexec-only case?) we always do > > xen_allocate_pirq_msi() which brings us new pirq. > > > > > In regards to why using 'xen_irq_from_pirq' instead of just checking the PIRQ - is > > > that we might be called twice by a buggy driver. As such we want to check > > > our PIRQ<->IRQ to figure this out. > > > > But if we're called twice we'll see the same pirq, right? Or there are > > Good point. > > some cases when we see 'crap' instead of pirq here? > > For PCI passthrough devices they will be zero until they are enabled. > But I am not sure about the emulated devices, such as e1000 or such, which > would also go through this path (I think - do we have MSI devices that > we emulate in QEMU?) > > > > > I think it would be nice to use the same pirq after kexec instead of > > allocating a new one even in case we can make remapping work. > > I concur. > > Stefano, do you recall why you used xen_irq_from_pirq instead of just > trusting the 'pirq' value? Was it to workaround broken QEMU? If I recall correctly the problem is that pirq == 0 is a valid pirq number. So the check pirq <= 0 is wrong. Can we rely on the fact that msg.data is always 0 on first read? If so, then we could simply: --- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ diff --git a/arch/x86/pci/xen.c b/arch/x86/pci/xen.c index 905956f..d824743 100644 --- a/arch/x86/pci/xen.c +++ b/arch/x86/pci/xen.c @@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type) __read_msi_msg(msidesc, &msg); pirq = MSI_ADDR_EXT_DEST_ID(msg.address_hi) | ((msg.address_lo >> MSI_ADDR_DEST_ID_SHIFT) & 0xff); - if (msg.data != XEN_PIRQ_MSI_DATA || - xen_irq_from_pirq(pirq) < 0) { + if (msg.data != XEN_PIRQ_MSI_DATA) { pirq = xen_allocate_pirq_msi(dev, msidesc); if (pirq < 0) { irq = -ENODEV;