From patchwork Wed Nov 19 11:29:10 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 41147 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-lb0-f197.google.com (mail-lb0-f197.google.com [209.85.217.197]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 4F667241C9 for ; Wed, 19 Nov 2014 11:29:39 +0000 (UTC) Received: by mail-lb0-f197.google.com with SMTP id b6sf273423lbj.0 for ; Wed, 19 Nov 2014 03:29:38 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:date:from:to:cc:subject:message-id :references:mime-version:in-reply-to:user-agent:sender:precedence :list-id:x-original-sender:x-original-authentication-results :mailing-list:list-post:list-help:list-archive:list-unsubscribe :content-type:content-disposition; bh=rfewQZWce/QdxsN3Uud0BUHEUKHtPeT/fhNhfqwCwHo=; b=A8i8xqqYAH88RwF9sbisD/ckBI3IWTjHp8HORkeIFCble/AL2pQ4c8ewY3IXSfPyqL zuziwDw2ijnknSOg4sxO1FGrlvEQ+JEakKyScyYhvfZ5bKF4pGER6QTTXaIsOZzEdWSS 8dzpO3kkhrlluqG8Cp+Ww6DkkhImrBTj5ycEbpKVHm9qGvJWPq3TsubpPqWBOPkOO9St 4+QiGXEJjYPPPtg+dUVkfD4gPMwz2dCuUncDG/+A5sl1X7fbFS95Z7I9ZBTax8s1UNwl 3t/KePdGj3s4Q76ywcUcqWKiEVF0FfOYXzJwYSNnHfrhVUAOZI+ZDjUZ9dvX7MjeC+fs m94w== X-Gm-Message-State: ALoCoQnxy03e32Zc94VWshed3ZmclPZfsiJ8lADUKrS9wJYV1vP1GCAqSOxBvNxab7AHPy/kijHI X-Received: by 10.112.142.36 with SMTP id rt4mr16110838lbb.3.1416396578339; Wed, 19 Nov 2014 03:29:38 -0800 (PST) X-BeenThere: patchwork-forward@linaro.org Received: by 10.152.23.4 with SMTP id i4ls1499585laf.59.gmail; Wed, 19 Nov 2014 03:29:37 -0800 (PST) X-Received: by 10.152.120.167 with SMTP id ld7mr2238803lab.77.1416396577847; Wed, 19 Nov 2014 03:29:37 -0800 (PST) Received: from mail-la0-f42.google.com (mail-la0-f42.google.com. [209.85.215.42]) by mx.google.com with ESMTPS id um10si1450938lbb.117.2014.11.19.03.29.37 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 19 Nov 2014 03:29:37 -0800 (PST) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.215.42 as permitted sender) client-ip=209.85.215.42; Received: by mail-la0-f42.google.com with SMTP id s18so335003lam.1 for ; Wed, 19 Nov 2014 03:29:37 -0800 (PST) X-Received: by 10.152.6.228 with SMTP id e4mr4817181laa.71.1416396577700; Wed, 19 Nov 2014 03:29:37 -0800 (PST) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.112.184.201 with SMTP id ew9csp65958lbc; Wed, 19 Nov 2014 03:29:36 -0800 (PST) X-Received: by 10.68.225.162 with SMTP id rl2mr19961251pbc.129.1416396575132; Wed, 19 Nov 2014 03:29:35 -0800 (PST) Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id fm3si2291329pab.94.2014.11.19.03.29.34 for ; Wed, 19 Nov 2014 03:29:35 -0800 (PST) Received-SPF: none (google.com: linux-kernel-owner@vger.kernel.org does not designate permitted sender hosts) client-ip=209.132.180.67; Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755990AbaKSL3c (ORCPT + 26 others); Wed, 19 Nov 2014 06:29:32 -0500 Received: from foss-mx-na.foss.arm.com ([217.140.108.86]:33203 "EHLO foss-mx-na.foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753595AbaKSL3b (ORCPT ); Wed, 19 Nov 2014 06:29:31 -0500 Received: from foss-smtp-na-1.foss.arm.com (unknown [10.80.61.8]) by foss-mx-na.foss.arm.com (Postfix) with ESMTP id 882C7D6; Wed, 19 Nov 2014 05:29:21 -0600 (CST) Received: from collaborate-mta1.arm.com (highbank-bc01-b06.austin.arm.com [10.112.81.134]) by foss-smtp-na-1.foss.arm.com (Postfix) with ESMTP id 1ABAF5FAD7; Wed, 19 Nov 2014 05:29:14 -0600 (CST) Received: from e104818-lin.cambridge.arm.com (e104818-lin.cambridge.arm.com [10.1.203.148]) by collaborate-mta1.arm.com (Postfix) with ESMTPS id E3A5213F6FA; Wed, 19 Nov 2014 05:29:12 -0600 (CST) Date: Wed, 19 Nov 2014 11:29:10 +0000 From: Catalin Marinas To: Arnd Bergmann Cc: Ding Tianhong , "linux-arm-kernel@lists.infradead.org" , Will Deacon , "linux-kernel@vger.kernel.org" Subject: Re: For the problem when using swiotlb Message-ID: <20141119112910.GD7156@e104818-lin.cambridge.arm.com> References: <5469E26B.2010905@huawei.com> <20141117180947.GI29595@e104818-lin.cambridge.arm.com> <546C0BBB.9070000@huawei.com> <1535751.CcvIi3DN4F@wuerfel> MIME-Version: 1.0 In-Reply-To: <1535751.CcvIi3DN4F@wuerfel> User-Agent: Mutt/1.5.23 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org Precedence: list List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: catalin.marinas@arm.com X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.215.42 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 List-Post: , List-Help: , List-Archive: List-Unsubscribe: , Content-Disposition: inline On Wed, Nov 19, 2014 at 08:45:43AM +0000, Arnd Bergmann wrote: > On Wednesday 19 November 2014 11:17:15 Ding Tianhong wrote: > > On 2014/11/18 2:09, Catalin Marinas wrote: > > > On Mon, Nov 17, 2014 at 12:18:42PM +0000, Arnd Bergmann wrote: > > >> On Monday 17 November 2014 19:56:27 Ding Tianhong wrote: > > >>> The commit 3690951fc6d42f3a0903987677d0e592c49dd8db(arm64: Use swiotlb late initialisation) > > >>> switches the DMA mapping code to swiotlb_tlb_late_init_with_default_size(), this will occur a problem > > >>> when I run the scsi stress tests, the message as below: > > >>> > > >>> sas_controller b1000000.sas: swiotlb buffer is full (sz: 65536 bytes).. > > >>> DMA: Out of SW-IOMMU space for 65536 bytes at device b1000000.sas > > >>> > > >>> The reason is that the swiotlb_tlb_late_init_with_default_size() could only alloc 16M memory for DMA-mapping, > > >>> and the param in cmdline "swiotlb=xxx" is useless because the get_free_pages() only use the buddy to assigned a > > >>> maximum memory of 16M(The MAX_ORDER is 13 for 4k pages), obviously 16M is too small in many scenes, but > > >>> the swiotlb_init() which could reserved a bigger memory as wished could work well for most drivers. > > >>> > > >>> I could not get a better way to fix this problem except to revert this patch, so could you please give me some > > >>> advise and help me, thanks very much. > > >> > > >> In general, you should not need to use swiotlb for most devices, in > > >> particular for high-performance devices like network or block. > > >> > > >> Please make sure that you have set up the dma-ranges properties in > > >> your DT properly to allow 64-bit DMA if the device supports it. > > > > > > That's the problem indeed, the DMA API ends up using swiotlb bounce > > > buffers because the physical address of the pages passed to (or > > > allocated by) the driver are beyond 32-bit limit (which is the default > > > dma mask). > > > > > > > Thanks everyone, I think I found the way to fix it, need to enable DMA_CMA, to reserve a big memory > > for CMA and set coherent mask for dev, then dma_alloc and dma_mapping will not use the swiotlb until > > the memory out of mask or swiotlb_force is enabled. > > > > If I still understand uncorrectly, please inform me. > > Please do not use CMA to work around the problem, but fix the underlying bug > instead. Agree. > The driver should call 'dma_set_mask_and_coherent()' with the appropriate > dma mask, and check whether that succeeded. However, the code implementing > dma_set_mask_and_coherent on arm64 also needs to be changed to look up > the dma-ranges property (see of_dma_configure()), and check if the mask > is possible. dma_set_mask_and_coherent() is a generic function. I think the of_dma_configure() should start with a coherent_dma_mask based on dma-ranges if given rather than defaulting to DMA_BIT_MASK(32). The comment in of_dma_configure() says that devices should set up the supported mask but it's not always up to them but the bus they are connected to. Something like below, untested: --- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ diff --git a/drivers/of/platform.c b/drivers/of/platform.c index 3b64d0bf5bba..dff34883db45 100644 --- a/drivers/of/platform.c +++ b/drivers/of/platform.c @@ -200,6 +200,10 @@ static void of_dma_configure(struct device *dev) /* DMA ranges found. Calculate and set dma_pfn_offset */ dev->dma_pfn_offset = PFN_DOWN(paddr - dma_addr); dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", dev->dma_pfn_offset); + + /* Set the coherent_dma_mask based on the dma-ranges property */ + if (size) + dev->coherent_dma_mask = DMA_BIT_MASK(ilog2(size)); } /**