From patchwork Sat Jan 2 15:17:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 356279 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A0BBC43333 for ; Sat, 2 Jan 2021 15:22:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 085D0224D2 for ; Sat, 2 Jan 2021 15:22:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726731AbhABPWI (ORCPT ); Sat, 2 Jan 2021 10:22:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47486 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726599AbhABPWF (ORCPT ); Sat, 2 Jan 2021 10:22:05 -0500 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B08DFC0613C1; Sat, 2 Jan 2021 07:21:24 -0800 (PST) Received: by mail-wm1-x32c.google.com with SMTP id 3so13738330wmg.4; Sat, 02 Jan 2021 07:21:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=rf6BUzrBX0nFEremfE+E6aI01/qO5RAbaAeeEZzdEU8=; b=b+W6Gio8PokvEN1p7hK+vmrtFsYuUXj5E1Pcurgm37QcJbo+IA1ctH7QYYoo7/bbAh CN1Ig8NDcGJCRgskTeuCehCCc3ApampAkhSMSg2g/2YalrbDFo/CHnPDKFyTmy/+JAGB WTZaFIVAU3/+edur0jyHbznCOvvLQgMzn3ycRPlQMd4vU60LafrpHS/ipZaf+XXm4IEX tlsU6fQjwRt3P8URGU8LEebx6TQBAIvGZsZGQNF/fgCj0JpB0HUSXNMuJkKfSKZurWme +ogJW22uQPLiT7k4Wy03Qa4DsVt11BbsmMtb4LQtIZN+IhFNvNGpiUgToKDkYZsoZrtD h/wA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rf6BUzrBX0nFEremfE+E6aI01/qO5RAbaAeeEZzdEU8=; b=o9436OLd5iMQhXloxcW6OV5rTksWhfRDHs/57xqYcRN9PkZkzelgrVevr5DE0uSmFm pZmQuA1YcjwbAz0JnU8YdgEuEJ9baQB73y5qbZyOzAYX1zfodX4xFVjSBPjUhO46cKNE qPckybt7iqGRyqWiLxXSFnp7LlsZkvOxTGvwD16cxgVNonY5TlftG0FciKIfofW4oXFZ Qqdgj98A2DgsTcr2MNaA5457OKmYffmB6toGKGZSKYtGFT0NDNVBDkorR4O9hOZk7D4J 7yWfPkJnBeMqzZ6f78wq7bWWXGIsxQA3cPq4JLnGQUZxF1OxL7fG02JPjN2F5H7TMl5w waAA== X-Gm-Message-State: AOAM531FZTEFHN1OjV5hB0O+YNQjAfOH1YPncLb7GM5xtia2nE+NK2zX 9pxX0MtOoL0MOGBcH9wdq7QUV9f/ootjyw== X-Google-Smtp-Source: ABdhPJx/WYak/g3xVcVBFTivN/j7PYb09nrxUAu6QP88WHUxu/G0S/WDG8I7eQUPvioar/5dabMjRQ== X-Received: by 2002:a1c:ba44:: with SMTP id k65mr19751159wmf.188.1609600883202; Sat, 02 Jan 2021 07:21:23 -0800 (PST) Received: from localhost.localdomain ([85.255.236.0]) by smtp.gmail.com with ESMTPSA id h13sm78671243wrm.28.2021.01.02.07.21.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 02 Jan 2021 07:21:22 -0800 (PST) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Matthew Wilcox , Ming Lei , Johannes Weiner , Alexander Viro , "Darrick J . Wong" , "Martin K . Petersen" , Jonathan Corbet , linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, linux-kernel@vger.kernel.org, target-devel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-doc@vger.kernel.org Subject: [PATCH v2 1/7] splice: don't generate zero-len segement bvecs Date: Sat, 2 Jan 2021 15:17:33 +0000 Message-Id: X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org iter_file_splice_write() may spawn bvec segments with zero-length. In preparation for prohibiting them, filter out by hand at splice level. Signed-off-by: Pavel Begunkov Reviewed-by: Christoph Hellwig --- fs/splice.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/fs/splice.c b/fs/splice.c index 866d5c2367b2..7299330c3270 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -644,7 +644,6 @@ iter_file_splice_write(struct pipe_inode_info *pipe, struct file *out, ret = splice_from_pipe_next(pipe, &sd); if (ret <= 0) break; - if (unlikely(nbufs < pipe->max_usage)) { kfree(array); nbufs = pipe->max_usage; @@ -662,12 +661,13 @@ iter_file_splice_write(struct pipe_inode_info *pipe, struct file *out, /* build the vector */ left = sd.total_len; - for (n = 0; !pipe_empty(head, tail) && left && n < nbufs; tail++, n++) { + for (n = 0; !pipe_empty(head, tail) && left && n < nbufs; tail++) { struct pipe_buffer *buf = &pipe->bufs[tail & mask]; size_t this_len = buf->len; - if (this_len > left) - this_len = left; + if (!this_len) + continue; + this_len = min(this_len, left); ret = pipe_buf_confirm(pipe, buf); if (unlikely(ret)) { @@ -680,6 +680,7 @@ iter_file_splice_write(struct pipe_inode_info *pipe, struct file *out, array[n].bv_len = this_len; array[n].bv_offset = buf->offset; left -= this_len; + n++; } iov_iter_bvec(&from, WRITE, array, n, sd.total_len - left); From patchwork Sat Jan 2 15:17:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 356278 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A8C0C43219 for ; Sat, 2 Jan 2021 15:22:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EE4F422482 for ; Sat, 2 Jan 2021 15:22:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726955AbhABPWb (ORCPT ); Sat, 2 Jan 2021 10:22:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47492 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726692AbhABPWH (ORCPT ); Sat, 2 Jan 2021 10:22:07 -0500 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 53661C0613CF; Sat, 2 Jan 2021 07:21:26 -0800 (PST) Received: by mail-wm1-x331.google.com with SMTP id e25so13776093wme.0; Sat, 02 Jan 2021 07:21:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=XV9zM74CTwn4+eOGP/R29Owp4dxby4wP3b3oxVzvafI=; b=SFOgPJzQdnQe9hMNyQxyQbgPlbohVEDn3pDwGhwgxKOSsT8dm8/2bAbej1/rjSTtri ukGLApzLogZWfv11VeVtgIPhW6awxqXpE7QP3IlLTnYIPyP+iFk5IGsXXgmVvQOPQxZJ TCphjgqog4CNIRgdfbE+pXjdH5B5VbGFy1aYwjG6mm7JM1Oh8PCoE4aQ4Y2EzwoFbFjI E/6LmzLKTkRVgnRsa5HlcvVzmcRJ4wXrxPM54VS/zG84Nv8uYfyIOnG5YqTrIUPT+4+Q x1zwgtU8+DEwQchxEcFpE1ikA3Qg3TbyJ0cGMNrqU3UEuyi35sfE5gXzZ3rw3LPWBlKi a/+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=XV9zM74CTwn4+eOGP/R29Owp4dxby4wP3b3oxVzvafI=; b=JHPvl42CtZQ+9ghptE4CRi99VwVCUQK5I/Wfua7wZuE+/AUhxngjFUsS7AklCo7Yix kfnUUrJLCLvwQHbMkjGNfVOjIDrfEIB/p1wpNwWoShAl8nhp8OD/UgD+NxLbDWX9v1is vsyxaPJhY0RdGDbVkj6p5IZ1ZclwP5U9+HuYRtw2rCpsPRG/HTuXAQL/jeLVwTqGqi14 bVOL6zVHnXvFHsimBYZZYO3W1gp+YR2oCZpEXT5KOXP+RNs2biIvaC1gJ2lgwxR85KBx 7qg4W75H2SQO0KcBRKEMJXzyaLYntN3Zky6I4q+2EA8CBFPB3eU5z9nJQwqXUsn5BcoH 6DqQ== X-Gm-Message-State: AOAM531OU0U/YorZBuev68TnwtEiny8QRgwVV0SrgUfWnWF8hKTWD48p U05jOzAhiWfnz6xn8Vj7p7vUAr1NY2Ewfg== X-Google-Smtp-Source: ABdhPJxwu7FOcxfliyqioKGivmZLgq8xfpPKYLmREFyeGepQEsU4cSXXuM6gY2U1D8AB/l/odvpL3Q== X-Received: by 2002:a7b:c04c:: with SMTP id u12mr20318416wmc.185.1609600884854; Sat, 02 Jan 2021 07:21:24 -0800 (PST) Received: from localhost.localdomain ([85.255.236.0]) by smtp.gmail.com with ESMTPSA id h13sm78671243wrm.28.2021.01.02.07.21.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 02 Jan 2021 07:21:24 -0800 (PST) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Matthew Wilcox , Ming Lei , Johannes Weiner , Alexander Viro , "Darrick J . Wong" , "Martin K . Petersen" , Jonathan Corbet , linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, linux-kernel@vger.kernel.org, target-devel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-doc@vger.kernel.org Subject: [PATCH v2 2/7] bvec/iter: disallow zero-length segment bvecs Date: Sat, 2 Jan 2021 15:17:34 +0000 Message-Id: X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org zero-length bvec segments are allowed in general, but not handled by bio and down the block layer so filtered out. This inconsistency may be confusing and prevent from optimisations. As zero-length segments are useless and places that were generating them are patched, declare them not allowed. Signed-off-by: Pavel Begunkov Reviewed-by: Christoph Hellwig --- Documentation/filesystems/porting.rst | 7 +++++++ lib/iov_iter.c | 2 -- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/Documentation/filesystems/porting.rst b/Documentation/filesystems/porting.rst index 867036aa90b8..c722d94f29ea 100644 --- a/Documentation/filesystems/porting.rst +++ b/Documentation/filesystems/porting.rst @@ -865,3 +865,10 @@ no matter what. Everything is handled by the caller. clone_private_mount() returns a longterm mount now, so the proper destructor of its result is kern_unmount() or kern_unmount_array(). + +--- + +**mandatory** + +zero-length bvec segments are disallowed, they must be filtered out before +passed on to an iterator. diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 1635111c5bd2..7de304269641 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -72,8 +72,6 @@ __start.bi_bvec_done = skip; \ __start.bi_idx = 0; \ for_each_bvec(__v, i->bvec, __bi, __start) { \ - if (!__v.bv_len) \ - continue; \ (void)(STEP); \ } \ } From patchwork Sat Jan 2 15:17:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 356088 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7774FC4332B for ; Sat, 2 Jan 2021 15:22:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 43079224D2 for ; Sat, 2 Jan 2021 15:22:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726823AbhABPWM (ORCPT ); Sat, 2 Jan 2021 10:22:12 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47494 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726766AbhABPWK (ORCPT ); Sat, 2 Jan 2021 10:22:10 -0500 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C29F1C0613ED; Sat, 2 Jan 2021 07:21:27 -0800 (PST) Received: by mail-wr1-x431.google.com with SMTP id 91so26566410wrj.7; Sat, 02 Jan 2021 07:21:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7FOerU/+uiaLOx8IjnYNL5LXRoiyAnmqji55A4AV5Zs=; b=LBsSYTbjwdI3QmQu4IbgQ7olNGDMBAqElqxAyPsojxLe7NcTh8El+Pke0BmUM22eYb AuZq9SBpNP53WKEFTQdHVDQPCC/8PDYsWLnhkSG7Q2av8EAE8APyACuUnwA3w3t7JqWh R4mCXlk4y9W+n2bx+NU4V6SqSobRuYDWexgtwJuMlhnrLQQMDddC3lR9NTJlDYGo4U1G Yntmje7tslRu50v12yeP/tlquNu8wXFd0g77n+FLG0DNgXfWPtBPtB8SMzz/3+rhpWhR Epwmsr+CSoG9EWj2xUflykAOxhsPhi2ABMO72oHO7eo5cj09XjFAKrK+DFY/UA4J8N0y OWaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7FOerU/+uiaLOx8IjnYNL5LXRoiyAnmqji55A4AV5Zs=; b=H54ehalIm4pDyQ20WUtcI4FW9hSb/7osqZR9HBk+rzMD32OM0cWqAMMMZMfZA57Cf3 vQi8Sj5yv6SHhQS95nr2OUK1mjnpOtKZUG8PVsCw6PAvEy4fiUmny0scjRb6fl4SZsZy o4b40qV+ruSi90OC0jZjPHrZI6g1T/dgX2Ugeb1HBzf4S7IUbhnmDMNmF/PFl5yVX7xm 4IclwqlzPwjK8Jg3lMyiNZjYTz12SBI0ZyCg2AAfZz7YYJAWa6rkZfXxuPDihImJn4Nu Gq7GXbvd2PkE5TkzKNekGWANcuToq2oH/VBogzBNfn+XvvXfUT84qfH/bendRWDaRiUF /zew== X-Gm-Message-State: AOAM532vVaNBSbbz8b10LLhrk1DTnnHmFPlkojsPahgQLncNsDl1DcF0 +NvOfaLIqM84eM+Qv+AfvBJCAsRBSy/3UQ== X-Google-Smtp-Source: ABdhPJzvzXyfPCTdTGrqpyWR+n7zH7gi0rGiGb6+ilO2BRa7Nk9Cv57P3HckjnOAEWjCCb1q8AjWLg== X-Received: by 2002:a5d:6789:: with SMTP id v9mr71303602wru.86.1609600886321; Sat, 02 Jan 2021 07:21:26 -0800 (PST) Received: from localhost.localdomain ([85.255.236.0]) by smtp.gmail.com with ESMTPSA id h13sm78671243wrm.28.2021.01.02.07.21.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 02 Jan 2021 07:21:25 -0800 (PST) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Matthew Wilcox , Ming Lei , Johannes Weiner , Alexander Viro , "Darrick J . Wong" , "Martin K . Petersen" , Jonathan Corbet , linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, linux-kernel@vger.kernel.org, target-devel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-doc@vger.kernel.org Subject: [PATCH v2 3/7] block/psi: remove PSI annotations from direct IO Date: Sat, 2 Jan 2021 15:17:35 +0000 Message-Id: X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org Direct IO does not operate on the current working set of pages managed by the kernel, so it should not be accounted as memory stall to PSI infrastructure. The block layer and iomap direct IO use bio_iov_iter_get_pages() to build bios, and they are the only users of it, so to avoid PSI tracking for them clear out BIO_WORKINGSET flag. Do same for dio_bio_submit() because fs/direct_io constructs bios by hand directly calling bio_add_page(). Reported-by: Christoph Hellwig Suggested-by: Christoph Hellwig Suggested-by: Johannes Weiner Signed-off-by: Pavel Begunkov Reviewed-by: Christoph Hellwig --- block/bio.c | 6 ++++++ fs/direct-io.c | 2 ++ 2 files changed, 8 insertions(+) diff --git a/block/bio.c b/block/bio.c index 1f2cc1fbe283..9f26984af643 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1099,6 +1099,9 @@ static int __bio_iov_append_get_pages(struct bio *bio, struct iov_iter *iter) * fit into the bio, or are requested in @iter, whatever is smaller. If * MM encounters an error pinning the requested pages, it stops. Error * is returned only if 0 pages could be pinned. + * + * It's intended for direct IO, so doesn't do PSI tracking, the caller is + * responsible for setting BIO_WORKINGSET if necessary. */ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) { @@ -1123,6 +1126,9 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) if (is_bvec) bio_set_flag(bio, BIO_NO_PAGE_REF); + + /* don't account direct I/O as memory stall */ + bio_clear_flag(bio, BIO_WORKINGSET); return bio->bi_vcnt ? 0 : ret; } EXPORT_SYMBOL_GPL(bio_iov_iter_get_pages); diff --git a/fs/direct-io.c b/fs/direct-io.c index d53fa92a1ab6..0e689233f2c7 100644 --- a/fs/direct-io.c +++ b/fs/direct-io.c @@ -426,6 +426,8 @@ static inline void dio_bio_submit(struct dio *dio, struct dio_submit *sdio) unsigned long flags; bio->bi_private = dio; + /* don't account direct I/O as memory stall */ + bio_clear_flag(bio, BIO_WORKINGSET); spin_lock_irqsave(&dio->bio_lock, flags); dio->refcount++; From patchwork Sat Jan 2 15:17:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 356087 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D086DC433E9 for ; Sat, 2 Jan 2021 15:22:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9B6CA224D4 for ; Sat, 2 Jan 2021 15:22:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726890AbhABPWV (ORCPT ); Sat, 2 Jan 2021 10:22:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47502 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726599AbhABPWK (ORCPT ); Sat, 2 Jan 2021 10:22:10 -0500 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7B46DC06179E; Sat, 2 Jan 2021 07:21:29 -0800 (PST) Received: by mail-wm1-x333.google.com with SMTP id c133so13165857wme.4; Sat, 02 Jan 2021 07:21:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=LOsfOtb4JF9iqyizeOjm4tVRPPo2GoTdvRn37AEB3XM=; b=OLiALIisaOX66Ofzuml3kC66wCIP9u2t1hFkBs0dEBL0aE8EBdga0VD5prANc6xETp TY9aUheJWJU6uSTxBN1daNkHXTUP8NPrlmGmVJdVYe26hS8drmoIrX5bVt+YOFOK7NS0 06iG4NlowbWLhJVZAv+YLXnsI8CN9SlFibCIW1Nsqv3OfI1Ox8k/kWFmOAfT8uxCMkSG CjagviERoypCMSqVlCZv2T+vPztbjieZK6tfDQXtVy8OUNP6Nfux0P87TYecslC5aXw2 uHkgUOKe4hizjipjPcmVFUCvtLfvEZTcgKXJkIGHO4xWKdCj6r0iQb+tUxmLTZSFwh+q WSEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=LOsfOtb4JF9iqyizeOjm4tVRPPo2GoTdvRn37AEB3XM=; b=gahcAZUpuy5xzI3niP5Bm+Tnnz0xCTtmvKttLE+HUEZQnwsxRcZqVTCHmpwbt46ysu w+2khoW6X7V8/SeX5qT8GQl28c4PAEn8OlU3OQAAclMxOFLtmJqXD2211ePCCz5LoPla qY/F51ZYaD5IeTXMZnQWijJKjjTXZIwDl0D0GCPKCEAOIrB5PMFwr1QaZTAQt17jXw25 iwwq7JqU4zuG/OzpWgxFpC9MqEbbWgn5FmoYWq8lqZcABdqw8/ElI3i1jJtb4wxxGCLn JLsnqK3V49pRNpXj/AlpmxskFTEvO8vK8LU774GAVUKDp5mH6Q2wMja2+w353epGd4CB 6SEQ== X-Gm-Message-State: AOAM532wPq4MQkWEJ6wIOw0krZkGZaOg7dqYCZlbVS18WOsRh+lBlsk3 Fgu3h5iOsFTJadr6As7nyjNVJUlfOxxsTw== X-Google-Smtp-Source: ABdhPJwLJ+vihedFjGQ3PU8Cex40WtPYz+zy0eYCghrDXR8HINdOVHJnDEzOjzgkOrGDOmWYk/f8TQ== X-Received: by 2002:a1c:6055:: with SMTP id u82mr20245480wmb.61.1609600887976; Sat, 02 Jan 2021 07:21:27 -0800 (PST) Received: from localhost.localdomain ([85.255.236.0]) by smtp.gmail.com with ESMTPSA id h13sm78671243wrm.28.2021.01.02.07.21.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 02 Jan 2021 07:21:27 -0800 (PST) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Matthew Wilcox , Ming Lei , Johannes Weiner , Alexander Viro , "Darrick J . Wong" , "Martin K . Petersen" , Jonathan Corbet , linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, linux-kernel@vger.kernel.org, target-devel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-doc@vger.kernel.org, Christoph Hellwig Subject: [PATCH v2 4/7] target/file: allocate the bvec array as part of struct target_core_file_cmd Date: Sat, 2 Jan 2021 15:17:36 +0000 Message-Id: <20d31d34eb6fb8a10ac001e5e6c3812eddd24aa3.1609461359.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org From: Christoph Hellwig This saves one memory allocation, and ensures the bvecs aren't freed before the AIO completion. This will allow the lower level code to be optimized so that it can avoid allocating another bvec array. Signed-off-by: Christoph Hellwig Signed-off-by: Pavel Begunkov --- drivers/target/target_core_file.c | 20 ++++++-------------- 1 file changed, 6 insertions(+), 14 deletions(-) diff --git a/drivers/target/target_core_file.c b/drivers/target/target_core_file.c index b0cb5b95e892..cce455929778 100644 --- a/drivers/target/target_core_file.c +++ b/drivers/target/target_core_file.c @@ -241,6 +241,7 @@ struct target_core_file_cmd { unsigned long len; struct se_cmd *cmd; struct kiocb iocb; + struct bio_vec bvecs[]; }; static void cmd_rw_aio_complete(struct kiocb *iocb, long ret, long ret2) @@ -268,29 +269,22 @@ fd_execute_rw_aio(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents, struct target_core_file_cmd *aio_cmd; struct iov_iter iter = {}; struct scatterlist *sg; - struct bio_vec *bvec; ssize_t len = 0; int ret = 0, i; - aio_cmd = kmalloc(sizeof(struct target_core_file_cmd), GFP_KERNEL); + aio_cmd = kmalloc(struct_size(aio_cmd, bvecs, sgl_nents), GFP_KERNEL); if (!aio_cmd) return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE; - bvec = kcalloc(sgl_nents, sizeof(struct bio_vec), GFP_KERNEL); - if (!bvec) { - kfree(aio_cmd); - return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE; - } - for_each_sg(sgl, sg, sgl_nents, i) { - bvec[i].bv_page = sg_page(sg); - bvec[i].bv_len = sg->length; - bvec[i].bv_offset = sg->offset; + aio_cmd->bvecs[i].bv_page = sg_page(sg); + aio_cmd->bvecs[i].bv_len = sg->length; + aio_cmd->bvecs[i].bv_offset = sg->offset; len += sg->length; } - iov_iter_bvec(&iter, is_write, bvec, sgl_nents, len); + iov_iter_bvec(&iter, is_write, aio_cmd->bvecs, sgl_nents, len); aio_cmd->cmd = cmd; aio_cmd->len = len; @@ -307,8 +301,6 @@ fd_execute_rw_aio(struct se_cmd *cmd, struct scatterlist *sgl, u32 sgl_nents, else ret = call_read_iter(file, &aio_cmd->iocb, &iter); - kfree(bvec); - if (ret != -EIOCBQUEUED) cmd_rw_aio_complete(&aio_cmd->iocb, ret, 0); From patchwork Sat Jan 2 15:17:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 356085 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC9F0C43217 for ; Sat, 2 Jan 2021 15:23:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B71EE224D2 for ; Sat, 2 Jan 2021 15:23:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727096AbhABPWr (ORCPT ); Sat, 2 Jan 2021 10:22:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47596 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726573AbhABPWp (ORCPT ); Sat, 2 Jan 2021 10:22:45 -0500 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 22906C06179F; Sat, 2 Jan 2021 07:21:31 -0800 (PST) Received: by mail-wm1-x335.google.com with SMTP id c133so13165892wme.4; Sat, 02 Jan 2021 07:21:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BfcAWicfY3ps8Cp/gsA4qs7Avo53ajYGpXmNYv+JbSY=; b=h3b+ygyZOCMZuu/SSxyE2DWy3gO46XVTWVl20xf69BlLvp9dyAKaIzAwn7UmW26Kb/ Ljv0oHCBxsXqorvd1qYO/nb8M0kRw/9DQQx6p+tfKVVwfj+/6ysumrx+Cn1GhfAuSm9S v+7NBHal1hmJCfS5zl+1GhhdLI3hHTIpQnNfgkqSKgB/I81jGpD95EU2eneLMNlNY4IH FgNLFwinxLsO4rsFs6D3gm3ZQzQI71uYRosnGEz/aeZdan7BrrrEt08QP5adFhjqMouz JWKb4wWJuJfsKMdwtZJd30UVuxO9MV9x4ger8/HPOf50eWCLPwOaRdQ2MS3H02te9WbG ByKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BfcAWicfY3ps8Cp/gsA4qs7Avo53ajYGpXmNYv+JbSY=; b=iUIhtk48KJ91qOW1SIRED0JtIPiFZF8NEJuOfqnWGqMCK+zAZNdd1fhU92fgwuT2DG 9PBXqvjDiWe782iEWD4tWc8ohfieNJtqOeh+sNg1Ot0EJJH0MkkYh5V5kqj3Ab56rmyy 762777UFpVnPtlIoGd4/bfA1ThKlI62Bfu7oZoVk/YQcP45xti/QRNpPFmqgsbDqFQdO hedkU1tmvJRdAvR4HCBMT3CZxBKjTpYzK8aYXmio0HNyJtkEDtrJHF8Pl9IYNajScQiM R1HMnG1pbg2KTSzvlPgTCutKztof2Env3dugFah86w+7CvU5CZyJxFxQgv6kqfEmKWCI AO2g== X-Gm-Message-State: AOAM531RBiiVYPWqFz5buDAf/mf2b5mTHjlxEkuHIdgBuxQwDymOeglX zTosUaalT949LHXbi0ugBf/8MEVPtoOJTw== X-Google-Smtp-Source: ABdhPJz+w9pXtmdIRErhfXjecmNBpBoEj5ZToxSuo4zC0pqEv8F+jXoDdzGVYwWU0q45PiWTWmj+DA== X-Received: by 2002:a1c:6383:: with SMTP id x125mr19797365wmb.46.1609600889718; Sat, 02 Jan 2021 07:21:29 -0800 (PST) Received: from localhost.localdomain ([85.255.236.0]) by smtp.gmail.com with ESMTPSA id h13sm78671243wrm.28.2021.01.02.07.21.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 02 Jan 2021 07:21:29 -0800 (PST) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Matthew Wilcox , Ming Lei , Johannes Weiner , Alexander Viro , "Darrick J . Wong" , "Martin K . Petersen" , Jonathan Corbet , linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, linux-kernel@vger.kernel.org, target-devel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-doc@vger.kernel.org, Christoph Hellwig Subject: [PATCH v2 5/7] iov_iter: optimise bvec iov_iter_advance() Date: Sat, 2 Jan 2021 15:17:37 +0000 Message-Id: X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org iov_iter_advance() is heavily used, but implemented through generic means. For bvecs there is a specifically crafted function for that, so use bvec_iter_advance() instead, it's faster and slimmer. Reviewed-by: Christoph Hellwig Signed-off-by: Pavel Begunkov --- lib/iov_iter.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 7de304269641..9b1c109dc8a9 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1065,6 +1065,21 @@ static void pipe_advance(struct iov_iter *i, size_t size) pipe_truncate(i); } +static void iov_iter_bvec_advance(struct iov_iter *i, size_t size) +{ + struct bvec_iter bi; + + bi.bi_size = i->count; + bi.bi_bvec_done = i->iov_offset; + bi.bi_idx = 0; + bvec_iter_advance(i->bvec, &bi, size); + + i->bvec += bi.bi_idx; + i->nr_segs -= bi.bi_idx; + i->count = bi.bi_size; + i->iov_offset = bi.bi_bvec_done; +} + void iov_iter_advance(struct iov_iter *i, size_t size) { if (unlikely(iov_iter_is_pipe(i))) { @@ -1075,6 +1090,10 @@ void iov_iter_advance(struct iov_iter *i, size_t size) i->count -= size; return; } + if (iov_iter_is_bvec(i)) { + iov_iter_bvec_advance(i, size); + return; + } iterate_and_advance(i, size, v, 0, 0, 0) } EXPORT_SYMBOL(iov_iter_advance); From patchwork Sat Jan 2 15:17:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 356277 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5090BC4332E for ; Sat, 2 Jan 2021 15:23:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3A198224D2 for ; Sat, 2 Jan 2021 15:23:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727043AbhABPWq (ORCPT ); Sat, 2 Jan 2021 10:22:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47598 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726692AbhABPWo (ORCPT ); Sat, 2 Jan 2021 10:22:44 -0500 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EC17FC0617A0; Sat, 2 Jan 2021 07:21:32 -0800 (PST) Received: by mail-wm1-x32e.google.com with SMTP id 190so13178531wmz.0; Sat, 02 Jan 2021 07:21:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ohSIp5ePWaz1w6S32NBHT2NvcCAqeYKdn5igq7tNqQI=; b=BEIk7nboPHlTUjalpiV3UauYsn/+YIZ6ApV/a38QtTCGDFE0O0B8g5OUqEAzD9RpBU 4nEM1qUGqsT8eVyKzZAXQXLjn7IRF2yLm0LxjbPlM+YimGIAqRNVEsuQ5SFBU6Ft1eP4 cnkJZMfEWLl0j8yTOmzIPIitBUzk5/8iLVC84bsdTjITUER+RBIa0ezp6VUSCi4cb/Jx Niag469Wd+k1znOnEhBPozfImiBQJERxwYLg2CO3Y5P4ILqQSf0atKKCFpPt79+xrbMs 81XqOQ7+NWeMTOEogSAur2iWee0K7sd1Jo89zWpwI2d0LnbvIDpGg7hhk2E9oD0TGx8R pgYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ohSIp5ePWaz1w6S32NBHT2NvcCAqeYKdn5igq7tNqQI=; b=r2LGmWaE63WmKZMDJvJB8Yx3FSm0I8TVNSZRXLEMU9gvTEx6Ts7jcgsQgSBGaufUSV 2JclfgLZiQ1TKwzHciYhaY01MvxJK+yjwhp8rdjxTqnkViOVECjC15iCIwbyk/0+DPqV kKQPw7CvXHVJF2SVEFFyGgePCU1buCP5m+AsiMI8HcOCvAofQkppqAaLdkvOFX1LCIcB pFwZ+OW9HUq/tr8n/fWLtVg4UCHDvg4DVcsbniHfOsmEQQNmRgPYBJQMtAZ/0RaVXu7L wYiyCB1Z93TwWlGrJg1sLh/rQvKyGuHHLl0RNR1nYKweysSwVCTiwNRdgOPkOWb+4rvn 10eA== X-Gm-Message-State: AOAM530at121qgHiITvtFOfvauHCF8J7cD7ELwNkZSB805JJWCHyG5oJ SW9G88A9TkyiRq4NlDsBEFMm2/Ia6m7ndA== X-Google-Smtp-Source: ABdhPJzRenSPI7p3GZoaEGEzPmw1o/yknelNfIvXGv60hGl85AWy2eJbIcUd/1xqAOFCbX3npNieyw== X-Received: by 2002:a7b:c35a:: with SMTP id l26mr19162941wmj.182.1609600891440; Sat, 02 Jan 2021 07:21:31 -0800 (PST) Received: from localhost.localdomain ([85.255.236.0]) by smtp.gmail.com with ESMTPSA id h13sm78671243wrm.28.2021.01.02.07.21.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 02 Jan 2021 07:21:31 -0800 (PST) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Matthew Wilcox , Ming Lei , Johannes Weiner , Alexander Viro , "Darrick J . Wong" , "Martin K . Petersen" , Jonathan Corbet , linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, linux-kernel@vger.kernel.org, target-devel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-doc@vger.kernel.org, Christoph Hellwig Subject: [PATCH v2 6/7] bio: add a helper calculating nr segments to alloc Date: Sat, 2 Jan 2021 15:17:38 +0000 Message-Id: X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org Add a helper function calculating the number of bvec segments we need to allocate to construct a bio. It doesn't change anything functionally, but will be used to not duplicate special cases in the future. Reviewed-by: Christoph Hellwig Signed-off-by: Pavel Begunkov --- fs/block_dev.c | 7 ++++--- fs/iomap/direct-io.c | 9 ++++----- include/linux/bio.h | 10 ++++++++++ 3 files changed, 18 insertions(+), 8 deletions(-) diff --git a/fs/block_dev.c b/fs/block_dev.c index 3e5b02f6606c..034e6c8cfcab 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -416,7 +416,7 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages) dio->size += bio->bi_iter.bi_size; pos += bio->bi_iter.bi_size; - nr_pages = iov_iter_npages(iter, BIO_MAX_PAGES); + nr_pages = bio_iov_vecs_to_alloc(iter, BIO_MAX_PAGES); if (!nr_pages) { bool polled = false; @@ -481,9 +481,10 @@ blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter) { int nr_pages; - nr_pages = iov_iter_npages(iter, BIO_MAX_PAGES + 1); - if (!nr_pages) + if (!iov_iter_count(iter)) return 0; + + nr_pages = bio_iov_vecs_to_alloc(iter, BIO_MAX_PAGES + 1); if (is_sync_kiocb(iocb) && nr_pages <= BIO_MAX_PAGES) return __blkdev_direct_IO_simple(iocb, iter, nr_pages); diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index 933f234d5bec..ea1e8f696076 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -250,11 +250,8 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length, orig_count = iov_iter_count(dio->submit.iter); iov_iter_truncate(dio->submit.iter, length); - nr_pages = iov_iter_npages(dio->submit.iter, BIO_MAX_PAGES); - if (nr_pages <= 0) { - ret = nr_pages; + if (!iov_iter_count(dio->submit.iter)) goto out; - } if (need_zeroout) { /* zero out from the start of the block to the write offset */ @@ -263,6 +260,7 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length, iomap_dio_zero(dio, iomap, pos - pad, pad); } + nr_pages = bio_iov_vecs_to_alloc(dio->submit.iter, BIO_MAX_PAGES); do { size_t n; if (dio->error) { @@ -308,7 +306,8 @@ iomap_dio_bio_actor(struct inode *inode, loff_t pos, loff_t length, dio->size += n; copied += n; - nr_pages = iov_iter_npages(dio->submit.iter, BIO_MAX_PAGES); + nr_pages = bio_iov_vecs_to_alloc(dio->submit.iter, + BIO_MAX_PAGES); iomap_dio_submit_bio(dio, iomap, bio, pos); pos += n; } while (nr_pages); diff --git a/include/linux/bio.h b/include/linux/bio.h index 1edda614f7ce..d8f9077c43ef 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -10,6 +10,7 @@ #include /* struct bio, bio_vec and BIO_* flags are defined in blk_types.h */ #include +#include #define BIO_DEBUG @@ -441,6 +442,15 @@ static inline void bio_wouldblock_error(struct bio *bio) bio_endio(bio); } +/* + * Calculate number of bvec segments that should be allocated to fit data + * pointed by @iter. + */ +static inline int bio_iov_vecs_to_alloc(struct iov_iter *iter, int max_segs) +{ + return iov_iter_npages(iter, max_segs); +} + struct request_queue; extern int submit_bio_wait(struct bio *bio); From patchwork Sat Jan 2 15:17:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 356276 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1778EC4161F for ; Sat, 2 Jan 2021 15:23:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E0220224D2 for ; Sat, 2 Jan 2021 15:23:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727187AbhABPW6 (ORCPT ); Sat, 2 Jan 2021 10:22:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47606 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727044AbhABPWq (ORCPT ); Sat, 2 Jan 2021 10:22:46 -0500 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8C95BC0617A1; Sat, 2 Jan 2021 07:21:34 -0800 (PST) Received: by mail-wm1-x32c.google.com with SMTP id g185so13756370wmf.3; Sat, 02 Jan 2021 07:21:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=3l7dQjIvN+V+hEfZaCkzKb+BBnG/zeqm2g1H963d/fU=; b=hKs4d/TU2wMEjpAXrytigQChKkUEsSDMoqcebeDllsKjoNG/spLXHO3ytny4jEZjuZ NcLxLnXZKDqEH9Uj2glbOUxlmtYSXcbYEEbzmRPtNysnmvxO4yeCEfgncm951hEtOUZ1 hzlJX09+eupyD4mtHblxJyOVZVwMKlcxcxD9nbw8GLsoh0Xym5yPfV9pg84AivjTPr3M a/yRyBwqMg6H/mJJ7/smc9rug0a80eRO6vJufbqg30jGrxcEEzL17qwrBGvdMV3jX7RK bzucefWdFbQBUZa9GTb4HqOHSwqVZBOZ4y+ECy+X8wyJr0JlAbTMarl7wqY4+bShvhwm eYZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3l7dQjIvN+V+hEfZaCkzKb+BBnG/zeqm2g1H963d/fU=; b=PhV45bzipliA8mXI/XqaQM6i22c87cathRqUyufcW3oIkqu55WKM8UugZbhVq/k5rJ qMLgLA+Ux+bUTojEAIyeAzNIr4H5Wc0VbiO8vDex39Vkv8oZzLhP5Hq4W3GU72b3cS7m guXI9528lhEacJQSWGwjs36YVrah1L6cXK1/ltT1g6aLpODNGRpsaCYsq2HbRz006GIL +I6ZqLvH3kZYWuz5xFl9sQfaHWHxc6g8vrqRvuBHIb65s3wPWyjgceAFR2sa4ib9NaSb NdBf1Xelty81sxIkmivCHDUHAqg4mPSBmlwcOFjrkSbE5oQzKS4YZGuCggDKYgd8L94R Zm1w== X-Gm-Message-State: AOAM530CBfJoevVAQqaN5+mtT6MvmEehDRy6n/thOO0ffJZ1zS7ZEJ77 W2qyCEXMHcFeeGpODNXlMDa1aGNu3MlnDg== X-Google-Smtp-Source: ABdhPJwGhnMFbzLGsmTiJIkT4AO1maR288iVD0rGXhF6tvzkOS5GoZ/R3t3iUBUVe2FW5oIam/JQSg== X-Received: by 2002:a1c:43c6:: with SMTP id q189mr19936956wma.7.1609600893086; Sat, 02 Jan 2021 07:21:33 -0800 (PST) Received: from localhost.localdomain ([85.255.236.0]) by smtp.gmail.com with ESMTPSA id h13sm78671243wrm.28.2021.01.02.07.21.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 02 Jan 2021 07:21:32 -0800 (PST) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: Jens Axboe , Christoph Hellwig , Matthew Wilcox , Ming Lei , Johannes Weiner , Alexander Viro , "Darrick J . Wong" , "Martin K . Petersen" , Jonathan Corbet , linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, linux-kernel@vger.kernel.org, target-devel@vger.kernel.org, linux-scsi@vger.kernel.org, linux-doc@vger.kernel.org Subject: [PATCH v2 7/7] bio: don't copy bvec for direct IO Date: Sat, 2 Jan 2021 15:17:39 +0000 Message-Id: <29ed343fa15eb4139f8ab9104d3f9b16fe025dfd.1609461359.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org The block layer spends quite a while in blkdev_direct_IO() to copy and initialise bio's bvec. However, if we've already got a bvec in the input iterator it might be reused in some cases, i.e. when new ITER_BVEC_FLAG_FIXED flag is set. Simple tests show considerable performance boost, and it also reduces memory footprint. Suggested-by: Matthew Wilcox Signed-off-by: Pavel Begunkov Reviewed-by: Christoph Hellwig --- Documentation/filesystems/porting.rst | 9 ++++ block/bio.c | 67 ++++++++++++--------------- include/linux/bio.h | 5 +- 3 files changed, 42 insertions(+), 39 deletions(-) diff --git a/Documentation/filesystems/porting.rst b/Documentation/filesystems/porting.rst index c722d94f29ea..1f8cf8e10b34 100644 --- a/Documentation/filesystems/porting.rst +++ b/Documentation/filesystems/porting.rst @@ -872,3 +872,12 @@ its result is kern_unmount() or kern_unmount_array(). zero-length bvec segments are disallowed, they must be filtered out before passed on to an iterator. + +--- + +**mandatory** + +For bvec based itererators bio_iov_iter_get_pages() now doesn't copy bvecs but +uses the one provided. Anyone issuing kiocb-I/O should ensure that the bvec and +page references stay until I/O has completed, i.e. until ->ki_complete() has +been called or returned with non -EIOCBQUEUED code. diff --git a/block/bio.c b/block/bio.c index 9f26984af643..6f031a04b59a 100644 --- a/block/bio.c +++ b/block/bio.c @@ -960,21 +960,17 @@ void bio_release_pages(struct bio *bio, bool mark_dirty) } EXPORT_SYMBOL_GPL(bio_release_pages); -static int __bio_iov_bvec_add_pages(struct bio *bio, struct iov_iter *iter) +static int bio_iov_bvec_set(struct bio *bio, struct iov_iter *iter) { - const struct bio_vec *bv = iter->bvec; - unsigned int len; - size_t size; - - if (WARN_ON_ONCE(iter->iov_offset > bv->bv_len)) - return -EINVAL; - - len = min_t(size_t, bv->bv_len - iter->iov_offset, iter->count); - size = bio_add_page(bio, bv->bv_page, len, - bv->bv_offset + iter->iov_offset); - if (unlikely(size != len)) - return -EINVAL; - iov_iter_advance(iter, size); + WARN_ON_ONCE(BVEC_POOL_IDX(bio) != 0); + + bio->bi_vcnt = iter->nr_segs; + bio->bi_max_vecs = iter->nr_segs; + bio->bi_io_vec = (struct bio_vec *)iter->bvec; + bio->bi_iter.bi_bvec_done = iter->iov_offset; + bio->bi_iter.bi_size = iter->count; + + iov_iter_advance(iter, iter->count); return 0; } @@ -1088,12 +1084,12 @@ static int __bio_iov_append_get_pages(struct bio *bio, struct iov_iter *iter) * This takes either an iterator pointing to user memory, or one pointing to * kernel pages (BVEC iterator). If we're adding user pages, we pin them and * map them into the kernel. On IO completion, the caller should put those - * pages. If we're adding kernel pages, and the caller told us it's safe to - * do so, we just have to add the pages to the bio directly. We don't grab an - * extra reference to those pages (the user should already have that), and we - * don't put the page on IO completion. The caller needs to check if the bio is - * flagged BIO_NO_PAGE_REF on IO completion. If it isn't, then pages should be - * released. + * pages. For bvec based iterators bio_iov_iter_get_pages() uses the provided + * bvecs rather than copying them. Hence anyone issuing kiocb based IO needs + * to ensure the bvecs and pages stay referenced until the submitted I/O is + * completed by a call to ->ki_complete() or returns with an error other than + * -EIOCBQUEUED. The caller needs to check if the bio is flagged BIO_NO_PAGE_REF + * on IO completion. If it isn't, then pages should be released. * * The function tries, but does not guarantee, to pin as many pages as * fit into the bio, or are requested in @iter, whatever is smaller. If @@ -1105,27 +1101,22 @@ static int __bio_iov_append_get_pages(struct bio *bio, struct iov_iter *iter) */ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) { - const bool is_bvec = iov_iter_is_bvec(iter); - int ret; - - if (WARN_ON_ONCE(bio->bi_vcnt)) - return -EINVAL; + int ret = 0; - do { - if (bio_op(bio) == REQ_OP_ZONE_APPEND) { - if (WARN_ON_ONCE(is_bvec)) - return -EINVAL; - ret = __bio_iov_append_get_pages(bio, iter); - } else { - if (is_bvec) - ret = __bio_iov_bvec_add_pages(bio, iter); + if (iov_iter_is_bvec(iter)) { + if (WARN_ON_ONCE(bio_op(bio) == REQ_OP_ZONE_APPEND)) + return -EINVAL; + bio_iov_bvec_set(bio, iter); + bio_set_flag(bio, BIO_NO_PAGE_REF); + return 0; + } else { + do { + if (bio_op(bio) == REQ_OP_ZONE_APPEND) + ret = __bio_iov_append_get_pages(bio, iter); else ret = __bio_iov_iter_get_pages(bio, iter); - } - } while (!ret && iov_iter_count(iter) && !bio_full(bio, 0)); - - if (is_bvec) - bio_set_flag(bio, BIO_NO_PAGE_REF); + } while (!ret && iov_iter_count(iter) && !bio_full(bio, 0)); + } /* don't account direct I/O as memory stall */ bio_clear_flag(bio, BIO_WORKINGSET); diff --git a/include/linux/bio.h b/include/linux/bio.h index d8f9077c43ef..1d30572a8c53 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -444,10 +444,13 @@ static inline void bio_wouldblock_error(struct bio *bio) /* * Calculate number of bvec segments that should be allocated to fit data - * pointed by @iter. + * pointed by @iter. If @iter is backed by bvec it's going to be reused + * instead of allocating a new one. */ static inline int bio_iov_vecs_to_alloc(struct iov_iter *iter, int max_segs) { + if (iov_iter_is_bvec(iter)) + return 0; return iov_iter_npages(iter, max_segs); }