From patchwork Thu Mar 7 16:25:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 159881 Delivered-To: patch@linaro.org Received: by 2002:a02:5cc1:0:0:0:0:0 with SMTP id w62csp7628343jad; Thu, 7 Mar 2019 08:26:20 -0800 (PST) X-Google-Smtp-Source: APXvYqzIG7IHYqkim4LtxVuWierdPjhd3dNX15HOTejdnbL3mGeVfwQogVTWpDo5BMmF59Gp3rGh X-Received: by 2002:a63:65c4:: with SMTP id z187mr4259263pgb.102.1551975980058; Thu, 07 Mar 2019 08:26:20 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1551975980; cv=none; d=google.com; s=arc-20160816; b=AZn/eZShRhnNxwCBY67m2nmYP0YF+yJ6SDcXxYqdTxGNQAOfLa2HTlttyDFOfW3cnv pG6x69TsfNlfhhwY1mL8CGi5i+xnTi49MWB1jwIrPBLWnOlKSO+a9dzhbU3CxjeHLWlo lBxDW6lD6sblI9MpNJTbnEPVRc2ojAX0anNIoacLz7OuIMKVkMbvH0aS5dkLSxitlTXX 6KpUl15yeyWu3yaVAeE0wivzbhdz5UV0u6LKtZ1Swybqm5Ju8JwfV0ngDzIIGqsbp1EM wRs1Bs3JmRiQEKjpTv+SMezlTLY7W2h5H4fxZmOQT2/tl+6jNpmv0enfjb2U2w+Q2c79 Wneg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=SY00KoFWeTwZXwxCAxRn3goBCIvGzbExPxId3YVmDAI=; b=nNjIsmShuyBNzwMud9kB2vnez2uIUKbR2hQekFhNCwtsUDkaIerjX+1e13lLPzL1Lz XmaPvVpHiFOTvYV8pFQZOrAaWzKrK+9cU6Ylxn+P3TaS1dPQ0lmzAwGZ7rJj5Ph2oAbY 1hWWK55aB6yB00eX4kfdLSkrw2JBwl2xcBjY2run8NUiujcjyq6V1+aIsLJukteIFNLl Fo+JzWWO5z1l3pPuggv94mIMy9RbOA5msuiwNn2fiNtCc9R88DVmCLgT7KOP7A1oBNYH DPtyK19qUTROjL18zHjysLGF98J4AsNRhisYlGj43ativ7lRZr2xEa678Voy7Y6Ypn87 h8YQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Z75T8hDp; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l16si2039022pgg.459.2019.03.07.08.26.19; Thu, 07 Mar 2019 08:26:20 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Z75T8hDp; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726414AbfCGQ0R (ORCPT + 31 others); Thu, 7 Mar 2019 11:26:17 -0500 Received: from mail-wm1-f68.google.com ([209.85.128.68]:37128 "EHLO mail-wm1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726275AbfCGQ0O (ORCPT ); Thu, 7 Mar 2019 11:26:14 -0500 Received: by mail-wm1-f68.google.com with SMTP id x10so9791189wmg.2 for ; Thu, 07 Mar 2019 08:26:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=SY00KoFWeTwZXwxCAxRn3goBCIvGzbExPxId3YVmDAI=; b=Z75T8hDpDi1+Pm2Su4PJKBxlbasjZjnSzjQhz6b470S0GFTTqJjFMS5NDxj7nS8LFE EJI/QQEJkG9kYknYgewNVTSsJ/HWO7vg9FxArz4inYurRrFOqySHiTUyNqILlvVb30vi sTOvXKmbe/sKdF1RarTKbSacGJHh8/XkrV0pvDu9XQGVRkAlc5ZNLBS2Lpa8NjUjEt+k iKHuxgZcj/U26ihOAHhUWWoBwC2O6hr3ZclZUCeVrHLQXuU46Q2WUL/jnaFZysIyQ6xS /Zy1PPDhalZ499zka/rYiXT90YBa6BrMCDipljv/JTNym10GlCxj39gRoaNPamySrQpu DbgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=SY00KoFWeTwZXwxCAxRn3goBCIvGzbExPxId3YVmDAI=; b=SHCqCulc5NZOQnOW9mz6AVldCM73f7b2g74gpUhKyNvUUh3Z6lWSay3e4+QhbkwdJD e0p4qcoWcBWJVsaJMoX9n0DcxsCF9sbSjGwQoDLbpLF+R8uc0NTPNUWTbKQILKra0Ojt 7ikIEFNn0ATdAgNEGuPIFniiMnbdljJrayHnsEaYDEuWV9kxZwzVyaKnnKKDRLW5zbXy XiLh90H3g9gC2jLNJ0T2nZW3NihXhvjqq6H+Ox8dtCiAaHjMBAcBjvoUGbWTgUletmKV tcfticbRIXw1OCNIhPVHVGezolH3KnpGi3bd2HOjGemGobFFwfc7d3gT3IlWokmEuQ7A AWHg== X-Gm-Message-State: APjAAAVMkquy0ezj7Q7FPCjjYK4bcNxBepsgbQZooZbUKs0mhGCMMtxV wPxxGdSBPfYkHvj8a+IBEmsgQw== X-Received: by 2002:a1c:9c0e:: with SMTP id f14mr6529463wme.78.1551975972267; Thu, 07 Mar 2019 08:26:12 -0800 (PST) Received: from localhost.localdomain (146-241-117-61.dyn.eolo.it. [146.241.117.61]) by smtp.gmail.com with ESMTPSA id o127sm5801797wmo.20.2019.03.07.08.26.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 Mar 2019 08:26:11 -0800 (PST) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, ulf.hansson@linaro.org, linus.walleij@linaro.org, broonie@kernel.org, bfq-iosched@googlegroups.com, oleksandr@natalenko.name, fra.fra.800@gmail.com, Paolo Valente Subject: [PATCH BUGFIX IMPROVEMENT 1/8] block, bfq: increase idling for weight-raised queues Date: Thu, 7 Mar 2019 17:25:47 +0100 Message-Id: <20190307162554.77205-2-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190307162554.77205-1-paolo.valente@linaro.org> References: <20190307162554.77205-1-paolo.valente@linaro.org> MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org If a sync bfq_queue has a higher weight than some other queue, and remains temporarily empty while in service, then, to preserve the bandwidth share of the queue, it is necessary to plug I/O dispatching until a new request arrives for the queue. In addition, a timeout needs to be set, to avoid waiting for ever if the process associated with the queue has actually finished its I/O. Even with the above timeout, the device is however not fed with new I/O for a while, if the process has finished its I/O. If this happens often, then throughput drops and latencies grow. For this reason, the timeout is kept rather low: 8 ms is the current default. Unfortunately, such a low value may cause, on the opposite end, a violation of bandwidth guarantees for a process that happens to issue new I/O too late. The higher the system load, the higher the probability that this happens to some process. This is a problem in scenarios where service guarantees matter more than throughput. One important case are weight-raised queues, which need to be granted a very high fraction of the bandwidth. To address this issue, this commit lower-bounds the plugging timeout for weight-raised queues to 20 ms. This simple change provides relevant benefits. For example, on a PLEXTOR PX-256M5S, with which gnome-terminal starts in 0.6 seconds if there is no other I/O in progress, the same applications starts in - 0.8 seconds, instead of 1.2 seconds, if ten files are being read sequentially in parallel - 1 second, instead of 2 seconds, if, in parallel, five files are being read sequentially, and five more files are being written sequentially Signed-off-by: Paolo Valente --- block/bfq-iosched.c | 2 ++ 1 file changed, 2 insertions(+) -- 2.20.1 diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 4c592496a16a..eb658de3cc40 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -2545,6 +2545,8 @@ static void bfq_arm_slice_timer(struct bfq_data *bfqd) if (BFQQ_SEEKY(bfqq) && bfqq->wr_coeff == 1 && bfq_symmetric_scenario(bfqd)) sl = min_t(u64, sl, BFQ_MIN_TT); + else if (bfqq->wr_coeff > 1) + sl = max_t(u32, sl, 20ULL * NSEC_PER_MSEC); bfqd->last_idling_start = ktime_get(); hrtimer_start(&bfqd->idle_slice_timer, ns_to_ktime(sl), From patchwork Thu Mar 7 16:25:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 159882 Delivered-To: patch@linaro.org Received: by 2002:a02:5cc1:0:0:0:0:0 with SMTP id w62csp7628368jad; Thu, 7 Mar 2019 08:26:21 -0800 (PST) X-Google-Smtp-Source: APXvYqx3ChKS8wm2AYYrPHnckAhte/NuULn5i3Boh92/FZBxG10kaoq2FlRzoAc0U6zFd2PKeheK X-Received: by 2002:a62:560f:: with SMTP id k15mr13398256pfb.231.1551975981120; Thu, 07 Mar 2019 08:26:21 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1551975981; cv=none; d=google.com; s=arc-20160816; b=ouIsRoZkDyxawi0hNeB9335CLiZV4pO4wLMX1eSyRED6N+/xCitGLbFvkM3NKfLFRv 6ct3erdE+uBq2CXXlxBBQ6WWJm5ey9rAqXNjF+2NY4peLkFb9YzFD0diyFYncnNzHOpe p97NOaHAl6VAXFqE3esXxlD+HnpPQjvr8ugxQVOTQds+VH4qEwTTvBe1+XWC/zZADT7q EE1U8o4NxvkGVFRBhOd7dROMIqFTdJPvKvDh9dRCfdfLqxPHvccHmS5VeSTQOVjPiBUZ ebsSWLW8i+gkcQqYN8jFsHvtRTEflwxoLRmkYWs9+7CTGC5+oGSzbY7eZPERoPAS6sBy MbmQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=0NAvrfo3Asn/Q7s9KDBK/T3qV+c+kR0mqhtwlAgJI8Y=; b=Be8mWzBUcnVQs1YBLOYvb4IGETJMfNOqAQLt4fyN6QUxt8E08M9ch7tVTHgGTU0Uxd owv6IJJzEaxWQSH6sil/vk8AXkgvYuUy7y9AqrUQuf+YP36GyQ72GcuVDBTo/ZD3xAZG bmU2QVxZvmyLCofbO1gxPyZhYIzG6SiQQ9RJd+rSfBUt4Eyr7urCgSlnw3CdHaIoZv62 +rO0fYdBQz3wn0O2wh0lyp38skEo7mNPIbd3t0bv9wcyX0d8V1qSAAIZIQEvQZ46rvfX BFczSN9VWEGAeJWwdYRvN+0evOdCiIirrVZ+LMEeFkAldhDq589ayRRCUUzBT93ZAlYP bOHQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=m9my2F8P; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id u185si4059388pgd.79.2019.03.07.08.26.20; Thu, 07 Mar 2019 08:26:21 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=m9my2F8P; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726436AbfCGQ0S (ORCPT + 31 others); Thu, 7 Mar 2019 11:26:18 -0500 Received: from mail-wr1-f45.google.com ([209.85.221.45]:38651 "EHLO mail-wr1-f45.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726127AbfCGQ0R (ORCPT ); Thu, 7 Mar 2019 11:26:17 -0500 Received: by mail-wr1-f45.google.com with SMTP id g12so18151927wrm.5 for ; Thu, 07 Mar 2019 08:26:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0NAvrfo3Asn/Q7s9KDBK/T3qV+c+kR0mqhtwlAgJI8Y=; b=m9my2F8PxT+VEZ6ZTPRJZ/yysIczbN1tS3x9zZ+21ou3rloElcIVaihb5lSi8/D2Kl gxAU9WdyhHy/rKGJDdhiG0l68VhkPTM3Q4eaJm3IYLIVNdPpPUlI4FKf6EIaeVjGtE7M 2lLxaOTQEbgaciU8+GLqgzTGPLiPAWPmZIqZ7qb0GL43yf5gScO0mLO/+wTTIz1V6TWH Oy57ZnVxmcWxDMdN2pFjLgAzyiPGpByhAdrxL8VTH4D2NxwfpSjUhr2L3CaSnV/GAtcJ lALerYJ/sceAqSxllZNOQqK3PW15I+nv49D/tjYOLRozWJJ6eAKzEeS15SKVR5pO0r5l rlBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0NAvrfo3Asn/Q7s9KDBK/T3qV+c+kR0mqhtwlAgJI8Y=; b=ohyfMkPbNbB7cHi9ZyC1hdpI1NNOerjNKEYaHxWwa9Z0TeiomzozsBeGV3QXHSMA10 09Rm+uy7+18n7f84qmDgoMqOMdCl2ek4nRDvW3vr1AxX6sBkidHjpQYh5bj2CTl1DtLo 4a4Zr+wGnNSL+1rvHwUxgThIRhoQm1mpgg0mQVrbQbUWrbJjmfTQ4VCROYn6jYKHFfaS qVSHISrZ0Ul4GsduHWlGt+YFyI4YDlg/VncaD0NQVWRkhOe6vco/Dv4PgNWC/uf6pVcd VlFOzfybtuotkzXlTKBGe5row/ZOFClKvYh7cSoJ4y/u90/u+SEKTPnygNtUdicl24j0 Ib4Q== X-Gm-Message-State: APjAAAWUJmdlQF2JzRGw44QQOMqK59cZOK00/dCxXuMWLNO+h9sWi+5R O8ft7a/RJsVKEC7dvFfpGcKd+w== X-Received: by 2002:adf:dfc4:: with SMTP id q4mr7703465wrn.276.1551975973567; Thu, 07 Mar 2019 08:26:13 -0800 (PST) Received: from localhost.localdomain (146-241-117-61.dyn.eolo.it. [146.241.117.61]) by smtp.gmail.com with ESMTPSA id o127sm5801797wmo.20.2019.03.07.08.26.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 Mar 2019 08:26:13 -0800 (PST) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, ulf.hansson@linaro.org, linus.walleij@linaro.org, broonie@kernel.org, bfq-iosched@googlegroups.com, oleksandr@natalenko.name, fra.fra.800@gmail.com, Paolo Valente Subject: [PATCH BUGFIX IMPROVEMENT 2/8] block, bfq: do not idle for lowest-weight queues Date: Thu, 7 Mar 2019 17:25:48 +0100 Message-Id: <20190307162554.77205-3-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190307162554.77205-1-paolo.valente@linaro.org> References: <20190307162554.77205-1-paolo.valente@linaro.org> MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In most cases, it is detrimental for throughput to plug I/O dispatch when the in-service bfq_queue becomes temporarily empty (plugging is performed to wait for the possible arrival, soon, of new I/O from the in-service queue). There is however a case where plugging is needed for service guarantees. If a bfq_queue, say Q, has a higher weight than some other active bfq_queue, and is sync, i.e., contains sync I/O, then, to guarantee that Q does receive a higher share of the throughput than other lower-weight queues, it is necessary to plug I/O dispatch when Q remains temporarily empty while being served. For this reason, BFQ performs I/O plugging when some active bfq_queue has a higher weight than some other active bfq_queue. But this is unnecessarily overkill. In fact, if the in-service bfq_queue actually has a weight lower than or equal to the other queues, then the queue *must not* be guaranteed a higher share of the throughput than the other queues. So, not plugging I/O cannot cause any harm to the queue. And can boost throughput. Taking advantage of this fact, this commit does not plug I/O for sync bfq_queues with a weight lower than or equal to the weights of the other queues. Here is an example of the resulting throughput boost with the dbench workload, which is particularly nasty for BFQ. With the dbench test in the Phoronix suite, BFQ reaches its lowest total throughput with 6 clients on a filesystem with journaling, in case the journaling daemon has a higher weight than normal processes. Before this commit, the total throughput was ~80 MB/sec on a PLEXTOR PX-256M5, after this commit it is ~100 MB/sec. Signed-off-by: Paolo Valente --- block/bfq-iosched.c | 204 +++++++++++++++++++++++++------------------- block/bfq-iosched.h | 6 +- block/bfq-wf2q.c | 2 +- 3 files changed, 118 insertions(+), 94 deletions(-) -- 2.20.1 diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index eb658de3cc40..2be504f25b09 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -629,12 +629,19 @@ void bfq_pos_tree_add_move(struct bfq_data *bfqd, struct bfq_queue *bfqq) } /* - * The following function returns true if every queue must receive the - * same share of the throughput (this condition is used when deciding - * whether idling may be disabled, see the comments in the function - * bfq_better_to_idle()). + * The following function returns false either if every active queue + * must receive the same share of the throughput (symmetric scenario), + * or, as a special case, if bfqq must receive a share of the + * throughput lower than or equal to the share that every other active + * queue must receive. If bfqq does sync I/O, then these are the only + * two cases where bfqq happens to be guaranteed its share of the + * throughput even if I/O dispatching is not plugged when bfqq remains + * temporarily empty (for more details, see the comments in the + * function bfq_better_to_idle()). For this reason, the return value + * of this function is used to check whether I/O-dispatch plugging can + * be avoided. * - * Such a scenario occurs when: + * The above first case (symmetric scenario) occurs when: * 1) all active queues have the same weight, * 2) all active queues belong to the same I/O-priority class, * 3) all active groups at the same level in the groups tree have the same @@ -654,30 +661,36 @@ void bfq_pos_tree_add_move(struct bfq_data *bfqd, struct bfq_queue *bfqq) * support or the cgroups interface are not enabled, thus no state * needs to be maintained in this case. */ -static bool bfq_symmetric_scenario(struct bfq_data *bfqd) +static bool bfq_asymmetric_scenario(struct bfq_data *bfqd, + struct bfq_queue *bfqq) { + bool smallest_weight = bfqq && + bfqq->weight_counter && + bfqq->weight_counter == + container_of( + rb_first_cached(&bfqd->queue_weights_tree), + struct bfq_weight_counter, + weights_node); + /* * For queue weights to differ, queue_weights_tree must contain * at least two nodes. */ - bool varied_queue_weights = !RB_EMPTY_ROOT(&bfqd->queue_weights_tree) && - (bfqd->queue_weights_tree.rb_node->rb_left || - bfqd->queue_weights_tree.rb_node->rb_right); + bool varied_queue_weights = !smallest_weight && + !RB_EMPTY_ROOT(&bfqd->queue_weights_tree.rb_root) && + (bfqd->queue_weights_tree.rb_root.rb_node->rb_left || + bfqd->queue_weights_tree.rb_root.rb_node->rb_right); bool multiple_classes_busy = (bfqd->busy_queues[0] && bfqd->busy_queues[1]) || (bfqd->busy_queues[0] && bfqd->busy_queues[2]) || (bfqd->busy_queues[1] && bfqd->busy_queues[2]); - /* - * For queue weights to differ, queue_weights_tree must contain - * at least two nodes. - */ - return !(varied_queue_weights || multiple_classes_busy + return varied_queue_weights || multiple_classes_busy #ifdef BFQ_GROUP_IOSCHED_ENABLED || bfqd->num_groups_with_pending_reqs > 0 #endif - ); + ; } /* @@ -694,10 +707,11 @@ static bool bfq_symmetric_scenario(struct bfq_data *bfqd) * should be low too. */ void bfq_weights_tree_add(struct bfq_data *bfqd, struct bfq_queue *bfqq, - struct rb_root *root) + struct rb_root_cached *root) { struct bfq_entity *entity = &bfqq->entity; - struct rb_node **new = &(root->rb_node), *parent = NULL; + struct rb_node **new = &(root->rb_root.rb_node), *parent = NULL; + bool leftmost = true; /* * Do not insert if the queue is already associated with a @@ -726,8 +740,10 @@ void bfq_weights_tree_add(struct bfq_data *bfqd, struct bfq_queue *bfqq, } if (entity->weight < __counter->weight) new = &((*new)->rb_left); - else + else { new = &((*new)->rb_right); + leftmost = false; + } } bfqq->weight_counter = kzalloc(sizeof(struct bfq_weight_counter), @@ -736,7 +752,7 @@ void bfq_weights_tree_add(struct bfq_data *bfqd, struct bfq_queue *bfqq, /* * In the unlucky event of an allocation failure, we just * exit. This will cause the weight of queue to not be - * considered in bfq_symmetric_scenario, which, in its turn, + * considered in bfq_asymmetric_scenario, which, in its turn, * causes the scenario to be deemed wrongly symmetric in case * bfqq's weight would have been the only weight making the * scenario asymmetric. On the bright side, no unbalance will @@ -750,7 +766,8 @@ void bfq_weights_tree_add(struct bfq_data *bfqd, struct bfq_queue *bfqq, bfqq->weight_counter->weight = entity->weight; rb_link_node(&bfqq->weight_counter->weights_node, parent, new); - rb_insert_color(&bfqq->weight_counter->weights_node, root); + rb_insert_color_cached(&bfqq->weight_counter->weights_node, root, + leftmost); inc_counter: bfqq->weight_counter->num_active++; @@ -765,7 +782,7 @@ void bfq_weights_tree_add(struct bfq_data *bfqd, struct bfq_queue *bfqq, */ void __bfq_weights_tree_remove(struct bfq_data *bfqd, struct bfq_queue *bfqq, - struct rb_root *root) + struct rb_root_cached *root) { if (!bfqq->weight_counter) return; @@ -774,7 +791,7 @@ void __bfq_weights_tree_remove(struct bfq_data *bfqd, if (bfqq->weight_counter->num_active > 0) goto reset_entity_pointer; - rb_erase(&bfqq->weight_counter->weights_node, root); + rb_erase_cached(&bfqq->weight_counter->weights_node, root); kfree(bfqq->weight_counter); reset_entity_pointer: @@ -889,7 +906,7 @@ static unsigned long bfq_serv_to_charge(struct request *rq, struct bfq_queue *bfqq) { if (bfq_bfqq_sync(bfqq) || bfqq->wr_coeff > 1 || - !bfq_symmetric_scenario(bfqq->bfqd)) + bfq_asymmetric_scenario(bfqq->bfqd, bfqq)) return blk_rq_sectors(rq); return blk_rq_sectors(rq) * bfq_async_charge_factor; @@ -2543,7 +2560,7 @@ static void bfq_arm_slice_timer(struct bfq_data *bfqd) * queue). */ if (BFQQ_SEEKY(bfqq) && bfqq->wr_coeff == 1 && - bfq_symmetric_scenario(bfqd)) + !bfq_asymmetric_scenario(bfqd, bfqq)) sl = min_t(u64, sl, BFQ_MIN_TT); else if (bfqq->wr_coeff > 1) sl = max_t(u32, sl, 20ULL * NSEC_PER_MSEC); @@ -3500,8 +3517,9 @@ static bool idling_boosts_thr_without_issues(struct bfq_data *bfqd, } /* - * There is a case where idling must be performed not for - * throughput concerns, but to preserve service guarantees. + * There is a case where idling does not have to be performed for + * throughput concerns, but to preserve the throughput share of + * the process associated with bfqq. * * To introduce this case, we can note that allowing the drive * to enqueue more than one request at a time, and hence @@ -3517,77 +3535,83 @@ static bool idling_boosts_thr_without_issues(struct bfq_data *bfqd, * concern about per-process throughput distribution, and * makes its decisions only on a per-request basis. Therefore, * the service distribution enforced by the drive's internal - * scheduler is likely to coincide with the desired - * device-throughput distribution only in a completely - * symmetric scenario where: - * (i) each of these processes must get the same throughput as - * the others; - * (ii) the I/O of each process has the same properties, in - * terms of locality (sequential or random), direction - * (reads or writes), request sizes, greediness - * (from I/O-bound to sporadic), and so on. - * In fact, in such a scenario, the drive tends to treat - * the requests of each of these processes in about the same - * way as the requests of the others, and thus to provide - * each of these processes with about the same throughput - * (which is exactly the desired throughput distribution). In - * contrast, in any asymmetric scenario, device idling is - * certainly needed to guarantee that bfqq receives its - * assigned fraction of the device throughput (see [1] for - * details). - * The problem is that idling may significantly reduce - * throughput with certain combinations of types of I/O and - * devices. An important example is sync random I/O, on flash - * storage with command queueing. So, unless bfqq falls in the - * above cases where idling also boosts throughput, it would - * be important to check conditions (i) and (ii) accurately, - * so as to avoid idling when not strictly needed for service - * guarantees. + * scheduler is likely to coincide with the desired throughput + * distribution only in a completely symmetric, or favorably + * skewed scenario where: + * (i-a) each of these processes must get the same throughput as + * the others, + * (i-b) in case (i-a) does not hold, it holds that the process + * associated with bfqq must receive a lower or equal + * throughput than any of the other processes; + * (ii) the I/O of each process has the same properties, in + * terms of locality (sequential or random), direction + * (reads or writes), request sizes, greediness + * (from I/O-bound to sporadic), and so on; + + * In fact, in such a scenario, the drive tends to treat the requests + * of each process in about the same way as the requests of the + * others, and thus to provide each of these processes with about the + * same throughput. This is exactly the desired throughput + * distribution if (i-a) holds, or, if (i-b) holds instead, this is an + * even more convenient distribution for (the process associated with) + * bfqq. + * + * In contrast, in any asymmetric or unfavorable scenario, device + * idling (I/O-dispatch plugging) is certainly needed to guarantee + * that bfqq receives its assigned fraction of the device throughput + * (see [1] for details). + * + * The problem is that idling may significantly reduce throughput with + * certain combinations of types of I/O and devices. An important + * example is sync random I/O on flash storage with command + * queueing. So, unless bfqq falls in cases where idling also boosts + * throughput, it is important to check conditions (i-a), i(-b) and + * (ii) accurately, so as to avoid idling when not strictly needed for + * service guarantees. * - * Unfortunately, it is extremely difficult to thoroughly - * check condition (ii). And, in case there are active groups, - * it becomes very difficult to check condition (i) too. In - * fact, if there are active groups, then, for condition (i) - * to become false, it is enough that an active group contains - * more active processes or sub-groups than some other active - * group. More precisely, for condition (i) to hold because of - * such a group, it is not even necessary that the group is - * (still) active: it is sufficient that, even if the group - * has become inactive, some of its descendant processes still - * have some request already dispatched but still waiting for - * completion. In fact, requests have still to be guaranteed - * their share of the throughput even after being - * dispatched. In this respect, it is easy to show that, if a - * group frequently becomes inactive while still having - * in-flight requests, and if, when this happens, the group is - * not considered in the calculation of whether the scenario - * is asymmetric, then the group may fail to be guaranteed its - * fair share of the throughput (basically because idling may - * not be performed for the descendant processes of the group, - * but it had to be). We address this issue with the - * following bi-modal behavior, implemented in the function - * bfq_symmetric_scenario(). + * Unfortunately, it is extremely difficult to thoroughly check + * condition (ii). And, in case there are active groups, it becomes + * very difficult to check conditions (i-a) and (i-b) too. In fact, + * if there are active groups, then, for conditions (i-a) or (i-b) to + * become false 'indirectly', it is enough that an active group + * contains more active processes or sub-groups than some other active + * group. More precisely, for conditions (i-a) or (i-b) to become + * false because of such a group, it is not even necessary that the + * group is (still) active: it is sufficient that, even if the group + * has become inactive, some of its descendant processes still have + * some request already dispatched but still waiting for + * completion. In fact, requests have still to be guaranteed their + * share of the throughput even after being dispatched. In this + * respect, it is easy to show that, if a group frequently becomes + * inactive while still having in-flight requests, and if, when this + * happens, the group is not considered in the calculation of whether + * the scenario is asymmetric, then the group may fail to be + * guaranteed its fair share of the throughput (basically because + * idling may not be performed for the descendant processes of the + * group, but it had to be). We address this issue with the following + * bi-modal behavior, implemented in the function + * bfq_asymmetric_scenario(). * * If there are groups with requests waiting for completion * (as commented above, some of these groups may even be * already inactive), then the scenario is tagged as * asymmetric, conservatively, without checking any of the - * conditions (i) and (ii). So the device is idled for bfqq. + * conditions (i-a), (i-b) or (ii). So the device is idled for bfqq. * This behavior matches also the fact that groups are created * exactly if controlling I/O is a primary concern (to * preserve bandwidth and latency guarantees). * - * On the opposite end, if there are no groups with requests - * waiting for completion, then only condition (i) is actually - * controlled, i.e., provided that condition (i) holds, idling - * is not performed, regardless of whether condition (ii) - * holds. In other words, only if condition (i) does not hold, - * then idling is allowed, and the device tends to be - * prevented from queueing many requests, possibly of several - * processes. Since there are no groups with requests waiting - * for completion, then, to control condition (i) it is enough - * to check just whether all the queues with requests waiting - * for completion also have the same weight. + * On the opposite end, if there are no groups with requests waiting + * for completion, then only conditions (i-a) and (i-b) are actually + * controlled, i.e., provided that conditions (i-a) or (i-b) holds, + * idling is not performed, regardless of whether condition (ii) + * holds. In other words, only if conditions (i-a) and (i-b) do not + * hold, then idling is allowed, and the device tends to be prevented + * from queueing many requests, possibly of several processes. Since + * there are no groups with requests waiting for completion, then, to + * control conditions (i-a) and (i-b) it is enough to check just + * whether all the queues with requests waiting for completion also + * have the same weight. * * Not checking condition (ii) evidently exposes bfqq to the * risk of getting less throughput than its fair share. @@ -3639,7 +3663,7 @@ static bool idling_boosts_thr_without_issues(struct bfq_data *bfqd, * compound condition that is checked below for deciding * whether the scenario is asymmetric. To explain this * compound condition, we need to add that the function - * bfq_symmetric_scenario checks the weights of only + * bfq_asymmetric_scenario checks the weights of only * non-weight-raised queues, for efficiency reasons (see * comments on bfq_weights_tree_add()). Then the fact that * bfqq is weight-raised is checked explicitly here. More @@ -3667,7 +3691,7 @@ static bool idling_needed_for_service_guarantees(struct bfq_data *bfqd, return (bfqq->wr_coeff > 1 && bfqd->wr_busy_queues < bfq_tot_busy_queues(bfqd)) || - !bfq_symmetric_scenario(bfqd); + bfq_asymmetric_scenario(bfqd, bfqq); } /* @@ -5505,7 +5529,7 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_type *e) HRTIMER_MODE_REL); bfqd->idle_slice_timer.function = bfq_idle_slice_timer; - bfqd->queue_weights_tree = RB_ROOT; + bfqd->queue_weights_tree = RB_ROOT_CACHED; bfqd->num_groups_with_pending_reqs = 0; INIT_LIST_HEAD(&bfqd->active_list); diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h index 062e1c4787f4..81cabf51a87e 100644 --- a/block/bfq-iosched.h +++ b/block/bfq-iosched.h @@ -450,7 +450,7 @@ struct bfq_data { * weight-raised @bfq_queue (see the comments to the functions * bfq_weights_tree_[add|remove] for further details). */ - struct rb_root queue_weights_tree; + struct rb_root_cached queue_weights_tree; /* * Number of groups with at least one descendant process that @@ -898,10 +898,10 @@ void bic_set_bfqq(struct bfq_io_cq *bic, struct bfq_queue *bfqq, bool is_sync); struct bfq_data *bic_to_bfqd(struct bfq_io_cq *bic); void bfq_pos_tree_add_move(struct bfq_data *bfqd, struct bfq_queue *bfqq); void bfq_weights_tree_add(struct bfq_data *bfqd, struct bfq_queue *bfqq, - struct rb_root *root); + struct rb_root_cached *root); void __bfq_weights_tree_remove(struct bfq_data *bfqd, struct bfq_queue *bfqq, - struct rb_root *root); + struct rb_root_cached *root); void bfq_weights_tree_remove(struct bfq_data *bfqd, struct bfq_queue *bfqq); void bfq_bfqq_expire(struct bfq_data *bfqd, struct bfq_queue *bfqq, diff --git a/block/bfq-wf2q.c b/block/bfq-wf2q.c index 63311d1ff1ed..0e3f344cc4d3 100644 --- a/block/bfq-wf2q.c +++ b/block/bfq-wf2q.c @@ -737,7 +737,7 @@ __bfq_entity_update_weight_prio(struct bfq_service_tree *old_st, struct bfq_queue *bfqq = bfq_entity_to_bfqq(entity); unsigned int prev_weight, new_weight; struct bfq_data *bfqd = NULL; - struct rb_root *root; + struct rb_root_cached *root; #ifdef CONFIG_BFQ_GROUP_IOSCHED struct bfq_sched_data *sd; struct bfq_group *bfqg; From patchwork Thu Mar 7 16:25:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 159884 Delivered-To: patch@linaro.org Received: by 2002:a02:5cc1:0:0:0:0:0 with SMTP id w62csp7628515jad; Thu, 7 Mar 2019 08:26:30 -0800 (PST) X-Google-Smtp-Source: APXvYqxoSk6P4GdIs7P3s78omJt3RvPI5q0SDQXvs9Aj9d+XOckxhOSyL0C2TIlj1jrY0K5Oqiw6 X-Received: by 2002:a17:902:4464:: with SMTP id k91mr13788929pld.287.1551975990192; Thu, 07 Mar 2019 08:26:30 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1551975990; cv=none; d=google.com; s=arc-20160816; b=GuhA6sb2F+uKdU9kaAt/EFTlLKjYQjoBhtJzZIihBd/t2vvHN93IPt+iAjeDN10DBR VHwqadNWzBPe/z2OBePJm1ztmH6UHdca2xDdIfTPcS5zKoSbO9nlSB2H8BkvbXw9L+bd 6zTzQ8REYA1enJsxfWUVrsLLavTQloWn3Hk3Xt87IztRRQfGhX+xRcWxJwEcUfQLt7rG hUaTtZx5Y/Qwl9gc+CIpxcpRAyNvOxPBVT+NogAGtje19iNNXL+8Xn4Apk4UIfhOdGiB RV2AZTLA4ktvueN/D8b/zpebtVmnkHOwjYTAmjESf3AeRmXh6dS0tQa7Swc3IZENy0Ti ZFmw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=eowyCyg6fE0mBjV4jlMIOF/lyn5g/MwdTODgsX3vJKg=; b=EyiN3xD+oPchGmwR7L2gdbpwzdgUfkeUA7AtW+/ORei9HSZEVXlZCq7MVsSzG35kgX 7KSBiq5wDw8QkkgYvUN7T77fIBMqoO/ttt6qGHno8QB8RBV+feAfceXvkvxjCdDnARp8 oNQoR9zTWuIBWNuOMF2vHDRoW/CuNRV111hdXQXK8PA4LGODc4hreSkImeMxYalfwZLv N8yzDxm44ChaCZU1a9ndJZNG+Kbdri22XlzK4mMsxSmqmUszfihZ9rV/xRZj0oOkr4Nm 864kpcAJgXdcsYxF4XIpgT0M4j0yFSGJZp+N3sssMtTnP32EZoqSqtJ2RFcN3F0U/XuS WUeA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=j1z8IoVL; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id w190si4146952pgd.105.2019.03.07.08.26.29; Thu, 07 Mar 2019 08:26:30 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=j1z8IoVL; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726536AbfCGQ01 (ORCPT + 31 others); Thu, 7 Mar 2019 11:26:27 -0500 Received: from mail-wr1-f47.google.com ([209.85.221.47]:41255 "EHLO mail-wr1-f47.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726380AbfCGQ0T (ORCPT ); Thu, 7 Mar 2019 11:26:19 -0500 Received: by mail-wr1-f47.google.com with SMTP id n2so18151531wrw.8 for ; Thu, 07 Mar 2019 08:26:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=eowyCyg6fE0mBjV4jlMIOF/lyn5g/MwdTODgsX3vJKg=; b=j1z8IoVLBlHnjS6c5OYsK0/iuUGE0NDDs/a9qeTCNomH1XJLqdrvT8LIZusLsXXTYg 9KNHJVjoHvJORgpT91vl6X22eWkGMbBmOp4nC+m8tVvBsmftlQSy0vQC/6q2uzjo7SPe XQtodrdvVfkNySDeJ04xsc6fjXJR0QVDNBYRMhO8HOADciYb+EFiUwmZaq0wLkSufy8z hmZ/eqLYM6u8s7+eqmCiD7hBX5m3QsONa3nGwx2FsYTWCLX5mAkAgxoKgqg7QKXbH0m3 1bUU8ygiL4whVNvQKC1B4EMozQ97smQjgvpC3uOk7nuPp0PyoV2nYqqRjMnwSq4ZslXq KoIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=eowyCyg6fE0mBjV4jlMIOF/lyn5g/MwdTODgsX3vJKg=; b=VzKz0N4ZVtSRKSdOW+Kw3bD2CvQVSwUcwgctlLvbsANSSFgjt8A1epkI+VQuaraQQN ywBov6orDwVwUbLdh880lHKOK7pmkJYI7uRjtRpHHH5T0N51nVrJ4anfO6uDhZCI7HOx aJgziUh37DMvbO6wTtgPxBEgETLF+PmqOj4jnYg9q9aAb97lfZNI+8BJgmug3rjg5fv2 8/7PNNkJMKjR8HPZie+0k8PHqRU9HrY4RHrBKHGIO68mR777qjf7k8+aarjfgB5AnwQv YXUOUwXLsDK88sWDkriECrj+LtgzBR5QUOd0lcMIbwwn5BvV+5Zh0VbKjt25bSj9stYM Jz7Q== X-Gm-Message-State: APjAAAWFYK1g0M+QsHoUEyzd0JTrY6tleX39c59yZlSvOpX22+GTtkdj TmHZXP2/BGZ1lBpxaFyF0w+c2Q== X-Received: by 2002:a5d:464b:: with SMTP id j11mr958768wrs.307.1551975975117; Thu, 07 Mar 2019 08:26:15 -0800 (PST) Received: from localhost.localdomain (146-241-117-61.dyn.eolo.it. [146.241.117.61]) by smtp.gmail.com with ESMTPSA id o127sm5801797wmo.20.2019.03.07.08.26.13 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 Mar 2019 08:26:14 -0800 (PST) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, ulf.hansson@linaro.org, linus.walleij@linaro.org, broonie@kernel.org, bfq-iosched@googlegroups.com, oleksandr@natalenko.name, fra.fra.800@gmail.com, Paolo Valente Subject: [PATCH BUGFIX IMPROVEMENT 3/8] block, bfq: tune service injection basing on request service times Date: Thu, 7 Mar 2019 17:25:49 +0100 Message-Id: <20190307162554.77205-4-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190307162554.77205-1-paolo.valente@linaro.org> References: <20190307162554.77205-1-paolo.valente@linaro.org> MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The processes associated with a bfq_queue, say Q, may happen to generate their cumulative I/O at a lower rate than the rate at which the device could serve the same I/O. This is rather probable, e.g., if only one process is associated with Q and the device is an SSD. It results in Q becoming often empty while in service. If BFQ is not allowed to switch to another queue when Q becomes empty, then, during the service of Q, there will be frequent "service holes", i.e., time intervals during which Q gets empty and the device can only consume the I/O already queued in its hardware queues. This easily causes considerable losses of throughput. To counter this problem, BFQ implements a request injection mechanism, which tries to fill the above service holes with I/O requests taken from other bfq_queues. The hard part in this mechanism is finding the right amount of I/O to inject, so as to both boost throughput and not break Q's bandwidth and latency guarantees. To this goal, the current version of this mechanism measures the bandwidth enjoyed by Q while it is being served, and tries to inject the maximum possible amount of extra service that does not cause Q's bandwidth to decrease too much. This solution has an important shortcoming. For bandwidth measurements to be stable and reliable, Q must remain in service for a much longer time than that needed to serve a single I/O request. Unfortunately, this does not hold with many workloads. This commit addresses this issue by changing the way the amount of injection allowed is dynamically computed. It tunes injection as a function of the service times of single I/O requests of Q, instead of Q's bandwidth. Single-request service times are evidently meaningful even if Q gets very few I/O requests completed while it is in service. As a testbed for this new solution, we measured the throughput reached by BFQ for one of the nastiest workloads and configurations for this scheduler: the workload generated by the dbench test (in the Phoronix suite), with 6 clients, on a filesystem with journaling, and with the journaling daemon enjoying a higher weight than normal processes. With this commit, the throughput grows from ~100 MB/s to ~150 MB/s on a PLEXTOR PX-256M5. Tested-by: Francesco Pollicino Signed-off-by: Paolo Valente --- block/bfq-iosched.c | 417 ++++++++++++++++++++++++++++++++++++++++---- block/bfq-iosched.h | 51 +++--- 2 files changed, 409 insertions(+), 59 deletions(-) -- 2.20.1 diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 2be504f25b09..41364c0cca8c 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -1721,6 +1721,123 @@ static void bfq_add_request(struct request *rq) bfqq->queued[rq_is_sync(rq)]++; bfqd->queued++; + if (RB_EMPTY_ROOT(&bfqq->sort_list) && bfq_bfqq_sync(bfqq)) { + /* + * Periodically reset inject limit, to make sure that + * the latter eventually drops in case workload + * changes, see step (3) in the comments on + * bfq_update_inject_limit(). + */ + if (time_is_before_eq_jiffies(bfqq->decrease_time_jif + + msecs_to_jiffies(1000))) { + /* invalidate baseline total service time */ + bfqq->last_serv_time_ns = 0; + + /* + * Reset pointer in case we are waiting for + * some request completion. + */ + bfqd->waited_rq = NULL; + + /* + * If bfqq has a short think time, then start + * by setting the inject limit to 0 + * prudentially, because the service time of + * an injected I/O request may be higher than + * the think time of bfqq, and therefore, if + * one request was injected when bfqq remains + * empty, this injected request might delay + * the service of the next I/O request for + * bfqq significantly. In case bfqq can + * actually tolerate some injection, then the + * adaptive update will however raise the + * limit soon. This lucky circumstance holds + * exactly because bfqq has a short think + * time, and thus, after remaining empty, is + * likely to get new I/O enqueued---and then + * completed---before being expired. This is + * the very pattern that gives the + * limit-update algorithm the chance to + * measure the effect of injection on request + * service times, and then to update the limit + * accordingly. + * + * On the opposite end, if bfqq has a long + * think time, then start directly by 1, + * because: + * a) on the bright side, keeping at most one + * request in service in the drive is unlikely + * to cause any harm to the latency of bfqq's + * requests, as the service time of a single + * request is likely to be lower than the + * think time of bfqq; + * b) on the downside, after becoming empty, + * bfqq is likely to expire before getting its + * next request. With this request arrival + * pattern, it is very hard to sample total + * service times and update the inject limit + * accordingly (see comments on + * bfq_update_inject_limit()). So the limit is + * likely to be never, or at least seldom, + * updated. As a consequence, by setting the + * limit to 1, we avoid that no injection ever + * occurs with bfqq. On the downside, this + * proactive step further reduces chances to + * actually compute the baseline total service + * time. Thus it reduces chances to execute the + * limit-update algorithm and possibly raise the + * limit to more than 1. + */ + if (bfq_bfqq_has_short_ttime(bfqq)) + bfqq->inject_limit = 0; + else + bfqq->inject_limit = 1; + bfqq->decrease_time_jif = jiffies; + } + + /* + * The following conditions must hold to setup a new + * sampling of total service time, and then a new + * update of the inject limit: + * - bfqq is in service, because the total service + * time is evaluated only for the I/O requests of + * the queues in service; + * - this is the right occasion to compute or to + * lower the baseline total service time, because + * there are actually no requests in the drive, + * or + * the baseline total service time is available, and + * this is the right occasion to compute the other + * quantity needed to update the inject limit, i.e., + * the total service time caused by the amount of + * injection allowed by the current value of the + * limit. It is the right occasion because injection + * has actually been performed during the service + * hole, and there are still in-flight requests, + * which are very likely to be exactly the injected + * requests, or part of them; + * - the minimum interval for sampling the total + * service time and updating the inject limit has + * elapsed. + */ + if (bfqq == bfqd->in_service_queue && + (bfqd->rq_in_driver == 0 || + (bfqq->last_serv_time_ns > 0 && + bfqd->rqs_injected && bfqd->rq_in_driver > 0)) && + time_is_before_eq_jiffies(bfqq->decrease_time_jif + + msecs_to_jiffies(100))) { + bfqd->last_empty_occupied_ns = ktime_get_ns(); + /* + * Start the state machine for measuring the + * total service time of rq: setting + * wait_dispatch will cause bfqd->waited_rq to + * be set when rq will be dispatched. + */ + bfqd->wait_dispatch = true; + bfqd->rqs_injected = false; + } + } + elv_rb_add(&bfqq->sort_list, rq); /* @@ -2566,6 +2683,8 @@ static void bfq_arm_slice_timer(struct bfq_data *bfqd) sl = max_t(u32, sl, 20ULL * NSEC_PER_MSEC); bfqd->last_idling_start = ktime_get(); + bfqd->last_idling_start_jiffies = jiffies; + hrtimer_start(&bfqd->idle_slice_timer, ns_to_ktime(sl), HRTIMER_MODE_REL); bfqg_stats_set_start_idle_time(bfqq_group(bfqq)); @@ -3240,13 +3359,6 @@ static unsigned long bfq_bfqq_softrt_next_start(struct bfq_data *bfqd, jiffies + nsecs_to_jiffies(bfqq->bfqd->bfq_slice_idle) + 4); } -static bool bfq_bfqq_injectable(struct bfq_queue *bfqq) -{ - return BFQQ_SEEKY(bfqq) && bfqq->wr_coeff == 1 && - blk_queue_nonrot(bfqq->bfqd->queue) && - bfqq->bfqd->hw_tag; -} - /** * bfq_bfqq_expire - expire a queue. * @bfqd: device owning the queue. @@ -3361,6 +3473,14 @@ void bfq_bfqq_expire(struct bfq_data *bfqd, "expire (%d, slow %d, num_disp %d, short_ttime %d)", reason, slow, bfqq->dispatched, bfq_bfqq_has_short_ttime(bfqq)); + /* + * bfqq expired, so no total service time needs to be computed + * any longer: reset state machine for measuring total service + * times. + */ + bfqd->rqs_injected = bfqd->wait_dispatch = false; + bfqd->waited_rq = NULL; + /* * Increase, decrease or leave budget unchanged according to * reason. @@ -3372,8 +3492,6 @@ void bfq_bfqq_expire(struct bfq_data *bfqd, if (ref == 1) /* bfqq is gone, no more actions on it */ return; - bfqq->injected_service = 0; - /* mark bfqq as waiting a request only if a bic still points to it */ if (!bfq_bfqq_busy(bfqq) && reason != BFQQE_BUDGET_TIMEOUT && @@ -3767,26 +3885,98 @@ static bool bfq_bfqq_must_idle(struct bfq_queue *bfqq) return RB_EMPTY_ROOT(&bfqq->sort_list) && bfq_better_to_idle(bfqq); } -static struct bfq_queue *bfq_choose_bfqq_for_injection(struct bfq_data *bfqd) +/* + * This function chooses the queue from which to pick the next extra + * I/O request to inject, if it finds a compatible queue. See the + * comments on bfq_update_inject_limit() for details on the injection + * mechanism, and for the definitions of the quantities mentioned + * below. + */ +static struct bfq_queue * +bfq_choose_bfqq_for_injection(struct bfq_data *bfqd) { - struct bfq_queue *bfqq; + struct bfq_queue *bfqq, *in_serv_bfqq = bfqd->in_service_queue; + unsigned int limit = in_serv_bfqq->inject_limit; + /* + * If + * - bfqq is not weight-raised and therefore does not carry + * time-critical I/O, + * or + * - regardless of whether bfqq is weight-raised, bfqq has + * however a long think time, during which it can absorb the + * effect of an appropriate number of extra I/O requests + * from other queues (see bfq_update_inject_limit for + * details on the computation of this number); + * then injection can be performed without restrictions. + */ + bool in_serv_always_inject = in_serv_bfqq->wr_coeff == 1 || + !bfq_bfqq_has_short_ttime(in_serv_bfqq); /* - * A linear search; but, with a high probability, very few - * steps are needed to find a candidate queue, i.e., a queue - * with enough budget left for its next request. In fact: + * If + * - the baseline total service time could not be sampled yet, + * so the inject limit happens to be still 0, and + * - a lot of time has elapsed since the plugging of I/O + * dispatching started, so drive speed is being wasted + * significantly; + * then temporarily raise inject limit to one request. + */ + if (limit == 0 && in_serv_bfqq->last_serv_time_ns == 0 && + bfq_bfqq_wait_request(in_serv_bfqq) && + time_is_before_eq_jiffies(bfqd->last_idling_start_jiffies + + bfqd->bfq_slice_idle) + ) + limit = 1; + + if (bfqd->rq_in_driver >= limit) + return NULL; + + /* + * Linear search of the source queue for injection; but, with + * a high probability, very few steps are needed to find a + * candidate queue, i.e., a queue with enough budget left for + * its next request. In fact: * - BFQ dynamically updates the budget of every queue so as * to accommodate the expected backlog of the queue; * - if a queue gets all its requests dispatched as injected * service, then the queue is removed from the active list - * (and re-added only if it gets new requests, but with - * enough budget for its new backlog). + * (and re-added only if it gets new requests, but then it + * is assigned again enough budget for its new backlog). */ list_for_each_entry(bfqq, &bfqd->active_list, bfqq_list) if (!RB_EMPTY_ROOT(&bfqq->sort_list) && + (in_serv_always_inject || bfqq->wr_coeff > 1) && bfq_serv_to_charge(bfqq->next_rq, bfqq) <= - bfq_bfqq_budget_left(bfqq)) - return bfqq; + bfq_bfqq_budget_left(bfqq)) { + /* + * Allow for only one large in-flight request + * on non-rotational devices, for the + * following reason. On non-rotationl drives, + * large requests take much longer than + * smaller requests to be served. In addition, + * the drive prefers to serve large requests + * w.r.t. to small ones, if it can choose. So, + * having more than one large requests queued + * in the drive may easily make the next first + * request of the in-service queue wait for so + * long to break bfqq's service guarantees. On + * the bright side, large requests let the + * drive reach a very high throughput, even if + * there is only one in-flight large request + * at a time. + */ + if (blk_queue_nonrot(bfqd->queue) && + blk_rq_sectors(bfqq->next_rq) >= + BFQQ_SECT_THR_NONROT) + limit = min_t(unsigned int, 1, limit); + else + limit = in_serv_bfqq->inject_limit; + + if (bfqd->rq_in_driver < limit) { + bfqd->rqs_injected = true; + return bfqq; + } + } return NULL; } @@ -3873,14 +4063,32 @@ static struct bfq_queue *bfq_select_queue(struct bfq_data *bfqd) * for a new request, or has requests waiting for a completion and * may idle after their completion, then keep it anyway. * - * Yet, to boost throughput, inject service from other queues if - * possible. + * Yet, inject service from other queues if it boosts + * throughput and is possible. */ if (bfq_bfqq_wait_request(bfqq) || (bfqq->dispatched != 0 && bfq_better_to_idle(bfqq))) { - if (bfq_bfqq_injectable(bfqq) && - bfqq->injected_service * bfqq->inject_coeff < - bfqq->entity.service * 10) + struct bfq_queue *async_bfqq = + bfqq->bic && bfqq->bic->bfqq[0] && + bfq_bfqq_busy(bfqq->bic->bfqq[0]) ? + bfqq->bic->bfqq[0] : NULL; + + /* + * If the process associated with bfqq has also async + * I/O pending, then inject it + * unconditionally. Injecting I/O from the same + * process can cause no harm to the process. On the + * contrary, it can only increase bandwidth and reduce + * latency for the process. + */ + if (async_bfqq && + icq_to_bic(async_bfqq->next_rq->elv.icq) == bfqq->bic && + bfq_serv_to_charge(async_bfqq->next_rq, async_bfqq) <= + bfq_bfqq_budget_left(async_bfqq)) + bfqq = bfqq->bic->bfqq[0]; + else if (!idling_boosts_thr_without_issues(bfqd, bfqq) && + (bfqq->wr_coeff == 1 || bfqd->wr_busy_queues > 1 || + !bfq_bfqq_has_short_ttime(bfqq))) bfqq = bfq_choose_bfqq_for_injection(bfqd); else bfqq = NULL; @@ -3972,15 +4180,15 @@ static struct request *bfq_dispatch_rq_from_bfqq(struct bfq_data *bfqd, bfq_bfqq_served(bfqq, service_to_charge); - bfq_dispatch_remove(bfqd->queue, rq); + if (bfqq == bfqd->in_service_queue && bfqd->wait_dispatch) { + bfqd->wait_dispatch = false; + bfqd->waited_rq = rq; + } - if (bfqq != bfqd->in_service_queue) { - if (likely(bfqd->in_service_queue)) - bfqd->in_service_queue->injected_service += - bfq_serv_to_charge(rq, bfqq); + bfq_dispatch_remove(bfqd->queue, rq); + if (bfqq != bfqd->in_service_queue) goto return_rq; - } /* * If weight raising has to terminate for bfqq, then next @@ -4411,13 +4619,6 @@ static void bfq_init_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq, bfq_mark_bfqq_has_short_ttime(bfqq); bfq_mark_bfqq_sync(bfqq); bfq_mark_bfqq_just_created(bfqq); - /* - * Aggressively inject a lot of service: up to 90%. - * This coefficient remains constant during bfqq life, - * but this behavior might be changed, after enough - * testing and tuning. - */ - bfqq->inject_coeff = 1; } else bfq_clear_bfqq_sync(bfqq); @@ -4976,6 +5177,147 @@ static void bfq_finish_requeue_request_body(struct bfq_queue *bfqq) bfq_put_queue(bfqq); } +/* + * The processes associated with bfqq may happen to generate their + * cumulative I/O at a lower rate than the rate at which the device + * could serve the same I/O. This is rather probable, e.g., if only + * one process is associated with bfqq and the device is an SSD. It + * results in bfqq becoming often empty while in service. In this + * respect, if BFQ is allowed to switch to another queue when bfqq + * remains empty, then the device goes on being fed with I/O requests, + * and the throughput is not affected. In contrast, if BFQ is not + * allowed to switch to another queue---because bfqq is sync and + * I/O-dispatch needs to be plugged while bfqq is temporarily + * empty---then, during the service of bfqq, there will be frequent + * "service holes", i.e., time intervals during which bfqq gets empty + * and the device can only consume the I/O already queued in its + * hardware queues. During service holes, the device may even get to + * remaining idle. In the end, during the service of bfqq, the device + * is driven at a lower speed than the one it can reach with the kind + * of I/O flowing through bfqq. + * + * To counter this loss of throughput, BFQ implements a "request + * injection mechanism", which tries to fill the above service holes + * with I/O requests taken from other queues. The hard part in this + * mechanism is finding the right amount of I/O to inject, so as to + * both boost throughput and not break bfqq's bandwidth and latency + * guarantees. In this respect, the mechanism maintains a per-queue + * inject limit, computed as below. While bfqq is empty, the injection + * mechanism dispatches extra I/O requests only until the total number + * of I/O requests in flight---i.e., already dispatched but not yet + * completed---remains lower than this limit. + * + * A first definition comes in handy to introduce the algorithm by + * which the inject limit is computed. We define as first request for + * bfqq, an I/O request for bfqq that arrives while bfqq is in + * service, and causes bfqq to switch from empty to non-empty. The + * algorithm updates the limit as a function of the effect of + * injection on the service times of only the first requests of + * bfqq. The reason for this restriction is that these are the + * requests whose service time is affected most, because they are the + * first to arrive after injection possibly occurred. + * + * To evaluate the effect of injection, the algorithm measures the + * "total service time" of first requests. We define as total service + * time of an I/O request, the time that elapses since when the + * request is enqueued into bfqq, to when it is completed. This + * quantity allows the whole effect of injection to be measured. It is + * easy to see why. Suppose that some requests of other queues are + * actually injected while bfqq is empty, and that a new request R + * then arrives for bfqq. If the device does start to serve all or + * part of the injected requests during the service hole, then, + * because of this extra service, it may delay the next invocation of + * the dispatch hook of BFQ. Then, even after R gets eventually + * dispatched, the device may delay the actual service of R if it is + * still busy serving the extra requests, or if it decides to serve, + * before R, some extra request still present in its queues. As a + * conclusion, the cumulative extra delay caused by injection can be + * easily evaluated by just comparing the total service time of first + * requests with and without injection. + * + * The limit-update algorithm works as follows. On the arrival of a + * first request of bfqq, the algorithm measures the total time of the + * request only if one of the three cases below holds, and, for each + * case, it updates the limit as described below: + * + * (1) If there is no in-flight request. This gives a baseline for the + * total service time of the requests of bfqq. If the baseline has + * not been computed yet, then, after computing it, the limit is + * set to 1, to start boosting throughput, and to prepare the + * ground for the next case. If the baseline has already been + * computed, then it is updated, in case it results to be lower + * than the previous value. + * + * (2) If the limit is higher than 0 and there are in-flight + * requests. By comparing the total service time in this case with + * the above baseline, it is possible to know at which extent the + * current value of the limit is inflating the total service + * time. If the inflation is below a certain threshold, then bfqq + * is assumed to be suffering from no perceivable loss of its + * service guarantees, and the limit is even tentatively + * increased. If the inflation is above the threshold, then the + * limit is decreased. Due to the lack of any hysteresis, this + * logic makes the limit oscillate even in steady workload + * conditions. Yet we opted for it, because it is fast in reaching + * the best value for the limit, as a function of the current I/O + * workload. To reduce oscillations, this step is disabled for a + * short time interval after the limit happens to be decreased. + * + * (3) Periodically, after resetting the limit, to make sure that the + * limit eventually drops in case the workload changes. This is + * needed because, after the limit has gone safely up for a + * certain workload, it is impossible to guess whether the + * baseline total service time may have changed, without measuring + * it again without injection. A more effective version of this + * step might be to just sample the baseline, by interrupting + * injection only once, and then to reset/lower the limit only if + * the total service time with the current limit does happen to be + * too large. + * + * More details on each step are provided in the comments on the + * pieces of code that implement these steps: the branch handling the + * transition from empty to non empty in bfq_add_request(), the branch + * handling injection in bfq_select_queue(), and the function + * bfq_choose_bfqq_for_injection(). These comments also explain some + * exceptions, made by the injection mechanism in some special cases. + */ +static void bfq_update_inject_limit(struct bfq_data *bfqd, + struct bfq_queue *bfqq) +{ + u64 tot_time_ns = ktime_get_ns() - bfqd->last_empty_occupied_ns; + unsigned int old_limit = bfqq->inject_limit; + + if (bfqq->last_serv_time_ns > 0) { + u64 threshold = (bfqq->last_serv_time_ns * 3)>>1; + + if (tot_time_ns >= threshold && old_limit > 0) { + bfqq->inject_limit--; + bfqq->decrease_time_jif = jiffies; + } else if (tot_time_ns < threshold && + old_limit < bfqd->max_rq_in_driver<<1) + bfqq->inject_limit++; + } + + /* + * Either we still have to compute the base value for the + * total service time, and there seem to be the right + * conditions to do it, or we can lower the last base value + * computed. + */ + if ((bfqq->last_serv_time_ns == 0 && bfqd->rq_in_driver == 0) || + tot_time_ns < bfqq->last_serv_time_ns) { + bfqq->last_serv_time_ns = tot_time_ns; + /* + * Now we certainly have a base value: make sure we + * start trying injection. + */ + bfqq->inject_limit = max_t(unsigned int, 1, old_limit); + } + + /* update complete, not waiting for any request completion any longer */ + bfqd->waited_rq = NULL; +} + /* * Handle either a requeue or a finish for rq. The things to do are * the same in both cases: all references to rq are to be dropped. In @@ -5020,6 +5362,9 @@ static void bfq_finish_requeue_request(struct request *rq) spin_lock_irqsave(&bfqd->lock, flags); + if (rq == bfqd->waited_rq) + bfq_update_inject_limit(bfqd, bfqq); + bfq_completed_request(bfqq, bfqd); bfq_finish_requeue_request_body(bfqq); diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h index 81cabf51a87e..26869cfbbfa9 100644 --- a/block/bfq-iosched.h +++ b/block/bfq-iosched.h @@ -240,6 +240,13 @@ struct bfq_queue { /* next ioprio and ioprio class if a change is in progress */ unsigned short new_ioprio, new_ioprio_class; + /* last total-service-time sample, see bfq_update_inject_limit() */ + u64 last_serv_time_ns; + /* limit for request injection */ + unsigned int inject_limit; + /* last time the inject limit has been decreased, in jiffies */ + unsigned long decrease_time_jif; + /* * Shared bfq_queue if queue is cooperating with one or more * other queues. @@ -357,29 +364,6 @@ struct bfq_queue { /* max service rate measured so far */ u32 max_service_rate; - /* - * Ratio between the service received by bfqq while it is in - * service, and the cumulative service (of requests of other - * queues) that may be injected while bfqq is empty but still - * in service. To increase precision, the coefficient is - * measured in tenths of unit. Here are some example of (1) - * ratios, (2) resulting percentages of service injected - * w.r.t. to the total service dispatched while bfqq is in - * service, and (3) corresponding values of the coefficient: - * 1 (50%) -> 10 - * 2 (33%) -> 20 - * 10 (9%) -> 100 - * 9.9 (9%) -> 99 - * 1.5 (40%) -> 15 - * 0.5 (66%) -> 5 - * 0.1 (90%) -> 1 - * - * So, if the coefficient is lower than 10, then - * injected service is more than bfqq service. - */ - unsigned int inject_coeff; - /* amount of service injected in current service slot */ - unsigned int injected_service; }; /** @@ -544,6 +528,26 @@ struct bfq_data { /* time of last request completion (ns) */ u64 last_completion; + /* time of last transition from empty to non-empty (ns) */ + u64 last_empty_occupied_ns; + + /* + * Flag set to activate the sampling of the total service time + * of a just-arrived first I/O request (see + * bfq_update_inject_limit()). This will cause the setting of + * waited_rq when the request is finally dispatched. + */ + bool wait_dispatch; + /* + * If set, then bfq_update_inject_limit() is invoked when + * waited_rq is eventually completed. + */ + struct request *waited_rq; + /* + * True if some request has been injected during the last service hole. + */ + bool rqs_injected; + /* time of first rq dispatch in current observation interval (ns) */ u64 first_dispatch; /* time of last rq dispatch in current observation interval (ns) */ @@ -553,6 +557,7 @@ struct bfq_data { ktime_t last_budget_start; /* beginning of the last idle slice */ ktime_t last_idling_start; + unsigned long last_idling_start_jiffies; /* number of samples in current observation interval */ int peak_rate_samples; From patchwork Thu Mar 7 16:25:50 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 159883 Delivered-To: patch@linaro.org Received: by 2002:a02:5cc1:0:0:0:0:0 with SMTP id w62csp7628468jad; Thu, 7 Mar 2019 08:26:27 -0800 (PST) X-Google-Smtp-Source: APXvYqxwiEEcYp/z1mc3SnNHWKFTUwGxw0cBYe7L+QaGtcZiWflAgFWD23tE9pGzLBITPn1d4Qrv X-Received: by 2002:a62:1e82:: with SMTP id e124mr13639426pfe.258.1551975987263; Thu, 07 Mar 2019 08:26:27 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1551975987; cv=none; d=google.com; s=arc-20160816; b=jU2WpNvSIIP5YzCeHk6oNUJwUqpi1ul580lsjY8dG3NRoUJt0z3Mn6o2vc0wJYMPBH XdrYs1OQdRGFWG6djsNUw0oTa/piXbBARa8h2KY+fC9JmMrjFQjLQlnTkidQuMSyNHvf ZQOJvZBOtq+1/vFe1CSa0Zo845IqA2hfYSSQ17YjwrDQm9+Sk589vyro6EEWQQaNpSRT pLuM9sFlMtap+LLe27XILDlFCaY7jhNvsKLK2OJWJfRufGwVzC09B4s4VFDg+JXO5ekS +9oJjmPFAJGqX/Ys155tnZU0AOdcPdi823U7mGw1uSuyo5/FK8SOOBo3dP9WWsRC5eOA Ae+w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=9QZLgskSCrdBD9un8CU62IVDYuz5MmLQQB3oIqrmxZE=; b=xYWFY2S9b3QihRKukV2B/hquTAl5XkzdJ3UV6hD5k86cLvbIEKTRh55BCb0mRZVPAh Ots10Tknb+kMhAEypkF4NaGh/eTNiLikeblREhWGwrVlRtsHbDcYKor3F/FdH/v9a87/ HqVslPgOk2DEE76odusPripvu3Z7VhkT9/6ktWrM5IUVodQwUnLYtUqyAP/c7IqL6zkY CLe6i7dhBf0Chddlz36EAMmiR2bVTzAju1H/EmK5USU6qEUB+9QFAIeK8jBuRuzFixB/ J22Kv+fVIYc2919nuZ7G9Z1il3dVSmbwB5jN9CI9ZKTThgKcYP05EHYLafh5ExnkIrpi l3aA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=tjyoaWfK; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id bd11si4283160plb.157.2019.03.07.08.26.25; Thu, 07 Mar 2019 08:26:27 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=tjyoaWfK; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726504AbfCGQ0Y (ORCPT + 31 others); Thu, 7 Mar 2019 11:26:24 -0500 Received: from mail-wr1-f68.google.com ([209.85.221.68]:36257 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726404AbfCGQ0T (ORCPT ); Thu, 7 Mar 2019 11:26:19 -0500 Received: by mail-wr1-f68.google.com with SMTP id o17so18166517wrw.3 for ; Thu, 07 Mar 2019 08:26:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9QZLgskSCrdBD9un8CU62IVDYuz5MmLQQB3oIqrmxZE=; b=tjyoaWfK+HDBZepAXYP0d91deWi5pF2hxjojvl2j4nqYnUeK/GX4i0hoxcN00Y6wVF XJnEKqwJoaMYkTU/dS6Oro+keZgr8s3/KgfWXyQ2SfCvWfcGvEH+PGg5cPc0/pIVfIG7 V+J2tM18i6kG7fuxnipRLAycXriyklmSCFOGbdIobZ96KAozWL4kf+0hRvwCXI6V/1UX bGDkesg4I0+7Dx6nB/GVgAC0qNDyaLI7ZoawkJjJiYBrmHB9NMcD6bLjc6YW06Rpc7hm L53tH2ZDmZqx21sMgX/8ErJRS7pQL18f5uwhN3M6q+h15AqOZrcXGOHRjsNMcZzoQNvT cWJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9QZLgskSCrdBD9un8CU62IVDYuz5MmLQQB3oIqrmxZE=; b=uHFKFotuN8n5RzvwYtrGWKgcd3+/+hUB5XX8Jq2mkQLG0Cr+obsn/oP8bgT1kIuExB U8EjS+RVCFep1uJbN/IxcyH+ivE3Vvfj3xW8hi1eX8aCyxkENHkmn4wlDPaNS2kRUsVQ nsR/rP00BFQWVGrZpm0cxn8Jmex/qy4UZnhc1WQhfiF7PjNI0xZ8phDlKLcj7DMvBu25 tjdKajF57zznz1eOvKL0mb3yXxMpbpvdskA6yt+ARSuPtFRPfsHVaqTGPSObiYAA+uTs h4coJiZKbSUAvHY/6+oEf4nA41umQk2sPwhVE2gXETw/JmpUjHK7KCvGBFJ7FY6HCI7D kD1g== X-Gm-Message-State: APjAAAXFuRO5fAoqvtFaGItJJL4gMVJrYt2C/dNXHxqawpVWzy6uDvM4 mwz1RuG6L2sPwnz3q8c//SiBsA== X-Received: by 2002:adf:eb89:: with SMTP id t9mr7264180wrn.21.1551975976529; Thu, 07 Mar 2019 08:26:16 -0800 (PST) Received: from localhost.localdomain (146-241-117-61.dyn.eolo.it. [146.241.117.61]) by smtp.gmail.com with ESMTPSA id o127sm5801797wmo.20.2019.03.07.08.26.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 Mar 2019 08:26:15 -0800 (PST) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, ulf.hansson@linaro.org, linus.walleij@linaro.org, broonie@kernel.org, bfq-iosched@googlegroups.com, oleksandr@natalenko.name, fra.fra.800@gmail.com, Paolo Valente , Alessio Masola Subject: [PATCH BUGFIX IMPROVEMENT 4/8] block, bfq: do not merge queues on flash storage with queueing Date: Thu, 7 Mar 2019 17:25:50 +0100 Message-Id: <20190307162554.77205-5-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190307162554.77205-1-paolo.valente@linaro.org> References: <20190307162554.77205-1-paolo.valente@linaro.org> MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org To boost throughput with a set of processes doing interleaved I/O (i.e., a set of processes whose individual I/O is random, but whose merged cumulative I/O is sequential), BFQ merges the queues associated with these processes, i.e., redirects the I/O of these processes into a common, shared queue. In the shared queue, I/O requests are ordered by their position on the medium, thus sequential I/O gets dispatched to the device when the shared queue is served. Queue merging costs execution time, because, to detect which queues to merge, BFQ must maintain a list of the head I/O requests of active queues, ordered by request positions. Measurements showed that this costs about 10% of BFQ's total per-request processing time. Request processing time becomes more and more critical as the speed of the underlying storage device grows. Yet, fortunately, queue merging is basically useless on the very devices that are so fast to make request processing time critical. To reach a high throughput, these devices must have many requests queued at the same time. But, in this configuration, the internal scheduling algorithms of these devices do also the job of queue merging: they reorder requests so as to obtain as much as possible a sequential I/O pattern. As a consequence, with processes doing interleaved I/O, the throughput reached by one such device is likely to be the same, with and without queue merging. In view of this fact, this commit disables queue merging, and all related housekeeping, for non-rotational devices with internal queueing. The total, single-lock-protected, per-request processing time of BFQ drops to, e.g., 1.9 us on an Intel Core i7-2760QM@2.40GHz (time measured with simple code instrumentation, and using the throughput-sync.sh script of the S suite [1], in performance-profiling mode). To put this result into context, the total, single-lock-protected, per-request execution time of the lightest I/O scheduler available in blk-mq, mq-deadline, is 0.7 us (mq-deadline is ~800 LOC, against ~10500 LOC for BFQ). Disabling merging provides a further, remarkable benefit in terms of throughput. Merging tends to make many workloads artificially more uneven, mainly because of shared queues remaining non empty for incomparably more time than normal queues. So, if, e.g., one of the queues in a set of merged queues has a higher weight than a normal queue, then the shared queue may inherit such a high weight and, by staying almost always active, may force BFQ to perform I/O plugging most of the time. This evidently makes it harder for BFQ to let the device reach a high throughput. As a practical example of this problem, and of the benefits of this commit, we measured again the throughput in the nasty scenario considered in previous commit messages: dbench test (in the Phoronix suite), with 6 clients, on a filesystem with journaling, and with the journaling daemon enjoying a higher weight than normal processes. With this commit, the throughput grows from ~150 MB/s to ~200 MB/s on a PLEXTOR PX-256M5 SSD. This is the same peak throughput reached by any of the other I/O schedulers. As such, this is also likely to be the maximum possible throughput reachable with this workload on this device, because I/O is mostly random, and the other schedulers basically just pass I/O requests to the drive as fast as possible. [1] https://github.com/Algodev-github/S Tested-by: Francesco Pollicino Signed-off-by: Alessio Masola Signed-off-by: Paolo Valente --- block/bfq-cgroup.c | 3 +- block/bfq-iosched.c | 73 +++++++++++++++++++++++++++++++++++++++++---- block/bfq-iosched.h | 3 ++ 3 files changed, 73 insertions(+), 6 deletions(-) -- 2.20.1 diff --git a/block/bfq-cgroup.c b/block/bfq-cgroup.c index c6113af31960..2a74a3f2a8f7 100644 --- a/block/bfq-cgroup.c +++ b/block/bfq-cgroup.c @@ -578,7 +578,8 @@ void bfq_bfqq_move(struct bfq_data *bfqd, struct bfq_queue *bfqq, bfqg_and_blkg_get(bfqg); if (bfq_bfqq_busy(bfqq)) { - bfq_pos_tree_add_move(bfqd, bfqq); + if (unlikely(!bfqd->nonrot_with_queueing)) + bfq_pos_tree_add_move(bfqd, bfqq); bfq_activate_bfqq(bfqd, bfqq); } diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 41364c0cca8c..b96be3764b8a 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -595,7 +595,16 @@ static bool bfq_too_late_for_merging(struct bfq_queue *bfqq) bfq_merge_time_limit); } -void bfq_pos_tree_add_move(struct bfq_data *bfqd, struct bfq_queue *bfqq) +/* + * The following function is not marked as __cold because it is + * actually cold, but for the same performance goal described in the + * comments on the likely() at the beginning of + * bfq_setup_cooperator(). Unexpectedly, to reach an even lower + * execution time for the case where this function is not invoked, we + * had to add an unlikely() in each involved if(). + */ +void __cold +bfq_pos_tree_add_move(struct bfq_data *bfqd, struct bfq_queue *bfqq) { struct rb_node **p, *parent; struct bfq_queue *__bfqq; @@ -1849,8 +1858,9 @@ static void bfq_add_request(struct request *rq) /* * Adjust priority tree position, if next_rq changes. + * See comments on bfq_pos_tree_add_move() for the unlikely(). */ - if (prev != bfqq->next_rq) + if (unlikely(!bfqd->nonrot_with_queueing && prev != bfqq->next_rq)) bfq_pos_tree_add_move(bfqd, bfqq); if (!bfq_bfqq_busy(bfqq)) /* switching to busy ... */ @@ -1990,7 +2000,9 @@ static void bfq_remove_request(struct request_queue *q, bfqq->pos_root = NULL; } } else { - bfq_pos_tree_add_move(bfqd, bfqq); + /* see comments on bfq_pos_tree_add_move() for the unlikely() */ + if (unlikely(!bfqd->nonrot_with_queueing)) + bfq_pos_tree_add_move(bfqd, bfqq); } if (rq->cmd_flags & REQ_META) @@ -2075,7 +2087,12 @@ static void bfq_request_merged(struct request_queue *q, struct request *req, */ if (prev != bfqq->next_rq) { bfq_updated_next_req(bfqd, bfqq); - bfq_pos_tree_add_move(bfqd, bfqq); + /* + * See comments on bfq_pos_tree_add_move() for + * the unlikely(). + */ + if (unlikely(!bfqd->nonrot_with_queueing)) + bfq_pos_tree_add_move(bfqd, bfqq); } } } @@ -2357,6 +2374,46 @@ bfq_setup_cooperator(struct bfq_data *bfqd, struct bfq_queue *bfqq, { struct bfq_queue *in_service_bfqq, *new_bfqq; + /* + * Do not perform queue merging if the device is non + * rotational and performs internal queueing. In fact, such a + * device reaches a high speed through internal parallelism + * and pipelining. This means that, to reach a high + * throughput, it must have many requests enqueued at the same + * time. But, in this configuration, the internal scheduling + * algorithm of the device does exactly the job of queue + * merging: it reorders requests so as to obtain as much as + * possible a sequential I/O pattern. As a consequence, with + * the workload generated by processes doing interleaved I/O, + * the throughput reached by the device is likely to be the + * same, with and without queue merging. + * + * Disabling merging also provides a remarkable benefit in + * terms of throughput. Merging tends to make many workloads + * artificially more uneven, because of shared queues + * remaining non empty for incomparably more time than + * non-merged queues. This may accentuate workload + * asymmetries. For example, if one of the queues in a set of + * merged queues has a higher weight than a normal queue, then + * the shared queue may inherit such a high weight and, by + * staying almost always active, may force BFQ to perform I/O + * plugging most of the time. This evidently makes it harder + * for BFQ to let the device reach a high throughput. + * + * Finally, the likely() macro below is not used because one + * of the two branches is more likely than the other, but to + * have the code path after the following if() executed as + * fast as possible for the case of a non rotational device + * with queueing. We want it because this is the fastest kind + * of device. On the opposite end, the likely() may lengthen + * the execution time of BFQ for the case of slower devices + * (rotational or at least without queueing). But in this case + * the execution time of BFQ matters very little, if not at + * all. + */ + if (likely(bfqd->nonrot_with_queueing)) + return NULL; + /* * Prevent bfqq from being merged if it has been created too * long ago. The idea is that true cooperating processes, and @@ -2986,8 +3043,10 @@ static void __bfq_bfqq_expire(struct bfq_data *bfqd, struct bfq_queue *bfqq) bfq_requeue_bfqq(bfqd, bfqq, true); /* * Resort priority tree of potential close cooperators. + * See comments on bfq_pos_tree_add_move() for the unlikely(). */ - bfq_pos_tree_add_move(bfqd, bfqq); + if (unlikely(!bfqd->nonrot_with_queueing)) + bfq_pos_tree_add_move(bfqd, bfqq); } /* @@ -5051,6 +5110,9 @@ static void bfq_update_hw_tag(struct bfq_data *bfqd) bfqd->hw_tag = bfqd->max_rq_in_driver > BFQ_HW_QUEUE_THRESHOLD; bfqd->max_rq_in_driver = 0; bfqd->hw_tag_samples = 0; + + bfqd->nonrot_with_queueing = + blk_queue_nonrot(bfqd->queue) && bfqd->hw_tag; } static void bfq_completed_request(struct bfq_queue *bfqq, struct bfq_data *bfqd) @@ -5882,6 +5944,7 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_type *e) INIT_HLIST_HEAD(&bfqd->burst_list); bfqd->hw_tag = -1; + bfqd->nonrot_with_queueing = blk_queue_nonrot(bfqd->queue); bfqd->bfq_max_budget = bfq_default_max_budget; diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h index 26869cfbbfa9..829730b96fb2 100644 --- a/block/bfq-iosched.h +++ b/block/bfq-iosched.h @@ -497,6 +497,9 @@ struct bfq_data { /* number of requests dispatched and waiting for completion */ int rq_in_driver; + /* true if the device is non rotational and performs queueing */ + bool nonrot_with_queueing; + /* * Maximum number of requests in driver in the last * @hw_tag_samples completed requests. From patchwork Thu Mar 7 16:25:51 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 159888 Delivered-To: patch@linaro.org Received: by 2002:a02:5cc1:0:0:0:0:0 with SMTP id w62csp7628890jad; Thu, 7 Mar 2019 08:26:52 -0800 (PST) X-Google-Smtp-Source: APXvYqxvWcrx4PBEILGX3tmhdO727ynR6mu98HztqFrSYAyQ1lsmMgCWO6hupmT4xQynxAv/q0ih X-Received: by 2002:a62:1342:: with SMTP id b63mr13444338pfj.7.1551976012296; Thu, 07 Mar 2019 08:26:52 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1551976012; cv=none; d=google.com; s=arc-20160816; b=ekdByeZoE6TS72pgv+L5wgiq+RXqsxMD68mRUGKsX5n4mMudab7LgaOC2ny0ImOx+L M3Poszac1rSg6haWazhLBGxfCO0gDJLEe3b01XVvom/lyAaMUzsitBsMooEjtc/lRv8M ioJh5QVgvDgdYW5c6Y1RVwG9WkmeVnWA2oQaQ/1V2ctTw5aYZQcBvIzE6Fs36pE9/FmB LbN3snWVeQxSzc4Hr6041qQEgoNMfrpkjB9ykAJPHXCiLRsmUdeDl6XIIHzjp++mYDtp xXQ2pHffavxJmxSIQqpaScskQWUS//3vUZXVX4jyCtYG1EquEcQsR8VZxUIaBHGAzOFX hpGw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=QUHw4uD/6PdjAyafvFS/vFaSGIdhMTKmdDdLyjrgLw8=; b=seUooYhECJbxx0FJvm8ZVYrec3gwjIqDI4rBZFE1AXfLw/Ot4kvPG7nU0nJE5VdZfG uGdV9gJ9K/e4btXPVsIP4Yd94x3EBXionII/5R+pemD7oBqC4TlNWqIQCQDUiIxZKMf/ ZZcN+VS2CraZPHPg3PBbf8FC49dl8tEq1ODzJsUW0YgwQIEHY2kneRZYYSdQPp+WdrP+ N/0c/QpE64pc4JXbfN+5kyXPIzUAX1/HyGuBKSpQJzvJGpYvBpYLY8kH0rzrZfrNTnRf /OeWmOuXmZi5t+aHkpLr8eE2V2zOCpAcUwc2oaMdQNjZ2mTt6O9tC5UOlO1o/JYNiR9n i4uQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=BiU1VLV7; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id h1si4513572pll.75.2019.03.07.08.26.51; Thu, 07 Mar 2019 08:26:52 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=BiU1VLV7; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726646AbfCGQ0u (ORCPT + 31 others); Thu, 7 Mar 2019 11:26:50 -0500 Received: from mail-wr1-f51.google.com ([209.85.221.51]:39018 "EHLO mail-wr1-f51.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726127AbfCGQ0U (ORCPT ); Thu, 7 Mar 2019 11:26:20 -0500 Received: by mail-wr1-f51.google.com with SMTP id l12so1695146wrp.6 for ; Thu, 07 Mar 2019 08:26:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=QUHw4uD/6PdjAyafvFS/vFaSGIdhMTKmdDdLyjrgLw8=; b=BiU1VLV7jOAsQD02bezreQzwIST1pRkoNjFNXNMIbkymkn6y1h3pGaiPA1uO5bH2tc cONVastuhjLUsr9yYGpMMEWqjG04/YeI0pJx2g9w5UdpvPEV9TEIFRNCqFwCRkGoU7SM 5EIN1eo1fuCtsw1vF1jDOJvTvgzWRq3xSJqvy3zwSarw+l3WmHSpxHKJlCsUqpIdDweM NUAETypSqhLe60fcKAgBcNIM+d3Yc0HGgMh8spHwJovuYjTV8wS7ygNDRX7m1D25lBDw HOQnCf2m2NqIidufGcDSNtuGKRqDTLQ7alooDns/2LyDYQvekS8HhSPCkWDybqkIzDQh DzBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QUHw4uD/6PdjAyafvFS/vFaSGIdhMTKmdDdLyjrgLw8=; b=R7mHnRiyKjExXTHnVHGaD9uej/3EtiWtFXNizlTNuiWAIcgpNFdwy9hmFZ0vMpWEEr pkuGFJRDKlkDef/3U0xFjq+7ftdp9fOU7DZNI6kXjeZBBrjVD9+OUh5DFU8eXJG47sSX ikCrqffZ/wusBNs4XqNrQQ+4aUhzW0rK3qMai18aWeteKpgaSvD3lKxRF47zKAF3Sx0b 3cgsfGZhxo4+6ZVcz5UYREn80FpTFhiz2uN3NIDvS2v6Y1g4xusyxTpOqt0IpNj6gVWS u+1nJxr5LluNEt8I+w4FvseziW+9Y3bUgv7iOd+WWhnY4Leww8bMidYsX5B8XJum0vCd ZuZw== X-Gm-Message-State: APjAAAVWYomzTp3NqZ/7+IU5fnbmw6zou4OcfktzeDdciAl1FufaTusx MMOmkM1SRaZDTLcyJEe6NS1na7viTVo= X-Received: by 2002:adf:a4c9:: with SMTP id h9mr7363248wrb.254.1551975978016; Thu, 07 Mar 2019 08:26:18 -0800 (PST) Received: from localhost.localdomain (146-241-117-61.dyn.eolo.it. [146.241.117.61]) by smtp.gmail.com with ESMTPSA id o127sm5801797wmo.20.2019.03.07.08.26.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 Mar 2019 08:26:17 -0800 (PST) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, ulf.hansson@linaro.org, linus.walleij@linaro.org, broonie@kernel.org, bfq-iosched@googlegroups.com, oleksandr@natalenko.name, fra.fra.800@gmail.com, Paolo Valente Subject: [PATCH BUGFIX IMPROVEMENT 5/8] block, bfq: do not tag totally seeky queues as soft rt Date: Thu, 7 Mar 2019 17:25:51 +0100 Message-Id: <20190307162554.77205-6-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190307162554.77205-1-paolo.valente@linaro.org> References: <20190307162554.77205-1-paolo.valente@linaro.org> MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Sync random I/O is likely to be confused with soft real-time I/O, because it is characterized by limited throughput and apparently isochronous arrival pattern. To avoid false positives, this commits prevents bfq_queues containing only random (seeky) I/O from being tagged as soft real-time. Signed-off-by: Paolo Valente --- block/bfq-iosched.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) -- 2.20.1 diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index b96be3764b8a..d34b80e5c47d 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -242,6 +242,14 @@ static struct kmem_cache *bfq_pool; blk_rq_sectors(rq) < BFQQ_SECT_THR_NONROT)) #define BFQQ_CLOSE_THR (sector_t)(8 * 1024) #define BFQQ_SEEKY(bfqq) (hweight32(bfqq->seek_history) > 19) +/* + * Sync random I/O is likely to be confused with soft real-time I/O, + * because it is characterized by limited throughput and apparently + * isochronous arrival pattern. To avoid false positives, queues + * containing only random (seeky) I/O are prevented from being tagged + * as soft real-time. + */ +#define BFQQ_TOTALLY_SEEKY(bfqq) (bfqq->seek_history & -1) /* Min number of samples required to perform peak-rate update */ #define BFQ_RATE_MIN_SAMPLES 32 @@ -1622,6 +1630,7 @@ static void bfq_bfqq_handle_idle_busy_switch(struct bfq_data *bfqd, */ in_burst = bfq_bfqq_in_large_burst(bfqq); soft_rt = bfqd->bfq_wr_max_softrt_rate > 0 && + !BFQQ_TOTALLY_SEEKY(bfqq) && !in_burst && time_is_before_jiffies(bfqq->soft_rt_next_start) && bfqq->dispatched == 0; @@ -4816,6 +4825,11 @@ bfq_update_io_seektime(struct bfq_data *bfqd, struct bfq_queue *bfqq, { bfqq->seek_history <<= 1; bfqq->seek_history |= BFQ_RQ_SEEKY(bfqd, bfqq->last_request_pos, rq); + + if (bfqq->wr_coeff > 1 && + bfqq->wr_cur_max_time == bfqd->bfq_wr_rt_max_time && + BFQQ_TOTALLY_SEEKY(bfqq)) + bfq_bfqq_end_wr(bfqq); } static void bfq_update_has_short_ttime(struct bfq_data *bfqd, From patchwork Thu Mar 7 16:25:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 159886 Delivered-To: patch@linaro.org Received: by 2002:a02:5cc1:0:0:0:0:0 with SMTP id w62csp7628714jad; Thu, 7 Mar 2019 08:26:41 -0800 (PST) X-Google-Smtp-Source: APXvYqw6smrteiYDdvslQkKQ83DHiu/fJIFXE49w05V8KUj40UcNkm5KE3YNiz1n1reBssF7ls6m X-Received: by 2002:a17:902:2884:: with SMTP id f4mr13006739plb.203.1551976001398; Thu, 07 Mar 2019 08:26:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1551976001; cv=none; d=google.com; s=arc-20160816; b=PAvpVVOb0kAtgwbJ1Cm+GM9Ar3v2SsL90/WDKPSC9yVwlTXu5WjtmQurVpAXtdjVMC sb5nokxJfdiYoycInmfrrlqJu3mPpLLXBp7XwmaK5WVIbHdkdsOz9hWD576Jda/LTHMb FLyhmxFwSh6gYQkSjYQHg/oIxVCul+3w7RT+21A4sslgXy9IkY2HINp3upRV4yMaDBXl W1FsRTzb7Dz0+xrTOaN66DfNvl2XBaNC87pT2tFIdSqX368L9t2NbJ9awBXalQg8ktYI Hhwd7iQtBvAi4ohagldBvgugi0YUG5z0jFREIO3xMOCQXQlS21ZtTjvBRfhF/FBWCMri Xhjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=7PDjwfDL/90N589WNTZ9R1L1ulLSIazVR8UDi7lsijE=; b=pXYnxlNG77xiu2RP3UoczSsES5BGAy99c9DUDSoIiQc8e6HoWZn7Kfbb0IEhNvOLNt hhgQECcpepdtgbsJOt62RT/CsvWNcxvTPGtQV1CvOjDHHMvUGJ9W5di7wC0L2FOQTfX6 6OEumuuS50xl0khCODdE/uccTGrs6E576JwNMC0IisGof7QRgTRjITYLUrc1Pd1Xvj4l Jr6H7pHTKQsoy00TRgCyDT8Y57UMPttLhVnUELNBIrhSoK55jV2LwEz2JbEeNAqr5Z4R h5KbkR7OnRyCDRmIrm2WKRjAqeKnS/rgPrIDtox58ti4Mb+K31WY1mHDNXg86r/jbrnX DSWA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=OPKokFeA; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l62si4059977pge.109.2019.03.07.08.26.41; Thu, 07 Mar 2019 08:26:41 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=OPKokFeA; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726620AbfCGQ0j (ORCPT + 31 others); Thu, 7 Mar 2019 11:26:39 -0500 Received: from mail-wr1-f66.google.com ([209.85.221.66]:40654 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726443AbfCGQ0W (ORCPT ); Thu, 7 Mar 2019 11:26:22 -0500 Received: by mail-wr1-f66.google.com with SMTP id y6so3003982wrn.7 for ; Thu, 07 Mar 2019 08:26:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7PDjwfDL/90N589WNTZ9R1L1ulLSIazVR8UDi7lsijE=; b=OPKokFeAtzikn8PllTFkktRpQm4KQ9HQVTg8Y3wHu77+UZVSy8NmL6Ng2aYKHF8xkE eLTWNr4DyzddpbcKVrLduSHicJyIqo+4StGrvjdEJFWWHBtao2ZwvBRNpWLn4XGin3ti RQNEPrnDaVsm/+WgTvwJT7L9AxZhfJkW+1NuyK7+zF/eCau7pVxjq+ANrv+SKUdwJwFV b0Wc12i4lOct6c4YQ0a7s3mpxiU/EufD0BfWpf6ICIYt/zW8b+A24kJeBkjljTJFEaGB F87x/j96fhngUFOGQEkGe9PsLL8QcpYFi5JC9wTu/4XZlnMCVXT9v7eGWY/1zxGdJLI1 sc1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7PDjwfDL/90N589WNTZ9R1L1ulLSIazVR8UDi7lsijE=; b=OYNikzck3A9vzf+UjAy15FxuHyY9aGZjOQqpgAfAVp6Hth+y3hzKpQKS4a9P09uw25 PPZj+xFI6VONht8UL9PehleDPfO7+0RLlhp+ndDwML3+DQFb4dc5F3dlm2zEzegS9ZGu 0KXrApsxAbKF7AOV6u0RJryH4U1jaTolux1pdUppQwKRrt1tywiugj4uGTZYdKhIrsM3 XGw8hqW0G/IvtrQOYtNa54VFuVn3odmCBATfuNEGJQRTtfXXONxUfiImWPNLQNxacWn2 X0UiIcq92cGoTO3gKr3befoFkm2AXnUasZySTDT49kzpD0AjCgjedib89jSbl21NDpjc iGCw== X-Gm-Message-State: APjAAAVkI/MA9Ih3xURMsesDha4AjyT2TSH1sOxaF8v0SY7JtmUjuV/8 HGy96+6DK41uXobhvPavTPGTaw== X-Received: by 2002:a5d:4e44:: with SMTP id r4mr7078347wrt.228.1551975979194; Thu, 07 Mar 2019 08:26:19 -0800 (PST) Received: from localhost.localdomain (146-241-117-61.dyn.eolo.it. [146.241.117.61]) by smtp.gmail.com with ESMTPSA id o127sm5801797wmo.20.2019.03.07.08.26.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 Mar 2019 08:26:18 -0800 (PST) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, ulf.hansson@linaro.org, linus.walleij@linaro.org, broonie@kernel.org, bfq-iosched@googlegroups.com, oleksandr@natalenko.name, fra.fra.800@gmail.com, Paolo Valente Subject: [PATCH BUGFIX IMPROVEMENT 6/8] block, bfq: always protect newly-created queues from existing active queues Date: Thu, 7 Mar 2019 17:25:52 +0100 Message-Id: <20190307162554.77205-7-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190307162554.77205-1-paolo.valente@linaro.org> References: <20190307162554.77205-1-paolo.valente@linaro.org> MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org If many bfq_queues belonging to the same group happen to be created shortly after each other, then the processes associated with these queues have typically a common goal. In particular, bursts of queue creations are usually caused by services or applications that spawn many parallel threads/processes. Examples are systemd during boot, or git grep. If there are no other active queues, then, to help these processes get their job done as soon as possible, the best thing to do is to reach a high throughput. To this goal, it is usually better to not grant either weight-raising or device idling to the queues associated with these processes. And this is exactly what BFQ currently does. There is however a drawback: if, in contrast, some other queues are already active, then the newly created queues must be protected from the I/O flowing through the already existing queues. In this case, the best thing to do is the opposite as in the other case: it is much better to grant weight-raising and device idling to the newly-created queues, if they deserve it. This commit addresses this issue by doing so if there are already other active queues. This change also helps eliminating false positives, which occur when the newly-created queues do not belong to an actual large burst of creations, but some background task (e.g., a service) happens to trigger the creation of new queues in the middle, i.e., very close to when the victim queues are created. These false positive may cause total loss of control on process latencies. Signed-off-by: Paolo Valente --- block/bfq-iosched.c | 64 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 51 insertions(+), 13 deletions(-) -- 2.20.1 diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index d34b80e5c47d..500b04df9efa 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -1075,8 +1075,18 @@ static void bfq_reset_burst_list(struct bfq_data *bfqd, struct bfq_queue *bfqq) hlist_for_each_entry_safe(item, n, &bfqd->burst_list, burst_list_node) hlist_del_init(&item->burst_list_node); - hlist_add_head(&bfqq->burst_list_node, &bfqd->burst_list); - bfqd->burst_size = 1; + + /* + * Start the creation of a new burst list only if there is no + * active queue. See comments on the conditional invocation of + * bfq_handle_burst(). + */ + if (bfq_tot_busy_queues(bfqd) == 0) { + hlist_add_head(&bfqq->burst_list_node, &bfqd->burst_list); + bfqd->burst_size = 1; + } else + bfqd->burst_size = 0; + bfqd->burst_parent_entity = bfqq->entity.parent; } @@ -1132,7 +1142,8 @@ static void bfq_add_to_burst(struct bfq_data *bfqd, struct bfq_queue *bfqq) * many parallel threads/processes. Examples are systemd during boot, * or git grep. To help these processes get their job done as soon as * possible, it is usually better to not grant either weight-raising - * or device idling to their queues. + * or device idling to their queues, unless these queues must be + * protected from the I/O flowing through other active queues. * * In this comment we describe, firstly, the reasons why this fact * holds, and, secondly, the next function, which implements the main @@ -1144,7 +1155,10 @@ static void bfq_add_to_burst(struct bfq_data *bfqd, struct bfq_queue *bfqq) * cumulatively served, the sooner the target job of these queues gets * completed. As a consequence, weight-raising any of these queues, * which also implies idling the device for it, is almost always - * counterproductive. In most cases it just lowers throughput. + * counterproductive, unless there are other active queues to isolate + * these new queues from. If there no other active queues, then + * weight-raising these new queues just lowers throughput in most + * cases. * * On the other hand, a burst of queue creations may be caused also by * the start of an application that does not consist of a lot of @@ -1178,14 +1192,16 @@ static void bfq_add_to_burst(struct bfq_data *bfqd, struct bfq_queue *bfqq) * are very rare. They typically occur if some service happens to * start doing I/O exactly when the interactive task starts. * - * Turning back to the next function, it implements all the steps - * needed to detect the occurrence of a large burst and to properly - * mark all the queues belonging to it (so that they can then be - * treated in a different way). This goal is achieved by maintaining a - * "burst list" that holds, temporarily, the queues that belong to the - * burst in progress. The list is then used to mark these queues as - * belonging to a large burst if the burst does become large. The main - * steps are the following. + * Turning back to the next function, it is invoked only if there are + * no active queues (apart from active queues that would belong to the + * same, possible burst bfqq would belong to), and it implements all + * the steps needed to detect the occurrence of a large burst and to + * properly mark all the queues belonging to it (so that they can then + * be treated in a different way). This goal is achieved by + * maintaining a "burst list" that holds, temporarily, the queues that + * belong to the burst in progress. The list is then used to mark + * these queues as belonging to a large burst if the burst does become + * large. The main steps are the following. * * . when the very first queue is created, the queue is inserted into the * list (as it could be the first queue in a possible burst) @@ -5695,7 +5711,29 @@ static struct bfq_queue *bfq_init_rq(struct request *rq) } } - if (unlikely(bfq_bfqq_just_created(bfqq))) + /* + * Consider bfqq as possibly belonging to a burst of newly + * created queues only if: + * 1) A burst is actually happening (bfqd->burst_size > 0) + * or + * 2) There is no other active queue. In fact, if, in + * contrast, there are active queues not belonging to the + * possible burst bfqq may belong to, then there is no gain + * in considering bfqq as belonging to a burst, and + * therefore in not weight-raising bfqq. See comments on + * bfq_handle_burst(). + * + * This filtering also helps eliminating false positives, + * occurring when bfqq does not belong to an actual large + * burst, but some background task (e.g., a service) happens + * to trigger the creation of new queues very close to when + * bfqq and its possible companion queues are created. See + * comments on bfq_handle_burst() for further details also on + * this issue. + */ + if (unlikely(bfq_bfqq_just_created(bfqq) && + (bfqd->burst_size > 0 || + bfq_tot_busy_queues(bfqd) == 0))) bfq_handle_burst(bfqd, bfqq); return bfqq; From patchwork Thu Mar 7 16:25:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 159887 Delivered-To: patch@linaro.org Received: by 2002:a02:5cc1:0:0:0:0:0 with SMTP id w62csp7628777jad; Thu, 7 Mar 2019 08:26:45 -0800 (PST) X-Google-Smtp-Source: APXvYqx3PIh8A26ZkyBsC6JIptkTVBnzJiLHkdeh0E2/CFqzEjqHUIjvZ/RzUWvzzjSWk5W+zDgM X-Received: by 2002:a17:902:822:: with SMTP id 31mr13705785plk.290.1551976004986; Thu, 07 Mar 2019 08:26:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1551976004; cv=none; d=google.com; s=arc-20160816; b=FUlc61cGh9YAtBqoE886Lq17/xwMtfoMODS/5zwfJRQ4D7I947saa/WqnWk1eqW84n wuoj+6BRduR73rZU4OyDMPU+TsQOnJnr6LeOwB8ZLqq/9iaNVhs210wO+w4qE+CyTgJD RJrxIknZU6dYlyuxyolWgtT2ckXeBuLnH9t5Zq7hM8d0RnAqTT1MuTQ9lcBJOUtOc7dt su41OfuvmYj7SoZio8CeTc+K9z3lLJ/jqM43Qt0TbHv2WgLykbf/HWvL6NGsroK355pk yqAbdR7YbGG2Xca3m+ZuVykDdT7l0sAl1l/T9aCsul/QTMqy3yuGUoyV1eaJOHisTOd8 Vecg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=xlQD5/IUhsVAwG7WZ+Uc2ij3RLscMuoaMIxAkg1O+iU=; b=kUYf1gsTE+CdLtBBbawErBkFI9m84QXM01F2tmH9KOh1M647or7GsivFacFu676/8H pis1JDH/pfZlYdzaDJWVekkBFbgG0T/P+ZmxcEzppAVqAl2gl/N8Rh8Z0PabvB4zRGrf 3X2xd6PCD4Hl6SoRJFHBl0bajOO7Aww8r0WBfKyLe+DFiSBT+jY/HVwg8c1Y2hW+hrbq P/OjYnekps8ltX7Lk+5uSIDwnFtsUGcfnl9/SvQ7AT1cS4gy280IF1hdVyozEGySYA/d uMhzL8Gn3B+gjhd3bRswa4sflVP2/MDfYlTFu7f5Gw6GjPyAUM+VDJ8hDYha1NVHxSXd zGOw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Q1E4T6UH; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l62si4059977pge.109.2019.03.07.08.26.44; Thu, 07 Mar 2019 08:26:44 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Q1E4T6UH; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726596AbfCGQ0i (ORCPT + 31 others); Thu, 7 Mar 2019 11:26:38 -0500 Received: from mail-wr1-f67.google.com ([209.85.221.67]:47038 "EHLO mail-wr1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726450AbfCGQ0W (ORCPT ); Thu, 7 Mar 2019 11:26:22 -0500 Received: by mail-wr1-f67.google.com with SMTP id i16so18132947wrs.13 for ; Thu, 07 Mar 2019 08:26:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xlQD5/IUhsVAwG7WZ+Uc2ij3RLscMuoaMIxAkg1O+iU=; b=Q1E4T6UHR4NyVlm1gbaXS3LV4WXKIF/F0NLzALIOjQ/JI8bU/ftJotOWy8XUY8UJ0j d1e7gj4cZHb4l21rHhfjQFcTvhG+OuqT9BIhi6ORBNZ/PP3NLmZYw1FXyv5HJk4Sbyrl Hjsa4b56ODAD1QEAuleKB/+IOgGMLiEFEhVhLeRy68WCrq5MsTbojThwSnDC++kO7HI6 YBQ6TAyLL+O4BHgIDct2NxkS96no/ITQ31rBbcoPwKjomcMgLlWY4mDnENBoBcRiFPca 8a3tTQ4omdGB+GToHNIlWVfuyaMS9bmCYT/V+tS6eKNPwYlvXR5+PY3ypSoEyCzul+ui WQWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xlQD5/IUhsVAwG7WZ+Uc2ij3RLscMuoaMIxAkg1O+iU=; b=Fi2FM8m45S16itL/c6DkFfPCvNRY1M60GQBiOm7YnjfiR47b7DR0VrtSsFTODi7FXV 4VNR6YNCBa7gpfxAfxUmuNBdpoZz0exuatCPh2DukaXMeGb1CgghY8GAKycRltWUARMi bzBP5Acu4KBsPcJyBWHtvMEcufh08kTfeFG77DH76MtLqJ20dKuoInnEwEpi+pTduQ6a Ij/t5YOCHguXyVNbDhKBP+W9iL6yfR0UC18NL+ECCJfWR+oNggD55BMatqW4sqdI1vo2 Qu1ZmdZyYQeDfyuhS/r+KJ2CHKSJBHAFiVimFYkpkZTWu3AAwhY98GkPX8kLqheEyojk DYmg== X-Gm-Message-State: APjAAAXjnF7i4hDVTL/ybYzsGb0R3X/bCFIY2ilEtot/NyHCk8QvMpZJ f4IDj8wmxY0zA4WLIS6q7J7bFw== X-Received: by 2002:a5d:4e50:: with SMTP id r16mr7028459wrt.8.1551975980402; Thu, 07 Mar 2019 08:26:20 -0800 (PST) Received: from localhost.localdomain (146-241-117-61.dyn.eolo.it. [146.241.117.61]) by smtp.gmail.com with ESMTPSA id o127sm5801797wmo.20.2019.03.07.08.26.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 Mar 2019 08:26:19 -0800 (PST) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, ulf.hansson@linaro.org, linus.walleij@linaro.org, broonie@kernel.org, bfq-iosched@googlegroups.com, oleksandr@natalenko.name, fra.fra.800@gmail.com, Paolo Valente Subject: [PATCH BUGFIX IMPROVEMENT 7/8] block, bfq: print SHARED instead of pid for shared queues in logs Date: Thu, 7 Mar 2019 17:25:53 +0100 Message-Id: <20190307162554.77205-8-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190307162554.77205-1-paolo.valente@linaro.org> References: <20190307162554.77205-1-paolo.valente@linaro.org> MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Francesco Pollicino The function "bfq_log_bfqq" prints the pid of the process associated with the queue passed as input. Unfortunately, if the queue is shared, then more than one process is associated with the queue. The pid that gets printed in this case is the pid of one of the associated processes. Which process gets printed depends on the exact sequence of merge events the queue underwent. So printing such a pid is rather useless and above all is often rather confusing because it reports a random pid between those of the associated processes. This commit addresses this issue by printing SHARED instead of a pid if the queue is shared. Signed-off-by: Francesco Pollicino Signed-off-by: Paolo Valente --- block/bfq-iosched.c | 10 ++++++++++ block/bfq-iosched.h | 18 ++++++++++++++++-- 2 files changed, 26 insertions(+), 2 deletions(-) -- 2.20.1 diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 500b04df9efa..7d95d9c01036 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -2590,6 +2590,16 @@ bfq_merge_bfqqs(struct bfq_data *bfqd, struct bfq_io_cq *bic, * assignment causes no harm). */ new_bfqq->bic = NULL; + /* + * If the queue is shared, the pid is the pid of one of the associated + * processes. Which pid depends on the exact sequence of merge events + * the queue underwent. So printing such a pid is useless and confusing + * because it reports a random pid between those of the associated + * processes. + * We mark such a queue with a pid -1, and then print SHARED instead of + * a pid in logging messages. + */ + new_bfqq->pid = -1; bfqq->bic = NULL; /* release process reference to bfqq */ bfq_put_queue(bfqq); diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h index 829730b96fb2..21ddb19cc322 100644 --- a/block/bfq-iosched.h +++ b/block/bfq-iosched.h @@ -32,6 +32,8 @@ #define BFQ_DEFAULT_GRP_IOPRIO 0 #define BFQ_DEFAULT_GRP_CLASS IOPRIO_CLASS_BE +#define MAX_PID_STR_LENGTH 12 + /* * Soft real-time applications are extremely more latency sensitive * than interactive ones. Over-raise the weight of the former to @@ -1016,13 +1018,23 @@ void bfq_add_bfqq_busy(struct bfq_data *bfqd, struct bfq_queue *bfqq); /* --------------- end of interface of B-WF2Q+ ---------------- */ /* Logging facilities. */ +void bfq_pid_to_str(int pid, char *str, int len) +{ + if (pid != -1) + snprintf(str, len, "%d", pid); + else + snprintf(str, len, "SHARED-"); +} + #ifdef CONFIG_BFQ_GROUP_IOSCHED struct bfq_group *bfqq_group(struct bfq_queue *bfqq); #define bfq_log_bfqq(bfqd, bfqq, fmt, args...) do { \ + char pid_str[MAX_PID_STR_LENGTH]; \ + bfq_pid_to_str((bfqq)->pid, pid_str, MAX_PID_STR_LENGTH); \ blk_add_cgroup_trace_msg((bfqd)->queue, \ bfqg_to_blkg(bfqq_group(bfqq))->blkcg, \ - "bfq%d%c " fmt, (bfqq)->pid, \ + "bfq%s%c " fmt, pid_str, \ bfq_bfqq_sync((bfqq)) ? 'S' : 'A', ##args); \ } while (0) @@ -1034,7 +1046,9 @@ struct bfq_group *bfqq_group(struct bfq_queue *bfqq); #else /* CONFIG_BFQ_GROUP_IOSCHED */ #define bfq_log_bfqq(bfqd, bfqq, fmt, args...) \ - blk_add_trace_msg((bfqd)->queue, "bfq%d%c " fmt, (bfqq)->pid, \ + char pid_str[MAX_PID_STR_LENGTH]; \ + bfq_pid_to_str((bfqq)->pid, pid_str, MAX_PID_STR_LENGTH); \ + blk_add_trace_msg((bfqd)->queue, "bfq%s%c " fmt, pid_str, \ bfq_bfqq_sync((bfqq)) ? 'S' : 'A', \ ##args) #define bfq_log_bfqg(bfqd, bfqg, fmt, args...) do {} while (0) From patchwork Thu Mar 7 16:25:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Valente X-Patchwork-Id: 159885 Delivered-To: patch@linaro.org Received: by 2002:a02:5cc1:0:0:0:0:0 with SMTP id w62csp7628592jad; Thu, 7 Mar 2019 08:26:34 -0800 (PST) X-Google-Smtp-Source: APXvYqxM8j6PEohYFyUOsqYu2oCD6wSuI/IJpWdArAun2Wvbfdq+oyM4vxUUZhPi18znGlrt+gVi X-Received: by 2002:a17:902:a50a:: with SMTP id s10mr13427360plq.223.1551975994345; Thu, 07 Mar 2019 08:26:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1551975994; cv=none; d=google.com; s=arc-20160816; b=hETOIbOvGhTV8MrjrTuHf8Mw67008Bk6ww0soj/Upj9ch3zzaJU+JMCPjKtBNVDugv saHvlxvqqMQEiz+cnKyHrk6RM5JoZV1otETpIqwYtxJ4hAI7qyzlfreIoefKZ0NxOPHT YHUa9x2XN1/eawMhFOGHQSOHAyoASfMfogmjqArY1NA/zN3CzZ7fiF0cFWybkeWaWmwV I0RYExwM8QqlIJno7ZqCLOkYFOppDZCLwcbxC3xSxsBGObtuFSCwoiSqInkuWFZbwV/k lFEkdYCqs6DsAOx7vIuxk7Pvl8nK6FUeDuu6Zt5Nc+s3DLd5RZb0GJON9Ovz3J//cAPV ysLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=rb+YsYLv5aeFW93p5pLsMf9umO1yuPDktu059+duwqc=; b=WrMibk6LhA8V1/3dIuT2cbnqIZi4yXwVaa39cM9km4wBM3gfk4iXIMKlwbdgUgOMOr AX2COapd+VYORYyXnU/L8ZAsHfDOZbI1J65YJVkT01IccqAGfOnW6zhYVo8kBD6/l9G0 b6VMct2IVbXnZMMG2aHwDUfkoBzgfVznOOy5NLywcj7wuAR9Bozxk5o0zfn5ViL6o5BM BfxhWBY/7ueFLe2hXwWLEoOnw7Clsxl4kWyuD29XHBgeGjWAzMdGbdsfGLY3PymER+yY 3GUG4IsqreKvJtglwezcn+uqwC18Xvw//Pq+GEAOaHVCNfEq5NbqkDGsoS87lWyCnTMP rY3A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="WQ4/460q"; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id w190si4146952pgd.105.2019.03.07.08.26.34; Thu, 07 Mar 2019 08:26:34 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="WQ4/460q"; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726561AbfCGQ0c (ORCPT + 31 others); Thu, 7 Mar 2019 11:26:32 -0500 Received: from mail-wm1-f68.google.com ([209.85.128.68]:38540 "EHLO mail-wm1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726465AbfCGQ0X (ORCPT ); Thu, 7 Mar 2019 11:26:23 -0500 Received: by mail-wm1-f68.google.com with SMTP id a188so9790483wmf.3 for ; Thu, 07 Mar 2019 08:26:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=rb+YsYLv5aeFW93p5pLsMf9umO1yuPDktu059+duwqc=; b=WQ4/460qM6FlUjNVkIEygVfdwtdGU0MVo3dAb9opKQ4YIY+njZ/t3bSyeLXDABk5Bm nabD9ZMJiJFyhcTjSRHDdvjJpzNmWPy1WmpctU3uppNVE3JMHQqfNuShfsyur5AqYxe0 HtKf0IZCMtqqJqF7sNB0rZ9yiO8n7Gy1fEK7JKQteAE97aNYyBwK62roCArXJ118L609 I3YJTpKmTs1imbEnclKCPdcIgMAD830znJpyFATC2ijZGG64rE3RcWWEI3w9VIwa4YdC GdGi4tUt4Voqynhw+88B2wyjZwVYXm3vTDix5O9xgG31dP+kOuX5mSykwsdzKW6ZcWrp bcCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rb+YsYLv5aeFW93p5pLsMf9umO1yuPDktu059+duwqc=; b=nBfVov2C+vBtdgk69k8lWzfLo3Ic9D7GNSgAAdq6X2BQFVBafTVGVjiWPhi02xyrvF 36F80et2glX9qP1y4JlsxC7ui/swC0Hg3DLu2S5CpsRK/qJvR0+VCj+8cv2XnF2P1mVZ LElUq/EnnOSB1FDxM9D7sVXoM9tHtlTWZzHQ3rdSvG5UCYMX336dFiR2Fv8SZvXLwJbY V6DM22eOW+KIZDq+14ySa6hWTMA8FlxyEctrsfqI52I8n0u/cFWgYneIB+bBxme+SSUP evCQFn5jJDF9pV7dpo3U1H0Y2CfTXqkMHHNW05iQ5pwCKvfK2I4F/2KPsdEF7nhsuwCr 9n6g== X-Gm-Message-State: APjAAAU+aYBP3VZqLcRp/ft4xkWDXZvhXeo2Y//F/sQ0fbr9GP8Vkam8 GL8arjmpqJO904hLjcGWC0yIhw== X-Received: by 2002:a1c:c644:: with SMTP id w65mr6170078wmf.19.1551975981462; Thu, 07 Mar 2019 08:26:21 -0800 (PST) Received: from localhost.localdomain (146-241-117-61.dyn.eolo.it. [146.241.117.61]) by smtp.gmail.com with ESMTPSA id o127sm5801797wmo.20.2019.03.07.08.26.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 Mar 2019 08:26:20 -0800 (PST) From: Paolo Valente To: Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, ulf.hansson@linaro.org, linus.walleij@linaro.org, broonie@kernel.org, bfq-iosched@googlegroups.com, oleksandr@natalenko.name, fra.fra.800@gmail.com, Paolo Valente Subject: [PATCH BUGFIX IMPROVEMENT 8/8] block, bfq: save & resume weight on a queue merge/split Date: Thu, 7 Mar 2019 17:25:54 +0100 Message-Id: <20190307162554.77205-9-paolo.valente@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190307162554.77205-1-paolo.valente@linaro.org> References: <20190307162554.77205-1-paolo.valente@linaro.org> MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Francesco Pollicino bfq saves the state of a queue each time a merge occurs, to be able to resume such a state when the queue is associated again with its original process, on a split. Unfortunately bfq does not save & restore also the weight of the queue. If the weight is not correctly resumed when the queue is recycled, then the weight of the recycled queue could differ from the weight of the original queue. This commit adds the missing save & resume of the weight. Signed-off-by: Francesco Pollicino Signed-off-by: Paolo Valente --- block/bfq-iosched.c | 2 ++ block/bfq-iosched.h | 9 +++++++++ 2 files changed, 11 insertions(+) -- 2.20.1 diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 7d95d9c01036..1712d12340c0 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -1028,6 +1028,7 @@ bfq_bfqq_resume_state(struct bfq_queue *bfqq, struct bfq_data *bfqd, else bfq_clear_bfqq_IO_bound(bfqq); + bfqq->entity.new_weight = bic->saved_weight; bfqq->ttime = bic->saved_ttime; bfqq->wr_coeff = bic->saved_wr_coeff; bfqq->wr_start_at_switch_to_srt = bic->saved_wr_start_at_switch_to_srt; @@ -2502,6 +2503,7 @@ static void bfq_bfqq_save_state(struct bfq_queue *bfqq) if (!bic) return; + bic->saved_weight = bfqq->entity.orig_weight; bic->saved_ttime = bfqq->ttime; bic->saved_has_short_ttime = bfq_bfqq_has_short_ttime(bfqq); bic->saved_IO_bound = bfq_bfqq_IO_bound(bfqq); diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h index 21ddb19cc322..18cc0e996abf 100644 --- a/block/bfq-iosched.h +++ b/block/bfq-iosched.h @@ -404,6 +404,15 @@ struct bfq_io_cq { */ bool was_in_burst_list; + /* + * Save the weight when a merge occurs, to be able + * to restore it in case of split. If the weight is not + * correctly resumed when the queue is recycled, + * then the weight of the recycled queue could differ + * from the weight of the original queue. + */ + unsigned int saved_weight; + /* * Similar to previous fields: save wr information. */