From patchwork Wed May 13 09:44:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Greg Kroah-Hartman X-Patchwork-Id: 225936 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE, SPF_PASS, URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12B8ECA90AF for ; Wed, 13 May 2020 10:02:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DEA3620575 for ; Wed, 13 May 2020 10:02:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1589364145; bh=H6RK4Fn9JCQrahNvg7ybV844UB/P0cnu7StCAIBei8I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=DI7c7Rvbk6FOnIJd4f+ocIH86EaWRupOcX1+ufcWDdnPY26Pb9kosE7bdKjUkIdXv G4q8e8d5e91YWjeeRCv8jes5bVvKpsV5O0p1tln6F2KgIhI8WxxpqhjfS4FFkNmMtD HVTq1Noeq14ffOLY7L4k9xUrzXXrz9ANq4x95kfs= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732785AbgEMJuD (ORCPT ); Wed, 13 May 2020 05:50:03 -0400 Received: from mail.kernel.org ([198.145.29.99]:49590 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2387433AbgEMJuC (ORCPT ); Wed, 13 May 2020 05:50:02 -0400 Received: from localhost (83-86-89-107.cable.dynamic.v4.ziggo.nl [83.86.89.107]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id C5D5C23128; Wed, 13 May 2020 09:50:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1589363402; bh=H6RK4Fn9JCQrahNvg7ybV844UB/P0cnu7StCAIBei8I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NIABCVsMFeUEcsIOh0l7E83e8/AJBwu6FIxD9JFjf5wzYgfwJOzn85/kf5TbSfyB+ YufsjvSRjyXbtYqOqJfyTOBsHqF0OoTFBTsauxFCkKVbeOrDPEhFnKrVC/2Rg3jmlQ n8dqLzLDb7mDStmfqrYPexHvmoIYq2dmGK75f1Qo= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Khazhismel Kumykov , Andrew Morton , Roman Penyaev , Alexander Viro , Heiher , Jason Baron , Linus Torvalds Subject: [PATCH 5.4 59/90] eventpoll: fix missing wakeup for ovflist in ep_poll_callback Date: Wed, 13 May 2020 11:44:55 +0200 Message-Id: <20200513094415.943881845@linuxfoundation.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200513094408.810028856@linuxfoundation.org> References: <20200513094408.810028856@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Sender: stable-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org From: Khazhismel Kumykov commit 0c54a6a44bf3d41e76ce3f583a6ece267618df2e upstream. In the event that we add to ovflist, before commit 339ddb53d373 ("fs/epoll: remove unnecessary wakeups of nested epoll") we would be woken up by ep_scan_ready_list, and did no wakeup in ep_poll_callback. With that wakeup removed, if we add to ovflist here, we may never wake up. Rather than adding back the ep_scan_ready_list wakeup - which was resulting in unnecessary wakeups, trigger a wake-up in ep_poll_callback. We noticed that one of our workloads was missing wakeups starting with 339ddb53d373 and upon manual inspection, this wakeup seemed missing to me. With this patch added, we no longer see missing wakeups. I haven't yet tried to make a small reproducer, but the existing kselftests in filesystem/epoll passed for me with this patch. [khazhy@google.com: use if/elif instead of goto + cleanup suggested by Roman] Link: http://lkml.kernel.org/r/20200424190039.192373-1-khazhy@google.com Fixes: 339ddb53d373 ("fs/epoll: remove unnecessary wakeups of nested epoll") Signed-off-by: Khazhismel Kumykov Signed-off-by: Andrew Morton Reviewed-by: Roman Penyaev Cc: Alexander Viro Cc: Roman Penyaev Cc: Heiher Cc: Jason Baron Cc: Link: http://lkml.kernel.org/r/20200424025057.118641-1-khazhy@google.com Signed-off-by: Linus Torvalds Signed-off-by: Greg Kroah-Hartman --- fs/eventpoll.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -1176,6 +1176,10 @@ static inline bool chain_epi_lockless(st { struct eventpoll *ep = epi->ep; + /* Fast preliminary check */ + if (epi->next != EP_UNACTIVE_PTR) + return false; + /* Check that the same epi has not been just chained from another CPU */ if (cmpxchg(&epi->next, EP_UNACTIVE_PTR, NULL) != EP_UNACTIVE_PTR) return false; @@ -1242,16 +1246,12 @@ static int ep_poll_callback(wait_queue_e * chained in ep->ovflist and requeued later on. */ if (READ_ONCE(ep->ovflist) != EP_UNACTIVE_PTR) { - if (epi->next == EP_UNACTIVE_PTR && - chain_epi_lockless(epi)) + if (chain_epi_lockless(epi)) + ep_pm_stay_awake_rcu(epi); + } else if (!ep_is_linked(epi)) { + /* In the usual case, add event to ready list. */ + if (list_add_tail_lockless(&epi->rdllink, &ep->rdllist)) ep_pm_stay_awake_rcu(epi); - goto out_unlock; - } - - /* If this file is already in the ready list we exit soon */ - if (!ep_is_linked(epi) && - list_add_tail_lockless(&epi->rdllink, &ep->rdllist)) { - ep_pm_stay_awake_rcu(epi); } /*