From patchwork Tue Feb 18 19:55:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Greg Kroah-Hartman X-Patchwork-Id: 231122 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09F3CC34026 for ; Tue, 18 Feb 2020 20:04:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D6E1021D56 for ; Tue, 18 Feb 2020 20:04:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1582056294; bh=r3Wt9Fqia8ELqoZcBvGRbikshi1Mn5+KDL7u1kFTmjA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=dhfVPE3rKDQWyYP4V80HjgpLX62UCN/8dazOjFe4sNHQLn9Qy9rsoBh4RaEkVQakH RV/KTCqeXwfz6Y4jbwhwDQ8k0Tzed5xRSQyrX5SY4S2E2oY68yxM3lMFAYvPk3/1Um pq9gq1uEqlyVAkXixEHB7YQnklHOM23hi2oXQUtI= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728973AbgBRUCg (ORCPT ); Tue, 18 Feb 2020 15:02:36 -0500 Received: from mail.kernel.org ([198.145.29.99]:43122 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728970AbgBRUCg (ORCPT ); Tue, 18 Feb 2020 15:02:36 -0500 Received: from localhost (83-86-89-107.cable.dynamic.v4.ziggo.nl [83.86.89.107]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id B9B2E21D56; Tue, 18 Feb 2020 20:02:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1582056156; bh=r3Wt9Fqia8ELqoZcBvGRbikshi1Mn5+KDL7u1kFTmjA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JBPj6XnXlmUHvT/28JZ3EOJ4YPCTZPWo5+zTbTHYod89lMH902tRVwdDsUqFPdtX+ teI2YF1MtXxac96W+C86o22BDHCM16R3RGUb5Ce1DURMpEgkE5KmscUjbm0yHy/Ivr JNvsELKM7kpp8ssZfVqkfJ8MyGSouIvsJQC/asto= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Zhu Yanjun , Leon Romanovsky , Jason Gunthorpe Subject: [PATCH 5.5 56/80] RDMA/rxe: Fix soft lockup problem due to using tasklets in softirq Date: Tue, 18 Feb 2020 20:55:17 +0100 Message-Id: <20200218190437.492485814@linuxfoundation.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200218190432.043414522@linuxfoundation.org> References: <20200218190432.043414522@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Sender: stable-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org From: Zhu Yanjun commit 8ac0e6641c7ca14833a2a8c6f13d8e0a435e535c upstream. When run stress tests with RXE, the following Call Traces often occur watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [swapper/2:0] ... Call Trace: create_object+0x3f/0x3b0 kmem_cache_alloc_node_trace+0x129/0x2d0 __kmalloc_reserve.isra.52+0x2e/0x80 __alloc_skb+0x83/0x270 rxe_init_packet+0x99/0x150 [rdma_rxe] rxe_requester+0x34e/0x11a0 [rdma_rxe] rxe_do_task+0x85/0xf0 [rdma_rxe] tasklet_action_common.isra.21+0xeb/0x100 __do_softirq+0xd0/0x298 irq_exit+0xc5/0xd0 smp_apic_timer_interrupt+0x68/0x120 apic_timer_interrupt+0xf/0x20 ... The root cause is that tasklet is actually a softirq. In a tasklet handler, another softirq handler is triggered. Usually these softirq handlers run on the same cpu core. So this will cause "soft lockup Bug". Fixes: 8700e3e7c485 ("Soft RoCE driver") Link: https://lore.kernel.org/r/20200212072635.682689-8-leon@kernel.org Signed-off-by: Zhu Yanjun Signed-off-by: Leon Romanovsky Signed-off-by: Jason Gunthorpe Signed-off-by: Greg Kroah-Hartman --- drivers/infiniband/sw/rxe/rxe_comp.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) --- a/drivers/infiniband/sw/rxe/rxe_comp.c +++ b/drivers/infiniband/sw/rxe/rxe_comp.c @@ -329,7 +329,7 @@ static inline enum comp_state check_ack( qp->comp.psn = pkt->psn; if (qp->req.wait_psn) { qp->req.wait_psn = 0; - rxe_run_task(&qp->req.task, 1); + rxe_run_task(&qp->req.task, 0); } } return COMPST_ERROR_RETRY; @@ -463,7 +463,7 @@ static void do_complete(struct rxe_qp *q */ if (qp->req.wait_fence) { qp->req.wait_fence = 0; - rxe_run_task(&qp->req.task, 1); + rxe_run_task(&qp->req.task, 0); } } @@ -479,7 +479,7 @@ static inline enum comp_state complete_a if (qp->req.need_rd_atomic) { qp->comp.timeout_retry = 0; qp->req.need_rd_atomic = 0; - rxe_run_task(&qp->req.task, 1); + rxe_run_task(&qp->req.task, 0); } } @@ -725,7 +725,7 @@ int rxe_completer(void *arg) RXE_CNT_COMP_RETRY); qp->req.need_retry = 1; qp->comp.started_retry = 1; - rxe_run_task(&qp->req.task, 1); + rxe_run_task(&qp->req.task, 0); } if (pkt) {