From patchwork Mon Jun 1 17:52:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Greg KH X-Patchwork-Id: 224977 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=3.0 tests=DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI, SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 432F4C433E0 for ; Mon, 1 Jun 2020 18:42:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 214B4206E2 for ; Mon, 1 Jun 2020 18:42:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1591036953; bh=zyOJLdQKVxJwVgLPFoUQkQN8r+16PCU3njpaITBRIc0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=j/A+fxRYbfY+77hIM6OM9vbt25Dus2wI0o0TPNBTJUob+XzPZYYow9GNhorR1J/xo oOiyyNBzjCH3avzyF9V9Z+Hg9+bsBh1iV6qHoa4KBhSTmHOL4tUfqUBGL6iUVUOtDO RgPG5aqxp7cvW9s2fGsc5VUH6jQgAEGo265AYs/4= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730710AbgFASm1 (ORCPT ); Mon, 1 Jun 2020 14:42:27 -0400 Received: from mail.kernel.org ([198.145.29.99]:58952 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730416AbgFASLo (ORCPT ); Mon, 1 Jun 2020 14:11:44 -0400 Received: from localhost (83-86-89-107.cable.dynamic.v4.ziggo.nl [83.86.89.107]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 92A2120825; Mon, 1 Jun 2020 18:11:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1591035103; bh=zyOJLdQKVxJwVgLPFoUQkQN8r+16PCU3njpaITBRIc0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qso2lNN5H9VqjP0Uj4RCMJfpO8FaIPsXRbwjQnAbVQd3LgUCfDgK5ZAXntbqSi56k XaaIXNzFCAYmx+ET8IX8QOyC3HL2zMyNXcHD5UiCs292Be+dIyXVbjt3IAeTETu5hE qNrcHkXm54RRjQHXXxCb6mPJeySFHtsTDQZgE0W4= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Moshe Shemesh , Eran Ben Elisha , Saeed Mahameed Subject: [PATCH 5.6 011/177] net/mlx5: Add command entry handling completion Date: Mon, 1 Jun 2020 19:52:29 +0200 Message-Id: <20200601174049.569200776@linuxfoundation.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200601174048.468952319@linuxfoundation.org> References: <20200601174048.468952319@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Sender: stable-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org From: Moshe Shemesh [ Upstream commit 17d00e839d3b592da9659c1977d45f85b77f986a ] When FW response to commands is very slow and all command entries in use are waiting for completion we can have a race where commands can get timeout before they get out of the queue and handled. Timeout completion on uninitialized command will cause releasing command's buffers before accessing it for initialization and then we will get NULL pointer exception while trying access it. It may also cause releasing buffers of another command since we may have timeout completion before even allocating entry index for this command. Add entry handling completion to avoid this race. Fixes: e126ba97dba9 ("mlx5: Add driver for Mellanox Connect-IB adapters") Signed-off-by: Moshe Shemesh Signed-off-by: Eran Ben Elisha Signed-off-by: Saeed Mahameed Signed-off-by: Greg Kroah-Hartman --- drivers/net/ethernet/mellanox/mlx5/core/cmd.c | 14 ++++++++++++++ include/linux/mlx5/driver.h | 1 + 2 files changed, 15 insertions(+) --- a/drivers/net/ethernet/mellanox/mlx5/core/cmd.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/cmd.c @@ -861,6 +861,7 @@ static void cmd_work_handler(struct work int alloc_ret; int cmd_mode; + complete(&ent->handling); sem = ent->page_queue ? &cmd->pages_sem : &cmd->sem; down(sem); if (!ent->page_queue) { @@ -978,6 +979,11 @@ static int wait_func(struct mlx5_core_de struct mlx5_cmd *cmd = &dev->cmd; int err; + if (!wait_for_completion_timeout(&ent->handling, timeout) && + cancel_work_sync(&ent->work)) { + ent->ret = -ECANCELED; + goto out_err; + } if (cmd->mode == CMD_MODE_POLLING || ent->polling) { wait_for_completion(&ent->done); } else if (!wait_for_completion_timeout(&ent->done, timeout)) { @@ -985,12 +991,17 @@ static int wait_func(struct mlx5_core_de mlx5_cmd_comp_handler(dev, 1UL << ent->idx, true); } +out_err: err = ent->ret; if (err == -ETIMEDOUT) { mlx5_core_warn(dev, "%s(0x%x) timeout. Will cause a leak of a command resource\n", mlx5_command_str(msg_to_opcode(ent->in)), msg_to_opcode(ent->in)); + } else if (err == -ECANCELED) { + mlx5_core_warn(dev, "%s(0x%x) canceled on out of queue timeout.\n", + mlx5_command_str(msg_to_opcode(ent->in)), + msg_to_opcode(ent->in)); } mlx5_core_dbg(dev, "err %d, delivery status %s(%d)\n", err, deliv_status_to_str(ent->status), ent->status); @@ -1026,6 +1037,7 @@ static int mlx5_cmd_invoke(struct mlx5_c ent->token = token; ent->polling = force_polling; + init_completion(&ent->handling); if (!callback) init_completion(&ent->done); @@ -1045,6 +1057,8 @@ static int mlx5_cmd_invoke(struct mlx5_c err = wait_func(dev, ent); if (err == -ETIMEDOUT) goto out; + if (err == -ECANCELED) + goto out_free; ds = ent->ts2 - ent->ts1; op = MLX5_GET(mbox_in, in->first.data, opcode); --- a/include/linux/mlx5/driver.h +++ b/include/linux/mlx5/driver.h @@ -761,6 +761,7 @@ struct mlx5_cmd_work_ent { struct delayed_work cb_timeout_work; void *context; int idx; + struct completion handling; struct completion done; struct mlx5_cmd *cmd; struct work_struct work;