[5.10,35/57] scsi: ibmvfc: Set default timeout to avoid crash during migration

From: Brian King <brking@linux.vnet.ibm.com>

From: Brian King <brking@linux.vnet.ibm.com>

[ Upstream commit 764907293edc1af7ac857389af9dc858944f53dc ]

While testing live partition mobility, we have observed occasional crashes
of the Linux partition. What we've seen is that during the live migration,
for specific configurations with large amounts of memory, slow network
links, and workloads that are changing memory a lot, the partition can end
up being suspended for 30 seconds or longer. This resulted in the following
scenario:

CPU 0                          CPU 1
-------------------------------  ----------------------------------
scsi_queue_rq                    migration_store
 -> blk_mq_start_request          -> rtas_ibm_suspend_me
  -> blk_add_timer                 -> on_each_cpu(rtas_percpu_suspend_me
              _______________________________________V
             |
             V
    -> IPI from CPU 1
     -> rtas_percpu_suspend_me
                                     -> __rtas_suspend_last_cpu

-- Linux partition suspended for > 30 seconds --
                                      -> for_each_online_cpu(cpu)
                                           plpar_hcall_norets(H_PROD
 -> scsi_dispatch_cmd
                                      -> scsi_times_out
                                       -> scsi_abort_command
                                        -> queue_delayed_work
  -> ibmvfc_queuecommand_lck
   -> ibmvfc_send_event
    -> ibmvfc_send_crq
     - returns H_CLOSED
   <- returns SCSI_MLQUEUE_HOST_BUSY
-> __blk_mq_requeue_request

                                      -> scmd_eh_abort_handler
                                       -> scsi_try_to_abort_cmd
                                         - returns SUCCESS
                                       -> scsi_queue_insert

Normally, the SCMD_STATE_COMPLETE bit would protect against the command
completion and the timeout, but that doesn't work here, since we don't
check that at all in the SCSI_MLQUEUE_HOST_BUSY path.

In this case we end up calling scsi_queue_insert on a request that has
already been queued, or possibly even freed, and we crash.

The patch below simply increases the default I/O timeout to avoid this race
condition. This is also the timeout value that nearly all IBM SAN storage
recommends setting as the default value.

Link: https://lore.kernel.org/r/1610463998-19791-1-git-send-email-brking@linux.vnet.ibm.com
Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
 drivers/scsi/ibmvscsi/ibmvfc.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

Message ID	20210205140657.471837264@linuxfoundation.org
State	Superseded
Headers	show Return-Path: <stable-owner@kernel.org> From: Greg Kroah-Hartman <gregkh@linuxfoundation.org> To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>, stable@vger.kernel.org, Brian King <brking@linux.vnet.ibm.com>, "Martin K. Petersen" <martin.petersen@oracle.com>, Sasha Levin <sashal@kernel.org> Subject: [PATCH 5.10 35/57] scsi: ibmvfc: Set default timeout to avoid crash during migration Date: Fri, 5 Feb 2021 15:07:01 +0100 Message-Id: <20210205140657.471837264@linuxfoundation.org> In-Reply-To: <20210205140655.982616732@linuxfoundation.org> References: <20210205140655.982616732@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Precedence: bulk
Series	None \| expand [5.10,02/57] net: stmmac: dwmac-intel-plat: remove config data on error [5.10,03/57] net: fec: put child node on error path [5.10,04/57] net: octeontx2: Make sure the buffer is 128 byte aligned [5.10,05/57] stmmac: intel: Configure EHL PSE0 GbE and PSE1 GbE to 32 bits DMA addressing [5.10,06/57] mlxsw: spectrum_span: Do not overwrite policer configuration [5.10,07/57] net: dsa: bcm_sf2: put device node before return [5.10,08/57] net: switchdev: dont set port_obj_info->handled true when -EOPNOTSUPP [5.10,09/57] ibmvnic: Ensure that CRQ entry read are correctly ordered [5.10,10/57] iommu/io-pgtable-arm: Support coherency for Mali LPAE [5.10,11/57] drm/panfrost: Support cache-coherent integrations [5.10,12/57] arm64: dts: meson: Describe G12b GPU as coherent [5.10,13/57] arm64: Fix kernel address detection of __is_lm_address() [5.10,14/57] arm64: Do not pass tagged addresses to __is_lm_address() [5.10,15/57] Revert "x86/setup: dont remove E820_TYPE_RAM for pfn 0" [5.10,16/57] ARM: 9025/1: Kconfig: CPU_BIG_ENDIAN depends on !LD_IS_LLD [5.10,17/57] iommu/vt-d: Do not use flush-queue when caching-mode is on [5.10,18/57] phy: cpcap-usb: Fix warning for missing regulator_disable [5.10,19/57] tools/power/x86/intel-speed-select: Set scaling_max_freq to base_frequency [5.10,20/57] tools/power/x86/intel-speed-select: Set higher of cpuinfo_max_freq or base_frequency [5.10,21/57] platform/x86: touchscreen_dmi: Add swap-x-y quirk for Goodix touchscreen on Estar Be... [5.10,22/57] platform/x86: intel-vbtn: Support for tablet mode on Dell Inspiron 7352 [5.10,23/57] habanalabs: fix dma_addr passed to dma_mmap_coherent [5.10,24/57] locking/lockdep: Avoid noinstr warning for DEBUG_LOCKDEP [5.10,25/57] x86: __always_inline __{rd,wr}msr() [5.10,26/57] scsi: scsi_transport_srp: Dont block target in failfast state [5.10,27/57] scsi: libfc: Avoid invoking response handler twice if ep is already completed [5.10,28/57] scsi: fnic: Fix memleak in vnic_dev_init_devcmd2 [5.10,29/57] ASoC: SOF: Intel: hda: Resume codec to do jack detection [5.10,30/57] ALSA: hda: Add AlderLake-P PCI ID and HDMI codec vid [5.10,31/57] objtool: Dont add empty symbols to the rbtree [5.10,32/57] mac80211: fix incorrect strlen of .write in debugfs [5.10,33/57] mac80211: fix fast-rx encryption check [5.10,34/57] mac80211: fix encryption key selection for 802.3 xmit [5.10,35/57] scsi: ibmvfc: Set default timeout to avoid crash during migration [5.10,36/57] ALSA: hda: Add Cometlake-R PCI ID [5.10,37/57] i2c: tegra: Create i2c_writesl_vi() to use with VI I2C for filling TX FIFO [5.10,38/57] udf: fix the problem that the disc content is not displayed [5.10,39/57] nvme: check the PRINFO bit before deciding the host buffer length [5.10,40/57] nvme-rdma: avoid request double completion for concurrent nvme_rdma_timeout [5.10,41/57] nvme-tcp: avoid request double completion for concurrent nvme_tcp_timeout [5.10,42/57] nvme-pci: allow use of cmb on v1.4 controllers [5.10,43/57] nvmet: set right status on error in id-ns handler [5.10,44/57] platform/x86: thinkpad_acpi: Add P53/73 firmware to fan_quirk_table for dual fan con... [5.10,45/57] selftests/powerpc: Only test lwm/stmw on big endian [5.10,46/57] drm/amd/display: Update dram_clock_change_latency for DCN2.1 [5.10,47/57] drm/amd/display: Allow PSTATE chnage when no displays are enabled [5.10,48/57] drm/amd/display: Change function decide_dp_link_settings to avoid infinite looping [5.10,49/57] drm/amd/display: Use hardware sequencer functions for PG control [5.10,50/57] drm/amd/display: Fixed corruptions on HPDRX link loss restore [5.10,51/57] habanalabs: zero pci counters packet before submit to FW [5.10,52/57] habanalabs: fix backward compatibility of idle check [5.10,53/57] habanalabs: disable FW events on device removal [5.10,54/57] objtool: Dont fail the kernel build on fatal errors [5.10,55/57] x86/cpu: Add another Alder Lake CPU to the Intel family [5.10,56/57] kthread: Extract KTHREAD_IS_PER_CPU [5.10,57/57] workqueue: Restrict affinity change to rescuer

[5.10,35/57] scsi: ibmvfc: Set default timeout to avoid crash during migration

Commit Message

Patch