From patchwork Wed Mar 15 12:35:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Ribalda X-Patchwork-Id: 663862 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A647C61DA4 for ; Wed, 15 Mar 2023 12:37:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232876AbjCOMhV (ORCPT ); Wed, 15 Mar 2023 08:37:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52784 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232576AbjCOMhG (ORCPT ); Wed, 15 Mar 2023 08:37:06 -0400 Received: from mail-ed1-x529.google.com (mail-ed1-x529.google.com [IPv6:2a00:1450:4864:20::529]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2FFF59F236 for ; Wed, 15 Mar 2023 05:36:09 -0700 (PDT) Received: by mail-ed1-x529.google.com with SMTP id eh3so18980715edb.11 for ; Wed, 15 Mar 2023 05:36:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1678883757; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=jmbQJ7O/6JS/WmDJaNmVYSgRY/QPgP7mF5MHe8za06A=; b=PFCpFqjiqIYZYC4iO9lzTESvAbW9pwjL3bpvWpgplyx+BTtFDFpwsNFEXAJl717pak ZOpFbOFllwVsXGELIa9NjexnQP7jDQZQOgStfNv6fdFvOcdI+qsImy5YZhlM4qdpanIH s9qUfAAUcIT2mXa1MqHNozS5Jt5A9KDGfynVU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678883757; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jmbQJ7O/6JS/WmDJaNmVYSgRY/QPgP7mF5MHe8za06A=; b=YHl4PvOsnZZPWbaVxvODarJt5Rl9zKttxxLqq1H7oJ8UYhpsFACpgcJvfwxuHmZutG lG3KRI4/Vjla9ZqC7tNXulNQoEGxVLa0nU4t6UrpuXU+60+H8AEA9WinmzbvxyduM/tl htou7MyHODZ+UWX00hLHDCKJZdSDc4GRltEA0IPXnuYZkiX7JO0UdlYBCmput36TU15V eywDFFHs+KhuN/20D6bbbGIZEFcGxOkIO7vngqKDD8emBes9b4F3nO4zsxPA3acfi/Re sL9bSLJIqmOd2MDnR5AejWjv+qD5XpKZoqbUBfyYsmJZB27Mh3K4IvMnwyax0tVqRKKf xuCA== X-Gm-Message-State: AO0yUKV41Cw46GGPvWF8IdlgqD9KKQ00n+HfC1HiWs7IFa+KcWVKWVgd +o6H6jMxDLF9k2E2Nw1EHOk+GuJ7oNo7gle0naM= X-Google-Smtp-Source: AK7set/iibasOIWFAw1GofhKQYU9qg0EbmQJ3920bLEA8G6wi1qXEVO462PZjGkwEcCD0Dl5T9d/3Q== X-Received: by 2002:a17:906:1952:b0:91e:52a8:9efc with SMTP id b18-20020a170906195200b0091e52a89efcmr7426685eje.43.1678883757726; Wed, 15 Mar 2023 05:35:57 -0700 (PDT) Received: from alco.roam.corp.google.com ([2620:0:1059:10:491c:f627:27f0:1c4]) by smtp.gmail.com with ESMTPSA id w3-20020a50c443000000b004c30e2fc6e5sm2363256edf.65.2023.03.15.05.35.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Mar 2023 05:35:57 -0700 (PDT) From: Ricardo Ribalda Date: Wed, 15 Mar 2023 13:35:37 +0100 Subject: [PATCH v8 1/6] media: uvcvideo: Fix negative modulus calculation MIME-Version: 1.0 Message-Id: <20220920-resend-hwtimestamp-v8-1-0edaca2e2ab3@chromium.org> References: <20220920-resend-hwtimestamp-v8-0-0edaca2e2ab3@chromium.org> In-Reply-To: <20220920-resend-hwtimestamp-v8-0-0edaca2e2ab3@chromium.org> To: Laurent Pinchart , Mauro Carvalho Chehab Cc: "hn.chen" , linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, Sergey Senozhatsky , Ricardo Ribalda X-Mailer: b4 0.11.0-dev-696ae X-Developer-Signature: v=1; a=openpgp-sha256; l=1078; i=ribalda@chromium.org; h=from:subject:message-id; bh=A9fEA5iTFV9nHf0HSMuMBzuGKXvge+iAmSWSfQdkDBc=; b=owEBbQKS/ZANAwAKAdE30T7POsSIAcsmYgBkEbugupcIaw0geteY56cHFpS8HuBVC+DUluIeJ2uy +NjN5raJAjMEAAEKAB0WIQREDzjr+/4oCDLSsx7RN9E+zzrEiAUCZBG7oAAKCRDRN9E+zzrEiA6GD/ wOEaWHot8B1ooRlrzHary8T4QxwMhOVnOtKIi1W0knaxjtWv6b8OowzDJ+qYjTTjsbJRV0JM2h+6N5 yGw1e+hSHXGQobiGbrr33oPSlT+3r/3kpedecDoOHVZCz0WC3WORXZgtq4Psr9FtusjR4V4yFTRUDD Pgnh+eczipCh5MC2Cpm9L6vXTLkeZbYf760nGR40taH/r6KhjZ2ThpkMTtnaKeZ2J0eIJV38MWpT3X pjvbKmCcOs6V7hAqbUMSKo0G//256XraLGK3q7R7nXW/dvKrZJlAK6OpHRqQigClWZlR+MiWWuBga0 MIc1TWE8SFcbi3uSCuh1ZmljFT/yo9rAMvgcSHGY7+BNQJy3JAlSmQatOwSDQOffc3HdqnM0DIrUZ6 hqN5R5t8qmzKKa+sfHf7QLCG+oMNaNmRzcmE6EB3z9JoQk4KoIcCJFwDKsE254d5nAjFj7GLOmz0A1 Lsi7ldmUiaCOglFLY7J1H0lsfKQcCJU0e8yFSSigOVSeGPWHjLKdkrxmd7idIlUGtWioJwh+eAIYhW zHM2OUBCoSl00EeIDT3gqOGtInTe1jcZgdanVv+02NuYYSY64wYDTuR2v3GgALJOGn8Y+SzUajjsoi mUwSFkakI0aahCFtsSThKfpxpWGZKdHr+bIfTZmHc71JPel7JZCmIQmEl5Ng== X-Developer-Key: i=ribalda@chromium.org; a=openpgp; fpr=9EC3BB66E2FC129A6F90B39556A0D81F9F782DA9 Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org If head is 0, last will be addressing the index 0 instead of clock->size -1. Luckily clock->head is unsiged, otherwise it would be addressing 0xffffffff. Nontheless, this is not the intented behaviour and should be fixed. Fixes: 66847ef013cc ("[media] uvcvideo: Add UVC timestamps support") Signed-off-by: Ricardo Ribalda --- drivers/media/usb/uvc/uvc_video.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/media/usb/uvc/uvc_video.c b/drivers/media/usb/uvc/uvc_video.c index d4b023d4de7c..4ff4ab4471fe 100644 --- a/drivers/media/usb/uvc/uvc_video.c +++ b/drivers/media/usb/uvc/uvc_video.c @@ -732,7 +732,7 @@ void uvc_video_clock_update(struct uvc_streaming *stream, goto done; first = &clock->samples[clock->head]; - last = &clock->samples[(clock->head - 1) % clock->size]; + last = &clock->samples[(clock->head - 1 + clock->size) % clock->size]; /* First step, PTS to SOF conversion. */ delta_stc = buf->pts - (1UL << 31); From patchwork Wed Mar 15 12:35:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Ribalda X-Patchwork-Id: 663861 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5FCAFC6FD1D for ; Wed, 15 Mar 2023 12:37:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232684AbjCOMhX (ORCPT ); Wed, 15 Mar 2023 08:37:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52792 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232584AbjCOMhG (ORCPT ); Wed, 15 Mar 2023 08:37:06 -0400 Received: from mail-ed1-x529.google.com (mail-ed1-x529.google.com [IPv6:2a00:1450:4864:20::529]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0F0869FBF5 for ; Wed, 15 Mar 2023 05:36:12 -0700 (PDT) Received: by mail-ed1-x529.google.com with SMTP id fd5so41012961edb.7 for ; Wed, 15 Mar 2023 05:36:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1678883760; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=Ks+I0iL9S/eJzdcQJQgtcbFNoYKHK8Hr97Z4+vCos3E=; b=jNDOTpVguRqmPsNwZMHd8LjaCyFGqB0h/vgUFZNsVbc4vJM5fkjfVSXBLB5tuMQGc9 AibuVUnU3RCV0RHGJwrd0V6/7frhHnpG0bb+ycHjnnJZhR+dRqn4Igxn1LhJZdpU/5VD 4LqGKTW1xyuwuq3F+MH8bNjeZBHxbem17nF60= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678883760; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ks+I0iL9S/eJzdcQJQgtcbFNoYKHK8Hr97Z4+vCos3E=; b=C4L0OcG4XEH6QNuQPcnG1c3+T4Ev99+aEllW7QHlzD2TWPMgcLZ699eKpM3dgTuBwG OJLZ88mdAxos3EvnOUEHxNIXQD3yuZnsiTITKM6eTSV1fVBP0POXtl45JvE3EcV+mdJC R13hjw+Ss3pOP17d41m2x2RqjnanoRWW6I3+XGrDLAO/oDFf5YNHAq/86543COo0VbgC GgwI4NiAnnjhX0EX7uYkN7umpkZX+x9bsGTB8qpKNJkp8oKprZygESIXn3qBSlLDjHeP b2B/y+M9kIhnXmUiCdLoa5QFGexCewBuoZJBIYDqQ4osRMF3PhOI4Ay+qv3nE3OUoEtn 0r7Q== X-Gm-Message-State: AO0yUKXFU94c6doPSqcPAnuxeifxqeoJ5aISDdlPFbFuuQcioq4mUbZb XwbPAm7UZF4LXJoY+IB3hcWN+A== X-Google-Smtp-Source: AK7set+6iMJqwTCKxbtSpa/DEZpAg+l9W6d0SLzFYfPsd3q00iUx64y4/CiIhyx5TsTvO7Flmon4+A== X-Received: by 2002:a05:6402:404:b0:4fc:b1f2:42eb with SMTP id q4-20020a056402040400b004fcb1f242ebmr2219182edv.20.1678883759997; Wed, 15 Mar 2023 05:35:59 -0700 (PDT) Received: from alco.roam.corp.google.com ([2620:0:1059:10:491c:f627:27f0:1c4]) by smtp.gmail.com with ESMTPSA id w3-20020a50c443000000b004c30e2fc6e5sm2363256edf.65.2023.03.15.05.35.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Mar 2023 05:35:59 -0700 (PDT) From: Ricardo Ribalda Date: Wed, 15 Mar 2023 13:35:38 +0100 Subject: [PATCH v8 2/6] media: uvcvideo: Ignore empty TS packets MIME-Version: 1.0 Message-Id: <20220920-resend-hwtimestamp-v8-2-0edaca2e2ab3@chromium.org> References: <20220920-resend-hwtimestamp-v8-0-0edaca2e2ab3@chromium.org> In-Reply-To: <20220920-resend-hwtimestamp-v8-0-0edaca2e2ab3@chromium.org> To: Laurent Pinchart , Mauro Carvalho Chehab Cc: "hn.chen" , linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, Sergey Senozhatsky , Ricardo Ribalda X-Mailer: b4 0.11.0-dev-696ae X-Developer-Signature: v=1; a=openpgp-sha256; l=3922; i=ribalda@chromium.org; h=from:subject:message-id; bh=+e1rJhTRZYFtyKcR4/N8/nQKunH8/Dqv1jSdJGbbaL4=; b=owEBbQKS/ZANAwAKAdE30T7POsSIAcsmYgBkEbujR7E2zx9xw86qmRztrnP4mXdUpFSOlP0hN7K3 xBV763+JAjMEAAEKAB0WIQREDzjr+/4oCDLSsx7RN9E+zzrEiAUCZBG7owAKCRDRN9E+zzrEiC5rD/ 9mI8NjUQ/dqMqieVdH8Zk2i68yUANex7arMlWP0u2OXtFOEHQ7eXRkzeo7w4fED2RBgpacUy//ZktZ YL5rIdili429BjEKDVLOXKTdVpmoX7B1JC6a95aNGxGZiPmhoc/P89+5VWBDdrkPtj/fBC617rdkxN hzyoI1Vcvb019dJpAJcFWsyC6UCz1ujkjy6sSAusz5jOsn9jWphBxtFX39NZxZpQ/a2f6h7Lf4ReGo KBUyJgLEDUg4F4IIRUL7omOfxh0uRALD0BDFiM5TU0TG22DP1327g9n7c85c/51J8Y9oI91iSVvs0Q ztD44jnNYI/re0K8wMBJ+H0zkywdpFuSh7BedIJB1I2yCnx6AvWzNdOH4kT6suz5CIRAcqRcUtiPbB nnRI3LqMYccQx0k6mCifj/BNnx7pdpuBquU3Sx+26NRrjzOgLlUsuqcP/M3nJMsO2W/ZOIkD4ShWc6 Q8ZS073PF7uPr0qnszaZUoUiQcA5iLSFy1d6zuHGY0hMBDDdZ1UewdHUq2Tuts9mJDeNjoPUYeCW0F C9havLCEWEeP2JHDAPqIbfJA4srKlBTJA6F6Wv9UcNbyBMQ2tL1gqsLb0flPJyBldRpSu0A0i1I/Tc y5juGqG781g0qU9eCGcE0qPydxI6JXyN+Ec/zQkQIfeqqxanzodtayRQX1cA== X-Developer-Key: i=ribalda@chromium.org; a=openpgp; fpr=9EC3BB66E2FC129A6F90B39556A0D81F9F782DA9 Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org Some SunplusIT cameras took a borderline interpretation of the UVC 1.5 standard, and fill the PTS and SCR fields with invalid data if the package does not contain data. "STC must be captured when the first video data of a video frame is put on the USB bus." Eg: buffer: 0xa7755c00 len 000012 header:0x8c stc 00000000 sof 0000 pts 00000000 buffer: 0xa7755c00 len 000012 header:0x8c stc 00000000 sof 0000 pts 00000000 buffer: 0xa7755c00 len 000668 header:0x8c stc 73779dba sof 070c pts 7376d37a This borderline/buggy interpretation has been implemented in a variety of devices, from directly SunplusIT and from other OEMs that rebrand SunplusIT products. So quirking based on VID:PID will be problematic. All the affected modules have the following extension unit: VideoControl Interface Descriptor: guidExtensionCode {82066163-7050-ab49-b8cc-b3855e8d221d} But the vendor plans to use that GUID in the future and fix the bug, this means that we should use heuristic to figure out the broken packets. This patch takes care of this. lsusb of one of the affected cameras: Bus 001 Device 003: ID 1bcf:2a01 Sunplus Innovation Technology Inc. Device Descriptor: bLength 18 bDescriptorType 1 bcdUSB 2.01 bDeviceClass 239 Miscellaneous Device bDeviceSubClass 2 ? bDeviceProtocol 1 Interface Association bMaxPacketSize0 64 idVendor 0x1bcf Sunplus Innovation Technology Inc. idProduct 0x2a01 bcdDevice 0.02 iManufacturer 1 SunplusIT Inc iProduct 2 HanChen Wise Camera iSerial 3 01.00.00 bNumConfigurations 1 Tested-by: HungNien Chen Reviewed-by: Sergey Senozhatsky Signed-off-by: Ricardo Ribalda --- drivers/media/usb/uvc/uvc_video.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/drivers/media/usb/uvc/uvc_video.c b/drivers/media/usb/uvc/uvc_video.c index 4ff4ab4471fe..1f416c494acc 100644 --- a/drivers/media/usb/uvc/uvc_video.c +++ b/drivers/media/usb/uvc/uvc_video.c @@ -478,6 +478,7 @@ uvc_video_clock_decode(struct uvc_streaming *stream, struct uvc_buffer *buf, ktime_t time; u16 host_sof; u16 dev_sof; + u32 dev_stc; switch (data[1] & (UVC_STREAM_PTS | UVC_STREAM_SCR)) { case UVC_STREAM_PTS | UVC_STREAM_SCR: @@ -526,6 +527,23 @@ uvc_video_clock_decode(struct uvc_streaming *stream, struct uvc_buffer *buf, if (dev_sof == stream->clock.last_sof) return; + dev_stc = get_unaligned_le32(&data[header_size - 6]); + + /* + * STC (Source Time Clock) is the clock used by the camera. The UVC 1.5 + * standard states that it "must be captured when the first video data + * of a video frame is put on the USB bus". + * Most of the vendors, clear the `UVC_STREAM_SCR` bit when the data is + * not valid, other vendors always set the `UVC_STREAM_SCR` bit and + * expect that the driver only samples the stc if there is data on the + * packet. + * Ignore all the hardware timestamp information if there is no data + * and stc and sof are zero. + */ + if (buf && buf->bytesused == 0 && len == header_size && + dev_stc == 0 && dev_sof == 0) + return; + stream->clock.last_sof = dev_sof; host_sof = usb_get_current_frame_number(stream->dev->udev); @@ -564,7 +582,7 @@ uvc_video_clock_decode(struct uvc_streaming *stream, struct uvc_buffer *buf, spin_lock_irqsave(&stream->clock.lock, flags); sample = &stream->clock.samples[stream->clock.head]; - sample->dev_stc = get_unaligned_le32(&data[header_size - 6]); + sample->dev_stc = dev_stc; sample->dev_sof = dev_sof; sample->host_sof = host_sof; sample->host_time = time; From patchwork Wed Mar 15 12:35:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Ribalda X-Patchwork-Id: 663860 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8765C6FD1D for ; Wed, 15 Mar 2023 12:37:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232604AbjCOMh0 (ORCPT ); Wed, 15 Mar 2023 08:37:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51692 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232659AbjCOMhI (ORCPT ); Wed, 15 Mar 2023 08:37:08 -0400 Received: from mail-ed1-x535.google.com (mail-ed1-x535.google.com [IPv6:2a00:1450:4864:20::535]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 27FDC9B2D1 for ; Wed, 15 Mar 2023 05:36:17 -0700 (PDT) Received: by mail-ed1-x535.google.com with SMTP id z21so9570161edb.4 for ; Wed, 15 Mar 2023 05:36:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1678883762; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=RN9dk0TN64Mo7WHVhpVGbABaRx7Q/q6Pwknupu3adDk=; b=Gq2Mx9D8fuLbueJb+f/zY1iLDil4UkqfW0MMwJ5IKoCwEcDhtr9BNZ8ysOnMrtg0yH G6hfWYP2T9NqIELhpG4wmR6FJ6SSczOsc/h5ETKoH3EQC1QV8sKPIjvR7uj9hR9QaY4A f2b6c1LEnY5wdGNriKYYvjdHLpUccVPydEPEg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678883762; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RN9dk0TN64Mo7WHVhpVGbABaRx7Q/q6Pwknupu3adDk=; b=iY8SGDTwAanBk+2JIid8cxpQqyrfqLzvTKZekqAb+d0Clr7xXxKZuMt4BKYb8CBAoN 3aO5PGHs1LOg5McUvRiNz45URs1YOAruGHTOCGwE4Ea7STwyBN8TSi61W6z6tJrgbZ9L wDTKmLiRHNbIeaWI/afq1kLav4w0craaRipBASiHkwAKW9kFdj42NMIt5Tn+qDsuknAk WDo1yrVS9k4lnKiOir8b1JuNZ98A3c/eoqDzTCf80x4EofYlmj7JCBMjdOMJxCyx8kZL USLYrupjh1+lj9eLJkq0kNWQAOVtDvmeuL2D2gfFSTGgRU5Vja85K28ghIQYJj6qtlMd VlAA== X-Gm-Message-State: AO0yUKVKDjzCyyCQaNftoSRiWPZ6xGYMu4cxA82dEFQQLqKE9BH6SLaZ vWOEPKV0OhZeyZlJ/UXLid+xRg== X-Google-Smtp-Source: AK7set8Tjvu6jP4xr0zFUSFxzqrD6Be0Lk+05/sKpmLx1SneJD4uOMfFIa1n/infgUFg1bLbjzLReA== X-Received: by 2002:a05:6402:1a27:b0:4fe:97a2:4b86 with SMTP id be7-20020a0564021a2700b004fe97a24b86mr2549451edb.8.1678883761818; Wed, 15 Mar 2023 05:36:01 -0700 (PDT) Received: from alco.roam.corp.google.com ([2620:0:1059:10:491c:f627:27f0:1c4]) by smtp.gmail.com with ESMTPSA id w3-20020a50c443000000b004c30e2fc6e5sm2363256edf.65.2023.03.15.05.36.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Mar 2023 05:36:01 -0700 (PDT) From: Ricardo Ribalda Date: Wed, 15 Mar 2023 13:35:40 +0100 Subject: [PATCH v8 4/6] media: uvcvideo: Allow hw clock updates with buffers not full MIME-Version: 1.0 Message-Id: <20220920-resend-hwtimestamp-v8-4-0edaca2e2ab3@chromium.org> References: <20220920-resend-hwtimestamp-v8-0-0edaca2e2ab3@chromium.org> In-Reply-To: <20220920-resend-hwtimestamp-v8-0-0edaca2e2ab3@chromium.org> To: Laurent Pinchart , Mauro Carvalho Chehab Cc: "hn.chen" , linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, Sergey Senozhatsky , Ricardo Ribalda X-Mailer: b4 0.11.0-dev-696ae X-Developer-Signature: v=1; a=openpgp-sha256; l=1966; i=ribalda@chromium.org; h=from:subject:message-id; bh=Ka10ItjemOVlrqrWVbbdhzTBAJcpymfvLrHEyyU17oM=; b=owEBbQKS/ZANAwAKAdE30T7POsSIAcsmYgBkEbun/QBdadLNvepqccbbDRz3RNClgucR14+xqoG0 FCPAZmmJAjMEAAEKAB0WIQREDzjr+/4oCDLSsx7RN9E+zzrEiAUCZBG7pwAKCRDRN9E+zzrEiBPYD/ 9Dp/6Sdm1tJh5mpdas5fMyoJI6E8BxLwH2bL4qlhWeJwcPAIG4B082jdRdDEmX/352uL9YZQ1KirlZ 8CcgEqdCBBCG/LnS7ZAoPXGXJhcv+sJ+YW3OG/FZwzzaVYoFoNwY9Pn63qk5BioWJWYkE8Ga94/v9P NRwyc3ppw6MdSkNKcaCuFxyacB5723VvXTr8/gpbsMUkCq4vv/aneVhLmIWFJdW1Zf1gLO4bojKBI/ SRhdnCdEKuG+Yy1MjuV+xV7GZJe9jgkK13z/nv9KK63x3mQCd0npDM0mfv0GiPwmUXiBtkANrULKQr aWTTf60BUgMc9EdxY7Dxm9Pry6xESQT7LxyzrkAsjs9FRAUkFbwj4R8I+QJWUEb38QOZQDaXiBZ3n9 pNwSI/9giCudGV8QMSO/rrkss8LV5B/U6PyPHNSEvZIIbIOcsFHsBSuyRxU4kl12C/x7OJAuLfxZ7I Z6qKa6LYm7QIegzHi+uU0qkUO7qgEJYWR25bkyurZdLi2Io5FzsHiEzXOH/sq2cUUVQnFzz0VtVWBx HBpP3b3hfbg2qzdMv49jm+xtSF3Z8cuGGmDZItZ1nwKzMuCkE2FnUiKMSxRJA2iR1SXldym8SmWZKZ oD+G9b/UiEb+PXxtPCj4vqP3sPwd6rls6WudgMPR7eBlxSn8GCcFa27oAKSQ== X-Developer-Key: i=ribalda@chromium.org; a=openpgp; fpr=9EC3BB66E2FC129A6F90B39556A0D81F9F782DA9 Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org With UVC 1.5 we get as little as one clock sample per frame. Which means that it takes 32 frames to move from the software timestamp to the hardware timestamp method. This results in abrupt changes in the timestamping after 32 frames (~1 second), resulting in noticeable artifacts when used for encoding. With this patch we modify the update algorithm to work with whatever amount of values are available. Tested-by: HungNien Chen Signed-off-by: Ricardo Ribalda --- drivers/media/usb/uvc/uvc_video.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/drivers/media/usb/uvc/uvc_video.c b/drivers/media/usb/uvc/uvc_video.c index 4d566edb73e7..6d0243ea0e07 100644 --- a/drivers/media/usb/uvc/uvc_video.c +++ b/drivers/media/usb/uvc/uvc_video.c @@ -754,10 +754,10 @@ void uvc_video_clock_update(struct uvc_streaming *stream, spin_lock_irqsave(&clock->lock, flags); - if (clock->count < clock->size) + if (clock->count < 2) goto done; - first = &clock->samples[clock->head]; + first = &clock->samples[(clock->head - clock->count + clock->size) % clock->size]; last = &clock->samples[(clock->head - 1 + clock->size) % clock->size]; /* First step, PTS to SOF conversion. */ @@ -772,6 +772,17 @@ void uvc_video_clock_update(struct uvc_streaming *stream, if (y2 < y1) y2 += 2048 << 16; + /* + * Have at least 1/4 of a second of timestamps before we + * try to do any calculation. Otherwise we do not have enough + * precision. This value was determined by running Android CTS + * on different devices. + * Dev_sof runs at 1KHz, and we have a fixed point precision of + * 16 bits. + */ + if ((y2 - y1) < ( (1000 / 4) << 16)) + goto done; + y = (u64)(y2 - y1) * (1ULL << 31) + (u64)y1 * (u64)x2 - (u64)y2 * (u64)x1; y = div_u64(y, x2 - x1); From patchwork Wed Mar 15 12:35:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Ribalda X-Patchwork-Id: 663859 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DAA86C6FD1D for ; Wed, 15 Mar 2023 12:37:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232908AbjCOMh3 (ORCPT ); Wed, 15 Mar 2023 08:37:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52984 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232212AbjCOMhK (ORCPT ); Wed, 15 Mar 2023 08:37:10 -0400 Received: from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com [IPv6:2a00:1450:4864:20::52e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 96D38A101F for ; Wed, 15 Mar 2023 05:36:17 -0700 (PDT) Received: by mail-ed1-x52e.google.com with SMTP id cn21so44723983edb.0 for ; Wed, 15 Mar 2023 05:36:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1678883763; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=tCkrkxPcp+pi7gWatTpAiQc1rhobUtJPz5SB1d3BoAs=; b=HcUeL/Gsuz2rsdYJ2uO0fa6VHfLGNxucpRGvZ1v26hYlGJ+MbjoFj6bpzEzwjQvo3X rfiuvdhsXlaq2a6utmzRzNIoDwXOlMDSWdRws60B76Sc4H0TbYS6ERX3p1jtxDOVO9qE vi1zN8sRadkxj7p6mEcosg0spK6fdUlqIA0RU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678883763; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tCkrkxPcp+pi7gWatTpAiQc1rhobUtJPz5SB1d3BoAs=; b=0prEbDKCUTLLUoU2xW5vyFA4OMnctSQP0VCtn8H4nuPe542MyzhhmY6J/nNhOmaunB 420rT78iyR84zSLkj75kcnUVQwKJcx5Q1L02KYpX3mZlhjyaDnmA247eS27XY4jIz0BZ Vk9Jz/FO2nM3vLXudhpEdA54HHPAizBOpzDxF2/2QVkG4ok54vhuvhV+O5tI8nvBl2oc HlFPhVChRrs/ZRG1ziOqvggtwxN/FDHVC2OkpB60idQCrty6xiFJ8x9dSieWKPcEmXcW 6H0xrpHB3ce19ZIr9T5HzXy3Nw/s51LseEIDinex6uZXHJWBie4Q5V9+RzpD9ntW7zX/ x6OQ== X-Gm-Message-State: AO0yUKUaF5WCFTh95VuqvMQo5+qCfQiNEScXCUqTfaWCzXJNOGG5o7Uj TXW1nPO2qK+6ZBLaoDAyo9+dUw== X-Google-Smtp-Source: AK7set8ooG5NmhmFwX3pgacTuf8SZ7jUdAHMo2GbSz5y8q1Ff7WUKJbBssgq7VWXi4Il2Z69verrWQ== X-Received: by 2002:a05:6402:887:b0:500:2a15:f86b with SMTP id e7-20020a056402088700b005002a15f86bmr1576900edy.42.1678883762837; Wed, 15 Mar 2023 05:36:02 -0700 (PDT) Received: from alco.roam.corp.google.com ([2620:0:1059:10:491c:f627:27f0:1c4]) by smtp.gmail.com with ESMTPSA id w3-20020a50c443000000b004c30e2fc6e5sm2363256edf.65.2023.03.15.05.36.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Mar 2023 05:36:02 -0700 (PDT) From: Ricardo Ribalda Date: Wed, 15 Mar 2023 13:35:41 +0100 Subject: [PATCH v8 5/6] media: uvcvideo: Refactor clock circular buffer MIME-Version: 1.0 Message-Id: <20220920-resend-hwtimestamp-v8-5-0edaca2e2ab3@chromium.org> References: <20220920-resend-hwtimestamp-v8-0-0edaca2e2ab3@chromium.org> In-Reply-To: <20220920-resend-hwtimestamp-v8-0-0edaca2e2ab3@chromium.org> To: Laurent Pinchart , Mauro Carvalho Chehab Cc: "hn.chen" , linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, Sergey Senozhatsky , Ricardo Ribalda X-Mailer: b4 0.11.0-dev-696ae X-Developer-Signature: v=1; a=openpgp-sha256; l=6688; i=ribalda@chromium.org; h=from:subject:message-id; bh=OHcMfCmrhkBCGNHm7ElXHEEK0vKCs58LRA2nG7cPcTo=; b=owEBbQKS/ZANAwAKAdE30T7POsSIAcsmYgBkEbupjxjnRk77SUQR0/MIksA+fOmz1p+1MXr0m6wG /W7qENuJAjMEAAEKAB0WIQREDzjr+/4oCDLSsx7RN9E+zzrEiAUCZBG7qQAKCRDRN9E+zzrEiBIKD/ 9KaGJwKe7leHuiXchoYFKwsc8k7asrzgqVT08RQwIXfVsHT48FlHaCtXSdkH7K3ldGGihnbriHBsSR 4eMaF01UiZkWjbmEQobFgctZn/FEYjJXmY9SfLrZ+oFvwVRgvKfpugNi18VFa22kI5/A/Y5GCsJkuK sQ+PsVfpbCjGmCQMR8sDiCa3CgrGdrfScPifEFtwCeJkS03n4dYzKylmDbdvICDAddwTMo4WVVs2O2 FnRRHAG/QeZQ8M09Vh6jMCn3XPKpbQUG8//g9B7LPnrnPGfdFZs6tzZecqV483Vs3EeWlTY+mBjK8x rYVK0WdRN+PY6sXZBpx2HASQTUHOLVZetApfTC3hJgcPc4QvtV9dmdHjbUyevQpasSaMIXY0KAIbZs v0bATtA9zjD8Gw7f01hmQNRDApGrAEzrcK/aooYEIWftI0ftrZq/eAxcIuUevyWSWYAmixtH8PBSie klxPo2R6rfDEIYHZpEW66GoopqmuojxJVGIoe0ozN2OzhPDWBLxKQCZGIozNfUTD+8HJo77nIKfQyu IVqw1NJt29K/W553YYrmA51cBiIbv1r0azMHYYS1l/APnmJaemznOel7aSpvlGtRlWnM/ZfsI9CI01 zEsIFDc4eztChaSTOH5O3uBAnDTMO6iTQayBmVGlCh+cZYawocpNFfQly7RA== X-Developer-Key: i=ribalda@chromium.org; a=openpgp; fpr=9EC3BB66E2FC129A6F90B39556A0D81F9F782DA9 Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org Isolate all the changes related to the clock circular buffer to its own function, that way we can make changes easier to the buffer logic. Also simplify the lock, by removing the circular buffer clock handling from uvc_video_clock_decode(). And now that we are at it, unify the API of the clock functions. Tested-by: HungNien Chen Reviewed-by: Sergey Senozhatsky Signed-off-by: Ricardo Ribalda --- drivers/media/usb/uvc/uvc_video.c | 84 ++++++++++++++++++--------------------- 1 file changed, 38 insertions(+), 46 deletions(-) diff --git a/drivers/media/usb/uvc/uvc_video.c b/drivers/media/usb/uvc/uvc_video.c index 6d0243ea0e07..df7c400fe82e 100644 --- a/drivers/media/usb/uvc/uvc_video.c +++ b/drivers/media/usb/uvc/uvc_video.c @@ -466,19 +466,30 @@ static inline ktime_t uvc_video_get_time(void) return ktime_get_real(); } +static void uvc_video_clock_add_sample(struct uvc_clock *clock, + const struct uvc_clock_sample *sample) +{ + unsigned long flags; + + spin_lock_irqsave(&clock->lock, flags); + + memcpy(&clock->samples[clock->head], sample, sizeof(*sample)); + clock->head = (clock->head + 1) % clock->size; + clock->count = min(clock->count + 1, clock->size); + + spin_unlock_irqrestore(&clock->lock, flags); + + stream->clock.last_sof = sample.dev_sof; +} + static void uvc_video_clock_decode(struct uvc_streaming *stream, struct uvc_buffer *buf, const u8 *data, int len) { - struct uvc_clock_sample *sample; + struct uvc_clock_sample sample; unsigned int header_size; bool has_pts = false; bool has_scr = false; - unsigned long flags; - ktime_t time; - u16 host_sof; - u16 dev_sof; - u32 dev_stc; switch (data[1] & (UVC_STREAM_PTS | UVC_STREAM_SCR)) { case UVC_STREAM_PTS | UVC_STREAM_SCR: @@ -523,11 +534,11 @@ uvc_video_clock_decode(struct uvc_streaming *stream, struct uvc_buffer *buf, * all the data packets of the same frame contains the same SOF. In that * case only the first one will match the host_sof. */ - dev_sof = get_unaligned_le16(&data[header_size - 2]); - if (dev_sof == stream->clock.last_sof) + sample.dev_sof = get_unaligned_le16(&data[header_size - 2]); + if (sample.dev_sof == stream->clock.last_sof) return; - dev_stc = get_unaligned_le32(&data[header_size - 6]); + sample.dev_stc = get_unaligned_le32(&data[header_size - 6]); /* * STC (Source Time Clock) is the clock used by the camera. The UVC 1.5 @@ -541,21 +552,19 @@ uvc_video_clock_decode(struct uvc_streaming *stream, struct uvc_buffer *buf, * and stc and sof are zero. */ if (buf && buf->bytesused == 0 && len == header_size && - dev_stc == 0 && dev_sof == 0) + sample.dev_stc == 0 && sample.dev_sof == 0) return; - stream->clock.last_sof = dev_sof; - - host_sof = usb_get_current_frame_number(stream->dev->udev); + sample.host_sof = usb_get_current_frame_number(stream->dev->udev); /* * On some devices, like the Logitech C922, the device SOF does not run * at a stable rate of 1kHz. For those devices use the host SOF instead. */ if (stream->dev->quirks & UVC_QUIRK_INVALID_DEVICE_SOF) - dev_sof = host_sof; + sample.dev_sof = sample.host_sof; - time = uvc_video_get_time(); + sample.host_time = uvc_video_get_time(); /* * The UVC specification allows device implementations that can't obtain @@ -578,46 +587,29 @@ uvc_video_clock_decode(struct uvc_streaming *stream, struct uvc_buffer *buf, * the 8 LSBs of the delta are kept. */ if (stream->clock.sof_offset == (u16)-1) { - u16 delta_sof = (host_sof - dev_sof) & 255; + u16 delta_sof = (sample.host_sof - sample.dev_sof) & 255; if (delta_sof >= 10) stream->clock.sof_offset = delta_sof; else stream->clock.sof_offset = 0; } - dev_sof = (dev_sof + stream->clock.sof_offset) & 2047; - - spin_lock_irqsave(&stream->clock.lock, flags); - - sample = &stream->clock.samples[stream->clock.head]; - sample->dev_stc = dev_stc; - sample->dev_sof = dev_sof; - sample->host_sof = host_sof; - sample->host_time = time; - - /* Update the sliding window head and count. */ - stream->clock.head = (stream->clock.head + 1) % stream->clock.size; + sample.dev_sof = (sample.dev_sof + stream->clock.sof_offset) & 2047; + sample.dev_stc = get_unaligned_le32(&data[header_size - 6]); - if (stream->clock.count < stream->clock.size) - stream->clock.count++; - - spin_unlock_irqrestore(&stream->clock.lock, flags); + uvc_video_clock_add_sample(&stream->clock, &sample); } -static void uvc_video_clock_reset(struct uvc_streaming *stream) +static void uvc_video_clock_reset(struct uvc_clock *clock) { - struct uvc_clock *clock = &stream->clock; - clock->head = 0; clock->count = 0; clock->last_sof = -1; clock->sof_offset = -1; } -static int uvc_video_clock_init(struct uvc_streaming *stream) +static int uvc_video_clock_init(struct uvc_clock *clock) { - struct uvc_clock *clock = &stream->clock; - spin_lock_init(&clock->lock); clock->size = 32; @@ -626,15 +618,15 @@ static int uvc_video_clock_init(struct uvc_streaming *stream) if (clock->samples == NULL) return -ENOMEM; - uvc_video_clock_reset(stream); + uvc_video_clock_reset(clock); return 0; } -static void uvc_video_clock_cleanup(struct uvc_streaming *stream) +static void uvc_video_clock_cleanup(struct uvc_clock *clock) { - kfree(stream->clock.samples); - stream->clock.samples = NULL; + kfree(clock->samples); + clock->samples = NULL; } /* @@ -2108,7 +2100,7 @@ int uvc_video_resume(struct uvc_streaming *stream, int reset) stream->frozen = 0; - uvc_video_clock_reset(stream); + uvc_video_clock_reset(&stream->clock); if (!uvc_queue_streaming(&stream->queue)) return 0; @@ -2257,7 +2249,7 @@ int uvc_video_start_streaming(struct uvc_streaming *stream) { int ret; - ret = uvc_video_clock_init(stream); + ret = uvc_video_clock_init(&stream->clock); if (ret < 0) return ret; @@ -2275,7 +2267,7 @@ int uvc_video_start_streaming(struct uvc_streaming *stream) error_video: usb_set_interface(stream->dev->udev, stream->intfnum, 0); error_commit: - uvc_video_clock_cleanup(stream); + uvc_video_clock_cleanup(&stream->clock); return ret; } @@ -2303,5 +2295,5 @@ void uvc_video_stop_streaming(struct uvc_streaming *stream) usb_clear_halt(stream->dev->udev, pipe); } - uvc_video_clock_cleanup(stream); + uvc_video_clock_cleanup(&stream->clock); }