From patchwork Wed Nov 13 17:02:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 179336 Delivered-To: patch@linaro.org Received: by 2002:a92:38d5:0:0:0:0:0 with SMTP id g82csp9873059ilf; Wed, 13 Nov 2019 09:02:42 -0800 (PST) X-Google-Smtp-Source: APXvYqxHAipDmmMdsNO0+9YGTlJWJey5sof5ts0uAVsWB25LFq7XQFjWxm00UY6JIJSvG7wkISe1 X-Received: by 2002:a17:90a:d792:: with SMTP id z18mr6452840pju.34.1573664562719; Wed, 13 Nov 2019 09:02:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1573664562; cv=none; d=google.com; s=arc-20160816; b=t/SNtIvh73YQCjp9qDyIAABMJGj5M/LXnitQCvrMPWoJ2SnnsL2Lw4kT/hBZcMbBof toljjdHai22AKVLbPNi+VrdURE/3r3V0CO0zRQG2a31CgIIoKAlVsHPUKmEQXIVJYhuo mxPKwkxPPFA2bDyDChMlw0QV4YMpsc4bHYeNRcO1OaNdMfMXZj1gASq/EvkY2uXUxlUy 0sLVoeMmMfW0TOl/rXhNBUYZT/Jijiwcck0qdZtk5Pyc+hdNHXioe0CiLlSdacrCb78m BVxDmtVnIMCiTxrRdgxOhR849HGh8ASPV+ZLYvmiwlfMgD249OYysbRTx4qPU6zw7woJ YUzg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:message-id:date:to:from:dkim-signature:delivered-to; bh=6+Qu2nL8hIm5rlmjkg8CQ5Cnppbl7BIwD3Pm1AJOxNk=; b=kAALkRmbSguqY2AsPBd0Z25XWTg2I9wTeIZrTLYI5IlwuG2RPVa9KSy+26BuHp8RfQ hvl+HR6NAIEC37kSsZnlIpaiCEIpKprZdb0u0+drQNSo+cQg0/hAxwcr9+lL3/k3URcF Gxf8CNeNt6RzsurEqOtCDrgZZBqMKIeQM3tfIV1Z5hN1US/L1bjbOFxy4xlngbI9g3sK r3uDRQYDM+VIPT6MdzO/PogrtKBPwIodg2c6PrEQbmfC9uc1z1lKAlzwOVTg5++bCoIF bp5vfkM/8GjHj04J/I0rBd04SWC2iHDR2vw540dx0b0R2Vom0DAmnWp5J1X5SzdrCeX0 YA7g== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=cAUA3dV2; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id h18si3497394pgg.259.2019.11.13.09.02.42; Wed, 13 Nov 2019 09:02:42 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=cAUA3dV2; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 61E827F7D3; Wed, 13 Nov 2019 17:02:39 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-wm1-f65.google.com (mail-wm1-f65.google.com [209.85.128.65]) by mail.openembedded.org (Postfix) with ESMTP id 1D23175E2D for ; Wed, 13 Nov 2019 17:02:37 +0000 (UTC) Received: by mail-wm1-f65.google.com with SMTP id b17so2860782wmj.2 for ; Wed, 13 Nov 2019 09:02:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=mf45DdTgWRs5x5BDGPO5ZG82r/eQAhxp1MbZLLqNFhQ=; b=cAUA3dV26YXaF3QTZ4HyoIEtqs3pI/pvi54Lc59B8LGe2315a8maSjErd5emOiDDaG hGvxsMWHLLU3SGlu7OpeaTdIyJJMTs5UkGGtE1muIZ0liyrZD20CIvTZ1rsRrH6TGdkh 15mV870ZhV1rGLexDz5env/eQN2UbX7qPWk5CNculm426XYyRwgxmj1ioGOm9+3unR3z 6mkgY5oY8PlPOc0zS+9hppkSRZNtzeD3X2TDI9TKQa64mmnizFZlPAVKU/H6GEFocgVl 0Gks4cNHC1ZjJX8SHpIsN70+ZKCGtkdXshqi7ajDlgVtw6UtVrnoL9I305D+2cXAWIv2 YxAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=mf45DdTgWRs5x5BDGPO5ZG82r/eQAhxp1MbZLLqNFhQ=; b=oQ1Il/OkJ1p33v4b5Ry8b8ORrATW8/XzP9RKh0834OI8yXGnWLGgy444PxMtOT67Cq qNYeNHRz8vhdjNZP+cCfpY6chYYL923kOa4tuPh3fcfwxGtO37NOhRKEGdNgpwkmQ7i9 lRKoJqPoNc/59U8T1ulRx/6DpNApJP+7kQlxfpdFUP9068WAgpVsds+EiUJ4m3/MECQP CvO7ZIecuAh65EVIA3neIaqABQXnyHFuzUUozfEZbuRmU5oqcwM5UcqttV6mhtNqZusV BhZ2vsMVp0m71SOfBszXFL2TNcXpeSjcESv0BbelHtuyVFpImC6i0l1QRS4nO+/T7iNR Dmvg== X-Gm-Message-State: APjAAAXkXcOGP8Pqb449LOpG2e3HXgdpFbJsydHKBuMMUVHrCanpuJKI jikTIedKQTcXWIcVcQKsWt13ibQO9GI= X-Received: by 2002:a1c:ab0a:: with SMTP id u10mr3978932wme.0.1573664558339; Wed, 13 Nov 2019 09:02:38 -0800 (PST) Received: from flashheart.burtonini.com (35.106.2.81.in-addr.arpa. [81.2.106.35]) by smtp.gmail.com with ESMTPSA id j3sm3318736wrs.70.2019.11.13.09.02.36 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Nov 2019 09:02:36 -0800 (PST) From: Ross Burton To: openembedded-core@lists.openembedded.org Date: Wed, 13 Nov 2019 17:02:35 +0000 Message-Id: <20191113170235.9907-1-ross.burton@intel.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Subject: [OE-core] [PATCH v3] cve-check: fetch CVE data once at a time instead of in a single call X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org This code used to construct a single SQL statement that fetched the NVD data for every CVE requested. For recipes such as the kernel where there are over 2000 CVEs to report this can hit the variable count limit and the query fails with "sqlite3.OperationalError: too many SQL variables". The default limit is 999 variables, but some distributions such as Debian set the default to 250000. As the NVD table has an index on the ID column, whilst requesting the data CVE-by-CVE is five times slower when working with 2000 CVEs the absolute time different is insignificant: 0.05s verses 0.01s on my machine. Signed-off-by: Ross Burton --- meta/classes/cve-check.bbclass | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass index e95716d9ded..19ed5548b3a 100644 --- a/meta/classes/cve-check.bbclass +++ b/meta/classes/cve-check.bbclass @@ -267,17 +267,17 @@ def get_cve_info(d, cves): cve_data = {} conn = sqlite3.connect(d.getVar("CVE_CHECK_DB_FILE")) - placeholders = ",".join("?" * len(cves)) - query = "SELECT * FROM NVD WHERE id IN (%s)" % placeholders - for row in conn.execute(query, tuple(cves)): - cve_data[row[0]] = {} - cve_data[row[0]]["summary"] = row[1] - cve_data[row[0]]["scorev2"] = row[2] - cve_data[row[0]]["scorev3"] = row[3] - cve_data[row[0]]["modified"] = row[4] - cve_data[row[0]]["vector"] = row[5] - conn.close() + for cve in cves: + for row in conn.execute("SELECT * FROM NVD WHERE ID IS ?", (cve,)): + cve_data[row[0]] = {} + cve_data[row[0]]["summary"] = row[1] + cve_data[row[0]]["scorev2"] = row[2] + cve_data[row[0]]["scorev3"] = row[3] + cve_data[row[0]]["modified"] = row[4] + cve_data[row[0]]["vector"] = row[5] + + conn.close() return cve_data def cve_write_data(d, patched, unpatched, cve_data):