From patchwork Sun Nov 24 23:50:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Armin Kuster X-Patchwork-Id: 180141 Delivered-To: patch@linaro.org Received: by 2002:a92:38d5:0:0:0:0:0 with SMTP id g82csp2443636ilf; Sun, 24 Nov 2019 15:52:06 -0800 (PST) X-Google-Smtp-Source: APXvYqz12atoKgvVGmAA4xADlI0eerr0sX6TyHF+DiYRtImsnH1GA7BcK3DZB4vRKsTLO0+0Ic7z X-Received: by 2002:a62:2b8b:: with SMTP id r133mr32166554pfr.7.1574639526844; Sun, 24 Nov 2019 15:52:06 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1574639526; cv=none; d=google.com; s=arc-20160816; b=fBpnQmDkK24ZI8ZiOEpyhvpdAYaqmfud74F+2dBv87ucXo22LUUqC7Lv4bKOHWQNwt /MBF7Jecevdq5hKjIjcjFHIyF+exSanqmBJgFm4h/Ez7cJ5dKopPA/5wMm5BIy2HxiOn zBUXPOTtoD/u/x03Zo9nQ2JB83QcZRdvkhHr55FI6uO3poRYjT9C4kjeXWg+jREmmfKC zbqIEEgk4e38MGQvVy6Df4hl+5Xz7PKkStHevm++vcIf+Dc8xXCKNGUyBC+hpMjWwgP0 kqYQNWe4T0HwpY4QJlmFAXCPZCBCGztcMadBPSUxBBxCBaGcjYc6DzAy55Wsllm+oBXp zAWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:mime-version :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to; bh=guDzzgQlfOonNA+HR11qRKdGCKH5kFz2wfii7Q92M8I=; b=GefaTXGH2pDHf2AW9Q+hIsbv+oDfvoONUdMyOinZcuUHi7s5mwpbcUAD0wYMeL6vsM GG0nG035xCELaJf5XhCNNN9ActDV/LbSHpzH8RMucxLv7HSMBVpDP8dLiDdJoddsu/yb 2rsdmWuw4Uaj+XxhftwOgtzNcGKbMBkPNaoCmWzIwwxuv46WAzazMG/ZfWO+m2D0Corx DdmmpP1KWK14UuF9DH0LxNA+AY1UY47lxk40huTEOXocJmnp5dviirx9gdBLWvF/aS4Q GmGnZxNavWoM42EcW/Mr8a4aaoHGMxYwkYKTE7kHi76TQeiuiCOby1BpzLjoo+e9EkZf wC6A== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b="AUBnaeh/"; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id k30si5827517pgi.217.2019.11.24.15.52.06; Sun, 24 Nov 2019 15:52:06 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b="AUBnaeh/"; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id A83F07FCA3; Sun, 24 Nov 2019 23:51:12 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-pj1-f66.google.com (mail-pj1-f66.google.com [209.85.216.66]) by mail.openembedded.org (Postfix) with ESMTP id 20AA47FC0D for ; Sun, 24 Nov 2019 23:50:52 +0000 (UTC) Received: by mail-pj1-f66.google.com with SMTP id v93so2337856pjb.6 for ; Sun, 24 Nov 2019 15:50:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references; bh=CTKHuYIdlGI1K3BGETzGGNJ3x3TD6nzJoEQ5WbWUBUo=; b=AUBnaeh//3Y6CGypebfnutQYuAr1kjaXt/Nr5sq+vWFdrSpDtJBU9FifzDCxaIy0XH TvpZHihaw1jxrWqrl3aMNcyW5a3zZSyL6hSp2zWBlYeKyOcv0aK6/k+aoSO72gYwiB32 mCUkNALZTrvKnDpn5zTH2CVjiEGx1HT7uJ3TC7J4T6GOnDHzAclQ3Yz4hDG+1wcer78y aEMFaO/Sy/Ip6VcOHeavYmi1/Sr18yvq3Ww2u8qBUtm3FhWANw8nSYHWutu3SGcc4PxP e3ACpEeuQlV0U2zsViMEeiSjfjYcvu0itOHqe8id7WZuy+keS9nRgv6/u2mUVB/q0tJW 3hFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=CTKHuYIdlGI1K3BGETzGGNJ3x3TD6nzJoEQ5WbWUBUo=; b=l06xsK4QmnjbtoBTJIKhOuR8ZN+rYA/wfI6PyRBOXe1Nga1v4AvwqIsAMq7xUS1R0g 6MvPF3NSVnHWdMVCMm9TYK2pWqITqM+CXMRhlpwwq/IwI7Qjsie3+w4rPClm+dgg4rwL YAXWHVKT9PXIoJdIAlyTeruLdO5DOPJ9Pxi0mZkRr6aitkfkBPWb7fpAPqg4NRzjgS61 ZCBQgliyiiik57DhIx4CwtCa6KHAYUNdU3Bia5IMilua2pZCfyLXRKuR95zh78vEfxPc HpSD52IN8F+kLlvfKN+JaDKT8rtBOOoagsNUBV0uAx/UgLLgMqRq57xrb2qqma4GgGcq s70A== X-Gm-Message-State: APjAAAVlwX3UTe9mtKO/ysPaCDlrLsuuORry5doflD3M9yAiaoVEQ5i7 47Qa+OPyrVyOhq6Msc8QFCCDWONB X-Received: by 2002:a17:902:6b47:: with SMTP id g7mr25829858plt.87.1574639453006; Sun, 24 Nov 2019 15:50:53 -0800 (PST) Received: from akuster-ThinkPad-T460s.hsd1.ca.comcast.net ([2601:202:4180:a5c0:69ac:e4d2:e89f:98da]) by smtp.gmail.com with ESMTPSA id q200sm5619783pfq.87.2019.11.24.15.50.52 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 24 Nov 2019 15:50:52 -0800 (PST) From: Armin Kuster To: openembedded-core@lists.openembedded.org Date: Sun, 24 Nov 2019 15:50:15 -0800 Message-Id: X-Mailer: git-send-email 2.7.4 In-Reply-To: References: Subject: [OE-core] [zeus 10/35] cve-check: fetch CVE data once at a time instead of in a single call X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton This code used to construct a single SQL statement that fetched the NVD data for every CVE requested. For recipes such as the kernel where there are over 2000 CVEs to report this can hit the variable count limit and the query fails with "sqlite3.OperationalError: too many SQL variables". The default limit is 999 variables, but some distributions such as Debian set the default to 250000. As the NVD table has an index on the ID column, whilst requesting the data CVE-by-CVE is five times slower when working with 2000 CVEs the absolute time different is insignificant: 0.05s verses 0.01s on my machine. (From OE-Core rev: 53d0cc1e9b7190fa66d7ff1c59518f91b0128d99) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie Signed-off-by: Anuj Mittal --- meta/classes/cve-check.bbclass | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) -- 2.7.4 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass index e95716d..19ed554 100644 --- a/meta/classes/cve-check.bbclass +++ b/meta/classes/cve-check.bbclass @@ -267,17 +267,17 @@ def get_cve_info(d, cves): cve_data = {} conn = sqlite3.connect(d.getVar("CVE_CHECK_DB_FILE")) - placeholders = ",".join("?" * len(cves)) - query = "SELECT * FROM NVD WHERE id IN (%s)" % placeholders - for row in conn.execute(query, tuple(cves)): - cve_data[row[0]] = {} - cve_data[row[0]]["summary"] = row[1] - cve_data[row[0]]["scorev2"] = row[2] - cve_data[row[0]]["scorev3"] = row[3] - cve_data[row[0]]["modified"] = row[4] - cve_data[row[0]]["vector"] = row[5] - conn.close() + for cve in cves: + for row in conn.execute("SELECT * FROM NVD WHERE ID IS ?", (cve,)): + cve_data[row[0]] = {} + cve_data[row[0]]["summary"] = row[1] + cve_data[row[0]]["scorev2"] = row[2] + cve_data[row[0]]["scorev3"] = row[3] + cve_data[row[0]]["modified"] = row[4] + cve_data[row[0]]["vector"] = row[5] + + conn.close() return cve_data def cve_write_data(d, patched, unpatched, cve_data):