From patchwork Mon Nov 18 16:46:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 179592 Delivered-To: patch@linaro.org Received: by 2002:a92:38d5:0:0:0:0:0 with SMTP id g82csp3126805ilf; Mon, 18 Nov 2019 08:46:56 -0800 (PST) X-Google-Smtp-Source: APXvYqwZohS3zODPkReEZtnq7CBV+ehsp8f7ODiaPTGdNf4UAQlcf9yt7Z+OkXRzXl5WAdUmh19O X-Received: by 2002:a17:90a:eb0f:: with SMTP id j15mr414584pjz.97.1574095616134; Mon, 18 Nov 2019 08:46:56 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1574095616; cv=none; d=google.com; s=arc-20160816; b=H5gP1bYbQPC4t2U6Y9jo8R3mDc79ToweNcF7bu0z6CmxI9/70lpLdvAwgSK0F8p8V2 /bQm1OkbeqG9A1rhduh7qJroIo72OWXV53iHSrkVn411v5TMqRKO+Cyh9LHS2XYzhKF9 M+dtTmrklYwBhOriOByk/fE5j+IlUSqiVSa7pC0m4uY4ki6x/XKzl01olWgy4t9reocP 8/btVJEbK8fQ1QBJEpZioyIo67K+w9J876LQ8Mx28BaomNM4iJX9Dka/+Bzp6z36jWOC n9f2QssLUgB9GDTLYHrGkGVJ0kT0QQDemxBjrzJc3Hjk/w7mZERJ+Rsvjn1ek7HQ5jie O6vw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:message-id:date:to:from:dkim-signature:delivered-to; bh=z2wTL46904tAm0Pqx1kKNWZdaS8FvssyWyrwpM2LYZA=; b=qNUjAZSGj8Fsfd98iSL4Bf31UjF/raUQv7LvEcRsv1rgMduy+Om8CWspZLxGm9khV+ fYWpZHwE9c1V0ZBkXBzaQqkH2Z8rXYWmPg/O3xOyqf1M0vMEj1REiX26KXJ1DyPhgIQk b1fKVDXL2UprT59ArupowKQoHi/uoV3+TvLs6Cn2CEh+F5xwWLVVHUD61MvLzufJWnFH /wHvPrrhQXxMrLNeYuw1jqA99htHfuVhkNy5AkxKgE0bCPZIOiDJAX9eQCcMsAvJtbAz ynErXyybWVQXAimBKpRxRjeH3pImEhe/vOtL17o3CdHnGPs3tzj53YGoesWdNN3k7rjB V71A== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=jCwvK+xf; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id m11si5508757pgq.72.2019.11.18.08.46.55; Mon, 18 Nov 2019 08:46:56 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=jCwvK+xf; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id C79A97F8B2; Mon, 18 Nov 2019 16:46:53 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-wm1-f67.google.com (mail-wm1-f67.google.com [209.85.128.67]) by mail.openembedded.org (Postfix) with ESMTP id F23A97F8A0 for ; Mon, 18 Nov 2019 16:46:52 +0000 (UTC) Received: by mail-wm1-f67.google.com with SMTP id t26so19742254wmi.4 for ; Mon, 18 Nov 2019 08:46:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=rWeYatW3qrAK2McH4FAaYl6YFFDJukQ8lNRElosSY2s=; b=jCwvK+xf8eYy//5qQyKFctpY9oMTIIFId5+yAHRsmFp0/j8Smgi3F742ep7MQ1obLm qkB6fRhFSm9wjPkpFVKjM96QFew9L8kLku1tWwSqdHAAiHfhBhJBHrJQjA4UaGmsgmWe 9eZE/C6ESToE1ELy8vvfWv1xw7EEiAU9dVHwNaw2AS1h9Maj4zqRl+jlxgzID2SRjqOY wGwKqhMmW3JFPNbmiy+UeaocxR2/p+nibLUZozsD24ijFhCEv78sI0YAUOO6KTwjJCJg dbVrx23P7hhZviI5JyrQrsOKUVpAeMzd8MQgp5Gmr2b/3UgVDVM4vjSnhf4nfsCEzUIk tkag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=rWeYatW3qrAK2McH4FAaYl6YFFDJukQ8lNRElosSY2s=; b=oSO2TCR8iyZhiUJWNvYUbsBHc/g74oVOypFvp1NhHuf2KkdeJC8/NRtMPL7IiK5Th4 hOaZ3Rsa5Rtyoig5g6J33EjceVrE3rAstu3qk+GBtTpw6rfILkvvSxlvorzEgrIhuj84 g9SZIIpcyQ/0RT3RFLRx7tsQji7GIOhLlkHQq4vXLUr8iBiQYPhgsMoSiLORliYQaqD5 2Owpl1csrNchGGhhvI0qfVIAF6U7amMKkvdnr9xgU4oAAwopp6MNQSRoQMo9b+Xrsj21 8FVY9YauRoYzrSk7oQIPxdY1FSvrmMiHlimAW4dVS0XKjSxvaSuoPCBzkLDjp+AfY7fK 11lA== X-Gm-Message-State: APjAAAWqWyI034AkjC4pXZ7Uy5vZkSOpg/bndy7228t3IzoQnbBt/FGA TF0hJRbpp0PvTswJ65DDb+ieDYi1nB8= X-Received: by 2002:a05:600c:210b:: with SMTP id u11mr31263547wml.170.1574095613398; Mon, 18 Nov 2019 08:46:53 -0800 (PST) Received: from flashheart.burtonini.com (35.106.2.81.in-addr.arpa. [81.2.106.35]) by smtp.gmail.com with ESMTPSA id y189sm20936208wmb.13.2019.11.18.08.46.51 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Nov 2019 08:46:51 -0800 (PST) From: Ross Burton To: openembedded-core@lists.openembedded.org Date: Mon, 18 Nov 2019 16:46:42 +0000 Message-Id: <20191118164647.29409-1-ross.burton@intel.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Subject: [OE-core] [PATCH 1/6] cve-update-db-native: don't hardcode the database name X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org Don't hardcode the database filename, there's a variable for this in cve-check.bbclass. Signed-off-by: Ross Burton --- meta/recipes-core/meta/cve-update-db-native.bb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb index 19875a49b1c..c15534de08b 100644 --- a/meta/recipes-core/meta/cve-update-db-native.bb +++ b/meta/recipes-core/meta/cve-update-db-native.bb @@ -28,8 +28,8 @@ python do_populate_cve_db() { BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-" YEAR_START = 2002 - db_dir = os.path.join(d.getVar("DL_DIR"), 'CVE_CHECK') - db_file = os.path.join(db_dir, 'nvdcve_1.0.db') + db_file = d.getVar("CVE_CHECK_DB_FILE") + db_dir = os.path.dirname(db_file) json_tmpfile = os.path.join(db_dir, 'nvd.json.gz') # Don't refresh the database more than once an hour From patchwork Mon Nov 18 16:46:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 179593 Delivered-To: patch@linaro.org Received: by 2002:a92:38d5:0:0:0:0:0 with SMTP id g82csp3126909ilf; Mon, 18 Nov 2019 08:47:01 -0800 (PST) X-Google-Smtp-Source: APXvYqyt3f2Tcb4jiOTL8+QMOM08DO+7oBvbZv1kMX/Ie6xIOOshfZfWCtruRgEVZBPSkrAhDVjF X-Received: by 2002:a63:5508:: with SMTP id j8mr219124pgb.97.1574095621409; Mon, 18 Nov 2019 08:47:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1574095621; cv=none; d=google.com; s=arc-20160816; b=rW54EgWiSwr9/oi1cprJHl+HiwWRJUtKXlZ4yy70D7SsuBiRyWU3qLf8wzWTrnr2TF NdPO0qgpMyw+8OLvZ6fNC2bJ9xAaheY96yq1OJYFMPaiOFN/feo2ADykNug/mh589dKK 6gdkrJoAlA7Gactb0vCiwPka3IEeYCR0W6ZQo0l11ZJ69PE7WpzJnRgsMBuHy83JhrZP vd9ZKqFddqOtdv0IQbQPjDF0V3Eb1WKhzX7USyutxBkEiNIy6ERgGOxpT3sq22zmETIJ GGObxP775LdnKHJnGeE7V+ME3IS5GMSmY6gcDV74RUMw1N8aKutB6Y6dwaKvuDDwiEBo LhWA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :dkim-signature:delivered-to; bh=0JeXNvaxsH4lEEd0w8gx9VtmlrweWozjF0hKkwx/z30=; b=seg0KUqC0HtlEfSC+JkTeFP5fY03n8YQ0zK0OqA73rdsTv0YAi48Pg66EN5WQzw5hT iRvEBiOmDngAN3cKRoXI1Ai1F7yemMCHR/OGl1GZ8gPr82V0Po5kwzWPIR8y/r8nY1JW zSklZVh/bMOmd8oYLFCQe7RB/Q8wZXUQ9Dh6GMJRnXQjYmRJ4a8aphFCpdbbZWbiBJeU /ItiXGA2PmvenAAy7TS94Jh5cG5osr3hDwCOKHGu9vlx76rDY8chVCofsspCn6z/ZLNS xQdkdGgFccsTcO9Alu3yhgSb7VsanGTfz+bzTpmhft1q7d/0aK1BddGbVbmgH4ZeZA1p sFdg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=mPheYBGy; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id f8si19175385plo.142.2019.11.18.08.47.01; Mon, 18 Nov 2019 08:47:01 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=mPheYBGy; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 68FC07F8BB; Mon, 18 Nov 2019 16:46:57 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-wr1-f46.google.com (mail-wr1-f46.google.com [209.85.221.46]) by mail.openembedded.org (Postfix) with ESMTP id C19C07F8A0 for ; Mon, 18 Nov 2019 16:46:53 +0000 (UTC) Received: by mail-wr1-f46.google.com with SMTP id a15so20334338wrf.9 for ; Mon, 18 Nov 2019 08:46:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=M4ctuwA6cZs27fmHe6BG8al2Ng7rxmll+fGGZm4Bf/M=; b=mPheYBGy7AJJQE34JQMAAYW550m7zMmvWaaRrdaXK56dhjaVQoptQhX1w7ovXiQsrp p3lxWeMpTYPHpgz8eqb8rfJdk9yrwLJPrSJTaYnhYHR0eGxnB9pM2hwuzOY+brjUGeRR 2pP9s65VGKfbkD+b8ybZiU4oPzmpuopmUXVeXs/V6pfvk+RGA2fDjLqSi3mpPPNcEtex RymCHt+if9D2KKoOTEzK6oZQiQticp2wO/Eq9UXnmdVu0+796vQQW07pE6PPiQeX8HjA F6MGLpd5ZKJOqJ46G8Ny8necS45VYCEeoq93EAqb2xAJU/lHscaJ8oN4aR0Tk7Vh8qWq DDCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=M4ctuwA6cZs27fmHe6BG8al2Ng7rxmll+fGGZm4Bf/M=; b=sH8FyDWaPbaB1n6+P//U2pzE64KYE4WMhoUasFF63gHPhjULcPiWvCVTnTaVMQu9X2 acyrZhVKoBSMAxx3s9z1Q3dYPdfUDnq4her/SJeC5FiNncOJRmptjjNBhReTFbEkwWQl iPf7QXmsREmozAkS1F6zaiQVJBN2wO4R31QFQL2r+LOSCMlro+fVA5v+qXmaOBSluAZ6 cQ5EgaGsyEdGd8iPmeVS5Rzy0XrmIbK6fei8glPIEW4qCtXlOcX4e/LktNDIB6RhTYEM IBpWa5kKQpyL4mrqO0wIxM10p8YzAQXBynkLh+bqma9rFaLrHoo/jwk6750POPDzFvg5 XBmA== X-Gm-Message-State: APjAAAXxGDq/WfiB2Zadq6tR0GO1YQceoYgR65pMIXJljta5oodd4vGK p98lR00KtKaHWgpjpXyIYso19NaBKRM= X-Received: by 2002:a5d:67ce:: with SMTP id n14mr31034305wrw.67.1574095614265; Mon, 18 Nov 2019 08:46:54 -0800 (PST) Received: from flashheart.burtonini.com (35.106.2.81.in-addr.arpa. [81.2.106.35]) by smtp.gmail.com with ESMTPSA id y189sm20936208wmb.13.2019.11.18.08.46.53 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Nov 2019 08:46:53 -0800 (PST) From: Ross Burton To: openembedded-core@lists.openembedded.org Date: Mon, 18 Nov 2019 16:46:43 +0000 Message-Id: <20191118164647.29409-2-ross.burton@intel.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191118164647.29409-1-ross.burton@intel.com> References: <20191118164647.29409-1-ross.burton@intel.com> MIME-Version: 1.0 Subject: [OE-core] [PATCH 2/6] cve-update-db-native: add an index on the CVE ID column X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org Create an index on the PRODUCTS table which contains a row for each CPE, drastically increasing the performance of lookups for a specific CVE. Signed-off-by: Ross Burton --- meta/recipes-core/meta/cve-update-db-native.bb | 3 +++ 1 file changed, 3 insertions(+) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb index c15534de08b..08b18f064f0 100644 --- a/meta/recipes-core/meta/cve-update-db-native.bb +++ b/meta/recipes-core/meta/cve-update-db-native.bb @@ -120,11 +120,14 @@ python do_populate_cve_db() { def initialize_db(c): c.execute("CREATE TABLE IF NOT EXISTS META (YEAR INTEGER UNIQUE, DATE TEXT)") + c.execute("CREATE TABLE IF NOT EXISTS NVD (ID TEXT UNIQUE, SUMMARY TEXT, \ SCOREV2 TEXT, SCOREV3 TEXT, MODIFIED INTEGER, VECTOR TEXT)") + c.execute("CREATE TABLE IF NOT EXISTS PRODUCTS (ID TEXT, \ VENDOR TEXT, PRODUCT TEXT, VERSION_START TEXT, OPERATOR_START TEXT, \ VERSION_END TEXT, OPERATOR_END TEXT)") + c.execute("CREATE INDEX IF NOT EXISTS PRODUCT_ID_IDX on PRODUCTS(ID);") def parse_node_and_insert(c, node, cveId): # Parse children node if needed From patchwork Mon Nov 18 16:46:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 179594 Delivered-To: patch@linaro.org Received: by 2002:a92:38d5:0:0:0:0:0 with SMTP id g82csp3127057ilf; Mon, 18 Nov 2019 08:47:08 -0800 (PST) X-Google-Smtp-Source: APXvYqzVDE4s58B8H5uCzEjhYEqeA6gOSvxiRUhadiX9FJDWHhOScwWljX8fUE4BP3LDymXlxDfh X-Received: by 2002:a63:a449:: with SMTP id c9mr235925pgp.53.1574095628805; Mon, 18 Nov 2019 08:47:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1574095628; cv=none; d=google.com; s=arc-20160816; b=HYZ0KoyxNInpoZx3c6LOke0t2poLz9vxshB2mhH0oHwaMEk+IwTrlReNJuzCaO1NRc VRf7JTozIoUZO4lAOznKhlAGMHKdDX8zm1omlL7ieAEPo4/Hp6Lok5TdN6KH4Z+buS0s xrAYK3XmosjppiylsCL2bx31NroyTsXtC21j5UB2FRuhKGytSWXiCNl4RfsitCjgZf5o mcGb0OdM7RdiMxtfG9d7xahxgapqeIKwVVRpf8n8GNyVJbhaK/DpOAhYm/LJ38kEsCeU 2J4SHUKKvowpTslown18sX2IjxbN17x7T6oIVBmF+LXRAe3GsfoAMRgz9voTpu5MMDmx WPzA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :dkim-signature:delivered-to; bh=9qeccblzfXfJ9AYYLWYOdZVdu57gF/oN9TKnrguKsFw=; b=wLnWHn4EPLjnZtATJBOAb8QIMURZZBqO4YGksMu6Rm/8538VYJ3LUf3coogSo2Xccm 4qmhja+OKS5uwr324jRoVOyHE4ckkcsCgSiVXQTttj52idQqpjvlrtMt1mhet+fNKqt4 XRa3VKHkY7FAvUe1su+Bkcb4Csu2z0gR8KPM+tLZouwFTxot8lhh7AhTYyWvmxd97E+E eWloXRGMQccHTaxO9Guf8FuaTrPqoJW3P+D7Yyg3gorxCiPdn8QADmNZQ5Nm9w5XaFIu i1OMqET53jjo+RlCFf61ai/9Jd11YMr8PYSwBQ/+8MBMulV5N9Y5ZHYkCZFav0DDv390 Y7pg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b="a9A7/Qoc"; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id 67si20680423ple.265.2019.11.18.08.47.08; Mon, 18 Nov 2019 08:47:08 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b="a9A7/Qoc"; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 8C1487F947; Mon, 18 Nov 2019 16:46:58 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-wr1-f53.google.com (mail-wr1-f53.google.com [209.85.221.53]) by mail.openembedded.org (Postfix) with ESMTP id D035E7F8C6 for ; Mon, 18 Nov 2019 16:46:54 +0000 (UTC) Received: by mail-wr1-f53.google.com with SMTP id q15so7538138wrw.7 for ; Mon, 18 Nov 2019 08:46:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=GJwE60olmB34OdDBRJlGoCyRwjwAxCOyFJr0nPQzq5w=; b=a9A7/QocVtg879NJKdPegUY0gnzkRYFWwRie6EYfN2NAMwPZlz/tjeVp7AEF4ahZsO aeOxF8h0AB60ili0SopVqVCkgO/dnh4Y8RNkrG3BjGEcQ/FYE9hVw6PhlkFPDGuTkpkN dsEuGmAp4qfpIkLB3rNg+L1j1p7gDYlGTP0BfSzx9Je5K0S52TqjJU8M7MWp/F/LZ/Wq pleXYav2mVvBjQUvZvMB1V6cCH0TITj/u8FLD49a2ZfpaU6kH35Yr0WsgaxcXSTLx1Fa uMNl4BSGHDaA1rC2BomBmuOIARXLqQ3d5XMV7vTwO/H6J3hAylOJlNFYB+9eQGQWz28v UK4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=GJwE60olmB34OdDBRJlGoCyRwjwAxCOyFJr0nPQzq5w=; b=kqajzjLsmhsZXQwX0r1e+qxKn3NHgp45Cy+itd3G2G/2REakGaz/aKdME4jCZmCu89 thUWx2UnpNhmoWZA43vIRqWwuhIQsFKD+kl4kO6uaGfJeUi+kQfdLS6hKopbqat0/n75 poGtEI770nLmSoDq7l38GdlwLevkOpP6jtvYuLh+b4PydYawqf39bIqdF6Jx6VjieHa0 KutWL1ZD36Q/pgH8sCXp3E2pgD4DswUQBmMQUBviaQhiRvo8pFY+ShIE5k/nCyhvvMdu Id7FZE+pzcs8LNYjSGO8MTXjubO9EQY6PSWkWF8p8wu1U+oVh3YKPoWroisevDjXB1/A A+gw== X-Gm-Message-State: APjAAAVxAykgZyyxvdeSVo5Xgt7l+WeAzmpkLdSR3e5uLCAB6okKEILN r5nKHkr2IYvC7TMjFEjfRemtM0lMPno= X-Received: by 2002:adf:cd0a:: with SMTP id w10mr17032533wrm.4.1574095615155; Mon, 18 Nov 2019 08:46:55 -0800 (PST) Received: from flashheart.burtonini.com (35.106.2.81.in-addr.arpa. [81.2.106.35]) by smtp.gmail.com with ESMTPSA id y189sm20936208wmb.13.2019.11.18.08.46.54 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Nov 2019 08:46:54 -0800 (PST) From: Ross Burton To: openembedded-core@lists.openembedded.org Date: Mon, 18 Nov 2019 16:46:44 +0000 Message-Id: <20191118164647.29409-3-ross.burton@intel.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191118164647.29409-1-ross.burton@intel.com> References: <20191118164647.29409-1-ross.burton@intel.com> MIME-Version: 1.0 Subject: [OE-core] [PATCH 3/6] cve-update-db-native: clean up proxy handling X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org urllib handles adding proxy handlers if the proxies are set in the environment, so call bb.utils.export_proxies() to do that and remove the manual setup. Signed-off-by: Ross Burton --- .../recipes-core/meta/cve-update-db-native.bb | 31 +++---------------- 1 file changed, 5 insertions(+), 26 deletions(-) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb index 08b18f064f0..db1d69a28e5 100644 --- a/meta/recipes-core/meta/cve-update-db-native.bb +++ b/meta/recipes-core/meta/cve-update-db-native.bb @@ -21,10 +21,12 @@ python do_populate_cve_db() { """ Update NVD database with json data feed """ - + import bb.utils import sqlite3, urllib, urllib.parse, shutil, gzip from datetime import date + bb.utils.export_proxies(d) + BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-" YEAR_START = 2002 @@ -40,16 +42,6 @@ python do_populate_cve_db() { except OSError: pass - proxy = d.getVar("https_proxy") - if proxy: - # instantiate an opener but do not install it as the global - # opener unless if we're really sure it's applicable for all - # urllib requests - proxy_handler = urllib.request.ProxyHandler({'https': proxy}) - proxy_opener = urllib.request.build_opener(proxy_handler) - else: - proxy_opener = None - cve_f = open(os.path.join(d.getVar("TMPDIR"), 'cve_check'), 'a') if not os.path.isdir(db_dir): @@ -67,15 +59,7 @@ python do_populate_cve_db() { json_url = year_url + ".json.gz" # Retrieve meta last modified date - - response = None - - if proxy_opener: - response = proxy_opener.open(meta_url) - else: - req = urllib.request.Request(meta_url) - response = urllib.request.urlopen(req) - + response = urllib.request.urlopen(meta_url) if response: for l in response.read().decode("utf-8").splitlines(): key, value = l.split(":", 1) @@ -95,12 +79,7 @@ python do_populate_cve_db() { # Update db with current year json file try: - if proxy_opener: - response = proxy_opener.open(json_url) - else: - req = urllib.request.Request(json_url) - response = urllib.request.urlopen(req) - + response = urllib.request.urlopen(json_url) if response: update_db(c, gzip.decompress(response.read()).decode('utf-8')) c.execute("insert or replace into META values (?, ?)", [year, last_modified]) From patchwork Mon Nov 18 16:46:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 179595 Delivered-To: patch@linaro.org Received: by 2002:a92:38d5:0:0:0:0:0 with SMTP id g82csp3127176ilf; Mon, 18 Nov 2019 08:47:15 -0800 (PST) X-Google-Smtp-Source: APXvYqw7kcWQ7X/W5lwT110Hls/bufZaWiPIr9ywP4G1BIX1K7sCu6ciqLfYc4aCmvZtnJygTJRx X-Received: by 2002:a17:90b:4006:: with SMTP id ie6mr490890pjb.50.1574095635439; Mon, 18 Nov 2019 08:47:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1574095635; cv=none; d=google.com; s=arc-20160816; b=SWf5Rpf0bArWCAM30qu0JdHxlL33+Dg+qWL2Kl/DQVDzm5YECYMz/YC11gwVqB8epv t7/PgiJBDVJXKaYzg26nbYVZm8qJELVhehIwOfObcLKa7KGxmuKdZmPrbrkeDUNiLO1u oc95YI61oWiUoBLlPCeJluHl1wt1fHfkqVENbFkFLhvEdKLGaYioKLk+YykGbUaKKU1M 09D/y5xzRiimu4R1FZWSTFDXaHPVRhlv3avVnG84ilmlptVaxLyoy2UJkAA+Eb2hxNn4 RpJkurY0rTg1FkSNuOmer8rMrE9NMSI0fBmQuqxf74iE8/7Lo01Um+9H/ByQkikFhyxA +oXg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :dkim-signature:delivered-to; bh=yRU08dYY9JT+LbfqLvKEETbJxRCp475H/XPpEl072YI=; b=sxqxqO4MUTIw7V2tWryBHy33qHuQwM7Sf70SjhfdC3WUrwarlMLOTM/szuuf/hWpvk w5pbFAa4uu5kUJohZrbTFQMXoZshrhPfr9g3H37nAM2CCL8+zW4BaiXMmbwPYxXrb9rR tP4Z4OOZAJ2icmwCMWXgSEcLN6yLc/dfgjK80gpx4//uud+T9BxIIEkns24r+kcWYt/y 2CSCzV0VFGtjdBd6NKpBGbuOXGBPOnQASdsHVsZ51J7ljuWuW5AqsChqMy4AMMvbaBrI CTRDysspr2f+7YNF3nwIyoXj571zPmV/J14R1kyX/dJLyEjXsn0th9Bcggnq8OhjFY/7 Ft9A== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=vlibkkv2; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id w128si21133989pfw.27.2019.11.18.08.47.15; Mon, 18 Nov 2019 08:47:15 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=vlibkkv2; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 42E027F8BC; Mon, 18 Nov 2019 16:47:00 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-wm1-f67.google.com (mail-wm1-f67.google.com [209.85.128.67]) by mail.openembedded.org (Postfix) with ESMTP id B5EBA7F8CC for ; Mon, 18 Nov 2019 16:46:55 +0000 (UTC) Received: by mail-wm1-f67.google.com with SMTP id b17so19717993wmj.2 for ; Mon, 18 Nov 2019 08:46:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=bXpTy66ZhmzjFqRhzkqbURNmX/UVYlGxO0A5tVJj/8M=; b=vlibkkv2d6MKO7QsI4chhZbn0VBvKUJSNvu/o7t+0xJPAEcwLAL2ApToFdl0zU63dL 9o5i/SKw/abiFjhLQJ0JbCG8pShXAW/d9lQPdCoq9W4Bj5zLuUqXXjMmKM8DuLCMwRsJ 72CMzPSy8X1NlmkE6abIQu/gajmuL17DoB9gbLykaNYpRE2YqApCGHyVOq8dgqQ77zDr RQ0m72cBe8KD50ntsaGq8B/7hicqEZcyx1H3CDLteUon2RsVHpxezBoURjdU0cZz/OBw p3L8tlXGMFqN5Kf2Pvs9cd7gHNkce+YjIIvedDBHBtbsA4MDqBBJBMYn/6+Cv+kSwoj/ JSdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=bXpTy66ZhmzjFqRhzkqbURNmX/UVYlGxO0A5tVJj/8M=; b=fr+r01ihFLun3dVbUgrUSVtPJ6DZBSV0TbbmF+Bc/l2UXU2j9WkO4HgjDbpZjfUFX0 fzbb9wfjxbuL8QicSdwEMmqc2dbd1NSKy/RfTDwqQF5TiibB93C6BQswCQutyVDIVDnD 5vLFIzdUsxd3h4eAUGfXIlsdpkH/7S0u3t+JswXO4pU6BumtX9mwpC/3FD+5xAM6Cu7T SBXwzCHutGlS8uV9BYkcD9goJQnrL58z5bxWiUiwIoX9ajuY7Bsg5Dx1A49WUk3Xngvq RVqz1BLbzC0VsAe+pxZ7sTA0AQ+P9WuS+K+F2RNgqjYPE6z/yyS2GOuHnGeNfJV3ow1Y tbFg== X-Gm-Message-State: APjAAAXmVtTzjbHPw2Dob25pCrWKI+B0GuJdxH6vAOuTZrQoAUut7v5D 5WRSOY58NDgaQLnHFg+hi3roJDOZcXQ= X-Received: by 2002:a7b:c776:: with SMTP id x22mr29090101wmk.144.1574095616145; Mon, 18 Nov 2019 08:46:56 -0800 (PST) Received: from flashheart.burtonini.com (35.106.2.81.in-addr.arpa. [81.2.106.35]) by smtp.gmail.com with ESMTPSA id y189sm20936208wmb.13.2019.11.18.08.46.55 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Nov 2019 08:46:55 -0800 (PST) From: Ross Burton To: openembedded-core@lists.openembedded.org Date: Mon, 18 Nov 2019 16:46:45 +0000 Message-Id: <20191118164647.29409-4-ross.burton@intel.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191118164647.29409-1-ross.burton@intel.com> References: <20191118164647.29409-1-ross.burton@intel.com> MIME-Version: 1.0 Subject: [OE-core] [PATCH 4/6] cve-check: rewrite look to fix false negatives X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org A previous optimisation was premature and resulted in false-negatives in the report. Rewrite the checking algorithm to first get the list of potential CVEs by vendor:product, then iterate through every matching CPE for that CVE to determine if the bounds match or not. By doing this in two stages we can know if we've checked every CPE, instead of accidentally breaking out of the scan too early. Signed-off-by: Ross Burton --- meta/classes/cve-check.bbclass | 63 ++++++++++++++++++---------------- 1 file changed, 34 insertions(+), 29 deletions(-) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass index 3326944d791..c1cbdbde7b7 100644 --- a/meta/classes/cve-check.bbclass +++ b/meta/classes/cve-check.bbclass @@ -165,7 +165,6 @@ def check_cves(d, patched_cves): """ Connect to the NVD database and find unpatched cves. """ - import ast, csv, tempfile, subprocess, io from distutils.version import LooseVersion cves_unpatched = [] @@ -187,68 +186,74 @@ def check_cves(d, patched_cves): cve_whitelist = d.getVar("CVE_CHECK_WHITELIST").split() import sqlite3 - db_file = d.getVar("CVE_CHECK_DB_FILE") - conn = sqlite3.connect(db_file) + db_file = d.expand("file:${CVE_CHECK_DB_FILE}?mode=ro") + conn = sqlite3.connect(db_file, uri=True) + # For each of the known product names (e.g. curl has CPEs using curl and libcurl)... for product in products: - c = conn.cursor() if ":" in product: vendor, product = product.split(":", 1) - c.execute("SELECT * FROM PRODUCTS WHERE PRODUCT IS ? AND VENDOR IS ?", (product, vendor)) else: - c.execute("SELECT * FROM PRODUCTS WHERE PRODUCT IS ?", (product,)) + vendor = "%" - for row in c: - cve = row[0] - version_start = row[3] - operator_start = row[4] - version_end = row[5] - operator_end = row[6] + # Find all relevant CVE IDs. + for cverow in conn.execute("SELECT DISTINCT ID FROM PRODUCTS WHERE PRODUCT IS ? AND VENDOR LIKE ?", (product, vendor)): + cve = cverow[0] if cve in cve_whitelist: bb.note("%s-%s has been whitelisted for %s" % (product, pv, cve)) # TODO: this should be in the report as 'whitelisted' patched_cves.add(cve) + continue elif cve in patched_cves: bb.note("%s has been patched" % (cve)) - else: - to_append = False + continue + + vulnerable = False + for row in conn.execute("SELECT * FROM PRODUCTS WHERE ID IS ? AND PRODUCT IS ? AND VENDOR LIKE ?", (cve, product, vendor)): + (_, _, _, version_start, operator_start, version_end, operator_end) = row + #bb.debug(2, "Evaluating row " + str(row)) + if (operator_start == '=' and pv == version_start): - to_append = True + vulnerable = True else: if operator_start: try: - to_append_start = (operator_start == '>=' and LooseVersion(pv) >= LooseVersion(version_start)) - to_append_start |= (operator_start == '>' and LooseVersion(pv) > LooseVersion(version_start)) + vulnerable_start = (operator_start == '>=' and LooseVersion(pv) >= LooseVersion(version_start)) + vulnerable_start |= (operator_start == '>' and LooseVersion(pv) > LooseVersion(version_start)) except: bb.warn("%s: Failed to compare %s %s %s for %s" % (product, pv, operator_start, version_start, cve)) - to_append_start = False + vulnerable_start = False else: - to_append_start = False + vulnerable_start = False if operator_end: try: - to_append_end = (operator_end == '<=' and LooseVersion(pv) <= LooseVersion(version_end)) - to_append_end |= (operator_end == '<' and LooseVersion(pv) < LooseVersion(version_end)) + vulnerable_end = (operator_end == '<=' and LooseVersion(pv) <= LooseVersion(version_end)) + vulnerable_end |= (operator_end == '<' and LooseVersion(pv) < LooseVersion(version_end)) except: bb.warn("%s: Failed to compare %s %s %s for %s" % (product, pv, operator_end, version_end, cve)) - to_append_end = False + vulnerable_end = False else: - to_append_end = False + vulnerable_end = False if operator_start and operator_end: - to_append = to_append_start and to_append_end + vulnerable = vulnerable_start and vulnerable_end else: - to_append = to_append_start or to_append_end + vulnerable = vulnerable_start or vulnerable_end - if to_append: + if vulnerable: bb.note("%s-%s is vulnerable to %s" % (product, pv, cve)) cves_unpatched.append(cve) - else: - bb.note("%s-%s is not vulnerable to %s" % (product, pv, cve)) - patched_cves.add(cve) + break + + if not vulnerable: + bb.note("%s-%s is not vulnerable to %s" % (product, pv, cve)) + # TODO: not patched but not vulnerable + patched_cves.add(cve) + conn.close() return (list(patched_cves), cves_unpatched) From patchwork Mon Nov 18 16:46:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 179596 Delivered-To: patch@linaro.org Received: by 2002:a92:38d5:0:0:0:0:0 with SMTP id g82csp3127311ilf; Mon, 18 Nov 2019 08:47:23 -0800 (PST) X-Google-Smtp-Source: APXvYqzSeWbYryOXDaj8rVOIz9s7FFkouYaZtdmMx4lnJK1vYbM1EPkb38R1p4dYtI2rfQzne7iL X-Received: by 2002:a17:902:aa42:: with SMTP id c2mr29835581plr.311.1574095642940; Mon, 18 Nov 2019 08:47:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1574095642; cv=none; d=google.com; s=arc-20160816; b=lnDkGK1mKruPuAXrqwBWoSwYOFZBxMjSO7GcMwRObbE7fJQP5RdXBcJNDPEF0qNgF+ oHr9zlE/jtwwdAyp7kU35NuFBmi9ykOL3Ffq4rdJoXGDuu7n84OfVNYNnYfHMI369/db VGBqPe20gMVyR90T+KkG5sDmVUsuTXOMY/AP2mxUf0SEvL36kTTGBlFV7v45KIqTogFU HMVJKtLhypCj+sUERc2WmJ3mxJLaQ0d78heTJbiUB/IrSgMUWx2AGtcZN7hB4TS47y5O Tuyi49gKtX+XrGfKBpHXE0Tu89HF5qpqKIhvPbK1NvYezgEEQ3da+g+nREuppDrhK9/G tpBA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :dkim-signature:delivered-to; bh=bp+3Ac6sxDLVMoZWnzoYsken88crdJb/V2puGnIrgNI=; b=Tx0XPAYCn0Yzk2iHUy7TOcYBsaGdBRMXwm788jog561nsGRtbSKhsaawrzJ2sG04Wl n1n7U3Y0mHMD6ohWGB56syp07NyMf8YSpRO+o5kO5iAsZSFHd5zqnuVcyGsOgaDD4HTq esq79SH5FM8zQp2zzhE8fVfJGO2JIJ1asVO9tVWrV6sJaI09Zy6IPQ25kxzDudY2UEdI WBEGby3+U0uPVZSUAbU0ZIbMyGvqWatEkoVteq0s9q7FuSK9P4+r6RREUz8WZ5Q8Vdih pNwYovYg5akr8iInQW1uf33L6vEXlQBXprBVlvFCwaf9Hx5x1KWPHQYEbmlfpiFNDUwa 8H+w== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=qI8UY7Xt; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id e4si22760430pfm.64.2019.11.18.08.47.22; Mon, 18 Nov 2019 08:47:22 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=qI8UY7Xt; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 110D27F8CC; Mon, 18 Nov 2019 16:47:05 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-wm1-f65.google.com (mail-wm1-f65.google.com [209.85.128.65]) by mail.openembedded.org (Postfix) with ESMTP id 477D67F8A6 for ; Mon, 18 Nov 2019 16:46:57 +0000 (UTC) Received: by mail-wm1-f65.google.com with SMTP id c22so19731500wmd.1 for ; Mon, 18 Nov 2019 08:46:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=6haDPYCzHqezaXGvQyoUaSEXVQuVUHGkZo5mjHvOvpQ=; b=qI8UY7XtMYI6NJSNP1z9U4wtLwyXHrpGOSmsnYuV/o07K6Lui+TIwbiQSIYxo/KoHe qqTp6B71LFaXRGse7IY4i2ng/okl9NwBXv2ShFtlYgn2YtZRM+ufquULI6T/PpIFE7kw XTcqStY27oQFGT8ejhQarq4GKsWc5V1dQMdBMTw1EVCImlA6BG7whyQCz2d/lsIrcQOS ZWyed4pyYcttSHwaWslSfUDdTRd6BrawrPnxyGhqm/H+5mr8FNjvdfw0TJO7s1ctf12A zFwPs8bnbNMRpIz/6u9KhAxbS11InG0+xn35aBb9qIgqT1t9ZQ2t3t6Pju//olme8Udj JQzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6haDPYCzHqezaXGvQyoUaSEXVQuVUHGkZo5mjHvOvpQ=; b=eDJpTO0nmnNv2iMPK7cLOZP6dpJOwvXGrtZUm18trLG7Em2GX+gyeHY3yZtQLQI3ON Tyw5ncEsvDEmlPkDP8jyvlymw/4qMSzpzFcOgPd2J164r5xJiwiNy+FrRHcIy8nEKKt9 ClmXNBixg+i7HcwxF57b/qBXOiqbuOsza0dq6U0JJLr2z6Tbbr/3J/qhS8WJLqDlAaZg 6NKatO8mRePYCTUTPtJBGVS8zqX0wyvdfFflSFUxfDRBvy2kMTyRCN5aqHAyh2ESgMUh vyUfUPILTHkCYw6C+zP2SNJ5SJ03OmZcOVQMZcXxeAr3c1kJx+WiqnY6eGo2U6trJGMv WI4w== X-Gm-Message-State: APjAAAX+IlWoaOAZer1QWfA9gKwEhp42urt8cVHUYXwPF1MTXjgEQJkY jYr70OnBUErruawuZI+shb0zLH9o6g0= X-Received: by 2002:a1c:2d17:: with SMTP id t23mr28990514wmt.59.1574095617691; Mon, 18 Nov 2019 08:46:57 -0800 (PST) Received: from flashheart.burtonini.com (35.106.2.81.in-addr.arpa. [81.2.106.35]) by smtp.gmail.com with ESMTPSA id y189sm20936208wmb.13.2019.11.18.08.46.56 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Nov 2019 08:46:56 -0800 (PST) From: Ross Burton To: openembedded-core@lists.openembedded.org Date: Mon, 18 Nov 2019 16:46:46 +0000 Message-Id: <20191118164647.29409-5-ross.burton@intel.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191118164647.29409-1-ross.burton@intel.com> References: <20191118164647.29409-1-ross.burton@intel.com> MIME-Version: 1.0 Subject: [OE-core] [PATCH 5/6] cve-check: neaten get_cve_info X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org Remove obsolete Python 2 code, and use convenience methods for neatness. Signed-off-by: Ross Burton --- meta/classes/cve-check.bbclass | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass index c1cbdbde7b7..e95716d9ded 100644 --- a/meta/classes/cve-check.bbclass +++ b/meta/classes/cve-check.bbclass @@ -261,23 +261,15 @@ def check_cves(d, patched_cves): def get_cve_info(d, cves): """ Get CVE information from the database. - - Unfortunately the only way to get CVE info is set the output to - html (hard to parse) or query directly the database. """ - try: - import sqlite3 - except ImportError: - from pysqlite2 import dbapi2 as sqlite3 + import sqlite3 cve_data = {} - db_file = d.getVar("CVE_CHECK_DB_FILE") - placeholder = ",".join("?" * len(cves)) - query = "SELECT * FROM NVD WHERE id IN (%s)" % placeholder - conn = sqlite3.connect(db_file) - cur = conn.cursor() - for row in cur.execute(query, tuple(cves)): + conn = sqlite3.connect(d.getVar("CVE_CHECK_DB_FILE")) + placeholders = ",".join("?" * len(cves)) + query = "SELECT * FROM NVD WHERE id IN (%s)" % placeholders + for row in conn.execute(query, tuple(cves)): cve_data[row[0]] = {} cve_data[row[0]]["summary"] = row[1] cve_data[row[0]]["scorev2"] = row[2] From patchwork Mon Nov 18 16:46:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 179597 Delivered-To: patch@linaro.org Received: by 2002:a92:38d5:0:0:0:0:0 with SMTP id g82csp3127481ilf; Mon, 18 Nov 2019 08:47:30 -0800 (PST) X-Google-Smtp-Source: APXvYqwQsAZtfwkoGKxHkayLBtGUpuzgtXQT3Bl3TwimQDMJfFeEVVC9DkKkz8NwBw4HHLm23NlJ X-Received: by 2002:a17:90a:bb94:: with SMTP id v20mr497432pjr.62.1574095650145; Mon, 18 Nov 2019 08:47:30 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1574095650; cv=none; d=google.com; s=arc-20160816; b=Uz7Sw2flwl98pSeSZuH7A7CS88ySNwtigAfmr6eAisV43/UbuX0o0Ac0m4vc/wJjqG k3s1Vmp31PTLTwIMD2v45H9P1Hs0bVUW4kVJJyR7h7thwfmHedTk+rZeTejriMcVYcvv tbtPxJvgzpTAlxWm0Hl9/p11zOS6PsXbHcCFPG5kj0eqju0gz/3ICBFPr94E965l3qeo 57wb72tgvr/UYOSW/z4/hIGRvzLJhR/ZeJeH+xjQbsTQVpokIVrQMbbRWkKh2pgHfZQM Mb+38rN5KSxO1zYB1Jy6LdFnJV9QvoqbIDC+ZxTAVCMGdDDmE4FQc8nrbD6RdDylmmZm tm5g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :dkim-signature:delivered-to; bh=6+Qu2nL8hIm5rlmjkg8CQ5Cnppbl7BIwD3Pm1AJOxNk=; b=vL5t+lqDf+AJXJ6MYgmhvONGuw05qao7ghe5mt9djBujEnV8LM6W0h5OjUWn+xi+qU VZR5ekvzpqdaCO1CeDdFdX/7E52DXeiEtQ71cSmk/ikpClN4b3EMGD86iUs+gOl+GRAu KApnwCNDhp+Z0eF3VlZeXO0YzQzsQzlvjhTiHDSO0sZAtG36pd0C1aIs0wVzO+O3APk9 e/tELKMAeOCRSHhdHYXHHFXBzaB5KFgnX3WwJq9bgwXWCqTQGfo1XnQqBTwb+za/Tpdq SywM2nlcx5ojGbkigMCyDi+NHlUoX2+MmvCsI0QpKm0GlMY2gl3O97j156PqSpIrLGPb YPbw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b="ZTpCgP/d"; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id bc8si19127352plb.88.2019.11.18.08.47.29; Mon, 18 Nov 2019 08:47:30 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b="ZTpCgP/d"; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 347097F96B; Mon, 18 Nov 2019 16:47:06 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-wm1-f67.google.com (mail-wm1-f67.google.com [209.85.128.67]) by mail.openembedded.org (Postfix) with ESMTP id 668D77F943 for ; Mon, 18 Nov 2019 16:46:58 +0000 (UTC) Received: by mail-wm1-f67.google.com with SMTP id u18so18185809wmc.3 for ; Mon, 18 Nov 2019 08:46:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=mf45DdTgWRs5x5BDGPO5ZG82r/eQAhxp1MbZLLqNFhQ=; b=ZTpCgP/d6YCPTAqSVX/4TEoW4bG5zuz855H/TTNbAPkNrEqzXBrcf7w+YHfZ751GOb kbsxnr3Aa8013p5kWCPP4NLzgtt3L7POYvyxyObPKLQN2Wb+ffMv8VP0RGsyTF7x9SFS /my/Z8AJ9okF6GpkD+RycXsL2nFyuuCHaY1kWovbCrnM6gFalf6HPf+IMhxeCEWOrP4W 16zaatV576Jg2f0jnlUrIG116U7JeTaLqVG8nVE4NA7YDpA5ydmJHttPANhxvDsM1Kk/ kx+k2n9vdfEWMUa06O2kxbxkbcSTOcjqfVQSw5L+QhAfbE4Zf4ZAwu/slck1+pxz5kEp jFOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mf45DdTgWRs5x5BDGPO5ZG82r/eQAhxp1MbZLLqNFhQ=; b=tCnkZ2VSzUdk3Hmrzi8cBE0U17CKv1haZ2udZcJ70yShK4rectYKxr0O7bYV3nWQnf 9MGIQgaJFiwzL+xR6/MapBt+FmVCnIt44ozt4FKRKDbvdrsMS8yuErFZ3d5SvFtnzFMW /1XkEzPxT6Q1wRrLoQGIPTvLEQNBp8vSTrI8AMs6U1shpvnaWaIW/B1yM9X3WXXWc5gn y+sjM7ARokMJW3AExth+QpCMwNDXJWPJl5UO5+cegHUhVkm//dHgfgQxrOPdBGHnsTDz IgkzqvOvsXWX0UDI3Xiezv4XBS8UWnLqrsR81rlzPo/D9S+0Fhq5Jcx1e+4VoP7rMJmb xTbg== X-Gm-Message-State: APjAAAWCB3DRx/D5QzaTuynRtiZ3XkNOBGJfiSgHA0fX8OMb/KfbNQ6m 2Y9ae8M+DWh78ZgLAoq8soqE47jtbj4= X-Received: by 2002:a1c:650b:: with SMTP id z11mr30156006wmb.149.1574095618686; Mon, 18 Nov 2019 08:46:58 -0800 (PST) Received: from flashheart.burtonini.com (35.106.2.81.in-addr.arpa. [81.2.106.35]) by smtp.gmail.com with ESMTPSA id y189sm20936208wmb.13.2019.11.18.08.46.57 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Nov 2019 08:46:58 -0800 (PST) From: Ross Burton To: openembedded-core@lists.openembedded.org Date: Mon, 18 Nov 2019 16:46:47 +0000 Message-Id: <20191118164647.29409-6-ross.burton@intel.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191118164647.29409-1-ross.burton@intel.com> References: <20191118164647.29409-1-ross.burton@intel.com> MIME-Version: 1.0 Subject: [OE-core] [PATCH 6/6] cve-check: fetch CVE data once at a time instead of in a single call X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org This code used to construct a single SQL statement that fetched the NVD data for every CVE requested. For recipes such as the kernel where there are over 2000 CVEs to report this can hit the variable count limit and the query fails with "sqlite3.OperationalError: too many SQL variables". The default limit is 999 variables, but some distributions such as Debian set the default to 250000. As the NVD table has an index on the ID column, whilst requesting the data CVE-by-CVE is five times slower when working with 2000 CVEs the absolute time different is insignificant: 0.05s verses 0.01s on my machine. Signed-off-by: Ross Burton --- meta/classes/cve-check.bbclass | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass index e95716d9ded..19ed5548b3a 100644 --- a/meta/classes/cve-check.bbclass +++ b/meta/classes/cve-check.bbclass @@ -267,17 +267,17 @@ def get_cve_info(d, cves): cve_data = {} conn = sqlite3.connect(d.getVar("CVE_CHECK_DB_FILE")) - placeholders = ",".join("?" * len(cves)) - query = "SELECT * FROM NVD WHERE id IN (%s)" % placeholders - for row in conn.execute(query, tuple(cves)): - cve_data[row[0]] = {} - cve_data[row[0]]["summary"] = row[1] - cve_data[row[0]]["scorev2"] = row[2] - cve_data[row[0]]["scorev3"] = row[3] - cve_data[row[0]]["modified"] = row[4] - cve_data[row[0]]["vector"] = row[5] - conn.close() + for cve in cves: + for row in conn.execute("SELECT * FROM NVD WHERE ID IS ?", (cve,)): + cve_data[row[0]] = {} + cve_data[row[0]]["summary"] = row[1] + cve_data[row[0]]["scorev2"] = row[2] + cve_data[row[0]]["scorev3"] = row[3] + cve_data[row[0]]["modified"] = row[4] + cve_data[row[0]]["vector"] = row[5] + + conn.close() return cve_data def cve_write_data(d, patched, unpatched, cve_data):