From patchwork Mon Nov 11 22:13:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 179155 Delivered-To: patch@linaro.org Received: by 2002:a92:38d5:0:0:0:0:0 with SMTP id g82csp7263210ilf; Mon, 11 Nov 2019 14:13:54 -0800 (PST) X-Google-Smtp-Source: APXvYqynVGzh5XCvVEAYh8/rasTf2TCDF0QleTltEh+yawvipIDc6mqB5gKtcBoBL+5VanF+XLUu X-Received: by 2002:a62:8490:: with SMTP id k138mr22881738pfd.186.1573510434317; Mon, 11 Nov 2019 14:13:54 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1573510434; cv=none; d=google.com; s=arc-20160816; b=MlnA9Jiy4RefhYtX6yzuGzu89MZUR2fWoi6xGSfvL8hkBEhYkBCiIMn+pGhoEttDKT GsfHh7q7fq634Itq9noMrbLcabyi9/gCAIS7JW+VRuQ5pe4Ve45mW1I4YcanvC5EY91s N7eKY1u50OMvdnT1kWrxdUUpEgIMaXhJ1IQQu/2+m6ZP2bPGroxA+vofH1rZFP8v+R4R sZZhuosGR4K2QDpwTS0Ojz/xK6n0mjSo0dSNjnzQbfeZGFhb30lPNlhQAwFY/3uIRXat pkVonHa5YNXnmwKFn3sOPxk6Y3tvjYwg37KqCITXm6vbr2VrNhXOFpNfzB5UJv2OYwCh n2wQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:message-id:date:to:from:dkim-signature:delivered-to; bh=z2wTL46904tAm0Pqx1kKNWZdaS8FvssyWyrwpM2LYZA=; b=dTIH8qOa34GF1/+qAwIoiR9i0sMSc9qI8xb8D+OTgRMhq6HIhmdkQSo6QYUH88touY wWcOvnfy35/tCg2UeTSQNv3Yo7LIpnNDU0LtFhBgCsrp/tm80hIyuOG6VEz+PHY1j0Ih Rd9pPLUC20qmC4VxSI6LorZX53l20i9BCYF6zX88t3IBb0Os0OLh0a+Q0U1j77hbllkH eOD12nIm6rEP2X4BruBaUI1umTRwuYvfO+NPvvlAkDZTPJK3Q6XzOb3YfXspCGhYIpZS u0YmIDV05LE0CF/nlvas6ZPiz9uK1khxBdLPNB2r9vRkpSYMsHhXihgrHOnPVGsyjbqO pICg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=YMTFPrut; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id g23si19373854pgm.405.2019.11.11.14.13.53; Mon, 11 Nov 2019 14:13:54 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=YMTFPrut; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 403387FA88; Mon, 11 Nov 2019 22:13:45 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-wm1-f66.google.com (mail-wm1-f66.google.com [209.85.128.66]) by mail.openembedded.org (Postfix) with ESMTP id A78ED7FA88 for ; Mon, 11 Nov 2019 22:13:40 +0000 (UTC) Received: by mail-wm1-f66.google.com with SMTP id b11so951849wmb.5 for ; Mon, 11 Nov 2019 14:13:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=rWeYatW3qrAK2McH4FAaYl6YFFDJukQ8lNRElosSY2s=; b=YMTFPrutYrRq7OGhmP4i6iuA8hXSVtPaVMhjQLshi9MlHf+tSjOx6nIrjRcfG6oV+Z JYwo3FkqzcF2Bea6fLg/ME3HHDBExICg2xS/qFN1uyZbKrgxldjzrNsgvB/xVqOK4r9+ 9wqbC0dr4Z0hfWplaUXjoSQxaDZQa8ORPxi8dwY+6GT70xnJH1QCXlTof/MILwkCF7cn 995CSltR1xKkqr5MJ/XhDEs6J+QoIKmqqONAYxrCqAx/tGzRjM4nSUlVBBSipIXB4buh xvRnkvylrb5M8I6tsdEL+diQ/e7o69aNezhSO4skQTqENxmzKyO+aPlPQV7cw2YrCatl ehMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=rWeYatW3qrAK2McH4FAaYl6YFFDJukQ8lNRElosSY2s=; b=nkxHd6oFjSroPsWx7EOJRhsXK7/uqNUUzR3v7Z5YbsgjsA6iqV6mjDVviLZAUMfa/u +UfHHNWBHxfcNjE6yq/Hk9wFRXjAwB7a6VNO4eBydSYBIsJd1lRtxJs4DNlNk7Fclfir wAZDlUQmzBqwmkrezX6aFkaU5iSqFjpFJ+T68IBTxF3npdBvargErmoO+0SPnZ18BotX GptaM1WMAEQrpJyqGjgEznl2+e9kikh2/b8xdS22Q+YniZlr9SGvYciaIN574PonGcrS NLJjA3AJKsdCKABwOGiTgsNRKmBNzNcw61bVCCPAOdPBfDKrtJl26IMvylvgDgN647bj tCig== X-Gm-Message-State: APjAAAVWkUOLBGzxJN6bQwBjFe654pUdkI9lc9Da2mEZpLNxCT2m/fD3 5jtzSPBHoUL/uqN2vRV2ErDMJB6Snio= X-Received: by 2002:a1c:152:: with SMTP id 79mr988658wmb.70.1573510421037; Mon, 11 Nov 2019 14:13:41 -0800 (PST) Received: from flashheart.burtonini.com (35.106.2.81.in-addr.arpa. [81.2.106.35]) by smtp.gmail.com with ESMTPSA id d11sm24900083wrf.80.2019.11.11.14.13.39 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Nov 2019 14:13:39 -0800 (PST) From: Ross Burton To: openembedded-core@lists.openembedded.org Date: Mon, 11 Nov 2019 22:13:34 +0000 Message-Id: <20191111221337.20332-1-ross.burton@intel.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Subject: [OE-core] [PATCH 1/4] cve-update-db-native: don't hardcode the database name X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org Don't hardcode the database filename, there's a variable for this in cve-check.bbclass. Signed-off-by: Ross Burton --- meta/recipes-core/meta/cve-update-db-native.bb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb index 19875a49b1c..c15534de08b 100644 --- a/meta/recipes-core/meta/cve-update-db-native.bb +++ b/meta/recipes-core/meta/cve-update-db-native.bb @@ -28,8 +28,8 @@ python do_populate_cve_db() { BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-" YEAR_START = 2002 - db_dir = os.path.join(d.getVar("DL_DIR"), 'CVE_CHECK') - db_file = os.path.join(db_dir, 'nvdcve_1.0.db') + db_file = d.getVar("CVE_CHECK_DB_FILE") + db_dir = os.path.dirname(db_file) json_tmpfile = os.path.join(db_dir, 'nvd.json.gz') # Don't refresh the database more than once an hour From patchwork Mon Nov 11 22:13:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 179157 Delivered-To: patch@linaro.org Received: by 2002:a92:38d5:0:0:0:0:0 with SMTP id g82csp7264008ilf; Mon, 11 Nov 2019 14:14:40 -0800 (PST) X-Google-Smtp-Source: APXvYqzjy5IORjVzRDa02RbX0wDaBlYHzo0DhS+Gie3hC9/08UfBOn3Q5FaL9JlJe4fkEZZGEm+/ X-Received: by 2002:a63:ec42:: with SMTP id r2mr13716021pgj.162.1573510480466; Mon, 11 Nov 2019 14:14:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1573510480; cv=none; d=google.com; s=arc-20160816; b=x2inx5fC46bX9TzfcYO2QMtEK5q1DvRDSzI064xsuyHeTjG265rBBlE3WyY2cRYI2U NXiQucMQEovZ7X3GoYPaef8HR3dxwranjxrrmqjBviipi4JcQITuqmzc9C9kivYxo94V upaVHVJ5gRzQ7R/ZEHjzTUKiiQFQ1RraltCIkqvCvdAUbJ7VkVw8EquZf9X6aaYV278x 7/yQB5V7vFbUCJI7pm3okz35wwFGCIEL4fRx8la8WBn543kPZ4zRdcTfJdrjNLUg5bkX 5cB5OuIlbLb5S3+24/s5TJRFPtPbKF814kfbE6znabYPxeZjq+DZf60BxWsFEaukyqx8 8Z1A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :dkim-signature:delivered-to; bh=0JeXNvaxsH4lEEd0w8gx9VtmlrweWozjF0hKkwx/z30=; b=xegoTMqZN23iv4VOX7R+UTPgYaipNqNwWdVg77vzeWYpaIrmKx3sO5rQajK+Koqpz9 4tjmYMZnXXT/icIFaqTlNT6utTEUOCkFc0EKG03rW3lVsD2E/i7EXLrnuusJLnH9Y80V JMt4n6lNw0OqT9sl52FoRPVHwMeYNn3dA57garPq59IyBDrm3Ujd4cXn73FGICFASrIr ENyLREQ2KSJ68fAvNvRBGPaKj1XEGUKQ1+bn3fVuqUS4WDEClnvxeDexqWLOQ7dlzFcc M0WSodluaqAmY6/v7mJPnZen0GNrYoDX2FHttaKMzDvslEdpwc9ykI6aH4xlC6shx8QS zkIA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=Cdfiuow2; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id y5si17661985plp.141.2019.11.11.14.14.40; Mon, 11 Nov 2019 14:14:40 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=Cdfiuow2; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id A4ABC7FBB5; Mon, 11 Nov 2019 22:14:32 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-wm1-f49.google.com (mail-wm1-f49.google.com [209.85.128.49]) by mail.openembedded.org (Postfix) with ESMTP id 5B4FE7FA88 for ; Mon, 11 Nov 2019 22:13:41 +0000 (UTC) Received: by mail-wm1-f49.google.com with SMTP id l1so976576wme.2 for ; Mon, 11 Nov 2019 14:13:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=M4ctuwA6cZs27fmHe6BG8al2Ng7rxmll+fGGZm4Bf/M=; b=Cdfiuow2q8I5EZQgaeuPn0sZAousQ1v8k4QEZp6twbVbZEJqH7+XSoD0m9WvabXL0Q jEMnyo3e183I4oDVBuvWNaLGXoe8pb4XUzqK+htjJI5FCrWli5dw890lzMPOiRm4mqAP gV4X3DYsO2JV9/iFWDITHrCA+AIcOWCxo24TRuDwIOu5QLwkQZxijOXJIjglTbzkRrcL Bw/pNeSKAdcu7b5ybrVSmR6k+oYlbTEqDOD8i1KYWbFa5In2TwncvDFAGfNgzti2qZuW yJAnlZMJtFl2CQsip0ZRIsaVJnctcrTrluyip/QL7d7qwNH6QNSJPwuKQFp0EDR7QpJf CdlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=M4ctuwA6cZs27fmHe6BG8al2Ng7rxmll+fGGZm4Bf/M=; b=QDHLFt1CPQYa6goksYjuYQXWZFCBVrXfDFMzpyrhhSJMSAxh9OqT5ow6hE3Bh3dkxp PvmNS8LIxxqcO0MKprMNdD6mp9BsDF6rUFbOFaHMUAGSnXlgygEFCWQc4COfGqPO4yV5 wlatJD8Ty+uU9GrMbe4f1x/8fdVVhG56N6L8mzZld8KSKkjpJsqyly465hOanekYwEgF LHejzeXvOF0Uqwtm/A1emvi83E7+iSfu4XVci6AEGM9q9poGANzeKah5bD/HeMwPW4LI BcTKC92ophBzRM5idJxAoyhHhMBa+UUkCfoupqmfx3pVAv9JulxOxIg6yespN2/yAwCc SvZQ== X-Gm-Message-State: APjAAAX9abqXoZEUZbQxlLpOXdhSHfuMPE/6E9y3Qy8LqdbtU4xk88aw lK1KOPpTNnq7tqEXIqpr6XMXMGRcG+E= X-Received: by 2002:a1c:3843:: with SMTP id f64mr918722wma.129.1573510421797; Mon, 11 Nov 2019 14:13:41 -0800 (PST) Received: from flashheart.burtonini.com (35.106.2.81.in-addr.arpa. [81.2.106.35]) by smtp.gmail.com with ESMTPSA id d11sm24900083wrf.80.2019.11.11.14.13.41 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Nov 2019 14:13:41 -0800 (PST) From: Ross Burton To: openembedded-core@lists.openembedded.org Date: Mon, 11 Nov 2019 22:13:35 +0000 Message-Id: <20191111221337.20332-2-ross.burton@intel.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191111221337.20332-1-ross.burton@intel.com> References: <20191111221337.20332-1-ross.burton@intel.com> MIME-Version: 1.0 Subject: [OE-core] [PATCH 2/4] cve-update-db-native: add an index on the CVE ID column X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org Create an index on the PRODUCTS table which contains a row for each CPE, drastically increasing the performance of lookups for a specific CVE. Signed-off-by: Ross Burton --- meta/recipes-core/meta/cve-update-db-native.bb | 3 +++ 1 file changed, 3 insertions(+) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb index c15534de08b..08b18f064f0 100644 --- a/meta/recipes-core/meta/cve-update-db-native.bb +++ b/meta/recipes-core/meta/cve-update-db-native.bb @@ -120,11 +120,14 @@ python do_populate_cve_db() { def initialize_db(c): c.execute("CREATE TABLE IF NOT EXISTS META (YEAR INTEGER UNIQUE, DATE TEXT)") + c.execute("CREATE TABLE IF NOT EXISTS NVD (ID TEXT UNIQUE, SUMMARY TEXT, \ SCOREV2 TEXT, SCOREV3 TEXT, MODIFIED INTEGER, VECTOR TEXT)") + c.execute("CREATE TABLE IF NOT EXISTS PRODUCTS (ID TEXT, \ VENDOR TEXT, PRODUCT TEXT, VERSION_START TEXT, OPERATOR_START TEXT, \ VERSION_END TEXT, OPERATOR_END TEXT)") + c.execute("CREATE INDEX IF NOT EXISTS PRODUCT_ID_IDX on PRODUCTS(ID);") def parse_node_and_insert(c, node, cveId): # Parse children node if needed From patchwork Mon Nov 11 22:13:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 179156 Delivered-To: patch@linaro.org Received: by 2002:a92:38d5:0:0:0:0:0 with SMTP id g82csp7263891ilf; Mon, 11 Nov 2019 14:14:34 -0800 (PST) X-Google-Smtp-Source: APXvYqwFimxaPJVl3Xrak6+uulL9P5fmV2nLLbKg8nrdbhh84CCHXMIkOPh2cxp1pCrqUdIt2/Fz X-Received: by 2002:a63:5966:: with SMTP id j38mr31083234pgm.304.1573510474110; Mon, 11 Nov 2019 14:14:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1573510474; cv=none; d=google.com; s=arc-20160816; b=lptWn6n5mnPI9lf9DvFwotvkIGLJ1nLYI9qfn3h3uFRb3F8blqda2pvN6D/kQQZhhM Q2lv2TC4va5qA9Ahho2K2ZQbJYOhDZRBdJx8NFz4OPAibaM0dlRnvt5XyJsrl191MO4I o9gRbpAN7AyqY12pK+L50PhfH34swH5DuaK6l/yeb35uNlBWGxFJ+vXlOxcV/mzEAWMf 2gHpe6Bq+abOkHb3Mx5J8KXZ/8nDi4EBTAknBpS/3gg6n2LI/Uk873arsN57b5dDXsT+ 6ZZzrWAkT0306xPEZV2inEaC8wt1xsJ8AuKrX/5dmjRcsgFNBlag98D390hy6Bj4+7tE kU7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :dkim-signature:delivered-to; bh=QgONmI5M0f6ZfLAa8/Mi60U3ELSHg8eXbDkFjXRNtLw=; b=C1qgWmDfU+nqKaPKPj2BPrdbjHCgUQZ1bEWX3LQlXe2Lyd6ZRKwgQRut308tcUHuhK DdlTYP5x69dRdKDxvJr9BOQ4YLE9TdfjO+IWkV9tsXHGSwA5PHZbLMqU70zmcYKnTkVL 3MXG+3zQ/hz3xQC3zmdzLjF+ZLlSxXYlva4QMNN42E11nIs6EUylP3xhoemuwv/cTi2/ OpoyQyW5c6T8KBSYNqLyNme+GYH+CL/4acwEvRL96wsBIJQFP9ygIdKht/cHLfJdLbI2 enB/bjn0Wz8q67ndWRlebGEIVndpSJnsCwLWQwT/knMNTA4rR8gNp3TFDBFmRdZi3o+v Dhsw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=P1QmOfbq; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id jx8si836785pjb.53.2019.11.11.14.14.33; Mon, 11 Nov 2019 14:14:34 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=P1QmOfbq; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 4C5F87FAAA; Mon, 11 Nov 2019 22:14:31 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-wr1-f67.google.com (mail-wr1-f67.google.com [209.85.221.67]) by mail.openembedded.org (Postfix) with ESMTP id 5FE277FAAA for ; Mon, 11 Nov 2019 22:13:42 +0000 (UTC) Received: by mail-wr1-f67.google.com with SMTP id t1so16369919wrv.4 for ; Mon, 11 Nov 2019 14:13:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=BvLw+Q5ig6tUnaYjdSmgh3xKOEyw45Rn18F3gm7mWBk=; b=P1QmOfbqqCwiu9CuYXVohJXXuydBw3FCN0F3GutpRENAHyUC7gr8OPIM6PsZ6HM/xz aAK7TYKXM1naoOMs+ch5eD7hVHZjVC5Jg+yP4vGPxEsaDD4j7tUkQzM1zMDvz/lNCd3j rEtzvmkgjFAYrLHQBUGtIYzqwZwDksLfbVoa2C7sUwsxoNl5EB+Fy4ufJy8lUR/noZJo ujZuMb+I5fQT+DuNvS2wGK35bHalACMPVcE2gIHRYeHkmD8U3d5JkLCo8v0e3EyRG3gj wsrqPj6BHTR8VMkK49iVkJxiN392q0yBjPCpkRaJPPRHo1Lw1pKroB+j5qoGZm62sIt8 dGuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BvLw+Q5ig6tUnaYjdSmgh3xKOEyw45Rn18F3gm7mWBk=; b=RixVwaTC3xzdfLE9/F17CCKRdWfyLeET7P6a1Fh7XrQ1ButCx7e3RycS4phwNn4Jyf s3RVX8xLpvUbEijkEDOcwJfNIQFWWtwHe55qoJ5Szu0AaiiZFZa1dY+vR/1vnYkIn6bS 3HtWhoFq0iiFA7mZARyXKEkn1ALEXWPkHjMkwLu6Y4CxKgAt9eaQiwjSH9F6seER3ZGY mRCVZoueoPxFO2O2rxPWcJYQpFhiptOGQN7VGZeVnh7npP7uPE3Dz0RucY7MJgesi9TM TRPLSu1VUf40Qg9qN31B6C5DRGHt24zqpAYGmJ2AERmxzJS5AGOZuBMbbn+/LyrMRdFj vpsQ== X-Gm-Message-State: APjAAAWydX0O/HstrN2c0j2gOHTmeeiJMF4uQXlInaWe2Ax6wG66VrV2 SvLWOSXh/CTcuXEDKAaF1SrXayVuew0= X-Received: by 2002:adf:e78c:: with SMTP id n12mr20979334wrm.94.1573510422808; Mon, 11 Nov 2019 14:13:42 -0800 (PST) Received: from flashheart.burtonini.com (35.106.2.81.in-addr.arpa. [81.2.106.35]) by smtp.gmail.com with ESMTPSA id d11sm24900083wrf.80.2019.11.11.14.13.41 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Nov 2019 14:13:42 -0800 (PST) From: Ross Burton To: openembedded-core@lists.openembedded.org Date: Mon, 11 Nov 2019 22:13:36 +0000 Message-Id: <20191111221337.20332-3-ross.burton@intel.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191111221337.20332-1-ross.burton@intel.com> References: <20191111221337.20332-1-ross.burton@intel.com> MIME-Version: 1.0 Subject: [OE-core] [PATCH 3/4] cve-check: neaten get_cve_info X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org Remove obsolete Python 2 code, and use convenience methods for neatness. Signed-off-by: Ross Burton --- meta/classes/cve-check.bbclass | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass index 3326944d791..419b2c9148f 100644 --- a/meta/classes/cve-check.bbclass +++ b/meta/classes/cve-check.bbclass @@ -256,23 +256,15 @@ def check_cves(d, patched_cves): def get_cve_info(d, cves): """ Get CVE information from the database. - - Unfortunately the only way to get CVE info is set the output to - html (hard to parse) or query directly the database. """ - try: - import sqlite3 - except ImportError: - from pysqlite2 import dbapi2 as sqlite3 + import sqlite3 cve_data = {} - db_file = d.getVar("CVE_CHECK_DB_FILE") - placeholder = ",".join("?" * len(cves)) - query = "SELECT * FROM NVD WHERE id IN (%s)" % placeholder - conn = sqlite3.connect(db_file) - cur = conn.cursor() - for row in cur.execute(query, tuple(cves)): + conn = sqlite3.connect(d.getVar("CVE_CHECK_DB_FILE")) + placeholders = ",".join("?" * len(cves)) + query = "SELECT * FROM NVD WHERE id IN (%s)" % placeholders + for row in conn.execute(query, tuple(cves)): cve_data[row[0]] = {} cve_data[row[0]]["summary"] = row[1] cve_data[row[0]]["scorev2"] = row[2] From patchwork Mon Nov 11 22:13:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 179158 Delivered-To: patch@linaro.org Received: by 2002:a92:38d5:0:0:0:0:0 with SMTP id g82csp7264116ilf; Mon, 11 Nov 2019 14:14:47 -0800 (PST) X-Google-Smtp-Source: APXvYqxCyd/Qeng5ffc4NuPYR4cKEVtsgs316be8Ix4/bmH4FQyiSCukYSkoUqFckDZz5lCeTGRP X-Received: by 2002:a17:90a:6d64:: with SMTP id z91mr1846383pjj.44.1573510486907; Mon, 11 Nov 2019 14:14:46 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1573510486; cv=none; d=google.com; s=arc-20160816; b=gDkclW/hMp5Tbw6+seE4qUJ/dALc+BRUjqJCXjvZOSo6mb765WQ+osiM0kw3cpMu4e 6nN3FtqciPZmgpkVFj32I/XhF8wBVxczHSnCx7fKt143FXenhaW66nWhNDPnr1G+kmGL csMmDD2nB5dmFdZPsXpVzSh2BtUMwjgyFzaAH61OwkE2EoPqIzHheTE5hVBKAAYbJvuv xfSFwHqd7NuylLDOxk8R8xByYtGOrhsRlywO3G0gocWVSkdi7us3YclpypyUZyGZ8te+ 7804KJpLEy+F5DrxKoQ8ydyBHD2DNXiZ7ooCc37Ut3W0FhPREqa9c7KbGm1peExoMFxX YFnQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :dkim-signature:delivered-to; bh=LdhccsxCkKbecsaSOnQysQtRD/19yY0fbmV8FGp/Qvo=; b=aRZ93zK9iyIhdv6upe9aeNG1if8ywVacHpiC2r4aeS49DitSx2vYeHj9yLFvmRWElu o/POol98w0VCRs9woEO0rqL9yFf8xIrnmZSUyJIyld09CF1QSe9h5WEsXpB6ehBAVwa1 cZMmxtmV40tsZdNK8hyg2GFwhp3HU1wcm3mqR6ZNGiGtNWIFZ0VUwEyYIMoOPzjneDUD atrJ0PcjG/JDMVhyXWJuBt1mSiybvsyOfFG9cS00qGF5n8bf2FTBjzI8Odfgdh/OCNbV W9F/ul8rCEu4wPMsvO6WaWNK+K5kDaUSbxKl2nYHviJydQVFER4qGidEjP64fr3Pesmt n+Bw== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=a+1ZiBvs; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id f21si17836292plj.56.2019.11.11.14.14.46; Mon, 11 Nov 2019 14:14:46 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=a+1ZiBvs; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id CF4037FBEB; Mon, 11 Nov 2019 22:14:33 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-wr1-f65.google.com (mail-wr1-f65.google.com [209.85.221.65]) by mail.openembedded.org (Postfix) with ESMTP id 61FB67FAAA for ; Mon, 11 Nov 2019 22:13:43 +0000 (UTC) Received: by mail-wr1-f65.google.com with SMTP id l7so4774828wrp.6 for ; Mon, 11 Nov 2019 14:13:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=3GW7ecThMk1nx7wVwGkBEYGD9LfM1p2nWPbbrKeaqx0=; b=a+1ZiBvsCk5qO3aUQ/g1SXafP07uU/9V/RbWlCMTB12xcznkBPZoR29lzXXJzYtB7z 9xdAhNEJv9BQ+dGtyAcdBX5hDeDZl/AgSqlZBlR3n3SWKgwWt4RUcWSckAUKSYBXVamH KIqC22SWQtvuOlbG0DsYoqVt4ukmWSDUbDqa/Z6cXtLVU2q8CLrLeR6zhdDi3eNQNbiT UxGrlKfmgZBhx/Oej9ViJ7SVF0fFBmqdUaG3EKYSy74g4QY90D+VKx6Tiq03XJpQe2AI fFoI+NJRhyQ6yVScCXpT2P5DDgCjEGhCNqWqzTJXfbSFEjfMY+lEMd7GRrEyx36DYbI2 XfIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3GW7ecThMk1nx7wVwGkBEYGD9LfM1p2nWPbbrKeaqx0=; b=F/kqY/4I8H4Y3C+OLKvHlWIKPH7yOsF38rs0sThph42CbzAYfchIAF0awniCKVdcG2 S89ZgACNAalp43qkYekPT0v7AIEdrbEe7Mi8gbpD46juGlBc/rEuOml8AcDpqAy94k13 FaIAr3UxwxdH1C/8J5XvSqpEAbd8DJKrcVob8foMJVfkURHKu8VYTUS/waOEOdrIAPId tH+7wKQqEOWtHKZEmQjvOP6Whh7jqa0RqtvpgmdSCfkUAajLg/J7WH6+J9SOS2n+tLOh +wThZ74UjecFper9gj0yXo6zD7kNgghyVU3Fi1UryIbkrOaz97kwTW1hmjr0sri1XreI z//A== X-Gm-Message-State: APjAAAUoKFA3N2FxELJ3tQCOKq5atZAat+50NrCLnoazgr/L0eOrP8Xv LKCH/Ic2Of/bSJvLETO94Vp8bVEMQEs= X-Received: by 2002:a5d:5224:: with SMTP id i4mr3142288wra.303.1573510423684; Mon, 11 Nov 2019 14:13:43 -0800 (PST) Received: from flashheart.burtonini.com (35.106.2.81.in-addr.arpa. [81.2.106.35]) by smtp.gmail.com with ESMTPSA id d11sm24900083wrf.80.2019.11.11.14.13.42 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Nov 2019 14:13:43 -0800 (PST) From: Ross Burton To: openembedded-core@lists.openembedded.org Date: Mon, 11 Nov 2019 22:13:37 +0000 Message-Id: <20191111221337.20332-4-ross.burton@intel.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191111221337.20332-1-ross.burton@intel.com> References: <20191111221337.20332-1-ross.burton@intel.com> MIME-Version: 1.0 Subject: [OE-core] [PATCH 4/4] cve-check: rewrite look to fix false negatives X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org A previous optimisation was premature and resulted in false-negatives in the report. Rewrite the checking algorithm to first get the list of potential CVEs by vendor:product, then iterate through every matching CPE for that CVE to determine if the bounds match or not. By doing this in two stages we can know if we've checked every CPE, instead of accidentally breaking out of the scan too early. Signed-off-by: Ross Burton --- meta/classes/cve-check.bbclass | 63 ++++++++++++++++++---------------- 1 file changed, 34 insertions(+), 29 deletions(-) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass index 419b2c9148f..e95716d9ded 100644 --- a/meta/classes/cve-check.bbclass +++ b/meta/classes/cve-check.bbclass @@ -165,7 +165,6 @@ def check_cves(d, patched_cves): """ Connect to the NVD database and find unpatched cves. """ - import ast, csv, tempfile, subprocess, io from distutils.version import LooseVersion cves_unpatched = [] @@ -187,68 +186,74 @@ def check_cves(d, patched_cves): cve_whitelist = d.getVar("CVE_CHECK_WHITELIST").split() import sqlite3 - db_file = d.getVar("CVE_CHECK_DB_FILE") - conn = sqlite3.connect(db_file) + db_file = d.expand("file:${CVE_CHECK_DB_FILE}?mode=ro") + conn = sqlite3.connect(db_file, uri=True) + # For each of the known product names (e.g. curl has CPEs using curl and libcurl)... for product in products: - c = conn.cursor() if ":" in product: vendor, product = product.split(":", 1) - c.execute("SELECT * FROM PRODUCTS WHERE PRODUCT IS ? AND VENDOR IS ?", (product, vendor)) else: - c.execute("SELECT * FROM PRODUCTS WHERE PRODUCT IS ?", (product,)) + vendor = "%" - for row in c: - cve = row[0] - version_start = row[3] - operator_start = row[4] - version_end = row[5] - operator_end = row[6] + # Find all relevant CVE IDs. + for cverow in conn.execute("SELECT DISTINCT ID FROM PRODUCTS WHERE PRODUCT IS ? AND VENDOR LIKE ?", (product, vendor)): + cve = cverow[0] if cve in cve_whitelist: bb.note("%s-%s has been whitelisted for %s" % (product, pv, cve)) # TODO: this should be in the report as 'whitelisted' patched_cves.add(cve) + continue elif cve in patched_cves: bb.note("%s has been patched" % (cve)) - else: - to_append = False + continue + + vulnerable = False + for row in conn.execute("SELECT * FROM PRODUCTS WHERE ID IS ? AND PRODUCT IS ? AND VENDOR LIKE ?", (cve, product, vendor)): + (_, _, _, version_start, operator_start, version_end, operator_end) = row + #bb.debug(2, "Evaluating row " + str(row)) + if (operator_start == '=' and pv == version_start): - to_append = True + vulnerable = True else: if operator_start: try: - to_append_start = (operator_start == '>=' and LooseVersion(pv) >= LooseVersion(version_start)) - to_append_start |= (operator_start == '>' and LooseVersion(pv) > LooseVersion(version_start)) + vulnerable_start = (operator_start == '>=' and LooseVersion(pv) >= LooseVersion(version_start)) + vulnerable_start |= (operator_start == '>' and LooseVersion(pv) > LooseVersion(version_start)) except: bb.warn("%s: Failed to compare %s %s %s for %s" % (product, pv, operator_start, version_start, cve)) - to_append_start = False + vulnerable_start = False else: - to_append_start = False + vulnerable_start = False if operator_end: try: - to_append_end = (operator_end == '<=' and LooseVersion(pv) <= LooseVersion(version_end)) - to_append_end |= (operator_end == '<' and LooseVersion(pv) < LooseVersion(version_end)) + vulnerable_end = (operator_end == '<=' and LooseVersion(pv) <= LooseVersion(version_end)) + vulnerable_end |= (operator_end == '<' and LooseVersion(pv) < LooseVersion(version_end)) except: bb.warn("%s: Failed to compare %s %s %s for %s" % (product, pv, operator_end, version_end, cve)) - to_append_end = False + vulnerable_end = False else: - to_append_end = False + vulnerable_end = False if operator_start and operator_end: - to_append = to_append_start and to_append_end + vulnerable = vulnerable_start and vulnerable_end else: - to_append = to_append_start or to_append_end + vulnerable = vulnerable_start or vulnerable_end - if to_append: + if vulnerable: bb.note("%s-%s is vulnerable to %s" % (product, pv, cve)) cves_unpatched.append(cve) - else: - bb.note("%s-%s is not vulnerable to %s" % (product, pv, cve)) - patched_cves.add(cve) + break + + if not vulnerable: + bb.note("%s-%s is not vulnerable to %s" % (product, pv, cve)) + # TODO: not patched but not vulnerable + patched_cves.add(cve) + conn.close() return (list(patched_cves), cves_unpatched)