From patchwork Fri Jul 19 20:33:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 169263 Delivered-To: patch@linaro.org Received: by 2002:a92:4782:0:0:0:0:0 with SMTP id e2csp4300196ilk; Fri, 19 Jul 2019 13:33:43 -0700 (PDT) X-Google-Smtp-Source: APXvYqzqjwxa1vS9olo1IGfp8YbbSNIHmpkpmZY+OSL1BNeJ2cQ1hiIEZWLuKzR9jhFP6xNIjSoF X-Received: by 2002:a63:bd0a:: with SMTP id a10mr55298499pgf.55.1563568423302; Fri, 19 Jul 2019 13:33:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1563568423; cv=none; d=google.com; s=arc-20160816; b=USTADVBl/onp1jrZ+1Yo8sQ/1Bz30R32Z751iUql2iRdATEv7VCPOB3+rxHdoF3eXO 4HwZLkdIUrdKz/2l2Aw4qT8ctLklrh+83d3VPps7MEWo105w1DaETcRXBnwvgfdUbOwY M5vNlJ7iLlpA2iiWnm9/7vwtJttlqc+UnBHk2ax0s7OCJNxVfFW0zHDYijVKnUw2a/oW geYQfN5xw9NP6zDDpJsZKTkBPhEvr1nhhBrJmiRxy6SrVENGxVV5XIxwmTYMjIJTTn8o 7nj8X5ej1DBvkpjD4Kf8GnScQqfxORWdM0DGtIgfnb2K8GX4Z2G8tW/X1oAOmAl6OH1d pxjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :dkim-signature:delivered-to; bh=3rEFA/REZfEiksIv9wFUFAGiCs4vrdWXE2m4tbSs81I=; b=Do0k8BrT/PGrbsJmfMcJR89A36QZIB0yml3u/0s+mhm1DyRDzcE9Euv/Y3NTDzyx00 aJp+ORcdGFXvoDpqv6VlAgRnv6QrQsKlAOCrLwSflDoVg3ywy1efH7hSGYmjpWQPN8JG 1t9aw3bJrPO5RG6MS13O/aoWVFCyW6mZ/j4OGN342ibgs0prdz2q1bYhPhhWAMUmjDhg alt+B7IJiiDoYRTDlR/RvMPMc3Lu0QYS38eDgRARcxXhdYaWErWoAT4GTyYq8LLd2Mdl dwUdNiQnbtxnobnan+KdXMFHezXtLDcyCcRlwXmtcOWwiSCLb+bd3DbG1Fqj64eGVbBG pqcA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=DFJJbO8m; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id d31si662615pla.84.2019.07.19.13.33.43; Fri, 19 Jul 2019 13:33:43 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=DFJJbO8m; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 437C77F1DB; Fri, 19 Jul 2019 20:33:40 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-wm1-f67.google.com (mail-wm1-f67.google.com [209.85.128.67]) by mail.openembedded.org (Postfix) with ESMTP id 6CFB97F1CC for ; Fri, 19 Jul 2019 20:33:26 +0000 (UTC) Received: by mail-wm1-f67.google.com with SMTP id h19so24447085wme.0 for ; Fri, 19 Jul 2019 13:33:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=gRZ2nKNkQHNFH21Pp6bphmKeLPDT4193/vucv/vuzII=; b=DFJJbO8mQQeI9V9A6WZy/OUUZa5Pg0U4IGdSNBqD2xRQlJQKBE1AJQSCvWJhULKfKi UZ34A3MfW7XGkzJ2+mY7Ae2aSIY6jfbv72pjGwnoMoZpxDEKdkGbvTItHVmXoD3GF2sW TnoGK56oKOikJgM+RUgCg0xd+fwMNgqiOmJrzVGAoKhgpphCui6nhO2nIEfvTz3kXCWF A38t+HwJ0AyqDK1DNldJ9oLjjm2y81bdqLvJgi+rdYIXLtpCn2TBEsaDz9+uanDO484r EgPIt+jtR8u6+wLeymdcVJzz4XppCIsKzwSR1c+XIFmiGSiOEbLZW1+IrIwItGGIFIgk H/pQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gRZ2nKNkQHNFH21Pp6bphmKeLPDT4193/vucv/vuzII=; b=DMZS///yOp7oE4Jqtzpk6mTa5vQGcxiom6S+3eY20VI/rdn958gbLB3v8wCxZtoFot JdPm63an2sBWS/8hZsIa3YhYTJ1IdJQcVAu/f18cFrE2UlR4K7V5d22sRHlznwpnVFtZ v7d3OgfFzpZt2gaixFl5D2OfL2UGff0epyOfFr8qsbDvK9KUNTcCL5bLSVUKDxCvr0O8 nmMVOMfkHwdmu7hfQiFcySR/si+vYv0TqVg/YYyM+uAuXkFkws8h2FeurExhfHUfnkjT TlQeS4q6D56JHPrh2EmrCSSUwmwVGyfdG9KlWFFJnfb/FQJ7mYI4HrNEdyU708heWluE lNzQ== X-Gm-Message-State: APjAAAUQXgDu+8k4YxqZLBfoUVuqbrGyzazsLD6R0g1aPtNKDBLJ3s9j xn8JBmWxuK76oVJ7MhptAnyz78QgtRc= X-Received: by 2002:a05:600c:228f:: with SMTP id 15mr46115667wmf.60.1563568406986; Fri, 19 Jul 2019 13:33:26 -0700 (PDT) Received: from flashheart.burtonini.com (35.106.2.81.in-addr.arpa. [81.2.106.35]) by smtp.gmail.com with ESMTPSA id z1sm35298988wrp.51.2019.07.19.13.33.25 for (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Fri, 19 Jul 2019 13:33:26 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Date: Fri, 19 Jul 2019 21:33:19 +0100 Message-Id: <20190719203319.20580-3-ross.burton@intel.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190719203319.20580-1-ross.burton@intel.com> References: <20190719203319.20580-1-ross.burton@intel.com> MIME-Version: 1.0 Subject: [OE-core] [PATCH 3/3] cve-update-db-native: clean up JSON fetching X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org Currently the code fetches the compressed JSON, writes it to a temporary file, uncompresses that with gzip and passes the fake file object to update_db(). Instead, uncompress the gzip'd data in memory and pass the JSON directly to update_db(). Signed-off-by: Ross Burton --- .../recipes-core/meta/cve-update-db-native.bb | 29 ++++++++----------- 1 file changed, 12 insertions(+), 17 deletions(-) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb index 41a2aa8f207..9c083bdc991 100644 --- a/meta/recipes-core/meta/cve-update-db-native.bb +++ b/meta/recipes-core/meta/cve-update-db-native.bb @@ -67,25 +67,20 @@ python do_populate_cve_db() { meta = c.fetchone() if not meta or meta[0] != last_modified: # Clear products table entries corresponding to current year - cve_year = 'CVE-' + str(year) + '%' - c.execute("delete from PRODUCTS where ID like ?", (cve_year,)) + c.execute("delete from PRODUCTS where ID like ?", ('CVE-%d%%' % year,)) # Update db with current year json file - req = urllib.request.Request(json_url) - if proxy: - req.set_proxy(proxy, 'https') try: - with urllib.request.urlopen(req, timeout=1) as r, \ - open(json_tmpfile, 'wb') as tmpfile: - shutil.copyfileobj(r, tmpfile) - except: + req = urllib.request.Request(json_url) + if proxy: + req.set_proxy(proxy, 'https') + with urllib.request.urlopen(req) as r: + update_db(c, gzip.decompress(r.read())) + c.execute("insert or replace into META values (?, ?)", [year, last_modified]) + except urllib.error.URLError as e: cve_f.write('Warning: CVE db update error, CVE data is outdated.\n\n') - break - - with gzip.open(json_tmpfile, 'rt') as jsonfile: - update_db(c, jsonfile) - c.execute("insert or replace into META values (?, ?)", - [year, last_modified]) + bb.warn("Cannot parse CVE data (%s), update failed" % e.reason) + return # Update success, set the date to cve_check file. if year == date.today().year: @@ -148,9 +143,9 @@ def parse_node_and_insert(c, node, cveId): c.executemany("insert into PRODUCTS values (?, ?, ?, ?, ?, ?, ?)", cpe_generator()) -def update_db(c, json_filename): +def update_db(c, jsondata): import json - root = json.load(json_filename) + root = json.loads(jsondata) for elt in root['CVE_Items']: if not elt['impact']: