From patchwork Wed Nov 6 15:37:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mikko Rapeli X-Patchwork-Id: 178741 Delivered-To: patch@linaro.org Received: by 2002:a92:38d5:0:0:0:0:0 with SMTP id g82csp875071ilf; Wed, 6 Nov 2019 08:55:59 -0800 (PST) X-Google-Smtp-Source: APXvYqwMkKQH6TfTO/mxC0Gq74dCBj43jcp8p+ggZZ+rWgHpWs0svXroWBqCNrmEqlU+PA4gf+sI X-Received: by 2002:a63:f94f:: with SMTP id q15mr4098062pgk.412.1573059359406; Wed, 06 Nov 2019 08:55:59 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1573059359; cv=none; d=google.com; s=arc-20160816; b=jfwscoc+bmtKDfJzrzEn0zz2xYNLMn8yCgkiHZZs9eySHb3wIGfc2goQdp6AwNuoAD 7qPJeg9gBnXaD0nUMtRWV8wqPIUnVAs0XPIFSuS6gmR1FWtU/Dvu9YPobVqOv8r3DNbD 8F0Z6QzBGKtOWngjB+G21cXu+qVsbKPKeyoiQQcn6g5Iqpza/nnvrAZD4YL0usb3mSJ3 EPhDwZRsD0td8RJnOOpamMVdqQMc5guCPmJU+Z/YA9A9VOdyC3ZMN7nUe6Xg4PoK3pOd KmBzjePFHqSR5G6Wqamqm7Q178nSCB2/VniYaGTxl9m2U1OSH4q96TzILKjt+9b3/HVw Z9BA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:mime-version :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:references :in-reply-to:message-id:date:to:from:dkim-signature:delivered-to; bh=hK50fvMlYl4amFlDe+mXz6Svxqafe1gAABza2HYV2+g=; b=wacW5j8+gaDA2YBEo8FnMXaS/0xEf+YDLYV7tKRhg3WDHLEfkMhMDhYWBd9mF1XvbH 37PB5fzL44fS0ocjN+X7FkVBOHXn7kl2d8JEJNo97fGvdDk86w2ywzuKSUrEVIpl1P29 8wq/8LOhSSXUHeqCTM0Er4UyOmdpXU85Vy2tUExZjsFW9B2hxni1RhTnr1x77qzhUK8t eEKoQn0tJoLXZlWfX5JLMA04Fn/isy4xoymkYFza4ZVMF1ytzMSOg77VXwLz1fKvfo3y 6DmeU1KaO/3fKk73VPN4BazMdn3S9MOW0hFHtmyaqd8t+wXL4o0BcoC/x2G8Ht7omYIU HOSQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@kapsi.fi header.s=20161220 header.b="y2f/L0qO"; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bmw.de Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id g21si7138540pgh.467.2019.11.06.08.55.59; Wed, 06 Nov 2019 08:55:59 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@kapsi.fi header.s=20161220 header.b="y2f/L0qO"; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bmw.de Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 175A27F9A7; Wed, 6 Nov 2019 16:55:39 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail.kapsi.fi (mail.kapsi.fi [91.232.154.25]) by mail.openembedded.org (Postfix) with ESMTP id 9BD5D7F895 for ; Wed, 6 Nov 2019 15:38:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=kapsi.fi; s=20161220; h=References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Sender:Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=YI8XShzA0FjpelZ+DPcqRe7psfe++srLUXtyfUNaodY=; b=y2f/L0qO7djiz/P9Gga5LRsJ23 cX/r6kH3bqWBn5W6Avr4kqkXEanTpfukgrQf+gN+lJzaB/QL10hI/gPlvWdyPT7W51lTiUzWhpopT A6GWCHScVofh7SH60aSkctrXRepiA8Qda89qk8aeEJiJyOXPSdC3qIYZxwEmfeQWfHweSJ2p86SCE b3YjqFtiFOx8EDB/QXtimkXnnOGofzOoeWlOoo+2F9ASsKJjgOc9klPuHXtNbxG9uLllWUN4+YqnM Tcopz+Pj0ZhQfR1m4myIYEXJJNMucPurte0HmZIuAQo1tLnA3WrRc3O8szoDI1tlW5ZOS/nQh23Ye bVjcJAJA==; Received: from kapsi.fi ([91.232.154.11] helo=lakka.kapsi.fi) by mail.kapsi.fi with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.89) (envelope-from ) id 1iSNOa-0000A1-1q; Wed, 06 Nov 2019 17:38:56 +0200 Received: from mcfrisk by lakka.kapsi.fi with local (Exim 4.84_2) (envelope-from ) id 1iSNO5-0007XI-RH; Wed, 06 Nov 2019 17:38:25 +0200 From: Mikko Rapeli To: openembedded-core@lists.openembedded.org Date: Wed, 6 Nov 2019 17:37:40 +0200 Message-Id: <487417678e7175395516806742a481b47fd0a151.1573047194.git.mikko.rapeli@bmw.de> X-Mailer: git-send-email 2.1.4 In-Reply-To: References: In-Reply-To: References: X-Rspam-Score: 6.0 (++++++) X-Rspam-Report: Action: add header Symbol: ARC_NA(0.00) Symbol: DMARC_POLICY_SOFTFAIL(0.10) Symbol: R_SPF_NEUTRAL(0.00) Symbol: FROM_HAS_DN(0.00) Symbol: TO_DN_SOME(0.00) Symbol: MULTIPLE_UNIQUE_HEADERS(4.89) Symbol: MIME_GOOD(-0.10) Symbol: TO_MATCH_ENVRCPT_ALL(0.00) Symbol: RCPT_COUNT_THREE(0.00) Symbol: NEURAL_SPAM(0.00) Symbol: RCVD_TLS_LAST(0.00) Symbol: MID_CONTAINS_FROM(1.00) Symbol: IP_SCORE(-0.15) Symbol: FORGED_SENDER(0.30) Symbol: R_DKIM_NA(0.00) Symbol: MIME_TRACE(0.00) Symbol: ASN(0.00) Symbol: FROM_NEQ_ENVFROM(0.00) Symbol: RCVD_COUNT_TWO(0.00) Message: (SPF): spf neutral Message-ID: 487417678e7175395516806742a481b47fd0a151.1573047194.git.mikko.rapeli@bmw.de X-Rspam-Status: Yes X-Rspam-Bar: ++++++ X-SA-Exim-Connect-IP: 91.232.154.11 X-SA-Exim-Mail-From: mcfrisk@kapsi.fi X-SA-Exim-Scanned: No (on mail.kapsi.fi); SAEximRunCond expanded to false Subject: [OE-core] [PATCH RFC CFH][sumo 25/47] cve-update-db-native: improve metadata parsing X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton The metadata parser is fragile: first it coerces a bytes() to a str() (so the string is b'LastModifiedDate:2019...'), assumes the first line is the date, and then uses a regex to parse (which then includes the trailing quote as part of the date). Clean this up by parsing the bytes as UTF-8 (ASCII is probably fine, but this is safer), iterate through the lines and split on colons to find the right key/value pair. (From OE-Core rev: bb4e53af33d6ca1e9346464adbdc1b39c47530f3) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie --- meta/recipes-core/meta/cve-update-db-native.bb | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) -- 1.9.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb index a5d8e32..6907197 100644 --- a/meta/recipes-core/meta/cve-update-db-native.bb +++ b/meta/recipes-core/meta/cve-update-db-native.bb @@ -17,7 +17,7 @@ python do_populate_cve_db() { Update NVD database with json data feed """ - import sqlite3, urllib, shutil, gzip, re + import sqlite3, urllib, shutil, gzip from datetime import date BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-" @@ -47,13 +47,15 @@ python do_populate_cve_db() { req = urllib.request.Request(meta_url) if proxy: req.set_proxy(proxy, 'https') - try: - with urllib.request.urlopen(req, timeout=1) as r: - date_line = str(r.read().splitlines()[0]) - last_modified = re.search('lastModifiedDate:(.*)', date_line).group(1) - except: - cve_f.write('Warning: CVE db update error, CVE data is outdated.\n\n') - break + with urllib.request.urlopen(req) as r: + for l in r.read().decode("utf-8").splitlines(): + key, value = l.split(":", 1) + if key == "lastModifiedDate": + last_modified = value + break + else: + bb.warn("Cannot parse CVE metadata, update failed") + return # Compare with current db last modified date c.execute("select DATE from META where YEAR = ?", (year,))