From patchwork Tue Nov 14 09:04:49 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Masahiro Yamada X-Patchwork-Id: 118863 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp2796974qgn; Tue, 14 Nov 2017 01:07:49 -0800 (PST) X-Google-Smtp-Source: AGs4zMZsC2eAiMtFk2IGEb8b1Ez0nOx2w0751NQ+fSmjOVyZWeHXZJX0kEZ9ffdC8yYGp/6n7ff2 X-Received: by 10.84.149.168 with SMTP id m37mr11655858pla.36.1510650469442; Tue, 14 Nov 2017 01:07:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1510650469; cv=none; d=google.com; s=arc-20160816; b=lW0vtePtAcAc3k2fpM/DPckeJ9R1eXzUcMApageT6tkc9Bzb9YCvpBK8Co70Fvy/p/ h0vxGYmBcpFcObNjCVG/3SuglTOJWBj95/+Ex8IpKwvu6gyNJvafG8Q5I9UgaNOzan3a n7ct8lhgVWLYtE4GKP0s7Nju/DJ9PGcusJZlUIn0jD9A8urgzsiNYloGh967nTTsX+/9 ynjlKSmzCvrF1RUzmEWFxLcLXPg6Re7V6TlMxHHSUz9U+AH4BdVdv0rIhCtlmC2dc4WN Qoc6Mo81UpGzFivWP1U/7A0pYMnixmjwdv2R22VX6mzkOrBnnnANGK+o4kDrrFq9XZlD 28Sw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:message-id:date:subject:cc:to:from :dkim-signature:dkim-filter:arc-authentication-results; bh=C+06YYO/XjUN42CX2TX9ChIHM8V3ZnjbA2MRjuoyMhA=; b=xqIsT0L3v2MdZt+b8yaWTWlDR15E+EZKFcQwJe4+EbUAySWI65ctE393erh+S/HWhI OFNfdEmieYTF+Q5T2x5Xb8UypvseOWFYkhihObUPgF0oqlxKcw6ZU8CrQDbpjvfYihot NrKY4NgDc76+P724869lwOIqUHINw6hxDGg1UWukQ3UKBI0CXC4ERcVE2KmCmjiBw+f3 pQ4JyFEyGbBxdjJEepn2B79e+7oQJN1hnBQi/yBPGyrl/QFUzU5wl/dVQTzd+BkwRJhI Tqv1Y9Z5fQJaYWROMUExvFGTXULqcDDjz98G82n+6gw1POWOonGmkIqxvaO2AInEXYav SUaQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@nifty.com header.s=dec2015msa header.b=lj3yZBZC; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id n7si8521353pgd.755.2017.11.14.01.07.49; Tue, 14 Nov 2017 01:07:49 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@nifty.com header.s=dec2015msa header.b=lj3yZBZC; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753750AbdKNJHs (ORCPT + 27 others); Tue, 14 Nov 2017 04:07:48 -0500 Received: from conuserg-11.nifty.com ([210.131.2.78]:27582 "EHLO conuserg-11.nifty.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751790AbdKNJHm (ORCPT ); Tue, 14 Nov 2017 04:07:42 -0500 Received: from pug.e01.socionext.com (p14092-ipngnfx01kyoto.kyoto.ocn.ne.jp [153.142.97.92]) (authenticated) by conuserg-11.nifty.com with ESMTP id vAE94xdr014001; Tue, 14 Nov 2017 18:05:00 +0900 DKIM-Filter: OpenDKIM Filter v2.10.3 conuserg-11.nifty.com vAE94xdr014001 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nifty.com; s=dec2015msa; t=1510650300; bh=C+06YYO/XjUN42CX2TX9ChIHM8V3ZnjbA2MRjuoyMhA=; h=From:To:Cc:Subject:Date:From; b=lj3yZBZChpyoKlNGLa6pvTj2osDdelnYNi0Fq9gWlAme+epzzZK4NKbM9D03PPZtV YGcNKW/KBl4aFRPezXNggqlbWwmU6IaRa+WIeGCRU2RxxDOZf46mFDKWxwNQZ4p3Yc XaUFfDQXzohOCZWCn/iXgnguXq0hUyILeajULr3fZGDlygEg75Sq7QtiU5xIt4L+Yf RNjAp+zHSztklbcRHZ4fUYZxnZ9S0/VTJbBm6vl/4TB7/fSWecX8AfBDw4vlgkmGrb y2VRVT+xpNyfE2joMmzhpP9uxducxiuUGWbPGN90I5HJSYyE5f+FeWV5eueQBCuu0R jH8MlsGWot5qw== X-Nifty-SrcIP: [153.142.97.92] From: Masahiro Yamada To: linux-kbuild@vger.kernel.org Cc: cocci@systeme.lip6.fr, Julia Lawall , Nicolas Palix , Gilles Muller , Masahiro Yamada , linux-kernel@vger.kernel.org, Michal Marek Subject: [PATCH v3] coccinelle: fix parallel build with CHECK=scripts/coccicheck Date: Tue, 14 Nov 2017 18:04:49 +0900 Message-Id: <1510650290-3363-1-git-send-email-yamada.masahiro@socionext.com> X-Mailer: git-send-email 2.7.4 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The command "make -j8 C=1 CHECK=scripts/coccicheck COCCI=..." produces lots of "coccicheck failed" error messages. Julia Lawall explained the coccinelle behavior as follows: "The problem on the Coccinelle side is that it uses a subdirectory with the name of the semantic patch to store standard output and standard error for the different threads. I didn't want to use a name with the pid, so that one could easily find this information while Coccinelle is running. Normally the subdirectory is cleaned up when Coccinelle completes, so there is only one of them at a time. Maybe it is best to just add the pid. There is the risk that these subdirectories will accumulate if Coccinelle crashes in a way such that they don't get cleaned up, but Coccinelle could print a warning if it detects this case, rather than failing." When scripts/coccicheck is used as CHECK tool and -j option is given to Make, the whole of build process runs in parallel. So, multiple processes try to get access to the same subdirectory. I notice spatch creates the subdirectory only when it runs in parallel (i.e. --jobs is given and is greater than 1). Setting NPROC=1 is a sensible solution; spatch does not create the subdirectory. Besides, ONLINE=1 mode takes a single file input for each spatch invocation, so there is no reason to parallelize it in the first place. Signed-off-by: Masahiro Yamada --- Changes in v3: - Set NPROC=1 because this is a more sensible solution given that there is no reason to run coccinelle in parallel for ONLINE=1 mode. - Move J= option handling to a proper place. - Add more detailed explanation Changes in v2: - Grep '-j' instead of '--jobserver-auth'. '--jobserver-*' is not a stable option flag. Make 4.2 change '--jobserver-fds' into '--jobserver-auth' - Add -q option to grep scripts/coccicheck | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) -- 2.7.4 diff --git a/scripts/coccicheck b/scripts/coccicheck index 040a8b1..7da82a1 100755 --- a/scripts/coccicheck +++ b/scripts/coccicheck @@ -30,12 +30,6 @@ else VERBOSE=0 fi -if [ -z "$J" ]; then - NPROC=$(getconf _NPROCESSORS_ONLN) -else - NPROC="$J" -fi - FLAGS="--very-quiet" # You can use SPFLAGS to append extra arguments to coccicheck or override any @@ -70,6 +64,13 @@ if [ "$C" = "1" -o "$C" = "2" ]; then # Take only the last argument, which is the C file to test shift $(( $# - 1 )) OPTIONS="$COCCIINCLUDE $1" + + # If -j option is given to Make, scripts/coccicheck runs in parallel. + # If coccinelle also runs in parallel, it fails because multiple processes + # try to get access to the same subdirectory that stores stdout/stderr. + # No need to parallelize coccinelle in this case - this mode takes only + # one file input. + NPROC=1 else ONLINE=0 if [ "$KBUILD_EXTMOD" = "" ] ; then @@ -77,6 +78,12 @@ else else OPTIONS="--dir $KBUILD_EXTMOD $COCCIINCLUDE" fi + + if [ -z "$J" ]; then + NPROC=$(getconf _NPROCESSORS_ONLN) + else + NPROC="$J" + fi fi if [ "$KBUILD_EXTMOD" != "" ] ; then