From patchwork Thu Apr 2 15:21:59 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stuart Haslam X-Patchwork-Id: 46726 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-wi0-f198.google.com (mail-wi0-f198.google.com [209.85.212.198]) by ip-10-151-82-157.ec2.internal (Postfix) with ESMTPS id 21B47216D1 for ; Thu, 2 Apr 2015 15:22:49 +0000 (UTC) Received: by wiaa2 with SMTP id a2sf19584091wia.1 for ; Thu, 02 Apr 2015 08:22:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:delivered-to:from:to:date :message-id:subject:precedence:list-id:list-unsubscribe:list-archive :list-post:list-help:list-subscribe:mime-version:content-type :content-transfer-encoding:errors-to:sender:x-original-sender :x-original-authentication-results:mailing-list; bh=saHVOjCdrYbTq4MTS8EaATpghZpxgPoQQ9Ubv9401cg=; b=GMvPZmxnR/jlQkqo3+679+0wi2GwlvzBglkZ8Q835CD6LbmI2KDAk5OPGf0KcWCs1u s+CytIOF2z8S7NqXf1faEpT6rgxfRdkcKgihW2jtD4RZeQKYeVpZz0DRq1Ok5jzsha99 8oKU9Yf9auWZr7/fTvI1NRNF0wI0wQUi9UEP3m1LZvw1Am+nFk/Yjtb6wvmq4JX0vlJg RWOD4jsM/EiHVypxKK04DQWS3Rd86cjJRKZCuHdvBxD7j6VXHsPk3tm6UPuuvhEI2O3R b9QGfYH+YZSl/lD0cY9Uexe/p3l+A8aCHgV2nY7Idw65aqPRad1ZTsIDnP+lWr2AmMiD ps7g== X-Gm-Message-State: ALoCoQlVX+J5BFa9o8N5c7KG0ohdj27BWbrdp5caZvGMeqGtGeDBSTwSwQGAVAYJUO0o1I7lshXg X-Received: by 10.180.106.136 with SMTP id gu8mr658279wib.6.1427988168470; Thu, 02 Apr 2015 08:22:48 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.152.205.33 with SMTP id ld1ls228219lac.11.gmail; Thu, 02 Apr 2015 08:22:48 -0700 (PDT) X-Received: by 10.152.1.70 with SMTP id 6mr41094988lak.83.1427988168290; Thu, 02 Apr 2015 08:22:48 -0700 (PDT) Received: from mail-la0-f53.google.com (mail-la0-f53.google.com. [209.85.215.53]) by mx.google.com with ESMTPS id ei9si968865lad.152.2015.04.02.08.22.48 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 02 Apr 2015 08:22:48 -0700 (PDT) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.215.53 as permitted sender) client-ip=209.85.215.53; Received: by lagg8 with SMTP id g8so62684985lag.1 for ; Thu, 02 Apr 2015 08:22:48 -0700 (PDT) X-Received: by 10.152.197.34 with SMTP id ir2mr14635540lac.36.1427988168187; Thu, 02 Apr 2015 08:22:48 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.112.57.201 with SMTP id k9csp1409553lbq; Thu, 2 Apr 2015 08:22:47 -0700 (PDT) X-Received: by 10.55.40.215 with SMTP id o84mr16882368qko.93.1427988166792; Thu, 02 Apr 2015 08:22:46 -0700 (PDT) Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id a63si2627892qga.120.2015.04.02.08.22.45; Thu, 02 Apr 2015 08:22:46 -0700 (PDT) Received-SPF: none (google.com: lng-odp-bounces@lists.linaro.org does not designate permitted sender hosts) client-ip=54.225.227.206; Received: by lists.linaro.org (Postfix, from userid 109) id B675D65099; Thu, 2 Apr 2015 15:22:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252.ec2.internal X-Spam-Level: X-Spam-Status: No, score=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from ip-10-142-244-252.ec2.internal (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id D4E2F65054; Thu, 2 Apr 2015 15:22:43 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 1846965096; Thu, 2 Apr 2015 15:22:42 +0000 (UTC) Received: from mail-wi0-f181.google.com (mail-wi0-f181.google.com [209.85.212.181]) by lists.linaro.org (Postfix) with ESMTPS id D4DA665054 for ; Thu, 2 Apr 2015 15:22:40 +0000 (UTC) Received: by widjs5 with SMTP id js5so11534996wid.1 for ; Thu, 02 Apr 2015 08:22:40 -0700 (PDT) X-Received: by 10.195.13.104 with SMTP id ex8mr93318133wjd.12.1427988160130; Thu, 02 Apr 2015 08:22:40 -0700 (PDT) Received: from e106441.cambridge.arm.com ([2001:41d0:a:3cb4::1]) by mx.google.com with ESMTPSA id u10sm30474697wib.1.2015.04.02.08.22.38 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 02 Apr 2015 08:22:39 -0700 (PDT) From: Stuart Haslam To: lng-odp@lists.linaro.org Date: Thu, 2 Apr 2015 16:21:59 +0100 Message-Id: <1427988119-17037-1-git-send-email-stuart.haslam@linaro.org> X-Mailer: git-send-email 2.1.1 X-Topics: patch Subject: [lng-odp] [PATCH] linux-generic: support running with restricted cpu set X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: , List-Help: , List-Subscribe: , MIME-Version: 1.0 Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: stuart.haslam@linaro.org X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.215.53 as permitted sender) smtp.mail=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 odp_cpu_count() returns the number of online CPUs, but when running in a control group access may be restricted to only a subset of these. This combined with a lack of error handling in odph_linux_pthread_create() means a number of the applications create more threads than there are cores available to them, leading to deadlocks in odp_barrier_wait(). Signed-off-by: Stuart Haslam --- helper/include/odp/helper/linux.h | 4 +- platform/linux-generic/odp_linux.c | 63 +++++++++++++++++--------------- platform/linux-generic/odp_system_info.c | 16 +++++--- 3 files changed, 47 insertions(+), 36 deletions(-) diff --git a/helper/include/odp/helper/linux.h b/helper/include/odp/helper/linux.h index 146e26c..44ee787 100644 --- a/helper/include/odp/helper/linux.h +++ b/helper/include/odp/helper/linux.h @@ -70,8 +70,10 @@ int odph_linux_cpumask_default(odp_cpumask_t *mask, int num); * @param mask CPU mask * @param start_routine Thread start function * @param arg Thread argument + * + * @return Number of threads created */ -void odph_linux_pthread_create(odph_linux_pthread_t *thread_tbl, +int odph_linux_pthread_create(odph_linux_pthread_t *thread_tbl, const odp_cpumask_t *mask, void *(*start_routine) (void *), void *arg); diff --git a/platform/linux-generic/odp_linux.c b/platform/linux-generic/odp_linux.c index 6865ab1..5a0d456 100644 --- a/platform/linux-generic/odp_linux.c +++ b/platform/linux-generic/odp_linux.c @@ -23,42 +23,38 @@ #include #include -int odph_linux_cpumask_default(odp_cpumask_t *mask, int num_in) + +int odph_linux_cpumask_default(odp_cpumask_t *mask, int num) { - int i; - int first_cpu = 1; - int num = num_in; - int cpu_count; + int ret, cpu, i; + cpu_set_t cpuset; - cpu_count = odp_cpu_count(); + ret = pthread_getaffinity_np(pthread_self(), + sizeof(cpu_set_t), &cpuset); + if (ret != 0) + ODP_ABORT("failed to read CPU affinity value\n"); + + odp_cpumask_zero(mask); /* * If no user supplied number or it's too large, then attempt * to use all CPUs */ - if (0 == num) - num = cpu_count; - if (cpu_count < num) - num = cpu_count; - - /* - * Always force "first_cpu" to a valid CPU - */ - if (first_cpu >= cpu_count) - first_cpu = cpu_count - 1; - - /* Build the mask */ - odp_cpumask_zero(mask); - for (i = 0; i < num; i++) { - int cpu; - - cpu = (first_cpu + i) % cpu_count; - odp_cpumask_set(mask, cpu); + if (0 == num || CPU_SETSIZE < num) + num = CPU_COUNT(&cpuset); + + /* build the mask, allocating down from highest numbered CPU */ + for (cpu = 0, i = CPU_SETSIZE-1; i >= 0 && cpu < num; --i) { + if (CPU_ISSET(i, &cpuset)) { + odp_cpumask_set(mask, i); + cpu++; + } } - return num; + return cpu; } + static void *odp_run_start_routine(void *arg) { odp_start_args_t *start_args = arg; @@ -80,7 +76,7 @@ static void *odp_run_start_routine(void *arg) } -void odph_linux_pthread_create(odph_linux_pthread_t *thread_tbl, +int odph_linux_pthread_create(odph_linux_pthread_t *thread_tbl, const odp_cpumask_t *mask_in, void *(*start_routine) (void *), void *arg) { @@ -89,6 +85,7 @@ void odph_linux_pthread_create(odph_linux_pthread_t *thread_tbl, odp_cpumask_t mask; int cpu_count; int cpu; + int ret; odp_cpumask_copy(&mask, mask_in); num = odp_cpumask_count(&mask); @@ -98,8 +95,9 @@ void odph_linux_pthread_create(odph_linux_pthread_t *thread_tbl, cpu_count = odp_cpu_count(); if (num < 1 || num > cpu_count) { - ODP_ERR("Bad num\n"); - return; + ODP_ERR("Invalid number of threads: %d (%d cores available)\n", + num, cpu_count); + return 0; } cpu = odp_cpumask_first(&mask); @@ -123,11 +121,18 @@ void odph_linux_pthread_create(odph_linux_pthread_t *thread_tbl, thread_tbl[i].start_args->start_routine = start_routine; thread_tbl[i].start_args->arg = arg; - pthread_create(&thread_tbl[i].thread, &thread_tbl[i].attr, + ret = pthread_create(&thread_tbl[i].thread, &thread_tbl[i].attr, odp_run_start_routine, thread_tbl[i].start_args); + if (ret != 0) { + ODP_ERR("Failed to start thread on cpu #%d\n", cpu); + free(thread_tbl[i].start_args); + break; + } cpu = odp_cpumask_next(&mask, cpu); } + + return i; } diff --git a/platform/linux-generic/odp_system_info.c b/platform/linux-generic/odp_system_info.c index 6b6c723..0aaaeda 100644 --- a/platform/linux-generic/odp_system_info.c +++ b/platform/linux-generic/odp_system_info.c @@ -4,11 +4,14 @@ * SPDX-License-Identifier: BSD-3-Clause */ +#define _GNU_SOURCE #include #include #include #include #include +#include +#include #include #include @@ -46,20 +49,21 @@ static odp_system_info_t odp_system_info; /* - * Report the number of online CPU's + * Report the number of CPUs in the affinity mask of the main thread */ static int sysconf_cpu_count(void) { - long ret; + cpu_set_t cpuset; + int ret; - ret = sysconf(_SC_NPROCESSORS_ONLN); - if (ret < 0) + ret = pthread_getaffinity_np(pthread_self(), + sizeof(cpuset), &cpuset); + if (ret != 0) return 0; - return (int)ret; + return CPU_COUNT(&cpuset); } - #if defined __x86_64__ || defined __i386__ || defined __OCTEON__ || \ defined __powerpc__ /*