From patchwork Thu Mar 31 13:08:14 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bill Fischofer X-Patchwork-Id: 64782 Delivered-To: patch@linaro.org Received: by 10.112.199.169 with SMTP id jl9csp150013lbc; Thu, 31 Mar 2016 06:12:27 -0700 (PDT) X-Received: by 10.140.20.104 with SMTP id 95mr16048927qgi.40.1459429947100; Thu, 31 Mar 2016 06:12:27 -0700 (PDT) Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id d32si7730640qgd.2.2016.03.31.06.12.26; Thu, 31 Mar 2016 06:12:27 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=pass (p=NONE dis=NONE) header.from=linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id B4061615BE; Thu, 31 Mar 2016 13:12:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id C65A8619CD; Thu, 31 Mar 2016 13:09:04 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 57537619E6; Thu, 31 Mar 2016 13:09:01 +0000 (UTC) Received: from mail-oi0-f50.google.com (mail-oi0-f50.google.com [209.85.218.50]) by lists.linaro.org (Postfix) with ESMTPS id AAC89619F3 for ; Thu, 31 Mar 2016 13:08:22 +0000 (UTC) Received: by mail-oi0-f50.google.com with SMTP id d205so59414129oia.0 for ; Thu, 31 Mar 2016 06:08:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=gAQe7rDgArxWn+2rLqLA8hzrfPhDwn6epqCje5P/0xc=; b=Cm6MKoCzdCs3NyhIQBANDLdv3d0rhqtsnj/fgwkO9z8zrVVnDfnbNhiQXpTc71Ui39 HQtXPRtB8MddnAktmeAPfKihVrRYGLB+XzMKflIMgQLfxxkbD1z/joiRENjL+jqbomxi tgzlu9185uccCwGg6/5/bYzv7119vOzVzgYrNKTn/RvWFMREJwjc5XKENo+jR03AU8Q/ C8zvAcVkAsRRfVtc80D9BggtmNgihklmPPF7NfAHnda0qa5g9RsjTM4uzIqWjQtFRJGx DcegDlOchI05i1jxv3OMDusq9wy8IsjAybLHBBgZ3wpAsGFnRpY8fEA0uwARSgKgPmZ1 TgzQ== X-Gm-Message-State: AD7BkJLPNInDXc+trLS+JUK7e3BslCyc4/pg+k70Nhzv7XX4CwLD0Ml7Ok9W4YT/BcY0rEtrIHw= X-Received: by 10.157.59.230 with SMTP id k93mr7699393otc.167.1459429702263; Thu, 31 Mar 2016 06:08:22 -0700 (PDT) Received: from Ubuntu15.localdomain (cpe-66-68-129-43.austin.res.rr.com. [66.68.129.43]) by smtp.gmail.com with ESMTPSA id yn3sm2668627obc.27.2016.03.31.06.08.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 31 Mar 2016 06:08:21 -0700 (PDT) From: Bill Fischofer To: lng-odp@lists.linaro.org Date: Thu, 31 Mar 2016 08:08:14 -0500 Message-Id: <1459429694-6832-6-git-send-email-bill.fischofer@linaro.org> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1459429694-6832-1-git-send-email-bill.fischofer@linaro.org> References: <1459429694-6832-1-git-send-email-bill.fischofer@linaro.org> X-Topics: patch Subject: [lng-odp] [API-NEXT PATCHv7 5/5] linux-generic: add support for initial cpumasks X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: "Gary S. Robertson" These code changes depend on the addition of control and worker cpumask pointers to the ODP initialization parameters, and implement the change in behavior suggested with that patch. They serve as the 'glue' between the input of the new ODP API initial cpuset masks and the use of those new cpumasks by the ODP application or instance. These changes allow multiple concurrent ODP instances to be given CPU resources which do not conflict with one another, so multiple ODP instances can coexist harmoniously with any isolation support on the underlying platform as well as with one another. Specifically: if either of the cpumask pointers are NULL when calling odp_init_global(), then that function creates 'best-guess' default values for the unspecified cpumasks and saves those defaults into global variables for later reference. However, if the cpumasks are valid and their pointers initialized prior to calling odp_init_global() then that routine saves their contents into the global variables instead of the default cpumasks. When odp_cpumask_default_control() or odp_cpumask_default_worker() are called they build the caller's cpumasks based on the saved contents of the global cpumask variables. Signed-off-by: Gary S. Robertson Signed-off-by: Bill Fischofer --- platform/linux-generic/include/odp_internal.h | 2 +- platform/linux-generic/odp_cpumask.c | 177 ++++++++++++++++++++++++-- platform/linux-generic/odp_init.c | 2 +- 3 files changed, 165 insertions(+), 16 deletions(-) diff --git a/platform/linux-generic/include/odp_internal.h b/platform/linux-generic/include/odp_internal.h index de9dc6a..7c8a12a 100644 --- a/platform/linux-generic/include/odp_internal.h +++ b/platform/linux-generic/include/odp_internal.h @@ -70,7 +70,7 @@ extern struct odp_global_data_s odp_global_data; int _odp_term_global(enum init_stage stage); int _odp_term_local(enum init_stage stage); -int odp_cpumask_init_global(void); +int odp_cpumask_init_global(const odp_init_t *params); int odp_cpumask_term_global(void); int odp_system_info_init(void); diff --git a/platform/linux-generic/odp_cpumask.c b/platform/linux-generic/odp_cpumask.c index 320ca8e..fd672e2 100644 --- a/platform/linux-generic/odp_cpumask.c +++ b/platform/linux-generic/odp_cpumask.c @@ -10,6 +10,7 @@ #include #include +#include #include #include @@ -215,8 +216,7 @@ int odp_cpumask_next(const odp_cpumask_t *mask, int cpu) /* * This function obtains system information specifying which cpus are - * available at boot time. These data are then used to produce cpumasks of - * configured CPUs without concern over isolation support. + * available at boot time. */ static int get_installed_cpus(void) { @@ -288,21 +288,22 @@ static int get_installed_cpus(void) } /* - * This function creates reasonable default cpumasks for control and worker - * tasks from the set of CPUs available at boot time. + * This function creates reasonable default cpumasks for control tasks + * from the set of CPUs available at boot time. + * This function assumes that the global control cpumask contains + * a list of all installed CPUs, and that no control cpumask was specified. */ -int odp_cpumask_init_global(void) +static void init_default_control_cpumask(int worker_cpus_default) { odp_cpumask_t *control_mask = &odp_global_data.control_cpus; odp_cpumask_t *worker_mask = &odp_global_data.worker_cpus; int i; - int retval = -1; - if (!get_installed_cpus()) { - /* CPU 0 is only used for workers on uniprocessor systems */ - if (odp_global_data.num_cpus_installed > 1) - odp_cpumask_clr(worker_mask, 0); + /* (Bits for all available CPUs are SET in control cpumask) */ + + if (worker_cpus_default) { /* + * The worker cpumask was also unspecified... * If only one or two CPUs installed, use CPU 0 for control. * Otherwise leave it for the kernel and start with CPU 1. */ @@ -319,15 +320,163 @@ int odp_cpumask_init_global(void) * reserve remaining CPUs for workers */ odp_cpumask_clr(control_mask, 0); - odp_cpumask_clr(worker_mask, 1); - for (i = 2; i < CPU_SETSIZE; i++) { + for (i = 2; i < odp_global_data.num_cpus_installed; i++) if (odp_cpumask_isset(worker_mask, i)) odp_cpumask_clr(control_mask, i); + } + } else { + /* + * The worker cpumask was specified so first ensure + * the control cpumask does not overlap any worker CPUs + */ + for (i = 0; i < odp_global_data.num_cpus_installed; i++) + if (odp_cpumask_isset(worker_mask, i)) + odp_cpumask_clr(control_mask, i); + + /* + * If only one or two CPUs installed, + * ensure availability of CPU 0 for control threads + */ + if (odp_global_data.num_cpus_installed < 3) { + odp_cpumask_set(control_mask, 0); + odp_cpumask_clr(control_mask, 1); + } else { + /* + * If three or more CPUs installed, + * then use CPU 0 for control threads if + * CPU 1 was allocated for workers - otherwise + * use CPU 1 for control and don't use CPU 0 + */ + if (odp_cpumask_isset(worker_mask, 1)) + odp_cpumask_set(control_mask, 0); + else + odp_cpumask_clr(control_mask, 0); + } + } +} + +/* + * This function creates reasonable default cpumasks for worker tasks + * from the set of CPUs available at boot time. + * This function assumes that the global worker cpumask contains + * a list of all installed CPUs, and that no worker cpumask was specified. + */ +static void init_default_worker_cpumask(int control_cpus_default) +{ + odp_cpumask_t *control_mask = &odp_global_data.control_cpus; + odp_cpumask_t *worker_mask = &odp_global_data.worker_cpus; + int i; + + /* (Bits for all available CPUs are SET in worker cpumask) */ + + if (control_cpus_default) { + /* + * The control cpumask was also unspecified... + * CPU 0 is only used for workers on uniprocessor systems + */ + if (odp_global_data.num_cpus_installed > 1) + odp_cpumask_clr(worker_mask, 0); + + if (odp_global_data.num_cpus_installed > 2) + /* + * If three or more CPUs, reserve CPU 0 for kernel, + * reserve CPU 1 for control, and + * reserve remaining CPUs for workers + */ + odp_cpumask_clr(worker_mask, 1); + } else { + /* + * The control cpumask was specified so first ensure + * the worker cpumask does not overlap any control CPUs + */ + for (i = 0; i < odp_global_data.num_cpus_installed; i++) + if (odp_cpumask_isset(control_mask, i)) + odp_cpumask_clr(worker_mask, i); + + /* + * If only one CPU installed, use CPU 0 for workers + * even though it is used for control as well. + */ + if (odp_global_data.num_cpus_installed < 2) + odp_cpumask_set(worker_mask, 0); + else + odp_cpumask_clr(worker_mask, 0); + } +} + +/* + * This function creates reasonable default cpumasks for control and worker + * tasks from the set of CPUs available at boot time. + * It also allows the default cpumasks to be overridden by + * externally specified cpumasks passed in as initialization parameters. + */ +int odp_cpumask_init_global(const odp_init_t *params) +{ + odp_cpumask_t *control_mask = &odp_global_data.control_cpus; + odp_cpumask_t *worker_mask = &odp_global_data.worker_cpus; + odp_cpumask_t check_mask; + int control_cpus_default = 1; + int worker_cpus_default = 1; + + /* + * Initialize the global control and worker cpumasks with lists of + * all installed CPUs. Return an error if this procedure fails. + */ + if (!get_installed_cpus()) { + if (params) { + if (params->control_cpus) { + /* + * If uninstalled control CPUs were specified, + * then return an error. Otherwise copy the + * specified control cpumask into the global + * control cpumask for later reference. + */ + odp_cpumask_and(&check_mask, control_mask, + params->control_cpus); + if (odp_cpumask_equal(params->control_cpus, + &check_mask)) { + odp_cpumask_copy(control_mask, + params->control_cpus); + control_cpus_default = 0; + } else { + return -1; + } + } + if (params->worker_cpus) { + /* + * If uninstalled worker CPUs were specified, + * then return an error. Otherwise copy the + * specified worker cpumask into the global + * worker cpumask for later reference. + */ + odp_cpumask_and(&check_mask, worker_mask, + params->worker_cpus); + if (odp_cpumask_equal(params->worker_cpus, + &check_mask)) { + odp_cpumask_copy(worker_mask, + params->worker_cpus); + worker_cpus_default = 0; + } else { + return -1; + } } } - retval = 0; + + /* + * Any caller-specified cpumasks have been validated + * and saved. Now fill in any unspecified masks with + * 'best guess' default configurations. + * (Worker mask gets to allocate CPUs before control mask) + */ + if (worker_cpus_default) + init_default_worker_cpumask(control_cpus_default); + if (control_cpus_default) + init_default_control_cpumask(worker_cpus_default); + + return 0; + } else { + return -1; } - return retval; } int odp_cpumask_term_global(void) diff --git a/platform/linux-generic/odp_init.c b/platform/linux-generic/odp_init.c index 9a923e4..71e074d 100644 --- a/platform/linux-generic/odp_init.c +++ b/platform/linux-generic/odp_init.c @@ -24,7 +24,7 @@ int odp_init_global(odp_instance_t *instance, odp_global_data.abort_fn = params->abort_fn; } - if (odp_cpumask_init_global()) { + if (odp_cpumask_init_global(params)) { ODP_ERR("ODP cpumask init failed.\n"); goto init_failed; }