From patchwork Mon Mar 28 16:11:38 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: gary.robertson@linaro.org X-Patchwork-Id: 64541 Delivered-To: patch@linaro.org Received: by 10.112.199.169 with SMTP id jl9csp1532920lbc; Mon, 28 Mar 2016 09:15:37 -0700 (PDT) X-Received: by 10.50.132.39 with SMTP id or7mr10644252igb.38.1459181736951; Mon, 28 Mar 2016 09:15:36 -0700 (PDT) Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id l9si24094273ioe.5.2016.03.28.09.15.36; Mon, 28 Mar 2016 09:15:36 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 40939616FC; Mon, 28 Mar 2016 16:15:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 3D4976184D; Mon, 28 Mar 2016 16:12:33 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 8D1916153C; Mon, 28 Mar 2016 16:12:27 +0000 (UTC) Received: from mail-ob0-f180.google.com (mail-ob0-f180.google.com [209.85.214.180]) by lists.linaro.org (Postfix) with ESMTPS id 5CD2D616FC for ; Mon, 28 Mar 2016 16:11:53 +0000 (UTC) Received: by mail-ob0-f180.google.com with SMTP id m7so102821592obh.3 for ; Mon, 28 Mar 2016 09:11:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=HQoEVJrOg/jWzSLYMJO0ZUa04dWK+TT4bEmqgupr28k=; b=OqtoVFj+yDdv89sr/yLkeqbnY8Gz9c/OAngMXdf9C+55q50uvS4eGj3EpNKsnv5kLE 3SoZMJQbPlkqkajfSzozWFPd4xmRDwx3qtQDOxFMM6XzgYyBQfLk/EW0Ez38N495T2UW ZdKJxpa02e+Ktx53/x2kV/yGlWnDnlf+NAXmXRbi3soDgjqlLbhlOqOfN08v91KMCJSU 02pxJStVDWsEm+vBA00xgU54U1ifpaPzTLfqdmXGscE4VCb0ALbgQYnBiUUfGCZJvtVp 3h3D+8pYC03Bwk5NVfOJ6c3tmiq1y4oxxwZQg3OuWU22wBLyHTTWCMwyASiyJ3d/XVZB oqUg== X-Gm-Message-State: AD7BkJJu6bB0OMe9b3IGF45iFvNhgxCrz3g5ciOzQmz7JYbSvkzQtmb8Mmk12b0o9iEY/GHgvm4= X-Received: by 10.60.132.208 with SMTP id ow16mr4402899oeb.67.1459181512859; Mon, 28 Mar 2016 09:11:52 -0700 (PDT) Received: from honkintosh.cybertech.lan (65-120-133-114.dia.static.qwest.net. [65.120.133.114]) by smtp.googlemail.com with ESMTPSA id 19sm8047972ote.18.2016.03.28.09.11.51 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 28 Mar 2016 09:11:52 -0700 (PDT) From: "Gary S. Robertson" To: mike.holmes@linaro.org, bill.fischofer@linaro.org, maxim.uvarov@linaro.org, anders.roxell@linaro.org, petri.savolainen@linaro.org Date: Mon, 28 Mar 2016 11:11:38 -0500 Message-Id: <1459181498-8134-6-git-send-email-gary.robertson@linaro.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1459181498-8134-1-git-send-email-gary.robertson@linaro.org> References: <1456929250-7720-1-git-send-email-gary.robertson@linaro.org> <1459181498-8134-1-git-send-email-gary.robertson@linaro.org> X-Topics: patch Cc: lng-odp@lists.linaro.org Subject: [lng-odp] [PATCH 5/5] linux-generic: add support for initial cpumasks X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" These code changes depend on the addition of control and worker cpumask pointers to the ODP initialization parameters, and implement the change in behavior suggested with that patch. They serve as the 'glue' between the input of the new ODP API initial cpuset masks and the use of those new cpumasks by the ODP application or instance. These changes allow multiple concurrent ODP instances to be given CPU resources which do not conflict with one another, so multiple ODP instances can coexist harmoniously with any isolation support on the underlying platform as well as with one another. Specifically: if either of the cpumask pointers are NULL when calling odp_init_global(), then that function creates 'best-guess' default values for the unspecified cpumasks and saves those defaults into global variables for later reference. However, if the cpumasks are valid and their pointers initialized prior to calling odp_init_global() then that routine saves their contents into the global variables instead of the default cpumasks. When odp_cpumask_default_control() or odp_cpumask_default_worker() are called they build the caller's cpumasks based on the saved contents of the global cpumask variables. Signed-off-by: Gary S. Robertson --- platform/linux-generic/include/odp_internal.h | 2 +- platform/linux-generic/odp_cpumask.c | 177 ++++++++++++++++++++++++-- platform/linux-generic/odp_init.c | 2 +- 3 files changed, 165 insertions(+), 16 deletions(-) diff --git a/platform/linux-generic/include/odp_internal.h b/platform/linux-generic/include/odp_internal.h index 567b1fb..ef165ac 100644 --- a/platform/linux-generic/include/odp_internal.h +++ b/platform/linux-generic/include/odp_internal.h @@ -69,7 +69,7 @@ extern struct odp_global_data_s odp_global_data; int _odp_term_global(enum init_stage stage); int _odp_term_local(enum init_stage stage); -int odp_cpumask_init_global(void); +int odp_cpumask_init_global(const odp_init_t *params); int odp_cpumask_term_global(void); int odp_system_info_init(void); diff --git a/platform/linux-generic/odp_cpumask.c b/platform/linux-generic/odp_cpumask.c index 320ca8e..fd672e2 100644 --- a/platform/linux-generic/odp_cpumask.c +++ b/platform/linux-generic/odp_cpumask.c @@ -10,6 +10,7 @@ #include #include +#include #include #include @@ -215,8 +216,7 @@ int odp_cpumask_next(const odp_cpumask_t *mask, int cpu) /* * This function obtains system information specifying which cpus are - * available at boot time. These data are then used to produce cpumasks of - * configured CPUs without concern over isolation support. + * available at boot time. */ static int get_installed_cpus(void) { @@ -288,21 +288,22 @@ static int get_installed_cpus(void) } /* - * This function creates reasonable default cpumasks for control and worker - * tasks from the set of CPUs available at boot time. + * This function creates reasonable default cpumasks for control tasks + * from the set of CPUs available at boot time. + * This function assumes that the global control cpumask contains + * a list of all installed CPUs, and that no control cpumask was specified. */ -int odp_cpumask_init_global(void) +static void init_default_control_cpumask(int worker_cpus_default) { odp_cpumask_t *control_mask = &odp_global_data.control_cpus; odp_cpumask_t *worker_mask = &odp_global_data.worker_cpus; int i; - int retval = -1; - if (!get_installed_cpus()) { - /* CPU 0 is only used for workers on uniprocessor systems */ - if (odp_global_data.num_cpus_installed > 1) - odp_cpumask_clr(worker_mask, 0); + /* (Bits for all available CPUs are SET in control cpumask) */ + + if (worker_cpus_default) { /* + * The worker cpumask was also unspecified... * If only one or two CPUs installed, use CPU 0 for control. * Otherwise leave it for the kernel and start with CPU 1. */ @@ -319,15 +320,163 @@ int odp_cpumask_init_global(void) * reserve remaining CPUs for workers */ odp_cpumask_clr(control_mask, 0); - odp_cpumask_clr(worker_mask, 1); - for (i = 2; i < CPU_SETSIZE; i++) { + for (i = 2; i < odp_global_data.num_cpus_installed; i++) if (odp_cpumask_isset(worker_mask, i)) odp_cpumask_clr(control_mask, i); + } + } else { + /* + * The worker cpumask was specified so first ensure + * the control cpumask does not overlap any worker CPUs + */ + for (i = 0; i < odp_global_data.num_cpus_installed; i++) + if (odp_cpumask_isset(worker_mask, i)) + odp_cpumask_clr(control_mask, i); + + /* + * If only one or two CPUs installed, + * ensure availability of CPU 0 for control threads + */ + if (odp_global_data.num_cpus_installed < 3) { + odp_cpumask_set(control_mask, 0); + odp_cpumask_clr(control_mask, 1); + } else { + /* + * If three or more CPUs installed, + * then use CPU 0 for control threads if + * CPU 1 was allocated for workers - otherwise + * use CPU 1 for control and don't use CPU 0 + */ + if (odp_cpumask_isset(worker_mask, 1)) + odp_cpumask_set(control_mask, 0); + else + odp_cpumask_clr(control_mask, 0); + } + } +} + +/* + * This function creates reasonable default cpumasks for worker tasks + * from the set of CPUs available at boot time. + * This function assumes that the global worker cpumask contains + * a list of all installed CPUs, and that no worker cpumask was specified. + */ +static void init_default_worker_cpumask(int control_cpus_default) +{ + odp_cpumask_t *control_mask = &odp_global_data.control_cpus; + odp_cpumask_t *worker_mask = &odp_global_data.worker_cpus; + int i; + + /* (Bits for all available CPUs are SET in worker cpumask) */ + + if (control_cpus_default) { + /* + * The control cpumask was also unspecified... + * CPU 0 is only used for workers on uniprocessor systems + */ + if (odp_global_data.num_cpus_installed > 1) + odp_cpumask_clr(worker_mask, 0); + + if (odp_global_data.num_cpus_installed > 2) + /* + * If three or more CPUs, reserve CPU 0 for kernel, + * reserve CPU 1 for control, and + * reserve remaining CPUs for workers + */ + odp_cpumask_clr(worker_mask, 1); + } else { + /* + * The control cpumask was specified so first ensure + * the worker cpumask does not overlap any control CPUs + */ + for (i = 0; i < odp_global_data.num_cpus_installed; i++) + if (odp_cpumask_isset(control_mask, i)) + odp_cpumask_clr(worker_mask, i); + + /* + * If only one CPU installed, use CPU 0 for workers + * even though it is used for control as well. + */ + if (odp_global_data.num_cpus_installed < 2) + odp_cpumask_set(worker_mask, 0); + else + odp_cpumask_clr(worker_mask, 0); + } +} + +/* + * This function creates reasonable default cpumasks for control and worker + * tasks from the set of CPUs available at boot time. + * It also allows the default cpumasks to be overridden by + * externally specified cpumasks passed in as initialization parameters. + */ +int odp_cpumask_init_global(const odp_init_t *params) +{ + odp_cpumask_t *control_mask = &odp_global_data.control_cpus; + odp_cpumask_t *worker_mask = &odp_global_data.worker_cpus; + odp_cpumask_t check_mask; + int control_cpus_default = 1; + int worker_cpus_default = 1; + + /* + * Initialize the global control and worker cpumasks with lists of + * all installed CPUs. Return an error if this procedure fails. + */ + if (!get_installed_cpus()) { + if (params) { + if (params->control_cpus) { + /* + * If uninstalled control CPUs were specified, + * then return an error. Otherwise copy the + * specified control cpumask into the global + * control cpumask for later reference. + */ + odp_cpumask_and(&check_mask, control_mask, + params->control_cpus); + if (odp_cpumask_equal(params->control_cpus, + &check_mask)) { + odp_cpumask_copy(control_mask, + params->control_cpus); + control_cpus_default = 0; + } else { + return -1; + } + } + if (params->worker_cpus) { + /* + * If uninstalled worker CPUs were specified, + * then return an error. Otherwise copy the + * specified worker cpumask into the global + * worker cpumask for later reference. + */ + odp_cpumask_and(&check_mask, worker_mask, + params->worker_cpus); + if (odp_cpumask_equal(params->worker_cpus, + &check_mask)) { + odp_cpumask_copy(worker_mask, + params->worker_cpus); + worker_cpus_default = 0; + } else { + return -1; + } } } - retval = 0; + + /* + * Any caller-specified cpumasks have been validated + * and saved. Now fill in any unspecified masks with + * 'best guess' default configurations. + * (Worker mask gets to allocate CPUs before control mask) + */ + if (worker_cpus_default) + init_default_worker_cpumask(control_cpus_default); + if (control_cpus_default) + init_default_control_cpumask(worker_cpus_default); + + return 0; + } else { + return -1; } - return retval; } int odp_cpumask_term_global(void) diff --git a/platform/linux-generic/odp_init.c b/platform/linux-generic/odp_init.c index d1b86ad..01e16c0 100644 --- a/platform/linux-generic/odp_init.c +++ b/platform/linux-generic/odp_init.c @@ -24,7 +24,7 @@ int odp_init_global(odp_instance_t *instance, odp_global_data.abort_fn = params->abort_fn; } - if (odp_cpumask_init_global()) { + if (odp_cpumask_init_global(params)) { ODP_ERR("ODP cpumask init failed.\n"); goto init_failed; }