diff mbox

[2/2] isolation: disable default odp cpuset

Message ID 1408347705-22886-1-git-send-email-santosh.shukla@linaro.org
State New
Headers show

Commit Message

Santosh Shukla Aug. 18, 2014, 7:41 a.m. UTC 
- Disable default odp thread cpuset. Use is-cpu-isolated.sh script to create
  cpuset for dplane, cplane for odp.
- Use odp-on-isolated-cpu.sh script to test this application.
- test script location [1]

- Refer readme for instruction to use.

[1]
https://git.linaro.org/people/santosh.shukla/test-definitions.git/shortlog/refs/heads/isol-v2

Signed-off-by: Santosh Shukla <santosh.shukla@linaro.org>
---
 example/isolation/odp_isolation.c  |    5 +++++
 example/isolation/readme           |   20 ++++++++++++++++++++
 platform/linux-generic/odp_linux.c |   12 ++++++++++--
 3 files changed, 35 insertions(+), 2 deletions(-)
 create mode 100644 example/isolation/readme

Comments

Stuart Haslam Aug. 18, 2014, 11 a.m. UTC | #1 
On Mon, Aug 18, 2014 at 08:41:45AM +0100, Santosh Shukla wrote:
> - Disable default odp thread cpuset. Use is-cpu-isolated.sh script to create
>   cpuset for dplane, cplane for odp.
> - Use odp-on-isolated-cpu.sh script to test this application.
> - test script location [1]
> 
> - Refer readme for instruction to use.
> 
> [1]
> https://git.linaro.org/people/santosh.shukla/test-definitions.git/shortlog/refs/heads/isol-v2
> 
> Signed-off-by: Santosh Shukla <santosh.shukla@linaro.org>
> ---
>  example/isolation/odp_isolation.c  |    5 +++++
>  example/isolation/readme           |   20 ++++++++++++++++++++
>  platform/linux-generic/odp_linux.c |   12 ++++++++++--
>  3 files changed, 35 insertions(+), 2 deletions(-)
>  create mode 100644 example/isolation/readme
> 
> diff --git a/example/isolation/odp_isolation.c b/example/isolation/odp_isolation.c
> index 8d8fffc..1ade4c7 100644
> --- a/example/isolation/odp_isolation.c
> +++ b/example/isolation/odp_isolation.c
> @@ -214,6 +214,11 @@ int main(int argc, char *argv[])
>  	thr_id = odp_thread_create(0);
>  	odp_init_local(thr_id);
>  
> +	/*
> +	 * create dummy dp thread and don't pin them on any core,
> +	 * use odp-on-isolated-cpu.sh script to define dplane cpuset
> +	 * and to pin, run app on them, Refer readme.
> +	 */
>  	for (i=0; i<num_workers; i++) {
>  
>  		first_core = atoi(args.core_name[i]);
> diff --git a/example/isolation/readme b/example/isolation/readme
> new file mode 100644
> index 0000000..d49d61a
> --- /dev/null
> +++ b/example/isolation/readme
> @@ -0,0 +1,20 @@
> +
> +README on How-to run odp app on isolated core for no_hz_full kernel mode.
> +
> +- Disbale timer init in odp application.
> +- avoid using odp's cpuset. Let odp thread launch unpinned.
> +- use odp-on-isolated-cpu.sh script to pin odp dp thread to
> + dplane cpuset.
> +
> +- download my local test-definition repo link [1]
> +- copy odp_isolation binary to /usr/local/bin
> +- run  ./common/scripts/odp-on-isolated-cpu.sh
> +Or
> +-  ./common/scripts/odp-on-isolated-cpu.sh 1,2 "odp_isolation -l 1,2"
> +- Above script shows isolation duration, before / after interrupt count.
> +
> +Known issue :
> +- x86 hw generates spurious ipi lead to break no_hz_full isolation (todo).
> +
> +[1] https://git.linaro.org/people/santosh.shukla/test-definitions.git/shortlog/refs/heads/isol-v2
> +
> diff --git a/platform/linux-generic/odp_linux.c b/platform/linux-generic/odp_linux.c
> index 6e2b448..7e853c7 100644
> --- a/platform/linux-generic/odp_linux.c
> +++ b/platform/linux-generic/odp_linux.c
> @@ -22,6 +22,11 @@
>  #include <odp_debug.h>
>  
>  
> +#define NO_HZ_FULL_ISOL /*
> +			 * don't use odp cpuset.
> +			 * enable this for odp isolation mode
> +			 */
> +
>  typedef struct {
>  	int thr_id;
>  	void *(*start_routine) (void *);
> @@ -49,7 +54,9 @@ void odp_linux_pthread_create(odp_linux_pthread_t *thread_tbl, int num,
>  		int first_core, void *(*start_routine) (void *), void *arg)
>  {
>  	int i;
> +#ifndef NO_HZ_FULL_ISOL
>  	cpu_set_t cpu_set;
> +#endif
>  	odp_start_args_t *start_args;
>  	int core_count;
>  	int cpu;
> @@ -62,16 +69,17 @@ void odp_linux_pthread_create(odp_linux_pthread_t *thread_tbl, int num,
>  	memset(thread_tbl, 0, num * sizeof(odp_linux_pthread_t));
>  
>  	for (i = 0; i < num; i++) {
> +		cpu = (first_core + i) % core_count;
> +#ifndef NO_HZ_FULL_ISOL
>  		pthread_attr_init(&thread_tbl[i].attr);
>  
>  		CPU_ZERO(&cpu_set);
>  
> -		cpu = (first_core + i) % core_count;
>  		CPU_SET(cpu, &cpu_set);
>  
>  		pthread_attr_setaffinity_np(&thread_tbl[i].attr,
>  					    sizeof(cpu_set_t), &cpu_set);
> -
> +#endif

There's no need for this to be a compile time decision - this changes
the behaviour of all applications linked against this library.

I would rather leave the affinity setting in odp_linux_pthread_create
and let the script just ensure that the specified cores are isolated. Is
there anything actually preventing that from being possible?

As it is, what happens if you build with NO_HZ_FULL_ISOL but don't run
the application via the odp-on-isolated-cpu.sh script? and if you do run
via the script, what's happening during the time between the application
launching the threads and the script moving those threads onto the
correct cores?
Santosh Shukla Aug. 18, 2014, 11:32 a.m. UTC | #2 
On 18 August 2014 16:30, Stuart Haslam <stuart.haslam@arm.com> wrote:
> On Mon, Aug 18, 2014 at 08:41:45AM +0100, Santosh Shukla wrote:
>> - Disable default odp thread cpuset. Use is-cpu-isolated.sh script to create
>>   cpuset for dplane, cplane for odp.
>> - Use odp-on-isolated-cpu.sh script to test this application.
>> - test script location [1]
>>
>> - Refer readme for instruction to use.
>>
>> [1]
>> https://git.linaro.org/people/santosh.shukla/test-definitions.git/shortlog/refs/heads/isol-v2
>>
>> Signed-off-by: Santosh Shukla <santosh.shukla@linaro.org>
>> ---
>>  example/isolation/odp_isolation.c  |    5 +++++
>>  example/isolation/readme           |   20 ++++++++++++++++++++
>>  platform/linux-generic/odp_linux.c |   12 ++++++++++--
>>  3 files changed, 35 insertions(+), 2 deletions(-)
>>  create mode 100644 example/isolation/readme
>>
>> diff --git a/example/isolation/odp_isolation.c b/example/isolation/odp_isolation.c
>> index 8d8fffc..1ade4c7 100644
>> --- a/example/isolation/odp_isolation.c
>> +++ b/example/isolation/odp_isolation.c
>> @@ -214,6 +214,11 @@ int main(int argc, char *argv[])
>>       thr_id = odp_thread_create(0);
>>       odp_init_local(thr_id);
>>
>> +     /*
>> +      * create dummy dp thread and don't pin them on any core,
>> +      * use odp-on-isolated-cpu.sh script to define dplane cpuset
>> +      * and to pin, run app on them, Refer readme.
>> +      */
>>       for (i=0; i<num_workers; i++) {
>>
>>               first_core = atoi(args.core_name[i]);
>> diff --git a/example/isolation/readme b/example/isolation/readme
>> new file mode 100644
>> index 0000000..d49d61a
>> --- /dev/null
>> +++ b/example/isolation/readme
>> @@ -0,0 +1,20 @@
>> +
>> +README on How-to run odp app on isolated core for no_hz_full kernel mode.
>> +
>> +- Disbale timer init in odp application.
>> +- avoid using odp's cpuset. Let odp thread launch unpinned.
>> +- use odp-on-isolated-cpu.sh script to pin odp dp thread to
>> + dplane cpuset.
>> +
>> +- download my local test-definition repo link [1]
>> +- copy odp_isolation binary to /usr/local/bin
>> +- run  ./common/scripts/odp-on-isolated-cpu.sh
>> +Or
>> +-  ./common/scripts/odp-on-isolated-cpu.sh 1,2 "odp_isolation -l 1,2"
>> +- Above script shows isolation duration, before / after interrupt count.
>> +
>> +Known issue :
>> +- x86 hw generates spurious ipi lead to break no_hz_full isolation (todo).
>> +
>> +[1] https://git.linaro.org/people/santosh.shukla/test-definitions.git/shortlog/refs/heads/isol-v2
>> +
>> diff --git a/platform/linux-generic/odp_linux.c b/platform/linux-generic/odp_linux.c
>> index 6e2b448..7e853c7 100644
>> --- a/platform/linux-generic/odp_linux.c
>> +++ b/platform/linux-generic/odp_linux.c
>> @@ -22,6 +22,11 @@
>>  #include <odp_debug.h>
>>
>>
>> +#define NO_HZ_FULL_ISOL /*
>> +                      * don't use odp cpuset.
>> +                      * enable this for odp isolation mode
>> +                      */
>> +
>>  typedef struct {
>>       int thr_id;
>>       void *(*start_routine) (void *);
>> @@ -49,7 +54,9 @@ void odp_linux_pthread_create(odp_linux_pthread_t *thread_tbl, int num,
>>               int first_core, void *(*start_routine) (void *), void *arg)
>>  {
>>       int i;
>> +#ifndef NO_HZ_FULL_ISOL
>>       cpu_set_t cpu_set;
>> +#endif
>>       odp_start_args_t *start_args;
>>       int core_count;
>>       int cpu;
>> @@ -62,16 +69,17 @@ void odp_linux_pthread_create(odp_linux_pthread_t *thread_tbl, int num,
>>       memset(thread_tbl, 0, num * sizeof(odp_linux_pthread_t));
>>
>>       for (i = 0; i < num; i++) {
>> +             cpu = (first_core + i) % core_count;
>> +#ifndef NO_HZ_FULL_ISOL
>>               pthread_attr_init(&thread_tbl[i].attr);
>>
>>               CPU_ZERO(&cpu_set);
>>
>> -             cpu = (first_core + i) % core_count;
>>               CPU_SET(cpu, &cpu_set);
>>
>>               pthread_attr_setaffinity_np(&thread_tbl[i].attr,
>>                                           sizeof(cpu_set_t), &cpu_set);
>> -
>> +#endif
>
> There's no need for this to be a compile time decision - this changes
> the behaviour of all applications linked against this library.
>

Agree, There might be better way to avoid linkage issue.

> I would rather leave the affinity setting in odp_linux_pthread_create
> and let the script just ensure that the specified cores are isolated. Is
> there anything actually preventing that from being possible?

Yes, Isolation is a specific requirement to application. current odp
pins thread to core in order which not necessarily be the case all the
time.

A  Flexibility to pin any of odp thread to any core is desired choice.
Therefore I had to break free from existing pinning assumption.
Keeping two set of cpuset - One is default ODP whome thread pinned
with then Second one created by isolation script - doesn't make sense
to me. Also unpinning a thread from odp cpuset group to script cpuset
group gives system error.

>
> As it is, what happens if you build with NO_HZ_FULL_ISOL but don't run
> the application via the odp-on-isolated-cpu.sh script? and if you do run

There is a readme, User must refer that in case he is looking for isolation.

> via the script, what's happening during the time between the application
> launching the threads and the script moving those threads onto the
> correct cores?

It is order, first application spwans a thread then script moves those
thread to isol cores. Therefore it should work.
>
> --
> Stuart.
>
Stuart Haslam Aug. 18, 2014, 2:12 p.m. UTC | #3 
On Mon, Aug 18, 2014 at 12:32:36PM +0100, Santosh Shukla wrote:
> On 18 August 2014 16:30, Stuart Haslam <stuart.haslam@arm.com> wrote:
> > On Mon, Aug 18, 2014 at 08:41:45AM +0100, Santosh Shukla wrote:
> >> - Disable default odp thread cpuset. Use is-cpu-isolated.sh script to create
> >>   cpuset for dplane, cplane for odp.
> >> - Use odp-on-isolated-cpu.sh script to test this application.
> >> - test script location [1]
> >>
> >> - Refer readme for instruction to use.
> >>
> >> [1]
> >> https://git.linaro.org/people/santosh.shukla/test-definitions.git/shortlog/refs/heads/isol-v2
> >>
> >> Signed-off-by: Santosh Shukla <santosh.shukla@linaro.org>
> >> ---
> >>  example/isolation/odp_isolation.c  |    5 +++++
> >>  example/isolation/readme           |   20 ++++++++++++++++++++
> >>  platform/linux-generic/odp_linux.c |   12 ++++++++++--
> >>  3 files changed, 35 insertions(+), 2 deletions(-)
> >>  create mode 100644 example/isolation/readme
> >>
> >> diff --git a/example/isolation/odp_isolation.c b/example/isolation/odp_isolation.c
> >> index 8d8fffc..1ade4c7 100644
> >> --- a/example/isolation/odp_isolation.c
> >> +++ b/example/isolation/odp_isolation.c
> >> @@ -214,6 +214,11 @@ int main(int argc, char *argv[])
> >>       thr_id = odp_thread_create(0);
> >>       odp_init_local(thr_id);
> >>
> >> +     /*
> >> +      * create dummy dp thread and don't pin them on any core,
> >> +      * use odp-on-isolated-cpu.sh script to define dplane cpuset
> >> +      * and to pin, run app on them, Refer readme.
> >> +      */
> >>       for (i=0; i<num_workers; i++) {
> >>
> >>               first_core = atoi(args.core_name[i]);
> >> diff --git a/example/isolation/readme b/example/isolation/readme
> >> new file mode 100644
> >> index 0000000..d49d61a
> >> --- /dev/null
> >> +++ b/example/isolation/readme
> >> @@ -0,0 +1,20 @@
> >> +
> >> +README on How-to run odp app on isolated core for no_hz_full kernel mode.
> >> +
> >> +- Disbale timer init in odp application.
> >> +- avoid using odp's cpuset. Let odp thread launch unpinned.
> >> +- use odp-on-isolated-cpu.sh script to pin odp dp thread to
> >> + dplane cpuset.
> >> +
> >> +- download my local test-definition repo link [1]
> >> +- copy odp_isolation binary to /usr/local/bin
> >> +- run  ./common/scripts/odp-on-isolated-cpu.sh
> >> +Or
> >> +-  ./common/scripts/odp-on-isolated-cpu.sh 1,2 "odp_isolation -l 1,2"
> >> +- Above script shows isolation duration, before / after interrupt count.
> >> +
> >> +Known issue :
> >> +- x86 hw generates spurious ipi lead to break no_hz_full isolation (todo).
> >> +
> >> +[1] https://git.linaro.org/people/santosh.shukla/test-definitions.git/shortlog/refs/heads/isol-v2
> >> +
> >> diff --git a/platform/linux-generic/odp_linux.c b/platform/linux-generic/odp_linux.c
> >> index 6e2b448..7e853c7 100644
> >> --- a/platform/linux-generic/odp_linux.c
> >> +++ b/platform/linux-generic/odp_linux.c
> >> @@ -22,6 +22,11 @@
> >>  #include <odp_debug.h>
> >>
> >>
> >> +#define NO_HZ_FULL_ISOL /*
> >> +                      * don't use odp cpuset.
> >> +                      * enable this for odp isolation mode
> >> +                      */
> >> +
> >>  typedef struct {
> >>       int thr_id;
> >>       void *(*start_routine) (void *);
> >> @@ -49,7 +54,9 @@ void odp_linux_pthread_create(odp_linux_pthread_t *thread_tbl, int num,
> >>               int first_core, void *(*start_routine) (void *), void *arg)
> >>  {
> >>       int i;
> >> +#ifndef NO_HZ_FULL_ISOL
> >>       cpu_set_t cpu_set;
> >> +#endif
> >>       odp_start_args_t *start_args;
> >>       int core_count;
> >>       int cpu;
> >> @@ -62,16 +69,17 @@ void odp_linux_pthread_create(odp_linux_pthread_t *thread_tbl, int num,
> >>       memset(thread_tbl, 0, num * sizeof(odp_linux_pthread_t));
> >>
> >>       for (i = 0; i < num; i++) {
> >> +             cpu = (first_core + i) % core_count;
> >> +#ifndef NO_HZ_FULL_ISOL
> >>               pthread_attr_init(&thread_tbl[i].attr);
> >>
> >>               CPU_ZERO(&cpu_set);
> >>
> >> -             cpu = (first_core + i) % core_count;
> >>               CPU_SET(cpu, &cpu_set);
> >>
> >>               pthread_attr_setaffinity_np(&thread_tbl[i].attr,
> >>                                           sizeof(cpu_set_t), &cpu_set);
> >> -
> >> +#endif
> >
> > There's no need for this to be a compile time decision - this changes
> > the behaviour of all applications linked against this library.
> >
> 
> Agree, There might be better way to avoid linkage issue.
> 
> > I would rather leave the affinity setting in odp_linux_pthread_create
> > and let the script just ensure that the specified cores are isolated. Is
> > there anything actually preventing that from being possible?
> 
> Yes, Isolation is a specific requirement to application. current odp
> pins thread to core in order which not necessarily be the case all the
> time.
> 
> A  Flexibility to pin any of odp thread to any core is desired choice.
> Therefore I had to break free from existing pinning assumption.

Hence my suggestion in the other thread about changing odp_linux_pthread_create.
If there's a limitation in the existing ODP API/implementation/apps then
it should be fixed rather than working around it.

> Keeping two set of cpuset - One is default ODP whome thread pinned
> with then Second one created by isolation script - doesn't make sense
> to me.

The two cpusets do different things (the cpusets created via /dev/cpuset
aren't related to the CPU_SET used with pthread_attr_setaffinity_np,
other than having a confusingly similar name).

The cpuset created by the script specifies a group of cores the
application is permitted to run threads on (and ensures those cores are
isolated). The pthread_attr_setaffinity_np in the application specifies
a specific core within that group for a particular thread to run on.

So I still don't see what the problem is with the problem is.

> Also unpinning a thread from odp cpuset group to script cpuset
> group gives system error.

Not sure what you mean exactly, but if the ODP application tries to
setaffinity to core that's not in the cpuset created by the script I
would expect that to fail. That's OK though as the list of cpus passed
to the script should be the same as that passed to the application,
anything else is an error.

> >
> > As it is, what happens if you build with NO_HZ_FULL_ISOL but don't run
> > the application via the odp-on-isolated-cpu.sh script? and if you do run
> 
> There is a readme, User must refer that in case he is looking for isolation.
> 
> > via the script, what's happening during the time between the application
> > launching the threads and the script moving those threads onto the
> > correct cores?
> 
> It is order, first application spwans a thread then script moves those
> thread to isol cores. Therefore it should work.

Yes but there is a window in between where they could all be running on
the same core. How long does it take for them to be moved to the correct
core? how does the application know when it's been done?.. there's no
need for the uncertainty when the application can set its own affinity.
Santosh Shukla Aug. 18, 2014, 4:26 p.m. UTC | #4 
On 18 August 2014 19:42, Stuart Haslam <stuart.haslam@arm.com> wrote:
> On Mon, Aug 18, 2014 at 12:32:36PM +0100, Santosh Shukla wrote:
>> On 18 August 2014 16:30, Stuart Haslam <stuart.haslam@arm.com> wrote:
>> > On Mon, Aug 18, 2014 at 08:41:45AM +0100, Santosh Shukla wrote:
>> >> - Disable default odp thread cpuset. Use is-cpu-isolated.sh script to create
>> >>   cpuset for dplane, cplane for odp.
>> >> - Use odp-on-isolated-cpu.sh script to test this application.
>> >> - test script location [1]
>> >>
>> >> - Refer readme for instruction to use.
>> >>
>> >> [1]
>> >> https://git.linaro.org/people/santosh.shukla/test-definitions.git/shortlog/refs/heads/isol-v2
>> >>
>> >> Signed-off-by: Santosh Shukla <santosh.shukla@linaro.org>
>> >> ---
>> >>  example/isolation/odp_isolation.c  |    5 +++++
>> >>  example/isolation/readme           |   20 ++++++++++++++++++++
>> >>  platform/linux-generic/odp_linux.c |   12 ++++++++++--
>> >>  3 files changed, 35 insertions(+), 2 deletions(-)
>> >>  create mode 100644 example/isolation/readme
>> >>
>> >> diff --git a/example/isolation/odp_isolation.c b/example/isolation/odp_isolation.c
>> >> index 8d8fffc..1ade4c7 100644
>> >> --- a/example/isolation/odp_isolation.c
>> >> +++ b/example/isolation/odp_isolation.c
>> >> @@ -214,6 +214,11 @@ int main(int argc, char *argv[])
>> >>       thr_id = odp_thread_create(0);
>> >>       odp_init_local(thr_id);
>> >>
>> >> +     /*
>> >> +      * create dummy dp thread and don't pin them on any core,
>> >> +      * use odp-on-isolated-cpu.sh script to define dplane cpuset
>> >> +      * and to pin, run app on them, Refer readme.
>> >> +      */
>> >>       for (i=0; i<num_workers; i++) {
>> >>
>> >>               first_core = atoi(args.core_name[i]);
>> >> diff --git a/example/isolation/readme b/example/isolation/readme
>> >> new file mode 100644
>> >> index 0000000..d49d61a
>> >> --- /dev/null
>> >> +++ b/example/isolation/readme
>> >> @@ -0,0 +1,20 @@
>> >> +
>> >> +README on How-to run odp app on isolated core for no_hz_full kernel mode.
>> >> +
>> >> +- Disbale timer init in odp application.
>> >> +- avoid using odp's cpuset. Let odp thread launch unpinned.
>> >> +- use odp-on-isolated-cpu.sh script to pin odp dp thread to
>> >> + dplane cpuset.
>> >> +
>> >> +- download my local test-definition repo link [1]
>> >> +- copy odp_isolation binary to /usr/local/bin
>> >> +- run  ./common/scripts/odp-on-isolated-cpu.sh
>> >> +Or
>> >> +-  ./common/scripts/odp-on-isolated-cpu.sh 1,2 "odp_isolation -l 1,2"
>> >> +- Above script shows isolation duration, before / after interrupt count.
>> >> +
>> >> +Known issue :
>> >> +- x86 hw generates spurious ipi lead to break no_hz_full isolation (todo).
>> >> +
>> >> +[1] https://git.linaro.org/people/santosh.shukla/test-definitions.git/shortlog/refs/heads/isol-v2
>> >> +
>> >> diff --git a/platform/linux-generic/odp_linux.c b/platform/linux-generic/odp_linux.c
>> >> index 6e2b448..7e853c7 100644
>> >> --- a/platform/linux-generic/odp_linux.c
>> >> +++ b/platform/linux-generic/odp_linux.c
>> >> @@ -22,6 +22,11 @@
>> >>  #include <odp_debug.h>
>> >>
>> >>
>> >> +#define NO_HZ_FULL_ISOL /*
>> >> +                      * don't use odp cpuset.
>> >> +                      * enable this for odp isolation mode
>> >> +                      */
>> >> +
>> >>  typedef struct {
>> >>       int thr_id;
>> >>       void *(*start_routine) (void *);
>> >> @@ -49,7 +54,9 @@ void odp_linux_pthread_create(odp_linux_pthread_t *thread_tbl, int num,
>> >>               int first_core, void *(*start_routine) (void *), void *arg)
>> >>  {
>> >>       int i;
>> >> +#ifndef NO_HZ_FULL_ISOL
>> >>       cpu_set_t cpu_set;
>> >> +#endif
>> >>       odp_start_args_t *start_args;
>> >>       int core_count;
>> >>       int cpu;
>> >> @@ -62,16 +69,17 @@ void odp_linux_pthread_create(odp_linux_pthread_t *thread_tbl, int num,
>> >>       memset(thread_tbl, 0, num * sizeof(odp_linux_pthread_t));
>> >>
>> >>       for (i = 0; i < num; i++) {
>> >> +             cpu = (first_core + i) % core_count;
>> >> +#ifndef NO_HZ_FULL_ISOL
>> >>               pthread_attr_init(&thread_tbl[i].attr);
>> >>
>> >>               CPU_ZERO(&cpu_set);
>> >>
>> >> -             cpu = (first_core + i) % core_count;
>> >>               CPU_SET(cpu, &cpu_set);
>> >>
>> >>               pthread_attr_setaffinity_np(&thread_tbl[i].attr,
>> >>                                           sizeof(cpu_set_t), &cpu_set);
>> >> -
>> >> +#endif
>> >
>> > There's no need for this to be a compile time decision - this changes
>> > the behaviour of all applications linked against this library.
>> >
>>
>> Agree, There might be better way to avoid linkage issue.
>>
>> > I would rather leave the affinity setting in odp_linux_pthread_create
>> > and let the script just ensure that the specified cores are isolated. Is
>> > there anything actually preventing that from being possible?
>>
>> Yes, Isolation is a specific requirement to application. current odp
>> pins thread to core in order which not necessarily be the case all the
>> time.
>>
>> A  Flexibility to pin any of odp thread to any core is desired choice.
>> Therefore I had to break free from existing pinning assumption.
>
> Hence my suggestion in the other thread about changing odp_linux_pthread_create.
> If there's a limitation in the existing ODP API/implementation/apps then
> it should be fixed rather than working around it.
>
>> Keeping two set of cpuset - One is default ODP whome thread pinned
>> with then Second one created by isolation script - doesn't make sense
>> to me.
>
> The two cpusets do different things (the cpusets created via /dev/cpuset
> aren't related to the CPU_SET used with pthread_attr_setaffinity_np,
> other than having a confusingly similar name).
>
> The cpuset created by the script specifies a group of cores the
> application is permitted to run threads on (and ensures those cores are
> isolated). The pthread_attr_setaffinity_np in the application specifies
> a specific core within that group for a particular thread to run on.
>
> So I still don't see what the problem is with the problem is.
>
>> Also unpinning a thread from odp cpuset group to script cpuset
>> group gives system error.
>
> Not sure what you mean exactly, but if the ODP application tries to
> setaffinity to core that's not in the cpuset created by the script I
> would expect that to fail. That's OK though as the list of cpus passed
> to the script should be the same as that passed to the application,
> anything else is an error.

Yes that was a problem which I wanted to avoid therefore choose to
mask off odp thread affinity part. Also it doesn't make sense to move
thread from odp cpuset to /dev/cpuset to me. However your concern is
valid in favor of keeping odp generic so I'll revisit my changes.

Thanks.

>
>> >
>> > As it is, what happens if you build with NO_HZ_FULL_ISOL but don't run
>> > the application via the odp-on-isolated-cpu.sh script? and if you do run
>>
>> There is a readme, User must refer that in case he is looking for isolation.
>>
>> > via the script, what's happening during the time between the application
>> > launching the threads and the script moving those threads onto the
>> > correct cores?
>>
>> It is order, first application spwans a thread then script moves those
>> thread to isol cores. Therefore it should work.
>
> Yes but there is a window in between where they could all be running on
> the same core.
yes, odp init time.

> How long does it take for them to be moved to the correct
> core?
In case of l2fwd - I am migrating thread just after thread
initialization.. no real IO started on those threads..So its fairly
quick for me. Are you asking about a use-case where dpthread spawned,
doing IO then migrated to other core by script?

> how does the application know when it's been done?..

I verify this way (command based approach)
ps -Leo pid,tid,psr,pcpu,comm | grep odp
# 23025  23025   0 99.9 odp_l2fwd
# 23025  23026   7  0.0 odp_l2fwd
# 23025  23027   1 99.9 odp_l2fwd
# 23025  23028   2  100 odp_l2fwd

Last 2 threads (dpthread) moved to isolated core as per my script instruction.

> there's no
> need for the uncertainty when the application can set its own affinity.

Right.
>
> --
> Stuart.
>
Stuart Haslam Aug. 18, 2014, 5:38 p.m. UTC | #5 
On Mon, Aug 18, 2014 at 05:26:56PM +0100, Santosh Shukla wrote:
> On 18 August 2014 19:42, Stuart Haslam <stuart.haslam@arm.com> wrote:
> > On Mon, Aug 18, 2014 at 12:32:36PM +0100, Santosh Shukla wrote:
> >> On 18 August 2014 16:30, Stuart Haslam <stuart.haslam@arm.com> wrote:
> >> > On Mon, Aug 18, 2014 at 08:41:45AM +0100, Santosh Shukla wrote:
> >> >> - Disable default odp thread cpuset. Use is-cpu-isolated.sh script to create
> >> >>   cpuset for dplane, cplane for odp.
> >> >> - Use odp-on-isolated-cpu.sh script to test this application.
> >> >> - test script location [1]
> >> >>
> >> >> - Refer readme for instruction to use.
> >> >>
> >> >> [1]
> >> >> https://git.linaro.org/people/santosh.shukla/test-definitions.git/shortlog/refs/heads/isol-v2
> >> >>
> >> >> Signed-off-by: Santosh Shukla <santosh.shukla@linaro.org>
> >> >> ---
> >> >>  example/isolation/odp_isolation.c  |    5 +++++
> >> >>  example/isolation/readme           |   20 ++++++++++++++++++++
> >> >>  platform/linux-generic/odp_linux.c |   12 ++++++++++--
> >> >>  3 files changed, 35 insertions(+), 2 deletions(-)
> >> >>  create mode 100644 example/isolation/readme
> >> >>
> >> >> diff --git a/example/isolation/odp_isolation.c b/example/isolation/odp_isolation.c
> >> >> index 8d8fffc..1ade4c7 100644
> >> >> --- a/example/isolation/odp_isolation.c
> >> >> +++ b/example/isolation/odp_isolation.c
> >> >> @@ -214,6 +214,11 @@ int main(int argc, char *argv[])
> >> >>       thr_id = odp_thread_create(0);
> >> >>       odp_init_local(thr_id);
> >> >>
> >> >> +     /*
> >> >> +      * create dummy dp thread and don't pin them on any core,
> >> >> +      * use odp-on-isolated-cpu.sh script to define dplane cpuset
> >> >> +      * and to pin, run app on them, Refer readme.
> >> >> +      */
> >> >>       for (i=0; i<num_workers; i++) {
> >> >>
> >> >>               first_core = atoi(args.core_name[i]);
> >> >> diff --git a/example/isolation/readme b/example/isolation/readme
> >> >> new file mode 100644
> >> >> index 0000000..d49d61a
> >> >> --- /dev/null
> >> >> +++ b/example/isolation/readme
> >> >> @@ -0,0 +1,20 @@
> >> >> +
> >> >> +README on How-to run odp app on isolated core for no_hz_full kernel mode.
> >> >> +
> >> >> +- Disbale timer init in odp application.
> >> >> +- avoid using odp's cpuset. Let odp thread launch unpinned.
> >> >> +- use odp-on-isolated-cpu.sh script to pin odp dp thread to
> >> >> + dplane cpuset.
> >> >> +
> >> >> +- download my local test-definition repo link [1]
> >> >> +- copy odp_isolation binary to /usr/local/bin
> >> >> +- run  ./common/scripts/odp-on-isolated-cpu.sh
> >> >> +Or
> >> >> +-  ./common/scripts/odp-on-isolated-cpu.sh 1,2 "odp_isolation -l 1,2"
> >> >> +- Above script shows isolation duration, before / after interrupt count.
> >> >> +
> >> >> +Known issue :
> >> >> +- x86 hw generates spurious ipi lead to break no_hz_full isolation (todo).
> >> >> +
> >> >> +[1] https://git.linaro.org/people/santosh.shukla/test-definitions.git/shortlog/refs/heads/isol-v2
> >> >> +
> >> >> diff --git a/platform/linux-generic/odp_linux.c b/platform/linux-generic/odp_linux.c
> >> >> index 6e2b448..7e853c7 100644
> >> >> --- a/platform/linux-generic/odp_linux.c
> >> >> +++ b/platform/linux-generic/odp_linux.c
> >> >> @@ -22,6 +22,11 @@
> >> >>  #include <odp_debug.h>
> >> >>
> >> >>
> >> >> +#define NO_HZ_FULL_ISOL /*
> >> >> +                      * don't use odp cpuset.
> >> >> +                      * enable this for odp isolation mode
> >> >> +                      */
> >> >> +
> >> >>  typedef struct {
> >> >>       int thr_id;
> >> >>       void *(*start_routine) (void *);
> >> >> @@ -49,7 +54,9 @@ void odp_linux_pthread_create(odp_linux_pthread_t *thread_tbl, int num,
> >> >>               int first_core, void *(*start_routine) (void *), void *arg)
> >> >>  {
> >> >>       int i;
> >> >> +#ifndef NO_HZ_FULL_ISOL
> >> >>       cpu_set_t cpu_set;
> >> >> +#endif
> >> >>       odp_start_args_t *start_args;
> >> >>       int core_count;
> >> >>       int cpu;
> >> >> @@ -62,16 +69,17 @@ void odp_linux_pthread_create(odp_linux_pthread_t *thread_tbl, int num,
> >> >>       memset(thread_tbl, 0, num * sizeof(odp_linux_pthread_t));
> >> >>
> >> >>       for (i = 0; i < num; i++) {
> >> >> +             cpu = (first_core + i) % core_count;
> >> >> +#ifndef NO_HZ_FULL_ISOL
> >> >>               pthread_attr_init(&thread_tbl[i].attr);
> >> >>
> >> >>               CPU_ZERO(&cpu_set);
> >> >>
> >> >> -             cpu = (first_core + i) % core_count;
> >> >>               CPU_SET(cpu, &cpu_set);
> >> >>
> >> >>               pthread_attr_setaffinity_np(&thread_tbl[i].attr,
> >> >>                                           sizeof(cpu_set_t), &cpu_set);
> >> >> -
> >> >> +#endif
> >> >
> >> > There's no need for this to be a compile time decision - this changes
> >> > the behaviour of all applications linked against this library.
> >> >
> >>
> >> Agree, There might be better way to avoid linkage issue.
> >>
> >> > I would rather leave the affinity setting in odp_linux_pthread_create
> >> > and let the script just ensure that the specified cores are isolated. Is
> >> > there anything actually preventing that from being possible?
> >>
> >> Yes, Isolation is a specific requirement to application. current odp
> >> pins thread to core in order which not necessarily be the case all the
> >> time.
> >>
> >> A  Flexibility to pin any of odp thread to any core is desired choice.
> >> Therefore I had to break free from existing pinning assumption.
> >
> > Hence my suggestion in the other thread about changing odp_linux_pthread_create.
> > If there's a limitation in the existing ODP API/implementation/apps then
> > it should be fixed rather than working around it.
> >
> >> Keeping two set of cpuset - One is default ODP whome thread pinned
> >> with then Second one created by isolation script - doesn't make sense
> >> to me.
> >
> > The two cpusets do different things (the cpusets created via /dev/cpuset
> > aren't related to the CPU_SET used with pthread_attr_setaffinity_np,
> > other than having a confusingly similar name).
> >
> > The cpuset created by the script specifies a group of cores the
> > application is permitted to run threads on (and ensures those cores are
> > isolated). The pthread_attr_setaffinity_np in the application specifies
> > a specific core within that group for a particular thread to run on.
> >
> > So I still don't see what the problem is with the problem is.
> >
> >> Also unpinning a thread from odp cpuset group to script cpuset
> >> group gives system error.
> >
> > Not sure what you mean exactly, but if the ODP application tries to
> > setaffinity to core that's not in the cpuset created by the script I
> > would expect that to fail. That's OK though as the list of cpus passed
> > to the script should be the same as that passed to the application,
> > anything else is an error.
> 
> Yes that was a problem which I wanted to avoid therefore choose to
> mask off odp thread affinity part. Also it doesn't make sense to move
> thread from odp cpuset to /dev/cpuset to me. However your concern is
> valid in favor of keeping odp generic so I'll revisit my changes.
> 
> Thanks.
> 
> >
> >> >
> >> > As it is, what happens if you build with NO_HZ_FULL_ISOL but don't run
> >> > the application via the odp-on-isolated-cpu.sh script? and if you do run
> >>
> >> There is a readme, User must refer that in case he is looking for isolation.
> >>
> >> > via the script, what's happening during the time between the application
> >> > launching the threads and the script moving those threads onto the
> >> > correct cores?
> >>
> >> It is order, first application spwans a thread then script moves those
> >> thread to isol cores. Therefore it should work.
> >
> > Yes but there is a window in between where they could all be running on
> > the same core.
> yes, odp init time.
> 
> > How long does it take for them to be moved to the correct
> > core?
> In case of l2fwd - I am migrating thread just after thread
> initialization.. no real IO started on those threads..So its fairly
> quick for me. Are you asking about a use-case where dpthread spawned,
> doing IO then migrated to other core by script?
> 
> > how does the application know when it's been done?..
> 
> I verify this way (command based approach)
> ps -Leo pid,tid,psr,pcpu,comm | grep odp
> # 23025  23025   0 99.9 odp_l2fwd
> # 23025  23026   7  0.0 odp_l2fwd
> # 23025  23027   1 99.9 odp_l2fwd
> # 23025  23028   2  100 odp_l2fwd
> 
> Last 2 threads (dpthread) moved to isolated core as per my script instruction.
> 

Actually the two questions were really a single question. I was
referring (not very clearly) to the fact that some parts of ODP
(spinlocks, barriers) depend on each ODP thread running on a different
core, so if this wasn't guaranteed until after the odp-on-isolated-cpu.sh
script had completed you'd need some way of signalling that completion
back to the application.. yuck, hopefully we are agreed on leaving the
affinity setting in ODP.

I just noticed the existing odp_linux_pthread_create doesn't cope well
with the presence of cpusets as it assumes that it can run threads on any
core in the system and doesn't check for success. So if you prevent the
process from being able to use some of the cores (e.g. by creating a
cpuset that it's not a member of and sticking some cores in it) it'll
end up trying to run two threads on one core and deadlock.

> > there's no
> > need for the uncertainty when the application can set its own affinity.
> 
> Right.
diff mbox

Patch

diff --git a/example/isolation/odp_isolation.c b/example/isolation/odp_isolation.c
index 8d8fffc..1ade4c7 100644
--- a/example/isolation/odp_isolation.c
+++ b/example/isolation/odp_isolation.c
@@ -214,6 +214,11 @@  int main(int argc, char *argv[])
 	thr_id = odp_thread_create(0);
 	odp_init_local(thr_id);
 
+	/*
+	 * create dummy dp thread and don't pin them on any core,
+	 * use odp-on-isolated-cpu.sh script to define dplane cpuset
+	 * and to pin, run app on them, Refer readme.
+	 */
 	for (i=0; i<num_workers; i++) {
 
 		first_core = atoi(args.core_name[i]);
diff --git a/example/isolation/readme b/example/isolation/readme
new file mode 100644
index 0000000..d49d61a
--- /dev/null
+++ b/example/isolation/readme
@@ -0,0 +1,20 @@ 
+
+README on How-to run odp app on isolated core for no_hz_full kernel mode.
+
+- Disbale timer init in odp application.
+- avoid using odp's cpuset. Let odp thread launch unpinned.
+- use odp-on-isolated-cpu.sh script to pin odp dp thread to
+ dplane cpuset.
+
+- download my local test-definition repo link [1]
+- copy odp_isolation binary to /usr/local/bin
+- run  ./common/scripts/odp-on-isolated-cpu.sh
+Or
+-  ./common/scripts/odp-on-isolated-cpu.sh 1,2 "odp_isolation -l 1,2"
+- Above script shows isolation duration, before / after interrupt count.
+
+Known issue :
+- x86 hw generates spurious ipi lead to break no_hz_full isolation (todo).
+
+[1] https://git.linaro.org/people/santosh.shukla/test-definitions.git/shortlog/refs/heads/isol-v2
+
diff --git a/platform/linux-generic/odp_linux.c b/platform/linux-generic/odp_linux.c
index 6e2b448..7e853c7 100644
--- a/platform/linux-generic/odp_linux.c
+++ b/platform/linux-generic/odp_linux.c
@@ -22,6 +22,11 @@ 
 #include <odp_debug.h>
 
 
+#define NO_HZ_FULL_ISOL /*
+			 * don't use odp cpuset.
+			 * enable this for odp isolation mode
+			 */
+
 typedef struct {
 	int thr_id;
 	void *(*start_routine) (void *);
@@ -49,7 +54,9 @@  void odp_linux_pthread_create(odp_linux_pthread_t *thread_tbl, int num,
 		int first_core, void *(*start_routine) (void *), void *arg)
 {
 	int i;
+#ifndef NO_HZ_FULL_ISOL
 	cpu_set_t cpu_set;
+#endif
 	odp_start_args_t *start_args;
 	int core_count;
 	int cpu;
@@ -62,16 +69,17 @@  void odp_linux_pthread_create(odp_linux_pthread_t *thread_tbl, int num,
 	memset(thread_tbl, 0, num * sizeof(odp_linux_pthread_t));
 
 	for (i = 0; i < num; i++) {
+		cpu = (first_core + i) % core_count;
+#ifndef NO_HZ_FULL_ISOL
 		pthread_attr_init(&thread_tbl[i].attr);
 
 		CPU_ZERO(&cpu_set);
 
-		cpu = (first_core + i) % core_count;
 		CPU_SET(cpu, &cpu_set);
 
 		pthread_attr_setaffinity_np(&thread_tbl[i].attr,
 					    sizeof(cpu_set_t), &cpu_set);
-
+#endif
 		start_args = malloc(sizeof(odp_start_args_t));
 		memset(start_args, 0, sizeof(odp_start_args_t));
 		start_args->start_routine = start_routine;