mbox

[PATCHv7,00/35] running things in process mode

Message ID 1463490282-23277-1-git-send-email-christophe.milard@linaro.org
State New
Headers show

Pull-request

https://git.linaro.org/people/christophe.milard/odp.git test_in_process_mode_v7

Message

Christophe Milard May 17, 2016, 1:04 p.m. UTC
Pull can be performed from the URL at the end of this cover letter
All patches sent to the list following Maxim's request.

Since v6
-rebased
-regrouping history, squashing patch 1 and 12; 4, 5 and 12 (Petri)
-mixed mode test removed (but still claimed to be supported in ODP). (Petri)

Since v5
-rebased
-patch removing old unused helper code dropped. (Petri)
-Doc update to discourage usage of the old functions. (Petri, Mike)

Since v4:
-updates following Brian's comments

Since v3:
-fixed rebase error (Christophe)
-rebased

Since v2:
-serious rebase following clash with dba05a28 (api: init: add instance handle)
-squashing the validation changes.

Since v1:
-variable declaration gathered in function 's head (Petri)
-linux prefix removed from helper's types and function names (Mike, Christophe)

Hi,

This patch series adds the ability to run tests/ exemples / perf-test
in "process mode" (i.e letting OPD threads being linux processes)
It it hence tackling ODP-171.

This is achieved in 2 main steps:

A]
The 2 pairs of helper functions:
odph_linux_pthread_create(), odph_linux_pthread_join()
and
odph_linux_process_fork_n(), odph_linux_process_wait_n()
are complemented by:
odph_linux_odpthreads_create() and odph_linux_odpthreads_join()
The latter's callers are unaware of the actual implementation of the ODP
thread (making test truly platform agnostic).
The helper functions decide at run time whether an odp thread is to be
a linux process or pthread based on the command line argument.

B] each test/example now calls a new helper function,
odph_linux_parse_options(), so that the helper can get its own arguments
out of the command line.
Currentely supported args is: --odph_proc.
Defaults assumes thread.

The changed are first done on the shmem tests, and thereafter propagated
to other helper users.
Note that this patch series enable the option but does not affect
make check at this time: make check still calls the tests with no options
which default to thread mode.


[GIT PULL ODPv7] running things in process mode:

The following changes since commit 9359d1b2fedb88c57ee4ef96e4a7282b4e07ffc6:

  linux-generic: scheduler: avoid starvation (2016-05-13 17:20:55 +0300)

are available in the git repository at:

  https://git.linaro.org/people/christophe.milard/odp.git test_in_process_mode_v7

for you to fetch changes up to 1ee35450efcd4efe513cf2bd959501b5521b6cbb:

  doc: implementers-guide: adding helper recommendations (2016-05-17 14:04:41 +0200)

----------------------------------------------------------------


Christophe Milard (35):
  helpers: adding command line argument parsing
  validation: common: adding command line argument parsing
  validation: shmem: adding command line argument parsing
  helpers: linux: creating functions to handle odpthreads
  helper: test: adding odpthread functions tests
  validation: using implementation agnostic function for ODP threads
  validation: most tests: adding command line argument parsing
  validation: init: adding command line argument parsing
  validation: pktio: adding command line argument parsing
  performance: odp_scheduling: proc mode done by helper
  performance: odp_pktio_perf: using agnostic function for ODP threads
  performance: odp_pktio_perf: adding helper cmd line parsing
  performance: odp_l2fwd: using agnostic function for ODP threads
  performance: odp_l2fwd: adding helper cmd line parsing
  performance: crypto: using agnostic function for ODP threads
  performance: crypto: adding helper cmd line parsing
  example: classifier: using agnostic function for ODP threads
  example: classifier: adding helper cmd line parsing
  example: generator: using agnostic function for ODP threads
  example: generator: adding helper cmd line parsing
  example: ipsec: using agnostic function for ODP threads
  example: ipsec: adding helper cmd line parsing
  example: l2fwd_simple: using agnostic function for ODP threads
  example: l2fwd_simple: adding helper cmd line parsing
  example: pktio: using agnostic function for ODP threads
  example: pktio: adding helper cmd line parsing
  example: time: using agnostic function for ODP threads
  example: time: adding helper cmd line parsing
  example: timer: using agnostic function for ODP threads
  example: timer: adding helper cmd line parsing
  example: switch: using agnostic function for ODP threads
  example: switch: adding helper cmd line parsing
  linux-generic: test: ringtest: using agnostic function for ODP threads
  linux-generic: test: ringtest: adding helper cmd line parsing
  doc: implementers-guide: adding helper recommendations

 doc/implementers-guide/implementers-guide.adoc     |  32 ++
 example/classifier/odp_classifier.c                |  39 +--
 example/generator/odp_generator.c                  |  45 +--
 example/ipsec/odp_ipsec.c                          |  27 +-
 example/l2fwd_simple/odp_l2fwd_simple.c            |  47 ++-
 example/packet/odp_pktio.c                         |  45 +--
 example/switch/odp_switch.c                        |  26 +-
 example/time/time_global_test.c                    |  17 +-
 example/timer/odp_timer_test.c                     |  26 +-
 helper/include/odp/helper/linux.h                  | 138 +++++++-
 helper/linux.c                                     | 382 +++++++++++++++++++++
 helper/test/.gitignore                             |   1 +
 helper/test/Makefile.am                            |   7 +-
 helper/test/odpthreads.c                           | 100 ++++++
 helper/test/odpthreads_as_processes                |  14 +
 helper/test/odpthreads_as_pthreads                 |  14 +
 platform/linux-generic/test/pktio/pktio_run        |  21 +-
 platform/linux-generic/test/pktio/pktio_run_dpdk   |  17 +-
 platform/linux-generic/test/pktio/pktio_run_netmap |   5 +-
 platform/linux-generic/test/pktio/pktio_run_pcap   |   5 +-
 platform/linux-generic/test/pktio/pktio_run_tap    |   6 +-
 platform/linux-generic/test/ring/ringtest.c        |  24 +-
 test/performance/odp_crypto.c                      |  24 +-
 test/performance/odp_l2fwd.c                       |  37 +-
 test/performance/odp_pktio_perf.c                  |  34 +-
 test/performance/odp_scheduling.c                  |  91 ++---
 test/validation/atomic/atomic.c                    |  40 ++-
 test/validation/atomic/atomic.h                    |   2 +-
 test/validation/atomic/atomic_main.c               |   4 +-
 test/validation/barrier/barrier.c                  |  14 +-
 test/validation/barrier/barrier.h                  |   2 +-
 test/validation/barrier/barrier_main.c             |   4 +-
 test/validation/buffer/buffer.c                    |  10 +-
 test/validation/buffer/buffer.h                    |   2 +-
 test/validation/buffer/buffer_main.c               |   4 +-
 test/validation/classification/classification.c    |  10 +-
 test/validation/classification/classification.h    |   2 +-
 .../classification/classification_main.c           |   4 +-
 test/validation/common/odp_cunit_common.c          |  24 +-
 test/validation/common/odp_cunit_common.h          |   6 +-
 test/validation/cpumask/cpumask.c                  |  10 +-
 test/validation/cpumask/cpumask.h                  |   2 +-
 test/validation/cpumask/cpumask_main.c             |   4 +-
 test/validation/crypto/crypto.c                    |   6 +-
 test/validation/crypto/crypto.h                    |   2 +-
 test/validation/crypto/crypto_main.c               |   4 +-
 test/validation/errno/errno.c                      |  10 +-
 test/validation/errno/errno.h                      |   2 +-
 test/validation/errno/errno_main.c                 |   4 +-
 test/validation/hash/hash.c                        |  10 +-
 test/validation/hash/hash.h                        |   2 +-
 test/validation/hash/hash_main.c                   |   4 +-
 test/validation/init/init.c                        |  18 +-
 test/validation/init/init.h                        |   6 +-
 test/validation/init/init_main_abort.c             |   4 +-
 test/validation/init/init_main_log.c               |   4 +-
 test/validation/init/init_main_ok.c                |   4 +-
 test/validation/lock/lock.c                        |  50 +--
 test/validation/lock/lock.h                        |   2 +-
 test/validation/lock/lock_main.c                   |   4 +-
 test/validation/packet/packet.c                    |  10 +-
 test/validation/packet/packet.h                    |   2 +-
 test/validation/packet/packet_main.c               |   4 +-
 test/validation/pktio/pktio.c                      |  10 +-
 test/validation/pktio/pktio.h                      |   2 +-
 test/validation/pktio/pktio_main.c                 |   4 +-
 test/validation/pool/pool.c                        |  10 +-
 test/validation/pool/pool.h                        |   2 +-
 test/validation/pool/pool_main.c                   |   4 +-
 test/validation/queue/queue.c                      |  10 +-
 test/validation/queue/queue.h                      |   2 +-
 test/validation/queue/queue_main.c                 |   4 +-
 test/validation/random/random.c                    |  10 +-
 test/validation/random/random.h                    |   2 +-
 test/validation/random/random_main.c               |   4 +-
 test/validation/scheduler/scheduler.c              |  18 +-
 test/validation/scheduler/scheduler.h              |   2 +-
 test/validation/scheduler/scheduler_main.c         |   4 +-
 test/validation/shmem/shmem.c                      |  16 +-
 test/validation/shmem/shmem.h                      |   2 +-
 test/validation/shmem/shmem_main.c                 |   4 +-
 test/validation/std_clib/std_clib.c                |  10 +-
 test/validation/std_clib/std_clib.h                |   2 +-
 test/validation/std_clib/std_clib_main.c           |   4 +-
 test/validation/system/system.c                    |  10 +-
 test/validation/system/system.h                    |   2 +-
 test/validation/system/system_main.c               |   4 +-
 test/validation/thread/thread.c                    |  14 +-
 test/validation/thread/thread.h                    |   2 +-
 test/validation/thread/thread_main.c               |   4 +-
 test/validation/time/time.c                        |  10 +-
 test/validation/time/time.h                        |   2 +-
 test/validation/time/time_main.c                   |   4 +-
 test/validation/timer/timer.c                      |  10 +-
 test/validation/timer/timer.h                      |   2 +-
 test/validation/timer/timer_main.c                 |   4 +-
 test/validation/traffic_mngr/traffic_mngr.c        |   6 +-
 test/validation/traffic_mngr/traffic_mngr.h        |   2 +-
 test/validation/traffic_mngr/traffic_mngr_main.c   |   4 +-
 99 files changed, 1316 insertions(+), 410 deletions(-)
 create mode 100644 helper/test/odpthreads.c
 create mode 100755 helper/test/odpthreads_as_processes
 create mode 100755 helper/test/odpthreads_as_pthreads

Comments

Maxim Uvarov May 18, 2016, 3:02 p.m. UTC | #1
On 05/18/16 14:14, Savolainen, Petri (Nokia - FI/Espoo) wrote:
> Reviewed patches 1-10
>
> Reviewed-by: Petri Savolainen <petri.savolainen@nokia.com>
Petri, Christophe, are you ok with merging 1-10 first, and next after that?

Maxim.
>
>> -----Original Message-----
>> From: lng-odp [mailto:lng-odp-bounces@lists.linaro.org] On Behalf Of
>> Christophe Milard
>> Sent: Tuesday, May 17, 2016 4:04 PM
>> To: lng-odp@lists.linaro.org; brian.brooks@linaro.org;
>> mike.holmes@linaro.org
>> Subject: [lng-odp] [PATCHv7 00/35] running things in process mode
>>
>> Pull can be performed from the URL at the end of this cover letter
>> All patches sent to the list following Maxim's request.
>>
>> Since v6
>> -rebased
>> -regrouping history, squashing patch 1 and 12; 4, 5 and 12 (Petri)
>> -mixed mode test removed (but still claimed to be supported in ODP).
>> (Petri)
>>
> _______________________________________________
> lng-odp mailing list
> lng-odp@lists.linaro.org
> https://lists.linaro.org/mailman/listinfo/lng-odp
Christophe Milard May 18, 2016, 3:18 p.m. UTC | #2
This is, of course very strange to me, as the usage of the "old" functions
will keep spreading, and we'll be stopping halv way done.
I guess Brian can apply his review-by, as patch 11-35 are the same as v6

But if the alternative is nothing done...

Christophe.

On 18 May 2016 at 17:02, Maxim Uvarov <maxim.uvarov@linaro.org> wrote:

> On 05/18/16 14:14, Savolainen, Petri (Nokia - FI/Espoo) wrote:
>
>> Reviewed patches 1-10
>>
>> Reviewed-by: Petri Savolainen <petri.savolainen@nokia.com>
>>
> Petri, Christophe, are you ok with merging 1-10 first, and next after that?
>
> Maxim.
>
>>
>> -----Original Message-----
>>> From: lng-odp [mailto:lng-odp-bounces@lists.linaro.org] On Behalf Of
>>> Christophe Milard
>>> Sent: Tuesday, May 17, 2016 4:04 PM
>>> To: lng-odp@lists.linaro.org; brian.brooks@linaro.org;
>>> mike.holmes@linaro.org
>>> Subject: [lng-odp] [PATCHv7 00/35] running things in process mode
>>>
>>> Pull can be performed from the URL at the end of this cover letter
>>> All patches sent to the list following Maxim's request.
>>>
>>> Since v6
>>> -rebased
>>> -regrouping history, squashing patch 1 and 12; 4, 5 and 12 (Petri)
>>> -mixed mode test removed (but still claimed to be supported in ODP).
>>> (Petri)
>>>
>>> _______________________________________________
>> lng-odp mailing list
>> lng-odp@lists.linaro.org
>> https://lists.linaro.org/mailman/listinfo/lng-odp
>>
>
> _______________________________________________
> lng-odp mailing list
> lng-odp@lists.linaro.org
> https://lists.linaro.org/mailman/listinfo/lng-odp
>
Mike Holmes May 18, 2016, 3:29 p.m. UTC | #3
From the call earlier I understood that we agreed to the whole series, we
just need the Reviewed-by, they should all go in if reviewed unless there
is a ncak that needs resloving.

On 18 May 2016 at 11:18, Christophe Milard <christophe.milard@linaro.org>
wrote:

> This is, of course very strange to me, as the usage of the "old" functions
> will keep spreading, and we'll be stopping halv way done.
> I guess Brian can apply his review-by, as patch 11-35 are the same as v6
>
> But if the alternative is nothing done...
>
> Christophe.
>
> On 18 May 2016 at 17:02, Maxim Uvarov <maxim.uvarov@linaro.org> wrote:
>
>> On 05/18/16 14:14, Savolainen, Petri (Nokia - FI/Espoo) wrote:
>>
>>> Reviewed patches 1-10
>>>
>>> Reviewed-by: Petri Savolainen <petri.savolainen@nokia.com>
>>>
>> Petri, Christophe, are you ok with merging 1-10 first, and next after
>> that?
>>
>> Maxim.
>>
>>>
>>> -----Original Message-----
>>>> From: lng-odp [mailto:lng-odp-bounces@lists.linaro.org] On Behalf Of
>>>> Christophe Milard
>>>> Sent: Tuesday, May 17, 2016 4:04 PM
>>>> To: lng-odp@lists.linaro.org; brian.brooks@linaro.org;
>>>> mike.holmes@linaro.org
>>>> Subject: [lng-odp] [PATCHv7 00/35] running things in process mode
>>>>
>>>> Pull can be performed from the URL at the end of this cover letter
>>>> All patches sent to the list following Maxim's request.
>>>>
>>>> Since v6
>>>> -rebased
>>>> -regrouping history, squashing patch 1 and 12; 4, 5 and 12 (Petri)
>>>> -mixed mode test removed (but still claimed to be supported in ODP).
>>>> (Petri)
>>>>
>>>> _______________________________________________
>>> lng-odp mailing list
>>> lng-odp@lists.linaro.org
>>> https://lists.linaro.org/mailman/listinfo/lng-odp
>>>
>>
>> _______________________________________________
>> lng-odp mailing list
>> lng-odp@lists.linaro.org
>> https://lists.linaro.org/mailman/listinfo/lng-odp
>>
>
>
Brian Brooks May 18, 2016, 7:31 p.m. UTC | #4
On 05/17 15:04:07, Christophe Milard wrote:
> Pull can be performed from the URL at the end of this cover letter
> All patches sent to the list following Maxim's request.
> 
> Since v6
> -rebased
> -regrouping history, squashing patch 1 and 12; 4, 5 and 12 (Petri)
> -mixed mode test removed (but still claimed to be supported in ODP). (Petri)
> 
> Since v5
> -rebased
> -patch removing old unused helper code dropped. (Petri)
> -Doc update to discourage usage of the old functions. (Petri, Mike)
> 
> Since v4:
> -updates following Brian's comments
> 
> Since v3:
> -fixed rebase error (Christophe)
> -rebased
> 
> Since v2:
> -serious rebase following clash with dba05a28 (api: init: add instance handle)
> -squashing the validation changes.
> 
> Since v1:
> -variable declaration gathered in function 's head (Petri)
> -linux prefix removed from helper's types and function names (Mike, Christophe)

This patch series allows existing tests and examples to run as pthread-per-core
or process-per-core. The default pthread-per-core is preserved, and
process-per-core is enabled via passing "--odph_proc" via the command line when
invoking a test or example binary. The auxiliary helper code for launching
pthreads and processes still exists, and a new "ODP thread" helper object was
created as an abstraction of the two task types. This review is contingent upon
the scope of ODP to include multi-process applications.

Reviewed-by: Brian Brooks <brian.brooks@linaro.org>