From patchwork Wed Feb 7 13:09:27 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adhemerval Zanella Netto X-Patchwork-Id: 127175 Delivered-To: patch@linaro.org Received: by 10.46.124.24 with SMTP id x24csp456014ljc; Wed, 7 Feb 2018 05:09:56 -0800 (PST) X-Google-Smtp-Source: AH8x225UERRUz6tBFDA7/g8hFRSbumS7OWCu4rXipmm/hClU9U5Gi/l0Qu/Q5FhX5jqFKRVPQFEW X-Received: by 10.101.70.12 with SMTP id v12mr4914590pgq.327.1518008995818; Wed, 07 Feb 2018 05:09:55 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1518008995; cv=none; d=google.com; s=arc-20160816; b=WUhhmemdaLPbYlko1ILbd1srjIBhKrnKW6pq93V6kfYLy3SHvhu1jGnNNAgU8kn5ae qYx9ma7+NEVUyCKFcsFSyl+uh/0RZlMk15ZGyhKauMbHCn369mDwavrUzmTTMygdWnt/ 1W9S2yAfRa+aAmjp3wutHUSA4e5UfC+W1gf4GCqmqYOb8wqnEjcMgTsoo8CigxgMP/Aw jaIctv2bMYdxwYt3aR2o1XasEZKnn2LumVaZCpw8pOD6BZkQ6j6qzv6VneA/1z+risDL k+WpvgvV48iCJHMo0Mzcmq0xiGRGcroNkwMAOmuBdqFV4GuBa6GUryH5rCl5tvzBNqOc lK5A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:to:from:delivered-to :sender:list-help:list-post:list-archive:list-subscribe :list-unsubscribe:list-id:precedence:mailing-list:dkim-signature :domainkey-signature:arc-authentication-results; bh=9edXBXbg2bScVzHX7jIdfZ17LpPELNfZlpR5GQYg/wA=; b=rsIE630uLtb9U2CUYm62a8ru+cpuVM6dbV+tHGUO/MPBPUaRwkGlUA7PwJFsjZF92R VKquMeClrUKU1gJc2UWRjn+VGsaW9XYnuigRgwTOxKkOrJkP57s3ZMxgMAobY+4V2va1 t3kKRYsarAWIX3WK7OWA37oRV7NZK1KSdivoDmHHhsmHZ7KsYAPJMuPvsLYh6HCKZQxX 9cuNrnzk5tzlrUmMzaIa8lUoZu8C9PBjoxcIBjwyUe3OsCFQ10/u4RF8zGWzX6bveYYw uPcFYTEEErn3iUWwZJWbALEIX4+Odd90wjr+MmDXmcuONE7/4cUnyfsp34htV5OT96bl fDCg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@sourceware.org header.s=default header.b=OCsGCcKX; spf=pass (google.com: domain of libc-alpha-return-90098-patch=linaro.org@sourceware.org designates 209.132.180.131 as permitted sender) smtp.mailfrom=libc-alpha-return-90098-patch=linaro.org@sourceware.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id x11si937337pgp.23.2018.02.07.05.09.55 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 07 Feb 2018 05:09:55 -0800 (PST) Received-SPF: pass (google.com: domain of libc-alpha-return-90098-patch=linaro.org@sourceware.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@sourceware.org header.s=default header.b=OCsGCcKX; spf=pass (google.com: domain of libc-alpha-return-90098-patch=linaro.org@sourceware.org designates 209.132.180.131 as permitted sender) smtp.mailfrom=libc-alpha-return-90098-patch=linaro.org@sourceware.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:from:to:subject:date:message-id:in-reply-to :references; q=dns; s=default; b=FPS7rv8sXHxv8wW4XwX+hPal2VQhixh 7OSkU+TWVvwrTfwRuvAeELg5n1gjv9I0+P9WDKT/BtMB1rbtkLjl9rEc4GW9wmS/ pUteQ0f4k0njQd7a2z/VbQcpdjT0H25aUAVz9B9ZDst5s0ZrQdvz5V18U59Nc7Fd EtK8S/BvdXXU= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:from:to:subject:date:message-id:in-reply-to :references; s=default; bh=TmtwhLrxvin12KYoTnMmQ2g9Cio=; b=OCsGC cKXBqqu4zSkFuw+4iWILAav4Z6AX0UgNX/yHsXFqgRUmYIBH8DnonptDJ36JTxUa LIHmUYuL3QIbNJrbyFlQ85AaFpwZvrT8oPyhYOXgmnY3UvuQ+Am/71TFzKsPIbsU waheh84Tkusm3BPP3KhduX40P9Yzlas8I8O1hE= Received: (qmail 11506 invoked by alias); 7 Feb 2018 13:09:46 -0000 Mailing-List: contact libc-alpha-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: libc-alpha-owner@sourceware.org Delivered-To: mailing list libc-alpha@sourceware.org Received: (qmail 11495 invoked by uid 89); 7 Feb 2018 13:09:45 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-24.1 required=5.0 tests=AWL, BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, RCVD_IN_DNSWL_NONE, SPF_PASS, UNSUBSCRIBE_BODY autolearn=ham version=3.3.2 spammy=WHO, acquires, preparation, trading X-HELO: mail-qk0-f172.google.com X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=9edXBXbg2bScVzHX7jIdfZ17LpPELNfZlpR5GQYg/wA=; b=q6RAtj8BOLD2JiM4K2lioDscbz0fL2JV6WOvzFQ90xZz3gcM29Oe+l/xJ58MKRtasR NvK5JAiVik9C0567jmsOBNLlbUQDY6boewQkuBM09hZYmtK8KJC7AEs8AGLj04N25BgX 3KMUnQyaYUMFkWPq46K58fd74LHKt9ZM8dcjZGG8Ds5SmcAq2t+Pjb41+rDUF/RTPYXj R1YOPcwDSeBm3Afd2mK2/Xd5f4oPe4FNxoCBjg1FcqlaMKqs8gE+rBCGk9SA4CFivZzg dE2onPlZxaRX+kSDA7nI4hFUjK76TdyYsTBYWNa9v4/NUSszLx1Pcrptnbtpnr0N0X+d rHtw== X-Gm-Message-State: APf1xPCrt99np/8LeUmkWXNHlAPILWCT+gWsaTefOBSg3DGUKLLgyJdS zs85Ok28NGtWWGPjrfboiX9U6Y+3SK8= X-Received: by 10.55.6.1 with SMTP id 1mr8769904qkg.38.1518008980445; Wed, 07 Feb 2018 05:09:40 -0800 (PST) From: Adhemerval Zanella To: libc-alpha@sourceware.org Subject: [PATCH 3/3] Refactor atfork handlers Date: Wed, 7 Feb 2018 11:09:27 -0200 Message-Id: <1518008967-8310-3-git-send-email-adhemerval.zanella@linaro.org> In-Reply-To: <1518008967-8310-1-git-send-email-adhemerval.zanella@linaro.org> References: <1518008967-8310-1-git-send-email-adhemerval.zanella@linaro.org> Current implementation (sysdeps/nptl/fork.c) replicates the atfork handlers list backward to invoke the child handlers after fork/clone syscall. The internal atfork handlers is implemented as a single-linked list so a lock-free algorithm can be used, trading fork mulithread call performance for some code complexity and dynamic stack allocation (since the backwards list should not fail). This patch refactor it to use a dynarary instead of a linked list. It simplifies the external variables need to be exported and also the internal atfork handler member definition. The downside is a serialization of fork call in multithread, since to operate on the dynarray the internal lock should be used. However as noted by Florian, it already acquires external locks for malloc and libio so it is already hitting some lock contention. Besides, posix_spawn should be faster and more scalable to run external programs in multithread environments. Checked on x86_64-linux-gnu. * nptl/Makefile (routines): Remove unregister-atfork. * nptl/register-atfork.c (fork_handler_pool): Remove variable. (fork_handler_alloc): Remove function. (fork_handlers, fork_handler_init): New variables. (__fork_lock): Rename to atfork_lock. (__register_atforki, __unregister_atfork, libc_freeres_fn): Rewrite to use a dynamic array to add/remove atfork handlers. * sysdeps/nptl/fork.c (__libc_fork): Likewise. * sysdeps/nptl/fork.h (__fork_lock, __fork_handlers, __linkin_atfork): Remove declaration. (fork_handler): Remove next, refcntr, and need_signal member. (__run_fork_handler_type): New enum. (__run_fork_handlers): New prototype. * sysdeps/nptl/libc-lockP.h (__libc_atfork): Remove declaration. --- ChangeLog | 15 +++++ nptl/Makefile | 2 +- nptl/register-atfork.c | 146 +++++++++++++++++++--------------------------- sysdeps/nptl/fork.c | 96 +----------------------------- sysdeps/nptl/fork.h | 31 +++++----- sysdeps/nptl/libc-lockP.h | 2 - 6 files changed, 97 insertions(+), 195 deletions(-) -- 2.7.4 diff --git a/nptl/Makefile b/nptl/Makefile index 6fc2c8b..be7ee3e 100644 --- a/nptl/Makefile +++ b/nptl/Makefile @@ -30,7 +30,7 @@ install-lib-ldscripts := libpthread.so routines = alloca_cutoff forward libc-lowlevellock libc-cancellation \ libc-cleanup libc_pthread_init libc_multiple_threads \ - register-atfork unregister-atfork pthread_self + register-atfork pthread_self shared-only-routines = forward # We need to provide certain routines for compatibility with existing diff --git a/nptl/register-atfork.c b/nptl/register-atfork.c index f309cec..0bc2fe9 100644 --- a/nptl/register-atfork.c +++ b/nptl/register-atfork.c @@ -22,123 +22,97 @@ #include #include +#define DYNARRAY_ELEMENT struct fork_handler +#define DYNARRAY_STRUCT fork_handler_list +#define DYNARRAY_PREFIX fork_handler_list_ +#define DYNARRAY_INITIAL_SIZE 48 +#include -struct fork_handler *__fork_handlers; - -/* Lock to protect allocation and deallocation of fork handlers. */ -int __fork_lock = LLL_LOCK_INITIALIZER; - - -/* Number of pre-allocated handler entries. */ -#define NHANDLER 48 - -/* Memory pool for fork handler structures. */ -static struct fork_handler_pool -{ - struct fork_handler_pool *next; - struct fork_handler mem[NHANDLER]; -} fork_handler_pool; - - -static struct fork_handler * -fork_handler_alloc (void) -{ - struct fork_handler_pool *runp = &fork_handler_pool; - struct fork_handler *result = NULL; - unsigned int i; - - do - { - /* Search for an empty entry. */ - for (i = 0; i < NHANDLER; ++i) - if (runp->mem[i].refcntr == 0) - goto found; - } - while ((runp = runp->next) != NULL); - - /* We have to allocate a new entry. */ - runp = (struct fork_handler_pool *) calloc (1, sizeof (*runp)); - if (runp != NULL) - { - /* Enqueue the new memory pool into the list. */ - runp->next = fork_handler_pool.next; - fork_handler_pool.next = runp; - - /* We use the last entry on the page. This means when we start - searching from the front the next time we will find the first - entry unused. */ - i = NHANDLER - 1; - - found: - result = &runp->mem[i]; - result->refcntr = 1; - result->need_signal = 0; - } - - return result; -} +static struct fork_handler_list fork_handlers; +static bool fork_handler_init = false; +static int atfork_lock = LLL_LOCK_INITIALIZER; int __register_atfork (void (*prepare) (void), void (*parent) (void), void (*child) (void), void *dso_handle) { - /* Get the lock to not conflict with other allocations. */ - lll_lock (__fork_lock, LLL_PRIVATE); + lll_lock (atfork_lock, LLL_PRIVATE); - struct fork_handler *newp = fork_handler_alloc (); + if (!fork_handler_init) + { + fork_handler_list_init (&fork_handlers); + fork_handler_init = true; + } + struct fork_handler *newp = fork_handler_list_emplace (&fork_handlers); if (newp != NULL) { - /* Initialize the new record. */ newp->prepare_handler = prepare; newp->parent_handler = parent; newp->child_handler = child; newp->dso_handle = dso_handle; - - __linkin_atfork (newp); } /* Release the lock. */ - lll_unlock (__fork_lock, LLL_PRIVATE); + lll_unlock (atfork_lock, LLL_PRIVATE); return newp == NULL ? ENOMEM : 0; } libc_hidden_def (__register_atfork) - void -attribute_hidden -__linkin_atfork (struct fork_handler *newp) +__unregister_atfork (void *dso_handle) { - do - newp->next = __fork_handlers; - while (catomic_compare_and_exchange_bool_acq (&__fork_handlers, - newp, newp->next) != 0); -} + lll_lock (atfork_lock, LLL_PRIVATE); + for (size_t i = 0; i < fork_handler_list_size (&fork_handlers); i++) + if (fork_handler_list_at (&fork_handlers, i)->dso_handle == dso_handle) + { + fork_handler_list_remove (&fork_handlers, i); + break; + } -libc_freeres_fn (free_mem) + lll_unlock (atfork_lock, LLL_PRIVATE); +} + +void +__run_fork_handlers (enum __run_fork_handler_type who) { - /* Get the lock to not conflict with running forks. */ - lll_lock (__fork_lock, LLL_PRIVATE); + struct fork_handler *runp; - /* No more fork handlers. */ - __fork_handlers = NULL; + if (who == atfork_run_prepare) + { + lll_lock (atfork_lock, LLL_PRIVATE); + size_t sl = fork_handler_list_size (&fork_handlers); + for (size_t i = sl; i > 0; i--) + { + runp = fork_handler_list_at (&fork_handlers, i - 1); + if (runp->prepare_handler != NULL) + runp->prepare_handler (); + } + } + else + { + size_t sl = fork_handler_list_size (&fork_handlers); + for (size_t i = 0; i < sl; i++) + { + runp = fork_handler_list_at (&fork_handlers, i); + if (who == atfork_run_child && runp->child_handler) + runp->child_handler (); + else if (who == atfork_run_parent && runp->parent_handler) + runp->parent_handler (); + } + lll_unlock (atfork_lock, LLL_PRIVATE); + } +} - /* Free eventually allocated memory blocks for the object pool. */ - struct fork_handler_pool *runp = fork_handler_pool.next; - memset (&fork_handler_pool, '\0', sizeof (fork_handler_pool)); +libc_freeres_fn (free_mem) +{ + lll_lock (atfork_lock, LLL_PRIVATE); - /* Release the lock. */ - lll_unlock (__fork_lock, LLL_PRIVATE); + fork_handler_list_free (&fork_handlers); - /* We can free the memory after releasing the lock. */ - while (runp != NULL) - { - struct fork_handler_pool *oldp = runp; - runp = runp->next; - free (oldp); - } + lll_unlock (atfork_lock, LLL_PRIVATE); } diff --git a/sysdeps/nptl/fork.c b/sysdeps/nptl/fork.c index 0061ee0..ec56a82 100644 --- a/sysdeps/nptl/fork.c +++ b/sysdeps/nptl/fork.c @@ -48,11 +48,6 @@ pid_t __libc_fork (void) { pid_t pid; - struct used_handler - { - struct fork_handler *handler; - struct used_handler *next; - } *allp = NULL; /* Determine if we are running multiple threads. We skip some fork handlers in the single-thread case, to make fork safer to use in @@ -60,60 +55,7 @@ __libc_fork (void) but our current fork implementation is not. */ bool multiple_threads = THREAD_GETMEM (THREAD_SELF, header.multiple_threads); - /* Run all the registered preparation handlers. In reverse order. - While doing this we build up a list of all the entries. */ - struct fork_handler *runp; - while ((runp = __fork_handlers) != NULL) - { - /* Make sure we read from the current RUNP pointer. */ - atomic_full_barrier (); - - unsigned int oldval = runp->refcntr; - - if (oldval == 0) - /* This means some other thread removed the list just after - the pointer has been loaded. Try again. Either the list - is empty or we can retry it. */ - continue; - - /* Bump the reference counter. */ - if (atomic_compare_and_exchange_bool_acq (&__fork_handlers->refcntr, - oldval + 1, oldval)) - /* The value changed, try again. */ - continue; - - /* We bumped the reference counter for the first entry in the - list. That means that none of the following entries will - just go away. The unloading code works in the order of the - list. - - While executing the registered handlers we are building a - list of all the entries so that we can go backward later on. */ - while (1) - { - /* Execute the handler if there is one. */ - if (runp->prepare_handler != NULL) - runp->prepare_handler (); - - /* Create a new element for the list. */ - struct used_handler *newp - = (struct used_handler *) alloca (sizeof (*newp)); - newp->handler = runp; - newp->next = allp; - allp = newp; - - /* Advance to the next handler. */ - runp = runp->next; - if (runp == NULL) - break; - - /* Bump the reference counter for the next entry. */ - atomic_increment (&runp->refcntr); - } - - /* We are done. */ - break; - } + __run_fork_handlers (atfork_run_prepare); /* If we are not running multiple threads, we do not have to preserve lock state. If fork runs from a signal handler, only @@ -192,29 +134,7 @@ __libc_fork (void) __rtld_lock_initialize (GL(dl_load_lock)); /* Run the handlers registered for the child. */ - while (allp != NULL) - { - if (allp->handler->child_handler != NULL) - allp->handler->child_handler (); - - /* Note that we do not have to wake any possible waiter. - This is the only thread in the new process. The count - may have been bumped up by other threads doing a fork. - We reset it to 1, to avoid waiting for non-existing - thread(s) to release the count. */ - allp->handler->refcntr = 1; - - /* XXX We could at this point look through the object pool - and mark all objects not on the __fork_handlers list as - unused. This is necessary in case the fork() happened - while another thread called dlclose() and that call had - to create a new list. */ - - allp = allp->next; - } - - /* Initialize the fork lock. */ - __fork_lock = LLL_LOCK_INITIALIZER; + __run_fork_handlers (atfork_run_child); } else { @@ -229,17 +149,7 @@ __libc_fork (void) } /* Run the handlers registered for the parent. */ - while (allp != NULL) - { - if (allp->handler->parent_handler != NULL) - allp->handler->parent_handler (); - - if (atomic_decrement_and_test (&allp->handler->refcntr) - && allp->handler->need_signal) - futex_wake (&allp->handler->refcntr, 1, FUTEX_PRIVATE); - - allp = allp->next; - } + __run_fork_handlers (atfork_run_parent); } return pid; diff --git a/sysdeps/nptl/fork.h b/sysdeps/nptl/fork.h index f0330cc..6eab61c 100644 --- a/sysdeps/nptl/fork.h +++ b/sysdeps/nptl/fork.h @@ -24,29 +24,37 @@ extern unsigned long int __fork_generation attribute_hidden; /* Pointer to the fork generation counter in the thread library. */ extern unsigned long int *__fork_generation_pointer attribute_hidden; -/* Lock to protect allocation and deallocation of fork handlers. */ -extern int __fork_lock attribute_hidden; - /* Elements of the fork handler lists. */ struct fork_handler { - struct fork_handler *next; void (*prepare_handler) (void); void (*parent_handler) (void); void (*child_handler) (void); void *dso_handle; - unsigned int refcntr; - int need_signal; }; -/* The single linked list of all currently registered for handlers. */ -extern struct fork_handler *__fork_handlers attribute_hidden; - - /* Function to call to unregister fork handlers. */ extern void __unregister_atfork (void *dso_handle) attribute_hidden; #define UNREGISTER_ATFORK(dso_handle) __unregister_atfork (dso_handle) +enum __run_fork_handler_type +{ + atfork_run_prepare, + atfork_run_child, + atfork_run_parent +}; + +/* Run the atfork handlers and lock/unlock the internal lock depending + of the WHO argument: + + - atfork_run_prepare: run all the PREPARE_HANDLER in reverse order of + insertion and locks the internal lock. + - atfork_run_child: run all the CHILD_HANDLER and unlocks the internal + lock. + - atfork_run_parent: run all the PARENT_HANDLER and unlocks the internal + lock. */ +extern void __run_fork_handlers (enum __run_fork_handler_type who) + attribute_hidden; /* C library side function to register new fork handlers. */ extern int __register_atfork (void (*__prepare) (void), @@ -54,6 +62,3 @@ extern int __register_atfork (void (*__prepare) (void), void (*__child) (void), void *dso_handle); libc_hidden_proto (__register_atfork) - -/* Add a new element to the fork list. */ -extern void __linkin_atfork (struct fork_handler *newp) attribute_hidden; diff --git a/sysdeps/nptl/libc-lockP.h b/sysdeps/nptl/libc-lockP.h index 8539bbf..989fefa 100644 --- a/sysdeps/nptl/libc-lockP.h +++ b/sysdeps/nptl/libc-lockP.h @@ -319,8 +319,6 @@ __libc_cleanup_routine (struct __pthread_cleanup_frame *f) /* Register handlers to execute before and after `fork'. Note that the last parameter is NULL. The handlers registered by the libc are never removed so this is OK. */ -#define __libc_atfork(PREPARE, PARENT, CHILD) \ - __register_atfork (PREPARE, PARENT, CHILD, NULL) extern int __register_atfork (void (*__prepare) (void), void (*__parent) (void), void (*__child) (void),