From patchwork Fri Aug 28 21:18:57 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bill Fischofer X-Patchwork-Id: 52814 Return-Path: X-Original-To: linaro@patches.linaro.org Delivered-To: linaro@patches.linaro.org Received: from mail-la0-f72.google.com (mail-la0-f72.google.com [209.85.215.72]) by patches.linaro.org (Postfix) with ESMTPS id E119122EBF for ; Fri, 28 Aug 2015 21:25:36 +0000 (UTC) Received: by laboe4 with SMTP id oe4sf7821624lab.3 for ; Fri, 28 Aug 2015 14:25:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:delivered-to:delivered-to:from:to:date :message-id:in-reply-to:references:subject:precedence:list-id :list-unsubscribe:list-archive:list-post:list-help:list-subscribe :mime-version:content-type:content-transfer-encoding:errors-to :sender:x-original-sender:x-original-authentication-results :mailing-list; bh=RwJqPd3mQ1GOynO8yLKPz8aSi9/hInu+gZLLjEnOXfA=; b=gdkfE717khdyEU1HNGUFx4PFzJyAljw1IOOXQ3aKdZJey30wi48bW+tdz1h0FL4d2g mrSJpugRZdA6JN/WaF8YVGtTPEWAHOZiNXmCZfqh3J/TkCaNTLQAeZvcpc/0iGQphDWh HegKgCAKMBmU2QG4HombFPiekygpUaFJiN6/dTctcFX59pkKvP37akQu+Nbj7QjXB6Bx jZaWcA7S51YDdU0UjaMxjD3klosxSZhqLDjOP5RTGMJ8B+Go627EDdNIIaehm3akG+Le pD5gXuHj0K+8Js8oGSFE57QgFB4MlVaICxkYogxJqKSUxr0H+EQy64gKrt7ypfntVRo5 zjKg== X-Gm-Message-State: ALoCoQlv8l6yY4u6Ez954K5gsDRIPkeQmWVcT45WH2slnBNRSJ0g9OrSN8x1XJfaX538XPRPjjux X-Received: by 10.180.90.106 with SMTP id bv10mr1670916wib.6.1440797135882; Fri, 28 Aug 2015 14:25:35 -0700 (PDT) X-BeenThere: patchwork-forward@linaro.org Received: by 10.152.27.234 with SMTP id w10ls256361lag.41.gmail; Fri, 28 Aug 2015 14:25:35 -0700 (PDT) X-Received: by 10.152.23.167 with SMTP id n7mr5773781laf.108.1440797135719; Fri, 28 Aug 2015 14:25:35 -0700 (PDT) Received: from mail-la0-f47.google.com (mail-la0-f47.google.com. [209.85.215.47]) by mx.google.com with ESMTPS id g6si6918237lbo.47.2015.08.28.14.25.35 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 28 Aug 2015 14:25:35 -0700 (PDT) Received-SPF: pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.215.47 as permitted sender) client-ip=209.85.215.47; Received: by laboe4 with SMTP id oe4so14238057lab.0 for ; Fri, 28 Aug 2015 14:25:35 -0700 (PDT) X-Received: by 10.112.209.106 with SMTP id ml10mr5656015lbc.112.1440797135592; Fri, 28 Aug 2015 14:25:35 -0700 (PDT) X-Forwarded-To: patchwork-forward@linaro.org X-Forwarded-For: patch@linaro.org patchwork-forward@linaro.org Delivered-To: patch@linaro.org Received: by 10.112.151.194 with SMTP id us2csp128696lbb; Fri, 28 Aug 2015 14:25:33 -0700 (PDT) X-Received: by 10.50.107.70 with SMTP id ha6mr4694699igb.75.1440797133145; Fri, 28 Aug 2015 14:25:33 -0700 (PDT) Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id vf13si3373400igb.16.2015.08.28.14.25.32; Fri, 28 Aug 2015 14:25:33 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Received: by lists.linaro.org (Postfix, from userid 109) id 29FF661988; Fri, 28 Aug 2015 21:25:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 93D9461D2C; Fri, 28 Aug 2015 21:20:00 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id E49A761D08; Fri, 28 Aug 2015 21:19:53 +0000 (UTC) Received: from mail-ob0-f179.google.com (mail-ob0-f179.google.com [209.85.214.179]) by lists.linaro.org (Postfix) with ESMTPS id 7A36361D27 for ; Fri, 28 Aug 2015 21:19:09 +0000 (UTC) Received: by obbfr1 with SMTP id fr1so55153529obb.1 for ; Fri, 28 Aug 2015 14:19:09 -0700 (PDT) X-Received: by 10.60.48.102 with SMTP id k6mr6875568oen.43.1440796748954; Fri, 28 Aug 2015 14:19:08 -0700 (PDT) Received: from Ubuntu15.localdomain (cpe-24-28-70-239.austin.res.rr.com. [24.28.70.239]) by smtp.gmail.com with ESMTPSA id k5sm4297965oev.16.2015.08.28.14.19.08 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 28 Aug 2015 14:19:08 -0700 (PDT) From: Bill Fischofer To: lng-odp@lists.linaro.org Date: Fri, 28 Aug 2015 16:18:57 -0500 Message-Id: <1440796737-8636-11-git-send-email-bill.fischofer@linaro.org> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1440796737-8636-1-git-send-email-bill.fischofer@linaro.org> References: <1440796737-8636-1-git-send-email-bill.fischofer@linaro.org> X-Topics: patch Subject: [lng-odp] [API-NEXT PATCHv4 10/10] linux-generic: schedule: implement ordered locks X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: , List-Help: , List-Subscribe: , MIME-Version: 1.0 Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" X-Removed-Original-Auth: Dkim didn't pass. X-Original-Sender: bill.fischofer@linaro.org X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of patch+caf_=patchwork-forward=linaro.org@linaro.org designates 209.85.215.47 as permitted sender) smtp.mailfrom=patch+caf_=patchwork-forward=linaro.org@linaro.org Mailing-list: list patchwork-forward@linaro.org; contact patchwork-forward+owners@linaro.org X-Google-Group-Id: 836684582541 Signed-off-by: Bill Fischofer --- .../linux-generic/include/odp_buffer_internal.h | 3 ++ .../linux-generic/include/odp_queue_internal.h | 3 ++ platform/linux-generic/odp_queue.c | 48 ++++++++++++++++++++++ platform/linux-generic/odp_schedule.c | 7 ++++ 4 files changed, 61 insertions(+) diff --git a/platform/linux-generic/include/odp_buffer_internal.h b/platform/linux-generic/include/odp_buffer_internal.h index ca4d314..6badeba 100644 --- a/platform/linux-generic/include/odp_buffer_internal.h +++ b/platform/linux-generic/include/odp_buffer_internal.h @@ -140,7 +140,10 @@ typedef struct odp_buffer_hdr_t { void *addr[ODP_BUFFER_MAX_SEG]; /* block addrs */ uint64_t order; /* sequence for ordered queues */ queue_entry_t *origin_qe; /* ordered queue origin */ + union { queue_entry_t *target_qe; /* ordered queue target */ + uint64_t sync; /* for ordered synchronization */ + }; } odp_buffer_hdr_t; /** @internal Compile time assert that the diff --git a/platform/linux-generic/include/odp_queue_internal.h b/platform/linux-generic/include/odp_queue_internal.h index 163172c..4cee9b6 100644 --- a/platform/linux-generic/include/odp_queue_internal.h +++ b/platform/linux-generic/include/odp_queue_internal.h @@ -82,6 +82,8 @@ struct queue_entry_s { uint64_t order_out; odp_buffer_hdr_t *reorder_head; odp_buffer_hdr_t *reorder_tail; + odp_atomic_u64_t sync_in; + odp_atomic_u64_t sync_out; }; typedef union queue_entry_u { @@ -120,6 +122,7 @@ int queue_sched_atomic(odp_queue_t handle); int release_order(queue_entry_t *origin_qe, uint64_t order, odp_pool_t pool, int enq_called); void get_sched_order(queue_entry_t **origin_qe, uint64_t *order); +void get_sched_sync(queue_entry_t **origin_qe, uint64_t **sync); void sched_enq_called(void); void sched_order_resolved(odp_buffer_hdr_t *buf_hdr); diff --git a/platform/linux-generic/odp_queue.c b/platform/linux-generic/odp_queue.c index 09b0398..1bd0de6 100644 --- a/platform/linux-generic/odp_queue.c +++ b/platform/linux-generic/odp_queue.c @@ -123,6 +123,8 @@ int odp_queue_init_global(void) /* init locks */ queue_entry_t *queue = get_qentry(i); LOCK_INIT(&queue->s.lock); + odp_atomic_init_u64(&queue->s.sync_in, 0); + odp_atomic_init_u64(&queue->s.sync_out, 0); queue->s.handle = queue_from_id(i); } @@ -599,6 +601,7 @@ odp_buffer_hdr_t *queue_deq(queue_entry_t *queue) if (queue_is_ordered(queue)) { buf_hdr->origin_qe = queue; buf_hdr->order = queue->s.order_in++; + buf_hdr->sync = odp_atomic_fetch_inc_u64(&queue->s.sync_in); buf_hdr->flags.sustain = 0; } else { buf_hdr->origin_qe = NULL; @@ -646,6 +649,8 @@ int queue_deq_multi(queue_entry_t *queue, odp_buffer_hdr_t *buf_hdr[], int num) if (queue_is_ordered(queue)) { buf_hdr[i]->origin_qe = queue; buf_hdr[i]->order = queue->s.order_in++; + buf_hdr[i]->sync = + odp_atomic_fetch_inc_u64(&queue->s.sync_in); buf_hdr[i]->flags.sustain = 0; } else { buf_hdr[i]->origin_qe = NULL; @@ -960,3 +965,46 @@ int release_order(queue_entry_t *origin_qe, uint64_t order, UNLOCK(&origin_qe->s.lock); return 0; } + +/* This routine is a no-op in linux-generic */ +int odp_schedule_order_lock_init(odp_schedule_order_lock_t *lock ODP_UNUSED, + odp_queue_t queue ODP_UNUSED) +{ + return 0; +} + +void odp_schedule_order_lock(odp_schedule_order_lock_t *lock ODP_UNUSED) +{ + queue_entry_t *origin_qe; + uint64_t *sync; + + get_sched_sync(&origin_qe, &sync); + if (!origin_qe) + return; + + /* Wait until we are in order. Note that sync_out will be incremented + * both by unlocks as well as order resolution, so we're OK if only + * some events in the ordered flow need to lock. + */ + while (*sync > odp_atomic_load_u64(&origin_qe->s.sync_out)) + odp_spin(); +} + +void odp_schedule_order_unlock(odp_schedule_order_lock_t *lock ODP_UNUSED) +{ + queue_entry_t *origin_qe; + uint64_t *sync; + + get_sched_sync(&origin_qe, &sync); + if (!origin_qe) + return; + + /* Get a new sync order for reusability, and release the lock. Note + * that this must be done in this sequence to prevent race conditions + * where the next waiter could lock and unlock before we're able to + * get a new sync order since that would cause order inversion on + * subsequent locks we may perform in this ordered context. + */ + *sync = odp_atomic_fetch_inc_u64(&origin_qe->s.sync_in); + odp_atomic_fetch_inc_u64(&origin_qe->s.sync_out); +} diff --git a/platform/linux-generic/odp_schedule.c b/platform/linux-generic/odp_schedule.c index 1c6cdbb..dc2d75f 100644 --- a/platform/linux-generic/odp_schedule.c +++ b/platform/linux-generic/odp_schedule.c @@ -86,6 +86,7 @@ typedef struct { queue_entry_t *qe; queue_entry_t *origin_qe; uint64_t order; + uint64_t sync; odp_pool_t pool; int enq_called; int num; @@ -799,6 +800,12 @@ void get_sched_order(queue_entry_t **origin_qe, uint64_t *order) *order = sched_local.order; } +void get_sched_sync(queue_entry_t **origin_qe, uint64_t **sync) +{ + *origin_qe = sched_local.origin_qe; + *sync = &sched_local.sync; +} + void sched_order_resolved(odp_buffer_hdr_t *buf_hdr) { if (buf_hdr)