From patchwork Sat Mar 19 17:30:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 553112 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07794C433FE for ; Sat, 19 Mar 2022 17:30:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243004AbiCSRcE (ORCPT ); Sat, 19 Mar 2022 13:32:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33602 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234986AbiCSRcD (ORCPT ); Sat, 19 Mar 2022 13:32:03 -0400 Received: from mail-pg1-x536.google.com (mail-pg1-x536.google.com [IPv6:2607:f8b0:4864:20::536]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EA39745506; Sat, 19 Mar 2022 10:30:41 -0700 (PDT) Received: by mail-pg1-x536.google.com with SMTP id o23so7348920pgk.13; Sat, 19 Mar 2022 10:30:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=DaQ7tIpYP0FN+ysFq8lKyeNWnVlaqiQkKqC0wh+aN8U=; b=gIIgdf3++Eq+4vAKrm3Bmzg6y37nT58dS7sKmznA/0Z0he8n3pQlzCSRFAZMUFkFUA a27Yi/AC9krchK6+tcCHfx+fOHarQKIpX/4zr3dJjpv/4iG8fDQcXDV2w1ehhqEfpzwz Tv2WW1/NT238AIev0argguYbMWsJVobYGPInR2KvXuJ8TVOe87qHdUckLRmNXaDEjJp7 p46jaVoNgeVwj5kXnrBpCbh3Gmc5lVNo78u8b/t18KVBwcqEQrclITzz5vOht+Ru1GNc tQ2vtQsBFanMpYrxzVmrMViPI4TSIiqZJI5migoLRX6ngj+g6DHtyXKYGTYF1wiYqb9/ TbjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DaQ7tIpYP0FN+ysFq8lKyeNWnVlaqiQkKqC0wh+aN8U=; b=znoPK8vllSaTYi6VaUSeWwXjxm1ZEincK+XnOWnnWV9A+B6xuzZU6T8m+MLkjX5zJf El5oIEEUGJdKPV/rnNw6h6Nhq6/EMbx3KE1AMMWQSRd8g9J1bJhSmXreVKN2Vczakona nRg4njX4T/8soN8eIKm82FV3ARDi869c2ru6Z+mKIOdNDQaikLDnpy1NyMoqI4tRBLcO t4+E8bZMSfvixWxMCSH7yYCY+gmlN7GSs2iW2RGCp+6s/MZYFctT46kmMtznh0mBQrne BxvfD+Vo2HKRpmHhhMNW1F5yxstipFw00SMc+T//W7vEh8d+OLtnmn6bRN8ot0rxndzG sTOA== X-Gm-Message-State: AOAM5322iDhWIughPAiMvn/m/he2Ohrn9AYf5DM4oZKyU4zKbXhxnBYh wmxCD1AkkKjQVm3e/J5GmEo= X-Google-Smtp-Source: ABdhPJzJfpVsPFMvRvZxlinLbP2ikPf+r/4KhMLR5qGL9f8tBztSFPKGq/Oaa6WjOipKpgwMZJ0XWQ== X-Received: by 2002:a63:ef4f:0:b0:375:6898:f6b8 with SMTP id c15-20020a63ef4f000000b003756898f6b8mr12183943pgk.59.1647711041349; Sat, 19 Mar 2022 10:30:41 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:4ab8:5400:3ff:fee9:a154]) by smtp.gmail.com with ESMTPSA id k21-20020aa788d5000000b004f71bff2893sm12722136pff.67.2022.03.19.10.30.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 19 Mar 2022 10:30:41 -0700 (PDT) From: Yafang Shao To: roman.gushchin@linux.dev, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, shuah@kernel.org Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, Yafang Shao Subject: [PATCH 01/14] bpf: Introduce no charge flag for bpf map Date: Sat, 19 Mar 2022 17:30:23 +0000 Message-Id: <20220319173036.23352-2-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220319173036.23352-1-laoar.shao@gmail.com> References: <20220319173036.23352-1-laoar.shao@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org A new map flag BPF_F_NO_CHARGE is introduced in bpf_attr, with which we can choose not to charge map memory while account it to root memcg only. At the map creation time, we can get the no charge flag from struct bpf_attr directly, while for other paths we can get it from struct bpf_map. The usecase of this flag is that sometimes we may create bpf maps with a process running in a container (with memcg) but these maps are targeted to the whole system, so we don't want to charge these memory into this container. That will be good for memory resource management for this container, as these shared bpf maps are always pinned which should belong to the system rather than this container. That can also help to make the charging behavior consistent, for example, if we charge the pinned memory into this container, after the contianer restarts these memory will not belong to it any more. Two helpers are introduced for followup usage. Signed-off-by: Yafang Shao --- include/linux/bpf.h | 15 ++++++++++++++- include/uapi/linux/bpf.h | 3 +++ kernel/bpf/syscall.c | 1 + tools/include/uapi/linux/bpf.h | 3 +++ 4 files changed, 21 insertions(+), 1 deletion(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 88449fbbe063..07c6603a6c81 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -184,7 +184,8 @@ struct bpf_map { char name[BPF_OBJ_NAME_LEN]; bool bypass_spec_v1; bool frozen; /* write-once; write-protected by freeze_mutex */ - /* 14 bytes hole */ + bool no_charge; /* Don't charge to memcg */ + /* 13 bytes hole */ /* The 3rd and 4th cacheline with misc members to avoid false sharing * particularly with refcounting. @@ -207,6 +208,18 @@ struct bpf_map { } owner; }; +static inline gfp_t +map_flags_no_charge(gfp_t flags, union bpf_attr *attr) +{ + return flags |= (attr->map_flags & BPF_F_NO_CHARGE) ? 0 : __GFP_ACCOUNT; +} + +static inline gfp_t +bpf_flags_no_charge(gfp_t flags, bool no_charge) +{ + return flags |= no_charge ? 0 : __GFP_ACCOUNT; +} + static inline bool map_value_has_spin_lock(const struct bpf_map *map) { return map->spin_lock_off >= 0; diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 7604e7d5438f..e2dba6cdd88d 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -1225,6 +1225,9 @@ enum { /* Create a map that is suitable to be an inner map with dynamic max entries */ BPF_F_INNER_MAP = (1U << 12), + +/* Don't charge memory to memcg */ + BPF_F_NO_CHARGE = (1U << 13), }; /* Flags for BPF_PROG_QUERY. */ diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index cdaa1152436a..029f04588b1a 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -368,6 +368,7 @@ void bpf_map_init_from_attr(struct bpf_map *map, union bpf_attr *attr) map->map_flags = bpf_map_flags_retain_permanent(attr->map_flags); map->numa_node = bpf_map_attr_numa_node(attr); map->map_extra = attr->map_extra; + map->no_charge = attr->map_flags & BPF_F_NO_CHARGE; } static int bpf_map_alloc_id(struct bpf_map *map) diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 7604e7d5438f..e2dba6cdd88d 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -1225,6 +1225,9 @@ enum { /* Create a map that is suitable to be an inner map with dynamic max entries */ BPF_F_INNER_MAP = (1U << 12), + +/* Don't charge memory to memcg */ + BPF_F_NO_CHARGE = (1U << 13), }; /* Flags for BPF_PROG_QUERY. */ From patchwork Sat Mar 19 17:30:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 553111 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EAE42C433EF for ; Sat, 19 Mar 2022 17:30:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243646AbiCSRcM (ORCPT ); Sat, 19 Mar 2022 13:32:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33824 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243300AbiCSRcF (ORCPT ); Sat, 19 Mar 2022 13:32:05 -0400 Received: from mail-pf1-x435.google.com (mail-pf1-x435.google.com [IPv6:2607:f8b0:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 564CB45505; Sat, 19 Mar 2022 10:30:44 -0700 (PDT) Received: by mail-pf1-x435.google.com with SMTP id s11so12095356pfu.13; Sat, 19 Mar 2022 10:30:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pfPDh91liqZm+BP4mghw0y9pBJj8lD1CecCVSBxX9RQ=; b=LiVGSCcXA6fHEXuuBuyhZr2cUPqAXeiwwy8Is9dF8j216O3XiRVmmBHNuvfaDGVv19 741ttud2pKa+CpCpdY27U034+eAiYDX4yOgjoX7e8VxXXWL+ze7fsqT7HfKNQgfXKH2o zZejoBD6UhxhZjz+912GqD8nwU+A/+GM99pTo1lSHZp8b9yP0z5AV3BUytsevTo742tr xeWn53M5qe+Uw2hkQIxmpQp22r6PCBMPEjxQd9HbJoQ7BcJYu1Hob0OSCGKWGveTpxK3 GJTitbn7tITZXV9f8dAQTy2ibHHjr/ZTfwzMoR8TAsnCAC7G1rhRfkwpbnyn99q9FD9X AcHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pfPDh91liqZm+BP4mghw0y9pBJj8lD1CecCVSBxX9RQ=; b=nQcSaYaWPeSuc6mQNWPA9HNgd18/JEYQmNt9VcEB7Lj5tqFjRghvhdB7N9HOzkLmEd Df56jECbHxtGz2LekvYFoXkuoZ/5NCOOB4EjTQwY5iVDValkTKPYtVFZH9RGx9qA9/Ms E0yfdRP+qy/B8vSIJe8L/xUwQBcbwGqFYujD89K16wkX4NClOrotOujHRwFhFbrKKFIV ysbyChz1t3+MSEe/8JMrRkhxRBtkvsdsG+Pf8H0oIrk3gDUyYOY9otPY2LldC30M5R96 JGz2cP6hBlNVhcPSnCHo4KbHaFPytN1uPcAzPBqA6lcbrgnmLMDzB0tFpe7/NfXO12ba QIdA== X-Gm-Message-State: AOAM530Y2UdL8bbWk1k6/DflN9E9U58H0gDXXkll1vKfZ7RvKQYVADyQ 0GF6z1jfypFe7N8UMwjrLPQ= X-Google-Smtp-Source: ABdhPJwXxdHfQIbhiS31nJsRloSgUOj73eXxsJggkaIikMTT44n46uK1Cl3+YoW4QIW7h+1LM3C1Zw== X-Received: by 2002:a63:af06:0:b0:378:3582:a49f with SMTP id w6-20020a63af06000000b003783582a49fmr12325169pge.125.1647711043808; Sat, 19 Mar 2022 10:30:43 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:4ab8:5400:3ff:fee9:a154]) by smtp.gmail.com with ESMTPSA id k21-20020aa788d5000000b004f71bff2893sm12722136pff.67.2022.03.19.10.30.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 19 Mar 2022 10:30:43 -0700 (PDT) From: Yafang Shao To: roman.gushchin@linux.dev, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, shuah@kernel.org Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, Yafang Shao Subject: [PATCH 03/14] bpf: Enable no charge in map _CREATE_FLAG_MASK Date: Sat, 19 Mar 2022 17:30:25 +0000 Message-Id: <20220319173036.23352-4-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220319173036.23352-1-laoar.shao@gmail.com> References: <20220319173036.23352-1-laoar.shao@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Many maps have their create masks to warn the invalid map flag. Set BPF_F_NO_CHARGE in all these masks to enable it. Signed-off-by: Yafang Shao --- kernel/bpf/arraymap.c | 2 +- kernel/bpf/bloom_filter.c | 4 ++-- kernel/bpf/bpf_local_storage.c | 3 ++- kernel/bpf/bpf_struct_ops.c | 5 ++++- kernel/bpf/cpumap.c | 5 ++++- kernel/bpf/devmap.c | 2 +- kernel/bpf/hashtab.c | 2 +- kernel/bpf/local_storage.c | 2 +- kernel/bpf/lpm_trie.c | 2 +- kernel/bpf/queue_stack_maps.c | 2 +- kernel/bpf/ringbuf.c | 2 +- kernel/bpf/stackmap.c | 2 +- net/core/sock_map.c | 2 +- net/xdp/xskmap.c | 5 ++++- 14 files changed, 25 insertions(+), 15 deletions(-) diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c index 7f145aefbff8..ac123747303c 100644 --- a/kernel/bpf/arraymap.c +++ b/kernel/bpf/arraymap.c @@ -16,7 +16,7 @@ #define ARRAY_CREATE_FLAG_MASK \ (BPF_F_NUMA_NODE | BPF_F_MMAPABLE | BPF_F_ACCESS_MASK | \ - BPF_F_PRESERVE_ELEMS | BPF_F_INNER_MAP) + BPF_F_PRESERVE_ELEMS | BPF_F_INNER_MAP | BPF_F_NO_CHARGE) static void bpf_array_free_percpu(struct bpf_array *array) { diff --git a/kernel/bpf/bloom_filter.c b/kernel/bpf/bloom_filter.c index b141a1346f72..f8ebfb4831e5 100644 --- a/kernel/bpf/bloom_filter.c +++ b/kernel/bpf/bloom_filter.c @@ -8,8 +8,8 @@ #include #include -#define BLOOM_CREATE_FLAG_MASK \ - (BPF_F_NUMA_NODE | BPF_F_ZERO_SEED | BPF_F_ACCESS_MASK) +#define BLOOM_CREATE_FLAG_MASK (BPF_F_NUMA_NODE | \ + BPF_F_ZERO_SEED | BPF_F_ACCESS_MASK | BPF_F_NO_CHARGE) struct bpf_bloom_filter { struct bpf_map map; diff --git a/kernel/bpf/bpf_local_storage.c b/kernel/bpf/bpf_local_storage.c index 092a1ac772d7..a92d3032fcde 100644 --- a/kernel/bpf/bpf_local_storage.c +++ b/kernel/bpf/bpf_local_storage.c @@ -15,7 +15,8 @@ #include #include -#define BPF_LOCAL_STORAGE_CREATE_FLAG_MASK (BPF_F_NO_PREALLOC | BPF_F_CLONE) +#define BPF_LOCAL_STORAGE_CREATE_FLAG_MASK \ + (BPF_F_NO_PREALLOC | BPF_F_CLONE | BPF_F_NO_CHARGE) static struct bpf_local_storage_map_bucket * select_bucket(struct bpf_local_storage_map *smap, diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c index 21069dbe9138..09eb848e6d12 100644 --- a/kernel/bpf/bpf_struct_ops.c +++ b/kernel/bpf/bpf_struct_ops.c @@ -21,6 +21,8 @@ enum bpf_struct_ops_state { refcount_t refcnt; \ enum bpf_struct_ops_state state +#define STRUCT_OPS_CREATE_FLAG_MASK (BPF_F_NO_CHARGE) + struct bpf_struct_ops_value { BPF_STRUCT_OPS_COMMON_VALUE; char data[] ____cacheline_aligned_in_smp; @@ -556,7 +558,8 @@ static void bpf_struct_ops_map_free(struct bpf_map *map) static int bpf_struct_ops_map_alloc_check(union bpf_attr *attr) { if (attr->key_size != sizeof(unsigned int) || attr->max_entries != 1 || - attr->map_flags || !attr->btf_vmlinux_value_type_id) + attr->map_flags & ~STRUCT_OPS_CREATE_FLAG_MASK || + !attr->btf_vmlinux_value_type_id) return -EINVAL; return 0; } diff --git a/kernel/bpf/cpumap.c b/kernel/bpf/cpumap.c index 650e5d21f90d..201226fc652b 100644 --- a/kernel/bpf/cpumap.c +++ b/kernel/bpf/cpumap.c @@ -39,6 +39,9 @@ */ #define CPU_MAP_BULK_SIZE 8 /* 8 == one cacheline on 64-bit archs */ + +#define CPU_MAP_CREATE_FLAG_MASK (BPF_F_NUMA_NODE | BPF_F_NO_CHARGE) + struct bpf_cpu_map_entry; struct bpf_cpu_map; @@ -93,7 +96,7 @@ static struct bpf_map *cpu_map_alloc(union bpf_attr *attr) if (attr->max_entries == 0 || attr->key_size != 4 || (value_size != offsetofend(struct bpf_cpumap_val, qsize) && value_size != offsetofend(struct bpf_cpumap_val, bpf_prog.fd)) || - attr->map_flags & ~BPF_F_NUMA_NODE) + attr->map_flags & ~CPU_MAP_CREATE_FLAG_MASK) return ERR_PTR(-EINVAL); cmap = kzalloc(sizeof(*cmap), GFP_USER | __GFP_ACCOUNT); diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index 038f6d7a83e4..39bf8b521f27 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -50,7 +50,7 @@ #include #define DEV_CREATE_FLAG_MASK \ - (BPF_F_NUMA_NODE | BPF_F_RDONLY | BPF_F_WRONLY) + (BPF_F_NUMA_NODE | BPF_F_RDONLY | BPF_F_WRONLY | BPF_F_NO_CHARGE) struct xdp_dev_bulk_queue { struct xdp_frame *q[DEV_MAP_BULK_SIZE]; diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index 65877967f414..5c6ec8780b09 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -16,7 +16,7 @@ #define HTAB_CREATE_FLAG_MASK \ (BPF_F_NO_PREALLOC | BPF_F_NO_COMMON_LRU | BPF_F_NUMA_NODE | \ - BPF_F_ACCESS_MASK | BPF_F_ZERO_SEED) + BPF_F_ACCESS_MASK | BPF_F_ZERO_SEED | BPF_F_NO_CHARGE) #define BATCH_OPS(_name) \ .map_lookup_batch = \ diff --git a/kernel/bpf/local_storage.c b/kernel/bpf/local_storage.c index 497916060ac7..865766c240d6 100644 --- a/kernel/bpf/local_storage.c +++ b/kernel/bpf/local_storage.c @@ -15,7 +15,7 @@ #include "../cgroup/cgroup-internal.h" #define LOCAL_STORAGE_CREATE_FLAG_MASK \ - (BPF_F_NUMA_NODE | BPF_F_ACCESS_MASK) + (BPF_F_NUMA_NODE | BPF_F_ACCESS_MASK | BPF_F_NO_CHARGE) struct bpf_cgroup_storage_map { struct bpf_map map; diff --git a/kernel/bpf/lpm_trie.c b/kernel/bpf/lpm_trie.c index 5763cc7ac4f1..f42edf613624 100644 --- a/kernel/bpf/lpm_trie.c +++ b/kernel/bpf/lpm_trie.c @@ -537,7 +537,7 @@ static int trie_delete_elem(struct bpf_map *map, void *_key) #define LPM_KEY_SIZE_MIN LPM_KEY_SIZE(LPM_DATA_SIZE_MIN) #define LPM_CREATE_FLAG_MASK (BPF_F_NO_PREALLOC | BPF_F_NUMA_NODE | \ - BPF_F_ACCESS_MASK) + BPF_F_ACCESS_MASK | BPF_F_NO_CHARGE) static struct bpf_map *trie_alloc(union bpf_attr *attr) { diff --git a/kernel/bpf/queue_stack_maps.c b/kernel/bpf/queue_stack_maps.c index f9c734aaa990..c78eed4659ce 100644 --- a/kernel/bpf/queue_stack_maps.c +++ b/kernel/bpf/queue_stack_maps.c @@ -11,7 +11,7 @@ #include "percpu_freelist.h" #define QUEUE_STACK_CREATE_FLAG_MASK \ - (BPF_F_NUMA_NODE | BPF_F_ACCESS_MASK) + (BPF_F_NUMA_NODE | BPF_F_ACCESS_MASK | BPF_F_NO_CHARGE) struct bpf_queue_stack { struct bpf_map map; diff --git a/kernel/bpf/ringbuf.c b/kernel/bpf/ringbuf.c index 710ba9de12ce..88779f688679 100644 --- a/kernel/bpf/ringbuf.c +++ b/kernel/bpf/ringbuf.c @@ -11,7 +11,7 @@ #include #include -#define RINGBUF_CREATE_FLAG_MASK (BPF_F_NUMA_NODE) +#define RINGBUF_CREATE_FLAG_MASK (BPF_F_NUMA_NODE | BPF_F_NO_CHARGE) /* non-mmap()'able part of bpf_ringbuf (everything up to consumer page) */ #define RINGBUF_PGOFF \ diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c index 38bdfcd06f55..b2e7dc1d9f5a 100644 --- a/kernel/bpf/stackmap.c +++ b/kernel/bpf/stackmap.c @@ -14,7 +14,7 @@ #define STACK_CREATE_FLAG_MASK \ (BPF_F_NUMA_NODE | BPF_F_RDONLY | BPF_F_WRONLY | \ - BPF_F_STACK_BUILD_ID) + BPF_F_STACK_BUILD_ID | BPF_F_NO_CHARGE) struct stack_map_bucket { struct pcpu_freelist_node fnode; diff --git a/net/core/sock_map.c b/net/core/sock_map.c index 2d213c4011db..7b0215bea413 100644 --- a/net/core/sock_map.c +++ b/net/core/sock_map.c @@ -22,7 +22,7 @@ struct bpf_stab { }; #define SOCK_CREATE_FLAG_MASK \ - (BPF_F_NUMA_NODE | BPF_F_RDONLY | BPF_F_WRONLY) + (BPF_F_NUMA_NODE | BPF_F_RDONLY | BPF_F_WRONLY | BPF_F_NO_CHARGE) static int sock_map_prog_update(struct bpf_map *map, struct bpf_prog *prog, struct bpf_prog *old, u32 which); diff --git a/net/xdp/xskmap.c b/net/xdp/xskmap.c index 65b53fb3de13..10a5ae727bd5 100644 --- a/net/xdp/xskmap.c +++ b/net/xdp/xskmap.c @@ -12,6 +12,9 @@ #include "xsk.h" +#define XSK_MAP_CREATE_FLAG_MASK \ + (BPF_F_NUMA_NODE | BPF_F_RDONLY | BPF_F_WRONLY | BPF_F_NO_CHARGE) + static struct xsk_map_node *xsk_map_node_alloc(struct xsk_map *map, struct xdp_sock __rcu **map_entry) { @@ -68,7 +71,7 @@ static struct bpf_map *xsk_map_alloc(union bpf_attr *attr) if (attr->max_entries == 0 || attr->key_size != 4 || attr->value_size != 4 || - attr->map_flags & ~(BPF_F_NUMA_NODE | BPF_F_RDONLY | BPF_F_WRONLY)) + attr->map_flags & ~XSK_MAP_CREATE_FLAG_MASK) return ERR_PTR(-EINVAL); numa_node = bpf_map_attr_numa_node(attr); From patchwork Sat Mar 19 17:30:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 553110 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A68E5C4332F for ; Sat, 19 Mar 2022 17:30:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243667AbiCSRcN (ORCPT ); Sat, 19 Mar 2022 13:32:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33964 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243028AbiCSRcH (ORCPT ); Sat, 19 Mar 2022 13:32:07 -0400 Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 99DAA45506; Sat, 19 Mar 2022 10:30:45 -0700 (PDT) Received: by mail-pj1-x1032.google.com with SMTP id bx24-20020a17090af49800b001c6872a9e4eso6908966pjb.5; Sat, 19 Mar 2022 10:30:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=59FYyOnE/ZfEE+/nWxi+Uw2ttiMNAftCWsX5hGEEb68=; b=K5CK3XbPteaXT9NC/Kt7ZweGODe0q1YbJP/or7LyRRCogBUhlcCxGunIOv4Rr42z0/ aLYZeU9LGqsD2ly1kCsXG2KH/njEAXgvUo7OFEPCN54Tt0Zr5+PJWhLCcfHxmHllqrbo OvYkhSz1Zt9vq3YSTHiOwkQ+EPq4fEUQpo+YzQi5mecwY3YEuY9svC4mQzse8AUJ73MC mXOYHHRQuA6b3X8nUzE29jOe5XMxu9boxYVfyxrwYQD1vREplesaUPQzaYD1RG7KKVh+ jgdbaOZG6hi18NmezjFFyTBr/gl3vOquUADRvmeQ2/zIUxidC0vgBglFYGGwP7jKC30A AekA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=59FYyOnE/ZfEE+/nWxi+Uw2ttiMNAftCWsX5hGEEb68=; b=DLS5R3pOCn/7yPgZ+UUybAO3WD3467kTpMpO1/4wbCieZjZ9Oi0pQGx7hWQyxtdAaK sW0vhM4zLIiVGP3WZ4JMO2Y0JEsv7IG3hUMHkAtLcUI4gv7CRhYwDguui2R2hkMHZPM4 4cnRnSm79XN3acuqlrjxRypLvp3PZxZR1D1PCsSGsIm8/xp2BODoCFPVFzG2xp3Cpyf9 +rr4ny8o5tKToGF/oSTnxNmrJyG2NfPbh6/jcf/IjHxF5yOFuY76CudmsbosyfvgUwQm CJSSIdyhzoRpZrXUBfSRs+4CfaSscdL5tNb7c3Qq42N4ZhaeEbLexgS0H5EUZhSFSgbe fNGA== X-Gm-Message-State: AOAM530ZMQLNiLTE/HXL/kbKxIF9ENvsflutpHhageYuHSMWjHQCDLOk 4A+bNRRPhqphWc8SSeUBb3E= X-Google-Smtp-Source: ABdhPJwfmnqCjaYsNruPjkN8VzGPjXY5rVHbzV6iD/dhuKWThJxTIsUb4037rl7jXLCT9K9X1zDU9Q== X-Received: by 2002:a17:902:db0c:b0:154:1b36:6a6b with SMTP id m12-20020a170902db0c00b001541b366a6bmr5091539plx.7.1647711045029; Sat, 19 Mar 2022 10:30:45 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:4ab8:5400:3ff:fee9:a154]) by smtp.gmail.com with ESMTPSA id k21-20020aa788d5000000b004f71bff2893sm12722136pff.67.2022.03.19.10.30.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 19 Mar 2022 10:30:44 -0700 (PDT) From: Yafang Shao To: roman.gushchin@linux.dev, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, shuah@kernel.org Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, Yafang Shao Subject: [PATCH 04/14] bpf: Introduce new parameter bpf_attr in bpf_map_area_alloc Date: Sat, 19 Mar 2022 17:30:26 +0000 Message-Id: <20220319173036.23352-5-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220319173036.23352-1-laoar.shao@gmail.com> References: <20220319173036.23352-1-laoar.shao@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Add a new parameter bpf_attr in bpf_map_area_alloc(), then we can get no charge flag from it. Currently there're two parameters, one of which is also got from bpf_attr, so we can remove it after this change. No functional change. Signed-off-by: Yafang Shao --- include/linux/bpf.h | 4 ++-- kernel/bpf/arraymap.c | 5 ++--- kernel/bpf/bloom_filter.c | 3 +-- kernel/bpf/bpf_struct_ops.c | 8 ++++---- kernel/bpf/cpumap.c | 3 +-- kernel/bpf/devmap.c | 10 ++++------ kernel/bpf/hashtab.c | 10 ++++------ kernel/bpf/queue_stack_maps.c | 3 +-- kernel/bpf/reuseport_array.c | 3 +-- kernel/bpf/ringbuf.c | 11 ++++++----- kernel/bpf/stackmap.c | 11 ++++++----- kernel/bpf/syscall.c | 13 +++++++------ net/core/sock_map.c | 6 ++---- net/xdp/xskmap.c | 4 +--- 14 files changed, 42 insertions(+), 52 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 07c6603a6c81..90a542d5a411 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1518,8 +1518,8 @@ void bpf_map_inc_with_uref(struct bpf_map *map); struct bpf_map * __must_check bpf_map_inc_not_zero(struct bpf_map *map); void bpf_map_put_with_uref(struct bpf_map *map); void bpf_map_put(struct bpf_map *map); -void *bpf_map_area_alloc(u64 size, int numa_node); -void *bpf_map_area_mmapable_alloc(u64 size, int numa_node); +void *bpf_map_area_alloc(u64 size, union bpf_attr *attr); +void *bpf_map_area_mmapable_alloc(u64 size, union bpf_attr *attr); void bpf_map_area_free(void *base); bool bpf_map_write_active(const struct bpf_map *map); void bpf_map_init_from_attr(struct bpf_map *map, union bpf_attr *attr); diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c index ac123747303c..e26aef906392 100644 --- a/kernel/bpf/arraymap.c +++ b/kernel/bpf/arraymap.c @@ -81,7 +81,6 @@ int array_map_alloc_check(union bpf_attr *attr) static struct bpf_map *array_map_alloc(union bpf_attr *attr) { bool percpu = attr->map_type == BPF_MAP_TYPE_PERCPU_ARRAY; - int numa_node = bpf_map_attr_numa_node(attr); u32 elem_size, index_mask, max_entries; bool bypass_spec_v1 = bpf_bypass_spec_v1(); u64 array_size, mask64; @@ -130,13 +129,13 @@ static struct bpf_map *array_map_alloc(union bpf_attr *attr) void *data; /* kmalloc'ed memory can't be mmap'ed, use explicit vmalloc */ - data = bpf_map_area_mmapable_alloc(array_size, numa_node); + data = bpf_map_area_mmapable_alloc(array_size, attr); if (!data) return ERR_PTR(-ENOMEM); array = data + PAGE_ALIGN(sizeof(struct bpf_array)) - offsetof(struct bpf_array, value); } else { - array = bpf_map_area_alloc(array_size, numa_node); + array = bpf_map_area_alloc(array_size, attr); } if (!array) return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/bloom_filter.c b/kernel/bpf/bloom_filter.c index f8ebfb4831e5..a35c664b4a02 100644 --- a/kernel/bpf/bloom_filter.c +++ b/kernel/bpf/bloom_filter.c @@ -90,7 +90,6 @@ static int bloom_map_get_next_key(struct bpf_map *map, void *key, void *next_key static struct bpf_map *bloom_map_alloc(union bpf_attr *attr) { u32 bitset_bytes, bitset_mask, nr_hash_funcs, nr_bits; - int numa_node = bpf_map_attr_numa_node(attr); struct bpf_bloom_filter *bloom; if (!bpf_capable()) @@ -141,7 +140,7 @@ static struct bpf_map *bloom_map_alloc(union bpf_attr *attr) } bitset_bytes = roundup(bitset_bytes, sizeof(unsigned long)); - bloom = bpf_map_area_alloc(sizeof(*bloom) + bitset_bytes, numa_node); + bloom = bpf_map_area_alloc(sizeof(*bloom) + bitset_bytes, attr); if (!bloom) return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c index 09eb848e6d12..1ca1407ae5e6 100644 --- a/kernel/bpf/bpf_struct_ops.c +++ b/kernel/bpf/bpf_struct_ops.c @@ -591,17 +591,17 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr) */ (vt->size - sizeof(struct bpf_struct_ops_value)); - st_map = bpf_map_area_alloc(st_map_size, NUMA_NO_NODE); + attr->map_flags &= ~BPF_F_NUMA_NODE; + st_map = bpf_map_area_alloc(st_map_size, attr); if (!st_map) return ERR_PTR(-ENOMEM); st_map->st_ops = st_ops; map = &st_map->map; - st_map->uvalue = bpf_map_area_alloc(vt->size, NUMA_NO_NODE); + st_map->uvalue = bpf_map_area_alloc(vt->size, attr); st_map->progs = - bpf_map_area_alloc(btf_type_vlen(t) * sizeof(struct bpf_prog *), - NUMA_NO_NODE); + bpf_map_area_alloc(btf_type_vlen(t) * sizeof(struct bpf_prog *), attr); st_map->image = bpf_jit_alloc_exec(PAGE_SIZE); if (!st_map->uvalue || !st_map->progs || !st_map->image) { bpf_struct_ops_map_free(map); diff --git a/kernel/bpf/cpumap.c b/kernel/bpf/cpumap.c index 201226fc652b..5a5b40e986ff 100644 --- a/kernel/bpf/cpumap.c +++ b/kernel/bpf/cpumap.c @@ -113,8 +113,7 @@ static struct bpf_map *cpu_map_alloc(union bpf_attr *attr) /* Alloc array for possible remote "destination" CPUs */ cmap->cpu_map = bpf_map_area_alloc(cmap->map.max_entries * - sizeof(struct bpf_cpu_map_entry *), - cmap->map.numa_node); + sizeof(struct bpf_cpu_map_entry *), attr); if (!cmap->cpu_map) goto free_cmap; diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index 39bf8b521f27..2857176c82bb 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -88,12 +88,12 @@ static DEFINE_SPINLOCK(dev_map_lock); static LIST_HEAD(dev_map_list); static struct hlist_head *dev_map_create_hash(unsigned int entries, - int numa_node) + union bpf_attr *attr) { int i; struct hlist_head *hash; - hash = bpf_map_area_alloc((u64) entries * sizeof(*hash), numa_node); + hash = bpf_map_area_alloc((u64) entries * sizeof(*hash), attr); if (hash != NULL) for (i = 0; i < entries; i++) INIT_HLIST_HEAD(&hash[i]); @@ -137,16 +137,14 @@ static int dev_map_init_map(struct bpf_dtab *dtab, union bpf_attr *attr) } if (attr->map_type == BPF_MAP_TYPE_DEVMAP_HASH) { - dtab->dev_index_head = dev_map_create_hash(dtab->n_buckets, - dtab->map.numa_node); + dtab->dev_index_head = dev_map_create_hash(dtab->n_buckets, attr); if (!dtab->dev_index_head) return -ENOMEM; spin_lock_init(&dtab->index_lock); } else { dtab->netdev_map = bpf_map_area_alloc((u64) dtab->map.max_entries * - sizeof(struct bpf_dtab_netdev *), - dtab->map.numa_node); + sizeof(struct bpf_dtab_netdev *), attr); if (!dtab->netdev_map) return -ENOMEM; } diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index 5c6ec8780b09..2c84045ff8e1 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -303,7 +303,7 @@ static struct htab_elem *prealloc_lru_pop(struct bpf_htab *htab, void *key, return NULL; } -static int prealloc_init(struct bpf_htab *htab) +static int prealloc_init(union bpf_attr *attr, struct bpf_htab *htab) { u32 num_entries = htab->map.max_entries; int err = -ENOMEM, i; @@ -311,8 +311,7 @@ static int prealloc_init(struct bpf_htab *htab) if (htab_has_extra_elems(htab)) num_entries += num_possible_cpus(); - htab->elems = bpf_map_area_alloc((u64)htab->elem_size * num_entries, - htab->map.numa_node); + htab->elems = bpf_map_area_alloc((u64)htab->elem_size * num_entries, attr); if (!htab->elems) return -ENOMEM; @@ -513,8 +512,7 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr) err = -ENOMEM; htab->buckets = bpf_map_area_alloc(htab->n_buckets * - sizeof(struct bucket), - htab->map.numa_node); + sizeof(struct bucket), attr); if (!htab->buckets) goto free_htab; @@ -535,7 +533,7 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr) htab_init_buckets(htab); if (prealloc) { - err = prealloc_init(htab); + err = prealloc_init(attr, htab); if (err) goto free_map_locked; diff --git a/kernel/bpf/queue_stack_maps.c b/kernel/bpf/queue_stack_maps.c index c78eed4659ce..0ff93c5bc184 100644 --- a/kernel/bpf/queue_stack_maps.c +++ b/kernel/bpf/queue_stack_maps.c @@ -66,14 +66,13 @@ static int queue_stack_map_alloc_check(union bpf_attr *attr) static struct bpf_map *queue_stack_map_alloc(union bpf_attr *attr) { - int numa_node = bpf_map_attr_numa_node(attr); struct bpf_queue_stack *qs; u64 size, queue_size; size = (u64) attr->max_entries + 1; queue_size = sizeof(*qs) + size * attr->value_size; - qs = bpf_map_area_alloc(queue_size, numa_node); + qs = bpf_map_area_alloc(queue_size, attr); if (!qs) return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/reuseport_array.c b/kernel/bpf/reuseport_array.c index 8251243022a2..b19fb70118a4 100644 --- a/kernel/bpf/reuseport_array.c +++ b/kernel/bpf/reuseport_array.c @@ -150,14 +150,13 @@ static void reuseport_array_free(struct bpf_map *map) static struct bpf_map *reuseport_array_alloc(union bpf_attr *attr) { - int numa_node = bpf_map_attr_numa_node(attr); struct reuseport_array *array; if (!bpf_capable()) return ERR_PTR(-EPERM); /* allocate all map elements and zero-initialize them */ - array = bpf_map_area_alloc(struct_size(array, ptrs, attr->max_entries), numa_node); + array = bpf_map_area_alloc(struct_size(array, ptrs, attr->max_entries), attr); if (!array) return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/ringbuf.c b/kernel/bpf/ringbuf.c index 88779f688679..a3b4d2a0a2c7 100644 --- a/kernel/bpf/ringbuf.c +++ b/kernel/bpf/ringbuf.c @@ -58,13 +58,14 @@ struct bpf_ringbuf_hdr { u32 pg_off; }; -static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) +static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, union bpf_attr *attr) { const gfp_t flags = GFP_KERNEL_ACCOUNT | __GFP_RETRY_MAYFAIL | __GFP_NOWARN | __GFP_ZERO; int nr_meta_pages = RINGBUF_PGOFF + RINGBUF_POS_PAGES; int nr_data_pages = data_sz >> PAGE_SHIFT; int nr_pages = nr_meta_pages + nr_data_pages; + int numa_node = bpf_map_attr_numa_node(attr); struct page **pages, *page; struct bpf_ringbuf *rb; size_t array_size; @@ -88,7 +89,7 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) * user-space implementations significantly. */ array_size = (nr_meta_pages + 2 * nr_data_pages) * sizeof(*pages); - pages = bpf_map_area_alloc(array_size, numa_node); + pages = bpf_map_area_alloc(array_size, attr); if (!pages) return NULL; @@ -126,11 +127,11 @@ static void bpf_ringbuf_notify(struct irq_work *work) wake_up_all(&rb->waitq); } -static struct bpf_ringbuf *bpf_ringbuf_alloc(size_t data_sz, int numa_node) +static struct bpf_ringbuf *bpf_ringbuf_alloc(size_t data_sz, union bpf_attr *attr) { struct bpf_ringbuf *rb; - rb = bpf_ringbuf_area_alloc(data_sz, numa_node); + rb = bpf_ringbuf_area_alloc(data_sz, attr); if (!rb) return NULL; @@ -169,7 +170,7 @@ static struct bpf_map *ringbuf_map_alloc(union bpf_attr *attr) bpf_map_init_from_attr(&rb_map->map, attr); - rb_map->rb = bpf_ringbuf_alloc(attr->max_entries, rb_map->map.numa_node); + rb_map->rb = bpf_ringbuf_alloc(attr->max_entries, attr); if (!rb_map->rb) { kfree(rb_map); return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c index b2e7dc1d9f5a..ed6bebef0132 100644 --- a/kernel/bpf/stackmap.c +++ b/kernel/bpf/stackmap.c @@ -42,14 +42,15 @@ static inline int stack_map_data_size(struct bpf_map *map) sizeof(struct bpf_stack_build_id) : sizeof(u64); } -static int prealloc_elems_and_freelist(struct bpf_stack_map *smap) +static int prealloc_elems_and_freelist(union bpf_attr *attr, + struct bpf_stack_map *smap) { u64 elem_size = sizeof(struct stack_map_bucket) + (u64)smap->map.value_size; int err; - smap->elems = bpf_map_area_alloc(elem_size * smap->map.max_entries, - smap->map.numa_node); + smap->elems = bpf_map_area_alloc(elem_size * + smap->map.max_entries, attr); if (!smap->elems) return -ENOMEM; @@ -101,7 +102,7 @@ static struct bpf_map *stack_map_alloc(union bpf_attr *attr) cost = n_buckets * sizeof(struct stack_map_bucket *) + sizeof(*smap); cost += n_buckets * (value_size + sizeof(struct stack_map_bucket)); - smap = bpf_map_area_alloc(cost, bpf_map_attr_numa_node(attr)); + smap = bpf_map_area_alloc(cost, attr); if (!smap) return ERR_PTR(-ENOMEM); @@ -113,7 +114,7 @@ static struct bpf_map *stack_map_alloc(union bpf_attr *attr) if (err) goto free_smap; - err = prealloc_elems_and_freelist(smap); + err = prealloc_elems_and_freelist(attr, smap); if (err) goto put_buffers; diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 0cca3d7d0d84..f70a7067ef4a 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -295,7 +295,7 @@ static int bpf_map_copy_value(struct bpf_map *map, void *key, void *value, * (e.g. in map update path) without taking care of setting the active * memory cgroup (see at bpf_map_kmalloc_node() for example). */ -static void *__bpf_map_area_alloc(u64 size, int numa_node, bool mmapable) +static void *__bpf_map_area_alloc(u64 size, union bpf_attr *attr, bool mmapable) { /* We really just want to fail instead of triggering OOM killer * under memory pressure, therefore we set __GFP_NORETRY to kmalloc, @@ -308,8 +308,9 @@ static void *__bpf_map_area_alloc(u64 size, int numa_node, bool mmapable) */ const gfp_t gfp = __GFP_NOWARN | __GFP_ZERO | __GFP_ACCOUNT; - unsigned int flags = 0; + int numa_node = bpf_map_attr_numa_node(attr); unsigned long align = 1; + unsigned int flags = 0; void *area; if (size >= SIZE_MAX) @@ -332,14 +333,14 @@ static void *__bpf_map_area_alloc(u64 size, int numa_node, bool mmapable) flags, numa_node, __builtin_return_address(0)); } -void *bpf_map_area_alloc(u64 size, int numa_node) +void *bpf_map_area_alloc(u64 size, union bpf_attr *attr) { - return __bpf_map_area_alloc(size, numa_node, false); + return __bpf_map_area_alloc(size, attr, false); } -void *bpf_map_area_mmapable_alloc(u64 size, int numa_node) +void *bpf_map_area_mmapable_alloc(u64 size, union bpf_attr *attr) { - return __bpf_map_area_alloc(size, numa_node, true); + return __bpf_map_area_alloc(size, attr, true); } void bpf_map_area_free(void *area) diff --git a/net/core/sock_map.c b/net/core/sock_map.c index 7b0215bea413..26b89d37944d 100644 --- a/net/core/sock_map.c +++ b/net/core/sock_map.c @@ -49,8 +49,7 @@ static struct bpf_map *sock_map_alloc(union bpf_attr *attr) raw_spin_lock_init(&stab->lock); stab->sks = bpf_map_area_alloc((u64) stab->map.max_entries * - sizeof(struct sock *), - stab->map.numa_node); + sizeof(struct sock *), attr); if (!stab->sks) { kfree(stab); return ERR_PTR(-ENOMEM); @@ -1093,8 +1092,7 @@ static struct bpf_map *sock_hash_alloc(union bpf_attr *attr) } htab->buckets = bpf_map_area_alloc(htab->buckets_num * - sizeof(struct bpf_shtab_bucket), - htab->map.numa_node); + sizeof(struct bpf_shtab_bucket), attr); if (!htab->buckets) { err = -ENOMEM; goto free_htab; diff --git a/net/xdp/xskmap.c b/net/xdp/xskmap.c index 10a5ae727bd5..50795c0c9b81 100644 --- a/net/xdp/xskmap.c +++ b/net/xdp/xskmap.c @@ -63,7 +63,6 @@ static void xsk_map_sock_delete(struct xdp_sock *xs, static struct bpf_map *xsk_map_alloc(union bpf_attr *attr) { struct xsk_map *m; - int numa_node; u64 size; if (!capable(CAP_NET_ADMIN)) @@ -74,10 +73,9 @@ static struct bpf_map *xsk_map_alloc(union bpf_attr *attr) attr->map_flags & ~XSK_MAP_CREATE_FLAG_MASK) return ERR_PTR(-EINVAL); - numa_node = bpf_map_attr_numa_node(attr); size = struct_size(m, xsk_map, attr->max_entries); - m = bpf_map_area_alloc(size, numa_node); + m = bpf_map_area_alloc(size, attr); if (!m) return ERR_PTR(-ENOMEM); From patchwork Sat Mar 19 17:30:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 553108 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0BC08C433EF for ; Sat, 19 Mar 2022 17:31:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243705AbiCSRcY (ORCPT ); Sat, 19 Mar 2022 13:32:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34330 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243632AbiCSRcM (ORCPT ); Sat, 19 Mar 2022 13:32:12 -0400 Received: from mail-pl1-x633.google.com (mail-pl1-x633.google.com [IPv6:2607:f8b0:4864:20::633]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3B9D445789; Sat, 19 Mar 2022 10:30:49 -0700 (PDT) Received: by mail-pl1-x633.google.com with SMTP id z3so9462505plg.8; Sat, 19 Mar 2022 10:30:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fbHuw2wN1RGMdDQanAXcScwfI02r7zqf8nuc7Y7i6So=; b=EftifGYUFgc8p+r1eKYESSfOtKu9MWFg12bUtv26Rl9uF8geVk4/r0RujVw831C6Lr T67W6T2AueI9nWi7cQR7+n+uc21vhyz/KzxW3obbga4ubV8ZHtNCvwkJGAKLOPgGPSIk 721tpCeodYzGhqsZqTUNTRUprE5xNnkes80HIOv2xps7OOSbI4Lp98xb3bUz+SxWyy92 XNIySUhtFQKYVe9ZqsF9YqZ/ri7ZiNx8cT+U6sIVIkPOO1xM20Jh5k9ZoQMAg7+S7/uQ WqQnWm1PEr6vu2BoH16fWCxx2Vvfw4H+Bd4T5qAkZRbbgyAi95RPz7rgZokXSTrDnvfG V2Dw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fbHuw2wN1RGMdDQanAXcScwfI02r7zqf8nuc7Y7i6So=; b=FfMgD7Y2QRcpoAOOJPFsnrXCpG+kOCPqnTfYg3ywLd4w0doJkE20pVBm/7y8Z1+qv9 khp1JskFdQolG8IDu/Mfv3RTf6BZWNhg4NZ8ED1uvjtGZQ4tHMS5XuZYYksn/734rFBs Wpch0Rf0yy1PtfnjqQfZJcxVU0lMS9QjXCXS1IKuTXERjEVYr4PW54ZTh8elAXuRNiS/ 6I7prBoOs8NgEYSVpbwt1Brp53Bj6rTfJWZE1tO9LYiJmlTAQjKv4TkJ1OCbYDjeLmYY h2XPH3+4cVF7ZMclhwbnx3SeYRbKjQ+66ppbqh6XTmbsiSI1T+AKtKqe4ve6k5axVneG RBzg== X-Gm-Message-State: AOAM5317qW4LAEB1d+Atg9Nj1o/ylXm6T4D+UB6BKXtqmDHWr3EB4+e3 PG+ODC85qUhddwaKj/RKk3V3cG2YornQN7z752Y= X-Google-Smtp-Source: ABdhPJwJva28WJ7jWfa33Znt81ZHpX/1QIXtCHZnxe+533hjN5affi9hym5cjzvBQn/oJ+7rhn5haA== X-Received: by 2002:a17:90a:1f4d:b0:1bb:a657:ace5 with SMTP id y13-20020a17090a1f4d00b001bba657ace5mr27796079pjy.39.1647711048648; Sat, 19 Mar 2022 10:30:48 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:4ab8:5400:3ff:fee9:a154]) by smtp.gmail.com with ESMTPSA id k21-20020aa788d5000000b004f71bff2893sm12722136pff.67.2022.03.19.10.30.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 19 Mar 2022 10:30:48 -0700 (PDT) From: Yafang Shao To: roman.gushchin@linux.dev, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, shuah@kernel.org Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, Yafang Shao Subject: [PATCH 07/14] bpf: Allow no charge in map specific allocation Date: Sat, 19 Mar 2022 17:30:29 +0000 Message-Id: <20220319173036.23352-8-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220319173036.23352-1-laoar.shao@gmail.com> References: <20220319173036.23352-1-laoar.shao@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Some maps have their own ->map_alloc, in which the no charge should also be allowed. Signed-off-by: Yafang Shao --- kernel/bpf/arraymap.c | 2 +- kernel/bpf/bpf_local_storage.c | 5 +++-- kernel/bpf/cpumap.c | 2 +- kernel/bpf/devmap.c | 2 +- kernel/bpf/hashtab.c | 2 +- kernel/bpf/local_storage.c | 2 +- kernel/bpf/lpm_trie.c | 2 +- kernel/bpf/ringbuf.c | 6 +++--- 8 files changed, 12 insertions(+), 11 deletions(-) diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c index e26aef906392..9df425ad769c 100644 --- a/kernel/bpf/arraymap.c +++ b/kernel/bpf/arraymap.c @@ -1062,7 +1062,7 @@ static struct bpf_map *prog_array_map_alloc(union bpf_attr *attr) struct bpf_array_aux *aux; struct bpf_map *map; - aux = kzalloc(sizeof(*aux), GFP_KERNEL_ACCOUNT); + aux = kzalloc(sizeof(*aux), map_flags_no_charge(GFP_KERNEL, attr)); if (!aux) return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/bpf_local_storage.c b/kernel/bpf/bpf_local_storage.c index a92d3032fcde..b626546d384d 100644 --- a/kernel/bpf/bpf_local_storage.c +++ b/kernel/bpf/bpf_local_storage.c @@ -582,11 +582,12 @@ int bpf_local_storage_map_alloc_check(union bpf_attr *attr) struct bpf_local_storage_map *bpf_local_storage_map_alloc(union bpf_attr *attr) { + gfp_t gfp_flags = map_flags_no_charge(GFP_USER | __GFP_NOWARN, attr); struct bpf_local_storage_map *smap; unsigned int i; u32 nbuckets; - smap = kzalloc(sizeof(*smap), GFP_USER | __GFP_NOWARN | __GFP_ACCOUNT); + smap = kzalloc(sizeof(*smap), gfp_flags); if (!smap) return ERR_PTR(-ENOMEM); bpf_map_init_from_attr(&smap->map, attr); @@ -597,7 +598,7 @@ struct bpf_local_storage_map *bpf_local_storage_map_alloc(union bpf_attr *attr) smap->bucket_log = ilog2(nbuckets); smap->buckets = kvcalloc(sizeof(*smap->buckets), nbuckets, - GFP_USER | __GFP_NOWARN | __GFP_ACCOUNT); + map_flags_no_charge(GFP_USER | __GFP_NOWARN, attr)); if (!smap->buckets) { kfree(smap); return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/cpumap.c b/kernel/bpf/cpumap.c index 5a5b40e986ff..fd3b3f05e76a 100644 --- a/kernel/bpf/cpumap.c +++ b/kernel/bpf/cpumap.c @@ -99,7 +99,7 @@ static struct bpf_map *cpu_map_alloc(union bpf_attr *attr) attr->map_flags & ~CPU_MAP_CREATE_FLAG_MASK) return ERR_PTR(-EINVAL); - cmap = kzalloc(sizeof(*cmap), GFP_USER | __GFP_ACCOUNT); + cmap = kzalloc(sizeof(*cmap), map_flags_no_charge(GFP_USER, attr)); if (!cmap) return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index 2857176c82bb..6aaa2e3ce795 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -160,7 +160,7 @@ static struct bpf_map *dev_map_alloc(union bpf_attr *attr) if (!capable(CAP_NET_ADMIN)) return ERR_PTR(-EPERM); - dtab = kzalloc(sizeof(*dtab), GFP_USER | __GFP_ACCOUNT); + dtab = kzalloc(sizeof(*dtab), map_flags_no_charge(GFP_USER, attr)); if (!dtab) return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index 2c84045ff8e1..74696d8196a5 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -474,7 +474,7 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr) struct bpf_htab *htab; int err, i; - htab = kzalloc(sizeof(*htab), GFP_USER | __GFP_ACCOUNT); + htab = kzalloc(sizeof(*htab), map_flags_no_charge(GFP_USER, attr)); if (!htab) return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/local_storage.c b/kernel/bpf/local_storage.c index 865766c240d6..741ab7cf3626 100644 --- a/kernel/bpf/local_storage.c +++ b/kernel/bpf/local_storage.c @@ -313,7 +313,7 @@ static struct bpf_map *cgroup_storage_map_alloc(union bpf_attr *attr) return ERR_PTR(-EINVAL); map = kmalloc_node(sizeof(struct bpf_cgroup_storage_map), - __GFP_ZERO | GFP_USER | __GFP_ACCOUNT, numa_node); + map_flags_no_charge(__GFP_ZERO | GFP_USER, attr), numa_node); if (!map) return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/lpm_trie.c b/kernel/bpf/lpm_trie.c index f42edf613624..9673f8e98c58 100644 --- a/kernel/bpf/lpm_trie.c +++ b/kernel/bpf/lpm_trie.c @@ -557,7 +557,7 @@ static struct bpf_map *trie_alloc(union bpf_attr *attr) attr->value_size > LPM_VAL_SIZE_MAX) return ERR_PTR(-EINVAL); - trie = kzalloc(sizeof(*trie), GFP_USER | __GFP_NOWARN | __GFP_ACCOUNT); + trie = kzalloc(sizeof(*trie), map_flags_no_charge(GFP_USER | __GFP_NOWARN, attr)); if (!trie) return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/ringbuf.c b/kernel/bpf/ringbuf.c index a3b4d2a0a2c7..3db07cd0ab60 100644 --- a/kernel/bpf/ringbuf.c +++ b/kernel/bpf/ringbuf.c @@ -60,8 +60,8 @@ struct bpf_ringbuf_hdr { static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, union bpf_attr *attr) { - const gfp_t flags = GFP_KERNEL_ACCOUNT | __GFP_RETRY_MAYFAIL | - __GFP_NOWARN | __GFP_ZERO; + const gfp_t flags = map_flags_no_charge(__GFP_RETRY_MAYFAIL | + __GFP_NOWARN | __GFP_ZERO, attr); int nr_meta_pages = RINGBUF_PGOFF + RINGBUF_POS_PAGES; int nr_data_pages = data_sz >> PAGE_SHIFT; int nr_pages = nr_meta_pages + nr_data_pages; @@ -164,7 +164,7 @@ static struct bpf_map *ringbuf_map_alloc(union bpf_attr *attr) return ERR_PTR(-E2BIG); #endif - rb_map = kzalloc(sizeof(*rb_map), GFP_USER | __GFP_ACCOUNT); + rb_map = kzalloc(sizeof(*rb_map), map_flags_no_charge(GFP_USER, attr)); if (!rb_map) return ERR_PTR(-ENOMEM); From patchwork Sat Mar 19 17:30:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 553109 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80927C43219 for ; Sat, 19 Mar 2022 17:31:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243703AbiCSRcX (ORCPT ); Sat, 19 Mar 2022 13:32:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35182 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243683AbiCSRcV (ORCPT ); Sat, 19 Mar 2022 13:32:21 -0400 Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CC52845ADC; Sat, 19 Mar 2022 10:30:53 -0700 (PDT) Received: by mail-pj1-x102e.google.com with SMTP id b8so9842126pjb.4; Sat, 19 Mar 2022 10:30:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=qHQu4l0DNA359MKDX3wgS7bak31X8wuVL1Qxy9vcN30=; b=aVlQw/X30t3guhPlzexjfi4ZuV1qG2ftDBMaM2cxYxqwAi3vbYD6Qvz749eSnnmN3I dcjSrNHmbXjAqWtWH9tlty2IMJT5ilANT557Z9fyWcqL3VhW8p+AWj02tlbZx2cPrTVs b1ipu9ey3X/BQRhLYasrQJrjczm7Qpmz3cCvBHfN8DtM/l5g0NO1OCC7+VT+o5NJ9HcD FW2ZjEo7o8iZXK9cbz6wNQ+H4kITaPS7gbBuVDOIm5JLoDEbKsZMAthE037au1hACQBN kATbhDZARdTdhdL0FTjfcGN0oNMFTSTSZqrbMuUJc6K4XFgMwQEKQ8PAzDUSE+aqN1U1 XiRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qHQu4l0DNA359MKDX3wgS7bak31X8wuVL1Qxy9vcN30=; b=1kKUSuEIBEri2WuLOsrtKm710NPF3xUsbNafiTV+CVzZPsLsQ8x+2uH3cttHl32XPk 4CrsIYelW1Qv1VI1Tt1By59hAnxoTcDGaDlZEkFkbAffkZWPWOEc59ZSkM3c4rrG8V04 yj9oSDSSvY8Y9o7JUqI+e6iCS3esG1gIGnijtRySEDJ3s4qLYs7u3Y67cc3/83nTwQAY 7UmT+//Q+W3DGhCFpWyXOPnUrCMOWzjg+Ucs4k2TdEMUvbegKQH/oUaYU3SCcAhJ/NRS 38QXWlRxNME9kJFm637ONboXRjANDcdjYnGbz1ohEKBWtaGIbN0VU+M9iWDH5xkoIF7j 6X2Q== X-Gm-Message-State: AOAM532CQb/VFS5CefiXp/3uO8Xozw16/QOditIROZz79oGiXuJVQtYA tEoIBWdEfglqWmtehVnMfw95sZkR4r1somi7sgU= X-Google-Smtp-Source: ABdhPJxbI/bKQUsvIS+pdtOIBg2Ba5EngaSH06pOcvVbNR2cQU79iWIhoSG40JbwcKqwIDkohnV3Ig== X-Received: by 2002:a17:903:230c:b0:151:93d0:5608 with SMTP id d12-20020a170903230c00b0015193d05608mr5013516plh.167.1647711053379; Sat, 19 Mar 2022 10:30:53 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:4ab8:5400:3ff:fee9:a154]) by smtp.gmail.com with ESMTPSA id k21-20020aa788d5000000b004f71bff2893sm12722136pff.67.2022.03.19.10.30.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 19 Mar 2022 10:30:52 -0700 (PDT) From: Yafang Shao To: roman.gushchin@linux.dev, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, shuah@kernel.org Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, Yafang Shao Subject: [PATCH 10/14] bpf: Only sys admin can set no charge flag for bpf prog Date: Sat, 19 Mar 2022 17:30:32 +0000 Message-Id: <20220319173036.23352-11-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220319173036.23352-1-laoar.shao@gmail.com> References: <20220319173036.23352-1-laoar.shao@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org When a bpf prog is loaded by a proccess running in a container (with memcg), only sys admin has privilege not to charge bpf prog memory into this container while account it to root memcg only. Signed-off-by: Yafang Shao --- kernel/bpf/syscall.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 346f3df9fa1d..ecc5de216f50 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -2234,6 +2234,9 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr) BPF_F_PROG_NO_CHARGE)) return -EINVAL; + if (attr->prog_flags & BPF_F_PROG_NO_CHARGE && !capable(CAP_SYS_ADMIN)) + return -EPERM; + if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) && (attr->prog_flags & BPF_F_ANY_ALIGNMENT) && !bpf_capable()) From patchwork Sat Mar 19 17:30:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 553107 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9AF7C4332F for ; Sat, 19 Mar 2022 17:31:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243739AbiCSRcm (ORCPT ); Sat, 19 Mar 2022 13:32:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35190 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243686AbiCSRcV (ORCPT ); Sat, 19 Mar 2022 13:32:21 -0400 Received: from mail-pl1-x633.google.com (mail-pl1-x633.google.com [IPv6:2607:f8b0:4864:20::633]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6AEFB46141; Sat, 19 Mar 2022 10:30:56 -0700 (PDT) Received: by mail-pl1-x633.google.com with SMTP id t22so9504713plo.0; Sat, 19 Mar 2022 10:30:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=UnzpRSS/vLYZGmFXrB6nYaZPHuwQlW9x/vmkhMBVTEQ=; b=RsczAP8C3Ig4HKDvm6X86NOreFdEhMYHBpRBoeKF/94wlV9+u9Egr4V/TBqS1FX8Ue lFFPQpIMJS4G5lpfiEw4KvUr8o5fqRs/XJsZMcZPpxldyQLPMhxPur699k75MNBgyaQ5 4dS5qDSS0tsiNXWZjsakb9ETsLt+JEePQaPATpMIr3Z/+Z86EFP3EGI9zrujz75CkY4V ahN24nCYzn1MoHhVLuq5xcKyvqaej5JUbg49KNmekfTtZuNF73XEQW4iYhSAIFbHtXCu LVoMb7GX+KBO8hWssyjZ8SSU8gCSwnDRoVlaBZRSYuVFbQtS176El/2RnVtm0bGcXaqC U0VQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UnzpRSS/vLYZGmFXrB6nYaZPHuwQlW9x/vmkhMBVTEQ=; b=xLhZ7OQ4E6dLm+gjHuIVZ2MS49DrlOAVmjIgXa1A96EXuTQMTvebZH4s0481xqTIZu Or9v6PerpPiTa4ZBYDutUrTG+zoflmqi9+KpILbleBtqfy52Q8cZrvAZHsfZvUn7x9ef MvIjltl/m2RaNV/Kcy33DtLgxe2UPR0D5FIW0Nol8OB5LmroNVITfNNaVu4jZQw/UnOe DJe4ng0XU3pSns8S3Bmy8EV3eFYKSaAm6YY0+xNvYuUSH7kqDazdTLP21gJ4EyOFYHTZ KSjUm+AJw2kCDqVldmT2MAL+djaViqmhuCeTVL/6meo94dHmTaWpWhqTGosOsdzb8HX/ mhKA== X-Gm-Message-State: AOAM532H6H9vW6Ab0EturzBedYh3O25Z0cr2tQpSE8ejF5BqcOciPdFI 9+DBsIQiUBEc+xuMkoYpFVQ= X-Google-Smtp-Source: ABdhPJzDsThUZW6p6KlCe6SKrOblLrhRM+o0ogBcfGoBqkKzlliLEkvGKbHYLlwwOKXRc+RIyMwfZw== X-Received: by 2002:a17:903:246:b0:153:87f0:a93e with SMTP id j6-20020a170903024600b0015387f0a93emr5130014plh.171.1647711055893; Sat, 19 Mar 2022 10:30:55 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:4ab8:5400:3ff:fee9:a154]) by smtp.gmail.com with ESMTPSA id k21-20020aa788d5000000b004f71bff2893sm12722136pff.67.2022.03.19.10.30.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 19 Mar 2022 10:30:55 -0700 (PDT) From: Yafang Shao To: roman.gushchin@linux.dev, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, shuah@kernel.org Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, Yafang Shao Subject: [PATCH 12/14] bpf: Allow no charge for bpf prog Date: Sat, 19 Mar 2022 17:30:34 +0000 Message-Id: <20220319173036.23352-13-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220319173036.23352-1-laoar.shao@gmail.com> References: <20220319173036.23352-1-laoar.shao@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Allow not to charge memory used by bpf progs. This includes the memory used by bpf progs itself, auxxiliary data, statistics and bpf line info. Signed-off-by: Yafang Shao --- kernel/bpf/core.c | 12 ++++++++++-- kernel/bpf/syscall.c | 6 ++++-- 2 files changed, 14 insertions(+), 4 deletions(-) diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 0f68b8203c18..7aa750e8bd7d 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -144,12 +144,17 @@ EXPORT_SYMBOL_GPL(bpf_prog_alloc); int bpf_prog_alloc_jited_linfo(struct bpf_prog *prog) { + gfp_t gfp_flags = GFP_KERNEL | __GFP_NOWARN; + if (!prog->aux->nr_linfo || !prog->jit_requested) return 0; + if (!prog->aux->no_charge) + gfp_flags |= __GFP_ACCOUNT; + prog->aux->jited_linfo = kvcalloc(prog->aux->nr_linfo, sizeof(*prog->aux->jited_linfo), - GFP_KERNEL_ACCOUNT | __GFP_NOWARN); + gfp_flags); if (!prog->aux->jited_linfo) return -ENOMEM; @@ -224,7 +229,7 @@ void bpf_prog_fill_jited_linfo(struct bpf_prog *prog, struct bpf_prog *bpf_prog_realloc(struct bpf_prog *fp_old, unsigned int size, gfp_t gfp_extra_flags) { - gfp_t gfp_flags = GFP_KERNEL_ACCOUNT | __GFP_ZERO | gfp_extra_flags; + gfp_t gfp_flags = GFP_KERNEL | __GFP_ZERO | gfp_extra_flags; struct bpf_prog *fp; u32 pages; @@ -233,6 +238,9 @@ struct bpf_prog *bpf_prog_realloc(struct bpf_prog *fp_old, unsigned int size, if (pages <= fp_old->pages) return fp_old; + if (!fp_old->aux->no_charge) + gfp_flags |= __GFP_ACCOUNT; + fp = __vmalloc(size, gfp_flags); if (fp) { memcpy(fp, fp_old, fp_old->pages * PAGE_SIZE); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index fdfbb4d0d5e0..e386b549fafc 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -2218,9 +2218,10 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr) enum bpf_prog_type type = attr->prog_type; struct bpf_prog *prog, *dst_prog = NULL; struct btf *attach_btf = NULL; - int err; + gfp_t gfp_flags = GFP_USER; char license[128]; bool is_gpl; + int err; if (CHECK_ATTR(BPF_PROG_LOAD)) return -EINVAL; @@ -2305,7 +2306,8 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr) } /* plain bpf_prog allocation */ - prog = bpf_prog_alloc(bpf_prog_size(attr->insn_cnt), GFP_USER | __GFP_ACCOUNT); + prog = bpf_prog_alloc(bpf_prog_size(attr->insn_cnt), + prog_flags_no_charge(gfp_flags, attr)); if (!prog) { if (dst_prog) bpf_prog_put(dst_prog); From patchwork Sat Mar 19 17:30:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 553106 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EDE22C4167B for ; Sat, 19 Mar 2022 17:31:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243681AbiCSRco (ORCPT ); Sat, 19 Mar 2022 13:32:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35152 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243702AbiCSRcW (ORCPT ); Sat, 19 Mar 2022 13:32:22 -0400 Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D9AC646B04; Sat, 19 Mar 2022 10:30:58 -0700 (PDT) Received: by mail-pj1-x102c.google.com with SMTP id mz9-20020a17090b378900b001c657559290so10657015pjb.2; Sat, 19 Mar 2022 10:30:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gNHwzX1XXzyuZEkRQLK5ljL7FKM2KTEIhvt/+nJ9w5k=; b=cWsSBz1wcJLbkEhvadWb1z178lDBINSMILOr0KYVKwhDbun9djaH9TCmc3V/X47pym DuDLtLZW9Mw1cTrnsjsKmlH7ZvuFPFkJ4rsarGoModU082cK1VA+dDJ+GNIc5CwoKx34 wvQQTZSYyrSVI7T6FfY1aefOUuwaZmC5wiCWxKZqWxV9RepJVsKZh34D2LgSsyFjKZuK BUthR6o8mnnz61PINJHyIrwp2uDJw5Xecw1vVn5dz30SLOTpiWnp0p0ojMl2ldLivLyo awA5jMvuzOhSmBdGifSsHR9p9ndyhKd83qk3MlNvtLjcTPqaODL9fz9n1vXPSd4moFCf +X/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gNHwzX1XXzyuZEkRQLK5ljL7FKM2KTEIhvt/+nJ9w5k=; b=XfqoWp/LEYVCUwVdRZ2WhIQ+Oe72Bao3Y5nw/uWn9wt5DQg37rXBw4HCYoiwAXQbeJ BdjOFxOaVUMdu+HvTOVe+YYbrKHX1umguWqnKcbzgJIiSeR7eGWkfE02SH3qdv97fbso yhcWy9uBR5/A3srgOvB2HvaKAFhW5Qi8KhvaQHZwhlUPCaRCzvKrxIxLdFQTVh5ulBsu x3FGU9WUOpuUy/HtDg/0wbAPLKuLSEgyyJawO8HM2oBWCpaaEJ8DSqibsxgFfdinHQAO /kdno53tPsg0EomsNp+0HDearF+h/eFraJH3BCfLHtS62FN2bH65P9V55df+o7qXTx17 cE3Q== X-Gm-Message-State: AOAM532GiV0mk9FaiBL1V2+9fnUIB8tBHshL6wI/ohMViHWoL5P5Q6VF RUHtTYgPhR5Y0KFSIj1jfS8= X-Google-Smtp-Source: ABdhPJx0WWiyXY4tSlfUjYz1EUtkAZiBSBQOTjQ9TIbe9waIxqXbbGnwv+SSon9nwqQjPI9mhc08Og== X-Received: by 2002:a17:90a:7305:b0:1bd:6972:faf5 with SMTP id m5-20020a17090a730500b001bd6972faf5mr17517556pjk.131.1647711058332; Sat, 19 Mar 2022 10:30:58 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:4ab8:5400:3ff:fee9:a154]) by smtp.gmail.com with ESMTPSA id k21-20020aa788d5000000b004f71bff2893sm12722136pff.67.2022.03.19.10.30.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 19 Mar 2022 10:30:57 -0700 (PDT) From: Yafang Shao To: roman.gushchin@linux.dev, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, shuah@kernel.org Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, Yafang Shao Subject: [PATCH 14/14] bpf: selftests: Add test case for BPF_F_PROG_NO_CHARGE Date: Sat, 19 Mar 2022 17:30:36 +0000 Message-Id: <20220319173036.23352-15-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220319173036.23352-1-laoar.shao@gmail.com> References: <20220319173036.23352-1-laoar.shao@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Test case to check if BPF_F_PROG_NO_CHARGE valid. The result as follows, $ ./test_progs ... #103 no_charge:OK ... Signed-off-by: Yafang Shao --- .../selftests/bpf/prog_tests/no_charge.c | 49 +++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/no_charge.c diff --git a/tools/testing/selftests/bpf/prog_tests/no_charge.c b/tools/testing/selftests/bpf/prog_tests/no_charge.c new file mode 100644 index 000000000000..85fbd2ccbc71 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/no_charge.c @@ -0,0 +1,49 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include + +#include "test_progs.h" + +#define BPF_ALU64_IMM(OP, DST, IMM) \ + ((struct bpf_insn) { \ + .code = BPF_ALU64 | BPF_OP(OP) | BPF_K, \ + .dst_reg = DST, \ + .src_reg = 0, \ + .off = 0, \ + .imm = IMM }) + +#define BPF_EXIT_INSN() \ + ((struct bpf_insn) { \ + .code = BPF_JMP | BPF_EXIT, \ + .dst_reg = 0, \ + .src_reg = 0, \ + .off = 0, \ + .imm = 0 }) + +void test_no_charge(void) +{ + struct bpf_insn prog[] = { + BPF_ALU64_IMM(BPF_MOV, BPF_REG_0, 0), + BPF_EXIT_INSN(), + }; + union bpf_attr attr; + int duration = 0; + int fd; + + bzero(&attr, sizeof(attr)); + attr.prog_type = BPF_PROG_TYPE_SCHED_CLS; + attr.insn_cnt = 2; + attr.insns = (__u64)prog; + attr.license = (__u64)("GPL"); + attr.prog_flags |= BPF_F_PROG_NO_CHARGE; + + fd = syscall(__NR_bpf, BPF_PROG_LOAD, &attr, sizeof(attr)); + CHECK(fd < 0 && fd != -EPERM, "no_charge", "error: %s\n", + strerror(errno)); + + if (fd > 0) + close(fd); +}