From patchwork Tue Feb 12 05:47:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Khem Raj X-Patchwork-Id: 158096 Delivered-To: patch@linaro.org Received: by 2002:a02:48:0:0:0:0:0 with SMTP id 69csp3445408jaa; Mon, 11 Feb 2019 21:48:25 -0800 (PST) X-Google-Smtp-Source: AHgI3IYUe7GMNE5w4Nz/4+uOwsooluKZfbBps9SvGigmg3cHarYqhfaVvDajTF4A0z7R7r28z8kW X-Received: by 2002:a63:6506:: with SMTP id z6mr2085386pgb.334.1549950505849; Mon, 11 Feb 2019 21:48:25 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1549950505; cv=none; d=google.com; s=arc-20160816; b=CKKCbzELIA8uEYf5G3k8+ExAaS2XSAlj6QVfoSte1GpkchyPpwYYjl15ggqm+fW+Zu E4DT6PblgFCSHmtExkd2aAno2cBbBAtbiZ0jCxypCT2rZ1nwGBXXD6x8XsvGrrIU9zv6 ILkB3YnmMQwH+t2KoovDcAfebegWL4wqWNe9OOslZCmyuASXvOcK8pI5XRYqb2hknMe+ oo63DKLH0iopS+j+GWsMB4MFwxnmGWIcZXsThPhvjOE9p4W4QxJ5wDm+GbEmUqIl5vJl 4AM1EvNlSWx5qxsDwcOrlyHSHgM+TPIlLhAXFv71IukRj+8pfedi//SksUiyMXljqp/5 H5fw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :dkim-signature:delivered-to; bh=GFJeSzxkecJlIr+JE7AW5ZrQe6V/DFHWsuNHrHv475g=; b=rUqTxXyVSMn+VmosBuLKzL8T78OP8Ew8cvmbi5ZS/RDb8nJyHQfTq0M8i3R0XjI2JC QdHBWO2KISxYoOqNArP5HESC2xwTAKdE2uzALmAq94rIu0gxe2Nlrz6VlMJLsY4lPor2 YKsc3cOlT5Dgv/vAmwx/XZ1VIA2xNyGYu8JbmEtmWhG6N//pTN0KVMhOvrpuB3ONuerv J9qNq4FhZz8AvwnC7ecix28y5/8P7bpIxsmpc2I3r7qO+iEfVAa4VDagMU97FKpFub7T erdFMNKIiPXDtXR6+UaQd2hXLmdzsFUtVRxRCbjZE1MJtCEwS3VjTDBvOMUZxhQ7IwMP x2oA== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=QYIsAUyi; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id 16si3001337pgt.548.2019.02.11.21.48.25; Mon, 11 Feb 2019 21:48:25 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=QYIsAUyi; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 727397C4A3; Tue, 12 Feb 2019 05:48:19 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-pf1-f195.google.com (mail-pf1-f195.google.com [209.85.210.195]) by mail.openembedded.org (Postfix) with ESMTP id BD11F7C445 for ; Tue, 12 Feb 2019 05:48:01 +0000 (UTC) Received: by mail-pf1-f195.google.com with SMTP id h1so751084pfo.7 for ; Mon, 11 Feb 2019 21:48:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=s8Of4ant40YGob0kjeYQ1/kt/nJgfkUNFQrF74QlVU4=; b=QYIsAUyiAGhLkcGUu0ojHo8GYnj5cTqs5aKUM1AcXQAeskfUpDA1HCPBhbZSPuaE/R X/ZLo011nrE42u3mGqtO8f3uryeZqOAr6bXEx97POT+cIiyVkFh2uTChPRZDnJgB1rNh JZvGu/EU8XuZepeZXeSLuDYzZKaJ6VZv9NOWY6JbcfkKVxOZyNVIts1f+zS73WA/62u8 gYvm6eHqulXoPnq7B2g+RR0Mou+cpOOusiuQwfpA3gNLM2xulwPANatxDpD9LSLFje2f /XckMWuzzZVlKsiTNQfh5u8IC532tmniTk4N3iCxO7LyzGn8mea1JsOIAlyzRIwuQwdE 7RoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=s8Of4ant40YGob0kjeYQ1/kt/nJgfkUNFQrF74QlVU4=; b=Rj9cmy8jGtIa02xy/RcPfebsQhmqACGpkNjyDBja6+vOebJJVti8achgtgLjhynF/3 5cmOU6p2oRpIpX5CAr2V62qWhxUH8TxS7rrds8XgP0hVKFTj+b69zbg4JdiruKdC5xZA Eh53u0KOSmpEmFq2caRwz7HrtuOY45A5pEhk0m2ZtEelqUU5uW4zi+pyky5Rv2o3m0YP Es4O1Jn7qSoIV+eAh/qm8C63waW9OcDZ4tkMFanbSJArusLCS1a4nIHLFgVXES3rTlFH jnJaywCk1P3Q2kgnLg3J3vMb8r0yPmZAJbIHP4N5wdmMq9HSD+aFRvHzY8jQSqsQ1JZj l74g== X-Gm-Message-State: AHQUAuazRhyc9wSndA3vNenYSPNpG1jTcbYKTQZvXIIEnGTO4ZMcUqrG yLd469BlX6vnlX/sCJUZFWxZq1XY X-Received: by 2002:a62:a1a:: with SMTP id s26mr2253366pfi.31.1549950482422; Mon, 11 Feb 2019 21:48:02 -0800 (PST) Received: from apollo.hsd1.ca.comcast.net ([2601:646:8500:6bc6::dc44]) by smtp.gmail.com with ESMTPSA id b15sm2020017pfj.44.2019.02.11.21.48.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 11 Feb 2019 21:48:01 -0800 (PST) From: Khem Raj To: openembedded-core@lists.openembedded.org Date: Mon, 11 Feb 2019 21:47:54 -0800 Message-Id: <20190212054754.2801-3-raj.khem@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190212054754.2801-1-raj.khem@gmail.com> References: <20190212054754.2801-1-raj.khem@gmail.com> MIME-Version: 1.0 Subject: [OE-core] [PATCH v3 3/3] image_types.bbclass: Set memory usage limit and CPU threads for xz X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org when building with opkg backend and huge packages e.g. chromium/llvm all going in parallel, memory pressure causes xz to catapult with do_package_write_ipk: Failed to create package, opkg-build failed with: xz: (stdin): Cannot allocate memory since there are many tasks going on in parallel, xz adds to memory pressure and it wants it all, put an upper limit for memory xz can use We add a variable XZ_MAXRAM with 30% of RAM limit and can be customized if builders have more memory one can set it like XZ_DEFAULTS = "-M 0 -T 0" Signed-off-by: Khem Raj --- V2: Consider all locations of xz use v3: Use ${@oe.utils.cpu_count()} in place of BB_NUMBER_THREADS meta/classes/image_types.bbclass | 9 ++++----- meta/classes/package_ipk.bbclass | 2 +- meta/classes/populate_sdk_base.bbclass | 2 +- meta/conf/bitbake.conf | 3 +++ 4 files changed, 9 insertions(+), 7 deletions(-) -- 2.20.1 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass index ddca5b624e..f9890581b3 100644 --- a/meta/classes/image_types.bbclass +++ b/meta/classes/image_types.bbclass @@ -56,7 +56,6 @@ def imagetypes_getdepends(d): XZ_COMPRESSION_LEVEL ?= "-3" XZ_INTEGRITY_CHECK ?= "crc32" -XZ_THREADS ?= "-T 0" ZIP_COMPRESSION_LEVEL ?= "-9" @@ -152,12 +151,12 @@ UBI_VOLNAME ?= "${MACHINE}-rootfs" multiubi_mkfs() { local mkubifs_args="$1" local ubinize_args="$2" - + # Added prompt error message for ubi and ubifs image creation. if [ -z "$mkubifs_args" ] || [ -z "$ubinize_args" ]; then bbfatal "MKUBIFS_ARGS and UBINIZE_ARGS have to be set, see http://www.linux-mtd.infradead.org/faq/ubifs.html for details" fi - + if [ -z "$3" ]; then local vname="" else @@ -284,7 +283,7 @@ CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum sha224sum sha25 CONVERSION_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}" CONVERSION_CMD_gz = "pigz -f -9 -n -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz" CONVERSION_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}" -CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS} --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz" +CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_DEFAULTS} --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz" CONVERSION_CMD_lz4 = "lz4 -9 -z -l ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4" CONVERSION_CMD_lzo = "lzop -9 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}" CONVERSION_CMD_zip = "zip ${ZIP_COMPRESSION_LEVEL} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zip ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}" @@ -319,7 +318,7 @@ CONVERSION_DEPENDS_base64 = "coreutils-native" RUNNABLE_IMAGE_TYPES ?= "ext2 ext3 ext4" RUNNABLE_MACHINE_PATTERNS ?= "qemu" -DEPLOYABLE_IMAGE_TYPES ?= "hddimg iso" +DEPLOYABLE_IMAGE_TYPES ?= "hddimg iso" # The IMAGE_TYPES_MASKED variable will be used to mask out from the IMAGE_FSTYPES, # images that will not be built at do_rootfs time: vmdk, vdi, qcow2, hdddirect, hddimg, iso, etc. diff --git a/meta/classes/package_ipk.bbclass b/meta/classes/package_ipk.bbclass index 508b7dcaff..d1b317b42b 100644 --- a/meta/classes/package_ipk.bbclass +++ b/meta/classes/package_ipk.bbclass @@ -8,7 +8,7 @@ IPKGCONF_SDK = "${WORKDIR}/opkg-sdk.conf" PKGWRITEDIRIPK = "${WORKDIR}/deploy-ipks" # Program to be used to build opkg packages -OPKGBUILDCMD ??= 'opkg-build -Z xz -a "--threads 0"' +OPKGBUILDCMD ??= 'opkg-build -Z xz -a "${XZ_DEFAULTS}"' OPKG_ARGS += "--force_postinstall --prefer-arch-to-version" OPKG_ARGS += "${@['', '--no-install-recommends'][d.getVar("NO_RECOMMENDATIONS") == "1"]}" diff --git a/meta/classes/populate_sdk_base.bbclass b/meta/classes/populate_sdk_base.bbclass index 80fa443e4c..ebc30d39b3 100644 --- a/meta/classes/populate_sdk_base.bbclass +++ b/meta/classes/populate_sdk_base.bbclass @@ -226,7 +226,7 @@ fakeroot tar_sdk() { # Package it up mkdir -p ${SDKDEPLOYDIR} cd ${SDK_OUTPUT}/${SDKPATH} - tar ${SDKTAROPTS} -cf - . | xz -T 0 > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz + tar ${SDKTAROPTS} -cf - . | xz ${XZ_DEFAULTS} > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.tar.xz } TOOLCHAIN_SHAR_EXT_TMPL ?= "${COREBASE}/meta/files/toolchain-shar-extract.sh" diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 21fd93e58d..6e063dc14d 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -791,6 +791,9 @@ BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}" # Default to setting automatically based on cpu count PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}" +# Default parallelism and resource usage for xz +XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count()}" + ################################################################## # Magic Cookie for SANITY CHECK ##################################################################