From patchwork Thu Jun 18 14:23:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Heiko_St=C3=BCbner?= X-Patchwork-Id: 242603 List-Id: U-Boot discussion From: heiko at sntech.de (Heiko Stuebner) Date: Thu, 18 Jun 2020 16:23:24 +0200 Subject: [PATCH v6 4/8] lib: rsa: fix allocated size for rr and rrtmp in rsa_gen_key_prop() In-Reply-To: <20200618142328.1753036-1-heiko@sntech.de> References: <20200618142328.1753036-1-heiko@sntech.de> Message-ID: <20200618142328.1753036-4-heiko@sntech.de> From: Heiko Stuebner When calculating rrtmp/rr rsa_gen_key_prop() tries to make (((rlen + 31) >> 5) + 1) steps in the rr uint32_t array and (((rlen + 7) >> 3) + 1) / 4 steps in uint32_t rrtmp[] with rlen being num_bits * 2 On a 4096bit key this comes down to to 257 uint32_t elements in rr and 256 elements in rrtmp but with the current allocation rr and rrtmp only have 129 uint32_t elements. On 2048bit keys this works by chance as the defined max_rsa_size=4096 allocates a suitable number of elements, but with an actual 4096bit key this results in other memory parts getting overwritten. So as suggested by Heinrich Schuchardt just use the actual bis-size of the key as base for the size calculation, in turn making the code compatible to any future keysizes. Suggested-by: Heinrich Schuchardt Signed-off-by: Heiko Stuebner Reviewed-by: Simon Glass --- changes in v6: - drop max_rsa_size and use the keysize as base changes in v4: - new patch lib/rsa/rsa-keyprop.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/lib/rsa/rsa-keyprop.c b/lib/rsa/rsa-keyprop.c index 4b54db44c4..83b942615f 100644 --- a/lib/rsa/rsa-keyprop.c +++ b/lib/rsa/rsa-keyprop.c @@ -654,14 +654,10 @@ int rsa_gen_key_prop(const void *key, uint32_t keylen, struct key_prop **prop) { struct rsa_key rsa_key; uint32_t *n = NULL, *rr = NULL, *rrtmp = NULL; - const int max_rsa_size = 4096; int rlen, i, ret; *prop = calloc(sizeof(**prop), 1); - n = calloc(sizeof(uint32_t), 1 + (max_rsa_size >> 5)); - rr = calloc(sizeof(uint32_t), 1 + (max_rsa_size >> 5)); - rrtmp = calloc(sizeof(uint32_t), 1 + (max_rsa_size >> 5)); - if (!(*prop) || !n || !rr || !rrtmp) { + if (!(*prop)) { ret = -ENOMEM; goto err; } @@ -682,6 +678,14 @@ int rsa_gen_key_prop(const void *key, uint32_t keylen, struct key_prop **prop) } memcpy((void *)(*prop)->modulus, &rsa_key.n[i], rsa_key.n_sz - i); + n = calloc(sizeof(uint32_t), 1 + ((*prop)->num_bits >> 5)); + rr = calloc(sizeof(uint32_t), 1 + (((*prop)->num_bits * 2) >> 5)); + rrtmp = calloc(sizeof(uint32_t), 1 + (((*prop)->num_bits * 2) >> 5)); + if (!n || !rr || !rrtmp) { + ret = -ENOMEM; + goto out; + } + /* exponent */ (*prop)->public_exponent = calloc(1, sizeof(uint64_t)); if (!(*prop)->public_exponent) {