| Message ID | 20241025191454.72616-5-ebiggers@kernel.org |
|---|---|
| State | Superseded |
| Headers | show |
| Series | Wire up CRC32 library functions to arch-optimized code | expand |
On Fri, Oct 25, 2024 at 10:47:15PM +0200, Ard Biesheuvel wrote: > On Fri, 25 Oct 2024 at 21:15, Eric Biggers <ebiggers@kernel.org> wrote: > > > > From: Eric Biggers <ebiggers@google.com> > > > > Instead of registering the crc32-$arch and crc32c-$arch algorithms if > > the arch-specific code was built, only register them when that code was > > built *and* is not falling back to the base implementation at runtime. > > > > This avoids confusing users like btrfs which checks the shash driver > > name to determine whether it is crc32c-generic. > > > > I think we agree that 'generic' specifically means a C implementation > that is identical across all architectures, which is why I updated my > patch to export -arch instead of wrapping the C code in yet another > driver just for the fuzzing tests. > > So why is this a problem? If no optimizations are available at > runtime, crc32-arch and crc32-generic are interchangeable, and so it > shouldn't matter whether you use one or the other. > > You can infer from the driver name whether the C code is being used, > not whether or not the implementation is 'fast', and the btrfs hack is > already broken on arm64. > > > (It would also make sense to change btrfs to test the crc32_optimization > > flags itself, so that it doesn't have to use the weird hack of parsing > > the driver name. This change still makes sense either way though.) > > > > Indeed. That hack is very dubious and I'd be inclined just to ignore > this. On x86 and arm64, it shouldn't make a difference, given that > crc32-arch will be 'fast' in the vast majority of cases. On other > architectures, btrfs may use the C implementation while assuming it is > something faster, and if anyone actually notices the difference, we > can work with the btrfs devs to do something more sensible here. Yes, we probably could get away without this. It's never really been appropriate to use the crypto driver names for anything important. And btrfs probably should just assume CRC32C == fast unconditionally, like what it does with xxHash64, or even do a quick benchmark to measure the actual speed of its hash algorithm (which can also be sha256 or blake2b which can be very fast too). Besides the btrfs case, my concern was there may be advice floating around about checking /proc/crypto to check what optimized code is being used. Having crc32-$arch potentially be running the generic code would make that misleading. It might make sense to keep it working similar to how it did before. But I do agree that we could probably get away without this. - Eric
On Fri, Oct 25, 2024 at 10:02:39PM +0000, Eric Biggers wrote: > On Fri, Oct 25, 2024 at 10:47:15PM +0200, Ard Biesheuvel wrote: > > On Fri, 25 Oct 2024 at 21:15, Eric Biggers <ebiggers@kernel.org> wrote: > > > > > > From: Eric Biggers <ebiggers@google.com> > > > > > > Instead of registering the crc32-$arch and crc32c-$arch algorithms if > > > the arch-specific code was built, only register them when that code was > > > built *and* is not falling back to the base implementation at runtime. > > > > > > This avoids confusing users like btrfs which checks the shash driver > > > name to determine whether it is crc32c-generic. > > > > > > > I think we agree that 'generic' specifically means a C implementation > > that is identical across all architectures, which is why I updated my > > patch to export -arch instead of wrapping the C code in yet another > > driver just for the fuzzing tests. > > > > So why is this a problem? If no optimizations are available at > > runtime, crc32-arch and crc32-generic are interchangeable, and so it > > shouldn't matter whether you use one or the other. > > > > You can infer from the driver name whether the C code is being used, > > not whether or not the implementation is 'fast', and the btrfs hack is > > already broken on arm64. > > > > > (It would also make sense to change btrfs to test the crc32_optimization > > > flags itself, so that it doesn't have to use the weird hack of parsing > > > the driver name. This change still makes sense either way though.) > > > > > > > Indeed. That hack is very dubious and I'd be inclined just to ignore > > this. On x86 and arm64, it shouldn't make a difference, given that > > crc32-arch will be 'fast' in the vast majority of cases. On other > > architectures, btrfs may use the C implementation while assuming it is > > something faster, and if anyone actually notices the difference, we > > can work with the btrfs devs to do something more sensible here. > > Yes, we probably could get away without this. It's never really been > appropriate to use the crypto driver names for anything important. And btrfs > probably should just assume CRC32C == fast unconditionally, like what it does > with xxHash64, or even do a quick benchmark to measure the actual speed of its > hash algorithm (which can also be sha256 or blake2b which can be very fast too). > > Besides the btrfs case, my concern was there may be advice floating around about > checking /proc/crypto to check what optimized code is being used. Having > crc32-$arch potentially be running the generic code would make that misleading. > It might make sense to keep it working similar to how it did before. > > But I do agree that we could probably get away without this. While testing this patchset I notice that none of the crypto API drivers for crc32 or crc32c even need to be loaded on my system anymore, as everything on my system that uses those algorithms (such as ext4) just uses the library APIs now. That makes the "check /proc/crypto" trick stop working anyway. I think you're right that we shouldn't bother with patches 3-4, and I'll plan to go back to leaving them out in the next version, unless someone yells. - Eric
On Sat, 26 Oct 2024 at 06:10, Eric Biggers <ebiggers@kernel.org> wrote: > > On Fri, Oct 25, 2024 at 10:02:39PM +0000, Eric Biggers wrote: > > On Fri, Oct 25, 2024 at 10:47:15PM +0200, Ard Biesheuvel wrote: > > > On Fri, 25 Oct 2024 at 21:15, Eric Biggers <ebiggers@kernel.org> wrote: > > > > > > > > From: Eric Biggers <ebiggers@google.com> > > > > > > > > Instead of registering the crc32-$arch and crc32c-$arch algorithms if > > > > the arch-specific code was built, only register them when that code was > > > > built *and* is not falling back to the base implementation at runtime. > > > > > > > > This avoids confusing users like btrfs which checks the shash driver > > > > name to determine whether it is crc32c-generic. > > > > > > > > > > I think we agree that 'generic' specifically means a C implementation > > > that is identical across all architectures, which is why I updated my > > > patch to export -arch instead of wrapping the C code in yet another > > > driver just for the fuzzing tests. > > > > > > So why is this a problem? If no optimizations are available at > > > runtime, crc32-arch and crc32-generic are interchangeable, and so it > > > shouldn't matter whether you use one or the other. > > > > > > You can infer from the driver name whether the C code is being used, > > > not whether or not the implementation is 'fast', and the btrfs hack is > > > already broken on arm64. > > > > > > > (It would also make sense to change btrfs to test the crc32_optimization > > > > flags itself, so that it doesn't have to use the weird hack of parsing > > > > the driver name. This change still makes sense either way though.) > > > > > > > > > > Indeed. That hack is very dubious and I'd be inclined just to ignore > > > this. On x86 and arm64, it shouldn't make a difference, given that > > > crc32-arch will be 'fast' in the vast majority of cases. On other > > > architectures, btrfs may use the C implementation while assuming it is > > > something faster, and if anyone actually notices the difference, we > > > can work with the btrfs devs to do something more sensible here. > > > > Yes, we probably could get away without this. It's never really been > > appropriate to use the crypto driver names for anything important. And btrfs > > probably should just assume CRC32C == fast unconditionally, like what it does > > with xxHash64, or even do a quick benchmark to measure the actual speed of its > > hash algorithm (which can also be sha256 or blake2b which can be very fast too). > > > > Besides the btrfs case, my concern was there may be advice floating around about > > checking /proc/crypto to check what optimized code is being used. Having > > crc32-$arch potentially be running the generic code would make that misleading. > > It might make sense to keep it working similar to how it did before. > > > > But I do agree that we could probably get away without this. > > While testing this patchset I notice that none of the crypto API drivers for > crc32 or crc32c even need to be loaded on my system anymore, as everything on my > system that uses those algorithms (such as ext4) just uses the library APIs now. > That makes the "check /proc/crypto" trick stop working anyway. > > I think you're right that we shouldn't bother with patches 3-4, and I'll plan to > go back to leaving them out in the next version, unless someone yells. > Agreed. If we need to make this distinction, it might be cleaner to use the static_call API instead, e.g., +DECLARE_STATIC_CALL(crc32_le_arch, crc32_le_base); + static inline u32 __pure crc32_le(u32 crc, const u8 *p, size_t len) { if (IS_ENABLED(CONFIG_CRC32_ARCH)) - return crc32_le_arch(crc, p, len); + return static_call(crc32_le_arch)(crc, p, len); return crc32_le_base(crc, p, len); } and use static_call_update() to update the target if the feature is supported. Then, we could check in the driver whether the static call points to the default or not: +static bool have_arch; + static int __init crc32_mod_init(void) { + have_arch = IS_ENABLED(CONFIG_CRC32_ARCH) && + static_call_query(crc32_le_arch) != crc32_le_base; + /* register the arch flavor only if it differs from the generic one */ - return crypto_register_shashes(algs, 1 + IS_ENABLED(CONFIG_CRC32_ARCH)); + return crypto_register_shashes(algs, 1 + have_arch); }
On Sat, 2 Nov 2024 at 10:45, Herbert Xu <herbert@gondor.apana.org.au> wrote: > > Eric Biggers <ebiggers@kernel.org> wrote: > > > > While testing this patchset I notice that none of the crypto API drivers for > > crc32 or crc32c even need to be loaded on my system anymore, as everything on my > > system that uses those algorithms (such as ext4) just uses the library APIs now. > > That makes the "check /proc/crypto" trick stop working anyway. > > What's stopping us from removing them altogether? > At least btrfs supports a variety of checksums/hashes (crc32c, xxhash, sha) via the shash API. There are some other remaining uses of crc32c using shash or sync ahash where the algo is hardcoded (NVMe, infiniband) so I imagine those might be candidates for conversion as well.
On Sat, Nov 02, 2024 at 10:58:53AM +0100, Ard Biesheuvel wrote: > > At least btrfs supports a variety of checksums/hashes (crc32c, xxhash, > sha) via the shash API. OK, given that btrfs is still doing this, I think we should still register crc32c-arch conditionally. Having it point to crc32c-generic is just confusing (at least if you use btrfs). Thanks,
On Sat, 2 Nov 2024 at 11:20, Herbert Xu <herbert@gondor.apana.org.au> wrote: > > On Sat, Nov 02, 2024 at 10:58:53AM +0100, Ard Biesheuvel wrote: > > > > At least btrfs supports a variety of checksums/hashes (crc32c, xxhash, > > sha) via the shash API. > > OK, given that btrfs is still doing this, I think we should still > register crc32c-arch conditionally. Having it point to crc32c-generic > is just confusing (at least if you use btrfs). > Agreed. So we should take this patch. The current issue with btrfs is that it will misidentify crc32c-generic on arm64 as being 'slow', but this was already fixed by my patches that are already in cryptodev. On arm64, crc32 instructions are always available (the only known micro-architecture that omitted them has been obsolete for years), and on x86_64 the situation is similar in practice (introduced in SSE 4.2), and so this patch changes very little for the majority of btrfs users. But on architectures such as 32-bit ARM, where these instructions are only available if you are booting a 32-bit kernel on a 64-bit CPU (which is more common than you might think), this patch will ensure that crc32-arm / crc32c-arm are only registered if the instructions are actually available, and btrfs will take the slow async patch for checksumming if they are not. (I seriously doubt that btrfs on 32-bit ARM is a thing but who knows)
On Sat, 2 Nov 2024 at 11:46, Ard Biesheuvel <ardb@kernel.org> wrote: > > On Sat, 2 Nov 2024 at 11:20, Herbert Xu <herbert@gondor.apana.org.au> wrote: > > > > On Sat, Nov 02, 2024 at 10:58:53AM +0100, Ard Biesheuvel wrote: > > > > > > At least btrfs supports a variety of checksums/hashes (crc32c, xxhash, > > > sha) via the shash API. > > > > OK, given that btrfs is still doing this, I think we should still > > register crc32c-arch conditionally. Having it point to crc32c-generic > > is just confusing (at least if you use btrfs). > > > > Agreed. So we should take this patch. > > The current issue with btrfs is that it will misidentify > crc32c-generic on arm64 as being 'slow', but this was already fixed by > my patches that are already in cryptodev. > > On arm64, crc32 instructions are always available (the only known > micro-architecture that omitted them has been obsolete for years), and > on x86_64 the situation is similar in practice (introduced in SSE > 4.2), and so this patch changes very little for the majority of btrfs > users. > > But on architectures such as 32-bit ARM, where these instructions are > only available if you are booting a 32-bit kernel on a 64-bit CPU > (which is more common than you might think), this patch will ensure > that crc32-arm / crc32c-arm are only registered if the instructions > are actually available, and btrfs will take the slow async patch for > checksumming if they are not. (I seriously doubt that btrfs on 32-bit > ARM is a thing but who knows) (actually, backpedalling a little bit - apologies) OTOH,btrfs is the only user where this makes a difference, and its use of the driver name is highly questionable IMO. On x86, it shouldn't make a difference in practice, on arm64, it was broken for a long time, and on the remaining architectures, I seriously doubt that anyone cares about this, and so we can fix this properly if there is a need. The only issue resulting from *not* taking this patch is that btrfs may misidentify the CRC32 implementation as being 'slow' and take an alternative code path, which does not necessarily result in worse performance. And I'd prefer static_call() / static_call_query() over a separate global variable to keep track of whether or not the generic code is in use.
On Sat, Nov 02, 2024 at 12:05:01PM +0100, Ard Biesheuvel wrote: > > The only issue resulting from *not* taking this patch is that btrfs > may misidentify the CRC32 implementation as being 'slow' and take an > alternative code path, which does not necessarily result in worse > performance. If we were removing crc32* (or at least crc32*-arch) from the Crypto API then these patches would be redundant. But if we're keeping them because btrfs uses them then we should definitely make crc32*-arch do the right thing. IOW they should not be registered if they're the same as crc32*-generic. Thanks,
On Sat, Nov 02, 2024 at 07:08:43PM +0800, Herbert Xu wrote: > On Sat, Nov 02, 2024 at 12:05:01PM +0100, Ard Biesheuvel wrote: > > > > The only issue resulting from *not* taking this patch is that btrfs > > may misidentify the CRC32 implementation as being 'slow' and take an > > alternative code path, which does not necessarily result in worse > > performance. > > If we were removing crc32* (or at least crc32*-arch) from the Crypto > API then these patches would be redundant. But if we're keeping them > because btrfs uses them then we should definitely make crc32*-arch > do the right thing. IOW they should not be registered if they're > the same as crc32*-generic. > > Thanks, I would like to eventually remove crc32 and crc32c from the crypto API, but it will take some time to get all the users converted. If there are AF_ALG users it could even be impossible, though the usual culprit, iwd, doesn't appear to use any CRCs, so hopefully we are fine there. I will plan to keep this patch, but change it to use a crc32_optimizations() function instead which was Ard's first suggestion. I don't think Ard's static_call suggestion would work as-is, since considering the following: static inline u32 __pure crc32_le(u32 crc, const u8 *p, size_t len) { if (IS_ENABLED(CONFIG_CRC32_ARCH)) return static_call(crc32_le_arch)(crc, p, len); return crc32_le_base(crc, p, len); } ... the 'static_call(crc32_le_arch)(crc, p, len)' will be inlined into every user, which could be a loadable module which gets loaded after crc32-${arch}.ko. And AFAIK, static calls in that module won't be updated in that case. That could be avoided by making crc32_le() a non-inline function in lib/crc32.c, so the static call would only be in that one place. That has the slight disadvantage that it would introduce an extra jump into the common case where the optimized function is enabled. Considering that some users are passing small amounts of data into the CRC functions (e.g., 4 bytes), I would like to minimize the overhead as much as possible. It could also be avoided by making CRC32 and CRC32_ARCH bool rather than tristate. I would prefer not to do that, since there can be situations where only loadable modules need these functions so they should not have to be built into the core kernel. So I plan to go with the crc32_optimizations() solution in v3. - Eric
On Sat, 2 Nov 2024 at 17:36, Eric Biggers <ebiggers@kernel.org> wrote: > > On Sat, Nov 02, 2024 at 07:08:43PM +0800, Herbert Xu wrote: > > On Sat, Nov 02, 2024 at 12:05:01PM +0100, Ard Biesheuvel wrote: > > > > > > The only issue resulting from *not* taking this patch is that btrfs > > > may misidentify the CRC32 implementation as being 'slow' and take an > > > alternative code path, which does not necessarily result in worse > > > performance. > > > > If we were removing crc32* (or at least crc32*-arch) from the Crypto > > API then these patches would be redundant. But if we're keeping them > > because btrfs uses them then we should definitely make crc32*-arch > > do the right thing. IOW they should not be registered if they're > > the same as crc32*-generic. > > > > Thanks, > > I would like to eventually remove crc32 and crc32c from the crypto API, but it > will take some time to get all the users converted. If there are AF_ALG users > it could even be impossible, though the usual culprit, iwd, doesn't appear to > use any CRCs, so hopefully we are fine there. > > I will plan to keep this patch, but change it to use a crc32_optimizations() > function instead which was Ard's first suggestion. > > I don't think Ard's static_call suggestion would work as-is, since considering > the following: > > static inline u32 __pure crc32_le(u32 crc, const u8 *p, size_t len) > { > if (IS_ENABLED(CONFIG_CRC32_ARCH)) > return static_call(crc32_le_arch)(crc, p, len); > return crc32_le_base(crc, p, len); > } > > ... the 'static_call(crc32_le_arch)(crc, p, len)' will be inlined into every > user, which could be a loadable module which gets loaded after crc32-${arch}.ko. > And AFAIK, static calls in that module won't be updated in that case. > Any call to static_call_update() will update all existing users, so this should work as expected. (Only x86 has a non-trivial implementation that patches callers inline - otherwise, it is either an indirect call involving a global function pointer variable, or a single trampoline that gets patched to point somewhere else) ... > > So I plan to go with the crc32_optimizations() solution in v3. > That is also fine with me.
On 11/2/24 5:36 PM, Eric Biggers wrote: > On Sat, Nov 02, 2024 at 07:08:43PM +0800, Herbert Xu wrote: >> On Sat, Nov 02, 2024 at 12:05:01PM +0100, Ard Biesheuvel wrote: >>> >>> The only issue resulting from *not* taking this patch is that btrfs >>> may misidentify the CRC32 implementation as being 'slow' and take an >>> alternative code path, which does not necessarily result in worse >>> performance. >> >> If we were removing crc32* (or at least crc32*-arch) from the Crypto >> API then these patches would be redundant. But if we're keeping them >> because btrfs uses them then we should definitely make crc32*-arch >> do the right thing. IOW they should not be registered if they're >> the same as crc32*-generic. >> >> Thanks, > > I would like to eventually remove crc32 and crc32c from the crypto API, but it > will take some time to get all the users converted. If there are AF_ALG users > it could even be impossible, though the usual culprit, iwd, doesn't appear to > use any CRCs, so hopefully we are fine there. Hi, Please do not forget about dm-integrity, it can use crc32/crc32c for non-cryptographic integrity through crypto API. To test it, cryptsetup testsuite should cover these variants (integrity-compat-test). Also, libcryptsetup can be compiled with userspace kernel crypto (AF_ALG) as a crypto backend. At least, I think we never used CRC32 though AF_ALG from userspace there... Thanks, Milan
diff --git a/crypto/crc32_generic.c b/crypto/crc32_generic.c index cc064ea8240e..cecd01e4d6e6 100644 --- a/crypto/crc32_generic.c +++ b/crypto/crc32_generic.c @@ -155,19 +155,23 @@ static struct shash_alg algs[] = {{ .base.cra_ctxsize = sizeof(u32), .base.cra_module = THIS_MODULE, .base.cra_init = crc32_cra_init, }}; +static int num_algs; + static int __init crc32_mod_init(void) { /* register the arch flavor only if it differs from the generic one */ - return crypto_register_shashes(algs, 1 + IS_ENABLED(CONFIG_CRC32_ARCH)); + num_algs = 1 + ((crc32_optimizations & CRC32_LE_OPTIMIZATION) != 0); + + return crypto_register_shashes(algs, num_algs); } static void __exit crc32_mod_fini(void) { - crypto_unregister_shashes(algs, 1 + IS_ENABLED(CONFIG_CRC32_ARCH)); + crypto_unregister_shashes(algs, num_algs); } subsys_initcall(crc32_mod_init); module_exit(crc32_mod_fini); diff --git a/crypto/crc32c_generic.c b/crypto/crc32c_generic.c index 04b03d825cf4..47d694da9d4a 100644 --- a/crypto/crc32c_generic.c +++ b/crypto/crc32c_generic.c @@ -195,19 +195,23 @@ static struct shash_alg algs[] = {{ .base.cra_ctxsize = sizeof(struct chksum_ctx), .base.cra_module = THIS_MODULE, .base.cra_init = crc32c_cra_init, }}; +static int num_algs; + static int __init crc32c_mod_init(void) { /* register the arch flavor only if it differs from the generic one */ - return crypto_register_shashes(algs, 1 + IS_ENABLED(CONFIG_CRC32_ARCH)); + num_algs = 1 + ((crc32_optimizations & CRC32C_OPTIMIZATION) != 0); + + return crypto_register_shashes(algs, num_algs); } static void __exit crc32c_mod_fini(void) { - crypto_unregister_shashes(algs, 1 + IS_ENABLED(CONFIG_CRC32_ARCH)); + crypto_unregister_shashes(algs, num_algs); } subsys_initcall(crc32c_mod_init); module_exit(crc32c_mod_fini);