[2/4] crypto: iaa - Add deflate-canned compression algorithm

The 'canned' compression mode implements a compression scheme that uses
statically-defined customized Huffman tables. Since the Deflate block
header is a constant, it is not stored with the compressed data. The
Huffman tables used were generated from statistics derived from swapped
4KB pages of SPEC CPU17 workloads. Canned mode provides better compression
than using the Deflate standard's (RFC-1951) "fixed" tables and better
latencies than dynamic compression.

Compressing a data set of 4300 4KB pages sampled from SPEC CPU17 workloads
produces a compression ratio of 2.88 for IAA canned compression and 2.69
for IAA fixed compression, which is a 7% improvement.

Either 'fixed' or 'canned' modes can be chosen as the mode to be used
by crypto facilities by selecting the corresponding algorithm.  For
example, to use IAA fixed mode in zswap:

  echo deflate-iaa > /sys/module/zswap/parameters/compressor

To choose 'canned' mode:

  echo deflate-iaa-canned > /sys/module/zswap/parameters/compressor

[ Based on work originally by George Powley, Jing Lin and Kyung Min
Park ]

Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Signed-off-by: Andre Glover <andre.glover@linux.intel.com>
---
 .../driver-api/crypto/iaa/iaa-crypto.rst      |  15 +-
 crypto/testmgr.c                              |  10 +
 crypto/testmgr.h                              |  72 +++++
 drivers/crypto/intel/iaa/Kconfig              |   1 +
 drivers/crypto/intel/iaa/Makefile             |   2 +-
 drivers/crypto/intel/iaa/iaa_crypto.h         |  38 +++
 .../crypto/intel/iaa/iaa_crypto_comp_canned.c | 116 ++++++++
 .../crypto/intel/iaa/iaa_crypto_comp_fixed.c  |   1 +
 drivers/crypto/intel/iaa/iaa_crypto_main.c    | 271 ++++++++++++++++--
 9 files changed, 500 insertions(+), 26 deletions(-)
 create mode 100644 drivers/crypto/intel/iaa/iaa_crypto_comp_canned.c

Hi Andre,

Some comments inline below...

On Thu, 2024-03-28 at 10:44 -0700, Andre Glover wrote:
> The 'canned' compression mode implements a compression scheme that uses
> statically-defined customized Huffman tables. Since the Deflate block
> header is a constant, it is not stored with the compressed data. The
> Huffman tables used were generated from statistics derived from swapped
> 4KB pages of SPEC CPU17 workloads. Canned mode provides better compression
> than using the Deflate standard's (RFC-1951) "fixed" tables and better
> latencies than dynamic compression.
> 
> Compressing a data set of 4300 4KB pages sampled from SPEC CPU17 workloads
> produces a compression ratio of 2.88 for IAA canned compression and 2.69
> for IAA fixed compression, which is a 7% improvement.
> 
> Either 'fixed' or 'canned' modes can be chosen as the mode to be used
> by crypto facilities by selecting the corresponding algorithm.  For
> example, to use IAA fixed mode in zswap:
> 
>   echo deflate-iaa > /sys/module/zswap/parameters/compressor
> 
> To choose 'canned' mode:
> 
>   echo deflate-iaa-canned > /sys/module/zswap/parameters/compressor
> 
> [ Based on work originally by George Powley, Jing Lin and Kyung Min
> Park ]
> 
> Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
> Signed-off-by: Andre Glover <andre.glover@linux.intel.com>
> ---
>  .../driver-api/crypto/iaa/iaa-crypto.rst      |  15 +-
>  crypto/testmgr.c                              |  10 +
>  crypto/testmgr.h                              |  72 +++++
>  drivers/crypto/intel/iaa/Kconfig              |   1 +
>  drivers/crypto/intel/iaa/Makefile             |   2 +-
>  drivers/crypto/intel/iaa/iaa_crypto.h         |  38 +++
>  .../crypto/intel/iaa/iaa_crypto_comp_canned.c | 116 ++++++++
>  .../crypto/intel/iaa/iaa_crypto_comp_fixed.c  |   1 +
>  drivers/crypto/intel/iaa/iaa_crypto_main.c    | 271 ++++++++++++++++--
>  9 files changed, 500 insertions(+), 26 deletions(-)
>  create mode 100644 drivers/crypto/intel/iaa/iaa_crypto_comp_canned.c
> 
> diff --git a/Documentation/driver-api/crypto/iaa/iaa-crypto.rst b/Documentation/driver-api/crypto/iaa/iaa-crypto.rst
> index 7b28aef39ba0..b64bd780ad87 100644
> --- a/Documentation/driver-api/crypto/iaa/iaa-crypto.rst
> +++ b/Documentation/driver-api/crypto/iaa/iaa-crypto.rst
> @@ -30,8 +30,8 @@ algorithm::
>  This will tell zswap to use the IAA 'fixed' compression mode for all
>  compresses and decompresses.
>  
> -Currently, there is only one compression modes available, 'fixed'
> -mode.
> +Currently, two compression modes are available to all IAA hardware,
> +'canned' and 'fixed' modes.
>  
>  The 'fixed' compression mode implements the compression scheme
>  specified by RFC 1951 and is given the crypto algorithm name
> @@ -42,6 +42,17 @@ which allows for a window of up to 32k.  Because of this limitation,
>  the IAA fixed mode deflate algorithm is given its own algorithm name
>  rather than simply 'deflate').
>  
> +The 'canned' compression mode implements a good general-purpose
> +compression scheme whose tables were generated from statistics derived
> +from a wide variety of SPEC17 workloads.  It provides much better
> +overall characteristics than the existing deflate-1951 tables
> +implemented by 'fixed'. 'Canned' mode is implemented by the
> +'deflate-iaa-canned' crypto algorithm.::

The double colon here isn't necessary.

> +
> +A zswap device can select the IAA 'canned' mode represented by
> +selecting the 'deflate-iaa-canned' crypto compression algorithm::
> +
> +  # echo deflate-iaa-canned > /sys/module/zswap/parameters/compressor
>  
>  Config options and other setup
>  ==============================
> diff --git a/crypto/testmgr.c b/crypto/testmgr.c
> index 94fd9ceef207..8981e8fb220b 100644
> --- a/crypto/testmgr.c
> +++ b/crypto/testmgr.c
> @@ -4806,6 +4806,16 @@ static const struct alg_test_desc alg_test_descs[] = {
>                                 .decomp = __VECS(deflate_decomp_tv_template)
>                         }
>                 }
> +       }, {
> +               .alg = "deflate-iaa-canned",
> +               .test = alg_test_comp,
> +               .fips_allowed = 1,
> +               .suite = {
> +                       .comp = {
> +                               .comp = __VECS(deflate_iaa_canned_comp_tv_template),
> +                               .decomp = __VECS(deflate_iaa_canned_decomp_tv_template)
> +                       }
> +               }
>         }, {
>                 .alg = "deflate-scomp-canned",
>                 .test = alg_test_comp,
> diff --git a/crypto/testmgr.h b/crypto/testmgr.h
> index 38c4c96e11e2..c79ba30c2194 100644
> --- a/crypto/testmgr.h
> +++ b/crypto/testmgr.h
> @@ -34526,6 +34526,78 @@ static const struct comp_testvec deflate_decomp_tv_template[] = {
>         },
>  };
>  
> +static const struct comp_testvec deflate_iaa_canned_comp_tv_template[] = {
> +       {
> +               .inlen  = 70,
> +               .outlen = 37,
> +               .input  = "Join us now and share the software "
> +                       "Join us now and share the software ",
> +               .output = "\x6d\x23\x43\x23\xa4\x71\x31\xd2"
> +                         "\x88\xc8\x61\x52\x75\x84\x56\x1a"
> +                         "\x13\xa2\x8e\xd6\x49\x63\x43\x74"
> +                         "\xd2\x98\xc8\xe0\xd8\x61\x58\x69"
> +                         "\xcb\x71\x01\xe5\x3f",
> +       }, {
> +               .inlen  = 191,
> +               .outlen = 128,
> +               .input  = "This document describes a compression method based on the DEFLATE"
> +                       "compression algorithm.  This document defines the application of "
> +                       "the DEFLATE algorithm to the IP Payload Compression Protocol.",
> +               .output = "\xdd\x42\x42\x63\xa4\xda\x48\x4d"
> +                         "\x5c\xb8\x2e\x22\x56\xaa\xd5\xc5"
> +                         "\x68\xa2\x43\x83\x74\x31\x52\xb5"
> +                         "\x54\x13\x19\x1e\x15\xad\x8b\x89"
> +                         "\x09\x8d\x8c\x90\x86\xeb\x62\x43"
> +                         "\x22\xb5\xd2\x20\x75\x8c\x4e\x2b"
> +                         "\x05\x3d\x36\x44\x27\xf5\x69\xe5"
> +                         "\xdb\xde\xbb\x5b\x2b\x7d\x37\x75"
> +                         "\xd8\xc0\xc8\xe8\xd0\xd8\x90\x70"
> +                         "\x7b\xa9\x54\x1c\x38\x38\x34\x02"
> +                         "\xc2\xe2\x8e\xea\xa8\xa8\xb0\x50"
> +                         "\x8d\x3a\x16\xf7\x88\x0c\xd6\x8f"
> +                         "\x95\x1f\x40\x1a\x1b\x29\x34\xb4"
> +                         "\xf1\x97\xfa\xab\x87\x87\x45\xaa"
> +                         "\xb5\xd2\x96\x7a\x03\xf9\x47\x47"
> +                         "\xc6\x46\x6a\x22\xc3\xec\xff\x03",
> +       },
> +};
> +
> +static const struct comp_testvec deflate_iaa_canned_decomp_tv_template[] = {
> +       {
> +               .inlen  = 128,
> +               .outlen = 191,
> +               .input  = "\xdd\x42\x42\x63\xa4\xda\x48\x4d"
> +                         "\x5c\xb8\x2e\x22\x56\xaa\xd5\xc5"
> +                         "\x68\xa2\x43\x83\x74\x31\x52\xb5"
> +                         "\x54\x13\x19\x1e\x15\xad\x8b\x89"
> +                         "\x09\x8d\x8c\x90\x86\xeb\x62\x43"
> +                         "\x22\xb5\xd2\x20\x75\x8c\x4e\x2b"
> +                         "\x05\x3d\x36\x44\x27\xf5\x69\xe5"
> +                         "\xdb\xde\xbb\x5b\x2b\x7d\x37\x75"
> +                         "\xd8\xc0\xc8\xe8\xd0\xd8\x90\x70"
> +                         "\x7b\xa9\x54\x1c\x38\x38\x34\x02"
> +                         "\xc2\xe2\x8e\xea\xa8\xa8\xb0\x50"
> +                         "\x8d\x3a\x16\xf7\x88\x0c\xd6\x8f"
> +                         "\x95\x1f\x40\x1a\x1b\x29\x34\xb4"
> +                         "\xf1\x97\xfa\xab\x87\x87\x45\xaa"
> +                         "\xb5\xd2\x96\x7a\x03\xf9\x47\x47"
> +                         "\xc6\x46\x6a\x22\xc3\xec\xff\x03",
> +               .output = "This document describes a compression method based on the DEFLATE"
> +                       "compression algorithm.  This document defines the application of "
> +                       "the DEFLATE algorithm to the IP Payload Compression Protocol.",
> +       }, {
> +               .inlen  = 37,
> +               .outlen = 70,
> +               .input  = "\x6d\x23\x43\x23\xa4\x71\x31\xd2"
> +                         "\x88\xc8\x61\x52\x75\x84\x56\x1a"
> +                         "\x13\xa2\x8e\xd6\x49\x63\x43\x74"
> +                         "\xd2\x98\xc8\xe0\xd8\x61\x58\x69"
> +                         "\xcb\x71\x01\xe5\x3f",
> +               .output = "Join us now and share the software "
> +                       "Join us now and share the software ",
> +       },
> +};
> +
>  static const struct comp_testvec deflate_scomp_canned_comp_tv_template[] = {
>         {
>                 .inlen  = 70,
> diff --git a/drivers/crypto/intel/iaa/Kconfig b/drivers/crypto/intel/iaa/Kconfig
> index d53f4b1d494f..a78fb0db814f 100644
> --- a/drivers/crypto/intel/iaa/Kconfig
> +++ b/drivers/crypto/intel/iaa/Kconfig
> @@ -2,6 +2,7 @@ config CRYPTO_DEV_IAA_CRYPTO
>         tristate "Support for Intel(R) IAA Compression Accelerator"
>         depends on CRYPTO_DEFLATE
>         depends on INTEL_IDXD
> +       select ZLIB_CANNED
>         default n
>         help
>           This driver supports acceleration for compression and
> diff --git a/drivers/crypto/intel/iaa/Makefile b/drivers/crypto/intel/iaa/Makefile
> index b64b208d2344..9667e89bd88a 100644
> --- a/drivers/crypto/intel/iaa/Makefile
> +++ b/drivers/crypto/intel/iaa/Makefile
> @@ -7,6 +7,6 @@ ccflags-y += -I $(srctree)/drivers/dma/idxd -DDEFAULT_SYMBOL_NAMESPACE=IDXD
>  
>  obj-$(CONFIG_CRYPTO_DEV_IAA_CRYPTO) := iaa_crypto.o
>  
> -iaa_crypto-y := iaa_crypto_main.o iaa_crypto_comp_fixed.o
> +iaa_crypto-y := iaa_crypto_main.o iaa_crypto_comp_fixed.o iaa_crypto_comp_canned.o
>  
>  iaa_crypto-$(CONFIG_CRYPTO_DEV_IAA_CRYPTO_STATS) += iaa_crypto_stats.o
> diff --git a/drivers/crypto/intel/iaa/iaa_crypto.h b/drivers/crypto/intel/iaa/iaa_crypto.h
> index 56985e395263..33ff0f95c543 100644
> --- a/drivers/crypto/intel/iaa/iaa_crypto.h
> +++ b/drivers/crypto/intel/iaa/iaa_crypto.h
> @@ -39,6 +39,15 @@
>                                          IAA_DECOMP_CHECK_FOR_EOB | \
>                                          IAA_DECOMP_STOP_ON_EOB)
>  
> +/*
> + * iaa_cap.dec_aecs_format_ver == 1 means the decompress internal state in
> + * the AECS is Format 2, which requires a different length from Format 1.
> + * See Intel® In-Memory Analytics Accelerator Architecture Specification
> + * Sections 6.4.1.1 and 7.2.2 for details.
> + */
> +#define DECOMP_INT_STATE_FMT1_LEN      1088
> +#define DECOMP_INT_STATE_FMT2_LEN      1248
> +
>  /* Representation of IAA workqueue */
>  struct iaa_wq {
>         struct list_head        list;
> @@ -59,8 +68,10 @@ struct iaa_device_compression_mode {
>         const char                      *name;
>  
>         struct aecs_comp_table_record   *aecs_comp_table;
> +       struct aecs_decomp_table_record *aecs_decomp_table;
>  
>         dma_addr_t                      aecs_comp_table_dma_addr;
> +       dma_addr_t                      aecs_decomp_table_dma_addr;
>  };
>  
>  /* Representation of IAA device with wqs, populated by probe */
> @@ -77,6 +88,7 @@ struct iaa_device {
>         atomic64_t                      comp_bytes;
>         atomic64_t                      decomp_calls;
>         atomic64_t                      decomp_bytes;

Could you add a space here, to visually separate the stats fields
from the unrelated field below?

> +       u32                             src2_size;
>  };
>  
>  struct wq_table_entry {
> @@ -105,8 +117,27 @@ struct aecs_comp_table_record {
>         u32 reserved_padding[2];
>  } __packed;
>  
> +/* AECS for decompress */
> +struct aecs_decomp_table_record {
> +       u32 crc;
> +       u32 xor_checksum;
> +       u32 low_filter_param;
> +       u32 high_filter_param;
> +       u32 output_mod_idx;
> +       u32 drop_init_decomp_out_bytes;
> +       u32 reserved[36];
> +       u32 output_accum_data[2];
> +       u32 out_bits_valid;
> +       u32 bit_off_indexing;
> +       u32 input_accum_data[64];
> +       u8  size_qw[32];
> +       u32 decomp_state[1220];
> +} __packed;
> +
>  int iaa_aecs_init_fixed(void);
>  void iaa_aecs_cleanup_fixed(void);
> +int iaa_aecs_init_canned(void);
> +void iaa_aecs_cleanup_canned(void);
>  
>  typedef int (*iaa_dev_comp_init_fn_t) (struct iaa_device_compression_mode *mode);
>  typedef int (*iaa_dev_comp_free_fn_t) (struct iaa_device_compression_mode *mode);
> @@ -117,6 +148,9 @@ struct iaa_compression_mode {
>         int                     ll_table_size;
>         u32                     *d_table;
>         int                     d_table_size;
> +       u8                      *header_table;
> +       int                     header_table_size;
> +       u16                     gen_decomp_table_flags;
>         iaa_dev_comp_init_fn_t  init;
>         iaa_dev_comp_free_fn_t  free;
>  };
> @@ -126,6 +160,9 @@ int add_iaa_compression_mode(const char *name,
>                              int ll_table_size,
>                              const u32 *d_table,
>                              int d_table_size,
> +                            const u8 *header_table,
> +                            int header_table_size,
> +                            u16 gen_decomp_table_flags,
>                              iaa_dev_comp_init_fn_t init,
>                              iaa_dev_comp_free_fn_t free);
>  
> @@ -133,6 +170,7 @@ void remove_iaa_compression_mode(const char *name);
>  
>  enum iaa_mode {
>         IAA_MODE_FIXED,
> +       IAA_MODE_CANNED,
>  };
>  
>  struct iaa_compression_ctx {
> diff --git a/drivers/crypto/intel/iaa/iaa_crypto_comp_canned.c b/drivers/crypto/intel/iaa/iaa_crypto_comp_canned.c
> new file mode 100644
> index 000000000000..26ea887978e6
> --- /dev/null
> +++ b/drivers/crypto/intel/iaa/iaa_crypto_comp_canned.c
> @@ -0,0 +1,116 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/* Copyright(c) 2021 Intel Corporation. All rights rsvd. */
> +
> +#include "iaa_crypto.h"
> +#include "idxd.h"
> +
> +#define IAA_AECS_ALIGN 32
> +
> +/*
> + * These tables were generated from statistics derived from a wide
> + * variety of SPEC17 workloads and implement a good general-purpose
> + * compression scheme called simply 'canned'.
> + */
> +
> +static const u32 canned_ll_iaa[286] = {
> +0x28002,  0x38024,  0x40066,  0x40067,  0x40068,  0x48144,  0x40069,  0x48145,
> +0x4006a,  0x48146,  0x4006b,  0x48147,  0x48148,  0x48149,  0x4814a,  0x4814b,
> +0x4006c,  0x4814c,  0x4814d,  0x4814e,  0x4814f,  0x48150,  0x48151,  0x48152,
> +0x4006d,  0x48153,  0x48154,  0x48155,  0x48156,  0x48157,  0x48158,  0x48159,
> +0x38025,  0x4815a,  0x4815b,  0x4815c,  0x4815d,  0x4815e,  0x4815f,  0x48160,
> +0x4006e,  0x48161,  0x48162,  0x48163,  0x48164,  0x48165,  0x4006f,  0x48166,
> +0x38026,  0x38027,  0x40070,  0x40071,  0x40072,  0x40073,  0x40074,  0x40075,
> +0x38028,  0x40076,  0x40077,  0x48167,  0x40078,  0x40079,  0x4007a,  0x38029,
> +0x3802a,  0x4007b,  0x48168,  0x48169,  0x4007c,  0x4816a,  0x4007d,  0x4816b,
> +0x4007e,  0x4816c,  0x4816d,  0x4816e,  0x4816f,  0x48170,  0x48171,  0x48172,
> +0x4007f,  0x48173,  0x48174,  0x48175,  0x48176,  0x48177,  0x48178,  0x48179,
> +0x40080,  0x4817a,  0x4817b,  0x4817c,  0x4817d,  0x4817e,  0x4817f,  0x48180,
> +0x40081,  0x3802b,  0x40082,  0x3802c,  0x3802d,  0x3802e,  0x40083,  0x48181,
> +0x40084,  0x40085,  0x48182,  0x48183,  0x40086,  0x40087,  0x40088,  0x40089,
> +0x4008a,  0x48184,  0x4008b,  0x4008c,  0x4008d,  0x4008e,  0x48185,  0x48186,
> +0x4008f,  0x48187,  0x48188,  0x48189,  0x4818a,  0x4818b,  0x4818c,  0x4818d,
> +0x40090,  0x4818e,  0x4818f,  0x48190,  0x48191,  0x48192,  0x48193,  0x48194,
> +0x40091,  0x48195,  0x48196,  0x48197,  0x48198,  0x48199,  0x4819a,  0x4819b,
> +0x40092,  0x4819c,  0x4819d,  0x4819e,  0x4819f,  0x481a0,  0x481a1,  0x481a2,
> +0x40093,  0x481a3,  0x481a4,  0x481a5,  0x481a6,  0x481a7,  0x481a8,  0x481a9,
> +0x40094,  0x481aa,  0x481ab,  0x481ac,  0x481ad,  0x481ae,  0x481af,  0x481b0,
> +0x481b1,  0x481b2,  0x481b3,  0x481b4,  0x481b5,  0x481b6,  0x481b7,  0x481b8,
> +0x40095,  0x481b9,  0x481ba,  0x481bb,  0x481bc,  0x481bd,  0x481be,  0x481bf,
> +0x40096,  0x481c0,  0x481c1,  0x481c2,  0x481c3,  0x481c4,  0x481c5,  0x40097,
> +0x40098,  0x481c6,  0x481c7,  0x481c8,  0x481c9,  0x481ca,  0x481cb,  0x481cc,
> +0x40099,  0x481cd,  0x481ce,  0x481cf,  0x481d0,  0x481d1,  0x481d2,  0x481d3,
> +0x4009a,  0x481d4,  0x481d5,  0x481d6,  0x481d7,  0x481d8,  0x481d9,  0x481da,
> +0x481db,  0x481dc,  0x481dd,  0x481de,  0x481df,  0x481e0,  0x481e1,  0x481e2,
> +0x4009b,  0x481e3,  0x481e4,  0x481e5,  0x481e6,  0x481e7,  0x481e8,  0x481e9,
> +0x481ea,  0x481eb,  0x481ec,  0x481ed,  0x481ee,  0x481ef,  0x481f0,  0x481f1,
> +0x4009c,  0x481f2,  0x481f3,  0x481f4,  0x481f5,  0x481f6,  0x481f7,  0x481f8,
> +0x481f9,  0x481fa,  0x481fb,  0x587fe,  0x481fc,  0x481fd,  0x481fe,  0x4009d,
> +0x503fe,  0x20000,  0x28003,  0x30010,  0x28004,  0x28005,  0x28006,  0x4009e,
> +0x4009f,  0x3802f,  0x38030,  0x30011,  0x400a0,  0x38031,  0x38032,  0x400a1,
> +0x28007,  0x73ff8,  0x73ff9,  0x7fff6,  0x7fff7,  0x7fff8,  0x7fff9,  0x7fffa,
> +0x7fffb,  0x7fffc,  0x7fffd,  0x7fffe,  0x7ffff,  0x73ffa,
> +};
> +
> +static const u32 canned_d_iaa[30] = {
> +0x3807e,  0x20004,  0x481fe,  0x18000,  0x400fe,  0x18001,  0x3003c,  0x20005,
> +0x20006,  0x28016,  0x20007,  0x28017,  0x20008,  0x28018,  0x28019,  0x20009,
> +0x2000a,  0x2801a,  0x2801b,  0x2801c,  0x2801d,  0x3003d,  0x3003e,  0x481ff,
> +0x0,  0x0,  0x0,  0x0,  0x0,  0x0,
> +};
> +
> +#define HEADER_SIZE_IN_BITS 656
> +static const u8 canned_header[] = {
> +0xed, 0xfd, 0x05, 0x5c, 0x54, 0xcf, 0x17, 0x06,
> +0x8c, 0xcf, 0x02, 0x22, 0x62, 0x61, 0x63, 0xaf,
> +0x8d, 0x85, 0xbb, 0xe4, 0x82, 0xc0, 0x2e, 0x8a,
> +0x88, 0x8d, 0x8d, 0xcd, 0xb2, 0xbb, 0x08, 0x4a,
> +0x49, 0x28, 0x36, 0x36, 0x36, 0x36, 0x36, 0x76,
> +0x2b, 0x36, 0x26, 0xd8, 0xd8, 0xd8, 0xad, 0xd8,
> +0xad, 0xd8, 0xf1, 0x53, 0xf9, 0x9e, 0xb9, 0x33,
> +0x03, 0xbb, 0x97, 0x5d, 0xe6, 0x7d, 0xff, 0x1d,
> +0xef, 0x1e, 0x1f, 0x9e, 0x7b, 0xce, 0x9c, 0x39,
> +0x33, 0xe7, 0x4c, 0xdc, 0xeb, 0xe7, 0xf3, 0xf9,
> +0x7c, 0x3e,
> +};
> +
> +#define CEIL(a, b)     (((a) + ((b) - 1)) / (b))
> +
> +int iaa_aecs_init_canned(void)
> +{
> +       u16 gen_decomp_table_flags;
> +       unsigned int slen;
> +       int ret;
> +
> +       slen = CEIL(HEADER_SIZE_IN_BITS, 8);
> +
> +       gen_decomp_table_flags = 0x1; /* enable decompression */
> +       gen_decomp_table_flags |= 1 << 9; /* suppress output */
> +
> +       /*
> +        * Bits 8:6 specify the number of bits to ignore at the end of the
> +        * compressed input stream.
> +        * See Intel® In-Memory Analytics Accelerator Architecture Specification
> +        * Sections 6.3.3.1 for details.
> +        */
> +       gen_decomp_table_flags |= (((slen * 8) - HEADER_SIZE_IN_BITS) << 6);
> +
> +       ret = add_iaa_compression_mode("canned",
> +                                      canned_ll_iaa,
> +                                      sizeof(canned_ll_iaa),
> +                                      canned_d_iaa,
> +                                      sizeof(canned_d_iaa),
> +                                      canned_header,
> +                                      sizeof(canned_header),
> +                                      gen_decomp_table_flags,
> +                                      NULL, NULL);
> +
> +       if (!ret)
> +               pr_debug("IAA canned compression mode initialized\n");
> +
> +       return ret;
> +}
> +
> +void iaa_aecs_cleanup_canned(void)
> +{
> +       remove_iaa_compression_mode("canned");
> +}
> diff --git a/drivers/crypto/intel/iaa/iaa_crypto_comp_fixed.c b/drivers/crypto/intel/iaa/iaa_crypto_comp_fixed.c
> index 19d9a333ac49..45cf5d74f0fb 100644
> --- a/drivers/crypto/intel/iaa/iaa_crypto_comp_fixed.c
> +++ b/drivers/crypto/intel/iaa/iaa_crypto_comp_fixed.c
> @@ -78,6 +78,7 @@ int iaa_aecs_init_fixed(void)
>                                        sizeof(fixed_ll_sym),
>                                        fixed_d_sym,
>                                        sizeof(fixed_d_sym),
> +                                      NULL, 0, 0,
>                                        init_fixed_mode, NULL);
>         if (!ret)
>                 pr_debug("IAA fixed compression mode initialized\n");
> diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c
> index 1ff6b7c77d89..f141a389f5ca 100644
> --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c
> +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c
> @@ -34,6 +34,7 @@ static unsigned int nr_cpus_per_node;
>  static unsigned int cpus_per_iaa;
>  
>  static struct crypto_comp *deflate_generic_tfm;
> +static struct crypto_acomp *deflate_canned_tfm;
>  
>  /* Per-cpu lookup table for balanced wqs */
>  static struct wq_table_entry __percpu *wq_table;
> @@ -91,7 +92,8 @@ DEFINE_MUTEX(iaa_devices_lock);
>  
>  /* If enabled, IAA hw crypto algos are registered, unavailable otherwise */
>  static bool iaa_crypto_enabled;
> -static bool iaa_crypto_registered;
> +static bool iaa_crypto_fixed_registered;
> +static bool iaa_crypto_canned_registered;
>  
>  /* Verify results of IAA compress or not */
>  static bool iaa_verify_compress = true;
> @@ -258,14 +260,16 @@ static void free_iaa_compression_mode(struct iaa_compression_mode *mode)
>         kfree(mode->name);
>         kfree(mode->ll_table);
>         kfree(mode->d_table);
> +       kfree(mode->header_table);
>  
>         kfree(mode);
>  }
>  
>  /*
> - * IAA Compression modes are defined by an ll_table and a d_table.
> - * These tables are typically generated and captured using statistics
> - * collected from running actual compress/decompress workloads.
> + * IAA Compression modes are defined by an ll_table, a d_table, and an
> + * optional header_table. These tables are typically generated and
> + * captured using statistics collected from running actual
> + * compress/decompress workloads.
>   *
>   * A module or other kernel code can add and remove compression modes
>   * with a given name using the exported @add_iaa_compression_mode()
> @@ -313,6 +317,9 @@ EXPORT_SYMBOL_GPL(remove_iaa_compression_mode);
>   * @ll_table_size: The ll table size in bytes
>   * @d_table: The d table
>   * @d_table_size: The d table size in bytes
> + * @header_table: Optional header table
> + * @header_table_size: Optional header table size in bytes
> + * @gen_decomp_table_flags: Otional flags used to generate the decomp table
>   * @init: Optional callback function to init the compression mode data
>   * @free: Optional callback function to free the compression mode data
>   *
> @@ -325,6 +332,9 @@ int add_iaa_compression_mode(const char *name,
>                              int ll_table_size,
>                              const u32 *d_table,
>                              int d_table_size,
> +                            const u8 *header_table,
> +                            int header_table_size,
> +                            u16 gen_decomp_table_flags,
>                              iaa_dev_comp_init_fn_t init,
>                              iaa_dev_comp_free_fn_t free)
>  {
> @@ -362,6 +372,16 @@ int add_iaa_compression_mode(const char *name,
>                 mode->d_table_size = d_table_size;
>         }
>  
> +       if (header_table) {
> +               mode->header_table = kzalloc(header_table_size, GFP_KERNEL);
> +               if (!mode->header_table)
> +                       goto free;
> +               memcpy(mode->header_table, header_table, header_table_size);
> +               mode->header_table_size = header_table_size;
> +       }
> +
> +       mode->gen_decomp_table_flags = gen_decomp_table_flags;
> +
>         mode->init = init;
>         mode->free = free;
>  
> @@ -394,14 +414,19 @@ get_iaa_device_compression_mode(struct iaa_device *iaa_device, int idx)
>  static void free_device_compression_mode(struct iaa_device *iaa_device,
>                                          struct iaa_device_compression_mode *device_mode)
>  {
> -       size_t size = sizeof(struct aecs_comp_table_record) + IAA_AECS_ALIGN;
> +       size_t c_size = sizeof(struct aecs_comp_table_record) + IAA_AECS_ALIGN;
> +       size_t d_size = sizeof(struct aecs_decomp_table_record) + IAA_AECS_ALIGN;
>         struct device *dev = &iaa_device->idxd->pdev->dev;
>  
>         kfree(device_mode->name);
>  
>         if (device_mode->aecs_comp_table)
> -               dma_free_coherent(dev, size, device_mode->aecs_comp_table,
> +               dma_free_coherent(dev, c_size, device_mode->aecs_comp_table,
>                                   device_mode->aecs_comp_table_dma_addr);
> +       if (device_mode->aecs_decomp_table)
> +               dma_free_coherent(dev, d_size, device_mode->aecs_decomp_table,
> +                                 device_mode->aecs_decomp_table_dma_addr);
> +
>         kfree(device_mode);
>  }
>  
> @@ -418,11 +443,79 @@ static int check_completion(struct device *dev,
>                             bool compress,
>                             bool only_once);
>  
> +static int decompress_header(struct iaa_device_compression_mode *device_mode,
> +                            struct iaa_compression_mode *mode,
> +                            struct idxd_wq *wq)
> +{
> +       dma_addr_t src_addr, src2_addr;
> +       struct idxd_desc *idxd_desc;
> +       struct iax_hw_desc *desc;
> +       struct device *dev;
> +       struct iaa_wq *iaa_wq;
> +       struct iaa_device *iaa_device;
> +       int ret = 0;
> +

Just a small style thing - when possible, it's good to use 'reverse
Christmas tree' order for locals so e.g. instead of the above it would
look like this:

+       dma_addr_t src_addr, src2_addr;
+       struct iaa_device *iaa_device;
+       struct idxd_desc *idxd_desc;
+       struct iax_hw_desc *desc;
+       struct iaa_wq *iaa_wq;
+       struct device *dev;
+       int ret = 0;

So, the longest at the top, followed by the next longest, etc.

> +       idxd_desc = idxd_alloc_desc(wq, IDXD_OP_BLOCK);
> +       if (IS_ERR(idxd_desc))
> +               return PTR_ERR(idxd_desc);
> +
> +       desc = idxd_desc->iax_hw;
> +
> +       iaa_wq = idxd_wq_get_private(wq);
> +       iaa_device = iaa_wq->iaa_device;
> +       dev = &wq->idxd->pdev->dev;
> +
> +       src_addr = dma_map_single(dev, (void *)mode->header_table,
> +                                 mode->header_table_size, DMA_TO_DEVICE);
> +       dev_dbg(dev, "%s: mode->name %s, src_addr %llx, dev %p, src %p, slen %d\n",
> +               __func__, mode->name, src_addr, dev,
> +               mode->header_table, mode->header_table_size);
> +       if (unlikely(dma_mapping_error(dev, src_addr))) {
> +               dev_dbg(dev, "dma_map_single err, exiting\n");
> +               ret = -ENOMEM;
> +               return ret;
> +       }
> +       src2_addr = device_mode->aecs_decomp_table_dma_addr;
> +
> +       desc->flags = IAX_AECS_GEN_FLAG;
> +       desc->opcode = IAX_OPCODE_DECOMPRESS;
> +       desc->decompr_flags = mode->gen_decomp_table_flags;
> +       desc->priv = 0;
> +
> +       desc->src1_addr = (u64)src_addr;
> +       desc->src1_size = mode->header_table_size;
> +       desc->src2_addr = (u64)src2_addr;
> +       desc->src2_size = iaa_device->src2_size;
> +       dev_dbg(dev, "%s: mode->name %s, src2_addr %llx, dev %p, src2_size %d\n",
> +               __func__, mode->name, desc->src2_addr, dev, desc->src2_size);
> +       desc->max_dst_size = 0; // suppressed output
> +       desc->completion_addr = idxd_desc->compl_dma;
> +
> +       ret = idxd_submit_desc(wq, idxd_desc);
> +       if (ret) {
> +               pr_err("%s: submit_desc failed ret=0x%x\n", __func__, ret);
> +               goto out;
> +       }
> +
> +       ret = check_completion(dev, idxd_desc->iax_completion, false, false);
> +       if (ret)
> +               dev_dbg(dev, "%s: mode->name %s check_completion failed ret=%d\n",
> +                       __func__, mode->name, ret);
> +       else
> +               dev_dbg(dev, "%s: mode->name %s succeeded\n", __func__,
> +                       mode->name);
> +out:
> +       dma_unmap_single(dev, src_addr, desc->src2_size, DMA_TO_DEVICE);
> +
> +       return ret;
> +}
> +
>  static int init_device_compression_mode(struct iaa_device *iaa_device,
>                                         struct iaa_compression_mode *mode,
>                                         int idx, struct idxd_wq *wq)
>  {
> -       size_t size = sizeof(struct aecs_comp_table_record) + IAA_AECS_ALIGN;
> +       size_t c_size = sizeof(struct aecs_comp_table_record) + IAA_AECS_ALIGN;
> +       size_t d_size = sizeof(struct aecs_decomp_table_record) + IAA_AECS_ALIGN;
>         struct device *dev = &iaa_device->idxd->pdev->dev;
>         struct iaa_device_compression_mode *device_mode;
>         int ret = -ENOMEM;
> @@ -435,16 +528,29 @@ static int init_device_compression_mode(struct iaa_device *iaa_device,
>         if (!device_mode->name)
>                 goto free;
>  
> -       device_mode->aecs_comp_table = dma_alloc_coherent(dev, size,
> +       device_mode->aecs_comp_table = dma_alloc_coherent(dev, c_size,
>                                                           &device_mode->aecs_comp_table_dma_addr, GFP_KERNEL);
>         if (!device_mode->aecs_comp_table)
>                 goto free;
>  
> +       device_mode->aecs_decomp_table = dma_alloc_coherent(dev, d_size,
> +                                                           &device_mode->aecs_decomp_table_dma_addr, GFP_KERNEL);
> +       if (!device_mode->aecs_decomp_table)
> +               goto free;
> +
>         /* Add Huffman table to aecs */
>         memset(device_mode->aecs_comp_table, 0, sizeof(*device_mode->aecs_comp_table));
>         memcpy(device_mode->aecs_comp_table->ll_sym, mode->ll_table, mode->ll_table_size);
>         memcpy(device_mode->aecs_comp_table->d_sym, mode->d_table, mode->d_table_size);
>  
> +       if (mode->header_table) {
> +               ret = decompress_header(device_mode, mode, wq);
> +               if (ret) {
> +                       pr_debug("iaa header decompression failed: ret=%d\n", ret);
> +                       goto free;
> +               }
> +       }
> +
>         if (mode->init) {
>                 ret = mode->init(device_mode);
>                 if (ret)
> @@ -786,6 +892,13 @@ static int save_iaa_wq(struct idxd_wq *wq)
>                         goto out;
>                 }
>  
> +               idxd = new_device->idxd;
> +               /* Set IAA device src2 size based on AECS Format Version */
> +               if (idxd->hw.iaa_cap.dec_aecs_format_ver)
> +                       new_device->src2_size = DECOMP_INT_STATE_FMT2_LEN;
> +               else
> +                       new_device->src2_size = DECOMP_INT_STATE_FMT1_LEN;
> +
>                 ret = add_iaa_wq(new_device, wq, &new_wq);
>                 if (ret) {
>                         del_iaa_device(new_device);
> @@ -1006,6 +1119,24 @@ static int deflate_generic_decompress(struct acomp_req *req)
>         return ret;
>  }
>  
> +static int deflate_canned_decompress(struct acomp_req *req)
> +{
> +       int ret;
> +       struct acomp_req *sw_req;
> +

Reverse Christmas tree here too (I won't go through them all, please
just check this in any new/changed functions throughout).


Thanks,

Tom