mbox series

[v3,00/47] qapi: static typing conversion, pt1

Message ID 20200925002900.465855-1-jsnow@redhat.com
Headers show
Series qapi: static typing conversion, pt1 | expand

Message

John Snow Sept. 25, 2020, 12:28 a.m. UTC 
Hi, this series adds static type hints to the QAPI module.
This is part one!

Part 1: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt1
Everything: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt6

- Requires Python 3.6+
- Requires mypy 0.770 or newer (for type analysis only)
- Requires pylint 2.6.0 or newer (for lint checking only)

In general, this series tackles the cleanup of one individual QAPI
module at a time. Once it passes pylint or mypy checks, those checks are
enabled for that file.

Type hints are added in patches that add *only* type hints and change no
other behavior. Any necessary changes to behavior to accommodate typing
are split out into their own tiny patches.

Notes:

- `make sphinxdocs` should work on every commit. It begins to include
  docstring content after the DO-NOT-MERGE patch 6.

- After patch 12, `isort -c` should pass 100% on this and every
  future commit.

- After patch 13, `flake8 qapi/` should pass 100% on this and every
  future commit.

- After patch 14, `pylint --rcfile=3Dqapi/pylintrc qapi/` should pass 100%
  on this and every future commit.

- After patch 23, `mypy --config-file=3Dqapi/mypy.ini qapi/` should pass
  100% on this and every future commit.

V3:
 - Use isort to enforce import consistency
 - Use sphinx apidoc to check docstring format
 - 16: Changed docstring formatting, changed import ordering.
 - 17: Changed import ordering.
 - 21: Changed formatting and added some comments for Sphinx.
 - 22: Changed import ordering.
 - 31: Fixed, thanks Eduardo.
 - 32: Shortened type annotation, thanks Cleber.
 - 39: Spiritually the same, but contextually extremely different.
 - 42: Very different, it now inlines the 'Extra' fields directly.
 - 43: Import ordering changes.
 - 45: Import ordering changes.
 - 47: Import ordering changes.

Status:

(This is my stgit summary with reviewer tags visible.)

+ [01] docs-replace-single-backtick   #
+ [02] docs-repair-broken-references  #
+ [03] docs-sphinx-change-default     #
+ [04] qapi-modify-docstrings-to-be   #
+ [05] qapi-doc-py-change-code        #
+ [06] docs-enable-sphinx-autodoc-for #
+ [07] qapi-gen-separate-arg-parsing  # [SOB] JS [RB] EH
+ [08] qapi-move-generator-entrypoint # [SOB] JS [RB] CR,EH [TB] CR
+ [09] do-not-merge-more-apidoc       #
+ [10] qapi-prefer-explicit-relative  # [SOB] JS [RB] CR,EH
+ [11] qapi-remove-wildcard-includes  # [SOB] JS [RB] CR,EH
+ [12] qapi-enforce-import-order      # [SOB] JS
+ [13] qapi-delint-using-flake8       # [SOB] JS [RB] CR,EH
+ [14] qapi-add-pylintrc              # [SOB] JS [TB] CR,EH [RB] CR
+ [15] qapi-common-py-remove-python   # [SOB] JS [RB] CR,EH
+ [16] qapi-common-add-indent-manager # [SOB] JS [RB] CR,EH
+ [17] qapi-common-py-delint-with     # [SOB] JS [RB] CR,EH
+ [18] replace-c-by-char              # [SOB] JS [RB] CR,EH
+ [19] qapi-common-py-check-with      # [SOB] JS [RB] CR [TB] CR,EH
+ [20] qapi-common-py-add-notational  # [SOB] JS [RB] CR,EH
+ [21] qapi-common-move-comments-into # [SOB] JS [RB] CR,EH
+ [22] qapi-split-build_params-into   # [SOB] JS [RB] CR,EH
+ [23] qapi-establish-mypy-type       # [SOB] JS [TB] EH
+ [24] qapi-events-py-add-notational  # [SOB] JS [RB] CR,EH
+ [25] qapi-events-move-comments-into # [SOB] JS [RB] CR,EH
+ [26] qapi-commands-py-don-t-re-bind # [SOB] JS [RB] CR,EH
+ [27] qapi-commands-py-add           # [SOB] JS [RB] CR,EH
+ [28] qapi-commands-py-enable        # [SOB] JS [RB] CR,EH
+ [29] qapi-source-py-add-notational  # [SOB] JS [RB] EH
+ [30] qapi-source-py-delint-with     # [SOB] JS [RB] CR,EH [TB] CR
+ [31] qapi-gen-py-fix-edge-case-of   #
+ [32] qapi-gen-py-add-notational     # [SOB] JS [RB] CR,EH
+ [33] qapi-gen-py-enable-checking    # [SOB] JS [RB] CR,EH [TB] CR
+ [34] qapi-gen-py-remove-unused      # [SOB] JS [RB] CR,EH
+ [35] qapi-gen-py-update-write-to-be # [SOB] JS [RB] CR,EH
+ [36] qapi-gen-py-delint-with-pylint # [SOB] JS [RB] CR,EH
+ [37] qapi-introspect-py-assert-obj  #
+ [38] qapi-introspect-py-create-a    # [SOB] EH,JS
+ [39] qapi-introspect-py-add         #
+ [40] qapi-introspect-py-unify       #
+ [41] qapi-introspect-py-replace     #
+ [42] qapi-introspect-py-create-a-0  #
+ [43] qapi-types-py-add-type-hint    # [SOB] JS [RB] CR,EH
+ [44] qapi-types-py-remove-one       # [SOB] JS [RB] CR,EH
+ [45] qapi-visit-py-assert           # [SOB] JS [RB] CR,EH
+ [46] qapi-visit-py-remove-unused    # [SOB] JS [RB] CR,EH [TB] CR
> [47] qapi-visit-py-add-notational   # [SOB] JS [RB] CR,EH [TB] CR


Changelog:

001/47:[down] '[DO-NOT-MERGE] docs: replace single backtick (`) with double-b=
acktick (``)'
002/47:[down] '[DO-NOT-MERGE] docs: repair broken references'
003/47:[down] '[DO-NOT-MERGE] docs/sphinx: change default role to "any"'
004/47:[down] 'qapi: modify docstrings to be sphinx-compatible'
005/47:[down] 'qapi/doc.py: Change code templates from function to string'
006/47:[down] '[DO-NOT-MERGE] docs: enable sphinx-autodoc for scripts/qapi'
009/47:[down] '[DO-NOT-MERGE] docs: add scripts/qapi/main to python manual'
012/47:[down] 'qapi: enforce import order/styling with isort'
016/47:[0012] [FC] 'qapi/common.py: Add indent manager'
017/47:[0003] [FC] 'qapi/common.py: delint with pylint'
021/47:[0006] [FC] 'qapi/common.py: Convert comments into docstrings, and ela=
borate'
022/47:[0020] [FC] 'qapi/common.py: move build_params into gen.py'
027/47:[down] 'qapi/commands.py: add type hint annotations'
031/47:[0002] [FC] 'qapi/gen.py: Fix edge-case of _is_user_module'
032/47:[0011] [FC] 'qapi/gen.py: add type hint annotations'
037/47:[down] 'qapi/introspect.py: assert obj is a dict when features are giv=
en'
038/47:[down] 'qapi/instrospect.py: add preliminary type hint annotations'
039/47:[0013] [FC] 'qapi/introspect.py: add _gen_features helper'
040/47:[down] 'qapi/introspect.py: Unify return type of _make_tree()'
041/47:[down] 'qapi/introspect.py: replace 'extra' dict with 'comment' argume=
nt'
042/47:[0062] [FC] 'qapi/introspect.py: create a typed 'Node' data structure'
043/47:[0004] [FC] 'qapi/types.py: add type hint annotations'
045/47:[0005] [FC] 'qapi/visit.py: assert tag_member contains a QAPISchemaEnu=
mType'
047/47:[0005] [FC] 'qapi/visit.py: add type hint annotations'

V2:
 - Removed Python3.6 patch in favor of Thomas Huth's
 - Addressed (most) feedback from Markus
 - Renamed type hint annotation commits

Eduardo Habkost (1):
  qapi/instrospect.py: add preliminary type hint annotations

John Snow (46):
  [DO-NOT-MERGE] docs: replace single backtick (`) with double-backtick
    (``)
  [DO-NOT-MERGE] docs: repair broken references
  [DO-NOT-MERGE] docs/sphinx: change default role to "any"
  qapi: modify docstrings to be sphinx-compatible
  qapi/doc.py: Change code templates from function to string
  [DO-NOT-MERGE] docs: enable sphinx-autodoc for scripts/qapi
  qapi-gen: Separate arg-parsing from generation
  qapi: move generator entrypoint into module
  [DO-NOT-MERGE] docs: add scripts/qapi/main to python manual
  qapi: Prefer explicit relative imports
  qapi: Remove wildcard includes
  qapi: enforce import order/styling with isort
  qapi: delint using flake8
  qapi: add pylintrc
  qapi/common.py: Remove python compatibility workaround
  qapi/common.py: Add indent manager
  qapi/common.py: delint with pylint
  qapi/common.py: Replace one-letter 'c' variable
  qapi/common.py: check with pylint
  qapi/common.py: add type hint annotations
  qapi/common.py: Convert comments into docstrings, and elaborate
  qapi/common.py: move build_params into gen.py
  qapi: establish mypy type-checking baseline
  qapi/events.py: add type hint annotations
  qapi/events.py: Move comments into docstrings
  qapi/commands.py: Don't re-bind to variable of different type
  qapi/commands.py: add type hint annotations
  qapi/commands.py: enable checking with mypy
  qapi/source.py: add type hint annotations
  qapi/source.py: delint with pylint
  qapi/gen.py: Fix edge-case of _is_user_module
  qapi/gen.py: add type hint annotations
  qapi/gen.py: Enable checking with mypy
  qapi/gen.py: Remove unused parameter
  qapi/gen.py: update write() to be more idiomatic
  qapi/gen.py: delint with pylint
  qapi/introspect.py: assert obj is a dict when features are given
  qapi/introspect.py: add _gen_features helper
  qapi/introspect.py: Unify return type of _make_tree()
  qapi/introspect.py: replace 'extra' dict with 'comment' argument
  qapi/introspect.py: create a typed 'Node' data structure
  qapi/types.py: add type hint annotations
  qapi/types.py: remove one-letter variables
  qapi/visit.py: assert tag_member contains a QAPISchemaEnumType
  qapi/visit.py: remove unused parameters from gen_visit_object
  qapi/visit.py: add type hint annotations

 docs/conf.py                           |   9 +-
 docs/devel/build-system.rst            | 118 +++++++-------
 docs/devel/index.rst                   |   1 +
 docs/devel/migration.rst               |  59 +++----
 docs/devel/multi-thread-tcg.rst        |   2 +-
 docs/devel/python/index.rst            |   7 +
 docs/devel/python/qapi.commands.rst    |   7 +
 docs/devel/python/qapi.common.rst      |   7 +
 docs/devel/python/qapi.doc.rst         |   7 +
 docs/devel/python/qapi.error.rst       |   7 +
 docs/devel/python/qapi.events.rst      |   7 +
 docs/devel/python/qapi.expr.rst        |   7 +
 docs/devel/python/qapi.gen.rst         |   7 +
 docs/devel/python/qapi.introspect.rst  |   7 +
 docs/devel/python/qapi.main.rst        |   7 +
 docs/devel/python/qapi.parser.rst      |   8 +
 docs/devel/python/qapi.rst             |  27 ++++
 docs/devel/python/qapi.schema.rst      |   7 +
 docs/devel/python/qapi.source.rst      |   7 +
 docs/devel/python/qapi.types.rst       |   7 +
 docs/devel/python/qapi.visit.rst       |   7 +
 docs/devel/tcg-plugins.rst             |  14 +-
 docs/devel/testing.rst                 |   4 +-
 docs/interop/live-block-operations.rst |   4 +-
 docs/system/arm/cpu-features.rst       | 110 ++++++-------
 docs/system/arm/nuvoton.rst            |   2 +-
 docs/system/s390x/protvirt.rst         |  10 +-
 scripts/qapi-gen.py                    |  59 ++-----
 scripts/qapi/.flake8                   |   2 +
 scripts/qapi/.isort.cfg                |   5 +
 scripts/qapi/commands.py               |  87 +++++++---
 scripts/qapi/common.py                 | 162 ++++++++++---------
 scripts/qapi/doc.py                    |  53 ++++---
 scripts/qapi/events.py                 |  56 +++++--
 scripts/qapi/expr.py                   |   6 +-
 scripts/qapi/gen.py                    | 184 +++++++++++++---------
 scripts/qapi/introspect.py             | 209 +++++++++++++++++--------
 scripts/qapi/main.py                   |  90 +++++++++++
 scripts/qapi/mypy.ini                  |  30 ++++
 scripts/qapi/parser.py                 |  15 +-
 scripts/qapi/pylintrc                  |  71 +++++++++
 scripts/qapi/schema.py                 |  35 +++--
 scripts/qapi/source.py                 |  34 ++--
 scripts/qapi/types.py                  | 124 ++++++++++-----
 scripts/qapi/visit.py                  | 116 ++++++++++----
 45 files changed, 1210 insertions(+), 594 deletions(-)
 create mode 100644 docs/devel/python/index.rst
 create mode 100644 docs/devel/python/qapi.commands.rst
 create mode 100644 docs/devel/python/qapi.common.rst
 create mode 100644 docs/devel/python/qapi.doc.rst
 create mode 100644 docs/devel/python/qapi.error.rst
 create mode 100644 docs/devel/python/qapi.events.rst
 create mode 100644 docs/devel/python/qapi.expr.rst
 create mode 100644 docs/devel/python/qapi.gen.rst
 create mode 100644 docs/devel/python/qapi.introspect.rst
 create mode 100644 docs/devel/python/qapi.main.rst
 create mode 100644 docs/devel/python/qapi.parser.rst
 create mode 100644 docs/devel/python/qapi.rst
 create mode 100644 docs/devel/python/qapi.schema.rst
 create mode 100644 docs/devel/python/qapi.source.rst
 create mode 100644 docs/devel/python/qapi.types.rst
 create mode 100644 docs/devel/python/qapi.visit.rst
 create mode 100644 scripts/qapi/.flake8
 create mode 100644 scripts/qapi/.isort.cfg
 create mode 100644 scripts/qapi/main.py
 create mode 100644 scripts/qapi/mypy.ini
 create mode 100644 scripts/qapi/pylintrc

--=20
2.26.2

Comments

John Snow Sept. 28, 2020, 3:17 p.m. UTC | #1 
On 9/24/20 8:28 PM, John Snow wrote:
> Hi, this series adds static type hints to the QAPI module.

> This is part one!

> 

> Part 1: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt1

> Everything: https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt6

> 


At this point, I have made only extremely minor edits to what has been 
reviewed so far, so I won't re-spin -- but I will announce that the most 
up-to-date version is rebased and pushed here:

https://gitlab.com/jsnow/qemu/-/tree/python-qapi-cleanup-pt1

Thanks,
--js