| Message ID | 20201014142957.763624-1-jsnow@redhat.com |
|---|---|
| Headers | show |
| Series | python: create installable package | expand |
On 10/19/20 6:02 AM, Daniel P. Berrangé wrote: > On Mon, Oct 19, 2020 at 11:45:09AM +0200, Andrea Bolognani wrote: >> On Wed, 2020-10-14 at 10:29 -0400, John Snow wrote: >>> Python infrastructure as it exists today is not capable reliably of >>> single-sourcing a package version from a parent directory. The authors >>> of pip are working to correct this, but as of today this is not possible >>> to my knowledge. >>> >>> The problem is that when using pip to build and install a python >>> package, it copies files over to a temporary directory and performs its >>> build there. This loses access to any information in the parent >>> directory, including git itself. >>> >>> Further, Python versions have a standard (PEP 440) that may or may not >>> follow QEMU's versioning. In general, it does; but naturally QEMU does >>> not follow PEP 440. To avoid any automatically-generated conflict, a >>> manual version file is preferred. >>> >>> >>> I am proposing: >>> >>> - Python core tooling synchronizes with the QEMU version directly >>> (5.2.0, 5.1.1, 5.3.0, etc.) >>> >>> - In the event that a Python package needs to be updated independently >>> of the QEMU version, a pre-release alpha version should be preferred, >>> but *only* after inclusion to the qemu development or stable branches. >>> >>> e.g. 5.2.0a1, 5.2.0a2, and so on should be preferred prior to 5.2.0's >>> release. >>> >>> - The Python core tooling makes absolutely no version compatibility >>> checks or constraints. It *may* work with releases of QEMU from the >>> past or future, but it is not required to. >>> >>> i.e., "qemu.core" will always remain in lock-step with QEMU. >>> >>> - We reserve the right to split out e.g. qemu.core.qmp to qemu.qmp >>> and begin indepedently versioning such a package separately from the >>> QEMU version it accompanies. >> >> I think this need to be considered very carefully. >> >> I'm not overly familiar with the Python ecosystem but it would appear >> that, despite PEP 440 not mandating this, many (most?) of the >> packages uploaded to PyPi are using semantic versioning. > > https://packaging.python.org/guides/distributing-packages-using-setuptools/#choosing-a-versioning-scheme > > Semver is the recommended approach, but they explicitly list date > based versioning as a valid alternative > > "Semantic versioning is not a suitable choice for all projects, > such as those with a regular time based release cadence and a > deprecation process that provides warnings for a number of > releases prior to removal of a feature." > > That paragraph describes QEMU's scenario. > > NB, historically we've made arbitrary changes to the python code > since it was not considered public API. If we make it official > public API, then we would actually need to start following our > deprecation process for the python code too. > I think our deprecation process is not tightly compatible with how Python programmers at-large expect packages to work. Semver is more or less the norm, despite the fact that it isn't explicitly required. setting requirements in requirements.txt, setup.[cfg|py], Pipfile, etc often hinge on a major version, e.g. qemu >= 5.0 qemu >= 3.0, < 6.0 would both be common forms of describing a requirement. Pinning specific versions is considered bad form, but in the context of releasing a package, I often see maintainers hedging their bets and preventing upgrades across a major version line. For that reason I am a little weary of adopting the deprecation policy as it exists in QEMU directly, and would propose a modification for my purposes here: - Features must be marked as deprecated - They must remain in a deprecated state for [at least] 2 releases - Deprecated features may not be removed until a major version change. In practice, this modification is a change from "2 releases" to "at least 2". However, I didn't intend to pay any mind to the deprecation policy "yet", as I have the package metadata listing the package status as "Alpha", see below: >> With that in mind, I think it would be unwise for qemu.* not to do >> the same; in particular, using a version number that's not <1.0.0 for >> a package that is very much in flux will almost certainly break >> people's expectations, and is also not something that you can easily >> take back at a later time. > > I don't think it is that big a deal, and there is clear benefit to > having the python code version match the QEMU version that it is > the companioon to. > Do you think it's fine if I start versioning at, say, "0.5.2", I could ignore a deprecation policy for now? I have a lot of changes I want to make and expect a lot of breaking changes very quickly. I just wanted to try -- somehow -- to conjure up a relationship to the QEMU package it's designed to work with. > Ultimately the versioning scheme just impacts on the version string > conditionals people list for their dependancies. Apps consuming QEMU > can handle any of the version schemes without much difference. > > Regards, > Daniel > Thanks for your input. This is the trickiest part of the process for me. I believe there is value in distributing these tools for other developers to help them prototype and experiment with new features, but realize it's a tightrope walk because we're flying dangerously close to providing a management utility that needs to care about cross-version compatibility and so on. I have no interest in providing stringent cross-version compatibility, but at the same time, libraries like QMP are not really at risk of changing all that much, actually. It will *probably* work for most QEMU versions. I have some further patches where I clean up the scripts in ./scripts/qmp and move them to ./python/qemu/qmp -- and that work is making me consider a rework to this series. I think that rework matches the spirit of an earlier suggestion of yours: - move ./python/qemu/core/qmp.py to ./python/qemu/qmp/qmp.py - (in a follow-up series) move ./scripts/qmp/* to ./python/qemu/qmp/* - rename ./python/qemu/core/ to ./python/qemu/machine (move accel.py, machine.py and qtest.py to a 'machine' pkg.) In effect, we'd then have: qemu.machine -- primarily a test interface for QEMU instances qemu.qmp -- fairly ageless QMP tools/lib for talking to QEMU and the different levels of support and "use at your own risk" become slightly more clear between the two packages. --js
On Mon, 2020-10-19 at 11:02 +0100, Daniel P. Berrangé wrote: > On Mon, Oct 19, 2020 at 11:45:09AM +0200, Andrea Bolognani wrote: > > I think this need to be considered very carefully. > > > > I'm not overly familiar with the Python ecosystem but it would appear > > that, despite PEP 440 not mandating this, many (most?) of the > > packages uploaded to PyPi are using semantic versioning. > > https://packaging.python.org/guides/distributing-packages-using-setuptools/#choosing-a-versioning-scheme > > Semver is the recommended approach, but they explicitly list date > based versioning as a valid alternative > > "Semantic versioning is not a suitable choice for all projects, > such as those with a regular time based release cadence and a > deprecation process that provides warnings for a number of > releases prior to removal of a feature." > > That paragraph describes QEMU's scenario. The section on date based versioning continues with A key advantage of date based versioning is that it is straightforward to tell how old the base feature set of a particular release is given just the version number. Version numbers for date based projects typically take the form of YEAR.MONTH (for example, 12.04, 15.10). The problem with QEMU's version numbers is that, while they are date based, they still *look* like semver, so it wouldn't be at all unreasonable for the user to expect that they also *behave* like semver. This is not much of a problem when it comes to the main binary, but it is certainly much more confusing when you start using the same version number for a Python library. > > With that in mind, I think it would be unwise for qemu.* not to do > > the same; in particular, using a version number that's not <1.0.0 for > > a package that is very much in flux will almost certainly break > > people's expectations, and is also not something that you can easily > > take back at a later time. > > I don't think it is that big a deal, and there is clear benefit to > having the python code version match the QEMU version that it is > the companioon to. > > Ultimately the versioning scheme just impacts on the version string > conditionals people list for their dependancies. Apps consuming QEMU > can handle any of the version schemes without much difference. The problem comes from the expectations: a Python programmer, who is used to semver due to its prominence on PyPi, when deciding whether to move from qemu.core 4.2.0 to 5.0.0 might expect to need code changes to cope with API-breaking changes - where in fact there are none, and at the same time might expect upgrading to 5.2.0 from 5.0.0 to be completely straightforward when in reality a feature their application depends on might have been removed after the usual deprecation period.
On Tue, Oct 20, 2020 at 10:52:14AM +0200, Andrea Bolognani wrote: > On Mon, 2020-10-19 at 11:02 +0100, Daniel P. Berrangé wrote: > > On Mon, Oct 19, 2020 at 11:45:09AM +0200, Andrea Bolognani wrote: > > > With that in mind, I think it would be unwise for qemu.* not to do > > > the same; in particular, using a version number that's not <1.0.0 for > > > a package that is very much in flux will almost certainly break > > > people's expectations, and is also not something that you can easily > > > take back at a later time. > > > > I don't think it is that big a deal, and there is clear benefit to > > having the python code version match the QEMU version that it is > > the companioon to. > > > > Ultimately the versioning scheme just impacts on the version string > > conditionals people list for their dependancies. Apps consuming QEMU > > can handle any of the version schemes without much difference. > > The problem comes from the expectations: a Python programmer, who is > used to semver due to its prominence on PyPi, when deciding whether > to move from qemu.core 4.2.0 to 5.0.0 might expect to need code > changes to cope with API-breaking changes - where in fact there are > none, and at the same time might expect upgrading to 5.2.0 from 5.0.0 > to be completely straightforward when in reality a feature their > application depends on might have been removed after the usual > deprecation period. The QEMU python modules are not like other python modules though, precisely because they are talking to QEMU. If we are shipping QEMU python releases on the same schedule as QEMU, then we can expect the normal ase to be updating both QEMU & Python together. So regardless of versioning in the python code, the QMP code they are talking to is liable to have removed deprecated features they are using. IMHO the upgrade issue is largely a problem of docs and testing, semver is no magic bullet. Regards, Daniel
On Tue, 2020-10-20 at 10:06 +0100, Daniel P. Berrangé wrote: > The QEMU python modules are not like other python modules though, > precisely because they are talking to QEMU. If we are shipping > QEMU python releases on the same schedule as QEMU, then we can > expect the normal ase to be updating both QEMU & Python together. Once you start uploading the Python packages to PyPi, you really have no way to ensure this will be the case. -- Andrea Bolognani / Red Hat / Virtualization
Based-on: https://gitlab.com/jsnow/qemu/-/tree/python This series factors the python/qemu directory as an installable module. It does not yet actually change the mechanics of how any other python source in the tree actually consumes it (yet), beyond the import path. The point of this series is primarily to formalize our dependencies on mypy, flake8, isort, and pylint alongside versions that are known to work. It also adds explicitly pinned versions of these dependencies that should behave in a repeatable and known way for developers and CI environments both. With the python tooling as a proper package, you can install this package in editable or production mode to a virtual environment, your local user environment, or your system packages. The primary benefit of this is to gain access to QMP tooling regardless of CWD, without needing to battle sys.path. For example: when developing, you may go to qemu/python/ and invoke `pipenv shell` to activate a virtual environment that contains the qemu packages. This package will always reflect the current version of the source files in the tree. When you are finished, you can simply exit the shell to remove these packages from your python environment. When not developing, you could install a version of this package to your environment outright to gain access to the QMP and QEMUMachine classes for lightweight scripting and testing by using pip: "pip install [--user] ." Finally, this package is formatted in such a way that it COULD be uploaded to https://pypi.org/project/qemu and installed independently of qemu.git with `pip install qemu`, but that button remains unpushed. TESTING THIS SERIES: CD to qemu/python first, and then: 1. Try "pipenv shell" to get a venv with the package installed to it in editable mode. Ctrl+d exits this venv shell. While in this shell, any python script that uses "from qemu.core import ..." should work correctly regardless of your CWD. 2. Try "pipenv sync --dev" to create/update the venv with the development packages without actually entering the venv. This should install isort, mypy, flake8 and pylint to the venv. 3. After the above sync, try "pipenv shell" again, and from the python project root, try any of the following: - pylint qemu - flake8 qemu - isort -c qemu - mypy qemu 4. Leave any venv you are in, and from the project root, try the following commands: - pipenv run pylint qemu - pipenv run flake8 qemu - pipenv run isort -c qemu - pipenv run mypy qemu John Snow (15): python: create qemu.core package python: add qemu package installer python: add VERSION file python: add directory structure README.rst files python: Add pipenv support python: add pylint exceptions to __init__.py python: move pylintrc into setup.cfg python: add pylint to pipenv python: move flake8 config to setup.cfg python: Add flake8 to pipenv python: move mypy.ini into setup.cfg python: add mypy to pipenv python: move .isort.cfg into setup.cfg python/qemu: add isort to pipenv python/qemu: add qemu package itself to pipenv python/PACKAGE.rst | 23 +++ python/README.rst | 27 +++ python/qemu/README.rst | 8 + python/qemu/core/README.rst | 9 + python/Pipfile | 16 ++ python/Pipfile.lock | 207 ++++++++++++++++++++++ python/VERSION | 1 + python/mypy.ini | 4 - python/qemu/.flake8 | 2 - python/qemu/.isort.cfg | 7 - python/qemu/__init__.py | 11 -- python/qemu/core/__init__.py | 47 +++++ python/qemu/{ => core}/accel.py | 0 python/qemu/{ => core}/console_socket.py | 0 python/qemu/{ => core}/machine.py | 0 python/qemu/{ => core}/qmp.py | 0 python/qemu/{ => core}/qtest.py | 0 python/{qemu/pylintrc => setup.cfg} | 66 +++---- python/setup.py | 23 +++ scripts/device-crash-test | 2 +- scripts/qmp/qemu-ga-client | 2 +- scripts/qmp/qmp | 2 +- scripts/qmp/qmp-shell | 2 +- scripts/qmp/qom-fuse | 2 +- scripts/qmp/qom-get | 2 +- scripts/qmp/qom-list | 2 +- scripts/qmp/qom-set | 2 +- scripts/qmp/qom-tree | 2 +- scripts/render_block_graph.py | 6 +- scripts/simplebench/bench_block_job.py | 4 +- tests/acceptance/avocado_qemu/__init__.py | 2 +- tests/acceptance/boot_linux.py | 3 +- tests/acceptance/virtio_check_params.py | 2 +- tests/acceptance/virtio_version.py | 2 +- tests/migration/guestperf/engine.py | 2 +- tests/qemu-iotests/235 | 2 +- tests/qemu-iotests/297 | 2 +- tests/qemu-iotests/300 | 4 +- tests/qemu-iotests/iotests.py | 4 +- tests/vm/basevm.py | 6 +- 40 files changed, 424 insertions(+), 84 deletions(-) create mode 100644 python/PACKAGE.rst create mode 100644 python/README.rst create mode 100644 python/qemu/README.rst create mode 100644 python/qemu/core/README.rst create mode 100644 python/Pipfile create mode 100644 python/Pipfile.lock create mode 100644 python/VERSION delete mode 100644 python/mypy.ini delete mode 100644 python/qemu/.flake8 delete mode 100644 python/qemu/.isort.cfg delete mode 100644 python/qemu/__init__.py create mode 100644 python/qemu/core/__init__.py rename python/qemu/{ => core}/accel.py (100%) rename python/qemu/{ => core}/console_socket.py (100%) rename python/qemu/{ => core}/machine.py (100%) rename python/qemu/{ => core}/qmp.py (100%) rename python/qemu/{ => core}/qtest.py (100%) rename python/{qemu/pylintrc => setup.cfg} (51%) mode change 100644 => 100755 create mode 100755 python/setup.py -- 2.26.2