| Message ID | 20201014142957.763624-4-jsnow@redhat.com |
|---|---|
| State | New |
| Headers | show |
| Series | python: create installable package | expand |
On 10/14/20 10:29 AM, John Snow wrote: > Python infrastructure as it exists today is not capable reliably of > single-sourcing a package version from a parent directory. The authors > of pip are working to correct this, but as of today this is not possible > to my knowledge. > > The problem is that when using pip to build and install a python > package, it copies files over to a temporary directory and performs its > build there. This loses access to any information in the parent > directory, including git itself. > See also: https://github.com/pypa/pip/issues/7549 https://github.com/pypa/pip/issues/7555 > Further, Python versions have a standard (PEP 440) that may or may not > follow QEMU's versioning. In general, it does; but naturally QEMU does > not follow PEP 440. To avoid any automatically-generated conflict, a > manual version file is preferred. > > > I am proposing: > > - Python core tooling synchronizes with the QEMU version directly > (5.2.0, 5.1.1, 5.3.0, etc.) > > - In the event that a Python package needs to be updated independently > of the QEMU version, a pre-release alpha version should be preferred, > but *only* after inclusion to the qemu development or stable branches. > > e.g. 5.2.0a1, 5.2.0a2, and so on should be preferred prior to 5.2.0's > release. > > - The Python core tooling makes absolutely no version compatibility > checks or constraints. It *may* work with releases of QEMU from the > past or future, but it is not required to. > > i.e., "qemu.core" will always remain in lock-step with QEMU. > > - We reserve the right to split out e.g. qemu.core.qmp to qemu.qmp > and begin indepedently versioning such a package separately from the > QEMU version it accompanies. > > > Implement this versioning scheme by adding a VERSION file and setting it > to 5.2.0a1. > > Signed-off-by: John Snow <jsnow@redhat.com> > --- > python/VERSION | 1 + > python/setup.cfg | 1 + > 2 files changed, 2 insertions(+) > create mode 100644 python/VERSION > > diff --git a/python/VERSION b/python/VERSION > new file mode 100644 > index 0000000000..2e81039c82 > --- /dev/null > +++ b/python/VERSION > @@ -0,0 +1 @@ > +5.2.0a1 > diff --git a/python/setup.cfg b/python/setup.cfg > index 12b99a796e..260f7f4e94 100755 > --- a/python/setup.cfg > +++ b/python/setup.cfg > @@ -1,5 +1,6 @@ > [metadata] > name = qemu > +version = file:VERSION > maintainer = QEMU Developer Team > maintainer_email = qemu-devel@nongnu.org > url = https://www.qemu.org/ >
On Wed, 2020-10-14 at 10:29 -0400, John Snow wrote: > Python infrastructure as it exists today is not capable reliably of > single-sourcing a package version from a parent directory. The authors > of pip are working to correct this, but as of today this is not possible > to my knowledge. > > The problem is that when using pip to build and install a python > package, it copies files over to a temporary directory and performs its > build there. This loses access to any information in the parent > directory, including git itself. > > Further, Python versions have a standard (PEP 440) that may or may not > follow QEMU's versioning. In general, it does; but naturally QEMU does > not follow PEP 440. To avoid any automatically-generated conflict, a > manual version file is preferred. > > > I am proposing: > > - Python core tooling synchronizes with the QEMU version directly > (5.2.0, 5.1.1, 5.3.0, etc.) > > - In the event that a Python package needs to be updated independently > of the QEMU version, a pre-release alpha version should be preferred, > but *only* after inclusion to the qemu development or stable branches. > > e.g. 5.2.0a1, 5.2.0a2, and so on should be preferred prior to 5.2.0's > release. > > - The Python core tooling makes absolutely no version compatibility > checks or constraints. It *may* work with releases of QEMU from the > past or future, but it is not required to. > > i.e., "qemu.core" will always remain in lock-step with QEMU. > > - We reserve the right to split out e.g. qemu.core.qmp to qemu.qmp > and begin indepedently versioning such a package separately from the > QEMU version it accompanies. I think this need to be considered very carefully. I'm not overly familiar with the Python ecosystem but it would appear that, despite PEP 440 not mandating this, many (most?) of the packages uploaded to PyPi are using semantic versioning. With that in mind, I think it would be unwise for qemu.* not to do the same; in particular, using a version number that's not <1.0.0 for a package that is very much in flux will almost certainly break people's expectations, and is also not something that you can easily take back at a later time. -- Andrea Bolognani / Red Hat / Virtualization
On Mon, Oct 19, 2020 at 11:45:09AM +0200, Andrea Bolognani wrote: > On Wed, 2020-10-14 at 10:29 -0400, John Snow wrote: > > Python infrastructure as it exists today is not capable reliably of > > single-sourcing a package version from a parent directory. The authors > > of pip are working to correct this, but as of today this is not possible > > to my knowledge. > > > > The problem is that when using pip to build and install a python > > package, it copies files over to a temporary directory and performs its > > build there. This loses access to any information in the parent > > directory, including git itself. > > > > Further, Python versions have a standard (PEP 440) that may or may not > > follow QEMU's versioning. In general, it does; but naturally QEMU does > > not follow PEP 440. To avoid any automatically-generated conflict, a > > manual version file is preferred. > > > > > > I am proposing: > > > > - Python core tooling synchronizes with the QEMU version directly > > (5.2.0, 5.1.1, 5.3.0, etc.) > > > > - In the event that a Python package needs to be updated independently > > of the QEMU version, a pre-release alpha version should be preferred, > > but *only* after inclusion to the qemu development or stable branches. > > > > e.g. 5.2.0a1, 5.2.0a2, and so on should be preferred prior to 5.2.0's > > release. > > > > - The Python core tooling makes absolutely no version compatibility > > checks or constraints. It *may* work with releases of QEMU from the > > past or future, but it is not required to. > > > > i.e., "qemu.core" will always remain in lock-step with QEMU. > > > > - We reserve the right to split out e.g. qemu.core.qmp to qemu.qmp > > and begin indepedently versioning such a package separately from the > > QEMU version it accompanies. > > I think this need to be considered very carefully. > > I'm not overly familiar with the Python ecosystem but it would appear > that, despite PEP 440 not mandating this, many (most?) of the > packages uploaded to PyPi are using semantic versioning. https://packaging.python.org/guides/distributing-packages-using-setuptools/#choosing-a-versioning-scheme Semver is the recommended approach, but they explicitly list date based versioning as a valid alternative "Semantic versioning is not a suitable choice for all projects, such as those with a regular time based release cadence and a deprecation process that provides warnings for a number of releases prior to removal of a feature." That paragraph describes QEMU's scenario. NB, historically we've made arbitrary changes to the python code since it was not considered public API. If we make it official public API, then we would actually need to start following our deprecation process for the python code too. > With that in mind, I think it would be unwise for qemu.* not to do > the same; in particular, using a version number that's not <1.0.0 for > a package that is very much in flux will almost certainly break > people's expectations, and is also not something that you can easily > take back at a later time. I don't think it is that big a deal, and there is clear benefit to having the python code version match the QEMU version that it is the companioon to. Ultimately the versioning scheme just impacts on the version string conditionals people list for their dependancies. Apps consuming QEMU can handle any of the version schemes without much difference. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On 10/19/20 6:02 AM, Daniel P. Berrangé wrote: > On Mon, Oct 19, 2020 at 11:45:09AM +0200, Andrea Bolognani wrote: >> On Wed, 2020-10-14 at 10:29 -0400, John Snow wrote: >>> Python infrastructure as it exists today is not capable reliably of >>> single-sourcing a package version from a parent directory. The authors >>> of pip are working to correct this, but as of today this is not possible >>> to my knowledge. >>> >>> The problem is that when using pip to build and install a python >>> package, it copies files over to a temporary directory and performs its >>> build there. This loses access to any information in the parent >>> directory, including git itself. >>> >>> Further, Python versions have a standard (PEP 440) that may or may not >>> follow QEMU's versioning. In general, it does; but naturally QEMU does >>> not follow PEP 440. To avoid any automatically-generated conflict, a >>> manual version file is preferred. >>> >>> >>> I am proposing: >>> >>> - Python core tooling synchronizes with the QEMU version directly >>> (5.2.0, 5.1.1, 5.3.0, etc.) >>> >>> - In the event that a Python package needs to be updated independently >>> of the QEMU version, a pre-release alpha version should be preferred, >>> but *only* after inclusion to the qemu development or stable branches. >>> >>> e.g. 5.2.0a1, 5.2.0a2, and so on should be preferred prior to 5.2.0's >>> release. >>> >>> - The Python core tooling makes absolutely no version compatibility >>> checks or constraints. It *may* work with releases of QEMU from the >>> past or future, but it is not required to. >>> >>> i.e., "qemu.core" will always remain in lock-step with QEMU. >>> >>> - We reserve the right to split out e.g. qemu.core.qmp to qemu.qmp >>> and begin indepedently versioning such a package separately from the >>> QEMU version it accompanies. >> >> I think this need to be considered very carefully. >> >> I'm not overly familiar with the Python ecosystem but it would appear >> that, despite PEP 440 not mandating this, many (most?) of the >> packages uploaded to PyPi are using semantic versioning. > > https://packaging.python.org/guides/distributing-packages-using-setuptools/#choosing-a-versioning-scheme > > Semver is the recommended approach, but they explicitly list date > based versioning as a valid alternative > > "Semantic versioning is not a suitable choice for all projects, > such as those with a regular time based release cadence and a > deprecation process that provides warnings for a number of > releases prior to removal of a feature." > > That paragraph describes QEMU's scenario. > > NB, historically we've made arbitrary changes to the python code > since it was not considered public API. If we make it official > public API, then we would actually need to start following our > deprecation process for the python code too. > I think our deprecation process is not tightly compatible with how Python programmers at-large expect packages to work. Semver is more or less the norm, despite the fact that it isn't explicitly required. setting requirements in requirements.txt, setup.[cfg|py], Pipfile, etc often hinge on a major version, e.g. qemu >= 5.0 qemu >= 3.0, < 6.0 would both be common forms of describing a requirement. Pinning specific versions is considered bad form, but in the context of releasing a package, I often see maintainers hedging their bets and preventing upgrades across a major version line. For that reason I am a little weary of adopting the deprecation policy as it exists in QEMU directly, and would propose a modification for my purposes here: - Features must be marked as deprecated - They must remain in a deprecated state for [at least] 2 releases - Deprecated features may not be removed until a major version change. In practice, this modification is a change from "2 releases" to "at least 2". However, I didn't intend to pay any mind to the deprecation policy "yet", as I have the package metadata listing the package status as "Alpha", see below: >> With that in mind, I think it would be unwise for qemu.* not to do >> the same; in particular, using a version number that's not <1.0.0 for >> a package that is very much in flux will almost certainly break >> people's expectations, and is also not something that you can easily >> take back at a later time. > > I don't think it is that big a deal, and there is clear benefit to > having the python code version match the QEMU version that it is > the companioon to. > Do you think it's fine if I start versioning at, say, "0.5.2", I could ignore a deprecation policy for now? I have a lot of changes I want to make and expect a lot of breaking changes very quickly. I just wanted to try -- somehow -- to conjure up a relationship to the QEMU package it's designed to work with. > Ultimately the versioning scheme just impacts on the version string > conditionals people list for their dependancies. Apps consuming QEMU > can handle any of the version schemes without much difference. > > Regards, > Daniel > Thanks for your input. This is the trickiest part of the process for me. I believe there is value in distributing these tools for other developers to help them prototype and experiment with new features, but realize it's a tightrope walk because we're flying dangerously close to providing a management utility that needs to care about cross-version compatibility and so on. I have no interest in providing stringent cross-version compatibility, but at the same time, libraries like QMP are not really at risk of changing all that much, actually. It will *probably* work for most QEMU versions. I have some further patches where I clean up the scripts in ./scripts/qmp and move them to ./python/qemu/qmp -- and that work is making me consider a rework to this series. I think that rework matches the spirit of an earlier suggestion of yours: - move ./python/qemu/core/qmp.py to ./python/qemu/qmp/qmp.py - (in a follow-up series) move ./scripts/qmp/* to ./python/qemu/qmp/* - rename ./python/qemu/core/ to ./python/qemu/machine (move accel.py, machine.py and qtest.py to a 'machine' pkg.) In effect, we'd then have: qemu.machine -- primarily a test interface for QEMU instances qemu.qmp -- fairly ageless QMP tools/lib for talking to QEMU and the different levels of support and "use at your own risk" become slightly more clear between the two packages. --js
On Mon, 2020-10-19 at 11:02 +0100, Daniel P. Berrangé wrote: > On Mon, Oct 19, 2020 at 11:45:09AM +0200, Andrea Bolognani wrote: > > I think this need to be considered very carefully. > > > > I'm not overly familiar with the Python ecosystem but it would appear > > that, despite PEP 440 not mandating this, many (most?) of the > > packages uploaded to PyPi are using semantic versioning. > > https://packaging.python.org/guides/distributing-packages-using-setuptools/#choosing-a-versioning-scheme > > Semver is the recommended approach, but they explicitly list date > based versioning as a valid alternative > > "Semantic versioning is not a suitable choice for all projects, > such as those with a regular time based release cadence and a > deprecation process that provides warnings for a number of > releases prior to removal of a feature." > > That paragraph describes QEMU's scenario. The section on date based versioning continues with A key advantage of date based versioning is that it is straightforward to tell how old the base feature set of a particular release is given just the version number. Version numbers for date based projects typically take the form of YEAR.MONTH (for example, 12.04, 15.10). The problem with QEMU's version numbers is that, while they are date based, they still *look* like semver, so it wouldn't be at all unreasonable for the user to expect that they also *behave* like semver. This is not much of a problem when it comes to the main binary, but it is certainly much more confusing when you start using the same version number for a Python library. > > With that in mind, I think it would be unwise for qemu.* not to do > > the same; in particular, using a version number that's not <1.0.0 for > > a package that is very much in flux will almost certainly break > > people's expectations, and is also not something that you can easily > > take back at a later time. > > I don't think it is that big a deal, and there is clear benefit to > having the python code version match the QEMU version that it is > the companioon to. > > Ultimately the versioning scheme just impacts on the version string > conditionals people list for their dependancies. Apps consuming QEMU > can handle any of the version schemes without much difference. The problem comes from the expectations: a Python programmer, who is used to semver due to its prominence on PyPi, when deciding whether to move from qemu.core 4.2.0 to 5.0.0 might expect to need code changes to cope with API-breaking changes - where in fact there are none, and at the same time might expect upgrading to 5.2.0 from 5.0.0 to be completely straightforward when in reality a feature their application depends on might have been removed after the usual deprecation period. -- Andrea Bolognani / Red Hat / Virtualization
diff --git a/python/VERSION b/python/VERSION new file mode 100644 index 0000000000..2e81039c82 --- /dev/null +++ b/python/VERSION @@ -0,0 +1 @@ +5.2.0a1 diff --git a/python/setup.cfg b/python/setup.cfg index 12b99a796e..260f7f4e94 100755 --- a/python/setup.cfg +++ b/python/setup.cfg @@ -1,5 +1,6 @@ [metadata] name = qemu +version = file:VERSION maintainer = QEMU Developer Team maintainer_email = qemu-devel@nongnu.org url = https://www.qemu.org/
Python infrastructure as it exists today is not capable reliably of single-sourcing a package version from a parent directory. The authors of pip are working to correct this, but as of today this is not possible to my knowledge. The problem is that when using pip to build and install a python package, it copies files over to a temporary directory and performs its build there. This loses access to any information in the parent directory, including git itself. Further, Python versions have a standard (PEP 440) that may or may not follow QEMU's versioning. In general, it does; but naturally QEMU does not follow PEP 440. To avoid any automatically-generated conflict, a manual version file is preferred. I am proposing: - Python core tooling synchronizes with the QEMU version directly (5.2.0, 5.1.1, 5.3.0, etc.) - In the event that a Python package needs to be updated independently of the QEMU version, a pre-release alpha version should be preferred, but *only* after inclusion to the qemu development or stable branches. e.g. 5.2.0a1, 5.2.0a2, and so on should be preferred prior to 5.2.0's release. - The Python core tooling makes absolutely no version compatibility checks or constraints. It *may* work with releases of QEMU from the past or future, but it is not required to. i.e., "qemu.core" will always remain in lock-step with QEMU. - We reserve the right to split out e.g. qemu.core.qmp to qemu.qmp and begin indepedently versioning such a package separately from the QEMU version it accompanies. Implement this versioning scheme by adding a VERSION file and setting it to 5.2.0a1. Signed-off-by: John Snow <jsnow@redhat.com> --- python/VERSION | 1 + python/setup.cfg | 1 + 2 files changed, 2 insertions(+) create mode 100644 python/VERSION