Message ID | 20201026101005.2940615-1-armbru@redhat.com |
---|---|
Headers | show |
Series | qemu-storage-daemon: QAPIfy --chardev the stupid way | expand |
On 26/10/20 11:10, Markus Armbruster wrote: > Kevin's "[PATCH v2 0/6] qemu-storage-daemon: QAPIfy --chardev" > involves surgery to the QAPI generator. Some (most?) of it should go > away if we deprecate the "data" wrappers due to simple unions in QMP. > > Do we really need to mess with the code generator to solve the problem > at hand? > > > Let's recapitulate the problem: > > * We want to QAPIfy --chardev, i.e. define its argument as a QAPI > type. Considering that this is not 5.2 stuff at this point, I would like to suggest again moving chardevs to -object, and ask you to evaluate that option with the agreement that I do the work instead of you. :) Paolo
Paolo Bonzini <pbonzini@redhat.com> writes: > On 26/10/20 11:10, Markus Armbruster wrote: >> Kevin's "[PATCH v2 0/6] qemu-storage-daemon: QAPIfy --chardev" >> involves surgery to the QAPI generator. Some (most?) of it should go >> away if we deprecate the "data" wrappers due to simple unions in QMP. >> >> Do we really need to mess with the code generator to solve the problem >> at hand? >> >> >> Let's recapitulate the problem: >> >> * We want to QAPIfy --chardev, i.e. define its argument as a QAPI >> type. > > Considering that this is not 5.2 stuff at this point, I would like to > suggest again moving chardevs to -object, and ask you to evaluate that > option with the agreement that I do the work instead of you. :) Replacing -chardev with -object without regressing features would be lovely. One feature in particular: introspection. If we manage to fully QAPIfy -object, we should be good. I understand Eduardo is cutting a path through the jungle. I can offer assistance with bridging QAPI schema to QOM.
Eric Blake <eblake@redhat.com> writes: > On 10/26/20 5:10 AM, Markus Armbruster wrote: >> chardev-add's arguments use an annoying amount of nesting. Example: >> >> {"execute": "chardev-add", >> "arguments": { >> "id":"sock0", >> "backend": { >> "type": "socket", >> "data": { >> "addr": { >> "type": "inet", >> "data": { >> "host": "0.0.0.0", >> "port": "2445"}}}}}} >> >> This is because chardev-add predates QAPI features that enable flatter >> data structures, both on the wire and in C: base types, flat unions, >> commands taking a union or alternate as 'data'. >> >> The nesting would be even more annoying in dotted key syntax: >> >> id=sock0,\ >> backend.type=socket,\ >> backend.data.addr.type=inet,\ >> backend.data.addr.data.host=0.0.0.0,\ >> backend.data.addr.data.port=2445 >> >> Relevant, because the next commit will QAPIfy qemu-storage-daemon >> --chardev. We really want this instead: >> >> --chardev socket,id=sock0,\ >> addr.type=inet,\ >> addr.host=0.0.0.0,\ >> addr.port=2445 >> >> To get it, define a new QAPI type ChardevOptions that is the flat >> equivalent to chardev-add's arguments. >> >> What we should do now is convert the internal interfaces to take this >> new type, and limit the nested old type to the external interface, >> similar to what commit bd269ebc82 "sockets: Limit SocketAddressLegacy >> to external interfaces" did. But we're too close to the freeze to >> pull that off safely. >> >> What I can do now is convert the new type to the old nested type, and >> promise to replace this by what should be done in the next development >> cycle. > > Nice evaluation of the trade-off. > >> >> In more detail: >> >> * Flat union ChardevOptions corresponds to chardev-add's implicit >> arguments type. It flattens a struct containing a simple union into >> a flat union. >> >> * The flat union's discriminator is named @backend, not @type. This >> avoids clashing with member @type of ChardevSpiceChannel. For what >> it's worth, -chardev also uses this name. >> >> * Its branches @socket, @udp use ChardevSocketFlat, ChardevUdpFlat >> instead of ChardevSocket, ChardevUdp. This flattens simple union >> SocketAddressLegacy members to flat union SocketAddress members. >> >> * New chardev_options_crumple() converts ChardevOptions to >> chardev-add's implict arguments type. > > implicit Yes. >> >> Only one existing QAPI definition is affected: some of ChardevSocket's >> members get moved to a new base type ChardevSocketBase, to reduce >> duplication. No change to the generated C type and the wire format. >> >> Signed-off-by: Markus Armbruster <armbru@redhat.com> >> --- >> qapi/char.json | 106 ++++++++++++++++++++++++++++--- >> include/chardev/char.h | 5 ++ >> include/qemu/sockets.h | 3 + >> chardev/char-legacy.c | 140 +++++++++++++++++++++++++++++++++++++++++ >> chardev/char-socket.c | 3 +- >> util/qemu-sockets.c | 38 +++++++++++ >> chardev/meson.build | 1 + >> 7 files changed, 287 insertions(+), 9 deletions(-) >> create mode 100644 chardev/char-legacy.c > > Big but worth it. I'm liking the simplicity of this alternative over > Kevin's proposal, especially if we're aiming to get this in 5.2 soft freeze. Kevin and I decided not to. We both would've like to get a sane, QAPIfied --chardev for qemu-storage-daemon in 5.2, but neither of the proposed solutions inspires sufficient confidence to rush it in at this point. >> diff --git a/qapi/char.json b/qapi/char.json >> index 43486d1daa..31b693bbb2 100644 >> --- a/qapi/char.json >> +++ b/qapi/char.json >> @@ -244,12 +244,8 @@ >> 'base': 'ChardevCommon' } >> >> ## >> -# @ChardevSocket: >> +# @ChardevSocketBase: >> # >> -# Configuration info for (stream) socket chardevs. >> -# >> -# @addr: socket address to listen on (server=true) >> -# or connect to (server=false) >> # @tls-creds: the ID of the TLS credentials object (since 2.6) >> # @tls-authz: the ID of the QAuthZ authorization object against which >> # the client's x509 distinguished name will be validated. This >> @@ -274,9 +270,8 @@ >> # >> # Since: 1.4 >> ## >> -{ 'struct': 'ChardevSocket', >> - 'data': { 'addr': 'SocketAddressLegacy', >> - '*tls-creds': 'str', >> +{ 'struct': 'ChardevSocketBase', >> + 'data': { '*tls-creds': 'str', >> '*tls-authz' : 'str', >> '*server': 'bool', >> '*wait': 'bool', >> @@ -287,6 +282,35 @@ >> '*reconnect': 'int' }, >> 'base': 'ChardevCommon' } > > Here we are subdividing ChardevSocket into everything that is already > flat, and excluding the awkward 'addr'... > >> >> +## >> +# @ChardevSocket: >> +# >> +# Configuration info for (stream) socket chardevs. >> +# >> +# @addr: socket address to listen on (server=true) >> +# or connect to (server=false) >> +# >> +# Since: 1.4 >> +## >> +{ 'struct': 'ChardevSocket', >> + # Do not add to 'data', it breaks chardev_options_crumple()! Add to >> + # ChardevSocketBase's 'data' instead. >> + 'data': { 'addr': 'SocketAddressLegacy' }, >> + 'base': 'ChardevSocketBase' } > > ...legacy use pulls in the legacy 'addr'... > >> + >> +## >> +# @ChardevSocketFlat: >> +# >> +# Note: This type should eventually replace ChardevSocket. The >> +# difference between the two: ChardevSocketFlat uses >> +# SocketAddressLegacy, ChardevSocket uses SocketAddress. >> +## > > Missing a 'Since: 5.2' tag, if you want one. Yes. >> +{ 'struct': 'ChardevSocketFlat', >> + # Do not add to 'data', it breaks chardev_options_crumple()! Add to >> + # ChardevSocketBase's 'data' instead. >> + 'data': { 'addr': 'SocketAddress' }, >> + 'base': 'ChardevSocketBase' } >> + > > ...and this is the new type with a saner 'addr'. Works for me so far. Kevin dislikes the "Do not add to 'data'" part. It's needed because chardev_options_crumple() open-codes the conversion of all local members. Vague idea: conversion visitor. Similar to clone, except you can hook a conversion function into the cloning of certain members. Perhaps less ambitious: somehow make the build fail when you add local members without updating chardev_options_crumple(). Kevin also dislikes the schema duplication. His solution avoids it by generating both flat and nested from the same schema. I doubt it's worthwhile, because there ist just one QAPI type in need of it. A more widely applicable evolution of the idea might be more useful, but also still more complex. Kevin's generator-based approach has another advantage: it enables deprecation of the nested form. On the one hand, I'd love to get rid of it. On the other hand, it's just syntactical cleanup, and whether making the syntax neater and more consistent outweighs the pain of changing things seems doubtful. >> ## >> # @ChardevUdp: >> # >> @@ -298,10 +322,26 @@ >> # Since: 1.5 >> ## >> { 'struct': 'ChardevUdp', >> + # Do not add to 'data', it breaks chardev_options_crumple()! Create >> + # ChardevUdpBase instead, similar to ChardevSocketBase. >> 'data': { 'remote': 'SocketAddressLegacy', >> '*local': 'SocketAddressLegacy' }, >> 'base': 'ChardevCommon' } >> >> +## >> +# @ChardevUdpFlat: >> +# >> +# Note: This type should eventually replace ChardevUdp. The >> +# difference between the two: ChardevUdpFlat uses >> +# SocketAddressLegacy, ChardevUdp uses SocketAddress. >> +## > > Another missing 'Since: 5.2' Yes. >> +{ 'struct': 'ChardevUdpFlat', >> + # Do not add to 'data', it breaks chardev_options_crumple()! Create >> + # ChardevUdpBase instead, similar to ChardevSocketBase. >> + 'data': { 'remote': 'SocketAddress', >> + '*local': 'SocketAddress' }, >> + 'base': 'ChardevCommon' } >> + >> ## >> # @ChardevMux: >> # >> @@ -422,6 +462,56 @@ >> # next one is just for compatibility >> 'memory': 'ChardevRingbuf' } } >> >> +## >> +# @ChardevBackendType: >> +# >> +# Since: 5.2 >> +## >> +{ 'enum': 'ChardevBackendType', >> + >> + 'data': [ 'file', 'serial', 'parallel', 'pipe', 'socket', 'udp', >> + 'pty', 'null', 'mux', 'msmouse', 'wctablet', 'braille', >> + 'testdev', 'stdio', 'console', 'spicevmc', 'spiceport', >> + 'vc', 'ringbuf' ] } >> + >> +## >> +# @ChardevOptions: >> +# >> +# Note: This type should eventually replace the implicit arguments >> +# type of chardev-add and chardev-chardev. The differences >> +# between the two: 1. ChardevSocketOptions is a flat union >> +# rather than a struct with a simple union member, and 2. it >> +# uses SocketAddress instead of SocketAddressLegacy. This >> +# avoids nesting on the wire, i.e. we need fewer {}. >> +# >> +# Since: 5.2 >> +## >> +{ 'union': 'ChardevOptions', >> + 'base': { 'backend': 'ChardevBackendType', >> + 'id': 'str' }, >> + 'discriminator': 'backend', >> + 'data': { 'file': 'ChardevFile', >> + 'serial': 'ChardevHostdev', >> + 'parallel': 'ChardevHostdev', >> + 'pipe': 'ChardevHostdev', >> + 'socket': 'ChardevSocketFlat', >> + 'udp': 'ChardevUdpFlat', >> + 'pty': 'ChardevCommon', >> + 'null': 'ChardevCommon', >> + 'mux': 'ChardevMux', >> + 'msmouse': 'ChardevCommon', >> + 'wctablet': 'ChardevCommon', >> + 'braille': 'ChardevCommon', >> + 'testdev': 'ChardevCommon', >> + 'stdio': 'ChardevStdio', >> + 'console': 'ChardevCommon', >> + 'spicevmc': { 'type': 'ChardevSpiceChannel', >> + 'if': 'defined(CONFIG_SPICE)' }, >> + 'spiceport': { 'type': 'ChardevSpicePort', >> + 'if': 'defined(CONFIG_SPICE)' }, >> + 'vc': 'ChardevVC', >> + 'ringbuf': 'ChardevRingbuf' } } > > Looks good from the QAPI point of view. > > >> +/* >> + * TODO Convert internal interfaces to ChardevOptions, replace this >> + * function by one that flattens (const char *str, ChardevBackend >> + * *backend) -> ChardevOptions. >> + */ >> +q_obj_chardev_add_arg *chardev_options_crumple(ChardevOptions *chr) >> +{ >> + q_obj_chardev_add_arg *arg; >> + ChardevBackend *be; >> + >> + if (!chr) { >> + return NULL; >> + } >> + >> + arg = g_malloc(sizeof(*arg)); >> + arg->id = g_strdup(chr->id); >> + arg->backend = be = g_malloc(sizeof(*be)); >> + >> + switch (chr->backend) { >> + case CHARDEV_BACKEND_TYPE_FILE: >> + be->type = CHARDEV_BACKEND_KIND_FILE; >> + be->u.file.data = QAPI_CLONE(ChardevFile, &chr->u.file); >> + break; > > Most branches are straightforward,... > >> + case CHARDEV_BACKEND_TYPE_SOCKET: >> + be->type = CHARDEV_BACKEND_KIND_SOCKET; >> + /* >> + * Clone with SocketAddress crumpled to SocketAddressLegacy. >> + * All other members are in the base type. >> + */ >> + be->u.socket.data = g_memdup(&chr->u.socket, sizeof(chr->u.socket)); >> + QAPI_CLONE_MEMBERS(ChardevSocketBase, >> + qapi_ChardevSocket_base(be->u.socket.data), >> + qapi_ChardevSocketFlat_base(&chr->u.socket)); >> + be->u.socket.data->addr = socket_address_crumple(chr->u.socket.addr); >> + break; > > ...and this looks correct as well. > >> +++ b/chardev/char-socket.c >> @@ -1404,7 +1404,8 @@ static void qemu_chr_parse_socket(QemuOpts *opts, ChardevBackend *backend, >> >> backend->type = CHARDEV_BACKEND_KIND_SOCKET; >> sock = backend->u.socket.data = g_new0(ChardevSocket, 1); >> - qemu_chr_parse_common(opts, qapi_ChardevSocket_base(sock)); >> + qemu_chr_parse_common(opts, >> + qapi_ChardevSocketBase_base(qapi_ChardevSocket_base(sock))); > > The double function call (for a double cast) looks unusual, but I don't > see any shorter expression, so it is fine. > > Reviewed-by: Eric Blake <eblake@redhat.com> Thanks!
Am 28.10.2020 um 08:01 hat Markus Armbruster geschrieben: > Paolo Bonzini <pbonzini@redhat.com> writes: > > > On 26/10/20 11:10, Markus Armbruster wrote: > >> Kevin's "[PATCH v2 0/6] qemu-storage-daemon: QAPIfy --chardev" > >> involves surgery to the QAPI generator. Some (most?) of it should go > >> away if we deprecate the "data" wrappers due to simple unions in QMP. > >> > >> Do we really need to mess with the code generator to solve the problem > >> at hand? > >> > >> > >> Let's recapitulate the problem: > >> > >> * We want to QAPIfy --chardev, i.e. define its argument as a QAPI > >> type. > > > > Considering that this is not 5.2 stuff at this point, I would like to > > suggest again moving chardevs to -object, and ask you to evaluate that > > option with the agreement that I do the work instead of you. :) I don't think this is the right thing to do at this point. Making more use of QOM is an orthogonal problem and would only make solving this one harder. The problem we have and we're trying to solve is that we have chardev-add (which has a QAPI schema) and -chardev (which doesn't). We want to get an option that is described by the schema, doesn't duplicate things and is still convenient to use. Whether this option starts with -chardev or with -object doesn't really make much of a difference. The QAPI schema you need behind it will be almost or even exactly the same. > Replacing -chardev with -object without regressing features would be > lovely. One feature in particular: introspection. > > If we manage to fully QAPIfy -object, we should be good. I understand > Eduardo is cutting a path through the jungle. I don't expect many difficulties with the existing -object (famous last words). But if you make chardevs user creatable objects first, it becomes much harder because you just combined two problems (one of which is already known to be hard) into one large problem. > I can offer assistance with bridging QAPI schema to QOM. So, the steps that I would suggest are: 1a. Finish the QAPI schema for object-add 1b. Find some way to unify chardev-add's ChardevBackend and a CLI-friendly version of it in the schema 2. Generate QOM boilerplate code from ObjectOptions instead of duplicating it in the implementation 3. Convert chardevs to using QOM properites only now If we do 3. earlier, we'll write code that we want to replace later anyway. If we even do it in parallel with 1b. we'll additionally get merge conflicts. And if we do it before 1b. we'll do it without considering QAPI in detail and will lose any flexibility to change things in the new interface, which will make 1b. much harder. Kevin
On 28/10/20 12:46, Kevin Wolf wrote: > I don't think this is the right thing to do at this point. Making more > use of QOM is an orthogonal problem and would only make solving this one > harder. Making more use of QOM will make this a non-problem. You'll just use object-add and -object and, when you figure out the QOM schema, it will just work. Paolo > The problem we have and we're trying to solve is that we have > chardev-add (which has a QAPI schema) and -chardev (which doesn't). We > want to get an option that is described by the schema, doesn't duplicate > things and is still convenient to use. > > Whether this option starts with -chardev or with -object doesn't really > make much of a difference. The QAPI schema you need behind it will be > almost or even exactly the same.
Am 28.10.2020 um 15:39 hat Paolo Bonzini geschrieben: > On 28/10/20 12:46, Kevin Wolf wrote: > > I don't think this is the right thing to do at this point. Making more > > use of QOM is an orthogonal problem and would only make solving this one > > harder. > > Making more use of QOM will make this a non-problem. You'll just use > object-add and -object and, when you figure out the QOM schema, it will > just work. Yes, but figuring out the QOM schema (or rather, what the interface represented by the schema should look like) is the hard part. It is exactly the same hard part that we're currently trying to figure out without also worrying about QOM at the same time. -object and object-add instead of -chardev and chardev-add change the spelling, but solve none of these problems. Kevin
On 28/10/20 15:59, Kevin Wolf wrote: >> Making more use of QOM will make this a non-problem. You'll just use >> object-add and -object and, when you figure out the QOM schema, it will >> just work. > > Yes, but figuring out the QOM schema (or rather, what the interface > represented by the schema should look like) is the hard part. I don't disagree with that, but it's a problem you have to solve anyway, isn't it? Once you figure out how to introspect QOM classes, that would apply just as well to character devices. On the other hand, creating character devices with -object does solve another problem, namely the distinction between "early" and "late" objects in vl.c, in a way that QAPIfied -chardev doesn't solve. Paolo
Am 28.10.2020 um 16:09 hat Paolo Bonzini geschrieben: > On 28/10/20 15:59, Kevin Wolf wrote: > >> Making more use of QOM will make this a non-problem. You'll just use > >> object-add and -object and, when you figure out the QOM schema, it will > >> just work. > > > > Yes, but figuring out the QOM schema (or rather, what the interface > > represented by the schema should look like) is the hard part. > > I don't disagree with that, but it's a problem you have to solve anyway, > isn't it? Once you figure out how to introspect QOM classes, that would > apply just as well to character devices. Yes, it's the problem I tried to address with my series, and Markus with this alternative series. We need to do this either way. > On the other hand, creating character devices with -object does solve > another problem, namely the distinction between "early" and "late" > objects in vl.c, in a way that QAPIfied -chardev doesn't solve. Right. Both are solving different problems, and solving one won't automatically make the other a non-problem as you suggested above. I just suggested leaving QOM for later because two people making big changes on the same subsystem is going to be painful for at least one of them, and because for adding QOM properties, you need to know what these properties should look like (unless you want to change them again later). If you don't wait for the QAPI work, you'll have solved the problem of having two separate ways to describe chardev options by making it three separate ways. Technically this might fulfill the condition of not having two separate ways, but it's not really what we had in mind. :-) Kevin
On 28/10/20 16:39, Kevin Wolf wrote: >> I don't disagree with that, but it's a problem you have to solve anyway, >> isn't it? Once you figure out how to introspect QOM classes, that would >> apply just as well to character devices. > Yes, it's the problem I tried to address with my series, and Markus with > this alternative series. We need to do this either way. Right, I appreciate that QOMifying chardev would only be a solution if QOM introspection gets into 6.0. This is why I only brought it up because neither of these series will be ready in time for 5.2. So maybe QOMifying chardev wouldn't make it a non-problem; it would make it someone else's (Eduardo's) problem. > you need to know what these properties should look like True that. But I think the existing QAPI structures do help for that. > If you don't wait for the QAPI work, you'll have solved the problem of > having two separate ways to describe chardev options by making it three > separate ways. Technically this might fulfill the condition of not > having two separate ways, but it's not really what we had in mind. :-) Actually four ways (-chardev, chardev-add, -object, object-add) but two of them would be implemented by the same code and qsd would be able to standardize on them. Paolo