| Message ID | 20241211-b4-ovpn-v15-0-314e2cad0618@openvpn.net |
|---|---|
| Headers | show |
| Series | Introducing OpenVPN Data Channel Offload | expand |
2024-12-11, 22:15:10 +0100, Antonio Quartulli wrote: > +static struct ovpn_socket *ovpn_socket_get(struct socket *sock) > +{ > + struct ovpn_socket *ovpn_sock; > + > + rcu_read_lock(); > + ovpn_sock = rcu_dereference_sk_user_data(sock->sk); > + if (WARN_ON(!ovpn_socket_hold(ovpn_sock))) Could we hit this situation when we're removing the last peer (so detaching its socket) just as we're adding a new one? ovpn_socket_new finds the socket already attached and goes through the EALREADY path, but the refcount has already dropped to 0? Then we'd also return NULL from ovpn_socket_new [1], which I don't think is handled well by the caller (at least the netdev_dbg call at the end of ovpn_nl_peer_modify, maybe other spots too). (I guess it's not an issue you would see with the existing userspace if it's single-threaded) [...] > +struct ovpn_socket *ovpn_socket_new(struct socket *sock, struct ovpn_peer *peer) > +{ > + struct ovpn_socket *ovpn_sock; > + int ret; > + > + ret = ovpn_socket_attach(sock, peer); > + if (ret < 0 && ret != -EALREADY) > + return ERR_PTR(ret); > + > + /* if this socket is already owned by this interface, just increase the > + * refcounter and use it as expected. > + * > + * Since UDP sockets can be used to talk to multiple remote endpoints, > + * openvpn normally instantiates only one socket and shares it among all > + * its peers. For this reason, when we find out that a socket is already > + * used for some other peer in *this* instance, we can happily increase > + * its refcounter and use it normally. > + */ > + if (ret == -EALREADY) { > + /* caller is expected to increase the sock refcounter before > + * passing it to this function. For this reason we drop it if > + * not needed, like when this socket is already owned. > + */ > + ovpn_sock = ovpn_socket_get(sock); > + sockfd_put(sock); [1] so we would need to add if (!ovpn_sock) return -EAGAIN; > + return ovpn_sock; > + } > + [...] > +int ovpn_udp_socket_attach(struct socket *sock, struct ovpn_priv *ovpn) > +{ > + struct ovpn_socket *old_data; > + int ret = 0; > + > + /* make sure no pre-existing encapsulation handler exists */ > + rcu_read_lock(); > + old_data = rcu_dereference_sk_user_data(sock->sk); > + if (!old_data) { > + /* socket is currently unused - we can take it */ > + rcu_read_unlock(); > + return 0; > + } > + > + /* socket is in use. We need to understand if it's owned by this ovpn > + * instance or by something else. > + * In the former case, we can increase the refcounter and happily > + * use it, because the same UDP socket is expected to be shared among > + * different peers. > + * > + * Unlikely TCP, a single UDP socket can be used to talk to many remote (since I'm commenting on this patch:) s/Unlikely/Unlike/ [I have some more nits/typos here and there but I worry the maintainers will get "slightly" annoyed if I make you repost 22 patches once again :) -- if that's all I find in the next few days, everyone might be happier if I stash them and we get them fixed after merging?]
On 12/12/2024 17:19, Sabrina Dubroca wrote: > 2024-12-11, 22:15:10 +0100, Antonio Quartulli wrote: >> +static struct ovpn_socket *ovpn_socket_get(struct socket *sock) >> +{ >> + struct ovpn_socket *ovpn_sock; >> + >> + rcu_read_lock(); >> + ovpn_sock = rcu_dereference_sk_user_data(sock->sk); >> + if (WARN_ON(!ovpn_socket_hold(ovpn_sock))) > > Could we hit this situation when we're removing the last peer (so > detaching its socket) just as we're adding a new one? ovpn_socket_new > finds the socket already attached and goes through the EALREADY path, > but the refcount has already dropped to 0? > hm good point. > Then we'd also return NULL from ovpn_socket_new [1], which I don't > think is handled well by the caller (at least the netdev_dbg call at > the end of ovpn_nl_peer_modify, maybe other spots too). > > (I guess it's not an issue you would see with the existing userspace > if it's single-threaded) The TCP patch 11/22 will convert the socket release routine to a scheduled worker. This means we can have the following flow: 1) userspace deletes a peer -> peer drops its reference to the ovpn_socket 2) ovpn_socket refcnt may hit 0 -> cleanup/detach work is scheduled, but not yet executed 3) userspace adds a new peer -> attach returns -EALREADY but refcnt is 0 So not so impossible, even with a single-threaded userspace software. > > [...] >> +struct ovpn_socket *ovpn_socket_new(struct socket *sock, struct ovpn_peer *peer) >> +{ >> + struct ovpn_socket *ovpn_sock; >> + int ret; >> + >> + ret = ovpn_socket_attach(sock, peer); >> + if (ret < 0 && ret != -EALREADY) >> + return ERR_PTR(ret); >> + >> + /* if this socket is already owned by this interface, just increase the >> + * refcounter and use it as expected. >> + * >> + * Since UDP sockets can be used to talk to multiple remote endpoints, >> + * openvpn normally instantiates only one socket and shares it among all >> + * its peers. For this reason, when we find out that a socket is already >> + * used for some other peer in *this* instance, we can happily increase >> + * its refcounter and use it normally. >> + */ >> + if (ret == -EALREADY) { >> + /* caller is expected to increase the sock refcounter before >> + * passing it to this function. For this reason we drop it if >> + * not needed, like when this socket is already owned. >> + */ >> + ovpn_sock = ovpn_socket_get(sock); >> + sockfd_put(sock); > > [1] so we would need to add > > if (!ovpn_sock) > return -EAGAIN; I am not sure returning -EAGAIN is the right move at this point. We don't know when the scheduled worker will execute, so we don't know when to try again. Maybe we should call cancel_sync_work(&ovpn_sock->work) inside ovpn_socket_get()? So the latter will return NULL only when it is sure that the socket has been detached. At that point we can skip the following return and continue along the "new socket" path. What do you think? However, this makes we wonder: what happens if we have two racing PEER_NEW with the same non-yet-attached UDP socket? Maybe we should lock the socket in ovpn_udp_socket_attach() when checking its user-data and setting it (in order to make the test-and-set atomic)? I am specifically talking about this in udp.c: 345 /* make sure no pre-existing encapsulation handler exists */ 346 rcu_read_lock(); 347 old_data = rcu_dereference_sk_user_data(sock->sk); 348 if (!old_data) { 349 /* socket is currently unused - we can take it */ 350 rcu_read_unlock(); 351 setup_udp_tunnel_sock(sock_net(sock->sk), sock, &cfg); 352 return 0; 353 } We will end up returning 0 in both contexts and thus allocate two ovpn_sockets instead of re-using the first one we allocated. Does it make sense? > >> + return ovpn_sock; >> + } >> + > > [...] >> +int ovpn_udp_socket_attach(struct socket *sock, struct ovpn_priv *ovpn) >> +{ >> + struct ovpn_socket *old_data; >> + int ret = 0; >> + >> + /* make sure no pre-existing encapsulation handler exists */ >> + rcu_read_lock(); >> + old_data = rcu_dereference_sk_user_data(sock->sk); >> + if (!old_data) { >> + /* socket is currently unused - we can take it */ >> + rcu_read_unlock(); >> + return 0; >> + } >> + >> + /* socket is in use. We need to understand if it's owned by this ovpn >> + * instance or by something else. >> + * In the former case, we can increase the refcounter and happily >> + * use it, because the same UDP socket is expected to be shared among >> + * different peers. >> + * >> + * Unlikely TCP, a single UDP socket can be used to talk to many remote > > (since I'm commenting on this patch:) > > s/Unlikely/Unlike/ ACK > > [I have some more nits/typos here and there but I worry the > maintainers will get "slightly" annoyed if I make you repost 22 > patches once again :) -- if that's all I find in the next few days, > everyone might be happier if I stash them and we get them fixed after > merging?] If we have to rework this socket attaching part, it may be worth throwing in those typ0 fixes too :) Thanks a lot. Regards,
On 16/12/2024 12:50, Antonio Quartulli wrote: > On 16/12/2024 12:09, Sabrina Dubroca wrote: > [...] >>> Maybe we should call cancel_sync_work(&ovpn_sock->work) inside >>> ovpn_socket_get()? >>> So the latter will return NULL only when it is sure that the socket >>> has been >>> detached. >>> >>> At that point we can skip the following return and continue along the >>> "new >>> socket" path. >>> >>> What do you think? >> >> The work may not have been scheduled yet? (small window between the >> last kref_put and schedule_work) >> >> Maybe a completion [Documentation/scheduler/completion.rst] would >> solve it (but it makes things even more complex, unfortunately): >> >> - at the end of ovpn_socket_detach: complete(&ovpn_sock->detached); >> - in ovpn_socket_new when handling EALREADY: >> wait_for_completion(&ovpn_sock->detached); >> - in ovpn_socket_new for the new socket: init_completion(&ovpn_sock- >> >detached); >> >> but ovpn_sock could be gone immediately after complete(). Maybe >> something with completion_done() before the kfree_rcu in >> ovpn_socket_detach? I'm not that familiar with the completion API. >> > > It seems the solution we are aiming for is more complex than the concept > of ovpn_socket per se :-D > > I'll think a bit more about this..maybe we can avoid entering this > situation at all.. I see that there are some kref_put variants that acquire a lock just before hitting zero and running the release cb. If I implement a kref_put variant that acquires the lock_sock, I could then perform the udp detach under lock, thus ensuring that zero'ing the refcount and erasing the sk_user_data happens while holding the lock_sock. This way I should be able to prevent the situation where "sk_user_data still says EALREADY, but the refcnt is actually 0". I hope adding this new API is fine. I am giving it a try now. Regards,
Notable changes since v14: * added socket-netnsid netlink peer attribute containing the socket netns * avoided dereferencing ovpn_priv if NULL in ovpn_udp_encap_recv() err path (reported by smatch) * released ref to peer in ovpn_recv() err path * made sa_len signed in ovpn_peer_get_by_transp_addr_p2p() (reported by smatch) * avoided starting non-doxygen comments with /** in ovpn-cli.c tool * shortened peer timeout waiting time in kselftest script Please note that some patches were already reviewed by Andre Lunn, Donald Hunter and Shuah Khan. They have retained the Reviewed-by tag since no major code modification has happened since the review. The latest code can also be found at: https://github.com/OpenVPN/linux-kernel-ovpn Thanks a lot! Best Regards, Antonio Quartulli OpenVPN Inc. --- Antonio Quartulli (22): net: introduce OpenVPN Data Channel Offload (ovpn) ovpn: add basic netlink support ovpn: add basic interface creation/destruction/management routines ovpn: keep carrier always on for MP interfaces ovpn: introduce the ovpn_peer object ovpn: introduce the ovpn_socket object ovpn: implement basic TX path (UDP) ovpn: implement basic RX path (UDP) ovpn: implement packet processing ovpn: store tunnel and transport statistics ovpn: implement TCP transport ovpn: implement multi-peer support ovpn: implement peer lookup logic ovpn: implement keepalive mechanism ovpn: add support for updating local UDP endpoint ovpn: add support for peer floating ovpn: implement peer add/get/dump/delete via netlink ovpn: implement key add/get/del/swap via netlink ovpn: kill key and notify userspace in case of IV exhaustion ovpn: notify userspace when a peer is deleted ovpn: add basic ethtool support testing/selftests: add test tool and scripts for ovpn module Documentation/netlink/specs/ovpn.yaml | 372 +++ MAINTAINERS | 11 + drivers/net/Kconfig | 14 + drivers/net/Makefile | 1 + drivers/net/ovpn/Makefile | 22 + drivers/net/ovpn/bind.c | 55 + drivers/net/ovpn/bind.h | 101 + drivers/net/ovpn/crypto.c | 211 ++ drivers/net/ovpn/crypto.h | 145 ++ drivers/net/ovpn/crypto_aead.c | 383 ++++ drivers/net/ovpn/crypto_aead.h | 33 + drivers/net/ovpn/io.c | 446 ++++ drivers/net/ovpn/io.h | 34 + drivers/net/ovpn/main.c | 339 +++ drivers/net/ovpn/main.h | 14 + drivers/net/ovpn/netlink-gen.c | 213 ++ drivers/net/ovpn/netlink-gen.h | 41 + drivers/net/ovpn/netlink.c | 1189 ++++++++++ drivers/net/ovpn/netlink.h | 18 + drivers/net/ovpn/ovpnstruct.h | 57 + drivers/net/ovpn/peer.c | 1266 +++++++++++ drivers/net/ovpn/peer.h | 163 ++ drivers/net/ovpn/pktid.c | 129 ++ drivers/net/ovpn/pktid.h | 87 + drivers/net/ovpn/proto.h | 118 + drivers/net/ovpn/skb.h | 59 + drivers/net/ovpn/socket.c | 180 ++ drivers/net/ovpn/socket.h | 55 + drivers/net/ovpn/stats.c | 21 + drivers/net/ovpn/stats.h | 47 + drivers/net/ovpn/tcp.c | 578 +++++ drivers/net/ovpn/tcp.h | 33 + drivers/net/ovpn/udp.c | 397 ++++ drivers/net/ovpn/udp.h | 23 + include/uapi/linux/if_link.h | 15 + include/uapi/linux/ovpn.h | 111 + include/uapi/linux/udp.h | 1 + net/ipv6/af_inet6.c | 1 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/net/ovpn/.gitignore | 2 + tools/testing/selftests/net/ovpn/Makefile | 17 + tools/testing/selftests/net/ovpn/config | 10 + tools/testing/selftests/net/ovpn/data64.key | 5 + tools/testing/selftests/net/ovpn/ovpn-cli.c | 2366 ++++++++++++++++++++ tools/testing/selftests/net/ovpn/tcp_peers.txt | 5 + .../testing/selftests/net/ovpn/test-chachapoly.sh | 9 + tools/testing/selftests/net/ovpn/test-float.sh | 9 + tools/testing/selftests/net/ovpn/test-tcp.sh | 9 + tools/testing/selftests/net/ovpn/test.sh | 182 ++ tools/testing/selftests/net/ovpn/udp_peers.txt | 5 + 50 files changed, 9603 insertions(+) --- base-commit: 65fb414c93f486cef5408951350f20552113abd0 change-id: 20241002-b4-ovpn-eeee35c694a2 Best regards,