[net-next,1/4] netlink: truncate overlength attribute list in nla_nest_end()

If a nested list of attributes is too long, then the length will
exceed the 16-bit nla_len of the parent nlattr. In such cases,
determine how many whole attributes can fit and truncate the
message to this length. This properly maintains the nesting
hierarchy, keeping the entire message valid, while fitting more
subelements inside the nest range than may result if the length
is wrapped modulo 64KB.

Marking truncated attributes, such that user space can determine
the precise attribute truncated, by means of an additional bit in
the nla_type was considered and rejected. The NLA_F_NESTED and
NLA_F_NET_BYTEORDER flags are supposed to be mutually exclusive.
So, in theory, the latter bit could have been redefined for nested
attributes in order to indicate truncation, but user space tools
(most notably iproute2) cannot be relied on to honor NLA_TYPE_MASK,
resulting in alteration of the perceived nla_type and subsequent
catastrophic failure.

Failing the entire message with a hard error must also be rejected,
as this would break existing user space functionality. The trigger
issue is evident for IFLA_VFINFO_LIST and a hard error here would
cause iproute2 to fail to render an entire interface list even if
only a single interface warranted a truncated VF list. Instead, set
NLM_F_NEST_TRUNCATED in the netlink header to inform user space
about the incomplete data. In this particular case, however, user
space can better ascertain which instance is truncated by consulting
the associated IFLA_NUM_VF to determine how many VFs were expected.

Fixes: bfa83a9e03cf ("[NETLINK]: Type-safe netlink messages/attributes interface")
Signed-off-by: Edwin Peer <edwin.peer@broadcom.com>
---
 include/net/netlink.h        | 11 +++++++++--
 include/uapi/linux/netlink.h |  1 +
 lib/nlattr.c                 | 27 +++++++++++++++++++++++++++
 3 files changed, 37 insertions(+), 2 deletions(-)

On 1/22/21 9:53 PM, Edwin Peer wrote:
> If a nested list of attributes is too long, then the length will
> exceed the 16-bit nla_len of the parent nlattr. In such cases,
> determine how many whole attributes can fit and truncate the
> message to this length. This properly maintains the nesting
> hierarchy, keeping the entire message valid, while fitting more
> subelements inside the nest range than may result if the length
> is wrapped modulo 64KB.
> 
> Marking truncated attributes, such that user space can determine
> the precise attribute truncated, by means of an additional bit in
> the nla_type was considered and rejected. The NLA_F_NESTED and
> NLA_F_NET_BYTEORDER flags are supposed to be mutually exclusive.
> So, in theory, the latter bit could have been redefined for nested
> attributes in order to indicate truncation, but user space tools
> (most notably iproute2) cannot be relied on to honor NLA_TYPE_MASK,
> resulting in alteration of the perceived nla_type and subsequent
> catastrophic failure.
> 

Did you look at using NETLINK_CB / netlink_skb_parms to keep a running
length of nested attributes to avoid the need to trim?

On Sat, Jan 23, 2021 at 11:14 AM David Ahern <dsahern@gmail.com> wrote:

> > Marking truncated attributes, such that user space can determine
> > the precise attribute truncated, by means of an additional bit in
> > the nla_type was considered and rejected. The NLA_F_NESTED and
> > NLA_F_NET_BYTEORDER flags are supposed to be mutually exclusive.
> > So, in theory, the latter bit could have been redefined for nested
> > attributes in order to indicate truncation, but user space tools
> > (most notably iproute2) cannot be relied on to honor NLA_TYPE_MASK,
> > resulting in alteration of the perceived nla_type and subsequent
> > catastrophic failure.
>
> Did you look at using NETLINK_CB / netlink_skb_parms to keep a running
> length of nested attributes to avoid the need to trim?

I did not, but thinking about it now, I don't think that's necessarily
the way to go. We shouldn't be concerned about the cost of iterating
over the list and trimming the skb for what should be a rare exception
path. Ideally, we want to make sure at compile time (by having correct
code) that we don't ever exceed this limit at run time. Perhaps we
should investigate static analysis approaches to prove nla_len can't
be exceeded?

Tracking the outer nest length during nla_put() would provide for
convenient error indication at the precise location where things go
wrong, but that's a fair amount of housekeeping that isn't free of
complexity and run time cost either. Instead of rarely (if ever)
undoing work, we'll always do extra work that we hardly ever need.

Then, if nla_put() can detect nesting errors, there's the issue of
what to do in the case of errors. Case in point, the IFLA_VFINFO_LIST
scenario would now require explicit error handling in the generator
logic, because we can't fail hard at that point. We would need to be
sure we propagate all possible nesting errors up to a common location
(probably where the nest ends, which is where we're dealing with the
problem in this solution), set the truncated flag and carry on (for
the same net effect the trim in nla_nest_end() has). If there are
other cases we don't know about today, they might turn from soft fails
into hard errors, breaking existing user space. Truncating the list is
the only non-obtrusive solution to any existing brokenness that is
guaranteed to not make things worse, but we can't know where we need
to do that apriori and would need to explicitly handle each case as
they come up.

Hard errors on nest overflow can only reliably work for new code. That
is, assuming it is tested to the extremes when it goes in, not after
user space comes to rely on the broken behavior.

Regards,
Edwin Peer

On Sat, Jan 23, 2021 at 12:42 PM Edwin Peer <edwin.peer@broadcom.com> wrote:

> Then, if nla_put() can detect nesting errors, there's the issue of
> what to do in the case of errors. Case in point, the IFLA_VFINFO_LIST
> scenario would now require explicit error handling in the generator
> logic, because we can't fail hard at that point. We would need to be
> sure we propagate all possible nesting errors up to a common location
> (probably where the nest ends, which is where we're dealing with the
> problem in this solution), set the truncated flag and carry on (for
> the same net effect the trim in nla_nest_end() has).

Also, the unwind here turns out to be just as complicated, requiring a
traversal from the start and a trim anyway, because the attribute that
triggers the overflow may be deep inside the hierarchy. We can't
simply truncate at this point. We should remove whole elements at the
uppermost level, or risk having partially filled attribute trees
rather than missing attributes at the level of the exceeded nest -
which may be worse.

Regards,
Edwin Peer

On Fri, 22 Jan 2021 20:53:18 -0800 Edwin Peer wrote:
> If a nested list of attributes is too long, then the length will

> exceed the 16-bit nla_len of the parent nlattr. In such cases,

> determine how many whole attributes can fit and truncate the

> message to this length. This properly maintains the nesting

> hierarchy, keeping the entire message valid, while fitting more

> subelements inside the nest range than may result if the length

> is wrapped modulo 64KB.

> 

> Marking truncated attributes, such that user space can determine

> the precise attribute truncated, by means of an additional bit in

> the nla_type was considered and rejected. The NLA_F_NESTED and

> NLA_F_NET_BYTEORDER flags are supposed to be mutually exclusive.

> So, in theory, the latter bit could have been redefined for nested

> attributes in order to indicate truncation, but user space tools

> (most notably iproute2) cannot be relied on to honor NLA_TYPE_MASK,

> resulting in alteration of the perceived nla_type and subsequent

> catastrophic failure.

> 

> Failing the entire message with a hard error must also be rejected,

> as this would break existing user space functionality. The trigger

> issue is evident for IFLA_VFINFO_LIST and a hard error here would

> cause iproute2 to fail to render an entire interface list even if

> only a single interface warranted a truncated VF list. Instead, set

> NLM_F_NEST_TRUNCATED in the netlink header to inform user space

> about the incomplete data. In this particular case, however, user

> space can better ascertain which instance is truncated by consulting

> the associated IFLA_NUM_VF to determine how many VFs were expected.

> 

> Fixes: bfa83a9e03cf ("[NETLINK]: Type-safe netlink messages/attributes interface")

> Signed-off-by: Edwin Peer <edwin.peer@broadcom.com>



> diff --git a/include/net/netlink.h b/include/net/netlink.h

> index 1ceec518ab49..fc8c57dafb05 100644

> --- a/include/net/netlink.h

> +++ b/include/net/netlink.h

> @@ -1785,19 +1785,26 @@ static inline struct nlattr *nla_nest_start(struct sk_buff *skb, int attrtype)

>  	return nla_nest_start_noflag(skb, attrtype | NLA_F_NESTED);

>  }

>  

> +int __nla_nest_trunc_msg(struct sk_buff *skb, const struct nlattr *start);

> +

>  /**

>   * nla_nest_end - Finalize nesting of attributes

>   * @skb: socket buffer the attributes are stored in

>   * @start: container attribute

>   *

>   * Corrects the container attribute header to include the all

> - * appeneded attributes.

> + * appeneded attributes. The list of attributes will be truncated

> + * if too long to fit within the parent attribute's maximum reach.

>   *

>   * Returns the total data length of the skb.

>   */

>  static inline int nla_nest_end(struct sk_buff *skb, struct nlattr *start)


What are the semantics for multiple nests? All attrs down will always
have the trunc flag set? Because I'd have expected the skb_tail to be
trimmed in __nla_nest_trunc_msg()..

In fact if there is another nest around the "full" one, and the "full"
one is close to 0xffff wouldn't that end up cascading all the way down
to the outermost attr yielding an empty top level attr? I feel like we
cater to a specific use case here where that doesn't happen.

After initially being concerned about using up another flag, I warmed
up to the concept of NLM_F_NEST_TRUNCATED, but I think this is all
better driven by the writer. The writer knows the size and count of the
attrs because it sizes the skb. So can the writer not drive the
truncation? Add a "how much space do we have left" check in
rtnl_fill_vf() ?

This would also avoid the assumption that the contents of the nla are
other nlas.

>  {

> -	start->nla_len = skb_tail_pointer(skb) - (unsigned char *)start;

> +	int len = skb_tail_pointer(skb) - (unsigned char *)start;

> +

> +	if (len > 0xffff)

> +		len = __nla_nest_trunc_msg(skb, start);

> +	start->nla_len = len;

>  	return skb->len;


This function really needs to start returning an error on overflow.
Perhaps even with a WARN(). 

Could you please check how many callers we'd need to change to have
nla_nest_end() have the same semantics as nla_put() (i.e. 0 or -errno)?
On a quick scan I see that most cases only care about errors already.

>  }

>  

> diff --git a/include/uapi/linux/netlink.h b/include/uapi/linux/netlink.h

> index 3d94269bbfa8..44a250825c30 100644

> --- a/include/uapi/linux/netlink.h

> +++ b/include/uapi/linux/netlink.h

> @@ -57,6 +57,7 @@ struct nlmsghdr {

>  #define NLM_F_ECHO		0x08	/* Echo this request 		*/

>  #define NLM_F_DUMP_INTR		0x10	/* Dump was inconsistent due to sequence change */

>  #define NLM_F_DUMP_FILTERED	0x20	/* Dump was filtered as requested */

> +#define NLM_F_NEST_TRUNCATED	0x40	/* Message contains truncated nested attribute */

>  

>  /* Modifiers to GET request */

>  #define NLM_F_ROOT	0x100	/* specify tree	root	*/

> diff --git a/lib/nlattr.c b/lib/nlattr.c

> index 5b6116e81f9f..2a267c0d3e16 100644

> --- a/lib/nlattr.c

> +++ b/lib/nlattr.c

> @@ -1119,4 +1119,31 @@ int nla_append(struct sk_buff *skb, int attrlen, const void *data)

>  	return 0;

>  }

>  EXPORT_SYMBOL(nla_append);

> +

> +/**

> + * __nla_nest_trunc_msg - Truncate list of nested netlink attributes to max len

> + * @skb: socket buffer with tail pointer positioned after end of nested list

> + * @start: container attribute designating the beginning of the list

> + *

> + * Trims the skb to fit only the attributes which are within the range of the

> + * containing nest attribute. This is a helper for nla_nest_end, to prevent

> + * adding unduly to the length of what is an inline function. It is not

> + * intended to be called from anywhere else.

> + *

> + * Returns the truncated length of the enclosing nest attribute in accordance

> + * with the number of whole attributes that can fit.

> + */

> +int __nla_nest_trunc_msg(struct sk_buff *skb, const struct nlattr *start)

> +{

> +	struct nlattr *attr = nla_data(start);

> +	int rem = 0xffff - NLA_HDRLEN;

> +

> +	while (nla_ok(attr, rem))

> +		attr = nla_next(attr, &rem);

> +	nlmsg_trim(skb, attr);

> +	nlmsg_hdr(skb)->nlmsg_flags |= NLM_F_NEST_TRUNCATED;

> +	return (unsigned char *)attr - (unsigned char *)start;

> +}

> +EXPORT_SYMBOL(__nla_nest_trunc_msg);

> +

>  #endif

On 1/23/21 1:42 PM, Edwin Peer wrote:
> On Sat, Jan 23, 2021 at 11:14 AM David Ahern <dsahern@gmail.com> wrote:

> 

>>> Marking truncated attributes, such that user space can determine

>>> the precise attribute truncated, by means of an additional bit in

>>> the nla_type was considered and rejected. The NLA_F_NESTED and

>>> NLA_F_NET_BYTEORDER flags are supposed to be mutually exclusive.

>>> So, in theory, the latter bit could have been redefined for nested

>>> attributes in order to indicate truncation, but user space tools

>>> (most notably iproute2) cannot be relied on to honor NLA_TYPE_MASK,

>>> resulting in alteration of the perceived nla_type and subsequent

>>> catastrophic failure.

>>

>> Did you look at using NETLINK_CB / netlink_skb_parms to keep a running

>> length of nested attributes to avoid the need to trim?

> 

> I did not, but thinking about it now, I don't think that's necessarily

> the way to go. We shouldn't be concerned about the cost of iterating

> over the list and trimming the skb for what should be a rare exception

> path. Ideally, we want to make sure at compile time (by having correct

> code) that we don't ever exceed this limit at run time. Perhaps we

> should investigate static analysis approaches to prove nla_len can't

> be exceeded?


It's not a rare exception path if VF data is dumped on every ip link
dump request. I think you would be surprised how often random s/w does a
link dump.

As for the static analysis, the number of VFs is dynamic so impossible
to detect at compile time. Limiting the number of VFs to what can fit
would be a different kind of regression.

> 

> Tracking the outer nest length during nla_put() would provide for

> convenient error indication at the precise location where things go

> wrong, but that's a fair amount of housekeeping that isn't free of

> complexity and run time cost either. Instead of rarely (if ever)

> undoing work, we'll always do extra work that we hardly ever need.

> 

> Then, if nla_put() can detect nesting errors, there's the issue of

> what to do in the case of errors. Case in point, the IFLA_VFINFO_LIST

> scenario would now require explicit error handling in the generator

> logic, because we can't fail hard at that point. We would need to be

> sure we propagate all possible nesting errors up to a common location

> (probably where the nest ends, which is where we're dealing with the

> problem in this solution), set the truncated flag and carry on (for

> the same net effect the trim in nla_nest_end() has). If there are

> other cases we don't know about today, they might turn from soft fails

> into hard errors, breaking existing user space. Truncating the list is

> the only non-obtrusive solution to any existing brokenness that is

> guaranteed to not make things worse, but we can't know where we need

> to do that apriori and would need to explicitly handle each case as

> they come up.


Yes, tracking the errors is hard given the flow of netlink helpers, but
ultimately all of the space checks come down to:

	if (unlikely(skb_tailroom(skb) < nla_total_size(attrlen)))
		return -EMSGSIZE;

so surely there is a solution. There are only a few entry points to
track space available (open coded checks deserve their fate) which can
be used to track overflow, nesting and multiple layers of nesting to
then avoid the memcpy in __nla_put, __nla_put_64bit, and __nla_put_nohdr

> 

> Hard errors on nest overflow can only reliably work for new code. That

> is, assuming it is tested to the extremes when it goes in, not after

> user space comes to rely on the broken behavior.

> 

> Regards,

> Edwin Peer

>

On 1/23/21 2:03 PM, Edwin Peer wrote:
> On Sat, Jan 23, 2021 at 12:42 PM Edwin Peer <edwin.peer@broadcom.com> wrote:

> 

>> Then, if nla_put() can detect nesting errors, there's the issue of

>> what to do in the case of errors. Case in point, the IFLA_VFINFO_LIST

>> scenario would now require explicit error handling in the generator

>> logic, because we can't fail hard at that point. We would need to be

>> sure we propagate all possible nesting errors up to a common location

>> (probably where the nest ends, which is where we're dealing with the

>> problem in this solution), set the truncated flag and carry on (for

>> the same net effect the trim in nla_nest_end() has).

> 

> Also, the unwind here turns out to be just as complicated, requiring a

> traversal from the start and a trim anyway, because the attribute that

> triggers the overflow may be deep inside the hierarchy. We can't

> simply truncate at this point. We should remove whole elements at the

> uppermost level, or risk having partially filled attribute trees

> rather than missing attributes at the level of the exceeded nest -

> which may be worse.

> 


I'm not a fan of the skb trim idea. I think it would be better to figure
out how to stop adding to the skb when an attr length is going to exceed
64kB. Not failing hard with an error (ip link sh needs to succeed), but
truncating the specific attribute of a message with a flag so userspace
knows it is short.

On Mon, Jan 25, 2021 at 8:56 PM David Ahern <dsahern@gmail.com> wrote:

> I'm not a fan of the skb trim idea. I think it would be better to figure

> out how to stop adding to the skb when an attr length is going to exceed

> 64kB. Not failing hard with an error (ip link sh needs to succeed), but

> truncating the specific attribute of a message with a flag so userspace

> knows it is short.


Absent the ability to do something useful in terms of actually
avoiding the overflow [1], I'm abandoning this approach entirely. I
have a different idea that I will propose in due course.

[1] https://marc.info/?l=linux-netdev&m=161163943811663

Regards,
Edwin Peer