diff mbox

[v6,08/10] xen/arm: Add relinquish_p2m_mapping to remove reference on every mapped page

Message ID 1387297678-17762-9-git-send-email-julien.grall@linaro.org
State Accepted, archived
Headers show

Commit Message

Julien Grall Dec. 17, 2013, 4:27 p.m. UTC
This function will be called when the domain relinquishes its memory.
It removes refcount on every mapped page to a valid MFN.

Currently, Xen doesn't take reference on every new mapping but only for foreign
mapping. Restrict the function only on foreign mapping.

Signed-off-by: Julien Grall <julien.grall@linaro.org>

---
    Changes in v6:
        - Typoes
        - Rework preempt
        - Clean up if ( p2mt == ... || p2mt == ... )
    Changes in v4:
        - Use LPAE_ENTRIES instead of hardcoded value
    Changes in v3:
        - Rework title
        - Reuse create_p2m_entries to remove reference
        - Don't forget to set relmem!
        - Fix compilation (missing include)
    Changes in v2:
        - Introduce the patch
---
 xen/arch/arm/domain.c        |    8 +++++++
 xen/arch/arm/p2m.c           |   49 +++++++++++++++++++++++++++++++++++++++++-
 xen/include/asm-arm/domain.h |    1 +
 xen/include/asm-arm/p2m.h    |   15 +++++++++++++
 4 files changed, 72 insertions(+), 1 deletion(-)

Comments

Ian Campbell Dec. 17, 2013, 4:36 p.m. UTC | #1
On Tue, 2013-12-17 at 16:27 +0000, Julien Grall wrote:
> This function will be called when the domain relinquishes its memory.
> It removes refcount on every mapped page to a valid MFN.
> 
> Currently, Xen doesn't take reference on every new mapping but only for foreign
> mapping. Restrict the function only on foreign mapping.
> 
> Signed-off-by: Julien Grall <julien.grall@linaro.org>
> 
> ---
>     Changes in v6:
>         - Typoes
>         - Rework preempt
>         - Clean up if ( p2mt == ... || p2mt == ... )

You don't seem to have addressed the issue I pointed out about
create_p2m_entries exiting on any holes which it finds at the first or
second level.

I think you missed my second reply to 08/10 on v5 see:
<1387279904.27441.47.camel@kazak.uk.xensource.com>

Ian.
Julien Grall Dec. 17, 2013, 4:58 p.m. UTC | #2
On 12/17/2013 04:36 PM, Ian Campbell wrote:
> On Tue, 2013-12-17 at 16:27 +0000, Julien Grall wrote:
>> This function will be called when the domain relinquishes its memory.
>> It removes refcount on every mapped page to a valid MFN.
>>
>> Currently, Xen doesn't take reference on every new mapping but only for foreign
>> mapping. Restrict the function only on foreign mapping.
>>
>> Signed-off-by: Julien Grall <julien.grall@linaro.org>
>>
>> ---
>>     Changes in v6:
>>         - Typoes
>>         - Rework preempt
>>         - Clean up if ( p2mt == ... || p2mt == ... )
> 
> You don't seem to have addressed the issue I pointed out about
> create_p2m_entries exiting on any holes which it finds at the first or
> second level.
> 
> I think you missed my second reply to 08/10 on v5 see:
> <1387279904.27441.47.camel@kazak.uk.xensource.com>

Sorry I forgot to address your comment (see the copy of your email here):

> If this function finds any non-present first or second level PTE then it
> will stop and exit (goto out), meaning it will miss any mappings which
> are higher up after the hole.
>
> e.g. if you have a guest p2m with RAM at 0-2M and 4-6M then
> relinquishing 0-6M will only actually free 0-2M, then abort on 4-6M.
>
> Perhaps this could be fixed by making relinquish_p2m_mapping loop over
> the address space relinquishing 2M chunks as it goes? This would
remove
> the need for the if ( op == RELINQUISH && .. && prempt() ) stuff,
> because you could add the preempt in that loop.

I think you are wrong:
	- every first pte exists. We return NULL in case of the address is very
high.
        - if the second pte doesn't exist, we will create it (if it
fails we go out).

For the last item, I think it's a bit stupid to create table if we are
removing/relinquish mapping. But I think it's an improvement for later.
There are lots of improvement to do in this function (eg: flushing).
Ian Campbell Dec. 17, 2013, 5:02 p.m. UTC | #3
On Tue, 2013-12-17 at 16:58 +0000, Julien Grall wrote:

> > If this function finds any non-present first or second level PTE then it
> > will stop and exit (goto out), meaning it will miss any mappings which
> > are higher up after the hole.
> >
> > e.g. if you have a guest p2m with RAM at 0-2M and 4-6M then
> > relinquishing 0-6M will only actually free 0-2M, then abort on 4-6M.
> >
> > Perhaps this could be fixed by making relinquish_p2m_mapping loop over
> > the address space relinquishing 2M chunks as it goes? This would
> remove
> > the need for the if ( op == RELINQUISH && .. && prempt() ) stuff,
> > because you could add the preempt in that loop.
> 
> I think you are wrong:
> 	- every first pte exists. We return NULL in case of the address is very
> high.
>         - if the second pte doesn't exist, we will create it (if it
> fails we go out).

Oops, I misread the code, sorry. I think you are correct.

> For the last item, I think it's a bit stupid to create table if we are
> removing/relinquish mapping. But I think it's an improvement for later.
> There are lots of improvement to do in this function (eg: flushing).

Agreed.

Ian.
Ian Campbell Dec. 18, 2013, 1:21 p.m. UTC | #4
(the cc line here is a bit odd, why Ian J? But not Tim or Stefano? I've
added those two) On Tue, 2013-12-17 at 17:02 +0000, Ian Campbell wrote:
> > For the last item, I think it's a bit stupid to create table if we are
> > removing/relinquish mapping. But I think it's an improvement for later.
> > There are lots of improvement to do in this function (eg: flushing).
> 
> Agreed.

Actually, there is an efficiency concern here.

If we were to skip non-present first and second levels then we would
skip vast swathes of the address space very quickly. As it stands we
actively spend time filling it in just so we can recurse over it.

This effectively turns this back into a loop over the entire gfn space,
which is what we wanted to avoid.

The use of next_gfn_to_relinquish..max_mapped_gfn mitigates this
somewhat, but max_mapped_gfn is guest controlled and can trivially be
made huge by the guest.

How hard would it be to skip missing entries for remove and relinquish
walks? Should just be roughtly:

  int populate = (op == INSERT || op == ALLOCATE)

  for ( addr = ... ; addr < ... ; addr ... )
  {
    [...]

    if ( !first[...].valid )
    {
       if ( !populate ) {
	 addr += skip size of a first supermapping,
         continue;
       }
       rc = p2m_create(...)
       [..error..]

       [same for second etc]
    }
  }

"size of a first" needs care vs the += PAGE_SIZE in the for loop, I'd be
inclined to turn this into a while loop and move the += PAGE_SIZE to the
end.

Ian.
Julien Grall Dec. 18, 2013, 1:30 p.m. UTC | #5
On 12/18/2013 01:21 PM, Ian Campbell wrote:
> (the cc line here is a bit odd, why Ian J? But not Tim or Stefano? I've
> added those two) On Tue, 2013-12-17 at 17:02 +0000, Ian Campbell wrote:
>>> For the last item, I think it's a bit stupid to create table if we are
>>> removing/relinquish mapping. But I think it's an improvement for later.
>>> There are lots of improvement to do in this function (eg: flushing).
>>
>> Agreed.
>
> Actually, there is an efficiency concern here.
>
> If we were to skip non-present first and second levels then we would
> skip vast swathes of the address space very quickly. As it stands we
> actively spend time filling it in just so we can recurse over it.
>
> This effectively turns this back into a loop over the entire gfn space,
> which is what we wanted to avoid.
>
> The use of next_gfn_to_relinquish..max_mapped_gfn mitigates this
> somewhat, but max_mapped_gfn is guest controlled and can trivially be
> made huge by the guest.
>
> How hard would it be to skip missing entries for remove and relinquish
> walks? Should just be roughtly:
>
>    int populate = (op == INSERT || op == ALLOCATE)
>
>    for ( addr = ... ; addr < ... ; addr ... )
>    {
>      [...]
>
>      if ( !first[...].valid )
>      {
>         if ( !populate ) {
> 	 addr += skip size of a first supermapping,
>           continue;
>         }
>         rc = p2m_create(...)
>         [..error..]
>
>         [same for second etc]
>      }
>    }
>
> "size of a first" needs care vs the += PAGE_SIZE in the for loop, I'd be
> inclined to turn this into a while loop and move the += PAGE_SIZE to the
> end.

Your solution should be fine. I will add a patch in the series.
Julien Grall Dec. 18, 2013, 2:06 p.m. UTC | #6
On 12/18/2013 01:21 PM, Ian Campbell wrote:
> (the cc line here is a bit odd, why Ian J? But not Tim or Stefano? I've
> added those two) On Tue, 2013-12-17 at 17:02 +0000, Ian Campbell wrote:
>>> For the last item, I think it's a bit stupid to create table if we are
>>> removing/relinquish mapping. But I think it's an improvement for later.
>>> There are lots of improvement to do in this function (eg: flushing).
>>
>> Agreed.
>
> Actually, there is an efficiency concern here.
>
> If we were to skip non-present first and second levels then we would
> skip vast swathes of the address space very quickly. As it stands we
> actively spend time filling it in just so we can recurse over it.
>
> This effectively turns this back into a loop over the entire gfn space,
> which is what we wanted to avoid.
>
> The use of next_gfn_to_relinquish..max_mapped_gfn mitigates this
> somewhat, but max_mapped_gfn is guest controlled and can trivially be
> made huge by the guest.
>
> How hard would it be to skip missing entries for remove and relinquish
> walks? Should just be roughtly:
>
>    int populate = (op == INSERT || op == ALLOCATE)
>
>    for ( addr = ... ; addr < ... ; addr ... )
>    {
>      [...]
>
>      if ( !first[...].valid )
>      {
>         if ( !populate ) {
> 	 addr += skip size of a first supermapping,
>           continue;
>         }
>         rc = p2m_create(...)
>         [..error..]
>
>         [same for second etc]
>      }
>    }
>
> "size of a first" needs care vs the += PAGE_SIZE in the for loop, I'd be
> inclined to turn this into a while loop and move the += PAGE_SIZE to the
> end.
>
> Ian.
>

Ian, is it fine for you if I send a patch on top of this patch series? 
So we can go ahead with the current series.
Ian Campbell Dec. 18, 2013, 2:12 p.m. UTC | #7
On Wed, 2013-12-18 at 14:06 +0000, Julien Grall wrote:
> 
> On 12/18/2013 01:21 PM, Ian Campbell wrote:
> > (the cc line here is a bit odd, why Ian J? But not Tim or Stefano? I've
> > added those two) On Tue, 2013-12-17 at 17:02 +0000, Ian Campbell wrote:
> >>> For the last item, I think it's a bit stupid to create table if we are
> >>> removing/relinquish mapping. But I think it's an improvement for later.
> >>> There are lots of improvement to do in this function (eg: flushing).
> >>
> >> Agreed.
> >
> > Actually, there is an efficiency concern here.
> >
> > If we were to skip non-present first and second levels then we would
> > skip vast swathes of the address space very quickly. As it stands we
> > actively spend time filling it in just so we can recurse over it.
> >
> > This effectively turns this back into a loop over the entire gfn space,
> > which is what we wanted to avoid.
> >
> > The use of next_gfn_to_relinquish..max_mapped_gfn mitigates this
> > somewhat, but max_mapped_gfn is guest controlled and can trivially be
> > made huge by the guest.
> >
> > How hard would it be to skip missing entries for remove and relinquish
> > walks? Should just be roughtly:
> >
> >    int populate = (op == INSERT || op == ALLOCATE)
> >
> >    for ( addr = ... ; addr < ... ; addr ... )
> >    {
> >      [...]
> >
> >      if ( !first[...].valid )
> >      {
> >         if ( !populate ) {
> > 	 addr += skip size of a first supermapping,
> >           continue;
> >         }
> >         rc = p2m_create(...)
> >         [..error..]
> >
> >         [same for second etc]
> >      }
> >    }
> >
> > "size of a first" needs care vs the += PAGE_SIZE in the for loop, I'd be
> > inclined to turn this into a while loop and move the += PAGE_SIZE to the
> > end.
> >
> > Ian.
> >
> 
> Ian, is it fine for you if I send a patch on top of this patch series? 
> So we can go ahead with the current series.

Yeah, lets do that.

Ian.
Ian Campbell Dec. 18, 2013, 2:20 p.m. UTC | #8
On Tue, 2013-12-17 at 16:27 +0000, Julien Grall wrote:
> This function will be called when the domain relinquishes its memory.
> It removes refcount on every mapped page to a valid MFN.
> 
> Currently, Xen doesn't take reference on every new mapping but only for foreign
> mapping. Restrict the function only on foreign mapping.
> 
> Signed-off-by: Julien Grall <julien.grall@linaro.org>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

With the proviso that this will be made to properly skip non-present
entries, instead of allocating an empty page to walk, ASAP and certainly
before the release.

> 
> ---
>     Changes in v6:
>         - Typoes
>         - Rework preempt
>         - Clean up if ( p2mt == ... || p2mt == ... )
>     Changes in v4:
>         - Use LPAE_ENTRIES instead of hardcoded value
>     Changes in v3:
>         - Rework title
>         - Reuse create_p2m_entries to remove reference
>         - Don't forget to set relmem!
>         - Fix compilation (missing include)
>     Changes in v2:
>         - Introduce the patch
> ---
>  xen/arch/arm/domain.c        |    8 +++++++
>  xen/arch/arm/p2m.c           |   49 +++++++++++++++++++++++++++++++++++++++++-
>  xen/include/asm-arm/domain.h |    1 +
>  xen/include/asm-arm/p2m.h    |   15 +++++++++++++
>  4 files changed, 72 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index 1590708..4099e88 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -717,6 +717,14 @@ int domain_relinquish_resources(struct domain *d)
>          if ( ret )
>              return ret;
>  
> +        d->arch.relmem = RELMEM_mapping;
> +        /* Fallthrough */
> +
> +    case RELMEM_mapping:
> +        ret = relinquish_p2m_mapping(d);
> +        if ( ret )
> +            return ret;
> +
>          d->arch.relmem = RELMEM_done;
>          /* Fallthrough */
>  
> diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
> index 410acb6..3f4ab37 100644
> --- a/xen/arch/arm/p2m.c
> +++ b/xen/arch/arm/p2m.c
> @@ -6,6 +6,8 @@
>  #include <xen/bitops.h>
>  #include <asm/flushtlb.h>
>  #include <asm/gic.h>
> +#include <asm/event.h>
> +#include <asm/hardirq.h>
>  
>  /* First level P2M is 2 consecutive pages */
>  #define P2M_FIRST_ORDER 1
> @@ -224,7 +226,8 @@ static int p2m_create_table(struct domain *d,
>  enum p2m_operation {
>      INSERT,
>      ALLOCATE,
> -    REMOVE
> +    REMOVE,
> +    RELINQUISH,
>  };
>  
>  static int create_p2m_entries(struct domain *d,
> @@ -242,6 +245,7 @@ static int create_p2m_entries(struct domain *d,
>      unsigned long cur_first_page = ~0,
>                    cur_first_offset = ~0,
>                    cur_second_offset = ~0;
> +    unsigned long count = 0;
>  
>      spin_lock(&p2m->lock);
>  
> @@ -326,13 +330,19 @@ static int create_p2m_entries(struct domain *d,
>                      maddr += PAGE_SIZE;
>                  }
>                  break;
> +            case RELINQUISH:
>              case REMOVE:
>                  {
>                      lpae_t pte = third[third_table_offset(addr)];
>                      unsigned long mfn = pte.p2m.base;
>  
>                      if ( !pte.p2m.valid )
> +                    {
> +                        count++;
>                          break;
> +                    }
> +
> +                    count += 0x10;
>  
>                      /* TODO: Handle other p2m type */
>                      if ( p2m_is_foreign(pte.p2m.type) )
> @@ -343,12 +353,35 @@ static int create_p2m_entries(struct domain *d,
>  
>                      memset(&pte, 0x00, sizeof(pte));
>                      write_pte(&third[third_table_offset(addr)], pte);
> +                    count++;
>                  }
>                  break;
>          }
>  
>          if ( flush )
>              flush_tlb_all_local();
> +
> +        /* Preempt every 2MiB (mapped) or 32 MiB (unmapped) - arbitrary */
> +        if ( op == RELINQUISH && count >= 0x2000 )
> +        {
> +            if ( hypercall_preempt_check() )
> +            {
> +                p2m->next_gfn_to_relinquish = maddr >> PAGE_SHIFT;
> +                rc = -EAGAIN;
> +                goto out;
> +            }
> +            count = 0;
> +        }
> +    }
> +
> +    if ( op == ALLOCATE || op == INSERT )
> +    {
> +        unsigned long sgfn = paddr_to_pfn(start_gpaddr);
> +        unsigned long egfn = paddr_to_pfn(end_gpaddr);
> +
> +        p2m->max_mapped_gfn = MAX(p2m->max_mapped_gfn, egfn);
> +        /* Use next_gfn_to_relinquish to store the lowest gfn mapped */
> +        p2m->next_gfn_to_relinquish = MIN(p2m->next_gfn_to_relinquish, sgfn);
>      }
>  
>      rc = 0;
> @@ -534,12 +567,26 @@ int p2m_init(struct domain *d)
>  
>      p2m->first_level = NULL;
>  
> +    p2m->max_mapped_gfn = 0;
> +    p2m->next_gfn_to_relinquish = ULONG_MAX;
> +
>  err:
>      spin_unlock(&p2m->lock);
>  
>      return rc;
>  }
>  
> +int relinquish_p2m_mapping(struct domain *d)
> +{
> +    struct p2m_domain *p2m = &d->arch.p2m;
> +
> +    return create_p2m_entries(d, RELINQUISH,
> +                              pfn_to_paddr(p2m->next_gfn_to_relinquish),
> +                              pfn_to_paddr(p2m->max_mapped_gfn),
> +                              pfn_to_paddr(INVALID_MFN),
> +                              MATTR_MEM, p2m_invalid);
> +}
> +
>  unsigned long gmfn_to_mfn(struct domain *d, unsigned long gpfn)
>  {
>      paddr_t p = p2m_lookup(d, pfn_to_paddr(gpfn), NULL);
> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> index 53c9895..28d39a0 100644
> --- a/xen/include/asm-arm/domain.h
> +++ b/xen/include/asm-arm/domain.h
> @@ -112,6 +112,7 @@ struct arch_domain
>          RELMEM_not_started,
>          RELMEM_xen,
>          RELMEM_page,
> +        RELMEM_mapping,
>          RELMEM_done,
>      } relmem;
>  
> diff --git a/xen/include/asm-arm/p2m.h b/xen/include/asm-arm/p2m.h
> index 5ccfa7f..8b7b6d0 100644
> --- a/xen/include/asm-arm/p2m.h
> +++ b/xen/include/asm-arm/p2m.h
> @@ -18,6 +18,15 @@ struct p2m_domain {
>  
>      /* Current VMID in use */
>      uint8_t vmid;
> +
> +    /* Highest guest frame that's ever been mapped in the p2m
> +     * Only takes into account ram and foreign mapping
> +     */
> +    unsigned long max_mapped_gfn;
> +
> +    /* When releasing mapped gfn's in a preemptible manner, recall where
> +     * to resume the search */
> +    unsigned long next_gfn_to_relinquish;
>  };
>  
>  /* List of possible type for each page in the p2m entry.
> @@ -48,6 +57,12 @@ int p2m_init(struct domain *d);
>  /* Return all the p2m resources to Xen. */
>  void p2m_teardown(struct domain *d);
>  
> +/* Remove mapping refcount on each mapping page in the p2m
> + *
> + * TODO: For the moment only foreign mappings are handled
> + */
> +int relinquish_p2m_mapping(struct domain *d);
> +
>  /* Allocate a new p2m table for a domain.
>   *
>   * Returns 0 for success or -errno.
Julien Grall Dec. 18, 2013, 4:48 p.m. UTC | #9
On 12/17/2013 04:27 PM, Julien Grall wrote:
> @@ -343,12 +353,35 @@ static int create_p2m_entries(struct domain *d,
>  
>                      memset(&pte, 0x00, sizeof(pte));
>                      write_pte(&third[third_table_offset(addr)], pte);
> +                    count++;
>                  }
>                  break;
>          }
>  
>          if ( flush )
>              flush_tlb_all_local();
> +
> +        /* Preempt every 2MiB (mapped) or 32 MiB (unmapped) - arbitrary */
> +        if ( op == RELINQUISH && count >= 0x2000 )
> +        {
> +            if ( hypercall_preempt_check() )
> +            {
> +                p2m->next_gfn_to_relinquish = maddr >> PAGE_SHIFT;

Oops, I made a mistake in this patch.

It should be addr instead of maddr.

As it's already in staging, I will send a fix right now.
Ian Campbell Dec. 18, 2013, 4:50 p.m. UTC | #10
On Wed, 2013-12-18 at 16:48 +0000, Julien Grall wrote:
> On 12/17/2013 04:27 PM, Julien Grall wrote:
> > @@ -343,12 +353,35 @@ static int create_p2m_entries(struct domain *d,
> >  
> >                      memset(&pte, 0x00, sizeof(pte));
> >                      write_pte(&third[third_table_offset(addr)], pte);
> > +                    count++;
> >                  }
> >                  break;
> >          }
> >  
> >          if ( flush )
> >              flush_tlb_all_local();
> > +
> > +        /* Preempt every 2MiB (mapped) or 32 MiB (unmapped) - arbitrary */
> > +        if ( op == RELINQUISH && count >= 0x2000 )
> > +        {
> > +            if ( hypercall_preempt_check() )
> > +            {
> > +                p2m->next_gfn_to_relinquish = maddr >> PAGE_SHIFT;
> 
> Oops, I made a mistake in this patch.
> 
> It should be addr instead of maddr.
> 
> As it's already in staging, I will send a fix right now.

Yes please!
diff mbox

Patch

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 1590708..4099e88 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -717,6 +717,14 @@  int domain_relinquish_resources(struct domain *d)
         if ( ret )
             return ret;
 
+        d->arch.relmem = RELMEM_mapping;
+        /* Fallthrough */
+
+    case RELMEM_mapping:
+        ret = relinquish_p2m_mapping(d);
+        if ( ret )
+            return ret;
+
         d->arch.relmem = RELMEM_done;
         /* Fallthrough */
 
diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index 410acb6..3f4ab37 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -6,6 +6,8 @@ 
 #include <xen/bitops.h>
 #include <asm/flushtlb.h>
 #include <asm/gic.h>
+#include <asm/event.h>
+#include <asm/hardirq.h>
 
 /* First level P2M is 2 consecutive pages */
 #define P2M_FIRST_ORDER 1
@@ -224,7 +226,8 @@  static int p2m_create_table(struct domain *d,
 enum p2m_operation {
     INSERT,
     ALLOCATE,
-    REMOVE
+    REMOVE,
+    RELINQUISH,
 };
 
 static int create_p2m_entries(struct domain *d,
@@ -242,6 +245,7 @@  static int create_p2m_entries(struct domain *d,
     unsigned long cur_first_page = ~0,
                   cur_first_offset = ~0,
                   cur_second_offset = ~0;
+    unsigned long count = 0;
 
     spin_lock(&p2m->lock);
 
@@ -326,13 +330,19 @@  static int create_p2m_entries(struct domain *d,
                     maddr += PAGE_SIZE;
                 }
                 break;
+            case RELINQUISH:
             case REMOVE:
                 {
                     lpae_t pte = third[third_table_offset(addr)];
                     unsigned long mfn = pte.p2m.base;
 
                     if ( !pte.p2m.valid )
+                    {
+                        count++;
                         break;
+                    }
+
+                    count += 0x10;
 
                     /* TODO: Handle other p2m type */
                     if ( p2m_is_foreign(pte.p2m.type) )
@@ -343,12 +353,35 @@  static int create_p2m_entries(struct domain *d,
 
                     memset(&pte, 0x00, sizeof(pte));
                     write_pte(&third[third_table_offset(addr)], pte);
+                    count++;
                 }
                 break;
         }
 
         if ( flush )
             flush_tlb_all_local();
+
+        /* Preempt every 2MiB (mapped) or 32 MiB (unmapped) - arbitrary */
+        if ( op == RELINQUISH && count >= 0x2000 )
+        {
+            if ( hypercall_preempt_check() )
+            {
+                p2m->next_gfn_to_relinquish = maddr >> PAGE_SHIFT;
+                rc = -EAGAIN;
+                goto out;
+            }
+            count = 0;
+        }
+    }
+
+    if ( op == ALLOCATE || op == INSERT )
+    {
+        unsigned long sgfn = paddr_to_pfn(start_gpaddr);
+        unsigned long egfn = paddr_to_pfn(end_gpaddr);
+
+        p2m->max_mapped_gfn = MAX(p2m->max_mapped_gfn, egfn);
+        /* Use next_gfn_to_relinquish to store the lowest gfn mapped */
+        p2m->next_gfn_to_relinquish = MIN(p2m->next_gfn_to_relinquish, sgfn);
     }
 
     rc = 0;
@@ -534,12 +567,26 @@  int p2m_init(struct domain *d)
 
     p2m->first_level = NULL;
 
+    p2m->max_mapped_gfn = 0;
+    p2m->next_gfn_to_relinquish = ULONG_MAX;
+
 err:
     spin_unlock(&p2m->lock);
 
     return rc;
 }
 
+int relinquish_p2m_mapping(struct domain *d)
+{
+    struct p2m_domain *p2m = &d->arch.p2m;
+
+    return create_p2m_entries(d, RELINQUISH,
+                              pfn_to_paddr(p2m->next_gfn_to_relinquish),
+                              pfn_to_paddr(p2m->max_mapped_gfn),
+                              pfn_to_paddr(INVALID_MFN),
+                              MATTR_MEM, p2m_invalid);
+}
+
 unsigned long gmfn_to_mfn(struct domain *d, unsigned long gpfn)
 {
     paddr_t p = p2m_lookup(d, pfn_to_paddr(gpfn), NULL);
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 53c9895..28d39a0 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -112,6 +112,7 @@  struct arch_domain
         RELMEM_not_started,
         RELMEM_xen,
         RELMEM_page,
+        RELMEM_mapping,
         RELMEM_done,
     } relmem;
 
diff --git a/xen/include/asm-arm/p2m.h b/xen/include/asm-arm/p2m.h
index 5ccfa7f..8b7b6d0 100644
--- a/xen/include/asm-arm/p2m.h
+++ b/xen/include/asm-arm/p2m.h
@@ -18,6 +18,15 @@  struct p2m_domain {
 
     /* Current VMID in use */
     uint8_t vmid;
+
+    /* Highest guest frame that's ever been mapped in the p2m
+     * Only takes into account ram and foreign mapping
+     */
+    unsigned long max_mapped_gfn;
+
+    /* When releasing mapped gfn's in a preemptible manner, recall where
+     * to resume the search */
+    unsigned long next_gfn_to_relinquish;
 };
 
 /* List of possible type for each page in the p2m entry.
@@ -48,6 +57,12 @@  int p2m_init(struct domain *d);
 /* Return all the p2m resources to Xen. */
 void p2m_teardown(struct domain *d);
 
+/* Remove mapping refcount on each mapping page in the p2m
+ *
+ * TODO: For the moment only foreign mappings are handled
+ */
+int relinquish_p2m_mapping(struct domain *d);
+
 /* Allocate a new p2m table for a domain.
  *
  * Returns 0 for success or -errno.