arm: mm: implement get_user_pages_fastlinux-lng-v3.10.14-finallinux-lng-v3.10.14linux-lng-3.10.14-2013.10

An implementation of get_user_pages_fast for ARM. It is based loosely
on the PowerPC implementation, but has a few subtle differences.

Under other architectures, the get_user_pages_fast implementations
disable the IRQs in the critical section. This protects against pages
backing page tables from being freed and from THP splits occurring as
these both call TLB invalidations which trigger an IPI which blocks
until the IRQs are re-enabled in get_user_page_fast.

Under ARM, TLB invalidations are usually broadcast in hardware thus
obviating the need for an IPI. After some discussion with Will Deacon:
http://marc.info/?l=linux-mm&m=138089480306901&w=2
It was decided that atomics should be used to protect the critical
section in get_user_pages_fast.

Calls to get_user_pages_fast, cause an atomic, gup_readers, to be
incremented. This guarantees that pages backing page tables won't be
freed from under it as both pte_free and tlb_flush_mmu will block on
non-zero values of gup_readers. Also, this guarantees that THPs will
not split, as an implementation of pmdp_splitting_flush is provided
that also blocks on non-zero values of gup_readers.

Signed-off-by: Steve Capper <steve.capper@linaro.org>
Signed-off-by: Kim Phillips <kim.phillips@linaro.org>

1 files changed, 9 insertions, 0 deletions

diff --git a/arch/arm/include/asm/pgalloc.h b/arch/arm/include/asm/pgalloc.h
index 943504f53f57..49f054cfaea1 100644
--- a/arch/arm/include/asm/pgalloc.h
+++ b/arch/arm/include/asm/pgalloc.h
@@ -123,6 +123,15 @@ static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte)
 static inline void pte_free(struct mm_struct *mm, pgtable_t pte)
 {
 	pgtable_page_dtor(pte);
+
+	/*
+	 * Before freeing page, check to see whether or not
+	 * __get_user_pages_fast is still walking pages in the mm.
+	 * If this is the case, wait until gup has finished.
+	 */
+	while (atomic_read(&mm->context.gup_readers) != 0)
+		cpu_relax();
+
 	__free_page(pte);
 }