Add IFN_COND_FMA functions

This patch adds conditional equivalents of the IFN_FMA built-in functions.
Most of it is just a mechanical extension of the binary stuff.

2018-07-12  Richard Sandiford  <richard.sandiford@linaro.org>

gcc/
	* doc/md.texi (cond_fma, cond_fms, cond_fnma, cond_fnms): Document.
	* optabs.def (cond_fma_optab, cond_fms_optab, cond_fnma_optab)
	(cond_fnms_optab): New optabs.
	* internal-fn.def (COND_FMA, COND_FMS, COND_FNMA, COND_FNMS): New
	internal functions.
	(FMA): Use DEF_INTERNAL_FLT_FN rather than DEF_INTERNAL_FLT_FLOATN_FN.
	* internal-fn.h (get_conditional_internal_fn): Declare.
	(get_unconditional_internal_fn): Likewise.
	* internal-fn.c (cond_ternary_direct): New macro.
	(expand_cond_ternary_optab_fn): Likewise.
	(direct_cond_ternary_optab_supported_p): Likewise.
	(FOR_EACH_COND_FN_PAIR): Likewise.
	(get_conditional_internal_fn): New function.
	(get_unconditional_internal_fn): Likewise.
	* gimple-match.h (gimple_match_op::MAX_NUM_OPS): Bump to 5.
	(gimple_match_op::gimple_match_op): Add a new overload for 5
	operands.
	(gimple_match_op::set_op): Likewise.
	(gimple_resimplify5): Declare.
	* genmatch.c (decision_tree::gen): Generate simplifications for
	5 operands.
	* gimple-match-head.c (gimple_simplify): Define an overload for
	5 operands.  Handle calls with 5 arguments in the top-level overload.
	(convert_conditional_op): Handle conversions from unconditional
	internal functions to conditional ones.
	(gimple_resimplify5): New function.
	(build_call_internal): Pass a fifth operand.
	(maybe_push_res_to_seq): Likewise.
	(try_conditional_simplification): Try converting conditional
	internal functions to unconditional internal functions.
	Handle 3-operand unconditional forms.
	* match.pd (UNCOND_TERNARY, COND_TERNARY): Operator lists.
	Define ternary equivalents of the current rules for binary conditional
	internal functions.
	* config/aarch64/aarch64.c (aarch64_preferred_else_value): Handle
	ternary operations.
	* config/aarch64/iterators.md (UNSPEC_COND_FMLA, UNSPEC_COND_FMLS)
	(UNSPEC_COND_FNMLA, UNSPEC_COND_FNMLS): New unspecs.
	(optab): Handle them.
	(SVE_COND_FP_TERNARY): New int iterator.
	(sve_fmla_op, sve_fmad_op): New int attributes.
	* config/aarch64/aarch64-sve.md (cond_<optab><mode>)
	(*cond_<optab><mode>_2, *cond_<optab><mode_4)
	(*cond_<optab><mode>_any): New SVE_COND_FP_TERNARY patterns.

gcc/testsuite/
	* gcc.dg/vect/vect-cond-arith-3.c: New test.
	* gcc.target/aarch64/sve/vcond_13.c: Likewise.
	* gcc.target/aarch64/sve/vcond_13_run.c: Likewise.
	* gcc.target/aarch64/sve/vcond_14.c: Likewise.
	* gcc.target/aarch64/sve/vcond_14_run.c: Likewise.
	* gcc.target/aarch64/sve/vcond_15.c: Likewise.
	* gcc.target/aarch64/sve/vcond_15_run.c: Likewise.
	* gcc.target/aarch64/sve/vcond_16.c: Likewise.
	* gcc.target/aarch64/sve/vcond_16_run.c: Likewise.


git-svn-id: https://gcc.gnu.org/svn/gcc/trunk@262587 138bc75d-0d04-0410-961f-82ee72b054a4

8 files changed, 380 insertions, 0 deletions

diff --git a/gcc/testsuite/gcc.target/aarch64/sve/vcond_13.c b/gcc/testsuite/gcc.target/aarch64/sve/vcond_13.c
new file mode 100644
index 00000000000..3c3aa00f674
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/sve/vcond_13.c
@@ -0,0 +1,58 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -ftree-vectorize" } */
+
+#define N 119
+
+#define DEF_LOOP(INV, TYPE, CMPTYPE, SUFFIX)			\
+  void __attribute__ ((noipa))					\
+  f_##INV##_##SUFFIX (TYPE *restrict a, TYPE *restrict b,	\
+		      TYPE *restrict c, TYPE *restrict d,	\
+		      CMPTYPE *restrict cond)			\
+  {								\
+    for (int i = 0; i < N; ++i)					\
+      {								\
+	TYPE mb = (INV & 1 ? -b[i] : b[i]);			\
+	TYPE mc = c[i];						\
+	TYPE md = (INV & 2 ? -d[i] : d[i]);			\
+	TYPE fma = __builtin_fma##SUFFIX (mb, mc, md);		\
+	TYPE truev = (INV & 4 ? -fma : fma);			\
+	a[i] = cond[i] < 10 ? truev : b[i];			\
+      }								\
+  }
+
+#define FOR_EACH_TYPE(T, INV)			\
+  T (INV, _Float16, short, f16)			\
+  T (INV, float, float, f32)			\
+  T (INV, double, double, f64)
+
+#define FOR_EACH_INV(T)				\
+  FOR_EACH_TYPE (T, 0)				\
+  FOR_EACH_TYPE (T, 1)				\
+  FOR_EACH_TYPE (T, 2)				\
+  FOR_EACH_TYPE (T, 3)				\
+  FOR_EACH_TYPE (T, 4)				\
+  FOR_EACH_TYPE (T, 5)				\
+  FOR_EACH_TYPE (T, 6)				\
+  FOR_EACH_TYPE (T, 7)
+
+FOR_EACH_INV (DEF_LOOP)
+
+/* { dg-final { scan-assembler-not {\tsel\t} } } */
+/* { dg-final { scan-assembler-not {\tmovprfx\t} } } */
+/* { dg-final { scan-assembler-not {\tmov\tz[0-9]+\.., z[0-9]+} } } */
+
+/* { dg-final { scan-assembler-times {\tfmad\tz[0-9]+\.h,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfmad\tz[0-9]+\.s,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfmad\tz[0-9]+\.d,} 2 } } */
+
+/* { dg-final { scan-assembler-times {\tfmsb\tz[0-9]+\.h,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfmsb\tz[0-9]+\.s,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfmsb\tz[0-9]+\.d,} 2 } } */
+
+/* { dg-final { scan-assembler-times {\tfnmad\tz[0-9]+\.h,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfnmad\tz[0-9]+\.s,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfnmad\tz[0-9]+\.d,} 2 } } */
+
+/* { dg-final { scan-assembler-times {\tfnmsb\tz[0-9]+\.h,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfnmsb\tz[0-9]+\.s,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfnmsb\tz[0-9]+\.d,} 2 } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/vcond_13_run.c b/gcc/testsuite/gcc.target/aarch64/sve/vcond_13_run.c
new file mode 100644
index 00000000000..f0f8385babc
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/sve/vcond_13_run.c
@@ -0,0 +1,37 @@
+/* { dg-do run { target aarch64_sve_hw } } */
+/* { dg-options "-O2 -ftree-vectorize" } */
+
+#include "vcond_13.c"
+
+#define TEST_LOOP(INV, TYPE, CMPTYPE, SUFFIX)		\
+  {							\
+    TYPE a[N], b[N], c[N], d[N];			\
+    CMPTYPE cond[N];					\
+    for (int i = 0; i < N; ++i)				\
+      {							\
+	b[i] = i % 15;					\
+	c[i] = i % 9 + 11;				\
+	d[i] = i % 13 + 14;				\
+	cond[i] = i % 17;				\
+	asm volatile ("" ::: "memory");			\
+      }							\
+    f_##INV##_##SUFFIX (a, b, c, d, cond);		\
+    for (int i = 0; i < N; ++i)				\
+      {							\
+	double mb = (INV & 1 ? -b[i] : b[i]);		\
+	double mc = c[i];				\
+	double md = (INV & 2 ? -d[i] : d[i]);		\
+	double fma = __builtin_fma (mb, mc, md);	\
+	double truev = (INV & 4 ? -fma : fma);		\
+	if (a[i] != (i % 17 < 10 ? truev : b[i]))	\
+	  __builtin_abort ();				\
+	asm volatile ("" ::: "memory");			\
+      }							\
+  }
+
+int
+main (void)
+{
+  FOR_EACH_INV (TEST_LOOP)
+  return 0;
+}
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/vcond_14.c b/gcc/testsuite/gcc.target/aarch64/sve/vcond_14.c
new file mode 100644
index 00000000000..1144461dec0
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/sve/vcond_14.c
@@ -0,0 +1,58 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -ftree-vectorize" } */
+
+#define N 119
+
+#define DEF_LOOP(INV, TYPE, CMPTYPE, SUFFIX)			\
+  void __attribute__ ((noipa))					\
+  f_##INV##_##SUFFIX (TYPE *restrict a, TYPE *restrict b,	\
+		      TYPE *restrict c, TYPE *restrict d,	\
+		      CMPTYPE *restrict cond)			\
+  {								\
+    for (int i = 0; i < N; ++i)					\
+      {								\
+	TYPE mb = (INV & 1 ? -b[i] : b[i]);			\
+	TYPE mc = c[i];						\
+	TYPE md = (INV & 2 ? -d[i] : d[i]);			\
+	TYPE fma = __builtin_fma##SUFFIX (mb, mc, md);		\
+	TYPE truev = (INV & 4 ? -fma : fma);			\
+	a[i] = cond[i] < 10 ? truev : c[i];			\
+      }								\
+  }
+
+#define FOR_EACH_TYPE(T, INV)			\
+  T (INV, _Float16, short, f16)			\
+  T (INV, float, float, f32)			\
+  T (INV, double, double, f64)
+
+#define FOR_EACH_INV(T)				\
+  FOR_EACH_TYPE (T, 0)				\
+  FOR_EACH_TYPE (T, 1)				\
+  FOR_EACH_TYPE (T, 2)				\
+  FOR_EACH_TYPE (T, 3)				\
+  FOR_EACH_TYPE (T, 4)				\
+  FOR_EACH_TYPE (T, 5)				\
+  FOR_EACH_TYPE (T, 6)				\
+  FOR_EACH_TYPE (T, 7)
+
+FOR_EACH_INV (DEF_LOOP)
+
+/* { dg-final { scan-assembler-not {\tsel\t} } } */
+/* { dg-final { scan-assembler-not {\tmovprfx\t} } } */
+/* { dg-final { scan-assembler-not {\tmov\tz[0-9]+\.., z[0-9]+} } } */
+
+/* { dg-final { scan-assembler-times {\tfmad\tz[0-9]+\.h,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfmad\tz[0-9]+\.s,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfmad\tz[0-9]+\.d,} 2 } } */
+
+/* { dg-final { scan-assembler-times {\tfmsb\tz[0-9]+\.h,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfmsb\tz[0-9]+\.s,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfmsb\tz[0-9]+\.d,} 2 } } */
+
+/* { dg-final { scan-assembler-times {\tfnmad\tz[0-9]+\.h,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfnmad\tz[0-9]+\.s,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfnmad\tz[0-9]+\.d,} 2 } } */
+
+/* { dg-final { scan-assembler-times {\tfnmsb\tz[0-9]+\.h,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfnmsb\tz[0-9]+\.s,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfnmsb\tz[0-9]+\.d,} 2 } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/vcond_14_run.c b/gcc/testsuite/gcc.target/aarch64/sve/vcond_14_run.c
new file mode 100644
index 00000000000..090f35be315
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/sve/vcond_14_run.c
@@ -0,0 +1,37 @@
+/* { dg-do run { target aarch64_sve_hw } } */
+/* { dg-options "-O2 -ftree-vectorize" } */
+
+#include "vcond_14.c"
+
+#define TEST_LOOP(INV, TYPE, CMPTYPE, SUFFIX)		\
+  {							\
+    TYPE a[N], b[N], c[N], d[N];			\
+    CMPTYPE cond[N];					\
+    for (int i = 0; i < N; ++i)				\
+      {							\
+	b[i] = i % 15;					\
+	c[i] = i % 9 + 11;				\
+	d[i] = i % 13 + 14;				\
+	cond[i] = i % 17;				\
+	asm volatile ("" ::: "memory");			\
+      }							\
+    f_##INV##_##SUFFIX (a, b, c, d, cond);		\
+    for (int i = 0; i < N; ++i)				\
+      {							\
+	double mb = (INV & 1 ? -b[i] : b[i]);		\
+	double mc = c[i];				\
+	double md = (INV & 2 ? -d[i] : d[i]);		\
+	double fma = __builtin_fma (mb, mc, md);	\
+	double truev = (INV & 4 ? -fma : fma);		\
+	if (a[i] != (i % 17 < 10 ? truev : c[i]))	\
+	  __builtin_abort ();				\
+	asm volatile ("" ::: "memory");			\
+      }							\
+  }
+
+int
+main (void)
+{
+  FOR_EACH_INV (TEST_LOOP)
+  return 0;
+}
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/vcond_15.c b/gcc/testsuite/gcc.target/aarch64/sve/vcond_15.c
new file mode 100644
index 00000000000..4967074072c
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/sve/vcond_15.c
@@ -0,0 +1,58 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -ftree-vectorize" } */
+
+#define N 119
+
+#define DEF_LOOP(INV, TYPE, CMPTYPE, SUFFIX)			\
+  void __attribute__ ((noipa))					\
+  f_##INV##_##SUFFIX (TYPE *restrict a, TYPE *restrict b,	\
+		      TYPE *restrict c, TYPE *restrict d,	\
+		      CMPTYPE *restrict cond)			\
+  {								\
+    for (int i = 0; i < N; ++i)					\
+      {								\
+	TYPE mb = (INV & 1 ? -b[i] : b[i]);			\
+	TYPE mc = c[i];						\
+	TYPE md = (INV & 2 ? -d[i] : d[i]);			\
+	TYPE fma = __builtin_fma##SUFFIX (mb, mc, md);		\
+	TYPE truev = (INV & 4 ? -fma : fma);			\
+	a[i] = cond[i] < 10 ? truev : d[i];			\
+      }								\
+  }
+
+#define FOR_EACH_TYPE(T, INV)			\
+  T (INV, _Float16, short, f16)			\
+  T (INV, float, float, f32)			\
+  T (INV, double, double, f64)
+
+#define FOR_EACH_INV(T)				\
+  FOR_EACH_TYPE (T, 0)				\
+  FOR_EACH_TYPE (T, 1)				\
+  FOR_EACH_TYPE (T, 2)				\
+  FOR_EACH_TYPE (T, 3)				\
+  FOR_EACH_TYPE (T, 4)				\
+  FOR_EACH_TYPE (T, 5)				\
+  FOR_EACH_TYPE (T, 6)				\
+  FOR_EACH_TYPE (T, 7)
+
+FOR_EACH_INV (DEF_LOOP)
+
+/* { dg-final { scan-assembler-not {\tsel\t} } } */
+/* { dg-final { scan-assembler-not {\tmovprfx\t} } } */
+/* { dg-final { scan-assembler-not {\tmov\tz[0-9]+\.., z[0-9]+} } } */
+
+/* { dg-final { scan-assembler-times {\tfmla\tz[0-9]+\.h,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfmla\tz[0-9]+\.s,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfmla\tz[0-9]+\.d,} 2 } } */
+
+/* { dg-final { scan-assembler-times {\tfmls\tz[0-9]+\.h,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfmls\tz[0-9]+\.s,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfmls\tz[0-9]+\.d,} 2 } } */
+
+/* { dg-final { scan-assembler-times {\tfnmla\tz[0-9]+\.h,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfnmla\tz[0-9]+\.s,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfnmla\tz[0-9]+\.d,} 2 } } */
+
+/* { dg-final { scan-assembler-times {\tfnmls\tz[0-9]+\.h,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfnmls\tz[0-9]+\.s,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfnmls\tz[0-9]+\.d,} 2 } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/vcond_15_run.c b/gcc/testsuite/gcc.target/aarch64/sve/vcond_15_run.c
new file mode 100644
index 00000000000..b08d4d8ebfb
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/sve/vcond_15_run.c
@@ -0,0 +1,37 @@
+/* { dg-do run { target aarch64_sve_hw } } */
+/* { dg-options "-O2 -ftree-vectorize" } */
+
+#include "vcond_15.c"
+
+#define TEST_LOOP(INV, TYPE, CMPTYPE, SUFFIX)		\
+  {							\
+    TYPE a[N], b[N], c[N], d[N];			\
+    CMPTYPE cond[N];					\
+    for (int i = 0; i < N; ++i)				\
+      {							\
+	b[i] = i % 15;					\
+	c[i] = i % 9 + 11;				\
+	d[i] = i % 13 + 14;				\
+	cond[i] = i % 17;				\
+	asm volatile ("" ::: "memory");			\
+      }							\
+    f_##INV##_##SUFFIX (a, b, c, d, cond);		\
+    for (int i = 0; i < N; ++i)				\
+      {							\
+	double mb = (INV & 1 ? -b[i] : b[i]);		\
+	double mc = c[i];				\
+	double md = (INV & 2 ? -d[i] : d[i]);		\
+	double fma = __builtin_fma (mb, mc, md);	\
+	double truev = (INV & 4 ? -fma : fma);		\
+	if (a[i] != (i % 17 < 10 ? truev : d[i]))	\
+	  __builtin_abort ();				\
+	asm volatile ("" ::: "memory");			\
+      }							\
+  }
+
+int
+main (void)
+{
+  FOR_EACH_INV (TEST_LOOP)
+  return 0;
+}
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/vcond_16.c b/gcc/testsuite/gcc.target/aarch64/sve/vcond_16.c
new file mode 100644
index 00000000000..7fa98716830
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/sve/vcond_16.c
@@ -0,0 +1,58 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -ftree-vectorize" } */
+
+#define N 119
+
+#define DEF_LOOP(INV, TYPE, CMPTYPE, SUFFIX)			\
+  void __attribute__ ((noipa))					\
+  f_##INV##_##SUFFIX (TYPE *restrict a, TYPE *restrict b,	\
+		      TYPE *restrict c, TYPE *restrict d,	\
+		      CMPTYPE *restrict cond)			\
+  {								\
+    for (int i = 0; i < N; ++i)					\
+      {								\
+	TYPE mb = (INV & 1 ? -b[i] : b[i]);			\
+	TYPE mc = c[i];						\
+	TYPE md = (INV & 2 ? -d[i] : d[i]);			\
+	TYPE fma = __builtin_fma##SUFFIX (mb, mc, md);		\
+	TYPE truev = (INV & 4 ? -fma : fma);			\
+	a[i] = cond[i] < 10 ? truev : 10;			\
+      }								\
+  }
+
+#define FOR_EACH_TYPE(T, INV)			\
+  T (INV, _Float16, short, f16)			\
+  T (INV, float, float, f32)			\
+  T (INV, double, double, f64)
+
+#define FOR_EACH_INV(T)				\
+  FOR_EACH_TYPE (T, 0)				\
+  FOR_EACH_TYPE (T, 1)				\
+  FOR_EACH_TYPE (T, 2)				\
+  FOR_EACH_TYPE (T, 3)				\
+  FOR_EACH_TYPE (T, 4)				\
+  FOR_EACH_TYPE (T, 5)				\
+  FOR_EACH_TYPE (T, 6)				\
+  FOR_EACH_TYPE (T, 7)
+
+FOR_EACH_INV (DEF_LOOP)
+
+/* { dg-final { scan-assembler-times {\tsel\t} 24 } } */
+/* { dg-final { scan-assembler-not {\tmovprfx\t} } } */
+/* { dg-final { scan-assembler-not {\tmov\tz[0-9]+\.., z[0-9]+} } } */
+
+/* { dg-final { scan-assembler-times {\tfmla\tz[0-9]+\.h,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfmla\tz[0-9]+\.s,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfmla\tz[0-9]+\.d,} 2 } } */
+
+/* { dg-final { scan-assembler-times {\tfmls\tz[0-9]+\.h,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfmls\tz[0-9]+\.s,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfmls\tz[0-9]+\.d,} 2 } } */
+
+/* { dg-final { scan-assembler-times {\tfnmla\tz[0-9]+\.h,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfnmla\tz[0-9]+\.s,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfnmla\tz[0-9]+\.d,} 2 } } */
+
+/* { dg-final { scan-assembler-times {\tfnmls\tz[0-9]+\.h,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfnmls\tz[0-9]+\.s,} 2 } } */
+/* { dg-final { scan-assembler-times {\tfnmls\tz[0-9]+\.d,} 2 } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/vcond_16_run.c b/gcc/testsuite/gcc.target/aarch64/sve/vcond_16_run.c
new file mode 100644
index 00000000000..d4e5afd4d57
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/sve/vcond_16_run.c
@@ -0,0 +1,37 @@
+/* { dg-do run { target aarch64_sve_hw } } */
+/* { dg-options "-O2 -ftree-vectorize" } */
+
+#include "vcond_16.c"
+
+#define TEST_LOOP(INV, TYPE, CMPTYPE, SUFFIX)		\
+  {							\
+    TYPE a[N], b[N], c[N], d[N];			\
+    CMPTYPE cond[N];					\
+    for (int i = 0; i < N; ++i)				\
+      {							\
+	b[i] = i % 15;					\
+	c[i] = i % 9 + 11;				\
+	d[i] = i % 13 + 14;				\
+	cond[i] = i % 17;				\
+	asm volatile ("" ::: "memory");			\
+      }							\
+    f_##INV##_##SUFFIX (a, b, c, d, cond);		\
+    for (int i = 0; i < N; ++i)				\
+      {							\
+	double mb = (INV & 1 ? -b[i] : b[i]);		\
+	double mc = c[i];				\
+	double md = (INV & 2 ? -d[i] : d[i]);		\
+	double fma = __builtin_fma (mb, mc, md);	\
+	double truev = (INV & 4 ? -fma : fma);		\
+	if (a[i] != (i % 17 < 10 ? truev : 10))		\
+	  __builtin_abort ();				\
+	asm volatile ("" ::: "memory");			\
+      }							\
+  }
+
+int
+main (void)
+{
+  FOR_EACH_INV (TEST_LOOP)
+  return 0;
+}