Skip to content

perf(fpx): keep the hardware result for FMUL/FMLA S when the flags are already set - #917

Merged
hckrman101 merged 3 commits into
NotRexed:mainfrom
klimovich008:perf-fp-sticky-rescue
Nov 9, 2026
Merged

hckrman101 merged 3 commits into
NotRexed:mainfrom
klimovich008:perf-fp-sticky-rescue

Conversation

@klimovich008

@klimovich008 klimovich008 commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

What this fixes

Single-precision FMUL/FNMUL/FMLA results that the FPX1 fast path can't prove fell back to the soft-float exact body every time. These are tiny or infinite results, or a subnormal input under FZ. Code that runs with those cumulative FPSR flags already set paid that cost on each such op.

Why merge

The rescue keeps the hardware result only when it can raise no new FPSR bit. Everything else still goes to the exact body. The review found no correctness problem.

Change

  • a64_fpx_sticky32 runs in the outlined wrappers a64_fpx_mul32_fma32 and a64_fpx_fma32_fma32, inside the gate.
  • It uses RECOMP_FPX_FMA64; without a hardware FMA only FMUL is rescued.
  • Cheap rejects run before any FP op.
  • Dead fz_mul32 code is removed.
  • The FPX1 contract comment now points to the wrappers.

Tested

tests/test_fpx_sticky.py builds at x86-64-v3 and x86-64-v2. Each build runs:

  • guest FMUL/FNMUL/FMADD/FMLA S words through the lifter's pipeline (96 runs);
  • 16M random checks;
  • 150M sweep checks.

There were 0 mismatches against the exact body. Rescues are counted inside the wrappers, and a rescue under a closed gate fails the test. Sabotages of the emitter route, the wrapper route, the FIZ/AH/NEP mask and the UFC/IDC checks each fail it.

Not covered

  • No Unicorn, insn or leaf test run.
  • No full re-lift.
  • No game re-measure of this revision.
  • No aarch64, wasm or GCC build.
  • The shadow build doesn't exercise the rescue.

🤖 Generated with Claude Code

klimovich008 and others added 3 commits November 9, 2026 18:56
…e already set

The fma32-shaped wrappers (a64_fpx_mul32_fma32, a64_fpx_fma32_fma32) fell
back to the exact soft-float body whenever the fast path declined: a
subnormal operand under FZ, a tiny or infinite result. FPSR's cumulative
flags are sticky, so when every flag the Arm op could raise is already set,
one hardware FMA (or multiply) result is the Arm result: a subnormal operand
under FZ flushed to a zero of its sign with IDC set, a tiny result with UFC
set, an infinite result with OFC set, an exact cancellation. a64_fpx_sticky32
keeps those, in place of a64_fpx_fz_mul32 for these two shapes, and falls back
otherwise.

tests/test_fpx_sticky.py checks both wrappers as the generator emits them
against recomp_fpe_fma32, value and FPSR: 116M checks, 0 mismatches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…FIZ/AH/NEP, drop dead mul32 code

Review of NotRexed#917:
- The generated wrappers call A64_FPX_STICKY_PROBE() (empty in the real build) when the
  sticky rescue keeps a result; the test counts there instead of calling a64_fpx_sticky32
  itself, so a broken route (fpx_header's branch, the mul argument) fails it (exit 4).
- The test builds and runs twice: -march=x86-64-v3 and x86-64-v2 (RECOMP_FPX_FMA64 off:
  FMUL rescued by a plain multiply, FMLA must decline every case). v2 runs without the
  AVX2 check.
- gen_state sets FPCR bits 0-2 (alone and in the random mask); the sweep adds one state per
  a: partial flags, IXC clear, FIZ/AH/NEP. A rescue under a closed gate fails the test
  (the exact body models none of FIZ/AH/NEP, so values alone cannot tell).
- A64_FPX_FMA32 is gone: the helper uses RECOMP_FPX_FMA64.
- Dead code removed: a64_fpx_fz_mul32, a64_fpx_fz_result's mul param, "mul32" in the
  fz elif, A64_FPX_FZ_OPEN; FPX_FZ_HEADER's comment updated.
- The wrapper tests RECOMP_FPX_OPEN once; a64_fpx_sticky32 rejects a NaN operand, and
  FMUL with OFC and UFC clear and nothing flushed, before computing (same rescue counts
  with and without these rejects).
- FPX1's contract comment in arm64_to_c.h points to the outlined wrappers' rescues and says
  the shadow build does not reach them.

tests/test_fpx_sticky.py: 2 x (16M random + 150M sweep) checks, 0 mismatches. Each
sabotage in its docstring fails it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…'s emit route

tests/test_fpx_sticky.py gains a "lifted" case in both builds: four hand-encoded guest
words (FMUL, FNMUL, FMADD, FMLA by element, S) go through a64emit.translate and
a64_lift.op_text, the lifter's per-instruction pipeline (and hot_text, the hot functions'
form), so the C calls the outlined wrapper exactly as lifted code does. Each runs against
the same word translated with FPX off (the exact body inline) on 5 inputs that must be
rescued and 7 that must fall back (IXC only, UFC/OFC/IDC clear, NaN, FZ|AH, a normal
result): whole Ctx and the A64_FPX_STICKY_PROBE count must match. 96 runs a build, 0
mismatches; 40 rescues at x86-64-v3, 20 (FMUL/FNMUL only) at x86-64-v2.

a64_lift.py: Lifter.write's per-instruction pipeline and the hot-function rewrite become
op_text() and hot_text(), which the test calls (no change to what they emit).

Sabotage: the emitter no longer outlining FMUL S (EmitFPMulAddValue's `FpxOutline()`
made `FpxOutline() && !mul`) fails the lifted case in both builds (20 FMUL/FNMUL runs
without a rescue) while the random and sweep runs, which call the wrappers by name, pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@klimovich008
klimovich008 force-pushed the perf-fp-sticky-rescue branch from e5f07b2 to 0a2b38e Compare November 9, 2026 18:13
@klimovich008 klimovich008 changed the title FPX: FMUL/FMLA S past the fast path keep the hardware result when the flags are already set perf(fpx): keep the hardware result for FMUL/FMLA S when the flags are already set Nov 9, 2026
@klimovich008

Copy link
Copy Markdown
Contributor Author

Thanks. All seven points are addressed in 1ff4abe and 0a2b38e:

  1. The wrappers now call A64_FPX_STICKY_PROBE() (empty by default) when the sticky rescue keeps a result, and the test counts those calls. A new lifted-route case runs hand-encoded FMUL, FNMUL, FMADD and FMLA S guest words through a64emit.translate and a64_lift.op_text/hot_text, the lifter's own pipeline. It checks the whole Ctx against the same words with FPX off, plus the probe count, on 5 must-rescue and 7 must-fall-back inputs. With the emitter no longer outlining FMUL (EmitFPMulAddValue), that case fails while the by-name runs pass. With fpx_header's tuple made never to match, every run fails.
  2. There's a second build at -march=x86-64-v2 (-DEXPECT_FMA=0, #error if RECOMP_FPX_FMA64 disagrees). It needs only SSSE3, SSE4.2 and POPCNT, not AVX2. It requires FMUL rescues > 0 and FMLA rescues == 0: 16.5M / 0 in the sweep.
  3. gen_state now sets FPCR bits 0-2. The sweep adds 16 states: partial IDC/UFC/OFC, IXC clear, and FIZ/AH/NEP. The exact body models none of FIZ/AH/NEP, so a rescue under a closed gate fails the test. Dropping bits 0-2 from the mask fails it (190k random and 6.1M sweep at v3).
  4. The FPX1 comment in FpxH now points to the wrappers' rescues and says the shadow build doesn't reach them. Nothing moved into the runtime.
  5. A64_FPX_FMA32 is gone; the helper uses RECOMP_FPX_FMA64.
  6. Removed a64_fpx_fz_mul32, fz_result's mul and its branch, "mul32" in the elif, and the unused A64_FPX_FZ_OPEN. Comment updated.
  7. One OPEN test per wrapper. Before any FP op, the rescue rejects NaN operands, FMLA without an FMA, and FMUL with OFC and UFC clear and nothing flushed. Rescue counts are unchanged, and the fallback microbenchmark is flat within noise.

Results: 2 × (16M random + 150M sweep + 96 lifted) checks, 0 mismatches.

@hckrman101
hckrman101 merged commit db5bd80 into NotRexed:main Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant