Single Instruction Isolation for RISC-V Vector Test Failures.

Testing complex RISC-V extensions such as RISC-V Vector (RVV) with its 600+ highly configurable instructions is crucial. For this reason, test suites have been developed over the last years, including both hand-written and automatically generated tests. Although the process of running these tests is often highly automated, a significant portion of the work, namely the result analysis, has to be conducted manually after the run. This paper introduces the modular, open-source framework RVVTS for positive and negative testing of RVV implementations, featuring a novel technique called Single Instruction Isolation with Code Minimization, which significantly reduces manual result analysis of failing tests. We demonstrate the effectiveness of RVVTS by automatically generating and applying test sets to the RISC-V VP++ Virtual Prototype and the QEMU emulator, achieving a functional coverage of >94%. For RISC-V VP++, our framework detects and minimizes similar to 1, 849 failures and associate them with 10 isolated, failing instructions. Similarly, for QEMU, it detects similar to 19k failures and relates them to 168 instructions for debugging. Overall, we confirmed 3 new bugs in the RISC-V VP++ and 2 in QEMU (and 7 more are to be analyzed).