Deep neural networks (DNNs) often accept high-dimensional media data (e.g., photos, text, and audio) and understand their perceptual content (e.g., a cat). To test DNNs,
diverse
inputs are needed to trigger mis-predictions. Some preliminary works use byte-level mutations or domain-specific filters (e.g., foggy), whose enabled mutations may be limited and likely error-prone. State-of-the-art (SOTA) works employ deep generative models to generate (infinite) inputs. Also, to keep the mutated inputs perceptually
valid
(e.g., a cat remains a “cat” after mutation), existing efforts rely on imprecise and less generalizable heuristics. This study revisits two key objectives in media input mutation—perception diversity (
Div
) and validity (
Val
) — in a rigorous manner based on manifold, a well-developed theory capturing perceptions of high-dimensional media data in a low-dimensional space. We show important results that
Div
and
Val
inextricably bound each other, and prove that SOTA generative model-based methods fundamentally fail to mutate
real-world media data
(either sacrificing
Div
or
Val
). In contrast, we discuss the feasibility of mutating real-world media data with provably high
Div
and
Val
based on manifold.
Following, we concretize the technical solution of mutating media data of various formats (images, audios, text) via a unified manner based on manifold. Specifically, when media data are projected into a low-dimensional manifold, the data can be mutated by walking on the manifold with certain directions and step sizes. When contrasted with the input data, the mutated data exhibit encouraging Div in the perceptual traits (e.g., lying vs. standing dog) while retaining reasonably high Val (i.e., a dog remains a dog).
We implement our techniques in Deepwalk for testing DNNs. Deepwalk constructs manifolds for media data offline. In online testing, Deepwalk walks on manifolds to generate mutated media data with provably high Div and Val. Our evaluation tests DNNs executing various tasks (e.g., classification, self-driving, machine translation) and media data of different types (image, audio, text). Deepwalk outperforms prior methods in terms of the testing comprehensiveness and can find more error-triggering inputs with higher quality. The tested DNNs, after repaired using Deepwalk’s findings, exhibit better accuracy.
更多
查看译文
关键词
Deep learning testing,deep neural networks,metamorphic testing
