Software System Exploration Using Library Call Analysis.

The ability to analyze software systems without access to the source code, offers many advantages including the detection of vulnerabilities so that they may be fixed before an adversary can exploit them in a zero day attack. This type of analysis also has an important role in education as it allows students to use their imagination and creativity in the exploration process. In this paper, we use two techniques for black-box testing based on our previous work, where we demonstrated how library calls may be intercepted using wrappers as well as using the kernel to separate the memory of a process into regions, based on the (statically/dynamically) linked libraries that a program uses. By monitoring function calls to libraries or the main executable, we can determine if a high-level execution signature (which depends not only on the occurrence, but also the sequence and number of calls) fits a pattern of a possible attack against a system under test. We can, then, (a) determine whether a call should go ahead, (b) determine whether the arguments are acceptable and (c) ensure that we will be informed when there is suspicion of foul play. We then demonstrate how these techniques may be used in student training sessions to explore the structure of software systems and determine how such systems respond to specific input sequences designed to trigger bugs or demonstrate unexpected behavior.