The goal of Human Computer interaction (HCI) is to design a system that improves the interaction between computers and users, mainly for people with disabilities. For this reason, this research describes how to make computers more usable by using speech based interaction. Proposed system will use speech and non speech characteristics of human voice for hands free computing. System will use human vocal characteristics to command mouse pointer. Words and sounds will be used to perform mouse 2-D movement and performing operating system’s operations. By doing this people with disabilities can use the computer systems normally as the normal users use it. Current speech recognition systems provide discrete interaction because they process user’s vocal utterances at word level, proposed system will be able to perform mouse-like operations fluidly.