The transient stability analysis (TSA) is a major numerical function in Energy Management System for large-scale power transmission system planning and evaluation. Ideally, the trajectory of system dynamics, such as bus voltage magnitude and generator phase angle, can be predicted to forecast issues and disturbances based on a time-domain solution of differential and algebraic equations (DAEs). However, the rise of system scales and complex machine models requires advanced computing techniques to achieve even faster than real-time (FTRT) criteria. For accelerating the execution, High-Performance Computing (HPC) on traditional CPU-based supercomputing clusters has been widely investigated for decades. Recently, as heterogeneous computing was introduced to power system domain areas, general-purpose computing devices such as graphics processing units (GPU) have been deployed to enhance computational performance and program portability further. This paper reviews the trials within the last 15 years on developing parallel TSA applications regarding mathematical formulations, programming approaches, and performance. We also address the future of TSA computation and its potential to improve the efficiency of FTRT executions.