Electromagnetic transient (EMT) simulation of power electronic converters is critical for analysis, design, and fast control prototyping of power and energy systems. This paper proposes a multi-granular GPU parallel-rate exponential integrator (EI) algorithm for fast offline EMT simulation of power electronic systems. The proposed parallel-rate EI algorithm utilizes the massively parallel GPU architecture to compute multiple discretization steps in parallel. The matrix-vector computations of the EI algorithm within each time step are also parallelized. Additionally, a novel GPU-based framework is proposed for numerically efficient precomputation of matrix exponentials before a simulation loop starts. The high degree of parallelism leads to large simulation speedups compared to single-thread CPU implementations. The discretization technique of high-order EI algorithm is absolutely stable with no numerical ringing and can achieve accurate differential equation discretization with large time-step sizes. The proposed parallel-rate EI solver is further applied to detect passive/diode switching events accurately. Two case studies are used to demonstrate the accuracy and efficiency of the parallel-rate EI algorithm. Two additional case studies showcase the benefit of the proposed parallel precomputation technique for matrix exponentials.