In order to calculate photonic devices with slowly varying structural variation along propagation direction, we develop finite element beam propagation method (FE-BPM) with coordinate transformation. In this approach, converting a longitudinally varying waveguide into equivalent straight waveguide, cumbersome processes in FE-BPM, such as mesh updating process, is avoided. We employ this simulation technique in shape optimization of photonic devices and show a design example of mode converter.