The aliasing effect in the discrete Fourier transform inherent will impose a serious detrimental effect on conventional phase retrieval measurement accuracy with under-sampled intensity. In this Letter, we describe a modal-based nonlinear optimization phase retrieval approach that is capable of retrieving wavefront measurements using under-sampled intensities. The extended Nijboer-Zernike theory is introduced to establish an analytic solution between wavefront phase and intensity image, and then nonlinear optimization is further utilized to solve wavefront aberration coefficients from under-sampled intensity data. The feasibility and accuracy of the algorithm are verified by simulations and experiments. This is a promising method that is especially suitable for full field phase recovery of optical systems with a relatively high numerical aperture.