Magnetic anisotropy is a fundamental property of magnetic materials that plays an essential role in the stability of magnetic domains and skyrmions. In this ferromagnetic resonance study, we report the evolution of magnetic anisotropy by substituting various 4d metals in the antiskyrmion host (Fe, Ni)3P with S4 tetragonal symmetry. In the undoped compound (Fe0.63Ni0.37)3P and in the Ru-doped (Fe0.59Ni0.32Ru0.09)3P, the uniaxial magnetic anisotropy has an easy-plane character, while Pd doping turns the material to an easy-axis-type magnet, as observed in (Fe0.62Ni0.29Pd0.09)3P. In addition to the dominant uniaxial anisotropy, we also quantify the fourfold anisotropy of the plane perpendicular to the tetragonal axis in (Fe0.63Ni0.30Pd0.07)3P. Using analytical calculations and micromagnetic simulations, we discuss how this planar anisotropy competes with the anisotropic Dzyaloshinskii-Moriya interaction in determining the orientation of the magnetic stripes and the antiskyrmions.