Fractional Chern insulators versus nonmagnetic states in twisted bilayer MoTe₂

Fractionally filled Chern bands with strong interactions may give rise to fractional Chern insulator (FCI) states, the zero -field analog of the fractional quantum Hall effect. Recent experiments have demonstrated the existence of FCIs in twisted bilayer MoTe2 without external magnetic fields-most robust at v = -2/3-as well as Chern insulators (CIs) at v = -1. Although the appearance of both of these states is theoretically natural in an interacting topological system, experiments repeatedly observe nonmagnetic (or weakly magnetic) states (lacking FCIs) at v = -1/3 and -4/3, a puzzling result, which has not been fully theoretically explained. In this paper, we perform Hartree-Fock and exact diagonalization calculations to test whether the standard MoTe2 moire model with the (greatly varying) parameter values available in the literature can reproduce the nonmagnetic/weakly magnetic states at v = -1/3 and -4/3 in unison with the FCI at v = -2/3 and CI state at v = -1. We focus on the experimentally relevant twist angles and, crucially, include remote bands. We find that the parameters proposed in Wang et al. [arXiv:2306.02501] can nearly capture the experimental phenomena at v = -1/3, -2/3, -1, -4/3 simultaneously, although the predicted ground states at v = -1/3 are still mostly FCIs and a larger dielectric constant c > 10 than is typical of hexagonal boron nitride (h-BN) substrate epsilon similar to 6 is required. Our results show the importance of remote bands in identifying the competing magnetic orders and lay the groundwork for further study of the realistic phase diagram.