Two-mode Dicke model from non-degenerate polarization modes.

We realize a nondegenerate two-mode Dicke model with competing interactions in a Bose-Einstein condensate (BEC) coupled to two orthogonal polarization modes of a single optical cavity. The BEC is coupled to the cavity modes via the scalar and vectorial part of the atomic polarizability. We can independently change these couplings and determine their effect on a self-organization phase transition. Measuring the phases of the system, we characterize a crossover from a single-mode to a two-mode Dicke model. This work provides perspectives for the realization of coupled phases of spin and density.