The factorization method can be applied for certain classes of inverse problems, where the shape of a domain has to be determined. It constructs a binary criterion which determines whether a given point is inside or outside the domain. The present paper develops the theory of the factorization method for the time harmonic Maxwell system where the support of the contrast of the index of refraction has to be determined from the knowledge of the far-field patterns of the scattered fields for plane incident waves. The cases of an absorbing medium and a non-absorbing medium (where in the latter case the dielectricity varies smoothly) are considered.