Frustrated competitive forces in the Et2Me2Sb[Pd(dmit)2]2 molecular conductor

The Et2Me2Sb[Pd(dmit)2]2 salt is a Mott insulator at room temperature with layers of equivalent dimers [Pd(dmit)2]2 forming a two-dimensional triangular lattice. Although spin liquid behavior has been reported in other systems with a quasitriangular lattice, Et2Me2Sb[Pd(dmit)2]2 shows no spin liquid behavior irrespective of the expected high spin frustration due to its almost perfect equilateral triangular structure. Here, we report a high resolution structural analysis that reveals the existence of an intermediate incommensurate phase between the highly frustrated room temperature phase and the charge separated phase characterized by differently charged dimers coupled to different intradimer Pd-Pd distances. The structure of the incommensurate phase exhibits no modulation of the Pd-Pd distance but a large steric effect between the cation and the terminal sulfur atom of the Pd(dmit)2 molecules. In addition to this steric frustration, the interdimer transfer integrals coupled to the arch distortion in the dimer are also modulated, suggesting that the spin state in the incommensurate phase is due to an incommensurate modulation of the transfer integral. We demonstrate that the presence of the incommensurate phase originates from different frustrated competitive forces and discuss its magnetic and optical properties.