We present end-to-end transformer training employing a realistic optical matrix multiplication simulator within the self-attention block. Our approach is validated in credit scoring, Tiny LLMs, and Tiny ViT tasks, and demonstrates that end-to-end fine-tuning effectively suppresses optical computation errors with up to 512 context lengths. Furthermore, a ‘generalized’ optical multiplication layer with trainable parameters outperforms digital baseline accuracy in the considered tasks.
