Accurate segmentation of brain MRI images is essential for early diagnosis and prevention of brain tumors and strokes. However, it is challenging due to data scarcity, domain shift, complexity, and blurred edges. Recent methods use mean-teacher model learning for the semantic details of an individual image based on the efficiency of teacher architecture. However, to enhance the segmentation in brain MRI, the paper introduces SemS4, a semi-supervised method on the basis of cross-image architecture. SemS4 uses a single architecture integrating an Edge Prototype Attention (EPA) and a foreground prototype attention (FPA). More specifically, EPA uses edge prototype integrated with adaptive edge container to improve the edge features for which global edge features are stored to produce edge prototype. Moreover, FPA stabilizes the segmentation by transferring complementary foreground information. Furthermore, the proposed method uses Pixel Affinity Loss (PAL) for missing contextual correlation during supervision to enhance segmentation performance on edges. The performance of the proposed approach in experimental results for two brain MRI benchmarks, LGG and BRISC, validates the superiority over existing methods under different partitioning settings.