Existing image-based fire segmentation techniques use convolutional neural networks to handle complicated scenes. Such approaches perform poorly when flame sizes vary greatly and when backgrounds are complex. In this paper, we describe a novel Context-Oriented Multi-Scale Network for fire segmentation. We construct a multi-scale aggregation module that combines semantic information at different levels in the neural network in order to recognize fires with different shapes and sizes. We also describe a Context-Oriented Module, which increases the receptive field of the network by utilizing relationships of all pixels in the feature map in order to obtain features that more effectively discriminate between fire and non-fire pixels. Experimental results demonstrate that our proposed model has a 2.7% higher mean Intersection over Union (mIoU) accuracy than previous fire detection methods.