In this article, we utilize an explainable AI approach, the Explainable Boosting Machine (EBM), to perform feature analysis on an extensive Internet of Things (IoT) dataset collected from real-world devices. After feature selection and data processing, our training dataset includes 2.88 million traffic data instances, categorized into six classes (DDoS, DoS, Mirai, Recon, Spoofing, and Benign). The EBM trained on this dataset achieved a impressive accuracy rate of 99.4% and an F1 score of 92.8%. Using the resultant model, we interpreted its predictions based on feature importance. The identified feature importance aligned well with established cybersecurity principles, indicating the model's potential. However, our analysis revealed that the machine learning model's predictions were strongly tied to the specific characteristics of the training IoT dataset, thereby raising concerns about the model's reliability when applied to real-world attack detection. Future research could explore the use of more diverse and balanced datasets or the applicability of the machine learning model in different IoT contexts, aiming to enhance the model's generalizability and practical relevance.