Identification of Straggler Node in Distributed Environment Using Soft Computing Algorithms

Node straggling is most impactful thing to improve the Quality of Service (QoS) in distributed as well as cloud environment. On the other hand, the reason for which node discharges are caused is complicated. The precedents work mainly focus on detecting Stragglers, optimizing the level of programming and analysis of the root cause. Without order to help users automate their services, these approaches cannot provide valuable information. We in proposed new approach, a general method that incorporates framework and characteristics of the system for the analysis of root cause for detection of stragglers in the big data system, which basically reduce the process execution time and generate job failure issues during execution. We proposed an framework for straggler node detection from distributed environment using machine learning algorithm based on large scale job performed log data. After execution of machine learning algorithm system will predict straggler nodes list and dynamically eliminate such nodes from execution list. Q-Learning based reinforcement learning algorithm initially proposed for execution and validate the system result in multi node environments.