Architecture, operation, and dependability of large-scale Internet services: three case studies

We describe the architecture and operational practices of three representative large-scale Internet services, and the causes of failure in two of them. We find convergence on a common architecture: division of nodes into service front-ends and back-ends, multiple levels of redundancy and load-balancing, and use of custom-written software for both production services and administrative tools. Operationally, we find a thin line between service developers and operators, and a need to coordinate problem detection and repair across administrative domains. Networking problems and operator error are the most significant contributors to failures in the systems we examined.