Outage: Shibboleth Overloaded on 9/28/2009
On Monday, 9/28/2009, UCLA's Shibboleth Identity Provider (IDP) experienced its heaviest traffic load to that date. It was the first day of class for Fall 2009. The pair of web servers running the IDP software started queuing requests under heavy load, causing widespread interruption to UCLA's services.
AIS engineers have traced the problem to a logging configuration error. The incorrect configuration causes Shibboleth to spend significantly longer time to process each log in transaction. AIS has determined that correcting this error improves system performance by as much as 10 fold.
During the outage, AIS replaced its older Shibboleth servers with 3 new virtualized servers one week ahead of schedule. These new servers are handling campus authentication traffic smoothly. AIS plans to update these servers with the corrected logging configuration file on October 7, 2009. Once updated, we expect the 3 new Shibboleth servers to easily handle campus authentication loads, including the additional traffic from URSA and.
An more detailed/technical analysis of this issue, including additional follow up action items is also available.