Child pages
  • 20090928ShibbolethOutage
Skip to end of metadata
Go to start of metadata

Outage: Shibboleth Overloaded on 9/28/2009

Problem

On Monday, 9/28/2009, UCLA's Shibboleth Identity Provider (IDP) experienced its heaviest traffic load to that date. It was the first day of class for Fall 2009. The pair of web servers running the IDP software started queuing requests under heavy load, causing widespread interruption to UCLA's services.

Cause

AIS engineers have traced the problem to a logging configuration error. The incorrect configuration causes Shibboleth to spend significantly longer time to process each log in transaction. AIS has determined that correcting this error improves system performance by as much as 10 fold.

Resolution

During the outage, AIS replaced its older Shibboleth servers with 3 new virtualized servers one week ahead of schedule. These new servers are handling campus authentication traffic smoothly. AIS plans to update these servers with the corrected logging configuration file on October 7, 2009. Once updated, we expect the 3 new Shibboleth servers to easily handle campus authentication loads, including the additional traffic from URSA and MyUCLA.

Additional Information

An more detailed/technical analysis of this issue, including additional follow up action items is also available.