Event Occurrence: January 19, 2021 12:47pm - 4:15pm
The Banner database BANP holds the information for the Banner enterprise resource planning (ERP) system. The system is critical to the success of the university tracking financial, personnel, course, and student information for all of UA. The database backends services such as UAOnline, Banner 8/9 admin pages, direct data integrations like ELMO, and data feeds to other UA Services like Google Apps and Blackboard Learn. Chef is used to manage and maintain the systems configuration.
On January 19th at 12:47pm technicians were alerted by monitoring and users of Banner 9 and UAOnline. Monitoring showed a connectivity issue for the database. New and existing Banner 9 and UAOnline were inoperable. Technicians determined that established network connections were still working, but all new connections to the database server were failing. Technicians noticed that the configuration management update failed leaving the firewall in an unexpected state. The firewall was restored and the issue was thought to be resolved at 3:00pm, but was not. By 3:15pm users were reporting they were still impacted. Technicians reviewing the system found that the memory consumption was so high that secondary memory (swap) was almost fully consumed. Secondary memory is much slower than regular memory so it was determined the fastest way to clear this and restore performance to the host was to stop the database service and restart the host. One database process (11996) that had been running for 4 days was terminated to facilitate clean shutdown of the database. The host came up and the configuration management utility was run to ensure the firewall had current settings. The service was restored and set to monitoring the incident at 4:18pm.
The production Banner database servers should operate within acceptable resource utilization levels. Service should be available to customers 24/7 except during pre-defined maintenance windows.
Chef configuration management runs every hour and failed to complete due to a corrupted Redhat package manager (RPM) database. The result left the server’s firewall in a partially configured state which rejected new network connection to the database. After correcting the RPM database, the memory consumption on the server was so high that it used a secondary memory system (swap) that is approximately 1000 times slower. Due to the secondary memory usage and the flood of new connection requests caused the server to respond extremely. To clear the secondary memory the host was rebooted, causing a further delay to service restoration.