Banner is one of if not the most important system at the University of Alaska and is critical to business needs during and outside of normal business hours. Core university business functions related to finance, HR, and SIS management is facilitated through the product or integrations to the set of products. The previous weekend we had worked to move all production Banner systems to our vendor cloud hosted environment. The starting day was the first day after the system was declared live in the new environment.
On March 14, at approximately 10:20 am reports were sent into the EAS team related to object timeout and related database connectivity issues. When technicians investigated they noticed that all dependencies on the database were stuck for a period of time which had a rippling effect across all banner related jobs and integration. The vendor was immediately engaged and helped determine the underlying cause related to specific processes. We worked with the vendor to optimize some specific jobs immediately, and then when others were discovered carried out similar optimizations.
Banner should be available 24/7 with the exception of the pre-defined maintenance windows.
Technicians worked with the vendor to determine that some very large tables, when accessed through stock banner forms, were taking an excessive amount of time to complete read queries. This was due to statistical data not having been captured in the weeks leading up to the go-live event, which would have fed execution plans to optimize queries against these tables. Due to a lack of full system use during these times, this data wasn’t generated to be applied for the go-live event.
Routinely collect statistical data to update execution plans.
March 15, 2023 - Started capturing statistical data based on system real usage.
We have adopted a nightly running of the statistics in the new environment. This feeds into updating execution plans periodically.