During routine database maintenance, a change to the database’s configuration file successfully completed on all but four production customer databases caused by a bug within Microsoft’s code. Due to the nature of the bug, it was not exposed until a database was restarted which happened on 5/3/21 at 6:54AM CST after routine database maintenance was initiated on all Provation Apex databases. During normal maintenance operations, databases are restarted using a phased approach. Importantly, this is done without incurring downtime. However, once the restart occurred, four databases that were impacted by the bug experienced a service disruption. In an attempt to resolve this issue, Microsoft’s on-call engineer manually revised the configuration file.
While mitigating the incorrect configuration file, Microsoft’s on-call engineer accidentally introduced an extra hidden space/character into the configuration file, causing periodic service disruptions on 5/3/21 at 12:03PM CST on a rolling basis. Once Microsoft’s on-call engineer identified and fixed the corrupted database configuration file, all databases came online and available at 1:55PM CST.
To prevent any similar issues:
We apologize for the impact this had on our customers. We continue to be committed to our purpose: To empower Providers to deliver quality healthcare for all. This means providing you with a great user experience with Provation Apex.