Around 9.00 AM EST on 4/5/2023, Provation received support tickets that many of their users were unable to login to iPro app.

The Provation support team evaluated the issue and noticed intermittent logins for several users.

It was to be noted that the Provation team completed and updated the production environment the night before (4/4/2023) as scheduled.

The Provation team noticed some database locks and increased CPU usage and resolved those database locks, updated the API and restarted the IIS that improved the resource utilization on the servers

This seemed to resolve the issue with many of the users being able to login again.

The following morning around 9.00 am EST on 4/6/2023, Provation support received communication that multiple users across several sites are reporting authentication errors.

The Provation team evaluated the issues and determined that the server resources are being utilized appropriately and restarted the web services that seemed to temporarily reduce the authentication errors users were receiving.

The team examined the deployment procedure and the code deployed for production. It was determined that the version deployed 2.5.9 may have stale code that can cause authentication errors particularly when large number of users use the system.

The team decided to revert the code to the previous core version retaining as many of the features promised in the release notes.

With approval the code was reverted in the test environment to core version 2.5.2.

The Provation team completed a thorough testing and load testing on the test environment.

With sign off, the Provation team reverted the production environment to core version 2.5.2 at 9.30 pm EST, followed by rigorous testing.

The Provation team monitored the situation the following morning (4/7/2023) and over the weekend and ensured that there are no longer any authentication errors reported.

Most of the features from the release notes are delivered except for a few that are noted and communicated. The Provation team will update the environment to the latest core version with all missed changes in the coming weeks after through regression testing.