Management UI performance degradation

Post Incident Review

Summary:

An unexpected surge in requests overloaded our origin services which prevented access to app.sorryapp.com

Timeline of events (UTC)

20th October 2025

• 10:31 Monitoring confirms errors with UI
• 10:35 Incident acknowledged
• 10:35 Updated customers about the issues with UI
• 11:05 First patch applied
• 11:12 Second patch applied
• 11:14 Monitoring confirms UI recovered
• 16:00 First round of permanent changes made

21st October 2025

• 09:00 Post incident review process started
• 15:00 Firewall improvement tasks completed
• 15:06 General notice posted with firewall improvements

Impact

Team members attempting to use app.sorryapp.com for a 43 minute duration would have experienced timeouts and errors.

Root cause

We were allowing POST requests to Status Pages, when we should have only been allowing non-POST requests. POST requests are not cacheable, and were therefore able to bypass our caching and make their way all the way down to the origin, which is what caused the overload.

Immediate Actions taken

Initial temporary blocks followed by blocking of POST requests except for endpoints where they are explicitly allowed.

Follow-up actions completed

Firewall changes with configuration changes made to our application firewall to better identify and block that kind of traffic in the future.

Performance of the management UI has remained stable, and we’re now marking this incident as resolved.

We’ll continue to review the underlying cause and make improvements to reduce the likelihood of similar issues in the future. A post-incident report will be shared once our investigation is complete.

Thank you for your patience during this incident.

We’ve identified the root cause and implemented a temporary fix, which has significantly improved performance in the management UI.

We’ll continue to monitor and will provide a final update once we’re confident the issue is fully resolved.

Thank you for your patience during this incident — we’ll follow up with a post-incident review once complete.

We’re investigating an issue affecting the management UI, where some customers may experience slow loading or timeouts.

We are actively investigating and we’ll provide updates as soon as we know more.