Delays to Page Updates and Message Distribution

Resolved

Following up on this morning's message delays, it seems the main cause of the problem was a bug which meant that reording the components on a page flooded our job queue with lots of unnecessary jobs - causing a bottleneck which led to the delays.

We've just made some changes which should stop this from happening again.

If anyone has any technical questions about this then please let me know, I'll be only too happy to talk you through it.

Once again, thanks for all being so understanding.

Avatar for Robert Rawlins
Robert Rawlins
Resolved

Things look to have returned to normal now folks. Thank you so much for understanding - letting you down twice in as many days is embarrassing.

On a positive note, I believe we now know where the issue lay, and we'll be patching this today.

I'll come back to you all once we've got something worked out.

Avatar for Robert Rawlins
Robert Rawlins
Ongoing

We're seeing delays akin to the ones we had yesterday, where changes to pages may take longer than expected to appear, and message delivery to your audience may be delayed.

Our monitoring was quicker to highlight this after the changes we made, however, we appear to still have a ghost in the machine which is causing the problem.

We're taking a closer look now, I'll update you as soon as we know more.

Avatar for Robert Rawlins
Robert Rawlins
Began at:

Affected components
  • Status Pages
  • Message Distribution
    • Email
      • Email by Sorry™
      • MailChimp
      • Mailgun
      • SendGrid
      • Postmark
    • Microsoft Teams
    • Slack
    • SMS
    • Twitter
    • Website Plugin
    • Intercom Messenger App