Possible delays in notifications being delivered

Uptime Impact: 42 minutes and 22 seconds
Resolved
Resolved

We've now resolved the incident. Thanks for your patience.

Avatar for Robert Rawlins
Robert Rawlins
Recovering

Things look to be much happier now. We'll keep an eye on things for a little longer, just to be sure.

Avatar for Robert Rawlins
Robert Rawlins
Identified

I've now got a better understanding of the issue. It looks as if AWS is currently having some performance issues on one of their APIs, which is having a knock-on effect. They haven't confirmed this on their status page at the moment, but that looks to be the case.

Hopefully the additional servers we added to handle the queue will help mitigate the effects.

Avatar for Robert Rawlins
Robert Rawlins

I've just increased the number of servers we have working on the queue, which should help keep things running. I'll continue to dig to find the culprit which is slowing things down.

Avatar for Robert Rawlins
Robert Rawlins
Investigating

It looks like our notifications queue is running a little slower than usual, which might cause a slight delay in messages being delivered. I'm diving in to take a closer look now, I'll update you as soon as a I know more.

Avatar for Robert Rawlins
Robert Rawlins
Began at:

Affected components
  • Message Distribution
    • Email
      • Email by Sorry™
      • MailChimp
      • Mailgun
      • SendGrid
      • Postmark
    • Microsoft Teams
    • Slack
    • SMS
    • Twitter
    • Website Plugin
    • Intercom Messenger App