URGENT: Service Disruption Notification

[Victoria - Pulseway Team]

Please be advised that we're experiencing storage issues with our hosting provider in the US hosting center. It is causing our database server to continuously crash. We apologise for any inconvenience and are actively working on a resolution. The billing system may be temporarily unavailable as well.

[Martin Astley]

Hope everyone is ok? 

[augment-tech]

Please keep us up to date with an ETA for resolution. Thanks!

[Paul]

Hey everyone,

Apologies for the post-Halloween scare. We've worked with the hosting provider and have restored the service. We will be announcing later this week a maintenance window when we will be switching to a more modern storage solution.

-Paul

[Martin Astley]

7 minutes ago, Paul said:

Hey everyone,

Apologies for the post-Halloween scare. We've worked with the hosting provider and have restored the service. We will be announcing later this week a maintenance window when we will be switching to a more modern storage solution.

-Paul

Thank you for the update Paul! - Very happy to be back in control of our infrastructure again 😎

[DarienAL]

What were the symptoms for us as users? I received a some offline notifications for machines that weren't offline, was that related to this?

[Josh Zets]

We are currently seeing an issue logging into web app as well as remote control.  Is this issue still ongoing?  The error is "Database Query Exception" so I assume is related to this issue.

[Paul]

Yes Josh, apologies again. The root cause of the issue is impacted disk I/O where the SAN has experienced multiple disk failures at the same time greatly reducing I/O capacity. We'll announce tomorrow a maintenance window for the window where we will transition to a SSD-backed SAN which will greatly increase the performance of our cluster.

Update: 5:14PM - We're down again. We've decided to switch to the new storage now.

Update: 5:50PM - We're 20% done with the move to the new storage, we anticipate another 2 hours before we can restore service.

Update: 6:24PM - 69% done with the move.

Update: 6:58PM - Move completed successfully, databases are starting up.

-Paul

Edited November 2, 2022 by Paul

[DarienAL]

I continue to receive multiple offline/online notifications from pulseway on many of our machines across multiple environments (onprem and cloud) is this related to the issues you are having?

[Paul]

Hi Darien,

Yes. That would be the cause of the database server finally starting up. Apologies for the inconvenience. Databases are still starting, will post another update when all services are completely back up.

Update: 7:10PM: All services are now fully operational.

-Paul

Edited November 2, 2022 by Paul