Jump to content

URGENT: Service Disruption Notification


Recommended Posts

  • Staff

Please be advised that we're experiencing storage issues with our hosting provider in the US hosting center. It is causing our database server to continuously crash. We apologise for any inconvenience and are actively working on a resolution. The billing system may be temporarily unavailable as well.

Link to comment
Share on other sites

  • Administrators

Hey everyone,

Apologies for the post-Halloween scare. We've worked with the hosting provider and have restored the service. We will be announcing later this week a maintenance window when we will be switching to a more modern storage solution.

-Paul

Link to comment
Share on other sites

7 minutes ago, Paul said:

Hey everyone,

Apologies for the post-Halloween scare. We've worked with the hosting provider and have restored the service. We will be announcing later this week a maintenance window when we will be switching to a more modern storage solution.

-Paul

Thank you for the update Paul! - Very happy to be back in control of our infrastructure again 😎

Link to comment
Share on other sites

We are currently seeing an issue logging into web app as well as remote control.  Is this issue still ongoing?  The error is "Database Query Exception" so I assume is related to this issue.

Link to comment
Share on other sites

  • Administrators

Yes Josh, apologies again. The root cause of the issue is impacted disk I/O where the SAN has experienced multiple disk failures at the same time greatly reducing I/O capacity. We'll announce tomorrow a maintenance window for the window where we will transition to a SSD-backed SAN which will greatly increase the performance of our cluster.

Update: 5:14PM - We're down again. We've decided to switch to the new storage now.

Update: 5:50PM - We're 20% done with the move to the new storage, we anticipate another 2 hours before we can restore service.

Update: 6:24PM - 69% done with the move.

Update: 6:58PM - Move completed successfully, databases are starting up.

-Paul

Edited by Paul
Link to comment
Share on other sites

I continue to receive multiple offline/online notifications from pulseway on many of our machines across multiple environments (onprem and cloud) is this related to the issues you are having?

Link to comment
Share on other sites

  • Administrators

Hi Darien,

Yes. That would be the cause of the database server finally starting up. Apologies for the inconvenience. Databases are still starting, will post another update when all services are completely back up.

Update: 7:10PM: All services are now fully operational.

-Paul

Edited by Paul
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...