Storage provider maintenance update - possible service disruption

[Anne]

Hi all, we do apologise for the inconvenience and service disruption, our storage provider had to reschedule the data maintenance works last minute and some instances might get affected. Most of the functionality should be working as normal, while the online billing portal is expected to be down till the maintenance works are over.

We do apologise for this inconvenience and service disruption. Moreover, we have officially taken a decision to switch the data providers and we are in the process of migrating to a new datacenter provider. We will be prioritising the customers who are expressing their concerns and would like to be moved first. Please mention this to your account manager and will prioritise you once the move begins in the upcoming weeks. 

if you would like to be among the customers who wishes to be moved to the new data storage please reach out to stefan.mclaughlin@pulseway.com  or to your account manager

[Connexium]

Hello Rescheduling a maintenance right at the beginning of a workday is absolutely insane and unacceptable.

[Rick Bestany]

Most of the functionality is NOT working as normal.

[John D]

Please keep us updated, if it does not come up soon it will be a total disaster today as our entire workforce will be unsupported.

I was told it would be back in an hour, this was 1 hour ago through e-mail.

Thank you

[ltintnteam]

Have you considered sending out an email to let everyone know that your service is down?
I know you can do this, because I receive promotional emails every day. If your support department can’t do it, contact your marketing department because they’ve got it figured out.

[Paul]

Some more context for this situation:

The hosting provider has rebuilt their storage gateway and they are having stability issues with it. Right now they are working on restoring it.

-Paul

[Ondrej Patrovic]

Is there an ETA?

We're still down.

Edited May 26, 2023 by Ondrej Patrovic

[Anne]

6 minutes ago, ltintnteam said:

Have you considered sending out an email to let everyone know that your service is down?
I know you can do this, because I receive promotional emails every day. If your support department can’t do it, contact your marketing department because they’ve got it figured out.

 

Hi 

We always try to communicate all service disruption as in advance as possible. There was a forum, and email communication sent out earlier this week on Tuesday. But due to the Data Centre change of schedule which was communicated to us at the last minute we did not get a chance to send the new global announcement once the service disruption started. 

 

As mentioned on our official post regarding this: we have officially taken a decision to switch the data providers and we are in the process of migrating to a new datacenter provider. We will be prioritizing the customers who are expressing their concerns and would like to be moved first. Please mention this to your account manager and will prioritize you once the move begins in the upcoming weeks. 

[ltintnteam]

Just now, StefanMcl_Pulseway said:

 

Hi 

We always try to communicate all service disruption as in advance as possible. There was a forum, and email communication sent out earlier this week on Tuesday. But due to the Data Centre change of schedule which was communicated to us at the last minute we did not get a chance to send the new global announcement once the service disruption started. 

 

As mentioned on our official post regarding this: we have officially taken a decision to switch the data providers and we are in the process of migrating to a new datacenter provider. We will be prioritizing the customers who are expressing their concerns and would like to be moved first. Please mention this to your account manager and will prioritize you once the move begins in the upcoming weeks. 

But you can email everyone right now and notify of the service disruption. 

[Connexium]

Sorry you have to deal with this Paul and Stephan, but when I cannot log in to support my customers, I don't see recent emails warning us about this urgent issue and calling your support line brings us to a full voicemail (that's right, no one is answering). It leaves us with a bitter aftertaste and a sense of insecurity.

 

We will now wait. Kindly keep us informed of any ETA you might have. Thank you for your assistance

[Ondrej Patrovic]

2 minutes ago, StefanMcl_Pulseway said:

 

Hi 

We always try to communicate all service disruption as in advance as possible. There was a forum, and email communication sent out earlier this week on Tuesday. But due to the Data Centre change of schedule which was communicated to us at the last minute we did not get a chance to send the new global announcement once the service disruption started. 

 

As mentioned on our official post regarding this: we have officially taken a decision to switch the data providers and we are in the process of migrating to a new datacenter provider. We will be prioritizing the customers who are expressing their concerns and would like to be moved first. Please mention this to your account manager and will prioritize you once the move begins in the upcoming weeks. 

Yeah, that's not really an excuse. As soon as you noticed a service disruption, a notice should've been sent out. I should not have to go onto a Forum for a paid product in order to find out what's going on.

[Paul]

Nobody likes such situations but everyone's frustration is warranted. We keep asking our hosting provider for an ETA but we haven't received any so far.

Our DR committee is evaluating the time it would take to restore the service in a failover DC.

-Paul

[ltintnteam]

Do you have a disaster recovery plan? If you have offsite backups, it should be trivial to spin them up on a different host (assuming you have an emergency backup host, right?) and redirect DNS.

[Paul]

3 minutes ago, ltintnteam said:

Do you have a disaster recovery plan? If you have offsite backups, it should be trivial to spin them up on a different host (assuming you have an emergency backup host, right?) and redirect DNS.

We do have a DR plan, the problem with it is that the DNS TTL is set to 24 hours. We're still evaluating our options.

-Paul

Update: The storage gateway is now online and our database cluster is starting up.

-Paul

[Paul]

Update: The database server has finished loading all the databases and the service is now restored.

-Paul

[Martin U.]

Quote

 email communication sent out earlier this week on Tuesday

How do we get on this list? I asked my team and none of us got this email. Thanks

[Anne]

2 minutes ago, Martin U. said:

How do we get on this list? I asked my team and none of us got this email. Thanks

Hi @Martin U.

If you could send me on your work email address I can have you added into the comms list, to which this email was sent out to. 

Thanks, 

Stefan