What happened on April 5th, 2023?

TL;DR: Scroll to the end of this post for a quick FAQ.



In the early morning of April 5th at 2 am, we were alerted that one of our database servers had an unusual occurrence. After an initial diagnosis, we ruled to take the server offline for a thorough inspection. As we have other database servers deployed (each with a copy of all the data), it was a routine audit done multiple times before.

Only after the audit did we see that some of our other servers started to slow down as they had information that wasn't fully synchronized. Hence, we added the server back online to sync up with its colleagues. Again, standard procedure.

However, this is where things quickly went sideways, i.e., the server started to pull complete copies of each of the other servers, causing reindexing and buffer issues on the server and the network. As we store millions of messages, we are talking TB of data here.

At this point, the whole Helpmonks team was already trying to devise a solution. We agreed on one solution: to take the server offline again so it won't impact you, as it was around 6 am.

Little did we know this made it worse, as the other servers were also looking for the data. Doing so caused a slowdown in traffic on all our servers.

Until now, we still thought that everything we did internally did not impact you, our customers. However, around 8 am, one of our support people mentioned needing help with our web interface. By then, we had started seeing massive issues with our web and application servers, which caused time-outs for our visitors.

At this point, it was already 10 am, and we were hoping to have the issue resolved without further impact for you. Unfortunately, we didn't consider all the amassed emails, which counted toward the ten thousand.

So, while our databases were "one man down," they also had to digest an immense amount of new emails and traffic from all our other servers, which were eager to deliver the messages to you as fast as possible. While we invested a lot to handle this traffic, it simply caused further slowdowns on your end, i.e., our web interface.

While we were able to re-establish the platform on April 5th, April 6th, and April 7th, our databases still needed to be fully optimized. During this time, you could sign in to Helpmonks, but some subsequent actions, timed out.

I want to apologize for this downtime to you personally. My whole team and I know how vital Helpmonks is for your business. We understand not being able to get to emails is frustrating.

However, I want to apologize even further for not being able to inform you sooner of the issues. We were all surprised when we saw that it impacted you directly. We were unaware of the impact as we only received your messages as delayed as you.

We understand that it was upsetting not to be able to reach us. To remedy this situation, we have now established a public channel on Telegram. Additionally, we've created a public group for discussions (something we have had planned anyhow).

We encourage you to join both of these channels. The channel is available at https://t.me/helpmonks and the group at https://t.me/helpmonks_group.

Please know that no emails or other data were lost during the outage.



FAQ


What happened?

One of our core database servers synchronized a faulty state which caused all database servers to adapt to that defective state. After investigation, we added it back, which inadvertently caused backlog and buffer issues.

Why was the platform unstable for a prolonged time?

We've had to turn off some services to keep all the data synced. Among them was email parsing. We also had to reduce our web to one server only. Otherwise, we would have unsynched data or, worse, missing data—something we wanted to avoid by all means.

Why did I not get a response from your team?

Simply put, we use Helpmonks in the same way as you do. We get and reply to all emails in Helpmonks. As we did not parse emails, we did not get your emails. The same applies to emails that you sent to team members directly.

What will you do to prevent this from happening again?

We were already running a large set of database servers and have extended this now. Furthermore, we set up real-time database copies to replace one which became corrupted.

Great, but what about communication?

We established a public channel on Telegram for announcements. Please join it at https://t.me/helpmonks.

In addition, we also created a public group for discussions of all kinds, e.g., questions, asking others, etc., at https://t.me/helpmonks_group.

Telegram is an independent platform with over 700 million users. You can use Telegram on every platform.


Helpmonks - retention policies
New in Helpmonks: Retention policies
We are excited to share that we have recently implemented Retention policies in Helpmonks. Retention policies allow to automatically remove conversations based on time.
Helpmonks login with a magic link
Introducing Magic Links for passwordless login
The Magic Link login process provides a seamless experience and allows for effortless login. You can now use Magic Links to log in to your Helpmonks account.
Helpmonks updates, Nov. 16th, 2023
This Helpsmonks update includes several enhancements and bug fixes we released over the last couple of weeks.
Helpmonks - public link
New in Helpmonks: Public links and vCards for Contacts
We have some exciting news to share with you. You can now share your email conversations with others and easily download contact information in vCard format.
Helpmonks updates, Nov. 10th, 2023
We continue to improve Helpmonks. Here are the most recent updates with fixes for real-time email services, email marketing, and more.
Helpmonks Email CRM updated
The new Helpmonks email CRM
Over four years ago, we added CRM capabilities to Helpmonks, allowing customers to track sales activities within their emails. The new enhanced version takes it further
Helpmonks - macOS Safari app
Helpmonks and macOS Sonoma
The latest macOS version features "Safari Apps". We are here to let you know that a Helpmonks Safari App is fully supported and works great with Helpmonks.
Enhanced compose options
We've been quite busy over the summer and can finally share with you what is enhanced and what's new. However, instead of creating some huge, long post,
Helpmonks - add many notes to emails, contacts, or companies
Multiple notes for contacts and companies
As of immediately, we enhanced the option to add a note to a contact or a company with being able to add multiple notes.
Helpmonks community for email management
The (re)launch of the Helpmonks community
Our new forum for Helpmonks customers has been (re)launched! It's the perfect place to connect about email management.
Helpmonks - assigned emails made easy
The re-design edition
We made a lot of UI enhancements to our customer engagement system in the last few weeks. Click here to read all about them.
New price plans with all features in all plans
Introducing new plans that provide more value to you
Hey everyone! I wanted to share some exciting news about our new price plans. We've been working hard to create the most flexible and affordable options for
Helpmonks - shared inbox for teams
Helpmonks Enhancements: User section and enhanced security
I am thrilled to share some updates about Helpmonks. Our team has worked tirelessly to improve your experience with Helpmonks and implement significant changes
Helpmonks - email sequences for follow-up emails
The new enhanced email sequences are here
With the enhanced email sequences in Helpmonks, you and your team can create automated follow-up emails for sales, marketing, and support.
Helpmonks - powerful file & attachment browser built in
File browser, Re-designed search & PDF and Word indexing
We are excited to announce several significant enhancements to our email management platform - a powerful email attachment and media library browser, a re-designed search
 
What can we do for you?
Found an issue or need an answer? Let us know here. We'll be happy to reply.