Postmortem on the incoming email delay issue on January 24th and 25th, 2018
Last Thursday morning on January 24th and Friday morning on January 25th, incoming emails to all Helpmonks mailboxes were delayed by two hours, respectively on the 25th for up to four hours.
The root cause was that we a recent update to the “message id” of an email (each email has a unique message id) and the reply-to-id were not correctly set. This in return, caused messages from another Helpmonks mailbox to be accepted, and if there was an auto-reply, it created an endless loop for some emails.
The endless loop created thousands of emails that came back to our mail server. As these emails got larger each time in size, it brought our parsing engine to its knees. Besides, our search server started to index all those massive emails causing a backlog in searches and an unaccounted overhead on our database.
We were actively testing and working on this update for several days and felt confident to roll it out to all of our customers. However, we failed to properly account for how to parse internal emails with auto-replies correctly.
In hindsight, it is clear to us that we could have avoided the problem. It’s bad enough that our customers had to experience a significant delay, but knowing that this could have been prevented is challenging to accept. I cannot express my apologies deeply enough.
Even though we had issues with relaying emails to our customers promptly, we did not lose a single email, as all emails automatically get stored on different incoming servers that are independently setup. That was another major upgrade we conducted in the recent months.
We work hard every day to bring the best team email collaboration platform for you. We have a weekly uptime record of 99.98% and in some weeks even a 100% uptime. The delay in delivering emails to you does not live up to our standards and we will take this incident and will discuss what we can do better to avoid this happening again.
I want to thank our support team who responded to each customer individually and also to our customers who despite the issue and the inconvenience this caused showed their understanding and encouragement.
If you have any questions, or if we can help in any way, please reach out to us. Thank you.
Features and enhancements that were added to Helpmonks in February 2019
Since the beginning of 2019, we've been working on many enhancements for your team emails. Here are the ones we made available in February. You can also follow public Helpmonks RoadMap to get updated and vote on features. Global shortcut to compose a new message Many of you like the…
Activity stream and notification panel
It’s been a while since we’ve had a chance to write on here. We’ve been working hard to build the next great destination for email collaboration. The surge of new customers in the last few months is proof that we are on the right track. Behind the scenes, we’ve been…
Helpmonks Update: Things we've added in the recent weeks
We've been busy here at Helpmonks and enhanced your email collaboration platform with many useful features. Without further ado, here are some of the most notable ones: User settings There is now an account-wide setting available to set the expiration time for your users. Also, we've added a permission setting…
Helpmonks in 2019: the year in review
2019 was an outstanding year for the Helpmonks email platform. I'll take you on a short journey to highlight some of the most notable features and enhancements that arrived for Helpmonks in 2019; An enhanced Dashboard At the start of 2019, we've introduced a new way for our customers to…
Here are a few updates you might have missed
Here at Helpmonks, we live and breathe our shared mailbox product daily, and as such, our team continues to improve and add features to the product. So much so, that we almost release something new weekly (you can check this on the Helpmonks RoadMap). Here are some enhancements you might…
What is Email Marketing and How Does it Work?
Email marketing uses email to promote products or services while developing relationships with potential customers or clients. Read about the perks of email marketing, methods to do it right, and how it compares to other marketing forms. Key Takeaways Email marketing is one of the most cost-effective ways to spread…
Trusted by over 2,000 organizations with millions of emails
Hosted, dedicated cloud server, or self-hosted
Helpmonks is the only vendor with a multi-deployment model
Hosted
Use our hosted shared mailbox software for your business. Everything is safe, secure and password-protected. Works with any email account and email client.
Over 2,000 organizations use Helpmonks daily on our platform. Get up and running within minutes.
Cloud Server
Want your own server with unlimited mailboxes? Our cloud based servers are deployed with the highest security in mind and are GDPR compliant.
Dedicated Helpmonks shared mailbox servers are the perfect companion to your team workflow.
