Postmortem on the incoming email delay issue on January 24th and 25th, 2018

Postmortem on the incoming email delay issue on January 24th and 25th, 2018

Last Thursday morning on January 24th and Friday morning on January 25th, incoming emails to all Helpmonks mailboxes were delayed by two hours, respectively on the 25th for up to four hours. The root cause was that we a recent update to the “message id” of an email (each email has a unique message id) and the reply-to-id were not correctly set. This in return, caused messages from another Helpmonks mailbox to be accepted, and if there was an auto-reply, it created an endless loop for some emails. The endless loop created thousands of emails that came back to our mail server. As these emails got larger each time in size, it brought our parsing engine to its knees. Besides, our search server started to index all those massive emails causing a backlog in searches and an unaccounted overhead on our database. We were actively testing and working on this update for several days and felt confident to roll it out to all of our customers. However, we failed to properly account for how to parse internal emails with auto-replies correctly. In hindsight, it is clear to us that we could have avoided the problem. It’s bad enough that our customers had to experience a significant delay, but knowing that this could have been prevented is challenging to accept. I cannot express my apologies deeply enough. Even though we had issues with relaying emails to our customers promptly, we did not lose a single email, as all emails automatically get stored on different incoming servers that are independently setup. That was another major upgrade we conducted in the recent months. We work hard every day to bring the best team email collaboration platform for you. We have a weekly uptime record of 99.98% and in some weeks even a 100% uptime. The delay in delivering emails to you does not live up to our standards and we will take this incident and will discuss what we can do better to avoid this happening again. I want to thank our support team who responded to each customer individually and also to our customers who despite the issue and the inconvenience this caused showed their understanding and encouragement. If you have any questions, or if we can help in any way, please reach out to us. Thank you.

What people are saying about Helpmonks

I really appreciate you always improving the product. Makes it apleasure to use.Have a great weekend!

Opticaltel

FYI,very happy customershere at Oosterpoort, we use helpmonks on a daily basis :-)

Oosterpoort

We really enjoy working with Helpmonks every day. It made ourcommunication with our customers so much easier.Thank you.

Medology

Did you know?

Helpmonks is the only shared mailbox solution that offers multiple deployment options

Hosted Edition

Use our hosted shared mailbox software for your business. Everything is safe, secure and password-protected. Your data is safe and your privacy is respected.

Over 2,000 organizations use Helpmonks daily on our platform. Get up and running within minutes.

See why over 2000 organizations trust us

Cloud Server Edition

Want your own server with unlimited mailboxes? Our cloud servers are deployed with the highest security in mind and are GDPR compliant.

Dedicated Helpmonks shared mailbox servers are the perfect companion to your team workflow.

Explore the benefit of the shared mailbox cloud edition

Self-Hosted Edition

Scalable and enterprise ready. The Helpmonks self-hosted edition grows with your company and has been deployed by some of the largest Fortune 500 companies with millions of emails

Secure and protected. Have full control over all data.

Read on to see if the on-premise shared mailbox edition makes sense for you

One more thing...

Helpmonks gives you the ability tocollaborate as a groupin your shared mailbox, features aTeam To-Do App, powerfulCustomer Relationship Management (CRM)capabilities, integratedLive-Chatand anEmail Marketing Platform, approval control, collision detection, email tracking, reminders, mentions, custom fields, custom applications, satisfaction ratings, Single-Sign-On & SAML authentication, Trello, Zapier, Slack, Livechat, Chatra, a Developer API, and much more.

Check outthe complete feature listor sign up for afree 30-day shared inbox trial now

Ready to create your own shared inbox?
Start collaborating on team emails within minutes

Try Helpmonks for FREE for 30 days