How we solved outages and performance issues in Helpmonks

How we solved outages and performance issues in Helpmonks

At the end of March 2019 and until the middle of April 2019, we experienced outages and performance issues in Helpmonks. What follows is a report of what happened and how we solved it.

Performance issues were caused because we moved our core database to a new cloud provider with a whole service package that did not perform the same in production as during our stress tests. After several support calls, upgrades to the underlining operating system and virtualization layer, we managed to bring the performance back. However, we experienced the “Support Service” that we wanted at first hand and realized that it was not any better than our team can provide. Long story short, after two weeks of, what seems like an endless nightmare of issues, we moved everything back to our private cloud. However, we added more servers to the database which is now scaled over several servers and networks.

Outages were caused for one by the provided Load-Balancer from our cloud provider and by issues in our code when parsing incoming emails. Furthermore, we identified a problem with our distributed storage that failed under some circumstances.

The provider confirmed the issue with the Load-Balancer and we were told that no fix is coming soon. Subsequently, we had to look for an alternative. However, as this is the first entry to our web app and API, we could not just “configure” something. Besides, we wanted to continue to cache calls to static assets and also create a fail-safe set up. In the end, we settled for haproxy with a second haproxy server and keepalived, i.e., if the load-balancer should become unavailable it will switch to another one automatically.

The storage issue was fixed by using several storage servers that are clustered with glusterfs. The significant benefit of this is that the whole glusterfs cluster (no pun intended) can be mounted with a “volume file” so that the disk is mounted even when some servers in the group are not available or have a network interruption.

In addition to all the hardware and fail-over configurations, we also enhanced our code by running multiple threads within one application so that one email that might throw a parsing error is not taking down the entire parsing application. This rarely ever was an issue, but murphy’s law hit us full time and caused another outage during the time we were having the problems that we outlined above. Nowadays, we deploy multiple threads within the parsing task that guarantees that this will not happen again.

We learned a lot during this time and have made sure that every single service that you depend on is configured in a fail-over matter, is available under any circumstance, and independent if a server or a network is down.

Today, we can report that all the issues are a thing of the past and that our performance is not only back to what it was before, but surpasses all expectations.

We’re fully aware that many of our customers around the world experienced performance issues and sometimes couldn’t access their emails in Helpmonks. We are sorry about this. We know this caused a severe interruption in the daily workflow for many of you. We work diligently that this will not happen again in Helpmonks.

Thank you for being a customer, thank you for your patience during these rough times, and thank you for your understanding.

Helpmonks All-in-one Customer Service Platform
Postmortem on the incoming email delay issue on January 24th and 25th, 2018

Last Thursday morning on January 24th and Friday morning on January 25th, incoming emails to all Helpmonks mailboxes were delayed by two hours, respectively on the 25th for up to four hours. The root cause was that we a recent update to the “message id” of an email (each email…

Helpmonks All-in-one Customer Service Platform
Deployment Models for Razuna Digital Asset Management

I often get the question about how Razuna can be deployed, so I decided to do a short blog post to outline the options. Some of our enterprise customers, prefer to to have Razuna deployed on their own servers, sometimes even behind firewalls. Other customers prefer to have us handle…

Helpmonks - Profile - Where to go next
New in Helpmonks: Set where to go next for actions

Navigating your emails in your Helpmonks shared inbox just became more comfortable - you can now determine "Where to go next" after an action, e.g., set a reminder and stay on the page, or assign the conversation and go back to the email list. Previously we've had the option to…

Helpmonks All-in-one Customer Service Platform
Announcing Razuna 1.9.5

As outlined in my last post, Razuna is going strong, and we see that many customers take to Razuna, which after all is still one of the best open source alternatives to any Enterprise Digital Asset Management. Today, I’m thrilled to announce the immediate availability of Razuna 1.9.5. Razuna 1.9.5, despite…

Helpmonks Usage Report
Helpmonks Release Notes May 2019

The month of May marked the arrival of some significant enhancements to Helpmonks - shared drafts, preview icons for attachments, including a new attachment view and an enhanced PDF viewer, a brand new usage report, and performance improvement across the board. Shared Drafts As of immediately, you have the option…

Helpmonks website 2019
Helpmonks in 2019: the year in review

2019 was an outstanding year for the Helpmonks email platform. I'll take you on a short journey to highlight some of the most notable features and enhancements that arrived for Helpmonks in 2019; An enhanced Dashboard At the start of 2019, we've introduced a new way for our customers to…

Trusted by over 2,000 organizations with millions of emails

Helpmonks helps us answer customer emails at lighting speed
We couldn't do our job without Helpmonks!
Helpmonks, the best thing since sliced bread
Excellent option for collaborating email amongst your team
A good platform that is here to grow and help you grow
Helpmonks is a great tool that our company utilizes and depends on
Great shared inbox for the team
Fantastic email support service for any business
Easy to navigate and catalog data for specific users
Solid Helpdesk software w/ responsive customer support
A really good solution to manage customer support mailbox
Good support, useful features

Hosted, dedicated cloud server, or self-hosted

Helpmonks is the only vendor with a multi-deployment model

Hosted

Use our hosted shared mailbox software for your business. Everything is safe, secure and password-protected. Works with any email account and email client.

Over 2,000 organizations use Helpmonks daily on our platform. Get up and running within minutes.

See why over 2000 organizations trust us

Cloud Server

Want your own server with unlimited mailboxes? Our cloud based servers are deployed with the highest security in mind and are GDPR compliant.

Dedicated Helpmonks shared mailbox servers are the perfect companion to your team workflow.

Explore the benefit of the cloud server mailbox

Self-Hosted

Scalable and enterprise ready. The Helpmonks self-hosted edition grows with your company and has been deployed by some of the largest Fortune 500 companies with millions of emails

Secure and protected. Have full control over all data.

Check out the on-premise shared mailbox

One more thing...

Helpmonks gives you the ability tocollaborate as a groupin your shared mailbox, features aTeam To-Do App, powerfulCustomer Relationship Management (CRM)capabilities, integratedLive-Chatand Email Marketing Tools for email newsletter and automated email campaigns, collision detection, email tracking, reminders, mentions, custom fields, custom applications, customer satisfaction ratings, Single-Sign-On (SSO) & SAML authentication, Trello, Zapier, Slack, an API, and much more.

Check outthe complete feature listor sign up for afree 30-day shared inbox trial now