System Updates and New Web Office

By now many of you have seen the new home page for the web office. Please feel free to use the LIVE CHAT feature (available Tuesday, March 25th) when you are needing support quickly.

History of System Fixes

Our servers and systems have been completely stabilized this week. Here is the chronology of the fixes:

There have been 3 facts in the recent server outages:

Mass Mailing DB problem
High Memory Usage problem
GSI’s maintanence – ~4hrs of downtime on Sunday.

Only problem 2 was affected by the themes and layouts release and only marginally. Items 1 and 2 just happened to coincide with the timing of the release.

Timeline

3rd March:

Major release – including Shop Theme and Layouts, Financial Reports, etc. It is noted that problem 1 was seen prior to the release; however it was exacerbated by the release.

11th March:

We released a patch release to reduce the extent of our caching and thus reduce our memory usage and therefore reduce the likelihood of problem 2. We note a reduction memory related crashes.

14th March:

We were able to reproduce problem 1 and thus were able to identified that the root cause was that the enormous size of our mailing recipients table. The size of this table combine with attempts to send new large mailings caused excessively heavy load on the database server. This caused other database requests to slower down so severely that the rate of processing requests was exceeded by the rate of incoming requests. This in turn caused the web server to error reporting the server to be too busy. Having identified this problem we immediately took measure to mitigate this problem, without stopping customers sending large mailing. The cost of this fix is that historical record of who was sent each mailing is temporarily unavailable. We will be putting a new more scalable architecture in place of the coming days.

16th March:

GSI, who host our servers, performed some scheduled maintenance, which we were told was estimated to cause “2-4 periods of less than 10 minutes each”. In fact GSI suffered some difficulties during this time which resulted in approximately 4hrs of downtime, which was completely outside of Endis’ control. This is problem 3.

17th March:

We released a patch to the web server to reduce ongoing memory problems. This patch should further reduce instances of problem 2.

Current situation:

The servers are now in as good, if not better, shape than they were before Major release of 3rd March. Problem 1 is past and should not be seen again. We are working to re-introduce the mail history in a scalable way. The frequency of crashes due to problem 2 has significantly reduce, there is continues to be ongoing development work to further optimise the servers performance. Problem 3 is also past and we are not aware of any future GSI maintenance at this time.

Can’t find what you’re looking for? Try our user community.