An update on Play-Cricket performance
Thursday 8th May 2014
As many of you will be aware we had problems with the performance of the new Play-Cricket website this weekend, for which I am sincerely sorry, as I know it caused many people considerable difficulty at a critical time when trying to upload and view your scorecards.
Starting at around 8:30pm on Saturday, some of you started to experience problems using Play-Cricket and got the infuriating automated message ‘website under heavy load’ when trying to update and view your results. Many of you gave up and tried again on Sunday morning and as a result, the same thing happened again on Sunday.
The reason this happened was due to a truly unprecedented load on the system. Over the weekend, each day we had an average of over 150,000 visitors to the site, peaking at 172,000 visitors on Sunday, and an average of 3.4 million hits a day, peaking at 4.1 million hits on Sunday. Play-Cricket last year received a total of 48 million hits across the whole of the year, so all in all we had nearly half our year’s usual traffic in one long weekend.
Because the new system is on a modern ‘virtualised platform', we were able to take steps to increase the performance of the system by adding more processor power as the problems started to occur. Unfortunately we hit the maximum amount of processor power we could apply to the system (without taking it down for a considerable time when people most wanted to use it). The good news is that we have been able to quickly add more servers to the setup this week which will massively increase the capacity of the website for this weekend and beyond.
Why couldn't we have foreseen this? We believed that we had exhaustively performance tested the system prior to launch with triple the maximum amount of visitors the system had ever received in the past. That, coupled with the additional capability to 'dial-up' the processing power at peak times, gave us the confidence that the system would cope with the heaviest of loads. However, with this week's work to add additional servers, we are looking forward to no more recurrence of the 'website under heavy load' message.
Please see below for a more detailed explanation of the numbers.
ECB Head of IT
As stated above, the reason this happened was due to a truly unprecedented load on the system. Play-Cricket last year received around 48 million hits across the whole of the year, and around 18-20% fewer per year prior to that. On Friday we received 2.2 million hits and on Saturday night we received 2.8 million hits; on Sunday we received 4.1 million hits and on Monday we also received over 3.5 million hits. On Friday we had 75,000 unique visitors, and on Saturday we had 112,000 unique visitors; on Sunday we had 172,000 unique visitors and on Monday we had 139,000 unique visitors. All in all we had nearly half our year's usual traffic in one long weekend.
Our initial thoughts on Saturday night were that we were being attacked by what is known as a 'Denial of Service Attack' but it appears that the requests to the system were all legitimate access from legitimate computers and devices. Some of the additional access may be due to the new layout of the system and we will investigate if the navigation is unnecessarily complex and simplify where necessary. A lot of the hits appear to be people having an extra look around when they accessed the site, probably because they hadn't seen it in its new guise. Finally, around 4% of the hits at the peak time on Saturday night appear to be from search engines, especially Google, attempting to index the site as new scorecards and statistics are added. We will be working with Google to manage this access over the coming days and weeks.
We have done our sums and can calculate that we will provide much more capacity than is required, but I still have a small doubt in the back of my mind that we don't know how many people didn't manage to get on to the system and hence gave up which meant that they did not figure in our statistics. I am also concerned that we may have removed one performance block only to expose another, although we have done everything in our power to mitigate this eventuality. Suffice it say that we will be watching the system like hawks this weekend and working tirelessly to avoid a repeat of last week's problems.
A long winded way of saying the system wasn't up to it.