The Uncomfortable Truth About Testing Website Speed And Hosting Speed

 

Intro

I’ve seen many times how bloggers test website speed or hosting speed. And it makes me sad.
It’s sad because the way they test it does not make sense. And it gives no value to the readers.
There are two problems:

• The first problem is that most people run the speed tests just once or maybe a couple of times. The results of such “testing” are misleading and do more harm than good.
• The second problem is that some people use server response time (Time-To-First-Byte) test as a substitute of speed test.

 

Running single speed test(s) makes very little sense

Let’s look inside the first problem now.

A small number of tests can be used manipulatively for drawing the conclusions which are needed. For example, one can make a number of hosting speed tests and select one or a few of them which look good and which will help to sell that host. In other words, it’s a pure manipulation.

If you have read my researches website speed and hosting speed, you already know that I make from about 100 to about 6,000 speed tests to make my conclusions for each testing instance. Only in rare cases I do less number of tests when the precision of measurements is not the point.

There is one big reason why you need to do multiple number of website or hosting speed tests if you want to get realistic results. And this reason is the speed of hosts and websites in general is not stable. Let me repeat this once again in different words – the speed of the website or the hosting you test may fluctuate enormously within a short period of time.

And if you measure it on a not typical performance peak, the speed estimate will be absolutely not realistic.

That’s why you need to make a big number of speed tests to get a realistic speed estimation.

For example, have a look at the following screenshot of one of the hosts that I monitor 24/7/365 (speed tests from each testing location run every 20 minutes):

As you can see, the average speed measured for my test website for this 24-hour interval is 3.03 seconds. This is the average (realistic) speed during that day.

Note that if you run a single test (or a couple of single tests) there are huge chances that you’ll get into much faster or slower intervals. See the same chart with my notes:

So, if you wanted for instance to bash or praise this hosting, you could simply run a handful of tests and select the worst (about 5 sec) or the best speed (about 2 sec) test results. So, single speed tests would give you in general from 2 to 5 seconds speed. Insane range for manipulation!

And even if your intention was not to manipulate with the presentation of the hosting speed, you need to know that just single tests are not enough since they are not reliable. You need to do at least several speed tests and analyze the results (for example, calculate the average). It will let you evaluate the speed of your website or hosting more correctly.

 

Running server response time tests instead of speed tests makes even less sense

The second problem is that instead of full page load time tests people use Time To First Byte tests.

Time To First Byte tests are more than hundred times cheaper than full page load time tests. That’s why some people use this method to evaluate hosting speed or website speed.

Full page load time tests (i.e. how fast the website loads from a visitor’s point of view) is the right way to measure the hosting speed. And Time To First Byte is just a little portion of full page load time. And Time To First Byte does not really correlate with hosting speed in general (only for specific hosts this can be so). By the way, you can see my research why Time To First Byte can not be used for measuring hosting speed.

Here is the screenshot of Time To First Byte (TTFB) tests results (uptime/TTFB tests from each testing location runs every minute):

And the same chart but with dots instead of line representation:

And now I zoomed the chart so that you could see single tests:

Each dot represents one test.

As you can see, Time To First Byte can change from 1 to 3 seconds in a short period of time. And you need to run a lot of tests to calculate the realistic average response time. After all, as I mentioned above, TTFB should not be used for evaluating hosting speed in the first place at all. It’s good for measuring uptime though.

 

What the right way to test your website (hosting) speed then?

Automated constant monitoring of your website (hosting) speed

Ideally, for the most realistic and accurate speed estimations you need to do a constant monitoring of your website (hosting) speed. Test website (hosting) speed regularly as often as you can. And then calculate averages and other metrics which will tell you the information on your website and hosting performance.

You can’t do it manually.

There are a number of services which allow doing it. For example, I use monitis.com for monitoring speed of my test websites which are hosted on different hosts. The service runs speed tests every 20 minutes and uptime tests every 1 minute automatically in a non-stop manner from two US locations for deeper insights.

And each month I get the averages values of speed and some other performance metrics. This is a very good way to estimate hosting speed realistically.

Here are examples of how I use automated testing of website speed and hosting speed:

• My Monthly Hosting Performance Contest (and the historical data of the hosting speed tests for all the tests recorded).
• Testing A2hosting Turbo plan.
• Testing heavy-loaded theme’s speed.
• Testing caching solutions to improve website speed.
• Research on if it makes sense to use Time to First Byte to estimate website speed (I already mentioned it above).
• Research of the year 2018 on how hosting speed and Server Response Time (Time To First Byte) changes throughout the time (15 hosts tested, tests run every minute during the whole year).

Testing the speed of your website or hosting manually

And what if you don’t want to use these (quite expensive) monitoring systems? There is an option. For one-time speed testing of your hosting or website you can use manual website speed tools. By the way, I’ve made a research on the precision and consistency of a number of these tools in this article.

Of course, using this manual method you can’t do constant monitoring of your hosting speed or website speed. But running dozens of tests can give you a good estimation of the speed.

For example I use this manual method to test hosting (website) speed when I want to compare different website configurations (e.g. with and without a particular plugin). I run the manual tests about 100 times for each configuration from each testing location.

Here are several examples of using manual method to test website speed or hosting speed:

• Testing speed improvement if using Cloudflare.
• Testing cache-everything rule in Cloudflare to make website as fast as possible.
• Comparing speed of cheaper and more expensive hosts.

By the way, there’s something else you need to keep in mind.

Different monitoring systems and website speed testing services can give you different results. That’s why if you want to compare different websites speed or different hosting speed you need to use the same tool or service. Because different speed tools have different environment and produce different speed results.

 

Conclusion

1. Single speed tests that you probably run to know your website or hosting speed make little sense.
2. Testing hosting speed or website speed is much better with the services which provide regular automated speed tests.
3. Manual testing using website speed testing tools can work for occasional tests too. But you need to run as many as possible tests for more or less reliable results.
4. Don’t trust website (hosting) speed tests which are done just once. Single test provide a wide field for manipulation or misleading. Besides, hosting performance (as well as website performance) can change throughout the time dramatically.
5. Don’t trust website speed tests and hosting speed tests which are done using response time metrics (Time To First Byte). This has little to realistic website performance measurements.
6. Comparing speed of different websites or hosts speed makes sense if you use the same testing tool. You can’t really compare the speed test results made in different speed testing tools.