The Uncomfortable Truth About Testing Website Speed And Hosting Speed

Share the knowledge...Share on Facebook
Facebook
Tweet about this on Twitter
Twitter
Share on LinkedIn
Linkedin

one speed test is not enough - title

 

Intro

I’ve seen many times how bloggers test website speed or hosting speed. And it makes me sad.
It’s sad because the way they test it does not make sense. And it gives no value to the readers.
There are two problems:

  • The first problem is that most people run the speed tests just once or maybe a couple of times. The results of such “testing” are misleading and do more harm than good.
  • The second problem is that some people use server response time (Time-To-First-Byte) test as a substitute of speed test.



 

Running single speed test(s) makes very little sense

Let’s look inside the first problem now.

A small number of tests can be used manipulatively for drawing the conclusions which are needed. For example, one can make a number of hosting speed tests and select one or a few of them which look good and which will help to sell that host. In other words, it’s a pure manipulation.

If you have read my researches website speed and hosting speed, you already know that I make from about 100 to about 6,000 speed tests to make my conclusions for each testing instance. Only in rare cases I do less number of tests when the precision of measurements is not the point.

There is one big reason why you need to do multiple number of website or hosting speed tests if you want to get realistic results. And this reason is the speed of hosts and websites in general is not stable. Let me repeat this once again in different words – the speed of the website or the hosting you test may fluctuate enormously within a short period of time.

And if you measure it on a not typical performance peak, the speed estimate will be absolutely not realistic.

That’s why you need to make a big number of speed tests to get a realistic speed estimation.

For example, have a look at the following screenshot of one of the hosts that I monitor 24/7/365 (speed tests from each testing location run every 20 minutes):

hosting speed monitoring

Hosting speed fluctuations during a day

As you can see, the average speed measured for my test website for this 24-hour interval is 3.03 seconds. This is the average (realistic) speed during that day.

Note that if you run a single test (or a couple of single tests) there are huge chances that you’ll get into much faster or slower intervals. See the same chart with my notes:

hosting speed monitoring - slower and faster intervals

Hosting speed fluctuations – difference from the average

So, if you wanted for instance to bash or praise this hosting, you could simply run a handful of tests and select the worst (about 5 sec) or the best speed (about 2 sec) test results. So, single speed tests would give you in general from 2 to 5 seconds speed. Insane range for manipulation!

And even if your intention was not to manipulate with the presentation of the hosting speed, you need to know that just single tests are not enough since they are not reliable. You need to do at least several speed tests and analyze the results (for example, calculate the average). It will let you evaluate the speed of your website or hosting more correctly.

 

Running server response time tests instead of speed tests makes even less sense

The second problem is that instead of full page load time tests people use Time To First Byte tests.

Time To First Byte tests are more than hundred times cheaper than full page load time tests. That’s why some people use this method to evaluate hosting speed or website speed.

Full page load time tests (i.e. how fast the website loads from a visitor’s point of view) is the right way to measure the hosting speed. And Time To First Byte is just a little portion of full page load time. And Time To First Byte does not really correlate with hosting speed in general (only for specific hosts this can be so). By the way, you can see my research why Time To First Byte can not be used for measuring hosting speed.

Here is the screenshot of Time To First Byte (TTFB) tests results (uptime/TTFB tests from each testing location runs every minute):

server response time monitoring

Server response time monitoring, day chart

And the same chart but with dots instead of line representation:

Server response time monitoring - dots

Server response time monitoring, day chart

And now I zoomed the chart so that you could see single tests:

Server response time monitoring piece

Server response time monitoring (zoom)

Each dot represents one test.

As you can see, Time To First Byte can change from 1 to 3 seconds in a short period of time. And you need to run a lot of tests to calculate the realistic average response time. After all, as I mentioned above, TTFB should not be used for evaluating hosting speed in the first place at all. It’s good for measuring uptime though.

 

What the right way to test your website (hosting) speed then?

Automated constant monitoring of your website (hosting) speed

Ideally, for the most realistic and accurate speed estimations you need to do a constant monitoring of your website (hosting) speed. Test website (hosting) speed regularly as often as you can. And then calculate averages and other metrics which will tell you the information on your website and hosting performance.

You can’t do it manually.

There are a number of services which allow doing it. For example, I use monitis.com for monitoring speed of my test websites which are hosted on different hosts. The service runs speed tests every 20 minutes and uptime tests every 1 minute automatically in a non-stop manner from two US locations for deeper insights.

And each month I get the averages values of speed and some other performance metrics. This is a very good way to estimate hosting speed realistically.

Here are examples of how I use automated testing of website speed and hosting speed:

Testing the speed of your website or hosting manually

And what if you don’t want to use these (quite expensive) monitoring systems? There is an option. For one-time speed testing of your hosting or website you can use manual website speed tools. By the way, I’ve made a research on the precision and consistency of a number of these tools in this article.

Of course, using this manual method you can’t do constant monitoring of your hosting speed or website speed. But running dozens of tests can give you a good estimation of the speed.

For example I use this manual method to test hosting (website) speed when I want to compare different website configurations (e.g. with and without a particular plugin). I run the manual tests about 100 times for each configuration from each testing location.

Here are several examples of using manual method to test website speed or hosting speed:

By the way, there’s something else you need to keep in mind.

Different monitoring systems and website speed testing services can give you different results. That’s why if you want to compare different websites speed or different hosting speed you need to use the same tool or service. Because different speed tools have different environment and produce different speed results.

 

Conclusion

  1. Single speed tests that you probably run to know your website or hosting speed make little sense.
  2. Testing hosting speed or website speed is much better with the services which provide regular automated speed tests.
  3. Manual testing using website speed testing tools can work for occasional tests too. But you need to run as many as possible tests for more or less reliable results.
  4. Don’t trust website (hosting) speed tests which are done just once. Single test provide a wide field for manipulation or misleading. Besides, hosting performance (as well as website performance) can change throughout the time dramatically.
  5. Don’t trust website speed tests and hosting speed tests which are done using response time metrics (Time To First Byte). This has little to realistic website performance measurements.
  6. Comparing speed of different websites or hosts speed makes sense if you use the same testing tool. You can’t really compare the speed test results made in different speed testing tools.

You can download a PDF version of this article (320 KB):


Subscribe to my Free Researches
Work on your blog and small business more efficiently!

subscribe
BTW, I respect your privacy, and of course I don't send spam, affiliate offers or trade your emails. What I send is information that I consider useful.

Share the knowledge...Share on Facebook
Facebook
Tweet about this on Twitter
Twitter
Share on LinkedIn
Linkedin

Comments

  1. Thanks to you for sharing about website speed and hosting speed . I have learn unknown information . now we can understand what is the problem of hosting providers .

  2. All hosting providers show their own standards thats so confusing most of the time.

  3. Prashant says

    Great article Michael as usual. Want to add one more thing – one must also look at the speed that your users are getting in real world. For example, I use the site speed data from Google Analytics to check on the actual site speed real world users get on my website. Google Pagespeed Insights also gives us real world insights (provided you have sufficient traffic on your site). What do you think?

    • Thanks, Prashant, for your feedback and valuable comment!

      Yes, the real-world speed of your website that your visitors experience matters the most. Speed tests with online tools just let us estimate it this way or the other. I’d say that online speed testing tools are better suited for estimating the dynamics of website speed, not the actual website speed. I mean they are good at estimating whether website became faster or slower rather than how exactly fast or slow a website is.

      Getting to know the real-world website speed is not very trivial task. I trust webpagespeed.org the most as it gives more consistent results compared to a number of other similar tools. And I find it to provide the most realistic results too for estimating website performance. However, the speed tests with this kind of tools are just imitations of real-world users experience.

      As regards GA’s page speed analytics, it is a great source of information. Unfortunately on my website it gives inconsistent results, probably due to insufficient traffic.

It's important for me to know what you think

*