UpriseUp - Up
UpriseUp - Rise
UpriseUp - Up
Back to EventsBack to Blog

Quality Score on Trial: What Does the Data Tell Us?

If you were to ask me what the single most important non-conversion metric for account performance is in Google Ads, I’d probably choose Quality Score.

Google has always said that Quality Score and CPC bids are what define your ad rank, which is what defines how powerful your ads will be in the ad auction. In today’s world, with automated bidding taking so much of the CPC bidding process away from advertisers, it’s more important than ever to make sure Quality Score is as high as possible on as many keywords as possible.

But what actually is Quality Score? How does it work? Is there anything that Google aren’t telling us about it? And what can we do to ensure that our Quality Scores are as high as possible?

Well, I took every keyword that received a Quality Score in any of our Grants Accounts in 2023 , and set out to find out if there were any insights hiding inside that data. Along the way, I’m going to explain the basics of Quality Score, why it is so important, and how you can work to get the highest Quality Score possible for keywords.

So, if you want to find out who would win: 20,000 keywords or one data analyst with a spreadsheet? Keep reading…

 

Back to Basics (or, How I Lost My Mind About Expected CTR)

Before we get into the numbers, I’m going to explain the basics of Quality Score, so that the many (many) graphs that follow make sense.

Quality Score is a value assigned to a keyword, once it has received a low number of impressions. It is a value between 1 and 10, and is based on three factors: Landing Page Experience, Ad Relevance, and Expected Clickthrough Rate (CTR).

Quality Score is essentially multiplied by your bid in the ad auction, so if you doubled your Quality Score you would essentially double your competitiveness in the auction. This is why Quality Score is such a key metric, and why we want it as high as possible.

To go into a little more detail in the factors, Landing Page Experience is a measure of how well your landing page is related to your keyword, along with general landing page best practices like short load times and low link density. Ad Relevance is a measure of how relevant the ad that the keyword serves is to the keyword. Expected CTR is a measure of…

Well, Google says: “how likely it is that your ads will get clicked when shown for that keyword, irrespective of your ad’s position, assets and other ad formats that may affect the prominence and visibility of your ads.”

But this is confusing. CTR is already a measure of how often your ads get clicks; are Google saying that this is a guess about how your CTR will change? Or is it a measure of your CTR against your competitors? Or is it something else entirely?

This was a large part of the motivation to do this analysis. Expected CTR has always been a huge black box in the Quality Score system, which has very unclear definitions and no clear explanation for how to make it go up. For years I’ve sat, watching this value in my accounts go up and down without knowing why. What did I do well to make it go up? What can I do when it goes down? My answer, up until now, has been to do everything else right, and hope it follows along.

But let’s see if the numbers can tell us more than Google has.

 

Quality Score Distribution (or, Why We’re Pretty Great at Google Ads)

The first, and most basic, graph to look at is just what the Quality Score distribution looked like across the 20,000 keywords that had a Quality Score in 36 of our Grants accounts, which encompass a range of different charity causes:

graph of quality score across 20,000 keywords in Ad Grants accounts

And already we begin to see some interesting results.

Firstly, we note that 7 is the most common Quality Score in our accounts, followed by 8 and then 10. This is all in the top half of the Quality Score range, and in particular the number of perfect Quality Scores is encouraging!

More interesting than patting ourselves on the back, however, is the fact that Quality Scores 6 and 9 are relatively rare in our accounts. This is especially surprising for 6, as it’s immediately followed by the most common Quality Score. This piqued my interest – is there a reason 6 and 9 might be so unlikely? Is there something more subtle at play here that can be investigated? It is a little beyond the scope of this blog, but stay turned for a follow-up blog where we delve into this in far more detail.

But let’s get into more important graphs. First of all, let’s ask the simple question – how much does Quality Score actually affect results?

 

Performance Metrics (or, How Quality is Quality Score?)

Let’s start with CTR:

graph of quality score vs click-through rate

The overall results are as expected. CTR is higher for high Quality Scores than for low ones, but it’s interesting that CTR doesn’t really begin to increase until we reach the highest Quality Scores of 8, 9 and 10. From the description Google gives of Quality Score, we would likely expect a more linear increase from low to high.

To an extent, this is unsurprising – the higher Quality Scores have gained those Quality Scores precisely because their CTR is so high. But the fact that it has collected in the final few scores is interesting.

On to Cost per click (CPC).

 

graph of quality score vs cost-per-click (CPC)

For the most part, this one is completely as expected. The higher your Quality Score, the lower you have to bid to outperform your competition. It’s somewhat interesting that Quality Score 1 keywords were abnormally high compared to the rest of the trend. Potentially, Google will only let these low-quality keywords show if you really show you want it.

It’s also notable that the CPC for Quality Score 2, 3 and 4 keywords is relatively flat. It’s only once you hit Quality Score 5 that the decreases begin in earnest. Our expectation was always that the graph would be an even decrease across all Quality Scores, but this data suggests you have to reach the higher Quality Scores before you begin to see significant drops in cost.

But all of these graphs were just the prologue. What I was really interested in were those Quality Score factors – how do they work, and which matter the most? Let’s get into that.

 

Quality Score Factors (or, The Main Event)

Let’s start with the basics. Each subfactor can be assigned a score of Below Average, Average, or Above Average. We can plot the different amounts of these scores for each subfactor, to see the numbers of above- and below-average scores achieved.

graphs of CTR, Ad Relevance, and Landing Page Experience

The fact that the average scores above  are dominant in all metrics is unsurprising. We already saw that our overall Quality Scores are high, which would suggest good subfactor scores.

Far more interesting here is that Expected CTR deviates from the other two factors in its distribution. Both Ad Relevance and Landing Page Experience show a high number of above-average scores, and then a roughly equal, lower amount of average and below-average scores. Expected CTR, however, shows a far smaller dropoff from above-average to average, before it drops to a small number of below-average.

This tells us two things. Firstly, it reinforces what I already said at the start of the blog. Expected CTR is the least understood subfactor, and therefore the one hardest to optimise towards. That means less of a push towards above average in the score split.

The second point it proves, however, is that it is a ‘picky’ factor, more willing to drop to average even when the other two factors are high.

But maybe that is unimportant, if having a low Expected CTR has little or no effect on results. Let’s take a look at that, shall we?

 

graphs of CPC vs CTR, Ad Relevance, and Landing Page Experience

We’ll start with CPC. A reminder that here a lower value is better. Our expected result here is that higher factor scores should lead to a lower CPC, as you are able to be more competitive with a lower bid.

When we look at Expected CTR, we see what we expect. Above-average has the lowest CPC, and it increases for each following factor. But when we look at Ad Relevance and Landing Page Experience, we see a different story. Both show far less adherence to the expected trend, with Ad Relevance having average scores achieve lowest CPC, and Landing Page Experience having average achieve the highest CPC.

This indicates, at least to me, that Expected CTR has far more effect on your final CPC than the other two factors.

Let’s take a look at CTR.

graphs of CTR vs Ad Relevance and Landing Page Experience

Now this is some interesting data. When we look at CTR statistics, both Expected CTR and Landing Page Experience show what we would expect – highest CTR occurs when the factors are above average, and it decreases as the factor score gets lower. However, Ad Relevance shows no relationship to CTR at all. All the scores have similar CTRs, and above-average actually has the worst performance.

This is an indication that Ad Relevance has, at least, a lower impact on your CTR than the other two factors of Quality Score. This is a very interesting result as we would expect this to be the metric that Ad Relevance influences the most strongly. It has been proven that Ad Relevance is the ‘weakest’ Quality Score factor when it comes to calculating final Quality Score (something I’ll back up in that follow-up blog I mentioned earlier) and this data shows that the theory matches our real world performance data.

 

Quality Score Factor Association (or, Relationship Counselling)

The last thing I want to investigate in this blog is a matter of correlation. If I have an above-average score in one factor, how likely is it that I will have above-average in another factor?
I did this using pivot tables, which are a good way of comparing the correlation of two different dimensions.

comparison of factors affecting quality score

This data is quite insightful. However, of all of my visualisations, I think that this is the hardest to actually read, so let me go through this piece-by-piece.

The first two tables compare Expected CTR to Ad Relevance and Landing Page Experience. They both show broadly similar data. The values are concentrated in the first cell, where both Expected CTR and the other factor are both above-average. Next most likely is expected CTR, being average with the other factor being above-average, and then several other factors are in third place.

But this is not nearly as interesting as the question of what’s lowest likelihood. On the Ad Relevance graph, it appears quite easy to have a below-average Expected CTR, with an above-average Ad Relevance. This was the 4th most common combination in our data, and over 5 times more likely than any other Ad Relevance score getting a below-average Expected CTR. This is one of those times where the data baffles me – why is this so concentrated?

For Landing Page Experience, the below-average Expected CTR scores are far more evenly distributed between all Landing Page Experience scores. It seems that these two scores are largely independent of each other.

The final table shows Ad Relevance and Landing Page Experience against one another. Although the heatmap looks more correlated, the data here is full of little deviations. For example, you’re more likely to have a below-average Landing Page Experience than average when you have an above-average Ad Relevance, for example.

This is, again, due to the fact that this is not random Quality Score data, but data pulled from our actively managed accounts. For this reason we’re likely to see more high scores as we are actively improving Quality Score at all times.

 

Conclusion (or, No More Graphs, I Promise)

So, what are the conclusions we can glean from all these graphs? Data is only as good as what you can learn from it, and I have three main conclusions to draw:

 

  1. Quality Score Works

This might seem like an obvious conclusion, but in recent times, where Google explaining how their systems work feels increasingly like a conspiracy theorist explaining how the Earth is flat, it is reassuring to see performance data match to what Google states should happen. You should care about your Quality Score, and you should put time aside to try and improve it as much as possible. From our data, getting your Quality Score to 7 or above is ideal, as this appears to be where the biggest performance improvements occur.

 

  1. Expected CTR is the most powerful factor.

The data shows that Expected CTR is the only Quality Score factor to have a strong correlation with all performance metrics. It seems like achieving a high Exp. CTR is the best way to guarantee higher performance. However, we also saw that expected CTR seems to be the hardest factor to improve.

 

  1. Quality Score factors should be treated as separate entities.

This is the gift of the final pivot tables. We don’t see major correlation between any of the Quality Score factors, even ones we would expect to see relationships between, such as Ad Relevance and Landing Page Experience. This means that improving one factor to above-average is not necessarily going to pull other factors up along with it. If you want to improve a subfactor score, you’ll have to focus on it.

 

This is all great information, but at the end of it I found myself still frustrated. None of this tells me what Expected CTR actually is. So, I knew I wasn’t done. There was more to find. Perhaps it was time to do some real spreadsheet magic.

But that will have to wait for following blogs. In my crusade to shine a light on Expected CTR, I have at least been able to show you some interesting features of Quality Score, and how it relates to other performance metrics. Happy quality scoring!

 

    Did you enjoy this blog post?

    Share this article:

    We use Mailchimp as our marketing platform. By clicking below to subscribe, you acknowledge that your information will be transferred to Mailchimp for processing. Learn more about Mailchimp's privacy practices here.

    Contact us