Tag Archive | "Keyword"

Make sense of your data with these essential keyword segments

Posted by TheMozTeam

This blog post was originally published on the STAT blog.


The first step to getting the most out of your SERP data is smart keyword segmentation — it surfaces targeted insights that will help you make data-driven decisions.

But knowing what to segment can feel daunting, especially when you’re working with thousands of keywords. That’s why we’re arming you with a handful of must-have tags.

Follow along as we walk through the different kinds of segments in STAT, how to create them, and which tags you’ll want to get started with. You’ll be a fanciful segment connoisseur by the time we’re through!

Segmentation in STAT

In STAT, keyword segments are called “tags” and come as two different types: standard or dynamic.

Standard tags are best used when you want to keep specific keywords grouped together because of shared characteristics — like term (brand, product type, etc), location, or device. Standard tags are static, so the keywords that populate those segments won’t change unless you manually add or remove them.

Dynamic tags, on the other hand, are a fancier kind of tag based on filter criteria. Just like a smart playlist, dynamic tags automatically populate with all of the keywords that meet said criteria, such as keywords with a search volume over 500 that rank on page one. This means that the keywords in a dynamic tag aren’t forever — they’ll filter in and out depending on the criteria you’ve set.

How to create a keyword segment

Tags are created in a few easy steps. At the Site level, pop over to the Keywords tab, click the down arrow on any table column header, and then select Filter keywords. From there, you can select the pre-populated options or enter your own metrics for a choose-your-own-filter adventure.

Once your filters are in place, simply click Tag All Filtered Keywords, enter a new tag name, and then pick the tag type best suited to your needs — standard or dynamic — and voila! You’ve created your very own segment.

Segments to get you started

Now that you know how to set up a tag, it’s time to explore some of the different segments you can implement and the filter criteria you’ll need to apply.

Rank and rank movement

Tracking your rank and ranking movements with dynamic tags will give you eyeballs on your keyword performance, making it easy to monitor and report on current and historical trends.

There’s a boatload of rank segments you can set up, but here’s just a sampling to get you started:

  • Keywords ranking in position 1–3; this will identify your top performing keywords.
  • Keywords ranking in position 11–15; this will suss out the low-hanging, top of page two fruit in need of a little nudge.
  • Keywords with a rank change of 10 or more (in either direction); this will show you keywords that are slipping off or shooting up the SERP.

Appearance and ownership of SERP features

Whether they’re images, carousels, or news results, SERP features have significantly altered the search landscape. Sometimes they push you down the page and other times, like when you manage to snag one, they can give you a serious leg up on the competition and drive loads more traffic to your site.

Whatever industry-related SERP features that you want to keep apprised of, you can create dynamic tags that show you the prevalence and movement of them within your keyword set. Segment even further for tags that show which keywords own those features and which have fallen short.

Below are a few segments you can set up for featured snippets and local packs.

Featured snippets

Everyone’s favourite SERP feature isn’t going anywhere anytime soon, so it wouldn’t be a bad idea to outfit yourself with a snippet tracking strategy. You can create as many tags as there are snippet options to choose from:

  • Keywords with a featured snippet.
  • Keywords with a paragraph, list, table, and/or carousel snippet.
  • Keywords with an owned paragraph, list, table, and/or carousel snippet.
  • Keywords with an unowned paragraph, list, table, and/or carousel snippet.

The first two will allow you to see over-arching snippet trends, while the last two will chart your ownership progress.

If you want to know the URL that’s won you a snippet, just take a peek at the URL column.

Local packs

If you’re a brick and mortar business, we highly advise creating tags for local packs since they provide a huge opportunity for exposure. These two tags will show you which local packs you have a presence in and which you need to work on

  • Keywords with an owned local pack.
  • Keywords with an unowned local pack.

Want all the juicy data squeezed into a local pack, like who’s showing up and with what URL? We created the Local pack report just for that.

Landing pages, subdomains, and other important URLs

Whether you’re adding new content or implementing link-building strategies around subdomains and landing pages, dynamic tags allow you to track and measure page performance, see whether your searchers are ending up on the pages you want, and match increases in page traffic with specific keywords.

For example, are your informational intent keywords driving traffic to your product pages instead of your blog? To check, a tag that includes your blog URL will pull in each post that ranks for one of your keywords.

Try these three dynamic tags for starters:

  • Keywords ranking for a landing page URL.
  • Keywords ranking for a subdomain URL.
  • Keywords ranking for a blog URL.

Is a page not indexed yet? That’s okay. You can still create a dynamic tag for its URL and keywords will start appearing in that segment when Google finally gets to it.

Location, location, location

Google cares a lot about location and so should you, which is why keyword segments centred around location are essential. You can tag in two ways: by geo-modifier and by geo-location.

For these, it’s better to go with the standard tag as the search term and location are fixed to the keyword.

Geo-modifier

A geo-modifier is the geographical qualifier that searchers manually include in their query — like in [sushi near me]. We advocate for adding various geo-modifiers to your keywords and then incorporating them into your tagging strategy. For instance, you can segment by:

  • Keywords with “in [city]” in them.
  • Keywords with “near me” in them.

The former will show you how you fare for city-wide searches, while the latter will let you see if you’re meeting the needs of searchers looking for nearby options.


Geo-location

Geo-location is where the keyword is being tracked. More tracked locations mean more searchers’ SERPs to sample. And the closer you can get to searchers standing on a street corner, the more accurate those SERPs will be. This is why we strongly recommend you track in multiple pin-point locations in every market you serve.

Once you’ve got your tracking strategy in place, get your segmentation on. You can filter and tag by:

  • Keywords tracked in specific locations; this will let you keep tabs on geographical trends.
  • Keywords tracked in each market; this will allow for market-level research.

Search volume & cost-per-click

Search volume might be a contentious metric thanks to Google’s close variants, but having a decent idea of what it’s up to is better than a complete shot in the dark. We suggest at least two dynamic segments around search volume:

  • Keywords with high search volume; this will show which queries are popular in your industry and have the potential to drive the most traffic.
  • Keywords with low search volume; this can actually help reveal conversion opportunities — remember, long-tail keywords typically have lower search volumes but higher conversion rates.

Tracking the cost-per-click of your keywords will also bring you and your PPC team tonnes of valuable insights — you’ll know if you’re holding the top organic spot for an outrageously high CPC keyword.

As with search volume, tags for high and low CPC should do you just fine. High CPC keywords will show you where the competition is the fiercest, while low CPC keywords will surface your easiest point of entry into the paid game — queries you can optimize for with less of a fight.

Device type

From screen size to indexing, desktop and smartphones produce substantially different SERPs from one another, making it essential to track them separately. So, filter and tag for:

  • Keywords tracked on a desktop.
  • Keywords tracked on a smartphone.

Similar to your location segments, it’s best to use the standard tag here.

Go crazy with multiple filters

We’ve shown you some really high-level segments, but you can actually filter down your keywords even further. In other words, you can get extra fancy and add multiple filters to a single tag. Go as far as high search volume, branded keywords triggering paragraph featured snippets that you own for smartphone searchers in the downtown core. Phew!

Want to make talk shop about segmentation or see dynamic tags in action? Say hello (don’t be shy) and request a demo.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!


Moz Blog

Posted in Latest NewsComments Off

How to Identify and Tackle Keyword Cannibalization in 2019

Posted by SamuelMangialavori

If you read the title of this blog and somehow, even only for a second, thought about the iconic movie “The Silence of the Lambs”, welcome to the club — you are not alone!

Despite the fact that the term “cannibalization” does not sound very suitable for digital marketing, this core concept has been around for a long time. This term simply identifies the issue of having multiple pages competing for the same (or very similar) keywords/keyword clusters, hence the cannibalization.

What do we mean by cannibalization in SEO?

This unfortunate and often unnoticed problem harms the SEO potential of the pages involved. When more than one page has the same/similar keyword target, it creates “confusion” in the eyes of the search engine, resulting in a struggle to decide what page to rank for what term.

For instance, say my imaginary e-commerce website sells shoes online and I have created a dedicated category page that targets the term ‘ankle boots’: www.distilledshoes.com/boots/ankle-boots/

Knowing the importance of editorial content, over time I decide to create two blog posts that cover topics related to ankle boots off the back of a keyword research: one post on how to wear ankle boots and another about the top 10 ways to wear ankle boots in 2019:

One month later, I realize that some of my blog pages are actually ranking for a few key terms that my e-commerce category page was initially visible for.

Now the question is: is this good or bad for my website?

Drum roll, please…and the answer is — It depends on the situation, the exact keywords, and the intent of the user when searching for a particular term.

Keyword cannibalization is not black or white — there are multiple grey areas and we will try and go though several scenarios in this blog post. I recommend you spend 5 minutes checking this awesome Whiteboard Friday which covers the topic of search intent extremely well.

How serious of a problem is keyword cannibalization?

Much more than what you might think — almost every website that I have worked on in the past few years have some degree of cannibalization that needs resolving. It is hard to estimate how much a single page might be held back by this issue, as it involves a group of pages whose potential is being limited. So, my suggestion is to treat this issue by analyzing clusters of pages that have some degree of cannibalization rather than single pages.

Where is most common to find cannibalization problems in SEO?

Normally, you can come across two main placements for cannibalization:

1) At meta data level:

When two or more pages have meta data (title tags and headings mainly) which target the same or very similar keywords, cannibalization occurs. This requires a less labour-intensive type of fix, as only meta data needs adjusting.

For example: my e-commerce site has three boots-related pages, which have the following meta data:

Page URL Title tag Header 1
/boots/all /Women’s Boots – Ankle & Chelsea Boots | Distilled Shoes Women’s Ankle & Chelsea Boots
/boots/ankle-boots/ Women’s Ankle Boots | Distilled Shoes Ankle Boots
boots/chelsea-boots/ Women’s Chelsea Boots | Distilled Shoes Chelsea Boots

These types of keyword cannibalization often occurs on e-commerce sites which have many category (or subcategory) pages with the intention to target specific keywords, such as the example above. Ideally, we would want to have a generic boots page to target generic boots related terms, while the other two pages should be focusing on the specific types of boots we are selling on those pages: ankle and chelsea.

Why not try the below instead?

Page URL New Title Tag New Header 1
/boots/all Women’s Boots – All Types of Winter Boots | Distilled Shoes Women’s Winter Boots
/boots/ankle-boots/ Women’s Ankle Boots | Distilled Shoes Ankle Boots
boots/chelsea-boots/ Women’s Chelsea Boots | Distilled Shoes Chelsea Boots

More often than not, we fail to differentiate our e-commerce site’s meta data to target the very specific subgroup of keywords that we should aim for — after all, this is the main point of having so many category pages, no? If interested in the topic, find here a blog post I wrote on the subject.

The fact that e-commerce pages tend to have very little text on them makes meta data very important, as it will be one of the main elements search engines look at to understand how a page differs from the other.

2) At page content level

When cannibalization occurs at page content level (meaning two or more pages tend to cover very similar topics in their body content), it normally needs more work than the above example, since it requires the webmaster to first find all the competing pages and then decide on the best approach to tackle the issue.

For example: say my e-commerce has two blog pages which cover the following topics:

Page URL Objective of the article
/blog/how-to-clean-leather-boots/ Suggests how to take care of leather boots so they last longer
/blog/boots-cleaning-guide-2019/ Shows a 121 guide on how to clean different types of boots

These types of keyword cannibalization typically occurs on editorial pages, or transactional pages provided with substantial amount of text.

It is fundamental to clarify something: SEO is often not the main driver when producing editorial content, as different teams are involved in producing content for social and engagement reasons, and fairly so. Especially in larger corporations, it is easy to underestimate how complex it is to find a balance between all departments and how easily things can be missed.

From a pure SEO standpoint, I can assure you that the two pages above are very likely to be subject to cannibalization. Despite the fact they have different editorial angles, they will probably display some degree of duplicated content between them (more on this later).

In the eyes of a search engine, how different are these two blog posts, both of which aim to address a fairly similar intent? That is the main question you should ask yourself when going through this task. My suggestion is the following: Before investing time and resources into creating new pages, make the effort to review your existing content.

What are the types of cannibalization in SEO?

Simply put, you could come across 2 main types:

1) Two or more landing pages on your website that are competing for the same keywords

For instance, it could be the case that, for the keyword “ankle boots”, two of my pages are ranking at the same time:

Page URL Title tag Ranking for the keyword “ankle boots”
Page A: /boots/all Women’s Boots – Ankle & Chelsea Boots | Distilled Shoes Position 8
Pabe B: /boots/ankle-boots/ Women’s Ankle Boots | Distilled Shoes Position 5

Is this a real cannibalization issue? The answer is both yes and no.

If multiple pages are ranking for the same term, it is because a search engine finds elements of both pages that they think respond to the query in some way — so technically speaking, they are potential ‘cannibals’.

Does it mean you need to panic and change everything on both pages? Surely not. It very much depends on the scenario and your objective.

Scenario 1

In the instances where both pages have really high rankings on the first page of the SERPS, this could work in your advantage: More space occupied means more traffic for your pages, so treat it as “good” cannibalization.

If this is the case, I recommend you do the following:

  • Consider changing the meta descriptions to make them more enticing and unique from each other. You do not want both pages to show the same message and fail to impress the user.
  • In case you realise that amongst the two pages, the “secondary/non-intended page” is ranking higher (for example: Page A /boots/all ranks higher than Page B /boots/ankle-boots/ for the term ‘ankle boots’), you should check on Google Search Console (GSC) to see which page is getting the most amount of clicks for that single term. Then, decide if it is worth altering other elements of your SEO to better address that particular keyword.

For instance, what would happen if I removed the term ankle boots from my /boots/all (Page A) title tag and page copy? If Google reacts by favouring my /boots/ankle-boots/ page instead (Page B), which may gain higher positions, then great! If not, the worst case scenario is you can revert the changes back and keep enjoying the two results on page one of the SERP.

Page URL Title tag Ranking for the keyword “ankle boots”
Page A: /boots/all Women’s Boots – Chelsea Boots & many more types | Distilled Shoes Test and decide

Scenario 2

In the instances where page A has high rankings page one of the SERPS and page B is nowhere to be seen (beyond the top 15–20 results), it is up to you to decide if this minor cannibalization is worth your time and resources, as this may not be an urgency.

If you decide that it is worth pursuing, I recommend you do the following:

  • Keep monitoring the keywords for which the two pages seem to show, in case Google might react differently in the future.
  • Come back to this minor cannibalization point after you have addressed your most important issues.

Scenario 3

In the instances where both pages are ranking in page two or three of the SERP, then it might be the case that your cannibalization is holding one or both of them back.

If this is the case, I recommend you do the following:

  • Check on GSC to see which of your pages is getting the most amount of clicks for that single keyword. You should also check on similar terms, since keywords on page two or three of the SERP will show very low clicks in GSC. Then, decide which page should be your primary focus — the one that is better suited from a content perspective — and be open to test changes for on-page SEO elements of both pages.
  • Review your title tags, headings, and page copies and try to find instances where both pages seem to overlap. If the degree of duplication between them is really high, it might be worth consolidating/canonicalising/redirecting one to the other (I’ll touch on this below).

2) Two or more landing pages on your website that are flip-flopping for the same keyword

It could be the case that, for instance, the keyword “ankle boots” for two of my pages are ranking at different times, as Google seems to have a difficult time deciding which page to choose for the term.

Page URL Ranking for the keyword “ankle boots” on 1st of January Ranking for the keyword “ankle boots” on 5th of January
Page A: /boots/all Position 6 Not ranking
Pabe B: /boots/ankle-boots/ Not ranking Position 8

If this happens to you, try and find an answer to the following questions:This is a common issue that I am sure many of you have encountered, in which landing pages seem to be very volatile and rank for a group of keywords in a non-consistent manner.

When did this flip-flopping start?

Pinpointing the right moment in time where this all began might help you understand how the problem originated in the first place. Maybe a canonical tag occurred or went missing, maybe some changes to your on-page elements or an algorithm update mixed things up?

How many pages flip-flop between each other for the same keyword?

The fewer pages subject to volatility, the better and easier to address. Try to identify which pages are involved and inspect all elements that might have triggered this instability.

How often do these pages flip-flop?

Try and find out how often the ranking page for a keyword has changed: the fewer times, the better. Cross reference the time of the changes with your knowledge of the site in case it might have been caused by other adjustments.

If the flip-flop has occurred only once and seems to have stopped, there is probably nothing to worry about, as it’s likely a one-off volatility in the SERP. At the end of the day, we need to remember that Google runs test and changes almost everyday.

How to identify which pages are victims of cannibalization

I will explain what tools I normally use to detect major cannibalization fluxes, but I am sure there are several ways to reach the same results — if you want to share your tips, please do comment below!

Tools to deploy for type 1 of cannibalization: When two of more landing pages are competing for the same keyword

I know we all love tools that help you speed up long tasks, and one of my favorites is Ahrefs. I recommend using their fantastic method which will find your ‘cannibals’ in minutes.

Watch their five minute video here to see how to do it.

I am certain SEMrush, SEOMonitor, and other similar tools offer the same ability to retrieve that kind of data, maybe just not as fast as Ahrefs’ method listed above. If you do not have any tools at your disposal, Google Search Console and Google Sheets will be your friends, but it will be more of a manual process.

Tools to deploy for Type 2 of cannibalization: When two or more landing pages are flip-flopping for the same keyword

Ideally, most rank tracking tools will be able to do this functionally discover when a keyword has changed ranking URL over time. Back in the day I used tracking tools like Linkdex and Pi Datametrics to do just this.

At Distilled, we use STAT, which displays this data under History, within the main Keyword tab — see screenshot below as example.

One caveat of these kinds of ranking tools is that this data is often accessible only by keyword and will require data analysis. This means it may take a bit of time to check all keywords involved in this cannibalization, but the insights you’ll glean are well worth the effort.

Google Data Studio Dashboard

If you’re looking for a speedier approach, you can build a Google Data Studio dashboard that connects to your GSC to provide data in real time, so you don’t have to check on your reports when you think there is a cannibalization issue (credit to my colleague Dom).

Our example of a dashboard comprises two tables (see screenshots below):

The table above captures the full list of keyword offenders for the period of time selected. For instance, keyword ‘X’ at the top of the table has generated 13 organic clicks (total_clicks) from GSC over the period considered and changed ranking URL approximately 24 times (num_of_pages).

The second table (shown above) indicates the individual pages that have ranked for each keyword for the period of time selected. In this particular example, for our keyword X (which, as we know, has changed URLs 24 times in the period of time selected) the column path would show the list of individual URLs that have been flip flopping.

What solutions should I implement to tackle cannibalization?

It is important to distinguish the different types of cannibalization you may encounter and try to be flexible with solutions — not every fix will be the same.

I started touching on possible solutions when I was talking about the different types of cannibalization, but let’s take a more holistic approach and explain what solutions are available.

301 redirection

Ask yourself this question: do I really need all the pages that I found cannibalizing each other?

In several instances the answer is no, and if that is the case, 301 redirects are your friends.

For instance, you might have created a new (or very similar) version of the same article your site posted years ago, so you may consider redirecting one of them — generally speaking, the older URL might have more equity in the eyes of search engines and potentially would have attracted some backlinks over time.

Page URL Date of blog post
Page A: blog/how-to-wear-ankle-boots May 2016
Page B: blog/how-to-wear-ankle-boots-in-2019 December 2018

Check if page A has backlinks and, if so, how many keywords it is ranking for (and how well it is ranking for those keywords)

What to do:

  • If page A has enough equity and visibility, do a 301 redirect from page B to page A, change all internal links (coming from the site to page B) to page A, and update metadata of page A if necessary (including the reference of 2019 for instance)
  • If not, do the opposite: complete a 301 redirect from page A to page B and change all internal links (coming from the site to page A) to page B.

Canonicalization

In case you do need all the pages that are cannibalizing for whatever reason (maybe PPC, social, or testing purposes, or maybe it is just because they are still relevant) then canonical tags are your friends. The main difference with a 301 redirect is that both pages will still exist, while the equity from page A will be transferred to page B.

Let’s say you created a new article that covers a similar topic to another existing one (but has a different angle) and you find out that both pages are cannibalizing each other. After a quick analysis, you may decide you want Page B to be your “primary”, so you can use a canonical tag from page A pointing to page B. You would want to use canonicalization if the content of the two pages is diverse enough that users should see it but not so much that search engines should think it’s different.

Page URL Date of blog post
Page A: blog/how-to-wear-ankle-boots-with-skinny-jeans December 2017
Page B: blog/how-to-wear-high-ankle-boots January 2019

What to do:

  • Use a canonical tag from page A to page B. As a reinforcement to Google, you could also use a self-referencing canonical tag on page B.
  • After having assessed accessibility and internal link equity of both pages, you may want to change all/some internal links (coming from the site to page A) to page B if you deem it useful.

Pages re-optimisation

As already touched on, it primarily involves a metadata type of cannibalization, which is what I named as type 1 in this article. After identifying the pages whose meta data seem to overlap or somehow target the same/highly similar keywords, you will need to decide which is your primary page for that keyword/keyword group and re-optimize the competing pages.

See the example earlier in the blog post to get a better idea.

Content consolidation

This type of solution involves consolidating a part or the entire content of a page into another. Once that has happened, it is down to you to decide if it is worth keeping the page you have stripped content from or just 301 redirect it to the other.

You would use consolidation as an option if you think the cannibalization is a result of similar or duplicated content between multiple pages, which is more likely to be the type 2 of cannibalization, as stated earlier. It is essential to establish your primary page first so you are able to act on the competing internal pages. Content consolidation requires you to move the offending content to your primary page in order to stop this problem and improve your rankings.

For example, you might have created a new article that falls under a certain content theme (in this instance, boots cleaning). You then realize that a paragraph of your new page B touches on leather boots and how to take care of them, which is something you have covered in page A. In case both articles respond to similar intents (one targeting cleaning leather only, the other targeting cleaning boots in general), then it is worth consolidating the offending content from page B to page A, and add an internal link to page A instead of the paragraph that covers leather boots in page B.

Page URL Date of blog post
Page A: blog/how-to-clean-leather-boots December 2017
Page B: /blog/boots-cleaning-guide-2019/ January 2019

What to do:

  • Find the offending part of content on page B, review it and consolidate the most compelling bits to page A
  • Replace the stripped content on page B with a direct internal link pointing to page A
  • Often after having consolidated the content of a page to another, there is no scope for the page where content has been stripped from so it should just be redirected (301).

How can I avoid cannibalization in the first place?

The best way to prevent cannibalization from happening is a simple, yet underrated task, that involves keyword mapping. Implementing a correct mapping strategy for your site is a key part of your SEO, as important as your keyword research.

Carson Ward has written an awesome Moz Blog post about the topic, I recommend you have a look.

Don’t take ‘intent’ for granted

Another way to avoid cannibalization, and the last tip I want to share with you, involves something most of you are familiar with: search intent.

Most of the time, we take things for granted, assuming Google will behave in a certain way and show certain type of results. What I mean by this is: When you work on your keyword mapping, don’t forget to check what kind of results search engines display before assuming a certain outcome. Often, even Google is not sure and will not always get intent right.

For instance, when searching for ‘shoes gift ideas’ and ‘gift ideas for shoe lovers’ I get two very different SERPs despite the fact that my intent is kind of the same: I am looking for ideas for a gift which involves shoes.

The SERP on the left shows a SERP for a query of “shoes gift ideas”. It displays a row of pictures from Google Images with the link to see more, one editorial page (informational content), and then the rest of results are all transactional/e-commerce pages for me to buy from. Google has assumed that I’d like to see commercial pages as I might be close to a conversion.

The SERP on the right shows a SERP for a query of “gift ideas for show loves”, displaying a row of Google Shopping ads and then a featured snippet, taken from an editorial page, while the rest are a mix of transactional and editorial pages, with Pinterest ranking twice in the top 10. Clearly Google is not sure what I would prefer to see here. Am I still in the consideration phase or am I moving to conversion?

The example above is just one of the many I encountered when going through my keyword research and mapping task. Before going after a certain keyword/keyword cluster, try and address all these points:

  • Check if one of your existing pages has already covered it.
  • If so, how well have you covered the keyword target? What can you do to improve my focus? Is there any cannibalization that is holding you back?
  • If you do not have a page for it, is it worth creating one and what implications will it have on your existing pages?
  • Check what results Google is displaying for that keyword target, as it might be different from your expectations.
  • Once you have created a new page/s, double check this has not created unintentional and unplanned cannibalization further down the line by using the tips in this post.

Conclusion

Keyword cannibalization is an underrated, but rather significant, problem, especially for sites that have been running for several years and end up having lots of pages. However, fear not — there are simple ways to monitor this issue and hopefully this post can help you speed up the whole process to find such instances.

Most of the times, it is just a matter of using the most logical approach while considering other SEO elements such as backlinks, crawlability, and content duplication. If possible, always test your changes first before applying it at site-wide level or making them permanent.

If you, like me, are a fan of knowledge sharing and you think there are better ways to help with cannibalization, please comment below!

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!


Moz Blog

Posted in Latest NewsComments Off

The Basics of Building an Intent-Based Keyword List

Posted by TheMozTeam

This post was originally published on the STAT blog.


In this article, we’re taking a deep dive into search intent.

It’s a topic we’ve covered before with some depth. This STAT whitepaper looked at how SERP features respond to intent, and a few bonus blog posts broke things down even further and examined how individual intent modifiers impact SERP features, the kind of content that Google serves at each stage of intent, and how you can set up your very own search intent projects. (And look out for Seer’s very own Scott Taft’s upcoming post this week on how to use STAT and Power BI to create your very own search intent dashboard.)

Search intent is the new demographics, so it only made sense to get up close and personal with it. Of course, in order to bag all those juicy search intent tidbits, we needed a great intent-based keyword list. Here’s how you can get your hands on one of those.

Gather your core keywords

First, before you can even think about intent, you need to have a solid foundation of core keywords in place. These are the products, features, and/or services that you’ll build your search intent funnel around.

But goodness knows that keyword list-building is more of an art than a science, and even the greatest writers (hi, Homer) needed to invoke the muses (hey, Calliope) for inspiration, so if staring at your website isn’t getting the creative juices flowing, you can look to a few different places for help.

Snag some good suggestions from keyword research tools

Lots of folks like to use the Google Keyword Planner to help them get started. Ubersuggest and Yoast’s Google Suggest Expander will also help add keywords to your arsenal. And Answer The Public gives you all of that, and beautifully visualized to boot.

Simply plunk in a keyword and watch the suggestions pour in. Just remember to be critical of these auto-generated lists, as odd choices sometimes slip into the mix. For example, apparently we should add [free phones] to our list of [rank tracking] keywords. Huh.

Spot inspiration on the SERPs

Two straight-from-the-SERP resources that we love for keyword research are the “People also ask” box and related searches. These queries are Google-vetted and plentiful, and also give you some insight into how the search engine giant links topics.

If you’re a STAT client, you can generate reports that will give you every question in a PAA box (before it gets infinite), as well as each of the eight related searches at the bottom of a SERP. Run the reports for a couple of days and you’ll get a quick sense of which questions and queries Google favours for your existing keyword set.

A quick note about language & location

When you’re in the UK, you push a pram, not a stroller; you don’t wear a sweater, you wear a jumper. This is all to say that if you’re in the business of global tracking, it’s important to keep different countries’ word choices in mind. Even if you’re not creating content with them, it’s good to see if you’re appearing for the terms your global searchers are using.

Add your intent modifiers

Now it’s time to tackle the intent bit of your keyword list. And this bit is going to require drawing some lines in the sand because the modifiers that occupy each intent category can be highly subjective — does “best” apply transactional intent instead of commercial?

We’ve put together a loose guideline below, but the bottom line is that intent should be structured and classified in a way that makes sense to your business. And if you’re stuck for modifiers to marry to your core keywords, here’s a list of 50+ to help with the coupling.

Informational intent

The searcher has identified a need and is looking for the best solution. These keywords are the core keywords from your earlier hard work, plus every question you think your searchers might have if they’re unfamiliar with your product or services.

Your informational queries might look something like:

  • [product name]
  • what is [product name]
  • how does [product name] work
  • how do I use [product name]

Commercial intent

At this stage, the searcher has zeroed in on a solution and is looking into all the different options available to them. They’re doing comparative research and are interested in specific requirements and features.

For our research, we used best, compare, deals, new, online, refurbished, reviews, shop, top, and used.

Your commercial queries might look something like:

  • best [product name]
  • [product name] reviews
  • compare [product name]
  • what is the top [product name]
  • [colour/style/size] [product name]

Transactional intent (including local and navigational intent)

Transactional queries are the most likely to convert and generally include terms that revolve around price, brand, and location, which is why navigational and local intent are nestled within this stage of the intent funnel.

For our research, we used affordable, buy, cheap, cost, coupon, free shipping, and price.

Your transactional queries might look something like:

  • how much does [product name] cost
  • [product name] in [location]
  • order [product name] online
  • [product name] near me
  • affordable [brand name] [product name]

A tip if you want to speed things up

A super quick way to add modifiers to your keywords and save your typing fingers is by using a keyword mixer like this one. Just don’t forget that using computer programs for human-speak means you’ll have to give them the ol’ once-over to make sure they still make sense.

Audit your list

Now that you’ve reached for the stars and got yourself a huge list of keywords, it’s time to bring things back down to reality and see which ones you’ll actually want to keep around.

No two audits are going to look the same, but here are a few considerations you’ll want to keep in mind when whittling your keywords down to the best of the bunch.

  1. Relevance. Are your keywords represented on your site? Do they point to optimized pages
  2. Search volume. Are you after highly searched terms or looking to build an audience? You can get the SV goods from the Google Keyword Planner.
  3. Opportunity. How many clicks and impressions are your keywords raking in? While not comprehensive (thanks, Not Provided), you can gather some of this info by digging into Google Search Console.
  4. Competition. What other websites are ranking for your keywords? Are you up against SERP monsters like Amazon? What about paid advertising like shopping boxes? How much SERP space are they taking up? Your friendly SERP analytics platform withshare of voice capabilities (hi!) can help you understand your search landscape.
  5. Difficulty. How easy is your keyword going to be to win? Search volume can give you a rough idea — the higher the search volume, the stiffer the competition is likely to be — but for a different approach, Moz’s Keyword Explorer has a Difficulty score that takes Page Authority, Domain Authority, and projected click-through-rate into account.

By now, you should have a pretty solid plan of attack to create an intent-based keyword list of your very own to love, nurture, and cherish.

If, before you jump headlong into it, you’re curious what a good chunk of this is going to looks like in practice, give this excellent article by Russ Jones a read, or drop us a line. We’re always keen to show folks why tracking keywords at scale is the best way to uncover intent-based insights.

Read on, readers!

More in our search intent series:

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!


Moz Blog

More Website Articles

Posted in Latest NewsComments Off

SearchCap: Quora keyword targets, Angular SEO & old stock SEO

Below is what happened in search today, as reported on Search Engine Land and from other places across the web.



Please visit Search Engine Land for the full article.


Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Posted in Latest NewsComments Off

Quora adds search-like keyword targeting, Auction Insights for advertisers

Keyword targeting offers the intent targeting of search and can be more granular than topic targeting. Auction Insights offer competitive visibility.



Please visit Search Engine Land for the full article.


Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Find More Articles

Posted in Latest NewsComments Off

Evolving Keyword Research to Match Your Buyer’s Journey

Posted by matthew_jkay

Keyword research has been around as long as the SEO industry has. Search engines built a system that revolves around users entering a term or query into a text entry field, hitting return, and receiving a list of relevant results. As the online search market expanded, one clear leader emerged — Google — and with it they brought AdWords (now Google Ads), an advertising platform that allowed organizations to appear on search results pages for keywords that organically they might not.

Within Google Ads came a tool that enabled businesses to look at how many searches there were per month for almost any query. Google Keyword Planner became the de facto tool for keyword research in the industry, and with good reason: it was Google’s data. Not only that, Google gave us the ability to gather further insights due to other metrics Keyword Planner provided: competition and suggested bid. Whilst these keywords were Google Ads-oriented metrics, they gave the SEO industry an indication of how competitive a keyword was.

The reason is obvious. If a keyword or phrase has higher competition (i.e. more advertisers bidding to appear for that term) it’s likely to be more competitive from an organic perspective. Similarly, a term that has a higher suggested bid means it’s more likely to be a competitive term. SEOs dined on this data for years, but when the industry started digging a bit more into the data, we soon realized that while useful, it was not always wholly accurate. Moz, SEMrush, and other tools all started to develop alternative volume and competitive metrics using Clickstream data to give marketers more insights.

Now industry professionals have several software tools and data outlets to conduct their keyword research. These software companies will only improve in the accuracy of their data outputs. Google’s data is unlikely to significantly change; their goal is to sell ad space, not make life easy for SEOs. In fact, they’ve made life harder by using volume ranges for Google Ads accounts with low activity. SEO tools have investors and customers to appease and must continually improve their products to reduce churn and grow their customer base. This makes things rosy for content-led SEO, right?

Well, not really.

The problem with historical keyword research is twofold:

1. SEOs spend too much time thinking about the decision stage of the buyer’s journey (more on that later).

2. SEOs spend too much time thinking about keywords, rather than categories or topics.

The industry, to its credit, is doing a lot to tackle issue number two. “Topics over keywords” is something that is not new as I’ll briefly come to later. Frameworks for topic-based SEO have started to appear over the last few years. This is a step in the right direction. Organizing site content into categories, adding appropriate internal linking, and understanding that one piece of content can rank for several variations of a phrase is becoming far more commonplace.

What is less well known (but starting to gain traction) is point one. But in order to understand this further, we should dive into what the buyer’s journey actually is.

What is the buyer’s journey?

The buyer’s or customer’s journey is not new. If you open marketing text books from years gone by, get a college degree in marketing, or even just go on general marketing blogs you’ll see it crop up. There are lots of variations of this journey, but they all say a similar thing. No matter what product or service is bought, everyone goes through this journey. This could be online or offline — the main difference is that depending on the product, person, or situation, the amount of time this journey takes will vary — but every buyer goes through it. But what is it, exactly? For the purpose of this article, we’ll focus on three stages: awareness, consideration, & decision.

Awareness

The awareness stage of the buyer’s journey is similar to problem discovery, where a potential customer realizes that they have a problem (or an opportunity) but they may not have figured out exactly what that is yet.

Search terms at this stage are often question-based — users are researching around a particular area.

Consideration

The consideration stage is where a potential consumer has defined what their problem or opportunity is and has begun to look for potential solutions to help solve the issue they face.

Decision

The decision stage is where most organizations focus their attention. Normally consumers are ready to buy at this stage and are often doing product or vendor comparisons, looking at reviews, and searching for pricing information.

To illustrate this process, let’s take two examples: buying an ice cream and buying a holiday.

Being low-value, the former is not a particularly considered purchase, but this journey still takes place. The latter is more considered. It can often take several weeks or months for a consumer to decide on what destination they want to visit, let alone a hotel or excursions. But how does this affect keyword research, and the content which we as marketers should provide?

At each stage, a buyer will have a different thought process. It’s key to note that not every buyer of the same product will have the same thought process but you can see how we can start to formulate a process.

The Buyer’s Journey – Holiday Purchase

The above table illustrates the sort of queries or terms that consumers might use at different stages of their journey. The problem is that most organizations focus all of their efforts on the decision end of the spectrum. This is entirely the right approach to take at the start because you’re targeting consumers who are interested in your product or service then and there. However, in an increasingly competitive online space you should try and find ways to diversify and bring people into your marketing funnel (which in most cases is your website) at different stages.

I agree with the argument that creating content for people earlier in the journey will likely mean lower conversion rates from visitor to customer, but my counter to this would be that you’re also potentially missing out on people who will become customers. Further possibilities to at least get these people into your funnel include offering content downloads (gated content) to capture user’s information, or remarketing activity via Facebook, Google Ads, or other retargeting platforms.

Moving from keywords to topics

I’m not going to bang this drum too loudly. I think many in of the SEO community have signed up to the approach that topics are more important than keywords. There are quite a few resources on this listed online, but what forced it home for me was Cyrus Shepard’s Moz article in 2014. Much, if not all, of that post still holds true today.

What I will cover is an adoption of HubSpot’s Topic Cluster model. For those unaccustomed to their model, HubSpot’s approach formalizes and labels what many search marketers have been doing for a while now. The basic premise is instead of having your site fragmented with lots of content across multiple sections, all hyperlinking to each other, you create one really in-depth content piece that covers a topic area broadly (and covers shorter-tail keywords with high search volume), and then supplement this page with content targeting the long-tail, such as blog posts, FAQs, or opinion pieces. HubSpot calls this “pillar” and “cluster” content respectively.

Source: Matt Barby / HubSpot

The process then involves taking these cluster pages and linking back to the pillar page using keyword-rich anchor text. There’s nothing particularly new about this approach aside from formalizing it a bit more. Instead of having your site’s content structured in such a way that it’s fragmented and interlinking between lots of different pages and topics, you keep the internal linking within its topic, or content cluster. This video explains this methodology further. While we accept this model may not fit every situation, and nor is it completely perfect, it’s a great way of understanding how search engines are now interpreting content.

At Aira, we’ve taken this approach and tried to evolve it a bit further, tying these topics into the stages of the buyer’s journey while utilizing several data points to make sure our outputs are based off as much data as we can get our hands on. Furthermore, because pillar pages tend to target shorter-tail keywords with high search volume, they’re often either awareness- or consideration-stage content, and thus not applicable for decision stage. We term our key decision pages “target pages,” as this should be a primary focus of any activity we conduct.

We’ll also look at the semantic relativity of the keywords reviewed, so that we have a “parent” keyword that we’re targeting a page to rank for, and then children of that keyword or phrase that the page may also rank for, due to its similarity to the parent. Every keyword is categorized according to its stage in the buyer’s journey and whether it’s appropriate for a pillar, target, or cluster page. We also add two further classifications to our keywords: track & monitor and ignore. Definitions for these five keyword types are listed below:

Pillar page

A pillar page covers all aspects of a topic on a single page, with room for more in-depth reporting in more detailed cluster blog posts that hyperlink back to the pillar page. A keyword tagged with pillar page will be the primary topic and the focus of a page on the website. Pillar pages should be awareness- or consideration-stage content.

A great pillar page example I often refer to is HubSpot’s Facebook marketing guide or Mosi-guard’s insect bites guide (disclaimer: probably don’t click through if you don’t like close-up shots of insects!).

Cluster page

A cluster topic page for the pillar focuses on providing more detail for a specific long-tail keyword related to the main topic. This type of page is normally associated with a blog article but could be another type of content, like an FAQ page.

Good examples within the Facebook marketing topic listed above are HubSpot’s posts:

For Mosi-guard, they’re not utilizing internal links within the copy of the other blogs, but the “older posts” section at the bottom of the blog is referencing this guide:

Target page

Normally a keyword or phrase linked to a product or service page, e.g. nike trainers or seo services. Target pages are decision-stage content pieces.

HubSpot’s target content is their social media software page, with one of Mosi-guard’s target pages being their natural spray product.

Track & monitor

A keyword or phrase that is not the main focus of a page, but could still rank due to its similarity to the target page keyword. A good example of this might be seo services as the target page keyword, but this page could also rank for seo agency, seo company, etc.

Ignore

A keyword or phrase that has been reviewed but is not recommended to be optimized for, possibly due to a lack of search volume, it’s too competitive, it won’t be profitable, etc.

Once the keyword research is complete, we then map our keywords to existing website pages. This gives us a list of mapped keywords and a list of unmapped keywords, which in turn creates a content gap analysis that often leads to a content plan that could last for three, six, or twelve-plus months.

Putting it into practice

I’m a firm believer in giving an example of how this would work in practice, so I’m going to walk through one with screenshots. I’ll also provide a template of our keyword research document for you to take away.

1. Harvesting keywords

The first step in the process is similar, if not identical, to every other keyword research project. You start off with a batch of keywords from the client or other stakeholders that the site wants to rank for. Most of the industry call this a seed keyword list. That keyword list is normally a minimum of 15–20 keywords, but can often be more if you’re dealing with an e-commerce website with multiple product lines.

This list is often based off nothing more than opinion: “What do we think our potential customers will search for?” It’s a good starting point, but you need the rest of the process to follow on to make sure you’re optimizing based off data, not opinion.

2. Expanding the list

Once you’ve got that keyword list, it’s time to start utilizing some of the tools you have at your disposal. There are lots, of course! We tend to use a combination of Moz Keyword Explorer, Answer the Public, Keywords Everywhere, Google Search Console, Google Analytics, Google Ads, ranking tools, and SEMrush.

The idea of this list is to start thinking about keywords that the organization may not have considered before. Your expanded list will include obvious synonyms from your list. Take the example below:

Seed Keywords

Expanded Keywords

ski chalet

ski chalet

ski chalet rental

ski chalet hire

ski chalet [location name]

etc

There are other examples that should be considered. A client I worked with in the past once gave a seed keyword of “biomass boilers.” But after keyword research was conducted, a more colloquial term for “biomass boilers” in the UK is “wood burners.” This is an important distinction and should be picked up as early in the process as possible. Keyword research tools are not infallible, so if budget and resource allows, you may wish to consult current and potential customers about which terms they might use to find the products or services being offered.

3. Filtering out irrelevant keywords

Once you’ve expanded the seed keyword list, it’s time to start filtering out irrelevant keywords. This is pretty labor-intensive and involves sorting through rows of data. We tend to use Moz’s Keyword Explorer, filter by relevancy, and work our way down. As we go, we’ll add keywords to lists within the platform and start to try and sort things by topic. Topics are fairly subjective, and often you’ll get overlap between them. We’ll group similar keywords and phrases together in a topic based off the semantic relativity of those phrases. For example:

Topic

Keywords

ski chalet

ski chalet

ski chalet rental

ski chalet hire

ski chalet [location name]

catered chalet

catered chalet

luxury catered chalet

catered chalet rental

catered chalet hire

catered chalet [location name]

ski accommodation

ski accommodation

cheap ski accommodation

budget ski accommodation

ski accomodation [location name]

Many of the above keywords are decision-based keywords — particularly those with rental or hire in them. They’re showing buying intent. We’ll then try to put ourselves in the mind of the buyer and come up with keywords towards the start of the buyer’s journey.

Topic

Keywords

Buyer’s stage

ski resorts

ski resorts

best ski resorts

ski resorts europe

ski resorts usa

ski resorts canada

top ski resorts

cheap ski resorts

luxury ski resorts

Consideration

skiing

skiing

skiing guide

skiing beginner’s guide

Consideration

family holidays

family holidays

family winter holidays

family trips

Awareness

This helps us cater to customers that might not be in the frame of mind to purchase just yet — they’re just doing research. It means we cast the net wider. Conversion rates for these keywords are unlikely to be high (at least, for purchases or enquiries) but if utilized as part of a wider marketing strategy, we should look to capture some form of information, primarily an email address, so we can send people relevant information via email or remarketing ads later down the line.

4. Pulling in data

Once you’ve expanded the seed keywords out, Keyword Explorer’s handy list function enables your to break things down into separate topics. You can then export that data into a CSV and start combining it with other data sources. If you have SEMrush API access, Dave Sottimano’s API Library is a great time saver; otherwise, you may want to consider uploading the keywords into the Keywords Everywhere Chrome extension and manually exporting the data and combining everything together. You should then have a spreadsheet that looks something like this:

You could then add in additional data sources. There’s no reason you couldn’t combine the above with volumes and competition metrics from other SEO tools. Consider including existing keyword ranking information or Google Ads data in this process. Keywords that convert well on PPC should do the same organically and should therefore be considered. Wil Reynolds talks about this particular tactic a lot.

5. Aligning phrases to the buyer’s journey

The next stage of the process is to start categorizing the keywords into the stage of the buyer’s journey. Something we’ve found at Aira is that keywords don’t always fit into a predefined stage. Someone looking for “marketing services” could be doing research about what marketing services are, but they could also be looking for a provider. You may get keywords that could be either awareness/consideration or consideration/decision. Use your judgement, and remember this is subjective. Once complete, you should end up with some data that looks similar to this:

This categorization is important, as it starts to frame what type of content is most appropriate for that keyword or phrase.

The next stage of this process is to start noticing patterns in keyphrases and where they get mapped to in the buyer’s journey. Often you’ll see keywords like “price” or ”cost” at the decision stage and phrases like “how to” at the awareness stage. Once you start identifying these patterns, possibly using a variation of Tom Casano’s keyword clustering approach, you can then try to find a way to automate so that when these terms appear in your keyword column, the intent automatically gets updated.

Once completed, we can then start to define each of our keywords and give them a type:

  • Pillar page
  • Cluster page
  • Target page
  • Track & monitor
  • Ignore

We use this document to start thinking about what type of content is most effective for that piece given the search volume available, how competitive that term is, how profitable the keyword could be, and what stage the buyer might be at. We’re trying to find that sweet spot between having enough search volume, ensuring we can actually rank for that keyphrase (there’s no point in a small e-commerce startup trying to rank for “buy nike trainers”), and how important/profitable that phrase could be for the business. The below Venn diagram illustrates this nicely:

We also reorder the keywords so keywords that are semantically similar are bucketed together into parent and child keywords. This helps to inform our on-page recommendations:

From the example above, you can see “digital marketing agency” as the main keyword, but “digital marketing services” & “digital marketing agency uk” sit underneath.

We also use conditional formatting to help identify keyword page types:

And then sheets to separate topics out:

Once this is complete, we have a data-rich spreadsheet of keywords that we then work with clients on to make sure we’ve not missed anything. The document can get pretty big, particularly when you’re dealing with e-commerce websites that have thousands of products.

5. Keyword mapping and content gap analysis

We then map these keywords to existing content to ensure that the site hasn’t already written about the subject in the past. We often use Google Search Console data to do this so we understand how any existing content is being interpreted by the search engines. By doing this we’re creating our own content gap analysis. An example output can be seen below:

The above process takes our keyword research and then applies the usual on-page concepts (such as optimizing meta titles, URLs, descriptions, headings, etc) to existing pages. We’re also ensuring that we’re mapping our user intent and type of page (pillar, cluster, target, etc), which helps us decide what sort of content the piece should be (such as a blog post, webinar, e-book, etc). This process helps us understand what keywords and phrases the site is not already being found for, or is not targeted to.

Free template

I promised a template Google Sheet earlier in this blog post and you can find that here.

Do you have any questions on this process? Ways to improve it? Feel free to post in the comments below or ping me over on Twitter!

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!


Moz Blog

Find More Articles

Posted in Latest NewsComments Off

The Advanced Guide to Keyword Clustering

Posted by tomcasano

If your goal is to grow your organic traffic, you have to think about SEO in terms of “product/market fit.”

Keyword research is the “market” (what users are actually searching for) and content is the “product” (what users are consuming). The “fit” is optimization.

To grow your organic traffic, you need your content to mirror the reality of what users are actually searching for. Your content planning and creation, keyword mapping, and optimization should all align with the market. This is one of the best ways to grow your organic traffic.

Why bother with keyword grouping?

One web page can rank for multiple keywords. So why aren’t we hyper-focused on planning and optimizing content that targets dozens of similar and related keywords?

Why target only one keyword with one piece of content when you can target 20?

The impact of keyword clustering to acquire more organic traffic is not only underrated, it is largely ignored. In this guide, I’ll share with you our proprietary process we’ve pioneered for keyword grouping so you can not only do it yourself, but you can maximize the number of keywords your amazing content can rank for.

Here’s a real-world example of a handful of the top keywords that this piece of content is ranking for. The full list is over 1,000 keywords.

17 different keywords one page is ranking for

Why should you care?

It’d be foolish to focus on only one keyword, as you’d lose out on 90%+ of the opportunity.

Here’s one of my favorite examples of all of the keywords that one piece of content could potentially target:

List of ~100 keywords one page ranks for

Let’s dive in!

Part 1: Keyword collection

Before we start grouping keywords into clusters, we first need our dataset of keywords from which to group from.

In essence, our job in this initial phase is to find every possible keyword. In the process of doing so, we’ll also be inadvertently getting many irrelevant keywords (thank you, Keyword Planner). However, it’s better to have many relevant and long-tail keywords (and the ability to filter out the irrelevant ones) than to only have a limited pool of keywords to target.

For any client project, I typically say that we’ll collect anywhere from 1,000 to 6,000 keywords. But truth be told, we’ve sometimes found 10,000+ keywords, and sometimes (in the instance of a local, niche client), we’ve found less than 1,000.

I recommend collecting keywords from about 8–12 different sources. These sources are:

  1. Your competitors
  2. Third-party data tools (Moz, Ahrefs, SEMrush, AnswerThePublic, etc.)
  3. Your existing data in Google Search Console/Google Analytics
  4. Brainstorming your own ideas and checking against them
  5. Mashing up keyword combinations
  6. Autocomplete suggestions and “Searches related to” from Google

There’s no shortage of sources for keyword collection, and more keyword research tools exist now than ever did before. Our goal here is to be so extensive that we never have to go back and “find more keywords” in the future — unless, of course, there’s a new topic we are targeting.

The prequel to this guide will expand upon keyword collection in depth. For now, let’s assume that you’ve spent a few hours collecting a long list of keywords, you have removed the duplicates, and you have semi-reliable search volume data.

Part 2: Term analysis

Now that you have an unmanageable list of 1,000+ keywords, let’s turn it into something useful.

We begin with term analysis. What the heck does that mean?

We break each keyword apart into its component terms that comprise the keyword, so we can see which terms are the most frequently occurring.

For example, the keyword: “best natural protein powder” is comprised of 4 terms: “best,” “natural,” “protein,” and “powder.” Once we break apart all of the keywords into their component parts, we can more readily analyze and understand which terms (as subcomponents of the keywords) are recurring the most in our keyword dataset.

Here’s a sampling of 3 keywords:

  • best natural protein powder
  • most powerful natural anti inflammatory
  • how to make natural deodorant

Take a closer look, and you’ll notice that the term “natural” occurs in all three of these keywords. If this term is occurring very frequently throughout our long list of keywords, it’ll be highly important when we start grouping our keywords.

You will need a word frequency counter to give you this insight. The ultimate free tool for this is Write Words’ Word Frequency Counter. It’s magical.

Paste in your list of keywords, click submit, and you’ll get something like this:

List of keywords and how frequently they occur

Copy and paste your list of recurring terms into a spreadsheet. You can obviously remove prepositions and terms like “is,” “for,” and “to.”

You don’t always get the most value by just looking at individual terms. Sometimes a two-word or three-word phrase gives you insights you wouldn’t have otherwise. In this example, you see the terms “milk” and “almond” appearing, but it turns out that this is actually part of the phrase “almond milk.”

To gather these insights, use the Phrase Frequency Counter from WriteWords and repeat the process for phrases that have two, three, four, five, and six terms in them. Paste all of this data into your spreadsheet too.

A two-word phrase that occurs more frequently than a one-word phrase is an indicator of its significance. To account for this, I use the COUNTA function in Google Sheets to show me the number of terms in a phrase:

=COUNTA(SPLIT(B2," "))

Now we can look at our keyword data with a second dimension: not only the number of times a term or phrase occurs, but also how many words are in that phrase.

Finally, to give more weighting to phrases that recur less frequently but have more terms in them, I put an exponent on the number of terms with a basic formula:

=(C4^2)*A4

In other words, take the number of terms and raise it to a power, and then multiply that by the frequency of its occurrence. All this does is give more weighting to the fact that a two-word phrase that occurs less frequently is still more important than a one-word phrase that might occur more frequently.

As I never know just the right power to raise it to, I test several and keep re-sorting the sheet to try to find the most important terms and phrases in the sheet.

Spreadsheet of keywords and their weighted importance

When you look at this now, you can already see patterns start to emerge and you’re already beginning to understand your searchers better.

In this example dataset, we are going from a list of 10k+ keywords to an analysis of terms and phrases to understand what people are really asking. For example, “what is the best” and “where can i buy” are phrases we can absolutely understand searchers using.

I mark off the important terms or phrases. I try to keep this number to under 50 and to a maximum of around 75; otherwise, grouping will get hairy in Part 5.

Part 3: Hot words

What are hot words?

Hot words are the terms or phrases from that last section that we have deemed to be the most important. We’ve explained hot words in greater depth here.

Why are hot words important?

We explain:

This exercise provides us with a handful of the most relevant and important terms and phrases for traffic and relevancy, which can then be used to create the best content strategies — content that will rank highly and, in turn, help us reap traffic rewards for your site.

When developing your hot words list, we identify the highest frequency and most relevant terms from a large range of keywords used by several of your highest-performing competitors to generate their traffic, and these become “hot words.”

When working with a client (or doing this for yourself), there are generally 3 questions we want answered for each hot word:

  1. Which of these terms are the most important for your business? (0–10)
  2. Which of these terms are negative keywords (we want to ignore or avoid)?
  3. Any other feedback about qualified or high-intent keywords?

We narrow down the list, removing any negative keywords or keywords that are not really important for the website.

Once we have our final list of hot words, we organize them into broad topic groups like this:

Organized spreadsheet of hot words by topic

The different colors have no meaning, but just help to keep it visually organized for when we group them.

One important thing to note is that word stems play an important part here.

For example, consider that all of these words below have the same underlying relevance and meaning:

  • blog
  • blogs
  • blogger
  • bloggers
  • blogging

Therefore, when we’re grouping keywords, to consider “blog” and “blogging” and “bloggers” as part of the same cluster, we’ll need to use the word stem of “blog” for all of them. Word stems are our best friend when grouping. Synonyms can be organized in a similar way, which are basically two different ways of saying the same thing (and the same user intent) such as “build” and “create” or “search” and “look for.”

Part 4: Preparation for keyword grouping

Now we’re going to get ourselves set up for our Herculean task of clustering.

To start, copy your list of hot words and transpose them horizontally across a row.

Screenshot of menu in spreadsheet

List your keywords in the first column.

Screenshot of keyword spreadsheet

Now, the real magic begins.

After much research and noodling around, I discovered the function in Google Sheets that tells us whether a stem or term is in a keyword or not. It uses RegEx:

=IF(RegExMatch(A5,"health"),"YES","NO")

This simply tells us whether this word stem or word is in that keyword or not. You have to individually set the term for each column to get your “YES” or “NO” answer. I then drag this formula down to all of the rows to get all of the YES/NO answers. Google Sheets often takes a minute or so to process all of this data.

Next, we have to “hard code” these formulas so we can remove the NOs and be left with only a YES if that terms exists in that keyword.

Copy all of the data and “Paste values only.”

Screenshot of spreadsheet menu

Now, use “Find and replace” to remove all of the NOs.

Screenshot of Find and Replace popup

What you’re left with is nothing short of a work of art. You now have the most powerful way to group your keywords. Let the grouping begin!

Screenshot of keyword spreadsheet

Part 5: Keyword grouping

At this point, you’re now set up for keyword clustering success.

This part is half art, half science. No wait, I take that back. To do this part right, you need:

  • A deep understanding of who you’re targeting, why they’re important to the business, user intent, and relevance
  • Good judgment to make tradeoffs when breaking keywords apart into groups
  • Good intuition

This is one of the hardest parts for me to train anyone to do. It comes with experience.

At the top of the sheet, I use the COUNTA function to show me how many times this word step has been found in our keyword set:

=COUNTA(C3:C10000)

This is important because as a general rule, it’s best to start with the most niche topics that have the least overlap with other topics. If you start too broadly, your keywords will overlap with other keyword groups and you’ll have a hard time segmenting them into meaningful groups. Start with the most narrow and specific groups first.

To begin, you want to sort the sheet by word stem.

The word stems that occur only a handful of times won’t have a large amount of overlap. So I start by sorting the sheet by that column, and copying and pasting those keywords into their own new tab.

Now you have your first keyword group!

Here’s a first group example: the “matcha” group. This can be its own project in its own right: for instance, if a website was all about matcha tea and there were other tangentially related keywords.

Screenshot of list of matcha-related keywords

As we continue breaking apart one keyword group and then another, by the end we’re left with many different keyword groups. If the groups you’ve arrived at are too broad, you can subdivide them even more into narrower keyword subgroups for more focused content pieces. You can follow the same process for this broad keyword group, and make it a microcosm of the same process of dividing the keywords into smaller groups based on word stems.

We can create an overview of the groups to see the volume and topical opportunities from a high level.

Screenshot of spreadsheet with keyword group overview

We want to not only consider search volume, but ideally also intent, competitiveness, and so forth.

Voilà!

You’ve successfully taken a list of thousands of keywords and grouped them into relevant keyword groups.

Wait, why did we do all of this hard work again?

Now you can finally attain that “product/market fit” we talked about. It’s magical.

You can take each keyword group and create a piece of optimized content around it, targeting dozens of keywords, exponentially raising your potential to acquire more organic traffic. Boo yah!

All done. Now what?

Now the real fun begins. You can start planning out new content that you never knew you needed to create. Alternatively, you can map your keyword groups (and subgroups) to existing pages on your website and add in keywords and optimizations to the header tags, body text, and so forth for all those long-tail keywords you had ignored.

Keyword grouping is underrated, overlooked, and ignored at large. It creates a massive new opportunity to optimize for terms where none existed. Sometimes it’s just adding one phrase or a few sentences targeting a long-tail keyword here and there that will bring in that incremental search traffic for your site. Do this dozens of times and you will keep getting incremental increases in your organic traffic.

What do you think?

Leave a comment below and let me know your take on keyword clustering.

Need a hand? Just give me a shout, I’m happy to help.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!


Moz Blog

Posted in Latest NewsComments Off

How to Get More Keyword Metrics for Your Target Keywords

Posted by Bill.Sebald

If you’re old in SEO years, you remember the day [not provided] was introduced. It was a dark, dark day. SEOs lost a vast amount of trusty information. Click data. Conversion data. This was incredibly valuable, allowing SEOs to prioritize their targets.

Google said the info was removed for security purposes, while suspicious SEOs thought this was a push towards spending more on AdWords (now Google Ads). I get it — since AdWords would give you the keyword data SEOs cherished, the “controversy” was warranted, in my opinion. The truth is out there.

But we’ve moved on, and learned to live with the situation. Then a few years later, Google Webmaster Tools (now Search Console) started providing some of the keyword data in the Search Analytics report. Through the years, the report got better and better.

But there’s still a finite set of keywords in the interface. You can’t get more than 999 in your report.

Search Analytics Report

Guess what? Google has more data for you!

The Google Search Console API is your friend. This summer it became even friendlier, providing 16 months worth of data. What you may not know is this API can give you more than 999 keywords. By way of example, the API provides more than 45,000 for our Greenlane site. And we’re not even a very large site. That’s right — the API can give you keywords, clicks, average position, impressions, and CTR %.

Salivating yet?

How to easily leverage the API

If you’re not very technical and the thought of an API frightens you, I promise there’s nothing to fear. I’m going to show you a way to leverage the data using Google Sheets.

Here is what you will need:

  1. Google Sheets (free)
  2. Supermetrics Add-On (free trial, but a paid tool)

If you haven’t heard of Google Sheets, it’s one of several tools Google provides for free. This directly competes with Microsoft Excel. It’s a cloud-based spreadsheet that works exceptionally well.

If you aren’t familiar with Supermetrics, it’s an add-on for Google Sheets that allows data to be pulled in from other sources. In this case, one of the sources will be Google Search Console. Now, while Supermetrics has a free trial, paid is the way to go. It’s worth it!

Installation of Supermetrics:

  1. Open Google Sheets and click the Add-On option
  2. Click Get Add-Ons
  3. A window will open where you can search for Supermetrics. It will look like this:

How To Install Supermetrics

From there, just follow the steps. It will immediately ask to connect to your Google account. I’m sure you’ve seen this kind of dialog box before:

Supermetrics wants to access your Google Account

You’ll be greeted with a message for launching the newly installed add-on. Just follow the prompts to launch. Next you’ll see a new window to the right of your Google Sheet.

Launch message

At this point, you should see the following note:

Great, you’re logged into Google Search Console! Now let’s run your first query. Pick an account from the list below.

Next, all you have to do is work down the list in Supermetrics. Data Source, Select Sites, and Select Dates are pretty self-explanatory. When you reach the “Select metrics” toggle, choose Impressions, Clicks, CTR (%), and Average Position.

Metrics

When you reach “Split by,” choose Search Query as the Split to rows option. And pick a large number for number of rows to fetch. If you also want the page URLs (perhaps you’d like your data divided by the page level), you just need to add Full URL as well.

Split By

You can play with the other Filter and Options if you’d like, but you’re ready to click Apply Changes and receive the data. It should compile like this:

Final result

Got the data. Now what?

Sometimes optimization is about taking something that’s working, and making it work better. This data can show you which keywords and topics are important to your audience. It’s also a clue towards what Google thinks you’re important for (thus, rewarding you with clicks).

SEMrush and Ahrefs can provide ranking keyword data with their estimated clicks, but impressions is an interesting metric here. High impression and low clicks? Maybe your title and description tags aren’t compelling enough. It’s also fun to VLOOKUP their data against this, to see just how accurate they are (or are not). Or you can use a tool like PowerBI to append other customer or paid search metrics to paint a bigger picture of your visitors’ mindset.

Conclusion

Sometimes the littlest hacks are the most fun. Google commonly holds some data back through their free products (the Greenlane Indexation Tester is a good example with the old interface). We know Search Planner and Google Analytics have more than they share. But in those cases, where directional information can sometimes be enough, digging out even more of your impactful keyword data is pure gold.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!


Moz Blog

Posted in Latest NewsComments Off

Ranking the 6 Most Accurate Keyword Difficulty Tools

Posted by Jeff_Baker

In January of 2018 Brafton began a massive organic keyword targeting campaign, amounting to over 90,000 words of blog content being published.

Did it work?

Well, yeah. We doubled the number of total keywords we rank for in less than six months. By using our advanced keyword research and topic writing process published earlier this year we also increased our organic traffic by 45% and the number of keywords ranking in the top ten results by 130%.

But we got a whole lot more than just traffic.

From planning to execution and performance tracking, we meticulously logged every aspect of the project. I’m talking blog word count, MarketMuse performance scores, on-page SEO scores, days indexed on Google. You name it, we recorded it.

As a byproduct of this nerdery, we were able to draw juicy correlations between our target keyword rankings and variables that can affect and predict those rankings. But specifically for this piece…

How well keyword research tools can predict where you will rank.

A little background

We created a list of keywords we wanted to target in blogs based on optimal combinations of search volume, organic keyword difficulty scores, SERP crowding, and searcher intent.

We then wrote a blog post targeting each individual keyword. We intended for each new piece of blog content to rank for the target keyword on its own.

With our keyword list in hand, my colleague and I manually created content briefs explaining how we would like each blog post written to maximize the likelihood of ranking for the target keyword. Here’s an example of a typical brief we would give to a writer:

This image links to an example of a content brief Brafton delivers to writers.

Between mid-January and late May, we ended up writing 55 blog posts each targeting 55 unique keywords. 50 of those blog posts ended up ranking in the top 100 of Google results.

We then paused and took a snapshot of each URL’s Google ranking position for its target keyword and its corresponding organic difficulty scores from Moz, SEMrush, Ahrefs, SpyFu, and KW Finder. We also took the PPC competition scores from the Keyword Planner Tool.

Our intention was to draw statistical correlations between between our keyword rankings and each tool’s organic difficulty score. With this data, we were able to report on how accurately each tool predicted where we would rank.

This study is uniquely scientific, in that each blog had one specific keyword target. We optimized the blog content specifically for that keyword. Therefore every post was created in a similar fashion.

Do keyword research tools actually work?

We use them every day, on faith. But has anyone ever actually asked, or better yet, measured how well keyword research tools report on the organic difficulty of a given keyword?

Today, we are doing just that. So let’s cut through the chit-chat and get to the results…

This image ranks each of the 6 keyword research tools, in order, Moz leads with 4.95 stars out of 5, followed by KW Finder, SEMrush, AHREFs, SpyFu, and lastly Keyword Planner Tool.

While Moz wins top-performing keyword research tool, note that any keyword research tool with organic difficulty functionality will give you an advantage over flipping a coin (or using Google Keyword Planner Tool).

As you will see in the following paragraphs, we have run each tool through a battery of statistical tests to ensure that we painted a fair and accurate representation of its performance. I’ll even provide the raw data for you to inspect for yourself.

Let’s dig in!

The Pearson Correlation Coefficient

Yes, statistics! For those of you currently feeling panicked and lobbing obscenities at your screen, don’t worry — we’re going to walk through this together.

In order to understand the relationship between two variables, our first step is to create a scatter plot chart.

Below is the scatter plot for our 50 keyword rankings compared to their corresponding Moz organic difficulty scores.

This image shows a scatter plot for Moz's keyword difficulty scores versus our keyword rankings. In general, the data clusters fairly tight around the regression line.

We start with a visual inspection of the data to determine if there is a linear relationship between the two variables. Ideally for each tool, you would expect to see the X variable (keyword ranking) increase proportionately with the Y variable (organic difficulty). Put simply, if the tool is working, the higher the keyword difficulty, the less likely you will rank in a top position, and vice-versa.

This chart is all fine and dandy, however, it’s not very scientific. This is where the Pearson Correlation Coefficient (PCC) comes into play.

The PCC measures the strength of a linear relationship between two variables. The output of the PCC is a score ranging from +1 to -1. A score greater than zero indicates a positive relationship; as one variable increases, the other increases as well. A score less than zero indicates a negative relationship; as one variable increases, the other decreases. Both scenarios would indicate a level of causal relationship between the two variables. The stronger the relationship between the two veriables, the closer to +1 or -1 the PCC will be. Scores near zero indicate a weak or no relatioship.

Phew. Still with me?

So each of these scatter plots will have a corresponding PCC score that will tell us how well each tool predicted where we would rank, based on its keyword difficulty score.

We will use the following table from statisticshowto.com to interpret the PCC score for each tool:

Coefficient Correlation R Score

Key

.70 or higher

Very strong positive relationship

.40 to +.69

Strong positive relationship

.30 to +.39

Moderate positive relationship

.20 to +.29

Weak positive relationship

.01 to +.19

No or negligible relationship

0

No relationship [zero correlation]

-.01 to -.19

No or negligible relationship

-.20 to -.29

Weak negative relationship

-.30 to -.39

Moderate negative relationship

-.40 to -.69

Strong negative relationship

-.70 or higher

Very strong negative relationship

In order to visually understand what some of these relationships would look like on a scatter plot, check out these sample charts from Laerd Statistics.

These scatter plots show three types of correlations: positive, negative, and no correlation. Positive correlations have data plots that move up and to the right. Negative correlations move down and to the right. No correlation has data that follows no linear pattern

And here are some examples of charts with their correlating PCC scores (r):

These scatter plots show what different PCC values look like visually. The tighter the grouping of data around the regression line, the higher the PCC value.

The closer the numbers cluster towards the regression line in either a positive or negative slope, the stronger the relationship.

That was the tough part – you still with me? Great, now let’s look at each tool’s results.

Test 1: The Pearson Correlation Coefficient

Now that we’ve all had our statistics refresher course, we will take a look at the results, in order of performance. We will evaluate each tool’s PCC score, the statistical significance of the data (P-val), the strength of the relationship, and the percentage of keywords the tool was able to find and report keyword difficulty values for.

In order of performance:

#1: Moz

This image shows a scatter plot for Moz's keyword difficulty scores versus our keyword rankings. In general, the data clusters fairly tight around the regression line.

Revisiting Moz’s scatter plot, we observe a tight grouping of results relative to the regression line with few moderate outliers.

Moz Organic Difficulty Predictability

PCC

0.412

P-val

.003 (P<0.05)

Relationship

Strong

% Keywords Matched

100.00%

Moz came in first with the highest PCC of .412. As an added bonus, Moz grabs data on keyword difficulty in real time, rather than from a fixed database. This means that you can get any keyword difficulty score for any keyword.

In other words, Moz was able to generate keyword difficulty scores for 100% of the 50 keywords studied.

#2: SpyFu

This image shows a scatter plot for SpyFu's keyword difficulty scores versus our keyword rankings. The plot is similar looking to Moz's, with a few larger outliers.

Visually, SpyFu shows a fairly tight clustering amongst low difficulty keywords, and a couple moderate outliers amongst the higher difficulty keywords.

SpyFu Organic Difficulty Predictability

PCC

0.405

P-val

.01 (P<0.05)

Relationship

Strong

% Keywords Matched

80.00%

SpyFu came in right under Moz with 1.7% weaker PCC (.405). However, the tool ran into the largest issue with keyword matching, with only 40 of 50 keywords producing keyword difficulty scores.

#3: SEMrush

This image shows a scatter plot for SEMrush's keyword difficulty scores versus our keyword rankings. The data has a significant amount of outliers relative to the regression line.

SEMrush would certainly benefit from a couple mulligans (a second chance to perform an action). The Correlation Coefficient is very sensitive to outliers, which pushed SEMrush’s score down to third (.364).

SEMrush Organic Difficulty Predictability

PCC

0.364

P-val

.01 (P<0.05)

Relationship

Moderate

% Keywords Matched

92.00%

Further complicating the research process, only 46 of 50 keywords had keyword difficulty scores associated with them, and many of those had to be found through SEMrush’s “phrase match” feature individually, rather than through the difficulty tool.

The process was more laborious to dig around for data.

#4: KW Finder

This image shows a scatter plot for KW Finder's keyword difficulty scores versus our keyword rankings. The data also has a significant amount of outliers relative to the regression line.

KW Finder definitely could have benefitted from more than a few mulligans with numerous strong outliers, coming in right behind SEMrush with a score of .360.

KW Finder Organic Difficulty Predictability

PCC

0.360

P-val

.01 (P<0.05)

Relationship

Moderate

% Keywords Matched

100.00%

Fortunately, the KW Finder tool had a 100% match rate without any trouble digging around for the data.

#5: Ahrefs

This image shows a scatter plot for AHREF's keyword difficulty scores versus our keyword rankings. The data shows tight clustering amongst low difficulty score keywords, and a wide distribution amongst higher difficulty scores.

Ahrefs comes in fifth by a large margin at .316, barely passing the “weak relationship” threshold.

Ahrefs Organic Difficulty Predictability

PCC

0.316

P-val

.03 (P<0.05)

Relationship

Moderate

% Keywords Matched

100%

On a positive note, the tool seems to be very reliable with low difficulty scores (notice the tight clustering for low difficulty scores), and matched all 50 keywords.

#6: Google Keyword Planner Tool

This image shows a scatter plot for Google Keyword Planner Tool's keyword difficulty scores versus our keyword rankings. The data shows randomly distributed plots with no linear relationship.

Before you ask, yes, SEO companies still use the paid competition figures from Google’s Keyword Planner Tool (and other tools) to assess organic ranking potential. As you can see from the scatter plot, there is in fact no linear relationship between the two variables.

Google Keyword Planner Tool Organic Difficulty Predictability

PCC

0.045

P-val

Statistically insignificant/no linear relationship

Relationship

Negligible/None

% Keywords Matched

88.00%

SEO agencies still using KPT for organic research (you know who you are!) — let this serve as a warning: You need to evolve.

Test 1 summary

For scoring, we will use a ten-point scale and score every tool relative to the highest-scoring competitor. For example, if the second highest score is 98% of the highest score, the tool will receive a 9.8. As a reminder, here are the results from the PCC test:

This bar chart shows the final PCC values for the first test, summarized.

And the resulting scores are as follows:

Tool

PCC Test

Moz

10

SpyFu

9.8

SEMrush

8.8

KW Finder

8.7

Ahrefs

7.7

KPT

1.1

Moz takes the top position for the first test, followed closely by SpyFu (with an 80% match rate caveat).

Test 2: Adjusted Pearson Correlation Coefficient

Let’s call this the “Mulligan Round.” In this round, assuming sometimes things just go haywire and a tool just flat-out misses, we will remove the three most egregious outliers to each tool’s score.

Here are the adjusted results for the handicap round:

Adjusted Scores (3 Outliers removed)

PCC

Difference (+/-)

SpyFu

0.527

0.122

SEMrush

0.515

0.150

Moz

0.514

0.101

Ahrefs

0.478

0.162

KWFinder

0.470

0.110

Keyword Planner Tool

0.189

0.144

As noted in the original PCC test, some of these tools really took a big hit with major outliers. Specifically, Ahrefs and SEMrush benefitted the most from their outliers being removed, gaining .162 and .150 respectively to their scores, while Moz benefited the least from the adjustments.

For those of you crying out, “But this is real life, you don’t get mulligans with SEO!”, never fear, we will make adjustments for reliability at the end.

Here are the updated scores at the end of round two:

Tool

PCC Test

Adjusted PCC

Total

SpyFu

9.8

10

19.8

Moz

10

9.7

19.7

SEMrush

8.8

9.8

18.6

KW Finder

8.7

8.9

17.6

AHREFs

7.7

9.1

16.8

KPT

1.1

3.6

4.7

SpyFu takes the lead! Now let’s jump into the final round of statistical tests.

Test 3: Resampling

Being that there has never been a study performed on keyword research tools at this scale, we wanted to ensure that we explored multiple ways of looking at the data.

Big thanks to Russ Jones, who put together an entirely different model that answers the question: “What is the likelihood that the keyword difficulty of two randomly selected keywords will correctly predict the relative position of rankings?”

He randomly selected 2 keywords from the list and their associated difficulty scores.

Let’s assume one tool says that the difficulties are 30 and 60, respectively. What is the likelihood that the article written for a score of 30 ranks higher than the article written on 60? Then, he performed the same test 1,000 times.

He also threw out examples where the two randomly selected keywords shared the same rankings, or data points were missing. Here was the outcome:

Resampling

% Guessed correctly

Moz

62.2%

Ahrefs

61.2%

SEMrush

60.3%

Keyword Finder

58.9%

SpyFu

54.3%

KPT

45.9%

As you can see, this tool was particularly critical on each of the tools. As we are starting to see, no one tool is a silver bullet, so it is our job to see how much each tool helps make more educated decisions than guessing.

Most tools stayed pretty consistent with their levels of performance from the previous tests, except SpyFu, which struggled mightily with this test.

In order to score this test, we need to use 50% as the baseline (equivalent of a coin flip, or zero points), and scale each tool relative to how much better it performed over a coin flip, with the top scorer receiving ten points.

For example, Ahrefs scored 11.2% better than flipping a coin, which is 8.2% less than Moz which scored 12.2% better than flipping a coin, giving AHREFs a score of 9.2.

The updated scores are as follows:

Tool

PCC Test

Adjusted PCC

Resampling

Total

Moz

10

9.7

10

29.7

SEMrush

8.8

9.8

8.4

27

Ahrefs

7.7

9.1

9.2

26

KW Finder

8.7

8.9

7.3

24.9

SpyFu

9.8

10

3.5

23.3

KPT

1.1

3.6

-.4

.7

So after the last statistical accuracy test, we have Moz consistently performing alone in the top tier. SEMrush, Ahrefs, and KW Finder all turn in respectable scores in the second tier, followed by the unique case of SpyFu, which performed outstanding in the first two tests (albeit, only returning results on 80% of the tested keywords), then falling flat on the final test.

Finally, we need to make some usability adjustments.

Usability Adjustment 1: Keyword Matching

A keyword research tool doesn’t do you much good if it can’t provide results for the keywords you are researching. Plain and simple, we can’t treat two tools as equals if they don’t have the same level of practical functionality.

To explain in practical terms, if a tool doesn’t have data on a particular keyword, one of two things will happen:

  1. You have to use another tool to get the data, which devalues the entire point of using the original tool.
  2. You miss an opportunity to rank for a high-value keyword.

Neither scenario is good, therefore we developed a penalty system. For each 10% match rate under 100%, we deducted a single point from the final score, with a maximum deduction of 5 points. For example, if a tool matched 92% of the keywords, we would deduct .8 points from the final score.

One may argue that this penalty is actually too lenient considering the significance of the two unideal scenarios outlined above.

The penalties are as follows:

Tool

Match Rate

Penalty

KW Finder

100%

0

Ahrefs

100%

0

Moz

100%

0

SEMrush

92%

-.8

Keyword Planner Tool

88%

-1.2

SpyFu

80%

-2

Please note we gave SEMrush a lot of leniency, in that technically, many of the keywords evaluated were not found in its keyword difficulty tool, but rather through manually digging through the phrase match tool. We will give them a pass, but with a stern warning!

Usability Adjustment 2: Reliability

I told you we would come back to this! Revisiting the second test in which we threw away the three strongest outliers that negatively impacted each tool’s score, we will now make adjustments.

In real life, there are no mulligans. In real life, each of those three blog posts that were thrown out represented a significant monetary and time investment. Therefore, when a tool has a major blunder, the result can be a total waste of time and resources.

For that reason, we will impose a slight penalty on those tools that benefited the most from their handicap.

We will use the level of PCC improvement to evaluate how much a tool benefitted from removing their outliers. In doing so, we will be rewarding the tools that were the most consistently reliable. As a reminder, the amounts each tool benefitted were as follows:

Tool

Difference (+/-)

Ahrefs

0.162

SEMrush

0.150

Keyword Planner Tool

0.144

SpyFu

0.122

KWFinder

0.110

Moz

0.101

In calculating the penalty, we scored each of the tools relative to the top performer, giving the top performer zero penalty and imposing penalties based on how much additional benefit the tools received over the most reliable tool, on a scale of 0–100%, with a maximum deduction of 5 points.

So if a tool received twice the benefit of the top performing tool, it would have had a 100% benefit, receiving the maximum deduction of 5 points. If another tool received a 20% benefit over of the most reliable tool, it would get a 1-point deduction. And so on.

Tool

% Benefit

Penalty

Ahrefs

60%

-3

SEMrush

48%

-2.4

Keyword Planner Tool

42%

-2.1

SpyFu

20%

-1

KW Finder

8%

-.4

Moz

-

0

Results

All told, our penalties were fairly mild, with a slight shuffling in the middle tier. The final scores are as follows:

Tool

Total Score

Stars (5 max)

Moz

29.7

4.95

KW Finder

24.5

4.08

SEMrush

23.8

3.97

Ahrefs

23.0

3.83

Spyfu

20.3

3.38

KPT

-2.6

0.00

Conclusion

Using any organic keyword difficulty tool will give you an advantage over not doing so. While none of the tools are a crystal ball, providing perfect predictability, they will certainly give you an edge. Further, if you record enough data on your own blogs’ performance, you will get a clearer picture of the keyword difficulty scores you should target in order to rank on the first page.

For example, we know the following about how we should target keywords with each tool:

Tool

Average KD ranking ≤10

Average KD ranking ≥ 11

Moz

33.3

37.0

SpyFu

47.7

50.6

SEMrush

60.3

64.5

KWFinder

43.3

46.5

Ahrefs

11.9

23.6

This is pretty powerful information! It’s either first page or bust, so we now know the threshold for each tool that we should set when selecting keywords.

Stay tuned, because we made a lot more correlations between word count, days live, total keywords ranking, and all kinds of other juicy stuff. Tune in again in early September for updates!

We hope you found this test useful, and feel free to reach out with any questions on our math!

Disclaimer: These results are estimates based on 50 ranking keywords from 50 blog posts and keyword research data pulled from a single moment in time. Search is a shifting landscape, and these results have certainly changed since the data was pulled. In other words, this is about as accurate as we can get from analyzing a moving target.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!


Moz Blog

Posted in Latest NewsComments Off

Rewriting the Beginner’s Guide to SEO, Chapter 3: Keyword Research

Posted by BritneyMuller

Welcome to the draft of Chapter Three of the new and improved Beginner’s Guide to SEO! So far you’ve been generous and energizing with your feedback for our outline, Chapter One, and Chapter Two. We’re asking for a little more of your time as we debut the our third chapter on keyword research. Please let us know what you think in the comments!


Chapter 3: Keyword Research

Understand what your audience wants to find.

Now that you’ve learned how to show up in search results, let’s determine which strategic keywords to target in your website’s content, and how to craft that content to satisfy both users and search engines.

The power of keyword research lies in better understanding your target market and how they are searching for your content, services, or products.

Keyword research provides you with specific search data that can help you answer questions like:

  • What are people searching for?
  • How many people are searching for it?
  • In what format do they want that information?

In this chapter, you’ll get tools and strategies for uncovering that information, as well as learn tactics that’ll help you avoid keyword research foibles and build strong content. Once you uncover how your target audience is searching for your content, you begin to uncover a whole new world of strategic SEO!

What terms are people searching for?

You may know what you do, but how do people search for the product, service, or information you provide? Answering this question is a crucial first step in the keyword research process.

Discovering keywords

You likely have a few keywords in mind that you would like to rank for. These will be things like your products, services, or other topics your website addresses, and they are great seed keywords for your research, so start there! You can enter those keywords into a keyword research tool to discover average monthly search volume and similar keywords. We’ll get into search volume in greater depth in the next section, but during the discovery phase, it can help you determine which variations of your keywords are most popular amongst searchers.

Once you enter in your seed keywords into a keyword research tool, you will begin to discover other keywords, common questions, and topics for your content that you might have otherwise missed.

Let’s use the example of a florist that specializes in weddings.

Typing “wedding” and “florist” into a keyword research tool, you may discover highly relevant, highly searched for related terms such as:

  • Wedding bouquets
  • Bridal flowers
  • Wedding flower shop

In the process of discovering relevant keywords for your content, you will likely notice that the search volume of those keywords varies greatly. While you definitely want to target terms that your audience is searching for, in some cases, it may be more advantageous to target terms with lower search volume because they’re far less competitive.

Since both high- and low-competition keywords can be advantageous for your website, learning more about search volume can help you prioritize keywords and pick the ones that will give your website the biggest strategic advantage.

Pro tip: Diversify!

It’s important to note that entire websites don’t rank for keywords, pages do. With big brands, we often see the homepage ranking for many keywords, but for most websites, this isn’t usually the case. Many websites receive more organic traffic to pages other than the homepage, which is why it’s so important to diversify your website’s pages by optimizing each for uniquely valuable keywords.

How often are those terms searched?

Uncovering search volume

The higher the search volume for a given keyword or keyword phrase, the more work is typically required to achieve higher rankings. This is often referred to as keyword difficulty and occasionally incorporates SERP features; for example, if many SERP features (like featured snippets, knowledge graph, carousels, etc) are clogging up a keyword’s result page, difficulty will increase. Big brands often take up the top 10 results for high-volume keywords, so if you’re just starting out on the web and going after the same keywords, the uphill battle for ranking can take years of effort.

Typically, the higher the search volume, the greater the competition and effort required to achieve organic ranking success. Go too low, though, and you risk not drawing any searchers to your site. In many cases, it may be most advantageous to target highly specific, lower competition search terms. In SEO, we call those long-tail keywords.

Understanding the long tail

It would be great to rank #1 for the keyword “shoes”… or would it?

It’s wonderful to deal with keywords that have 50,000 searches a month, or even 5,000 searches a month, but in reality, these popular search terms only make up a fraction of all searches performed on the web. In fact, keywords with very high search volumes may even indicate ambiguous intent, which, if you target these terms, it could put you at risk for drawing visitors to your site whose goals don’t match the content your page provides.

Does the searcher want to know the nutritional value of pizza? Order a pizza? Find a restaurant to take their family? Google doesn’t know, so they offer these features to help you refine. Targeting “pizza” means that you’re likely casting too wide a net.

The remaining 75% lie in the “chunky middle” and “long tail” of search.

Don’t underestimate these less popular keywords. Long tail keywords with lower search volume often convert better, because searchers are more specific and intentional in their searches. For example, a person searching for “shoes” is probably just browsing. Whereas, someone searching for “best price red womens size 7 running shoe,” practically has their wallet out!

Pro tip: Questions are SEO gold!

Discovering what questions people are asking in your space, and adding those questions and their answers to an FAQ page, can yield incredible organic traffic for your website.

Getting strategic with search volume

Now that you’ve discovered relevant search terms for your site and their corresponding search volumes, you can get even more strategic by looking at your competitors and figuring out how searches might differ by season or location.

Keywords by competitor

You’ll likely compile a lot of keywords. How do you know which to tackle first? It could be a good idea to prioritize high-volume keywords that your competitors are not currently ranking for. On the flip side, you could also see which keywords from your list your competitors are already ranking for and prioritize those. The former is great when you want to take advantage of your competitors’ missed opportunities, while the latter is an aggressive strategy that sets you up to compete for keywords your competitors are already performing well for.

Keywords by season

Knowing about seasonal trends can be advantageous in setting a content strategy. For example, if you know that “christmas box” starts to spike in October through December in the United Kingdom, you can prepare content months in advance and give it a big push around those months.

Keywords by region

You can more strategically target a specific location by narrowing down your keyword research to specific towns, counties, or states in the Google Keyword Planner, or evaluate “interest by subregion” in Google Trends. Geo-specific research can help make your content more relevant to your target audience. For example, you might find out that in Texas, the preferred term for a large truck is “big rig,” while in New York, “tractor trailer” is the preferred terminology.

Which format best suits the searcher’s intent?

In Chapter 2, we learned about SERP features. That background is going to help us understand how searchers want to consume information for a particular keyword. The format in which Google chooses to display search results depends on intent, and every query has a unique one. While there are thousands of of possible search types, there are five major categories to be aware of:

1. Informational queries: The searcher needs information, such as the name of a band or the height of the Empire State Building.

2. Navigational queries: The searcher wants to go to a particular place on the Internet, such as Facebook or the homepage of the NFL.

3. Transactional queries: The searcher wants to do something, such as buy a plane ticket or listen to a song.

4. Commercial investigation: The searcher wants to compare products and find the best one for their specific needs.

5. Local queries: The searcher wants to find something locally, such as a nearby coffee shop, doctor, or music venue.

An important step in the keyword research process is surveying the SERP landscape for the keyword you want to target in order to get a better gauge of searcher intent. If you want to know what type of content your target audience wants, look to the SERPs!

Google has closely evaluated the behavior of trillions of searches in an attempt to provide the most desired content for each specific keyword search.

Take the search “dresses,” for example:

By the shopping carousel, you can infer that Google has determined many people who search for “dresses” want to shop for dresses online.

There is also a Local Pack feature for this keyword, indicating Google’s desire to help searchers who may be looking for local dress retailers.

If the query is ambiguous, Google will also sometimes include the “refine by” feature to help searchers specify what they’re looking for further. By doing so, the search engine can provide results that better help the searcher accomplish their task.

Google has a wide array of result types it can serve up depending on the query, so if you’re going to target a keyword, look to the SERP to understand what type of content you need to create.

Tools for determining the value of a keyword

How much value would a keyword add to your website? These tools can help you answer that question, so they’d make great additions to your keyword research arsenal:

  • Moz Keyword Explorer – Our own Moz Keyword Explorer tool extracts accurate search volume data, keyword difficulty, and keyword opportunity metrics by using live clickstream data. To learn more about how we’re producing our keyword data, check out Announcing Keyword Explorer.
  • Google Keyword Planner – Google’s AdWords Keyword Planner has historically been the most common starting point for SEO keyword research. However, Keyword Planner does restrict search volume data by lumping keywords together into large search volume range buckets. To learn more, check out Google Keyword Planner’s Dirty Secrets.
  • Google Trends – Google’s keyword trend tool is great for finding seasonal keyword fluctuations. For example, “funny halloween costume ideas” will peak in the weeks before Halloween.
  • AnswerThePublic – This free tool populates commonly searched for questions around a specific keyword. Bonus! You can use this tool in tandem with another free tool, Keywords Everywhere, to prioritize ATP’s suggestions by search volume.
  • SpyFu Keyword Research Tool – Provides some really neat competitive keyword data.

Download our free keyword research template!

Keyword research can yield a ton of data. Stay organized by downloading our free keyword research template. You can customize the template to fit your unique needs (ex: remove the “Seasonal Trends” column), sort keywords by volume, and categorize by Priority Score. Happy keyword researching!

Now that you know how to uncover what your target audience is searching for and how often, it’s time to move onto the next step: crafting pages in a way that users will love and search engines can understand.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!


Moz Blog

More Articles

Posted in Latest NewsComments Off

Advert