Tag Archive | "mobilefirst"

How Mobile-First Indexing Disrupts the Link Graph

Posted by rjonesx.

It’s happened to all of us. You bring up a webpage on your mobile device, only to find out that a feature you were accustomed to using on desktop simply isn’t available on mobile. While frustrating, it has always been a struggle for web developers and designers alike to simplify and condense their site on mobile screens without needing to strip features or content that would otherwise clutter a smaller viewport. The worst-case scenario for these trade-offs is that some features would be reserved for desktop environments, or perhaps a user might be able to opt out of the mobile view. Below is an example of how my personal blog displays the mobile version using a popular plugin by ElegantThemes called HandHeld. As you can see, the vast page is heavily stripped down and is far easier to read… but at what cost? And at what cost to the link graph?

My personal blog drops 75 of the 87 links, and all of the external links, when the mobile version is accessed. So what happens when the mobile versions of sites become the primary way the web is accessed, at scale, by the bots which power major search engines?

Google’s announcement to proceed with a mobile-first index raises new questions about how the link structure of the web as a whole might be influenced once these truncated web experiences become the first (and sometimes only) version of the web Googlebot encounters.

So, what’s the big deal?

The concern, which no doubt Google engineers have studied internally, is that mobile websites often remove content and links in order to improve user experience on a smaller screen. This abbreviated content fundamentally alters the link structure which underlies one of the most important factors in Google’s rankings. Our goal is to try and understand the impact this might have.

Before we get started, one giant unknown variable which I want to be quick to point out is we don’t know what percentage of the web Google will crawl with both its desktop and mobile bots. Perhaps Google will choose to be “mobile-first” only on sites that have historically displayed an identical codebase to both the mobile and desktop versions of Googlebot. However, for the purposes of this study, I want to show the worst-case scenario, as if Google chose not only to go “mobile-first,” but in fact to go “mobile-only.”

Methodology: Comparing mobile to desktop at scale

For this brief research, I decided to grab 20,000 random websites from the Quantcast Top Million. I would then crawl two levels deep, spoofing both the Google mobile and Google desktop versions of Googlebot. With this data, we can begin to compare how different the link structure of the web might look.

Homepage metrics

Let’s start with some descriptive statistics of the home pages of these 20,000 randomly selected sites. Of the sites analyzed, 87.42% had the same number of links on their homepage regardless of whether the bot was mobile- or desktop-oriented. Of the remaining 12.58%, 9% had fewer links and 3.58% had more. This doesn’t seem too disparate at first glance.

Perhaps more importantly, only 79.87% had identical links on the homepage when visited by desktop and mobile bots. Just because the same number of links were found didn’t mean they were actually the same links. This is important to take into consideration because links are the pathways which bots use to find content on the web. Different paths mean a different index.

Among the homepage links, we found a 7.4% drop in external links. This could mean a radical shift in some of the most important links on the web, given that homepage links often carry a great deal of link equity. Interestingly, the biggest “losers” as a percentage tended to be social sites. In retrospect, it seems reasonable that one of the common types of links a website might remove from their mobile version would be social share buttons because they’re often incorporated into the “chrome” of a page rather than the content, and the “chrome” often changes to accommodate a mobile version.

The biggest losers as a percentage in order were:

  1. linkedin.com
  2. instagram.com
  3. twitter.com
  4. facebook.com

So what’s the big deal about 5–15% differences in links when crawling the web? Well, it turns out that these numbers tend to be biased towards sites with lots of links that don’t have a mobile version. However, most of those links are main navigation links. When you crawl deeper, you just find the same links. But those that do deviate end up having radically different second-level crawl links.

Second-level metrics

Now this is where the data gets interesting. As we continue to crawl out on the web using crawl sets that are influenced by the links discovered by a mobile bot versus a desktop bot, we’ll continue to get more and more divergent results. But how far will they diverge? Let’s start with size. While we crawled an identical number of home pages, the second-tier results diverged based on the number of links found on those original home pages. Thus, the mobile crawlset was 977,840 unique URLs, while the desktop crawlset was 1,053,785. Already we can see a different index taking shape — the desktop index would be much larger. Let’s dig deeper.

I want you to take a moment and really focus on this graph. Notice there are three categories:

  • Mobile Unique: Blue bars represent unique items found by the mobile bot
  • Desktop Unique: Orange bars represent unique items found by the desktop bot
  • Shared: Gray bars represent items found by both

Notice also that there are there are four tests:

  • Number of URLs discovered
  • Number of Domains discovered
  • Number of Links discovered
  • Number of Root Linking Domains discovered

Now here is the key point, and it’s really big. There are more URLs, Domains, Links, and Root Linking Domains unique to the desktop crawl result than there are shared between the desktop and mobile crawler. The orange bar is always taller than the gray. This means that by just the second level of the crawl, the majority of link relationships, pages, and domains are different in the indexes. This is huge. This is a fundamental shift in the link graph as we have come to know it.

And now for the big question, what we all care about the most — external links.

A whopping 63% of external links are unique to the desktop crawler. In a mobile-only crawling world, the total number of external links was halved.

What is happening at the micro level?

So, what’s really causing this huge disparity in the crawl? Well, we know it has something to do with a few common shortcuts to making a site “mobile-friendly,” which include:

  1. Subdomain versions of the content that have fewer links or features
  2. The removal of links and features by user-agent detecting plugins

Of course, these changes might make the experience better for your users, but it does create a different experience for bots. Let’s take a closer look at one site to see how this plays out.

This site has ~10,000 pages according to Google and has a Domain Authority of 72 and 22,670 referring domains according to the new Moz Link Explorer. However, the site uses a popular WordPress plugin that abbreviates the content down to just the articles and pages on the site, removing links from descriptions in the articles on the category pages and removing most if not all extraneous links from the sidebar and footer. This particular plugin is used on over 200,000 websites. So, what happens when we fire up a six-level-deep crawl with Screaming Frog? (It’s great for this kind of analysis because we can easily change the user-agent and restrict settings to just crawl HTML content.)

The difference is shocking. First, notice that in the mobile crawl on the left, there is clearly a low number of links per page and that number of links is very steady as you crawl deeper through the site. This is what produces such a steady, exponential growth curve. Second, notice that the crawl abruptly ended at level four. The site just didn’t have any more pages to offer the mobile crawler! Only ~3,000 of the ~10,000 pages Google reports were found.

Now, compare this to the desktop crawler. It explodes in pages at level two, collecting nearly double the total pages of the mobile crawl at this level alone. Now, recall the graph before showing that there were more unique desktop pages than there were shared pages when we crawled 20,000 sites. Here is confirmation of exactly how it happens. Ultimately, 6x the content was made available to the desktop crawler in the same level of crawl depth.

But what impact did this have on external links?

Wow. 75% of the external, outbound links were culled in the mobile version. 4,905 external links were found in the desktop version while only 1,162 were found in the mobile. Remember, this is a DA 72 site with over twenty thousand referring domains. Imagine losing that link because the mobile index no longer finds the backlink. What should we do? Is the sky falling?

Take a deep breath

Mobile-first isn’t mobile-only

The first important caveat to all this research is that Google isn’t giving up on the desktop — they’re simply prioritizing the mobile crawl. This makes sense, as the majority of search traffic is now mobile. If Google wants to make sure quality mobile content is served, they need to shift their crawl priorities. But they also have a competing desire to find content, and doing so requires using a desktop crawler so long as webmasters continue to abbreviate the mobile versions of their sites.

This reality isn’t lost on Google. In the Original Official Google Mobile First Announcement, they write…

If you are building a mobile version of your site, keep in mind that a functional desktop-oriented site can be better than a broken or incomplete mobile version of the site.

Google took the time to state that a desktop version can be better than an “incomplete mobile version.” I don’t intend to read too much into this statement other than to say that Google wants a full mobile version, not just a postcard.

Good link placements will prevail

One anecdotal outcome of my research was that the external links which tended to survive the cull of a mobile version were often placed directly in the content. External links in sidebars like blog-rolls were essentially annihilated from the index, but in-content links survived. This may be a signal Google picks up on. External links that are both in mobile and desktop tend to be the kinds of links people might click on.

So, while there may be fewer links powering the link graph (or at least there might be a subset that is specially identified), if your links are good, content-based links, then you have a chance to see improved performance.

I was able to confirm this by looking at a subset of known good links. Using Fresh Web Explorer, I looked up fresh links to toysrus.com which is currently gaining a great deal of attention due to stores closing. We can feel confident that most of these links will be in-content because the articles themselves are about the relevant, breaking news regarding Toys R Us. Sure enough, after testing 300+ mentions, we found the links to be identical in the mobile and desktop crawls. These were good, in-content links and, subsequently, they showed up in both versions of the crawl.

Selection bias and convergence

It is probably the case that popular sites are more likely to have a mobile version than non-popular sites. Now, they might be responsive — at which point they would yield no real differences in the crawl — but at least some percentage would likely be m.* domains or utilize plugins like those mentioned above which truncate the content. At the lower rungs of the web, older, less professional content is likely to have only one version which is shown to mobile and desktop devices alike. If this is the case, we can expect that over time the differences in the index might begin to converge rather than diverge, as my study looked only at sites that were in the top million and only crawled two levels deep.

Moreover (this one is a bit speculative), but I think over time that there will be convergence between a mobile and desktop index. I don’t think the link graphs will grow exponentially different as the linked web is only so big. Rather, the paths to which certain pages are reached, and the frequency with which they are reached, will change quite a bit. So, while the link graph will differ, the set of URLs making up the link graph will largely be the same. Of course, some percentage of the mobile web will remain wholly disparate. The large number of sites that use dedicated mobile subdomains or plugins that remove substantial sections of content will remain like mobile islands in the linked web.

Impact on SERPs

It’s difficult at this point to say what the impact on search results will be. It will certainly not leave the SERPs unchanged. What would be the point of Google making and announcing a change to its indexing methods if it didn’t improve the SERPs?

That being said, this study wouldn’t be complete without some form of impact assessment. Hat tip to JR Oakes for giving me this critique, otherwise I would have forgotten to take a look.

First, there are a couple of things which could mitigate dramatic shifts in the SERPs already, regardless of the veracity of this study:

  • A slow rollout means that shifts in SERPs will be lost to the natural ranking fluctuations we already see.
  • Google can seed URLs found by mobile or by desktop into their respective crawlers, thereby limiting index divergence. (This is a big one!)
  • Google could choose to consider, for link purposes, the aggregate of both mobile and desktop crawls, not counting one to the exclusion of the other.

Second, the relationships between domains may be less affected than other index metrics. What is the likelihood that the relationship between Domain X and Domain Y (more or less links) is the same for both the mobile- and desktop-based indexes? If the relationships tend to remain the same, then the impact on SERPs will be limited. We will call this relationship being “directionally consistent.”

To accomplish this part of the study, I took a sample of domain pairs from the mobile index and compared their relationship (more or less links) to their performance in the desktop index. Did the first have more links than the second in both the mobile and desktop? Or did they perform differently?

It turns out that the indexes were fairly close in terms of directional consistency. That is to say that while the link graphs as a whole were quite different, when you compared one domain to another at random, they tended in both data sets to be directionally consistent. Approximately 88% of the domains compared maintained directional consistency via the indexes. This test was only run comparing the mobile index domains to the desktop index domains. Future research might explore the reverse relationship.

So what’s next?: Moz and the mobile-first index

Our goal for the Moz link index has always been to be as much like Google as possible. It is with that in mind that our team is experimenting with a mobile-first index as well. Our new link index and Link Explorer in Beta seeks to be more than simply one of the largest link indexes on the web, but the most relevant and useful, and we believe part of that means shaping our index with methods similar to Google. We will keep you updated!

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!


Moz Blog

Related Website Articles

Posted in Latest NewsComments Off

How Does Mobile-First Indexing Work, and How Does It Impact SEO?

Posted by bridget.randolph

We’ve been hearing a lot about mobile-first indexing lately, as the latest development in Google’s ever-continuing efforts to make the web more mobile-friendly and reflect user behavior trends.

But there’s also a lot of confusion around what this means for the average business owner. Do you have to change anything? Everything? If your site is mobile-friendly, will that be good enough?

IS THIS GOING TO BE ANOTHER MOBILEGEDDON?!!

In this post I’ll go over the basics of what “mobile-first indexing” means, and what you may need to do about it. I’ll also answer some frequently asked questions about mobile-first indexing and what it means for our SEO efforts.

What is “mobile-first indexing”?

Mobile-first indexing is exactly what it sounds like. It just means that the mobile version of your website becomes the starting point for what Google includes in their index, and the baseline for how they determine rankings. If you monitor crawlbot traffic to your site, you may see an increase in traffic from Smartphone Googlebot, and the cached versions of pages will usually be the mobile version of the page.

It’s called “mobile-first” because it’s not a mobile-only index: for instance, if a site doesn’t have a mobile-friendly version, the desktop site can still be included in the index. But the lack of a mobile-friendly experience could impact negatively on the rankings of that site, and a site with a better mobile experience would potentially receive a rankings boost even for searchers on a desktop.

You may also want to think of the phrase “mobile-first” as a reference to the fact that the mobile version will be considered the primary version of your website. So if your mobile and desktop versions are equivalent — for instance if you’ve optimized your content for mobile, and/or if you use responsive design — this change should (in theory) not have any significant impact in terms of your site’s performance in search results.

However it does represent a fundamental reversal in the way Google is thinking about your website content and how to prioritize crawling and indexation. Remember that up until now the desktop site was considered the primary version (similar to a canonical URL) and the mobile site was treated as an “alternate” version for a particular use case. This is why Google encouraged webmasters with a separate mobile site (m.domain.com) to implement switchboard tags (which indicated the existence of a mobile URL version with a special rel=alternate tag). Google might not even make the effort to crawl and cache the mobile versions of all of these pages, as they could simply display that mobile URL to mobile searchers.

This view of the desktop version as the primary one often meant in practice that the desktop site would be prioritized by SEOs and marketing teams and was treated as the most comprehensive version of a website, with full content, structured data markup, hreflang (international tags), the majority of backlinks, etc.; while the mobile version might have lighter content, and/or not include the same level of markup and structure, and almost certainly would not receive the bulk of backlinks and external attention.

What should I do about mobile-first indexing?

The first thing to know is that there’s no need to panic. So far this change is only in the very earliest stages of testing, and is being rolled out very gradually only to websites which Google considers to be “ready” enough for this change to have a minimal impact.

According to Google’s own latest guidance on the topic, if your website is responsive or otherwise identical in its desktop and mobile versions, you may not have to do anything differently (assuming you’re happy with your current rankings!).

That said, even with a totally responsive site, you’ll want to ensure that mobile page speed and load time are prioritized and that images and other (potentially) dynamic elements are optimized correctly for the mobile experience. Note that with mobile-first indexing, content which is collapsed or hidden in tabs, etc. due to space limitations will not be treated differently than visible content (as it may have been previously), since this type of screen real estate management is actually a mobile best practice.

If you have a separate mobile site, you’ll want to check the following:

  • Content: make sure your mobile version has all the high-quality, valuable content that exists on your desktop site. This could include text, videos and images. Make sure the formats used on the mobile version are crawlable and indexable (including alt-attributes for images).
  • Structured data: you should include the same structured data markup on both the mobile and desktop versions of the site. URLs shown within structured data on mobile pages should be the mobile version of the URL. Avoid adding unnecessary structured data if it isn’t relevant to the specific content of a page.
  • Metadata: ensure that titles and meta descriptions are equivalent on both versions of all pages.
    • Note that the official guidance says “equivalent” rather than “identical” – you may still want to optimize your mobile titles for shorter character counts, but make sure the same information and relevant keywords are included.
  • Hreflang: if you use rel=hreflang for internationalization, your mobile URLs’ hreflang annotations should point to the mobile version of your country or language variants, and desktop URLs should point to the desktop versions.
  • Social metadata: OpenGraph tags, Twitter cards and other social metadata should be included on the mobile version as well as the desktop version.
  • XML and media sitemaps: ensure that any links to sitemaps are accessible from the mobile version of the site. This also applies to robots directives (robots.txt and on-page meta-robots tags) and potentially even trust signals, like links to your privacy policy page.
  • Search Console verification: if you have only verified your desktop site in Google Search Console, make sure you also add and verify the mobile version.
  • App indexation: if you have app indexation set up for your desktop site, you may want to ensure that you have verified the mobile version of the site in relation to app association files, etc.
  • Server capacity: Make sure that your host servers can handle increased crawl rate.
    • (This only applies for sites with their mobile version on a separate host, such as m.domain.com.)
  • Switchboard tags: if you currently have mobile switchboard tags implemented, you do not need to change this implementation. These should remain as they are.

Common questions about mobile-first indexing

Is mobile-first indexing adding mobile pages to a separate mobile index?

With mobile-first indexing, there is only one index (the same one Google uses now). The change to mobile-first indexing does not generate a new “mobile-first” index, nor is it creating a separate “mobile index” with a “desktop index” remaining active. Instead, it simply changes how content is added to the existing index.

Is the mobile-first index live and affecting my site now? If not, when does it go live?

Google has been experimenting with this approach to indexing on a small number of sites, which were selected based on perceived “readiness”. A wider rollout is likely going to take a long time and in June 2017, Gary Illyes stated that it will probably take a few years before “we reach an index that is only mobile-first.”

Google has also stated the following on the Webmasters Blog, in a blog post dated Dec 18 2017:

“We will be evaluating sites independently on their readiness for mobile-first indexing based on the above criteria and transitioning them when ready. This process has already started for a handful of sites and is closely being monitored by the search team.

“We continue to be cautious with rolling out mobile-first indexing. We believe taking this slowly will help webmasters get their sites ready for mobile users, and because of that, we currently don’t have a timeline for when it’s going to be completed.”

Will Google only use my mobile site to determine my rankings?

Mobile-first means that the mobile version will be considered the primary version when it comes to how rankings are determined. However, there may be circumstances where the desktop version could be taken into consideration (for instance, if you don’t have a mobile version of a page).

That being said, you will potentially still see varying ranking results between mobile search results and desktop search results, so you’ll still want to track both. (In the same way that now, Google primarily uses the desktop site to determine rankings but you still want to track mobile rankings as these vary from desktop rankings based on user behavior and other factors).

When might Google use the desktop site to determine rankings vs. the mobile site?

The primary use case I’ve seen referred to so far is that they will use the desktop site to determine rankings when there is no mobile version.

It is possible that for websites where the desktop version has additional ranking information (such as backlinks), that information could also be taken into consideration – but there is no guarantee that they will crawl or index the desktop version once they’ve seen the mobile version, and I haven’t seen any official statements that this would be the case.

Therefore one of the official recommendations is that once the mobile-first indexing rollout happens, if you’re in the process of building your mobile site or have a “placeholder” type mobile version currently live it would actually be better to have no mobile site than a broken or incomplete one. In this case, you should wait to launch your mobile site until it is fully ready.

What if I don’t have a mobile version of my site?

If you don’t have a mobile version of your site and your desktop version is not mobile-friendly, your content can still be indexed; however you may not rank as well in comparison to mobile-friendly websites. This may even negatively impact your overall rankings on desktop search as well as mobile search results because it will be perceived as having a poorer user experience than other sites (since the crawler will be a “mobile” crawler).

What could happen to sites with a large desktop site and a small mobile site? Will content on your desktop site that does not appear on the mobile version be indexed and appear for desktop searches?

The end goal for this rollout is that the index will be based predominantly on crawling mobile content. If you have a heavily indexed desktop version, they’re not going to suddenly purge your desktop content from the existing index and start fresh with just your thin mobile site indexed; but the more you can ensure that your mobile version contains all relevant and valuable content, the more likely it is to continue to rank well, particularly as they cut back on crawling desktop versions of websites.

How does this change ranking factors and strategy going forward?

This may impact a variety of ranking factors and strategy in the future; Cindy Krum at Mobile Moxie has written two excellent articles on what could be coming in the future around this topic.

Cindy talks about the idea that mobile-first indexing may be “an indication that Google is becoming less dependent on traditional links and HTML URLS for ranking.” It seems that Google is moving away from needing to rely so much on a “URL” system of organizing content, in favor of a more API type approach based on “entities” (thanks, structured data!) rather than URL style links. Check out Cindy’s posts for more explanation of how this could impact the future of search and SEO.

Is there a difference between how responsive sites and separate mobile sites will be treated?

Yes and no. The main difference will be in terms of how much work you have to do to get ready for this change.

If you have a fully responsive site, you should already have everything present on your mobile version that is currently part of the desktop version, and your main challenge will simply be to ensure that the mobile experience is well optimized from a user perspective (e.g. page speed, load time, navigation, etc).

With a separate mobile site, you’ll need to make sure that your mobile version contains everything that your desktop site does, which could be a lot of work depending on your mobile strategy so far.

Will this change how I should serve ads/content/etc. on my mobile site?

If your current approach to ads is creating a slow or otherwise poor user experience you will certainly need to address that.

If you currently opt to hide some of your mobile site content in accordions or tabs to save space, this is actually not an issue as this content will be treated in the same way as if it was loaded fully visible (as long as the content is still crawlable/accessible).

Does this change how I use rel=canonical/switchboard tags?

No. For now, Google has stated that if you have already implemented switchboard tags, you should leave them as they are.


Has this overview helped you to feel more prepared for the shift to mobile-first indexing? Are there any questions you still have?

I’d love to hear what you’re thinking about in the comments!

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!


Moz Blog

Posted in Latest NewsComments Off

SMX Advanced session: Mobile-First For The Advanced SEO

How can websites prepare for the mobile-first index? Columnist Greg Gifford recaps a session from SMX Advanced dealing with the impending rollout of Google’s new index, which prioritizes mobile content over desktop.

The post SMX Advanced session: Mobile-First For The Advanced SEO appeared first on…



Please visit Search Engine Land for the full article.


Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Posted in Latest NewsComments Off

Google’s mobile-first index likely not coming until 2018 at earliest

At SMX Advanced, Gary Illyes says Google will “communicate a lot” before rolling out the mobile-first index.

The post Google’s mobile-first index likely not coming until 2018 at earliest appeared first on Search Engine Land.



Please visit Search Engine Land for the full article.


Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Posted in Latest NewsComments Off

SearchCap: Google Fred confirmation, mobile-first index status & future of search

Below is what happened in search today, as reported on Search Engine Land and from other places across the web.

The post SearchCap: Google Fred confirmation, mobile-first index status & future of search appeared first on Search Engine Land.



Please visit Search Engine Land for the full article.


Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Find More Articles

Posted in Latest NewsComments Off

SearchCap: Google link command, mobile-first index & SEO problems

Below is what happened in search today, as reported on Search Engine Land and from other places across the web.

The post SearchCap: Google link command, mobile-first index & SEO problems appeared first on Search Engine Land.



Please visit Search Engine Land for the full article.


Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Posted in Latest NewsComments Off

Finally going mobile-friendly, but on a collision course with Google’s mobile-first index [Case Study]

Google’s index has been shifting toward ‘mobile-first,’ making mobile pages the default pages for ranking purposes. Columnist Glenn Gabe uses a case study to illustrate some of the issues this might pose for site owners.

The post Finally going mobile-friendly, but on a collision course with…



Please visit Search Engine Land for the full article.


Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Posted in Latest NewsComments Off

Google begins mobile-first indexing, using mobile content for all search rankings

While called an ‘experiment,’ it’s actually the first move in Google’s planned shift to looking primarily at mobile content, rather than desktop, when deciding how to rank results.

The post Google begins mobile-first indexing, using mobile content for all search rankings appeared first on Search…



Please visit Search Engine Land for the full article.


Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing

Posted in Latest NewsComments Off


Advert